Generate chat completion
Description
Interact with the LLM model using a chat completions interface. The LLM model will respond with an assistant message.
{
"model": "gpt-4o",
"messages": [
{ "role": "system", "content": "You are a name generator. Do not generate anything else than names" },
{ "role": "user", "content": "Generate 5 names" },
]
}
Authorizations
Body
The name of the model.
Should be formatted as vendor/model_name or vendor/model_name/deployment (ie: openai/gpt-3.5-turbo).
The list of messages in the conversation.
Expand each message type to see how it works and when to use it. Most conversations should begin with a single user message.
A Jinja template to use for this conversion. As of transformers v4.44, default chat template is no longer allowed, so you must provide a chat template if the model's tokenizer does not define one and no override template is given
Additional kwargs to pass to the template renderer. Will be accessible by the chat template.
JSON schema for guided decoding. Only supported in vllm.
Regex for guided decoding. Only supported in vllm.
Choices for guided decoding. Only supported in vllm.
Context-free grammar for guided decoding. Only supported in vllm.
If specified, will override the default guided decoding backend of the server for this specific request. If set, must be either 'outlines' / 'lm-format-enforcer'
If specified, will override the default whitespace pattern for guided json decoding.
Whether to include the stop string in the output.
The memory strategy to use for the conversation. A memory strategy is a way to prevent the underlying LLM's context limit from being exceeded. Only the last_k memory strategy is supported at this time.
Whether or not to stream the response.
Setting this to True will stream the response in real-time.
Customize streaming behavior
Up to 4 sequences where the API will stop generating further tokens.
Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim.
Modify the likelihood of specified tokens appearing in the completion.
Accepts a JSON object that maps tokens (specified by their token ID in the tokenizer) to an associated bias value from -100 to 100. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model, but values between -1 and 1 should decrease or increase likelihood of selection; values like -100 or 100 should result in a ban or exclusive selection of the relevant token.
Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message.
An integer between 0 and 20 specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used.
The maximum number of tokens that can be generated.
The token count of your prompt plus max_tokens cannot exceed the model's context length.
How many completions to generate for each prompt.
Note: Because this parameter generates many completions, it can quickly consume your token quota. Use carefully and ensure that you have reasonable settings for max_tokens and stop.
Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics.
If specified, our system will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result.
The suffix that comes after a completion of inserted text.This parameter is only supported for gpt-3.5-turbo-instruct.
The sampling temperature to use for the completion. Higher values mean the model will take more risks.
An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered.
It is generally recommended to alter this or temperature but not both.
An object specifying the format that the model must output
A unique identifier representing your end-user.
Audio-related parameters for chat completion
Timeout for the API request
Maximum number of tokens to generate in the completion
List of modalities for the chat completion
text, audio Additional params to be included in the API request
Currently only supported for OpenAI and Anthropic models.A list of tools the model may call.Currently, only functions are supported as a tool.Use this to provide a list of functions the model may generate JSON inputs for.
Currently only supported for OpenAI and Anthropic models.Controls which (if any) tool is called by the model.
none, auto, required Currently only supported for OpenAI models.Enables calling tools in parallel.
Generates best_of completions server-side and returns the best (the one with the highest log probability per token). Results cannot be streamed.
Additional headers to be included in the API request
Configure a predicted output to reduce latency
Only sample from the top K options for each subsequent token
Response
Usage statistics.
A unique identifier for the completion.
The model used for completion.
The object type, ex text_completion or chat.completion
This fingerprint represents the backend configuration that the model runs with.
The Unix timestamp (in seconds) of when the chat completion was created.