> ## Documentation Index > Fetch the complete documentation index at: https://docs.gp.scale.com/llms.txt > Use this file to discover all available pages before exploring further. # Create Chat Completion > ### Description Given a list of messages representing a conversation history, runs LLM inference to produce the next message. ### Details Like [completions](https://scale-egp.readme.io/docs/completions-1), [chat completions]( https://scale-egp.readme.io/docs/chat-completions-intro) involve an LLM's response to input. However, chat completions take a conversation history as input, instead of a single prompt, which enables the LLM to create responses that take past context into account. ### Messages The primary input to the LLM is a list of messages represented by the `messages` array, which forms the conversation. The `messages` array must contain at least one `message` object. Each `message` object is attributed to a specific entity through its `role`. The available roles are: - `user`: Represents the human querying the model. - `assistant`: Represents the model responding to user. - `system`: Represents a non-user entity that provides information to guide the behavior of the assistant. When the `role` of a `message` is set to `user`, `assistant`, or `system`, the `message` must also contain a `content` field which is a string representing the actual text of the message itself. Semantically, when the `role` is `user`, `content` contains the user's query. When the `role` is `assistant`, `content` is the model's response to the user. When the `role` is `system`, `content` represents the instruction for the assistant. ### Instructions You may provide instructions to the assistant by supplying by supplying `instructions` in the HTTP request body or by specifying a `message` with `role` set to `system` in the `messages` array. By convention, the system message should be the first message in the array. Do **not** specify both an instruction and a system message in the `messages` array. ## OpenAPI ````yaml https://app.stainlessapi.com/api/spec/documented/sgp/openapi.yml post /v4/chat-completions openapi: 3.1.0 info: title: EGP API V4 description: >- This is the parent API for all EGP APIs. If you are looking for the EGP API, please go to https://api.egp.scale.com/docs. contact: name: Scale Generative AI Platform url: https://scale.com/genai-platform version: 0.1.0 servers: - url: https://api.egp.scale.com security: [] tags: - name: Models description: Model API. paths: /v4/chat-completions: post: tags: - Chat Completions summary: Create Chat Completion description: >- ### Description Given a list of messages representing a conversation history, runs LLM inference to produce the next message. ### Details Like [completions](https://scale-egp.readme.io/docs/completions-1), [chat completions]( https://scale-egp.readme.io/docs/chat-completions-intro) involve an LLM's response to input. However, chat completions take a conversation history as input, instead of a single prompt, which enables the LLM to create responses that take past context into account. ### Messages The primary input to the LLM is a list of messages represented by the `messages` array, which forms the conversation. The `messages` array must contain at least one `message` object. Each `message` object is attributed to a specific entity through its `role`. The available roles are: - `user`: Represents the human querying the model. - `assistant`: Represents the model responding to user. - `system`: Represents a non-user entity that provides information to guide the behavior of the assistant. When the `role` of a `message` is set to `user`, `assistant`, or `system`, the `message` must also contain a `content` field which is a string representing the actual text of the message itself. Semantically, when the `role` is `user`, `content` contains the user's query. When the `role` is `assistant`, `content` is the model's response to the user. When the `role` is `system`, `content` represents the instruction for the assistant. ### Instructions You may provide instructions to the assistant by supplying by supplying `instructions` in the HTTP request body or by specifying a `message` with `role` set to `system` in the `messages` array. By convention, the system message should be the first message in the array. Do **not** specify both an instruction and a system message in the `messages` array. operationId: POST-V4-/v2/chat-completions requestBody: content: application/json: schema: $ref: '#/components/schemas/CreateChatCompletionRequest' required: true responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/CreateChatCompletionResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-codeSamples: - lang: Python source: |- import os from scale_gp import SGPClient client = SGPClient( api_key=os.environ.get("SGP_API_KEY"), # This is the default and can be omitted ) for chat_completion in client.chat_completions.create( messages=[{ "content": "string", "role": "user", }], model="gpt-oss-120b", ): print(chat_completion) - lang: Go source: "package main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\n\t\"github.com/stainless-sdks/sgp-go\"\n\t\"github.com/stainless-sdks/sgp-go/option\"\n\t\"github.com/stainless-sdks/sgp-go/shared\"\n)\n\nfunc main() {\n\tclient := sgp.NewClient(\n\t\toption.WithAPIKey(\"My API Key\"),\n\t)\n\tchatCompletionsResponse, err := client.ChatCompletions.New(context.TODO(), sgp.ChatCompletionNewParams{\n\t\tMessages: sgp.F([]sgp.ChatCompletionNewParamsMessageUnion{sgp.ChatCompletionNewParamsMessagesUserMessage{\n\t\t\tContent: sgp.F[sgp.ChatCompletionNewParamsMessagesUserMessageContentUnion](shared.UnionString(\"string\")),\n\t\t\tRole: sgp.F(sgp.ChatCompletionNewParamsMessagesUserMessageRoleUser),\n\t\t}}),\n\t\tModel: sgp.F(sgp.ChatCompletionNewParamsModelGptOss120b),\n\t})\n\tif err != nil {\n\t\tpanic(err.Error())\n\t}\n\tfmt.Printf(\"%+v\\n\", chatCompletionsResponse.ChatCompletion)\n}\n" components: schemas: CreateChatCompletionRequest: properties: account_id: title: Account Id description: >- The account ID to use for usage tracking. This will be gradually enforced. type: string model: type: string enum: - gpt-oss-120b - gpt-oss-20b - o1 - o1-mini - o3-mini - gpt-4 - gpt-4-0613 - gpt-4-32k - gpt-4-32k-0613 - gpt-4o - gpt-4o-mini - gpt-4o-2024-08-06 - gpt-3.5-turbo - gpt-3.5-turbo-0613 - gpt-3.5-turbo-16k - gpt-3.5-turbo-16k-0613 - gemini-pro - gemini-1.5-pro-001 - gemini-1.5-pro-002 - gemini-1.5-pro-preview-0409 - gemini-1.5-pro-preview-0514 - llama-2-7b-chat - llama-2-13b-chat - llama-2-70b-chat - llama-3-8b-instruct - llama-3-70b-instruct - llama-3-1-8b-instruct - llama-3-1-70b-instruct - llama-3-2-1b-instruct - llama-3-2-3b-instruct - llama-3-3-70b-instruct - Meta-Llama-3-8B-Instruct-RMU - Meta-Llama-3-8B-Instruct-RR - Meta-Llama-3-8B-Instruct-DERTA - Meta-Llama-3-8B-Instruct-LAT - mixtral-8x7b-instruct - mixtral-8x22b-instruct - claude-3-opus-20240229 - claude-3-sonnet-20240229 - claude-3-haiku-20240307 - claude-3-5-sonnet-20240620 - claude-3-5-sonnet-20241022 - mistral-large-latest - phi-3-mini-4k-instruct - phi-3-cat-merged - zephyr-cat-merged - dolphin-2.9-llama3-8b - dolphin-2.9-llama3-70b - defense-llama-3-8b-instruct - donovan-combat-llama - llama3-1-405b-instruct-v1 title: Model description: >- The ID of the model to use for chat completions. We only support the models listed here so far. memory_strategy: $ref: '#/components/schemas/MemoryStrategy' description: >- The memory strategy to use for the agent. A memory strategy is a way to prevent the underlying LLM's context limit from being exceeded. Each memory strategy uses a different technique to condense the input message list into a smaller payload for the underlying LLM. We only support the Last K memory strategy right now, but will be adding new strategies soon. messages: items: $ref: '#/components/schemas/ChatMessage' type: array title: Messages description: >- The list of messages in the conversation. Expand each message type to see how it works and when to use it. Most conversations should begin with a single `user` message. model_parameters: $ref: '#/components/schemas/ModelParameters' description: >- Configuration parameters for the chat completion model, such as temperature, max_tokens, and stop_sequences. If not specified, the default value are: - temperature: 0.2 - max_tokens: None (limited by the model's max tokens) - stop_sequences: None default: temperature: 0.2 instructions: title: Instructions description: >- The initial instructions to provide to the chat completion model. Use this to guide the model to act in more specific ways. For example, if you have specific rules you want to restrict the model to follow you can specify them here. Good prompt engineering is crucial to getting performant results from the model. If you are having trouble getting the model to perform well, try writing more specific instructions here before trying more expensive techniques such as swapping in other models or finetuning the underlying LLM. default: >- You are an AI assistant that helps users with their questions by chatting back and forth with them. When asked a question, you should answer it as best as you can with the information you have. If you need more information, you can ask the user for it. type: string chat_template: title: Chat Template description: >- Currently only supported for LLM-Engine models. A Jinja template string that defines how the chat completion API formats the string prompt. For Llama models, the template must take in at most a `messages` object, `bos_token` string, and `eos_token` string. The `messages` object is a list of dictionaries, each with keys `role` and `content`. For Mixtral models, the template must take in at most a `messages` object and `eos_token` string. The `messages` object looks identical to the Llama model's `messages` object, but the template can assume the `role` key takes on the values `user` or `assistant`, or `system` for the first message. The chat template either needs to handle this system message (which gets set via the `instructions` field or by the messages), or the `instructions` field must be set to `null` and the `messages` object must not contain any system messages.See the default chat template present in the Llama and Mixtral tokenizers for examples. type: string stream: type: boolean title: Stream description: |- Whether or not to stream the response. Setting this to True will stream the completion in real-time. default: false type: object required: - model - messages title: CreateChatCompletionRequest CreateChatCompletionResponse: properties: chat_completion: $ref: >- #/components/schemas/egp_api_backend__server__api__models__egp_models__ChatCompletion token_usage: $ref: '#/components/schemas/TokenUsage' type: object required: - chat_completion title: CreateChatCompletionResponse HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError MemoryStrategy: $ref: '#/components/schemas/LastKMemoryStrategy' title: MemoryStrategy ChatMessage: oneOf: - $ref: '#/components/schemas/UserMessage' - $ref: '#/components/schemas/AssistantMessage' - $ref: '#/components/schemas/SystemMessage' title: ChatMessage discriminator: propertyName: role mapping: assistant: $ref: '#/components/schemas/AssistantMessage' system: $ref: '#/components/schemas/SystemMessage' user: $ref: '#/components/schemas/UserMessage' ModelParameters: properties: temperature: title: Temperature description: >- What sampling temperature to use, between [0, 1]. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. Setting temperature=0.0 will enable fully deterministic (greedy) sampling. Temperature is ignored for OpenAI reasoning models. default: 0.2 type: number maximum: 1 minimum: 0 stop_sequences: title: Stop Sequences description: >- List of up to 4 sequences where the API will stop generating further tokens. The returned text will not contain the stop sequence. items: type: string type: array maxItems: 4 max_tokens: title: Max Tokens description: > The maximum number of tokens to generate in the completion. The token count of your prompt plus max_tokens cannot exceed the model's context length. If not, specified, max_tokens will be determined based on the model used: | Model API family | Model API default | EGP applied default | | --- | --- | --- | | OpenAI Completions | [`16`](https://platform.openai.com/docs/api-reference/completions/create#max_tokens) | `context window - prompt size` | | OpenAI Chat Completions | [`context window - prompt size`](https://platform.openai.com/docs/api-reference/chat/create#max_tokens) | `context window - prompt size` | | LLM Engine | [`max_new_tokens`](https://github.com/scaleapi/launch-python-client/blob/207adced1c88c1c2907266fa9dd1f1ff3ec0ea5b/launch/client.py#L2910) parameter is required | `100` | | Anthropic Claude 2 | [`max_tokens_to_sample`](https://docs.anthropic.com/claude/reference/complete_post) parameter is required | `10000` | type: integer topP: title: Topp description: >- The cumulative probability cutoff for token selection. Lower values mean sampling from a smaller, more top-weighted nucleus. Available for models from Anthropic, Google, Mistral, LLM Engine, and OpenAI. type: number topK: title: Topk description: >- Sample from the k most likely next tokens at each step. Lower k focuses on higher probability tokens. Available for models from Anthropic, Google, and LLM Engine. type: integer frequency_penalty: title: Frequency Penalty description: >- Penalize tokens based on how much they have already appeared in the text. Positive values encourage the model to generate new tokens and negative values encourage the model to repeat tokens. Available for models from LLM Engine, and OpenAI type: number presence_penalty: title: Presence Penalty description: >- Penalize tokens based on if they have already appeared in the text. Positive values encourage the model to generate new tokens and negative values encourage the model to repeat tokens. Available for models from LLM Engine, and OpenAI. type: number reasoning_effort: title: Reasoning Effort description: >- The effort level for the model to reason. Available for models from OpenAI. type: string enum: - low - medium - high type: object title: ModelParameters egp_api_backend__server__api__models__egp_models__ChatCompletion: properties: message: $ref: '#/components/schemas/ChatMessage' finish_reason: title: Finish Reason type: string type: object required: - message title: ChatCompletion TokenUsage: properties: prompt: title: Prompt description: Number of tokens in the prompt. type: integer completion: title: Completion description: Number of tokens in the completion. type: integer total: type: integer title: Total description: Total number of tokens in both the prompt and the completion. type: object required: - total title: TokenUsage ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type input: title: Input ctx: type: object title: Context additionalProperties: true type: object required: - loc - msg - type title: ValidationError LastKMemoryStrategy: properties: name: type: string const: last_k title: Name description: >- Name of the memory strategy. Must be `last_k`. This strategy truncates the message history to the last `k` messages. It is the simplest way to prevent the model's context limit from being exceeded. However, this strategy only allows the model to have short term memory. For longer term memory, please use one of the other strategies. default: last_k params: $ref: '#/components/schemas/LastKMemoryStrategyParams' description: Configuration parameters for the memory strategy. type: object required: - params title: Last K Memory Strategy UserMessage: properties: role: type: string const: user title: Role description: >- The role of the message. Must be set to 'user'. A user message is a message from the user to the AI. This should be the message used to send end user input to the AI. default: user content: anyOf: - type: string - items: $ref: '#/components/schemas/UserMessageContentParts' type: array title: Content description: >- Input from the user. Can either be text or a list of content parts. Not all models support image content parts, or multiple parts. type: object required: - content title: User Message AssistantMessage: properties: role: type: string const: assistant title: Role description: >- The role of the message. Must be set to 'assistant'. An assistant message is a message from the AI to the client. It is different from an agent message in that it cannot contain a tool request. It is simply a direct response from the AI to the client. default: assistant content: type: string title: Content description: Text response from the assistant type: object required: - content title: Assistant Message SystemMessage: properties: role: type: string const: system title: Role description: >- The role of the message. Must be set to 'system'. A system message is different from other messages in that it does not originate from a party engaged in a user/AI conversation. Instead, it is a message that is injected by either the application or system to guide the conversation. For example, a system message may be used as initial instructions for an AI entity or to tell the AI that it did not do something correctly. default: system content: type: string title: Content description: Text input from the system. type: object required: - content title: System Message LastKMemoryStrategyParams: properties: k: type: integer minimum: 1 title: K description: The maximum number of previous messages to remember. type: object required: - k title: LastKMemoryStrategyParams UserMessageContentParts: oneOf: - $ref: '#/components/schemas/TextUserMessageContentParts' - $ref: '#/components/schemas/ImageUrlUserMessageContentParts' - $ref: '#/components/schemas/ImageDataUserMessageContentParts' title: UserMessageContentParts discriminator: propertyName: type mapping: image_data: $ref: '#/components/schemas/ImageDataUserMessageContentParts' image_url: $ref: '#/components/schemas/ImageUrlUserMessageContentParts' text: $ref: '#/components/schemas/TextUserMessageContentParts' TextUserMessageContentParts: properties: type: type: string const: text title: Type default: text text: type: string title: Text type: object required: - text title: TextUserMessageContentParts ImageUrlUserMessageContentParts: properties: type: type: string const: image_url title: Type default: image_url image_url: $ref: '#/components/schemas/ImageUrlSubPart' description: >- Specifies the image URL and level of detail. Only supported by OpenAI models type: object required: - image_url title: ImageUrlUserMessageContentParts ImageDataUserMessageContentParts: properties: type: type: string const: image_data title: Type default: image_data image_data: $ref: '#/components/schemas/ImageDataSubPart' description: Specifies inline image data type: object required: - image_data title: ImageDataUserMessageContentParts ImageUrlSubPart: properties: url: type: string title: Url description: 'The URL of the image. Note: only OpenAI supports this.' detail: title: Detail description: >- Only used for OpenAI. Corresponds to OpenAI's image detail parameter. type: string enum: - low - high - auto type: object required: - url title: ImageUrlSubPart ImageDataSubPart: properties: type: type: string const: base64 title: Type description: The type of the image data. Only base64 is supported. default: base64 media_type: type: string title: Media Type description: >- The media/mime type of the image data. For example, 'image/png'. Check providers' documentation for supported media types. data: type: string title: Data description: The base64-encoded image data. detail: title: Detail description: >- Only used for OpenAI. Corresponds to OpenAI's image detail parameter. type: string enum: - low - high - auto type: object required: - media_type - data title: ImageDataSubPart ````