> ## Documentation Index > Fetch the complete documentation index at: https://docs.gp.scale.com/llms.txt > Use this file to discover all available pages before exploring further. # Generate chat completion > ### Description Interact with the LLM model using the specified model_deployment_id. You can include a list of messages as the conversation history. The conversation can feature multiple messages from the roles user, assistant, and system. If the chosen model does not support chat completion, the API will revert to simple completion, disregarding the provided history. The endpoint manages context length exceedance optimistically: it estimates the token count from the provided history and prompt, and if it exceeds the context or approaches 80% of it, the exact token count will be calculated, and the history will be trimmed to fit the context. ```json { "prompt": "Generate 5 more", "chat_history": [ { "role": "system", "content": "You are a name generator. Do not generate anything else than names" }, { "role": "user", "content": "Generate 5 names" }, { "role": "assistant", "content": "1. Olivia Bennett\n2. Ethan Carter\n3. Sophia Ramirez\n4. Liam Thompson\n5. Ava Mitchell" } ], } ``` ## OpenAPI ````yaml https://app.stainlessapi.com/api/spec/documented/sgp/openapi.yml post /v4/models/{model_deployment_id}/chat-completions openapi: 3.1.0 info: title: EGP API V4 description: >- This is the parent API for all EGP APIs. If you are looking for the EGP API, please go to https://api.egp.scale.com/docs. contact: name: Scale Generative AI Platform url: https://scale.com/genai-platform version: 0.1.0 servers: - url: https://api.egp.scale.com security: [] tags: - name: Models description: Model API. paths: /v4/models/{model_deployment_id}/chat-completions: post: tags: - Models summary: Generate chat completion description: >- ### Description Interact with the LLM model using the specified model_deployment_id. You can include a list of messages as the conversation history. The conversation can feature multiple messages from the roles user, assistant, and system. If the chosen model does not support chat completion, the API will revert to simple completion, disregarding the provided history. The endpoint manages context length exceedance optimistically: it estimates the token count from the provided history and prompt, and if it exceeds the context or approaches 80% of it, the exact token count will be calculated, and the history will be trimmed to fit the context. ```json { "prompt": "Generate 5 more", "chat_history": [ { "role": "system", "content": "You are a name generator. Do not generate anything else than names" }, { "role": "user", "content": "Generate 5 names" }, { "role": "assistant", "content": "1. Olivia Bennett\n2. Ethan Carter\n3. Sophia Ramirez\n4. Liam Thompson\n5. Ava Mitchell" } ], } ``` operationId: POST-V4-/v4/models/{model_deployment_id}/chat-completions parameters: - name: model_deployment_id in: path required: true schema: title: Model Deployment Id type: string requestBody: required: true content: application/json: schema: $ref: >- #/components/schemas/egp_api_backend__server__utils__model_schemas__completions__ChatCompletionRequest responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/CompletionResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-codeSamples: - lang: Python source: |- import os from scale_gp import SGPClient client = SGPClient( api_key=os.environ.get("SGP_API_KEY"), # This is the default and can be omitted ) for chat_completion in client.models.chat_completions.create( model_deployment_id="model_deployment_id", chat_history=[{ "content": "string", "role": "user", }], prompt="prompt", ): print(chat_completion) - lang: Go source: "package main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\n\t\"github.com/stainless-sdks/sgp-go\"\n\t\"github.com/stainless-sdks/sgp-go/option\"\n\t\"github.com/stainless-sdks/sgp-go/shared\"\n)\n\nfunc main() {\n\tclient := sgp.NewClient(\n\t\toption.WithAPIKey(\"My API Key\"),\n\t)\n\tcompletionResponse, err := client.Models.ChatCompletions.New(\n\t\tcontext.TODO(),\n\t\t\"model_deployment_id\",\n\t\tsgp.ModelChatCompletionNewParams{\n\t\t\tChatHistory: sgp.F([]sgp.FlexibleMessageUnionParam{sgp.FlexibleMessageUserMessageParam{\n\t\t\t\tContent: sgp.F[sgp.FlexibleMessageUserMessageContentUnionParam](shared.UnionString(\"string\")),\n\t\t\t\tRole: sgp.F(sgp.FlexibleMessageUserMessageRoleUser),\n\t\t\t}}),\n\t\t\tPrompt: sgp.F(\"prompt\"),\n\t\t},\n\t)\n\tif err != nil {\n\t\tpanic(err.Error())\n\t}\n\tfmt.Printf(\"%+v\\n\", completionResponse.Completions)\n}\n" components: schemas: egp_api_backend__server__utils__model_schemas__completions__ChatCompletionRequest: properties: model_request_parameters: $ref: '#/components/schemas/ParameterBindings' temperature: title: Temperature description: >- What sampling temperature to use, between [0, 2]. Higher values like 1.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. Setting temperature=0.0 will enable fully deterministic (greedy) sampling.NOTE: The temperature parameter range for some model is limited to [0, 1] if the given value is above the available range, it defaults to the max value. Temperature is ignored for OpenAI reasoning models. default: 0.2 type: number maximum: 2 minimum: 0 stop_sequences: title: Stop Sequences description: >- List of up to 4 sequences where the API will stop generating further tokens. The returned text will not contain the stop sequence. items: type: string type: array maxItems: 4 max_tokens: title: Max Tokens description: > The maximum number of tokens to generate in the completion. The token count of your prompt plus max_tokens cannot exceed the model's context length. If not, specified, max_tokens will be determined based on the model used: | Model API family | Model API default | EGP applied default | | --- | --- | --- | | OpenAI Completions | [`16`](https://platform.openai.com/docs/api-reference/completions/create#max_tokens) | `context window - prompt size` | | OpenAI Chat Completions | [`context window - prompt size`](https://platform.openai.com/docs/api-reference/chat/create#max_tokens) | `context window - prompt size` | | LLM Engine | [`max_new_tokens`](https://github.com/scaleapi/launch-python-client/blob/207adced1c88c1c2907266fa9dd1f1ff3ec0ea5b/launch/client.py#L2910) parameter is required | `100` | | Anthropic Claude 2 | [`max_tokens_to_sample`](https://docs.anthropic.com/claude/reference/complete_post) parameter is required | `10000` | type: integer top_p: title: Top P description: >- The cumulative probability cutoff for token selection. Lower values mean sampling from a smaller, more top-weighted nucleus. Available for models provided by Google, LLM Engine, and OpenAI. type: number top_k: title: Top K description: >- Sample from the k most likely next tokens at each step. Lower k focuses on higher probability tokens. Available for models provided by Google and LLM Engine. type: number frequency_penalty: title: Frequency Penalty description: >- Penalize tokens based on how much they have already appeared in the text. Positive values encourage the model to generate new tokens and negative values encourage the model to repeat tokens. Available for models provided by LLM Engine and OpenAI. type: number presence_penalty: title: Presence Penalty description: >- Penalize tokens based on if they have already appeared in the text. Positive values encourage the model to generate new tokens and negative values encourage the model to repeat tokens. Available for models provided by LLM Engine and OpenAI. type: number stream: title: Stream description: Flag indicating whether to stream the completion response default: false type: boolean logprobs: title: Logprobs description: >- Whether to return logprobs. Currently only supported for llmengine chat models. type: boolean top_logprobs: title: Top Logprobs description: >- Number of top logprobs to return. Currently only supported for llmengine chat models. type: integer chat_template: title: Chat Template description: >- The chat template to use for the completion. Currently only supported for llmengine chat models. type: string chat_template_kwargs: title: Chat Template Kwargs description: >- Additional keyword arguments for the chat template. Currently only supported for llmengine chat models. additionalProperties: true type: object reasoning_effort: title: Reasoning Effort description: >- The reasoning effort to use for the completion. Currently only supported for openai models. type: string enum: - low - medium - high chat_history: items: $ref: '#/components/schemas/ChatMessageV2' type: array title: Chat History description: >- Chat history entries with roles and messages. If there's no history, pass an empty list. prompt: type: string title: Prompt description: New user prompt. This will be sent to the model with a user role. additionalProperties: false type: object required: - chat_history - prompt title: ChatCompletionRequest description: |- Represents a chat completion request. Attributes: chat_history (List[Message]): Chat history CompletionResponse: properties: finish_reason: title: Finish Reason type: string prompt_tokens: title: Prompt Tokens default: 0 type: integer response_tokens: title: Response Tokens default: 0 type: integer completions: items: prefixItems: - type: string - items: type: string type: array type: array maxItems: 2 minItems: 2 type: array title: Completions choices: title: Choices items: $ref: >- #/components/schemas/egp_api_backend__server__utils__model_schemas__completions__Choice type: array additionalProperties: false type: object required: - completions title: CompletionResponse HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError ParameterBindings: properties: bindings: additionalProperties: anyOf: - type: string - type: integer - type: number - type: boolean type: object title: Bindings additionalProperties: false type: object required: - bindings title: ParameterBindings ChatMessageV2: oneOf: - $ref: '#/components/schemas/UserMessage' - $ref: '#/components/schemas/AssistantMessage' - $ref: '#/components/schemas/SystemMessage' title: ChatMessageV2 discriminator: propertyName: role mapping: assistant: $ref: '#/components/schemas/AssistantMessage' system: $ref: '#/components/schemas/SystemMessage' user: $ref: '#/components/schemas/UserMessage' egp_api_backend__server__utils__model_schemas__completions__Choice: properties: index: type: integer title: Index logprobs: $ref: >- #/components/schemas/egp_api_backend__server__utils__model_schemas__completions__ChoiceLogprobs type: object required: - index title: Choice ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type input: title: Input ctx: type: object title: Context additionalProperties: true type: object required: - loc - msg - type title: ValidationError UserMessage: properties: role: type: string const: user title: Role description: >- The role of the message. Must be set to 'user'. A user message is a message from the user to the AI. This should be the message used to send end user input to the AI. default: user content: anyOf: - type: string - items: $ref: '#/components/schemas/UserMessageContentParts' type: array title: Content description: >- Input from the user. Can either be text or a list of content parts. Not all models support image content parts, or multiple parts. type: object required: - content title: User Message AssistantMessage: properties: role: type: string const: assistant title: Role description: >- The role of the message. Must be set to 'assistant'. An assistant message is a message from the AI to the client. It is different from an agent message in that it cannot contain a tool request. It is simply a direct response from the AI to the client. default: assistant content: type: string title: Content description: Text response from the assistant type: object required: - content title: Assistant Message SystemMessage: properties: role: type: string const: system title: Role description: >- The role of the message. Must be set to 'system'. A system message is different from other messages in that it does not originate from a party engaged in a user/AI conversation. Instead, it is a message that is injected by either the application or system to guide the conversation. For example, a system message may be used as initial instructions for an AI entity or to tell the AI that it did not do something correctly. default: system content: type: string title: Content description: Text input from the system. type: object required: - content title: System Message egp_api_backend__server__utils__model_schemas__completions__ChoiceLogprobs: properties: content: title: Content items: $ref: >- #/components/schemas/egp_api_backend__server__utils__model_schemas__completions__ChatCompletionTokenLogprob type: array type: object title: ChoiceLogprobs UserMessageContentParts: oneOf: - $ref: '#/components/schemas/TextUserMessageContentParts' - $ref: '#/components/schemas/ImageUrlUserMessageContentParts' - $ref: '#/components/schemas/ImageDataUserMessageContentParts' title: UserMessageContentParts discriminator: propertyName: type mapping: image_data: $ref: '#/components/schemas/ImageDataUserMessageContentParts' image_url: $ref: '#/components/schemas/ImageUrlUserMessageContentParts' text: $ref: '#/components/schemas/TextUserMessageContentParts' egp_api_backend__server__utils__model_schemas__completions__ChatCompletionTokenLogprob: properties: token: type: string title: Token bytes: title: Bytes items: type: integer type: array logprob: type: number title: Logprob top_logprobs: items: $ref: >- #/components/schemas/egp_api_backend__server__utils__model_schemas__completions__TopLogprob type: array title: Top Logprobs type: object required: - token - logprob - top_logprobs title: ChatCompletionTokenLogprob TextUserMessageContentParts: properties: type: type: string const: text title: Type default: text text: type: string title: Text type: object required: - text title: TextUserMessageContentParts ImageUrlUserMessageContentParts: properties: type: type: string const: image_url title: Type default: image_url image_url: $ref: '#/components/schemas/ImageUrlSubPart' description: >- Specifies the image URL and level of detail. Only supported by OpenAI models type: object required: - image_url title: ImageUrlUserMessageContentParts ImageDataUserMessageContentParts: properties: type: type: string const: image_data title: Type default: image_data image_data: $ref: '#/components/schemas/ImageDataSubPart' description: Specifies inline image data type: object required: - image_data title: ImageDataUserMessageContentParts egp_api_backend__server__utils__model_schemas__completions__TopLogprob: properties: token: type: string title: Token bytes: title: Bytes items: type: integer type: array logprob: type: number title: Logprob type: object required: - token - logprob title: TopLogprob ImageUrlSubPart: properties: url: type: string title: Url description: 'The URL of the image. Note: only OpenAI supports this.' detail: title: Detail description: >- Only used for OpenAI. Corresponds to OpenAI's image detail parameter. type: string enum: - low - high - auto type: object required: - url title: ImageUrlSubPart ImageDataSubPart: properties: type: type: string const: base64 title: Type description: The type of the image data. Only base64 is supported. default: base64 media_type: type: string title: Media Type description: >- The media/mime type of the image data. For example, 'image/png'. Check providers' documentation for supported media types. data: type: string title: Data description: The base64-encoded image data. detail: title: Detail description: >- Only used for OpenAI. Corresponds to OpenAI's image detail parameter. type: string enum: - low - high - auto type: object required: - media_type - data title: ImageDataSubPart ````