Chat Completions

Create Chat Completion

Description

Given a list of messages representing a conversation history, runs LLM inference to produce the next message.

Details

Like completions, chat completions involve an LLM’s response to input. However, chat completions take a conversation history as input, instead of a single prompt, which enables the LLM to create responses that take past context into account.

Messages

The primary input to the LLM is a list of messages represented by the messages array, which forms the conversation. The messages array must contain at least one message object. Each message object is attributed to a specific entity through its role. The available roles are:

user: Represents the human querying the model. - assistant: Represents the model responding to user. - system: Represents a non-user entity that provides information to guide the behavior of the assistant.

When the role of a message is set to user, assistant, or system, the message must also contain a content field which is a string representing the actual text of the message itself. Semantically, when the role is user, content contains the user’s query. When the role is assistant, content is the model’s response to the user. When the role is system, content represents the instruction for the assistant.

Instructions

You may provide instructions to the assistant by supplying by supplying instructions in the HTTP request body or by specifying a message with role set to system in the messages array. By convention, the system message should be the first message in the array. Do not specify both an instruction and a system message in the messages array.

POST

chat-completions

import SGPClient from 'sgp';

const client = new SGPClient({
  apiKey: process.env['SGP_API_KEY'], // This is the default and can be omitted
});

async function main() {
  const chatCompletionsResponse = await client.chatCompletions.create({
    messages: [{ content: 'string' }],
    model: 'o1',
  });

  console.log(chatCompletionsResponse.chat_completion);
}

main();

{
  "chat_completion": {
    "message": {
      "role": "user",
      "content": "<string>"
    },
    "finish_reason": "<string>"
  },
  "token_usage": {
    "prompt": 123,
    "completion": 123,
    "total": 123
  }
}

Authorizations

x-api-key

string

header

required

Body

application/json

model

enum<string>

required

The ID of the model to use for chat completions. We only support the models listed here so far.

Available options:

o1,

o1-mini,

o3-mini,

gpt-4,

gpt-4-0613,

gpt-4-32k,

gpt-4-32k-0613,

gpt-4o,

gpt-4o-mini,

gpt-4o-2024-08-06,

gpt-3.5-turbo,

gpt-3.5-turbo-0613,

gpt-3.5-turbo-16k,

gpt-3.5-turbo-16k-0613,

gemini-pro,

gemini-1.5-pro-001,

gemini-1.5-pro-002,

gemini-1.5-pro-preview-0409,

gemini-1.5-pro-preview-0514,

llama-2-7b-chat,

llama-2-13b-chat,

llama-2-70b-chat,

llama-3-8b-instruct,

llama-3-70b-instruct,

llama-3-1-8b-instruct,

llama-3-1-70b-instruct,

llama-3-2-1b-instruct,

llama-3-2-3b-instruct,

llama-3-3-70b-instruct,

Meta-Llama-3-8B-Instruct-RMU,

Meta-Llama-3-8B-Instruct-RR,

Meta-Llama-3-8B-Instruct-DERTA,

Meta-Llama-3-8B-Instruct-LAT,

mixtral-8x7b-instruct,

mixtral-8x22b-instruct,

claude-3-opus-20240229,

claude-3-sonnet-20240229,

claude-3-haiku-20240307,

claude-3-5-sonnet-20240620,

claude-3-5-sonnet-20241022,

mistral-large-latest,

phi-3-mini-4k-instruct,

phi-3-cat-merged,

zephyr-cat-merged,

dolphin-2.9-llama3-8b,

dolphin-2.9-llama3-70b,

defense-llama-3-8b-instruct,

donovan-combat-llama,

llama3-1-405b-instruct-v1

messages

object[]

required

The list of messages in the conversation.

Expand each message type to see how it works and when to use it. Most conversations should begin with a single user message.

account_id

string

The account ID to use for usage tracking. This will be gradually enforced.

memory_strategy

object

The memory strategy to use for the agent. A memory strategy is a way to prevent the underlying LLM's context limit from being exceeded. Each memory strategy uses a different technique to condense the input message list into a smaller payload for the underlying LLM.

We only support the Last K memory strategy right now, but will be adding new strategies soon.

model_parameters

object

Configuration parameters for the chat completion model, such as temperature, max_tokens, and stop_sequences.

If not specified, the default value are:

temperature: 0.2
max_tokens: None (limited by the model's max tokens)
stop_sequences: None

model_parameters.temperature

number

default:0.2

What sampling temperature to use, between [0, 1]. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. Setting temperature=0.0 will enable fully deterministic (greedy) sampling. Temperature is ignored for OpenAI reasoning models.

Required range: 0 <= x <= 1

model_parameters.stop_sequences

string[]

List of up to 4 sequences where the API will stop generating further tokens. The returned text will not contain the stop sequence.

model_parameters.max_tokens

integer

The maximum number of tokens to generate in the completion. The token count of your prompt plus max_tokens cannot exceed the model's context length. If not, specified, max_tokens will be determined based on the model used:

Model API family	Model API default	EGP applied default
OpenAI Completions	`16`	`context window - prompt size`
OpenAI Chat Completions	`context window - prompt size`	`context window - prompt size`
LLM Engine	`max_new_tokens` parameter is required	`100`
Anthropic Claude 2	`max_tokens_to_sample` parameter is required	`10000`

model_parameters.topP

number

The cumulative probability cutoff for token selection. Lower values mean sampling from a smaller, more top-weighted nucleus. Available for models from Anthropic, Google, Mistral, LLM Engine, and OpenAI.

model_parameters.topK

integer

Sample from the k most likely next tokens at each step. Lower k focuses on higher probability tokens. Available for models from Anthropic, Google, and LLM Engine.

model_parameters.frequency_penalty

number

Penalize tokens based on how much they have already appeared in the text. Positive values encourage the model to generate new tokens and negative values encourage the model to repeat tokens. Available for models from LLM Engine, and OpenAI

model_parameters.presence_penalty

number

Penalize tokens based on if they have already appeared in the text. Positive values encourage the model to generate new tokens and negative values encourage the model to repeat tokens. Available for models from LLM Engine, and OpenAI.

model_parameters.reasoning_effort

enum<string>

The effort level for the model to reason. Available for models from OpenAI.

Available options:

low,

medium,

high

instructions

string

default:You are an AI assistant that helps users with their questions by chatting back and forth with them. When asked a question, you should answer it as best as you can with the information you have. If you need more information, you can ask the user for it.

The initial instructions to provide to the chat completion model.

Use this to guide the model to act in more specific ways. For example, if you have specific rules you want to restrict the model to follow you can specify them here.

Good prompt engineering is crucial to getting performant results from the model. If you are having trouble getting the model to perform well, try writing more specific instructions here before trying more expensive techniques such as swapping in other models or finetuning the underlying LLM.

chat_template

string

Currently only supported for LLM-Engine models. A Jinja template string that defines how the chat completion API formats the string prompt. For Llama models, the template must take in at most a messages object, bos_token string, and eos_token string. The messages object is a list of dictionaries, each with keys role and content. For Mixtral models, the template must take in at most a messages object and eos_token string. The messages object looks identical to the Llama model's messages object, but the template can assume the role key takes on the values user or assistant, or system for the first message. The chat template either needs to handle this system message (which gets set via the instructions field or by the messages), or the instructions field must be set to null and the messages object must not contain any system messages.See the default chat template present in the Llama and Mixtral tokenizers for examples.

stream

boolean

default:false

Whether or not to stream the response.

Setting this to True will stream the completion in real-time.

Response

200

application/json

Successful Response

chat_completion

object

required

token_usage

object

Create Completion Execute Model Deployment

import SGPClient from 'sgp';

const client = new SGPClient({
  apiKey: process.env['SGP_API_KEY'], // This is the default and can be omitted
});

async function main() {
  const chatCompletionsResponse = await client.chatCompletions.create({
    messages: [{ content: 'string' }],
    model: 'o1',
  });

  console.log(chatCompletionsResponse.chat_completion);
}

main();

{
  "chat_completion": {
    "message": {
      "role": "user",
      "content": "<string>"
    },
    "finish_reason": "<string>"
  },
  "token_usage": {
    "prompt": 123,
    "completion": 123,
    "total": 123
  }
}

Knowledge Bases

Chunks

Agents

Completions

Chat Completions

Models

Users

Accounts

Organizations

Question Sets

Evaluations

Evaluation Configs

Evaluation Datasets

Studio Projects

Application Specs

Questions

Knowledge Base Data Sources

Model Templates V3 (Beta)

Model server

API Reference

Fine Tuning Jobs V3 (Beta)

Training Datasets V3 (Beta)

package deployments

Beta

Applications

ChatThreads

Interactions

MonitoringDashboard

Chat Themes

account groups

account

Guardrails

Create Chat Completion

Description

Details

Messages

Instructions

Authorizations

Body

Response