Completions

Create Completion

Description

Given a user’s input, runs LLM inference to produce the model’s response.

Details

LLM completions have many use cases, such as content summarization, question-answering, and text generation.

The model parameter determines which LLM will be used to generate the completion. Keep in mind that different models have varying sizes, costs, and may perform differently across different tasks.

The user input, commonly referred to as the “prompt”, is a required field in the request body. The quality of the model’s response can vary greatly depending on the input prompt. Good prompt engineering can significantly enhance the response quality. If you encounter suboptimal results, consider writing more specific instructions or providing examples to the LLM before trying more expensive techniques such as swapping in other models or finetuning.

By default, the endpoint will return the entire response as one whole object. If you would prefer to stream the completion in real-time, you can achieve this by setting the stream flag to true.

POST

completions

import SGPClient from 'sgp';

const client = new SGPClient({
  apiKey: process.env['SGP_API_KEY'], // This is the default and can be omitted
});

async function main() {
  const completionsResponse = await client.completions.create({ model: 'gpt-4', prompt: 'prompt' });

  console.log(completionsResponse.completion);
}

main();

{
  "completion": {
    "text": "<string>",
    "finish_reason": "<string>",
    "response_metadata": {}
  },
  "token_usage": {
    "prompt": 123,
    "completion": 123,
    "total": 123
  }
}

Authorizations

x-api-key

string

header

required

Body

application/json

model

required

The ID of the model to use for completions.

Users have two options:

Option 1: Use one of the supported models from the dropdown.
Option 2: Enter the ID of a custom model.

Note: For custom models we currently only support models finetuned using using the Scale-hosted LLM-Engine API.

Available options:

gpt-4,

gpt-4-0613,

gpt-4-32k,

gpt-4-32k-0613,

gpt-4-vision-preview,

gpt-4o,

gpt-4o-mini,

gpt-4o-2024-08-06,

gpt-3.5-turbo,

gpt-3.5-turbo-0613,

gpt-3.5-turbo-16k,

gpt-3.5-turbo-16k-0613,

gemini-pro,

gemini-1.5-pro-001,

gemini-1.5-pro-002,

gemini-1.5-pro-preview-0409,

gemini-1.5-pro-preview-0514,

text-davinci-003,

text-davinci-002,

text-curie-001,

text-babbage-001,

text-ada-001,

claude-instant-1,

claude-instant-1.1,

claude-2,

claude-2.0,

llama-7b,

llama-2-7b,

llama-2-7b-chat,

llama-2-13b,

llama-2-13b-chat,

llama-2-70b,

llama-2-70b-chat,

llama-3-8b,

llama-3-8b-instruct,

llama-3-1-8b-instruct,

llama-3-1-70b-instruct,

llama-3-70b-instruct,

llama-3-2-1b-instruct,

llama-3-2-3b-instruct,

llama-3-3-70b-instruct,

Meta-Llama-3-8B-Instruct-RMU,

Meta-Llama-3-8B-Instruct-RR,

Meta-Llama-3-8B-Instruct-DERTA,

Meta-Llama-3-8B-Instruct-LAT,

falcon-7b,

falcon-7b-instruct,

falcon-40b,

falcon-40b-instruct,

mpt-7b,

mpt-7b-instruct,

flan-t5-xxl,

mistral-7b,

mistral-7b-instruct,

mixtral-8x7b,

mixtral-8x7b-instruct,

mixtral-8x22b-instruct,

llm-jp-13b-instruct-full,

llm-jp-13b-instruct-full-dolly,

zephyr-7b-alpha,

zephyr-7b-beta,

zephyr-cat-merged,

codellama-7b,

codellama-7b-instruct,

codellama-13b,

codellama-13b-instruct,

codellama-34b,

codellama-34b-instruct,

phi-3-mini-4k-instruct,

phi-3-cat-merged,

dolphin-2.9-llama3-8b,

dolphin-2.9-llama3-70b,

defense-llama-3-8b-instruct,

donovan-combat-llama

prompt

string

required

Prompt for which to generate the completion.

Good prompt engineering is crucial to getting performant results from the model. If you are having trouble getting the model to perform well, try writing a more specific prompt here before trying more expensive techniques such as swapping in other models or finetuning the underlying LLM.

account_id

string

The account ID to use for usage tracking. This will be gradually enforced.

images

object[]

List of image urls to be used for image based completions. Leave empty for text based completions.

model_parameters

object

Configuration parameters for the completion model, such as temperature, max_tokens, and stop_sequences.

If not specified, the default value are:

temperature: 0.2
max_tokens: None (limited by the model's max tokens)
stop_sequences: None

model_parameters.temperature

number

default:0.2

What sampling temperature to use, between [0, 1]. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. Setting temperature=0.0 will enable fully deterministic (greedy) sampling. Temperature is ignored for OpenAI reasoning models.

Required range: 0 <= x <= 1

model_parameters.stop_sequences

string[]

List of up to 4 sequences where the API will stop generating further tokens. The returned text will not contain the stop sequence.

model_parameters.max_tokens

integer

The maximum number of tokens to generate in the completion. The token count of your prompt plus max_tokens cannot exceed the model's context length. If not, specified, max_tokens will be determined based on the model used:

Model API family	Model API default	EGP applied default
OpenAI Completions	`16`	`context window - prompt size`
OpenAI Chat Completions	`context window - prompt size`	`context window - prompt size`
LLM Engine	`max_new_tokens` parameter is required	`100`
Anthropic Claude 2	`max_tokens_to_sample` parameter is required	`10000`

model_parameters.topP

number

The cumulative probability cutoff for token selection. Lower values mean sampling from a smaller, more top-weighted nucleus. Available for models from Anthropic, Google, Mistral, LLM Engine, and OpenAI.

model_parameters.topK

integer

Sample from the k most likely next tokens at each step. Lower k focuses on higher probability tokens. Available for models from Anthropic, Google, and LLM Engine.

model_parameters.frequency_penalty

number

Penalize tokens based on how much they have already appeared in the text. Positive values encourage the model to generate new tokens and negative values encourage the model to repeat tokens. Available for models from LLM Engine, and OpenAI

model_parameters.presence_penalty

number

Penalize tokens based on if they have already appeared in the text. Positive values encourage the model to generate new tokens and negative values encourage the model to repeat tokens. Available for models from LLM Engine, and OpenAI.

model_parameters.reasoning_effort

enum<string>

The effort level for the model to reason. Available for models from OpenAI.

Available options:

low,

medium,

high

stream

boolean

default:false

Whether or not to stream the response.

Setting this to True will stream the completion in real-time.

Response

200

application/json

Successful Response

completion

object

required

The actual completion text and the finish reason.

token_usage

object

Token usage numbers. If streaming, this field is null until the stream completes, at which point it will be populated (if supported).

Execute Agent Create Chat Completion

import SGPClient from 'sgp';

const client = new SGPClient({
  apiKey: process.env['SGP_API_KEY'], // This is the default and can be omitted
});

async function main() {
  const completionsResponse = await client.completions.create({ model: 'gpt-4', prompt: 'prompt' });

  console.log(completionsResponse.completion);
}

main();

{
  "completion": {
    "text": "<string>",
    "finish_reason": "<string>",
    "response_metadata": {}
  },
  "token_usage": {
    "prompt": 123,
    "completion": 123,
    "total": 123
  }
}

Knowledge Bases

Chunks

Agents

Completions

Chat Completions

Models

Users

Accounts

Organizations

Question Sets

Evaluations

Evaluation Configs

Evaluation Datasets

Studio Projects

Application Specs

Questions

Knowledge Base Data Sources

Model Templates V3 (Beta)

Model server

API Reference

Fine Tuning Jobs V3 (Beta)

Training Datasets V3 (Beta)

package deployments

Beta

Applications

ChatThreads

Interactions

MonitoringDashboard

Chat Themes

account groups

account

Guardrails

Create Completion

Description

Details

Authorizations

Body

Response