> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gp.scale.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Execute Agent
> ### Description
Executes one Agent inference step. Given a list of messages and a list of tools to ask for help from, the Agent will either respond with a final answer directly or ask the user to execute a tool to provide more information.
### Details
An Agent is a component that utilizes a Language Model (LLM) as an interpreter and decision maker. Unlike asking an LLM for a direct response, communicating with an agent consists of a running dialogue where an agent can optionally ask the user to execute specialized tools for specific tasks, such as calculations, web searches, or accessing custom data from private knowledge bases.
An agent is designed to be stateless, emitting outputs one step at a time. This means that client-side applications are responsible for managing message history, tool execution, and responses. This grants users greater flexibility to write and execute custom tools and maintain explicit control over their message history.
#### Message Types
- `User Message`: A message from the user to the agent.
- `System Message`: An informational text message from the system to guide the agent. It is not a user message or agent message because it did not come from either entity.
- `Agent Message`: A message from the agent to the client. It will contain either a final answer as `content` or a request for the user to execute a tool as a `tool_request`.
- `Tool Message`: A message from the user to the agent that contains the output of a tool execution. The tool message will be processed by the agent and the agent will respond with either a final answer or another tool request.
#### Agent Instructions
Instructions are used to guide the agent's decision making process and output generation.
Good prompt engineering is crucial to getting performant results from the agent. If you are having trouble getting the agent to perform well, try writing more specific instructions before trying more expensive techniques such as swapping in other models or finetuning the underlying LLM.
For example, the default instructions we set for the agent are the following:
> You are an AI assistant that helps users with their questions. You can answer questions directly or acquire information from any of the attached tools to assist you. Always answer the user's most recent query to the best of your knowledge.
> When asked about what tools are available, you must list each attached tool's name and description. When asked about what you can do, mention that in addition to your normal capabilities, you can also use the attached tools by listing their names and descriptions. You cannot use any other tools other than the ones provided to you explicitly.
### Restrictions and Limits
**Message Limits**:
- The message list is not limited by length, but by the context limit of the underlying language model. If you are getting an error regarding the underlying model's context limit, try using a memory strategy to condense the input messages.
**Model Restrictions**:
- Currently, only closed source models like GPT and Claude are supported due to the limitations of open source models when it comes to tool selection, generating tool arguments in valid JSON, and planning out multi-step tool execution. Specialized fine-tuning will likely be required to make open source models compatible with agents.
## OpenAPI
````yaml https://app.stainlessapi.com/api/spec/documented/sgp/openapi.yml post /v4/agents/execute
openapi: 3.1.0
info:
title: EGP API V4
description: >-
This is the parent API for all EGP APIs. If you are looking for the EGP API,
please go to https://api.egp.scale.com/docs.
contact:
name: Scale Generative AI Platform
url: https://scale.com/genai-platform
version: 0.1.0
servers:
- url: https://api.egp.scale.com
security: []
tags:
- name: Models
description: Model API.
paths:
/v4/agents/execute:
post:
tags:
- Agents
summary: Execute Agent
description: >-
### Description
Executes one Agent inference step. Given a list of messages and a list
of tools to ask for help from, the Agent will either respond
with a final answer directly or ask the user to execute a tool
to provide more information.
### Details
An Agent is a component that utilizes a Language Model (LLM) as an
interpreter and decision maker. Unlike asking an LLM for a
direct response, communicating with an agent consists of a
running dialogue where an agent can optionally ask the user to execute
specialized tools for specific tasks, such as calculations, web
searches, or accessing custom data from private knowledge bases.
An agent is designed to be stateless, emitting outputs one step at a
time. This means that client-side applications are responsible
for managing message history, tool execution, and responses.
This grants users greater flexibility to write and execute custom
tools and maintain explicit control over their message history.
#### Message Types
- `User Message`: A message from the user to the agent.
- `System Message`: An informational text message from the system to
guide the agent. It is not a user message or agent message
because it did not come from either entity.
- `Agent Message`: A message from the agent to the client. It will
contain either a final answer as `content` or a request for the
user to execute a tool as a `tool_request`.
- `Tool Message`: A message from the user to the agent that contains the
output of a tool execution. The tool message will be processed
by the agent and the agent will respond with either a final
answer or another tool request.
#### Agent Instructions
Instructions are used to guide the agent's decision making process and
output generation.
Good prompt engineering is crucial to getting performant results from
the agent. If you are having trouble getting the agent to
perform well, try writing more specific instructions before
trying more expensive techniques such as swapping in other models or
finetuning the underlying LLM.
For example, the default instructions we set for the agent are the
following:
> You are an AI assistant that helps users with their questions. You can
answer questions directly or acquire information from any of the
attached tools to assist you. Always answer the user's most
recent query to the best of your knowledge.
> When asked about what tools are available, you must list each attached
tool's name and description. When asked about what you can do,
mention that in addition to your normal capabilities, you can
also use the attached tools by listing their names and
descriptions. You cannot use any other tools other than the ones
provided to you explicitly.
### Restrictions and Limits
**Message Limits**:
- The message list is not limited by length, but by the context limit of the underlying language model. If you are getting an error regarding the underlying model's context limit, try using a memory strategy to condense the input messages.
**Model Restrictions**:
- Currently, only closed source models like GPT and Claude are supported due to the limitations of open source models when it comes to tool selection, generating tool arguments in valid JSON, and planning out multi-step tool execution. Specialized fine-tuning will likely be required to make open source models compatible with agents.
operationId: POST-V4-/v2/agents/execute
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/ExecuteAgentRequest'
required: true
responses:
'200':
description: Successful Response
content:
application/json:
schema:
$ref: '#/components/schemas/ExecuteAgentResponse'
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
x-codeSamples:
- lang: Python
source: |-
import os
from scale_gp import SGPClient
client = SGPClient(
api_key=os.environ.get("SGP_API_KEY"), # This is the default and can be omitted
)
execute_agent_response = client.agents.execute(
messages=[{
"content": "string",
"role": "user",
}],
model="gpt-4",
tools=[{
"arguments": {
"type": "object"
},
"description": "description",
"name": "name",
}],
)
print(execute_agent_response.action)
- lang: Go
source: "package main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\n\t\"github.com/stainless-sdks/sgp-go\"\n\t\"github.com/stainless-sdks/sgp-go/option\"\n\t\"github.com/stainless-sdks/sgp-go/shared\"\n)\n\nfunc main() {\n\tclient := sgp.NewClient(\n\t\toption.WithAPIKey(\"My API Key\"),\n\t)\n\texecuteAgentResponse, err := client.Agents.Execute(context.TODO(), sgp.AgentExecuteParams{\n\t\tMessages: sgp.F([]sgp.AgentExecuteParamsMessageUnion{sgp.AgentExecuteParamsMessagesUserMessage{\n\t\t\tContent: sgp.F[sgp.AgentExecuteParamsMessagesUserMessageContentUnion](shared.UnionString(\"string\")),\n\t\t\tRole: sgp.F(sgp.AgentExecuteParamsMessagesUserMessageRoleUser),\n\t\t}}),\n\t\tModel: sgp.F(sgp.AgentExecuteParamsModelGpt4),\n\t\tTools: sgp.F([]sgp.AgentExecuteParamsTool{{\n\t\t\tArguments: sgp.F(sgp.AgentExecuteParamsToolsArguments{\n\t\t\t\tType: sgp.F(sgp.AgentExecuteParamsToolsArgumentsTypeObject),\n\t\t\t}),\n\t\t\tDescription: sgp.F(\"description\"),\n\t\t\tName: sgp.F(\"name\"),\n\t\t}}),\n\t})\n\tif err != nil {\n\t\tpanic(err.Error())\n\t}\n\tfmt.Printf(\"%+v\\n\", executeAgentResponse.Action)\n}\n"
components:
schemas:
ExecuteAgentRequest:
properties:
model:
type: string
enum:
- gpt-4
- gpt-4-0613
- gpt-4-32k
- gpt-4-32k-0613
- gpt-3.5-turbo
- gpt-3.5-turbo-0613
- gpt-3.5-turbo-16k
- gpt-3.5-turbo-16k-0613
- claude-instant-1
- claude-instant-1.1
- claude-2
- claude-2.0
title: Model
description: >-
The ID of the model to use for the agent. We only support the models
listed here so far.
memory_strategy:
$ref: '#/components/schemas/MemoryStrategy'
description: >-
The memory strategy to use for the agent. A memory strategy is a way
to prevent the underlying LLM's context limit from being exceeded.
Each memory strategy uses a different technique to condense the
input message list into a smaller payload for the underlying LLM.
We only support the Last K memory strategy right now, but will be
adding new strategies soon.
tools:
items:
$ref: '#/components/schemas/Tool'
type: array
title: Tools
description: >+
The list of specs of tools that the agent can use. Each spec must
contain a `name` key set to the name of the tool, a `description`
key set to the description of the tool, and an `arguments` key set
to a JSON Schema compliant object describing the tool arguments.
The name and description of each tool is used by the agent to decide
when to use certain tools. Because some queries are complex and may
require multiple tools to complete, it is important to make these
descriptions as informative as possible. If a tool is not being
chosen when it should, it is common practice to tune the description
of the tool to make it more apparent to the agent when the tool can
be used effectively.
messages:
items:
$ref: '#/components/schemas/Message'
type: array
title: Messages
description: >-
The list of messages in the conversation.
Expand each message type to see how it works and when to use it.
Most conversations should begin with a single `user` message.
model_parameters:
$ref: '#/components/schemas/ModelParameters'
description: >-
Configuration parameters for the agent model, such as temperature,
max_tokens, and stop_sequences.
If not specified, the default value are:
- temperature: 0.2
- max_tokens: None (limited by the model's max tokens)
- stop_sequences: None
default:
temperature: 0.2
instructions:
title: Instructions
description: >-
The initial instructions to provide to the agent.
Use this to guide the agent to act in more specific ways. For
example, if you have specific rules you want to restrict the agent
to follow you can specify them here. For example, if I want the
agent to always use certain tools before others, I can write that
rule in these instructions.
Good prompt engineering is crucial to getting performant results
from the agent. If you are having trouble getting the agent to
perform well, try writing more specific instructions here before
trying more expensive techniques such as swapping in other models or
finetuning the underlying LLM.
default: >-
You are an AI assistant that helps users with their questions. You
can answer questions directly or acquire information from any of the
attached tools to assist you. Always answer the user's most recent
query to the best of your knowledge.
When asked about what tools are available, you must list each
attached tool's name and description. When asked about what you can
do, mention that in addition to your normal capabilities, you can
also use the attached tools by listing their names and descriptions.
You cannot use any other tools other than the ones provided to you
explicitly.
type: string
type: object
required:
- model
- tools
- messages
title: ExecuteAgentRequest
ExecuteAgentResponse:
properties:
action:
$ref: '#/components/schemas/AgentAction'
description: >-
The action that the agent performed.
The context will contain a key for each action that the agent can
perform. However, only the key corresponding to the action that the
agent actually performed will have a populated value. The rest of
the values will be `null`.
context:
$ref: '#/components/schemas/ActionContext'
description: >-
Context object containing the output payload. This will contain a
key for all actions that the agent can perform. However, only the
key corresponding to the action that the agent performed have a
populated value. The rest of the values will be `null`.
type: object
required:
- action
- context
title: ExecuteAgentResponse
HTTPValidationError:
properties:
detail:
items:
$ref: '#/components/schemas/ValidationError'
type: array
title: Detail
type: object
title: HTTPValidationError
MemoryStrategy:
$ref: '#/components/schemas/LastKMemoryStrategy'
title: MemoryStrategy
Tool:
properties:
name:
type: string
pattern: ^[a-zA-Z0-9_-]{1,64}$
title: Name
description: >-
Name of the tool.
A tool is a function that the _client application_ has at its
disposal. The tool name is the name the client wishes the Agent to
use to refer to this function when it decides if it wants the user
to use the tool or not. Itmust be unique amongst the set of tools
provided in a single API call.
description:
type: string
title: Description
description: >-
Description of the tool.
Because some queries are complex and may require multiple tools to
complete, it is important to make these descriptions as informative
as possible. If a tool is not being chosen when it should, it is
common practice to tune the description of the tool to make it more
apparent to the agent when the tool can be used effectively.
arguments:
$ref: '#/components/schemas/ToolArguments'
description: >-
An JSON Schema-compliant schema for the tool arguments. To describe
a function that accepts no parameters, provide the value `{"type":
""object", "properties": {}}`.
For more information on how to define a valid JSON Schema object,
visit
https://json-schema.org/understanding-json-schema/reference/object.html
type: object
required:
- name
- description
- arguments
title: Tool
Message:
oneOf:
- $ref: '#/components/schemas/UserMessage'
- $ref: '#/components/schemas/ToolMessage'
- $ref: '#/components/schemas/AgentMessage'
- $ref: '#/components/schemas/SystemMessage'
- $ref: '#/components/schemas/AssistantMessage'
title: Message
discriminator:
propertyName: role
mapping:
agent:
$ref: '#/components/schemas/AgentMessage'
assistant:
$ref: '#/components/schemas/AssistantMessage'
system:
$ref: '#/components/schemas/SystemMessage'
tool:
$ref: '#/components/schemas/ToolMessage'
user:
$ref: '#/components/schemas/UserMessage'
ModelParameters:
properties:
temperature:
title: Temperature
description: >-
What sampling temperature to use, between [0, 1]. Higher values like
0.8 will make the output more random, while lower values like 0.2
will make it more focused and deterministic. Setting temperature=0.0
will enable fully deterministic (greedy) sampling. Temperature is
ignored for OpenAI reasoning models.
default: 0.2
type: number
maximum: 1
minimum: 0
stop_sequences:
title: Stop Sequences
description: >-
List of up to 4 sequences where the API will stop generating further
tokens. The returned text will not contain the stop sequence.
items:
type: string
type: array
maxItems: 4
max_tokens:
title: Max Tokens
description: >
The maximum number of tokens to generate in the completion. The
token count of your prompt plus max_tokens cannot exceed the model's
context length. If not, specified, max_tokens will be determined
based on the model used:
| Model API family | Model API default | EGP applied default |
| --- | --- | --- |
| OpenAI Completions |
[`16`](https://platform.openai.com/docs/api-reference/completions/create#max_tokens)
| `context window - prompt size` |
| OpenAI Chat Completions | [`context window - prompt
size`](https://platform.openai.com/docs/api-reference/chat/create#max_tokens)
| `context window - prompt size` |
| LLM Engine |
[`max_new_tokens`](https://github.com/scaleapi/launch-python-client/blob/207adced1c88c1c2907266fa9dd1f1ff3ec0ea5b/launch/client.py#L2910)
parameter is required | `100` |
| Anthropic Claude 2 |
[`max_tokens_to_sample`](https://docs.anthropic.com/claude/reference/complete_post)
parameter is required | `10000` |
type: integer
topP:
title: Topp
description: >-
The cumulative probability cutoff for token selection. Lower values
mean sampling from a smaller, more top-weighted nucleus. Available
for models from Anthropic, Google, Mistral, LLM Engine, and OpenAI.
type: number
topK:
title: Topk
description: >-
Sample from the k most likely next tokens at each step. Lower k
focuses on higher probability tokens. Available for models from
Anthropic, Google, and LLM Engine.
type: integer
frequency_penalty:
title: Frequency Penalty
description: >-
Penalize tokens based on how much they have already appeared in the
text. Positive values encourage the model to generate new tokens and
negative values encourage the model to repeat tokens. Available for
models from LLM Engine, and OpenAI
type: number
presence_penalty:
title: Presence Penalty
description: >-
Penalize tokens based on if they have already appeared in the text.
Positive values encourage the model to generate new tokens and
negative values encourage the model to repeat tokens. Available for
models from LLM Engine, and OpenAI.
type: number
reasoning_effort:
title: Reasoning Effort
description: >-
The effort level for the model to reason. Available for models from
OpenAI.
type: string
enum:
- low
- medium
- high
type: object
title: ModelParameters
AgentAction:
type: string
enum:
- tool_request
- content
title: AgentAction
ActionContext:
properties:
content:
title: Content
description: The final output of the agent when it no longer needs any tools
type: string
tool_request:
$ref: '#/components/schemas/ToolRequest'
description: The tool request if the agent needs more information.
type: object
title: ActionContext
ValidationError:
properties:
loc:
items:
anyOf:
- type: string
- type: integer
type: array
title: Location
msg:
type: string
title: Message
type:
type: string
title: Error Type
input:
title: Input
ctx:
type: object
title: Context
additionalProperties: true
type: object
required:
- loc
- msg
- type
title: ValidationError
LastKMemoryStrategy:
properties:
name:
type: string
const: last_k
title: Name
description: >-
Name of the memory strategy. Must be `last_k`.
This strategy truncates the message history to the last `k`
messages. It is the simplest way to prevent the model's context
limit from being exceeded. However, this strategy only allows the
model to have short term memory. For longer term memory, please use
one of the other strategies.
default: last_k
params:
$ref: '#/components/schemas/LastKMemoryStrategyParams'
description: Configuration parameters for the memory strategy.
type: object
required:
- params
title: Last K Memory Strategy
ToolArguments:
properties:
type:
type: string
const: object
title: Type
description: Type of argument. Currently only "object" is supported
properties:
additionalProperties:
$ref: '#/components/schemas/ToolPropertyValue'
type: object
title: Properties
description: >-
An object where each key is the name of a keyword argument and each
value is a schema used to validate that property. Each schema must
have a type and description, but can also have a default value and
examples.
For more information on how to define a valid property, visit
https://json-schema.org/understanding-json-schema/reference/object.html
required:
title: Required
description: List of required property names.
items:
type: string
type: array
type: object
required:
- type
title: ToolArguments
UserMessage:
properties:
role:
type: string
const: user
title: Role
description: >-
The role of the message. Must be set to 'user'.
A user message is a message from the user to the AI. This should be
the message used to send end user input to the AI.
default: user
content:
anyOf:
- type: string
- items:
$ref: '#/components/schemas/UserMessageContentParts'
type: array
title: Content
description: >-
Input from the user. Can either be text or a list of content parts.
Not all models support image content parts, or multiple parts.
type: object
required:
- content
title: User Message
ToolMessage:
properties:
role:
type: string
const: tool
title: Role
description: >-
The role of the message. Must be set to 'tool'.
A tool message is a message the client application uses to send tool
output to the AI. It should contain the name of the tool and the
output from the tool encoded as text.
default: tool
name:
type: string
title: Name
description: Name of the tool
content:
type: string
title: Content
description: Text output from the tool
type: object
required:
- name
- content
title: Tool Message
AgentMessage:
properties:
role:
type: string
const: agent
title: Role
description: >-
The role of the message. Must be set to 'agent'.
An agent message is a message generated by an AI agent. It is
different than an assistant message in that it can either contain a
direct content output or a tool request that the client application
must handle.
default: agent
content:
title: Content
description: Text output of the agent if no more tools are needed.
type: string
tool_request:
$ref: '#/components/schemas/ToolRequest'
description: The tool request if the agent needs more information.
type: object
title: Agent Message
SystemMessage:
properties:
role:
type: string
const: system
title: Role
description: >-
The role of the message. Must be set to 'system'.
A system message is different from other messages in that it does
not originate from a party engaged in a user/AI conversation.
Instead, it is a message that is injected by either the application
or system to guide the conversation. For example, a system message
may be used as initial instructions for an AI entity or to tell the
AI that it did not do something correctly.
default: system
content:
type: string
title: Content
description: Text input from the system.
type: object
required:
- content
title: System Message
AssistantMessage:
properties:
role:
type: string
const: assistant
title: Role
description: >-
The role of the message. Must be set to 'assistant'.
An assistant message is a message from the AI to the client. It is
different from an agent message in that it cannot contain a tool
request. It is simply a direct response from the AI to the client.
default: assistant
content:
type: string
title: Content
description: Text response from the assistant
type: object
required:
- content
title: Assistant Message
ToolRequest:
properties:
name:
type: string
title: Name
description: Name of the tool that the AI wants the client to use.
arguments:
type: string
title: Arguments
description: >-
Arguments to pass to the tool. The format must be a JSON
Schema-compliant object serialized into a string.
type: object
required:
- name
- arguments
title: ToolRequest
LastKMemoryStrategyParams:
properties:
k:
type: integer
minimum: 1
title: K
description: The maximum number of previous messages to remember.
type: object
required:
- k
title: LastKMemoryStrategyParams
ToolPropertyValue:
properties:
type:
type: string
enum:
- string
- number
- integer
- boolean
- object
- array
- 'null'
title: Type
description: >-
The argument's type.
The type is used to help the Agent generate valid arguments for the
tool.
For more information about types, see:
https://json-schema.org/understanding-json-schema/reference/type.html#type-specific-keywords
description:
type: string
title: Description
description: >-
Description of what the argument is used for.
This description is used to help the Agent generate sensible
arguments for the tool. It is very important that this description
is succinct, clear, and accurate.
default:
title: Default
description: A default value for the argument if unspecified.
type: string
examples:
title: Examples
description: >-
Example values demonstrating how the argument should look.
This can be used to help the agent understand what a valid argument
should look like.
items:
type: string
type: array
type: object
required:
- type
- description
title: property
UserMessageContentParts:
oneOf:
- $ref: '#/components/schemas/TextUserMessageContentParts'
- $ref: '#/components/schemas/ImageUrlUserMessageContentParts'
- $ref: '#/components/schemas/ImageDataUserMessageContentParts'
title: UserMessageContentParts
discriminator:
propertyName: type
mapping:
image_data:
$ref: '#/components/schemas/ImageDataUserMessageContentParts'
image_url:
$ref: '#/components/schemas/ImageUrlUserMessageContentParts'
text:
$ref: '#/components/schemas/TextUserMessageContentParts'
TextUserMessageContentParts:
properties:
type:
type: string
const: text
title: Type
default: text
text:
type: string
title: Text
type: object
required:
- text
title: TextUserMessageContentParts
ImageUrlUserMessageContentParts:
properties:
type:
type: string
const: image_url
title: Type
default: image_url
image_url:
$ref: '#/components/schemas/ImageUrlSubPart'
description: >-
Specifies the image URL and level of detail. Only supported by
OpenAI models
type: object
required:
- image_url
title: ImageUrlUserMessageContentParts
ImageDataUserMessageContentParts:
properties:
type:
type: string
const: image_data
title: Type
default: image_data
image_data:
$ref: '#/components/schemas/ImageDataSubPart'
description: Specifies inline image data
type: object
required:
- image_data
title: ImageDataUserMessageContentParts
ImageUrlSubPart:
properties:
url:
type: string
title: Url
description: 'The URL of the image. Note: only OpenAI supports this.'
detail:
title: Detail
description: >-
Only used for OpenAI. Corresponds to OpenAI's image detail
parameter.
type: string
enum:
- low
- high
- auto
type: object
required:
- url
title: ImageUrlSubPart
ImageDataSubPart:
properties:
type:
type: string
const: base64
title: Type
description: The type of the image data. Only base64 is supported.
default: base64
media_type:
type: string
title: Media Type
description: >-
The media/mime type of the image data. For example, 'image/png'.
Check providers' documentation for supported media types.
data:
type: string
title: Data
description: The base64-encoded image data.
detail:
title: Detail
description: >-
Only used for OpenAI. Corresponds to OpenAI's image detail
parameter.
type: string
enum:
- low
- high
- auto
type: object
required:
- media_type
- data
title: ImageDataSubPart
````