Inference
Create Completion and Chat Completions
Recipes
- Evaluations
- Applications
- Datasets
- Inference
Inference
Create Completion and Chat Completions
Use this recipe to use the Scale GenAI Platform SDK to perform completion and chat completions
Follow the instructions in the Quickstart Guide to setup the SGP Client
from scale_gp import SGPClient
client = SGPClient(api_key=api_key)
Define model name and parameters to perform a completion.
model_parameters = ModelParameters(max_tokens=200, temperature=0.5, top_k=1, top_p=1)
completion = client.completions.create(
model="gpt-4",
prompt="Why is the sky blue?",
account_id=account_id,
model_parameters=model_parameters,
)
Call a chat model to perform a completion using a message and the parameters defined above. Chat completion allows for a series of messages to be sent to the model, serving as a conversation between the user and the model
chat_model_parameters = ChatModelParameters(max_tokens=200, temperature=0.5, top_k=1, top_p=1)
message = MessageEgpAPIBackendServerAPIModelsEgpModelsUserMessage(role="user", content="What is the capital of Canada?")
chat_completion = client.chat_completions.create(
model="gpt-4",
messages=[
message
], # messages is a list of historical messages in the conversation, with roles usually alternating between user, assistant and system
account_id=account_id,
model_parameters=model_parameters,
instructions="Answer the question like an elementary school teacher.",
)