List Model Templates

GET

model-templates

import SGPClient from 'sgp';

const client = new SGPClient({
  apiKey: process.env['SGP_API_KEY'], // This is the default and can be omitted
});

async function main() {
  // Automatically fetches more pages as needed.
  for await (const modelTemplate of client.modelTemplates.list()) {
    console.log(modelTemplate.id);
  }
}

main();

{
  "items": [
    {
      "name": "<string>",
      "endpoint_type": "SYNC",
      "model_type": "COMPLETION",
      "vendor_configuration": {
        "vendor": "LAUNCH",
        "bundle_config": {
          "registry": "<string>",
          "image": "<string>",
          "tag": "<string>",
          "command": [
            "<string>"
          ],
          "env": {},
          "streaming_command": [
            "<string>"
          ],
          "readiness_initial_delay_seconds": 120,
          "healthcheck_route": "/readyz",
          "predict_route": "/predict",
          "streaming_predict_route": "/generate_streaming"
        },
        "endpoint_config": {
          "cpus": 3,
          "memory": "8Gi",
          "storage": "16Gi",
          "gpus": 0,
          "gpu_type": "nvidia-tesla-t4",
          "min_workers": 0,
          "max_workers": 1,
          "per_worker": 10,
          "endpoint_type": "ASYNC",
          "high_priority": false
        },
        "fine_tuning_job_bundle_config": {
          "registry": "<string>",
          "image": "<string>",
          "tag": "<string>",
          "command": [
            "<string>"
          ],
          "env": {},
          "mount_location": "/workspace/launch_specific/config.json",
          "training_dataset_schema_type": "GENERATION",
          "resources": {
            "cpus": 3,
            "memory": "8Gi",
            "storage": "16Gi",
            "gpus": 0,
            "gpu_type": "nvidia-tesla-t4"
          }
        }
      },
      "model_creation_parameters_schema": {
        "parameters": [
          {
            "name": "<string>",
            "type": "str",
            "description": "<string>",
            "required": true
          }
        ]
      },
      "model_request_parameters_schema": {
        "parameters": [
          {
            "name": "<string>",
            "type": "str",
            "description": "<string>",
            "required": true
          }
        ]
      },
      "endpoint_protocol": "SGP",
      "id": "<string>",
      "created_at": "2023-11-07T05:31:56Z",
      "account_id": "<string>",
      "created_by_user_id": "<string>"
    }
  ],
  "total_item_count": 123,
  "current_page": 123,
  "items_per_page": 123
}

Authorizations

x-api-key

string

header

required

Headers

x-selected-account-id

string | null

Query Parameters

page

integer

default:

Page number for pagination to be returned by the given endpoint. Starts at page 1

Required range: x > 1

limit

integer

default:

100

Maximum number of artifacts to be returned by the given endpoint. Defaults to 100 and cannot be greater than 10k.

Required range: 1 < x < 10000

account_id

string | null

Response

200

application/json

Successful Response

items

object[]

required

The data returned for the current page.

items.name

string

required

items.endpoint_type

enum<string>

required

An enum representing the different types of model endpoint types supported.

Attributes: SYNC: Denotes that the model endpoint type is sync. ASYNC: Denotes that the model endpoint type is async. STREAMING: Denotes that the model endpoint type is streaming. BATCH: Denotes that the model endpoint type is batch.

Available options:

SYNC,

ASYNC,

STREAMING,

BATCH

items.model_type

enum<string>

required

An enum representing the different types of models supported.

Attributes: COMPLETION: Denotes that the model type is completion. CHAT_COMPLETION: Denotes that the model type is chat completion. AGENT: Denotes that the model type is agent. EMBEDDING: Denotes that the model type is embedding. RERANKING: Denotes that the model type is reranking. GENERIC: Denotes that the model type is generic.

Available options:

COMPLETION,

CHAT_COMPLETION,

AGENT,

EMBEDDING,

RERANKING,

GENERIC

items.vendor_configuration

object

required

Configuration for launching a model using the Launch service which is an internal and self-hosted service developed by Scale that deploys models on Kubernetes.

Attributes: vendor: The vendor of the model template bundle_config: The bundle configuration of the model template endpoint_config: The endpoint configuration of the model template

items.vendor_configuration.bundle_config

object

required

items.vendor_configuration.vendor

enum<string>

default:

LAUNCH

Available options:

LAUNCH

items.vendor_configuration.endpoint_config

object

items.vendor_configuration.endpoint_config.cpus

integer

default:

items.vendor_configuration.endpoint_config.memory

string

default:

8Gi

items.vendor_configuration.endpoint_config.storage

string

default:

16Gi

items.vendor_configuration.endpoint_config.gpus

integer

default:

items.vendor_configuration.endpoint_config.gpu_type

enum<string>

Available options:

nvidia-tesla-t4,

nvidia-ampere-a10,

nvidia-ampere-a100,

nvidia-ampere-a100e,

nvidia-hopper-h100,

nvidia-hopper-h100-1g20gb,

nvidia-hopper-h100-3g40gb

items.vendor_configuration.endpoint_config.min_workers

integer

default:

items.vendor_configuration.endpoint_config.max_workers

integer

default:

items.vendor_configuration.endpoint_config.per_worker

integer

default:

The maximum number of concurrent requests that an individual worker can service. Launch automatically scales the number of workers for the endpoint so that each worker is processing per_worker requests, subject to the limits defined by min_workers and max_workers.

If the average number of concurrent requests per worker is lower than per_worker, then the number of workers will be reduced. - Otherwise, if the average number of concurrent requests per worker is higher than per_worker, then the number of workers will be increased to meet the elevated traffic.

Here is our recommendation for computing per_worker:

Compute min_workers and max_workers per your minimum and maximum throughput requirements. 2. Determine a value for the maximum number of concurrent requests in the workload. Divide this number by max_workers. Doing this ensures that the number of workers will "climb" to max_workers.

items.vendor_configuration.endpoint_config.endpoint_type

enum<string>

An enum representing the different types of model endpoint types supported.

Available options:

SYNC,

ASYNC,

STREAMING,

BATCH

items.vendor_configuration.endpoint_config.high_priority

boolean

default:

false

items.vendor_configuration.fine_tuning_job_bundle_config

object

items.id

string

required

The unique identifier of the entity.

items.created_at

string

required

The date and time when the entity was created in ISO format.

items.account_id

string

required

The ID of the account that owns the given entity.

items.created_by_user_id

string

required

The user who originally created the entity.

items.model_creation_parameters_schema

object

The field names and types of available parameter fields which may be specified during model creation

items.model_request_parameters_schema

object

The field names and types of available parameter fields which may be specified in a model execution API's model_request_parameters field.

items.endpoint_protocol

enum<string>

The name of the calling convention expected by the Launch model endpoint

Available options:

SGP,

COHERE,

VLLM

total_item_count

integer

required

The total number of items of the query

current_page

integer

required

The current page number.

items_per_page

integer

required

The number of items per page.

Update Knowledge Base Data Source Create Model Template

import SGPClient from 'sgp';

const client = new SGPClient({
  apiKey: process.env['SGP_API_KEY'], // This is the default and can be omitted
});

async function main() {
  // Automatically fetches more pages as needed.
  for await (const modelTemplate of client.modelTemplates.list()) {
    console.log(modelTemplate.id);
  }
}

main();

{
  "items": [
    {
      "name": "<string>",
      "endpoint_type": "SYNC",
      "model_type": "COMPLETION",
      "vendor_configuration": {
        "vendor": "LAUNCH",
        "bundle_config": {
          "registry": "<string>",
          "image": "<string>",
          "tag": "<string>",
          "command": [
            "<string>"
          ],
          "env": {},
          "streaming_command": [
            "<string>"
          ],
          "readiness_initial_delay_seconds": 120,
          "healthcheck_route": "/readyz",
          "predict_route": "/predict",
          "streaming_predict_route": "/generate_streaming"
        },
        "endpoint_config": {
          "cpus": 3,
          "memory": "8Gi",
          "storage": "16Gi",
          "gpus": 0,
          "gpu_type": "nvidia-tesla-t4",
          "min_workers": 0,
          "max_workers": 1,
          "per_worker": 10,
          "endpoint_type": "ASYNC",
          "high_priority": false
        },
        "fine_tuning_job_bundle_config": {
          "registry": "<string>",
          "image": "<string>",
          "tag": "<string>",
          "command": [
            "<string>"
          ],
          "env": {},
          "mount_location": "/workspace/launch_specific/config.json",
          "training_dataset_schema_type": "GENERATION",
          "resources": {
            "cpus": 3,
            "memory": "8Gi",
            "storage": "16Gi",
            "gpus": 0,
            "gpu_type": "nvidia-tesla-t4"
          }
        }
      },
      "model_creation_parameters_schema": {
        "parameters": [
          {
            "name": "<string>",
            "type": "str",
            "description": "<string>",
            "required": true
          }
        ]
      },
      "model_request_parameters_schema": {
        "parameters": [
          {
            "name": "<string>",
            "type": "str",
            "description": "<string>",
            "required": true
          }
        ]
      },
      "endpoint_protocol": "SGP",
      "id": "<string>",
      "created_at": "2023-11-07T05:31:56Z",
      "account_id": "<string>",
      "created_by_user_id": "<string>"
    }
  ],
  "total_item_count": 123,
  "current_page": 123,
  "items_per_page": 123
}

Knowledge Bases

Chunks

Agents

Completions

Chat Completions

Models

Users

Accounts

Organizations

Question Sets

Evaluations

Evaluation Configs

Evaluation Datasets

Studio Projects

Application Specs

Questions

Knowledge Base Data Sources

Model Templates V3 (Beta)

Model server

API Reference

Fine Tuning Jobs V3 (Beta)

Training Datasets V3 (Beta)

package deployments

Beta

Applications

ChatThreads

Interactions

MonitoringDashboard

Chat Themes

account groups

account

Guardrails

List Model Templates

Description

Details

Authorizations

Headers

Query Parameters

Response