Skip to main content

What is a knowledge base?

A knowledge base organizes ingested artifacts and their searchable chunks. In the V2 knowledge-base model used by the APIs in this guide, ingestion is managed through knowledge-base uploads. An upload can ingest files directly or use a connection to a data source. How is data indexed? An upload extracts content, splits it into chunks, and indexes the chunks using the knowledge base’s embedding configuration. You can then query those chunks for retrieval. How are data sources connected? A data-source connection identifies the source and upload configuration used to ingest data into the knowledge base. You can run one-time or scheduled uploads from that connection.
The legacy V1 knowledge-base, vector-store, and data-connector APIs use a different architecture and are not backward compatible with this model.

Creating a new knowledge base in the UI

To create a new knowledge base, you need to:
  1. Define a Knowledge Base Name.
  2. Select an Embedding Model.
  3. Select a Data Source.
  4. Configure Upload Source.

Knowledge Base Name

This will be used to locate your knowledge base after you create it.

Embedding Model

Embedding models are ML models that convert text into a numerical representation. The representation is then used to compare and match text data. You can select an existing model on the SGP Platform or install a new model and use the new model.

Data Source

Data sources provide the data for knowledge bases to ingest. When a knowledge base upload is created from a data source, it will read data from it, extract text from relevant files, split it into chunks, embed the chunks, and store the embeddings in a vector database for future retrieval.

Configure Upload

Upload configurations define how the data source is split into chunks. The chunking configuration affects the relevance of the content retrieved from a knowledge base when the LLM is used to embed content. You can choose to go with the default configuration or choose your own Chunk Strategy, Size, Overlap, and Separator.

Knowledge Base API

The starter code below creates an SGP knowledge base using the Create Knowledge Base endpoint. Set SGP_API_BASE_URL to your deployment’s API URL and API_KEY to the key from Authentication, then choose a name and an available embedding model for your knowledge base.
Python
You should see a response like the following, which will show your knowledge base ID. {"knowledge_base_id":"clk123456789"} Keep this value—you will need it later when getting information about your knowledge base, creating uploads and data-source connections, and more.