Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.gp.scale.com/llms.txt

Use this file to discover all available pages before exploring further.

Vector stores enable semantic search and RAG-enhanced extraction for large documents or multi-file processing.

​
When to Use Vector Stores?

Is your document > 50 pages?
├─ Yes → Use vector store
└─ No → Do you have multiple documents to query together?
   ├─ Yes → Use vector store
   └─ No → Extract directly from parse result

​
Vector Store Engines

Dex supports two vector store engines. Prefer SGP_VECTOR_STORE for new implementations.
EngineBest ForEmbedding OptionsSearch Features
SGP_VECTOR_STORERecommended. Document RAG, advanced search, rerankingBase embedding modelsSemantic, lexical, hybrid search; optional reranking
SGP_KNOWLEDGE_BASELegacy onlyBase models or Models API deploymentSemantic search
You must update your SGP platform to support the SGP Vector Store engine. If SGP_VECTOR_STORE is not available, use SGP_KNOWLEDGE_BASE as a fallback.
The SGP Vector Store engine is the preferred vector store implementation. Use it for:
  • Document RAG workflows within Dex
  • Lexical (keyword) or hybrid search in addition to semantic search
  • Reranking search results for higher relevance
Embedding configuration: Base embedding models only (e.g., openai/text-embedding-3-large, sentence-transformers/all-mpnet-base-v2). Search features:
  • Query types: semantic, lexical, or hybrid
  • Reranking: Optional rerank_model and rerank_top_k for improved result ordering

​
SGP Knowledge Base — Legacy

The SGP Knowledge Base engine is SGP’s legacy vector store implementation. Its use is discouraged for new projects. Consider it only when:
  • You need integration with SGP data connectors for ingesting from external sources
  • You must use a custom embedding model deployed via the Models API (embedding_type="models_api")
  • Your SGP platform does not yet support SGP Vector Store
Embedding configuration:
  • Base models (embedding_type="base"): Use pre-configured embedding models such as openai/text-embedding-3-large
  • Models API (embedding_type="models_api"): Use a custom model deployment via model_deployment_id
Search: Semantic search only. Reranking and query type options are not supported.
create_vector_store returns DexSGPKnowledgeBase for SGP_KNOWLEDGE_BASE and DexSGPVectorStore for SGP_VECTOR_STORE. Use DexSGPVectorStore.get_by_id only for SGP Vector Store IDs—it will raise if given a Knowledge Base ID.

​
Creating a Vector Store

from dex_sdk.types import VectorStoreEngines

# Create an SGP Vector Store (recommended)
vs_store = await project.create_vector_store(
    name="Financial Documents Store",
    engine=VectorStoreEngines.SGP_VECTOR_STORE,
    embedding_model="openai/text-embedding-3-large",
)

# Or create a Knowledge Base vector store (legacy; for data connector integration)
kb_store = await project.create_vector_store(
    name="Financial Documents Store",
    engine=VectorStoreEngines.SGP_KNOWLEDGE_BASE,
    embedding_model="openai/text-embedding-3-large",
)

​
Adding Documents to Vector Store

# Add parsed files to the vector store
await vector_store.add_parse_results([parse_result.id])
print("Parse results added to vector store")

# Perform semantic search across all documents
search_results = await vector_store.search(
    query="What is the total taxable income?",
    top_k=5,
)

# Search within a specific file
file_search_results = await vector_store.search_in_file(
    file_id=dex_file.id,
    query="What is the total taxable income?",
    top_k=5,
    filters=None,  # Optional filters like {"file_id": "file_123"}
)

# Access search results
for chunk in search_results.chunks:
    print(f"Score: {chunk.score:.2f}")
    print(f"Content: {chunk.content[:100]}...")
    print(f"File: {chunk.file_id}, Page: {chunk.blocks[0].page_number if chunk.blocks else 'N/A'}")

    # Access additional metadata from the chunk
    if chunk.metadata:
        print(f"Metadata: {chunk.metadata}")
SGP Vector Store only: use query_type and reranking for improved results: 
# Hybrid search with reranking (SGP Vector Store only)
search_results = await vs_store.search(
    query="What is the total taxable income?",
    top_k=10,
    query_type="hybrid",
    rerank_model="cohere.rerank-english-v3.0",
    rerank_top_k=5,
)

​
Custom Metadata Schema

You can define custom metadata fields on your vector store and attach metadata to each parse result. This metadata is indexed and filterable when searching, so you can narrow results by document attributes such as department, document type, or priority. 1. Define the schema when creating the vector store Specify vector_store_metadata_schema as a dict mapping field names to types: "string", "int", "double", or "boolean". 
vector_store = await project.create_vector_store(
    name="Documents by Department",
    engine=VectorStoreEngines.SGP_VECTOR_STORE,
    embedding_model="openai/text-embedding-3-large",
    vector_store_metadata_schema={
        "department": "string",
        "priority": "int",
        "is_public": "boolean",
    },
)
2. Attach metadata when parsing Pass vector_store_metadata in your parse job parameters. The metadata is stored with the parse result and indexed when you add it to the vector store. 
from dex_sdk.types import (
    ParseEngine,
    ReductoParseJobParams,
    ReductoChunkingMethod,
    ReductoChunkingOptions,
    ReductoParseEngineOptions,
)

parse_result = await dex_file.parse(
    ReductoParseJobParams(
        engine=ParseEngine.REDUCTO,
        options=ReductoParseEngineOptions(
            chunking=ReductoChunkingOptions(chunk_mode=ReductoChunkingMethod.VARIABLE)
        ),
        vector_store_metadata={"department": "legal", "priority": 1, "is_public": False},
    )
)
await vector_store.add_parse_results([parse_result.id])
For custom parse results, pass vector_store_metadata when creating the parse result via DexParseResult.from_custom_results. 3. Filter on custom metadata when searching Use the filters parameter in search. Filter format depends on the engine:
OperationSGP Vector StoreSGP Knowledge BaseExample
Equal$eqeq{"department": {"$eq": "legal"}}
Not equal$nene{"department": {"$ne": "archived"}}
Greater than$gtgt{"priority": {"$gt": 1}}
Greater than or equal$gtegte{"priority": {"$gte": 1}}
Less than$ltlt{"priority": {"$lt": 5}}
Less than or equal$ltelte{"priority": {"$lte": 5}}
In (list of values)$inin{"department": {"$in": ["legal", "finance"]}}
Not in (list of values)$ninnin{"department": {"$nin": ["archived"]}}
And (combine conditions)$andand{"$and": [{"department": {"$eq": "legal"}}, {"priority": {"$gte": 1}}]}
Or (any condition matches)$oror{"$or": [{"department": {"$eq": "legal"}}, {"department": {"$eq": "finance"}}]}
Not (negate condition)$notnot{"department": {"$not": {"$eq": "archived"}}}
# Filter by department (SGP Vector Store)
search_results = await vector_store.search(
    query="contract terms",
    top_k=10,
    filters={"department": {"$eq": "legal"}},
)

# Filter by multiple values (SGP Vector Store)
search_results = await vector_store.search(
    query="quarterly report",
    top_k=10,
    filters={"department": {"$in": ["legal", "finance"]}},
)

# Filter by numeric value (SGP Vector Store)
search_results = await vector_store.search(
    query="policy",
    top_k=10,
    filters={"priority": {"$gte": 1}},
)
The schema must be defined when creating the vector store. Fields in vector_store_metadata that are not in vector_store_metadata_schema are not indexed and cannot be used for filtering.

​
RAG-Enhanced Extraction

Extract data using vector store context for improved accuracy on large documents: 
# Extract from vector store with RAG context
extract_result = await vector_store.extract(
    extraction_schema=FinancialData,
    user_prompt="""Extract financial data from all documents in this store.
    Focus on quarterly income statements.""",
    model="openai/gpt-4o",
    generate_citations=True,
    generate_confidence=True,
)

# Access extracted data with citations
result = extract_result.result
for field_name, field in result.data.items():
    print(f"{field_name}: {field.value} (confidence: {field.confidence:.2f})")
    if field.citations:
        for cite in field.citations:
            print(f"  → From {cite.file_id}, page {cite.page}")

​
Pattern: RAG for Large Documents

# 1. Create vector store
vs = await project.create_vector_store(
    name="Store",
    engine=VectorStoreEngines.SGP_VECTOR_STORE,
    embedding_model="openai/text-embedding-3-large",
)

# 2. Add documents
await vs.add_parse_results([parse_result.id])

# 3. Extract with context
result = await vs.extract(
    extraction_schema=Schema,
    user_prompt="Extract from all docs",
    model="openai/gpt-4o",
)

​
Next Steps

  • Extract: Extract structured data from parse results or vector stores
  • Chunking: Optimize chunking for vector store embeddings
  • API Reference: Complete SDK documentation