Vector Stores

Vector stores enable semantic search and RAG-enhanced extraction for large documents or multi-file processing.

When to Use Vector Stores?

Is your document > 50 pages?
├─ Yes → Use vector store
└─ No → Do you have multiple documents to query together?
   ├─ Yes → Use vector store
   └─ No → Extract directly from parse result

Vector Store Engines

Dex supports two vector store engines. Prefer SGP_VECTOR_STORE for new implementations.

Engine	Best For	Embedding Options	Search Features
`SGP_VECTOR_STORE`	Recommended. Document RAG, advanced search, reranking	Base embedding models	Semantic, lexical, hybrid search; optional reranking
`SGP_KNOWLEDGE_BASE`	Legacy only	Base models or Models API deployment	Semantic search

You must update your SGP platform to support the SGP Vector Store engine. If SGP_VECTOR_STORE is not available, use SGP_KNOWLEDGE_BASE as a fallback.

SGP Vector Store — Recommended

The SGP Vector Store engine is the preferred vector store implementation. Use it for:

Document RAG workflows within Dex
Lexical (keyword) or hybrid search in addition to semantic search
Reranking search results for higher relevance

Embedding configuration: Base embedding models only (e.g., openai/text-embedding-3-large, sentence-transformers/all-mpnet-base-v2). Search features:

Query types: semantic, lexical, or hybrid
Reranking: Optional rerank_model and rerank_top_k for improved result ordering

SGP Knowledge Base — Legacy

The SGP Knowledge Base engine is SGP’s legacy vector store implementation. Its use is discouraged for new projects. Consider it only when:

You need integration with SGP data connectors for ingesting from external sources
You must use a custom embedding model deployed via the Models API (embedding_type="models_api")
Your SGP platform does not yet support SGP Vector Store

Embedding configuration:

Base models (embedding_type="base"): Use pre-configured embedding models such as openai/text-embedding-3-large
Models API (embedding_type="models_api"): Use a custom model deployment via model_deployment_id

Search: Semantic search only. Reranking and query type options are not supported.

create_vector_store returns DexSGPKnowledgeBase for SGP_KNOWLEDGE_BASE and DexSGPVectorStore for SGP_VECTOR_STORE. Use DexSGPVectorStore.get_by_id only for SGP Vector Store IDs—it will raise if given a Knowledge Base ID.

Creating a Vector Store

from dex_sdk.types import VectorStoreEngines

# Create an SGP Vector Store (recommended)
vs_store = await project.create_vector_store(
    name="Financial Documents Store",
    engine=VectorStoreEngines.SGP_VECTOR_STORE,
    embedding_model="openai/text-embedding-3-large",
)

# Or create a Knowledge Base vector store (legacy; for data connector integration)
kb_store = await project.create_vector_store(
    name="Financial Documents Store",
    engine=VectorStoreEngines.SGP_KNOWLEDGE_BASE,
    embedding_model="openai/text-embedding-3-large",
)

Adding Documents to Vector Store

# Add parsed files to the vector store
await vector_store.add_parse_results([parse_result.id])
print("Parse results added to vector store")

Semantic Search

# Perform semantic search across all documents
search_results = await vector_store.search(
    query="What is the total taxable income?",
    top_k=5,
)

# Search within a specific file
file_search_results = await vector_store.search_in_file(
    file_id=dex_file.id,
    query="What is the total taxable income?",
    top_k=5,
    filters=None,  # Optional filters like {"file_id": "file_123"}
)

# Access search results
for chunk in search_results.chunks:
    print(f"Score: {chunk.score:.2f}")
    print(f"Content: {chunk.content[:100]}...")
    print(f"File: {chunk.file_id}, Page: {chunk.blocks[0].page_number if chunk.blocks else 'N/A'}")

    # Access additional metadata from the chunk
    if chunk.metadata:
        print(f"Metadata: {chunk.metadata}")

SGP Vector Store only: use query_type and reranking for improved results:

# Hybrid search with reranking (SGP Vector Store only)
search_results = await vs_store.search(
    query="What is the total taxable income?",
    top_k=10,
    query_type="hybrid",
    rerank_model="cohere.rerank-english-v3.0",
    rerank_top_k=5,
)

Custom Metadata Schema

You can define custom metadata fields on your vector store and attach metadata to each parse result. This metadata is indexed and filterable when searching, so you can narrow results by document attributes such as department, document type, or priority. 1. Define the schema when creating the vector store Specify vector_store_metadata_schema as a dict mapping field names to types: "string", "int", "double", or "boolean".

vector_store = await project.create_vector_store(
    name="Documents by Department",
    engine=VectorStoreEngines.SGP_VECTOR_STORE,
    embedding_model="openai/text-embedding-3-large",
    vector_store_metadata_schema={
        "department": "string",
        "priority": "int",
        "is_public": "boolean",
    },
)

2. Attach metadata when parsing Pass vector_store_metadata in your parse job parameters. The metadata is stored with the parse result and indexed when you add it to the vector store.

from dex_sdk.types import ParseEngine, IrisParseJobParams, IrisParseEngineOptions

parse_result = await dex_file.parse(
    IrisParseJobParams(
        engine=ParseEngine.IRIS,
        options=IrisParseEngineOptions(
            layout="rt_detr_bce",
            text_ocr="openai/gpt-5.4",
            table_ocr="openai/gpt-5.4",
            confidence_threshold=0.5,
            img_method="description",
        ),
        vector_store_metadata={
            "department": "legal",
            "priority": 1,
            "is_public": False,
        },
    )
)
await vector_store.add_parse_results([parse_result.id])

For custom parse results, pass vector_store_metadata when creating the parse result via DexParseResult.from_custom_results. 3. Filter on custom metadata when searching Use the filters parameter in search. Filter format depends on the engine:

Operation	SGP Vector Store	SGP Knowledge Base	Example
Equal	`$eq`	`eq`	`{"department": {"$eq": "legal"}}`
Not equal	`$ne`	`ne`	`{"department": {"$ne": "archived"}}`
Greater than	`$gt`	`gt`	`{"priority": {"$gt": 1}}`
Greater than or equal	`$gte`	`gte`	`{"priority": {"$gte": 1}}`
Less than	`$lt`	`lt`	`{"priority": {"$lt": 5}}`
Less than or equal	`$lte`	`lte`	`{"priority": {"$lte": 5}}`
In (list of values)	`$in`	`in`	`{"department": {"$in": ["legal", "finance"]}}`
Not in (list of values)	`$nin`	`nin`	`{"department": {"$nin": ["archived"]}}`
And (combine conditions)	`$and`	`and`	`{"$and": [{"department": {"$eq": "legal"}}, {"priority": {"$gte": 1}}]}`
Or (any condition matches)	`$or`	`or`	`{"$or": [{"department": {"$eq": "legal"}}, {"department": {"$eq": "finance"}}]}`
Not (negate condition)	`$not`	`not`	`{"department": {"$not": {"$eq": "archived"}}}`

# Filter by department (SGP Vector Store)
search_results = await vector_store.search(
    query="contract terms",
    top_k=10,
    filters={"department": {"$eq": "legal"}},
)

# Filter by multiple values (SGP Vector Store)
search_results = await vector_store.search(
    query="quarterly report",
    top_k=10,
    filters={"department": {"$in": ["legal", "finance"]}},
)

# Filter by numeric value (SGP Vector Store)
search_results = await vector_store.search(
    query="policy",
    top_k=10,
    filters={"priority": {"$gte": 1}},
)

# Same filter on SGP Knowledge Base — operators have no $ prefix
search_results = await knowledge_base.search(
    query="contract terms",
    top_k=10,
    filters={"department": {"eq": "legal"}},
)

The schema must be defined when creating the vector store. Fields in vector_store_metadata that are not in vector_store_metadata_schema are not indexed and cannot be used for filtering.

RAG-Enhanced Extraction

Extract data using vector store context for improved accuracy on large documents:

# Extract from vector store with RAG context
extract_result = await vector_store.extract(
    extraction_schema=FinancialData,
    user_prompt="""Extract financial data from all documents in this store.
    Focus on quarterly income statements.""",
    model="openai/gpt-5.4",
    generate_citations=True,
    generate_confidence=True,
)

# Access extracted data with citations
result = extract_result.result
for field_name, field in result.data.items():
    print(f"{field_name}: {field.value} (confidence: {field.confidence:.2f})")
    if field.citations:
        for cite in field.citations:
            print(f"  → From {cite.file_id}, page {cite.page}")

Pattern: RAG for Large Documents

# 1. Create vector store
vs = await project.create_vector_store(
    name="Store",
    engine=VectorStoreEngines.SGP_VECTOR_STORE,
    embedding_model="openai/text-embedding-3-large",
)

# 2. Add documents
await vs.add_parse_results([parse_result.id])

# 3. Extract with context
result = await vs.extract(
    extraction_schema=Schema,
    user_prompt="Extract from all docs",
    model="openai/gpt-5.4",
)

Next Steps

Extract: Extract structured data from parse results or vector stores
Chunking: Optimize chunking for vector store embeddings
API Reference: Complete SDK documentation

Getting Started

Document Understanding

OCR

Workflows

Training

When to Use Vector Stores?

Vector Store Engines

SGP Vector Store — Recommended

SGP Knowledge Base — Legacy

Creating a Vector Store

Adding Documents to Vector Store

Semantic Search

Custom Metadata Schema

RAG-Enhanced Extraction

Pattern: RAG for Large Documents

Next Steps

​When to Use Vector Stores?

​Vector Store Engines

​SGP Vector Store — Recommended

​SGP Knowledge Base — Legacy

​Creating a Vector Store

​Adding Documents to Vector Store

​Semantic Search

​Custom Metadata Schema

​RAG-Enhanced Extraction

​Pattern: RAG for Large Documents

​Next Steps

When to Use Vector Stores?

Vector Store Engines

SGP Vector Store — Recommended

SGP Knowledge Base — Legacy

Creating a Vector Store

Adding Documents to Vector Store

Semantic Search

Custom Metadata Schema

RAG-Enhanced Extraction

Pattern: RAG for Large Documents

Next Steps