> ## Documentation Index > Fetch the complete documentation index at: https://docs.gp.scale.com/llms.txt > Use this file to discover all available pages before exploring further. # Vector Stores > Create vector stores for semantic search and RAG-enhanced extraction from large or multi-file documents. Vector stores enable semantic search and RAG-enhanced extraction for large documents or multi-file processing. *** ## When to Use Vector Stores? ``` Is your document > 50 pages? ├─ Yes → Use vector store └─ No → Do you have multiple documents to query together? ├─ Yes → Use vector store └─ No → Extract directly from parse result ``` *** ## Vector Store Engines Dex supports two vector store engines. Prefer `SGP_VECTOR_STORE` for new implementations. | Engine | Best For | Embedding Options | Search Features | | -------------------- | --------------------------------------------------------- | ------------------------------------ | ---------------------------------------------------- | | `SGP_VECTOR_STORE` | **Recommended.** Document RAG, advanced search, reranking | Base embedding models | Semantic, lexical, hybrid search; optional reranking | | `SGP_KNOWLEDGE_BASE` | Legacy only | Base models or Models API deployment | Semantic search | You must update your SGP platform to support the SGP Vector Store engine. If `SGP_VECTOR_STORE` is not available, use `SGP_KNOWLEDGE_BASE` as a fallback. ### SGP Vector Store — Recommended The SGP Vector Store engine is the preferred vector store implementation. Use it for: * Document RAG workflows within Dex * Lexical (keyword) or hybrid search in addition to semantic search * Reranking search results for higher relevance **Embedding configuration:** Base embedding models only (e.g., `openai/text-embedding-3-large`, `sentence-transformers/all-mpnet-base-v2`). **Search features:** * **Query types:** `semantic`, `lexical`, or `hybrid` * **Reranking:** Optional `rerank_model` and `rerank_top_k` for improved result ordering ### SGP Knowledge Base — Legacy The SGP Knowledge Base engine is SGP's legacy vector store implementation. Its use is discouraged for new projects. Consider it only when: * You need integration with SGP data connectors for ingesting from external sources * You must use a custom embedding model deployed via the Models API (`embedding_type="models_api"`) * Your SGP platform does not yet support SGP Vector Store **Embedding configuration:** * **Base models** (`embedding_type="base"`): Use pre-configured embedding models such as `openai/text-embedding-3-large` * **Models API** (`embedding_type="models_api"`): Use a custom model deployment via `model_deployment_id` **Search:** Semantic search only. Reranking and query type options are not supported. `create_vector_store` returns `DexSGPKnowledgeBase` for `SGP_KNOWLEDGE_BASE` and `DexSGPVectorStore` for `SGP_VECTOR_STORE`. Use `DexSGPVectorStore.get_by_id` only for SGP Vector Store IDs—it will raise if given a Knowledge Base ID. *** ## Creating a Vector Store ```python theme={null} from dex_sdk.types import VectorStoreEngines # Create an SGP Vector Store (recommended) vs_store = await project.create_vector_store( name="Financial Documents Store", engine=VectorStoreEngines.SGP_VECTOR_STORE, embedding_model="openai/text-embedding-3-large", ) # Or create a Knowledge Base vector store (legacy; for data connector integration) kb_store = await project.create_vector_store( name="Financial Documents Store", engine=VectorStoreEngines.SGP_KNOWLEDGE_BASE, embedding_model="openai/text-embedding-3-large", ) ``` *** ## Adding Documents to Vector Store ```python theme={null} # Add parsed files to the vector store await vector_store.add_parse_results([parse_result.id]) print("Parse results added to vector store") ``` *** ## Semantic Search ```python theme={null} # Perform semantic search across all documents search_results = await vector_store.search( query="What is the total taxable income?", top_k=5, ) # Search within a specific file file_search_results = await vector_store.search_in_file( file_id=dex_file.id, query="What is the total taxable income?", top_k=5, filters=None, # Optional filters like {"file_id": "file_123"} ) # Access search results for chunk in search_results.chunks: print(f"Score: {chunk.score:.2f}") print(f"Content: {chunk.content[:100]}...") print(f"File: {chunk.file_id}, Page: {chunk.blocks[0].page_number if chunk.blocks else 'N/A'}") # Access additional metadata from the chunk if chunk.metadata: print(f"Metadata: {chunk.metadata}") ``` SGP Vector Store only: use `query_type` and reranking for improved results: ```python theme={null} # Hybrid search with reranking (SGP Vector Store only) search_results = await vs_store.search( query="What is the total taxable income?", top_k=10, query_type="hybrid", rerank_model="cohere.rerank-english-v3.0", rerank_top_k=5, ) ``` *** ## Custom Metadata Schema You can define custom metadata fields on your vector store and attach metadata to each parse result. This metadata is indexed and filterable when searching, so you can narrow results by document attributes such as department, document type, or priority. **1. Define the schema when creating the vector store** Specify `vector_store_metadata_schema` as a dict mapping field names to types: `"string"`, `"int"`, `"double"`, or `"boolean"`. ```python theme={null} vector_store = await project.create_vector_store( name="Documents by Department", engine=VectorStoreEngines.SGP_VECTOR_STORE, embedding_model="openai/text-embedding-3-large", vector_store_metadata_schema={ "department": "string", "priority": "int", "is_public": "boolean", }, ) ``` **2. Attach metadata when parsing** Pass `vector_store_metadata` in your parse job parameters. The metadata is stored with the parse result and indexed when you add it to the vector store. ```python theme={null} from dex_sdk.types import ParseEngine, IrisParseJobParams, IrisParseEngineOptions parse_result = await dex_file.parse( IrisParseJobParams( engine=ParseEngine.IRIS, options=IrisParseEngineOptions( layout="rt_detr_bce", text_ocr="openai/gpt-5.4", table_ocr="openai/gpt-5.4", confidence_threshold=0.5, img_method="description", ), vector_store_metadata={ "department": "legal", "priority": 1, "is_public": False, }, ) ) await vector_store.add_parse_results([parse_result.id]) ``` For custom parse results, pass `vector_store_metadata` when creating the parse result via `DexParseResult.from_custom_results`. **3. Filter on custom metadata when searching** Use the `filters` parameter in search. Filter format depends on the engine: | Operation | SGP Vector Store | SGP Knowledge Base | Example | | -------------------------- | ---------------- | ------------------ | --------------------------------------------------------------------------------- | | Equal | `$eq` | `eq` | `{"department": {"$eq": "legal"}}` | | Not equal | `$ne` | `ne` | `{"department": {"$ne": "archived"}}` | | Greater than | `$gt` | `gt` | `{"priority": {"$gt": 1}}` | | Greater than or equal | `$gte` | `gte` | `{"priority": {"$gte": 1}}` | | Less than | `$lt` | `lt` | `{"priority": {"$lt": 5}}` | | Less than or equal | `$lte` | `lte` | `{"priority": {"$lte": 5}}` | | In (list of values) | `$in` | `in` | `{"department": {"$in": ["legal", "finance"]}}` | | Not in (list of values) | `$nin` | `nin` | `{"department": {"$nin": ["archived"]}}` | | And (combine conditions) | `$and` | `and` | `{"$and": [{"department": {"$eq": "legal"}}, {"priority": {"$gte": 1}}]}` | | Or (any condition matches) | `$or` | `or` | `{"$or": [{"department": {"$eq": "legal"}}, {"department": {"$eq": "finance"}}]}` | | Not (negate condition) | `$not` | `not` | `{"department": {"$not": {"$eq": "archived"}}}` | ```python theme={null} # Filter by department (SGP Vector Store) search_results = await vector_store.search( query="contract terms", top_k=10, filters={"department": {"$eq": "legal"}}, ) # Filter by multiple values (SGP Vector Store) search_results = await vector_store.search( query="quarterly report", top_k=10, filters={"department": {"$in": ["legal", "finance"]}}, ) # Filter by numeric value (SGP Vector Store) search_results = await vector_store.search( query="policy", top_k=10, filters={"priority": {"$gte": 1}}, ) # Same filter on SGP Knowledge Base — operators have no $ prefix search_results = await knowledge_base.search( query="contract terms", top_k=10, filters={"department": {"eq": "legal"}}, ) ``` The schema must be defined when creating the vector store. Fields in `vector_store_metadata` that are not in `vector_store_metadata_schema` are not indexed and cannot be used for filtering. *** ## RAG-Enhanced Extraction Extract data using vector store context for improved accuracy on large documents: ```python theme={null} # Extract from vector store with RAG context extract_result = await vector_store.extract( extraction_schema=FinancialData, user_prompt="""Extract financial data from all documents in this store. Focus on quarterly income statements.""", model="openai/gpt-5.4", generate_citations=True, generate_confidence=True, ) # Access extracted data with citations result = extract_result.result for field_name, field in result.data.items(): print(f"{field_name}: {field.value} (confidence: {field.confidence:.2f})") if field.citations: for cite in field.citations: print(f" → From {cite.file_id}, page {cite.page}") ``` ### Pattern: RAG for Large Documents ```python theme={null} # 1. Create vector store vs = await project.create_vector_store( name="Store", engine=VectorStoreEngines.SGP_VECTOR_STORE, embedding_model="openai/text-embedding-3-large", ) # 2. Add documents await vs.add_parse_results([parse_result.id]) # 3. Extract with context result = await vs.extract( extraction_schema=Schema, user_prompt="Extract from all docs", model="openai/gpt-5.4", ) ``` *** ## Next Steps * **[Extract](extract)**: Extract structured data from parse results or vector stores * **[Chunking](chunking)**: Optimize chunking for vector store embeddings * **[API Reference](dex-sdk-api-reference)**: Complete SDK documentation