Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.gp.scale.com/llms.txt

Use this file to discover all available pages before exploring further.

Dex provides flexible chunking for parsed documents. You can apply chunking in two ways: Dex rechunking (parse once, then rechunk with different strategies) or engine-specific chunking during the initial parse (Reducto only).

​
Dex Chunking Strategies

Dex offers four chunking strategies that work with any parse result via rechunking. Use these when you need consistent, configurable chunk boundaries across different parse engines.
StrategyDescriptionBest ForEmbedding Suitability
token_sizeSplits by token count using a tokenizer (e.g., tiktoken)LLM APIs with token limits, cost optimizationExcellent
recursiveRecursively splits using separators (paragraphs → sentences → words)Articles, documentation, RAG systemsExcellent
by_pageSplits by page boundaries, grouping complete pagesLegal documents, forms, reportsMay be large
by_sectionSplits by section headers (e.g., markdown #, ##, ###)Technical manuals, wikis, academic papersGood

​
Parse Once, Rechunk as Needed

Parse with disabled chunking (Reducto only) to get raw blocks, then apply Dex chunking strategies. This lets you experiment with different strategies without re-parsing. 
from dex_sdk.types import (
    ParseEngine,
    ReductoParseJobParams,
    ReductoChunkingMethod,
    ReductoChunkingOptions,
    ReductoParseEngineOptions,
)
from dex_core.models.chunking import (
    TokenSizeChunkingOptions,
    RecursiveChunkingOptions,
    PageChunkingOptions,
    SectionChunkingOptions,
)

# Step 1: Parse with disabled chunking (Reducto only)
parse_result = await dex_file.parse(
    ReductoParseJobParams(
        engine=ParseEngine.REDUCTO,
        options=ReductoParseEngineOptions(
            chunking=ReductoChunkingOptions(chunk_mode=ReductoChunkingMethod.DISABLED)
        ),
    )
)

# Step 2: Rechunk with Dex strategies
# Token-based (for LLM APIs, embedding models)
rechunked = await parse_result.rechunk(
    TokenSizeChunkingOptions(
        chunk_size=512,
        chunk_overlap=50,
        encoding_name="cl100k_base",
    )
)

# Recursive (preserves paragraphs and sentences)
rechunked = await parse_result.rechunk(
    RecursiveChunkingOptions(
        chunk_size=1000,
        chunk_overlap=200,
    )
)

# Page-based (legal docs, forms)
rechunked = await parse_result.rechunk(
    PageChunkingOptions(pages_per_chunk=1)
)

# Section-based (structured documents with headers)
rechunked = await parse_result.rechunk(
    SectionChunkingOptions(
        section_headers=None,  # Auto-detect from block types
        include_header_in_chunk=True,
    )
)

​
Async Rechunking

For long-running documents, start the rechunk job and poll for completion: 
# Start rechunk job (returns immediately)
job = await parse_result.start_rechunk_job(
    TokenSizeChunkingOptions(chunk_size=512, chunk_overlap=50)
)

# Wait for completion and get rechunked result
rechunked = await job.get_rechunked_result()

​
Chunking with Iris

The Iris parse engine supports Dex chunking strategies via chunking_options in IrisParseJobParams, or rechunking after the parse.

​
Option 1: Configure Chunking in Parse Job

Pass chunking_options when creating the parse job. Chunking is applied automatically after Iris parses the document: 
from dex_sdk.types import IrisParseJobParams, IrisParseEngineOptions
from dex_core.models.chunking import (
    TokenSizeChunkingOptions,
    RecursiveChunkingOptions,
    PageChunkingOptions,
    SectionChunkingOptions,
)

# Parse with Iris and apply chunking in one step
parse_result = await dex_file.parse(
    IrisParseJobParams(
        options=IrisParseEngineOptions(),
        chunking_options=TokenSizeChunkingOptions(
            chunk_size=512,
            chunk_overlap=50,
        ),
    )
)

# Or use other strategies
parse_result = await dex_file.parse(
    IrisParseJobParams(
        options=IrisParseEngineOptions(),
        chunking_options=RecursiveChunkingOptions(
            chunk_size=1000,
            chunk_overlap=200,
        ),
    )
)

​
Option 2: Rechunk After Parsing

Parse first, then rechunk to experiment with different strategies without re-parsing: 
# Step 1: Parse with Iris (no chunking)
parse_result = await dex_file.parse(
    IrisParseJobParams(options=IrisParseEngineOptions())
)

# Step 2: Rechunk with Dex strategies
rechunked = await parse_result.rechunk(
    TokenSizeChunkingOptions(chunk_size=512, chunk_overlap=50)
)
# Or: RecursiveChunkingOptions, PageChunkingOptions, SectionChunkingOptions

​
Reducto Chunking (Parse-Time)

When using the Reducto parse engine, you can chunk during the initial parse instead of rechunking. Reducto’s methods are layout-aware and use document structure.
MethodChunk SizeBest ForEmbeddingLocation Tracking
VARIABLEAuto (optimal)General use, embeddingsExcellentGood
BLOCKSmall (~100-500 chars)Precise locations, UI overlaysToo smallExcellent
SECTIONMedium (~1000-3000 chars)Structured documentsGoodGood
PAGELarge (full page)Page-oriented docsMay be largeExcellent
PAGE_SECTIONSMedium-LargeHybrid needsGoodGood
DISABLEDVery large (entire doc)Special casesToo largeExcellent
Recommendation: Use VARIABLE for most cases, especially with embeddings. 
from dex_sdk.types import (
    ParseEngine,
    ReductoParseJobParams,
    ReductoChunkingMethod,
    ReductoChunkingOptions,
    ReductoParseEngineOptions,
)

# Reducto variable chunking (parse-time)
parse_result = await dex_file.parse(
    ReductoParseJobParams(
        engine=ParseEngine.REDUCTO,
        options=ReductoParseEngineOptions(
            chunking=ReductoChunkingOptions(
                chunk_mode=ReductoChunkingMethod.VARIABLE,
                chunk_size=None,
            )
        ),
    )
)

# Reducto block chunking for precise location tracking
parse_result = await dex_file.parse(
    ReductoParseJobParams(
        engine=ParseEngine.REDUCTO,
        options=ReductoParseEngineOptions(
            chunking=ReductoChunkingOptions(chunk_mode=ReductoChunkingMethod.BLOCK)
        ),
    )
)

​
Pattern: Retry with Different Chunking

# Try 1: Variable chunking
result1 = await file.parse(
    ReductoParseJobParams(
        engine=ParseEngine.REDUCTO,
        options=ReductoParseEngineOptions(
            chunking=ReductoChunkingOptions(
                chunk_mode=ReductoChunkingMethod.VARIABLE,
            )
        ),
    )
)

# If not satisfactory, try block chunking
result2 = await file.parse(
    ReductoParseJobParams(
        engine=ParseEngine.REDUCTO,
        options=ReductoParseEngineOptions(
            chunking=ReductoChunkingOptions(
                chunk_mode=ReductoChunkingMethod.BLOCK,
            )
        ),
    )
)

​
Chunking Decision Tree

Use Dex token_size when:
  • Working with LLM APIs that have token limits
  • Embedding models with specific token limits
  • Cost optimization
Use Dex recursive when:
  • General document chunking for RAG
  • Preserving paragraphs and sentences
  • Articles, blog posts, documentation
Use Dex by_page when:
  • Legal documents, forms, reports
  • Page references matter
  • Page structure should be preserved
Use Dex by_section when:
  • Documents have clear section headers
  • Technical manuals, wikis, academic papers
  • Semantic coherence within topics
Use Reducto VARIABLE when:
  • Parsing with Reducto and want layout-aware chunking
  • General document processing
  • You want optimal chunk sizes from the parser
Use Reducto BLOCK when:
  • Need precise bounding box information
  • Building UI overlays on documents
  • Not using for embeddings
Use Reducto DISABLED when:
  • You plan to rechunk with Dex strategies
  • Parse once, experiment with multiple chunking approaches

​
Which Chunking Method?

What's your primary goal?
├─ Embeddings/Semantic Search → VARIABLE
├─ Precise bounding boxes/UI → BLOCK
├─ Page-by-page processing → PAGE
└─ Structured document navigation → SECTION

​
Next Steps

  • Vector Stores: Add chunks to vector stores for semantic search
  • Extract: Extract structured data from parse results
  • Parse: Parse engine options and configuration