> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gp.scale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Chunking

> Choose chunking strategies for parsed documents, Dex rechunking or Reducto parse-time chunking.

Dex provides flexible chunking for parsed documents. You can apply chunking in two ways: **Dex rechunking** (parse once, then rechunk with different strategies) or **engine-specific chunking** during the initial parse (Reducto only).

***

## IRIS v2 Chunk Structure

IRIS v2 always returns a `chunks` array in the parse result. `chunk_mode: disabled` is a Reducto-only concept (soon to be deprecated) that does not exist in IRIS v2.
Chunk count and size are driven by **how IRIS segments each page**, not by a separate chunking enum.

| Control                           | Values                                                                                               | Effect on chunks                                                 |
| --------------------------------- | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| `layout`                          | `"rt_detr_bce"` (default)                                                                            | One chunk per detected layout region (text, table, image, etc.)  |
| `layout`                          | `"whole_page"`                                                                                       | One region per page → **one chunk per page**                     |
| `e2e_ocr` + `e2e_response_parser` | e.g. DeepSeek OCR2                                                                                   | Full-page e2e OCR; parser defines region/box structure           |
| Layout filters                    | `confidence_threshold`, `containment_threshold`, `strict_containment_filter`, table/image thresholds | Fewer or more regions → fewer or more chunks                     |
| `img_method`                      | `"description"`, `"base64"`, `"skip"`                                                                | Whether image regions become chunks (`skip` omits image regions) |

These options live on `IrisParseEngineOptions` inside `IrisParseJobParams`. They are **not** the same as Dex’s four rechunk strategies below—they control IRIS’s native output only.

### Parse with IRIS v2

```python theme={null}
from dex_sdk.types import (
    ParseEngine,
    IrisParseJobParams,
    IrisParseEngineOptions,
)
parse_result = await dex_file.parse(
    IrisParseJobParams(
        engine=ParseEngine.IRIS,
        options=IrisParseEngineOptions(
            layout="rt_detr_bce",   # default: region-level chunks
            # layout="whole_page",  # alternative: one chunk per page
            text_ocr="openai/gpt-5.4",
        ),
    )
)
```

## Dex Chunking Strategies

Dex offers four chunking strategies that work on any parse result—including IRIS v2 via rechunking.
Use these when you need consistent, configurable chunk boundaries across documents or for embeddings/RAG

| Strategy     | Description                                                          | Best For                                      | Embedding Suitability |
| ------------ | -------------------------------------------------------------------- | --------------------------------------------- | --------------------- |
| `token_size` | Splits by token count using a tokenizer (e.g., tiktoken)             | LLM APIs with token limits, cost optimization | Excellent             |
| `recursive`  | Recursively splits using separators (paragraphs → sentences → words) | Articles, documentation, RAG systems          | Excellent             |
| `by_page`    | Splits by page boundaries, grouping complete pages                   | Legal documents, forms, reports               | May be large          |
| `by_section` | Splits by section headers (e.g., markdown #, ##, ###)                | Technical manuals, wikis, academic papers     | Good                  |

***

## Parse Once, Post-chunking as needed

With IRIS 2, you parse without Dex rechunking (omit chunking\_options), then apply Dex strategies on the parse result.
This is the recommended pattern: one parse, many chunking experiments.

```python theme={null}
from dex_sdk.types import IrisParseJobParams, IrisParseEngineOptions, ParseEngine
from dex_core.models.chunking import (
    TokenSizeChunkingOptions,
    RecursiveChunkingOptions,
    PageChunkingOptions,
    SectionChunkingOptions,
)
# Step 1: Parse with IRIS v2 (native region- or page-level chunks)
parse_result = await dex_file.parse(
    IrisParseJobParams(
        engine=ParseEngine.IRIS,
        options=IrisParseEngineOptions(
            layout="whole_page",  # optional: coarser native chunks
        ),
    )
)

# Step 2: Rechunk with Dex strategies
# Token-based (for LLM APIs, embedding models)
rechunked = await parse_result.rechunk(
    TokenSizeChunkingOptions(
        chunk_size=512,
        chunk_overlap=50,
        encoding_name="cl100k_base",
    )
)

# Recursive (preserves paragraphs and sentences)
rechunked = await parse_result.rechunk(
    RecursiveChunkingOptions(
        chunk_size=1000,
        chunk_overlap=200,
    )
)

# Page-based (legal docs, forms)
rechunked = await parse_result.rechunk(
    PageChunkingOptions(pages_per_chunk=1)
)

# Section-based (structured documents with headers)
rechunked = await parse_result.rechunk(
    SectionChunkingOptions(
        section_headers=None,  # Auto-detect from block types
        include_header_in_chunk=True,
    )
)
```

***

## Async Rechunking

For long-running documents, start the rechunk job and poll for completion:

```python theme={null}
# Start rechunk job (returns immediately)
job = await parse_result.start_rechunk_job(
    TokenSizeChunkingOptions(chunk_size=512, chunk_overlap=50)
)

# Wait for completion and get rechunked result
rechunked = await job.get_rechunked_result()
```

***

## Chunking Decision Tree

**IRIS v2 native (`IrisParseEngineOptions`):**

* **Many small, layout-aware chunks** → `layout="rt_detr_bce"` (default)
* **One chunk per page** → `layout="whole_page"`
* **Full-page e2e model** → `e2e_ocr` + `e2e_response_parser`
* **Fewer regions (fewer chunks)** → raise confidence thresholds or tighten containment filters

**Dex rechunk (after IRIS v2 parse):**

**Use `token_size` when:**

* Working with LLM APIs that have token limits
* Embedding models with specific token limits
* Cost optimization

**Use `recursive` when:**

* General document chunking for RAG
* Preserving paragraphs and sentences
* Articles, blog posts, documentation

**Use `by_page` when:**

* Legal documents, forms, reports
* Page references matter
* Page structure should be preserved

**Use `by_section` when:**

* Documents have clear section headers
* Technical manuals, wikis, academic papers
* Semantic coherence within topics

***

## Reducto Chunking Parse-Time - (Legacy)

When using the **Reducto** parse engine (soon to be deprecated), you can chunk during the initial parse instead of rechunking.
Reducto's methods are layout-aware and use document structure.

| Method             | Chunk Size                 | Best For                       | Embedding    | Location Tracking |
| ------------------ | -------------------------- | ------------------------------ | ------------ | ----------------- |
| **VARIABLE**       | Auto (optimal)             | General use, embeddings        | Excellent    | Good              |
| **BLOCK**          | Small (\~100-500 chars)    | Precise locations, UI overlays | Too small    | Excellent         |
| **SECTION**        | Medium (\~1000-3000 chars) | Structured documents           | Good         | Good              |
| **PAGE**           | Large (full page)          | Page-oriented docs             | May be large | Excellent         |
| **PAGE\_SECTIONS** | Medium-Large               | Hybrid needs                   | Good         | Good              |
| **DISABLED**       | Very large (entire doc)    | Special cases                  | Too large    | Excellent         |

**Recommendation:** Use `VARIABLE` for most cases, especially with embeddings.

```python theme={null}
# (Legacy) Reducto parse-time chunking
from dex_sdk.types import (
    ParseEngine,
    ReductoParseJobParams,
    ReductoChunkingMethod,
    ReductoChunkingOptions,
    ReductoParseEngineOptions,
)

# Reducto variable chunking (parse-time)
parse_result = await dex_file.parse(
    ReductoParseJobParams(
        engine=ParseEngine.REDUCTO,
        options=ReductoParseEngineOptions(
            chunking=ReductoChunkingOptions(
                chunk_mode=ReductoChunkingMethod.VARIABLE,
                chunk_size=None,
            )
        ),
    )
)

# Reducto block chunking for precise location tracking
parse_result = await dex_file.parse(
    ReductoParseJobParams(
        engine=ParseEngine.REDUCTO,
        options=ReductoParseEngineOptions(
            chunking=ReductoChunkingOptions(chunk_mode=ReductoChunkingMethod.BLOCK)
        ),
    )
)
```

### Pattern: Retry with Different Chunking

```python theme={null}
# (Legacy) Reducto retry pattern
# Try 1: Variable chunking
result1 = await file.parse(
    ReductoParseJobParams(
        engine=ParseEngine.REDUCTO,
        options=ReductoParseEngineOptions(
            chunking=ReductoChunkingOptions(
                chunk_mode=ReductoChunkingMethod.VARIABLE,
            )
        ),
    )
)

# If not satisfactory, try block chunking
result2 = await file.parse(
    ReductoParseJobParams(
        engine=ParseEngine.REDUCTO,
        options=ReductoParseEngineOptions(
            chunking=ReductoChunkingOptions(
                chunk_mode=ReductoChunkingMethod.BLOCK,
            )
        ),
    )
)
```

***

### Reducto-only decision tree (Legacy)

**Use Reducto `VARIABLE` when:**

* Still on Reducto and want layout-aware chunking at parse time
* General document processing with parser-optimal chunk sizes

**Use Reducto `BLOCK` when:**

* Need precise bounding box information
* Building UI overlays on documents
* Not using for embeddings

**Use Reducto `DISABLED` when:**

* You plan to rechunk with Dex strategies (same pattern as IRIS v2)

```
What's your primary goal? (Reducto only)
├─ Embeddings/Semantic Search → VARIABLE
├─ Precise bounding boxes/UI → BLOCK
├─ Page-by-page processing → PAGE
└─ Structured document navigation → SECTION
```

***

## Next Steps

* **[Vector Stores](vector-stores)**: Add chunks to vector stores for semantic search
* **[Extract](extract)**: Extract structured data from parse results
* **[Parse](parse)**: Parse engine options and configuration (IRIS v2)