Choose a chunking method

The chunking_method parameter on create_ingestion_job controls how SeekrFlow segments your content into chunks before embedding. Choose the method that best fits your documents.

Markdown chunking (default)

Intelligent structure-aware chunking that automatically detects logical content breaks:

Respects document structure and heading hierarchies
Keeps related content together (headings with their content)
Preserves tables with their headers
Groups small sections to optimize chunk sizes

Semantic chunking

Call with chunking_method="semantic" to enable meaning-aware segmentation. This method detects sentence boundaries, embeds paragraphs, and groups them by meaning using hierarchical clustering, rather than relying on document position:

Searches for topic shifts instead of raw heading boundaries to keep tightly related sentences together
Automatically merges short paragraphs or bullets when they express the same idea
Honors document structure when it provides strong signals but can span across headings if the semantics match
Applies token_count and overlap_tokens as safety caps, splitting only when a semantic chunk would exceed those limits

When to use it:

Long narrative content (wikis, blogs, requirements) where sections do not follow strict Markdown hierarchy
Mixed-format documents where context spans multiple small headings or callouts

No additional markup is required. Tune token_count and overlap_tokens to control chunk granularity.

Sliding window chunking

Call with chunking_method="sliding" to split content into fixed-size overlapping windows. This method has no structural awareness, which makes it fast and predictable. Use it for plain-text or homogeneous content that lacks clear document structure. To force chunk boundaries at specific points, insert ---DOCUMENT_BREAK--- markers in your Markdown:

# Section 1

This content will be in one chunk...

---DOCUMENT_BREAK---

# Section 2

This content will be in a separate chunk...

---DOCUMENT_BREAK---

# Section 3

This starts another chunk...

Set token_count (for example, 1000) and overlap_tokens (for example, 100) in the same create_ingestion_job call to control chunk size and overlap. Document break markers apply only to sliding window chunking. With markdown or semantic chunking, they are ignored.

If the content between two document break markers exceeds token_count, the sliding window splits it into multiple chunks.

Add per-chunk metadata

With sliding window chunking, you can attach different metadata to different chunks by embedding a metadata block inside a section. Place the block between ---CHUNK_META_START--- and ---CHUNK_META_END---, with a single line of JSON in between. The block is scoped to the section it appears in and is removed from the text before indexing and never becomes part of the chunk content.

# Introduction

---CHUNK_META_START---
{"author": "Jane Smith", "source": "chapter_1", "classification": "public", "year": 2024}
---CHUNK_META_END---

This is the first chunk of the document.

---DOCUMENT_BREAK---

# Conclusion

This chunk has no metadata block and falls back to the job-level metadata from the ingestion request.

Per-chunk metadata follows the same rules as job-level metadata and behaves as follows:

It is supported only with sliding window chunking. Including these markers with any other chunking method is an error.
Only one metadata block is allowed per section. A second block in the same section is an error.
A section’s block replaces the job-level metadata for that chunk. The two are not merged.
A section with no block inherits the job-level metadata from the ingestion request, or no metadata if the request supplied none.

​Markdown chunking (default)

​Semantic chunking

​Sliding window chunking

​Add per-chunk metadata

Markdown chunking (default)

Semantic chunking

Sliding window chunking

Add per-chunk metadata