Faster file ingestion

Significantly faster document processing across all file sizes and ingestion modes (speed and accuracy optimized).

Performance improvements:

• 5 MB PDFs: complete in under 35 seconds
• Small text-based documents: complete in under 6 seconds

Actual ingestion times vary based on file size and content. Both speed and accuracy modes now process documents substantially faster regardless of workload,

Chunking strategies for document ingestion

Multiple chunking strategies for vector database ingestion, enabling optimized document processing based on content type and use case.

Available strategies:

• Manual chunking: Use custom delimiters (---DOCUMENT_BREAK---) to define chunk boundaries, ideal for resume collections or document compilations where each document should remain intact.
• Semantic chunking: Group content based on semantic similarity using embedding models, maintaining coherence for long-form content.
• Sliding window chunking: Simple fixed-size chunks with configurable overlap, suitable for uniform processing without content-aware splitting.
• Markdown chunking (existing): Split documents based on markdown heading hierarchy while preserving table structures.

All strategies respect token limits and include fallback handling for sections exceeding maximum token counts.