Core capabilities
The data engine provides two primary functions:Storage
Storage manages raw content through file ingestion and vector database creation, with ingestion insights that surface real-time processing status and diagnostics. Vector databases transform these files into searchable knowledge bases through chunking and embedding, with support for user-defined chunk metadata.
AI-ready data
AI-ready data generates and transforms datasets, converting raw content into training-ready formats. These jobs produce structured outputs optimized for model fine-tuning and alignment, including standard instruction datasets and context-grounded datasets.
Data workflow
The typical data engine workflow:1
Upload raw content files to storage.
2
Create vector stores for retrieval applications.
3
Generate AI-ready datasets from selected files.
4
Use outputs for fine-tuning, agent knowledge bases, or evaluations.
Integration points
Data engine outputs integrate across SeekrFlow:- Fine-tuning – AI-ready datasets feed model training pipelines
- Agents – Vector stores power FileSearch tool for knowledge retrieval
- Evaluations – Structured datasets support model testing and validation