Create and populate a vector database

This guide covers setting up the vector database, processing your documents, creating embeddings, and monitoring ingestion jobs. A complete example is available following the step-by-step guide, along with guidance for managing your vector databases.

Step 1: Set up a vector database

Seekr’s Vector Database SDK provides advanced semantic search capabilities by transforming text into vector embeddings, making it possible to perform semantic searches that focus on meaning and context. This approach provides a smarter and more intuitive way to retrieve documents compared to traditional keyword-based methods. First, choose an embedding model: Supported embedding models

Model	Dimensions	Max input tokens	Language support	Availability
E5-Mistral-7B-Instruct `intfloat/e5-mistral-7b-instruct`	4096	4096	English (best); limited multilingual	All deployments
Titan Text Embeddings V2 `bedrock:amazon.titan-embed-text-v2:0`	256, 512, or 1024	8192	100+ languages	Self-hosted AWS/EKS only (recommended Bedrock model)
Titan Text Embeddings V1 `bedrock:amazon.titan-embed-text-v1`	1536	8192	25+ languages	Self-hosted AWS/EKS only (legacy)
Titan Text Embeddings G1 `bedrock:amazon.titan-embed-g1-text-02`	1536	8192	25+ languages	Self-hosted AWS/EKS only (legacy)
Titan Multimodal Embeddings `bedrock:amazon.titan-embed-image-v1`	256, 384, or 1024	128	Multimodal (text + image)	Self-hosted AWS/EKS only

Avoid inputs longer than a model’s maximum input tokens.

Bedrock embedding models are available for self-hosted AWS/EKS deployments only. See Use AWS Bedrock for ingestion and inference for setup instructions.

Create an empty vector database

Create the vector database with your chosen model:

from seekrai import SeekrFlow

# Initialize the client. Omit api_key if your key is set as an environment variable.
client = SeekrFlow(api_key="YOUR_KEY_HERE")

# Create the vector database
vector_db = client.vector_database.create(
    model="intfloat/e5-mistral-7b-instruct",
    name="QuickStart_DB",
    description="Quick start example database",
)

database_id = vector_db.id
print(f"Created database: {vector_db.name} (ID: {database_id})")

Sample response:

Created database: QuickStart_DB (ID: b7123456789-09876-4567)

Step 2: Upload files

Upload your source documents to get the file_ids for ingestion. Files can be up to 4GB each. For supported file types and file-preparation guidance, see Prepare and ingest files.

If you already have file_ids from a separate upload, skip this step and reuse them.

# Upload a file
# Replace with the path to your file.
# On Windows, use a raw string to avoid backslash issues: r"C:\Users\username\Downloads\document.pdf"
file_path = "/Users/username/Downloads/document.pdf"

upload_response = client.files.upload(file_path, purpose="alignment")
file_id = upload_response.id
print(f"Uploaded file with ID: {file_id}")

To upload several files at once, or to list and delete uploaded files, see Prepare and ingest files.

Step 3: Start a vector database ingestion job

Next, create a job to ingest documents into your vector database. This step converts the files and creates embeddings from them. Choose an ingestion method and a chunking method, set the chunk size, then start the job.

Choose an ingestion method

Accuracy-optimized (default) When you use method="accuracy-optimized" or omit the method parameter, the system prioritizes accuracy. Depending on what data is available in your PDF document (bookmarks, tables, text layers), the system combines multiple extraction techniques for best results. Key features:

Uses both OCR and direct text extraction, then blends them together
Employs LLM agents to correct and enhance document hierarchy
Applies advanced table detection algorithms for accurate table formatting

Documents over 100 pages can take up to 30 minutes to process.

Speed-optimized When you use method="speed-optimized", the system balances quality with processing speed. It automatically selects faster methods based on document size while maintaining reasonable accuracy for smaller documents. Key features:

Small documents still use high-accuracy methods
Larger documents use speed optimized algorithms to meet time constraints

Optimized to complete in approximately 3 minutes regardless of document size.

Once ingestion is complete, you’ll receive a Markdown file that you can use for fine-tuning.

Ingestion mode and the UIWhen ingesting files through the SeekrFlow UI, speed-optimized mode is always used. The SDK lets you choose between speed-optimized and accuracy-optimized.

Choose a chunking method

Set chunking_method to control how SeekrFlow segments content into chunks. The default is markdown; semantic and sliding are also available. See Choose a chunking method for how each method works and how to attach per-chunk metadata.

Set chunk size and overlap

The token_count parameter specifies the target size of each chunk, ensuring each chunk is neither too large (risking truncation by model limits) nor too small (losing semantic coherence). Best practices:

Common ranges: For embedding and retrieval, 200–500 tokens per chunk is a widely used range, balancing context and efficiency. The example here uses a token count of 512.
Adjust for document type: If your documents are dense or have complex structure (e.g., legal, technical), consider slightly larger chunks; for conversational or highly variable content, smaller chunks may work better.

The overlap_tokens parameter creates overlapping regions between adjacent chunks at chunk boundaries, reducing the risk of missing relevant information that spans two chunks. Adjust chunking parameters based on document characteristics:

Document type	Recommended `token_count`	Recommended `overlap_tokens`
Technical documentation	384-512	50-75
Legal documents	512-768	75-100
Conversational content	256-384	25-50

Create the ingestion job

# Create the ingestion job
ingestion_job = client.vector_database.create_ingestion_job(
    database_id=database_id,
    files=[file_id],
    method="accuracy-optimized",
    chunking_method="markdown",
    token_count=512,
    overlap_tokens=50,
)

job_id = ingestion_job.id
print(f"Created ingestion job: {job_id}")

Sample response:

Created ingestion job: ij-d80bd45a-4bb5-4bac-bbf3-7e3345409bc8

Attach metadata at ingestion

To attach user-defined metadata to the chunks created by an ingestion job, include an optional metadata object in the request. The metadata is job-level: it is copied onto every chunk produced from every file in the job. You can later filter or edit it with the chunk metadata methods (see Manage chunk metadata).

ingestion_job = client.vector_database.create_ingestion_job(
    database_id=database_id,
    files=[file_id],
    method="accuracy-optimized",
    chunking_method="markdown",
    token_count=512,
    overlap_tokens=50,
    metadata={
        "year": 2024,
        "doc_type": "annual_report",
        "department": "finance",
        "is_confidential": True,
    },
)

The metadata object must follow a few constraints (flat object, typed values, 20 keys maximum); see Metadata rules for the full list. To set different metadata on different chunks within one job, use the per-chunk metadata blocks described under Add per-chunk metadata.

Step 4: Monitor ingestion status (optional)

After starting an ingestion job, you can track job progress, view per-file statuses, and diagnose any failures. See Monitor ingestion for details on checking job states, interpreting file_records, and resolving errors. Once status shows completed, your vector database is ready to query. Every ingested chunk also captures provenance metadata (source page, line ranges, heading path) automatically. To trace query results back to their source, see Source tracing.

Complete example

This example demonstrates the entire workflow for creating a vector database, adding files, and kicking off an ingestion job:

from seekrai import SeekrFlow
import time
import os

client = SeekrFlow()

# Step 1: Create vector database
print("Creating vector database...")
db_name = f"QuickStart_DB_{int(time.time())}"
vector_db = client.vector_database.create(
    model="intfloat/e5-mistral-7b-instruct",
    name=db_name,
    description="Quick start example database"
)
database_id = vector_db.id
print(f"Created database: {vector_db.name} (ID: {database_id})")

# Step 2: Upload file
print("Uploading file...")
file_path = "document.pdf"  # Replace with your file path
upload_response = client.files.upload(file_path, purpose="alignment")
file_id = upload_response.id
print(f"Uploaded file with ID: {file_id}")

# Step 3: Begin vector database ingestion
print("Creating ingestion job...")
ingestion_job = client.vector_database.create_ingestion_job(
    database_id=database_id,
    files=[file_id],
    method="accuracy-optimized",
    token_count=512,
    overlap_tokens=50
)
job_id = ingestion_job.id
print(f"Created ingestion job with ID: {job_id}")

# Step 4: Monitor ingestion status
# For per-file tracking and error diagnostics, see Monitor ingestion.
print("Waiting for ingestion job to complete...")
interval = 5    # Check every 5 seconds

while True:
    job_status = client.vector_database.retrieve_ingestion_job(database_id, job_id)
    status = job_status.status
    print(f"Ingestion job status: {status}")

    if status == "completed":
        print(f"Vector database ready with ID: {database_id}")
        break
    elif status == "failed":
        error = getattr(job_status, "error_message", "Unknown error")
        print(f"Ingestion job failed: {error}")
        break

    time.sleep(interval)

print("Setup complete!")

​Step 1: Set up a vector database

​Create an empty vector database

​Step 2: Upload files

​Step 3: Start a vector database ingestion job

​Choose an ingestion method

​Choose a chunking method

​Set chunk size and overlap

​Create the ingestion job

​Attach metadata at ingestion

​Step 4: Monitor ingestion status (optional)

​Complete example

Step 1: Set up a vector database

Create an empty vector database

Step 2: Upload files

Step 3: Start a vector database ingestion job

Choose an ingestion method

Choose a chunking method

Set chunk size and overlap

Create the ingestion job

Attach metadata at ingestion

Step 4: Monitor ingestion status (optional)

Complete example