Create and populate a vector database

​

Step 1: Set up a vector database

​

Create an empty vector database

from seekrai import SeekrFlow

# Initialize client

client = SeekrFlow(api_key="YOUR KEY HERE") # you can leave this empty if your key is stored as an environment variable

# Create vector database

vector_db = client.vector_database.create(
model="intfloat/e5-mistral-7b-instruct",
name="QuickStart_DB",
description="Quick start example database"
)

database_id = vector_db.id
print(f"Created database: {vector_db.name} (ID: {database_id})")

Created database: Quickstart_DB123456789 (ID: b7123456789-09876-4567)

​

Step 2: Upload files

• PDF (.pdf)
• Word documents (.docx)
• Markdown (.md)

• All files must have correctly ordered headers (# followed by ##, and so on) with titles and meaningful content. For example:

# Customer service plan

Some content, separated from the header by a line.

## Unaccompanied minor service

Some more content

### Etc.

• Avoid using headers with more than 6 hashtags (e.g., ####### Pointlessly small md header)

​

Upload a file for ingestion

# Upload a file

# Windows (use raw string to avoid backslash issues):
file_path = r"C:\Users\username\Downloads\document.pdf" # Replace with your file path

# Mac/Linux
file_path = "/Users/username/Downloads/document.pdf" # Replace with your file path

upload_response = client.files.upload(file_path, purpose="alignment")
file_id = upload_response.id
print(f"Uploaded file with ID: {file_id}")

​

Upload a batch of files for ingestion (optional)

from seekrai import SeekrFlow
client = SeekrFlow()

bulk_resp = client.files.bulk_upload(
["documentation.pdf", "policies.md", "guidelines.docx"], purpose="alignment"
)
print("Upload complete")

# Access and print the ID of each uploaded file

print("\nFile IDs:")
for resp in bulk_resp:
print(resp.id)

Uploading file documentation.pdf: 100%|██████████████████████████████████████████████████████████████████████| 7.16M/7.16M [00:08<00:00, 857kB/s]
Uploading file policies.md: 100%|█████████████████████████████████████████████████████████████████| 1.26M/1.26M [00:08<00:00, 151kB/s]
Uploading file guidelines.docx: 100%|███████████████████████████████████████████████████████████| 21.9k/21.9k [00:08<00:00, 2.62kB/s]
Upload complete

File IDs:
file-457989bc-2cf5-11f0-8b3b-56f95a5e9ef4
file-45f909da-2cf5-11f0-8b3b-56f95a5e9ef4
file-46226d20-2cf5-11f0-8b3b-56f95a5e9ef4

​

Step 3: Start a vector database ingestion job

1. Common ranges: For embedding and retrieval, 200–500 tokens per chunk is a widely used range, balancing context and efficiency. The example here uses a token count of 512.
2. Adjust for document type: If your documents are dense or have complex structure (e.g., legal, technical), consider slightly larger chunks; for conversational or highly variable content, smaller chunks may work better.

# Create ingestion job
ingestion_job = client.vector_database.create_ingestion_job(
    database_id=database_id,
    files=[file_id],
    method="accuracy-optimized",
		chunking_method="markdown",
    token_count=512,
    overlap_tokens=50
)

job_id = ingestion_job.id
print(f"Created ingestion job: {job_id}")

Created ingestion job: ij-d80bd45a-4bb5-4bac-bbf3-7e3345409bc8

​

Attach metadata at ingestion

ingestion_job = client.vector_database.create_ingestion_job(
    database_id=database_id,
    files=[file_id],
    method="accuracy-optimized",
    chunking_method="markdown",
    token_count=512,
    overlap_tokens=50,
    metadata={
        "year": 2024,
        "doc_type": "annual_report",
        "department": "finance",
        "is_confidential": True,
    },
)

​

Choose an ingestion method

• Uses both OCR and direct text extraction, then blends them together
• Employs LLM agents to correct and enhance document hierarchy
• Applies advanced table detection algorithms for accurate table formatting

• Small documents still use high-accuracy methods
• Larger documents use speed optimized algorithms to meet time constraints

• Respects document structure and header hierarchies
• Keeps related content together (headers with their content)
• Preserves tables with their headers
• Groups small sections to optimize chunk sizes

• Searches for topic shifts instead of raw heading boundaries to keep tightly related sentences together
• Automatically merges short paragraphs or bullets when they express the same idea
• Honors document structure when it provides strong signals but can span across headings if the semantics match
• Applies Chunk Size and Chunk Overlap as safety caps, splitting only when the semantic chunk would exceed those limits

• Long narrative content (wikis, blogs, requirements) where sections do not follow strict Markdown hierarchy
• Mixed-format documents where context spans multiple small headers or callouts

• Insert ---DOCUMENT_BREAK--- markers to define exactly where chunks should be separated
• Chunk Size (token count) - Maximum tokens per chunk (used as fallback when content exceeds limit)
• Chunk Overlap (token count) - Overlap between chunks when sliding window is applied

# Section 1
This content will be in one chunk...

---DOCUMENT_BREAK---

# Section 2

This content will be in a separate chunk...

---DOCUMENT_BREAK---

# Section 3

This starts another chunk...

​

Add per-chunk metadata

# Introduction

---CHUNK_META_START---
{"author": "Jane Smith", "source": "chapter_1", "classification": "public", "year": 2024}
---CHUNK_META_END---

This is the first chunk of the document.

---DOCUMENT_BREAK---

# Conclusion

This chunk has no metadata block and falls back to the job-level metadata from the ingestion request.

• It is supported only with manual window chunking. Including these markers with any other chunking method is an error.
• Only one metadata block is allowed per section. A second block in the same section is an error.
• A section’s block replaces the job-level metadata for that chunk. The two are not merged.
• A section with no block inherits the job-level metadata from the ingestion request, or no metadata if the request supplied none.

​

Embedding model configuration

​

Step 4: Monitor ingestion status (optional)

​

Source tracing

• line_number_start / line_number_end — line range within the ingested Markdown
• char_start / char_end — character offsets within the section
• hierarchy — heading path from the document root to the chunk (e.g. ["Chapter 3", "3.2 Pricing", "Cancellation"])
• page_number — source document page number, 1-indexed (null for native Markdown or JSON uploads)

Line numbers refer to lines in the ingested Markdown, not the original file. Use page_number to navigate back to a source PDF.

​

Complete example

from seekrai import SeekrFlow
import time
import os

client = SeekrFlow()

# Step 1: Create vector database
print("Creating vector database...")
db_name = f"QuickStart_DB_{int(time.time())}"
vector_db = client.vector_database.create(
    model="intfloat/e5-mistral-7b-instruct",
    name=db_name,
    description="Quick start example database"
)
database_id = vector_db.id
print(f"Created database: {vector_db.name} (ID: {database_id})")

# Step 2: Upload file
print("Uploading file...")
file_path = "document.pdf"  # Replace with your file path
upload_response = client.files.upload(file_path, purpose="alignment")
file_id = upload_response.id
print(f"Uploaded file with ID: {file_id}")

# Step 3: Begin vector database ingestion
print("Creating ingestion job...")
ingestion_job = client.vector_database.create_ingestion_job(
    database_id=database_id,
    files=[file_id],
    method="accuracy-optimized",
    token_count=512,
    overlap_tokens=50
)
job_id = ingestion_job.id
print(f"Created ingestion job with ID: {job_id}")

# Step 4: Monitor ingestion status
# For per-file tracking and error diagnostics, see Monitor ingestion.
print("Waiting for ingestion job to complete...")
interval = 5    # Check every 5 seconds

while True:
    job_status = client.vector_database.retrieve_ingestion_job(database_id, job_id)
    status = job_status.status
    print(f"Ingestion job status: {status}")

    if status == "completed":
        print(f"Vector database ready with ID: {database_id}")
        break
    elif status == "failed":
        error = getattr(job_status, "error_message", "Unknown error")
        print(f"Ingestion job failed: {error}")
        break

    time.sleep(interval)

print("Setup complete!")

​

Manage your vector databases

​

List all vector databases

databases = client.vector_database.list()

for db in databases.data:
print(f"ID: {db.id}, Name: {db.name}")

​

Get a specific vector database

# Get vector database details
db_details = client.vector_database.retrieve(database_id)

print(f"Name: {db_details.name}")
print(f"Last updated: {db_details.updated_at}")

​

Delete a vector database

client.vector_database.delete(database_id) print(f"Successfully deleted
database {database_id}") ```
</CodeGroup>

### List all files in a vector database

<CodeGroup>
```python Python
# List files in vector database
db_files = client.vector_database.list_files(database_id)

for file in db_files.data:
print(f"ID: {file.id}, Filename: {file.filename}")

​

Delete a file from a vector database

# Delete a file from vector database
client.vector_database.delete_file(database_id, file_id)
print(f"Successfully deleted file {file_id} from {database_id}")

​

Troubleshoot common issues

​

Document processing issues

​

File ingestion issues