> ## Documentation Index
> Fetch the complete documentation index at: https://docs.seekr.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Create and populate a vector database

> Set up a vector database and ingest documents to generate embeddings for semantic search and retrieval.

This guide covers setting up the vector database, processing your documents, creating embeddings, and monitoring ingestion jobs. A complete example is available following the step-by-step guide, along with guidance for managing your vector databases.

## Step 1: Set up a vector database

Seekr's Vector Database SDK provides advanced semantic search capabilities by transforming text into vector embeddings, making it possible to perform semantic searches that focus on meaning and context. This approach provides a smarter and more intuitive way to retrieve documents compared to traditional keyword-based methods.

First, choose an embedding model:

**Supported embedding models**

| Model                                                                      | Dimensions        | Max input tokens | Language support                     | Availability                                         |
| -------------------------------------------------------------------------- | ----------------- | ---------------- | ------------------------------------ | ---------------------------------------------------- |
| **E5-Mistral-7B-Instruct**<br />`intfloat/e5-mistral-7b-instruct`          | 4096              | 4096             | English (best); limited multilingual | All deployments                                      |
| **Titan Text Embeddings V2**<br />`bedrock:amazon.titan-embed-text-v2:0`   | 256, 512, or 1024 | 8192             | 100+ languages                       | Self-hosted AWS/EKS only (recommended Bedrock model) |
| **Titan Text Embeddings V1**<br />`bedrock:amazon.titan-embed-text-v1`     | 1536              | 8192             | 25+ languages                        | Self-hosted AWS/EKS only (legacy)                    |
| **Titan Text Embeddings G1**<br />`bedrock:amazon.titan-embed-g1-text-02`  | 1536              | 8192             | 25+ languages                        | Self-hosted AWS/EKS only (legacy)                    |
| **Titan Multimodal Embeddings**<br />`bedrock:amazon.titan-embed-image-v1` | 256, 384, or 1024 | 128              | Multimodal (text + image)            | Self-hosted AWS/EKS only                             |

Avoid inputs longer than a model's maximum input tokens.

<Note>
  Bedrock embedding models are available for self-hosted AWS/EKS deployments only. See [Use AWS Bedrock for ingestion and inference](/flow/sdk/data-engine/aws-bedrock) for setup instructions.
</Note>

### Create an empty vector database

Create the vector database with your chosen model:

<CodeGroup>
  ```python Python theme={null}
  from seekrai import SeekrFlow

  # Initialize the client. Omit api_key if your key is set as an environment variable.
  client = SeekrFlow(api_key="YOUR_KEY_HERE")

  # Create the vector database
  vector_db = client.vector_database.create(
      model="intfloat/e5-mistral-7b-instruct",
      name="QuickStart_DB",
      description="Quick start example database",
  )

  database_id = vector_db.id
  print(f"Created database: {vector_db.name} (ID: {database_id})")
  ```
</CodeGroup>

**Sample response:**

<CodeGroup>
  ```text Output theme={null}
  Created database: QuickStart_DB (ID: b7123456789-09876-4567)
  ```
</CodeGroup>

## Step 2: Upload files

Upload your source documents to get the `file_id`s for ingestion. Files can be up to 4GB each. For supported file types and file-preparation guidance, see [Prepare and ingest files](/flow/sdk/data-engine/file-ingestion).

<Tip>
  If you already have `file_id`s from a separate upload, skip this step and reuse them.
</Tip>

<CodeGroup>
  ```python Python theme={null}
  # Upload a file
  # Replace with the path to your file.
  # On Windows, use a raw string to avoid backslash issues: r"C:\Users\username\Downloads\document.pdf"
  file_path = "/Users/username/Downloads/document.pdf"

  upload_response = client.files.upload(file_path, purpose="alignment")
  file_id = upload_response.id
  print(f"Uploaded file with ID: {file_id}")
  ```
</CodeGroup>

To upload several files at once, or to list and delete uploaded files, see [Prepare and ingest files](/flow/sdk/data-engine/file-ingestion).

## Step 3: Start a vector database ingestion job

Next, create a job to ingest documents into your vector database. This step converts the files and creates embeddings from them. Choose an ingestion method and a chunking method, set the chunk size, then start the job.

### Choose an ingestion method

**Accuracy-optimized (default)**

When you use `method="accuracy-optimized"` or omit the method parameter, the system prioritizes accuracy. Depending on what data is available in your PDF document (bookmarks, tables, text layers), the system combines multiple extraction techniques for best results.

**Key features:**

* Uses both OCR and direct text extraction, then blends them together
* Employs LLM agents to correct and enhance document hierarchy
* Applies advanced table detection algorithms for accurate table formatting

<Info>
  Documents over 100 pages can take up to 30 minutes to process.
</Info>

**Speed-optimized**

When you use `method="speed-optimized"`, the system balances quality with processing speed. It automatically selects faster methods based on document size while maintaining reasonable accuracy for smaller documents.

**Key features:**

* Small documents still use high-accuracy methods
* Larger documents use speed optimized algorithms to meet time constraints

<Info>
  Optimized to complete in approximately 3 minutes regardless of document size.
</Info>

Once ingestion is complete, you'll receive a Markdown file that you can use for fine-tuning.

<Info>
  **Ingestion mode and the UI**

  When ingesting files through the SeekrFlow UI, speed-optimized mode is always used. The SDK lets you choose between speed-optimized and accuracy-optimized.
</Info>

### Choose a chunking method

Set `chunking_method` to control how SeekrFlow segments content into chunks. The default is `markdown`; `semantic` and `sliding` are also available. See [Choose a chunking method](/flow/sdk/data-engine/choose-chunking-method) for how each method works and how to attach per-chunk metadata.

### Set chunk size and overlap

The `token_count` parameter specifies the target size of each chunk, ensuring each chunk is neither too large (risking truncation by model limits) nor too small (losing semantic coherence).

**Best practices:**

* Common ranges: For embedding and retrieval, 200–500 tokens per chunk is a widely used range, balancing context and efficiency. The example here uses a token count of 512.
* Adjust for document type: If your documents are dense or have complex structure (e.g., legal, technical), consider slightly larger chunks; for conversational or highly variable content, smaller chunks may work better.

The `overlap_tokens` parameter creates overlapping regions between adjacent chunks at chunk boundaries, reducing the risk of missing relevant information that spans two chunks.

Adjust chunking parameters based on document characteristics:

| Document type           | Recommended `token_count` | Recommended `overlap_tokens` |
| ----------------------- | ------------------------- | ---------------------------- |
| Technical documentation | 384-512                   | 50-75                        |
| Legal documents         | 512-768                   | 75-100                       |
| Conversational content  | 256-384                   | 25-50                        |

### Create the ingestion job

<CodeGroup>
  ```python Python theme={null}
  # Create the ingestion job
  ingestion_job = client.vector_database.create_ingestion_job(
      database_id=database_id,
      files=[file_id],
      method="accuracy-optimized",
      chunking_method="markdown",
      token_count=512,
      overlap_tokens=50,
  )

  job_id = ingestion_job.id
  print(f"Created ingestion job: {job_id}")
  ```
</CodeGroup>

**Sample response:**

<CodeGroup>
  ```text Output theme={null}
  Created ingestion job: ij-d80bd45a-4bb5-4bac-bbf3-7e3345409bc8
  ```
</CodeGroup>

### Attach metadata at ingestion

To attach user-defined metadata to the chunks created by an ingestion job, include an optional `metadata` object in the request. The metadata is job-level: it is copied onto every chunk produced from every file in the job. You can later filter or edit it with the chunk metadata methods (see [Manage chunk metadata](/flow/sdk/data-engine/manage-chunk-metadata)).

<CodeGroup>
  ```python Python theme={null}
  ingestion_job = client.vector_database.create_ingestion_job(
      database_id=database_id,
      files=[file_id],
      method="accuracy-optimized",
      chunking_method="markdown",
      token_count=512,
      overlap_tokens=50,
      metadata={
          "year": 2024,
          "doc_type": "annual_report",
          "department": "finance",
          "is_confidential": True,
      },
  )
  ```
</CodeGroup>

The metadata object must follow a few constraints (flat object, typed values, 20 keys maximum); see [Metadata rules](/flow/sdk/data-engine/manage-chunk-metadata#metadata-rules) for the full list. To set different metadata on different chunks within one job, use the per-chunk metadata blocks described under [Add per-chunk metadata](/flow/sdk/data-engine/choose-chunking-method#add-per-chunk-metadata).

## Step 4: Monitor ingestion status (optional)

After starting an ingestion job, you can track job progress, view per-file statuses, and diagnose any failures. See [Monitor ingestion](/flow/sdk/data-engine/monitor-ingestion) for details on checking job states, interpreting `file_records`, and resolving errors.

Once `status` shows `completed`, your vector database is ready to query. Every ingested chunk also captures provenance metadata (source page, line ranges, heading path) automatically. To trace query results back to their source, see [Source tracing](/flow/sdk/explainability/source-tracing).

## Complete example

This example demonstrates the entire workflow for creating a vector database, adding files, and kicking off an ingestion job:

<CodeGroup>
  ```python Python expandable theme={null}
  from seekrai import SeekrFlow
  import time
  import os

  client = SeekrFlow()

  # Step 1: Create vector database
  print("Creating vector database...")
  db_name = f"QuickStart_DB_{int(time.time())}"
  vector_db = client.vector_database.create(
      model="intfloat/e5-mistral-7b-instruct",
      name=db_name,
      description="Quick start example database"
  )
  database_id = vector_db.id
  print(f"Created database: {vector_db.name} (ID: {database_id})")

  # Step 2: Upload file
  print("Uploading file...")
  file_path = "document.pdf"  # Replace with your file path
  upload_response = client.files.upload(file_path, purpose="alignment")
  file_id = upload_response.id
  print(f"Uploaded file with ID: {file_id}")

  # Step 3: Begin vector database ingestion
  print("Creating ingestion job...")
  ingestion_job = client.vector_database.create_ingestion_job(
      database_id=database_id,
      files=[file_id],
      method="accuracy-optimized",
      token_count=512,
      overlap_tokens=50
  )
  job_id = ingestion_job.id
  print(f"Created ingestion job with ID: {job_id}")

  # Step 4: Monitor ingestion status
  # For per-file tracking and error diagnostics, see Monitor ingestion.
  print("Waiting for ingestion job to complete...")
  interval = 5    # Check every 5 seconds

  while True:
      job_status = client.vector_database.retrieve_ingestion_job(database_id, job_id)
      status = job_status.status
      print(f"Ingestion job status: {status}")

      if status == "completed":
          print(f"Vector database ready with ID: {database_id}")
          break
      elif status == "failed":
          error = getattr(job_status, "error_message", "Unknown error")
          print(f"Ingestion job failed: {error}")
          break

      time.sleep(interval)

  print("Setup complete!")
  ```
</CodeGroup>