> ## Documentation Index
> Fetch the complete documentation index at: https://docs.seekr.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure models

> Choose where models run, pick the model each SeekrFlow service uses, and define the self-hosted models to run.

SeekrFlow can serve models from GPU inference servers inside your cluster, from an external provider, or from both at once.

## Choose where models run

Two settings decide which models users can choose from: models on your GPU nodes, models on Amazon Bedrock, or both. `sfe bootstrap` sets them from your answers to **Sync self-hosted (unprefixed) models** and **Sync the managed cloud set**.

| Setting | Set to `true` to offer |
| - | - |
| `global.modelPolicy.enableSelfHostedModels` | Models that run on your GPU nodes |
| `global.modelPolicy.enableExternalModels` | Models on Amazon Bedrock |

```yaml theme={null}
global:
  modelPolicy:
    enableSelfHostedModels: true
    enableExternalModels: true
```

To keep models from certain countries of origin out of the list, add those countries to `global.modelPolicy.deniedCountriesOfOrigin`.

The models users can choose from come from a catalog that ships with the chart. SeekrFlow loads the catalog into its database automatically when you install or upgrade. It includes only the models these settings allow. A catalog model that runs on your GPU nodes also needs an entry in [Define self-hosted models](#define-self-hosted-models).

To replace the chart's catalog with your own, list your models under `ModelCatalog.models`, using the same fields as the chart's `MODEL_CATALOG.yaml`. Your deployment then stops receiving catalog changes from new releases, including newly added models.

## Choose the model for each job

SeekrFlow's services call models for their own work: reading documents during ingestion, planning agent steps, and explaining fine-tuned models. Each job has a setting in your values file.

<Note>
  A job you don't set uses a self-hosted default model. If that model isn't running on your GPU nodes, the job fails the first time a service calls it. For example, an agent fails on its first run.
</Note>

A model ID starts with a prefix that says where the model runs.

| Prefix | Where the model runs | Example |
| - | - | - |
| `aws-bedrock:` | Amazon Bedrock | `aws-bedrock:openai.gpt-oss-120b-1:0` |
| None | Your GPU nodes | `meta-llama/Llama-3.1-8B-Instruct` |

Before you use a model on Amazon Bedrock, make sure your account can access it. See [Amazon Bedrock access](/self-managed/flow/provision/checklist#amazon-bedrock-access).

In AWS GovCloud (US), Amazon Bedrock offers fewer models. Most use the same model ID as in commercial Regions. An ID that starts with `us-gov.`, such as `aws-bedrock:us-gov.anthropic.claude-sonnet-4-5-20250929-v1:0`, names a GovCloud inference profile.

Set each job's model with these settings:

| Job | What it does | Setting | Default |
| - | - | - | - |
| Document reading | Extracts text and tables from PDFs during ingestion. Needs a vision model. | `SEEKR_VISION_MODEL` under `Ingestion.env` | `meta-llama/Llama-3.2-11B-Vision-Instruct` |
| Document cleanup | Fixes headings and detects tables of contents during ingestion. | `SEEKR_LANGUAGE_MODEL` under `Ingestion.env` | `meta-llama/Llama-3.1-8B-Instruct` |
| Agent responses | Writes an agent's replies. | `Agents.models.llm_model` | `meta-llama/Llama-3.1-8B-Instruct` |
| Tool selection | Chooses which tools an agent calls. | `Agents.models.tool_model` | `openai/gpt-oss-120b` |
| Agent planning | Plans an agent's steps. The performance and speed planners serve agents set to higher or lower effort. | `Agents.models.planner_model`, `performance_planner_model`, and `speed_planner_model` | `meta-llama/Llama-3.1-8B-Instruct` |
| Fine-tuning explanations | Explains which training data shaped a fine-tuned model. | `EXPLAIN_JUDGE_MODEL_NAME` under `Explainability.env` | `meta-llama/Llama-3.1-8B-Instruct` |
| Training data preparation | Prepares and refines training data for alignment. | `ALIGNMENT_MODEL_NAME` under `Training.env` | `meta-llama/Llama-3.1-8B-Instruct` |
| Training data index | Builds the searchable index behind fine-tuning explanations. Needs an embedding model. | `EXPLAINABILITY_DB_EMBEDDING_MODEL` under `Training.env` | None |

These lists show the models each job supports. Every job except document reading and the training data index uses a language model.

<AccordionGroup>
  <Accordion title="Language models">
    | Where it runs | Model IDs |
    | - | - |
    | Your GPU nodes | `openai/gpt-oss-120b`<br />`meta-llama/Llama-3.1-8B-Instruct`<br />`meta-llama/Llama-3.1-70B-Instruct`<br />`meta-llama/Llama-3.2-1B-Instruct` |
    | Amazon Bedrock | `aws-bedrock:global.anthropic.claude-sonnet-4-5-20250929-v1:0`<br />`aws-bedrock:openai.gpt-oss-120b-1:0`<br />`aws-bedrock:nvidia.nemotron-super-3-120b`<br />`aws-bedrock:nvidia.nemotron-nano-12b-v2`<br />`aws-bedrock:nvidia.nemotron-nano-9b-v2`<br />`aws-bedrock:nvidia.nemotron-nano-3-30b`<br />`aws-bedrock:meta.llama3-70b-instruct-v1:0` |
    | Amazon Bedrock in Amazon Web Services (AWS) GovCloud (US) | `aws-bedrock:us-gov.anthropic.claude-sonnet-4-5-20250929-v1:0`<br />`aws-bedrock:openai.gpt-oss-120b-1:0`<br />`aws-bedrock:nvidia.nemotron-super-3-120b`<br />`aws-bedrock:nvidia.nemotron-nano-3-30b`<br />`aws-bedrock:nvidia.nemotron-nano-12b-v2`<br />`aws-bedrock:nvidia.nemotron-nano-9b-v2`<br />`aws-bedrock:meta.llama3-70b-instruct-v1:0`<br />`aws-bedrock:us-gov.openai.gpt-oss-120b-1:0`<br />`aws-bedrock:us-gov.nvidia.nemotron-super-3-120b`<br />`aws-bedrock:us-gov.nvidia.nemotron-nano-3-30b`<br />`aws-bedrock:us-gov.nvidia.nemotron-nano-9b-v2` |
  </Accordion>

  <Accordion title="Vision models">
    | Where it runs | Model IDs |
    | - | - |
    | Your GPU nodes | `ds4sd/SmolDocling-256M-preview`<br />`meta-llama/Llama-3.2-11B-Vision-Instruct`<br />`Qwen/Qwen2.5-VL-7B-Instruct` |
    | Amazon Bedrock | `aws-bedrock:global.anthropic.claude-sonnet-4-5-20250929-v1:0`<br />`aws-bedrock:us.amazon.nova-pro-v1:0` |
    | Amazon Bedrock in AWS GovCloud (US) | `aws-bedrock:us-gov.anthropic.claude-sonnet-4-5-20250929-v1:0`<br />`aws-bedrock:amazon.nova-pro-v1:0` |
  </Accordion>

  <Accordion title="Embedding models">
    | Where it runs | Model IDs |
    | - | - |
    | Your GPU nodes | `intfloat/e5-mistral-7b-instruct` |
    | Amazon Bedrock | `aws-bedrock:amazon.titan-embed-text-v2:0`<br />`aws-bedrock:amazon.titan-embed-g1-text-02` |
    | Amazon Bedrock in AWS GovCloud (US) | `aws-bedrock:amazon.titan-embed-text-v2:0` |
  </Accordion>
</AccordionGroup>

Context attribution also uses a dedicated model. `sfe bootstrap` sets it up. See [Create the values file](/self-managed/flow/prepare/values-file).

A model on Amazon Bedrock needs `global.modelPolicy.enableExternalModels` set to `true`. See [Choose where models run](#choose-where-models-run).

This example uses Amazon Bedrock for every job:

```yaml theme={null}
global:
  modelPolicy:
    enableExternalModels: true

Agents:
  models:
    llm_model: "aws-bedrock:openai.gpt-oss-120b-1:0"
    tool_model: "aws-bedrock:openai.gpt-oss-120b-1:0"
    planner_model: "aws-bedrock:openai.gpt-oss-120b-1:0"
    performance_planner_model: "aws-bedrock:openai.gpt-oss-120b-1:0"
    speed_planner_model: "aws-bedrock:openai.gpt-oss-120b-1:0"

Ingestion:
  env:
    - name: SEEKR_VISION_MODEL
      value: "aws-bedrock:global.anthropic.claude-sonnet-4-5-20250929-v1:0"
    - name: SEEKR_LANGUAGE_MODEL
      value: "aws-bedrock:openai.gpt-oss-120b-1:0"

Explainability:
  env:
    - name: EXPLAIN_JUDGE_MODEL_NAME
      value: "aws-bedrock:openai.gpt-oss-120b-1:0"

Training:
  env:
    - name: EXPLAINABILITY_DB_EMBEDDING_MODEL
      value: "aws-bedrock:amazon.titan-embed-text-v2:0"
    - name: ALIGNMENT_MODEL_NAME
      value: "aws-bedrock:openai.gpt-oss-120b-1:0"
```

To change these settings after install, update your values file, and then upgrade. See [Upgrade a SeekrFlow deployment](/self-managed/flow/operate/upgrade).

## Define self-hosted models

Each model that runs on your GPU nodes needs an entry under `FlowModel.models`. That includes any model from [Choose the model for each job](#choose-the-model-for-each-job) without the `aws-bedrock:` prefix, and any self-hosted model you want to offer users. Your Seekr account team provides a `flowmodel.yaml` file with entries for your account. Merge its `FlowModel.models` block into your values file.

```yaml theme={null}
FlowModel:
  models:
    meta-llama/Llama-3.1-8B-Instruct:
      toolCalling:
        enabled: true
        parser: "llama3_json"
      gpu:
        request: 1
        memory: 24
        cpu: "2000m"
```

| Setting | What it sets | Default |
| - | - | - |
| `gpu.request` | The number of GPUs the model uses | `1` |
| `gpu.memory` | GPU memory per GPU, in GB | `24` |
| `gpu.cpu` | CPU per GPU, in Kubernetes CPU units, such as `2000m` for two cores | `2000m` |

<Note>
  A model can't span nodes. The node that runs a model needs enough GPUs, with enough memory on each, for the whole model. Different models can run on different nodes.
</Note>

### Run models on reserved GPU nodes

If you reserved your GPU nodes for models by adding the `nvidia.com/gpu=true:NoSchedule` [taint](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/), each model needs a matching toleration to run there. On Amazon Elastic Kubernetes Service (Amazon EKS), models get the toleration automatically, and you don't need to add it.

```yaml theme={null}
FlowModel:
  models:
    meta-llama/Llama-3.1-8B-Instruct:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
```

### Load model weights

By default, each self-hosted model downloads its weights from Hugging Face the first time it starts, and SeekrFlow caches them on your shared file storage. Large models can take several minutes to start. Gated models, such as most Llama models, need the Hugging Face token from [Request credentials](/self-managed/flow/provision/checklist#request-credentials).

To load a model from your own object storage instead, set `fileStorePath` in its FlowModel entry to the bucket and folder that hold its weights.

### Stage weights for an offline install

If your cluster can't reach Hugging Face, stage the weights in your SeekrFlow bucket and turn on offline mode. Offline mode also turns off tools that need the internet, such as web search.

<Steps>
  <Step title="Download the weights">
    On a machine that can reach Hugging Face, run:

    ```bash theme={null}
    hf download meta-llama/Llama-3.1-8B-Instruct --cache-dir ./hf-cache
    ```
  </Step>

  <Step title="Copy them to your bucket">
    Keep the folder name Hugging Face created:

    ```bash theme={null}
    aws s3 cp --recursive ./hf-cache/models--meta-llama--Llama-3.1-8B-Instruct s3://<bucket-name>/models--meta-llama--Llama-3.1-8B-Instruct/
    ```

    The copy in your bucket takes about twice the model's size. Hugging Face links files within its download folder, and Amazon S3 stores a full copy of each linked file.
  </Step>

  <Step title="Turn on offline mode">
    Set `global.offlineMode` to `true` in your values file.
  </Step>
</Steps>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.