Configure models

SeekrFlow can serve models from GPU inference servers inside your cluster, from an external provider, or from both at once.

Choose where models run

Two settings decide which models users can choose from: models on your GPU nodes, models on Amazon Bedrock, or both. sfe bootstrap sets them from your answers to Sync self-hosted (unprefixed) models and Sync the managed cloud set.

Setting	Set to `true` to offer
`global.modelPolicy.enableSelfHostedModels`	Models that run on your GPU nodes
`global.modelPolicy.enableExternalModels`	Models on Amazon Bedrock

global:
  modelPolicy:
    enableSelfHostedModels: true
    enableExternalModels: true

To keep models from certain countries of origin out of the list, add those countries to global.modelPolicy.deniedCountriesOfOrigin. The models users can choose from come from a catalog that ships with the chart. SeekrFlow loads the catalog into its database automatically when you install or upgrade. It includes only the models these settings allow. A catalog model that runs on your GPU nodes also needs an entry in Define self-hosted models. To replace the chart’s catalog with your own, list your models under ModelCatalog.models, using the same fields as the chart’s MODEL_CATALOG.yaml. Your deployment then stops receiving catalog changes from new releases, including newly added models.

Choose the model for each job

SeekrFlow’s services call models for their own work: reading documents during ingestion, planning agent steps, and explaining fine-tuned models. Each job has a setting in your values file.

A job you don’t set uses a self-hosted default model. If that model isn’t running on your GPU nodes, the job fails the first time a service calls it. For example, an agent fails on its first run.

A model ID starts with a prefix that says where the model runs.

Prefix	Where the model runs	Example
`aws-bedrock:`	Amazon Bedrock	`aws-bedrock:openai.gpt-oss-120b-1:0`
None	Your GPU nodes	`meta-llama/Llama-3.1-8B-Instruct`

Before you use a model on Amazon Bedrock, make sure your account can access it. See Amazon Bedrock access. In AWS GovCloud (US), Amazon Bedrock offers fewer models. Most use the same model ID as in commercial Regions. An ID that starts with us-gov., such as aws-bedrock:us-gov.anthropic.claude-sonnet-4-5-20250929-v1:0, names a GovCloud inference profile. Set each job’s model with these settings:

Job	What it does	Setting	Default
Document reading	Extracts text and tables from PDFs during ingestion. Needs a vision model.	`SEEKR_VISION_MODEL` under `Ingestion.env`	`meta-llama/Llama-3.2-11B-Vision-Instruct`
Document cleanup	Fixes headings and detects tables of contents during ingestion.	`SEEKR_LANGUAGE_MODEL` under `Ingestion.env`	`meta-llama/Llama-3.1-8B-Instruct`
Agent responses	Writes an agent’s replies.	`Agents.models.llm_model`	`meta-llama/Llama-3.1-8B-Instruct`
Tool selection	Chooses which tools an agent calls.	`Agents.models.tool_model`	`openai/gpt-oss-120b`
Agent planning	Plans an agent’s steps. The performance and speed planners serve agents set to higher or lower effort.	`Agents.models.planner_model`, `performance_planner_model`, and `speed_planner_model`	`meta-llama/Llama-3.1-8B-Instruct`
Fine-tuning explanations	Explains which training data shaped a fine-tuned model.	`EXPLAIN_JUDGE_MODEL_NAME` under `Explainability.env`	`meta-llama/Llama-3.1-8B-Instruct`
Training data preparation	Prepares and refines training data for alignment.	`ALIGNMENT_MODEL_NAME` under `Training.env`	`meta-llama/Llama-3.1-8B-Instruct`
Training data index	Builds the searchable index behind fine-tuning explanations. Needs an embedding model.	`EXPLAINABILITY_DB_EMBEDDING_MODEL` under `Training.env`	None

These lists show the models each job supports. Every job except document reading and the training data index uses a language model.

Language models

Where it runs	Model IDs
Your GPU nodes	`openai/gpt-oss-120b` `meta-llama/Llama-3.1-8B-Instruct` `meta-llama/Llama-3.1-70B-Instruct` `meta-llama/Llama-3.2-1B-Instruct`
Amazon Bedrock	`aws-bedrock:global.anthropic.claude-sonnet-4-5-20250929-v1:0` `aws-bedrock:openai.gpt-oss-120b-1:0` `aws-bedrock:nvidia.nemotron-super-3-120b` `aws-bedrock:nvidia.nemotron-nano-12b-v2` `aws-bedrock:nvidia.nemotron-nano-9b-v2` `aws-bedrock:nvidia.nemotron-nano-3-30b` `aws-bedrock:meta.llama3-70b-instruct-v1:0`
Amazon Bedrock in Amazon Web Services (AWS) GovCloud (US)	`aws-bedrock:us-gov.anthropic.claude-sonnet-4-5-20250929-v1:0` `aws-bedrock:openai.gpt-oss-120b-1:0` `aws-bedrock:nvidia.nemotron-super-3-120b` `aws-bedrock:nvidia.nemotron-nano-3-30b` `aws-bedrock:nvidia.nemotron-nano-12b-v2` `aws-bedrock:nvidia.nemotron-nano-9b-v2` `aws-bedrock:meta.llama3-70b-instruct-v1:0` `aws-bedrock:us-gov.openai.gpt-oss-120b-1:0` `aws-bedrock:us-gov.nvidia.nemotron-super-3-120b` `aws-bedrock:us-gov.nvidia.nemotron-nano-3-30b` `aws-bedrock:us-gov.nvidia.nemotron-nano-9b-v2`

Vision models

Where it runs	Model IDs
Your GPU nodes	`ds4sd/SmolDocling-256M-preview` `meta-llama/Llama-3.2-11B-Vision-Instruct` `Qwen/Qwen2.5-VL-7B-Instruct`
Amazon Bedrock	`aws-bedrock:global.anthropic.claude-sonnet-4-5-20250929-v1:0` `aws-bedrock:us.amazon.nova-pro-v1:0`
Amazon Bedrock in AWS GovCloud (US)	`aws-bedrock:us-gov.anthropic.claude-sonnet-4-5-20250929-v1:0` `aws-bedrock:amazon.nova-pro-v1:0`

Embedding models

Where it runs	Model IDs
Your GPU nodes	`intfloat/e5-mistral-7b-instruct`
Amazon Bedrock	`aws-bedrock:amazon.titan-embed-text-v2:0` `aws-bedrock:amazon.titan-embed-g1-text-02`
Amazon Bedrock in AWS GovCloud (US)	`aws-bedrock:amazon.titan-embed-text-v2:0`

Context attribution also uses a dedicated model. sfe bootstrap sets it up. See Create the values file. A model on Amazon Bedrock needs global.modelPolicy.enableExternalModels set to true. See Choose where models run. This example uses Amazon Bedrock for every job:

global:
  modelPolicy:
    enableExternalModels: true

Agents:
  models:
    llm_model: "aws-bedrock:openai.gpt-oss-120b-1:0"
    tool_model: "aws-bedrock:openai.gpt-oss-120b-1:0"
    planner_model: "aws-bedrock:openai.gpt-oss-120b-1:0"
    performance_planner_model: "aws-bedrock:openai.gpt-oss-120b-1:0"
    speed_planner_model: "aws-bedrock:openai.gpt-oss-120b-1:0"

Ingestion:
  env:
    - name: SEEKR_VISION_MODEL
      value: "aws-bedrock:global.anthropic.claude-sonnet-4-5-20250929-v1:0"
    - name: SEEKR_LANGUAGE_MODEL
      value: "aws-bedrock:openai.gpt-oss-120b-1:0"

Explainability:
  env:
    - name: EXPLAIN_JUDGE_MODEL_NAME
      value: "aws-bedrock:openai.gpt-oss-120b-1:0"

Training:
  env:
    - name: EXPLAINABILITY_DB_EMBEDDING_MODEL
      value: "aws-bedrock:amazon.titan-embed-text-v2:0"
    - name: ALIGNMENT_MODEL_NAME
      value: "aws-bedrock:openai.gpt-oss-120b-1:0"

To change these settings after install, update your values file, and then upgrade. See Upgrade a SeekrFlow deployment.

Define self-hosted models

Each model that runs on your GPU nodes needs an entry under FlowModel.models. That includes any model from Choose the model for each job without the aws-bedrock: prefix, and any self-hosted model you want to offer users. Your Seekr account team provides a flowmodel.yaml file with entries for your account. Merge its FlowModel.models block into your values file.

FlowModel:
  models:
    meta-llama/Llama-3.1-8B-Instruct:
      toolCalling:
        enabled: true
        parser: "llama3_json"
      gpu:
        request: 1
        memory: 24
        cpu: "2000m"

Setting	What it sets	Default
`gpu.request`	The number of GPUs the model uses	`1`
`gpu.memory`	GPU memory per GPU, in GB	`24`
`gpu.cpu`	CPU per GPU, in Kubernetes CPU units, such as `2000m` for two cores	`2000m`

A model can’t span nodes. The node that runs a model needs enough GPUs, with enough memory on each, for the whole model. Different models can run on different nodes.

Run models on reserved GPU nodes

If you reserved your GPU nodes for models by adding the nvidia.com/gpu=true:NoSchedule taint, each model needs a matching toleration to run there. On Amazon Elastic Kubernetes Service (Amazon EKS), models get the toleration automatically, and you don’t need to add it.

FlowModel:
  models:
    meta-llama/Llama-3.1-8B-Instruct:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

Load model weights

By default, each self-hosted model downloads its weights from Hugging Face the first time it starts, and SeekrFlow caches them on your shared file storage. Large models can take several minutes to start. Gated models, such as most Llama models, need the Hugging Face token from Request credentials. To load a model from your own object storage instead, set fileStorePath in its FlowModel entry to the bucket and folder that hold its weights.

Stage weights for an offline install

If your cluster can’t reach Hugging Face, stage the weights in your SeekrFlow bucket and turn on offline mode. Offline mode also turns off tools that need the internet, such as web search.

Download the weights

On a machine that can reach Hugging Face, run:

hf download meta-llama/Llama-3.1-8B-Instruct --cache-dir ./hf-cache

Copy them to your bucket

Keep the folder name Hugging Face created:

aws s3 cp --recursive ./hf-cache/models--meta-llama--Llama-3.1-8B-Instruct s3://<bucket-name>/models--meta-llama--Llama-3.1-8B-Instruct/

The copy in your bucket takes about twice the model’s size. Hugging Face links files within its download folder, and Amazon S3 stores a full copy of each linked file.

Turn on offline mode

Set global.offlineMode to true in your values file.

SeekrFlow Self-Managed

Plan a SeekrFlow deployment

Provision SeekrFlow dependencies

Prepare to install SeekrFlow

Deploy SeekrFlow

Operate SeekrFlow

Reference

Choose where models run

Choose the model for each job

Define self-hosted models

Run models on reserved GPU nodes

Load model weights

Stage weights for an offline install

​Choose where models run

​Choose the model for each job

​Define self-hosted models

​Run models on reserved GPU nodes

​Load model weights

​Stage weights for an offline install

Choose where models run

Choose the model for each job

Define self-hosted models

Run models on reserved GPU nodes

Load model weights

Stage weights for an offline install