Worker nodes
Worker nodes run the platform services.
Under heavy load, SeekrFlow runs additional copies of its services, up to the peak figures. Size worker nodes for the peak figures. Each agent runs as a single pod. If you taint GPU nodes to keep platform services off them, the worker nodes must hold every platform service at peak.
Core services include MLflow. They don’t include Argo Workflows, which runs training and ingestion jobs. The Argo server and controller don’t reserve CPU or memory. Size worker nodes with spare capacity beyond the totals in this table. Ingestion jobs run on worker nodes and use 1 vCPU and about 3.8 GiB of memory while they run.
The Valkey caches are off by default. Turn them on for any deployment reachable by untrusted users. Enabling them requires at least three worker nodes. See Review SeekrFlow requirements.
GPU nodes
SeekrFlow assigns each model at least one dedicated GPU. Each model reserves most of its GPUs’ memory, regardless of model size. Memory not used by the model weights stores active requests, and SeekrFlow sets the maximum context length to fit the remaining memory. Additional GPU memory supports a longer maximum context and more concurrent requests. Model servers reserve 2 vCPUs and 24 GiB of system memory for each GPU they use, unless you change these defaults when you configure models. To leave room for the operating system and Kubernetes, give each GPU node more CPU and memory than its model servers reserve.Decide where the platform models run
SeekrFlow uses four platform models: document ingestion, embeddings, agent planning, and context attribution for explainability. You choose and configure each one. On AWS, the ingestion, embedding, and agent planning models can run on your GPU nodes or on Amazon Bedrock, which requires no GPU nodes. The context attribution model always requires a dedicated GPU in your cluster.
Self-hosting all four platform models requires at least four GPUs, one per model. If Amazon Bedrock serves the other three, only the context attribution model requires a GPU.
Add the models you plan to run
Each additional model you self-host also requires at least one dedicated GPU. The minimum GPU memory depends on the model.
All figures are for one replica. A model with two replicas requires twice the resources.
Plan for fine-tuning jobs
Fine-tuning jobs run on GPU nodes and need far more CPU and memory than model servers, plus temporary disk space. Seekr measured these figures on typical jobs:
Size each GPU node’s root volume, or its local non-volatile memory express (NVMe) storage, for the largest fine-tuning job you plan to run.
Add it up
- Worker nodes – Core services at peak, the Valkey caches if you enable them, and each agent.
- GPU nodes – The platform models you self-host and your other models, multiplied by their replicas.
- Each GPU node – Enough GPUs, CPU, and memory for the largest model or fine-tuning job it runs, and enough disk space for the largest fine-tuning job. A model can’t span nodes.