Provision the Kubernetes cluster for SeekrFlow

SeekrFlow runs on a Kubernetes cluster you create and manage. Create it with your provider’s standard process, and make sure it meets the requirements in Review SeekrFlow requirements.

Create the cluster

Amazon Web Services (AWS)
On-premises

Create an Amazon Elastic Kubernetes Service (Amazon EKS) cluster with the AWS Management Console, eksctl, or the AWS Command Line Interface (AWS CLI). See Create an Amazon EKS cluster. Use the settings in this table, and leave the rest at their defaults.

Setting	What SeekrFlow needs
Name and AWS Region	Any name. Any commercial Region, or `us-gov-west-1` or `us-gov-east-1` for AWS GovCloud (US). You enter both in the values file later.
EKS Auto Mode	Off. SeekrFlow runs on managed node groups. Auto Mode replaces and consolidates nodes on its own schedule, which can interrupt model serving and fine-tuning and release GPU instances that are hard to replace.
Kubernetes version	One of the three most recent Kubernetes minor versions. Versions in Amazon EKS extended support aren’t supported.
Cluster IAM role	An AWS Identity and Access Management (IAM) role that Amazon EKS can assume, with the `AmazonEKSClusterPolicy` managed policy attached. If you already have one, such as `AmazonEKSClusterRole`, use it. Otherwise, create one. See Amazon EKS cluster IAM role.
Virtual private cloud (VPC) and subnets	Any layout your organization uses. Amazon EKS requires subnets in at least two Availability Zones. The nodes must reach the container registry, object storage, PostgreSQL, OpenSearch, and any cloud model services you use.
Endpoint access	Any. If you select Private, run `kubectl` and `sfe` from a machine inside your network, such as a bastion host.
Add-ons	The Amazon EKS Pod Identity Agent, which SeekrFlow uses to reach your cloud services. See Set up the Amazon EKS Pod Identity Agent. Also keep Amazon VPC CNI, CoreDNS, and kube-proxy, which the cluster needs, and Metrics Server, which SeekrFlow needs to scale its services. SeekrFlow doesn’t need the other add-ons the console preselects, such as External DNS. Remove any you don’t plan to configure.

Add worker and GPU nodes

Add two sets of nodes to the cluster:

Worker nodes – Run the platform services. Provision the virtual CPUs (vCPUs) and memory from Size your deployment. Enabling the Valkey caches takes at least three worker nodes.
GPU nodes – Run model servers, fine-tuning, and explainability workloads. Use an instance type or server that meets the GPU requirements, with the GPU count from Size your deployment. Each GPU node needs enough GPUs and memory for the largest model it hosts, because a model can’t span nodes.

Both must be nodes you manage. Serverless compute, such as AWS Fargate, isn’t supported.

Amazon Web Services (AWS)
On-premises

Create two managed node groups, one for worker nodes and one for GPU nodes. See Create a managed node group for your cluster. Use the settings in this table, and leave the rest at their defaults.

Setting	Worker node group	GPU node group
Node IAM role	A role with the `AmazonEKSWorkerNodePolicy`, `AmazonEC2ContainerRegistryReadOnly`, and `AmazonEKS_CNI_Policy` managed policies attached. If you already have one, use it. Otherwise, create one. See Amazon EKS node IAM role.	The same role.
Amazon Machine Image (AMI) type	A standard x86_64 AMI type, such as Amazon Linux 2023 (x86_64) Standard.	An NVIDIA AMI type, such as Amazon Linux 2023 NVIDIA or Bottlerocket NVIDIA. On Kubernetes 1.34 and later, these include driver 580 or newer. Earlier Kubernetes versions get an older driver. If your compliance requirements call for FIPS mode on the nodes, use an NVIDIA AMI with FIPS mode enabled, such as Amazon Linux 2023 NVIDIA. See Use EKS-optimized accelerated AMIs for GPU instances.
Instance type	Enough instances for the vCPUs and memory in your sizing, and at least three if you enable the Valkey caches.	An instance family with a supported GPU, such as G5 (A10G), G6 (L4), G6e (L40S), P4d (A100), or P5 (H100). Check that your Region offers it and that your quota covers it.
Root volume	At least 100 GiB.	At least 200 GiB, plus temporary space for fine-tuning jobs. See Size your deployment.
Kubernetes taints	None.	Optional. To keep platform services off GPU nodes, add the taint `nvidia.com/gpu=true:NoSchedule`. Model pods on Amazon EKS accept this taint automatically.
Subnets	The subnets you chose for the cluster.	The same subnets.

Join the worker and GPU servers to the cluster as nodes, using your Kubernetes distribution’s process.On each GPU node, install an NVIDIA driver that meets the driver requirement.To keep platform services off GPU nodes, taint the GPU nodes with nvidia.com/gpu=true:NoSchedule. Each model then needs a matching toleration in FlowModel.models.<name>.tolerations. See Run models on reserved GPU nodes.

Check the cluster

Run these checks after the nodes are ready. Each one tells you a value to record.

Amazon Web Services (AWS)
On-premises

Connect to the cluster

Add the cluster to your kubectl configuration:

aws eks update-kubeconfig --name <cluster-name> --region <region>

Then confirm every node shows Ready:

kubectl get nodes

Check Metrics Server

To check that Metrics Server is running, run:

kubectl top nodes

If the command prints CPU and memory for each node, Metrics Server is running. If it returns an error, add the Metrics Server add-on to the cluster. It takes about a minute to start reporting.

Check the NVIDIA device plugin

The NVIDIA device plugin makes each node’s GPUs available to Kubernetes. To check whether it’s running, run:

kubectl get daemonset --all-namespaces | grep nvidia-device-plugin

If the command prints nothing, the plugin isn’t running. Install it before you install SeekrFlow, which doesn’t install it for you:

helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm install nvidia-device-plugin nvdp/nvidia-device-plugin --namespace kube-system

For other installation options, see the NVIDIA device plugin documentation.The Bottlerocket NVIDIA AMIs include the plugin. The Amazon Linux 2023 NVIDIA AMIs don’t.

Find the GPU node label

Kubernetes needs a label to identify which nodes are GPU nodes. Any key-value pair works as long as it matches what is on your nodes. To see the labels on your nodes, run:

kubectl get nodes --show-labels

The Amazon Linux 2023 NVIDIA AMIs add nvidia.com/gpu.present: "true". If all your GPU nodes share one instance type, you can also use node.kubernetes.io/instance-type, such as node.kubernetes.io/instance-type: g5.xlarge.

Check for a JobSet controller

Multi-node fine-tuning needs the JobSet controller. SeekrFlow can install it during configuration, but a cluster must not run two. To check whether one is installed, run:

kubectl get crd jobsets.jobset.x-k8s.io

If the command returns NotFound, JobSet isn’t installed. Record true for jobset.enabled only if you need multi-node fine-tuning. Otherwise, record false. If the command returns the resource, JobSet is already installed. Record false.

Connect to the cluster

Set your kubectl context to the cluster, using your Kubernetes distribution’s process. Then confirm every node shows Ready:

kubectl get nodes

Check Metrics Server

To check that Metrics Server is running, run:

kubectl top nodes

If the command prints CPU and memory for each node, Metrics Server is running. If it returns an error, install Metrics Server. It takes about a minute to start reporting.

Check the NVIDIA driver

Run nvidia-smi on each GPU node. The Driver Version it reports must be 580 or later.

Check the NVIDIA device plugin

The NVIDIA device plugin makes each node’s GPUs available to Kubernetes. To check whether it’s running, run:

kubectl get daemonset --all-namespaces | grep nvidia-device-plugin

If the command prints nothing, the plugin isn’t running. Install it before you install SeekrFlow, which doesn’t install it for you:

helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm install nvidia-device-plugin nvdp/nvidia-device-plugin --namespace kube-system

For other installation options, see the NVIDIA device plugin documentation.

Find the GPU node label

Kubernetes needs a label to identify which nodes are GPU nodes. Any key-value pair works as long as it matches what is on your nodes. To see the labels on your nodes, run:

kubectl get nodes --show-labels

Use whatever label your GPU nodes carry.

Check for a JobSet controller

Multi-node fine-tuning needs the JobSet controller. SeekrFlow can install it during configuration, but a cluster must not run two. To check whether one is installed, run:

kubectl get crd jobsets.jobset.x-k8s.io

Create the namespaces

SeekrFlow installs into the seekrflow namespace and runs training and ingestion jobs in the argo-workflows namespace. Create both before you continue. Later steps add network policies, secrets, and cloud identity settings to them.

kubectl create namespace seekrflow
kubectl create namespace argo-workflows

If a namespace already exists, kubectl returns an AlreadyExists error. You can continue.

Record the values

Note these values. You will enter them when you create the values file.

Value	Used for	Example or format
Cluster name	`global.cloudProvider.clusterName`	`seekrflow-prod`
AWS Region, AWS only	`global.cloudProvider.aws.region`	`us-east-1`
Kubernetes context	Your `sfe` profile	The `kubectl` context for this cluster.
GPU node label	`global.defaults.gpuNodeLabel`	`nvidia.com/gpu.present: "true"`
Whether a JobSet controller is running	`jobset.enabled`	`true` only if you need multi-node fine-tuning and the cluster has no JobSet controller.

SeekrFlow Self-Managed

Plan a SeekrFlow deployment

Provision SeekrFlow dependencies

Prepare to install SeekrFlow

Deploy SeekrFlow

Operate SeekrFlow

Reference

Provision the Kubernetes cluster for SeekrFlow

Create the cluster

Add worker and GPU nodes

Check the cluster

Create the namespaces

Record the values

​Create the cluster

​Add worker and GPU nodes

​Check the cluster

​Create the namespaces

​Record the values

Create the cluster

Add worker and GPU nodes

Check the cluster

Create the namespaces

Record the values