> ## Documentation Index
> Fetch the complete documentation index at: https://docs.seekr.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshoot a SeekrFlow deployment

> Symptoms during install, after install, and during upgrades, with their causes and fixes.

Find your symptom in the table for when it happened. For anything not listed, see [Investigate a resource](#investigate-a-resource).

## During install

| Symptom | Cause and fix |
| - | - |
| A pod shows `ImagePullBackOff` | The image pull secret is missing. Check that it exists in both the `seekrflow` and `argo-workflows` namespaces. See [Create SeekrFlow secrets](/self-managed/flow/prepare/secrets-and-tokens). |
| `seekrflow-model-sync` shows `ImagePullBackOff` | The catalog sync job doesn't get your image pull secret. Add it to the `seekrflow-operator` service account. See [Install SeekrFlow](/self-managed/flow/deploy/install). |
| A pod shows `CreateContainerConfigError` | A secret is missing. The pod's events name it. Store the secret, then apply it with `sfe upgrade -p <profile-name>`. |
| Auth, Training, Ingestion, and other services never start, and `seekrflow-data-migration` shows `secret "hf-token" not found` | The data migration needs the `hf-token` secret, even if you don't download gated models. The services that own databases wait for the migration. Store the token, then run `sfe upgrade -p <profile-name>`. |
| Most pods stuck, none obviously failing | Check the `seekrflow-license` secret first. Other services wait for the License service. A corrected key can take up to five minutes to take effect. |
| A migration job stays `Pending` | The migration jobs have no tolerations. If every node has a taint, Kubernetes can't schedule them. Make at least one untainted node available. |
| A volume stays `Pending` | The storage class in `global.SharedModelCache.storageClassName` doesn't exist. Create it, or set `global.SharedModelCache.createStorageClass` to `true`. |
| An admission policy rejects pods, with a Kyverno or Gatekeeper error | Your policy engine is blocking SeekrFlow's security context. Allow user IDs `65532` and `1001`, and exempt the Argo server and the model catalog sync job. See [Set up networking and cluster policies](/self-managed/flow/provision/networking). |

## After install

| Symptom | Cause and fix |
| - | - |
| The ingress has no address, and its events say `no certificate found for host` | The load balancer has no certificate for your hostname. In Amazon Web Services (AWS) Certificate Manager, check that the certificate's status is Issued and that it covers the `global.ingress.host` value. |
| `ModelCatalogSynced` is `False`, with reason `SyncFailed` | The catalog sync started before the database was ready. Check the `seekrflow-model-sync` pod's logs, then load the catalog with `sfe models sync -p <profile-name> -r <version> --yes`. |
| Users can't choose models you expect | Check that `ModelCatalogSynced` is `True` with `sfe resources describe SeekrFlowEverywhere/seekrflow`. To compare the catalog with the database, run `sfe models verify -p <profile-name> -r <version>`. |
| The `brave-search-api-key` or `opensearch-credentials` external secret shows `SecretSyncedError`, with `AccessDeniedException` in its events | You didn't store this optional secret, and SeekrFlow can't search your secret store for it. If you don't use it, you can ignore the error. To use it, store it with `sfe secrets create -f my-values.yaml --secrets <secret-name>`, then run `sfe upgrade -p <profile-name>`. |
| `sfe org setup` stops because the deployment has accounts but no administrator | Someone signed in before you created the administrator. Sign in to `sfe` as one of those accounts with `sfe org login`, then run `sfe org setup` again with `--recover-admin`. |

## During an upgrade

If the upgrade's data migration stops, `sfe resources describe SeekrFlowEverywhere/seekrflow` shows a reason on the `DataMigrationComplete` or `RouterMigrationComplete` condition. Services that depend on the migration keep running on the previous version.

| Reason | What to do |
| - | - |
| `SchemaFailed` | A database schema upgrade failed. Read the log of the init container the message names, fix the cause, then re-run the migration. |
| `NotDispatchable` | SeekrFlow couldn't create the migration job. Fix the values the message names, then run `sfe upgrade` again. |
| `AgentRollFailed` | Moving existing agents to the new version failed. SeekrFlow retries on its own. If it continues, contact Seekr support. |
| `DuplicateAccounts` | Two accounts have email addresses that differ only in capitalization, linked to different single sign-on identities. The job's `dedupe-emails` log names them. Delete or change one of each pair, then re-run the migration. |
| `NeedsReview` | The migration finished, but found data that needs review. Read the job's log and contact Seekr support. Don't delete the job until support has reviewed it. |
| `SelfTeamConflict`, `SchemaNotReady`, or `Failed` | Read the job's log, and contact Seekr support. |

SeekrFlow never re-runs a failed migration job on its own. To re-run one after you fix the cause, save its log, then delete the job. SeekrFlow starts a new one, which continues from where the previous one stopped:

```bash theme={null}
kubectl logs job/<job-name> -n seekrflow --all-containers > <job-name>-failed.log
kubectl delete job <job-name> -n seekrflow
```

## Errors from sfe commands

| Error | Cause and fix |
| - | - |
| `Missing Region` | Your machine has no default AWS Region. Set one with `aws configure set region <region>`. |
| `sfe db setup` asks for a Keycloak password, but you didn't enable Keycloak | Add `-r <version>`. Without it, `sfe` can't tell whether you enabled Keycloak. |
| `sfe images mirror` fails with `unauthorized` | Mirroring reads registry sign-ins from Docker's credentials file. Sign in with `docker login` to both registries. |
| `sfe secrets create` fails after you enter every value | The `seekrflow` namespace doesn't exist. Create it, then run the command again. See [Create the namespaces](/self-managed/flow/provision/cluster#create-the-namespaces). |
| `basic credential not found` | The chart isn't on disk, and `sfe` has no saved registry credentials. Sign in with `sfe chart login`. |

## Investigate a resource

<Tabs>
  <Tab title="With the sfe CLI">
    ```bash theme={null}
    sfe resources describe <kind>/<name>
    sfe resources logs <kind>/<name>
    ```

    SeekrFlow's images have no shell. To investigate inside a pod, attach a debug container:

    ```bash theme={null}
    sfe resources debug <kind>/<name>
    ```
  </Tab>

  <Tab title="Without the sfe CLI">
    ```bash theme={null}
    kubectl describe pod <pod-name> -n seekrflow
    kubectl logs <pod-name> -n seekrflow
    ```

    The `Events` section at the end of the `describe` output usually names the problem.
  </Tab>
</Tabs>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.