# Health checks

> Run health checks on your Self-Hosted Deployment.

## Check the health of the pods

To check the health of the pods, run this command:

```sh
kubectl get pods -n <namespace>
```

Replace `<namespace>` with the Kubernetes namespace where the solution is deployed, as defined in your `manifest.yaml` file.

All pods must have the status **Running** and show the expected number of ready containers. The exception is the pods of one-off jobs, such as `novu-manager-job` and, with database monitoring enabled, `pmm-integration-job`: they finish with the status **Completed**.

## Inspect unhealthy pods

If a pod is not in a healthy state, inspect its details:

```sh
kubectl describe pod <pod-name> -n <namespace>
```

Check the pod logs:

```sh
kubectl logs <pod-name> -n <namespace>
```

For pods with multiple containers, specify the container:

```sh
kubectl logs <pod-name> -c <container-name> -n <namespace>
```

## Recover Keycloak

A Keycloak pod that stays unready for 13 minutes restarts itself. If Keycloak still ends up short of a replica, delete every Keycloak pod together and let them come back in order. For example, with two replicas:

```sh
kubectl delete pod keycloak-0 keycloak-1 -n <namespace>
```

`kubectl rollout restart` does nothing while a pod is unready, and deleting only the pod that still looks healthy leaves you with no working Keycloak.

## Database monitoring

With database monitoring enabled, the `pmm-integration-job` job creates the PMM tokens that MongoDB and PostgreSQL report with. If PMM stays unreachable, the job fails after about 8 minutes without affecting the databases, and it doesn't run again on its own. Once PMM is healthy, delete the failed job, then deploy again:

```sh
kubectl delete job -n <namespace> -l app=pmm-integration-job
```

To issue new tokens, for example after one was revoked in PMM, set `FORCE_ROTATE` for the job through the manifest's per-chart override, and deploy again. Both databases get new tokens and restart once. Remove the override afterward: left in place, the next upgrade issues new tokens and restarts both databases again.

```yaml
configuration:
  overrides:
    pmm-integration-job:
      values:
        env:
          FORCE_ROTATE: "true"
```

Next to the PostgreSQL instance, PMM lists a second service, `<instance>-patroni-external`, with the status `Unspecified` or `Failed` even though it works. Its metrics are present and queryable.

## Uninstall and redeploy the solution

If a component is in a bad state and can't recover on its own, you can use shdctl to uninstall and redeploy a specific chart:

```sh
shdctl release uninstall --name <chart-name> --dry-run
shdctl release uninstall --name <chart-name>
shdctl release deploy --name <chart-name>
```