文档

Monitor your deployment

Check the health of the pods in your Self-Hosted Deployment, sign in to Grafana, and manage the credentials that monitoring uses.
阅读时间3 分钟最后更新于 13 天前

Check that the pods in your Self-Hosted Deployment are healthy, and use Grafana and Prometheus to monitor its services. If a check finds a problem, refer to Troubleshooting Self-Hosted Deployment components.

Prerequisites

Before you start, check the following:
  • To use Grafana or collect service metrics, turn on Prometheus monitoring. In the manifest, set
    configuration.monitoring.prometheus.enabled: true
    or
    configuration.monitoring.prometheus.installedInCluster: true
    .
  • To issue new PMM tokens, enable database monitoring.

Check the health of the pods

To check the health of the pods, run this command:
kubectl get pods -n <namespace>
Replace
<namespace>
with the Kubernetes namespace where the solution is deployed, as defined in your
manifest.yaml
file.
All pods must have the status Running and show the expected number of ready containers. The exception is the pods of one-off jobs, such as
novu-manager-job
and, with database monitoring enabled,
pmm-integration-job
. These pods finish with the status Completed.

Inspect unhealthy pods

If a pod isn't in a healthy state, inspect its details:
kubectl describe pod <pod-name> -n <namespace>
Check the pod logs:
kubectl logs <pod-name> -n <namespace>
For pods with multiple containers, specify the container:
kubectl logs <pod-name> -c <container-name> -n <namespace>

Sign in to Grafana

Grafana is at
https://<appDomain>/grafana
. Sign in as
admin
with the password that
shdctl secret generate
created. The password is in your persisted secrets file under
grafana-credentials
. To read it from the cluster instead, run this command:
kubectl -n <namespace> get secret grafana-credentials -o jsonpath='{.data.admin-password}' | base64 -d
Grafana applies this secret only when it's first installed. After that, Grafana's dashboard and datasource updates still sign in with the password in the secret, so the password in the secret and the password in Grafana must match.

Change the Grafana password

To change the Grafana admin password, follow these steps:
  1. In your persisted secrets file, set the new password under
    grafana-credentials
    .
  2. Generate and deploy the secrets, as Regenerate and deploy the secrets on the upgrade page describes.
  3. Apply the new password to Grafana, and restart Grafana:
    kubectl -n <namespace> get secret grafana-credentials -o jsonpath='{.data.admin-password}' | base64 -d | kubectl -n <namespace> exec -i statefulset/kube-prometheus-stack-grafana -c grafana -- grafana cli --config /etc/grafana/grafana.ini admin reset-admin-password --password-from-stdinkubectl -n <namespace> rollout restart statefulset/kube-prometheus-stack-grafana

Collect service metrics with your own Prometheus Operator

If you use your own Prometheus Operator (
installedInCluster: true
), set its
serviceMonitorSelector
to match the label
release: kube-prometheus-stack
. Every ServiceMonitor that the release creates carries this label. The bundled Prometheus already selects it.
For the metrics that Prometheus collects, refer to Service metrics reference.

Issue new PMM tokens

To issue new tokens, for example after someone revokes a token in PMM, set
FORCE_ROTATE
for the
pmm-integration-job
job through the manifest's per-chart override, and deploy again:
configuration: overrides: pmm-integration-job: values: env: FORCE_ROTATE: "true"
Both databases get new tokens and restart once. Remove the override afterward. If you leave it in place, the next upgrade issues new tokens and restarts both databases again.

Additional resources