Monitor your deployment
Check the health of the pods in your Self-Hosted Deployment, sign in to Grafana, and manage the credentials that monitoring uses.
阅读时间3 分钟最后更新于 13 天前
Check that the pods in your Self-Hosted Deployment are healthy, and use Grafana and Prometheus to monitor its services. If a check finds a problem, refer to Troubleshooting Self-Hosted Deployment components.
Prerequisites
Before you start, check the following:
- To use Grafana or collect service metrics, turn on Prometheus monitoring. In the manifest, set or
configuration.monitoring.prometheus.enabled: true.configuration.monitoring.prometheus.installedInCluster: true - To issue new PMM tokens, enable database monitoring.
Check the health of the pods
To check the health of the pods, run this command:
kubectl get pods -n <namespace>
Replace with the Kubernetes namespace where the solution is deployed, as defined in your file.
<namespace>manifest.yamlAll pods must have the status Running and show the expected number of ready containers. The exception is the pods of one-off jobs, such as and, with database monitoring enabled, . These pods finish with the status Completed.
novu-manager-jobpmm-integration-jobInspect unhealthy pods
If a pod isn't in a healthy state, inspect its details:
kubectl describe pod <pod-name> -n <namespace>
Check the pod logs:
kubectl logs <pod-name> -n <namespace>
For pods with multiple containers, specify the container:
kubectl logs <pod-name> -c <container-name> -n <namespace>
Sign in to Grafana
Grafana is at . Sign in as with the password that created. The password is in your persisted secrets file under . To read it from the cluster instead, run this command:
https://<appDomain>/grafanaadminshdctl secret generategrafana-credentialskubectl -n <namespace> get secret grafana-credentials -o jsonpath='{.data.admin-password}' | base64 -d
Grafana applies this secret only when it's first installed. After that, Grafana's dashboard and datasource updates still sign in with the password in the secret, so the password in the secret and the password in Grafana must match.
Change the Grafana password
To change the Grafana admin password, follow these steps:
-
In your persisted secrets file, set the new password under.
grafana-credentials -
Generate and deploy the secrets, as Regenerate and deploy the secrets on the upgrade page describes.
-
Apply the new password to Grafana, and restart Grafana:kubectl -n <namespace> get secret grafana-credentials -o jsonpath='{.data.admin-password}' | base64 -d | kubectl -n <namespace> exec -i statefulset/kube-prometheus-stack-grafana -c grafana -- grafana cli --config /etc/grafana/grafana.ini admin reset-admin-password --password-from-stdinkubectl -n <namespace> rollout restart statefulset/kube-prometheus-stack-grafana
Collect service metrics with your own Prometheus Operator
If you use your own Prometheus Operator (), set its to match the label . Every ServiceMonitor that the release creates carries this label. The bundled Prometheus already selects it.
installedInCluster: trueserviceMonitorSelectorrelease: kube-prometheus-stackFor the metrics that Prometheus collects, refer to Service metrics reference.
Issue new PMM tokens
To issue new tokens, for example after someone revokes a token in PMM, set for the job through the manifest's per-chart override, and deploy again:
FORCE_ROTATEpmm-integration-jobconfiguration: overrides: pmm-integration-job: values: env: FORCE_ROTATE: "true"
Both databases get new tokens and restart once. Remove the override afterward. If you leave it in place, the next upgrade issues new tokens and restarts both databases again.