Monitoring and Observability

The Metrics Pipeline

Prometheus scrapes metrics from the cluster and applications; Grafana visualises them; Loki collects logs; Alertmanager fires alerts.

Getting Cluster Metrics

Most clusters ship a metrics server; if installed you can see usage directly:

kubectl top nodes
kubectl top pods

Installing a Monitoring Stack

The kube-prometheus-stack chart installs Prometheus + Grafana + Alertmanager quickly:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack
kubectl port-forward svc/monitoring-grafana 3000:80

Then open http://localhost:3000 (default login admin/prom-operator).

What to Watch

• Pod restarts and CrashLoopBackOff.
• CPU/memory utilisation per node and per pod.
• API server latency and errors.
• Disk fills on nodes running the container runtime.

Key Points

• Prometheus + Grafana is the standard Kubernetes monitoring stack.
• Watch trends, not just alerts; set alerts for sustained problems.
• Logs live outside pods - centralise them with Loki or an external log service.