Note: If you have missed my previous articles on Docker and Kubernetes, you can find them here:
Application deployment models evolution.
Getting started with Docker.Docker file and images.
Publishing images to Docker Hub and re-using them.
Docker- Find out what's going on.
Docker Networking- Part 1.
Docker Networking- Part 2.
Docker Swarm-Multi-Host container Cluster.Docker Networking- Part 3 (Overlay Driver).
Introduction to Kubernetes.Kubernetes- Diving in (Part 1)-Installing Kubernetes multi-node cluster.
Kubernetes-Diving in (Part2)- Services.
Kubernetes- Infrastructure As Code with Yaml (part 1).
Kubernetes- Infrastructure As Code Part 2- Creating PODs with YAML.
Kubernetes Infrastructure-as-Code part 3- Replicasets with YAML.
Kubernetes Infrastructure-as-Code part 4 - Deployments and Services with YAML.
Deploying a microservices APP with Kubernetes.
Kubernetes- Time based scaling of deployments with python client.
Kubernetes Networking - The Flannel network explained.
Kubernetes- Installing and using kubectl top for monitoring nodes and PoDs
Kubernetes Administration- Scheduling
Kubernetes Administration- Storage
Kubernetes Administration- Users
Kubernetes Administration - Network Policies with Calico network plugin
Kubernetes Administration - Managing Kubernetes Clusters with Rancher
Kubernetes Administration - Package Management with Helm
Kubernetes Administration - Monitoring cluster health with Prometheus
Kubernetes Administration - DevOps- Prometheus- The need and high-level architecturePrometheus is a powerful tool that can be used to monitor nodes and microservice applications. Prometheus can be installed with Helm and its architecture enables third-party tools supporting PromQL to query data. Grafana is an observability tool that can be used to visualize Prometheus data. In addition to supporting Prometheus, Grafana also supports datasets from various other sources like cloud providers, hosts, etc.
If you intended to follow along, ensure that you have Helm installed- you can refer to my previous article for this.
Installing Grafana
The simplest way to install Grafana is with a helm chart for Kube-Prometheus-stack (Grafana is part of the install). This chart installs Prometheus, Grafana, and a bunch of other tools that enable easy monitoring.
root@sathish-vm2:/home/sathish# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
root@sathish-vm2:/home/sathish# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "prometheus-community" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈
root@sathish-vm2:/home/sathish# helm install prometheous-vm prometheus-community/kube-prometheus-stack
NAME: prometheous-vm
LAST DEPLOYED: Sun Apr 4 07:49:44 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
kubectl --namespace default get pods -l "release=prometheous-vm"
Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.
root@sathish-vm2:/home/sathish/example-voting-app# kubectl get pods
NAME READY STATUS RESTARTS AGE
alertmanager-prometheous-vm-kube-promet-alertmanager-0 2/2 Running 0 16s
prometheous-vm-grafana-75cdd4c75-qv9wd 2/2 Running 0 19s
prometheous-vm-kube-promet-operator-55d66df7db-bll85 1/1 Running 0 19s
prometheous-vm-kube-state-metrics-858f884bc8-dxknj 1/1 Running 0 19s
prometheous-vm-prometheus-node-exporter-bmz5p 1/1 Running 0 19s
prometheous-vm-prometheus-node-exporter-lnnhq 1/1 Running 0 19s
prometheus-prometheous-vm-kube-promet-prometheus-0 2/2 Running 1 15sAccessing the service
All the installed services are of type ClusterIP and cannot be accessed outside of the cluster. If you are running it on the cloud, it is best to use Ingress to expose Grafana and Prometheus. For local networks, you can use the NodePort service type.
root@sathish-vm2:/home/sathish/example-voting-app# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 61s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 171d
prometheous-vm-grafana ClusterIP 10.105.48.224 <none> 80/TCP 65s
prometheous-vm-kube-promet-alertmanager ClusterIP 10.107.67.154 <none> 9093/TCP 64s
prometheous-vm-kube-promet-operator ClusterIP 10.102.177.200 <none> 443/TCP 64s
prometheous-vm-kube-promet-prometheus ClusterIP 10.106.158.240 <none> 9090/TCP 64s
prometheous-vm-kube-state-metrics ClusterIP 10.96.39.244 <none> 8080/TCP 64s
prometheous-vm-prometheus-node-exporter ClusterIP 10.104.169.34 <none> 9100/TCP 64s
prometheus-operated ClusterIP None <none> 9090/TCP 60s
prometheus-prometheous-vm-kube-promet-prometheus-0 ClusterIP 10.102.92.28 <none> 9090/TCP,8080/TCP 2m29s
Adding a Data Source
As I mentioned before, Grafanna can collect data from various data sources. What we are interested in is Prometheus. Click the "Add your first data source" option and add Prometheus.
My Prometheus service is of type ClusterIP, so I need to use the corresponding IP
root@sathish-vm2:/home/sathish# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 22m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 171d
prometheous-vm-grafana NodePort 10.96.123.173 <none> 80:31389/TCP,3000:30067/TCP 15m
prometheous-vm-kube-promet-alertmanager ClusterIP 10.104.94.128 <none> 9093/TCP 22m
prometheous-vm-kube-promet-operator ClusterIP 10.101.130.227 <none> 443/TCP 22m
prometheous-vm-kube-promet-prometheus ClusterIP 10.102.234.84 <none> 9090/TCP 22m
prometheous-vm-kube-state-metrics ClusterIP 10.104.106.132 <none> 8080/TCP 22m
prometheous-vm-prometheus-node-exporter ClusterIP 10.100.242.56 <none> 9100/TCP 22m
prometheus-operated ClusterIP None <none> 9090/TCP 22mNote: It might be a good idea to change the service type of Prometheus UI, so it can be accessed from another machine. I have changed it to NodePort on my setup
root@sathish-vm2:/home/sathish/example-voting-app# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 15m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 171d
prometheous-vm-grafana NodePort 10.104.141.104 <none> 80:30067/TCP 12m
prometheous-vm-kube-promet-alertmanager ClusterIP 10.102.236.148 <none> 9093/TCP 15m
prometheous-vm-kube-promet-operator ClusterIP 10.102.57.69 <none> 443/TCP 15m
prometheous-vm-kube-promet-prometheus ClusterIP 10.100.201.230 <none> 9090/TCP 15m
prometheous-vm-kube-state-metrics ClusterIP 10.105.155.65 <none> 8080/TCP 15m
prometheous-vm-prometheus-node-exporter ClusterIP 10.96.183.206 <none> 9100/TCP 15m
prometheus-operated ClusterIP None <none> 9090/TCP 15m
prometheus-prometheous-vm-kube-promet-prometheus-0 NodePort 10.102.92.28 <none> 9090:30090/TCP,8080:31151/TCP 2m24s
Once the data source is added, you can create a dashboard to monitor various things.
Prometheus monitors a list of objects exposed by the Service Monitor object. Here is a list of default service monitor objects that the Helm chart installs.
root@sathish-vm2:/home/sathish/example-voting-app# kubectl get servicemonitor
NAME AGE
prometheous-vm-kube-promet-alertmanager 19m
prometheous-vm-kube-promet-apiserver 19m
prometheous-vm-kube-promet-coredns 19m
prometheous-vm-kube-promet-grafana 19m
prometheous-vm-kube-promet-kube-controller-manager 19m
prometheous-vm-kube-promet-kube-etcd 19m
prometheous-vm-kube-promet-kube-proxy 19m
prometheous-vm-kube-promet-kube-scheduler 19m
prometheous-vm-kube-promet-kube-state-metrics 19m
prometheous-vm-kube-promet-kubelet 19m
prometheous-vm-kube-promet-node-exporter 19m
prometheous-vm-kube-promet-operator 19m
prometheous-vm-kube-promet-prometheus 19mThe release label registers the serviceMonitor with Prometheus enabling it to collect data from a particular serviceMonitor object.
root@sathish-vm2:/home/sathish/example-voting-app# kubectl describe servicemonitor prometheous-vm-kube-promet-grafana
Name: prometheous-vm-kube-promet-grafana
Namespace: default
Labels: app=kube-prometheus-stack-grafana
app.kubernetes.io/managed-by=Helm
chart=kube-prometheus-stack-14.5.0
heritage=Helm
release=prometheous-vm
Monitoring Apps
Grafanna can be used to monitor the status of various components of applications. I have already deployed the voting app (described here ) and now components of app can be monitored with Grafanna. To enable the app component to export data an exporter is required. An exporter converts data to a format that Prometheus understands and also exposes a metrics endpoint. A list of prebuilt exporters is available here:
If desirable exporter for a particular app component can be installed after customizing parameters.
helm show values prometheus-community/prometheus-postgres-exporter > postgres.yaml
# Edit and change the highlighted fields
# Default values for prometheus-postgres-exporter.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
image:
repository: quay.io/prometheuscommunity/postgres-exporter
tag: v0.9.0
pullPolicy: IfNotPresent
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# pullSecrets:
# - myRegistrKeySecretName
service:
type: ClusterIP
port: 80
targetPort: 9187
name: http
labels:
release: prometheous-vm
annotations: {}
...........................................
datasource:
# Specify one of both datasource or datasourceSecret
#ClusterIP of DB pod
host: 10.101.175.235
user: postgres
# Only one of password and passwordSecret can be specified
password: postgres
#Deploy the chart
root@sathish-vm2:~# helm install postgres-exporter prometheus-community/prometheus-postgres-exporter -n vote -f postgres.yaml
NAME: postgres-exporter
LAST DEPLOYED: Sun Apr 4 11:12:44 2021
NAMESPACE: vote
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace vote -l "app=prometheus-postgres-exporter,release=postgres-exporter" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl port-forward $POD_NAME 8080:80Grafana dashboards
Grafana has a list of prebuilt dashboards that can be used to monitor the status of various resources.
Customized dashboards can also be created from UI.
Playlists could be created from UI that automatically circles through dashboards- this could be useful for DevOps operators.
Grafanna is a very useful tool for DevOps monitoring and is widely deployed. I addition to the capabilities described in this article, Grafanna supports lots of features and plugins. More details about Grafanna can be found here.
Hope this short introduction to Grafana was useful. Till next time, ciao and have a great week ahead.
Comments