Integrating Prometheus with Grafana on Kubernetes

6 min readAug 16, 2020

In this Article I Will cover how to make data persistent with the help of Kubernetes PVC and Config Map

What is Prometheus?

Prometheus is an open-source software application used for event monitoring and alerting. It records real-time metrics in a time series database (allowing for high dimensionality) built using an HTTP pull model, with flexible queries and real-time alerting. It is now a standalone open source project and maintained independently of any company

What is Grafana?

Grafana is an open-source solution for running data analytics, pulling up metrics that make sense of the massive amount of data & to monitor our apps with the help of cool customizable dashboards.

Grafana connects with every possible data source, commonly referred to as databases such as Graphite, Prometheus, Influx DB, ElasticSearch, MySQL, PostgreSQL, etc.

Grafana being an open-source solution also enables us to write plugins from scratch for integration with several different data sources.

The tool helps us study, analyze & monitor data over a period of time, technically called time-series analytics

Task Description:

Integrate Prometheus and Grafana and perform in the following way:

1. Deploy them as pods on top of Kubernetes by creating resources Deployment, ReplicaSet, Pods or Services

2. And make their data to remain persistent

3. And both of them should be exposed to the outside world

Let’s start with Prometheus:

First of all, we have to create PVC For Prometheus

Here is the Code:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:storage: 1Gi

I have allocated 1Gb to the Prometheus for its Persistent Storage.

Now we will create a Kubernetes config file of Prometheus:

apiVersion: v1
kind: ConfigMap
metadata: 
  name: prometheus-configuration
data:
  prometheus.yml: |-
    global:
      scrape_interval:     5s 
      evaluation_interval: 5s 
    scrape_configs:
     - job_name: 'prometheus'
       static_configs:
       - targets: ['localhost:9090']

This will create a file named prometheus.yml inside the deployed pod. This file is used by the Prometheus as Config file.

Now we are creating the Prometheus Deployment:

apiVersion: apps/v1
kind: Deployment
metadata: 
  name: prometheus-deployment
spec:
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
     app: prometheus
  template:
    metadata:
      name: prometheus-pod
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus-con
        image: prom/prometheus
        ports:
        - containerPort: 9090
          name: prometheus-con   
        volumeMounts:
        - name: configuration-mount 
          mountPath: /etc/prometheus
        - name: volume-mount
          mountPath: /prometheus
      volumes:
      - name: configuration-mount
        configMap:
          name: prometheus-configuration
      - name: volume-mount 
        persistentVolumeClaim:claimName: prometheus-pvc

Here i have mounted the previously created config file and PVC. The prometheus stores the config at /etc/prometheus/prometheus.yml, and all the data are stored in /prometheus folder.

To expose this Deployment to outside world, we use the following code

apiVersion: v1
kind: Service
metadata:
  name: prometheus-service
  labels:
    app: prometheus
spec:
  ports:
    - port: 9090
      nodePort: 30877
  selector:
    app: prometheustype: NodePort

I am setting the nodePort value to 30877. The prometheus is going to run on this port number and port value is set to 9090, because Prometheus default port is 9090.

Step 2: Setting up Kubernetes for Grafana

Here also we create the same files for Grafana with some different settings.

So, Here is the PVC file:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:storage: 1Gi

Now let’s create the Grafana Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana-deployment
  labels:
    app: grafana
spec:
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: grafana    
  template:
    metadata:
      name: grafana-pod
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana-con
        image: grafana/grafana
        ports:
        - name: grafana-con
          containerPort: 3000
        volumeMounts:
        - name: grafana-vol
          mountPath: /var/lib/grafana            
   
      volumes:
      - name: grafana-vol
        persistentVolumeClaim:claimName: grafana-pvc

Here i have only mounted the PVC to the path i.e. /var/lib/grafana, In this folder grafana stores its files and other configurations.

Now we Expose this Deployment to the outside world.

apiVersion: v1
kind: Service
metadata:
  name: grafana-svc
  labels:
    app: grafana
spec:
  selector:
    app: grafana
  ports:
  - port: 3000
    nodePort: 30878type: NodePort

Here i have given the value of nodePort as 30878 . This port is going to be used by us for connecting to the Grafana. Its internal port no is 3000. Basically we are connecting our 30878 port to the internal port 3000 of the Grafana.

Now we have two choices:

We can run these files one by one.
We can create one more file called kustomization.yml which will deploy all the files in one go.

So i am going to use the second approach. Let’s create the kustomization.yml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - prometheus-pvc.yml
  - grafana-pvc.yml
  - prometheus-config.yml
  - prometheus-deployment.yml
  - grafana-deployment.yml
  - grafana-svc.yml- prometheus-svc.yml

Step 3: Deployment on Kubernetes

For deploying we are going to use the following command inside the folder where our kustomization.yml file present.

kubectl create -k .

Here is the output of this command:

Now we know the port no of our the pods but we don’t know the IP. So for getting the IP we can use this command:

Now we know the IP address. Let’s connect to our Pods.

PROMETHEUS

Here you can see Prometheus is Running successfully.

GRAFANA

Our Grafana is successfully Running. But as you can see it is asking for username & password.

The Default username: admin

The Default password: admin

After entering the username and password it will ask you to change the password just give any password to change it. After that this will be the Home Screen: