Prometheus: running Pushgateway on Kubernetes with Helm and Terraform

By | 04/28/2023

We have a lot of AWS Lambda functions in the project, and developers want to be able to send metrics to our Prometheus to add their own alerts and graphs in Grafana.

For this, the functions use the Prometheus library, which allows these metrics to be created (see Prometheus: Building a Custom Prometheus Exporter in Python). But we cannot normally get this data from Prometheus, because the functions can live for several seconds, and Prometheus uses PULL -model, that is, he himself goes to exporters and extracts data from there.

What we can do in the case of AWS Lambda is to add Prometheus Pushgateway, which will have an external endpoint to which functions will send their metrics. Then, Prometheus will take this data from Pushgateway, which acts as a “proxy role”.

The general scheme of how Prometheus works and the role of Pushgateway is well represented in this diagram:

So today we will launch Prometheus Pushgateway in Kubernetes using its Helm chart and will install this chart using Terraform.

Pushgateway Helm chart

First, let’s see what we have in the values ​​of the Pushgateway chart.

In fact, for now, we can be interested in only two parameters – the Ingress and ServiceMonitor.

Let’s write our values, deploy the environment to Dev for the test, and then will add it to Terraform, through which Pushgateway will be deployed to all our monitoring environments:

ingress:
  enabled: true
  className: alb
  path: /
  pathType: Prefix
  annotations:
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-west-2:638***021:certificate/e97f17e9-33f9-46e7-9b2b-d50de8a72219
    alb.ingress.kubernetes.io/target-type: ip
  hosts:
    - test-pushgateway.monitoring.dev.example.com

serviceMonitor:
  enabled: true
  namespace: monitoring
  additionalLabels:
    release: prometheus

Here we describe the creation of Kubernetes Ingress, to which we will connect an SSL certificate from Amazon Certificate Manager to port 443 of the corresponding AWS Application Load Balancer. In the hosts field, specify a hostname that will be created in AWS Route53 using ExternalDNS.

And at the end, we are adding a ServiceMonitor with the label release: prometheus so that Prometheus from the Kube Prometheus Stack can see it.

Add the Prometheus repository:

[simterm]

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update

[/simterm]

And install Pushgateway to the namespace monitoring:

[simterm]

$ helm -n monitoring upgrade --install prometheus-pushgateway-test prometheus-community/prometheus-pushgateway -f pushgateway-values-dev.yaml 
...
NOTES:
1. Get the application URL by running these commands:
  http://test-pushgateway.monitoring.example.com/

[/simterm]

Check its Pod and Ingress:

[simterm]

$ kubectl get pod,ingress --namespace monitoring -l app.kubernetes.io/instance=prometheus-pushgateway-test
NAME                                               READY   STATUS    RESTARTS   AGE
pod/prometheus-pushgateway-test-6b5dfbdd7f-chzkf   1/1     Running   0          53s

NAME                                                    CLASS   HOSTS                                         ADDRESS                                                              PORTS   AGE
ingress.networking.k8s.io/prometheus-pushgateway-test   alb     test-pushgateway.monitoring.dev.example.com   internal-k8s-monitori-promethe-***-***.us-west-2.elb.amazonaws.com   80      54s

[/simterm]

Pushgateway has its own web interface – open access to its Pod on port 9091:

[simterm]

$ kubectl -n monitoring port-forward pod/prometheus-pushgateway-test-6b5dfbdd7f-chzkf 9091
Forwarding from 127.0.0.1:9091 -> 9091
Forwarding from [::1]:9091 -> 9091

[/simterm]

Go to http://localhost:9091 in the browser :

Next, let’s check whether Prometheus itself sees this Pushgateway – open port 9090 to the Prometheus Pod:

[simterm]

$ kk -n monitoring port-forward prometheus-prometheus-kube-prometheus-prometheus-0 9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

[/simterm]

Go to Status > Targets and look for the Pushgateway target:

And let’s check if the metrics from Pushgateway will get to Prometheus.

Since our Ingress/ALB is of type internal, i.e., not accessible from the Internet, let’s run a Kubernetes Pod with Ubuntu in the Kubernetes cluster:

[simterm]

$ kubectl run pod --rm -i --tty --image ubuntu -- bash
If you don't see a command prompt, try pressing enter.
root@pod:/# 

[/simterm]

Install curl:

[simterm]

root@pod:/# apt update && apt -y install curl

[/simterm]

But using the documentation, send a request, just do not specify the port here, because it is 443, SSL:

[simterm]

root@pod:/# echo "some_metric 3.14" | curl --data-binary @- https://test-pushgateway.monitoring.dev.exmaple.com/metrics/job/some_job

[/simterm]

Check in the Pushgateway:

And in the Prometheus:

Good, we’ve got our metric.

Pushgateway authentication

Since our Pushgateway works on the internal ALB, I haven’t done authentication yet, because it’s still in the process of testing, and it’s not yet known how it will work for developers.

But in values, there is a block with an example of starting a container with openshift/oauth-proxy, and in the Using OpenShift OAuth Proxy to secure your Applications on OpenShift, there is an example of its use, may be will try it later.

Terraform and Helm for Pushgateway

In our project, most of the resources in Kubernetes are deployed using Terraform and its helm_release resource.

We have many AWS accounts, and each one will have its own certificate ARN, so let’s get it using the Terraform Data Source (or you can create your own certificate using aws_acm_certificate):

data "aws_acm_certificate" "wildcard" {
  domain = "*.${var.root_domain}"
}

The root_domain variable is passed from the main module and looks like monitoring.dev.example.com.

Create a template for our values pushgateway-values.tpl.yaml:

ingress:
  enabled: true
  className: alb 
  path: /
  pathType: Prefix 
  annotations:
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/certificate-arn: ${acmWildcardArn}
    alb.ingress.kubernetes.io/target-type: ip
  hosts: 
    - ${pushgatewayUrl}
  
serviceMonitor:
  enabled: true
  namespace: monitoring
  additionalLabels:
    release: prometheus

And add helm_release, where in the values[] we set the template file and a couple of variables – the ARN of the certificate received with the help of data.aws_acm_certificate, and the domain for Ingress hosts:

resource "helm_release" "pushgateway" {
  name = "prometheus-pushgateway"

  namespace = kubernetes_namespace.monitoring.id

  repository  = "https://prometheus-community.github.io/helm-charts"
  chart       = "prometheus-pushgateway"
  version     = "2.1.3"
  max_history = 10

  values = [
    templatefile("${path.module}/configs/prometheus/pushgateway-values.tpl.yaml", {
      acmWildcardArn : data.aws_acm_certificate.wildcard.arn,
      pushgatewayUrl : "pushgateway.${var.root_domain}"
    })
  ]
}

Deploy it:

[simterm]

$ terraform apply
...
module.monitoring.helm_release.pushgateway: Creating...
module.monitoring.helm_release.pushgateway: Still creating... [10s elapsed]
module.monitoring.helm_release.pushgateway: Still creating... [20s elapsed]
module.monitoring.helm_release.pushgateway: Still creating... [30s elapsed]
module.monitoring.helm_release.pushgateway: Creation complete after 32s [id=prometheus-pushgateway]
...
Apply complete! Resources: 1 added, 1 changed, 0 destroyed.

[/simterm]

Check corresponding Ingress and Pod:

[simterm]

$ kk -n monitoring get ingress,pod -l app.kubernetes.io/instance=prometheus-pushgateway
NAME                                               CLASS   HOSTS                                    ADDRESS                                                              PORTS   AGE
ingress.networking.k8s.io/prometheus-pushgateway   alb     pushgateway.monitoring.dev.example.com   internal-k8s-monitori-promethe-***-***.us-west-2.elb.amazonaws.com   80      10m

NAME                                          READY   STATUS    RESTARTS   AGE
pod/prometheus-pushgateway-55b9f5ffd6-sm9ck   1/1     Running   0          10m

[/simterm]

Done.