Continuing with the Kubernetes: monitoring with Prometheus – exporters, a Service Discovery, and its roles, where we configured Prometheus manually to see how it’s working – now, let’s try to use Prometheus Operator installed via Helm chart.

So, the task is spin up a Prometheus server and all necessary exporter in an AWS Elastic Kubernetes Cluster and then via /federation pass metrics to our “central” Prometheus server with Alertmanager alerts and Grafana dashboards.

A bit confusing is the whole set of such Helm charts – there is a “simple” Prometheus chart, and  kube-prometheus, and prometheus-operator:

• Bitnami Prometheus Operator – https://github.com/bitnami/charts/tree/master/bitnami/prometheus-operator/
• CoreOS Prometheus Operator – https://github.com/coreos/prometheus-operator
• Helm Community Prometheus Operator – https://github.com/helm/charts/tree/master/stable/prometheus-operator
• CoreOS kube-prometheus – https://github.com/coreos/kube-prometheus
• And “just a Prometheus” – https://github.com/helm/charts/tree/master/stable/prometheus

Although if look for it via helm search – it returns the only one prometheus-operator:

The difference between  stable/prometheus and stable/prometheus-operator is that Operator has built-in Grafana with a set of ready for use dashboards and set of ServiceMonitors to collect metrics from a cluster’s services such as the CoreDNS, API Server, Scheduler, etc.

So, as mentioned – we will use the stable/prometheus-operator.

Prometheus Operator deployment

Deploy it with Helm:

Check pods:

Note: alias kk="kubectl" >> ~/.bashrc

So, the Prometheus Operator’s Helm chart created a whole bunch of services – Prometheus itself, Alertmanager, Grafana, plus a set of ServiceMonitors:

The ServiceMonitors role will be reviewed in this post later in the Adding Kubernetes ServiceMonitor part.

Grafana access

Let’s go to see which dashboards are shipped with Grafana.

Find Grafana’s pod:

Run port-forward:

Open localhost:3000, log in with the admin username and prom-operator password and you’ll see a lot of ready for user graphs:

At the time of writing Prometheus Operator is shipped with Grafana version 7.0.3.

Actually, we don’t need Grafana and Alertmanager here, as they are used on our “central” monitoring server, so let’s remove them from here.

Prometheus Operator configuration

Prometheus Operator uses Custom Resource Definitions which describes all its components:

For example, the prometheuses.monitoring.coreos.com CRD describes a Custom Resource named Prometheus:

And then you can use this name as a common Kubernetes object like pods, nodes, volumes, etc by using its name from the names field:

For now, we are interested in the serviceMonitorSelector record:

...
serviceMonitorSelector: 
  matchLabels:
    release: prometheus

Which defines which ServiceMonitors will be added under the observation of the Prometheus server in the cluster.

Adding new application under monitoring

Now, let’s try to add an additional service under monitoring:

1. spin up a Redis server
2. and redis_exporter
3. will add ServiceMonitor
4. and eventually, we will configure Prometheus Operator to use the ServiceMonitor to collect metrics from the redis_exporter

Redis server launch

Create a namespace to make this setup more realistic, when a monitored application is located in one namespace, while monitorings services are living in an another:

Run Redis:

Create a Service object for Redis network communication:

Check it:

Okay – Redis is working, now let’s add its Prometheus exporter.

redis_exporter launch

Install it with Helm:

Check its Service:

And its endpoint:

Go to check metrics – spin up a new pod, for example with Debian, and install curl:

And make a request to the redis_exporter:

Or without an additional port – just by running port-forward to the redis-svc:

And run from your local PC:

Okay – we’ve got our Redis Server application and its Prometheus exporter with metrics available via the /metrics URI and 9121 port.

The next thing is t configure Prometheus Operator to collect those metrics to its database so later they are pulled by the “central” monitoring via the Prometheus federation.

Adding Kubernetes ServiceMonitor

Check our redis_exporter‘s labels:

And check serviceMonitorSelector‘s Selector of the prometheus resource created above:

At the end of output find the following lines:

...
serviceMonitorSelector: 
  matchLabels:
    release: prometheus

So, Prometheus will look for ServiceMonitors with the release tag with the prometheus value.

Create an additional ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    serviceapp: redis-servicemonitor
    release: prometheus
  name: redis-servicemonitor
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 15s
    port: redis-exporter
  namespaceSelector:
    matchNames:
    - monitoring
  selector:
    matchLabels:
      release: redis-exporter

In its labels we’ve set the release: prometheus so Prometheus can find it, and in the selector.matchLabels – specified to look for any Service tagged as release: redis-exporter.

Apply it:

Check Prometheus’ Targets:

Redis Server metrics:

Great – “It works!” (c)

What is next?

And the next thing to do is to remove Alertmanager and Grafana from our Prometheus Operator stack.

Prometheus Operator Helm deployment and its services configuration

Actually, why need to remove Grafana? It has really useful dashboards ready, so let’s leave it there.

The better way seems to be:

• on each EKS we will have Prometheus Operator with Grafana but without Alertmanager
• on cluster’s local Prometheus servers  let’s leave the default retention period to store metrics set to 2 weeks
• will remove the Alertmanager from there – will use our “central” monitoring’s Alertmanager with already defined routs and alerts (see the Prometheus: Alertmanager’s alerts receivers and routing based on severity level and tags post for details about routing)
• and on the central monitoring server – we will keep our metrics for one year and will have some Grafana dashboards there

So, we need to add two LoadBalacners – one with the internet-facing type for the Grafana service and another one – internal – for the Prometheus, because it will communicate with our central monitoring host via AWS VPC Peering.

And later well need to add this to our automation – AWS Elastic Kubernetes Service: a cluster creation automation, part 2 – Ansible, eksctl.

But for now, let’s do it manually.

Well, what do we need to change in Prometheus Operator’s default deployment?

• drop Alertmanager
• add settings for:
  • Prometheus and Grafana – they must be deployed behind an AWS LoadBalancer
  • set a login:pass for Grafana

All available options can be found in the documentation – https://github.com/helm/charts/tree/master/stable/prometheus-operator.

At his moment, let’s start with removing Alertmanager.

To do so need to pass the alertmanager.enabled parameter.

Check its pod now:

Redeploy it and via the --set specify the alertmanager.enabled=false:

Check pods again – no Alertmanager must be present now in the stack.

An AWS LoadBalancer configuration

To make our Grafana available externally from the Internet we need to add an Ingress resource for Grafana and a dedicated  Ingress for the Prometheus.

What do we have in the documentation:

Add the Ingress for Grafana:

Er…

The documentation says nothing about LoadBalancers configuration, but I googled some in the Jupyter’s docs here.

Let’s try it – now via the values.yaml file to avoid a bunch of --set.

Add the hosts to the file:

grafana:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: "alb"
      alb.ingress.kubernetes.io/scheme: "internet-facing"
    hosts:
      - "dev-0.eks.monitor.example.com"

Deploy:

Check the Kubernetes ALB Controller logs:

I0617 11:37:48.272749       1 tags.go:43] monitoring/prometheus-grafana: modifying tags {  ingress.k8s.aws/cluster: “bttrm-eks-dev-0”,  ingress.k8s.aws/stack: “monitoring/prometheus-grafana”,  kubernetes.io/service-name: “prometheus-grafa
na”,  kubernetes.io/service-port: “80”,  ingress.k8s.aws/resource: “monitoring/prometheus-grafana-prometheus-grafana:80”,  kubernetes.io/cluster/bttrm-eks-dev-0: “owned”,  kubernetes.io/namespace: “monitoring”,  kubernetes.io/ingress-name
: “prometheus-grafana”} on arn:aws:elasticloadbalancing:us-east-2:534***385:targetgroup/96759da8-e0b8253ac04c7ceacd7/35de144cca011059
E0617 11:37:48.310083       1 controller.go:217] kubebuilder/controller “msg”=”Reconciler error” “error”=”failed to reconcile targetGroups due to failed to reconcile targetGroup targets due to prometheus-grafana service is not of type NodePort or LoadBalancer and target-type is instance”  “controller”=”alb-ingress-controller” “request”={“Namespace”:”monitoring”,”Name”:”prometheus-grafana”}

Okay – add the /target-type: "ip" so AWS ALB will send traffic directly to the Grafana’s pod instead of WorkerNodes port and let’s add valid HTTP codes:

grafana:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: "alb"  
      alb.ingress.kubernetes.io/scheme: "internet-facing"
      alb.ingress.kubernetes.io/target-type: "ip"
      alb.ingress.kubernetes.io/success-codes: 200,302
    hosts:
      - "dev-0.eks.monitor.example.com"

Or another way – reconfigure the Grafana’s Service to use the NodePort:

grafana:
  service:
    type: NodePort
    port: 80
    annotations: {}
    labels: {}  
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: "alb"
      alb.ingress.kubernetes.io/scheme: "internet-facing"
      alb.ingress.kubernetes.io/success-codes: 200,302
    hosts:
      - "dev-0.eks.monitor.example.com"

Redeploy Operator’s stack:

Wait for a minute and check AWS LoadBalancer again:

But now dev-0.eks.monitor.example.com will return the 404 code from our LoadBalancer:

Why does this happen?

1. the ALB accepts a request to the dev-0.eks.monitor.example.com URL and redirects it to the WorkerNodes TargetGroup with Grafana’s pod(s)
2. Grafana returns the 302 redirect code to the /login URI
3. the request is going back to the ALB but at this time with the /login URI

Check the ALB’s Listener rules:

Well, here it is – we will return 404 code for any URI requests excepting /.  Accordingly the /login request also will be dropped with the 404.

I’d like to see any comments from developers, who set this as the default behavior for the path.

Go back to our values.yaml, replace the path from the / to the /*.

And update the hosts field to avoid the “Invalid value: []networking.IngressRule(nil): either `backend` or `rules` must be specified” error.

To not tie yourself to any specific domain name set it to the "" value:

grafana:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: "alb"
      alb.ingress.kubernetes.io/target-type: "ip"
      alb.ingress.kubernetes.io/scheme: "internet-facing"
      alb.ingress.kubernetes.io/success-codes: 200,302
    hosts:
      - ""
    path: /*

Redeploy, check:

Open in a browser:

And even more – now we are able to see all graphs in the Lens  utility used by our developers – earlier they got the “Metrics are not available due to missing or invalid Prometheus configuration” и “Metrics not available at the moment” error messages:

Actually, that’s all to start using Prometheus Operator to monitor Your Kubernetes cluster

Useful links

• Kubernetes monitoring with Prometheus – Prometheus operator tutorial
• Prometheus Operator – Installing Prometheus Monitoring Within The Kubernetes Environment
• Quick Start with Prometheus Monitoring
• Prometheus Operator — How to monitor an external service
• How to Set Up DigitalOcean Kubernetes Cluster Monitoring with Helm and Prometheus Operator
• Configuring prometheus-operator helm chart with AWS EKS
• Zero to JupyterHub with Kubernetes
• Howto expose prometheus, grafana and alertmanager with nginx ingress
• Howto expose prometheus, grafana and alertmanager with nginx ingressEKS: failed to reconcile targetGroups due to failed to load serviceAnnotation due to no object matching key

Also published on Medium.