Kubernetes HorizontalPodAutoscaler automatically scales Kubernetes Pods under ReplicationController, Deployment, or ReplicaSet controllers basing on its CPU, memory, or other metrics.

It was shortly discussed in the Kubernetes: running metrics-server in AWS EKS for a Kubernetes Pod AutoScaler post, now let’s go deeper to check all options available for scaling.

For HPA you can use three API types:

• metrics.k8s.io: default metrics, basically provided by the  metrics-server
• custom.metrics.k8s.io: metrics, provided by adapters from inside of a cluster, for example – Microsoft Azure Adapter, Google Stackdriver, Prometheus Adapter (the Prometheus Adapter will be used in this post later), check the full list here>>>
• external.metrics.k8s.io: similar to the Custom Metrics API, but metrics are provided by an external system, such as AWS CloudWatch

Documentation: Support for metrics APIs, and Custom and external metrics for autoscaling workloads.

Besides the HorizontalPodAutoscaler (HPA) you also can use Vertical Pod Autoscaling (VPA) and they can be used together although with some limitations, see Horizontal Pod Autoscaling Limitations.

Create HorizontalPodAutoscaler

Let’s start with a simple HPA which will scale pods basing on CPU usage:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-example
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: deployment-example
  minReplicas: 1
  maxReplicas: 5
  targetCPUUtilizationPercentage: 10

Here:

• apiVersion: autoscaling/v1 – an API groupautoscaling, pay attention to the API version, as  in the v1 at the time of writing, scaling was available by the CPU metrics only, thus memory and custom metrics can be used only with the API v2beta2 (still, you can use v1 with annotations), see API Object.
• spec.scaleTargetRef: specify for НРА which controller will be scaled (ReplicationController, Deployment, ReplicaSet), in this case, HPA will look for the Deployment object called deployment-example
• spec.minReplicas, spec.maxReplicas: minimal and maximum pods to be running by this HPA
• targetCPUUtilizationPercentage: CPU usage % from the requests when HPA will add or remove pods

Create it:

Check:

Currently, its TARGETS has the <unknown> value as there are no pods created yet, but metrics are already available:

Add the called deployment-example Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-example
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      application: deployment-example
  template:
    metadata:
      labels:
        application: deployment-example
    spec: 
      containers:
      - name: deployment-example-pod
        image: nginx
        ports:
          - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 100Mi

Here we defined Deployment which will spin up one pod with NINGX with requests for 100 millicores and 100 mebibyte memory, see Kubernetes best practices: Resource requests and limits.

Create it:

Check the HPA now:

Our НРА found the deployment and started checking its pods’ metrics.

Let’s check those metrics – find a pod:

And run the following API request:

CPU usage – zero, memory – 2 megabytes, let’s confirm with top:

“Alright, these guys!” (с)

Okay – we got our metrics, we’ve created the HPA and deployment – let’s go to see how the scaling will work here.

Load testing HorizontalPodAutoscaler scaling

For load testing, we can use the loadimpact/loadgentest-wrk utility image.

Now, run ports redirect from the local workstation to the pod with NGINX, as we didn’t add any LoadBalancer (see Kubernetes: ClusterIP vs NodePort vs LoadBalancer, Services, and Ingress – an overview with examples):

Check resources once again:

0% CPU is used, one pod is running (REPLICAS 1).

Run the test:

Here:

• open 100 connections using 100 threads
• run the test for 5 minutes

Check the pod:

CPU usage now 49mi, and in the requests, we’ve set the 10 milicpu limit – check the HPA:

TARGETS 49% from the 10% limit and our HPA started new pods – REPLICAS 4:

Multi-metrics scaling

Okay, we were able to scale by the CPU usage, but what if you want to scale by both CPU and memory usage?

Add another Resource with memory limit set to the same 10%:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-example
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: deployment-example
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 10
  - type: Resource
    resource:
      name: memory 
      targetAverageUtilization: 10

autoscaling API group versions

Let’s go back to the API versions.

In the first manifest for our HPA, we’ve used the autoscaling/v1 API which has the only targetCPUUtilizationPercentage parameter.

Check the autoscaling/v2beta1 – now, it has the metrics field added which is the MetricSpec array which can hold for new fields – the external, object, pods, resource.

in its turn the resource has the ResourceMetricSource, which holds two fields – targetAverageUtilization, and targetAverageValue, which are used now in themertics insted of the  targetCPUUtilizationPercentage.

Apply the HPA update:

Check it:

The TARGETS now displaying two metrics – CPU and memory.

It’s hard to make NGINX consume a lot of memory, so let’s go to see how much it uses now with the following kubectl command:

2 megabytes.

Let’s update our PHA once again and set a new limit, but at this time we’ll use raw values instead of the percent – 1024Ki, 1 megabyte using targetAverageUtilization instead of the previously used targetAverageUtilization:

...
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 10
  - type: Resource
    resource:
      name: memory
      targetAverageValue: 1024Ki

Apply and check:

REPLICAS == 3 now, pods were scaled, check the value from the TARGETS – convert it to kilobytes:

And check real memory usage:

2492 ~= 2496Ki.

Okay, so – we are able now to scale the Deployment by both CPU and memory usage.

Custom Metrics

Memory metrics scaling

Apart from metrics provided by the API server and cAdvisor we can use any other metrics, for example – metrics, collected by Prometheus.

It can be metrics collected by a Cloudwatch exporter, Prometheus’ node_exporter, or metrics from an application.

Documention is here>>>.

As we are using Prometheus (see Kubernetes: monitoring with Prometheus – exporters, a Service Discovery, and its roles and Kubernetes: a cluster’s monitoring with the Prometheus Operator) so let’s add its adapter:

If you’ll try to access external or custom API endpoints now you’ll get the error:

Install the adapter from the Helm chart:

Wait for a minute or two and check API again:

Well, but why the "resources":[] is empty?

Check the adapter’s pod logs:

Here is the error:

dial tcp: lookup prometheus.default.svc on 172.20.0.10:53: no such host

Let try to access our Prometheus Operator from the pod by its Service DNS name:

Okay, we are able to reach it using the prometheus-prometheus-oper-prometheus.monitoring.svc.cluster.local:9090 URL.

Edit the adapter’s Deployment:

Update its prometheus-url:

...
    spec:
      affinity: {}
      containers:
      - args:
        - /adapter
        - --secure-port=6443
        - --cert-dir=/tmp/cert
        - --logtostderr=true
        - --prometheus-url=http://prometheus-prometheus-oper-prometheus.monitoring.svc.cluster.local:9090
...

Apply changes and check again:

Nice – we’ve got our metrics and can use them now in the HPA.

Check the API server for the memory_usage_bytes metric:

Update the НРА’s manifest:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-example
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: deployment-example
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metricName: memory_usage_bytes
      targetAverageValue: 1024000

Check the HPA’s values now:

Apply the latest changes:

Check again:

Still have 1 replica, check events:

And HPA again:

Great – “It works!” (c)

And it’s good for metrics which are already present in the cluster like the  memory_usage_bytes by default collected by the cAdvisor from all containers in the cluster.

Let’s try to use more custom metrics, for example – let’s scale a Gorush-server by using its own metrics, see Kubernetes: running a push-server with Gorush behind an AWS LoadBalancer.

Application-based metrics scaling

So, we have the Gorush server running in our cluster which is used to send push-notifications to mobile clients.

It has the built-in  /metrics endpoint which returns standard time-series metrics that can be used in Prometheus.

To run the testing Gorush server we can use such Service, ConfigMap, andDeployment:

apiVersion: v1
kind: Service
metadata:
  name: gorush
  labels:
    app: gorush
    tier: frontend
spec:
  selector:
    app: gorush
    tier: frontend
  type: ClusterIP
  ports:
  - name: gorush
    protocol: TCP
    port: 80
    targetPort: 8088
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: gorush-config
data:
  stat.engine: memory
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gorush
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gorush
      tier: frontend
  template:
    metadata:
      labels:
        app: gorush
        tier: frontend
    spec:
      containers:
      - image: appleboy/gorush
        name: gorush
        imagePullPolicy: Always
        ports:
        - containerPort: 8088
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8088
          initialDelaySeconds: 3
          periodSeconds: 3
        env:
        - name: GORUSH_STAT_ENGINE
          valueFrom:
            configMapKeyRef:
              name: gorush-config
              key: stat.engine

Create a dedciated namespace:

Create the application:

Check pods:

Gorush Service:

Run the port-forward to its pod:

Check metrcis:

Or another way: by using its Service name, we can reach it directly.

Find the Service:

Open proxy to the API-server:

And connect to the Service:

Kubernetes ServiceMonitor

The next thing to do is to add a ServiceMonitor to the Kubernetes cluster for our Prometheus Operator which will collect those metrics, check the Adding Kubernetes ServiceMonitor.

Check metrics now with port-forward:

Try to access them:

The "data":[] is empty now – or Prometheus doesn’t collect those metrics yet.

Define the ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    serviceapp: gorush-servicemonitor
    release: prometheus
  name: gorush-servicemonitor
  namespace: monitoring
spec:     
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 15s
    port: gorush
  namespaceSelector:
    matchNames: 
    - eks-dev-1-gorush
  selector: 
    matchLabels:
      app: gorush

Note: The Prometheus resource includes a field called serviceMonitorSelector, which defines a selection of ServiceMonitors to be used. By default and before the version v0.19.0, ServiceMonitors must be installed in the same namespace as the Prometheus instance. With the Prometheus Operator v0.19.0 and above, ServiceMonitors can be selected outside the Prometheus namespace via the serviceMonitorNamespaceSelector field of the Prometheus resource.
See Prometheus Operator

Create this ServiceMonitor:

Check it in the Targets:

UP, good.

And in a couple of minutes check for metrics again:

Or in this way:

Nice, we’ve got our metrics, let’s go ahead and use them in the HorizontalPodAutoscaler of this Deployment.

Check the metrics groups available here:

Add a new manifest with the HPA which will use the gorush_queue_usage from the Pods group:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: gorush-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: gorush
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metricName: gorush_total_push_count
      targetAverageValue: 2

With such settings, the HPA has to scale pods once the gorush_total_push_count‘s value will be over 2.

Create it:

Check its value now:

Check the НРА:

TARGETS 0/1, okay.

Send a push:

Check the gorush_total_push_count metric again:

One push was sent.

Check the HPA one more time:

TARGETS 1/2, REPLICAS still has 1, send another push and check the events:

And the HPA:

Great! So, scaling by the gorush_total_push_count is working.

But here a trap: the gorush_total_push_count is cumulative metric, e.g. on the Production graph it will look like the next:

And in such a case our HPA will scale pods till the end of time.

Prometheus Adapter ConfigMap – seriesQuery and metricsQuery

To mitigate this let’s add an own metric.

The Prometheus Adapter has its own ConfigMap:

Wich contains the config.yaml, see its example here>>>.

Create a PromQL query which will return pushes count per second:

rate(gorush_total_push_count{instance="push.server.com:80",job="push-server"}[5m])

Update the ConfigMap and add new query there:

apiVersion: v1
data:
  config.yaml: |
    rules:
    - seriesQuery: '{__name__=~"gorush_total_push_count"}'
      seriesFilters: []
      resources:
        overrides:
          namespace:
            resource: namespace
          pod:
            resource: pod
      name:
        matches: ""
        as: "gorush_push_per_second"
      metricsQuery: rate(<<.Series>>{<<.LabelMatchers>>}[5m])

Save and exit, check it:

Re-create the pod so it will apply the changes (see Kubernetes: ConfigMap and Secrets – data auto-reload in pods):

Check it:

Update the НРА to use the gorush_push_per_second:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: gorush-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: gorush
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metricName: gorush_push_per_second
      targetAverageValue: 1m

Check it:

Events:

And tje HPA now:

Done.

Useful links

• Kubernetes Autoscaling in Production: Best Practices for Cluster Autoscaler, HPA and VPA
• Ultimate Kubernetes Resource Planning Guide
• Horizontal Autoscaling in Kubernetes #2 – Custom Metrics
• Kubernetes HPA : ExternalMetrics+Prometheus
• Prometheus Custom Metrics Adapter
• Horizontal Pod Autoscaling (HPA) triggered by Kafka event
• Custom and external metrics for autoscaling workloads
• Prometheus Metrics Based Autoscaling in Kubernetes
• Kubernetes best practices: Resource requests and limits
• Знакомство с Kubernetes. Часть 19: HorizontalPodAutoscaler
• How to Use Kubernetes for Autoscaling
• Horizontal Pod Autoscaling by memory
• Autoscaling apps on Kubernetes with the Horizontal Pod Autoscaler
• Horizontally autoscale Kubernetes deployments on custom metrics
• Kubernetes pod autoscaler using custom metrics
• Kubernetes HPA Autoscaling with Custom and External Metrics
• Horizontal pod auto scaling by using custom metrics
• Horizontal Pod Autoscale with Custom Prometheus Metrics
• Kubernetes HPA using Custom Metrics
• Kubernetes HPA Autoscaling with Custom Metrics
• Building a K8s Autoscaler with Custom Metrics

Also published on Medium.