apiVersion: autoscaling/v1 – an API groupautoscaling, pay attention to the API version, as in the v1 at the time of writing, scaling was available by the CPU metrics only, thus memory and custom metrics can be used only with the API v2beta2 (still, you can use v1 with annotations), see API Object.
spec.scaleTargetRef: specify for НРА which controller will be scaled (ReplicationController, Deployment, ReplicaSet), in this case, HPA will look for the Deployment object called deployment-example
spec.minReplicas, spec.maxReplicas: minimal and maximum pods to be running by this HPA
targetCPUUtilizationPercentage: CPU usage % from the requests when HPA will add or remove pods
Create it:
kubectl apply -f hpa-example.yaml
horizontalpodautoscaler.autoscaling/hpa-example created
Check:
kubectl get hpa hpa-example
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
In the first manifest for our HPA, we’ve used the autoscaling/v1 API which has the only targetCPUUtilizationPercentage parameter.
Check the autoscaling/v2beta1 – now, it has the metrics field added which is the MetricSpec array which can hold for new fields – the external, object, pods, resource.
in its turn the resource has the ResourceMetricSource, which holds two fields – targetAverageUtilization, and targetAverageValue, which are used now in themertics insted of the targetCPUUtilizationPercentage.
The TARGETS now displaying two metrics – CPU and memory.
It’s hard to make NGINX consume a lot of memory, so let’s go to see how much it uses now with the following kubectl command:
kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/default/pods/deployment-example-c4d6f96db-jv6nm | jq '.containers[].usage.memory'
"2824Ki"
2 megabytes.
Let’s update our PHA once again and set a new limit, but at this time we’ll use raw values instead of the percent – 1024Ki, 1 megabyte using targetAverageUtilization instead of the previously used targetAverageUtilization:
I0808 10:45:47.706771 1 adapter.go:94] successfully using in-cluster auth
E0808 10:45:47.752737 1 provider.go:209] unable to update list of all metrics: unable to fetch metrics for query "{__name__=~\"^container_.*\",container!=\"POD\",namespace!=\"\",pod!=\"\"}": Get http://prometheus.default.svc:9090/api/v1/series?match%5B%5D=%7B__name__%3D~%22%5Econtainer_.%2A%22%2Ccontainer%21%3D%22POD%22%2Cnamespace%21%3D%22%22%2Cpod%21%3D%22%22%7D&start=1596882347.736: dial tcp: lookup prometheus.default.svc on 172.20.0.10:53: no such host
And it’s good for metrics which are already present in the cluster like the memory_usage_bytes by default collected by the cAdvisor from all containers in the cluster.
HELP go_goroutines Number of goroutines that currently exist.
Kubernetes ServiceMonitor
The next thing to do is to add a ServiceMonitor to the Kubernetes cluster for our Prometheus Operator which will collect those metrics, check the Adding Kubernetes ServiceMonitor.
Check metrics now with port-forward:
kk -n monitoring port-forward prometheus-prometheus-prometheus-oper-prometheus-0 9090:9090
Note: The Prometheus resource includes a field called serviceMonitorSelector, which defines a selection of ServiceMonitors to be used. By default and before the version v0.19.0, ServiceMonitors must be installed in the same namespace as the Prometheus instance. With the Prometheus Operator v0.19.0 and above, ServiceMonitors can be selected outside the Prometheus namespace via the serviceMonitorNamespaceSelector field of the Prometheus resource. See Prometheus Operator