We have a VictoriaLogs Helm chart with a PVC size of 30 GB, which is no longer enough for us, and we need to increase it.
But the problem is that .spec.volumeClaimTemplates[*].spec.resources.requests.storage in STS is immutable, that is, we can’t just change the size through values.yaml file, because it will lead to the error“Forbidden: updates to statefulset spec for fields other than ‘replicas’, ‘ordinals’, ‘template’, ‘updateStrategy’, ‘revisionHistoryLimit’, ‘persistentVolumeClaimRetentionPolicy’ and ‘minReadySeconds’ are forbidden“.
The chart values now look like this:
victoria-logs-single: server: persistentVolume: enabled: true storageClassName: gp2-retain size: 30Gi retentionPeriod: 7d
And with the default type of StatefulSet in the chart, the volumeClaimTemplates is used to create PVCs:
...
volumeClaimTemplates:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: server-volume
...
spec:
...
resources:
requests:
storage: {{ $app.persistentVolume.size }}
...
If instead of STS there was a Deployment type, then in the VictoriaLogs chart this would lead to the creation of a separate PVC – see the pvc.yaml.
You could simply create a separate PVC yourself and connect it through the existingClaim value, but you already have a PersistentVolume, and you don’t want to create a new one and migrate data (although you can if you need to, see VictoriaMetrics: migrating VMSingle and VictoriaLogs data between Kubernetes clusters, but there will be a down time), so let’s see how we can solve this differently – without deleting Pods and without stopping the service.
Contents
storageClassName and AllowVolumeExpansion
The storageClass used to create a Persistent Volume must support AllowVolumeExpansion – see Volume expansion:
$ kk describe storageclass gp2-retain Name: gp2-retain ... Provisioner: kubernetes.io/aws-ebs Parameters: <none> AllowVolumeExpansion: True MountOptions: <none> ReclaimPolicy: Retain VolumeBindingMode: WaitForFirstConsumer ...
Create this storageClass when creating an EKS cluster from a simple manifest:
...
resource "kubectl_manifest" "storageclass_gp2_retain" {
yaml_body = <<YAML
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp2-retain
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
YAML
}
...
Although there is a dedicated storage_class resource for Terraform, and would be better to use it instead f the kubectl_manifest.
And the kubernetes.io/aws-ebs driver is already deprecated (OMG, since Kubernetes 1.17!), it’s time to update to ebs.csi.aws.com.
But we’ll fix this later, right now the goal is to simply increase the disk.
Reproducing the issue
For the test, let’s write our own STS with volumeClaimTemplates:
apiVersion: apps/v1 kind: StatefulSet metadata: name: demo-sts spec: serviceName: demo-sts-svc replicas: 1 selector: matchLabels: app: demo template: metadata: labels: app: demo spec: containers: - name: app image: busybox command: ["sh", "-c", "sleep 3600"] volumeMounts: - name: data mountPath: /data volumeClaimTemplates: - metadata: name: data spec: accessModes: ["ReadWriteOnce"] storageClassName: gp2-retain resources: requests: storage: 1Gi
In volumeClaimTemplates, set the storageClassName and the size to 1 gigabyte.
Deploy:
$ kk apply -f test-sts-pvc.yaml statefulset.apps/demo-sts created
Check the PVC:
$ kk get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE data-demo-sts-0 Bound pvc-31a9a547-7547-4d34-bb2d-2c7015b9e0f3 1Gi RWO gp2-retain <unset> 15s
Now, if we want to increase the size via volumeClaimTemplates from 1Gi to 2Gi:
... volumeClaimTemplates: ... resources: requests: storage: 2Gi
Then we get an error:
$ kk apply -f test-sts-pvc.yaml The StatefulSet "demo-sts" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
The Fix.
But we can get around this very easily:
- edit the PVC manually – set a new size
- delete STS with the
--cascade=orphan– see Delete owner objects and orphan dependents - create STS again
- …
- profit!
Let’s try it.
Note: before changing disks, don’t forget about backups!
Edit the PVC manually – change resources.requests.storage from 1Gi to 2Gi:
Check the Events of this PVC:
$ kk describe pvc data-demo-sts-0 ... Normal ExternalExpanding 40s volume_expand CSI migration enabled for kubernetes.io/aws-ebs; waiting for external resizer to expand the pvc Normal Resizing 40s external-resizer ebs.csi.aws.com External resizer is resizing volume pvc-31a9a547-7547-4d34-bb2d-2c7015b9e0f3 Normal FileSystemResizeRequired 35s external-resizer ebs.csi.aws.com Require file system resize of volume on node
And after a few more seconds, it’s done:
... Normal FileSystemResizeSuccessful 19s kubelet
Check CAPACITY:
$ kk get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE data-demo-sts-0 Bound pvc-31a9a547-7547-4d34-bb2d-2c7015b9e0f3 2Gi RWO gp2-retain <unset> 4m7s
2Gi, everything is OK.
And now we also have 2 gigabytes in the Pod itself:
$ kk exec -ti demo-sts-0 -- df -h /data Filesystem Size Used Available Use% Mounted on /dev/nvme7n1 1.9G 24.0K 1.9G 0% /data
But if we try to deploy the changes to volumeClaimTemplates.spec.resources.requests.storage again, we will still get an error:
$ kk apply -f test-sts-pvc.yaml The StatefulSet "demo-sts" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
So, delete the STS itself, but leave all its dependent objects:
$ kubectl delete statefulset demo-sts --cascade=orphan statefulset.apps "demo-sts" deleted
Check if the Pod is alive:
$ kk get pod NAME READY STATUS RESTARTS AGE demo-sts-0 1/1 Running 0 3m13s
And now we just create STS again, with a new value in the volumeClaimTemplates.spec.resources.requests.storage:
$ kk apply -f test-sts-pvc.yaml statefulset.apps/demo-sts created
Done.
