We have a VictoriaLogs Helm chart with a PVC size of 30 GB, which is no longer enough for us, and we need to increase it.

But the problem is that .spec.volumeClaimTemplates[*].spec.resources.requests.storage in STS is immutable, that is, we can’t just change the size through values.yaml file, because it will lead to the error“Forbidden: updates to statefulset spec for fields other than ‘replicas’, ‘ordinals’, ‘template’, ‘updateStrategy’, ‘revisionHistoryLimit’, ‘persistentVolumeClaimRetentionPolicy’ and ‘minReadySeconds’ are forbidden“.

The chart values now look like this:

victoria-logs-single:
  server:
    persistentVolume:
      enabled: true
      storageClassName: gp2-retain
      size: 30Gi
    retentionPeriod: 7d

And with the default type of StatefulSet in the chart, the volumeClaimTemplates is used to create PVCs:

...  
volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: server-volume
        ...
      spec:
        ...
        resources:
          requests:
            storage: {{ $app.persistentVolume.size }}
...

If instead of STS there was a Deployment type, then in the VictoriaLogs chart this would lead to the creation of a separate PVC – see the pvc.yaml.

You could simply create a separate PVC yourself and connect it through the existingClaim value, but you already have a PersistentVolume, and you don’t want to create a new one and migrate data (although you can if you need to, see VictoriaMetrics: migrating VMSingle and VictoriaLogs data between Kubernetes clusters, but there will be a down time), so let’s see how we can solve this differently – without deleting Pods and without stopping the service.

storageClassName and AllowVolumeExpansion

The storageClass used to create a Persistent Volume must support AllowVolumeExpansion – see Volume expansion:

$ kk describe storageclass gp2-retain
Name:            gp2-retain
...
Provisioner:           kubernetes.io/aws-ebs
Parameters:            <none>
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Retain
VolumeBindingMode:     WaitForFirstConsumer
...

Create this storageClass when creating an EKS cluster from a simple manifest:

...
resource "kubectl_manifest" "storageclass_gp2_retain" {

  yaml_body = <<YAML
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: gp2-retain
    provisioner: kubernetes.io/aws-ebs
    reclaimPolicy: Retain
    allowVolumeExpansion: true
    volumeBindingMode: WaitForFirstConsumer
  YAML
}
...

Although there is a dedicated storage_class resource for Terraform, and would be better to use it instead f the kubectl_manifest.

And the kubernetes.io/aws-ebs driver is already deprecated (OMG, since Kubernetes 1.17!), it’s time to update to ebs.csi.aws.com.

But we’ll fix this later, right now the goal is to simply increase the disk.

Reproducing the issue

For the test, let’s write our own STS with volumeClaimTemplates:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: demo-sts
spec:
  serviceName: demo-sts-svc
  replicas: 1
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
        - name: app
          image: busybox
          command: ["sh", "-c", "sleep 3600"]
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp2-retain
        resources:
          requests:
            storage: 1Gi

In volumeClaimTemplates, set the storageClassName and the size to 1 gigabyte.

Deploy:

$ kk apply -f test-sts-pvc.yaml 
statefulset.apps/demo-sts created

Check the PVC:

$ kk get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-demo-sts-0   Bound    pvc-31a9a547-7547-4d34-bb2d-2c7015b9e0f3   1Gi        RWO            gp2-retain     <unset>                 15s

Now, if we want to increase the size via volumeClaimTemplates from 1Gi to 2Gi:

...
  volumeClaimTemplates:
    ...
        resources:
          requests:
            storage: 2Gi

Then we get an error:

$ kk apply -f test-sts-pvc.yaml 
The StatefulSet "demo-sts" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

The Fix.

But we can get around this very easily:

1. edit the PVC manually – set a new size
2. delete STS with the --cascade=orphan – see Delete owner objects and orphan dependents
3. create STS again
4. …
5. profit!

Let’s try it.

Note: before changing disks, don’t forget about backups!

Edit the PVC manually – change resources.requests.storage from 1Gi to 2Gi:

Check the Events of this PVC:

$ kk describe pvc data-demo-sts-0
...
  Normal  ExternalExpanding         40s                    volume_expand                                                                             CSI migration enabled for kubernetes.io/aws-ebs; waiting for external resizer to expand the pvc
  Normal  Resizing                  40s                    external-resizer ebs.csi.aws.com                                                          External resizer is resizing volume pvc-31a9a547-7547-4d34-bb2d-2c7015b9e0f3
  Normal  FileSystemResizeRequired  35s                    external-resizer ebs.csi.aws.com                                                          Require file system resize of volume on node

And after a few more seconds, it’s done:

...
  Normal FileSystemResizeSuccessful 19s kubelet

Check CAPACITY:

$ kk get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-demo-sts-0   Bound    pvc-31a9a547-7547-4d34-bb2d-2c7015b9e0f3   2Gi        RWO            gp2-retain     <unset>                 4m7s

2Gi, everything is OK.

And now we also have 2 gigabytes in the Pod itself:

$ kk exec -ti demo-sts-0 -- df -h /data
Filesystem                Size      Used Available Use% Mounted on
/dev/nvme7n1              1.9G     24.0K      1.9G   0% /data

But if we try to deploy the changes to volumeClaimTemplates.spec.resources.requests.storage again, we will still get an error:

$ kk apply -f test-sts-pvc.yaml 
The StatefulSet "demo-sts" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

So, delete the STS itself, but leave all its dependent objects:

$ kubectl delete statefulset demo-sts --cascade=orphan 
statefulset.apps "demo-sts" deleted

Check if the Pod is alive:

$ kk get pod
NAME         READY   STATUS    RESTARTS   AGE
demo-sts-0   1/1     Running   0          3m13s

And now we just create STS again, with a new value in the volumeClaimTemplates.spec.resources.requests.storage:

$ kk apply -f test-sts-pvc.yaml 
statefulset.apps/demo-sts created

Done.