We have an AWS Elastic Kubernetes Service with the VictoriaMetrics stack deployed (see VictoriaMetrics: deploying a Kubernetes monitoring stack).
I need to migrate the data from the old VMSingle Pod to the new one on the new cluster, and to do this, I need to find VMSingle’s data on an EC2.
Note: regarding the migration of VMSingle data, there is a vmctl utility for it, somewhere in the drafts there is a post, when I do the next migration, I will write iabout it
Contents
Checking the VMSingle Kubernetes Pod
Check the data of the VMSingle Kubernetes Pod – find its EC2 instance, a corresponding Container ID and the Pod’s Mounts:
$ kk describe pod vmsingle-vm-k8s-stack-67d585c9cd-jt4f7 Name: vmsingle-vm-k8s-stack-67d585c9cd-jt4f7 ... Node: ip-10-0-46-247.ec2.internal/10.0.46.247 ... Containers: vmsingle: Container ID: containerd://57398c3184cd229be564b140f32a9214b38a507137522904eab6ae38b676432a ... Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5xmmq (ro) /victoria-metrics-data from data (rw) ...
Connect to the server (see AWS: Karpenter and SSH for Kubernetes WorkerNodes):
$ ssh -i .ssh/hOS/atlas-eks-ec2 [email protected] [ec2-user@ip-10-0-46-247 ~]$ sudo -s [root@ip-10-0-46-247 ec2-user]#
But the usual docker CLI is not available here, because since version 1.24 we have ContainerD instead of dockershim (see Everything you need to know about moving to containerd on Amazon EKS):
[root@ip-10-0-46-247 ec2-user]# docker bash: docker: command not found
To work with containerd, we have the crictl utility (Container Runtime Interface CTL):
[root@ip-10-0-46-247 ec2-user]# crictl NAME: crictl - client for CRI USAGE: crictl [global options] command [command options] [arguments...] COMMANDS: attach Attach to a running container create Create a new container exec Run a command in a running container version Display runtime version information images, image, img List images inspect Display the status of one or more containers inspecti Return the status of one or more images ...
Option 1: using crictl inspect and hostPath
Using the Container ID from kubectl describe pod, we can get information about this container on the host and its mounts:
[root@ip-10-0-46-247 ec2-user]# crictl inspect 57398c3184cd229be564b140f32a9214b38a507137522904eab6ae38b676432a
...
"mounts": [
{
"containerPath": "/victoria-metrics-data",
"gidMappings": [],
"hostPath": "/var/lib/kubelet/pods/7b2b0205-8c7e-430f-995b-a45cd79ecb9f/volumes/kubernetes.io~csi/pvc-ed3831bc-56a2-4660-9aef-b47cd252edac/mount",
...
And check the directory from the hostPath field on the EC2:
[root@ip-10-0-46-247 ec2-user]# ll /var/lib/kubelet/pods/7b2b0205-8c7e-430f-995b-a45cd79ecb9f/volumes/kubernetes.io~csi/pvc-ed3831bc-56a2-4660-9aef-b47cd252edac/mount total 40 drwxr-xr-x 6 root root 4096 Oct 1 00:51 cache drwxr-xr-x 4 root root 4096 Dec 4 2023 data -rw-r--r-- 1 root root 0 Oct 1 00:52 flock.lock drwxr-xr-x 6 root root 4096 Sep 29 04:00 indexdb drwx------ 2 root root 16384 Dec 4 2023 lost+found drwxr-xr-x 2 root root 4096 Dec 4 2023 metadata drwxr-xr-x 2 root root 4096 Dec 4 2023 snapshots drwxr-xr-x 3 root root 4096 Oct 1 00:52 tmp
Option 2: using /proc/PID/root
Another option is through the PID of the process that we also saw in the crictl inspect output, in this case it will is “57398c3184cd229be564b140f32a9214b38a507137522904eab6ae38b676432a”:
...
"info": {
...
"pid": 6933,
...
"config": {
"metadata": {
"name": "vmsingle"
},
...
Then on the host we can check /proc/<PID>/root/<containerPath> directory:
[root@ip-10-0-46-247 ec2-user]# ll /proc/6933/root/victoria-metrics-data/ total 40 drwxr-xr-x 6 root root 4096 Oct 1 00:51 cache drwxr-xr-x 4 root root 4096 Dec 4 2023 data -rw-r--r-- 1 root root 0 Oct 1 00:52 flock.lock drwxr-xr-x 6 root root 4096 Sep 29 04:00 indexdb drwx------ 2 root root 16384 Dec 4 2023 lost+found drwxr-xr-x 2 root root 4096 Dec 4 2023 metadata drwxr-xr-x 2 root root 4096 Dec 4 2023 snapshots drwxr-xr-x 3 root root 4096 Oct 1 00:52 tmp
That’s it. Now you can rsync from directory to a new Kubernetes WorkerNode.
