• How to check the ETCD version?
    • Check the etcd logs: 
      kubectl -n kube-system logs etcd-controlplane | grep -i 'etcd-version'
  • Or check the image: 
    kubectl -n kube-system describe pod etcd-controlplane | grep Image
  • How to check which ETCD address and port it is reachable on? 
    kubectl -n kube-system describe pod etcd-controlplane | grep '\--listen-client-urls'
  • The ETCD certificates are located under: 
    /etc/kubernetes/pki/etcd/
  • Example snapshot command: 
    ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/snapshot-pre-boot.db
  • On Snapshot Restore: First Restore the snapshot:

root@controlplane:~# ETCDCTL_API=3 etcdctl –data-dir /var/lib/etcd-from-backup
snapshot restore /opt/snapshot-pre-boot.db

2022-03-25 09:19:27.175043 I | mvcc: restore compact to 2552 2022-03-25 09:19:27.266709 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32 root@controlplane:~#

Note: In this case, we are restoring the snapshot to a different directory but in the same server where we took the backup (the controlplane node) As a result, the only required option for the restore command is the –data-dir.

Next, update the /etc/kubernetes/manifests/etcd.yaml:

We have now restored the etcd snapshot to a new path on the controlplane - /var/lib/etcd-from-backup, so, the only change to be made in the YAML file, is to change the hostPath for the volume called etcd-data from old directory (/var/lib/etcd) to the new directory (/var/lib/etcd-from-backup).

volumes:

  • hostPath: path: /var/lib/etcd-from-backup type: DirectoryOrCreate name: etcd-data

With this change, /var/lib/etcd on the container points to /var/lib/etcd-from-backup on the controlplane (which is what we want).

When this file is updated, the ETCD pod is automatically re-created as this is a static pod placed under the /etc/kubernetes/manifests directory.

Note 1: As the ETCD pod has changed it will automatically restart, and also kube-controller-manager and kube-scheduler. Wait 1-2 to mins for this pods to restart. You can run the command: watch "crictl ps | grep etcd" to see when the ETCD pod is restarted.

Note 2: If the etcd pod is not getting Ready 1/1, then restart it by kubectl delete pod -n kube-system etcd-controlplane and wait 1 minute.

Note 3: This is the simplest way to make sure that ETCD uses the restored data after the ETCD pod is recreated. You don't have to change anything else.

If you do change –data-dir to /var/lib/etcd-from-backup in the ETCD YAML file, make sure that the volumeMounts for etcd-data is updated as well, with the mountPath pointing to /var/lib/etcd-from-backup (THIS COMPLETE STEP IS OPTIONAL AND NEED NOT BE DONE FOR COMPLETING THE RESTORE)

  • etcd runs in the kube-system namespace.
  • Can check the manifests directory.
  • Can do export ETCDCTL_API=3 so don’t need to add ETCDCTL_API=3 every time in front of the etcdctl command.
  • Check deployments, pods and services once the snapshot is complete.

Updated: