Etcd Cluster Backup and Restore for Kubernetes

In Unix-like operating systems, the /etc folder holds vital configuration data. In the realm of Kubernetes, a similarly crucial role is played by Etcd, a key-value data store responsible for housing both configuration data and cluster information. The name “Etcd” is a nod to the /etc folder, with an added “d” symbolizing distributed systems.

Etcd is the cornerstone of your Kubernetes cluster, storing data essential for service discovery and cluster management. Safeguarding your Etcd data through regular backups is paramount in preparing for unforeseen failures. In this article, we will walk you through the process of securely backing up and restoring your Etcd cluster in a Kubernetes environment using the powerful etcdctl tool.

Backing up Your Etcd Cluster

1. Verify etcdctl Installation: To begin, ensure that you have etcdctl installed on your system. Run the following command to check:

etcdctl version

If etcdctl is correctly installed, you will receive an output indicating the version. If not, you’ll encounter a ‘command not found’ error.

2. Install etcdctl (if needed): If etcdctl is missing, follow the installation instructions provided in the URL. Detailed guides are available for Linux, MacOS, and Docker.

3. Gather Information: With etcdctl at your disposal, the next step is to obtain the necessary information to initiate the backup process. You’ll need:

• Endpoint Information: Determine the endpoints for Etcd. If Etcd runs on the same server, the default endpoint is https://127.0.0.1:2379. If Etcd is on a different server, replace 127.0.0.1 with the server’s IP.
• Certificates: You will require the following certificates for authentication:

• –cert
• –cacert
• –key

Retrieve endpoint information using the command:

cat /etc/kubernetes/manifests/etcd.yaml | grep liste

For certificate information:

cat /etc/kubernetes/manifests/etcd.yaml | grep file

4. Run Snapshot Save Command: Armed with the necessary information, you can execute the snapshot save command using etcdctl:

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save <backup-file-location>

Ensure to prepend ETCDCTL_API=3 at the beginning of the command. This ensures that etcdctl uses the v3 API by default.

5. Set Environment Variable (Optional): If you prefer not to specify the API version with each command, set the environment variable with:

export ETCDCTL_API=3

6. Confirm Backup: After running the snapshot save command, you’ll have successfully created a backup named etcd-backup.db. The last line of the output should confirm this with the message “snapshot saved at etcd-backup.db.”

Restoring Etcd from the Snapshot

Before we dive into the restoration process, let’s briefly review the initial steps taken to make the transition smoother.

1. Verify Pods: Check the pods in the default namespace. As you observed during the backup process, no new pods were created, and you should have no resources.
2. Create a New Pod: Just to demonstrate the impact of the restoration process, create a new pod using:

 kubectl run newpod --image=nginx

3. Begin Restoration: Assume Etcd has encountered a failure, and you need to revert to the last saved state. You have the etcd-backup.db snapshot saved earlier.

4. Restore Using etcdctl: To restore from the saved snapshot, use a command that is similar to the one used for backup, but this time, specify a new data directory where the cluster data will be copied:

ETCDCTL_API=3 etcdctl --data-dir="/var/lib/etcd-backup" \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot restore etcd-backup.db

5. Edit Etcd Manifest: As the data is restored, you need to modify the Etcd pod’s manifest file to use the new restored data directory as a volume. Open the Etcd manifest file for editing:

vi /etc/kubernetes/manifests/etcd.yaml

Modify relevant sections in the etcd.yaml file.

6. Wait for Pod Restart: Give it a few minutes for the Etcd pod to restart with the new state. During this time, you may not receive responses from the API server.

7. Verify Restoration: To ensure a successful restoration, check the pods in the default namespace. If you initially created a new pod, it should not exist after the restoration. Use the following command to verify:

kubectl get pods

Congratulations! You’ve effectively secured your Etcd cluster with a backup and demonstrated your ability to restore it, safeguarding your Kubernetes environment against unexpected failures.

Check out our Kubernetes Services.