Blog

Advanced Etcd Cluster Backup and Restore for Kubernetes

Picture of Boyan Kirov
Boyan Kirov
DevOps & Cloud Engineer
16.11.2023
Reading time: 4 mins.
Last Updated: 03.01.2024

Table of Contents

In Unix-like operating systems, the /etc folder holds vital configuration data. In the realm of Kubernetes, a similarly crucial role is played by Etcd, a key-value data store responsible for housing both configuration data and cluster information. The name “Etcd” is a nod to the /etc folder, with an added “d” symbolizing distributed systems.

Etcd is the cornerstone of your Kubernetes cluster, storing data essential for service discovery and cluster management. Safeguarding your Etcd data through regular backups is paramount in preparing for unforeseen failures. In this article, we will walk you through the process of securely backing up and restoring your Etcd cluster in a Kubernetes environment using the powerful etcdctl tool.

Backing up Your Etcd Cluster

  1. Verify etcdctl Installation: To begin, ensure that you have etcdctl installed on your system. Run the following command to check:
etcdctl version

 If etcdctl is correctly installed, you will receive an output indicating the version. If not, you’ll encounter a ‘command not found’ error.

  1. Install etcdctl (if needed): If etcdctl is missing, follow the installation instructions provided in the URL. Detailed guides are available for Linux, MacOS, and Docker.
  2. Gather Information: With etcdctl at your disposal, the next step is to obtain the necessary information to initiate the backup process. You’ll need:
  • Endpoint Information: Determine the endpoints for Etcd. If Etcd runs on the same server, the default endpoint is https://127.0.0.1:2379. If Etcd is on a different server, replace 127.0.0.1 with the server’s IP.
  • Certificates: You will require the following certificates for authentication:
  • –cert
  • –cacert
  • –key

Retrieve endpoint information using the command:

cat /etc/kubernetes/manifests/etcd.yaml | grep listen

For certificate information:

cat /etc/kubernetes/manifests/etcd.yaml | grep file

4. Run Snapshot Save Command: Armed with the necessary information, you can execute the snapshot save command using etcdctl:

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save <backup-file-location>

Ensure to prepend ETCDCTL_API=3 at the beginning of the command. This ensures that etcdctl uses the v3 API by default.

  1. Set Environment Variable (Optional): If you prefer not to specify the API version with each command, set the environment variable with:
 export ETCDCTL_API=3

6. Confirm Backup: After running the snapshot save command, you’ll have successfully created a backup named etcd-backup.db. The last line of the output should confirm this with the message “snapshot saved at etcd-backup.db.”

Restoring Etcd from the Snapshot

Before we dive into the restoration process, let’s briefly review the initial steps taken to make the transition smoother.

  1. Verify Pods: Check the pods in the default namespace. As you observed during the backup process, no new pods were created, and you should have no resources.
  2. Create a New Pod: Just to demonstrate the impact of the restoration process, create a new pod using:
 kubectl run newpod --image=nginx
  1. Begin Restoration: Assume Etcd has encountered a failure, and you need to revert to the last saved state. You have the etcd-backup.db snapshot saved earlier.
  2. Restore Using etcdctl: To restore from the saved snapshot, use a command that is similar to the one used for backup, but this time, specify a new data directory where the cluster data will be copied:
ETCDCTL_API=3 etcdctl --data-dir="/var/lib/etcd-backup" \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot restore etcd-backup.db

5. Edit Etcd Manifest: As the data is restored, you need to modify the Etcd pod’s manifest file to use the new restored data directory as a volume. Open the Etcd manifest file for editing:

vi /etc/kubernetes/manifests/etcd.yaml
 Modify relevant sections in the etcd.yaml file.
  1. Wait for Pod Restart: Give it a few minutes for the Etcd pod to restart with the new state. During this time, you may not receive responses from the API server.
  2. Verify Restoration: To ensure a successful restoration, check the pods in the default namespace. If you initially created a new pod, it should not exist after the restoration. Use the following command to verify:
kubectl get pods

Congratulations! You’ve effectively secured your Etcd cluster with a backup and demonstrated your ability to restore it, safeguarding your Kubernetes environment against unexpected failures.

Conclusion

To sum up, a strong Etcd cluster backup and restore plan is essential to guaranteeing the stability of your Kubernetes setup. Gaining proficiency with the sophisticated methods described in this tutorial can strengthen your system’s defenses against unanticipated malfunctions, protecting important information and ensuring continuous functioning. The key to a successful Kubernetes deployment is being able to safeguard and recover your Etcd cluster while you traverse the challenging terrain of distributed systems. Adopt these procedures, and in the face of difficulties, confidently drive your Kubernetes deployment.

Leave a Reply

Your email address will not be published. Required fields are marked *

More Posts

Note: The following example demonstrates upgrading a Kubernetes cluster from version 1.23 to 1.24. Replace the version numbers according to your specific setup. To ensure a seamless upgrade, it’s crucial...
Reading
What is Renovate? Renovate is an automated dependency update service. It works by scanning private or public repositories and creates Merge Requests, based on the requirements we have and the...
Reading
Get In Touch
ITGix provides you with expert consultancy and tailored DevOps services to accelerate your business growth.
Newsletter for
Tech Experts
Join 12,000+ business leaders and engineers who receive blogs, e-Books, and case studies on emerging technology.