Why go for a multi-master setup?

To have a truly reliable, highly available distributed system for your applications. Master fault tolerance.

Why go for a multi-zone setup?

To improve availability in case of a single zone outage/maintenance. Zone failure tolerance.

Why 3 masters?

Some of the master components need to run as a single actor to avoid a split-brain situation. More on the ‘Master elected components section’ below

Why 3 workers?

In case one of the zones goes down the applications to be able to start on the other zone. We can use “Pod affinity” to run our pods on a zone with a higher number of workers

How to scale?

By using Amazon’s Single auto-scaling group spanning multiple zones.

How to set persistent storage?

AWS S3 is Kubernetes compatible. Amazon EBS provides persistent block storage volumes for use with Amazon EC2 instances in the AWS Cloud.

How to monitor in AWS?

We can use open-source tools like Icinga2, Prometheus, etc. or we can go for Amazon CloudWatch Service. Both are suitable solutions.

THE CHALLENGE

Build a highly available Kubernetes cluster with features such as Master fault tolerance and zone failure tolerance.

THE SOLUTION

Establish redundant reliable storage:

The number one rule of high availability is to protect the data. Whatever else happens, whatever catches on fire, if you have the data, you can rebuild.

Clustering ETCD

Use persistent storage provided by Amazon S3 which is designed to deliver 99.999999999% durability
https://aws.amazon.com/s3/  OR Each Amazon EBS volume is designed for 99.999% availability and automatically replicates within its Availability Zone to protect your applications from component failure. https://aws.amazon.com/ebs/

Replicated API:

Install the API server on all master nodes with the YAML definition.
Setup Load Balancing between the master nodes using the desired method. With AWS you go with ELB.

Master elected components:

So far we haven’t done anything that actually modifies cluster state, such as the controller manager and scheduler. To achieve reliability, we want only one actor modifying the state of the cluster, but we also want replicated instances of these actors, in case the machine dies. Lease-lock is used in the API to perform master election (leader-elect flag).
The scheduler and controller manager can be configured to communicate using the load-balanced IP address of the API servers. Regardless of how they are configured, the scheduler and controller manager will complete the leader-election process mentioned above when using the leader-elect flag.
In case of a failure to access the API server, the elected leader will not be able to renew the lease, causing a new leader to be elected. This is especially relevant when configuring the scheduler and controller manager to access the API server via 127.0.0.1, with the API server on the same node being unavailable.

Adding Workers:

Fort a fresh cluster: we need to install the kubelet and kube-proxy on each worker node, and set the –apiserver flag to your replicated endpoint.

Multi-zone setup:

Multi-zone support is deliberately limited: a single Kubernetes cluster can run in multiple zones, but only within the same region (and cloud provider).
With multiple-zone clusters, this spreading behavior is extended across zones (to reduce the impact of zone failures.) (This is achieved via SelectorSpreadPriority).

Limitations for multi-zone setup:

We need to assure that the different zones are located close to each other in the network, so we don’t perform any zone-aware routing. Particularly the traffic that goes via services might cross zones (even if pods in some pods backing that service exists in the same zone as the client), and this may incur additional latency and cost.