Advanced Troubleshooting for Multi-Tenant Web Application with SSL Termination and Rate Limiting on Kubernetes

Introduction

Running a multi-tenant web application on Kubernetes brings numerous benefits but also introduces complexities, especially when implementing SSL termination and rate limiting. This guide walks you through advanced troubleshooting steps to ensure your application runs smoothly and securely.

Scenario

You manage a multi-tenant web application on a Kubernetes cluster with custom domains for each tenant. An NGINX Ingress Controller handles SSL termination and rate limiting. Recently, a new tenant was added, but traffic isn’t routing correctly to the tenant’s service, and SSL termination and rate limiting are malfunctioning.

Setup

• Kubernetes Cluster: Running on a cloud provider (e.g., GKE, EKS, or AKS)
• NGINX Ingress Controller: Installed and configured for SSL and rate limiting
• Ingress Resources: Defined for routing multiple custom domains with SSL termination and rate limiting
• Services: Separate service for each tenant within the cluster
• Cert-Manager: Manages SSL certificates automatically
• ConfigMap and Annotations: Custom configurations for the Ingress controller

Ingress Configuration Example

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: multi-tenant-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/limit-rps: "10"
    nginx.ingress.kubernetes.io/limit-burst-multiplier: "3"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - tenant1.example.com
    - tenant2.example.com
    - tenant3.example.com
    secretName: multi-tenant-tls
  rules:
  - host: tenant1.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: tenant1-service
            port:
              number: 80
  - host: tenant2.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: tenant2-service
            port:
              number: 80
  - host: tenant3.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: tenant3-service
            port:
              number: 80

Problem: Traffic Not Routing Correctly for a New Tenant with SSL and Rate Limiting

Despite configuring everything correctly, traffic for the new tenant’s domain (tenant3.example.com) does not reach the intended service (tenant3-service). Additionally, SSL termination and rate limiting are not functioning as expected.

Advanced Troubleshooting Steps

1. Verify DNS Configuration

• Ensure that the DNS record for tenant3.example.com points to the Ingress controller’s external IP address

• Use tools like nslookup or dig to confirm the DNS configuration

nslookup tenant3.example.com
dig tenant3.example.com

2. Check SSL Certificate Status

Verify the SSL certificate for tenant3.example.com is correctly issued and attached to the Ingress resource.

kubectl describe certificate tenant3-cert

3. Inspect Ingress Controller Logs:

• Access the logs of the NGINX Ingress controller to look for any errors or misconfigurations.
• Example command:

kubectl logs -n ingress-nginx <pod-ingress-controller>

4. Inspect Ingress Resource

• Ensure the Ingress resource for tenant3.example.com is correctly defined and applied.
• Example command to view the Ingress resource:

kubectl get ingress multi-tenant-ingress -o yaml

5. Check Service Status:

Verify the tenant3-service is running and reachable within the cluster.

kubectl get service tenant3-service

6. Test Internal Connectivity:

Use a temporary pod to test connectivity to the tenant3-service within the cluster.

kubectl run curlpod --image=radial/busyboxplus:curl -i --tty
curl http://tenant3-service

7. Check Ingress Rules and Annotations

Ensure there are no conflicts or misconfigurations in the Ingress rules and annotations.

8. Review NGINX Configuration

Check the NGINX configuration generated by the Ingress controller for any anomalies.

kubectl exec -it <nginx-ingress-controller-pod> -- cat /etc/nginx/nginx.conf

9. Check the Rate Limiting Configuration

Ensure the rate-limiting annotations are correctly applied and not conflicting with other settings.

10. Check the Network Policies

Ensure Network Policies allow traffic between the Ingress controller and the tenant3-service.

11. Update the Ingress Annotations

Add specific annotations to help debug or mitigate routing issues.

nginx.ingress.kubernetes.io/ssl-redirect: "false"

12. Enable Debug Logging

Increase the log level of the NGINX Ingress controller to gather more detailed information.

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
data:
  log-level: debug

Conclusion

By following these advanced troubleshooting steps, you can identify and resolve issues with traffic routing, SSL termination, and rate limiting for your multi-tenant web application on Kubernetes. Proper DNS configuration, correct Ingress resource definitions, detailed SSL certificate management, and thorough troubleshooting are crucial for ensuring smooth and secure operations.