NGINX Ingress Controller for Amazon EKS — Frequently Used Annotations

Table of Contents

Introduction

Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service that you can use to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane or nodes. Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized applications.

— Amazon EKS User Guide

An Ingress controller is a specialized load balancer for Kubernetes (and other containerized) environments. An Ingress controller abstracts away the complexity of Kubernetes application traffic routing and provides a bridge between Kubernetes services and external ones.

— NGINX Glossary

The DevOps team here at QloudX, was recently involved in setting up an EKS cluster for one of our enterprise clients. As always, we encountered the question of which ingress controller to use for our EKS cluster.

AWS Load Balancer Controller

We first tried to use AWS Load Balancer Controller. However, the AWS LBC didn’t work out for us for a few reasons:

  1. We were planning to run 250+ apps from a single large cluster. Since the AWS LBC delegates many of its functionality to AWS services (like path based routing rules), we starting running into many of the limits imposed by AWS’s Application Load Balancer (ALB), like “target groups per ALB” & others described at Quotas for your Application Load Balancers.
  2. We didn’t want to deploy & maintain multiple ALBs (to workaround these limits) since all our apps combined could be easily served by the capacity of a single ALB.

So we decided to go with the most popular ingress controller for Kubernetes: NGINX!

NGINX Ingress Controller

Installation

Installing the NGINX ingress controller onto our cluster was fairly straight-forward using Helm:

helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace

The controller will create its own load balancer in AWS as soon as you install it. Unfortunately, no ingress controller except AWS LBC can create & manage an AWS ALB. The best you can get is an NLB.

However, since we really needed an ALB, we prefer to create our own ALB & configure it to route traffic to NGINX ingress controller, configured as a NodePort service inside the cluster.

In case you’re in the same boat as us, here are the Helm values we used while installing the controller:

controller:
  ingressClassResource:
    controllerValue: k8s.io/ingress-nginx
    enabled: true
    name: nginx
  kind: DaemonSet
  metrics:
    enabled: true
    port: 10254
    service:
      annotations: {}
      nodePort: 31254
      servicePort: 10254
      type: NodePort
  service:
    enableHttp: true
    enableHttps: false
    nodePorts:
      http: 32080
    type: NodePort

Basic Usage

The simple use case for an ingress is as follows. Here is an ingress for an app:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app
            port:
              number: 80

Ingress Annotations

Advanced behavior (beyond basic usage) can be achieved by annotating ingresses.

All annotation keys & values must always be strings! Values like true must be quoted as “true”.

All annotations always start with nginx.ingress.kubernetes.io.

Blue/Green Deployments

If you employ blue/green deployments in your cluster, this section is for you!

Reference: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary

From the NGINX ingress controller’s perspective, blue/green deployments are essentially canary deployments that have been configured to send 100% of a particular type of incoming traffic to the green environment.

The type of requests to send to the green environment can be determined by many factors like:

  1. Requests that include a specific HTTP header
  2. Requests that include a specific value for a pre-defined HTTP header
  3. Requests that include a specific cookie

In each of these cases, the blue service has a standard ingress like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-v1
spec:
  ingressClassName: nginx
  rules:
  - host: app-v1.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-v1
            port:
              number: 80

The green service always has a separate ingress resource. How it’s configured, depends on what kind of request routing is desired.

The green ingress must always have this annotation:

nginx.ingress.kubernetes.io/canary: "true"

HTTP Header Based Request Routing

To route all requests containing a particular HTTP header to the green environment, the green ingress must be:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-v2
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: canary
spec:
  ingressClassName: nginx
  rules:
  - host: app-v2.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-v2
            port:
              number: 80

All requests containing the HTTP header canary = always will be routed to app v2. All other requests will be routed to app v1.

HTTP Header Value Based Request Routing

To route all requests containing an HTTP header app-version = v2 to app v2:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-v2
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: app-version
    nginx.ingress.kubernetes.io/canary-by-header-value: v2
spec:
  ingressClassName: nginx
  rules:
  - host: app-v2.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-v2
            port:
              number: 80

HTTP Header Value Pattern Based Request Routing

To route requests containing any value for the canary header to app v2:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-v2
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: canary
    nginx.ingress.kubernetes.io/canary-by-header-pattern: .*
spec:
  ingressClassName: nginx
  rules:
  - host: app-v2.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-v2
            port:
              number: 80

canary-by-header-pattern can be any PCRE regular expression.

To route all requests containing the cookie canary = always to app v2:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-v2
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-cookie: canary
spec:
  ingressClassName: nginx
  rules:
  - host: app-v2.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-v2
            port:
              number: 80

Canary Deployments

To route 10% of all incoming live traffic to app v2:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-v2
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  ingressClassName: nginx
  rules:
  - host: app-v2.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-v2
            port:
              number: 80

Session Affinity / Stickiness

To ensure that users that were served by canaries, continue to be served by canaries, set this annotation on the app v1 ingress (not the canary ingress):

nginx.ingress.kubernetes.io/affinity: cookie

HTTP Response Headers

To include additional HTTP headers in all responses from canary, add this annotation to the canary ingress:

nginx.ingress.kubernetes.io/configuration-snippet: |
  add_header "app-version: v2";

NOTE: This feature may not work out of the box. It was disabled due to a CVE. Click here for details.

CORS: Cross-Origin Resource Sharing

Reference: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#enable-cors

To enable CORS on your service, use:

nginx.ingress.kubernetes.io/enable-cors: "true"

To fine-tune CORS behavior, use:

nginx.ingress.kubernetes.io/cors-allow-origin: https://*.example.com
nginx.ingress.kubernetes.io/cors-allow-methods: OPTIONS, GET, POST

Nginx Server Configuration

For use cases not covered by annotations, it’s possible to provide Nginx server configuration to ingresses using a special annotation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
        set $agentflag 0;
 
        if ($http_user_agent ~* "(Mobile)" ){
          set $agentflag 1;
        }
 
        if ( $agentflag = 1 ) {
          return 301 https://m.example.com;
        }

Redirect Requests

To redirect incoming requests, use:

nginx.ingress.kubernetes.io/permanent-redirect: https://other-app.example.com

for a permanent HTTP 301 redirect, or

nginx.ingress.kubernetes.io/temporal-redirect: https://other-app.example.com

for a temporary HTTP 302 redirect.

Request Mirroring

Incoming requests can be mirrored to another backend by applying this annotation:

nginx.ingress.kubernetes.io/mirror-target: https://mirror-app.example.com/$request_uri

Responses from mirror backends are ignored.

Regular Expressions in Paths

Reference: https://kubernetes.github.io/ingress-nginx/user-guide/ingress-path-matching/

Regular expressions cannot be used in the host URL, but they can be used in the URL paths if needed, by using the use-regex annotation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /app/.*
        pathType: Prefix
        backend:
          service:
            name: app
            port:
              number: 80

Conclusion

In this post, we presented a curated list of commonly-used NGINX ingress annotations. Hopefully, this will serve as a handy reference for you, as it does for us. 😊

About the Author ✍🏻

Harish KM is a Principal DevOps Engineer at QloudX & a top-ranked AWS Ambassador since 2020. 👨🏻‍💻

With over a decade of industry experience as everything from a full-stack engineer to a cloud architect, Harish has built many world-class solutions for clients around the world! 👷🏻‍♂️

With over 20 certifications in cloud (AWS, Azure, GCP), containers (Kubernetes, Docker) & DevOps (Terraform, Ansible, Jenkins), Harish is an expert in a multitude of technologies. 📚

These days, his focus is on the fascinating world of DevOps & how it can transform the way we do things! 🚀

Leave a Reply

Your email address will not be published. Required fields are marked *