## Nginx Ingress **This article applies to K8S version 1.26+. For other versions, please refer to: [Ingress Support](/uk8s/service/ingress/README)** ## What is Ingress Ingress serves as the entry point for accessing internal cluster services from outside the Kubernetes cluster, providing layer-7 load balancing capabilities for in-cluster Services. Generally, Services and Pods can only be accessed via IP addresses within the cluster, with all traffic reaching the cluster boundary either discarded or forwarded elsewhere. In the Service chapter, we introduced creating LoadBalancer-type Services: UK8S leverages Kubernetes' extension interfaces to create a corresponding ULB load balancer for each Service, which receives external traffic and routes it to the cluster. However, in scenarios like microservices, where each Service requires a dedicated load balancer, management costs become prohibitively high — hence the emergence of Ingress. Think of Ingress as a "Service for Services," providing proxy-based load balancing for backend Services. It allows configuration of externally accessible URLs, load balancing, SSL, name-based virtual hosting, and more. ## I. Deploying the Ingress Controller For Ingress to function, an Ingress Controller must be deployed in the cluster. Unlike other controllers (e.g., Deployment, which runs as part of the kube-controller-manager binary at cluster startup), Ingress Controller requires manual deployment. The Kubernetes community offers the following Ingress Controllers to choose from: 1. Nginx 2. HAProxy 3. Envoy 4. Traefik Here we choose Nginx as the Ingress Controller. Deploying the Nginx Ingress Controller is very simple — just execute the following command. #### Notes * The sample Ingress Controller creates an internal ULB. To create an external ULB, download the YAML file and change the Service's `metadata.annotations."service.beta.kubernetes.io/ucloud-load-balancer-type"` to "outer". For more parameters, please refer to the official documentation on [ULB Parameter Description](/docs/uk8s/service/annotations). * Containers use UTC by default. To use the host's time zone, refer to [Pod Time Zone Issues](/docs/uk8s/troubleshooting/pod_debug_summary?id=_10-pod%e7%9a%84%e6%97%b6%e5%8c%ba%e9%97%ae%e9%a2%98). * If you need to access Services via ULB addresses from within the cluster, you need to change the Ingress Controller Service's externalTrafficPolicy to `Cluster`. Otherwise, if a Pod is not on the same node as the Ingress Controller, access will fail. **After modifying this parameter, real client source IPs will no longer be obtainable.** ```bash kubectl apply -f https://docs.scloudsg.com/docs/uk8s/yaml/ingress_nginx/mandatory_1.26.yaml ``` #### Explanation This YAML defines a ReplicaSet of Pods using the ingress-nginx-controller image. The primary function of this Pod is to act as a controller that monitors Ingress objects and changes to their backend Services. When a new Ingress object is created by a user, the controller generates a corresponding Nginx configuration file (i.e., `/etc/nginx/nginx.conf`) and starts an Nginx service with this configuration. When the Ingress object is updated, the configuration file is refreshed. Notably, if only the proxied Service object is updated, the Nginx service managed by the controller does not need to reload, as ingress-nginx-controller uses Nginx Lua to enable dynamic upstream configuration. Additionally, the YAML file defines a ConfigMap, which allows ingress-nginx-controller to customize the Nginx configuration. Example: ```yaml apiVersion: v1 data: allow-snippet-annotations: "false" kind: ConfigMap metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.11.5 name: ingress-nginx-controller namespace: ingress-nginx ``` **Note:** In ConfigMap, keys and values only support strings, so values such as integers need to be wrapped in double quotes, for example "100". For detailed information, see [Nginx-Ingress-ConfigMap](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/). ## II. External Access to Nginx Ingress Above, we have deployed a Nginx Ingress Controller in UK8S, and to make it externally accessible, we have also created an externally accessible LoadBalancer-type Service, as shown below. ```yaml apiVersion: v1 kind: Service metadata: annotations: "service.beta.kubernetes.io/ucloud-load-balancer-type": "inner" "service.beta.kubernetes.io/ucloud-load-balancer-listentype": "network" labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.11.5 name: ingress-nginx-controller namespace: ingress-nginx spec: externalTrafficPolicy: Local ports: - appProtocol: http name: http port: 80 protocol: TCP targetPort: http - appProtocol: https name: https port: 443 protocol: TCP targetPort: https selector: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx type: LoadBalancer ``` The sole job of this Service is to expose ports 80 and 443 of all Pods with the ingress-nginx label. You can get the external access entry of this Service in the following way. ```bash $ kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller LoadBalancer 172.30.48.77 xxx.yy.xxx.yy 80:30052/TCP,443:31285/TCP 14m ...... ``` After deploying the Ingress Controller and its required Service, we can now use it to proxy other Services inside the cluster. ## III. Creating Two Applications In the following YAML, we define 2 applications using the echo-nginx image, which mainly outputs some global variables of the Nginx application itself. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: demo-app-1 labels: app: demo-app-1 spec: replicas: 2 selector: matchLabels: app: demo-app-1 template: metadata: labels: app: demo-app-1 spec: containers: - name: demo-app-1 image: uhub.scloudsg.com/jenkins_k8s_cicd/echo_nginx:v11 ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: demo-app-1-svc spec: ports: - port: 80 targetPort: 80 protocol: TCP name: http selector: app: demo-app-1 --- apiVersion: apps/v1 kind: Deployment metadata: name: demo-app-2 labels: app: demo-app-2 spec: replicas: 3 selector: matchLabels: app: demo-app-2 template: metadata: labels: app: demo-app-2 spec: containers: - name: demo-app-2 image: uhub.scloudsg.com/jenkins_k8s_cicd/echo_nginx:v11 ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: demo-app-2-svc spec: ports: - port: 80 targetPort: 80 protocol: TCP name: http selector: app: demo-app-2 ``` Save the above YAML as `demo-app.yaml` and create the applications with the following command. ```bash kubectl apply -f demo-app.yaml ``` ## IV. Defining the Ingress Object We have deployed the Nginx Ingress Controller and exposed it to the external network, and created two applications. Next, we can define an Ingress object to proxy the two applications out of the cluster. ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: demo-app-ingress spec: ingressClassName: nginx defaultBackend: service: name: demo-app-1-svc port: number: 80 rules: - host: demo-app.example.com http: paths: - path: /demo-app-1 pathType: Prefix backend: service: name: demo-app-1-svc port: number: 80 - path: /demo-app-2 pathType: Prefix backend: service: name: demo-app-2-svc port: number: 80 ``` The above YAML file defines an Ingress object, where `ingress.spec.rules` is the set of proxy rules for the Ingress. #### ingressClassName In the above Ingress configuration, we set `ingressClassName: nginx`, which specifies which Ingress Controller should handle this Ingress object. If `ingressClassName` is not specified, Kubernetes will check whether a default IngressClass exists in the cluster. **Setting the Default IngressClass:** You can set the default by adding an annotation to IngressClass. This has been added in `mandatory_1.26.yaml`: ```yaml apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: nginx annotations: ingressclass.kubernetes.io/is-default-class: "true" spec: controller: k8s.io/ingress-nginx ``` #### defaultBackend When no host and path in the rules are matched, the default backend defined in `ingress.spec.defaultBackend` takes effect, forwarding traffic that does not match any host or path to the defaultBackend for processing. #### rules * host: Its value must be a standard domain name format string and cannot be an IP address. The value defined by the host field is the entry point of this Ingress. When a user accesses `demo-app.example.com`, they are actually accessing this Ingress object. This allows Kubernetes to use IngressRule to forward the request to the next step. * path: Each path here corresponds to a backend Service. In our example, two paths are defined, corresponding to the Services of the demo-app-1 and demo-app-2 Deployments (i.e., demo-app-1-svc and demo-app-2-svc). * http: Each HTTP rule contains the following information: a host configuration item (e.g., `demo-app.example.com`), a path list (e.g., `/demo-app-1` and `/demo-app-2`). Each path is associated with a backend (e.g., port 80 of `demo-app-1-svc`). Before the LoadBalancer forwards traffic to the backend, all inbound requests must first match the host and path. Save the above YAML as ingress.yaml and create the Ingress object directly with the following command: ```bash kubectl apply -f ingress.yaml ``` Next, we can inspect this Ingress object: ```bash $ kubectl get ingresses.networking.k8s.io NAME CLASS HOSTS ADDRESS PORTS AGE demo-app-ingress demo-app.example.com 80 5m39s $ kubectl describe ingresses.networking.k8s.io demo-app-ingress Name: demo-app-ingress Labels: Namespace: ingress-nginx Address: Default backend: demo-app-1-svc:80 (172.20.145.127:80,172.20.187.251:80) Rules: Host Path Backends ---- ---- -------- demo-app.example.com /demo-app-1 demo-app-1-svc:80 (172.20.145.127:80,172.20.187.251:80) /demo-app-2 demo-app-2-svc:80 (172.20.40.180:80,172.20.43.118:80,172.20.62.63:80) Annotations: Events: ``` From the rules we can see that the Host we defined is `demo-app.example.com`, and it has two forwarding rules (Path), forwarding to demo-app-1-svc and demo-app-2-svc respectively. Of course, in the Ingress YAML file, you can also define multiple Hosts to provide load balancing services for more domain names. #### Access You can access the applications we deployed earlier by visiting the address and port of this Ingress. For example, when we visit `http://demo-app.example.com/demo-app-2`, the demo-app-2 Deployment should respond to the request. When creating the LoadBalancer, if you choose external network mode and bind an EIP, you can access it directly through the external network. We can add the Service's external IP and domain name to the local `/etc/hosts` file to access via domain name from the external network. If it is in internal network mode, only resources within the VPC can access through Ingress. ```bash $ cat /etc/hosts ...... xxx.yy.xxx.yy demo-app.example.com ``` ```bash $ curl http://demo-app.example.com/demo-app-2 Scheme: http Server address: 172.20.43.118:80 Server name: demo-app-2-5f6c5df698-rvsmc Date: 12/Jan/2022:02:56:38 +0000 URI: /demo-app-2 Request ID: ba34c07f5cc78e74629041df5568977a ``` ## V. TLS Support In the above Ingress object, we did not specify a TLS certificate for the Host. The Ingress Controller supports encrypting the site by specifying a Secret containing the TLS private key and certificate. First, create a Secret containing tls.crt and tls.key. When generating the certificate, ensure that the CN contains `demo-app.example.com` and base64 encode the certificate content. ```bash $ HOST=demo-app.example.com $ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=${HOST}/O=${HOST}" Generating a RSA private key ................................+++++ ................................+++++ writing new private key to 'tls.key' ``` ```bash $ kubectl create secret tls demo-app-tls --key tls.key --cert tls.crt secret/demo-app-tls created $ kubectl describe secret demo-app-tls Name: demo-app-tls Namespace: ingress-nginx Labels: Annotations: Type: kubernetes.io/tls Data ==== tls.crt: 1229 bytes tls.key: 1708 bytes ``` Then in the Ingress object, reference the Secret via the `ingress.spec.tls` field. The Ingress Controller will encrypt the communication pipeline between the client and the Ingress. ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: demo-app-ingress spec: ingressClassName: nginx tls: - hosts: - demo-app.example.com secretName: demo-app-tls defaultBackend: service: name: demo-app-1-svc port: number: 80 rules: - host: demo-app.example.com http: paths: - path: /demo-app-1 pathType: Prefix backend: service: name: demo-app-1-svc port: number: 80 - path: /demo-app-2 pathType: Prefix backend: service: name: demo-app-2-svc port: number: 80 ``` You can now access the Ingress via the HTTPS protocol. Note that since the Ingress certificate is self-signed, you need to skip certificate verification when using tools like curl. **In a production environment, it is recommended to use a TLS certificate signed by a CA.** ```bash $ curl --insecure https://demo-app.example.com/demo-app-2 Scheme: http Server address: 172.20.40.180:80 Server name: demo-app-2-5f6c5df698-pvr45 Date: 12/Jan/2022:03:19:58 +0000 URI: /demo-app-2 Request ID: 4cc35ddb1a301977f0477ded4c09d5df ``` ## VI. Setting Access Whitelist In some scenarios, your application only allows access from specified IP addresses. This can be achieved by adding annotations, specifically `nginx.ingress.kubernetes.io/whitelist-source-range`. The value is a list of CIDRs separated by commas ",". Example: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: demo-app-ingress annotations: nginx.ingress.kubernetes.io/whitelist-source-range: 172.16.0.0/16,172.18.0.0/16 spec: ingressClassName: nginx tls: - hosts: - demo-app.example.com secretName: demo-app-tls defaultBackend: service: name: demo-app-1-svc port: number: 80 rules: - host: demo-app.example.com http: paths: - path: /demo-app-1 pathType: Prefix backend: service: name: demo-app-1-svc port: number: 80 - path: /demo-app-2 pathType: Prefix backend: service: name: demo-app-2-svc port: number: 80 ``` ## VII. Retrieving Client Source IP with ALB/CLB7 When a layer-7 load balancer (ALB/CLB7) is deployed in front of the Ingress Controller, the source IP obtained by backend service Pods is the proxy IP of the load balancer, not the real client IP. To enable backend service Pods to obtain the real client IP, you need to modify the ConfigMap of `ingress-nginx-controller`. When `use-forwarded-headers: "true"` is set, if the client customizes the `X-Forwarded-For` header, the real client IP may not be obtainable. By adding the `compute-full-forwarded-for` parameter, you can get all IP addresses in the complete forwarding chain, thereby obtaining the real client IP. Additionally, you need to configure `proxy-real-ip-cidr` to specify the trusted load balancer proxy IP address ranges. Only `X-Forwarded-For` headers from these IPs will be trusted, preventing spoofing. 1. Run the following command to edit the ConfigMap: ```shell kubectl edit configmap ingress-nginx-controller -n ingress-nginx ``` 2. Add or modify the following configuration in the `data` field: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: ingress-nginx-controller namespace: ingress-nginx # ... data: # ... compute-full-forwarded-for: "true" forwarded-for-header: X-Forwarded-For use-forwarded-headers: "true" proxy-real-ip-cidr: 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 ``` **Parameter Description:** * `compute-full-forwarded-for`: Appends the client IP to the `X-Forwarded-For` header instead of replacing it. * `use-forwarded-headers`: Preserves the `X-Forwarded-*` series of request headers. * `forwarded-for-header`: Sets the request header used to obtain the real client IP, here using `X-Forwarded-For`. * `proxy-real-ip-cidr`: Trusted load balancer proxy IP ranges, separated by commas. Only `X-Forwarded-For` headers from these IPs will be trusted, preventing spoofing. After configuration, backend service Pods can correctly obtain the real client IP.