Configure mTLS Between Gateway and Upstream

Learn how to use APISIX or API7 Ingress Controller to configure the gateway to forward traffic to upstream services over mutual TLS (mTLS).

Applies to:API7 Ingress Controller (drives API7 Gateway)APISIX Ingress Controller (drives Apache APISIX)

Goal

Configure the gateway to authenticate itself to an upstream HTTPS service and verify the upstream's certificate using mutual TLS (mTLS).

Before you begin

Complete Set Up Ingress Controller and Gateway.
You have kubectl configured against the cluster used in the setup guide.
You have OpenSSL installed to create the test CA and certificates.
Choose a free local port for the upstream NGINX port-forward. The examples use 8443.

Set the variables this guide uses:

export UPSTREAM_HOST="test.com"
export UPSTREAM_LOCAL_PORT="8443"
export UPSTREAM_URL="https://127.0.0.1:${UPSTREAM_LOCAL_PORT}"
export GATEWAY_URL="http://127.0.0.1:9080"   # kubectl port-forward, or your Service's address

Steps

Step 1: Generate the test CA and certificates

Generate the Certificate Authority (CA) key and certificate:

openssl genrsa -out ca.key 2048
openssl req -new -x509 -days 36500 -sha256 \
  -key ca.key \
  -out ca.crt \
  -subj "/CN=MyTestCA" \
  -extensions v3_ca \
  -config <(printf "[req]\ndistinguished_name=req\n[ v3_ca ]\nbasicConstraints=critical,CA:TRUE\nkeyUsage=critical,keyCertSign,cRLSign\nsubjectKeyIdentifier=hash\nauthorityKeyIdentifier=keyid:always,issuer")

Generate the server key and certificate signing request (CSR), then sign it with the CA:

openssl genrsa -out server.key 2048
openssl req -new -sha256 \
  -key server.key \
  -out server.csr \
  -subj "/CN=${UPSTREAM_HOST}"

openssl x509 -req -days 36500 -sha256 \
  -in server.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out server.crt \
  -extensions v3_req \
  -extfile <(printf "[v3_req]\nbasicConstraints=CA:FALSE\nkeyUsage=digitalSignature,keyEncipherment\nextendedKeyUsage=serverAuth")

Generate the client key and CSR, then sign it with the CA:

openssl genrsa -out client.key 2048
openssl req -new -sha256 \
  -key client.key \
  -out client.csr \
  -subj "/CN=CLIENT"

openssl x509 -req -days 36500 -sha256 \
  -in client.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out client.crt \
  -extensions v3_req \
  -extfile <(printf "[v3_req]\nbasicConstraints=CA:FALSE\nkeyUsage=digitalSignature,keyEncipherment\nextendedKeyUsage=clientAuth")

Step 2: Deploy the upstream NGINX service

This guide uses NGINX as the example upstream service to demonstrate mutual TLS (mTLS).

Create a Kubernetes secret for NGINX:

kubectl create secret generic nginx-certs \
  --from-file=server.crt=server.crt \
  --from-file=server.key=server.key \
  --from-file=ca.crt=ca.crt \
  --namespace=aic

Create a ConfigMap for NGINX, in which TLS certificates are configured. The example uses the SNI test.com:

nginx-cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: aic
  name: nginx-config
data:
  default.conf: |
    server {
        listen 8443 ssl;
        server_name test.com;
        ssl_certificate        /etc/nginx/ssl/server.crt;
        ssl_certificate_key    /etc/nginx/ssl/server.key;
        ssl_client_certificate /etc/nginx/ssl/ca.crt;
        ssl_verify_client on;
        location /hello {
            return 200 "Hello World!";
        }
    }

❶ server_name: set to test.com to be consistent with the server certificate CN value.

❷ ssl_certificate: configure the path to the server certificate public key server.crt.

❸ ssl_certificate_key: configure the path to the server certificate private key server.key.

❹ ssl_client_certificate: configure the path to the CA certificate public key ca.crt.

❺ ssl_verify_client: set to on to verify the client certificate.

Create a Kubernetes manifest file for the NGINX example upstream service deployment and service:

nginx.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: quickstart-nginx
spec:
  type: NodePort
  selector:
    app: quickstart-nginx
  ports:
    - name: https
      port: 8443
      targetPort: 8443
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: aic
  name: quickstart-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: quickstart-nginx
  template:
    metadata:
      labels:
        app: quickstart-nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 8443
          volumeMounts:
            - name: config
              mountPath: /etc/nginx/conf.d
            - name: certs
              mountPath: /etc/nginx/ssl
      volumes:
        - name: config
          configMap:
            name: nginx-config
        - name: certs
          secret:
            secretName: nginx-certs

Apply the configuration to your cluster:

kubectl apply -f nginx-cm.yaml -f nginx.yaml

Step 3: Check upstream mTLS directly

Forward the NGINX service port to your local machine:

kubectl port-forward service/quickstart-nginx "${UPSTREAM_LOCAL_PORT}:8443" &

Send a request to the NGINX service's route with the client certificate and key:

curl -ik "${UPSTREAM_URL}/hello" --cert client.crt --key client.key

Expected: an HTTP/1.1 200 OK response and the following message:

Hello World!

To verify that the upstream rejects a request without a client certificate, run:

curl -ik "${UPSTREAM_URL}/hello"

Expected: an HTTP/1.1 400 Bad Request response.

Step 4: Configure gateway-to-upstream mTLS and create the route

The Ingress Controller currently does not support the configuration of upstream mTLS using Gateway API.

Create a Kubernetes secret that holds the client certificate and key:

kubectl create secret tls test-mtls-secret \
  --cert=client.crt \
  --key=client.key \
  --namespace=aic

Create a Kubernetes manifest file for a route to the NGINX service:

upstream-mtls-route.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: aic
  name: quickstart-nginx  # should match the service name
spec:
  ingressClassName: apisix
  scheme: https
  tlsSecret:
    name: test-mtls-secret
    namespace: aic
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: mtls-nginx
spec:
  ingressClassName: apisix
  http:
  - name: mtls-nginx
    match:
      paths:
      - /hello
    backends:
    - serviceName: quickstart-nginx
      servicePort: 8443

Apply the configuration to your cluster:

kubectl apply -f upstream-mtls-route.yaml

Validate

To verify mTLS between the gateway and the upstream service, send a request to the route through the gateway:

curl -ikv "${GATEWAY_URL}/hello"

Expected: an HTTP/1.1 200 OK response and the following message:

Hello World!

This verifies the successful establishment of mTLS between the gateway and the upstream service.

Roll back / Clean up

Delete the manifests and secrets created in this guide:

kubectl delete -f upstream-mtls-route.yaml
kubectl delete -f nginx.yaml
kubectl delete -f nginx-cm.yaml
kubectl delete secret nginx-certs -n aic
kubectl delete secret test-mtls-secret -n aic
rm -f ca.key ca.crt ca.srl server.key server.csr server.crt client.key client.csr client.crt

Stop the local port-forward process if it is still running:

kill %1 2>/dev/null || true

Troubleshoot

Symptom	Cause	Fix
Direct curl to NGINX returns `400 Bad Request` with the client certificate.	The client certificate is not signed by the CA trusted by NGINX, or the certificate/key files are mismatched.	Verify `client.crt` and `client.key` are the pair generated in Step 1 and that `ca.crt` is the CA used to sign `server.crt`.
Direct curl to NGINX returns `200 OK` without a client certificate.	`ssl_verify_client on` is missing from the NGINX ConfigMap.	Re-apply `nginx-cm.yaml` with `ssl_verify_client on;` and wait for the Pod to reload the configuration.
Curl through the gateway returns `502` or `503`.	The gateway cannot establish mTLS to the upstream, or the `ApisixUpstream` scheme/tlsSecret is wrong.	Confirm the `ApisixUpstream` uses `scheme: https` and that `tlsSecret` references the client certificate Secret created in Step 4.
Curl through the gateway returns `404`.	The `ApisixRoute` is not reconciled or the path does not match.	Check that the route `ingressClassName` and `serviceName` match the upstream Service.