Upstream mTLS

When an upstream service requires mutual TLS, API7 Gateway can act as a TLS client — presenting a client certificate to the upstream and optionally validating the upstream's server certificate against a trusted CA.

Configure it on the service's upstream, with three fields that reference stored certificate objects:

An older upstream.tls block accepts the same material inline as PEM. It still works, but it is marked deprecated in the Admin API and cannot reference stored secrets. See The deprecated inline tls block if you are maintaining a configuration that uses it, or driving the gateway with ADC.

How it works

1. The gateway opens a TLS connection to the upstream using the upstream's host/SNI.
2. The upstream presents its server certificate. If tls_verify is true, the gateway validates it against the CAs in ca_certificates; otherwise the certificate is accepted without verification.
3. The upstream requests a client certificate. The gateway presents the certificate referenced by client_certificate.
4. The upstream validates the client certificate against its own trusted CA. If validation succeeds, the request is proxied; otherwise the gateway returns HTTP 502.

Prerequisites

• A TLS-enabled upstream service that requests a client certificate and validates it against a trusted CA.

• jq, for reading IDs out of the API responses.

• A token from the Dashboard, and the variables every command below uses:

  export API7_ADMIN_URL="https://localhost:7443"
  export GATEWAY_GROUP_ID="default"   # the group the quickstart creates; otherwise its UUID from Gateway Groups in the Dashboard
  read -rsp "Dashboard token: " API_KEY && export API_KEY && echo

Prepare the following PEM-encoded material:

The examples below use an nginx instance with the following configuration:

events {}
http {
  server {
    listen 8443 ssl;
    server_name upstream.test.local;
    ssl_certificate /certs/upstream.crt;
    ssl_certificate_key /certs/upstream.key;
    ssl_client_certificate /certs/ca.crt;
    ssl_verify_client on;
    location / {
      return 200 "upstream-mtls-ok client=$ssl_client_s_dn\n";
      add_header Content-Type text/plain;
    }
  }
}

If you do not have a set of certificates for testing, generate one with:

# CA
openssl req -x509 -newkey rsa:2048 -keyout ca.key -out ca.crt -days 365 -nodes \
  -subj "/CN=Test CA"

# Upstream server certificate
openssl req -newkey rsa:2048 -keyout upstream.key -out upstream.csr -nodes \
  -subj "/CN=upstream.test.local"
openssl x509 -req -in upstream.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out upstream.crt -days 365

# Client certificate (presented by the gateway)
openssl req -newkey rsa:2048 -keyout client.key -out client.csr -nodes \
  -subj "/CN=api7-gateway"
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out client.crt -days 365

Configure upstream mTLS

Step 1: Store the client certificate

Create a certificate object holding the certificate and key the gateway will present. Note the id in the response — the upstream references it.

CLIENT_CRT=$(awk 1 ORS='\\n' client.crt)
CLIENT_KEY=$(awk 1 ORS='\\n' client.key)

CLIENT_CERT_ID=$(curl -sk "${API7_ADMIN_URL}/apisix/admin/certificates?gateway_group_id=${GATEWAY_GROUP_ID}" -X POST \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{
    \"name\": \"upstream-mtls-client\",
    \"cert\": \"${CLIENT_CRT}\",
    \"key\": \"${CLIENT_KEY}\"
  }" | jq -r '.value.id')

echo "${CLIENT_CERT_ID}"

The POST returns the created object, so its id comes straight out of the response — do not look it up by name afterwards, because re-running this guide leaves two certificates with the same name and a lookup would pick one at random.

Build the PEM variables with a top-level assignment as shown. Substituting $(awk ...) directly inside the quoted -d string does not work: the outer double quotes collapse \\n to \n before awk runs, so awk emits real newlines into the JSON string and the request is rejected as malformed.

Step 2: Store the upstream CA certificate

CA certificates are a separate resource, and they take only a certificate — there is no key:

CA_CRT=$(awk 1 ORS='\\n' ca.crt)

CA_CERT_ID=$(curl -sk "${API7_ADMIN_URL}/apisix/admin/ca_certificates?gateway_group_id=${GATEWAY_GROUP_ID}" -X POST \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{
    \"name\": \"upstream-mtls-ca\",
    \"cert\": \"${CA_CRT}\"
  }" | jq -r '.value.id')

echo "${CA_CERT_ID}"

Skip this step if you are not verifying the upstream's certificate.

Step 3: Create a service that references them

Create a service whose upstream uses the https scheme (or grpcs for gRPC) and references the two certificate objects by ID.

pass_host: rewrite together with upstream_host ensures the SNI sent during the TLS handshake matches the upstream server's certificate, even when the node is addressed by IP.

curl -k "${API7_ADMIN_URL}/apisix/admin/services/upstream-mtls?gateway_group_id=${GATEWAY_GROUP_ID}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{
    \"name\": \"upstream-mtls\",
    \"upstream\": {
      \"type\": \"roundrobin\",
      \"scheme\": \"https\",
      \"pass_host\": \"rewrite\",
      \"upstream_host\": \"upstream.test.local\",
      \"nodes\": [
        { \"host\": \"192.0.2.10\", \"port\": 8443, \"weight\": 1 }
      ],
      \"client_certificate\": \"${CLIENT_CERT_ID}\",
      \"ca_certificates\": [\"${CA_CERT_ID}\"],
      \"tls_verify\": true
    }
  }"

Replace the node host with the address of your upstream.

The control plane resolves each ID at write time and rejects the request if a certificate does not exist in that gateway group, so a typo fails immediately rather than at the first proxied request.

To skip upstream certificate verification — for example during development against a self-signed upstream certificate — set tls_verify to false and omit ca_certificates. The gateway still presents its client certificate but does not validate the upstream's. This is not recommended for production.

Step 4: Create a route on the service

curl -k "${API7_ADMIN_URL}/apisix/admin/routes/upstream-mtls?gateway_group_id=${GATEWAY_GROUP_ID}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "upstream-mtls",
    "service_id": "upstream-mtls",
    "paths": ["/*"]
  }'

Step 5: Verify

Send a request through the gateway:

curl -i "http://127.0.0.1:9080/anything"

You should receive a 200 OK response and a body of the form:

upstream-mtls-ok client=CN=api7-gateway

The client= portion echoes the subject DN of the client certificate the gateway presented, confirming that the upstream completed the mTLS handshake against it.

If you remove client_certificate and re-apply the service, the same request no longer succeeds — but the exact failure mode depends on your upstream's nginx/OpenSSL version. Older combinations reject the missing certificate at the TLS handshake, which the gateway surfaces as HTTP 502 with an SSL alert (peer did not return a certificate) in the gateway error log. Newer combinations complete the TLS handshake and reject the request at the HTTP layer instead, so the upstream's own error response (for example 400 No required SSL certificate was sent) is proxied through unchanged, with no SSL alert logged on the gateway side. Either way, the request stops succeeding — check the upstream's own logs if the response code does not match what you expect here.

Rotate the certificate

Because the upstream holds an ID rather than a copy of the PEM, rotation does not touch the service: update the certificate object in place and every upstream referencing it picks up the new material.

NEW_CRT=$(awk 1 ORS='\\n' client-new.crt)
NEW_KEY=$(awk 1 ORS='\\n' client-new.key)

curl -k "${API7_ADMIN_URL}/apisix/admin/certificates/${CLIENT_CERT_ID}?gateway_group_id=${GATEWAY_GROUP_ID}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{
    \"name\": \"upstream-mtls-client\",
    \"cert\": \"${NEW_CRT}\",
    \"key\": \"${NEW_KEY}\"
  }"

Roll the CA the other way around: add the new CA as a second entry in ca_certificates and let both be trusted while the upstream switches over, then remove the old one. Replacing the only entry in a single step means every request in flight against the old certificate fails.

For the certificate rotation between the control plane and data planes, which is a separate mechanism, see Rotate Certificates.

Reference certificates from a secret provider

A certificate object's cert and key accept a $secret:// or $env:// reference instead of inline PEM, so the private key never has to be stored in the control plane:

curl -k "https://localhost:7443/apisix/admin/certificates?gateway_group_id={gateway_group_id}" -X POST \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "upstream-mtls-client",
    "cert": "$secret://vault/1/upstream-mtls/cert",
    "key": "$secret://vault/1/upstream-mtls/key"
  }'

See Secure Credentials for the available providers and the URI format.

The deprecated inline tls block

The older form embeds the PEM material in the upstream itself:

"upstream": {
  "scheme": "https",
  "nodes": [{ "host": "192.0.2.10", "port": 8443, "weight": 1 }],
  "tls": {
    "client_cert": "-----BEGIN CERTIFICATE-----\n...",
    "client_key": "-----BEGIN PRIVATE KEY-----\n...",
    "verify": true,
    "ca_certs": ["-----BEGIN CERTIFICATE-----\n..."]
  }
}

All four fields are implemented and enforced by the data plane, so an existing configuration keeps working. Prefer the certificate objects for new work: the inline block is marked deprecated in the Admin API schema, it cannot reference a secret provider, and rotating a certificate means rewriting every service that embeds it.

tls.client_cert_id is the one bridge between the two forms — it references a certificate object while keeping the rest of the block inline.

With ADC

ADC's schema supports only the inline form, and only four of its fields: client_cert, client_key, client_cert_id and verify.

services:
  - name: upstream-mtls
    upstream:
      type: roundrobin
      scheme: https
      pass_host: rewrite
      upstream_host: upstream.test.local
      nodes:
        - host: 192.0.2.10
          port: 8443
          weight: 1
      tls:
        client_cert_id: "<certificate object id>"
        verify: false
    routes:
      - name: upstream-mtls
        uris:
          - /*

adc sync -f adc.yaml

Troubleshooting

Next steps

• Client mTLS Authentication: Configure mTLS for clients connecting to the gateway.
• SSL Certificates: Manage certificates in API7 Gateway.
• Secure Credentials: Store certificates in external secret managers.