Integrate with Kubernetes Service Discovery

Grant namespace-scoped discovery access, configure Apache APISIX to watch ready Kubernetes endpoints securely, validate routing, and clean up.

Deployments:KubernetesRoles:Platform engineerTime:25 minutesImpact:Changes live traffic

Goal

Configure an in-cluster APISIX deployment to watch EndpointSlices for one namespace and route to two ready sample pods. Use a dedicated service account, namespace-scoped read-only RBAC, verified Kubernetes API TLS, an authenticated fixed-ID Route, and complete cleanup.

Before you begin

Run APISIX in Kubernetes with network access to the Kubernetes API and pod addresses.
Install kubectl, cURL, and jq.
Select a dedicated APISIX service account. Changing an existing workload's service account can remove permissions it already needs, so review and preserve those permissions before rollout.
Know how your deployment manages config.yaml and its trusted CA bundle. Back up those resources before changing them.
The commands below create and later delete the entire apisix-discovery-guide namespace. Stop if that namespace already exists.

Set the gateway endpoints and the identity used by the APISIX pods:

export APISIX_ADMIN_URL="http://127.0.0.1:9180"
export GATEWAY_URL="http://127.0.0.1:9080"   # kubectl port-forward, or your APISIX Service address
export APISIX_NAMESPACE="apisix"             # namespace containing the APISIX workload
export APISIX_SERVICE_ACCOUNT="apisix"       # dedicated ServiceAccount selected for that workload
export APISIX_WORKLOAD="deployment/apisix"   # Deployment or StatefulSet that runs APISIX
export DISCOVERY_NAMESPACE="apisix-discovery-guide"
read -rsp "Admin API key: " APISIX_ADMIN_KEY && export APISIX_ADMIN_KEY && echo

Verify the selected service account and protect the fixed Route and namespace:

kubectl get serviceaccount "${APISIX_SERVICE_ACCOUNT}" -n "${APISIX_NAMESPACE}" >/dev/null
kubectl get "${APISIX_WORKLOAD}" -n "${APISIX_NAMESPACE}" >/dev/null
test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${APISIX_ADMIN_URL}/apisix/admin/routes/kubernetes-discovery-guide-route" -H "X-API-KEY: ${APISIX_ADMIN_KEY}")" = 404 || { echo 'Stop: route kubernetes-discovery-guide-route already exists'; exit 1; }
if kubectl get namespace "${DISCOVERY_NAMESPACE}" >/dev/null 2>&1; then
  echo "Stop: namespace ${DISCOVERY_NAMESPACE} already exists"
  exit 1
fi

Steps

Step 1: Grant namespace-scoped discovery access

Create an isolated namespace, a read-only Role, and a RoleBinding to the service account used by APISIX:

kubectl create namespace "${DISCOVERY_NAMESPACE}"
kubectl create role apisix-discovery-guide \
  --verb=get,list,watch \
  --resource=endpoints,endpointslices.discovery.k8s.io \
  -n "${DISCOVERY_NAMESPACE}"
kubectl create rolebinding apisix-discovery-guide \
  --role=apisix-discovery-guide \
  --serviceaccount="${APISIX_NAMESPACE}:${APISIX_SERVICE_ACCOUNT}" \
  -n "${DISCOVERY_NAMESPACE}"

Assert the exact read operations APISIX needs:

for check in \
  'get endpoints' \
  'list endpoints' \
  'watch endpoints' \
  'get endpointslices.discovery.k8s.io' \
  'list endpointslices.discovery.k8s.io' \
  'watch endpointslices.discovery.k8s.io'; do
  set -- ${check}
  test "$(kubectl auth can-i "$1" "$2" --as="system:serviceaccount:${APISIX_NAMESPACE}:${APISIX_SERVICE_ACCOUNT}" -n "${DISCOVERY_NAMESPACE}")" = yes
done

This Role grants no write operation and no access outside apisix-discovery-guide. Repeat a namespace-scoped RoleBinding deliberately for each additional namespace; do not replace it with cluster-wide access merely for convenience.

What changed: the selected APISIX service account can read only Endpoints and EndpointSlices in the isolated guide namespace.

Step 2: Deploy two controlled endpoints

Create a two-replica HTTPBin Deployment and a Service with the named port http:

kubectl apply -f - <<YAML
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-discovery-guide
  namespace: ${DISCOVERY_NAMESPACE}
spec:
  replicas: 2
  selector:
    matchLabels:
      app: kubernetes-discovery-guide
  template:
    metadata:
      labels:
        app: kubernetes-discovery-guide
    spec:
      containers:
        - name: httpbin
          image: mccutchen/go-httpbin:v2.15.0
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /status/204
              port: http
            periodSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: kubernetes-discovery-guide
  namespace: ${DISCOVERY_NAMESPACE}
spec:
  selector:
    app: kubernetes-discovery-guide
  ports:
    - name: http
      port: 80
      targetPort: http
YAML
kubectl rollout status deployment/kubernetes-discovery-guide -n "${DISCOVERY_NAMESPACE}" --timeout=120s
kubectl get endpointslices -n "${DISCOVERY_NAMESPACE}" \
  -l kubernetes.io/service-name=kubernetes-discovery-guide -o json | \
  jq -e '[.items[].endpoints[] | select(.conditions.ready == true)] | length == 2'

What changed: Kubernetes publishes exactly two ready endpoints behind the named Service port.

Step 3: Configure APISIX discovery and trust

Ensure the APISIX workload uses ${APISIX_SERVICE_ACCOUNT} as spec.template.spec.serviceAccountName. Kubernetes then mounts a short-lived, rotating token and the cluster CA into each pod. Roll out the workload after changing its pod specification.

Merge this configuration into the deployment's existing config.yaml; do not replace other top-level fields or append a second apisix or discovery key:

config.yaml

apisix:
  ssl:
    ssl_trusted_certificate: /usr/local/apisix/conf/trusted-ca-bundle.pem

discovery:
  kubernetes:
    shared_size: 64m
    service:
      schema: https
      host: kubernetes.default.svc
      port: "443"
      ssl_verify: true
    client:
      token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    namespace_selector:
      equal: apisix-discovery-guide
    watch_endpoint_slices: true

Build trusted-ca-bundle.pem from the mounted Kubernetes CA and all roots already trusted by APISIX. The apisix.ssl.ssl_trusted_certificate setting is global; replacing an existing bundle with only the Kubernetes CA can break other HTTPS connections.

Before rollout, validate the rendered APISIX configuration in a candidate pod with apisix test. If validation fails, restore the complete configuration backup and do not reload or restart with the rejected file. After a successful rollout, verify the workload uses the selected service account:

kubectl rollout status "${APISIX_WORKLOAD}" -n "${APISIX_NAMESPACE}" --timeout=120s
kubectl get "${APISIX_WORKLOAD}" -n "${APISIX_NAMESPACE}" -o json | \
  jq -e --arg account "${APISIX_SERVICE_ACCOUNT}" '.spec.template.spec.serviceAccountName == $account'

What changed: APISIX watches only the guide namespace using a projected service-account token and verifies the Kubernetes API certificate against a combined trust bundle.

Step 4: Create the discovery Route

For a single cluster, service_name uses namespace/service:port-name. Create one authenticated fixed-ID Route and assert its stored discovery settings:

jq -n --arg service "${DISCOVERY_NAMESPACE}/kubernetes-discovery-guide:http" '{
  name:"kubernetes-discovery-guide-route",
  uri:"/kubernetes-discovery-guide",
  methods:["GET"],
  upstream:{type:"roundrobin",discovery_type:"kubernetes",service_name:$service}
}' | \
curl -sS --connect-timeout 3 --max-time 15 \
  "${APISIX_ADMIN_URL}/apisix/admin/routes/kubernetes-discovery-guide-route" -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" -H 'Content-Type: application/json' \
  --data-binary @- | \
jq -e --arg service "${DISCOVERY_NAMESPACE}/kubernetes-discovery-guide:http" '.value.id == "kubernetes-discovery-guide-route" and .value.upstream.discovery_type == "kubernetes" and .value.upstream.service_name == $service'

What changed: GET /kubernetes-discovery-guide resolves the named Service port through Kubernetes discovery.

Validate

Wait for discovery to converge and require a successful response from the controlled service:

for attempt in {1..30}; do
  curl --fail-with-body -sS --connect-timeout 3 --max-time 10 \
    "${GATEWAY_URL}/kubernetes-discovery-guide" | \
    jq -e '.url | endswith("/kubernetes-discovery-guide")' && break
  sleep 1
done
curl --fail-with-body -sS --connect-timeout 3 --max-time 10 \
  "${GATEWAY_URL}/kubernetes-discovery-guide" | \
  jq -e '.url | endswith("/kubernetes-discovery-guide")'

If the Control API is enabled on a protected operator endpoint, inspect /v1/discovery/kubernetes/dump and confirm the namespace, service, port name, and two ready endpoint addresses. Do not publish the Control API merely to complete this guide.

Roll back / Clean up

Delete the Route first, restore the complete pre-guide APISIX configuration and workload service account through the same deployment mechanism, validate with apisix test, and complete that rollout. Then delete the isolated namespace:

curl --fail-with-body -sS --connect-timeout 3 --max-time 15 -X DELETE \
  "${APISIX_ADMIN_URL}/apisix/admin/routes/kubernetes-discovery-guide-route" \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}"
kubectl delete namespace "${DISCOVERY_NAMESPACE}" --wait=true --timeout=120s
unset APISIX_ADMIN_URL APISIX_ADMIN_KEY GATEWAY_URL APISIX_NAMESPACE APISIX_SERVICE_ACCOUNT APISIX_WORKLOAD DISCOVERY_NAMESPACE

Expected: the fixed Route and isolated namespace are absent, the namespace-scoped Role and RoleBinding were deleted with that namespace, and APISIX is running with its complete pre-guide configuration and identity.

Troubleshoot

Symptom	Cause	Fix
An RBAC assertion prints `no`.	The RoleBinding names a different service account, or the binding is in the wrong namespace.	Compare the APISIX pod's `serviceAccountName` with the binding subject and keep the binding in the namespace being watched.
APISIX reports certificate verification failure.	The bundle lacks the Kubernetes API CA, the API hostname does not match the certificate, or system time is wrong.	Correct the combined CA bundle, certificate chain, hostname, or time. Do not set `ssl_verify: false` as a production fix.
The Route returns `503` or `502`.	Discovery has not converged, no endpoint is ready, or pod addresses are unreachable from APISIX.	Re-run the exact EndpointSlice assertion, inspect the protected discovery dump, and verify network policy and routing.
Old endpoints remain visible after a configuration change.	Shared memory can retain earlier discovery state during reload.	Confirm a fresh successful Kubernetes API watch and the current pod configuration; do not treat an old cache entry as proof that TLS or RBAC now works.

Multiple clusters

Use an array only when APISIX must watch more than one cluster. Each item requires a unique lowercase alphanumeric id, an explicit API server address, its own token file, and a CA bundle that trusts every configured API server. The service name becomes cluster-id/namespace/service:port-name. Grant the service account namespace-scoped read access in each required cluster and namespace.

APISIX Kubernetes discovery — single- and multi-cluster configuration fields and service-name syntax.
Kubernetes RBAC — namespace scope, RoleBindings, and service-account least privilege.
Kubernetes ServiceAccounts — projected, rotating pod credentials.
Discovery Control API — inspect discovery state without exposing it publicly.