Docs
Ingress ControllerManage traffic and APIsTraffic ManagementConfigure Upstream Health Checks

Configure Upstream Health Checks

Configure active and passive upstream health checks for APISIX or API7 Ingress Controller.

Applies to:API7 Ingress Controller (drives API7 Gateway)APISIX Ingress Controller (drives Apache APISIX)

Goal

Configure active and passive health checks for upstream Services using the Ingress Controller and verify how the gateway marks nodes healthy or unhealthy.

Before you begin

  • Complete Set Up Ingress Controller and Gateway.
  • You have kubectl configured against the cluster used in the setup guide.
  • The examples use the aic namespace and APISIX Control API on port 9090. For API7 Gateway, replace GATEWAY_POD_NAME with a gateway Pod name.

Set the variables this guide uses:

export UPSTREAM1="nginx1"
export UPSTREAM2="nginx2"
export UPSTREAM1_PORT="8080"
export UPSTREAM2_PORT="8081"
export UPSTREAM_TARGET_PORT="80"
export GATEWAY_SERVICE_NAME="apisix-gateway"
export GATEWAY_HTTP_PORT="80"
export FORWARD_PORT="9080"
export CONTROL_PORT="9090"
export CONTROL_FORWARD_PORT="9090"
export APISIX_CONTROL_SERVICE="apisix-control"
export GATEWAY_POD_NAME="api7-gateway-0"   # replace with your API7 Gateway Pod name

Steps

Step 1: Deploy the example upstream services

Create a manifest for two NGINX Services in the aic namespace:

nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: aic
  name: nginx1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx1
  template:
    metadata:
      labels:
        app: nginx1
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: nginx1
spec:
  selector:
    app: nginx1
  ports:
    - name: http
      port: 8080
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: aic
  name: nginx2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx2
  template:
    metadata:
      labels:
        app: nginx2
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: nginx2
spec:
  selector:
    app: nginx2
  ports:
    - name: http
      port: 8081
      targetPort: 80

Apply the manifest and wait for the Pods to become ready:

kubectl apply -f nginx.yaml
kubectl wait --for=condition=ready pod -l app=nginx1 -n aic --timeout=120s
kubectl wait --for=condition=ready pod -l app=nginx2 -n aic --timeout=120s

Port-forward both Services to your local machine and verify they respond:

kubectl port-forward -n aic "svc/${UPSTREAM1}" "${UPSTREAM1_PORT}:${UPSTREAM1_PORT}" &
kubectl port-forward -n aic "svc/${UPSTREAM2}" "${UPSTREAM2_PORT}:${UPSTREAM2_PORT}" &
for port in ${UPSTREAM1_PORT} ${UPSTREAM2_PORT}; do
  curl -s "http://127.0.0.1:${port}" | grep -q "Welcome to nginx" &&
  echo "NGINX welcome page available on port ${port}."
done

Expected output:

NGINX welcome page available on port 8080.
NGINX welcome page available on port 8081.

Step 2: Enable the Control API

Export the current Helm values:

helm get values -n aic apisix --all > values.yaml

Add the Control API configuration in values.yaml:

values.yaml
apisix:
  enabled: true
control:
  enabled: true

Upgrade the release:

helm upgrade -n aic apisix apisix/apisix -f values.yaml

Port-forward the Control API Service:

kubectl port-forward -n aic "service/${APISIX_CONTROL_SERVICE}" "${CONTROL_FORWARD_PORT}:${CONTROL_PORT}" &

Step 3: Configure active health checks

Active checks determine the health of upstream services by periodically sending requests and seeing how they respond.

Create manifests that configure active health checks on the two NGINX Services and a route to both:

active-health-checks.yaml
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: aic
  name: nginx1
spec:
  targetRefs:
    - name: nginx1
      kind: Service
      group: ""
      sectionName: http
  healthCheck:
    active:
      type: http
      httpPath: /
      healthy:
        interval: 2s
        successes: 1
      unhealthy:
        interval: 1s
        timeouts: 3
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: aic
  name: nginx2
spec:
  targetRefs:
    - name: nginx2
      kind: Service
      group: ""
      sectionName: http
  healthCheck:
    active:
      type: http
      httpPath: /
      healthy:
        interval: 2s
        successes: 1
      unhealthy:
        interval: 1s
        timeouts: 3
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: example-hc-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /
      backendRefs:
        - name: nginx1
          port: 8080
          weight: 1
        - name: nginx2
          port: 8081
          weight: 1

Apply the manifest:

kubectl apply -f active-health-checks.yaml

sectionName selects a Service port by spec.ports[].name. If the named port does not exist, the policy is not attached. Check the policy status with:

kubectl describe backendtrafficpolicy ${UPSTREAM1} -n aic

Step 4: Check active health check status changes

Forward the gateway HTTP port to your local machine:

kubectl port-forward -n aic "svc/${GATEWAY_SERVICE_NAME}" "${FORWARD_PORT}:${GATEWAY_HTTP_PORT}" &

Send a request to the route to start health checks:

curl "http://127.0.0.1:${FORWARD_PORT}/"

Query the health check endpoint:

curl "http://127.0.0.1:${CONTROL_FORWARD_PORT}/v1/healthcheck"

You should see both upstream nodes with status: healthy.

Make one upstream unavailable

Scale one deployment down:

kubectl scale deployment ${UPSTREAM1} -n aic --replicas=0

Wait a few seconds and query the health check endpoint again:

curl "http://127.0.0.1:${CONTROL_FORWARD_PORT}/v1/healthcheck"

You should see one node marked unhealthy with timeout_failure counts.

Send a request to the route:

curl -i "http://127.0.0.1:${FORWARD_PORT}/"

You should receive HTTP/1.1 200 OK.

Make both upstreams unavailable

Scale both deployments down:

kubectl scale deployment ${UPSTREAM2} -n aic --replicas=0

Query the health check endpoint:

curl "http://127.0.0.1:${CONTROL_FORWARD_PORT}/v1/healthcheck"

Both nodes should be unhealthy. Send a request to the route:

curl -i "http://127.0.0.1:${FORWARD_PORT}/"

You should receive HTTP/1.1 502 Bad Gateway.

Recover both upstreams

Scale both deployments back up:

kubectl scale deployment -n aic ${UPSTREAM1} ${UPSTREAM2} --replicas=1

Wait a few seconds and query the health check endpoint:

curl "http://127.0.0.1:${CONTROL_FORWARD_PORT}/v1/healthcheck"

Both nodes should return to healthy.

Step 5: Configure active health checks that always report unhealthy

The gateway can still forward client requests to upstream services even when all active health statuses are unhealthy. To demonstrate, configure active health checks that probe a path that returns 404:

active-health-checks.yaml
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: aic
  name: nginx1
spec:
  targetRefs:
    - name: nginx1
      kind: Service
      group: ""
  healthCheck:
    active:
      type: http
      httpPath: /404
      healthy:
        interval: 2s
        successes: 1
      unhealthy:
        interval: 1s
        httpFailures: 2
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: aic
  name: nginx2
spec:
  targetRefs:
    - name: nginx2
      kind: Service
      group: ""
  healthCheck:
    active:
      type: http
      httpPath: /404
      healthy:
        interval: 2s
        successes: 1
      unhealthy:
        interval: 1s
        httpFailures: 2
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: example-hc-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /
      backendRefs:
        - name: nginx1
          port: 8080
          weight: 1
        - name: nginx2
          port: 8081
          weight: 1

Apply the manifest:

kubectl apply -f active-health-checks.yaml

Step 6: Test request forwarding when all upstreams are unhealthy

Send a request to the route:

curl -i "http://127.0.0.1:${FORWARD_PORT}/"

You should receive HTTP/1.1 200 OK.

Query the health check endpoint:

curl "http://127.0.0.1:${CONTROL_FORWARD_PORT}/v1/healthcheck"

Both nodes should be marked unhealthy with http_failure counters. This confirms the gateway still forwards client requests even when all upstream health statuses are unhealthy.

Step 7: Configure passive health checks

known issue

There is a known issue in the gateway where the passive health check data displayed through the Control API do not accurately reflect the actual health statuses, so your testing results may differ from the example shown. This issue is being actively resolved. However, the passive health check mechanism itself is functioning correctly and continues to route requests as intended.

To use passive health checks, you must also configure active health checks so the gateway can detect when an upstream recovers. The following manifest adds passive health checks to the existing active checks:

passive-health-checks.yaml
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: aic
  name: nginx1
spec:
  targetRefs:
    - name: nginx1
      kind: Service
      group: ""
  healthCheck:
    active:
      type: http
      httpPath: /
      healthy:
        interval: 99999s
        successes: 1
      unhealthy:
        interval: 30s
    passive:
      healthy:
        httpCodes: [200, 201, 202, 300, 301, 302]
        successes: 1
      unhealthy:
        httpCodes: [429, 404, 500, 501, 502, 503, 504, 505]
        httpFailures: 3
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: aic
  name: nginx2
spec:
  targetRefs:
    - name: nginx2
      kind: Service
      group: ""
  healthCheck:
    active:
      type: http
      httpPath: /
      healthy:
        interval: 99999s
        successes: 1
      unhealthy:
        interval: 30s
    passive:
      healthy:
        httpCodes: [200, 201, 202, 300, 301, 302]
        successes: 1
      unhealthy:
        httpCodes: [429, 404, 500, 501, 502, 503, 504, 505]
        httpFailures: 3
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: example-hc-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /404
      backendRefs:
        - name: nginx1
          port: 8080
          weight: 1
        - name: nginx2
          port: 8081
          weight: 1

Apply the manifest:

kubectl apply -f passive-health-checks.yaml

Step 8: Test passive health check behavior

Send a request to the /404 route:

curl -i "http://127.0.0.1:${FORWARD_PORT}/404"

You should see HTTP/1.1 404 Not Found.

Query the health check endpoint:

curl "http://127.0.0.1:${CONTROL_FORWARD_PORT}/v1/healthcheck"

One of the nodes should show an http_failure count and a mostly_healthy status, meaning the node is currently healthy but has started to receive unhealthy indications.

Generate consecutive 404 responses:

resp=$(seq 10 | xargs -I{} curl "http://127.0.0.1:${FORWARD_PORT}/404" -o /dev/null -s -w "%{http_code}\n") && \
  count=$(echo "$resp" | grep "404" | wc -l) && \
  echo "Invoked ${count} responses with 404 status code."

Query the health check endpoint again:

curl "http://127.0.0.1:${CONTROL_FORWARD_PORT}/v1/healthcheck"

Both nodes should now be unhealthy. Wait at least 30 seconds for active checks to probe / and mark them as healthy again. Then query the endpoint once more:

curl "http://127.0.0.1:${CONTROL_FORWARD_PORT}/v1/healthcheck"

Both nodes should return to healthy.

Validate

As a final check, confirm the health check endpoint reports both nodes as healthy after the upstreams recover:

kubectl scale deployment -n aic ${UPSTREAM1} ${UPSTREAM2} --replicas=1
curl "http://127.0.0.1:${CONTROL_FORWARD_PORT}/v1/healthcheck"

Expected: both nodes have status: healthy.

Roll back / Clean up

Delete the manifests and stop the port-forwards:

kubectl delete -f passive-health-checks.yaml
kubectl delete -f active-health-checks.yaml
kubectl delete -f nginx.yaml
kill %1 2>/dev/null || true
kill %2 2>/dev/null || true

If you updated the gateway Helm values for the Control API, re-apply the original values file or remove the control section.

Troubleshoot

SymptomCauseFix
The health check endpoint returns no data.The Control API is not enabled, or the port-forward is not running.Verify the control section is enabled and that CONTROL_FORWARD_PORT is forwarded.
Scaling down one deployment does not mark the node unhealthy.The active check intervals are too long, or the health check has not had time to run.Wait for the unhealthy.interval and timeouts thresholds configured in the BackendTrafficPolicy or ApisixUpstream.
Passive checks show mostly_healthy but not unhealthy.Not enough httpFailures have been accumulated from 404 responses.Send more consecutive requests to the /404 route or lower httpFailures.
The gateway returns 502 when only one node is down.Both deployments are scaled to zero, or the HTTPRoute status is not Accepted.Check kubectl get httproute example-hc-route -n aic and scale at least one upstream back up.

Proxy WebSocket Connections

Previous Page

Detect Upstream Protocol with appProtocol

Next Page