Configure Upstream Health Checks

Configure active and passive checks and verify exact upstream state through the Control API.

Goal

Create one two-node service that demonstrates active failure detection, failover, and recovery, plus one single-node service that demonstrates passive HTTP failure counting and active recovery. Verify the exact node states through the Data Plane Control API.

When at least one node is healthy, API7 Gateway selects from the healthy subset. If every node is marked unhealthy, the current balancer falls back to the complete configured node set; health checks do not guarantee that all-unhealthy traffic is rejected.

The following active-check flow shows an unhealthy node being excluded while another healthy node remains:

Before you begin

This procedure requires API7 Gateway 3.10.7 or later for the checkers Control API sub-resource used below.

Set the variables used in this guide:

export API7_ADMIN_URL="https://localhost:7443"
export API7_GATEWAY_GROUP_ID="default"   # only the group the quickstart creates; otherwise copy the gateway group's UUID from Gateway Groups in the Dashboard
read -rp "Gateway group name for ADC (if used): " API7_GATEWAY_GROUP_NAME && export API7_GATEWAY_GROUP_NAME
export GATEWAY_URL="http://127.0.0.1:9080"
read -rp "Operator-reachable Data Plane Control API URL: " CONTROL_API_URL && export CONTROL_API_URL
read -rp "Host or DNS name the gateway uses to reach this machine: " HEALTH_UPSTREAM_HOST && export HEALTH_UPSTREAM_HOST
read -rsp "Dashboard token: " API7_ADMIN_KEY && export API7_ADMIN_KEY && echo

CONTROL_API_URL is commonly reachable only from the gateway host or network namespace, for example http://127.0.0.1:9090 there. Keep the Control API on an operator-only network; do not expose it through a public data-plane route.

Confirm that the control plane and selected gateway group are ready:

curl -ksS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}\n' "${API7_ADMIN_URL}/api/gateway_groups" -H "X-API-KEY: ${API7_ADMIN_KEY}"
curl -ksS --connect-timeout 3 --max-time 10 "${API7_ADMIN_URL}/api/gateway_groups/${API7_GATEWAY_GROUP_ID}/instances" -H "X-API-KEY: ${API7_ADMIN_KEY}" | grep -o '"status": *"Healthy"' | wc -l
curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}\n' "${CONTROL_API_URL}/v1/healthcheck"

Expected: 200, 1 or more, then 200.

Python 3, curl, jq, and GNU envsubst are available. The gateway can reach ${HEALTH_UPSTREAM_HOST} on TCP ports 18083, 18084, and 18085.

Reserve the ports and local files:

for file in health-upstream.py health-1.mode health-2.mode health-passive.mode health-1.pid health-2.pid health-passive.pid health-1.log health-2.log health-passive.log health-check-adc.yaml.tmpl health-check-adc.yaml health-check-cleanup.yaml; do test ! -e "${file}" || { echo "Stop: ${file} already exists"; exit 1; }; done
python3 - <<'PY'
import socket

sockets = []
try:
    for port in (18083, 18084, 18085):
        probe = socket.socket()
        probe.bind(("0.0.0.0", port))
        sockets.append(probe)
finally:
    for probe in sockets:
        probe.close()
print("sample upstream ports are free")
PY

For ADC, confirm label-selector support:
```
adc --version
adc sync --help | grep -- '--label-selector'
```
ADC resolves --gateway-group by display name; the compatibility Admin API uses the group's UUID.

Check all fixed Admin API IDs before using PUT:

for id in health-active-service-id health-passive-service-id; do
  curl -ksS --connect-timeout 3 --max-time 10 -w "\nservice ${id}: %{http_code}\n" "${API7_ADMIN_URL}/apisix/admin/services/${id}?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -H "X-API-KEY: ${API7_ADMIN_KEY}"
done
for id in health-active-route-id health-passive-route-id; do
  curl -ksS --connect-timeout 3 --max-time 10 -w "\nroute ${id}: %{http_code}\n" "${API7_ADMIN_URL}/apisix/admin/routes/${id}?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -H "X-API-KEY: ${API7_ADMIN_KEY}"
done

Expected: every resource is absent. A 200 response containing a resource means the ID is in use; stop or change every occurrence because PUT replaces that ID. Admin URL IDs deliberately differ from names. ADC names are not backend IDs; continue only if its diff creates exactly this guide's labeled resources.

Steps

Step 1: Start controllable sample upstreams

Create an HTTP server whose /health endpoint always returns 200, while /demo reads its status from a mode file:

cat > health-upstream.py <<'PY'
import os
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

port = int(os.environ["PORT"])
mode_file = os.environ["MODE_FILE"]

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        status = 200 if self.path == "/health" else int(open(mode_file, encoding="utf-8").read().strip())
        body = ("upstream=%s status=%s\n" % (port, status)).encode()
        self.send_response(status)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        pass

ThreadingHTTPServer(("0.0.0.0", port), Handler).serve_forever()
PY

printf '200\n' > health-1.mode
printf '200\n' > health-2.mode
printf '200\n' > health-passive.mode
PORT=18083 MODE_FILE="$(pwd)/health-1.mode" python3 health-upstream.py > health-1.log 2>&1 & echo $! > health-1.pid
PORT=18084 MODE_FILE="$(pwd)/health-2.mode" python3 health-upstream.py > health-2.log 2>&1 & echo $! > health-2.pid
PORT=18085 MODE_FILE="$(pwd)/health-passive.mode" python3 health-upstream.py > health-passive.log 2>&1 & echo $! > health-passive.pid
for port in 18083 18084 18085; do curl -sS --connect-timeout 3 --max-time 5 -o /dev/null -w "${port}: %{http_code}\n" "http://127.0.0.1:${port}/health"; done

Expected: all three direct checks report 200.

Step 2: Create services and routes with the Admin API

Choose this step or step 3; do not apply both.

jq -n --arg host "${HEALTH_UPSTREAM_HOST}" '{name:"health-active-service",labels:{guide:"health-check"},upstream:{name:"default",type:"roundrobin",scheme:"http",nodes:[{host:$host,port:18083,weight:100},{host:$host,port:18084,weight:100}],checks:{active:{type:"http",http_path:"/health",timeout:1,concurrency:2,healthy:{interval:1,successes:1,http_statuses:[200]},unhealthy:{interval:1,http_failures:2,tcp_failures:2,timeouts:2,http_statuses:[429,500,502,503,504]}}}}}' \
  | curl -ksS --connect-timeout 3 --max-time 15 "${API7_ADMIN_URL}/apisix/admin/services/health-active-service-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -X PUT -H "X-API-KEY: ${API7_ADMIN_KEY}" -H "Content-Type: application/json" --data-binary @-
curl -ksS --connect-timeout 3 --max-time 15 "${API7_ADMIN_URL}/apisix/admin/routes/health-active-route-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -X PUT -H "X-API-KEY: ${API7_ADMIN_KEY}" -H "Content-Type: application/json" \
  -d '{"name":"health-active-route","labels":{"guide":"health-check"},"paths":["/health/active"],"methods":["GET"],"service_id":"health-active-service-id","plugins":{"proxy-rewrite":{"uri":"/demo"}}}'

jq -n --arg host "${HEALTH_UPSTREAM_HOST}" '{name:"health-passive-service",labels:{guide:"health-check"},upstream:{name:"default",type:"roundrobin",scheme:"http",nodes:[{host:$host,port:18085,weight:100}],checks:{active:{type:"http",http_path:"/health",timeout:1,concurrency:1,healthy:{interval:3600,successes:1,http_statuses:[200]},unhealthy:{interval:5,http_failures:2,tcp_failures:2,timeouts:2,http_statuses:[429,500,502,503,504]}},passive:{type:"http",healthy:{successes:1,http_statuses:[200]},unhealthy:{http_failures:3,tcp_failures:2,timeouts:2,http_statuses:[500,502,503,504]}}}}}' \
  | curl -ksS --connect-timeout 3 --max-time 15 "${API7_ADMIN_URL}/apisix/admin/services/health-passive-service-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -X PUT -H "X-API-KEY: ${API7_ADMIN_KEY}" -H "Content-Type: application/json" --data-binary @-
curl -ksS --connect-timeout 3 --max-time 15 "${API7_ADMIN_URL}/apisix/admin/routes/health-passive-route-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -X PUT -H "X-API-KEY: ${API7_ADMIN_KEY}" -H "Content-Type: application/json" \
  -d '{"name":"health-passive-route","labels":{"guide":"health-check"},"paths":["/health/passive"],"methods":["GET"],"service_id":"health-passive-service-id","plugins":{"proxy-rewrite":{"uri":"/demo"}}}'

Active intervals apply according to current state: healthy targets are probed every second in the first service, while unhealthy targets are probed every second. Failure counters are separate; two HTTP failures, TCP failures, or timeouts each reach their corresponding threshold. Passive checks have no interval because proxied traffic supplies their observations. Passive checks require active checks in the upstream schema.

Step 3: Apply one complete desired state with ADC

Skip this alternative if you completed step 2. Save both services and routes in one template:

health-check-adc.yaml.tmpl

services:
  - name: health-active-service
    labels: {guide: health-check}
    upstream:
      name: default
      scheme: http
      nodes:
        - {host: "${HEALTH_UPSTREAM_HOST}", port: 18083, weight: 100}
        - {host: "${HEALTH_UPSTREAM_HOST}", port: 18084, weight: 100}
      checks:
        active:
          type: http
          http_path: /health
          timeout: 1
          concurrency: 2
          healthy: {interval: 1, successes: 1, http_statuses: [200]}
          unhealthy: {interval: 1, http_failures: 2, tcp_failures: 2, timeouts: 2, http_statuses: [429, 500, 502, 503, 504]}
    routes:
      - name: health-active-route
        labels: {guide: health-check}
        uris: [/health/active]
        methods: [GET]
        plugins:
          proxy-rewrite: {uri: /demo}
  - name: health-passive-service
    labels: {guide: health-check}
    upstream:
      name: default
      scheme: http
      nodes:
        - {host: "${HEALTH_UPSTREAM_HOST}", port: 18085, weight: 100}
      checks:
        active:
          type: http
          http_path: /health
          timeout: 1
          concurrency: 1
          healthy: {interval: 3600, successes: 1, http_statuses: [200]}
          unhealthy: {interval: 5, http_failures: 2, tcp_failures: 2, timeouts: 2, http_statuses: [429, 500, 502, 503, 504]}
        passive:
          type: http
          healthy: {successes: 1, http_statuses: [200]}
          unhealthy: {http_failures: 3, tcp_failures: 2, timeouts: 2, http_statuses: [500, 502, 503, 504]}
    routes:
      - name: health-passive-route
        labels: {guide: health-check}
        uris: [/health/passive]
        methods: [GET]
        plugins:
          proxy-rewrite: {uri: /demo}

Render only the gateway-reachable upstream host, then lint, validate, inspect, and apply the selected state:

envsubst '${HEALTH_UPSTREAM_HOST}' < health-check-adc.yaml.tmpl > health-check-adc.yaml
adc lint -f health-check-adc.yaml
adc validate -f health-check-adc.yaml --backend api7ee --server "${API7_ADMIN_URL}" --token "${API7_ADMIN_KEY}" --gateway-group "${API7_GATEWAY_GROUP_NAME}" --tls-skip-verify --label-selector guide=health-check
adc diff -f health-check-adc.yaml --backend api7ee --server "${API7_ADMIN_URL}" --token "${API7_ADMIN_KEY}" --gateway-group "${API7_GATEWAY_GROUP_NAME}" --tls-skip-verify --label-selector guide=health-check
adc sync -f health-check-adc.yaml --backend api7ee --server "${API7_ADMIN_URL}" --token "${API7_ADMIN_KEY}" --gateway-group "${API7_GATEWAY_GROUP_NAME}" --tls-skip-verify --label-selector guide=health-check

Expected: validation succeeds and the reviewed diff affects only the two labeled services and their two routes.

Validate

Resolve each service's actual backend ID by label and display name. This is required for ADC because its name is not the Control API ID:

ACTIVE_SERVICE_ID=$(curl -ksS --connect-timeout 3 --max-time 15 "${API7_ADMIN_URL}/apisix/admin/services?gateway_group_id=${API7_GATEWAY_GROUP_ID}&search=health-active-service&page=1&page_size=100" -H "X-API-KEY: ${API7_ADMIN_KEY}" \
  | jq -r '[.list[] | select(.name == "health-active-service" and .labels.guide == "health-check") | .id] | if length == 1 then .[0] else empty end')
PASSIVE_SERVICE_ID=$(curl -ksS --connect-timeout 3 --max-time 15 "${API7_ADMIN_URL}/apisix/admin/services?gateway_group_id=${API7_GATEWAY_GROUP_ID}&search=health-passive-service&page=1&page_size=100" -H "X-API-KEY: ${API7_ADMIN_KEY}" \
  | jq -r '[.list[] | select(.name == "health-passive-service" and .labels.guide == "health-check") | .id] | if length == 1 then .[0] else empty end')
test -n "${ACTIVE_SERVICE_ID}" && test -n "${PASSIVE_SERVICE_ID}"

Prime both checkers, then require their initial target counts and healthy states within ten seconds:

curl -sS --connect-timeout 3 --max-time 10 -o /dev/null "${GATEWAY_URL}/health/active"
curl -sS --connect-timeout 3 --max-time 10 -o /dev/null "${GATEWAY_URL}/health/passive"
for attempt in {1..10}; do
  ACTIVE_CHECKERS=$(curl -sS --connect-timeout 3 --max-time 5 "${CONTROL_API_URL}/v1/healthcheck/services/${ACTIVE_SERVICE_ID}/checkers")
  PASSIVE_CHECKERS=$(curl -sS --connect-timeout 3 --max-time 5 "${CONTROL_API_URL}/v1/healthcheck/services/${PASSIVE_SERVICE_ID}/checkers")
  jq -e 'length == 1 and ([.[0].nodes[] | select(.status == "healthy")] | length == 2)' <<<"${ACTIVE_CHECKERS}" >/dev/null \
    && jq -e 'length == 1 and (.[0].nodes[0].status == "healthy")' <<<"${PASSIVE_CHECKERS}" >/dev/null && break
  sleep 1
done
jq -e 'length == 1 and ([.[0].nodes[] | select(.status == "healthy")] | length == 2)' <<<"${ACTIVE_CHECKERS}"
jq -e 'length == 1 and (.[0].nodes[0].status == "healthy")' <<<"${PASSIVE_CHECKERS}"

Stop active node 18083, require that exact port to become unhealthy, and verify five requests still succeed through node 18084:

kill "$(cat health-1.pid)"
for attempt in {1..15}; do
  ACTIVE_CHECKERS=$(curl -sS --connect-timeout 3 --max-time 5 "${CONTROL_API_URL}/v1/healthcheck/services/${ACTIVE_SERVICE_ID}/checkers")
  jq -e 'any(.[0].nodes[]; .port == 18083 and .status == "unhealthy")' <<<"${ACTIVE_CHECKERS}" >/dev/null && break
  sleep 1
done
jq -e 'any(.[0].nodes[]; .port == 18083 and .status == "unhealthy") and any(.[0].nodes[]; .port == 18084 and .status == "healthy")' <<<"${ACTIVE_CHECKERS}"
for request in {1..5}; do test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${GATEWAY_URL}/health/active")" = 200; done

Restart that node and require active recovery:

PORT=18083 MODE_FILE="$(pwd)/health-1.mode" python3 health-upstream.py > health-1.log 2>&1 & echo $! > health-1.pid
for attempt in {1..15}; do
  ACTIVE_CHECKERS=$(curl -sS --connect-timeout 3 --max-time 5 "${CONTROL_API_URL}/v1/healthcheck/services/${ACTIVE_SERVICE_ID}/checkers")
  jq -e 'any(.[0].nodes[]; .port == 18083 and .status == "healthy")' <<<"${ACTIVE_CHECKERS}" >/dev/null && break
  sleep 1
done
jq -e 'any(.[0].nodes[]; .port == 18083 and .status == "healthy")' <<<"${ACTIVE_CHECKERS}"

Make the passive upstream return a configured unhealthy status, send enough real traffic to cross the threshold, and require an unhealthy observation:

printf '503\n' > health-passive.mode
PASSIVE_CODES=()
for request in {1..5}; do
  PASSIVE_CODES+=("$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${GATEWAY_URL}/health/passive")")
done
test "${PASSIVE_CODES[*]}" = "503 503 503 503 503"
for attempt in {1..10}; do
  PASSIVE_CHECKERS=$(curl -sS --connect-timeout 3 --max-time 5 "${CONTROL_API_URL}/v1/healthcheck/services/${PASSIVE_SERVICE_ID}/checkers")
  jq -e 'any(.[0].nodes[]; .port == 18085 and .status == "unhealthy")' <<<"${PASSIVE_CHECKERS}" >/dev/null && break
  sleep 1
done
jq -e 'any(.[0].nodes[]; .port == 18085 and .status == "unhealthy")' <<<"${PASSIVE_CHECKERS}"

The single passive node can still receive requests after it is unhealthy because the all-unhealthy balancer fallback uses the configured set. Restore /demo without sending client traffic; the required active checker should recover it on its five-second unhealthy interval:

printf '200\n' > health-passive.mode
for attempt in {1..15}; do
  PASSIVE_CHECKERS=$(curl -sS --connect-timeout 3 --max-time 5 "${CONTROL_API_URL}/v1/healthcheck/services/${PASSIVE_SERVICE_ID}/checkers")
  jq -e 'any(.[0].nodes[]; .port == 18085 and .status == "healthy")' <<<"${PASSIVE_CHECKERS}" >/dev/null && break
  sleep 1
done
jq -e 'any(.[0].nodes[]; .port == 18085 and .status == "healthy")' <<<"${PASSIVE_CHECKERS}"

Roll back / Clean up

For Admin API resources, remove routes before services:

for id in health-passive-route-id health-active-route-id; do
  curl -ksS --connect-timeout 3 --max-time 15 -X DELETE "${API7_ADMIN_URL}/apisix/admin/routes/${id}?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -H "X-API-KEY: ${API7_ADMIN_KEY}"
done
for id in health-passive-service-id health-active-service-id; do
  curl -ksS --connect-timeout 3 --max-time 15 -X DELETE "${API7_ADMIN_URL}/apisix/admin/services/${id}?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -H "X-API-KEY: ${API7_ADMIN_KEY}"
done

For ADC, review and apply an empty selected state:

health-check-cleanup.yaml

services: []

adc lint -f health-check-cleanup.yaml
adc diff -f health-check-cleanup.yaml --backend api7ee --server "${API7_ADMIN_URL}" --token "${API7_ADMIN_KEY}" --gateway-group "${API7_GATEWAY_GROUP_NAME}" --tls-skip-verify --label-selector guide=health-check
adc sync -f health-check-cleanup.yaml --backend api7ee --server "${API7_ADMIN_URL}" --token "${API7_ADMIN_KEY}" --gateway-group "${API7_GATEWAY_GROUP_NAME}" --tls-skip-verify --label-selector guide=health-check

Stop the local upstreams and remove their files:

for file in health-1.pid health-2.pid health-passive.pid; do if test -f "${file}" && kill -0 "$(cat "${file}")" 2>/dev/null; then kill "$(cat "${file}")"; fi; done
rm -f health-upstream.py health-1.mode health-2.mode health-passive.mode health-1.pid health-2.pid health-passive.pid health-1.log health-2.log health-passive.log health-check-adc.yaml.tmpl health-check-adc.yaml health-check-cleanup.yaml

Troubleshoot

The Control API returns an empty nodes array: send one request through the route to create its checker targets, then repeat the bounded poll.
The Control API returns 404: query the management API for exactly one service with the expected label and name; do not put an ADC display name in the Control API URL.
A stopped node remains healthy: verify that healthy.interval, the relevant TCP/timeout threshold, and the poll deadline agree, and test reachability from the gateway network.
Passive 503 responses do not change state: passive checks require real traffic and count only statuses listed in passive.unhealthy.http_statuses. Confirm requests reached port 18085.
An unhealthy node still receives traffic: when every configured node is unhealthy, the current balancer falls back to all configured nodes. Keep at least one independently healthy target if unhealthy-node exclusion is required.
ADC proposes unrelated changes: stop and verify the gateway-group display name, guide=health-check selector, labels, rendered host, and complete desired-state file.