Configure Upstream Health Checks

Configure active and passive upstream health checks in Apache APISIX and verify exact node state through the Control API.

Goal

Create two isolated routes that demonstrate active failure detection, failover, and recovery, plus passive HTTP failure counting followed by active recovery. You will verify exact node ports and states through the Control API instead of relying on fixed delays or successful proxy responses alone.

When at least one node is healthy, APISIX selects from the healthy subset. If every configured node is marked unhealthy, the balancer falls back to the complete configured node set. Health checks therefore do not guarantee that all-unhealthy traffic is rejected; an actually unavailable node can still produce 502 or 503 during that fallback.

Before you begin

You have an APISIX installation and cURL. Get APISIX starts APISIX in Docker with proxy, Admin API, and Control API ports exposed.
Python 3 and jq are installed on the machine that will host the sample upstreams.
APISIX can reach that machine on TCP ports 18083, 18084, and 18085. For Docker Desktop, the gateway-reachable host is commonly host.docker.internal.
Keep the Control API on an operator-only network. Do not expose it through a public data-plane route.

Set the variables used in this guide:

export APISIX_ADMIN_URL="http://127.0.0.1:9180"
export GATEWAY_URL="http://127.0.0.1:9080"
export CONTROL_API_URL="http://127.0.0.1:9090"
read -rp "Host or DNS name APISIX uses to reach this machine: " HEALTH_UPSTREAM_HOST && export HEALTH_UPSTREAM_HOST
read -rsp "Admin API key: " APISIX_ADMIN_KEY && export APISIX_ADMIN_KEY && echo

Confirm all three APISIX endpoints are reachable:

test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${APISIX_ADMIN_URL}/apisix/admin/routes" -H "X-API-KEY: ${APISIX_ADMIN_KEY}")" = 200
test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${GATEWAY_URL}")" != 000
test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${CONTROL_API_URL}/v1/healthcheck")" = 200

Protect existing resources and local files before using this guide's fixed names:

for id in health-active-route health-passive-route; do
  test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${APISIX_ADMIN_URL}/apisix/admin/routes/${id}" -H "X-API-KEY: ${APISIX_ADMIN_KEY}")" = 404 || { echo "Stop: route ${id} already exists"; exit 1; }
done
for file in health-upstream.py health-1.mode health-2.mode health-passive.mode health-1.pid health-2.pid health-passive.pid health-1.log health-2.log health-passive.log; do
  test ! -e "${file}" || { echo "Stop: ${file} already exists"; exit 1; }
done
python3 - <<'PY'
import socket

sockets = []
try:
    for port in (18083, 18084, 18085):
        probe = socket.socket()
        probe.bind(("0.0.0.0", port))
        sockets.append(probe)
finally:
    for probe in sockets:
        probe.close()
print("sample upstream ports are free")
PY

Steps

Step 1: Start controllable upstreams

Create a server whose /health endpoint always returns 200. Other paths return the status in that server's mode file, making failure and recovery deterministic:

cat > health-upstream.py <<'PY'
import os
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

port = int(os.environ["PORT"])
mode_file = os.environ["MODE_FILE"]

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        status = 200 if self.path == "/health" else int(open(mode_file, encoding="utf-8").read().strip())
        body = ("upstream=%s status=%s\n" % (port, status)).encode()
        self.send_response(status)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        pass

ThreadingHTTPServer(("0.0.0.0", port), Handler).serve_forever()
PY

printf '200\n' > health-1.mode
printf '200\n' > health-2.mode
printf '200\n' > health-passive.mode
PORT=18083 MODE_FILE="$(pwd)/health-1.mode" python3 health-upstream.py > health-1.log 2>&1 & echo $! > health-1.pid
PORT=18084 MODE_FILE="$(pwd)/health-2.mode" python3 health-upstream.py > health-2.log 2>&1 & echo $! > health-2.pid
PORT=18085 MODE_FILE="$(pwd)/health-passive.mode" python3 health-upstream.py > health-passive.log 2>&1 & echo $! > health-passive.pid
for port in 18083 18084 18085; do
  test "$(curl -sS --connect-timeout 3 --max-time 5 -o /dev/null -w '%{http_code}' "http://127.0.0.1:${port}/health")" = 200
done

Step 2: Configure active health checks

This route probes /health every second. Two consecutive connection failures, HTTP failures, or timeouts mark only the affected node unhealthy.

jq -n --arg host "${HEALTH_UPSTREAM_HOST}" '{
  uri: "/health/active",
  plugins: {"proxy-rewrite": {uri: "/demo"}},
  upstream: {
    type: "roundrobin",
    nodes: [
      {host: $host, port: 18083, weight: 1},
      {host: $host, port: 18084, weight: 1}
    ],
    checks: {
      active: {
        type: "http",
        http_path: "/health",
        timeout: 1,
        concurrency: 2,
        healthy: {interval: 1, successes: 1, http_statuses: [200]},
        unhealthy: {
          interval: 1,
          http_failures: 2,
          tcp_failures: 2,
          timeouts: 2,
          http_statuses: [429, 500, 502, 503, 504]
        }
      }
    }
  }
}' | curl -sS --connect-timeout 3 --max-time 15 "${APISIX_ADMIN_URL}/apisix/admin/routes/health-active-route" -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" -H "Content-Type: application/json" --data-binary @-

Step 3: Create the passive-check route

Passive checks learn from proxied requests and have no interval. APISIX requires an active check alongside a passive check so that an unhealthy node can be probed and restored. Here, three proxied 503 responses mark the node unhealthy; once unhealthy, the active checker probes /demo every second until it recovers.

jq -n --arg host "${HEALTH_UPSTREAM_HOST}" '{
  uri: "/health/passive",
  plugins: {"proxy-rewrite": {uri: "/demo"}},
  upstream: {
    type: "roundrobin",
    nodes: [{host: $host, port: 18085, weight: 1}],
    checks: {
      active: {
        type: "http",
        http_path: "/demo",
        timeout: 1,
        concurrency: 1,
        healthy: {interval: 3600, successes: 1, http_statuses: [200]},
        unhealthy: {
          interval: 1,
          http_failures: 2,
          tcp_failures: 2,
          timeouts: 2,
          http_statuses: [429, 500, 502, 503, 504]
        }
      },
      passive: {
        type: "http",
        healthy: {successes: 1, http_statuses: [200]},
        unhealthy: {
          http_failures: 3,
          tcp_failures: 2,
          timeouts: 2,
          http_statuses: [500, 502, 503, 504]
        }
      }
    }
  }
}' | curl -sS --connect-timeout 3 --max-time 15 "${APISIX_ADMIN_URL}/apisix/admin/routes/health-passive-route" -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" -H "Content-Type: application/json" --data-binary @-

The active failure counters are independent: two HTTP failures, two TCP failures, or two timeouts reach their respective thresholds. They are not added together.

Validate

Prime both health checkers with one request, then require the expected initial states within ten seconds. A new checker can report an empty nodes array until its first request initializes the targets.

curl -sS --connect-timeout 3 --max-time 10 -o /dev/null "${GATEWAY_URL}/health/active"
curl -sS --connect-timeout 3 --max-time 10 -o /dev/null "${GATEWAY_URL}/health/passive"
for attempt in {1..10}; do
  ACTIVE_STATE=$(curl -sS --connect-timeout 3 --max-time 5 "${CONTROL_API_URL}/v1/healthcheck/routes/health-active-route")
  PASSIVE_STATE=$(curl -sS --connect-timeout 3 --max-time 5 "${CONTROL_API_URL}/v1/healthcheck/routes/health-passive-route")
  jq -e '([.nodes[] | select(.status == "healthy")] | length == 2)' <<<"${ACTIVE_STATE}" >/dev/null \
    && jq -e '(.nodes | length == 1) and (.nodes[0].status == "healthy")' <<<"${PASSIVE_STATE}" >/dev/null && break
  sleep 1
done
jq -e '([.nodes[] | select(.status == "healthy")] | length == 2)' <<<"${ACTIVE_STATE}"
jq -e '(.nodes | length == 1) and (.nodes[0].status == "healthy")' <<<"${PASSIVE_STATE}"

Stop active node 18083, require that exact port to become unhealthy, and prove that traffic uses the remaining healthy node:

kill "$(cat health-1.pid)"
for attempt in {1..15}; do
  ACTIVE_STATE=$(curl -sS --connect-timeout 3 --max-time 5 "${CONTROL_API_URL}/v1/healthcheck/routes/health-active-route")
  jq -e 'any(.nodes[]; .port == 18083 and .status == "unhealthy")' <<<"${ACTIVE_STATE}" >/dev/null && break
  sleep 1
done
jq -e 'any(.nodes[]; .port == 18083 and .status == "unhealthy") and any(.nodes[]; .port == 18084 and .status == "healthy")' <<<"${ACTIVE_STATE}"
for request in {1..5}; do
  test "$(curl -sS --connect-timeout 3 --max-time 10 "${GATEWAY_URL}/health/active")" = "upstream=18084 status=200"
done

Restart node 18083 and require active recovery:

PORT=18083 MODE_FILE="$(pwd)/health-1.mode" python3 health-upstream.py > health-1.log 2>&1 & echo $! > health-1.pid
for attempt in {1..15}; do
  ACTIVE_STATE=$(curl -sS --connect-timeout 3 --max-time 5 "${CONTROL_API_URL}/v1/healthcheck/routes/health-active-route")
  jq -e 'any(.nodes[]; .port == 18083 and .status == "healthy")' <<<"${ACTIVE_STATE}" >/dev/null && break
  sleep 1
done
jq -e 'any(.nodes[]; .port == 18083 and .status == "healthy")' <<<"${ACTIVE_STATE}"

Make the passive upstream return 503, send exactly the configured number of failing requests, and require the exact node to become unhealthy:

printf '503\n' > health-passive.mode
PASSIVE_CODES=()
for request in {1..3}; do
  PASSIVE_CODES+=("$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${GATEWAY_URL}/health/passive")")
done
test "${PASSIVE_CODES[*]}" = "503 503 503"
for attempt in {1..5}; do
  PASSIVE_STATE=$(curl -sS --connect-timeout 3 --max-time 5 "${CONTROL_API_URL}/v1/healthcheck/routes/health-passive-route")
  jq -e 'any(.nodes[]; .port == 18085 and .status == "unhealthy")' <<<"${PASSIVE_STATE}" >/dev/null && break
  sleep 1
done
jq -e 'any(.nodes[]; .port == 18085 and .status == "unhealthy")' <<<"${PASSIVE_STATE}"

The single passive node can still receive traffic while unhealthy because all-unhealthy fallback uses the configured set. Restore the response without sending more client traffic; the required active checker should recover it:

printf '200\n' > health-passive.mode
for attempt in {1..15}; do
  PASSIVE_STATE=$(curl -sS --connect-timeout 3 --max-time 5 "${CONTROL_API_URL}/v1/healthcheck/routes/health-passive-route")
  jq -e 'any(.nodes[]; .port == 18085 and .status == "healthy")' <<<"${PASSIVE_STATE}" >/dev/null && break
  sleep 1
done
jq -e 'any(.nodes[]; .port == 18085 and .status == "healthy")' <<<"${PASSIVE_STATE}"

Roll back / Clean up

Remove the routes, stop any sample process still running, and remove only this guide's local files:

for id in health-passive-route health-active-route; do
  curl -sS --connect-timeout 3 --max-time 15 -X DELETE "${APISIX_ADMIN_URL}/apisix/admin/routes/${id}" -H "X-API-KEY: ${APISIX_ADMIN_KEY}"
done
for file in health-1.pid health-2.pid health-passive.pid; do
  if test -f "${file}" && kill -0 "$(cat "${file}")" 2>/dev/null; then kill "$(cat "${file}")"; fi
done
rm -f health-upstream.py health-1.mode health-2.mode health-passive.mode health-1.pid health-2.pid health-passive.pid health-1.log health-2.log health-passive.log

Troubleshoot

Symptom	Cause	Fix
The Control API returns an empty nodes array.	No request has initialized the checker targets yet.	Send one request through that route, then repeat the bounded poll.
A node never appears or changes state.	APISIX cannot reach the host and port from its own network namespace, or the deadline does not match the interval and threshold.	Test the configured host and port from the APISIX container or host and verify the check settings.
Passive 503 responses do not change state.	Passive checks count only real proxied responses whose status is configured as unhealthy.	Confirm /health/passive reaches port 18085 and returns exactly 503 503 503.
An unhealthy node still receives traffic.	Every configured node is unhealthy, so APISIX falls back to the full configured set.	Keep at least one independently healthy target if unhealthy-node exclusion is required.
The Admin API preflight returns 401 or 403.	The key is absent, wrong, or unauthorized.	Set APISIX_ADMIN_KEY from the active deployment before creating anything.