Health Checks

AISIX exposes separate health and status endpoints for process, traffic, configuration, and model state. The same endpoints apply to the open-source AISIX gateway and to AISIX gateways connected to AISIX Cloud. Use the endpoint that matches the condition you want to monitor. To verify the complete caller-to-provider path, send a test request through the gateway.

These endpoints do not require authentication. The metrics/status endpoints are available when Prometheus metrics are enabled, as they are by default. Keep that listener private to monitoring and operations systems.

Proxy Liveness

Use /livez for process liveness:

curl -i "http://127.0.0.1:3000/livez"

A healthy process returns 200 OK with an ok body.

A draining instance answers 200 as well. Liveness decides whether to restart the instance, and draining is deliberate work: an instance that has been told to shut down is finishing the requests it already accepted, and restarting it would kill exactly those. It is /readyz that reports the drain, so that traffic is withdrawn without the instance being replaced under its own in-flight work.

Append ?verbose=1 when investigating manually. Do not make an automated probe depend on the verbose response body.

Liveness is intentionally narrow. It does not prove that the gateway has loaded configuration, that a model is available, or that a provider request can succeed.

Traffic Readiness

Use /readyz to decide whether an instance should receive traffic:

curl -i "http://127.0.0.1:3000/readyz"

It returns 503 Service Unavailable while the instance is draining or before it applies its first configuration. After a valid configuration is available, the gateway remains ready for as long as it can serve that configuration.

A control-plane or configuration-store interruption does not make a running gateway unready merely because no recent update arrived. A stalled source commonly affects every instance, so withdrawing them would remove the traffic path rather than shift traffic to a healthy peer. Monitor configuration freshness separately.

Append ?verbose=1 when investigating why an instance is not ready. Do not make an automated probe depend on the verbose response body.

For Kubernetes, point the liveness and readiness probes at /livez and /readyz on the proxy listener. Give the gateway enough termination time to drain in-flight and streaming requests.

Shutdown and Draining

A load balancer learns that an instance is withdrawing on its next health check, not the moment the instance decides to. Between those two points it keeps routing new connections. Closing the listener as soon as the shutdown signal arrives would refuse every connection routed inside that interval. Callers then see gateway errors during an ordinary rolling update or scale-down.

So the gateway separates the two events. On SIGTERM or SIGINT it:

1. Answers /readyz with 503 Service Unavailable immediately, so the next health check withdraws it. /livez stays 200: the process is healthy and must not be restarted while it drains.
2. Keeps accepting new connections for at least shutdown.min_drain_secs, which defaults to 30 seconds.
3. Adds Connection: close to every HTTP/1.1 response, so a client that pools connections retires them as it uses them instead of holding idle ones open. HTTP/2 forbids that header, so an HTTP/2 client is sent a GOAWAY frame instead, at the moment the drain starts. GOAWAY asks the peer to finish the streams it has already opened and to start no new ones; it does not close the connection or interrupt anything in flight.
4. After the minimum window, waits without a deadline of its own for the in-flight count to reach zero, then stops accepting new connections and exits.

Set min_drain_secs above the detection latency of whatever load-balances the instance. A Kubernetes readiness probe needs periodSeconds x failureThreshold. An external load balancer needs its own check interval multiplied by its retry count. Setting it too low closes the listener while traffic is still arriving. Setting it too high only delays the exit.

shutdown:
  min_drain_secs: 30

The window is a minimum, not a deadline. After it elapses the gateway still waits for the in-flight count to reach zero, so a load balancer slower than configured cannot make it close under live traffic. 0 drops the window entirely and is only correct when nothing routes to the instance by health check.

Because the wait for in-flight work to reach zero is unbounded, the deployment platform sets the real limit on the whole sequence: terminationGracePeriodSeconds in Kubernetes, TimeoutStopSec under systemd. Size it above shutdown.min_drain_secs plus your longest request or stream, with operational margin. Add the duration of any preStop hook because it consumes the same Kubernetes termination budget.

Watch a Drain in the Logs

The drain logs its own progress, because the record a request normally leaves is written when the request completes — and a request still running when the platform's grace period expires never gets there. These lines are what separate work the gateway had already taken on from traffic still being routed at it after the signal:

The two counts answer different questions and neither is derivable from the other. in_flight counts requests being served, and a streaming response holds its slot for as long as bytes may still flow. open_connections counts downstream connections open on the proxy listener, including pooled ones sitting idle with no request on them. So a drain that will not end while in_flight stays up is the gateway finishing work it already accepted; one where in_flight has reached zero while open_connections stays up is clients holding connections they are not using; one where open_connections keeps climbing is something still routing new connections here.

Together, the two per-event lines explain a request that failed during a rolling update. An arrival line with a matching accept line for the same peer means the connection itself was routed here after the gateway had already asked to be withdrawn — a load balancer that has not caught up, which is the case min_drain_secs exists to absorb. An arrival line with no matching accept line means the connection predates the signal and the client reused it out of its pool, which is what the Connection: close header retires.

The accept line cannot tell your callers from your platform, though. /livez and /readyz are served on the proxy listener too, and probes keep arriving on their own connections throughout the drain, so they raise open_connections and produce accept lines like any other connection. The arrival line is where that judgement is made: it knows the path, and deliberately excludes those two endpoints so the handful of real arrivals is not buried under a probe every few seconds. Read the accept line for what the arrival line cannot see — a connection opened and never used.

peer and downstream_request_id are the connection identity described in Metrics and Logs. Matching them against the records of whatever fronts the gateway is how a gateway line and a balancer line are shown to be about the same connection.

Configuration State

The metrics/status listener provides two configuration checks.

GET /status/ready is a configuration-only startup gate:

• 503 Service Unavailable before the first valid configuration is applied;
• 200 OK after a valid configuration is available, including while AISIX serves a last-known-good snapshot after a later update fails.

GET /status/config explains what AISIX observed and applied:

curl -sS "http://127.0.0.1:9090/status/config"

Use it when a resource update does not appear in proxy behavior. Compare the source and applied state, then inspect rejected resources and the latest load failure. See Configuration Status for the complete response fields, state meanings, Prometheus metrics, and alert examples.

Configuration status does not replace caller-facing verification. After the expected snapshot applies, query GET /v1/models and send the request whose behavior changed. See Configuration Propagation.

Per-Model Runtime Health

Use GET /status/models when the gateway is ready but routing avoids a model or reports no eligible target:

curl -sS "http://127.0.0.1:9090/status/models"

Each configured model reports one of these high-level states:

• healthy: available for routing;
• cooldown: temporarily removed after recent upstream failures; only a model that enables cooldown enters this state;
• unhealthy: excluded after failed background model checks;
• not_applicable: a virtual model whose availability derives from its targets.

The status view helps identify cooldown and background-check failures, but it does not validate caller access or provider credentials. A model can report healthy while a caller API key, provider key, or upstream response still prevents a request from succeeding.

See Configuration Status for the full response fields.

Combined Health Signals

Work from the earliest failing layer:

Finish with the same path the application uses:

AISIX_API_KEY="YOUR_CALLER_API_KEY"

curl -sS "http://127.0.0.1:3000/v1/models" \
  -H "Authorization: Bearer ${AISIX_API_KEY}"

Then send a real request through the required endpoint and model. That final probe verifies conditions that runtime health endpoints deliberately do not evaluate.

Next Steps

Use Troubleshooting to narrow a failed health or request-path check to configuration, caller policy, AISIX Cloud projection, or the upstream provider.