Troubleshooting

Narrow an AISIX failure to startup, configuration, caller policy, the provider path, or AISIX Enterprise projection before changing resources. Work through these checks in order until the failing layer is clear, then use the matching section to decide the next action.

Fast triage

Start with the runtime path before changing configuration.

Check listener health first:

curl -i "http://127.0.0.1:3000/livez"

A refused connection here is a signal in its own right, not a failed command. With etcd or AISIX Enterprise as the resource source, the gateway does not bind the proxy listener until it has applied its first configuration, so an instance that has never reached that source has no proxy port to answer on. See The Gateway Runs but the Proxy Port Refuses Connections.

When Prometheus metrics are enabled, check the latest observed and applied configuration on the private metrics/status listener, which binds as soon as the process starts and therefore answers in that case too:

curl -sSi "http://127.0.0.1:9090/status/ready"
curl -sS "http://127.0.0.1:9090/status/config"

Verify model discovery with the same caller API key the application uses:

AISIX_API_KEY="YOUR_CALLER_API_KEY"

curl -sS "http://127.0.0.1:3000/v1/models" \
  -H "Authorization: Bearer ${AISIX_API_KEY}"

Then send one real request to the endpoint that failed. For example, check the OpenAI-compatible chat path with the caller API key:

curl -sS "http://127.0.0.1:3000/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": "Say hello."
      }
    ]
  }'

Use the response status, error type, and correlation header to choose the next check.

For request correlation, use x-aisix-request-id, which AISIX adds to proxy responses across endpoint families. Successful Chat Completions responses also include x-aisix-call-id. For exact header scope, see Headers and Error Codes.

Verify Configuration Visibility

Use this step when resources were created or updated recently, or when the proxy behaves as if it has older configuration.

For a store-backed gateway, etcd provides the dynamic-resource source. If etcd becomes unavailable after a snapshot has loaded, the gateway can continue using that snapshot, but it cannot receive new or updated dynamic resources until the configuration connection recovers.

The Gateway Runs but the Proxy Port Refuses Connections

Use this step when the process is up — it has not exited and has reported no error — but every request to the proxy port, including /livez, is refused.

A gateway that reads its resources from etcd or from AISIX Enterprise binds the proxy listener only after it has applied its first configuration. Before then the port does not exist, so the failure looks like a process that never started rather than one that is waiting. It is genuinely waiting: it does not exit, does not bind a degraded listener, and does not fall back to serving nothing. The one exception is the boot connection itself being answered and refused — etcd credentials that are wrong, or credentials presented to a cluster with authentication disabled — which ends the boot with that error instead, because waiting cannot repair them. A refusal that arrives after that connection, a user without permission on etcd.prefix included, leaves the gateway in exactly this wait. It binds the listener the moment a read succeeds, and retries a read that fails on an exponential backoff that grows from one second to a sixty-second ceiling. See Startup and the First Configuration for the full contract.

Confirm the diagnosis on the metrics/status listener, which is bound throughout:

curl -sSi "http://127.0.0.1:9090/status/ready"
curl -sS "http://127.0.0.1:9090/status/config"

A 503 with no configuration available, alongside state: never_loaded in /status/config, is this case. The gateway log carries it as well: one waiting for the first configuration before binding the proxy listener line when the wait starts, then proxy listener still not bound: no configuration has been applied yet every ten seconds until it ends.

A gateway that restored a usable snapshot from its on-disk cache, or one configured from a resources file, binds immediately and never reaches this state. See Offline Resilience.

Verify Caller Access and Policy

Use this step when AISIX rejects the request before a provider call, or when model discovery differs between caller API keys.

For exact proxy error envelopes, status codes, and retry headers, see Proxy Errors and Retries.

When rate limits differ across gateway instances, identify the counter backend before changing a policy. Memory counters are local to each process. Redis shares counters, but a runtime Redis failure falls back to process-local enforcement. Outage-time counts are not reconciled into Redis after recovery, so restore Redis and let the active windows roll over before assuming the counters are aligned again.

Verify the Provider Path

Use this step after AISIX authenticates the caller, resolves the model alias, and starts dispatching to the configured provider.

Compare the failing route with the matching provider upstream guide when the problem is provider-specific.

Read a Transport Error

A transport error means the request never completed at the HTTP layer, so there is no upstream status code to interpret. Gateway logs record the cause chain alongside the message, which is what distinguishes the possible faults:

Errors that appear only intermittently against a provider that is otherwise healthy usually point to connection reuse rather than the provider. Lower pool_idle_timeout_secs below the shortest idle timeout on the path and re-check.

The same failure has a mirror image on the inbound side. If a gateway or load balancer in front of AISIX reports intermittent resets or 502s against a healthy AISIX, check whether downstream.idle_timeout_secs is set below that node's own pool idle timeout — the gateway would then be closing connections the node in front still considers usable. Leaving it at 0 (the default) rules this out. See Tune the Downstream Connection Layer.

Attribute an Upstream Error to the Right Hop

When the recorded message is upstream returned HTTP <status>: <body>, the status and body came from whatever answered at api_base — which is the provider only when nothing sits in front of it. Response bodies that mention a proxy's own vocabulary indicate an intermediate hop generated the error rather than the model provider:

• upstream connect error or disconnect/reset before headers with a reset reason, or exceeded request buffer limit while retrying upstream, are emitted by Envoy-based proxies, service meshes, and API gateways.
• A generic HTML error page is emitted by a reverse proxy or load balancer, not by a provider's JSON API.

On modeled routes, the gateway records these bodies for diagnosis but does not return them to the caller. Upstream 5xx responses reach the application as a generic 502 envelope. Passthrough routes instead forward the upstream provider's native status and body. Use the per-attempt records to identify the failing provider_key_id and target model, then compare them with access logs from the upstream endpoint configured in api_base.

Verify AISIX Enterprise Projection

Use this step when control-plane state and live gateway behavior do not match, or when an AISIX gateway cannot receive projected configuration.

Use the relevant feature guide or reference page after identifying the failing layer. Re-run the caller-facing request after each correction to verify the complete path rather than only the component that failed.