API7 Gateway 3.9.19

The gateway has always overwritten X-Forwarded-Proto, X-Forwarded-Host and X-Forwarded-Port and cleared Forwarded for requests from peers it does not trust.

Release Date: 2026-08-26

Upgrade Notes

Upgrade note — access log formats naming $http_x_forwarded_* now record what the client sent

The gateway has always overwritten X-Forwarded-Proto, X-Forwarded-Host and X-Forwarded-Port and cleared Forwarded for requests from peers it does not trust. That work now happens in the NGINX configuration rather than in Lua. What the upstream receives is unchanged, and so is what plugins see: core.request.header and ctx.var.http_x_forwarded_* still return the overwritten values, and ctx.var.original_x_forwarded_* still returns what the client sent.

The NGINX configuration level differs. Naming $http_x_forwarded_proto, $http_x_forwarded_host, $http_x_forwarded_port or $http_forwarded in an access log format, an if, or a map now reads what the client sent, where it previously read the overwritten value. A log format that records $http_x_forwarded_host for auditing therefore starts recording a value the client controls: a request forging X-Forwarded-Host: evil.example.com was logged as the real host before the upgrade and is logged as evil.example.com after it.

Use $scheme, $var_x_forwarded_host and $var_x_forwarded_port to log the overwritten values. $http_x_forwarded_for is not affected. What the client sent is also available in the new $original_x_forwarded_proto, $original_x_forwarded_host, $original_x_forwarded_port, $original_x_forwarded_for and $original_forwarded NGINX variables.

Upgrade note — logging plugins now discard entries once the backlog reaches 8192

The batch processor behind every logging plugin kept undelivered entries in worker memory without a limit unless max_pending_entries was set in the plugin's metadata, so a log server that was slow or unreachable grew the worker's memory with the request rate. The limit now defaults to 8192 entries, and datadog, lago, loggly, sls-logger and the stream subsystem's syslog expose the option for the first time.

A deployment whose log server keeps up is unaffected: the backlog tracks batch_max_size rather than the request rate and stays well under a thousand entries. Once the limit is reached, entries are discarded and reported at most once per second with a running count. If you log request or response bodies larger than a few kilobytes, lower max_pending_entries in the plugin metadata so the backlog fits your memory budget; if you raised batch_max_size, raise this with it.

Upgrade note — OpenID Connect enforces the authorization checks it was configured with

Three checks that could be configured on OpenID Connect (opens in Plugin Hub docs) were not applied in every case, so requests that should have been rejected were let through. Configurations that relied on them are now enforced, and traffic that was passing before may start receiving HTTP 401 or 403.

required_scopes was only applied to tokens validated by introspection. A route using the authorization code flow — the default, with bearer_only unset — never read the scopes granted to the session, so every user who completed the login was authorized. The scopes are now read from the access token, and a session whose granted scopes cannot be determined is denied rather than allowed. Check that your identity provider issues the access token as a JWT carrying scope, or that the ID token carries it, before relying on required_scopes with the authorization code flow.

claim_validator.audience.match_with_client_id compared the aud claim against the client id only when the claim was present, so a token that omitted aud skipped the check. The option now implies that the claim is required, and a token without aud receives HTTP 403.

In bearer_only mode with no valid_issuers configured, the trusted issuer comes from the provider's discovery document; when that fetch failed the gateway logged a warning and verified the token with no issuer constraint at all. It now answers HTTP 401 with error_description="issuer validation unavailable" instead. Configure valid_issuers explicitly if you do not want availability of the discovery endpoint on the request path.

Upgrade note — AI Proxy Multi rejects instances that share a name

AI Proxy Multi (opens in Plugin Hub docs) accepted two instances configured with the same name, which made the instance a request was routed to ambiguous. Such a configuration is now rejected with HTTP 400 at write time.

Configurations already stored keep working but are reported in the Data Plane's configuration compatibility report at error level after the upgrade. Rename the duplicated instances so each name is unique; the report clears once the configuration is rewritten.

Upgrade note — Proxy Cache in-memory entries are fetched again once after the upgrade

The storage key layout of Proxy Cache (opens in Plugin Hub docs) in memory strategy changed to close the variant collision described under Fixes, and the cache version was bumped alongside it. Entries written by an earlier release are therefore not reachable under the new layout: the first request for each key is a MISS and repopulates the cache, and a PURGE for a URL that only has pre-upgrade entries answers HTTP 404. The old entries stay in the shared dictionary until their own TTL expires. Deployments using the default disk strategy are unaffected.

Features

Plugins

Prometheus (opens in Plugin Hub docs)
- Added three metrics for Layer 4 traffic, joining the apisix_stream_connection_total counter that was already exported: apisix_stream_status counting completed sessions by termination status, listening address and upstream node, apisix_stream_active_connections as a gauge of TCP connections and UDP sessions per listening address, and apisix_stream_bandwidth counting bytes proxied by listening address, direction and side. Enable the plugin on the stream route to collect them. The stream_status key can be added to the disabled_labels plugin metadata to drop labels from apisix_stream_status, as status already does for the HTTP counter.
AI Proxy Multi (opens in Plugin Hub docs)
- Added fallback_http_statuses, which lists the HTTP status codes an instance may answer with to make the gateway retry the request against the next instance. Previously a fallback only happened when an instance could not be reached at all, so a provider answering 401 for an expired key or 429 for a quota exhausted on that account was returned to the client instead of being retried elsewhere.

Control Plane

Routes can now match the PURGE method, which is how Proxy Cache (opens in Plugin Hub docs) and GraphQL Proxy Cache (opens in Plugin Hub docs) invalidate a cached entry. The gateway has always accepted it; only the Dashboard's own schema rejected it, so a route serving cache invalidation could not be created through the API or the Dashboard.
Upstream and route timeouts now accept fractional seconds, so 0.5 can be configured where the API previously required a whole number. The Data Plane has always supported sub-second timeouts. The smallest value is 0.001, because anything below a millisecond is truncated.

Data Plane

The X-Forwarded-* headers a request carries are now sanitized by the NGINX configuration instead of by Lua on every request. Plugins see the same values as before, and the values the client sent — already available to Lua as ctx.var.original_x_forwarded_* — are now NGINX variables as well, so a log format can record them. See Upgrade Notes for the one place this changes what is logged.

Fixes

Plugins

GraphQL Limit Count (opens in Plugin Hub docs)
- Fixed issue: A GraphQL document whose fragments spread one another was measured by expanding each spread every time it was referenced, so a small request body could cost an unbounded amount of CPU. A 1.4 KB document with 34 chained fragments held a worker at 100% CPU for over 45 seconds, during which the gateway answered nothing on any route and had to be restarted. Each fragment is now measured once, and a document whose fragment spreads form a cycle is rejected with HTTP 400.
AI Proxy (opens in Plugin Hub docs)
- Fixed issue: When a streaming upstream answered HTTP 200 with Content-Type: text/event-stream and then ended the response without writing anything, the request was neither answered nor terminated, so the client received no HTTP response at all and the access log recorded a 500 with a zero-length body. The gateway now returns HTTP 502.
- Fixed issue: When a streaming upstream failed part way through — the connection dropping or timing out after some of the stream had already reached the client — the gateway still tried to answer HTTP 500 or 504 on a response it had already committed as HTTP 200. It logged attempt to set ngx.status after sending out response headers, returned the broken connection to the keepalive pool, and on AI Proxy Multi (opens in Plugin Hub docs) treated the failure as retryable, sending a second billable request to the next instance after the client had already received part of the answer. The stream now ends where the read failed, with no synthesized terminator, so the client detects the truncation from the missing [DONE], message_stop or response.completed. A read error that arrives before the first byte reaches the client is unchanged: still 504 for a timeout and 500 otherwise, and still eligible for a fallback.
AI Proxy Multi (opens in Plugin Hub docs)
- Fixed issue: When a request was retried against the next instance, it was built from the body the previous attempt had already rewritten, so the previous instance's options — for example its temperature or max_tokens — leaked into the request sent to the fallback and the model name could be overwritten. Each attempt now starts from the client's own request body.
AI AWS Content Moderation (opens in Plugin Hub docs) and AI Aliyun Content Moderation (opens in Plugin Hub docs)
- Fixed issue: When the gateway cut a stream short itself — max_response_bytes or max_stream_duration_ms on AI Proxy (opens in Plugin Hub docs) reaching its limit — these plugins still appended the protocol's completion event to the stream they re-encode and deliver. The client received a well-formed [DONE] and had no way to tell that content had been dropped, so a truncated answer was indistinguishable from a complete one: in one measured case 7 of 9 events were discarded and the terminator was sent anyway. The terminator is no longer synthesized for a stream that was aborted.
Redirect
- Fixed issue: With http_to_https enabled, the X-Forwarded-Proto header was compared case-sensitively, so a client or load balancer sending HTTPS was redirected to the HTTPS URL it had already reached, producing a redirect loop. The comparison is now case-insensitive.
Data Mask (opens in Plugin Hub docs)
- Fixed issue: Request header masking had no effect on a request the gateway answered itself instead of proxying — one rejected by an authentication plugin, a rate limit, or Fault Injection (opens in Plugin Hub docs), for example. Logging plugins running afterwards read the header from the unmodified request, so a credential that was configured to be masked was shipped to the log sink in plaintext. Masking now holds on every path.
- Fixed issue: Removing an element from a JSON array left a hole in it rather than compacting the array, so the elements after the removed one were dropped from the logged body: masking $.items[1] out of ["a","b","c"] logged ["a"] instead of ["a","c"].
Proxy Cache (opens in Plugin Hub docs)
- Fixed issue: In memory strategy, the storage key of a Vary variant was the cache key with the variant signature appended, and the cache key is derived from the request URI. A request crafted to carry another request's variant signature in its own URI therefore produced the same storage key, so it was served that request's cached response and its own response was stored where the next request would look that variant up. Storage keys are now derived so that no crafted request can reproduce another one's key. See Upgrade Notes for what this means for entries cached before the upgrade.
OpenID Connect (opens in Plugin Hub docs)
- Fixed issue: An identity provider that redirected back with error=temporarily_unavailable — which the specification defines as a transient condition — made the gateway answer HTTP 500. The authentication flow is now restarted from the original URL, up to three times per session, and other error codes such as access_denied are still treated as a final answer.
- Fixed issue: required_scopes, claim_validator.audience.match_with_client_id and issuer validation were not applied in every case, letting through requests the configuration should have rejected. See Upgrade Notes.
HMAC Auth (opens in Plugin Hub docs)
- Fixed issue: With hide_credentials enabled, the Authorization header was removed from the request sent upstream but stayed in the cached request headers, so a plugin ordered after HMAC Auth on the same route could still read the signature credential.
UA Restriction (opens in Plugin Hub docs)
- Fixed issue: A request carrying more than one User-Agent header inverted the denylist decision. A client whose user agent was on the denylist was let through by sending the header twice, and a request whose first user agent matched nothing on the list was rejected with HTTP 403. Requests with a single User-Agent header, and the allowlist mode, were not affected.
AWS Lambda (opens in Plugin Hub docs)
- Fixed issue: A request body sent with Transfer-Encoding: chunked was forwarded to the function without being reframed, so the invocation failed and the client received HTTP 503 with failed to process aws-lambda, err: closed in the error log. The body is now forwarded with a Content-Length. The same applies to the other serverless upstream plugins, which share this code path.

Data Plane

Fixed issue: When resolving an upstream host that is a CNAME, the resolver's answer was only collapsed onto the queried name when the last record of the answer happened to be of the requested type. A resolver that appends an EDNS(0) OPT record, or returns the answer section out of chain order, defeated that test, so the resolved record was recorded under the canonical name rather than the name that was queried — and a same-type record owned by an unrelated name could be renamed onto the queried name and cached there. The chain is now walked to the name it ends at and only the records that name owns are collapsed.
Fixed issue: The gateway rewrote conf/apisix.uid with the instance id it had just read out of it. A deployment that mounts the file without write access for the runtime user — which is how the Docker Compose package ships it — therefore logged failed to open file [/usr/local/apisix//conf/apisix.uid] for writing at error level on every start — four times in 3.9.18, and six when apisix.stream_proxy was configured. The id was always read and registered correctly; the redundant write, and the errors it produced, are gone.

Control Plane

Fixed issue: Any signed-in user could list the upstreams of a service, including their node addresses, because the endpoint that lists them carried no permission check while every other view of the same data was filtered by permission. Listing a service's upstreams now requires view permission on that service.
Fixed issue: An SSL configuration that carried both cert/key and certs/keys for the same SNI — the way an RSA and an ECDSA certificate are served side by side — was rejected with input matches more than one oneOf schemas. Both are now accepted together, and the Data Plane negotiates whichever the client prefers.
Fixed issue: A jwt-auth consumer credential using an asymmetric algorithm was written to the configuration store with a placeholder private_key field the user never configured, which the Data Plane reported as an unrecognized field and which was stored in the clear. The field is no longer added. Credentials written before the upgrade keep it until they are next saved.
Fixed issue: A gateway instance running a data-plane-only hotfix — a version such as 3.9.18-patch.1, which ships without a matching Control Plane release — was reported as PartiallyCompatible with an Upgrade Recommended hint next to its version, even though it was newer than the Control Plane, while another instance on the base release in the same gateway group was reported as Compatible. An error-level item in its compatibility report also turned it Incompatible and raised the banner asking for a gateway upgrade. A version that differs from the Control Plane's only in its pre-release part is now judged as the release it is built on.

Console (Dashboard)

Fixed issue: Reopening the plugin editor in YAML mode repeated every field in the suggestion list once per open, so after five opens each field appeared five times. The editor's YAML support is now configured once per page.