AISIX 1.4.0

Release date: September 22, 2026

A gateway now re-sends a usage batch the control plane failed to answer, so a control-plane outage delays usage instead of losing it. A Redis that is unreachable when the gateway starts no longer holds its listeners closed. An etcd whose dial never completes no longer holds the boot open either. The gateway starts, serves its cached configuration when there is one, and retries in the background. A resource kind the gateway does not know is forward compatibility rather than a failed reload. Logging, the configuration apply, the Prometheus scrape, and upstream DNS no longer contend with request serving. The Dashboard generates only the basic gateway deployment template, and advanced gateway configuration is documented instead. The offline package forwards the CORS allowlist, and the origin allowlists accept the spellings a browser actually sends.

Behavior Changes

• A rate-limit Redis that is down when the gateway starts no longer stops the gateway. A gateway configured with ratelimit.backend: redis used to bind no listener at all while that Redis accepted or dropped packets without answering, and the process then exited after several minutes. /livez and /metrics were refused for that whole stretch. The gateway now binds its listeners and serves, counting per replica. It logs one warning naming the backend, the Redis host and port, and the connect budget it spent; the configured URL is never logged, because it carries the password. The degradation is restated every five minutes. A background task attaches the shared backend as soon as Redis answers. This is a temporary degradation, never a permanent switch to the memory backend, and cluster-wide limits are not enforced while it lasts.

• A cache Redis that is down at startup no longer stops the gateway either. The listeners bind, and every backend: redis cache policy is served as a miss until Redis answers. The warning, the five-minute restatement, and the background attach work the same way. Vector search settles only on a real probe answer, so nothing is published speculatively. While the probe has not passed, a semantic policy logs once and serves exact matches only, with no embedding call and no Redis round trip per request. An operator who wanted the boot to fail on an unreachable cache no longer gets that.

• etcd.dial_timeout_ms now defaults to 5000 ms, where the dial was previously unbounded. Only a deployment that sets etcd.user is affected, because without credentials the dial performs no I/O. The value bounds one connection attempt, and the whole dial gets it once per configured endpoint, so the budget is dial_timeout_ms × max(1, endpoints). An expired dial reports an etcd that could not be reached, and the existing path runs: warn, bind, serve from the snapshot cache, and retry in the background. That snapshot cache is read after the dial, so an unbounded dial also kept unread the one thing that exists for a control-plane outage. Write dial_timeout_ms: 0 to keep the old unbounded behavior. request_timeout_ms is untouched and still defaults to unbounded.

• A failing Redis cache reports an outage once instead of once per request. The cache lookup failed, cache write failed, and cache backfill write failed messages, and their two semantic equivalents, log the first failure of an outage at warning level and the rest at debug level. The next success re-arms them, so a later outage is reported again. aisix_redis_failures_total{operation} is untouched and still counts every failed operation.

• A Redis that answers and refuses the connection settings is told apart from an outage, and does not end the boot. A rejected credential, a NOAUTH or DENIED answer, and a database the server does not have were all reported as an outage. The warning therefore blamed the network for a server that had replied in milliseconds. Such a failure now takes the degraded path under its own name. The boot warning carries reason=refused and the driver's error, and the five-minute restatement carries the same word, so one filter finds both. An unreachable backend keeps reason=unreachable. The gateway serves degraded — per-replica counting for the limiter, misses for the cache — and keeps re-attaching. A credential corrected on the Redis side is therefore adopted without restarting the gateway. How specific the error is depends on the driver: a refused database carries the server's own text, while a refused credential arrives as a fixed authentication message.

• What still ends the boot is configuration the gateway rejects before any network call. A Redis url the driver cannot parse and TLS material that cannot be read are refused locally, for both the limiter and the cache. So are the block's own checks: a missing url, nodes, sentinels or master_name for the selected mode, and timeout_secs: 0. A typo therefore still announces itself at startup.

• The shared Redis username, password, and database fields now apply in every mode. single mode, which is the default, previously parsed them and then discarded them. A credential supplied through password never reached the handshake, so the gateway logged connected at boot and then failed every Redis operation for the life of the process. The explicit fields also now override a credential embedded in the url, because a value injected through the environment is no use if a stale one in the URL outranks it. The credential is taken as a pair, so setting either half replaces both. database applies in single and sentinel modes; Redis Cluster has only DB 0. A boot warning names the case where a credential is configured in both places at once.

• An unknown resource kind no longer reads as a failed reload. A gateway that reads a document whose kind its build does not know now reports aisix_config_last_reload_successful 1 and state: "synced", where it previously reported 0 and "degraded". The row moves from rejected[] to the new unknown_kinds[] on GET /status/config, and from aisix_config_rejected_resources to the new aisix_config_unknown_kind_resources{kind}. The two series are disjoint. A genuine rejection still flips the gauge, and the sticky last failure names it. Unknown-kind rows also get their own retention budget, so a flood of them no longer evicts real rejections from /status/config and from the heartbeat. Nothing has to be edited. An alert that counted aisix_config_rejected_resources across all kinds to mean "something is unloadable" now excludes unknown kinds.

• The Metric labels editor is removed from the deploy-a-gateway page. The Dashboard now generates only the basic gateway deployment template: the mTLS bundle, AISIX_CONFIG_PATH, the dp-manager endpoint, and the proxy port. observability.metrics.labels is still a supported gateway startup setting, and it is now configured from the documentation rather than from a console form. The panel held no server-side state, so a gateway already running from a previously generated snippet is unaffected.

• A usage batch that fails is re-sent rather than dropped. Against a 1.4.0 control plane, a gateway re-sends the same batch unchanged, under the same batch id. It stops when the batch is accepted, or when the batch's oldest event is 30 minutes old. It also gives up after eight consecutive failures that carried a response. A control-plane outage therefore delays usage rather than losing it, and usage can appear in the console minutes late. Delivery of later batches waits behind the batch being re-sent. aisix_usage_event_drops_total gains two reason values: send_failed, for a batch given up on without re-sending, and retry_budget_exhausted. Against a control plane that does not advertise batch de-duplication, nothing changes and the batch is dropped as before.

• A stalled log consumer costs log lines instead of stalling the gateway. Log events go into a bounded queue of 32,768 entries, drained by a dedicated thread. When the container runtime stops draining the gateway's stderr, and a log rotation is enough to do that, the new event is dropped and counted in aisix_log_lines_dropped_total. One summary warning follows once the sink catches up. The request worker previously blocked in write and the whole gateway stalled, /livez included. Loss is zero unless the queue fills.

• Telemetry exporters ride out a longer outage and honor Retry-After. A transiently failing exporter keeps a batch in flight for up to 5 minutes, backing off from 200 ms to a 30-second ceiling. This covers OTLP, Datadog, SLS, and object stores. It previously gave up after four attempts spanning about 3 seconds. Retry-After on a 429 or a 503 is honored in both of its forms, capped at the same 30 seconds. The queue behind a sink now holds 8,192 records, up from 1,024. A full queue drops the newest record as queue_full, where records were previously lost further upstream as retries_exhausted. A full-capture exporter owns its records, so a long outage holds more memory. Permanent errors still drop on the first attempt.

• No request pays for Redis failure detection during an outage. When the Redis circuit breaker opens, a background probe now tests recovery as the 30-second window expires. The response cache and the shared rate-limit counters share that breaker. The next business request is no longer admitted as the probe. Recovery timing is unchanged; it is simply no longer billed to a caller. A PING proves less than the real command the old probe used, so a server that answers PING while this subsystem's own operations still time out closes the breaker. The commands behind it then each pay one timeout budget until the first failure re-opens it.

• GET /metrics is served as a chunked response. The exposition is streamed in 256 KiB pieces instead of being built whole in memory. The text itself is unchanged, and pieces break between whole series. At high cardinality, the peak memory a scrape adds fell from hundreds of megabytes to a few. There are two consequences. A render that fails after the first piece ends the body with an error rather than answering 500, so a truncated scrape cannot be ingested as a complete one. And a reader that stops reading without closing the connection loses its own scrape after a bounded wait, instead of parking every later scrape. Two scrapes that overlap no longer share one render; the second waits one render and produces its own current exposition.

• Upstream host names are resolved through a short-lived shared cache. Every outbound client on the request path resolves through one process-wide cache. Concurrent lookups of a name collapse into one, answers are reused for 30 seconds, failures for one second, and the cache holds at most 1,024 names. An upstream whose address changes is therefore dialed at the previous address for up to 30 seconds. Bedrock and the /v1/realtime dial build their own transports and are not covered. A provider key's resolve_addresses pins still apply on top and never reach the cache.

• The control plane answers 503 instead of 401 when it cannot read the revocation list. Every /dp route a gateway calls now separates two cases. A revocation store that could not be read is 503 MTLS_UNAVAILABLE, which the gateway retries. A certificate that is not acceptable is 401. The routes covered are the heartbeat, certificate rotation, the budget check, and usage ingestion. Neither case fails open. The most likely transient control-plane fault previously reached the gateway as a final refusal.

• The origin allowlists accept the spellings a browser actually sends. AISIX_CLOUD_CORS_ALLOWED_ORIGINS and AISIX_TRUSTED_ORIGINS now accept an IPv4-mapped address in its hex form, such as https://[::ffff:c0a8:1], which is what an Origin header carries. The Helm chart accepts a canonical address carrying a written-out zero hextet, such as https://[1::1:0], instead of refusing the install. The chart also refuses at render time the malformed IPv6 entries that previously rendered and then crash-looped cp-api. It refuses a wildcard suffix carrying more than one leading separator, which cp-api has always refused. No entry cp-api accepted before is refused now.

New Features

• Usage ingestion is idempotent per batch. A gateway mints one id per usage batch and sends it as X-Aisix-Usage-Batch-Id. dp-manager claims that id in the same transaction that inserts the rows. A batch that committed is therefore answered with exactly the first delivery's response, and writes nothing a second time. Every /dp/telemetry response carries X-Aisix-Usage-Batch-Dedup: 1, which is how a gateway learns that re-sending is safe. The capability is judged per response and never cached, so mixed-version dp-manager replicas and a rollback are both safe. A batch the control plane can never store is answered 422 and dropped at once, rather than re-sent. Claims are kept at least 24 hours and swept by the existing prune daemon.

• The offline package can configure CORS. AISIX_CLOUD_CORS_ALLOWED_ORIGINS is now forwarded by the packaged docker-compose.yaml, offered commented out in .env.example, and documented in the README beside AISIX_TRUSTED_ORIGINS. A deployment whose Dashboard is served from a different origin than cp-api previously had to hand-edit the compose file, which the next package extract overwrote. The source-tree compose stack gained the same knob, plus the notification and MCP-spec opt-ins and the CA-bootstrap override.

• New gateway metrics. The gateway adds aisix_config_unknown_kind_resources{kind}, aisix_config_apply_duration_seconds{trigger}, aisix_config_apply_batch_events{trigger}, and aisix_log_lines_dropped_total. The trigger label is watch for a coalesced batch of watch events and full for a resync. All four are registered in the metric-label catalog, so observability.metrics.labels can select on them, and trigger is a new label variable.

Improvements

• Background computation no longer competes with request serving. The configuration apply, the Prometheus render, and the snapshot reclaimer run on a demoted-priority thread, while request workers, the /livez and /readyz listener, and the log writer keep their priority.
• The configuration digest is computed when something reads it, rather than on every apply. On a configuration of roughly 70,000 resources, an apply fell from 0.165 to 0.0275 core-seconds, with the digests byte-identical and apply_seq and applied_at keeping their documented rule.
• The per-caller budget decision cache evicts in constant time, instead of scanning all 10,000 entries on every miss.
• The queue between the proxy and the usage sender now holds 16,384 events, up from 1,024. That absorbs a multi-second control-plane stall which previously dropped events as sink_full. A timed flush now fills from the ready queue, instead of sending an undersized batch while a backlog waits.
• Environment-scoped budget totals are answered from a new partial covering index, instead of scanning charged usage from other environments. The index is built concurrently at startup under the existing migration lock, and an interrupted build is repaired.
• Usage ingestion resolves a batch's pricing in two queries, rather than one per model plus one per miss. A batch of 100 models with 50 misses now costs two reads instead of 150.
• The public aisix-cp chart's values table now describes all 117 values, including the four an install cannot skip. The render-time message for a placeholder PostgreSQL password now says openssl rand -hex 24, because base64 characters corrupt the postgres:// DSN the password is embedded in.

Fixes

• cp-api no longer exhausts its connection pool when many gateways request certificates at once. POST /api/environments/{env_id}/gateway_certificates held one pooled connection while taking up to three more, so a rollout of sixteen gateway replicas could wedge the pool until the callers gave up. The certificate, its credential identity, and its audit row now commit or roll back together. A rolled-back issuance therefore leaves no live mTLS credential whose private key the caller never received.
• DELETE /api/environments/{env_id} now revokes certificates and deletes the environment atomically. A failure part-way could previously leave certificates revoked against an environment that survived, kicking a fleet of gateways off a working environment.
• A transient database failure while reading the registered gateway fleet no longer turns a resource save into 500 with nothing stored. An unreadable fleet gates nothing, as it did before 1.3.0. The compatibility gate's own verdict, 422 DP_INCOMPATIBLE, is unchanged when the fleet can be read.
• GET /api/data_export answers 409 DATA_EXPORT_UNAVAILABLE, naming dp-manager as the thing to start, instead of 500 with a raw SQL error, on a control plane where dp-manager has never run.
• The data export and import audit records are now written for attempts that fail or are abandoned mid-download. They carry an outcome of completed, failed, or client_disconnected, and an export also carries the bytes_delivered counted at the response writer. A client that took part of an organization's data and vanished previously left no record at all.
• dp-manager no longer stays in gRPC graceful shutdown indefinitely behind a long-lived watch. Draining is capped within the existing five-second shutdown budget, and startup failures cancel and join the workers they created.
• cp-api and dp-manager no longer run Gin in debug mode. The 187 route-dump lines per start, each naming an internal handler, are gone from production logs. GIN_MODE=debug still reaches the dump.
• The Dashboard no longer warns at boot about Google and GitHub OAuth providers it was never configured with. A provider given only a client id is no longer registered with an empty secret.
• Deleting a budget while the aggregator is mid-tick no longer produces warnings from the aggregator or from the alert notifier.
• A usage event carrying an over-long value in one of four bounded fields is now clipped at ingest, with one warning per batch. It previously failed the whole batch, taking every other billing row in it. The four fields are cache_status, inbound_protocol, operation, and guardrail_bypassed_reason.

API Changes

The structural comparison reports no operation added, removed, or changed. Both documents carry 129 operations, no schema was added or removed, and there is no breaking change. The only structural difference is the description of GET /data_export. One change carries meaning the structural comparison cannot see.

The gateway-to-control-plane protocol, /dp/*, is not part of the Cloud Admin API. It gained X-Aisix-Usage-Batch-Id on POST /dp/telemetry, X-Aisix-Usage-Batch-Dedup: 1 on every /dp response, a 422 for a batch that can never be stored, and 503 MTLS_UNAVAILABLE when the revocation list cannot be read.

Upgrade Notes

These notes apply when you upgrade from any release earlier than 1.4.0, including an upgrade that passes over it on the way to a later release.

• Do not add the new metric families to observability.metrics.labels until every gateway in the environment runs 1.4.0. A gateway refuses an unknown metric family in that setting at startup. A snippet naming aisix_config_unknown_kind_resources, aisix_config_apply_duration_seconds, aisix_config_apply_batch_events, or aisix_log_lines_dropped_total therefore stops a 1.3.0 gateway from starting.
• An alert on aisix_config_rejected_resources now excludes unknown resource kinds. Add aisix_config_unknown_kind_resources to keep the previous, wider view of "something is unloadable".
• The first cp-api start after the upgrade builds a second partial index on dpmgr_usage_events, concurrently and under the existing migration lock. On a large usage table, expect startup to do extra work once, plus the index's storage and write maintenance from then on.
• A credentialed etcd whose dial legitimately takes longer than 5 seconds must now set etcd.dial_timeout_ms explicitly. The key defaults to 5000 ms, where the dial used to be unbounded, and 0 restores the old behavior. Deployments without etcd.user are unaffected, because their dial performs no I/O.
• Remove or correct a stale redis.password or redis.username field before upgrading. In single mode those fields had no effect until 1.4.0, so a deployment carrying a stale one beside a working URL credential now sends the field value. The shared backend then degrades under reason=refused, where the field was previously ignored.
• Expect a boot warning on a Redis that requires authentication and has no credential configured. Such a deployment already failed every Redis operation. It now reports reason=refused at startup and serves degraded, instead of logging connected and failing per request.
• Usage re-sending requires both planes at 1.4.0. A 1.4.0 gateway talking to an older control plane drops a failed batch exactly as before. No configuration enables or disables this.