Disaster Recovery
Recover AISIX from region, control-plane, gateway-fleet, and database loss, with the recovery path for each tied to how AISIX actually behaves in that scenario.
Scope
This page is a scenario narrative, not a new source of facts: the underlying behavior lives in High Availability and Offline Resilience, pulled forward here into specific failure scenarios rather than restated in full. Read those two pages for the complete picture; read this one for what to do when one of the scenarios below happens.
Recovery Objectives
AISIX does not publish a blanket RTO or RPO SLA that covers every deployment and scenario. There is no single number to plan against. What follows is the concrete recovery characteristic of each specific failure mode — a stale-budget window, a retry tolerance, a "nothing to recover" case — stated in the section it belongs to. Where AISIX establishes no number for a scenario, that section says so rather than asserting one.
Region or Availability-Zone Loss
With the active-active pattern in High Availability, AISIX gateways run in two or more active deployments across independent failure domains, each behind its own load balancer, with a global load balancer steering traffic between them. Losing one region or availability zone means the global load balancer stops sending new requests to the deployment behind it and continues serving from the surviving deployment — nothing in AISIX itself needs to react.
Without that pattern — a single gateway deployment — a region or AZ loss removes the whole traffic path until it is redeployed elsewhere. Gateways themselves are stateless: they hold no data of their own beyond the configuration snapshot they have applied, so a redeployed instance in a new location has nothing to recover except reconnecting to the control plane and re-applying configuration (see Gateway Fleet Loss below). If the control plane survives the region loss, the effective RPO for the traffic path is close to zero: a redeployed gateway pulls the current configuration and starts serving from it, with no data to reconstruct.
Control-Plane Loss and Connectivity Loss
A gateway's connection to the AISIX Enterprise control plane is a management path, not the traffic path. Offline Resilience covers this in full; the facts that matter for recovery planning are pulled forward here.
Continue Serving
A running gateway keeps serving its latest accepted configuration snapshot from memory for as long as the control-plane connection is down. AISIX does not withdraw a gateway from traffic merely because no newer configuration event has arrived — treat stale configuration as an operator alert, not a load-balancer health condition.
Restart from Cached Configuration
This only helps a gateway that restarts mid-outage, and it depends on a setting most deployments do not have turned on: on-disk snapshot caching is disabled by default, in both managed and self-hosted etcd modes. Without managed.snapshot_cache_enabled: true (and a writable snapshot_cache_path) set ahead of time, a gateway that restarts during a control-plane outage has nothing to serve until it reconnects — its proxy listener stays unbound until then. If persistence was enabled beforehand, a restarted gateway with a valid cached snapshot can serve while reconnecting instead.
What Stops, What Keeps Running
Not every connectivity-dependent workflow degrades the same way:
| Workflow | Behavior while disconnected |
|---|---|
| AISIX Enterprise budget checks | AISIX reuses the last decision for up to AISIX_DP_BUDGET_STALE_MAX_SECONDS, which defaults to 600 seconds. Without a cached decision, it denies the request. After the stale window, it applies the failure mode returned with the last decision. |
| AISIX Enterprise usage telemetry | Failed batches are dropped. AISIX does not persist, retry, or replay them after connectivity returns — that usage history is gone, not delayed. |
| Certificate rotation | Requires control-plane connectivity; the gateway keeps using its current certificate while it remains valid. |
Both the budget-cache window and the telemetry-drop behavior are worth planning a drill around: a control-plane outage longer than 10 minutes changes budget-check behavior for every affected gateway at once, and any usage telemetry generated during the outage is permanently unrecoverable, not merely delayed.
Recovery Path
What happens next depends on how the control plane itself is deployed:
- A control plane run in the HA pattern — redundant
cp-api,dp-manager, and dashboard replicas behind an HA Postgres endpoint — self-heals: a failed replica is replaced, gateways reconnect, and heartbeats and resource projection resume without an operator restoring anything. - A single-instance control plane — the common case for a Docker Compose On-Premises install, or any deployment that has not adopted the HA pattern — needs to be restarted, and if the database itself was affected, restored first. That is exactly the database restore procedure in Backup and Restore.
Gateway Fleet Loss
AISIX gateways are disposable compute. High Availability's deployment guidance treats them that way: redeploy the instances, let them re-authenticate to the control plane over mTLS with their certificate bundle, and they pull the current projected configuration on connect. There is no gateway-side backup to restore, and none is needed — a gateway holds no state of its own beyond what it re-derives on startup from the control plane, or, for AISIX OSS, from its resources file or etcd.
The one thing worth keeping backed up per instance, if you rely on Restart from Cached Configuration above, is the persistent state directory holding the certificate bundle and the snapshot cache — but that is a per-instance persistence-volume decision, not a fleet-recovery backup.
Database Loss (Control-Plane Postgres)
How a database loss recovers depends on the same bundled-versus-external split as backing it up.
The bundled Postgres that ships with the Docker Compose packages has no built-in high availability: High Availability states that "the packaged Docker Compose deployments are single-host and do not support this topology" — the replicated PostgreSQL behind a stable endpoint that the HA reference pattern assumes. Losing that single host loses the database outright; recovery is a full restore, not a failover.
The Helm chart's own bundled PostgreSQL sub-chart (postgresql.builtin: true, the chart default) has the same limitation: High Availability states that "the bundled PostgreSQL chart does not provide the replicated database shown in this reference pattern by default." A single-pod loss there is also a full restore, using the bundled-PostgreSQL backup you took as the postgres superuser role.
An external Postgres reached through Helm (postgresql.builtin: false) can be made highly available at the database layer — that is what the HA reference pattern describes as "a replicated PostgreSQL deployment behind a stable endpoint." Whether a given external-Postgres deployment actually has that replication is a decision your DBA team makes outside AISIX; where it does, a single-node failure there fails over the way any HA Postgres cluster does, without an AISIX-specific procedure.
Either way, a database loss that replication does not absorb funnels into the same place: the restore procedure in Backup and Restore, run against your most recent backup.
One tolerance is worth knowing about during that restore. External Database documents that on a fresh database, dp-manager retries missing application-schema errors for about two minutes while cp-api completes its migration — that fact is stated there about first-install bootstrap, but the same tolerance applies to a freshly restored database, which also needs cp-api to complete its migration before the schema is ready. Give cp-api that window before concluding a restore has failed.
Run a DR Drill
Pick one scenario above and run it for real, in a non-production environment:
- Choose a scenario — region loss, control-plane connectivity loss, gateway fleet loss, or database loss — and define what "simulate it" means concretely: kill the pods, block the route, drop the database.
- Execute the matching recovery steps from the section above, not an idealized version of them.
- Record the actual time to recovery and what, if anything, was lost. There is no published number to compare against, so the drill's own result is the baseline the next drill improves on.
- Fix whatever the drill exposed — a missing
snapshot_cache_enabled, a stale backup, an alert that never fired — before the next one.
Related
- Backup and Restore — the database backup and restore procedure this page's recovery paths depend on.
- Organization Data Export and Import — a portable, organization-scoped export and import that recovers one organization's configuration and credentials without a database-level restore, on Hybrid Cloud as well as On-Premises.
- High Availability — the failure-domain design and deployment guidance behind the region-loss and fleet-loss sections above.
- Offline Resilience — the full connectivity-loss behavior the control-plane section above summarizes.
- Multi-Target Routing and Failover — an upstream provider outage is a routing concern, not a disaster-recovery one; configure retries and fallback targets there.