Back Up and Restore etcd

Take an etcd snapshot of an Apache APISIX configuration store, restore it into a new data directory, and reconnect the gateway to it.

Deployments:DockerRoles:Platform engineerSRETime:20 minutesImpact:Restarts the data plane

Goal

An etcd snapshot on disk that a restore turns back into a working configuration store, with the gateway serving the routes it held before.

etcd clusters tolerate the loss of a minority of their nodes on their own. A snapshot is for what they do not tolerate: several nodes lost at once, a deleted data directory, or an upgrade you want a way back from. The examples run etcd and Apache APISIX in Docker; the same two commands apply anywhere etcd runs, with the data directory and the restart mechanism of your own deployment.

Before you begin

APISIX is running and its Admin API accepts your key. Check it with:
```
curl -s -o /dev/null -w '%{http_code}\n' "${APISIX_ADMIN_URL}/apisix/admin/routes" \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}"
```
Expected: 200. Get APISIX installs an instance in Docker if you have none.
There is configuration in etcd worth restoring. Configure Routes creates the getting-started-ip route this guide validates against, and you can count what is there with:
```
curl -s "${APISIX_ADMIN_URL}/apisix/admin/routes" -H "X-API-KEY: ${APISIX_ADMIN_KEY}" \
  | grep -o '"total":[0-9]*'
```
Expected: "total": and a number of one or more.
The etcd container is running in the same Docker network as APISIX. Check it with:
```
docker inspect --format '{{.State.Status}}' "${ETCD_CONTAINER}"
```
Expected: running.

Set the variables this guide uses:

export APISIX_ADMIN_URL="http://127.0.0.1:9180"
export GATEWAY_URL="http://127.0.0.1:9080"
export ETCD_CONTAINER="etcd-quickstart"       # the etcd container name in your deployment
export ETCD_LISTEN_PORT="2379"                # the etcd client port
export ETCD_NETWORK="apisix-quickstart-net"   # the Docker network APISIX and etcd share
export ETCD_IMAGE="gcr.io/etcd-development/etcd:v3.5.32"   # supplies both etcdctl and etcdutl
export APISIX_CONTAINER="apisix-quickstart"   # the APISIX container name in your deployment
read -rsp "Admin API key: " APISIX_ADMIN_KEY && export APISIX_ADMIN_KEY && echo

Steps

Three steps: take the snapshot, restore it into a directory that is not the live one, then point etcd at the restored directory and let the gateway reconnect. The official etcd distribution carries both tools this needs — etcdctl saves a snapshot from a running member, and etcdutl restores one into a data directory.

Step 1: Take the snapshot

Create the directory the snapshot lands in:

mkdir -p ./backup

Then run etcdctl snapshot save against the etcd client port. The three tabs are the same command in three places: from a throwaway container on the same network, inside the etcd container, or from the host.

The backup directory on the host is mounted into the container, so the snapshot is written outside the container that made it:

docker run --rm \
    --mount type=bind,source="$(pwd)/backup",target=/backup \
    -e ETCDCTL_API=3 \
    --network "${ETCD_NETWORK}" \
    "${ETCD_IMAGE}" \
    etcdctl --endpoints="http://${ETCD_CONTAINER}:${ETCD_LISTEN_PORT}" snapshot save /backup/snapshot.db

{"level":"info","ts":"2026-07-21T06:04:56.103946Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/backup/snapshot.db.part"}
{"level":"info","ts":"2026-07-21T06:04:56.107775Z","logger":"client","caller":"v3@v3.5.32/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2026-07-21T06:04:56.107820Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"http://etcd-quickstart:2379"}
{"level":"info","ts":"2026-07-21T06:04:56.113786Z","logger":"client","caller":"v3@v3.5.32/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2026-07-21T06:04:56.114407Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"http://etcd-quickstart:2379","size":"45 kB","took":"now"}
{"level":"info","ts":"2026-07-21T06:04:56.114681Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/backup/snapshot.db"}
Snapshot saved at /backup/snapshot.db

etcdctl ships inside the etcd image, so the snapshot can be written in the container and copied out. This needs the container's directory permissions to allow the write:

docker exec -it "${ETCD_CONTAINER}" \
  etcdctl --endpoints="http://127.0.0.1:${ETCD_LISTEN_PORT}" snapshot save /tmp/snapshot.db

…
Snapshot saved at /tmp/snapshot.db

Copy it onto the host and remove the copy inside the container, so a snapshot does not sit in the container whose failure it exists for:

docker cp "${ETCD_CONTAINER}:/tmp/snapshot.db" ./backup/snapshot.db
docker exec "${ETCD_CONTAINER}" rm /tmp/snapshot.db

etcdctl can run on the host when the etcd client endpoint is reachable from it. Install it from the etcd distribution for your operating system, or with a package manager, and expose the endpoint over TLS rather than in the clear if it leaves the host:

etcdctl --endpoints="http://127.0.0.1:${ETCD_LISTEN_PORT}" snapshot save ./backup/snapshot.db

…
Snapshot saved at ./backup/snapshot.db

What changed: nothing in etcd or in the gateway; ./backup/snapshot.db now holds the state of the configuration store at the moment the command ran.

Step 2: Restore the snapshot into a new data directory

etcdutl snapshot restore writes a complete etcd data directory from the snapshot. Restore into a directory that is not the live one, so a failed restore costs nothing and the original data directory is still there to compare against:

mkdir -p ./restore
docker run --rm \
    --mount type=bind,source="$(pwd)/backup",target=/backup \
    --mount type=bind,source="$(pwd)/restore",target=/restore \
    "${ETCD_IMAGE}" \
    etcdutl snapshot restore /backup/snapshot.db --data-dir /restore/data

Expected: the command exits without an error and ./restore/data exists on the host. It writes its progress to standard error and creates the directory; it does not print the keys it restored.

etcdutl also runs on the host when it is installed there, with the same arguments and no container.

What changed: ./restore/data is a data directory an etcd member can start on. Nothing is serving from it yet.

Step 3: Start etcd on the restored directory and reconnect the gateway

Replace the etcd container, or the volume it mounts, with the restored data directory, and start etcd with the bootstrap configuration your deployment already uses. The gateway then reconnects on its own in most cases; restart it when it does not:

docker restart "${APISIX_CONTAINER}"

The gateway is not serving traffic while it restarts. On a deployment with more than one instance, restart them one at a time.

What changed: etcd is serving the restored configuration and the gateway is reading from it.

Validate

Ask the gateway for a route it held before the snapshot, and send a request through it. The first command proves the configuration store came back; the second proves the traffic path did:

curl -s -o /dev/null -w '%{http_code}\n' \
  "${APISIX_ADMIN_URL}/apisix/admin/routes/getting-started-ip" -H "X-API-KEY: ${APISIX_ADMIN_KEY}"

curl -is "${GATEWAY_URL}/ip" | sed -n '1p'

HTTP/1.1 200 OK

A 404 from the first command means the restored directory is not the one etcd started on. A 404 from the second, with 200 from the first, means the route is back but the gateway has not picked it up yet — step 3's restart is what does that.

Roll back / Clean up

The restore did not touch the original data directory, so going back is starting etcd on it again with the same bootstrap configuration, then restarting the gateway as in step 3.

Remove the working directories once the restored store is the one you are keeping:

rm -rf ./restore

Keep ./backup/snapshot.db. A snapshot is the only artifact here that cannot be recreated after the fact, and it is what an upgrade's rollback depends on — see Upgrade and Roll Back APISIX. Store it where a host failure does not take it with it.

Troubleshoot

Symptom	Cause	Fix
`snapshot save` fails with a permission error on the path inside the container.	The etcd container's user cannot write to the directory the snapshot was aimed at.	Take the snapshot from the temporary container in step 1 instead, which writes to a host directory mounted into it, or from the host.
etcd starts on the restored directory, but the gateway answers `404` on every route.	The gateway is still connected to the old store, or it reconnected before etcd finished starting.	Restart the gateway as in step 3, then repeat the Validate check.
The restore does not produce the data directory you expected, or it fails on the path given to `--data-dir`.	`--data-dir` pointed at a directory that already has contents. A restore recreates a complete etcd data directory from the snapshot rather than merging into an existing one, which is why step 2 restores into a new path.	Restore into an empty path — `./restore/data` under a fresh `./restore`, as in step 2 — then move it into place.

Upgrade and Roll Back APISIX — the procedure this snapshot is a prerequisite of.
Deployment Modes — which deployments keep configuration in etcd at all, and which read a file instead.
Configuration Files — deployment.etcd, including the prefix and the TLS settings a restored cluster has to keep.
Install APISIX with Docker — the container names, network, and mounts the examples above assume.
etcd disaster recovery — the upstream procedure for a multi-node cluster, including restoring every member (external).