Implement Traffic Mirroring

Mirror an Apache APISIX request to a secondary service and verify both the primary response and the mirrored copy.

Deployments:DockerRoles:API developerPlatform engineerTime:10 minutesImpact:Changes live traffic

Goal

Create one isolated Route that sends a request to a primary upstream and copies the same request to a secondary receiver with the proxy-mirror (opens in Plugin Hub docs) plugin. Prove independently that the primary response succeeds and that the receiver observes the mirrored request.

Mirrored requests can contain credentials, personal data, and other sensitive headers or bodies. Use a protected receiver with suitable access controls and retention when adapting this procedure outside a disposable environment.

Before you begin

Complete Get APISIX on a machine with Docker.
Install cURL and jq.

Set the Admin API and gateway endpoints, then enter the Admin API key:

export APISIX_ADMIN_URL="http://127.0.0.1:9180"
export GATEWAY_URL="http://127.0.0.1:9080"
read -rsp "Admin API key (press Enter for an unchanged quickstart): " APISIX_ADMIN_KEY && export APISIX_ADMIN_KEY && echo

Confirm the quickstart network, Admin API, and gateway are available:

docker network inspect apisix-quickstart-net >/dev/null
test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${APISIX_ADMIN_URL}/apisix/admin/routes" -H "X-API-KEY: ${APISIX_ADMIN_KEY}")" = 200
test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${GATEWAY_URL}")" != 000

Protect the fixed Route ID, URI, and container names. Stop if any check reports a collision rather than replacing an existing resource:

test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${APISIX_ADMIN_URL}/apisix/admin/routes/traffic-mirror-guide-route" -H "X-API-KEY: ${APISIX_ADMIN_KEY}")" = 404 || { echo 'Stop: route traffic-mirror-guide-route already exists'; exit 1; }
ROUTE_COLLISIONS=$(curl -sS --connect-timeout 3 --max-time 10 "${APISIX_ADMIN_URL}/apisix/admin/routes" -H "X-API-KEY: ${APISIX_ADMIN_KEY}" | jq -r --arg uri '/traffic-mirror-guide' '.list[] | select(.value.uri == $uri or ((.value.uris // []) | index($uri))) | .key')
test -z "${ROUTE_COLLISIONS}" || { printf 'Stop: URI is already used by %s\n' "${ROUTE_COLLISIONS}"; exit 1; }
for container in traffic-mirror-guide-primary traffic-mirror-guide-receiver; do
  test -z "$(docker ps -aq --filter name=^/${container}$)" || { echo "Stop: container ${container} already exists"; exit 1; }
done

Steps

Step 1: Start the primary service and mirror receiver

Start two disposable containers on the quickstart network. The primary returns a fixed response. The NGINX receiver deliberately has no matching resource, so its access log records the mirrored request and its 404 response cannot be confused with the primary result:

docker run -d --name traffic-mirror-guide-primary \
  --network apisix-quickstart-net \
  hashicorp/http-echo:1.0.0 -listen=:5678 -text=primary-response

docker run -d --name traffic-mirror-guide-receiver \
  --network apisix-quickstart-net \
  nginx:1.29-alpine

Wait up to 15 seconds for both listeners from the APISIX container:

for attempt in {1..15}; do
  docker exec apisix-quickstart bash -c '</dev/tcp/traffic-mirror-guide-primary/5678' >/dev/null 2>&1 && \
    docker exec apisix-quickstart bash -c '</dev/tcp/traffic-mirror-guide-receiver/80' >/dev/null 2>&1 && break
  sleep 1
done
docker exec apisix-quickstart bash -c '</dev/tcp/traffic-mirror-guide-primary/5678'
docker exec apisix-quickstart bash -c '</dev/tcp/traffic-mirror-guide-receiver/80'

Step 2: Create the mirrored Route

Create the Route at its fixed resource URL. sample_ratio: 1 mirrors every matched request. The mirror host contains only a scheme and network-reachable host; because path is omitted, the receiver gets the original request URI and query string.

curl -sS --connect-timeout 3 --max-time 15 "${APISIX_ADMIN_URL}/apisix/admin/routes/traffic-mirror-guide-route" -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" -H "Content-Type: application/json" \
  -d '{
    "uri": "/traffic-mirror-guide",
    "methods": ["GET"],
    "plugins": {
      "proxy-mirror": {
        "host": "http://traffic-mirror-guide-receiver",
        "sample_ratio": 1
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {"traffic-mirror-guide-primary:5678": 1}
    }
  }' | jq -e '.value.id == "traffic-mirror-guide-route" and .value.plugins["proxy-mirror"].sample_ratio == 1'

APISIX ignores the mirror receiver's response content, but the mirror subrequest can still affect request latency until its configured connect, send, or read timeout expires. Size those global proxy-mirror timeouts (opens in Plugin Hub docs) for the receiver's failure mode.

Validate

Send a request with a unique query marker. Require the primary response to match exactly, then wait up to 10 seconds for the same request target to appear in the receiver's access log:

MIRROR_MARKER="$(date +%s)-$$"
MAIN_RESPONSE=$(curl --fail-with-body -sS --connect-timeout 3 --max-time 15 "${GATEWAY_URL}/traffic-mirror-guide?marker=${MIRROR_MARKER}")
test "${MAIN_RESPONSE}" = "primary-response"

for attempt in {1..10}; do
  docker logs traffic-mirror-guide-receiver 2>&1 | grep -F "GET /traffic-mirror-guide?marker=${MIRROR_MARKER} HTTP/1.1" >/dev/null && break
  sleep 1
done
docker logs traffic-mirror-guide-receiver 2>&1 | grep -F "GET /traffic-mirror-guide?marker=${MIRROR_MARKER} HTTP/1.1"
printf 'Primary response: %s\n' "${MAIN_RESPONSE}"

Expected: the first printed line is the NGINX access-log entry for the unique marker and the final line is Primary response: primary-response. The two assertions are independent: the primary response does not prove that the mirror received a copy, and the receiver's intentional 404 does not replace the primary response.

sample_ratio is an independent probability for each request. Use 1 for deterministic verification. A value such as 0.1 does not guarantee that exactly one of every ten requests is mirrored; validate sampling statistically over sustained traffic.

Roll back / Clean up

Delete the Route before removing its upstream containers, then clear the variables:

curl -sS --connect-timeout 3 --max-time 15 -X DELETE "${APISIX_ADMIN_URL}/apisix/admin/routes/traffic-mirror-guide-route" -H "X-API-KEY: ${APISIX_ADMIN_KEY}"
docker rm -f traffic-mirror-guide-receiver traffic-mirror-guide-primary
unset APISIX_ADMIN_URL APISIX_ADMIN_KEY GATEWAY_URL ROUTE_COLLISIONS MIRROR_MARKER MAIN_RESPONSE

Troubleshoot

Symptom	Cause	Fix
APISIX returns `502`.	The primary container is stopped, on another network, or not listening on port 5678.	Inspect `docker logs traffic-mirror-guide-primary`, its network attachment, and repeat the TCP check.
The primary response succeeds but the receiver log has no marker.	The plugin is absent, `sample_ratio` is below `1`, or APISIX cannot resolve the receiver.	Read back the fixed Route, restore `sample_ratio: 1`, and inspect the receiver plus APISIX logs.
The main request becomes slow when the receiver is unavailable.	The mirror subrequest is waiting for a configured timeout.	Restore the receiver, then review the global connect, send, and read timeouts before testing failures.
The receiver logs a `404`.	NGINX has no file at the mirrored URI.	This is intentional in the disposable test; use the presence of the uniquely marked access-log entry as the delivery assertion.