Docs
Apache APISIXObservabilityTrace Requests with Zipkin

Trace Requests with Zipkin

Trace requests from Apache APISIX to a controlled local upstream, verify the resulting spans through the Zipkin API, and clean up.

Applies to:Apache APISIX 3.18.0

Deployments:DockerRoles:Platform engineerTime:15 minutesImpact:Changes live traffic

Goal

Enable the zipkin plugin on one isolated Route, send spans to a pinned local Zipkin server, and prove through the Zipkin query API that a request produced spans for the intended service. The example does not enable tracing globally or depend on a public upstream.

Tracing can record request metadata and adds processing and storage overhead. This local test samples every request for deterministic validation; select a lower production sampling ratio and apply access control, encryption, and retention to the collector.

Before you begin

  • Complete Get APISIX with the Docker quickstart.
  • Install Docker, cURL, and jq.
  • Use a shell that supports Bash or Zsh heredocs.
  • Ensure TCP port 9411 on loopback is available for the temporary Zipkin UI and query API.

Set the endpoints, current directory, and Admin API key:

export APISIX_ADMIN_URL="http://127.0.0.1:9180"
export GATEWAY_URL="http://127.0.0.1:9080"
export ZIPKIN_URL="http://127.0.0.1:9411"
export GUIDE_DIR="$(pwd)"
read -rsp "Admin API key (press Enter for an unchanged quickstart): " APISIX_ADMIN_KEY && export APISIX_ADMIN_KEY && echo

Protect the fixed Route, containers, and local file used by this guide:

test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${APISIX_ADMIN_URL}/apisix/admin/routes/zipkin-guide-route" -H "X-API-KEY: ${APISIX_ADMIN_KEY}")" = 404 || { echo 'Stop: route zipkin-guide-route already exists'; exit 1; }
for container in zipkin-guide-server zipkin-guide-upstream; do
  test -z "$(docker ps -aq --filter "name=^/${container}$")" || { echo "Stop: container ${container} already exists"; exit 1; }
done
test ! -e zipkin-guide-nginx.conf || { echo 'Stop: zipkin-guide-nginx.conf already exists'; exit 1; }

Steps

Step 1: Start a pinned Zipkin server

Start Zipkin on the APISIX quickstart network and publish its UI and query API only on loopback:

docker run -d --name zipkin-guide-server \
  --network apisix-quickstart-net \
  -p 127.0.0.1:9411:9411 \
  openzipkin/zipkin:3.6.1
for attempt in {1..60}; do
  curl --fail --silent --show-error --connect-timeout 2 --max-time 5 \
    "${ZIPKIN_URL}/health" >/dev/null && break
  sleep 1
done
curl --fail --silent --show-error --connect-timeout 2 --max-time 5 \
  "${ZIPKIN_URL}/health" >/dev/null

What changed: a fresh in-memory Zipkin 3.6.1 server is reachable by APISIX and from local loopback, but not from other host interfaces.

Step 2: Start a controlled upstream

Create a deterministic NGINX response and start it without publishing an upstream port:

cat > zipkin-guide-nginx.conf <<'NGINX'
events {}
http {
  access_log off;
  server {
    listen 80;
    location = /zipkin-guide { return 200 '{"source":"zipkin-guide-upstream"}'; }
  }
}
NGINX
docker run -d --name zipkin-guide-upstream \
  --network apisix-quickstart-net \
  -v "${GUIDE_DIR}/zipkin-guide-nginx.conf:/etc/nginx/nginx.conf:ro" \
  nginx:1.29.1-alpine
for attempt in {1..15}; do
  docker exec apisix-quickstart bash -c '</dev/tcp/zipkin-guide-upstream/80' >/dev/null 2>&1 && break
  sleep 1
done
docker exec apisix-quickstart bash -c '</dev/tcp/zipkin-guide-upstream/80'

What changed: APISIX has one controlled local HTTP endpoint for the traced request.

Step 3: Create the traced Route

Create one authenticated fixed-ID Route. sample_ratio: 1 makes this isolated validation deterministic, while span_version: 2 emits the current two-span representation:

curl -sS --connect-timeout 3 --max-time 15 \
  "${APISIX_ADMIN_URL}/apisix/admin/routes/zipkin-guide-route" -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" -H 'Content-Type: application/json' \
  -d '{
    "name":"zipkin-guide-route",
    "uri":"/zipkin-guide",
    "methods":["GET"],
    "plugins":{
      "zipkin":{
        "endpoint":"http://zipkin-guide-server:9411/api/v2/spans",
        "sample_ratio":1,
        "service_name":"apisix-zipkin-guide",
        "span_version":2
      }
    },
    "upstream":{"type":"roundrobin","nodes":{"zipkin-guide-upstream:80":1}}
  }' | \
jq -e '.value.id == "zipkin-guide-route" and .value.plugins.zipkin.endpoint == "http://zipkin-guide-server:9411/api/v2/spans" and .value.plugins.zipkin.service_name == "apisix-zipkin-guide"'

What changed: only GET /zipkin-guide is sampled and reported to the isolated Zipkin server.

Validate

Require the controlled upstream response:

curl --fail-with-body -sS --connect-timeout 3 --max-time 10 \
  "${GATEWAY_URL}/zipkin-guide" | \
  jq -e '. == {source:"zipkin-guide-upstream"}'

Poll the Zipkin v2 query API until it returns a trace containing the configured local service and both APISIX span names:

for attempt in {1..30}; do
  curl --fail --silent --show-error --connect-timeout 3 --max-time 10 \
    "${ZIPKIN_URL}/api/v2/traces?serviceName=apisix-zipkin-guide&limit=10" | \
    jq -e 'length > 0 and any(.[][]; .localEndpoint.serviceName == "apisix-zipkin-guide") and (([.[][] | .name] | index("proxy")) != null) and (([.[][] | .name] | index("response")) != null)' >/dev/null && break
  sleep 1
done
curl --fail --silent --show-error --connect-timeout 3 --max-time 10 \
  "${ZIPKIN_URL}/api/v2/traces?serviceName=apisix-zipkin-guide&limit=10" | \
  jq -e 'length > 0 and any(.[][]; .localEndpoint.serviceName == "apisix-zipkin-guide") and (([.[][] | .name] | index("proxy")) != null) and (([.[][] | .name] | index("response")) != null)'

Both assertions must pass. The first proves traffic reached the intended upstream; the second proves Zipkin stored the current APISIX span representation for that request. The optional UI is available at ${ZIPKIN_URL}/zipkin/ while the container is running.

Roll back / Clean up

Delete the Route before removing its collector and upstream, then delete the local configuration file:

curl --fail-with-body -sS --connect-timeout 3 --max-time 15 -X DELETE \
  "${APISIX_ADMIN_URL}/apisix/admin/routes/zipkin-guide-route" \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}"
docker rm -f zipkin-guide-upstream zipkin-guide-server
rm -f zipkin-guide-nginx.conf
unset APISIX_ADMIN_URL APISIX_ADMIN_KEY GATEWAY_URL ZIPKIN_URL GUIDE_DIR

Expected: the fixed Route, two containers, local file, loopback listener, and in-memory traces are absent.

Troubleshoot

SymptomCauseFix
Zipkin does not become healthy.Port 9411 is occupied, the image could not start, or the host lacks resources.Inspect docker logs zipkin-guide-server and the port binding; keep the listener on loopback.
The gateway response succeeds but no trace appears.APISIX cannot resolve or reach zipkin-guide-server, or reporting has not completed.Verify both containers share apisix-quickstart-net, inspect APISIX errors, and keep the bounded poll rather than assuming immediate delivery.
The trace exists without both expected span names.The Route uses a different span_version or the stored trace came from another service.Read the fixed Route and query only apisix-zipkin-guide; do not loosen the span assertion.
Tracing overhead is too high in a load test.Sampling every request creates work and storage for every request.Reduce sample_ratio for the tested production workload and measure the resulting coverage and overhead.

Log Requests to Elasticsearch

Previous Page

Log Consumer Labels in the Access Log

Next Page