Docs
Apache APISIXObservabilityMonitor APISIX Metrics with Prometheus

Monitor APISIX Metrics with Prometheus

Scrape a route-scoped Apache APISIX metric with a pinned local Prometheus server, assert the exact series, and clean up.

Applies to:Apache APISIX 3.18.0

Deployments:DockerRoles:Platform engineerTime:20 minutesImpact:Changes live traffic

Goal

Enable prometheus on one isolated Route, scrape the APISIX metrics export server with a pinned local Prometheus container, and assert both target health and the Route's HTTP status counter. Remove every guide-owned resource afterward.

Metrics labels create time series. Keep Route scope for this example and evaluate label cardinality, retention, authentication, network policy, storage, and alert ownership before production rollout.

Before you begin

  • Complete Get APISIX with the Docker quickstart.
  • Install Docker, cURL, and jq.
  • Use a shell that supports Bash or Zsh heredocs.
  • Confirm that the quickstart metrics export server is reachable as apisix-quickstart:9091 from apisix-quickstart-net. If a custom deployment binds it only to loopback, configure a protected scrape path using the Prometheus static configuration (opens in Plugin Hub docs) before continuing.

Set the endpoints and protect all fixed resources and files used by this guide:

export APISIX_ADMIN_URL="http://127.0.0.1:9180"
export GATEWAY_URL="http://127.0.0.1:9080"
export PROMETHEUS_URL="http://127.0.0.1:9092"
export GUIDE_DIR="$(pwd)"
read -rsp "Admin API key (press Enter for an unchanged quickstart): " APISIX_ADMIN_KEY && export APISIX_ADMIN_KEY && echo
test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${APISIX_ADMIN_URL}/apisix/admin/routes/prometheus-guide-route" -H "X-API-KEY: ${APISIX_ADMIN_KEY}")" = 404 || { echo 'Stop: route prometheus-guide-route already exists'; exit 1; }
for container in prometheus-guide-server prometheus-guide-upstream; do
  test -z "$(docker ps -aq --filter "name=^/${container}$")" || { echo "Stop: container ${container} already exists"; exit 1; }
done
for file in prometheus-guide.yml prometheus-guide-nginx.conf; do
  test ! -e "${file}" || { echo "Stop: ${file} already exists"; exit 1; }
done

Verify network-local access to the APISIX metrics endpoint without publishing another listener:

docker run --rm --network apisix-quickstart-net curlimages/curl:8.16.0 \
  --fail --silent --show-error --connect-timeout 3 --max-time 10 \
  "http://apisix-quickstart:9091/apisix/prometheus/metrics" | \
grep -q '^apisix_etcd_reachable '

Stop here if this assertion fails. Do not expose an unauthenticated metrics endpoint broadly just to make the tutorial continue.

Steps

Step 1: Start a controlled upstream

Create a deterministic NGINX response and start it without publishing an upstream port:

cat > prometheus-guide-nginx.conf <<'NGINX'
events {}
http {
  access_log off;
  server {
    listen 80;
    location = /prometheus-guide { return 200 '{"source":"prometheus-guide-upstream"}'; }
  }
}
NGINX
docker run -d --name prometheus-guide-upstream \
  --network apisix-quickstart-net \
  -v "${GUIDE_DIR}/prometheus-guide-nginx.conf:/etc/nginx/nginx.conf:ro" \
  nginx:1.29.1-alpine
for attempt in {1..15}; do
  docker exec apisix-quickstart bash -c '</dev/tcp/prometheus-guide-upstream/80' >/dev/null 2>&1 && break
  sleep 1
done
docker exec apisix-quickstart bash -c '</dev/tcp/prometheus-guide-upstream/80'

What changed: APISIX has one controlled local endpoint for generating metrics.

Step 2: Create one monitored Route

Enable prometheus only on the authenticated fixed-ID Route. prefer_name: false makes the route label use the stable Route ID:

jq -n '{
  name:"prometheus-guide-route",
  uri:"/prometheus-guide",
  methods:["GET"],
  plugins:{prometheus:{prefer_name:false}},
  upstream:{type:"roundrobin",nodes:{"prometheus-guide-upstream:80":1}}
}' | \
curl -sS --connect-timeout 3 --max-time 15 \
  "${APISIX_ADMIN_URL}/apisix/admin/routes/prometheus-guide-route" -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" -H 'Content-Type: application/json' \
  --data-binary @- | \
jq -e '.value.id == "prometheus-guide-route" and .value.plugins.prometheus.prefer_name == false'

What changed: only the guide Route contributes route-scoped HTTP metrics.

Step 3: Start a pinned Prometheus server

Create a one-target configuration with a five-second scrape interval:

cat > prometheus-guide.yml <<'YAML'
global:
  scrape_interval: 5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: apisix
    metrics_path: /apisix/prometheus/metrics
    static_configs:
      - targets:
          - apisix-quickstart:9091
YAML
docker run -d --name prometheus-guide-server \
  --network apisix-quickstart-net \
  -p 127.0.0.1:9092:9090 \
  -v "${GUIDE_DIR}/prometheus-guide.yml:/etc/prometheus/prometheus.yml:ro" \
  prom/prometheus:v3.7.3 \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/prometheus \
  --storage.tsdb.retention.time=1h
for attempt in {1..30}; do
  curl --fail --silent --show-error --connect-timeout 2 --max-time 5 \
    "${PROMETHEUS_URL}/-/ready" >/dev/null 2>&1 && break
  sleep 1
done
curl --fail --silent --show-error --connect-timeout 3 --max-time 10 \
  "${PROMETHEUS_URL}/-/ready" | grep -q 'Prometheus Server is Ready'

The Prometheus UI is available only on host loopback. No persistent volume is used, and the one-hour retention limit bounds this disposable example.

What changed: a pinned local Prometheus server scrapes only the APISIX quickstart endpoint.

Validate

Poll the Prometheus API until the APISIX target is up, send one controlled request, then require a non-zero 200 counter for the exact Route ID:

for attempt in {1..12}; do
  curl --fail --silent --show-error --get "${PROMETHEUS_URL}/api/v1/query" \
    --data-urlencode 'query=up{job="apisix"}' | \
    jq -e '.status == "success" and any(.data.result[]?; .value[1] == "1")' && break
  sleep 5
done
curl --fail --silent --show-error --get "${PROMETHEUS_URL}/api/v1/query" \
  --data-urlencode 'query=up{job="apisix"}' | \
jq -e '.status == "success" and any(.data.result[]?; .value[1] == "1")'
curl --fail-with-body -sS --connect-timeout 3 --max-time 10 \
  "${GATEWAY_URL}/prometheus-guide" | \
jq -e '. == {source:"prometheus-guide-upstream"}'
for attempt in {1..12}; do
  PROMETHEUS_RESPONSE="$(curl --fail --silent --show-error --get "${PROMETHEUS_URL}/api/v1/query" \
    --data-urlencode 'query=apisix_http_status{route="prometheus-guide-route",code="200"}')"
  jq -e '.status == "success" and any(.data.result[]?; (.["value"][1] | tonumber) > 0)' <<<"${PROMETHEUS_RESPONSE}" && break
  sleep 5
done
jq -e '.status == "success" and any(.data.result[]?; (.["value"][1] | tonumber) > 0)' <<<"${PROMETHEUS_RESPONSE}"

All assertions must pass. They prove the scrape target is healthy, the intended upstream responded, and Prometheus stored the HTTP status series for this exact Route.

Roll back / Clean up

Delete the Route, then remove the collector, upstream, files, and shell variables:

curl --fail-with-body -sS --connect-timeout 3 --max-time 15 -X DELETE \
  "${APISIX_ADMIN_URL}/apisix/admin/routes/prometheus-guide-route" \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}"
docker rm -f prometheus-guide-server prometheus-guide-upstream
rm -f prometheus-guide.yml prometheus-guide-nginx.conf
unset APISIX_ADMIN_URL APISIX_ADMIN_KEY GATEWAY_URL PROMETHEUS_URL GUIDE_DIR PROMETHEUS_RESPONSE

Expected: the fixed Route, both containers, in-container metrics data, local files, and shell variables are absent.

Troubleshoot

SymptomCauseFix
The preflight metrics assertion fails.The export server is bound to container loopback, disabled, or uses another address.Configure a protected scrape endpoint from the static configuration reference; verify it from the collector network before proceeding.
Prometheus stays unready.The YAML is invalid, port 9092 is occupied, or local resources are insufficient.Run docker logs prometheus-guide-server and docker inspect prometheus-guide-server; correct the issue without changing the fixed Route.
The up query returns 0.Prometheus cannot resolve or connect to apisix-quickstart:9091.Confirm both containers use apisix-quickstart-net and repeat the preflight from the same network.
The upstream response passes but the Route series is absent.The Route configuration has not propagated or the next scrape has not completed.Read the fixed Route through the authenticated Admin API, inspect APISIX errors, and retain the bounded polling interval.
Prometheus uses excessive memory or storage.Retention, scrape frequency, or label cardinality is too high.Measure active series, disable only optional labels documented for APISIX, and size production storage from retention and ingestion targets.

Integrate APISIX with Coraza WAF

Previous Page

Monitor APISIX Metrics with Datadog

Next Page