Monitor APISIX Metrics with Datadog

Send metrics from one Apache APISIX Route through a pinned DogStatsD agent and verify the uniquely tagged series with the Datadog API.

Deployments:DockerRoles:Platform engineerTime:25 minutesImpact:Changes live traffic

Goal

Enable datadog on one isolated Route, send its metrics through a pinned DogStatsD agent, and use a unique run tag to prove that Datadog received the request counter. Remove the Route, plugin metadata, containers, file, and shell secrets afterward.

Datadog custom metrics can affect billing. This guide generates one request and six metric names. Review custom metric billing before adapting the configuration or increasing tag cardinality.

Before you begin

Complete Get APISIX with the Docker quickstart.
Install Docker, cURL, and jq.
Have a Datadog API key, an application key with timeseries_query, and the API hostname for the selected Datadog site. Read-only metric queries require both keys.
Use an isolated quickstart where datadog plugin metadata does not already exist. Plugin metadata affects every datadog plugin instance in the gateway.

Set endpoints and read secrets without placing them in shell history. For example, the US5 API hostname is api.us5.datadoghq.com:

export APISIX_ADMIN_URL="http://127.0.0.1:9180"
export GATEWAY_URL="http://127.0.0.1:9080"
export GUIDE_DIR="$(pwd)"
export DATADOG_RUN_ID="$(date -u +%Y%m%d%H%M%S)-$$"
read -rp "Datadog site (for example, us5.datadoghq.com): " DD_SITE && export DD_SITE
read -rp "Datadog API hostname (for example, api.us5.datadoghq.com): " DATADOG_API_HOST && export DATADOG_API_HOST
read -rsp "Datadog API key: " DD_API_KEY && export DD_API_KEY && echo
read -rsp "Datadog application key: " DD_APP_KEY && export DD_APP_KEY && echo
read -rsp "Admin API key (press Enter for an unchanged quickstart): " APISIX_ADMIN_KEY && export APISIX_ADMIN_KEY && echo
test -n "${DD_SITE}" && test -n "${DATADOG_API_HOST}" && test -n "${DD_API_KEY}" && test -n "${DD_APP_KEY}"

Protect every fixed resource and the global plugin-metadata slot used by this guide:

test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${APISIX_ADMIN_URL}/apisix/admin/routes/datadog-guide-route" -H "X-API-KEY: ${APISIX_ADMIN_KEY}")" = 404 || { echo 'Stop: route datadog-guide-route already exists'; exit 1; }
test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${APISIX_ADMIN_URL}/apisix/admin/plugin_metadata/datadog" -H "X-API-KEY: ${APISIX_ADMIN_KEY}")" = 404 || { echo 'Stop: datadog plugin metadata already exists'; exit 1; }
for container in datadog-guide-agent datadog-guide-upstream; do
  test -z "$(docker ps -aq --filter "name=^/${container}$")" || { echo "Stop: container ${container} already exists"; exit 1; }
done
test ! -e datadog-guide-nginx.conf || { echo 'Stop: datadog-guide-nginx.conf already exists'; exit 1; }

Steps

Step 1: Start a pinned DogStatsD agent

Attach the standalone agent to the APISIX quickstart network. The UDP listener is not published to the host:

docker run -d --name datadog-guide-agent \
  --network apisix-quickstart-net \
  -e DD_API_KEY="${DD_API_KEY}" \
  -e DD_SITE="${DD_SITE}" \
  -e DD_HOSTNAME=apisix-datadog-guide \
  -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true \
  datadog/dogstatsd:7.81.2
for attempt in {1..30}; do
  docker logs datadog-guide-agent 2>&1 | grep -Eq 'DogStatsD.*(running|started|listening)|dogstatsd.*(running|started|listening)' && break
  sleep 1
done
test "$(docker inspect -f '{{.State.Running}}' datadog-guide-agent)" = true

The API key is present in the disposable container environment and is visible to operators with Docker access. Do not treat this pattern as a production secret-delivery mechanism.

What changed: one pinned DogStatsD agent can accept UDP packets only from the shared Docker network and forward them to the selected Datadog site.

Step 2: Point the plugin at the agent

Create plugin metadata with a unique constant tag. The fixed metadata slot was confirmed absent before this write:

jq -n --arg run_id "${DATADOG_RUN_ID}" '{
  host:"datadog-guide-agent",
  port:8125,
  namespace:"apisix",
  constant_tags:["source:apisix-docs-guide",("guide_run:" + $run_id)],
  max_pending_entries:1024
}' | \
curl -sS --connect-timeout 3 --max-time 15 \
  "${APISIX_ADMIN_URL}/apisix/admin/plugin_metadata/datadog" -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" -H 'Content-Type: application/json' \
  --data-binary @- | \
jq -e --arg run_id "${DATADOG_RUN_ID}" '.value.host == "datadog-guide-agent" and .value.constant_tags == ["source:apisix-docs-guide",("guide_run:" + $run_id)]'

What changed: datadog instances on this isolated gateway use the guide's network-local agent and unique run tag.

Step 3: Start a controlled upstream

Create a deterministic NGINX response and start it without publishing an upstream port:

cat > datadog-guide-nginx.conf <<'NGINX'
events {}
http {
  access_log off;
  server {
    listen 80;
    location = /datadog-guide { return 200 '{"source":"datadog-guide-upstream"}'; }
  }
}
NGINX
docker run -d --name datadog-guide-upstream \
  --network apisix-quickstart-net \
  -v "${GUIDE_DIR}/datadog-guide-nginx.conf:/etc/nginx/nginx.conf:ro" \
  nginx:1.29.1-alpine
for attempt in {1..15}; do
  docker exec apisix-quickstart bash -c '</dev/tcp/datadog-guide-upstream/80' >/dev/null 2>&1 && break
  sleep 1
done
docker exec apisix-quickstart bash -c '</dev/tcp/datadog-guide-upstream/80'

What changed: APISIX has one controlled local endpoint for generating metrics.

Step 4: Create one monitored Route

Enable datadog only on the fixed Route. Immediate batching keeps this one-request validation bounded:

jq -n '{
  name:"datadog-guide-route",
  uri:"/datadog-guide",
  methods:["GET"],
  plugins:{datadog:{prefer_name:false,batch_max_size:1,inactive_timeout:1,buffer_duration:10,max_retry_count:2,retry_delay:1}},
  upstream:{type:"roundrobin",nodes:{"datadog-guide-upstream:80":1}}
}' | \
curl -sS --connect-timeout 3 --max-time 15 \
  "${APISIX_ADMIN_URL}/apisix/admin/routes/datadog-guide-route" -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" -H 'Content-Type: application/json' \
  --data-binary @- | \
jq -e '.value.id == "datadog-guide-route" and .value.plugins.datadog.prefer_name == false and .value.plugins.datadog.batch_max_size == 1'

What changed: only the guide Route emits the six documented APISIX metric names.

Validate

Require the controlled upstream response, then query a ten-minute window until Datadog returns a non-zero uniquely tagged request counter:

curl --fail-with-body -sS --connect-timeout 3 --max-time 10 \
  "${GATEWAY_URL}/datadog-guide" | \
  jq -e '. == {source:"datadog-guide-upstream"}'
for attempt in {1..30}; do
  DATADOG_TO_EPOCH="$(date +%s)"
  DATADOG_FROM_EPOCH="$((DATADOG_TO_EPOCH - 600))"
  DATADOG_QUERY="sum:apisix.request.counter{guide_run:${DATADOG_RUN_ID}}.as_count()"
  DATADOG_RESPONSE="$(curl --fail --silent --show-error --get \
    "https://${DATADOG_API_HOST}/api/v1/query" \
    -H "DD-API-KEY: ${DD_API_KEY}" \
    -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
    --data-urlencode "from=${DATADOG_FROM_EPOCH}" \
    --data-urlencode "to=${DATADOG_TO_EPOCH}" \
    --data-urlencode "query=${DATADOG_QUERY}")"
  jq -e '.status == "ok" and any(.series[]?.pointlist[]?; .[1] != null and .[1] > 0)' <<<"${DATADOG_RESPONSE}" && break
  sleep 10
done
jq -e '.status == "ok" and any(.series[]?.pointlist[]?; .[1] != null and .[1] > 0)' <<<"${DATADOG_RESPONSE}"

Both assertions must pass. They prove the intended upstream responded and Datadog ingested the counter emitted by this exact guide run. Metric ingestion is asynchronous, so the bounded poll allows up to five minutes.

Roll back / Clean up

Delete the Route before its metadata and agent, then remove the upstream, file, and shell secrets:

curl --fail-with-body -sS --connect-timeout 3 --max-time 15 -X DELETE \
  "${APISIX_ADMIN_URL}/apisix/admin/routes/datadog-guide-route" \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}"
curl --fail-with-body -sS --connect-timeout 3 --max-time 15 -X DELETE \
  "${APISIX_ADMIN_URL}/apisix/admin/plugin_metadata/datadog" \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}"
docker rm -f datadog-guide-upstream datadog-guide-agent
rm -f datadog-guide-nginx.conf
unset APISIX_ADMIN_URL APISIX_ADMIN_KEY GATEWAY_URL GUIDE_DIR DATADOG_RUN_ID DD_SITE DATADOG_API_HOST DD_API_KEY DD_APP_KEY DATADOG_TO_EPOCH DATADOG_FROM_EPOCH DATADOG_QUERY DATADOG_RESPONSE

Expected: the fixed Route, guide-owned plugin metadata, both containers, local file, and shell variables are absent. The already-ingested custom metric remains subject to Datadog retention and billing policies.

Troubleshoot

Symptom	Cause	Fix
The agent exits or reports intake errors.	The API key or site is wrong, or outbound HTTPS is blocked.	Inspect `docker logs datadog-guide-agent`; verify the key and site in Datadog before creating the Route.
The Admin API refuses plugin metadata.	Metadata already exists or the selected APISIX release does not support a field.	Do not overwrite shared metadata. Use an isolated quickstart matching this site's APISIX 3.18.0 release.
The gateway response passes but the Datadog query stays empty.	UDP cannot reach the agent, the agent cannot reach intake, ingestion is delayed, or the query uses the wrong site.	Verify both containers share `apisix-quickstart-net`, inspect APISIX and agent logs, and confirm `DATADOG_API_HOST` matches `DD_SITE`.
The query returns `403`.	The application key lacks `timeseries_query`.	Grant the minimum read permission to a dedicated validation key; do not broaden the API key.
Custom metric usage grows unexpectedly.	A tag has unbounded values or the plugin is enabled too broadly.	Keep Route scope, use bounded tag sets, and review Metrics Summary before production rollout.

Datadog plugin (opens in Plugin Hub docs) — exported metrics, tags, batching, and metadata.
DogStatsD — agent transport and packet behavior.
Query timeseries points — API authentication, permission, and query parameters.