Docs
API7 GatewayAI GatewayAI TrafficUse CasesMonitor AI Traffic and Track LLM Costs

Monitor AI Traffic and Track LLM Costs

Gain visibility into LLM usage, token consumption, latency, and costs with API7 Gateway's observability features.

This guide explains how to observe AI traffic and estimate LLM costs using built-in AI logging, APISIX context variables, and standard observability integrations.

Overview

AI observability is different from traditional API observability. For LLM workloads, you need token-level visibility, model attribution, and latency breakdowns such as time to first token.

With API7 Gateway, you can collect:

  • Request and response model metadata.
  • Prompt and completion token counts.
  • End-to-end and upstream timing signals.
  • Optional payload-level logs for request/response content.

The gateway does not calculate billing directly. Cost tracking is derived by mapping token usage to provider pricing.

Prerequisites

  • Install Docker.

  • Install cURL to send requests to the services for validation.

  • Have a running API7 Gateway instance.

  • Create a token from the Dashboard and save it to an environment variable:

    export API_KEY=your-dashboard-token   # replace with your Dashboard token

  • Replace {gateway_group_id} with your gateway group ID. Use default if you are following the quickstart.

  • If you are following the Admin API examples, create or reuse a service in API7 Gateway. If you do not have one yet, follow Create or Reuse a Service, then save its ID to an environment variable:

    export SERVICE_ID=your-service-id         # replace with your service ID

Built-in AI Logging

ai-proxy and ai-proxy-multi support a logging configuration with:

  • logging.summaries (boolean): logs request_model, model, duration, prompt_tokens, completion_tokens, and upstream_response_time.
  • logging.payloads (boolean): logs request messages, stream flag, and response text content.

Enable logging at route scope (or in shared plugin policy where applicable):

curl -k "https://localhost:7443/apisix/admin/routes?gateway_group_id={gateway_group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  --data-binary @- <<EOF
  {
    "id": "ai-observability",
    "service_id": "$SERVICE_ID",
    "paths": ["/ai"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": { "header": { "Authorization": "Bearer $OPENAI_API_KEY" } },
        "options": { "model": "gpt-4o" },
        "logging": {
          "summaries": true,
          "payloads": false
        }
      }
    }
  }
EOF

❶ Enable AI logging on the ai-proxy plugin.

❷ Log summary-level fields (model + token/timing metadata) for cost and performance analysis.

❸ Keep payload logging disabled by default to reduce sensitive content exposure.

For multi-model routes, apply the same logging fields in each ai-proxy-multi instance:

curl -k "https://localhost:7443/apisix/admin/routes?gateway_group_id={gateway_group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  --data-binary @- <<EOF
  {
    "id": "ai-observability-multi",
    "service_id": "$SERVICE_ID",
    "paths": ["/ai-multi"],
    "plugins": {
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "openai-primary",
            "provider": "openai",
            "auth": { "header": { "Authorization": "Bearer $OPENAI_API_KEY" } },
            "options": { "model": "gpt-4o-mini" },
            "logging": {
              "summaries": true,
              "payloads": false
            },
            "weight": 1
          }
        ]
      }
    }
  }
EOF

❶ Configure logging.summaries and logging.payloads per instance on ai-proxy-multi.

For the full configuration reference, see ai-proxy (opens in Plugin Hub docs) and ai-proxy-multi (opens in Plugin Hub docs).

APISIX Variables for AI Traffic

The most useful variables for cost tracking are below. They can be used in access log formats, in external logging plugins, and anywhere else a variable is accepted:

VariableDescription
llm_prompt_tokensPrompt token count reported by the provider.
llm_completion_tokensCompletion token count reported by the provider.
llm_cache_read_input_tokensPrompt tokens the provider served from its cache. Usually billed at a lower rate.
llm_cache_creation_input_tokensPrompt tokens the provider wrote to its cache.
llm_reasoning_tokensTokens spent on reasoning, for models that report them separately.
llm_modelThe model the gateway sent upstream.
request_llm_modelThe model the client asked for.
llm_time_to_first_tokenTime to first token, in milliseconds.
apisix_upstream_response_timeTotal time the provider took, in milliseconds.

For the full set of 14, including the streaming and tool-call variables, see Built-In Variables.

llm_response_text is not a variable. It is a field the logging plugins add to the JSON log entry when logging.payloads is enabled, so $llm_response_text in an access log format resolves to nothing. Read it from the structured log entry instead — and remember that enabling payload logging writes model output, which usually contains whatever the user put in.

Prometheus Integration

Use the standard prometheus (opens in Plugin Hub docs) plugin to export gateway metrics. For AI traffic, the built-in metrics are:

MetricTypeWhat it holds
apisix_llm_latencyhistogramLLM request latency in milliseconds. The type label separates total (whole response, both chat and streaming) from ttft (time to first token, streaming only).
apisix_llm_prompt_tokenscounterCumulative prompt tokens.
apisix_llm_completion_tokenscounterCumulative completion tokens.
apisix_llm_prompt_tokens_disthistogramPrompt tokens per request, so you can see the distribution rather than just the total.
apisix_llm_completion_tokens_disthistogramCompletion tokens per request.
apisix_llm_active_connectionsgaugeConnections currently open to the provider.
apisix_ai_cache_hits_totalcounterResponses served from the AI cache, with a layer label for which cache layer answered.
apisix_ai_cache_misses_totalcounterRequests that missed the AI cache.
apisix_ai_cache_bypasses_totalcounterRequests that skipped the cache entirely.
apisix_ai_cache_embedding_latencyhistogramTime spent generating the embedding used for a semantic cache lookup.

Every one of these carries route, route_id, service, service_id, consumer, node, gateway_group_id, instance_id, portal_id, api_product_id, request_type, request_llm_model and llm_model, and all but apisix_llm_latency also carry matched_uri and matched_host.

There is no provider label. Group by llm_model for what the gateway actually sent, or by request_llm_model for what clients asked for — the two differ whenever a route pins a model with options.model.

If you have set plugin_attr.prometheus.allow_degradation: true, these metrics stop being recorded while the gateway is shedding load, so a gap in a cost report is not necessarily a gap in traffic.

Example PromQL queries:

# Current active LLM connections across all routes
sum(apisix_llm_active_connections)
# Top routes by active LLM connections
topk(10, sum by (route) (apisix_llm_active_connections))
# Alert condition example: sustained high active connections
avg_over_time(sum(apisix_llm_active_connections)[5m]) > 200
# Prompt tokens per minute by model — the shape of a cost report
sum by (llm_model) (rate(apisix_llm_prompt_tokens[1m])) * 60
# Cache hit ratio
sum(rate(apisix_ai_cache_hits_total[5m]))
  / (sum(rate(apisix_ai_cache_hits_total[5m])) + sum(rate(apisix_ai_cache_misses_total[5m])))
# 95th percentile time to first token, by model
histogram_quantile(0.95,
  sum by (le, llm_model) (rate(apisix_llm_latency_bucket{type="ttft"}[5m])))

Adjust labels and thresholds to your deployment and traffic profile.

Logging AI Requests to External Systems

You can forward structured logs with AI metadata to systems such as ELK, Loki, or Splunk by combining APISIX logging plugins with AI context variables.

Example structured log payload:

{
  "route_id": "27bcdea2-7586-47ad-9262-a3adf4b6699e",
  "service_id": "d8402098-2f80-4e08-afab-21b9ebc62090",
  "model": "deepseek-chat",
  "request_model": "",
  "prompt_tokens": 11,
  "completion_tokens": 16,
  "duration_ms": 3991,
  "ttft_ms": 2159,
  "upstream_response_time": 1133
}
  • route_id and service_id are the gateway-assigned UUIDs for the route and service.
  • request_model is empty when the client does not specify a model explicitly; the model is then determined by the plugin configuration.
  • duration_ms is not a built-in AI log field — use $latency (total request time) and $upstream_latency (upstream response time) in your log format configuration to measure request duration.

Use summary logging for default production telemetry. Enable payload logging only when you need short-term debugging with appropriate data handling controls.

Building a Cost Dashboard

API7 Gateway exposes token usage signals, while cost is calculated externally:

estimated_cost = (prompt_tokens × prompt_price_per_token) + (completion_tokens × completion_price_per_token)

Typical dashboard panels in Grafana:

  • Cost by consumer (hourly/daily).
  • Cost by model.
  • Prompt vs. completion token split.
  • Spend trend and budget threshold alerts.

Implementation approach:

  1. Export AI logs/metrics.
  2. Enrich records with provider pricing metadata.
  3. Compute estimated cost in your log or metrics pipeline.
  4. Visualize and alert in Grafana.

For token governance controls, see Token-Based Rate Limiting and Quota Management.

OpenTelemetry Tracing

Use the standard opentelemetry (opens in Plugin Hub docs) plugin to trace AI request lifecycle latency across:

  • Client request ingress.
  • Gateway processing.
  • Upstream LLM call.
  • Response egress.

This helps isolate whether latency is caused by client-side behavior, gateway plugins, or upstream model/provider response time.

Next Steps

Control AI Costs with Token-Based Rate Limiting

Previous Page

Enforce AI Guardrails and Protect PII

Next Page