API7 Docs
Observability

RocketMQ Logger

The rocketmq-logger plugin pushes request and response logs as JSON objects to RocketMQ clusters in batches, allowing for customizable log formats to enhance data management.

The rocketmq-logger plugin pushes request and response logs as JSON objects to your RocketMQ clusters in batches and supports the customization of log formats.

Examples

The examples below demonstrate how you can configure rocketmq-logger plugin for different scenarios.

To follow along the examples, start a sample RocketMQ cluster:

docker-compose.yml
version: "3"

services:
  rocketmq_namesrv:
    image: apacherocketmq/rocketmq:4.6.0
    container_name: rmqnamesrv
    restart: unless-stopped
    ports:
      - "9876:9876"
    command: sh mqnamesrv
    networks:
      rocketmq_net:

  rocketmq_broker:
    image: apacherocketmq/rocketmq:4.6.0
    container_name: rmqbroker
    restart: unless-stopped
    ports:
      - "10909:10909"
      - "10911:10911"
      - "10912:10912"
    depends_on:
      - rocketmq_namesrv
    command: sh mqbroker -n rmqnamesrv:9876 -c ../conf/broker.conf
    networks:
      rocketmq_net:

networks:
  rocketmq_net:

Start containers:

docker compose up -d

In a few seconds, the name server and broker should start.

Create the TopicTest topic:

docker exec -i rmqnamesrv rm /home/rocketmq/rocketmq-4.6.0/conf/tools.yml
docker exec -i rmqnamesrv /home/rocketmq/rocketmq-4.6.0/bin/mqadmin updateTopic -n rmqnamesrv:9876 -t TopicTest -c DefaultCluster

Wait for messages in the configured RocketMQ topic:

docker run -it --name rockemq_consumer -e NAMESRV_ADDR=localhost:9876 --net host apacherocketmq/rocketmq:4.6.0 sh tools.sh org.apache.rocketmq.example.quickstart.Consumer

In a few seconds, the consumer should start and listen for messages from APISIX:

01:32:17.823 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
Consumer Started.

Open a new terminal session for the following steps working with APISIX.

Log in Different Meta Log Formats

The following example demonstrates how you can enable the rocketmq-logger plugin on a route, which logs client requests to the route and pushes logs to RocketMQ. You will also understand the differences between the default and origin meta log formats.

Create a route with rocketmq-logger as follows:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "rocketmq-logger-route",
    "uri": "/anything",
    "plugins": {
      "rocketmq-logger": {
        "nameserver_list": [ "127.0.0.1:9876" ],
        "topic": "TopicTest",
        "key": "key1",
        "timeout": 30,
        "meta_format": "default",
        "batch_max_size": 1
      }
    },
    "upstream": {
      "nodes": {
        "httpbin.org:80": 1
      },
      "type": "roundrobin"
    }
  }'

❶ meta_format: set to the default log format.

❷ batch_max_size: set to 1 to send the log entry immediately.

Send a request to the route to generate a log entry:

curl -i "http://127.0.0.1:9080/anything"

You should see a log entry similar to the following:

{
  "client_ip": "127.0.0.1",
  "upstream": "34.197.122.172:80",
  "start_time": 1744727400000,
  "request": {
    "headers": {
      "host": "127.0.0.1:9080",
      "accept": "*/*",
      "user-agent": "curl/8.6.0"
    },
    "querystring": {},
    "size": 86,
    "uri": "/anything",
    "url": "http://127.0.0.1:9080/anything",
    "method": "GET"
  },
  "route_id": "rocketmq-logger-route",
  "apisix_latency": 8.9998455047607,
  "upstream_latency": 503,
  "latency": 511.99984550476,
  "response": {
    "size": 617,
    "headers": {
      "content-length": "391",
      "connection": "close",
      "date": "Tue, 15 Apr 2025 14:30:00 GMT",
      "server": "APISIX/3.15.0",
      "content-type": "application/json"
    },
    "status": 200
  },
  "server": {
    "hostname": "apisix",
    "version": "3.15.0"
  },
  "service_id": ""
}

Update the rocketmq-logger meta log format to origin:

curl "http://127.0.0.1:9180/apisix/admin/routes/rocketmq-logger-route" -X PATCH \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "plugins": {
      "rocketmq-logger": {
        "meta_format": "origin"
      }
    }
  }'

Send a request to the route again to generate a new log entry:

curl -i "http://127.0.0.1:9080/anything"

You should see a log entry in the raw HTTP request format:

GET /anything HTTP/1.1
host: 127.0.0.1:9080
user-agent: curl/8.6.0
accept: */*

Log Request and Response Headers With Plugin Metadata

The following example demonstrates how you can customize log format using plugin metadata and built-in variables to log specific headers from request and response.

In APISIX, plugin metadata is used to configure the common metadata fields of all plugin instances of the same plugin. It is useful when a plugin is enabled across multiple resources and requires a universal update to their metadata fields.

First, create a route with rocketmq-logger as follows:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "rocketmq-logger-route",
    "uri": "/anything",
    "plugins": {
      "rocketmq-logger": {
        "nameserver_list": [ "127.0.0.1:9876" ],
        "topic": "TopicTest",
        "key": "key1",
        "timeout": 30,
        "meta_format": "default",
        "batch_max_size": 1
      }
    },
    "upstream": {
      "nodes": {
        "httpbin.org:80": 1
      },
      "type": "roundrobin"
    }
  }'

❶ meta_format: set to the default log format. It is important to note that this is mandatory if you would like to customize log format with plugin metadata. If meta_format is set to origin, the log entries will remain in origin format.

❷ batch_max_size: set to 1 to send the log entry immediately.

Next, configure the plugin metadata for rocketmq-logger:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/rocketmq-logger" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "log_format": {
      "host": "$host",
      "@timestamp": "$time_iso8601",
      "client_ip": "$remote_addr",
      "env": "$http_env",
      "resp_content_type": "$sent_http_Content_Type"
    }
  }'

❶ log the custom request header env.

❷ log the response header Content-Type.

Send a request to the route with the env header:

curl -i "http://127.0.0.1:9080/anything" -H "env: dev"

You should see a log entry similar to the following:

{
  "host": "127.0.0.1",
  "client_ip": "127.0.0.1",
  "resp_content_type": "application/json",
  "route_id": "rocketmq-logger-route",
  "env": "dev",
  "@timestamp": "2025-04-15T14:30:00+00:00"
}

Log Request Bodies Conditionally

The following example demonstrates how you can conditionally log request body.

Create a route with rocketmq-logger as follows:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "plugins": {
      "rocketmq-logger": {
        "nameserver_list": [ "127.0.0.1:9876" ],
        "topic": "TopicTest",
        "key": "key1",
        "timeout": 30,
        "meta_format": "default",
        "batch_max_size": 1,
        "include_req_body": true,
        "include_req_body_expr": [["arg_log_body", "==", "yes"]]
      }
    },
    "upstream": {
      "nodes": {
        "httpbin.org:80": 1
      },
      "type": "roundrobin"
    },
  "uri": "/anything",
  "id": "rocketmq-logger-route"
}'

❶ include_req_body: set to true to include request body.

❷ include_req_body_expr: only include request body if the URL query string log_body is yes.

Send a request to the route with an URL query string satisfying the condition:

curl -i "http://127.0.0.1:9080/anything?log_body=yes" -X POST -d '{"env": "dev"}'

You should see the request body logged:

{
  ...,
    "method": "POST",
    "body": "{\"env\": \"dev\"}",
    "size": 183
  }
}

Send a request to the route without any URL query string:

curl -i "http://127.0.0.1:9080/anything" -X POST -d '{"env": "dev"}'

You should not observe the request body in the log.

If you have customized the log_format in addition to setting include_req_body or include_resp_body to true, the plugin would not include the bodies in the logs.

As a workaround, you may be able to use the NGINX variable $request_body in the log format, such as:

{
  "rocketmq-logger": {
    ...,
    "log_format": {"body": "$request_body"}
  }
}

Configuration

Previous Page

Configuration

Next Page