API7 Docs
Observability

Error Log Logger

The error-log-logger plugin pushes APISIX's error logs to TCP, Apache SkyWalking, Apache Kafka, or ClickHouse servers, in batches. You can specify the severity level of which the plugin should send the corresponding logs.

The error-log-logger plugin pushes APISIX's error logs (error.log) to TCP, Apache SkyWalking, Apache Kafka, or ClickHouse servers, in batches. You can specify the severity level of which the plugin should send the corresponding logs.

The plugin is disabled by default. Once enabled, it will automatically start pushing error logs to remote servers. You should configure remote server details in plugin metadata only, instead of on other resources, such as routes.

Examples

The examples below demonstrate how you can configure error-log-logger plugin for different scenarios.

The APISIX and API7 Gateway runtime configurations do not load error-log-logger by default. Enable it in the gateway static configuration before configuring plugin metadata.

Keep the existing plugin list in config.yaml and add error-log-logger:

config.yaml
plugins:
  # Keep the complete plugin list used by your gateway.
  - error-log-logger

Reload the gateway for changes to take effect.

Send Logs to TCP Server

The following example demonstrates how you can configure the error-log-logger plugin to send error logs to a TCP server.

Start a TCP server listening on port 19000:

nc -l 19000

Configure the plugin metadata for error-log-logger:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/error-log-logger" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "tcp": {
      "host": "192.168.2.103",
      "port": 19000
    },
    "level": "INFO"
  }'

❶ Configure the host to the TCP server address.

❷ Configure the port to your TCP server listening port.

❸ Configure the severity level to INFO so most logs would be sent, for easier verification.

To verify, you can manually generate a log at warn level by reloading APISIX.

If you are using Docker, in the terminal session where netcat is listening, you should see a log entry. If you are using Kubernetes, check the tcp-server pod logs:

kubectl logs -n aic -l app=tcp-server

You should see a log entry similar to the following:

2025/01/26 20:15:29 [warn] 211#211: *35552 [lua] plugin.lua:205: load(): new plugins: {"cas-auth":true,"real-ip":true,"ai":true,"client-control":true,"proxy-control":true,"request-id":true,"zipkin":true,"ext-plugin-pre-req":true,"fault-injection":true,"mocking":true,"serverless-pre-function":true,"cors":true,"ip-restriction":true,"ua-restriction":true,"referer-restriction":true,"csrf":true,"uri-blocker":true,"request-validation":true,"chaitin-waf":true,"multi-auth":true,"openid-connect":true,"authz-casbin":true,"authz-casdoor":true,"wolf-rbac":true,"ldap-auth":true,"hmac-auth":true,"basic-auth":true,"jwt-auth":true,"redirect":true,"key-auth":true,"consumer-restriction":true,"attach-consumer-label":true,"authz-keycloak":true,"proxy-cache":true,"body-transformer":true,"ai-prompt-template":true,"ai-prompt-decorator":true,"proxy-mirror":true,"proxy-rewrite":true,"workflow":true,"api-breaker":true,"ai-proxy":true,"limit-conn":true,"limit-count":true,"limit-req":true,"gzip":true,"server-info":true,"traffic-split":true,"response-rewrite":true,"degraphql":true,"kafka-proxy":true,"grpc-transcode":true,"grpc-web":true,"http-dubbo":true,"public-api":true,"prometheus":true,"datadog":true,"loki-logger":true,"elasticsearch-logger":true,"echo":true,"loggly":true,"http-logger":true,"splunk-hec-logging":true,"skywalking-logger":true,"google-cloud-logging":true,"sls-logger":true,"tcp-logger":true,"kafka-logger":true,"rocketmq-logger":true,"syslog":true,"udp-logger":true,"file-logger":true,"clickhouse-logger":true,"tencent-cloud-cls":true,"inspect":true,"example-plugin":true,"aws-lambda":true,"azure-functions":true,"openwhisk":true,"openfunction":true,"error-log-logger":true,"ext-plugin-post-req":true,"ext-plugin-post-resp":true,"serverless-post-function":true,"opa":true,"forward-auth":true,"jwe-decrypt":true}, context: init_worker_by_lua*

Send Logs to SkyWalking

The following example demonstrates how you can configure the error-log-logger plugin to send error logs to SkyWalking.

Set up SkyWalking OAP server:

Start a SkyWalking storage, OAP and Booster UI with Docker Compose, following Skywalking's documentation. Once set up, the OAP server should be listening on 12800 and you should be able to access the UI at http://localhost:8080.

Configure the plugin metadata for error-log-logger:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/error-log-logger" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "skywalking": {
      "endpoint_addr": "http://192.168.2.103:12800/v3/logs"
    },
    "level": "INFO"
  }'

❶ Configure the endpoint address to the SkyWalking server.

❷ Configure the severity level to INFO so most logs would be sent, for easier verification.

To verify, you can manually generate a log at warn level by reloading APISIX.

In Skywalking UI, navigate to General Service > Services. You should see a service called APISIX with the following log entry:

2025/01/27 07:40:06 [warn] 211#211: *35552 [lua] plugin.lua:205: load(): new plugins: {"cas-auth":true,"real-ip":true,"ai":true,"client-control":true,"proxy-control":true,"request-id":true,"zipkin":true,"ext-plugin-pre-req":true,"fault-injection":true,"mocking":true,"serverless-pre-function":true,"cors":true,"ip-restriction":true,"ua-restriction":true,"referer-restriction":true,"csrf":true,"uri-blocker":true,"request-validation":true,"chaitin-waf":true,"multi-auth":true,"openid-connect":true,"authz-casbin":true,"authz-casdoor":true,"wolf-rbac":true,"ldap-auth":true,"hmac-auth":true,"basic-auth":true,"jwt-auth":true,"redirect":true,"key-auth":true,"consumer-restriction":true,"attach-consumer-label":true,"authz-keycloak":true,"proxy-cache":true,"body-transformer":true,"ai-prompt-template":true,"ai-prompt-decorator":true,"proxy-mirror":true,"proxy-rewrite":true,"workflow":true,"api-breaker":true,"ai-proxy":true,"limit-conn":true,"limit-count":true,"limit-req":true,"gzip":true,"server-info":true,"traffic-split":true,"response-rewrite":true,"degraphql":true,"kafka-proxy":true,"grpc-transcode":true,"grpc-web":true,"http-dubbo":true,"public-api":true,"prometheus":true,"datadog":true,"loki-logger":true,"elasticsearch-logger":true,"echo":true,"loggly":true,"http-logger":true,"splunk-hec-logging":true,"skywalking-logger":true,"google-cloud-logging":true,"sls-logger":true,"tcp-logger":true,"kafka-logger":true,"rocketmq-logger":true,"syslog":true,"udp-logger":true,"file-logger":true,"clickhouse-logger":true,"tencent-cloud-cls":true,"inspect":true,"example-plugin":true,"aws-lambda":true,"azure-functions":true,"openwhisk":true,"openfunction":true,"error-log-logger":true,"ext-plugin-post-req":true,"ext-plugin-post-resp":true,"serverless-post-function":true,"opa":true,"forward-auth":true,"jwe-decrypt":true}, context: init_worker_by_lua*

You should also observe logs at other severity levels, such as error, emerg, and info, when they are generated.

Send Logs to Kafka over TLS

The following example sends error-level gateway logs to a TLS-enabled Kafka broker. Complete the trusted-CA setup in Send Logs to a TLS-Enabled Broker, then set the broker address and topic:

export KAFKA_TLS_HOST="kafka-tls"
export KAFKA_TLS_PORT="9093"
export KAFKA_ERROR_TOPIC="apisix-error-logs"

Create the dedicated error-log topic:

docker exec kafka-tls /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-tls:9093 \
  --command-config /etc/kafka/secrets/client.properties \
  --create \
  --if-not-exists \
  --topic "${KAFKA_ERROR_TOPIC}" \
  --partitions 1 \
  --replication-factor 1

Configure the plugin metadata:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/error-log-logger" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  --data-binary @- <<EOF
{
  "kafka": {
    "brokers": [
      {
        "host": "${KAFKA_TLS_HOST}",
        "port": ${KAFKA_TLS_PORT}
      }
    ],
    "kafka_topic": "${KAFKA_ERROR_TOPIC}",
    "tls": {
      "verify": true
    }
  },
  "level": "ERROR",
  "inactive_timeout": 1
}
EOF

To generate an error for verification, create a route whose upstream port is closed, then send a request:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "trigger-error-log",
    "uri": "/trigger-error-log",
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "127.0.0.1:1": 1
      }
    }
  }'

curl -i "http://127.0.0.1:9080/trigger-error-log"

The request should return 502 Bad Gateway. After the one-second batch timeout, consume one record from the dedicated topic:

docker exec kafka-tls /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server kafka-tls:9093 \
  --topic "${KAFKA_ERROR_TOPIC}" \
  --consumer.config /etc/kafka/secrets/client.properties \
  --from-beginning \
  --max-messages 1

The record should contain an error entry similar to the following:

connect() failed (111: Connection refused) while connecting to upstream

Kafka broker information written by the plugin to its own logs contains only host and port; SASL passwords are redacted.

Send Logs to ClickHouse

The following example demonstrates how you can configure the error-log-logger plugin to send error logs to ClickHouse.

Start a sample ClickHouse server with user default and empty password:

docker run -d -p 8123:8123 -p 9000:9000 -p 9009:9009 --name clickhouse-server clickhouse/clickhouse-server

In ClickHouse database default, create a table named default_logs with a data column. Note that the data column is expected by the plugin to push logs to.

curl "http://127.0.0.1:8123" -X POST -d '
  CREATE TABLE default.default_logs (
    data String,
    PRIMARY KEY(`data`)
  )
  ENGINE = MergeTree()
' --user default:

Configure the plugin metadata for error-log-logger:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/error-log-logger" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "clickhouse": {
      "endpoint_addr": "http://192.168.2.103:8123",
      "user": "default",
      "password": "",
      "database": "default",
      "logtable": "default_logs"
    },
    "level": "INFO"
  }'

❶ Replace with your ClickHouse server address.

❷ Set the username to default.

❸ Set the password to empty.

❹ Set the database to default.

❺ Set the database table to default_logs.

❻ Configure the severity level to INFO so most logs would be sent, for easier verification.

To verify, you can manually generate a log at warn level by reloading APISIX.

Send a request to ClickHouse to see the log entries:

echo 'SELECT * FROM default.default_logs FORMAT Pretty' | curl "http://127.0.0.1:8123/?" -d @-

You should see a log entry similar to the following:

2025/01/27 08:21:13 [warn] 353#353: *106572 [lua] plugin.lua:205: load(): new plugins: {"client-control":true,"proxy-control":true,"request-id":true,"zipkin":true,"ext-plugin-pre-req":true,"fault-injection":true,"mocking":true,"serverless-pre-function":true,"cors":true,"ip-restriction":true,"ua-restriction":true,"referer-restriction":true,"csrf":true,"uri-blocker":true,"request-validation":true,"chaitin-waf":true,"multi-auth":true,"openid-connect":true,"authz-casbin":true,"authz-casdoor":true,"wolf-rbac":true,"ldap-auth":true,"hmac-auth":true,"basic-auth":true,"jwt-auth":true,"jwe-decrypt":true,"key-auth":true,"consumer-restriction":true,"attach-consumer-label":true,"forward-auth":true,"opa":true,"authz-keycloak":true,"proxy-cache":true,"body-transformer":true,"ai-prompt-template":true,"ai-prompt-decorator":true,"proxy-mirror":true,"proxy-rewrite":true,"workflow":true,"api-breaker":true,"ai-proxy":true,"limit-conn":true,"limit-count":true,"limit-req":true,"gzip":true,"server-info":true,"traffic-split":true,"response-rewrite":true,"degraphql":true,"kafka-proxy":true,"grpc-transcode":true,"grpc-web":true,"http-dubbo":true,"public-api":true,"error-log-logger":true,"google-cloud-logging":true,"sls-logger":true,"tcp-logger":true,"kafka-logger":true,"rocketmq-logger":true,"syslog":true,"udp-logger":true,"file-logger":true,"clickhouse-logger":true,"tencent-cloud-cls":true,"inspect":true,"example-plugin":true,"aws-lambda":true,"azure-functions":true,"openwhisk":true,"openfunction":true,"serverless-post-function":true,"ext-plugin-post-req":true,"ext-plugin-post-resp":true,"redirect":true,"skywalking-logger":true,"splunk-hec-logging":true,"http-logger":true,"loggly":true,"echo":true,"elasticsearch-logger":true,"cas-auth":true,"prometheus":true,"datadog":true,"loki-logger":true,"real-ip":true,"ai":true}, context: init_worker_by_lua* │

Configuration

Previous Page

Configuration

Next Page