Kafka Logger Configuration

Parameters

See plugin common configurations for configuration options available to all plugins.

broker_list — object · optional

Deprecated. Use brokers instead. A map of Kafka broker hosts to their ports. Configure either broker_list or brokers.
brokers — array · optional

Valid values: greater than 0

List of Kafka broker nodes. Configure either brokers or the deprecated broker_list.
- host — string · required
  
  The host of Kafka broker.
- port — integer · required
  
  The port of Kafka broker.
- sasl_config — object · optional
  
  The SASL configuration of Kafka broker
  - mechanism — string · optional · default: PLAIN
    
    Valid values: PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512
    
    The mechanism of SASL configuration.
    
    The SCRAM-SHA-256 and SCRAM-SHA-512 options are available in API7 Enterprise from version 3.8.16 and APISIX from version 3.15.0.
  - user — string · required
    
    The user of SASL configuration.
  - password — string · required
    
    The password of SASL configuration. The value is encrypted with AES before being stored in etcd.
tls — object · optional

TLS configuration for connecting to Kafka brokers. Setting this object makes the plugin connect over TLS; omit it to connect in plaintext. Introduced in API7 Enterprise 3.9.17 and 3.10.4, and APISIX 3.18.0.
- verify — boolean · optional · default: false
  
  If true, verify the Kafka broker's TLS certificate against the configured trusted CA store.
  
  The default is false, so enabling tls encrypts the connection but does not authenticate the broker, which leaves it open to an active man-in-the-middle. Set it to true in production.
kafka_topic — string · required

Target topic to push the logs for organization.
producer_type — string · optional · default: async

Valid values: async or sync

Kafka producer mode. In async mode, messages are buffered locally before being sent to Kafka. In sync mode, messages are sent without using the async producer buffer.
required_acks — integer · optional · default: 1

Valid values: -1 or 1

Number of acknowledgements the leader needs to receive for the producer to consider the request complete. This controls the durability of the sent records. See Kafka documentation for more information. acks=0 is not yet supported.
api_version — integer · optional · default: 1

Valid values: 0, 1, or 2

Kafka Produce API version used to send messages to the broker. Only version 2 carries the message timestamp, allowing the broker to store it; with the default 1, messages may be recorded without a usable timestamp. Kafka 0.10 or later is required for version 2. Introduced in API7 Enterprise 3.9.14 and 3.10.1, and APISIX 3.18.0.
key — string · optional

Key used for allocating partitions for messages.
timeout — integer · optional · default: 3

Valid values: greater than 0

Timeout for the upstream to send data.
meta_format — string · optional · default: default

Valid values: default or origin

Format to collect the request information. Setting to default collects the information in JSON format and origin collects the information with the original HTTP request. See the example for more details.
log_format — object · optional

Custom log format using key-value pairs in JSON format. Values can reference built-in variables.

In APISIX from 3.15.0, log format nested structures are supported up to five levels deep. In API7 Enterprise, only flat key-value structures are supported; nested structures are not yet supported.

You can also configure log format on a global scale using the plugin metadata, which configures the log format for all kafka-logger plugin instances. If the log format configured on the individual plugin instance differs from the log format configured on plugin metadata, the log format configured on the individual plugin instance takes precedence. See the example for more details.
log_format_extra — object · optional

Additional fields to add to the default log entry, using key-value pairs in JSON format. Values can reference built-in variables. A configured field does not overwrite an existing default field. A plugin instance takes precedence over plugin metadata; setting an empty object on the instance disables the metadata value. When log_format is configured, log_format_extra is ignored. Introduced in API7 Enterprise 3.9.15 and 3.10.2, and APISIX 3.18.0.
include_req_body — boolean · optional · default: false

If true, include the request body in the log. Note that if the request body is too big to be kept in the memory, it can not be logged due to NGINX's limitations.
include_req_body_expr — array[array] · optional

An array of one or more conditions in the form of APISIX expressions. Used when the include_req_body is true. Request body would only be logged when the expressions configured here evaluate to true.
max_req_body_bytes — integer · optional · default: 524288

Valid values: greater than or equal to 1

Maximum request body allowed in bytes. Request bodies falling within this limit will be pushed to Kafka. If the size exceeds the configured value, the body will be truncated before being pushed to Kafka.
include_resp_body — boolean · optional · default: false

If true, include the response body in the log.
include_resp_body_expr — array[array] · optional

An array of one or more conditions in the form of APISIX expressions. Used when the include_resp_body is true. Response body would only be logged when the expressions configured here evaluate to true.
max_resp_body_bytes — integer · optional · default: 524288

Valid values: greater than or equal to 1

Maximum response body allowed in bytes. Response bodies falling within this limit will be pushed to Kafka. If the size exceeds the configured value, the body will be truncated before being pushed to Kafka.
cluster_name — integer · optional · default: 1

Valid values: greater than or equal to 1

Name of the cluster. Used when there are two or more Kafka clusters. Only works if producer_type is set to async.
producer_batch_num — integer · optional · default: 200

Valid values: greater than or equal to 1

The number of messages to send in one batch. Same as the batch_num parameter in lua-resty-kafka.
producer_batch_size — integer · optional · default: 1048576

Valid values: greater than or equal to 0

The size of the TCP send buffer to use when sending data. Same as the batch_size parameter in lua-resty-kafka, but in bytes.
producer_max_buffering — integer · optional · default: 50000

Valid values: greater than or equal to 1

Maximum number of Kafka producer messages that the async producer can buffer locally. Same as the max_buffering parameter in lua-resty-kafka. This buffer is not capped by max_pending_entries. Memory usage also depends on worker count, batch_max_size, log format, and whether request or response bodies are logged.
producer_time_linger — integer · optional · default: 1

Valid values: greater than or equal to 1

Flush time. Same as the flush_time parameter in lua-resty-kafka, but in seconds.
meta_refresh_interval — integer · optional · default: 30

Valid values: greater than or equal to 1

Time interval to auto refresh the metadata. Same as the refresh_interval parameter in lua-resty-kafka, but in seconds.
name — string · optional · default: kafka logger

Unique identifier of the plugin for the batch processor. If you use Prometheus to monitor APISIX metrics, the name is exported in apisix_batch_process_entries.
batch_max_size — integer · optional · default: 1000

Valid values: greater than 0

The number of log entries allowed in one batch. Once reached, the batch will be sent to the logging service. Setting this parameter to 1 means immediate processing.
inactive_timeout — integer · optional · default: 5

Valid values: greater than 0

The maximum time in seconds to wait for new logs before sending the batch to the logging service. The value should be smaller than buffer_duration.
buffer_duration — integer · optional · default: 60

Valid values: greater than 0

The maximum time in seconds from the earliest entry allowed before sending the batch to the logging service.
retry_delay — integer · optional · default: 1

Valid values: greater than or equal to 0

The time interval in seconds to retry sending the batch to the logging service if the batch was not successfully sent.
max_retry_count — integer · optional · default: 0

Valid values: greater than or equal to 0

The maximum number of unsuccessful retries allowed before dropping the log entries.

Plugin Metadata

log_format — object · optional

Custom log format using key-value pairs in JSON format. Values can reference built-in variables.

In APISIX from 3.15.0, log format nested structures are supported up to five levels deep. In API7 Enterprise, only flat key-value structures are supported; nested structures are not yet supported.
log_format_extra — object · optional

Additional fields to add to the default log entry, using key-value pairs in JSON format. Values can reference built-in variables. A configured field does not overwrite an existing default field. A plugin instance takes precedence over plugin metadata; setting an empty object on the instance disables the metadata value. When log_format is configured, log_format_extra is ignored. Introduced in API7 Enterprise 3.9.15 and 3.10.2, and APISIX 3.18.0.
max_pending_entries — integer · optional · default: 8192 in APISIX 3.18.0 and in API7 Enterprise 3.9.19 and 3.10.6; none in API7 Enterprise 3.9.18 and 3.10.5

Valid values: greater than or equal to 1

Maximum number of entries waiting in the batch processor. New entries are discarded when the backlog reaches the limit. This setting does not limit the async Kafka producer's local buffer.

The default changed to 8192 in APISIX 3.18.0 and in API7 Enterprise 3.9.19 on the 3.9 line and 3.10.6 on the 3.10 line. In API7 Enterprise 3.9.18 and 3.10.5, and in earlier APISIX versions, omitting the parameter leaves the backlog unlimited.

See Batch Processor for sizing and verification guidance.