AI Cache Configuration

Configuration reference for the AI Cache plugin (`ai-cache`): the parameters, each attribute with its type, whether it is required, its default and its accepted values.

Parameters

See plugin common configurations (opens in Apache APISIX docs) for configuration options available to all plugins.

This plugin supports referencing sensitive parameter values from environment variables using the env:// prefix, or from a secret manager, such as HashiCorp Vault’s KV secrets engine, using the secret:// prefix. For more information, see environment variables in plugin (opens in Apache APISIX docs) and secrets (opens in Apache APISIX docs).

exact — object · optional · since APISIX 3.18.0 · API7 Gateway 3.9.16

Settings for exact-match caching, where a response is reused only when the normalized request is identical to a previously cached one.

ttl — integer · optional · default: 3600 · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: greater than or equal to 1

Time-to-live in seconds for a cached entry.

cache_key — object · optional · since APISIX 3.18.0 · API7 Gateway 3.9.16

Settings that control how the cache key is scoped.

share_across_routes — boolean · optional · default: false · since APISIX 3.18.0 · API7 Gateway 3.9.16

If true, the cache key does not include the route ID, so identical requests on different routes can share cached responses. If false, each route has its own cache scope.

include_consumer — boolean · optional · default: false · since APISIX 3.18.0 · API7 Gateway 3.9.16

If true, the consumer name is included in the cache key, so cached responses are isolated per consumer.

include_vars — array[string] · optional · default: [] · since APISIX 3.18.0 · API7 Gateway 3.9.16

Names of additional context variables to include in the cache key scope, so requests with different values of these variables do not share cached responses.

max_cache_body_size — integer · optional · default: 1048576 · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: greater than or equal to 0

Maximum size in bytes of a response body that will be cached. Larger responses are not cached.

cache_headers — boolean · optional · default: true · since APISIX 3.18.0 · API7 Gateway 3.9.16

If true, the plugin adds the X-AI-Cache-Status response header (and X-AI-Cache-Age on a cache hit). Set to false to omit these headers.

fail_mode — string · optional · default: skip · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: skip, warn, or error

Behavior when the request cannot be cached because no AI instance was selected, for example when the route does not also configure ai-proxy or ai-proxy-multi. With skip, the request is passed through unchecked. With warn, the request is passed through and a warning is logged. With error, the request is rejected with HTTP 500.

bypass_on — array[object] · optional · since APISIX 3.18.0 · API7 Gateway 3.9.16

A list of request-header matching rules. If a request matches any rule, the cache is bypassed and the response is marked with X-AI-Cache-Status set to BYPASS.

equals — string · required · since APISIX 3.18.0 · API7 Gateway 3.9.16

Value that the header must equal for the rule to match.

policy — string · optional · default: redis · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: redis

Cache storage backend. Currently only redis is supported.

layers — array[string] · optional · default: ["exact"] · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: exact, or both exact and semantic

Cache layers to enable. exact is always required. Add semantic to look for similar prompts through embeddings and vector search after an exact miss.

Semantic caching was introduced in API7 Enterprise 3.9.16 and 3.10.3, and APISIX 3.18.0.

semantic — object · optional · since APISIX 3.18.0 · API7 Gateway 3.9.16

Settings for semantic caching. Required when layers includes semantic.

Introduced in API7 Enterprise 3.9.16 and 3.10.3, and APISIX 3.18.0.

similarity_threshold — number · optional · default: 0.95 · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: between 0 and 1 inclusive

Minimum similarity score required for a semantic cache hit.

top_k — integer · optional · default: 1 · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: greater than or equal to 1

Number of nearest vector matches to retrieve.

distance_metric — string · optional · default: cosine · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: cosine

Distance metric used by vector search.

ttl — integer · optional · default: 86400 · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: greater than or equal to 1

Time-to-live in seconds for semantic cache entries.

match — object · optional · since APISIX 3.18.0 · API7 Gateway 3.9.16

Settings that control which conversation content is embedded for semantic matching.

message_countback — integer · optional · default: 1 · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: greater than or equal to 1

Number of recent user-message turns to include in the embedding input.

ignore_system_prompts — boolean · optional · default: true · since APISIX 3.18.0 · API7 Gateway 3.9.16

If true, system prompts are excluded from the embedding input.

ignore_assistant_prompts — boolean · optional · default: true · since APISIX 3.18.0 · API7 Gateway 3.9.16

If true, assistant messages are excluded from the embedding input.

ignore_tool_prompts — boolean · optional · default: true · since APISIX 3.18.0 · API7 Gateway 3.9.16

If true, tool messages are excluded from the embedding input.

embedding — object · required · since APISIX 3.18.0 · API7 Gateway 3.9.16

Embedding provider configuration. Configure exactly one of openai or azure_openai.

openai — object · optional · since APISIX 3.18.0 · API7 Gateway 3.9.16

OpenAI-compatible embedding provider settings. Requires model and api_key.

endpoint — string · optional · since APISIX 3.18.0 · API7 Gateway 3.9.16

OpenAI-compatible embedding API endpoint. When not configured, the public OpenAI embeddings endpoint is used.

model — string · required · since APISIX 3.18.0 · API7 Gateway 3.9.16

Embedding model name, such as text-embedding-3-small.

api_key — string · required · since APISIX 3.18.0 · API7 Gateway 3.9.16

API key used to authenticate with the embedding provider. The value is encrypted with AES before being stored in etcd.

dimensions — integer · optional · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: greater than or equal to 1

Number of dimensions in the embedding output. Configure this only for models that support overriding the output dimensions.

ssl_verify — boolean · optional · default: true · since APISIX 3.18.0 · API7 Gateway 3.9.16

If true, verify the embedding provider's TLS certificate.

timeout — integer · optional · default: 5000 · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: greater than or equal to 1

Timeout in milliseconds for requests to the embedding provider.

azure_openai — object · optional · since APISIX 3.18.0 · API7 Gateway 3.9.16

Azure OpenAI embedding provider settings. Requires endpoint and api_key.

endpoint — string · required · since APISIX 3.18.0 · API7 Gateway 3.9.16

Azure OpenAI embeddings endpoint.

api_key — string · required · since APISIX 3.18.0 · API7 Gateway 3.9.16

API key used to authenticate with Azure OpenAI. The value is encrypted with AES before being stored in etcd.

dimensions — integer · optional · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: greater than or equal to 1

Number of dimensions in the embedding output. Configure this only for models that support overriding the output dimensions.

ssl_verify — boolean · optional · default: true · since APISIX 3.18.0 · API7 Gateway 3.9.16

If true, verify the Azure OpenAI endpoint's TLS certificate.

timeout — integer · optional · default: 5000 · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: greater than or equal to 1

Timeout in milliseconds for requests to Azure OpenAI.

vector_search — object · required · since APISIX 3.18.0 · API7 Gateway 3.9.16

Vector search backend configuration.

redis — object · required · since APISIX 3.18.0 · API7 Gateway 3.9.16

RediSearch vector index settings.

index — string · optional · default: ai-cache · since APISIX 3.18.0 · API7 Gateway 3.9.16

Name of the RediSearch index used by semantic caching.

redis_host — string · required · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: at least 2 characters

Address of the Redis server. Required when policy is redis.

redis_port — integer · optional · default: 6379 · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: greater than or equal to 1

Port of the Redis server.

redis_username — string · optional · since APISIX 3.18.0 · API7 Gateway 3.9.16

Username for Redis authentication when using Redis ACLs.

redis_password — string · optional · since APISIX 3.18.0 · API7 Gateway 3.9.16

Password for Redis authentication.

In API7 Gateway 3.10.2 or later in the 3.10 release series, and 3.9.16 or later in the 3.9 release series, the value is encrypted with AES256 before being saved to the database.

In APISIX 3.18.0 or later, the value is encrypted with AES before being stored in etcd.

redis_database — integer · optional · default: 0 · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: greater than or equal to 0

Database number of the Redis server.

redis_timeout — integer · optional · default: 1000 · since APISIX 3.18.0 · API7 Gateway 3.9.16

Valid values: greater than or equal to 1

Timeout in milliseconds for Redis operations.

redis_ssl — boolean · optional · default: false · since APISIX 3.18.0 · API7 Gateway 3.9.16

If true, use TLS for the connection to Redis.

redis_ssl_verify — boolean · optional · default: false · since APISIX 3.18.0 · API7 Gateway 3.9.16

If true, verify the TLS certificate of the Redis server.

redis_keepalive_timeout — integer · optional · default: 10000 · since APISIX 3.18.0 · API7 Gateway 3.9.17

Valid values: greater than or equal to 1000

Keepalive timeout in milliseconds for the Redis connection pool. Available in API7 Enterprise from version 3.9.17 on the 3.9 line and from version 3.10.4 on the 3.10 line, and in APISIX from version 3.18.0.

redis_keepalive_pool — integer · optional · default: 100 · since APISIX 3.18.0 · API7 Gateway 3.9.17

Valid values: greater than or equal to 1

Keepalive pool size for Redis connections. Available in API7 Enterprise from version 3.9.17 on the 3.9 line and from version 3.10.4 on the 3.10 line, and in APISIX from version 3.18.0.