AI Proxy Multi Configuration
Configuration reference for the AI Proxy Multi plugin (`ai-proxy-multi`): the static configurations and parameters, each attribute with its type, whether it is required, its default and its accepted values.
Static Configurations
APISIX 3.18.0 uses ngx_http_ffi_client by default for upstream requests from ai-proxy, ai-proxy-multi, and ai-request-rewrite. Set http_client to lua-resty-http to use the Lua client instead. API7 Gateway 3.9 and 3.10 use the Lua client and do not expose this setting.
To use the Lua client in an APISIX host or Docker deployment, configure the following setting:
plugin_attr:
ai-proxy:
http_client: lua-resty-httpThen reload APISIX for the change to take effect.
Export the full effective values for the installed APISIX release:
helm get values <release-name> -n <namespace> --all -o yaml > values.yamlAdd or update the following value:
apisix:
pluginAttrs:
ai-proxy:
http_client: lua-resty-httpThen apply the values file with the chart used for this APISIX release:
helm upgrade <release-name> apisix/apisix -n <namespace> -f values.yamlParameters
See plugin common configurations (opens in Apache APISIX docs) for configuration options available to all plugins.
fallback_strategy — string or array · optional · since APISIX 3.12.0 · API7 Gateway 3.7.0
Valid values: string: instance_health_and_rate_limiting, http_429, or http_5xx
array: Any combination of rate_limiting, http_429, and http_5xx
Fallback strategy. The option instance_health_and_rate_limiting is kept for backward compatibility and is functionally the same as rate_limiting.
With rate_limiting or instance_health_and_rate_limiting, when the current instance's quota is exhausted, the request is forwarded to the next instance regardless of priority. With http_429, if an instance returns status code 429, the request is retried with other instances. With http_5xx, if an instance returns a 5xx status code, the request is retried with other instances. If all instances fail, the plugin returns the last upstream status, response body, and Content-Type.
When not set, the plugin will not forward the request to low priority instances when tokens of the high priority instance are exhausted.
max_retries — integer · optional · since APISIX 3.17.0 · API7 Gateway 3.9.14
Valid values: greater than or equal to 0
Maximum number of fallback retries after the initial request fails. This bounds how many additional instances a single request tries, so it does not exhaust every configured instance. Only takes effect together with fallback_strategy. When not set, there is no explicit cap and the plugin retries until an instance succeeds or all instances have been tried. Available in API7 Enterprise from version 3.9.14 and APISIX from version 3.17.0.
retry_on_failure_within_ms — integer · optional · since APISIX 3.17.0 · API7 Gateway 3.9.14
Valid values: greater than or equal to 1
Only fall back to another instance when the upstream fails within this many milliseconds. Fast failures (such as connection errors and quick 429 or 5xx responses) are retried, while a slow failure that takes longer than this is returned to the client directly to avoid doubling the total wait time. Only takes effect together with fallback_strategy. When not set, the plugin retries regardless of how long the failed attempt took. Available in API7 Enterprise from version 3.9.14 and APISIX from version 3.17.0.
fallback_http_statuses — array[integer] · optional · since API7 Gateway 3.9.19
Valid values: each between 400 and 599, no duplicates
Additional upstream HTTP status codes that make the request fall back to another instance, on top of the http_429 and http_5xx entries of fallback_strategy. Use it for statuses that mean the instance's own credential or quota is the problem rather than the request, such as 401 for an expired key or 403 for a disabled account, so the request is retried elsewhere instead of being returned to the client. Only takes effect together with fallback_strategy. Available in API7 Enterprise from version 3.9.19 on the 3.9 line and from version 3.10.6 on the 3.10 line.
balancer — object · optional · since APISIX 3.12.0 · API7 Gateway 3.7.0
Load balancing configurations.
algorithm — string · optional · default: roundrobin · since APISIX 3.12.0 · API7 Gateway 3.7.0
Valid values: roundrobin, chash, or semantic
Load balancing algorithm. When set to roundrobin, weighted round robin algorithm is used. When set to chash, consistent hashing algorithm is used. When set to semantic, the instance whose examples are semantically closest to the prompt is used, configured under semantic_opts.
The semantic algorithm does not participate in health checks, fallback_strategy, or max_retries. An upstream failure on the selected instance is returned to the client; the algorithm falls back only when no instance clears its threshold or the embedding request fails.
The semantic algorithm is available in API7 Enterprise from version 3.9.18 on the 3.9 line and from version 3.10.5 on the 3.10 line, and in APISIX from version 3.18.0.
hash_on — string · optional · default: vars · since APISIX 3.12.0 · API7 Gateway 3.7.0
Valid values: vars, header, cookie, consumer, or vars_combinations
Used when type is chash. Support hashing on built-in variables (opens in Apache APISIX docs), header, cookie, consumer, or a combination of built-in variables (opens in Apache APISIX docs).
key — string · optional · since APISIX 3.12.0 · API7 Gateway 3.7.0
Used when type is chash. When hash_on is set to header or cookie, key is required. When hash_on is set to consumer, key is not required as the consumer name will be used as the key automatically.
semantic_opts — object · optional · since APISIX 3.18.0 · API7 Gateway 3.9.18
Configurations for the semantic balancer algorithm. Required when balancer.algorithm is semantic, and ignored otherwise.
Available in API7 Enterprise from version 3.9.18 on the 3.9 line and from version 3.10.5 on the 3.10 line, and in APISIX from version 3.18.0.
embeddings — object · required · since APISIX 3.18.0 · API7 Gateway 3.9.18
Embedding service used to turn the prompt and each instance's examples into vectors. The prompt is embedded on every request, so this service is on the request path.
provider — string · required · since APISIX 3.18.0 · API7 Gateway 3.9.18
Valid values: openai or azure-openai
Embedding service provider.
model — string · required · since APISIX 3.18.0 · API7 Gateway 3.9.18
Name of the embedding model, such as text-embedding-3-small.
endpoint — string · optional · since APISIX 3.18.0 · API7 Gateway 3.9.18
Embedding API endpoint. Optional for openai, which defaults to the public API. Required for azure-openai, where it has to be the full URL, such as https://{resource}.openai.azure.com/openai/deployments/{deployment}/embeddings?api-version={version}.
auth — object · required · since APISIX 3.18.0 · API7 Gateway 3.9.18
Authentication for the embedding service, carried either as headers or as query parameters.
header — object · optional · since APISIX 3.18.0 · API7 Gateway 3.9.18
Key-value pairs sent as request headers to the embedding service.
query — object · optional · since APISIX 3.18.0 · API7 Gateway 3.9.18
Key-value pairs sent as query parameters to the embedding service.
timeout — integer · optional · default: 3000 · since APISIX 3.18.0 · API7 Gateway 3.9.18
Valid values: greater than or equal to 1
Timeout in milliseconds for an embedding request. Because the prompt is embedded synchronously, this bounds the latency added to each request when the embedding service is slow. On a timeout the request is routed to the fallback instance rather than failed.
ssl_verify — boolean · optional · default: true · since APISIX 3.18.0 · API7 Gateway 3.9.18
If true, verify the embedding service's TLS certificate.
threshold — number · optional · default: 0 · since APISIX 3.18.0 · API7 Gateway 3.9.18
Valid values: between -1 and 1 inclusive
Global minimum cosine similarity an instance has to reach to be selected. An instance's own threshold overrides this value. The default of 0 admits almost any prompt, so the fallback instance is only ever reached once a threshold above 0 is set.
fallback — string · optional · since APISIX 3.18.0 · API7 Gateway 3.9.18
Name of the instance to route to when no instance clears its threshold or the embedding request fails. It is otherwise a normally ranked instance and needs its own examples. Defaults to the first instance when unset.
debugging — boolean · optional · default: false · since APISIX 3.18.0 · API7 Gateway 3.9.18
If true, return the per-instance similarity scores and the routing decision in the X-AI-Semantic-Scores and X-AI-Semantic-Picked-Instance response headers. Intended for tuning examples and thresholds, not for production traffic.
instances — array[object] · required · since APISIX 3.12.0 · API7 Gateway 3.7.0
LLM instance configurations.
name — string · required · since APISIX 3.12.0 · API7 Gateway 3.7.0
Name of the LLM service instance.
examples — array[string] · optional · since APISIX 3.18.0 · API7 Gateway 3.9.18
Valid values: between 1 and 64 items
Example utterances representing what this instance handles. Each one is embedded into its own reference vector, and the semantic balancer routes a request to the instance whose closest example is most similar to the prompt.
Required for every instance when balancer.algorithm is semantic, including the instance named by semantic_opts.fallback. Ignored by the other algorithms.
Available in API7 Enterprise from version 3.9.18 on the 3.9 line and from version 3.10.5 on the 3.10 line, and in APISIX from version 3.18.0.
threshold — number · optional · since APISIX 3.18.0 · API7 Gateway 3.9.18
Valid values: between -1 and 1 inclusive
Minimum cosine similarity a prompt has to reach for this instance to be selected by the semantic balancer. Overrides semantic_opts.threshold for this instance.
Available in API7 Enterprise from version 3.9.18 on the 3.9 line and from version 3.10.5 on the 3.10 line, and in APISIX from version 3.18.0.
provider — string · required · since APISIX 3.12.0 · API7 Gateway 3.7.0
Valid values: openai, deepseek, azure-openai, aimlapi, gemini, vertex-ai, anthropic, openrouter, bedrock, openai-compatible
LLM service provider.
When set to openai, the plugin sends detected Chat Completions, Responses API, and Embeddings requests to their corresponding OpenAI endpoints.
When set to deepseek, the plugin will proxy requests to https://api.deepseek.com/chat/completions.
When set to gemini (available from APISIX 3.15.0 and Enterprise 3.9.2), the plugin will proxy requests to https://generativelanguage.googleapis.com/v1beta/openai/chat/completions. If you are proxying requests to an embedding model, you should configure the embedding model endpoint in the override.
When set to vertex-ai (available from APISIX 3.15.0 and Enterprise 3.9.2), the plugin proxies requests to Google Cloud Vertex AI. For chat completions, the plugin will proxy requests to https://{region}-aiplatform.googleapis.com/v1beta1/projects/{project_id}/locations/{region}/endpoints/openapi/chat/completions. For embeddings, the plugin will proxy requests to https://{region}-aiplatform.googleapis.com/v1/projects/{project_id}/locations/{region}/publishers/google/models/{model}:predict. These require configuring provider_conf with project_id and region. Alternatively, you can configure override for a custom endpoint.
When set to anthropic (available from APISIX 3.15.0 and Enterprise 3.9.2), the plugin sends detected Chat Completions requests to https://api.anthropic.com/v1/chat/completions and native Anthropic Messages requests to https://api.anthropic.com/v1/messages.
When set to openrouter (available from APISIX 3.15.0 and Enterprise 3.9.2), the plugin will proxy requests to https://openrouter.ai/api/v1/chat/completions.
When set to bedrock (available from API7 Enterprise 3.9.12 and APISIX 3.17.0), the plugin proxies requests to AWS Bedrock using the Converse API.
When set to aimlapi (available from APISIX 3.14.0 and Enterprise 3.8.17), the plugin uses the OpenAI-compatible driver and proxies the request to https://api.aimlapi.com/v1/chat/completions.
When set to openai-compatible, the plugin proxies requests to the custom endpoint configured in override.
When set to azure-openai, the plugin also proxies requests to the custom endpoint configured in override and additionally removes the model parameter from user requests.
priority — integer · optional · default: 0 · since APISIX 3.12.0 · API7 Gateway 3.7.0
Priority of the LLM instance in load balancing. priority takes precedence over weight.
weight — integer · required · since APISIX 3.12.0 · API7 Gateway 3.7.0
Valid values: greater than or equal to 0
Weight of the LLM instance in load balancing.
auth — object · required · since APISIX 3.12.0 · API7 Gateway 3.7.0
Authentication configurations.
header — object · optional · since APISIX 3.12.0 · API7 Gateway 3.7.0
Authentication headers. At least one of the header and query should be configured. You can configure additional custom headers that will be forwarded to the upstream LLM service.
query — object · optional · since APISIX 3.12.0 · API7 Gateway 3.7.0
Authentication query parameters. At least one of the header and query should be configured.
gcp — object · optional · since APISIX 3.15.0 · API7 Gateway 3.9.2
GCP service account authentication for Vertex AI. Available in API7 Enterprise from 3.9.2 and APISIX from version 3.17.0.
service_account_json — string · optional · since APISIX 3.15.0 · API7 Gateway 3.9.2
GCP service account JSON content used for authentication. This can be configured using this parameter or by setting the GCP_SERVICE_ACCOUNT environment variable.
max_ttl — integer · optional · min: 1 · since APISIX 3.15.0 · API7 Gateway 3.9.2
Maximum TTL for GCP access token caching, in seconds.
expire_early_secs — integer · optional · default: 60 · min: 0 · since APISIX 3.15.0 · API7 Gateway 3.9.2
Number of seconds to expire the access token before its actual expiration time. This prevents edge cases where tokens expire during active requests.
aws — object · optional · since APISIX 3.17.0 · API7 Gateway 3.9.11
AWS IAM credentials for SigV4 signing. Required when provider is bedrock (for Bedrock, auth.aws is sufficient and auth.header/auth.query are not required). Available in API7 Enterprise from version 3.9.12 and APISIX from version 3.17.0.
access_key_id — string · required · since APISIX 3.17.0 · API7 Gateway 3.9.11
AWS IAM access key ID.
secret_access_key — string · required · since APISIX 3.17.0 · API7 Gateway 3.9.11
AWS IAM secret access key.
session_token — string · optional · since APISIX 3.17.0 · API7 Gateway 3.9.11
AWS session token for temporary credentials (e.g. from STS AssumeRole).
options — object · optional · since APISIX 3.12.0 · API7 Gateway 3.7.0
Model configurations.
In addition to model, you can configure additional parameters and they will be forwarded to the upstream LLM service in the request body. For instance, if you are working with OpenAI or DeepSeek, you can configure additional parameters such as max_tokens, temperature, top_p, and stream. See your LLM provider's API documentation for more available options.
model — string · optional · since APISIX 3.12.0 · API7 Gateway 3.7.0
Name of the LLM model, such as gpt-4 or gpt-3.5. See your LLM provider's API documentation for more available models.
provider_conf — object · optional · since APISIX 3.15.0 · API7 Gateway 3.9.2
Provider-specific configuration. Required when provider is bedrock. When provider is vertex-ai, configure either provider_conf or override.endpoint.
Available in API7 Enterprise from 3.9.2 and APISIX from version 3.17.0.
project_id — string · optional · since APISIX 3.15.0 · API7 Gateway 3.9.2
Google Cloud Project ID for Vertex AI.
region — string · required · since APISIX 3.15.0 · API7 Gateway 3.9.2
Cloud region. For vertex-ai, this is the GCP region. For bedrock, this is the AWS region (e.g. us-east-1).
override — object · optional · since APISIX 3.12.0 · API7 Gateway 3.7.0
Override setting.
endpoint — string · optional · since APISIX 3.12.0 · API7 Gateway 3.7.0
LLM provider endpoint to replace the endpoint selected for the detected request protocol.
llm_options — object · optional · since APISIX 3.17.0 · API7 Gateway 3.9.10
Provider-aware LLM option overrides. Available in API7 Enterprise from version 3.9.10 and APISIX from version 3.17.0.
max_tokens — integer · optional · min: 1 · since APISIX 3.17.0 · API7 Gateway 3.9.10
Maximum number of output tokens. The gateway automatically maps this to the correct field name for the target provider, such as max_completion_tokens for OpenAI Chat or max_output_tokens for OpenAI Responses API, and overwrites the client value.
request_body — object · optional · since APISIX 3.17.0 · API7 Gateway 3.9.10
Per target-protocol request body overrides. Keys are target protocol names, such as openai-chat, openai-responses, openai-embeddings, anthropic-messages, bedrock-converse, and passthrough. Values are partial request bodies that are deep-merged into the outgoing body. Available in API7 Enterprise from version 3.9.10 and APISIX from version 3.17.0.
request_body_force_override — boolean · optional · default: false · since APISIX 3.17.0 · API7 Gateway 3.9.10
When false (default), client request body fields take priority and request_body override values only fill in missing fields. When true, request_body override values overwrite client fields. Available in API7 Enterprise from version 3.9.10 and APISIX from version 3.17.0.
checks — object · optional · since APISIX 3.14.0 · API7 Gateway 3.7.0
Health check configurations.
Note that at the moment, OpenAI and DeepSeek do not provide an official health check endpoint. Other LLM services that you can configure under openai-compatible provider may have available health check endpoints.
active — object · required · since APISIX 3.14.0 · API7 Gateway 3.7.0
Active health check configurations.
type — string · optional · default: http · since APISIX 3.14.0 · API7 Gateway 3.7.0
Valid values: http, https, or tcp
Type of health check connection.
timeout — number · optional · default: 1 · since APISIX 3.14.0 · API7 Gateway 3.7.0
Health check timeout in seconds.
concurrency — integer · optional · default: 10 · since APISIX 3.14.0 · API7 Gateway 3.7.0
Number of upstream nodes to be checked at the same time.
host — string · optional · since APISIX 3.14.0 · API7 Gateway 3.7.0
HTTP host.
port — integer · optional · since APISIX 3.14.0 · API7 Gateway 3.7.0
Valid values: between 1 and 65535 inclusive
HTTP port.
http_path — string · optional · default: / · since APISIX 3.14.0 · API7 Gateway 3.7.0
Path for HTTP probing requests.
http_method — string · optional · default: GET · since APISIX 3.18.0 · API7 Gateway 3.8.0
Valid values: CONNECT, DELETE, GET, HEAD, OPTIONS, PATCH, POST, PURGE, PUT, or TRACE
HTTP method for active health check probing requests. Available in API7 Enterprise and in APISIX from version 3.18.0.
http_req_body — string · optional · since APISIX 3.18.0 · API7 Gateway 3.9.11
Request body to send in active health check probing requests. This is useful when http_method is set to POST. Defaults to empty string. Available in API7 Enterprise and in APISIX from version 3.18.0.
https_verify_certificate — boolean · optional · default: true · since APISIX 3.14.0 · API7 Gateway 3.7.0
If true, verify the node's TLS certificate.
healthy — object · optional · since APISIX 3.14.0 · API7 Gateway 3.7.0
Healthy check configurations.
interval — integer · optional · default: 1 · since APISIX 3.14.0 · API7 Gateway 3.7.0
Time interval of checking healthy nodes, in seconds.
http_statuses — array[integer] · optional · default: [200,302] · since APISIX 3.14.0 · API7 Gateway 3.7.0
Valid values: status code between 200 and 599 inclusive
An array of HTTP status codes that defines a healthy node.
successes — integer · optional · default: 2 · since APISIX 3.14.0 · API7 Gateway 3.7.0
Valid values: between 1 and 254 inclusive
Number of successful probes to define a healthy node.
req_headers — array[string] · optional · since APISIX 3.14.0 · API7 Gateway 3.7.0
List of additional HTTP headers to send in health check probing requests, in "Header: Value" format.
unhealthy — object · optional · since APISIX 3.14.0 · API7 Gateway 3.7.0
Unhealthy check configurations.
interval — integer · optional · default: 1 · since APISIX 3.14.0 · API7 Gateway 3.7.0
Time interval of checking unhealthy nodes, in seconds.
http_statuses — array[integer] · optional · default: [429,404,500,501,502,503,504,505] · since APISIX 3.14.0 · API7 Gateway 3.7.0
Valid values: status code between 200 and 599 inclusive
An array of HTTP status codes that defines an unhealthy node.
http_failures — integer · optional · default: 5 · since APISIX 3.14.0 · API7 Gateway 3.7.0
Valid values: between 1 and 254 inclusive
Number of HTTP failures to define an unhealthy node.
tcp_failures — integer · optional · default: 2 · since APISIX 3.14.0 · API7 Gateway 3.7.0
Valid values: between 1 and 254 inclusive
Number of TCP failures to define an unhealthy node.
timeouts — integer · optional · default: 3 · since APISIX 3.14.0 · API7 Gateway 3.7.0
Valid values: between 1 and 254 inclusive
Number of probe timeouts to define an unhealthy node.
logging — object · optional · since APISIX 3.15.0 · API7 Gateway 3.8.0
Logging configurations. These configurations apply to access logs and logs sent to logging plugins, and do not affect the error log.
summaries — boolean · optional · default: false · since APISIX 3.15.0 · API7 Gateway 3.8.0
If true, add an llm_summary object to logger entries with model, latency, and token usage. In API7 Enterprise 3.9.18 and 3.10.5, and in APISIX 3.18.0, the summary also includes stream status, tool count and usage, end-user ID, cache read and creation tokens, reasoning tokens, and content risk level when available.
payloads — boolean · optional · default: false · since APISIX 3.15.0 · API7 Gateway 3.8.0
If true, log request and response payload.
timeout — integer · optional · default: 30000 · since APISIX 3.12.0 · API7 Gateway 3.7.0
Valid values: between 1 and 600000 inclusive
Timeout in milliseconds for each connect, send, or blocking read operation to the LLM service. It does not limit the total duration of a streaming response; use max_stream_duration_ms for that limit.
max_req_body_size — integer · optional · default: 67108864 · since APISIX 3.17.0 · API7 Gateway 3.9.14
Valid values: greater than or equal to 1
Maximum request body size in bytes that the plugin reads into memory. Larger requests are rejected with HTTP 413. This prevents unbounded memory buffering of large request bodies. The default is 67108864 bytes (64 MiB). Available in API7 Enterprise from versions 3.9.14 and 3.10.1 in their respective release lines, and APISIX from version 3.17.0.
max_stream_duration_ms — integer · optional · since APISIX 3.17.0 · API7 Gateway 3.9.10
Valid values: greater than or equal to 1
Maximum wall-clock duration, in milliseconds, for a streaming AI response. The limit is optional. When reached, the gateway closes the connection; if output has already started, the stream ends without a protocol terminator such as [DONE], message_stop, or response.completed. Enforcement occurs between upstream reads, so the final chunk can exceed the configured duration. Available in API7 Enterprise from version 3.9.10 and APISIX from version 3.17.0.
max_response_bytes — integer · optional · since APISIX 3.17.0 · API7 Gateway 3.9.10
Valid values: greater than or equal to 1
Maximum total bytes read from the upstream for one streaming or non-streaming AI response. The limit is optional and checked between upstream reads, so the final chunk can exceed it. If the limit is exceeded before output starts, the gateway returns 502 Bad Gateway; after output starts, the gateway closes the stream without a protocol terminator. Available in API7 Enterprise from version 3.9.10 and APISIX from version 3.17.0.
streaming_flush_interval_ms — integer · optional · default: 10 · since APISIX 3.17.0 · API7 Gateway 3.9.13
Valid values: greater than or equal to 0
Background flush interval in milliseconds for streaming responses. A positive value periodically flushes buffered output to bound client latency when the upstream sends tokens in bursts. Set to 0 to flush each chunk synchronously. Available in API7 Enterprise from version 3.9.13 and APISIX from version 3.17.0.
keepalive — boolean · optional · default: true · since APISIX 3.12.0 · API7 Gateway 3.7.0
If true, keep the connection alive when requesting the LLM service.
keepalive_timeout — integer · optional · default: 60000 · since APISIX 3.13.0 · API7 Gateway 3.8.0
Valid values: greater than or equal to 1000
Keepalive timeout in milliseconds when requesting the LLM service.
keepalive_pool — integer · optional · default: 30 · since APISIX 3.12.0 · API7 Gateway 3.7.0
Valid values: greater than or equal to 1
Keepalive pool size for when connecting with the LLM service.
ssl_verify — boolean · optional · default: true · since APISIX 3.12.0 · API7 Gateway 3.7.0
If true, verify the LLM service's certificate.