Fault Injection
The fault-injection plugin tests application resiliency by simulating controlled faults or delays, making it ideal for chaos engineering and failure condition analysis.
The fault-injection plugin is designed to test your application's resiliency by simulating controlled faults or delays. It executes before other configured plugins, ensuring that faults are applied consistently. This makes it ideal for scenarios like chaos engineering, where the behavior of your system under failure conditions is analyzed.
The plugin supports two key actions: abort, which immediately terminates a request with a specified HTTP status code (e.g., 503 Service Unavailable), skipping all subsequent plugins; and delay, which introduces a specified delay before processing the request further. These features allow you to simulate scenarios such as service outages or latency, helping you validate error-handling logic and improve system reliability.
Examples
The examples below demonstrate how you can configure the fault-injection plugin for different scenarios.
Inject Faults
The following example demonstrate how you can configure the fault-injection plugin on a route to intercept further request sending, and respond with a specific HTTP code.
Create a route using the fault-injection plugin with the abort action to respond any request with 404 and the specified response body:
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"id": "fault-injection-route",
"uri": "/anything",
"plugins": {
"fault-injection": {
"abort": {
"http_status": 404,
"body": "APISIX Fault Injection"
}
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"httpbin.org": 1
}
}
}'services:
- name: httpbin
routes:
- name: fault-injection-route
uris:
- /anything
plugins:
fault-injection:
abort:
http_status: 404
body: "APISIX Fault Injection"
upstream:
type: roundrobin
nodes:
- host: httpbin.org
port: 80
weight: 1Synchronize the configuration to the gateway:
adc sync -f adc.yamlapiVersion: v1
kind: Service
metadata:
namespace: aic
name: httpbin-external-domain
spec:
type: ExternalName
externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
namespace: aic
name: fault-injection-plugin-config
spec:
plugins:
- name: fault-injection
config:
abort:
http_status: 404
body: "APISIX Fault Injection"
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
namespace: aic
name: fault-injection-route
spec:
parentRefs:
- name: apisix
rules:
- matches:
- path:
type: Exact
value: /anything
filters:
- type: ExtensionRef
extensionRef:
group: apisix.apache.org
kind: PluginConfig
name: fault-injection-plugin-config
backendRefs:
- name: httpbin-external-domain
port: 80Apply the configuration to your cluster:
kubectl apply -f fault-injection-ic.yamlapiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
namespace: aic
name: httpbin-external-domain
spec:
ingressClassName: apisix
externalNodes:
- type: Domain
name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
namespace: aic
name: fault-injection-route
spec:
ingressClassName: apisix
http:
- name: fault-injection-route
match:
paths:
- /anything
upstreams:
- name: httpbin-external-domain
plugins:
- name: fault-injection
enable: true
config:
abort:
http_status: 404
body: "APISIX Fault Injection"Apply the configuration to your cluster:
kubectl apply -f fault-injection-ic.yamlSend a request to the route:
curl -i "http://127.0.0.1:9080/anything"You should receive an HTTP/1.1 404 Not Found response and see the following response body, without the request being forwarded to the upstream service:
APISIX Fault InjectionInject Latencies
The following example demonstrate how you can configure the fault-injection plugin on a route to inject request latencies.
Create a route using the fault-injection plugin with the delay action to delay response sending for 3 seconds:
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"id": "fault-injection-route",
"uri": "/anything",
"plugins": {
"fault-injection": {
"delay": {
"duration": 3
}
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"httpbin.org": 1
}
}
}'services:
- name: httpbin
routes:
- name: fault-injection-route
uris:
- /anything
plugins:
fault-injection:
delay:
duration: 3
upstream:
type: roundrobin
nodes:
- host: httpbin.org
port: 80
weight: 1Synchronize the configuration to the gateway:
adc sync -f adc.yamlapiVersion: v1
kind: Service
metadata:
namespace: aic
name: httpbin-external-domain
spec:
type: ExternalName
externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
namespace: aic
name: fault-injection-plugin-config
spec:
plugins:
- name: fault-injection
config:
delay:
duration: 3
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
namespace: aic
name: fault-injection-route
spec:
parentRefs:
- name: apisix
rules:
- matches:
- path:
type: Exact
value: /anything
filters:
- type: ExtensionRef
extensionRef:
group: apisix.apache.org
kind: PluginConfig
name: fault-injection-plugin-config
backendRefs:
- name: httpbin-external-domain
port: 80Apply the configuration to your cluster:
kubectl apply -f fault-injection-ic.yamlapiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
namespace: aic
name: httpbin-external-domain
spec:
ingressClassName: apisix
externalNodes:
- type: Domain
name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
namespace: aic
name: fault-injection-route
spec:
ingressClassName: apisix
http:
- name: fault-injection-route
match:
paths:
- /anything
upstreams:
- name: httpbin-external-domain
plugins:
- name: fault-injection
enable: true
config:
delay:
duration: 3Apply the configuration to your cluster:
kubectl apply -f fault-injection-ic.yamlSend a request to the route and use time command to summarize of how long the request took to complete:
time curl -i "http://127.0.0.1:9080/anything"You should receive an HTTP/1.1 200 OK response from the upstream service and see the following timing summary:
0.01s user
0.01s system
0% cpu
3.685 totalInject Faults Conditionally
The following example demonstrate how you can configure the fault-injection plugin on a route to intercept further request sending, and respond with a specific HTTP code.
Create a route using the fault-injection plugin with the abort action as such:
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"id": "fault-injection-route",
"uri": "/anything",
"plugins": {
"fault-injection": {
"abort": {
"http_status": 404,
"body": "APISIX Fault Injection",
"headers": {
"X-APISIX-Remote-Addr": "$remote_addr"
},
"vars": [
[
[ "arg_name","==","john" ]
]
]
}
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"httpbin.org": 1
}
}
}'services:
- name: httpbin
routes:
- name: fault-injection-route
uris:
- /anything
plugins:
fault-injection:
abort:
http_status: 404
body: "APISIX Fault Injection"
headers:
X-APISIX-Remote-Addr: $remote_addr
vars:
- - - arg_name
- "=="
- john
upstream:
type: roundrobin
nodes:
- host: httpbin.org
port: 80
weight: 1Synchronize the configuration to the gateway:
adc sync -f adc.yamlapiVersion: v1
kind: Service
metadata:
namespace: aic
name: httpbin-external-domain
spec:
type: ExternalName
externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
namespace: aic
name: fault-injection-plugin-config
spec:
plugins:
- name: fault-injection
config:
abort:
http_status: 404
body: "APISIX Fault Injection"
headers:
X-APISIX-Remote-Addr: $remote_addr
vars:
- - - arg_name
- "=="
- john
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
namespace: aic
name: fault-injection-route
spec:
parentRefs:
- name: apisix
rules:
- matches:
- path:
type: Exact
value: /anything
filters:
- type: ExtensionRef
extensionRef:
group: apisix.apache.org
kind: PluginConfig
name: fault-injection-plugin-config
backendRefs:
- name: httpbin-external-domain
port: 80Apply the configuration to your cluster:
kubectl apply -f fault-injection-ic.yamlapiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
namespace: aic
name: httpbin-external-domain
spec:
ingressClassName: apisix
externalNodes:
- type: Domain
name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
namespace: aic
name: fault-injection-route
spec:
ingressClassName: apisix
http:
- name: fault-injection-route
match:
paths:
- /anything
upstreams:
- name: httpbin-external-domain
plugins:
- name: fault-injection
enable: true
config:
abort:
http_status: 404
body: "APISIX Fault Injection"
headers:
X-APISIX-Remote-Addr: $remote_addr
vars:
- - - arg_name
- "=="
- johnApply the configuration to your cluster:
kubectl apply -f fault-injection-ic.yaml❶ Respond requests with HTTP status code 404.
❷ Respond requests with APISIX Fault Injection as the body.
❸ Respond requests with header X-APISIX-Remote-Addr and the IP the request originates from.
❹ Respond requests with the above specifications only if the URL parameter name value is john.
Send a request to the route with the URL parameter name being john:
curl -i "http://127.0.0.1:9080/anything?name=john"You should receive an HTTP/1.1 404 Not Found response:
HTTP/1.1 404 Not Found
...
X-APISIX-Remote-Addr: 192.168.65.1
APISIX Fault InjectionSend a request to the route with the URL parameter name being a different value:
curl -i "http://127.0.0.1:9080/anything?name=jane"You should receive an HTTP/1.1 200 OK response from the upstream service without the injections.