Configure Fault Injection

Inject deterministic HTTP errors and delays, then choose a safe sampling percentage.

Goal

Create two independent routes that inject a synthetic 503 response or a two-second delay. Validate both actions deterministically with percentage: 100, then understand how lower percentages sample requests during a controlled resilience test.

Before you begin

Use fault injection only in a controlled environment where synthetic errors and latency cannot affect unrelated traffic.

Set the variables used in this guide:

export API7_ADMIN_URL="https://localhost:7443"
export API7_GATEWAY_GROUP_ID="default"   # only the group the quickstart creates; otherwise copy the gateway group's UUID from Gateway Groups in the Dashboard
read -rp "Gateway group name for ADC (if used): " API7_GATEWAY_GROUP_NAME && export API7_GATEWAY_GROUP_NAME
export GATEWAY_URL="http://127.0.0.1:9080"
read -rsp "Dashboard token: " API7_ADMIN_KEY && export API7_ADMIN_KEY && echo

Confirm that the control plane and selected gateway group are ready:

curl -ksS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}\n' "${API7_ADMIN_URL}/api/gateway_groups" -H "X-API-KEY: ${API7_ADMIN_KEY}"
curl -ksS --connect-timeout 3 --max-time 10 "${API7_ADMIN_URL}/api/gateway_groups/${API7_GATEWAY_GROUP_ID}/instances" -H "X-API-KEY: ${API7_ADMIN_KEY}" | grep -o '"status": *"Healthy"' | wc -l

Expected: 200, then 1 or more.

The gateway can resolve and reach httpbin.org:443. The abort action returns before proxying, but the delay route uses this public demonstration service; its availability is outside API7 Gateway's control.
curl and awk are available.
For ADC, confirm label-selector support and reserve the working files:
```
adc --version
adc sync --help | grep -- '--label-selector'
for file in fault-injection-adc.yaml fault-injection-cleanup.yaml; do test ! -e "${file}" || { echo "Stop: ${file} already exists"; exit 1; }; done
```
ADC resolves --gateway-group by display name; the compatibility Admin API uses the group's UUID. Enter the display name of the group identified by API7_GATEWAY_GROUP_ID.

This guide creates one service and two routes. Check every fixed Admin API identifier before using PUT:

curl -ksS --connect-timeout 3 --max-time 10 -w '\nservice: %{http_code}\n' "${API7_ADMIN_URL}/apisix/admin/services/fault-injection-guide-service-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -H "X-API-KEY: ${API7_ADMIN_KEY}"
for id in fault-injection-abort-id fault-injection-delay-id; do
  curl -ksS --connect-timeout 3 --max-time 10 -w "\nroute ${id}: %{http_code}\n" "${API7_ADMIN_URL}/apisix/admin/routes/${id}?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -H "X-API-KEY: ${API7_ADMIN_KEY}"
done

Expected: every resource is absent. A 200 response containing a resource means the ID is in use; stop or change every occurrence. Correct authentication, group, and request errors until absence is unambiguous because PUT replaces a resource at the same ID. The Admin API URL IDs deliberately differ from the name fields below. ADC also treats a display name and backend ID as separate fields; do not infer one from the other. For ADC, continue only if its diff shows creates for exactly the labeled resources in this guide.

Steps

Complete steps 1 and 2 for the Admin API path, or skip them and complete step 3 for ADC. Do not apply both paths.

Step 1: Create the upstream service

curl -ksS --connect-timeout 3 --max-time 15 "${API7_ADMIN_URL}/apisix/admin/services/fault-injection-guide-service-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -X PUT \
  -H "X-API-KEY: ${API7_ADMIN_KEY}" -H "Content-Type: application/json" \
  -d '{
    "name": "fault-injection-guide-service",
    "labels": {"guide": "fault-injection"},
    "upstream": {
      "name": "default", "type": "roundrobin", "scheme": "https", "pass_host": "node",
      "nodes": [{"host": "httpbin.org", "port": 443, "weight": 100}]
    }
  }'

What changed: the service points to the HTTPS demonstration upstream. No public route uses it yet.

Step 2: Create deterministic abort and delay routes

curl -ksS --connect-timeout 3 --max-time 15 "${API7_ADMIN_URL}/apisix/admin/routes/fault-injection-abort-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -X PUT \
  -H "X-API-KEY: ${API7_ADMIN_KEY}" -H "Content-Type: application/json" \
  -d '{
    "name":"fault-injection-abort","labels":{"guide":"fault-injection"},"paths":["/fault/abort"],"methods":["GET"],"service_id":"fault-injection-guide-service-id",
    "plugins":{"fault-injection":{"abort":{"http_status":503,"body":"Service Temporarily Unavailable","percentage":100}}}
  }'

curl -ksS --connect-timeout 3 --max-time 15 "${API7_ADMIN_URL}/apisix/admin/routes/fault-injection-delay-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -X PUT \
  -H "X-API-KEY: ${API7_ADMIN_KEY}" -H "Content-Type: application/json" \
  -d '{
    "name":"fault-injection-delay","labels":{"guide":"fault-injection"},"paths":["/fault/delay"],"methods":["GET"],"service_id":"fault-injection-guide-service-id",
    "plugins":{"proxy-rewrite":{"uri":"/get"},"fault-injection":{"delay":{"duration":2,"percentage":100}}}
  }'

What changed: every request to /fault/abort receives a gateway-generated 503 with the configured body and does not reach the upstream. Every request to /fault/delay waits two seconds at the gateway before continuing to httpbin /get. abort returns an HTTP response; it does not reset the transport connection.

Step 3: Apply the complete desired state with ADC

Skip this alternative if you completed steps 1 and 2. Save the service and both routes in one file so each sync describes the complete selected state.

fault-injection-adc.yaml

services:
  - name: fault-injection-guide-service
    labels: {guide: fault-injection}
    upstream:
      name: default
      scheme: https
      pass_host: node
      nodes:
        - {host: httpbin.org, port: 443, weight: 100}
    routes:
      - name: fault-injection-abort
        labels: {guide: fault-injection}
        uris: [/fault/abort]
        methods: [GET]
        plugins:
          fault-injection:
            abort:
              http_status: 503
              body: Service Temporarily Unavailable
              percentage: 100
      - name: fault-injection-delay
        labels: {guide: fault-injection}
        uris: [/fault/delay]
        methods: [GET]
        plugins:
          proxy-rewrite: {uri: /get}
          fault-injection:
            delay:
              duration: 2
              percentage: 100

Lint, validate, inspect, and then apply the reviewed diff:

adc lint -f fault-injection-adc.yaml
adc validate -f fault-injection-adc.yaml --backend api7ee --server "${API7_ADMIN_URL}" --token "${API7_ADMIN_KEY}" --gateway-group "${API7_GATEWAY_GROUP_NAME}" --tls-skip-verify --label-selector guide=fault-injection
adc diff -f fault-injection-adc.yaml --backend api7ee --server "${API7_ADMIN_URL}" --token "${API7_ADMIN_KEY}" --gateway-group "${API7_GATEWAY_GROUP_NAME}" --tls-skip-verify --label-selector guide=fault-injection
adc sync -f fault-injection-adc.yaml --backend api7ee --server "${API7_ADMIN_URL}" --token "${API7_ADMIN_KEY}" --gateway-group "${API7_GATEWAY_GROUP_NAME}" --tls-skip-verify --label-selector guide=fault-injection

Expected: lint and validation succeed; the diff creates only the labeled service and its two labeled routes; sync affects only that reviewed set.

Step 4: Choose a sampling percentage

The deterministic checks use 100, meaning every request receives the action. After those checks pass, you can change either percentage to an integer from 0 through 100 and reapply the same complete configuration. 0 selects no requests. If percentage is omitted, the action runs for every matching request.

A value such as 10 independently samples each request; it does not guarantee one fault in every ten requests or any fault in a small batch. Treat the observed share over a sufficiently large test as statistical evidence, not an exact acceptance assertion. Keep delay and abort on separate routes. When both actions are configured on one route, they are sampled independently and a selected delay runs before a selected abort; their percentages do not divide traffic into complementary groups.

Validate

Assert the abort status and body with two requests. Both are deterministic because the percentage is 100:

ABORT_STATUS=$(curl -sS --connect-timeout 3 --max-time 15 -o /dev/null -w '%{http_code}' "${GATEWAY_URL}/fault/abort")
test "${ABORT_STATUS}" = "503" || { echo "Expected 503, got ${ABORT_STATUS}"; exit 1; }
ABORT_BODY=$(curl -sS --connect-timeout 3 --max-time 15 "${GATEWAY_URL}/fault/abort")
test "${ABORT_BODY}" = "Service Temporarily Unavailable" || { echo "Unexpected abort body"; exit 1; }
echo 'Abort status and body verified'

Measure the delayed request and confirm the upstream still succeeds:

DELAY_RESULT=$(curl -sS --connect-timeout 3 --max-time 15 -o /dev/null -w '%{http_code} %{time_total}' "${GATEWAY_URL}/fault/delay")
read -r DELAY_STATUS DELAY_SECONDS <<< "${DELAY_RESULT}"
test "${DELAY_STATUS}" = "200" || { echo "Expected 200, got ${DELAY_STATUS}"; exit 1; }
awk -v elapsed="${DELAY_SECONDS}" 'BEGIN { if (elapsed < 1.5) exit 1 }'
printf 'Delay verified: %s seconds\n' "${DELAY_SECONDS}"

Expected: the abort check succeeds, and the delay check reports a successful response taking at least 1.5 seconds. The margin avoids failing on timer-reporting precision around the configured two seconds. Network and upstream time can make the total longer.

Roll back / Clean up

For the Admin API path, delete routes before their service:

for id in fault-injection-delay-id fault-injection-abort-id; do
  curl -ksS --connect-timeout 3 --max-time 15 -X DELETE "${API7_ADMIN_URL}/apisix/admin/routes/${id}?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -H "X-API-KEY: ${API7_ADMIN_KEY}"
done
curl -ksS --connect-timeout 3 --max-time 15 -X DELETE "${API7_ADMIN_URL}/apisix/admin/services/fault-injection-guide-service-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -H "X-API-KEY: ${API7_ADMIN_KEY}"

For ADC, save an empty selected state and review the deletion diff before syncing it:

fault-injection-cleanup.yaml

services: []

adc lint -f fault-injection-cleanup.yaml
adc diff -f fault-injection-cleanup.yaml --backend api7ee --server "${API7_ADMIN_URL}" --token "${API7_ADMIN_KEY}" --gateway-group "${API7_GATEWAY_GROUP_NAME}" --tls-skip-verify --label-selector guide=fault-injection
adc sync -f fault-injection-cleanup.yaml --backend api7ee --server "${API7_ADMIN_URL}" --token "${API7_ADMIN_KEY}" --gateway-group "${API7_GATEWAY_GROUP_NAME}" --tls-skip-verify --label-selector guide=fault-injection
rm -f fault-injection-adc.yaml fault-injection-cleanup.yaml

Expected: the reviewed diff and sync delete only this guide's labeled service and routes.

Troubleshoot

The abort route reaches the upstream: confirm that percentage is still 100 and that the request selected fault-injection-abort.
The delay is shorter than 1.5 seconds: confirm that the plugin is attached to the delay route and that duration and percentage are 2 and 100 respectively.
The delay request fails or times out: verify DNS and outbound HTTPS access to httpbin.org:443; the abort route does not establish upstream connectivity.
A lower percentage produces an unexpected small-sample count: restore 100 for deterministic verification. Lower values are independent random samples rather than fixed request intervals.
ADC proposes unrelated changes: stop and verify the gateway-group display name, guide=fault-injection selector, labels, and complete desired-state file.