Docs
API7 GatewayManage traffic and APIsTraffic ManagementImplement Canary Release

Implement Canary Release

Qualify a canary deterministically, shift traffic by relative weight, and verify rollback.

Applies to:API7 Gateway 3.10.x

Roles:Platform engineer

Goal

Route requests with X-Canary: true to a controlled canary upstream, then progressively shift unqualified traffic by relative weight and prove an exact rollback to stable.

traffic-split

traffic-split

Client

Gateway

Upstream V1 (90%)

Upstream V2 (10%)

traffic-split

traffic-split

Client

Gateway

Upstream V1 (90%)

Upstream V2 (10%)

Define application readiness, error-rate, latency, and business-metric gates before a real rollout. Weighted routing is probabilistic, so a small sample diagnoses configuration but cannot prove an exact percentage.

Before you begin

export API7_ADMIN_URL="https://localhost:7443"
export API7_GATEWAY_GROUP_ID="default"   # only the group the quickstart creates; otherwise copy the gateway group's UUID from Gateway Groups in the Dashboard
read -rp "Gateway group name for ADC (if used): " API7_GATEWAY_GROUP_NAME && export API7_GATEWAY_GROUP_NAME
export GATEWAY_URL="http://127.0.0.1:9080"
read -rp "Host or DNS name the gateway uses to reach this machine: " RELEASE_UPSTREAM_HOST && export RELEASE_UPSTREAM_HOST
read -rsp "Dashboard token: " API7_ADMIN_KEY && export API7_ADMIN_KEY && echo

  • Confirm the selected group is ready:

    curl -ksS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}\n' "${API7_ADMIN_URL}/api/gateway_groups" -H "X-API-KEY: ${API7_ADMIN_KEY}"
curl -ksS --connect-timeout 3 --max-time 10 "${API7_ADMIN_URL}/api/gateway_groups/${API7_GATEWAY_GROUP_ID}/instances" -H "X-API-KEY: ${API7_ADMIN_KEY}" | grep -o '"status": *"Healthy"' | wc -l

    Expected: 200, then 1 or more.

  • Python 3, curl, jq, awk, grep, and GNU envsubst are available. The gateway can reach ${RELEASE_UPSTREAM_HOST} on ports 18088 and 18089.

  • Reserve files and ports:

    for file in canary-upstream.py stable.pid canary.pid stable.log canary.log canary-adc.yaml.tmpl canary-adc.yaml canary-rollback-adc.yaml.tmpl canary-rollback-adc.yaml canary-cleanup.yaml canary-sample.txt; do test ! -e "${file}" || { echo "Stop: ${file} already exists"; exit 1; }; done
python3 - <<'PY'
import socket
sockets = []
try:
    for port in (18088, 18089):
        probe = socket.socket()
        probe.bind(("0.0.0.0", port))
        sockets.append(probe)
finally:
    for probe in sockets:
        probe.close()
print("canary upstream ports are free")
PY

  • For ADC, confirm label-selector support:

    adc --version
adc sync --help | grep -- '--label-selector'

    ADC resolves --gateway-group by display name; the compatibility Admin API uses the group's UUID.

Preflight both fixed Admin API IDs:

curl -ksS --connect-timeout 3 --max-time 10 -w '\nservice: %{http_code}\n' "${API7_ADMIN_URL}/apisix/admin/services/canary-guide-service-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -H "X-API-KEY: ${API7_ADMIN_KEY}"
curl -ksS --connect-timeout 3 --max-time 10 -w '\nroute: %{http_code}\n' "${API7_ADMIN_URL}/apisix/admin/routes/canary-guide-route-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -H "X-API-KEY: ${API7_ADMIN_KEY}"

Expected: both are absent. A 200 containing a resource means the ID is in use; stop or change every occurrence because PUT replaces that ID. URL IDs deliberately differ from names.

Steps

Step 1: Start controlled upstreams

cat > canary-upstream.py <<'PY'
import os
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
port = int(os.environ["PORT"])
release = os.environ["RELEASE"]
class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        body = ("release=%s\n" % release).encode()
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, format, *args):
        pass
ThreadingHTTPServer(("0.0.0.0", port), Handler).serve_forever()
PY
PORT=18088 RELEASE=stable python3 canary-upstream.py > stable.log 2>&1 & echo $! > stable.pid
PORT=18089 RELEASE=canary python3 canary-upstream.py > canary.log 2>&1 & echo $! > canary.pid
test "$(curl -fsS --connect-timeout 3 --max-time 5 http://127.0.0.1:18088/)" = "release=stable"
test "$(curl -fsS --connect-timeout 3 --max-time 5 http://127.0.0.1:18089/)" = "release=canary"

Step 2: Define complete traffic operations

Choose one tab and continue in the same shell; later steps call the functions defined here.

jq -n --arg host "${RELEASE_UPSTREAM_HOST}" '{name:"canary-guide-service",labels:{guide:"canary-release"},upstream:{name:"default",type:"roundrobin",scheme:"http",nodes:[{host:$host,port:18088,weight:100}]}}' \
  | curl -ksS --connect-timeout 3 --max-time 15 "${API7_ADMIN_URL}/apisix/admin/services/canary-guide-service-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -X PUT -H "X-API-KEY: ${API7_ADMIN_KEY}" -H "Content-Type: application/json" --data-binary @-
apply_canary() {
  local canary_weight="$1" stable_weight="$2"
  jq -n --arg host "${RELEASE_UPSTREAM_HOST}" --argjson canary "${canary_weight}" --argjson stable "${stable_weight}" '{name:"canary-guide-route",labels:{guide:"canary-release"},paths:["/release/canary"],methods:["GET"],service_id:"canary-guide-service-id",plugins:{"traffic-split":{rules:[{match:[{vars:[["http_x_canary","==","true"]]}],weighted_upstreams:[{weight:100,upstream:{type:"roundrobin",scheme:"http",nodes:[{host:$host,port:18089,weight:100}]}}]},{weighted_upstreams:[{weight:$canary,upstream:{type:"roundrobin",scheme:"http",nodes:[{host:$host,port:18089,weight:100}]}},{weight:$stable}]}]}}}' \
    | curl -ksS --connect-timeout 3 --max-time 15 "${API7_ADMIN_URL}/apisix/admin/routes/canary-guide-route-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -X PUT -H "X-API-KEY: ${API7_ADMIN_KEY}" -H "Content-Type: application/json" --data-binary @-
}
rollback_canary() {
  jq -n --arg host "${RELEASE_UPSTREAM_HOST}" '{name:"canary-guide-route",labels:{guide:"canary-release"},paths:["/release/canary"],methods:["GET"],service_id:"canary-guide-service-id",plugins:{"traffic-split":{rules:[{weighted_upstreams:[{weight:0,upstream:{type:"roundrobin",scheme:"http",nodes:[{host:$host,port:18089,weight:100}]}},{weight:100}]}]}}}' \
    | curl -ksS --connect-timeout 3 --max-time 15 "${API7_ADMIN_URL}/apisix/admin/routes/canary-guide-route-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -X PUT -H "X-API-KEY: ${API7_ADMIN_KEY}" -H "Content-Type: application/json" --data-binary @-
}
apply_canary 0 100

Rules run in array order and the first matching rule wins. Keep the conditional rule first and the unconditional weighted rule last. In each weighted list, the item without upstream or upstream_id selects the stable service upstream. Weights are nonnegative integers and act as relative proportions.

Step 3: Prove deterministic qualification

for request in {1..10}; do
  test "$(curl -fsS --connect-timeout 3 --max-time 10 "${GATEWAY_URL}/release/canary")" = "release=stable"
  test "$(curl -fsS --connect-timeout 3 --max-time 10 "${GATEWAY_URL}/release/canary" -H 'X-Canary: true')" = "release=canary"
done

X-Canary is only a reproducible qualification signal for this guide. Use an authenticated, trusted condition for a production rollout.

Step 4: Shift unqualified traffic progressively

observe_canary() {
  : > canary-sample.txt
  for request in $(seq 1 200); do
    curl -fsS --connect-timeout 3 --max-time 10 "${GATEWAY_URL}/release/canary" >> canary-sample.txt
  done
  stable_count="$(awk '$0 == "release=stable" {n++} END {print n+0}' canary-sample.txt)"
  canary_count="$(awk '$0 == "release=canary" {n++} END {print n+0}' canary-sample.txt)"
  test "$((stable_count + canary_count))" -eq 200
  test "${stable_count}" -gt 0
  test "${canary_count}" -gt 0
  printf 'observed stable=%s canary=%s of 200\n' "${stable_count}" "${canary_count}"
}
apply_canary 10 90
observe_canary

Inspect your release gates. If they pass, continue:

apply_canary 50 50
observe_canary

Inspect the gates again. Promote only when they pass:

apply_canary 100 0
for request in {1..10}; do
  test "$(curl -fsS --connect-timeout 3 --max-time 10 "${GATEWAY_URL}/release/canary")" = "release=canary"
done

The observed counts need not equal the configured ratio. At 0/100 and 100/0, exact release-body assertions are appropriate.

Validate

Confirm exactly one owned route exists:

curl -ksS --connect-timeout 3 --max-time 15 "${API7_ADMIN_URL}/apisix/admin/routes?gateway_group_id=${API7_GATEWAY_GROUP_ID}&service_id=canary-guide-service-id&search=canary-guide-route&page=1&page_size=100" -H "X-API-KEY: ${API7_ADMIN_KEY}" \
  | jq -e '[.list[] | select(.name == "canary-guide-route" and .labels.guide == "canary-release")] | length == 1'

This checks ownership and uniqueness; it does not infer an ADC backend ID from the display name. Route priority does not control the order of traffic-split.rules.

Roll back / Clean up

Apply a complete known-good route that removes the qualification rule and assigns all traffic to stable:

rollback_canary
for request in {1..10}; do
  test "$(curl -fsS --connect-timeout 3 --max-time 10 "${GATEWAY_URL}/release/canary")" = "release=stable"
  test "$(curl -fsS --connect-timeout 3 --max-time 10 "${GATEWAY_URL}/release/canary" -H 'X-Canary: true')" = "release=stable"
done

Keep the rolled-back route if stable should continue serving. Otherwise choose the matching cleanup tab.

curl -ksS --connect-timeout 3 --max-time 15 -X DELETE "${API7_ADMIN_URL}/apisix/admin/routes/canary-guide-route-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -H "X-API-KEY: ${API7_ADMIN_KEY}"
curl -ksS --connect-timeout 3 --max-time 15 -X DELETE "${API7_ADMIN_URL}/apisix/admin/services/canary-guide-service-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -H "X-API-KEY: ${API7_ADMIN_KEY}"
for file in stable.pid canary.pid; do
  if test -f "${file}"; then
    pid="$(cat "${file}")"
    if kill -0 "${pid}" 2>/dev/null && ps -p "${pid}" -o command= | grep -Fq 'python3 canary-upstream.py'; then kill "${pid}"; fi
  fi
done
rm -f canary-upstream.py stable.pid canary.pid stable.log canary.log canary-adc.yaml.tmpl canary-adc.yaml canary-rollback-adc.yaml.tmpl canary-rollback-adc.yaml canary-cleanup.yaml canary-sample.txt
unset CANARY_WEIGHT STABLE_WEIGHT

Troubleshoot

  • Qualified requests reach stable: confirm the header value is exactly true, the conditional rule is first, and the latest complete route reached the selected group.
  • A weighted sample contains only one release: confirm both weights are positive and both upstreams are reachable, then collect a larger diagnostic sample.
  • The gateway returns 502 or 504: test both ports from the gateway network and verify RELEASE_UPSTREAM_HOST.
  • ADC proposes unrelated changes: stop and verify the group name, selector, labels, and complete desired state.
  • Application metrics fail: call rollback_canary immediately and investigate the application.

Rewrite Upstream Requests

Previous Page

Implement Blue-Green Deployment

Next Page