Implement Blue-Green Deployment

Switch all traffic between controlled blue and green upstreams and verify exact rollback.

Goal

Create a blue service and inline green upstream, switch all requests to green with traffic-split, and prove a complete rollback sends every request to blue.

Define application readiness, error-rate, latency, and business-metric gates before a real cutover. This guide validates routing mechanics, not production readiness.

Before you begin

export API7_ADMIN_URL="https://localhost:7443"
export API7_GATEWAY_GROUP_ID="default"   # only the group the quickstart creates; otherwise copy the gateway group's UUID from Gateway Groups in the Dashboard
read -rp "Gateway group name for ADC (if used): " API7_GATEWAY_GROUP_NAME && export API7_GATEWAY_GROUP_NAME
export GATEWAY_URL="http://127.0.0.1:9080"
read -rp "Host or DNS name the gateway uses to reach this machine: " RELEASE_UPSTREAM_HOST && export RELEASE_UPSTREAM_HOST
read -rsp "Dashboard token: " API7_ADMIN_KEY && export API7_ADMIN_KEY && echo

Confirm the selected group is ready:

curl -ksS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}\n' "${API7_ADMIN_URL}/api/gateway_groups" -H "X-API-KEY: ${API7_ADMIN_KEY}"
curl -ksS --connect-timeout 3 --max-time 10 "${API7_ADMIN_URL}/api/gateway_groups/${API7_GATEWAY_GROUP_ID}/instances" -H "X-API-KEY: ${API7_ADMIN_KEY}" | grep -o '"status": *"Healthy"' | wc -l

Expected: 200, then 1 or more.

Python 3, curl, jq, grep, and GNU envsubst are available. The gateway can reach ${RELEASE_UPSTREAM_HOST} on ports 18086 and 18087.

Reserve files and ports:

for file in blue-green-upstream.py blue.pid green.pid blue.log green.log blue-green-adc.yaml.tmpl blue-green-adc.yaml blue-green-cleanup.yaml; do test ! -e "${file}" || { echo "Stop: ${file} already exists"; exit 1; }; done
python3 - <<'PY'
import socket
sockets = []
try:
    for port in (18086, 18087):
        probe = socket.socket()
        probe.bind(("0.0.0.0", port))
        sockets.append(probe)
finally:
    for probe in sockets:
        probe.close()
print("blue-green upstream ports are free")
PY

For ADC, confirm label-selector support:
```
adc --version
adc sync --help | grep -- '--label-selector'
```
ADC resolves --gateway-group by display name; the compatibility Admin API uses the group's UUID.

Preflight both fixed Admin API IDs:

curl -ksS --connect-timeout 3 --max-time 10 -w '\nservice: %{http_code}\n' "${API7_ADMIN_URL}/apisix/admin/services/blue-green-guide-service-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -H "X-API-KEY: ${API7_ADMIN_KEY}"
curl -ksS --connect-timeout 3 --max-time 10 -w '\nroute: %{http_code}\n' "${API7_ADMIN_URL}/apisix/admin/routes/blue-green-guide-route-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -H "X-API-KEY: ${API7_ADMIN_KEY}"

Expected: both are absent. A 200 containing a resource means the ID is in use; stop or change every occurrence because PUT replaces that ID. URL IDs deliberately differ from names.

Steps

Step 1: Start controlled upstreams

cat > blue-green-upstream.py <<'PY'
import os
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
port = int(os.environ["PORT"])
release = os.environ["RELEASE"]
class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        body = ("release=%s\n" % release).encode()
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, format, *args):
        pass
ThreadingHTTPServer(("0.0.0.0", port), Handler).serve_forever()
PY
PORT=18086 RELEASE=blue python3 blue-green-upstream.py > blue.log 2>&1 & echo $! > blue.pid
PORT=18087 RELEASE=green python3 blue-green-upstream.py > green.log 2>&1 & echo $! > green.pid
test "$(curl -fsS --connect-timeout 3 --max-time 5 http://127.0.0.1:18086/)" = "release=blue"
test "$(curl -fsS --connect-timeout 3 --max-time 5 http://127.0.0.1:18087/)" = "release=green"

Step 2: Define one complete switch operation

Choose one tab and continue in the same shell; later steps call the function defined here.

jq -n --arg host "${RELEASE_UPSTREAM_HOST}" '{name:"blue-green-guide-service",labels:{guide:"blue-green-deployment"},upstream:{name:"default",type:"roundrobin",scheme:"http",nodes:[{host:$host,port:18086,weight:100}]}}' \
  | curl -ksS --connect-timeout 3 --max-time 15 "${API7_ADMIN_URL}/apisix/admin/services/blue-green-guide-service-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -X PUT -H "X-API-KEY: ${API7_ADMIN_KEY}" -H "Content-Type: application/json" --data-binary @-
apply_blue_green() {
  local green_weight="$1" blue_weight="$2"
  jq -n --arg host "${RELEASE_UPSTREAM_HOST}" --argjson green "${green_weight}" --argjson blue "${blue_weight}" '{name:"blue-green-guide-route",labels:{guide:"blue-green-deployment"},paths:["/release/blue-green"],methods:["GET"],service_id:"blue-green-guide-service-id",plugins:{"traffic-split":{rules:[{weighted_upstreams:[{weight:$green,upstream:{type:"roundrobin",scheme:"http",nodes:[{host:$host,port:18087,weight:100}]}},{weight:$blue}]}]}}}' \
    | curl -ksS --connect-timeout 3 --max-time 15 "${API7_ADMIN_URL}/apisix/admin/routes/blue-green-guide-route-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -X PUT -H "X-API-KEY: ${API7_ADMIN_KEY}" -H "Content-Type: application/json" --data-binary @-
}
apply_blue_green 0 100

blue-green-adc.yaml.tmpl

services:
  - name: blue-green-guide-service
    labels: {guide: blue-green-deployment}
    upstream:
      name: default
      scheme: http
      nodes:
        - {host: "${RELEASE_UPSTREAM_HOST}", port: 18086, weight: 100}
    routes:
      - name: blue-green-guide-route
        labels: {guide: blue-green-deployment}
        uris: [/release/blue-green]
        methods: [GET]
        plugins:
          traffic-split:
            rules:
              - weighted_upstreams:
                  - weight: ${GREEN_WEIGHT}
                    upstream:
                      type: roundrobin
                      scheme: http
                      nodes:
                        - {host: "${RELEASE_UPSTREAM_HOST}", port: 18087, weight: 100}
                  - weight: ${BLUE_WEIGHT}

apply_blue_green() {
  export GREEN_WEIGHT="$1"
  export BLUE_WEIGHT="$2"
  envsubst '${RELEASE_UPSTREAM_HOST} ${GREEN_WEIGHT} ${BLUE_WEIGHT}' < blue-green-adc.yaml.tmpl > blue-green-adc.yaml
  adc lint -f blue-green-adc.yaml
  adc validate -f blue-green-adc.yaml --backend api7ee --server "${API7_ADMIN_URL}" --token "${API7_ADMIN_KEY}" --gateway-group "${API7_GATEWAY_GROUP_NAME}" --tls-skip-verify --label-selector guide=blue-green-deployment
  adc diff -f blue-green-adc.yaml --backend api7ee --server "${API7_ADMIN_URL}" --token "${API7_ADMIN_KEY}" --gateway-group "${API7_GATEWAY_GROUP_NAME}" --tls-skip-verify --label-selector guide=blue-green-deployment
  local adc_confirm
  read -rp "Apply only this label-scoped diff? Type yes: " adc_confirm
  test "${adc_confirm}" = "yes"
  adc sync -f blue-green-adc.yaml --backend api7ee --server "${API7_ADMIN_URL}" --token "${API7_ADMIN_KEY}" --gateway-group "${API7_GATEWAY_GROUP_NAME}" --tls-skip-verify --label-selector guide=blue-green-deployment
}
apply_blue_green 0 100

Continue only if the diff affects this labeled service and route. ADC names are display names, not backend IDs.

The first item is green. The item without upstream or upstream_id is default blue. Relative weights 0 and 100 send all traffic to blue.

Step 3: Switch traffic to green

assert_blue_green_release() {
  local expected="$1"
  for request in {1..10}; do
    test "$(curl -fsS --connect-timeout 3 --max-time 10 "${GATEWAY_URL}/release/blue-green")" = "release=${expected}"
  done
}
assert_blue_green_release blue

Use the same configuration method selected in Step 2.

apply_blue_green 100 0
assert_blue_green_release green

apply_blue_green 100 0
assert_blue_green_release green

Review the label-scoped diff before the function syncs the complete switched state.

Hold green only after your release gates pass. The inline green upstream has no health checks because inline traffic-split upstreams do not support checks.

Validate

Confirm exactly one owned route exists:

curl -ksS --connect-timeout 3 --max-time 15 "${API7_ADMIN_URL}/apisix/admin/routes?gateway_group_id=${API7_GATEWAY_GROUP_ID}&service_id=blue-green-guide-service-id&search=blue-green-guide-route&page=1&page_size=100" -H "X-API-KEY: ${API7_ADMIN_KEY}" \
  | jq -e '[.list[] | select(.name == "blue-green-guide-route" and .labels.guide == "blue-green-deployment")] | length == 1'

This checks ownership and uniqueness; it does not infer an ADC backend ID from the display name.

Roll back / Clean up

Apply the complete known-good state with the same configuration method selected in Step 2 and require only blue:

apply_blue_green 0 100
assert_blue_green_release blue

apply_blue_green 0 100
assert_blue_green_release blue

Review the label-scoped diff before the function syncs the complete rolled-back state.

Keep the rolled-back route if blue should continue serving. Otherwise choose the matching cleanup tab.

curl -ksS --connect-timeout 3 --max-time 15 -X DELETE "${API7_ADMIN_URL}/apisix/admin/routes/blue-green-guide-route-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -H "X-API-KEY: ${API7_ADMIN_KEY}"
curl -ksS --connect-timeout 3 --max-time 15 -X DELETE "${API7_ADMIN_URL}/apisix/admin/services/blue-green-guide-service-id?gateway_group_id=${API7_GATEWAY_GROUP_ID}" -H "X-API-KEY: ${API7_ADMIN_KEY}"

blue-green-cleanup.yaml

services: []

adc lint -f blue-green-cleanup.yaml
adc diff -f blue-green-cleanup.yaml --backend api7ee --server "${API7_ADMIN_URL}" --token "${API7_ADMIN_KEY}" --gateway-group "${API7_GATEWAY_GROUP_NAME}" --tls-skip-verify --label-selector guide=blue-green-deployment
adc sync -f blue-green-cleanup.yaml --backend api7ee --server "${API7_ADMIN_URL}" --token "${API7_ADMIN_KEY}" --gateway-group "${API7_GATEWAY_GROUP_NAME}" --tls-skip-verify --label-selector guide=blue-green-deployment

for file in blue.pid green.pid; do
  if test -f "${file}"; then
    pid="$(cat "${file}")"
    if kill -0 "${pid}" 2>/dev/null && ps -p "${pid}" -o command= | grep -Fq 'python3 blue-green-upstream.py'; then kill "${pid}"; fi
  fi
done
rm -f blue-green-upstream.py blue.pid green.pid blue.log green.log blue-green-adc.yaml.tmpl blue-green-adc.yaml blue-green-cleanup.yaml
unset GREEN_WEIGHT BLUE_WEIGHT

Troubleshoot

A zero-weight release still appears: verify the latest complete route reached the selected group and this path matches only one route.
The gateway returns 502 or 504: test both ports from the gateway network and verify RELEASE_UPSTREAM_HOST.
ADC proposes unrelated changes: stop and verify the group name, selector, labels, and complete desired state.
The sample passes but application metrics fail: apply rollback and investigate the application.