Manage APISIX Credentials in Google Cloud Secret Manager

Store a generated key-auth credential in Google Cloud Secret Manager, reference it from Apache APISIX without embedding the service-account key in an Admin API request, and clean up every test resource.

Deployments:DockerRoles:Platform engineerTime:30 minutesImpact:Changes live traffic

Goal

Create a least-privilege Google Cloud service account and secret, configure APISIX to read the secret from a protected credential file, and prove valid and invalid key-auth requests against a controlled upstream.

This procedure creates billable Google Cloud resources and is intended for an isolated APISIX quickstart; use workload identity or the organization's approved credential delivery mechanism in production.

Before you begin

Complete Get APISIX with the Docker quickstart.
Install Google Cloud CLI, cURL, jq, and OpenSSL.
Select a Google Cloud project where Secret Manager is enabled and the current identity can create a secret, service account, service-account key, and secret-level IAM binding.

Set the endpoints, current project, and unique resource names:

export APISIX_ADMIN_URL="http://127.0.0.1:9180"
export GATEWAY_URL="http://127.0.0.1:9080"
export GCP_PROJECT_ID="$(gcloud config get-value project 2>/dev/null)"
export GUIDE_SUFFIX="$(date +%s)"
export GCP_SECRET_NAME="apisix-key-auth-guide-${GUIDE_SUFFIX}"
export GCP_SERVICE_ACCOUNT_ID="apisix-secret-guide-${GUIDE_SUFFIX}"
export GCP_SERVICE_ACCOUNT_EMAIL="${GCP_SERVICE_ACCOUNT_ID}@${GCP_PROJECT_ID}.iam.gserviceaccount.com"
read -rsp "Admin API key (press Enter for an unchanged quickstart): " APISIX_ADMIN_KEY && export APISIX_ADMIN_KEY && echo

Confirm the selected project and API, then protect all fixed APISIX resources, cloud names, containers, and local files:

test -n "${GCP_PROJECT_ID}" && test "${GCP_PROJECT_ID}" != '(unset)'
gcloud auth list --filter=status:ACTIVE --format='value(account)' | grep -q .
test "$(gcloud services list --enabled --project="${GCP_PROJECT_ID}" --filter='name:secretmanager.googleapis.com' --format='value(name)')" = "secretmanager.googleapis.com"
test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${APISIX_ADMIN_URL}/apisix/admin/secrets/gcp/gcp-secret-guide" -H "X-API-KEY: ${APISIX_ADMIN_KEY}")" = 404 || { echo 'Stop: APISIX secret gcp/gcp-secret-guide already exists'; exit 1; }
test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${APISIX_ADMIN_URL}/apisix/admin/consumers/gcp-secret-guide" -H "X-API-KEY: ${APISIX_ADMIN_KEY}")" = 404 || { echo 'Stop: consumer gcp-secret-guide already exists'; exit 1; }
test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${APISIX_ADMIN_URL}/apisix/admin/routes/gcp-secret-guide-route" -H "X-API-KEY: ${APISIX_ADMIN_KEY}")" = 404 || { echo 'Stop: route gcp-secret-guide-route already exists'; exit 1; }
test -z "$(docker ps -aq --filter name=^/gcp-secret-guide-upstream$)" || { echo 'Stop: container gcp-secret-guide-upstream already exists'; exit 1; }
! gcloud secrets describe "${GCP_SECRET_NAME}" --project="${GCP_PROJECT_ID}" >/dev/null 2>&1 || { echo 'Stop: cloud secret already exists'; exit 1; }
! gcloud iam service-accounts describe "${GCP_SERVICE_ACCOUNT_EMAIL}" --project="${GCP_PROJECT_ID}" >/dev/null 2>&1 || { echo 'Stop: service account already exists'; exit 1; }
for file in gcp-secret-guide-key.txt gcp-secret-guide-service-account.json gcp-secret-guide-config.yaml.backup gcp-secret-guide-config.yaml.updated gcp-secret-guide-body.txt; do
  test ! -e "${file}" || { echo "Stop: ${file} already exists"; exit 1; }
done

Steps

Step 1: Create the cloud secret and least-privilege identity

Generate a test credential without printing it, create the secret, and add one version:

umask 077
openssl rand -hex 24 > gcp-secret-guide-key.txt
gcloud secrets create "${GCP_SECRET_NAME}" \
  --project="${GCP_PROJECT_ID}" \
  --replication-policy=automatic
gcloud secrets versions add "${GCP_SECRET_NAME}" \
  --project="${GCP_PROJECT_ID}" \
  --data-file=gcp-secret-guide-key.txt

Expected: the commands report a created secret and an enabled version.

Create a dedicated service account, grant access only to this secret, and write its key to a permission-restricted file:

gcloud iam service-accounts create "${GCP_SERVICE_ACCOUNT_ID}" \
  --project="${GCP_PROJECT_ID}" \
  --display-name='APISIX Secret Manager guide'
gcloud secrets add-iam-policy-binding "${GCP_SECRET_NAME}" \
  --project="${GCP_PROJECT_ID}" \
  --member="serviceAccount:${GCP_SERVICE_ACCOUNT_EMAIL}" \
  --role='roles/secretmanager.secretAccessor'
gcloud iam service-accounts keys create gcp-secret-guide-service-account.json \
  --project="${GCP_PROJECT_ID}" \
  --iam-account="${GCP_SERVICE_ACCOUNT_EMAIL}"
chmod 600 gcp-secret-guide-service-account.json

What changed: Google Cloud now has one uniquely named secret and one service account whose accessor permission is scoped to that secret.

Step 2: Preserve APISIX configuration and trusted CAs

Back up the exact configuration. If it already defines a trusted CA, preserve it; if it has no apisix.ssl block, add the quickstart system bundle. Stop for a source-controlled merge instead of replacing an existing SSL block:

umask 077
docker cp apisix-quickstart:/usr/local/apisix/conf/config.yaml gcp-secret-guide-config.yaml.backup
test "$(grep -Ec '^apisix:[[:space:]]*$' gcp-secret-guide-config.yaml.backup)" = 1
if grep -q '^[[:space:]]*ssl_trusted_certificate:' gcp-secret-guide-config.yaml.backup; then
  cp gcp-secret-guide-config.yaml.backup gcp-secret-guide-config.yaml.updated
elif grep -q '^  ssl:[[:space:]]*$' gcp-secret-guide-config.yaml.backup; then
  echo 'Stop: merge the system CA bundle with the existing apisix.ssl configuration'
  exit 1
else
  awk '1; /^apisix:[[:space:]]*$/ {print "  ssl:\n    ssl_trusted_certificate: /etc/ssl/certs/ca-certificates.crt"}' \
    gcp-secret-guide-config.yaml.backup > gcp-secret-guide-config.yaml.updated
fi
docker exec apisix-quickstart test -r /etc/ssl/certs/ca-certificates.crt

Apply and validate the updated file only when it differs from the backup:

if ! cmp -s gcp-secret-guide-config.yaml.backup gcp-secret-guide-config.yaml.updated; then
  docker cp gcp-secret-guide-config.yaml.updated apisix-quickstart:/usr/local/apisix/conf/config.yaml
  if ! docker exec apisix-quickstart apisix test; then
    docker cp gcp-secret-guide-config.yaml.backup apisix-quickstart:/usr/local/apisix/conf/config.yaml
    docker exec apisix-quickstart apisix reload
    exit 1
  fi
  docker exec apisix-quickstart apisix reload
fi

What changed: APISIX retained its complete original configuration and has a readable CA bundle for verified Google API connections.

Step 3: Install the protected credential file and provider

Copy the service-account file without embedding its private key in an Admin API request:

docker cp gcp-secret-guide-service-account.json apisix-quickstart:/usr/local/apisix/conf/gcp-secret-guide-service-account.json
docker exec apisix-quickstart chown apisix:apisix /usr/local/apisix/conf/gcp-secret-guide-service-account.json
docker exec apisix-quickstart chmod 600 /usr/local/apisix/conf/gcp-secret-guide-service-account.json

Create the fixed APISIX secret-provider resource with certificate verification enabled:

curl -sS --connect-timeout 3 --max-time 15 "${APISIX_ADMIN_URL}/apisix/admin/secrets/gcp/gcp-secret-guide" -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" -H "Content-Type: application/json" \
  -d '{
    "auth_file": "/usr/local/apisix/conf/gcp-secret-guide-service-account.json",
    "ssl_verify": true
  }' | jq -e '.value.id == "gcp-secret-guide" and .value.ssl_verify == true'

What changed: APISIX can authenticate to Secret Manager through a protected local file while the provider resource contains only its path.

Step 4: Create the Consumer credential

Create a fixed Consumer, then build the secret URI from the unique Google Cloud secret name:

curl -sS --connect-timeout 3 --max-time 15 "${APISIX_ADMIN_URL}/apisix/admin/consumers/gcp-secret-guide" -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" -H "Content-Type: application/json" \
  -d '{"username":"gcp-secret-guide"}' | jq -e '.value.username == "gcp-secret-guide"'
export GCP_KEY_REFERENCE="\$secret://gcp/gcp-secret-guide/${GCP_SECRET_NAME}"
jq -n --arg key_ref "${GCP_KEY_REFERENCE}" '{plugins:{"key-auth":{key:$key_ref}}}' | \
curl -sS --connect-timeout 3 --max-time 15 "${APISIX_ADMIN_URL}/apisix/admin/consumers/gcp-secret-guide/credentials/gcp-secret-guide-key-auth" -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" -H "Content-Type: application/json" \
  --data-binary @- | jq -e '.value.id == "gcp-secret-guide-key-auth"'

What changed: the Consumer credential refers to the cloud secret by URI; its plaintext value is not stored in the APISIX resource.

Step 5: Create a controlled upstream and authenticated Route

Start a pinned local upstream and wait for its listener:

docker run -d --name gcp-secret-guide-upstream \
  --network apisix-quickstart-net \
  hashicorp/http-echo:1.0.0 -listen=:5678 -text=gcp-secret-guide-upstream
for attempt in {1..15}; do
  docker exec apisix-quickstart bash -c '</dev/tcp/gcp-secret-guide-upstream/5678' >/dev/null 2>&1 && break
  sleep 1
done
docker exec apisix-quickstart bash -c '</dev/tcp/gcp-secret-guide-upstream/5678'

Create a fixed-ID Route protected by key-auth:

curl -sS --connect-timeout 3 --max-time 15 "${APISIX_ADMIN_URL}/apisix/admin/routes/gcp-secret-guide-route" -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" -H "Content-Type: application/json" \
  -d '{
    "uri": "/gcp-secret-guide",
    "plugins": {"key-auth": {}},
    "upstream": {
      "type": "roundrobin",
      "nodes": {"gcp-secret-guide-upstream:5678": 1}
    }
  }' | jq -e '.value.id == "gcp-secret-guide-route" and (.value.plugins["key-auth"] | type == "object")'

What changed: the fixed path now authenticates against a key resolved from Google Cloud Secret Manager before proxying to the local upstream.

Validate

Require an exact 200 response and body with the generated secret, then require a wrong key to return 401:

test "$(curl --fail-with-body -sS --connect-timeout 3 --max-time 15 -o gcp-secret-guide-body.txt -w '%{http_code}' \
  "${GATEWAY_URL}/gcp-secret-guide" -H "apikey: $(cat gcp-secret-guide-key.txt)")" = 200
test "$(cat gcp-secret-guide-body.txt)" = "gcp-secret-guide-upstream"
test "$(curl -sS --connect-timeout 3 --max-time 15 -o /dev/null -w '%{http_code}' \
  "${GATEWAY_URL}/gcp-secret-guide" -H 'apikey: wrong-key')" = 401

All three assertions must pass. They prove successful remote secret resolution, authentication enforcement, and routing to the intended upstream without printing the credential.

Roll back / Clean up

Remove APISIX dependencies first, restore the exact configuration backup, then delete the cloud identity and secret:

curl -sS --connect-timeout 3 --max-time 15 -X DELETE "${APISIX_ADMIN_URL}/apisix/admin/routes/gcp-secret-guide-route" -H "X-API-KEY: ${APISIX_ADMIN_KEY}"
curl -sS --connect-timeout 3 --max-time 15 -X DELETE "${APISIX_ADMIN_URL}/apisix/admin/consumers/gcp-secret-guide/credentials/gcp-secret-guide-key-auth" -H "X-API-KEY: ${APISIX_ADMIN_KEY}"
curl -sS --connect-timeout 3 --max-time 15 -X DELETE "${APISIX_ADMIN_URL}/apisix/admin/consumers/gcp-secret-guide" -H "X-API-KEY: ${APISIX_ADMIN_KEY}"
curl -sS --connect-timeout 3 --max-time 15 -X DELETE "${APISIX_ADMIN_URL}/apisix/admin/secrets/gcp/gcp-secret-guide" -H "X-API-KEY: ${APISIX_ADMIN_KEY}"
docker rm -f gcp-secret-guide-upstream
docker cp gcp-secret-guide-config.yaml.backup apisix-quickstart:/usr/local/apisix/conf/config.yaml
docker exec apisix-quickstart apisix reload
docker exec apisix-quickstart rm -f /usr/local/apisix/conf/gcp-secret-guide-service-account.json
gcloud iam service-accounts delete "${GCP_SERVICE_ACCOUNT_EMAIL}" --project="${GCP_PROJECT_ID}" --quiet
gcloud secrets delete "${GCP_SECRET_NAME}" --project="${GCP_PROJECT_ID}" --quiet
rm -f gcp-secret-guide-key.txt gcp-secret-guide-service-account.json \
  gcp-secret-guide-config.yaml.backup gcp-secret-guide-config.yaml.updated gcp-secret-guide-body.txt
unset APISIX_ADMIN_URL APISIX_ADMIN_KEY GATEWAY_URL GCP_PROJECT_ID GUIDE_SUFFIX \
  GCP_SECRET_NAME GCP_SERVICE_ACCOUNT_ID GCP_SERVICE_ACCOUNT_EMAIL GCP_KEY_REFERENCE

Expected: the fixed APISIX resources, local upstream, credential files, service account, and cloud secret are absent, and APISIX is running with the exact pre-guide configuration.

Troubleshoot

Symptom	Cause	Fix
APISIX reports a certificate verification error.	The active trusted bundle does not contain the system roots needed for Google APIs.	Restore the backup, build a reviewed combined CA bundle in source-controlled deployment configuration, and keep `ssl_verify: true`.
The valid key returns `401`.	The service account lacks access to the exact secret, the secret URI is wrong, or APISIX cannot read the credential file.	Verify the secret-level IAM binding, `GCP_KEY_REFERENCE`, and file owner/mode without printing either credential.
The valid key returns `502`.	Authentication passed but the local upstream is unavailable.	Confirm the pinned container is running on `apisix-quickstart-net` and its port `5678` is reachable from APISIX.
Cleanup cannot delete a cloud resource.	The active gcloud identity lacks delete permission or the selected project changed.	Re-select `GCP_PROJECT_ID` and delete only the unique names recorded by this guide.

Secrets — explains APISIX secret references.
Key Authentication (opens in Plugin Hub docs) — documents the plugin used for validation.
Manage Secrets in AWS Secrets Manager — uses the AWS provider.
Manage Secrets in HashiCorp Vault — uses the Vault provider.