Manage APISIX Credentials in AWS Secrets Manager

Store a generated key-auth credential in AWS Secrets Manager, give Apache APISIX least-privilege access to it, validate authentication, and clean up every test resource.

Deployments:DockerRoles:Platform engineerTime:30 minutesImpact:Changes live traffic

Goal

Create a uniquely named AWS Secrets Manager secret and a dedicated IAM user that can read only that secret. Configure APISIX to resolve a key-auth credential from the secret, then prove valid and invalid requests against a controlled upstream.

This procedure creates a billable AWS secret and a long-lived IAM access key. It is intended for an isolated APISIX quickstart. In production, use temporary credentials delivered through your platform's workload-identity mechanism and protect the APISIX control plane and etcd according to your organization's security requirements.

Before you begin

Complete Get APISIX with the Docker quickstart.
Install AWS CLI, cURL, jq, and OpenSSL.
Authenticate AWS CLI with an identity that can create a secret, create an IAM user and access key, and attach an inline policy.

Set the endpoints, selected region, and unique resource names:

export APISIX_ADMIN_URL="http://127.0.0.1:9180"
export GATEWAY_URL="http://127.0.0.1:9080"
export AWS_REGION="$(aws configure get region)"
export GUIDE_SUFFIX="$(date +%s)"
export AWS_SECRET_NAME="apisix-key-auth-guide-${GUIDE_SUFFIX}"
export AWS_IAM_USER_NAME="apisix-secret-guide-${GUIDE_SUFFIX}"
read -rsp "Admin API key (press Enter for an unchanged quickstart): " APISIX_ADMIN_KEY && export APISIX_ADMIN_KEY && echo

Confirm the AWS identity and region, then protect all fixed APISIX resources, cloud names, containers, and local files:

test -n "${AWS_REGION}"
aws sts get-caller-identity >/dev/null
test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${APISIX_ADMIN_URL}/apisix/admin/secrets/aws/aws-secret-guide" -H "X-API-KEY: ${APISIX_ADMIN_KEY}")" = 404 || { echo 'Stop: APISIX secret aws/aws-secret-guide already exists'; exit 1; }
test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${APISIX_ADMIN_URL}/apisix/admin/consumers/aws-secret-guide" -H "X-API-KEY: ${APISIX_ADMIN_KEY}")" = 404 || { echo 'Stop: consumer aws-secret-guide already exists'; exit 1; }
test "$(curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w '%{http_code}' "${APISIX_ADMIN_URL}/apisix/admin/routes/aws-secret-guide-route" -H "X-API-KEY: ${APISIX_ADMIN_KEY}")" = 404 || { echo 'Stop: route aws-secret-guide-route already exists'; exit 1; }
test -z "$(docker ps -aq --filter name=^/aws-secret-guide-upstream$)" || { echo 'Stop: container aws-secret-guide-upstream already exists'; exit 1; }
! aws secretsmanager describe-secret --region "${AWS_REGION}" --secret-id "${AWS_SECRET_NAME}" >/dev/null 2>&1 || { echo 'Stop: cloud secret already exists'; exit 1; }
! aws iam get-user --user-name "${AWS_IAM_USER_NAME}" >/dev/null 2>&1 || { echo 'Stop: IAM user already exists'; exit 1; }
for file in aws-secret-guide-key.txt aws-secret-guide-payload.json aws-secret-guide-policy.json aws-secret-guide-access-key.json aws-secret-guide-config.yaml.backup aws-secret-guide-config.yaml.updated aws-secret-guide-body.txt; do
  test ! -e "${file}" || { echo "Stop: ${file} already exists"; exit 1; }
done

Steps

Step 1: Create the cloud secret

Generate a test credential without printing it, store it as a JSON field, and create the uniquely named secret:

umask 077
openssl rand -hex 24 > aws-secret-guide-key.txt
jq -n --rawfile key aws-secret-guide-key.txt '{"key-auth":($key | rtrimstr("\n"))}' > aws-secret-guide-payload.json
export AWS_SECRET_ARN="$(aws secretsmanager create-secret \
  --region "${AWS_REGION}" \
  --name "${AWS_SECRET_NAME}" \
  --description 'Temporary secret for the APISIX AWS guide' \
  --secret-string file://aws-secret-guide-payload.json \
  --query ARN --output text)"
test -n "${AWS_SECRET_ARN}" && test "${AWS_SECRET_ARN}" != None

What changed: AWS Secrets Manager now contains one uniquely named secret whose key-auth field holds the generated credential.

Step 2: Create a least-privilege IAM credential

Create an IAM user and an inline policy that permits only GetSecretValue on the exact secret ARN:

aws iam create-user --user-name "${AWS_IAM_USER_NAME}" >/dev/null
jq -n --arg secret_arn "${AWS_SECRET_ARN}" '{
  Version:"2012-10-17",
  Statement:[{
    Effect:"Allow",
    Action:"secretsmanager:GetSecretValue",
    Resource:$secret_arn
  }]
}' > aws-secret-guide-policy.json
aws iam put-user-policy \
  --user-name "${AWS_IAM_USER_NAME}" \
  --policy-name APISIXSecretGuideRead \
  --policy-document file://aws-secret-guide-policy.json
aws iam create-access-key --user-name "${AWS_IAM_USER_NAME}" > aws-secret-guide-access-key.json
chmod 600 aws-secret-guide-access-key.json
jq -e '.AccessKey.AccessKeyId and .AccessKey.SecretAccessKey' aws-secret-guide-access-key.json >/dev/null

Do not print, commit, or reuse aws-secret-guide-access-key.json. The key is long-lived until the cleanup step deletes it.

What changed: the dedicated IAM user can retrieve only the guide's secret and has one local, permission-restricted access-key file.

Step 3: Preserve APISIX configuration and trusted CAs

Back up the exact configuration. If it already defines a trusted CA, preserve it; if it has no apisix.ssl block, add the quickstart system bundle. Stop for a source-controlled merge instead of replacing an existing SSL block:

umask 077
docker cp apisix-quickstart:/usr/local/apisix/conf/config.yaml aws-secret-guide-config.yaml.backup
test "$(grep -Ec '^apisix:[[:space:]]*$' aws-secret-guide-config.yaml.backup)" = 1
if grep -q '^[[:space:]]*ssl_trusted_certificate:' aws-secret-guide-config.yaml.backup; then
  cp aws-secret-guide-config.yaml.backup aws-secret-guide-config.yaml.updated
elif grep -q '^  ssl:[[:space:]]*$' aws-secret-guide-config.yaml.backup; then
  echo 'Stop: merge the system CA bundle with the existing apisix.ssl configuration'
  exit 1
else
  awk '1; /^apisix:[[:space:]]*$/ {print "  ssl:\n    ssl_trusted_certificate: /etc/ssl/certs/ca-certificates.crt"}' \
    aws-secret-guide-config.yaml.backup > aws-secret-guide-config.yaml.updated
fi
docker exec apisix-quickstart test -r /etc/ssl/certs/ca-certificates.crt

Apply and validate the updated file only when it differs from the backup:

if ! cmp -s aws-secret-guide-config.yaml.backup aws-secret-guide-config.yaml.updated; then
  docker cp aws-secret-guide-config.yaml.updated apisix-quickstart:/usr/local/apisix/conf/config.yaml
  if ! docker exec apisix-quickstart apisix test; then
    docker cp aws-secret-guide-config.yaml.backup apisix-quickstart:/usr/local/apisix/conf/config.yaml
    docker exec apisix-quickstart apisix reload
    exit 1
  fi
  docker exec apisix-quickstart apisix reload
fi

What changed: APISIX retained its complete original configuration and has a readable CA bundle for verified AWS API connections.

Step 4: Configure the AWS secret provider

Build the request from the protected access-key file and create the fixed provider resource:

jq -n \
  --arg region "${AWS_REGION}" \
  --arg access_key_id "$(jq -r '.AccessKey.AccessKeyId' aws-secret-guide-access-key.json)" \
  --arg secret_access_key "$(jq -r '.AccessKey.SecretAccessKey' aws-secret-guide-access-key.json)" \
  '{region:$region, access_key_id:$access_key_id, secret_access_key:$secret_access_key}' | \
curl -sS --connect-timeout 3 --max-time 15 "${APISIX_ADMIN_URL}/apisix/admin/secrets/aws/aws-secret-guide" -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" -H "Content-Type: application/json" \
  --data-binary @- | jq -e '.value.id == "aws-secret-guide"'

The Admin API resource necessarily contains the AWS credential required by the APISIX AWS provider. Restrict access to the Admin API and etcd, and delete the provider and IAM access key immediately after the test. For the request schema, see Create or Replace a Secret.

What changed: APISIX can authenticate to AWS Secrets Manager with a credential limited to the exact guide secret.

Step 5: Create the Consumer credential

Create a fixed Consumer, then build the secret URI from the unique AWS secret name and JSON field:

curl -sS --connect-timeout 3 --max-time 15 "${APISIX_ADMIN_URL}/apisix/admin/consumers/aws-secret-guide" -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" -H "Content-Type: application/json" \
  -d '{"username":"aws-secret-guide"}' | jq -e '.value.username == "aws-secret-guide"'
export AWS_KEY_REFERENCE="\$secret://aws/aws-secret-guide/${AWS_SECRET_NAME}/key-auth"
jq -n --arg key_ref "${AWS_KEY_REFERENCE}" '{plugins:{"key-auth":{key:$key_ref}}}' | \
curl -sS --connect-timeout 3 --max-time 15 "${APISIX_ADMIN_URL}/apisix/admin/consumers/aws-secret-guide/credentials/aws-secret-guide-key-auth" -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" -H "Content-Type: application/json" \
  --data-binary @- | jq -e '.value.id == "aws-secret-guide-key-auth"'

What changed: the Consumer credential refers to the AWS secret by URI; its plaintext authentication value is not stored in the credential resource.

Step 6: Create a controlled upstream and authenticated Route

Start a pinned local upstream and wait for its listener:

docker run -d --name aws-secret-guide-upstream \
  --network apisix-quickstart-net \
  hashicorp/http-echo:1.0.0 -listen=:5678 -text=aws-secret-guide-upstream
for attempt in {1..15}; do
  docker exec apisix-quickstart bash -c '</dev/tcp/aws-secret-guide-upstream/5678' >/dev/null 2>&1 && break
  sleep 1
done
docker exec apisix-quickstart bash -c '</dev/tcp/aws-secret-guide-upstream/5678'

Create a fixed-ID Route protected by key-auth:

curl -sS --connect-timeout 3 --max-time 15 "${APISIX_ADMIN_URL}/apisix/admin/routes/aws-secret-guide-route" -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" -H "Content-Type: application/json" \
  -d '{
    "uri": "/aws-secret-guide",
    "plugins": {"key-auth": {}},
    "upstream": {
      "type": "roundrobin",
      "nodes": {"aws-secret-guide-upstream:5678": 1}
    }
  }' | jq -e '.value.id == "aws-secret-guide-route" and (.value.plugins["key-auth"] | type == "object")'

What changed: the fixed path now authenticates against a key resolved from AWS Secrets Manager before proxying to the local upstream.

Validate

Require an exact 200 response and body with the generated secret, then require a wrong key to return 401:

test "$(curl --fail-with-body -sS --connect-timeout 3 --max-time 15 -o aws-secret-guide-body.txt -w '%{http_code}' \
  "${GATEWAY_URL}/aws-secret-guide" -H "apikey: $(cat aws-secret-guide-key.txt)")" = 200
test "$(cat aws-secret-guide-body.txt)" = "aws-secret-guide-upstream"
test "$(curl -sS --connect-timeout 3 --max-time 15 -o /dev/null -w '%{http_code}' \
  "${GATEWAY_URL}/aws-secret-guide" -H 'apikey: wrong-key')" = 401

All three assertions must pass. They prove successful remote secret resolution, authentication enforcement, and routing to the intended upstream without printing the credential.

Roll back / Clean up

Remove APISIX dependencies first, restore the exact configuration backup, delete the IAM access key and user, and schedule the uniquely named AWS secret for recoverable deletion:

curl -sS --connect-timeout 3 --max-time 15 -X DELETE "${APISIX_ADMIN_URL}/apisix/admin/routes/aws-secret-guide-route" -H "X-API-KEY: ${APISIX_ADMIN_KEY}"
curl -sS --connect-timeout 3 --max-time 15 -X DELETE "${APISIX_ADMIN_URL}/apisix/admin/consumers/aws-secret-guide/credentials/aws-secret-guide-key-auth" -H "X-API-KEY: ${APISIX_ADMIN_KEY}"
curl -sS --connect-timeout 3 --max-time 15 -X DELETE "${APISIX_ADMIN_URL}/apisix/admin/consumers/aws-secret-guide" -H "X-API-KEY: ${APISIX_ADMIN_KEY}"
curl -sS --connect-timeout 3 --max-time 15 -X DELETE "${APISIX_ADMIN_URL}/apisix/admin/secrets/aws/aws-secret-guide" -H "X-API-KEY: ${APISIX_ADMIN_KEY}"
docker rm -f aws-secret-guide-upstream
docker cp aws-secret-guide-config.yaml.backup apisix-quickstart:/usr/local/apisix/conf/config.yaml
docker exec apisix-quickstart apisix reload
export AWS_ACCESS_KEY_ID_TO_DELETE="$(jq -r '.AccessKey.AccessKeyId' aws-secret-guide-access-key.json)"
aws iam delete-access-key --user-name "${AWS_IAM_USER_NAME}" --access-key-id "${AWS_ACCESS_KEY_ID_TO_DELETE}"
aws iam delete-user-policy --user-name "${AWS_IAM_USER_NAME}" --policy-name APISIXSecretGuideRead
aws iam delete-user --user-name "${AWS_IAM_USER_NAME}"
aws secretsmanager delete-secret --region "${AWS_REGION}" --secret-id "${AWS_SECRET_ARN}" >/dev/null
rm -f aws-secret-guide-key.txt aws-secret-guide-payload.json aws-secret-guide-policy.json \
  aws-secret-guide-access-key.json aws-secret-guide-config.yaml.backup \
  aws-secret-guide-config.yaml.updated aws-secret-guide-body.txt
unset APISIX_ADMIN_URL APISIX_ADMIN_KEY GATEWAY_URL AWS_REGION GUIDE_SUFFIX AWS_SECRET_NAME \
  AWS_SECRET_ARN AWS_IAM_USER_NAME AWS_KEY_REFERENCE AWS_ACCESS_KEY_ID_TO_DELETE

Expected: the fixed APISIX resources, local upstream, IAM user, access key, and local secret files are absent. APISIX is running with the exact pre-guide configuration, and AWS reports the secret as scheduled for deletion within its recovery window.

Troubleshoot

Symptom	Cause	Fix
APISIX reports a certificate verification error.	The active trusted bundle does not contain the system roots needed for AWS APIs.	Restore the backup, build a reviewed combined CA bundle in source-controlled deployment configuration, and do not disable certificate verification.
The valid key returns `401`.	The IAM policy has not propagated, the secret URI or JSON field is wrong, or the IAM access key is inactive.	Verify the exact secret ARN in the inline policy, `AWS_KEY_REFERENCE`, and access-key status without printing secret values.
The valid key returns `502`.	Authentication passed but the local upstream is unavailable.	Confirm the pinned container is running on `apisix-quickstart-net` and its port `5678` is reachable from APISIX.
Cleanup cannot delete an IAM resource.	The inline policy or access key still exists, or the active AWS identity lacks delete permission.	Delete only the access key and inline policy recorded by this guide, then retry deletion of the uniquely named user.
The secret remains visible after cleanup.	AWS uses a recovery window for deletion.	Confirm its `DeletedDate`; restore it only if needed, otherwise let the recovery window expire.

Secrets — explains APISIX secret references.
Key Authentication (opens in Plugin Hub docs) — documents the plugin used for validation.
Manage APISIX Credentials in Google Cloud Secret Manager — uses the Google Cloud provider.
Manage Secrets in HashiCorp Vault — uses the Vault provider.