Kubernetes Production Tuning

Goal

Tune an existing API7 Gateway Kubernetes deployment for production: size CPU and memory, add resilience with PodDisruptionBudget and HorizontalPodAutoscaler, spread pods with anti-affinity, and expose the gateway with a cloud-provider load balancer.

This guide assumes the control plane and data plane are already installed in a cluster. If you have not installed them yet, follow Deploy on Kubernetes first.

Before you begin

• A working API7 Gateway deployment in a Kubernetes cluster, installed from Deploy on Kubernetes.
• kubectl and Helm configured against the cluster.
• Permissions to create or update PodDisruptionBudget, HorizontalPodAutoscaler, and Service resources in the target namespace.

Set the variables this guide uses:

export API7_NAMESPACE="api7"
export GATEWAY_RELEASE="api7-ee-3-gateway"
export GATEWAY_EXTERNAL_IP="203.0.113.10"   # the address the LoadBalancer Service reports after Step 4

Steps

Step 1: Size CPU and worker processes

CPU requests, CPU limits, and NGINX worker processes control different things about a gateway pod, and they are configured together so that each worker has enough CPU capacity while leaving room for the other processes in the container.

If your cluster policy allows it, leave apisix.resources.limits.cpu unset, so the CPU quota never throttles gateway processes. If the policy requires a limit, set it no lower than the number of worker processes:

apisix.resources.limits.cpu >= nginx.workerProcesses

Each NGINX worker is a single-threaded process, so more traffic does not let one worker use more than roughly one core, and a CPU limit above the worker count does not change that — the extra capacity is for auxiliary processes and other container overhead.

apisix.resources.requests.cpu is used for Kubernetes scheduling and resource guarantees; it is not a hard per-process limit. For workloads without sustained high load or strict performance requirements, start from one requested core per worker process:

This 1:1 ratio is a practical baseline centred on worker processes, not a strict resource equation for every process in the container: auxiliary processes such as the privileged process and the cache manager consume CPU too. For high-load or performance-sensitive environments, add headroom on top of the baseline and use benchmark results and runtime monitoring to settle the final CPU request and, where one is required, the CPU limit.

Four fixed workers at the baseline:

nginx:
  workerProcesses: 4

apisix:
  resources:
    requests:
      cpu: "4"

The same, where the cluster policy also requires a CPU limit, with one additional core as container-level headroom:

nginx:
  workerProcesses: 4

apisix:
  resources:
    requests:
      cpu: "4"
    limits:
      cpu: "5"

Kubernetes accepts fractional CPU requests and limits such as 500m, but API7 Gateway uses the NGINX single-threaded worker model, so gateway performance may be constrained when each worker has well under one core. The effective number of worker processes also determines licensed DP core usage — see How Gateway Worker Processes Affect Licensed Cores.

Step 2: Set memory requests and limits

Memory is configured separately from CPU and worker processes. These are example starter values, not chart defaults:

apisix:
  resources:
    requests:
      memory: "4Gi"
    limits:
      memory: "8Gi"

Adjust both against your traffic profile, the plugins you enable, benchmark results and runtime monitoring. The general data plane capacity baseline is in System Requirements.

Step 3: Add pod disruption budget, autoscaling and anti-affinity

A PodDisruptionBudget keeps a gateway available through node maintenance, a HorizontalPodAutoscaler adds instances under load, and anti-affinity spreads the instances across nodes. Apply the PodDisruptionBudget directly; the autoscaler and the anti-affinity both come from the chart's values:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api7-gateway-pdb
  namespace: api7
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: gateway

kubectl apply -f pdb.yaml

poddisruptionbudget.policy/api7-gateway-pdb created

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70

apisix:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                    - gateway
            topologyKey: kubernetes.io/hostname

The chart renders the autoscaler only when the gateway runs as a Deployment; on a DaemonSet install it produces nothing. Autoscaling on a CPU target also requires a CPU request on the pods and a Metrics Server in the cluster — see Autoscale Data Plane on Kubernetes.

Step 4: Expose the gateway with a cloud load balancer

The gateway chart creates a NodePort Service by default, which is reachable through any node IP and is enough to verify the deployment from your own machine. On a cloud provider, switch the Service to LoadBalancer and read the address it provisions:

helm upgrade "${GATEWAY_RELEASE}" api7/gateway \
  --set "gateway.type=LoadBalancer" \
  -n "${API7_NAMESPACE}" --reuse-values

STATUS: deployed

The provider then takes a minute or two to attach an address:

kubectl -n "${API7_NAMESPACE}" get svc "${GATEWAY_RELEASE}-gateway" \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

203.0.113.10

--reuse-values keeps every value from the previous install and changes only the Service type, so the upgrade replaces the Service without touching the gateway pods.

EKS

On EKS, the AWS Load Balancer Controller provisions a Network Load Balancer:

apisix:
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "external"
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
      service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"

EKS also wants the AWS Load Balancer Controller installed before you use an NLB, the EBS CSI driver add-on for PostgreSQL persistent storage, and a kubeconfig entry for the cluster:

export AWS_REGION="us-west-2"          # the region your cluster is in
export EKS_CLUSTER_NAME="api7"         # the name of your cluster

aws eks update-kubeconfig --region "${AWS_REGION}" --name "${EKS_CLUSTER_NAME}"

Updated context arn:aws:eks:us-west-2:…:cluster/api7 in ~/.kube/config

For performance-sensitive deployments, put the two halves on dedicated node groups and schedule them with a node selector or node affinity:

apisix:
  nodeSelector:
    nodeName: api7-dp

GKE

GKE provisions a TCP load balancer for a LoadBalancer Service on its own. For HTTP(S) load balancing with Google Cloud Armor, or for internal-only access:

apisix:
  service:
    annotations:
      cloud.google.com/l4-rls: "enabled"
      networking.gke.io/load-balancer-type: "Internal"  # For internal-only access

GKE also wants Workload Identity enabled if you use Cloud SQL for PostgreSQL, and a Cloud SQL PostgreSQL instance with a private IP in your VPC. Open the firewall ports listed in System Requirements to your GKE nodes.

AKS

AKS uses the Standard SKU load balancer by default. For internal access:

apisix:
  service:
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"

On AKS, use the Azure Disk CSI driver — enabled by default on AKS 1.21+ — for PostgreSQL persistent storage, provision an Azure Database for PostgreSQL server for an external database, and use Azure CNI for predictable pod IP allocation in production.

Validate

Check that the data plane pods are ready, the HPA exists, and the gateway Service has an external endpoint:

kubectl -n "${API7_NAMESPACE}" get pods -l app.kubernetes.io/name=gateway
kubectl -n "${API7_NAMESPACE}" get hpa
kubectl -n "${API7_NAMESPACE}" get svc "${GATEWAY_RELEASE}-gateway"

Expected: all gateway pods are Running and Ready, the HPA is OK, and the Service shows an EXTERNAL-IP when LoadBalancer is configured.

Send one request to the gateway from outside the cluster:

curl -so /dev/null -w '%{http_code}\n' "http://${GATEWAY_EXTERNAL_IP}/"

Expected: 404 if no routes are configured, which proves the gateway is up and accepting traffic.

Roll back / Clean up

To remove only the production-tuning resources added in this guide:

kubectl -n "${API7_NAMESPACE}" delete poddisruptionbudget api7-gateway-pdb
kubectl -n "${API7_NAMESPACE}" delete horizontalpodautoscaler "${GATEWAY_RELEASE}-gateway"

The chart names the HorizontalPodAutoscaler after the release, the same as the Service (${GATEWAY_RELEASE}-gateway) — not the literal api7-gateway-hpa. Run kubectl -n "${API7_NAMESPACE}" get hpa first if you are unsure of the exact name in your cluster.

To revert the Service to NodePort:

helm upgrade "${GATEWAY_RELEASE}" api7/gateway \
  --set "gateway.type=NodePort" \
  -n "${API7_NAMESPACE}" --reuse-values

Troubleshoot

Related

• Deploy on Kubernetes — install the control plane and data plane before applying this guide.
• Running in Production — general production checklist and recommendations.
• Production Best Practices — security and operational best practices.
• System Requirements — hardware and software baseline.
• Helm Chart — every value accepted by the gateway chart.