Troubleshooting intermittent 502 Bad Gateway errors behind a load balancer

Symptom

Intermittent HTTP 502 Bad Gateway errors on sites behind the load balancer, typically after a maintenance window or an upstream service restart.

Root causes we see most often

1. PHP-FPM pool exhaustion — workers not restarted after config changes; check systemctl status php8.2-fpm and the FPM log for "server reached pm.max_children".
2. Stale upstream in the load balancer — backend restarted with a new IP but the balancer kept the old target; reload the upstream config or update the target group.
3. Keepalive mismatch — upstream keepalive timeout shorter than the proxy's, so the LB reuses a connection the backend already closed.

Resolution steps

1. Confirm backend health directly from the LB host: curl -I http://BACKEND_IP:PORT/health
2. Restart PHP-FPM and verify the worker count: systemctl restart php8.2-fpm
3. Reload the proxy so upstreams re-resolve: nginx -s reload
4. Align keepalive: upstream keepalive_timeout 75s or higher than the proxy timeout.
5. Watch the error rate for 15 minutes: tail -f /var/log/nginx/error.log | grep 502

Escalation

If 502s persist above 0.5% of requests after these steps, escalate to the infrastructure team with the LB error log excerpt and health-check output.