Nginx as a reverse proxy

nginx sits between the internet and the Java service: it terminates TLS, serves the static files, forwards /api to the JVM, adds the headers, and answers with a 502 when the JVM does not answer at all. Every backend engineer ends up reading its configuration during an incident, and this lesson is what that configuration means, using this repository's own.

#What a reverse proxy is for

A forward proxy sits in front of clients; a reverse proxy sits in front of servers, and clients talk to it as if it were the server. In front of a Java service it does the work the JVM should not:

• TLS termination — certificates, ciphers, HTTP/2, renewal — in one place, so the service speaks plain HTTP on the loopback interface.
• Static files, from disk, at a speed and a memory cost no servlet container matches; this repository's whole site is a static export nginx serves, and only /api/*, /admin/api/* and the blog covers reach Java.
• One public port in front of several processes, and the routing between them by path.
• Buffering slow clients so a worker thread in Tomcat is not held for the duration of a mobile upload.
• Limits — body size, request rate, connections — enforced before a request costs the JVM anything.

location /api/ {
    proxy_pass http://127.0.0.1:8082;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    client_max_body_size 8m;
}
location / {
    root /srv/code10x/current/site;
    try_files $uri $uri.html $uri/index.html =404;
}

#TLS

listen 443 ssl http2;
ssl_certificate     /etc/letsencrypt/live/code10x.in/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/code10x.in/privkey.pem;

Certbot writes those lines and renews the certificate on a timer; the http2 flag on the listen line enables multiplexing for the static assets (the file's comment notes it is the 1.24 spelling; newer versions use http2 on;). Port 80 exists only to redirect to 443. The security headers — HSTS, X-Content-Type-Options, the Content-Security-Policy — are added here on every response, and in this repository they are generated from the same table next.config.mjs uses for development, so the two cannot disagree; add_header has one trap worth knowing, which the generated file's comment states: a location that adds any header of its own drops the inherited ones, so the include appears in every such block.

#Forwarded headers: the service must know who really asked

Behind a proxy, the connection the JVM sees comes from 127.0.0.1, over plain HTTP. If the application reads request.getRemoteAddr() it gets the proxy; if it builds an absolute URL it gets http://. The proxy_set_header lines carry the truth: X-Real-IP and X-Forwarded-For for the client's address, X-Forwarded-Proto for the scheme, Host for the name the client used. Spring reads them when told to — server.forward-headers-strategy=native for Tomcat's own valve, or framework for Spring's ForwardedHeaderFilter — and only then, because trusting X-Forwarded-For from an arbitrary client is how rate limits and audit logs get a forged address. This repository's throttles read X-Real-IP, else the last X-Forwarded-For entry — the one nginx appended — and never the first, which the client controls; the learner-test that checks this names exactly that reason.

#Timeouts and buffering

Three timeouts on the proxy side, and they must agree with the JVM's:

• proxy_connect_timeout — to open the connection to the upstream (seconds; the JVM is on loopback, so a failure here is "not listening").
• proxy_read_timeout — between two reads from the upstream, default 60 s. A request the JVM takes 90 s to answer is a 504 from nginx at 60 s while the JVM keeps working, which is the microservices course's "timeout that was measured, then wrong" from the other side. Set it at or above the slowest legitimate request, and make that request faster instead.
• proxy_send_timeout — between two writes to the upstream.

Buffering (proxy_buffering on, the default) reads the upstream's whole response into memory or a temp file and then sends it to the client at the client's pace, freeing the JVM's thread early — right for a JSON API, wrong for a server-sent event stream or a large download, where proxy_buffering off (or X-Accel-Buffering: no from the application) streams through. client_max_body_size is the request-body cap, 8 MB here for cover uploads, and exceeding it is a 413 from nginx before the JVM sees a byte. proxy_request_buffering is the same idea for uploads.

#Reading a 502, a 504 and a 499

• 502 Bad Gateway — nginx could not get a valid response: the upstream is not listening (the JVM is restarting — the deploy script's few-second window), refused the connection, closed it mid-response, or answered with something nginx could not parse. systemctl status the service, ss -tlnp | grep 8082, then nginx's error log: connect() failed (111: Connection refused) is the JVM down; upstream prematurely closed connection is the JVM dying mid-request, usually the OOM killer, so the kernel log is next.
• 504 Gateway Timeout — the upstream was reached and did not answer within proxy_read_timeout. The JVM is alive and slow: a thread dump or the slow-query checklist from the production course.
• 499 — nginx's own code for the client went away before the response; a burst of them means clients timing out on their side, which is a latency problem that the 504s have not caught yet.
• 413 — the body limit above. 404 on a static site is try_files finding nothing, and this repository's export check exists to make sure every promised page is on disk before the switchover.

The two logs: /var/log/nginx/access.log with the status, the upstream time ($upstream_response_time, worth adding to the log format) and the request; /var/log/nginx/error.log with the reason. nginx -t validates a configuration before systemctl reload nginx applies it without dropping connections — which is what the deploy script runs, in that order, at every switchover.