Linux for backend engineers

The service runs on Linux, and the JVM lessons stopped at the boundary where the kernel takes over: the process that ended with no Java log, the Too many open files that stopped accepting connections, the port that was already in use after a restart. This lesson is the kernel's side of those boundaries — enough to read what the system is telling you, and the commands that show it.

#Processes and signals

A running JVM is a process: a PID, a parent, a user, an environment, open files, and a set of threads the kernel schedules (ps -T -p <pid> lists them; the thread-dump lesson's nid is the kernel's thread id in hex). ps aux, top and htop show them; /proc/<pid>/ is the truth they read from — status for memory and threads, fd/ for open files, environ for the environment, limits for the caps.

The kernel talks to a process with signals, and a service must handle three:

The rule from the Docker course applies here too: the signal reaches the JVM only if the JVM is the process it was sent to. A sh -c "java ..." wrapper receives the SIGTERM and does not forward it, so the JVM sees nothing until the SIGKILL arrives. exec java ... in a script, or the exec-form entrypoint, makes Java the process. Exit codes tell the story afterwards: 0 clean, 1 a Java exception or System.exit(1), 137 ended by signal 9 (128 + 9), 143 terminated by signal 15.

#File descriptors

Every open file, socket, pipe and epoll instance is a file descriptor, an integer in the process's table, and the table has a limit: ulimit -n in a shell, LimitNOFILE= in a systemd unit, the container runtime's default. The default of 1024 on some systems is a number a Java service reaches in minutes — every accepted connection is a descriptor, every outbound HTTP call, every open log file, every jar on the classpath while it is being read — and the failure is java.io.IOException: Too many open files, which stops accepting connections as well as opening files, so a leak of file handles becomes an outage of the whole listener.

ls /proc/<pid>/fd | wc -l is the count; lsof -p <pid> says what they are, and a leak shows as the same file or the same remote address repeated hundreds of times — the Files.lines without a try from the I/O course, or an HTTP client created per request instead of once. Set the limit high (65536 is common) and find the leak; raising the limit alone moves the outage to next week.

#Memory, and the OOM killer

The memory-areas lesson measured what a JVM uses; the kernel sees only the total, the process's resident set size (RSS, in ps and top), and enforces a limit through the cgroup the container or the systemd unit sets (MemoryMax=1G on this repository's API unit). When the limit is hit — or, on a host without limits, when the whole machine runs out — the OOM killer picks a process, by a score that favours the largest, and sends SIGKILL. The evidence is not in the application's log; it is in the kernel's: dmesg -T | grep -i 'killed process' or journalctl -k, with the process name, its RSS, and the cgroup.

Two things to know when reading it. free -m shows memory "used" by the page cache as available, which it is — the kernel drops cache under pressure — so a host at 95% "used" may be fine and a container at 95% of its limit is not. And swap on a JVM host is a latency disaster rather than a safety net: a heap page swapped out turns a 10 ms GC pause into seconds; most JVM hosts run with swap off (vm.swappiness=0 or none configured) and a limit that ends the process instead.

#Ports and sockets

A service listens on a port; ss -tlnp (the modern netstat -tlnp) lists who is listening on what, with the PID, and it is the first command when the answer to "is it up" is unclear. ss -tnp | grep :5432 shows the connections to the database, and their states: ESTABLISHED is a live connection, TIME_WAIT is a closed one the kernel holds for a minute (thousands of them after a load test is normal), CLOSE_WAIT in large numbers is your process not closing what the other side closed — a client leak, and the Too many open files above is where it ends.

Ports below 1024 need root or the CAP_NET_BIND_SERVICE capability, which is why a service listens on 8080 and nginx on 443. Address already in use after a restart is the old process still holding the port, or a socket in TIME_WAIT without SO_REUSEADDR, which Java's server sockets set by default. And a service that should not be reachable from outside binds 127.0.0.1, as this repository's API does on 8082 — the firewall is a second layer, not the first.

#systemd and journald

On a virtual machine the service is a unit: a file that says what to run, as whom, with what environment and limits, and what to do when it dies.

[Service]
User=code10x
EnvironmentFile=/etc/code10x.env
ExecStart=/usr/lib/jvm/java-25-openjdk-amd64/bin/java -Xms128m -Xmx384m -XX:MaxMetaspaceSize=160m -jar /srv/code10x/current/backend/api/code10x-api.jar
Restart=always
RestartSec=5
MemoryMax=1G
LimitNOFILE=65536

Restart=always is the supervisor; EnvironmentFile keeps credentials out of the unit (which anyone who can run systemctl cat would otherwise see — the file's own comment says so); MemoryMax is the cgroup limit; User is the non-root rule. systemctl status, start, stop, restart, enable (start at boot) and daemon-reload after editing the file are the whole interface.

Its logs go to journald, which is where you read them: journalctl -u code10x-api -f follows, --since "1 hour ago" bounds, -p err filters by priority, -k is the kernel log where the OOM killer writes. The observability course's structured logging still applies — the JSON goes into the journal as the message — and a log shipper reads the journal to send it onward.