Leveling Up: An Advanced Linux Tutorial for Real World Ops

The 3 AM Wake-Up Call That Changed My Configs

I still remember the cold sweat dripping down my back around 3 AM on a Tuesday in 2018. We had just scaled our primary application cluster to handle a massive influx of traffic—think 50,000 concurrent connections hitting a fleet of fresh Debian servers. Everything looked green on the dashboard until, suddenly, it wasn't. The database connections started dropping, the web servers locked up, and SSH became unresponsive. It wasn't a hacker, and it wasn't a DDoS attack. It was a default file descriptor limit.

See, I’d deployed standard images. I assumed the defaults were sane enough to handle the load. I was wrong. The default `ulimit` of 1024 open files choked the Nginx processes faster than you can say "Connection Refused." That night taught me a painful lesson: Linux out of the box is designed for compatibility, not high-performance production. If you aren't tuning your kernel and hardening your shell before traffic hits, you're just waiting for a disaster.

Why Defaults Are Your Enemy in Production

Here's the thing about Linux distributions—whether you're running RHEL 9, Ubuntu 22.04 LTS, or Alpine. The maintainers have to ship a kernel that boots on a Raspberry Pi, a dusty laptop from 2012, and an AWS `c6g.metal` instance. To do that, they leave the settings incredibly conservative.

When we talk about "Advanced Linux in Production," we aren't talking about installing packages or managing users. We're talking about taking that generic, vanilla kernel and molding it into a specialized tool that can survive the hostile internet. I’ve audited hundreds of servers in the last decade, and honestly, 90% of the issues I see stem from administrators treating a server like a desktop. They install the app and walk away. But in a production environment, specifically when you're dealing with high throughput or sensitive data, "it works" isn't good enough. You need it to be resilient, and you need it to be boring. Boring is good. Excitement usually means downtime.

1. Stop Disabling SELinux (Seriously, Stop It)

I know, I know. The first thing most of us learned to do when a service wouldn't start was to run `setenforce 0` and edit `/etc/selinux/config` to `SELINUX=disabled`. I did it for years. It felt like SELinux existed solely to break my httpd configurations or stop my reverse proxies from talking to upstream ports. But disabling it in 2024 is irresponsible.

SELinux (or AppArmor, if you're in the Debian/Ubuntu camp) provides Mandatory Access Control (MAC). It limits what a process can do, even if that process is running as root. If a vulnerability is found in your web server daemon—like the Nginx zero-days we see pop up—SELinux can prevent that compromised process from reading /etc/shadow or opening a reverse shell to an attacker's IP.

Instead of turning it off, learn to use `audit2allow`. When something breaks:

• Check the audit log: `grep avc /var/log/audit/audit.log`
• Pipe it to the tool: `grep avc /var/log/audit/audit.log | audit2allow -w`
• Generate a custom policy module if the action is legitimate.

It takes maybe five extra minutes during the setup phase, but it saves you from a total compromise later. I’ve seen SELinux stop a remote code execution exploit dead in its tracks because the attacker's shell couldn't execute binaries in `/tmp`. That’s worth the headache.

2. Kernel Tuning via sysctl: The Performance Layer

Most default TCP stacks are tuned for a 100Mbps LAN connection, not a 10Gbps cloud link. If you aren't touching `/etc/sysctl.conf`, you are leaving performance on the table. I usually apply a standard set of tunables to any node serving web traffic.

First, look at your connection tracking. If you see "nf_conntrack: table full, dropping packet" in `dmesg`, you've hit the wall. You need to bump `net.netfilter.nf_conntrack_max`. But more importantly, look at how your server handles closing connections.

A few settings I rely on (check your kernel docs, specifically for 5.15+):

• net.core.somaxconn: Bump this from the default (often 128 or 4096) to something like 65535. This is the backlog of connections waiting to be accepted.
• net.ipv4.tcp_tw_reuse = 1: This allows reusing sockets in TIME_WAIT state for new connections. Without this, in high-load scenarios, you’ll run out of ephemeral ports.
• vm.swappiness: The default is usually 60. On a dedicated database server with 64GB of RAM, having the kernel swap out application memory to disk just because the filesystem cache is growing is terrible for latency. I usually drop this to 1 or 10.

Mistake I made: I once set `tcp_tw_recycle = 1` on a NAT gateway. It broke connections for half our users because of timestamp issues. It was deprecated and removed in later kernels, but I learned the hard way to read the RFCs before copy-pasting sysctl configs from old forum posts.

3. Observability: eBPF is the New Top

If you're still relying solely on `top`, `htop`, and `iotop` to debug production performance, you're looking at the world through a keyhole. Those tools give you averages. They tell you CPU is high, but they don't tell you why. A few years back, I started digging into eBPF (Extended Berkeley Packet Filter), and it completely changed how I troubleshoot.

With tools like `bcc-tools` or `bpftrace`, you can inspect the kernel in real-time with zero overhead. For example, if disk I/O is slow, `iotop` shows you which process is writing. But `biolatency` (from bcc-tools) shows you a histogram of exactly how long disk I/O requests are taking at the driver level.

I remember chasing a "ghost" latency issue on a Redis instance. The CPU was low, memory was fine, but requests were taking 200ms randomly. Using `tcptop` and `ext4slower` via eBPF, we found that a specific background logging rotation was causing micro-stalls in the filesystem journaling. Standard monitoring tools never even registered a spike. If you're running Linux kernel 4.15 or newer (which you should be), start learning these tools.

4. SSH Hardening and Access Control

This seems basic, but I audit environments every month where this is messed up. SSH is the front door. You need to bolt it shut. In `/etc/ssh/sshd_config`, there are lines that should be non-negotiable in 2024.

PermitRootLogin no. Never log in as root directly. Log in as a user and elevate. This creates an audit trail.
PasswordAuthentication no. SSH keys or certificates only. Brute force attacks are constant background noise on the internet; don't give them a chance to guess "password123".
AllowUsers or AllowGroups. Explicitly list who can log in. If a service account exists on the box but shouldn't have shell access, ensure it's not in the allow list.

Another tip that saved me: implement 2FA on SSH. Using the Google Authenticator PAM module isn't hard to set up. It adds a `Verification code:` prompt after your SSH key handshake. It’s a minor annoyance for you, but a massive wall for an attacker who managed to steal your private key from your laptop.

5. Immutable Infrastructure and "Drift"

The old way of managing Linux was treating servers like pets. You gave them names (Zeus, Apollo, Gandalf), you nursed them back to health when they got sick, and you manually patched them on Friday nights. This doesn't work at scale. It leads to "Configuration Drift," where Server A behaves slightly differently than Server B because someone made a manual change six months ago and forgot to document it.

The advanced practice here is to treat servers as immutable. If you need to change a configuration, you don't SSH in and edit a file. You update your Ansible playbook or your Terraform code, build a new image, and replace the server.

I use Ansible for configuration management. It’s agentless, which keeps the production servers clean. But the discipline requires that you never make manual changes. If you fix a bug manually on the server to stop the bleeding during an outage, your next step must be to backport that fix to the code. If you don't, that fix disappears the next time the server is redeployed. I've wiped out critical hotfixes because I forgot this step. It’s embarrassing to have the same outage twice in one week.

FAQ: Common Production Questions

Should I use Swap space on modern servers with lots of RAM?

This is controversial, but yes, I think you should have a small amount of swap. Even with 128GB of RAM, anonymous memory pages (memory not backed by a file) can't be reclaimed easily without swap. If you have absolutely zero swap and hit memory pressure, the OOM (Out of Memory) killer engages immediately and shoots your database in the head. A small swap partition gives the kernel a place to push inactive pages, buying you time to react. Just keep `vm.swappiness` low.

Which filesystem should I use: EXT4 or XFS?

For years, EXT4 was the safe, default choice. However, in the RHEL/CentOS ecosystem (and increasingly elsewhere), XFS is the default for a reason. XFS handles large files and parallel I/O operations significantly better than EXT4. If you are running a large database or a file server, XFS is generally the superior choice. However, EXT4 is still slightly more resilient to power failures and easier to shrink if you need to resize volumes downwards (XFS can only grow, not shrink).

How do I handle logs so they don't fill the disk?

Log rotation is critical, but often configured poorly. Don't rely solely on `logrotate` running once a day. If an app goes crazy and writes 100GB of errors in an hour, `logrotate` won't save you. I recommend setting up remote logging (rsyslog or filebeat) to ship logs off the server immediately. Locally, use `journald` limits. Set `SystemMaxUse=1G` in `/etc/systemd/journald.conf`. This ensures system logs never chew up your entire root partition.

Is it worth compiling my own kernel?

Honestly? Almost never. Ten or fifteen years ago, we did this to strip out drivers and gain a 1% performance boost. Today, the maintenance burden is massive. You become responsible for security patching. The generic kernels provided by Red Hat, Canonical, or Amazon are highly optimized and tested. Unless you need a specific hardware driver that isn't upstream yet or you are doing high-frequency trading where microseconds equal millions of dollars, stick to the distro kernel.

My Take: It's a Moving Target

Here is the reality of working with Linux in production: you are never "done." Security standards change. New kernel features like eBPF or io_uring come out and change the performance landscape. The settings I used in 2019 are obsolete today.

The best admins I know aren't the ones who memorized the man pages for `tar`. They are the ones who are constantly auditing their own assumptions. They assume their current config is probably slightly broken and are looking for ways to improve it. Start with the basics: lock down SSH, tune your sysctl, enable SELinux, and automate everything. Once you have that foundation, you can sleep a lot better at night. And trust me, avoiding that 3 AM wake-up call is worth the effort.

Related Posts