The Linux toolbox: namespaces, networking, systemd, and the commands that earn their place.
94 posts
Everyone assumes a bigger network buffer is safer. Run this simulation and watch a large buffer turn a fast link into a link that feels slow — while the throughput number stays exactly the same.
Homelab power consumption measured, not guessed: real RAPL readings of idle power vs busy on an i7-8550U, a CPU governor A/B test, and an honest checklist for a power-efficient server.
A hands-on guide to gdb for debugging C crashes: backtraces, core dumps, conditional breakpoints, watchpoints, threads, and the -O0 vs -O2 story — with real output from real programs.
A hands-on session with valgrind on Linux: real memcheck reports for memory leaks in C, invalid reads and writes, uninitialised values, plus massif heap profiles, callgrind instruction counts and helgrind data races — with actual output and the fixes.
A hands-on tour of inotify, the Linux kernel's file-change event API: real inotifywait and inotifywatch output, a Python ctypes reader, and a reproduced event-queue overflow.
Policy routing with iproute2 lets `ip rule` pick a routing table by source address, fwmark, or interface — not just the destination. Real `ip route get` output for two uplinks, source-based routing, split tunnels, and IPv6.
Real fio numbers on a Docker 29 host: how the cgroup v2 io controller caps per-container disk throughput, what --device-write-bps actually writes, io.max vs io.weight, and why buffered dd lies.
What Linux swap actually is (and why it is not RAM), what vm.swappiness really controls, how to read memory pressure from vmstat si/so, pswpin/pswpout and /proc/pressure, and when zram beats a disk swapfile — real output, plus a 400 MiB allocation test under a Docker cgroup limit.
A practical guide to tmux sessions, windows, and panes: how a terminal multiplexer keeps your work alive across dropped SSH connections, how to script reproducible layouts, and how send-keys and capture-pane enable real automation.
Restart=on-failure only catches processes that exit. This shows how WatchdogSec plus sd_notify heartbeats catch the ones that hang instead, with real systemd journal output, StartLimitBurst crash-loop protection, and Type=notify readiness.
Linux signals are asynchronous notifications, not message queues. Learn the difference between SIGTERM and SIGKILL, how signal handlers enable graceful shutdown, why sigprocmask defers delivery, and the PID 1 container trap where your app must handle SIGTERM itself.
Why a 500 Mbit/s link delivers 14 Mbit/s: the receive window, TCP window scaling, the bandwidth-delay product, and how socket buffers and autotuning set your real ceiling — measured with real iperf3, tcpdump, netem, and ss -ti output.
Where do file reads really happen? I measure the Linux page cache with /proc/meminfo, show a 25x cold-vs-warm read, count read() vs mmap() syscalls with strace, and demonstrate msync, fadvise, and why drop_caches is the wrong tool.
io_uring trades one syscall per I/O for a shared ring between user space and the kernel. Real fio ioengine comparisons on Linux 7.2: what queue depth changes, what it doesn't, and where SQPOLL and registered buffers actually help.
Forward-simulating the optics behind glass-bead 3D pictures: a per-pixel GPU shader ray tracer for a glass bead lens array, thin-lens deflection vs Snell's law refraction, why moiré magnification and parallax appear, the Qt+moderngl context trap, and inverse design that prints a 3D model.
Recursive CTEs walk hierarchical data with a base case and a self-referencing term. Learn WITH RECURSIVE, depth/path accumulation, CYCLE protection, SEARCH ORDER, EXPLAIN plans, and when to use materialized path or ltree instead.
A hands-on Linux capabilities guide with real Docker demos: why root is too much power, how ~40 capabilities split it up, the five sets, capsh, /proc/self/status, getcap/setcap, bounding sets, CAP_SYS_ADMIN, and the drop-all-add-back recipe.
Why 'my disk is slow' is usually the wrong question. Real fio benchmarks showing IOPS vs throughput vs latency, the queue-depth effect, the true cost of fsync, and how tmpfs vs a LUKS-encrypted NVMe SSD changes everything.
How a host gets an IPv6 address without DHCP — RS/RA exchange, prefix information, A/M/O flags, EUI-64 vs stable-privacy, and real radvd output.
A hands-on tour of Linux CPU scheduling on an 8-core box: how CFS shares a core by weight, why nice/renice only bites under load, when CPU affinity and taskset help vs hurt, the SCHED_FIFO danger, how docker --cpus throttles via cpu.max, and how to spot a pegged core with mpstat -P ALL and pidstat.
Postgres triggers explained with real output from a throwaway container: BEFORE/AFTER row triggers, FOR EACH STATEMENT, NEW/OLD, WHEN conditions, INSTEAD OF on views, RAISE EXCEPTION, trigger ordering, the visibility gotcha, and the per-row performance trap.
A hands-on tour of journalctl and the systemd journal on a real Arch host: filter by unit, priority and time, follow logs live, read journal fields like _SYSTEMD_UNIT and _PID, add custom fields, script with -o json, and manage journald retention with --disk-usage and vacuum.
What memory overcommit is, the three vm.overcommit_memory modes, Committed_AS vs CommitLimit, why the OOM killer picks the biggest RSS via oom_score, cgroup limits vs the global killer, swap's role, and how to read the evidence — all with real output from throwaway containers.
systemd socket activation, made concrete: systemd owns the listening socket (ListenStream), starts your service on the first connection, and passes the fd via sd_listen_fds. Real systemctl --user output, the .socket/.service pair, systemd-socket-activate, and the honest tradeoffs.
A half-open connection can linger for hours. Learn how TCP keepalive (SO_KEEPALIVE, TCP_KEEPIDLE) and application-level heartbeats detect dead peers, with real experiments and timings.
Postgres has a built-in pub/sub mechanism called LISTEN/NOTIFY. It is small, fast, and already running in your database. Here is how to use it, its real limits, and when to reach for a real queue instead.
What a file descriptor is, why files, sockets and pipes share one limit, how to read soft vs hard limits with ulimit -n and /proc/<pid>/limits, what EMFILE means, and a checklist for fixing 'too many open files' — raise the limit, then find the leak.
Use tc netem and tbf to emulate WAN latency, packet loss, and rate limits on Linux. Real measurements from a throwaway Docker container: ping RTT, throughput before/after, and tc -s counters.
Window functions compute across related rows without collapsing them. Learn ROW_NUMBER, RANK, LAG, running totals, frames, and the top-N-per-group pattern with real query output.
A hands-on tour of Go concurrency bugs with real race-detector output: a data race reproduced with go run -race and go test -race, a deadlock that trips the runtime, and a goroutine leak — plus the sync.Mutex, channel-ownership, and context cancellation fixes that pass -race clean.
A ground-up walkthrough of the Linux boot process on a real Arch Linux laptop: UEFI firmware and the ESP, GRUB, kernel and initramfs, systemd as PID 1, the sysinit → basic → multi-user → graphical target chain, systemd-analyze timing, and a debugging playbook for slow or failed boots.
How I hunt a Python memory leak without guessing: build a reproducer, read RSS from /proc, diff tracemalloc snapshots to name the growing allocation, walk gc.get_referrers to find who holds it, then prove the fix with the same numbers.
A hands-on guide to Postgres JSONB: jsonb operators, GIN indexing with jsonb_ops vs jsonb_path_ops, containment queries, and when a document column is the right — or wrong — tool.
A hands-on tour of bash strict mode: what set -euo pipefail really does, the cases where set -e silently gives up, how word splitting and IFS break scripts, cleaning up with trap EXIT and mktemp, reading PIPESTATUS, and why shellcheck is the real safety net. Every pitfall shown with a real script that fails.
The systemd unit file directives I actually use: ExecStart, EnvironmentFile, User, After vs WantedBy, Restart=on-failure and start limits, Type=simple vs exec vs oneshot, journal logging, timers, systemctl edit drop-ins, and a real systemd-analyze verify failure I fix.
A practical awk tutorial and sed tutorial with real output: field selection, sums and averages, BEGIN/END, grouped counts, sed replace, capture groups, line ranges, in-place edits with backups, and when to stop and use jq or Python instead.
A practical walkthrough of the find command and ripgrep as a grep replacement: search files by name, type, size, mtime, user and permission; why -print0 and xargs -0 matter; ripgrep's smart filtering, ignore files and binary detection; fd as a friendlier find; and real recipes.
A hands-on Linux permissions guide with real terminal output: what rwx means for files versus directories, how umask sets default modes, setuid/setgid/sticky demos, ACLs and default ACLs with setfacl, the classic mistakes, and a security checklist.
A hands-on tour of Postgres index types on one 2M-row table: when B-tree, GIN, GiST, BRIN and hash indexes are the right tool, when they are a mistake, real EXPLAIN (ANALYZE, BUFFERS) plans, real index sizes, and how CREATE INDEX CONCURRENTLY, INCLUDE, partial and expression options fit in.
A hands-on tour of cgroups v2 on a real Docker host: why v2 unified the controllers, the filesystem layout, how --memory and --cpus map to memory.max and cpu.max, CPU throttling, memory accounting, OOM kills, memory.high vs memory.max, and using PSI to debug containers that die or crawl.
A hands-on look at Postgres connection pooling with PgBouncer: why every connection is a process, why raising max_connections only moves the ceiling, how pool modes differ, and the ways transaction mode broke session state, temp tables and prepared statements — with real output.
A hands-on comparison of Postgres full-text search and Elasticsearch on the same corpus: tsvector, tsquery, ts_rank, GIN indexes, pg_trgm for typo search, real EXPLAIN plans and ES responses, what Postgres does not do, and the decision rule for when a search cluster earns its keep.
Postgres vacuum explained with real output: I generated table bloat on a throwaway Postgres 17, tuned autovacuum thresholds, watched VACUUM mark dead tuples reusable while VACUUM FULL shrank the file, and checked wraparound with pg_stat_user_tables and pgstattuple.
A throwaway Postgres 16 container, a 3-million-row table and its declaratively partitioned twin: real EXPLAIN plans showing partition pruning, the index-per-partition reality, manual monthly partitions, ATTACH and DETACH, and why dropping a partition beats DELETE for retention.
A load test answers three questions — your capacity, where it breaks, and whether your last change helped. Run a hey benchmark against your own container, read latency percentiles instead of the mean, find the knee where throughput plateaus and p99 climbs, and correlate it with server CPU.
A restic backup restore drill, start to finish with real output: init, encrypted deduplicated backup, restic forget retention and prune, check --read-data, a wipe-and-restore, content-hash and diff verification, a timer plus a dead man's switch, and the 3-2-1 anti-patterns.
Both databases are excellent, so the real differences are defaults, types, and SQL dialect. Real output from throwaway MySQL 8.4 and PostgreSQL 17 containers: isolation levels, NULL and boolean quirks, identifier case, ON CONFLICT vs ON DUPLICATE KEY, RETURNING, jsonb, arrays, and a decision list.
Postgres replication reproduced end to end: building a physical standby with pg_basebackup and primary_conninfo, reading pg_stat_replication and pg_stat_wal_receiver, measuring real lag, the synchronous tradeoff, and why failover stays manual.
Stand up a container DNS server in minutes: a minimal CoreDNS Corefile serving a custom authoritative zone from a zone file, forwarding unknown names upstream, and every answer poked with dig @127.0.0.1 -p 15353 — including real NXDOMAIN, SERVFAIL, CNAME-chain and TTL output.
EXPLAIN, EXPLAIN ANALYZE and BUFFERS explained with real plans from a 1.5M-row table: how to read a Postgres query plan bottom-up, estimates vs actual rows, stale statistics, seq scan vs index scan, nested loop vs hash join, index-only scans, and a repeatable method for finding the slow query.
Postgres locks and deadlocks, reproduced on a throwaway Postgres 17. What UPDATE and SELECT ... FOR UPDATE lock, finding blockers with pg_locks and pg_stat_activity, the lock queue that stalls everything, and avoiding deadlocks with lock ordering, SKIP LOCKED and lock_timeout.
A practical method for git archaeology: use git log -S (the pickaxe) and -G to find when a string changed, git log -L to follow one function, git blame and its -w/-C limits, --follow across renames, and git bisect run to catch the commit that broke it — then read the why, not just the what.
A practical regex tutorial that makes regular expressions click: the mental model, the atoms, capture groups and greedy vs lazy quantifiers, alternation and precedence, word boundaries in real patterns, why catastrophic backtracking happens, and how to test regex examples against real inputs.
A practical set of openssl commands to inspect a certificate, verify a certificate chain, and check a live TLS server with openssl s_client. Real output from x509, req, verify, and pkcs12, plus PEM vs DER vs PKCS#12 conversion and the leaf-vs-chain and key-vs-cert confusions.
Hands-on with the sqlite3 CLI: open a database, use the dot commands, import and export CSV, run one-off sqlite commands with -line/-json/-csv, script them with .read, take a safe sqlite backup instead of cp, set WAL and foreign keys with sqlite pragma, and read EXPLAIN QUERY PLAN.
The curl flags I use daily, explained with real output: -v/-i/-D to read requests, curl -w timing, -L and --max-redirs, --fail and exit codes, -d/--json/-F bodies, headers and auth, HTTP/2 and HTTP/3, --resolve for testing, and why -v beats guessing.
A hands-on jq tutorial for real work: pretty-print and compact output, selecting and select, map and to_entries, // defaults, slurp, --arg, -e exit codes, group_by, sort, unique, and --stream. Every jq command here comes with real, copy-pasteable output.
A practical guide to BGP in a homelab: what the protocol actually does, when multi-homing, ISP failover, or your own IPv6 PI space justify it, the BIRD and FRR options, the concepts (AS, prefix, local-pref, route filtering) you must understand, a minimal config, and an honest recommendation.
TCP congestion control is how the kernel picks its own send rate with almost no knowledge of the path. Slow start, congestion avoidance and fast recovery; why CUBIC is Linux's default; how BBR models the bandwidth-delay product; reading `ss -ti`; switching to BBR with sysctl; and bufferbloat.
A practical guide to syscall debugging with strace: when it beats a debugger or logs, how to attach to a process safely, how to read open/read/connect/futex output and errno, the filters worth memorising (-e trace, -f, -p, -y, -T, -tt), three real workthroughs, and where ltrace and bpftrace fit.
A practical nftables guide: the tables/chains/rules model, a complete default-deny nftables ruleset for a server, sets and named intervals for IP lists, masquerade and port-forwarding NAT, persistence, and how to read the ruleset without locking yourself out.
A practical guide to an IPv6 home network: check that your ISP really does prefix delegation, get the router advertising a /64 over SLAAC, lock the IPv6 firewall to default-deny inbound, publish the right AAAA record, and debug the stall a broken IPv6 path causes.
authorized_keys sprawl breaks the moment you hire someone or rotate a key. SSH certificates fix it: one SSH CA signs user certs with principals and expiry, host certificates kill the TOFU prompt, and short-lived credentials remove the need for revocation. Copyable ssh-keygen -s and sshd config.
How to sandbox systemd services without breaking them: the directives that matter (NoNewPrivileges, ProtectSystem=strict, PrivateTmp, SystemCallFilter, CapabilityBoundingSet, dynamic user), an incremental workflow measured with systemd-analyze security, and how to test and roll a unit back.
A field report after a month of running NixOS: the appeal of declaring a whole system in one file, what a configuration.nix actually looks like, the real learning curve around the Nix language and Nix flakes, where it shines on servers and CI, where it hurts, and an honest recommendation.
A repeatable method for reading a codebase you didn't write: start outside in, trace one real request end to end, lean on the debugger and logs, find the seams, and use git log sparingly. The honest core of it is deciding what not to read.
A practical, honest look at encryption at rest: the real threat model, what full disk encryption with LUKS covers and does not, why your passphrase is the weak point, TPM/clevis and crypttab unlock, and doing LUKS on a VPS, encrypted ZFS and encrypted backups.
High load with an idle CPU is usually a disk problem. How to read iostat -x for average await, %util and queue depth, separate throughput problems from latency problems, catch a p99 stall with iotop and /proc/<pid>/io, and fix fsync-heavy writers, WAL, swap thrash and log spam.
Timezones, UTC, and the bugs between them: why you store UTC and convert at the edges, the difference between an instant, a local date-time, and a duration, why the wall clock is not monotonic, DST gaps and ambiguity, the tz database, cron across DST, and Python/JS traps — with a checklist.
A slow build is rarely the compiler's fault. A checklist for measuring cold vs warm builds, finding the real culprits — reinstalling dependencies, no build cache, giant images, single-threaded steps — and fixing them with caching, incremental builds, and a shorter critical path.
Local, remote and dynamic SSH port forwarding made concrete, plus ProxyJump for bastions, autossh and a systemd unit for persistence, and the honest point where an ssh tunnel stops being enough and you should run WireGuard or a mesh overlay instead.
A practical guide to database indexing: what an index really is and what it costs, why "add an index" is often wrong, composite index column order and the leftmost prefix, covering indexes, when the query planner ignores you, and how to read EXPLAIN ANALYZE.
A no-theory ZFS basics guide: the four concepts that matter (pool, dataset, snapshot, scrub), the commands you actually type, why ZFS snapshots are not backups and how zfs send receive fixes that, the honest ZFS ARC answer for small machines, the ECC myth, and how to recover.
Why copying a live Postgres data directory fails, and how to choose among pg_dump, volume snapshots, and WAL archiving. Concrete commands, restore drills with pg_restore and pg_ctl recovery, the gotchas, and a recommendation for a small self-hosted database.
Why print statements and unstructured logs fail you at 3am, what structured logging with JSON logs really means, what to log at each log level, how to propagate a correlation id, the journald reality including journalctl -o json, sampling, and a quick Python/Go/Node comparison.
Linux load average counts runnable and uninterruptible tasks, not CPU usage. Here is what the 1/5/15-minute numbers really measure, why I/O wait and D-state processes inflate them, why per-core normalisation matters, how container CPU quotas break the old rules, and what to run instead — vmstat, mpstat, PSI and iostat.
A practical btrfs vs ZFS comparison for a homelab: what they share (checksums, snapshots, send/receive, RAID-like redundancy), where they diverge (in-kernel per-file vs integrated volume management and RAIDZ), the memory/ARC and licensing trade-offs, and a recommendation per scenario.
The ssh config file is the best productivity upgrade I ever made on the command line. Then come ssh tunnels: local, remote and dynamic port forwarding, ProxyJump for bastion hosts, ed25519 keys and ssh-agent. Concrete configs and commands, plus hardening that also improves daily UX.
A crisp end-to-end walk of what happens when you type a URL: browser cache, DNS resolution through stub, recursive and authoritative servers, the TCP handshake and slow start, the TLS 1.3 handshake, the reverse-proxy hop, cache headers, rendering, and HTTP/2 vs HTTP/3.
A practical case for SQLite in production: what it really is, its one-writer concurrency model and WAL mode, busy_timeout, online backups with the backup API, migrations, and exactly when SQLite vs Postgres becomes a real decision instead of a reflex.
An honest answer on using Ansible for a single server: what configuration management really buys you (idempotent re-runs, disaster recovery) versus what it costs (YAML sprawl, over-abstraction), a minimal readable playbook, and when a shell script or a Nix/container image is the better answer.
Git Flow and PR-heavy processes exist to coordinate teams. If you are the only committer, they are pure overhead. My git workflow: trunk on main, short-lived branches or direct commits, tags for releases, tight commit hygiene, a few pre-commit hooks, and bisect plus reflog as the safety net.
Reading your first packet capture without the overwhelm: when a capture beats logs, how to capture correctly with tcpdump (-i, BPF filters, -w, snaplen), what the TCP handshake, retransmissions, RST and TLS bytes look like in Wireshark, and a worked 'why is this slow?' example.
Beyond the pricing page: which VPS specs actually matter for a small self-hosting server — RAM before CPU, disk IO, network caps, IPv4/IPv6, snapshots, latency — which marketing numbers lie, the real cheap-vs-premium tradeoff, KVM vs OpenVZ, port 25 and abuse policy, and a decision checklist.
You can run LLM locally on ordinary hardware, but the real limits are memory bandwidth and VRAM, not parameter count. I cover quantization, when to pick llama.cpp, Ollama, MLX or vLLM, a sizing rule of thumb, realistic local use cases, and an honest API comparison.
Classic traceroute is a flattering cartoon: ICMP rate limits, per-flow load balancing, hidden MPLS labels, asymmetric return paths and NAT all bend it. Learn to read paths honestly with tcptraceroute, mtr and fixed-flow probes, tell queuing from distance, and diagnose a real path problem.
Why outbound mail is the hardest thing to self-host — reputation, IP warming, blocklists — while inbound is easy. The must-haves (PTR, SPF, DKIM, DMARC, MTA-STS), a pragmatic Postfix + Dovecot stack, and the blunt decision: receive it yourself, send via a relay.
You do not need Prometheus and Grafana to know your servers are healthy. What actually breaks on a handful of boxes, the minimal toolset of node_exporter (or not), systemd checks and a dead man's switch, plus journald retention — and when a real stack earns its keep.
A packet-level walkthrough of DNS resolution: the stub resolver, the recursive resolver, root and TLD servers, the authoritative answer, caching and TTLs, the record types that matter, CNAME rules, glue and apex ALIAS/ANAME, authoritative vs recursive roles, and debugging with dig +trace, kdig, DoH and DoT.
Why the 1500-byte MTU breaks under VPNs, overlays, and tunnels: fragmentation and the DF bit, MSS clamping with a working nftables rule, path MTU discovery and how filtered ICMP kills it, the classic TLS-hang symptom, and a diagnosis walkthrough with ping -M do, tracepath, and tcpdump.
A blunt 3-2-1 backup strategy for a one-person homelab: why the only metric that matters is a successful restore, what to back up (data, config, secrets), how to choose between rsync, restic and borg, how to automate a run-and-verify loop, and how to run a real test restore before you need one.
A practical IPv6 tutorial for people who avoided it: why the address math is simpler (prefixes, /64, SLAAC, no NAT), what changes operationally (nftables, ip -6, AAAA, listening on [::]), why dual stack is pragmatic, the gotchas, and how to test it on one box.
Build a tiny multi-host network inside one Linux box with ip netns, veth pairs, a bridge and NAT. A hands-on walkthrough for testing routing, overlays and firewall rules without extra hardware, verified with ping, ip route and tcpdump.
A look at systemd timers vs cron: same job as a crontab line vs a .service + .timer pair, journalctl logging, Persistent=true catch-up, OnCalendar syntax, RandomizedDelaySec, resource limits, environment handling, and the systemd-analyze calendar trick — plus when cron is still the right call.