The same Helm chart deploys my orders service six different ways: in memory, on PostgreSQL with in-memory messaging, on PostgreSQL and RabbitMQ backed by local disk, the same pair on EBS, both pointed at external endpoints, or PostgreSQL with AWS SQS. Six values files, one chart, no forks.
I got there by being wrong about what portability means.
I set app.messaging.sqs.topic in a values file. Helm accepted it. The chart rendered without error. The deployed pod did not have the environment variable, because the chart contains no reference to SQS anywhere.
I found that by reading the template, not by debugging the application. That is the skill this post is about, because most Helm charts you deploy are charts someone else wrote.
A freshly provisioned EKS cluster cannot provision a load balancer, cannot create a persistent volume on a modern disk type, cannot route Gateway API traffic, and cannot create a DNS record. It runs pods and that is roughly it.
Four add-ons close that gap, and three of them have a default that silently does the wrong thing.
None of these appeared in documentation. Each one cost me somewhere between twenty minutes and an afternoon, and every fix turned out to be a one-line change once I understood the cause. Here they are with the symptom, the root cause and the resolution, in the order I hit them.
Building the image is the easy part. What actually accumulates over months of running DebugBox against real clusters is a list of things that look like bugs the first time and turn out to be the correct behaviour of something underneath: musl, network namespaces, kube-proxy, ps column order. Six of them, with what actually caused each one.
Every other post in this series is about a machine that runs a service: Jenkins, SonarQube, Nexus, the jump host that only forwards SSH. This one is about the machine I actually sit at. dev-machine-rootfs is a full DevOps workstation, more than 40 tools installed at build time, that boots ready in a browser tab instead of accumulating over a year of apt install on whatever laptop I happen to be using.
I write a Dockerfile and what comes out the other end is a bootable server. Not a container image that happens to run systemd, but a root filesystem that a hypervisor mounts as a block device and boots with its own kernel. Running any of these images with docker run fails immediately, and that is the correct behaviour.
Most Kubernetes debugging images ship as a monolithic package. netshoot is 202 MB. That 202 MB gets pulled whether the task requires dig or tshark. On edge clusters, metered connections, or environments with registry pull rate limits, that bandwidth cost accumulates across teams and incidents.
DebugBox takes a different approach: three variants, each scoped to a specific category of debugging work.
DebugBox v1.0.0 shipped with every tool I could justify. The power variant included bird (a BGP routing daemon), bridge-utils, nano, py3-pip, and speedtest-cli. In v1.1.0, all five were removed. Several other tools were moved between variants. The result: balanced dropped from 51 MB to 47 MB, power from 112 MB to 91 MB.
This post covers the decision framework used to evaluate each tool, the specific changes made, and how the project's manifest system prevents tool lists from drifting across documentation.