Skip to content

2026

My Builds Were Reproducible, So My Deployments Stopped

Ten microservices on EKS, images landing in the registry on every push, and a controller whose entire job is to notice new images and roll the pods. It noticed nothing. No errors, no warnings, just a log line reporting that it had considered ten images and updated zero of them.

The cause turned out to be a feature working correctly in one tool making another tool's core assumption false.

Building Only What Changed, and the Permissions Ceiling Nobody Warns You About

Ten services in one repository. Most commits touch one of them. Rebuilding all ten on every push is the default and it is wrong, so the pipeline detects which service directories changed and fans out one job per service.

Two things about that turned out to be harder than the idea: computing "what changed" correctly across every way a workflow can be triggered, and discovering that a reusable workflow cannot ask for permissions its caller did not grant.

The Jenkinsfile Rewrite Was Mostly Deletions

I rewrote a working Jenkins pipeline. The version that replaced it is longer, and the changes that mattered most were the two blocks I removed.

Both of them did the same thing: they asked the Jenkins server to supply something the pipeline needed. Taking them out is what made the file portable, and it is the single change I would make first to any Jenkinsfile I inherited.

Zero Compute, and Still a Dozen Decisions

Seven days of globally distributed static hosting on AWS cost me $1.45 across CloudFront, three S3 buckets, CloudTrail and KMS. There is no compute in it at all. No Lambda, no container, no instance.

"Static site on S3" gets treated as the trivial deployment, and the hosting genuinely is trivial. Everything around it is not, and the decisions turn out to be about who can read the bucket, which key encrypted the object, and what happens when a path does not exist.

401 and 403 Are Different Failures, and One of Them Is Unrecoverable

kubectl returned 403 on a cluster I had just created. Later, on the same cluster, it returned 401. Those two numbers look like degrees of the same problem and they are not: one meant my identity was known and unauthorised, the other meant it was not known at all, and the second one could not be fixed without destroying the cluster.

The reason both were possible is a setting that can only be chosen once.

A Pod With AWS Permissions and No Credentials

Two services in this cluster talk to AWS. Neither has an access key, a Secret, or anything in a values file that could leak one. They assume an IAM role through their service account, and the SDK inside them picks it up without being told.

That mechanism is IRSA, and the part worth writing down is not how to enable it. It is where the boundary sits: which object owns the binding, and why keeping it out of the Helm chart is what lets the same chart still run on a laptop.

A Site That Loads Proves Almost Nothing

The home page rendering proves CloudFront serves content. It does not prove the bucket is private, that replication runs, that audit logs are recorded, that encryption is uniform, or that the cache is working. All five can be wrong on a site that looks perfect, and four of the five fail without producing an error.

This is the verification pass for that architecture, starting with the status code that means four different things.

One Chart, a Ladder of Dependencies

The same Helm chart deploys my orders service six different ways: in memory, on PostgreSQL with in-memory messaging, on PostgreSQL and RabbitMQ backed by local disk, the same pair on EBS, both pointed at external endpoints, or PostgreSQL with AWS SQS. Six values files, one chart, no forks.

I got there by being wrong about what portability means.