Practical engineering write-ups on infrastructure, containers, and distributed systems. Every post comes from a real setup: no theory without execution.https://blog.ibtisam-iq.com/ Muhammad Ibtisamen Thu, 01 Oct 2026 00:18:01 -0000 Thu, 01 Oct 2026 00:18:01 -0000 1440 MkDocs RSS plugin - v1.19.0 https://blog.ibtisam-iq.com/images/favicon.png https://blog.ibtisam-iq.com/ Muhammad Ibtisam Career Career & Growth DevOps Documentation <h1>I Gave Hyperagent One Handle and Nothing Else: It Verified the Rest</h1> <p>Not my name, not a CV, not a list of my sites. Six words containing a GitHub handle, typed into an empty account. It came back with seven live subdomains, more than thirty repositories, two certification badges, a Docker Hub namespace and a tutorial on somebody else's platform, every claim graded against an independent source. Twice, a day apart, for $21.02.</p> https://blog.ibtisam-iq.com/ai-agent-audit-devops-portfolio/ Tue, 04 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/ai-agent-audit-devops-portfolio/ Muhammad Ibtisam ArgoCD BuildKit CI/CD & Automation Debugging Docker GitOps Kubernetes <h1>My Builds Were Reproducible, So My Deployments Stopped</h1> <p>Ten microservices on EKS, images landing in the registry on every push, and a controller whose entire job is to notice new images and roll the pods. It noticed nothing. No errors, no warnings, just a log line reporting that it had considered ten images and updated zero of them.</p> <p>The cause turned out to be a feature working correctly in one tool making another tool's core assumption false.</p> https://blog.ibtisam-iq.com/buildkit-epoch-broke-image-updater/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/buildkit-epoch-broke-image-updater/ Muhammad Ibtisam Architecture ArgoCD CI/CD CI/CD & Automation GitHub Actions GitOps Kubernetes <h1>Push or Pull: Two Ways to Hand Off a Deployment</h1> <p>I built the same handoff twice, in opposite directions. In three monolith pipelines, CI ends by committing an image tag into the deployment repository. For ten microservices, CI stops at the registry and a controller notices. Both are GitOps. Only one of them survived ten services.</p> https://blog.ibtisam-iq.com/gitops-push-vs-pull-handoff/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/gitops-push-vs-pull-handoff/ Muhammad Ibtisam Automation CI/CD CI/CD & Automation Debugging GitHub Actions Monorepo <h1>Building Only What Changed, and the Permissions Ceiling Nobody Warns You About</h1> <p>Ten services in one repository. Most commits touch one of them. Rebuilding all ten on every push is the default and it is wrong, so the pipeline detects which service directories changed and fans out one job per service.</p> <p>Two things about that turned out to be harder than the idea: computing "what changed" correctly across every way a workflow can be triggered, and discovering that a reusable workflow cannot ask for permissions its caller did not grant.</p> https://blog.ibtisam-iq.com/monorepo-matrix-dispatch-permissions/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/monorepo-matrix-dispatch-permissions/ Muhammad Ibtisam Architecture CI/CD CI/CD & Automation Jenkins Maven SonarQube <h1>The Jenkinsfile Rewrite Was Mostly Deletions</h1> <p>I rewrote a working Jenkins pipeline. The version that replaced it is longer, and the changes that mattered most were the two blocks I removed.</p> <p>Both of them did the same thing: they asked the Jenkins server to supply something the pipeline needed. Taking them out is what made the file portable, and it is the single change I would make first to any Jenkinsfile I inherited.</p> https://blog.ibtisam-iq.com/why-i-rewrote-my-jenkinsfile/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/why-i-rewrote-my-jenkinsfile/ Muhammad Ibtisam AWS AWS KMS Amazon S3 Cloud Platforms CloudFront Cost Optimisation Security <h1>Zero Compute, and Still a Dozen Decisions</h1> <p>Seven days of globally distributed static hosting on AWS cost me $1.45 across CloudFront, three S3 buckets, CloudTrail and KMS. There is no compute in it at all. No Lambda, no container, no instance.</p> <p>"Static site on S3" gets treated as the trivial deployment, and the hosting genuinely is trivial. Everything around it is not, and the decisions turn out to be about who can read the bucket, which key encrypted the object, and what happens when a path does not exist.</p> https://blog.ibtisam-iq.com/cloudfront-oac-dual-kms-static-hosting/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/cloudfront-oac-dual-kms-static-hosting/ Muhammad Ibtisam AWS Amazon EKS Cloud Platforms Debugging IAM Kubernetes Security <h1>401 and 403 Are Different Failures, and One of Them Is Unrecoverable</h1> <p><code>kubectl</code> returned 403 on a cluster I had just created. Later, on the same cluster, it returned 401. Those two numbers look like degrees of the same problem and they are not: one meant my identity was known and unauthorised, the other meant it was not known at all, and the second one could not be fixed without destroying the cluster.</p> <p>The reason both were possible is a setting that can only be chosen once.</p> https://blog.ibtisam-iq.com/eks-authentication-modes/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/eks-authentication-modes/ Muhammad Ibtisam AWS Amazon EKS Cloud Platforms IAM Kubernetes Security Serverless <h1>A Pod With AWS Permissions and No Credentials</h1> <p>Two services in this cluster talk to AWS. Neither has an access key, a Secret, or anything in a values file that could leak one. They assume an IAM role through their service account, and the SDK inside them picks it up without being told.</p> <p>That mechanism is IRSA, and the part worth writing down is not how to enable it. It is where the boundary sits: which object owns the binding, and why keeping it out of the Helm chart is what lets the same chart still run on a laptop.</p> https://blog.ibtisam-iq.com/irsa-serverless-event-pipeline/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/irsa-serverless-event-pipeline/ Muhammad Ibtisam AWS AWS KMS Amazon S3 Cloud Platforms CloudFront Debugging Verification <h1>A Site That Loads Proves Almost Nothing</h1> <p>The home page rendering proves CloudFront serves content. It does not prove the bucket is private, that replication runs, that audit logs are recorded, that encryption is uniform, or that the cache is working. All five can be wrong on a site that looks perfect, and four of the five fail without producing an error.</p> <p>This is the verification pass for that architecture, starting with the status code that means four different things.</p> https://blog.ibtisam-iq.com/static-hosting-verification-and-teardown/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/static-hosting-verification-and-teardown/ Muhammad Ibtisam Amazon EKS Architecture Containers & Orchestration Helm Helmfile Kubernetes Microservices <h1>One Chart, a Ladder of Dependencies</h1> <p>The same Helm chart deploys my orders service six different ways: in memory, on PostgreSQL with in-memory messaging, on PostgreSQL and RabbitMQ backed by local disk, the same pair on EBS, both pointed at external endpoints, or PostgreSQL with AWS SQS. Six values files, one chart, no forks.</p> <p>I got there by being wrong about what portability means.</p> https://blog.ibtisam-iq.com/helmfile-dependency-ladder/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/helmfile-dependency-ladder/ Muhammad Ibtisam Architecture Containers & Orchestration Debugging Helm Kubernetes Microservices <h1>Reading a Helm Chart You Did Not Write</h1> <p>I set <code>app.messaging.sqs.topic</code> in a values file. Helm accepted it. The chart rendered without error. The deployed pod did not have the environment variable, because the chart contains no reference to SQS anywhere.</p> <p>I found that by reading the template, not by debugging the application. That is the skill this post is about, because most Helm charts you deploy are charts someone else wrote.</p> https://blog.ibtisam-iq.com/reading-a-helm-chart-you-did-not-write/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/reading-a-helm-chart-you-did-not-write/ Muhammad Ibtisam AWS Amazon EKS Containers & Orchestration Gateway API Kubernetes Networking Storage <h1>Four Add-ons Between a Bare EKS Cluster and a Working One</h1> <p>A freshly provisioned EKS cluster cannot provision a load balancer, cannot create a persistent volume on a modern disk type, cannot route Gateway API traffic, and cannot create a DNS record. It runs pods and that is roughly it.</p> <p>Four add-ons close that gap, and three of them have a default that silently does the wrong thing.</p> https://blog.ibtisam-iq.com/eks-cluster-addons/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/eks-cluster-addons/ Muhammad Ibtisam AWS Amazon EKS Debugging IAM Infrastructure & IaC Infrastructure as Code Terraform <h1>The EKS Module Accepted the Setting and Never Sent It</h1> <p>Terraform applied cleanly. The cluster reached ACTIVE. Then <code>kubectl</code> returned 401, and it turned out that nobody, including the account that created it, had permission to talk to the thing that had just been created successfully.</p> <p>The setting that grants that permission was in my configuration. The module took it, validated it, and did not pass it to AWS.</p> https://blog.ibtisam-iq.com/terraform-eks-module-silent-drop/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/terraform-eks-module-silent-drop/ Muhammad Ibtisam AWS Amazon EKS Debugging IAM Infrastructure & IaC Infrastructure as Code Terraform <h1>Seventeen Errors, and the Ones That Were Caused by the Fix</h1> <p>Provisioning EKS in a restricted AWS account produced seventeen distinct failures. What made it hard was not the count. It was that several of them only existed because of the fix for the previous one.</p> <p>Three of the chains are worth walking through, because the pattern they share is the actual lesson: in a constrained environment, the correct move is often to back out rather than push forward.</p> https://blog.ibtisam-iq.com/terraform-error-cascades/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/terraform-error-cascades/ Muhammad Ibtisam Configuration Databases Debugging Linux & System Administration Spring Boot Twelve-Factor <h1>The Ampersand That Silently Truncated My Database URL</h1> <p>The connection string was correct. I checked it several times. The application still could not reach MySQL, and the error said nothing useful about why.</p> <p>The URL was being cut in half before Java ever saw it, by the shell, at a character I had not thought of as a character.</p> https://blog.ibtisam-iq.com/two-local-databases-one-artifact/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/two-local-databases-one-artifact/ Muhammad Ibtisam AWS Debugging Load Balancing Networking & Storage Spring Boot TLS Troubleshooting <h1>Two Failures Behind One Load Balancer, and Why Both Components Were Right</h1> <p>The Auto Scaling Group kept replacing instances. The application kept starting cleanly. Neither of those statements was wrong, and the gap between them took most of an afternoon to close. Then the moment the health check passed, a second failure appeared underneath it that had been invisible until the first one was gone.</p> https://blog.ibtisam-iq.com/alb-health-check-https-redirect-loop/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/alb-health-check-https-redirect-loop/ Muhammad Ibtisam Gateway API Kubernetes Lets Encrypt Networking & Storage TLS Troubleshooting cert-manager <h1>An ACME Challenge That Could Not Reach Itself</h1> <p>Let's Encrypt could not reach my cluster to validate a domain. That part I expected to fix. What took the rest of the day was that the cluster could not reach itself either, and cert-manager will not ask Let's Encrypt for anything until it can.</p> https://blog.ibtisam-iq.com/cert-manager-gateway-http01-baremetal/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/cert-manager-gateway-http01-baremetal/ Muhammad Ibtisam Cloudflare Kubernetes Lets Encrypt NAT Networking & Storage TLS cert-manager <h1>TLS Behind NAT Is Two Problems, Not One</h1> <p>A Kubernetes cluster on a private address cannot use the HTTP-01 challenge. Not because something is misconfigured, but because the challenge requires Let's Encrypt to open a connection to you, and nothing routable exists to connect to.</p> <p>The mistake I made next was assuming that solving the certificate solved the problem. It does not. Getting a trusted certificate and being reachable from the internet are separate problems with separate solutions, and one of the three approaches below fixes only the first.</p> https://blog.ibtisam-iq.com/tls-three-approaches-behind-nat/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/tls-three-approaches-behind-nat/ Muhammad Ibtisam Gateway API Ingress Kubernetes Networking & Storage TLS Troubleshooting cert-manager <h1>Two cert-manager Solvers, and the Flag That Is Off By Default</h1> <p>cert-manager has two HTTP-01 solvers. Which one you need is not a preference, it is determined entirely by what is routing traffic in your cluster. Pick the wrong one and the challenge never gets served.</p> <p>Pick the right one and it still may not work, because on a manifest install the Gateway API solver is compiled in and switched off.</p> https://blog.ibtisam-iq.com/cert-manager-ingress-vs-gateway-solver/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/cert-manager-ingress-vs-gateway-solver/ Muhammad Ibtisam Amazon EKS Architecture Gateway API Ingress Kubernetes Networking Networking & Storage <h1>Gateway API Three Ways, and What It Actually Buys Over Ingress</h1> <p>I ran the same routing problem three times: NGINX Gateway Fabric on a bare metal cluster, the AWS Load Balancer Controller on EKS, and an ALB Ingress Group sharing one load balancer across three services. Two used Gateway API and one used Ingress, and the differences between them were not the ones I expected.</p> https://blog.ibtisam-iq.com/gateway-api-three-ways/ Sun, 02 Aug 2026 00:00:00 +0000 Muhammad Ibtisam | Engineering Bloghttps://blog.ibtisam-iq.com/gateway-api-three-ways/