Four Machines, One Playground: Composing a CI/CD Stack That Boots Ready¶

The first time I stood this up by hand it took a few hundred commands across four servers. It now takes one click and about ninety seconds, and anyone can launch the same thing in a browser. This is the composition: how four machines are wired, how a fixed resource budget was allocated, and the one step that stays manual on purpose.

What Comes Up¶

Four machines on a shared private network at 172.16.0.0/24, each booting from its own pre-built OCI image with no install step:

Every service is running by the time the browser tab is usable. The installing happened when the images were built, which is a separate topic and the reason this composition is possible at all.

The Jump Host Is Not the Workstation¶

Worth stating early, because the naming invites confusion. The node called dev-machine in this stack boots from dev-cicd-rootfs, which is deliberately minimal: no services, no installs, nothing but the base image plus SSH aliases.

FROM ghcr.io/ibtisam-iq/ubuntu-24-04-rootfs:latest

USER $USER
ENV HOME=/home/$USER

COPY welcome $HOME/.welcome
RUN --mount=type=bind,source=scripts,target=/tmp/scripts \
    bash /tmp/scripts/customize-bashrc.sh

EXPOSE 22

That is the entire Dockerfile. It exists to give you somewhere to stand while you drive the other three, and it inherits the base toolset (jq, yq, fzf, btop, code-server and the rest) without adding anything on top.

The full workstation image with roughly 40 tools is a different image and a different playground. Putting it here would spend vCPU and several GiB on a machine whose job is to run ssh.

Its .bashrc carries the aliases that make it useful:

alias stack-jenkins='ssh -o StrictHostKeyChecking=no ibtisam@jenkins-server'
alias stack-sonarqube='ssh -o StrictHostKeyChecking=no ibtisam@sonarqube-server'
alias stack-nexus='ssh -o StrictHostKeyChecking=no ibtisam@nexus-server'

StrictHostKeyChecking=no is intentional here and would be wrong almost anywhere else. Host keys on these machines are generated fresh at every boot, so strict checking would prompt on every new playground.

Allocating a Fixed Budget¶

The playground type provides a hard pool: 10 vCPU, 16 GiB RAM, 150 GiB disk, shared across every machine. Not a target, a ceiling.

The allocation above consumes exactly all three:

vCPU:  1 +  3 +  3 +  3 =  10
RAM:   1 +  4 +  6 +  5 =  16 GiB
Disk: 30 + 40 + 40 + 40 = 150 GiB

Every unit is placed on purpose.

The jump host gets the smallest slice because it runs no services. One vCPU and 1 GiB is enough for SSH and an editor.

SonarQube gets the most memory because it is three memory-consuming processes in one machine: a web server, a compute engine, and an embedded Elasticsearch node, sitting alongside PostgreSQL. Six GiB is not generosity, it is the minimum that avoids an OOM kill during startup.

Nexus gets the full 40 GiB disk because artifact storage is the thing that actually grows. Maven JARs, npm tarballs and Docker layers accumulate in a way that Jenkins job history does not.

Jenkins gets 4 GiB for build concurrency and the plugin set, which is comfortable for two executors.

Sizing the JVMs to match¶

Allocating RAM to a machine is half the job. The processes inside have to be told about it, or they will make their own assumptions and get killed.

SonarQube's sonar.properties is tuned against that 6 GiB:

sonar.web.javaOpts=-Xmx1G -Xms256m -XX:+UseG1GC
sonar.ce.javaOpts=-Xmx2G -Xms512m -XX:+UseG1GC
sonar.search.javaOpts=-Xms1G -Xmx1G

Three heaps totalling 4 GiB maximum, leaving room for PostgreSQL and the OS. The Elasticsearch heap sets -Xms equal to -Xmx deliberately: resizing a search heap at runtime causes GC pressure that shows up as analysis timeouts. I tried 512 MB there first and it was not enough for indexing.

Under-size any of these and the failure is unhelpful. SonarQube shows activating for two or three minutes and then dies, with the real cause buried in an Elasticsearch log rather than in the systemd status.

The Interface Is Part of the Definition¶

The manifest declares the tabs, so the playground opens with everything already wired. This section shows the fields as they are used here; the full schema, including networks, drives, and access control, is covered in Publishing a Custom Playground on iximiuz Labs.

tabs:
  - kind: ide
    name: IDE
    machine: dev-machine
  - kind: terminal
    name: dev
    machine: dev-machine
  - kind: terminal
    name: jenkins
    machine: jenkins-server
  - kind: http-port
    name: "Jenkins UI"
    machine: jenkins-server
    number: 80
    access: public

An IDE, four terminals and three service UIs, without anyone opening a menu.

All three UI tabs point at port 80, not the service port. Port 80 is Nginx. That indirection means the tab keeps working if a service port changes, and the same reverse proxy serves the browser tab and the Cloudflare Tunnel identically. One entry point, two consumers.

Intra-Stack Networking¶

The four machines resolve each other by hostname on the private network:

# from jenkins-server
ping sonarqube-server
ping nexus-server

Which means a Jenkins pipeline reaches SonarQube and Nexus without leaving the playground. Analysis traffic and artifact uploads stay on the private network rather than going out to the internet and back through a tunnel.

Public domains are for humans. Hostnames are for the pipeline.

Verifying It Came Up¶

Each service machine boots lab-init → nginx → service, with SonarQube inserting postgresql before Nginx. The check is the same shape everywhere:

ssh jenkins-server
systemctl is-active lab-init nginx jenkins    # all three: active
curl -f http://localhost/health               # healthy

ssh sonarqube-server
systemctl is-active lab-init postgresql nginx sonarqube
curl -f http://localhost/health

The /health endpoint is served by Nginx, not by the application, which is the point. It answers 200 while SonarQube is still starting, so you can distinguish "the machine is fine and the app is warming up" from "nothing is listening".

SonarQube takes two to three minutes on first boot. systemctl status sonarqube showing activating during that window is expected, not a fault.

The One Manual Step¶

Everything above happens with no intervention. Exposing the services on public domains does not, and that is deliberate.

Each service image ships cloudflared pre-installed, but no tunnel token. Adding one is a per-playground action:

sudo cloudflared service install <token>

The token is a credential. Baking it into an image means everyone who pulls that image gets the ability to register as your tunnel. There is no version of that which is acceptable, so the token stays out and the step stays manual.

The same applies to everything else that is per-instance rather than per-image: Jenkins generates its initial admin password at first boot, SonarQube starts at its default credentials expecting the setup wizard, and Nexus writes its password to /opt/sonatype-work/nexus3/admin.password. None of those belong in a public image either.

The tunnel mechanics are covered separately.

How the Images Get Built¶

Six GitHub Actions workflows, one per image, each triggered by changes under its own path:

on:
  push:
    branches: [main]
    paths:
      - 'iximiuz/rootfs/jenkins/**'
      - '!iximiuz/rootfs/jenkins/README.md'
      - '.github/workflows/build-jenkins-rootfs.yml'

Path filtering with negation for documentation, so a README edit does not trigger a rebuild.

One correction to something I wrote in an earlier version of this post: only the base image builds multi-arch. ubuntu-24-04-rootfs builds for linux/amd64 and linux/arm64 with QEMU. All five child images build linux/amd64 only, with QEMU deliberately omitted. Which means the arm64 base currently has no consumers, and that is a gap rather than a decision.

What the Composition Buys¶

The manifest is the environment. It is a file in a repository, versioned like anything else, and it produces the same four machines every time.

Beyond reproducibility, two things surprised me.

The resource budget forced honest design. With a soft limit I would have over-allocated everything and never thought about why SonarQube needs more memory than Jenkins. A hard ceiling made me justify every GiB, and the JVM tuning came out of that exercise rather than out of a later incident.

Separating image from composition made both simpler. Each image is a machine that works on its own and is published as its own playground. The stack manifest only decides how many run together and how they are sized. Neither layer knows much about the other, so I can change the Jenkins image without touching the stack, and resize the stack without rebuilding anything.

Source¶

• Stack manifest
• All six images
• Post-provisioning runbook covering credentials, the SonarQube webhook and the Nexus Docker registry
• Launch it