Skip to content

Moving the team's platform

13.7 → 4.1minutes median pull request check, old server to new

I moved the team's code, checks and releases off an on-⁠premises server and onto GitHub. Then I built the path from a pull request to a release on an office server I set up. Pull request checks went from a median of 13.7 minutes to 4.1.

  • Isherwoods
  • Sep to Oct 2026
  • In use, private

13.7 → 4.1minutes median pull request check, old server to new

How a change travels from code to the office serverA tall stack, read top to bottom in the order a change travels. At the top left, the old server: one on-premises machine running Azure DevOps, drawn as a single block. A short arrow carries the move across into the top plate, code and pull requests on GitHub: the history was cleaned first, then all 67 open pull requests moved with their review threads (step 1). A line runs down to the second plate, the checks: a grid of projects with the three a change affects raised, because only what changed is built and tested (step 2). The median pull request check went from 13.7 minutes to 4.1. The line runs on to the third plate, the images: each merge builds one, kept by commit, and that exact image is shipped by its digest (step 3). It lands on the widest plate at the foot, the office server, built from a bare VM (step 4). On it stand three stacks: the infrastructure stack with the database, backed up nightly, and dev and stable, both from one stack file, and the runner that deploys. On the stable stack, the old image stands in front of the live one with an arrow from it back into the live one's place: a release that fails its health check is rolled back by itself (step 5). In the rehearsal, a release that never became healthy was rolled back in about 2 minutes. In the left column, two dim twins of the office server: the developer machine where the whole server was rehearsed first (step 6), and the cloud sandbox, built from the same stack file.Code and PRsOn GitHubChecksOnly what changedImagesBuilt once, shipped by digestOffice serverBuilt from a bare VMHealth checkRolls a bad release backDatabaseBacked up nightlyRunnerDeploys onlyDev and stableOne stack fileOld serverAzure DevOpsDeveloper machineRehearsed here firstCloud sandboxSame stack fileHistory cleaned first12345667 open PRs67 open PRs~2 min~2 min
A change's way down: code, checks, one image, then the server I built on a bare VM. Each number is a decision below. Press one to read it.Scroll sideways to see all of it
How a change travels from code to the office serverA tall stack, read top to bottom in the order a change travels. At the top left, the old server: one on-premises machine running Azure DevOps, drawn as a single block. A short arrow carries the move across into the top plate, code and pull requests on GitHub: the history was cleaned first, then all 67 open pull requests moved with their review threads (step 1). A line runs down to the second plate, the checks: a grid of projects with the three a change affects raised, because only what changed is built and tested (step 2). The median pull request check went from 13.7 minutes to 4.1. The line runs on to the third plate, the images: each merge builds one, kept by commit, and that exact image is shipped by its digest (step 3). It lands on the widest plate at the foot, the office server, built from a bare VM (step 4). On it stand three stacks: the infrastructure stack with the database, backed up nightly, and dev and stable, both from one stack file, and the runner that deploys. On the stable stack, the old image stands in front of the live one with an arrow from it back into the live one's place: a release that fails its health check is rolled back by itself (step 5). In the rehearsal, a release that never became healthy was rolled back in about 2 minutes. In the left column, two dim twins of the office server: the developer machine where the whole server was rehearsed first (step 6), and the cloud sandbox, built from the same stack file.Code and PRsOn GitHubChecksOnly what changedImagesBuilt once, shipped by digestOffice serverBuilt from a bare VMHealth checkRolls a bad release backDatabaseBacked up nightlyRunnerDeploys onlyDev and stableOne stack fileOld serverAzure DevOpsDeveloper machineRehearsed here firstCloud sandboxSame stack fileHistory cleaned first12345667 open PRs67 open PRs~2 min~2 min
A change's way down: code, checks, one image, then the server I built on a bare VM. Numbers match the decisions below.Scroll sideways to see all of it

Diagram. A change's way down: code, checks, one image, then the server I built on a bare VM. Numbers match the decisions below.

A change's way down: code, checks, one image, then the server I built on a bare VM. Numbers match the decisions below.

This is employer work, so it's described rather than shown. Names are generic, the diagram is redrawn and simplified, and there's no code or product UI. I can say more in conversation.

01Problem

Our source control, pull requests, checks and releases all ran on Azure DevOps, on one on-⁠premises Windows server. The pipelines that tested and shipped our code were set up in its web interface, outside version control, so a change to how we released couldn't be reviewed like code. The server had one runner, so only one check ran at a time and pull requests waited in a queue. Every check was a full run of everything, with a median of 13.7 minutes over 36 runs in August and September.

The team's main reason to move was AI. GitHub works much better with the AI tools we wanted, so we could hand work to agents. Faster checks came with the move.

Releases now had to go to a bare VM that IT handed over on 2 October, with nothing of ours on it. Leaving wasn't a simple push either. The history had to be cleaned first, which rewrote the commits, and 67 pull requests were open on the day of the move. And our GitHub plan has a fixed pool of minutes, so faster checks had to mean doing less work, not buying bigger machines.

02Decisions

  1. Clean the history, then move the pull requests with a tool

    I cleaned the history first, and that rewrote every commit. Reopening 67 pull requests by hand would then have been slow, tedious work, and it would have lost their review threads. So I built a migration tool with 440 tests to carry them onto the cleaned history. It dry-⁠runs by default, and a rerun posts nothing new. Each colleague ran it for their own pull requests, so every comment kept its real author. It moved 208 threads, and 190 went back on their lines, where the code hadn't changed. The rest went into the review text.

  2. Run only what changed

    Slow checks were already holding the team up, so I'd made the frontend tests 40% faster on the old server, merged in July. The next saving had to come from running less. nx affected works out which projects a change touches, and a pull request builds and tests only those. Each job restores three caches: the package store, the prebuilt shared libraries and Nx's task cache. Nine jobs run in parallel behind one gate check. I ruled out larger runners, which are billed all the time, and one big job, which took about 30 minutes.

    Step 2 in the drawing: checks run only on what a change touched. Redrawn, with generic names.Step 2 in the drawing: checks run only on what a change touched, and the median check went from 13.7 minutes to 4.1. Redrawn, with generic names.

  3. Build each release once, and ship that exact image

    Every merge into dev or main builds the release as container images and pushes them to GitHub's container registry. Each image gets a tag for its commit that never moves. On main, a check reuses the image already tested on dev when the code is the same, so main ships the exact image that was tested. The deploy names each image by its digest, never by a tag that could point somewhere else tomorrow. A monthly job clears out old images.

  4. Build the office server from a bare VM

    IT handed over the VM on 2 October. I built everything we run on it: Docker Engine, then a single-⁠node Docker Swarm running three stacks. One holds the database, with a backup every night kept for seven days. The stable and dev apps come from the same stack file with different variables, kept apart on their own networks. The GitHub runner lives on the server as a non-⁠root service, and only the deploy workflow can use it. The first pipeline deploys ran on 6 October.

  5. Let the health check decide

    Every service has a health check. If an update's new containers don't pass it, Swarm rolls the update back by itself. The deploy job doesn't take that on trust. It waits for the update to finish, then checks the stack is running what it asked for. In the rehearsal, a release that crashed on start-⁠up was caught in 17 seconds, and the old version was serving again in about 30. One that never became healthy was rolled back in about 2 minutes.

  6. Rehearse the whole server on a developer machine first

    On 7 September, weeks before the VM arrived, I rehearsed the whole server on a developer machine. It found eight defects: five on the server side, two in the app and one in the test design. My rollback test couldn't fail, so I replaced it with a real one that deploys a broken image. A restore brought back 48,295 records in 8 seconds. It all went into a runbook of 2,366 lines, with 13 sections and 22 troubleshooting entries.

03Result

13.7 → 4.1median pull request check, in minutes

  • The old server
  • GitHub Actions
  • 67open PRs moved with their review threads
  • ~2minutes back up after a bad release

The main reason for the move is done. Our code and pull requests are on GitHub now, where the team can work with AI tools and agents. Everything else here came with it.

From the first pipeline deploy on 6 October to 23 October, it deployed 40 times: 36 to dev and 4 releases. The median deploy took about 4 minutes, most of it waiting on the health check. Six attempts failed, all on the runner's side, not the app's, and a re-⁠run fixed each one. Nothing has needed a rollback yet.

The workflows and the server's files now live in the repo, so a change to how we release gets reviewed like code. A recent cloud sandbox reuses the same stack and host files, plus one overlay that adds an HTTPS proxy at the edge and keeps it to a single app stack. It runs the same deploy steps once a reviewer approves, and its notes record only what's different.

Pull requests no longer wait in a queue, because each one's jobs run on their own machines. Small changes now finish in 1 to 2 minutes, and 87% of runs did less than a full run. A cold full run takes 6 to 9 minutes. That's quicker on the clock, but it uses 34 to 40 runner-⁠minutes across up to nine runners, against about 14 on the old bare-metal server.

04How I worked

We were three developers. IT built the VM and backs it up. One of my two senior colleagues brought in Nx and pnpm in June 2026, which the CI work builds on. Both reviewed my pull requests, and our tech lead reviewed the server and release ones. I wrote 42 of the 45 commits on the pipeline and deploy, and a bot wrote the other 3. I also wrote the stack and host files, the workflows, the rehearsal and the runbook.

I documented everything as I went, in runbooks and notes, so someone else could understand the setup and do it themselves. It's all Markdown files for now.

AI agents wrote much of the code and helped with every commit in the release pipeline. I set the direction and made the calls on what to move first, what to rule out and how a release proves it's healthy. I reviewed and tested every change. The rehearsal in step 6 was the real check.

  • GitHub Actions
  • Docker Swarm
  • Ubuntu
  • Nx
  • pnpm
  • TypeScript

05What I'd do differently

I set up the office server first, from a runbook and scripts in the repo. Then a cloud sandbox came up that we hadn't planned for, and I had to repeat the whole setup by hand, a good few hours of steps. Next time I'd write the setup as code from the start, with a tool like Ansible, so building another machine is one command.