A team moved all of its CI from hosted runners to two self-hosted runners on a single Linux box. The usual reasons: lower cost, no queue, machines that sit idle anyway. Each runner got its own Unix user and its own work directory, which felt like a clean separation. Two runners, two users, two directories — nothing shared.

Day one brought three failures. None of them were about the runners being slow. All three traced back to one fact that is easy to miss: a hosted runner is thrown away after every job, and a self-hosted runner is not. Everything a hosted runner cleaned up for free — temp files, ports, the workspace — is now your job.

Key facts

  • /tmp is one directory shared by every user on the host. A fixed log path there works for the first runner and fails for the second.
  • A hosted runner is discarded after each job. A self-hosted one keeps whatever the job left running, and those processes keep their ports.
  • The workspace is reused between jobs, not fresh. A checkout can land in a tree that is sparse or dirty.
  • The fixes are small: write per-job files under $RUNNER_TEMP, kill background processes in an if: always() step, keep checkouts clean.
  • One Unix user per runner contains the damage; one runner per machine avoids the sharing entirely.

What happened

Two runners on one host, each registered under its own Unix user with its own work directory. On paper the only things they share are the kernel and the CPU. In practice they share /tmp, the port space on loopback, and every global resource on the box — and the first day's failures came from exactly those.

Trap 1: the log file both runners wanted

The first failure came from a deploy step that wrote its log to a fixed path: /tmp/firebase-deploy.log. Nothing exotic — redirect the deploy tool's output to a file so a later step can print it if the deploy fails.

The first job ran on runner one, whose user created the file. The next job landed on runner two, running as the other user, and died with:

/tmp/firebase-deploy.log: Permission denied (os error 13)

Every later run on the other runner hit the same wall, and the error pointed at the deploy tool, not at the runners, so the first look went to the wrong place.

The cause is simple: /tmp is one directory for all users on the host. Runner one's user created the file and owns it; runner two's user cannot write to it. With a single runner, the same user wins every time and the conflict never appears. Add a second user and the assumption breaks on the first cross-over job.

The fix is to stop sharing. $RUNNER_TEMP is a fresh per-job temp directory the runner provides and wipes with the job; anything a job writes for itself belongs there. If you need a scratch path inline, mktemp gives the same guarantee.

Trap 2: the port nobody freed

The second failure looked like a port conflict with no opponent. A test job that starts the local emulators failed with:

Port 4407 is not open on 127.0.0.1, could not start emulator hub

with 4507 and 8087 failing the same way — and no other test run on the host.

There was an opponent; it was just invisible. An earlier job had started the emulators and never stopped them. On a hosted runner that process would have died with the machine, because the whole box is discarded after the job. On a self-hosted runner the machine persists, so a process left behind keeps running for hours or days, holding its ports, until something kills it.

Three fixes, and they stack. Kill leftovers in a cleanup step that runs even when the job fails — a post: step or a step with if: always(). Give each runner its own fixed port range, so two runners cannot fight even when both leak. And put a process sweep in the runner's service definition, so a leaked process dies when the runner restarts.

Trap 3: the workspace that was not fresh

The third failure looked like a broken checkout. A deploy job failed with:

cd: console/api: No such file or directory

in a step that ran right after checkout. The directory should have been there. It was always there on a hosted runner, because a hosted runner starts every job from an empty disk.

A self-hosted runner reuses the same workspace directory job after job. A reused workspace can end up sparse or dirty: an interrupted run, a step that moved files, a clean setting somebody disabled. The checkout tries to be fast against what is already there, and the tree is not what the job assumed.

The fix: keep clean: true on checkout — it is the default, but verify nobody turned it off — or wipe $GITHUB_WORKSPACE yourself at job start if you have reason to distrust the tree.

The fix

The traps share one shape: the job assumed a private, disposable machine and got a shared, permanent one. The steps below close all three gaps:

- name: Deploy
  run: |
    ./deploy.sh console/api > "$RUNNER_TEMP/deploy.log" 2>&1 || {
      cat "$RUNNER_TEMP/deploy.log"; exit 1; }

- name: Clean up background processes
  if: always()
  run: pkill -u "$(id -un)" -f emulator || true

Two details carry the weight. The log goes under $RUNNER_TEMP, so each job writes to its own directory and two runners never fight over a file. The cleanup step runs with if: always(), so the emulators die even when the test step fails — which is exactly when a job is most likely to leave them behind.

The checklist

Before you point real traffic at a shared self-hosted box:

  • Per-job temp files. Anything a job writes goes under $RUNNER_TEMP, or mktemp if you need a path inline. Never a fixed path in /tmp.
  • A cleanup step for background processes. if: always(), scoped to the job's own user, killing what the job started.
  • Clean checkouts. clean: true is the default; verify it stayed on, or wipe the workspace at job start.
  • One Unix user per runner. It does not remove the sharing, but it contains the blast radius and turns silent corruption into a loud permission error.
  • A disk watch. A long-lived machine fills. Hosted runners were erased after every job; yours will not be.

The lesson in one line: a self-hosted runner is a long-lived machine, and anything a hosted runner cleaned for free — temp files, ports, the workspace — is now your job.

Common questions

Why does /tmp break with two runners but not one?

/tmp is one directory for every user on the host. The first runner's user created the log file and owns it; the second runner's user cannot write to it. With a single runner the same user wins every time, so the conflict never appears.

Why was the port still taken with no test running?

A previous job's emulator process never exited. The machine persists between jobs, unlike a hosted runner, so anything left running keeps its ports until something kills it.

Doesn't checkout clean the workspace?

clean: true is the default on checkout, but a reused workspace can still end up sparse or dirty: an interrupted run, a disabled clean setting, a step that moved files. Verify the setting, or wipe the workspace at job start.

Melta is free; join the waitlist.