A green run that shipped nothing: when a skipped job counts as success

A team ships over-the-air updates to a phone app from CI. Every push to the main branch runs a workflow, and when the workflow finishes, the new code reaches phones a few minutes later. For six days the runs kept finishing with a green tick. For six days nothing reached a single phone.

Nothing was broken. The workflow did exactly what it was written to do. The problem was in what people read into the green tick.

What happened

The workflow has three jobs. gate answers one question: is this push allowed to deploy at all? decide compares the push against a baseline git tag. The tag marks the last commit the store binaries are compatible with, the builds people actually have installed. If only app code changed since the tag, decide answers ota. If something that needs a new store build changed, it answers native.

The third job, publish, sends the update. It carries one line that matters:

if: needs.decide.outputs.verdict == 'ota'

The baseline tag dated from five weeks earlier. Some time after it, a native-affecting change landed: an edit to the app config and the package manifest. From then on, decide compared every push against that old tag, found the native change in the range, and answered native. Every push. So publish was skipped on every run.

Why it stayed green

In GitHub Actions, a job whose if: is false is skipped, and a skipped job does not fail the workflow. The run's conclusion is success. The tick on the commit is green. The list of runs looks exactly like a week of good deploys.

That is what people saw: about 40 green runs in six days. Zero updates delivered to phones. People on the team, and an automated assistant that reported deploys, read the green tick as "shipped". Users were told that two fixes were live. They were not.

Nobody caught it from the run list. It was caught only when someone opened one run and read the jobs one by one, and saw that the job that does the actual work had not run.

The one command that shows the truth

The run's conclusion hides the answer. The job conclusions show it. One command lists them:

gh run view <run-id> --json jobs --jq '.jobs[]|[.name,.conclusion]|@tsv'

On any of those 40 runs, it printed:

gate	success
decide	success
publish	skipped

Three lines, and the third one is the whole story. The same field is in the REST API as jobs[].conclusion, so a bot that reports deploys can check it as easily as a person can.

The gate was right

It is tempting to call the skip the bug. It was not. Shipping new app code over the air to a store binary that is older than the native change can crash the app on launch. The gate saw a native change since the baseline and refused to publish. That is its job, and it did it every time.

The bug was that a correct "no" looked exactly like a "yes". The one job that is the whole point of the workflow could stop running, and the only place that said so was a grey icon inside a run nobody opened.

Make the skip loud

Add a last job that always runs and says, in plain words, what happened to the publish:

report:
  needs: [decide, publish]
  if: always()
  runs-on: ubuntu-latest
  steps:
    - name: Say what happened
      run: |
        if [ "${{ needs.publish.result }}" = "skipped" ]; then
          echo "NOT published: verdict ${{ needs.decide.outputs.verdict }}, native change since baseline" >> "$GITHUB_STEP_SUMMARY"
          echo "::warning::NOT published: native change since baseline. A store build is needed."
        else
          echo "Published: ${{ needs.publish.result }}" >> "$GITHUB_STEP_SUMMARY"
        fi

Two details carry the weight. Without if: always(), a job that needs a skipped job is skipped too, so the report would vanish exactly when it matters. And needs.publish.result is the publish job's own result (success, failure, cancelled or skipped), not the run's. The summary line sits at the top of the run page. The ::warning:: line puts a yellow annotation on the run, which is hard to scroll past. If decide also exposes the baseline commit, put it in the line too: "NOT published: native change since baseline <sha>".

The checklist

Anyone with a conditional deploy job can do these tomorrow:

• Read job conclusions, not the run's. Before anyone says "deployed", check that the publish job says success, with the command above or jobs[].conclusion in the API.
• Make the skip visible. A report job with if: always() that writes a plain line to $GITHUB_STEP_SUMMARY and emits a ::warning:: annotation when the publish was skipped.
• Put the verdict where people look. run-name: cannot see job results, so the run title cannot say "not published". Put the verdict in the summary and in the deploy notification instead.
• Alert on staleness, not only on failure. A scheduled check that fails when the last real publish is older than N days while commits kept landing. It catches every quiet stop, not only this one.
• Keep the gate honest. Do not weaken the check to make the runs ship. The skip was right; fix how it is reported.

The lesson in one line: the run's conclusion says "nothing broke", not "the thing happened". A conditional job that is the whole point of the workflow needs its own loud signal.

Common questions

Why does a skipped job leave the run green?

A skip is not an error. The job's condition was false, so it did not run, and nothing failed. A run fails only when one of its jobs fails, so a run whose main job was skipped ends with conclusion success.

Can run-name show the verdict?

No. run-name is set when the run starts, before any job has a result, so it cannot say whether the publish ran. Put the verdict in the step summary, a warning annotation and the deploy notification instead.

Should the publish job just fail instead?

Usually not. A red run every time the gate says no trains people to ignore red. Keep the skip, and add a report job that says plainly that nothing was published and why, plus a staleness alert that fails when nothing has shipped for too long.

Melta is free; join the waitlist.