In my career I've never been a full-time security hire, but most of my roles have at least overlapped with cybersecurity. Having been around many hackers of ill repute, it's both comical and inexplicable that I've wound up entirely white hat / blue team.
The latest generation of models is changing the cybersecurity field very quickly, and while there are some hiccups, I think it's likely for the better. But one consequence is that I've only earned one "human-authored" security bounty, and it's likely to be my last as many programs are already dramatically reducing eligibility.
I thought it'd be fun to write up how I found a security vulnerability on GitHub, why we disagreed about severity, and why I don't think it'll happen again.
The problem
In early 2024, I had just started a new role. One of my first projects was figuring out why our CI was giving users a poor experience.
Like a lot of companies, we use GitHub Actions. But unlike a lot of companies, we have Docker images that can take an hour to build and be 10-50 GiB in size.1
We had a mandate to provide better user performance, and to save time and money. The decision was made to bring workloads in-house using the GitHub Actions self-hosted runners capability. I've been pretty honest how I feel about that:
"Now, to be fair, you can bring your own runners to GitHub Actions. Self-hosted runners exist." haha. don't ever do this btw
But that was my assignment, so I got to it.
Building a repro case
You can skip this part if you don't care about what I was working on when I discovered a security vulnerability.
During this period users reported that builds were "slow" or "flaky". We investigated and realized that many failures were correlated with chaotic messages fairly deep in the network stack: failed Docker image pulls, sockets closed, HTTP/2 handshake and frame errors, Go channels closing, that sort of thing.
Many applications incorrectly assume that they will have a stable and reliable network. When that assumption breaks, the errors don't necessarily surface close to the consuming code thanks to intermediary layers like daemons and pools. For example, some test suites used the network2 and failures would surface as "test failures". Connection failures in pip3 would become "missing packages" rather than hard failures. And builds were sometimes "slow" because retries were papering over massive packet loss issues.
We used log aggregation to look for trends in network-related application errors, and came to a few conclusions:
It was a low level network misconfiguration that surfaced closer to the application, often in networking libraries
There was always some background presence, but it got worse during periods of heavy builds
It hit our self-hosted CI the most, but also "everything else"
When dealing with haunted software you generally want to build a minimal repro case that still exhibits the behavior. In our case, the observed behavior was saturation-based; so our repro case required a harness to run hundreds of GitHub Actions workflows doing various Docker pulls and pip installs at the same time (and then a random subset would exhibit the issue).
We started repro work on a dedicated branch to prove out a proof of concept, and later moved to a dedicated repo when it became clear it wasn't a quick fix. Ideally, you'd even want to do this on a dedicated set of runners so that you don't degrade user experience when creating artificially high load.4
All told, we never found a smoking gun issue. Instead we kept tripping over bottlenecks in networks and clusters that had never originally planned to host CI servers. Mostly they were both obscure and tedious:
"Stacked PRs" tooling that would often start a lot of wasteful builds around the same time
Subtle network differences between "Docker" and "Docker-in-Docker" run modes in Kubernetes
Packet loss due to MTU mismatch between Docker networking and GCP networking
DNS misconfiguration within k8s causing various issues when large numbers of domains were looked up (and failed resolution) in parallel
Cloud NAT port saturation due to misconfiguration in total IPs and in VM port allocation
Funneling all outbound traffic (including the office) through a single VPC and Cloud NAT
Lack of network-level retries in various places
We just kept gradually working on these issues, and on our overall network architecture5, until random disconnections and packet loss no longer blocked users from using CI.
But that is a boring outcome, and not why I wrote this post.
Not knowing when to quit pays off
During the repro process I built a Datadog dashboard (using their dedicated CI Pipeline Visibility product) to keep tabs on aggregate failure rates of harness runs, and what types of error messages we saw.
One day, I saw this:
...and I immediately realized I might have stumbled on a much bigger problem.
(Do you know enough about the GitHub API to know why?)
Coincidentally, by this point we'd been working directly with some GitHub employees to talk about future Actions product features. They also helped us determine if any of our issues were related to their self-host stack6. I was able to directly ask whether I'd found a security issue.
They confirmed that I should submit my finding through the bug bounty program. After going through some intra-company approvals (since I found the issue on work time and infrastructure), I signed up.
The report
Here's a redacted version of the report I submitted.
Description
Github Actions customers can employ "self-hosted runners", which run the Github Actions runner software stack (https://github.com/actions/runner/releases) within a customer environment. This feature is part of both Enterprise Cloud and Enterprise Server, and can be used to run both private and public repository builds.
When a self-hosted runner claims and begins executing a job, it performs a variety of remote calls to Github Actions APIs for both initialization and for heartbeating. Under recommended and typical configurations, the runner process will be on the same host as the runner job being executed. This often means they are sharing underlying network resources.
When the runner process and the runner job share the same underlying network resource, network degradation (such as extreme packet loss) can be triggered (accidentally or intentionally as a denial-of-service) as a "noisy neighbor", even when the workload is isolated to a container and does not have direct network access to the runner. Under these conditions, it is possible to cause the runner process to be unable to call back to the Github Actions APIs that control logging, state transitions, and other aspects of the control plane.
This denial-of-service attack becomes a privilege escalation due to an error in the Github API control plane for Github Runners. If a runner has claimed work and posted to the API successfully, but subsequently fails to heartbeat in time, the API will eventually assume the runner has disconnected and mark the in-progress job as "skipped" rather than "error" (exceptions/failures) or "cancelled" (manual cancellations).
An abridged version of the resulting output:
Workflow: {"status":"completed","conclusion":"failure"}
Job "A": {"status":"completed","conclusion":"skipped"}
Job "B": {"status":"completed","conclusion":"success"}This fallback case presents several problems for API integrity. Moving from "running" to "skipped" is not a valid state transition, and it's not valid to have a failed workflow when the jobs in that workflow all either succeeded or skipped. But what makes this a privilege escalation is that the "skipped" status in the Pull Request API has the meaning of "allow this check to pass even if it didn't run", to account for PRs not requiring checks to pass if their jobs weren't required to run: https://emmer.dev/blog/skippable-github-status-checks-aren-t-really-required/
Required checks on repositories are both a form of access control, and also are frequently used to implement other security controls (like requiring code to pass a security suite). This means the impact could be significant and impact customers in unpredictable ways. Additionally, this issue may also affect Github-hosted runners, which run the same software and use the same APIs (though that specific combination has not been tested).
This issue was first observed while [REDACTED].
Steps To Reproduce
- 1.
Create a new pool of Github Actions self-hosted runners, using the most up-to-date version of the Github Actions runner software stack: https://github.com/actions/runner
- 2.
Create a Github Actions workflow that uses the target runner pool, and runs one or more long-lived jobs.
- 3.
Wait for the job to be claimed and to start executing on the runner. The affected component is the heartbeat status update of the runner, so the denial of service has to run in parallel with the runner job.
- 4.
Trigger partial or total loss of network connectivity from the Github Actions self-hosted runner, either from contention from the runner job, or via simulation.
- 5.
Eventually, the self-hosted runner will fail to report back for long enough that the Github API will mark the job as "skipped" rather than "failed" or "canceled". This may require multiple attempts depending on the networking approach (we saw it ~1 in 100 times).
- 6.
Skipped state counts as a success for required checks: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/collaborating-on-repositories-with-code-quality-features/about-status-checks
"Note: A job that is skipped will report its status as "Success". It will not prevent a pull request from merging, even if it is a required check."
Impact
Attackers with access to submit a pull request can bypass any Github required check (the "Require status checks to pass before merging" checkbox) on any PR check that would normally be reported from a Github Actions runner.
As attackers can include a modified/alternate workflow in their PR, and checks operate on job name; this privilege escalation may also function to trigger and then bypass required checks that wouldn't normally run based on the current repo contents (such as allowing a check to pass when it would normally not run due to path filtering).
This does not affect certain other commit-based access controls, like CODEOWNERS.md checks.
This may impact Github-hosted runners as they use the same software stack and APIs.
The state of security
If you haven't worked with the GitHub Actions APIs before, you might be a bit confused: why the fuck does "skipping" a check count as "passing" it?
For readers who haven't dealt with this API before, I asked Claude to create a diagram of GitHub Actions runner states based on public documentation:
(statuses and conclusions are documented on the Github API documentation.)
A common failure mode in GitHub Actions is to add a workflow with a path filter ("only run on changes to this file"), and then mark that workflow as required. This causes the check to remain in a "pending" state indefinitely, blocking the PR from merging. The inability to use these two common features together has been a problem for years and the most common workaround is a total hack where you reimplement path filtering with a third-party action.
One could imagine similar issues anywhere else that a job only runs "conditionally". But that's even more common than path filtering! You'll conditionally not do work whenever you use if: statements (example: only run this job on certain branches) or needs: statements (example: only run this job if a prior job ran).
Unlike path filtering, though, these statements will set a skipped status on jobs that didn't run. And that's why skipped is treated as mergeable by required checks: so that you can't brick your workflows by adding if: statements to required checks.
I think that path filtering being logically similar to if:, but working very differently, is indicative of deeper problems in this API's design. Claude apparently agrees; just asking it to render a state machine diagram resulted in paragraphs of unsolicited feedback about how the diagram didn't represent various other API oddities.
Understanding what went wrong
I enjoy analyzing complex systems as a "black box", where I can see observed behavior but none of the internals or the rationale for why it's implemented this way. This kind of state-oriented API immediately makes me think of finite state automata (or, "state machines"), which are a kind of rules system governing when state transitions can occur.
And this API is kind of like one, if nothing goes wrong, if you don't look at it too hard. But the performance issues injected by our test harness triggered several observable incorrect behaviors:
The "skipped" job conclusion was reached despite the workflow file used for this run having no job dependencies nor `if:` statement that could've skipped it (the two documented ways to be "skipped").
The "skipped" job conclusion was reached after the job was received by a runner, when logically, it should only be permitted before scheduling and execution.
The overall pipeline conclusion was marked as an error despite one "success" and one "skipped" job, which is irregular/incorrect according to the public documentation.
My guess is that the API internals (at the time, at least) represented more of an ad hoc state machine implemented in a non-systematic way, crossing service boundaries, and under some circumstances allowing illegal state transitions.
{"status": "in_progress", "conclusion": ""}
->
{"status": "completed", "conclusion": "skipped"}
I believe it would instead want to only allow a transition to a new "lost connection with runner" state:
{"status": "requested", "conclusion": ""}
->
{"status": "completed", "conclusion": "NEW_DISCONNECT_STATE_HERE"}
It's also possible it could make sense to use the existing "failure" status with no conclusion, though the intent/behavior of that status is poorly documented:
{"status": "requested", "conclusion": ""}
->
{"status": "failure", "conclusion": ""}
Business illogic
If there had been a more formalized description of this API surface, would it have made a difference?
Maybe.
There were definitely some correctness issues that resulted from checks being in a "hybrid" state that was sort-of-failed, sort-of-skipped. During this investigation I even got the self-hosted runner to write various record combinations that caused the UI to fail to load.
But in the real world, these kinds of failures are common: few systems are very rigorous about state transitions! This security issue only occurred because of a second issue in the control plane: a disagreement about what it means for something to be skipped.
In the execution layer, "skipped" was used to mean "remote end became unreachable, intentionally continuing execution without this job"; while in the access control layer "skipped" was used to mean "check will never run, intentionally allowing merge without this job". They are both "skipping" something, but it's not the same thing.
In large scale web SaaS it's very common for features to be broken up into multiple (micro)services, often supported by different teams. On a mechanical level, this can increase the odds of schemas becoming misaligned (running slightly-incompatible versions). But the distance between teams also increases the odds that they will be misinterpreted when writing business logic on top of them.
...and all I got was this t-shirt
First off, I did get paid a decent amount of cash for this one7. But more importantly: I got a shirt.
Here's an interesting question: what severity level would you guess this landed at? Take your guess in the replies.
On the one hand, this was an exploitable issue in the control plane logic that could have allowed merging PRs without approval. Since the broken component was on GitHub's side, there wasn't a customer-patchable component. And the denial-of-service payload can be inserted into the PR and built on CI, potentially including on Github-hosted runners too (since the skipped state was being set by the control plane after not receiving heartbeats).
On the other hand, it was really well hidden, and relies on either chaining exploits to break out of a sandbox or on saturating a network (which is unpredictable and noisy). And most repos using "required checks" are also using other merge gates, like required approvers or CODEOWNERS, that this wouldn't be able to bypass. I think you'd be hard pressed to take over that many repos with just this exploit.
The future of cybering
I'm happy to have had the experience of doing a deep dive on a third-party system and proving a novel way in which it's broken. That doesn't reflect what the day to day of most security work is, but when I get to do it, I find it deeply engaging engineering work.
Yes, it's a bonus that it's both against a recognizable name, and a part of their stack I don't like.
However, I do doubt that I'll ever work on a security issue in quite the same way. Fable-tier models can do everything that I did above, but tens to hundreds of times faster. They may take wrong branches, but they can investigate them so quickly that they're still more effective than I am. Overall, I expect them to find almost all categories of security bugs both faster and cheaper than I can.
Open question, though: would these models miss the initial intuition of why a single "invalid" state is worth poking at deeper?
Haha, no, a Fable without this blog post as context figured it out instantly BTW. Then asking it to rate it as a hypothetical vulnerability downgraded me to Opus, which said "High bordering on Critical" (AV:N/AC:L/PR:N/UI:R/S:C/C:H/I:H/A:N).
Good luck out there!