How often do AI agent pull requests break each other?

This study measures two things. First, it measures how often Git itself stops with a conflict when it merges two overlapping AI agent pull requests. This repeats a measurement from Xu et al. on new data. Second, it measures what gitta catches that Git misses, on the same pairs. We replayed 733 pairs of overlapping AI agent pull requests from public GitHub projects. Git reported a textual conflict in 24.4% of them. We could run the tests end to end on 44 clean merges. In one of them, each side built alone and the merge failed. Git let that merge through without a word. gitta flagged the pair before the merge, on all three clouds.

Niklas Femerstrand, founder · 21 Oct 2026 · Updated 22 Oct 2026

The question

Coding agents now open pull requests on their own, and they rarely open just one. Xu, Subramanian and Karthik studied the AIDev-pop dataset of 33,596 agent pull requests in 2,807 GitHub repositories. In 40.2% of the repositories, two agent pull requests were open at the same time. These co-active pairs cover 79.4% of all the agent pull requests in the set. Most pairs come from one agent. In a replay of 747 pairs, they measured a textual conflict rate of 41.7% when two different agents wrote the pull requests and 19.8% when one agent wrote both [1].

A textual conflict is the visible kind of collision. It happens when both pull requests change the same lines, so Git cannot combine them by itself. Git stops and asks a person. The harder case is a pair that Git merges without complaint and that no longer works after the merge. Each pull request passed its own checks, and a reviewer read each diff alone. The break appears only on the main branch. We wanted to know how often that happens with real agent work, and whether gitta's pull request check notices it.

Method

Sampling the pairs

We used the AIDev dataset from Hugging Face at tag v3, whose pull request table holds exactly the 33,596 pull requests and 2,807 repositories of AIDev-pop [2]. We counted two pull requests as co-active when their open intervals overlap. Pull requests that were still open count as closed at the dataset cutoff. This reproduces the paper's prevalence figures, 580,914 co-active pairs against the paper's 580,913.

Following the paper, we took one pair per repository. That gave all 122 repositories with a cross-agent pair, plus 625 repositories drawn at random from the 1,007 that only have same-agent pairs. We shuffled the 747 pairs with a fixed seed (20261020), wrote them once to pairs.jsonl and uploaded that file unchanged to every machine. The paper does not say how it chose its 625 same-agent repositories. Our draw is therefore a different random sample from the same population.

What a replay is

For each pair, we fetched both pull request heads from GitHub into an empty repository. We found their merge base and merged them with git merge-tree --write-tree. That is a three-way merge of the two pull requests on their common base, the same merge Git would do when the second one lands after the first. A non-zero exit means a textual conflict.

A clean merge is one where Git combined both sides with no conflict. When the merge was clean, we built a merge commit from the merged tree. We then ran the project's own tests three times, on pull request A, on pull request B and on the merge. Each run used a pinned Docker image (Python 3.12, Node 22 or Go 1.23) with 2 CPUs and 5 GB of memory. Dependencies installed with the network on. The tests then ran with the network off. The runner looked for a test command at the repository root. For Node it ran the test script, plus tsc --noEmit when the project has a TypeScript config. For Python it ran pytest, and for Go it ran go build ./... && go test ./....

A pair counts as a break when both sides pass and the merge fails twice in a row. A merge that fails once and passes on the second run counts as flaky and working.

What was left out

Most pairs dropped out before the test stage. Nearly always the reason was that we could not run the project's tests. The merge was not the reason. If your own agents work in a repository whose tests run offline in a container, your pairs are the kind this study could judge.

The last group is the most important. If a side fails alone, a failing merge tells us nothing about the combination, so we cannot judge the pair. Most of these suites need a database, a browser, secrets or the network. One example is Karma, which needs Chrome system libraries.

Of 747 pairs, 554 merged with no complaint from Git. We could test only 44 of those, and one of the 44 broke.

Machines and clouds

The whole run went five times on three clouds. Each run used a fresh virtual machine, which Terraform created and destroyed afterwards.

Each run took between 10.1 and 19.2 hours. Git gave the same result on every pair in all five runs, and the number of tested pairs differed by at most three.

Every pair had the same base, head and merge SHAs in every run. The textual results were identical, with 179 conflicts out of 733 everywhere. The single break and gitta's result on it were also identical. The number of pairs that reached the end of the test stage was 43 and 42 on Google Cloud, 44 in both AWS runs and 45 on Azure. The difference comes from a few timing-dependent tests that passed on one machine and failed on another. The numbers below come from the AWS 8 vCPU run unless a table says otherwise. Each run cost between $3.35 and $4.72 in on-demand compute.

Results

How often Git stops with a conflict

One pair in four stopped with a conflict, so Git could not combine the two changes on its own. When two different agents made the changes, almost half of the pairs clashed. When the same agent made both, about one in five did.

The paper's rates replicate closely. We repeated their measurement on new data and got nearly the same numbers. Pairs from two different agents conflict about twice as often as pairs from one agent, and the paper's numbers are inside our intervals. Intervals here and below are 95% Wilson intervals. A 95% interval is the range where the true rate very likely is, given how many pairs we counted. Fewer pairs give a wider range.

Our bars and the paper's bars are almost the same length. If you run agents from more than one vendor on one repository, expect about twice as many conflicts as with one agent.

A clean merge that broke

Git merged 554 of the 733 pairs cleanly. In 44 of those, the tests ran end to end, on both sides and on the merge. 43 merges passed and one failed.

The pair is #72 and #73 in firecrawl-mcp-server, an MCP server for the Firecrawl scraping API. Devin opened both within three minutes on 30 June 2025 with the same title, "Implement maxAge fast scraping parameter". Each adds a maxAge property to the input schema of SCRAPE_TOOL in src/index.ts, about a hundred lines apart.

+      maxAge: {
+        type: 'number',
+        description: 'Maximum age in
+          milliseconds for cached content. ...',
+        default: 0,
+      },
       actions: {

         description: 'Location settings for scraping',
       },
+      maxAge: {
+        type: 'number',
+        description: 'Maximum age in
+          milliseconds for cached content. ...',
+      },
     },

Git sees two separate hunks and merges them. The merged object literal then has two maxAge keys. The package builds with tsc in its prepare script, so npm ci stops on the merge, twice, on every cloud.

> tsc && node -e "require('fs').chmodSync('dist/index.js', '755')"

src/index.ts(198,7): error TS1117: An object literal cannot have multiple properties with the same name.

Each pull request alone passes npm ci && npm test, and so does the base.

What gitta caught that Git missed

On every replayed pair we ran gitta pr-check, the check that gitta runs on open pull requests. It ran once without tests. On the pairs that reached the merge test, it also ran with the same Docker test routine as its gate. In the gate step, gitta runs the tests on the combined change and holds the change back when they fail. The run used gitta at commit e2dbbb0.

On firecrawl, the check without tests warned both edited: src/index.ts assign SCRAPE_TOOL (#72 + #73). With the gate it reported the tests as failing, with the same TS1117 error.

gitta warned about the one broken pair on every cloud, with and without tests. It also warned on about one working pair in three, which is noise for a reviewer. With tests, it almost never stopped a pair that worked.

The one gated failure on Google Cloud is a timing-dependent test in pterm that passed on the other clouds. In many of the warnings on working pairs, both pull requests edited a changelog, a CI workflow, a test function or an import line. After this run we changed the check. Such overlaps are now listed as info when the file merges cleanly, and they no longer count as warnings (commit 43972d1). In a replay of the 44 clean pairs with that change, warned pairs went from 14 to 6, and firecrawl still warns. That leaves 5 warnings on the 43 working pairs.

On all 545 clean merges where the check finished, it never reported "does not merge". On 150 of the 170 Git conflicts where it finished, it reported "does not merge". On the other 20, its structure-aware merge combined the two sides. We did not test whether those combinations build.

How long the check takes

The check needs only the two pull request heads and their base. In the AWS run, about half of the 733 pairs finished within 5 seconds and 62% within 10 seconds. 90.5% finished within two minutes. 18 pairs hit the 600 second limit, all in very large repositories such as pytorch, sentry and the Azure SDK for Java, fetched as partial clones.

For most pairs the check gives an answer in a few seconds, so you can run it on every pull request. Only very large repositories took minutes.

The tail is the small group of slowest pairs at the right end of the chart. Most of that time went to fetching file contents one at a time in a partial clone. The same commit 43972d1 fetches them in one batched request. We reran the 18 former timeouts with a release build at 9981f65, one pair at a time on a laptop, with the same steps as the runner. 17 of them now finish in 2 to 22 seconds, with a median of 6.1 seconds. The eighteenth, OpenMetadata, was not run because its checkout would have filled the disk.

How to read these numbers

Conflicts between agent pull requests are common, and you can trust that number. Clean merges that break are real. But we found only one, so nobody can yet say how often they happen.

The textual conflict rates rest on 733 pairs and match an independent study, so they are solid. The break rate is much less certain. Only 44 clean pairs could be tested end to end, and one of them broke. That is 2.3%, with a 95% interval from 0.4% to 11.8%. It shows that clean merges of real agent work do break, and it gives one well documented case. It is too small a sample to quote as a rate for agent pull requests in general.

The testable pairs are not a random slice. They are projects whose test suites run offline in a stock container from the repository root. Projects that need services, browsers or secrets to test, about 92% of the clean merges here, may break more or less often. A break also counts only when the project's tests catch it. If a merge is wrong in a way that no test covers, it looks like a pass here, as it would in CI.

With one break, the check's recall cannot be estimated. What we can say is that gitta warned on the one break we found, both with and without tests. The warning counts on working pairs are a better guide to how much a reviewer will see from the check, and those counts moved from 13 to 5 with the change described above.

Repositories and pull request refs were fetched in October 2026, not while the pull requests were open. Closed pull requests keep their head refs on GitHub, so this changes little. But 12 deleted or private repositories dropped out. The paper excluded 31 pairs at its own time, and we excluded 14.

gitta's authors designed and ran this study, and gitta is our product. For that reason, the sample, the runner and every result row are in the gitta repository under bench/aidev-replay, and the merge and test stages do not depend on gitta at all. Anyone can rerun them and check our numbers.

Reproduce it

The harness is in the gitta repository under bench/aidev-replay, with the Terraform for the machines under infra/bench. The files are these.

python3 bench/aidev-replay/build_pairs.py pull_request.parquet pairs.jsonl   # gives the same pairs.jsonl
VCPU=8 bench/aidev-replay/run.sh aws all      # or gcp with GCP_PROJECT set; VCPU=4 azure for the 4 vCPU run
python3 bench/aidev-replay/analyze.py         # metrics.json and graphs

The firecrawl merge takes a minute to check by hand.

git init -q firecrawl && cd firecrawl
git fetch --filter=blob:none https://github.com/mendableai/firecrawl-mcp-server \
  refs/pull/72/head:pr72 refs/pull/73/head:pr73
git merge-tree --write-tree pr72 pr73          # exit 0, no conflict
git checkout -q "$(git commit-tree "$(git merge-tree --write-tree pr72 pr73)" -p pr72 -p pr73 -m merge)"
grep -n "maxAge: {" src/index.ts                # lines 95 and 198 are both in SCRAPE_TOOL

For more on catching these merges in day to day work, read Why git worktrees don't prevent agent conflicts and How to review AI-generated pull requests.

References

1. George Xu, Arjun Subramanian and Nithilan Karthik. AI Agent Pull Requests on GitHub, Frequency, Structure, and Merge Conflict Rates. arXiv 2607.04697, July 2026. arxiv.org/abs/2607.04697
2. AIDev dataset, tag v3. huggingface.co/datasets/hao-li/AIDev
3. mendableai/firecrawl-mcp-server pull requests #72 and #73.