TS-Bench

A benchmark that tests whether AI coding agents can fix real bugs.

TS-Bench mines real fixes from open-source projects, proves that each one is a fair task, runs models against them in isolated sandboxes, and scores every patch with the project's own tests. It covers TypeScript, Python and Java.

See the architecture Read the code
  1. Find a real fixA merged pull request that closed an issue and changed a test.
  2. Prove it is fairTests fail before the fix and pass after, three times over.
  3. Run the agentIsolated sandbox, fixed budget, no access to the answer.
  4. Score the patchThe project's own hidden tests decide, nothing else.
Task set
24 real bugs7 repositories, version v0.2
Languages
TypeScript, Python, JavaOne adapter interface
Real attempts
6674 models, $9.30 spent
Runs on
Docker, Kafka, KubernetesPostgres for results
Codebase
About 4,500 lines of Python46 passing tests, CI on every push

What problem this solves

Leaderboards for AI coding agents report one number. Whether that number means anything depends on questions that are rarely answered in public. Was the task actually solvable? Could the model see the answer? Was the test result stable across runs? When an attempt failed, was that the model's fault or the infrastructure's?

TS-Bench is built around those questions. Most of the engineering here is not the agent. It is the machinery that makes a score believable.

Design commitments

  • A task exists only if its tests fail before the human fix and pass after it, confirmed over three runs.
  • Agents never see the project's git history or the hidden tests, and any test file they edit is restored before scoring.
  • Infrastructure failures are kept apart from model failures and never counted against a model.
  • Every number on this page is generated from the raw run files, not typed in.

System architecture

Five layers. Each one can be tested and swapped on its own, which is why the same harness now runs three languages, two sandboxes and two ways of scheduling work.

Swipe sideways to see the whole diagram.

PresentationWhat people seeOrchestrationRuns many jobs atonceExecution harnessRuns one attempt andscores itTask pipelineTurns real fixes intochecked tasksData and sourcesWhere everythinglivesLeaderboard site: Static HTML, CSS and JavaScript, generated from the results data.Leaderboard siteStatic HTML, CSS and JavaScript, generated from the resultsdata.Statistics: Fix rate with Wilson 95% ranges, pass@k with bootstrap error bars, cost per attempt.StatisticsFix rate with Wilson 95% ranges, pass@k with bootstrap errorbars, cost per attempt.Job producer: One job per model, task and repeat.Job producerOne job per model, taskand repeat.Kafka topic: KRaft mode, 6 partitions.Kafka topicKRaft mode, 6partitions.Worker pods: Kubernetes consumer group of 3. Each result is written once, even if a job is replayed.Worker podsKubernetes consumer group of 3.Each result is written once, evenif a job is replayed.Sequential driver: run_driver.py. Same harness, one process, resumable results file.Sequential driverrun_driver.py. Sameharness, one process,resumable results file.run jobModel gateway: LiteLLM. One interface for Ollama, OpenRouter, Anthropic and a mock model.Model gatewayLiteLLM. Oneinterface forOllama, OpenRouter,Anthropic and a mockmodel.Agent scaffold: One command per turn. 40 turns and 15 minutes, identical for every model.Agent scaffoldOne command perturn. 40 turns and15 minutes,identical for everymodel.Eval runner: Resets test files, applies the patch, adds hidden tests, runs, scores.Eval runnerResets test files,applies the patch,adds hidden tests,runs, scores.LanguageAdapter: Four methods. TypeScript, Python and Java plug in here.LanguageAdapterFour methods.TypeScript, Pythonand Java plug inhere.Sandbox: A local process, or Docker with warm package caches.SandboxA local process, orDocker with warm packagecaches.Miner: GitHub GraphQL. Merged pull requests that close an issue and change a test.MinerGitHub GraphQL. Merged pullrequests that close an issueand change a test.Validator: gold and empty gate: Tests fail before the fix and pass after. Three runs drop flaky tests.Validator: gold and empty gateTests fail before the fix and passafter. Three runs drop flaky tests.Dataset export: Versioned JSONL, plus merge dates.Dataset exportVersioned JSONL, plusmerge dates.Rigor audit: Re-runs the gate to catch drift.Rigor auditRe-runs the gateto catch drift.GitHub API: Issues, pull requests, changed files.GitHub APIIssues, pull requests,changed files.Git mirrors: One bare clone per repository. Source of every diff.Git mirrorsOne bare clone per repository.Source of every diff.Task set v0.2: 24 tasks, 3 languages.Task set v0.224 tasks, 3 languages.Postgres: tasks, runs, results.Postgrestasks, runs,results.
Boxes are components in the repository; the dashed box is an external service and the tinted ones are outputs. Moving dashes show the direction of work. Hover a box for its role.

How a task is proven and an attempt is scored

The hard part of a benchmark is trusting the tasks. Every task passes a gate before it is admitted, and every attempt is scored on a clean rebuild the agent never touched.

Swipe sideways to see the whole diagram.

1 Candidate: A merged pull request that closed an issue and changed a test.1 CandidateA merged pullrequest thatclosed an issueand changed atest.2 Materialize: git archive of the commit before the fix into a plain folder. No history.2 Materializegit archive ofthe commit beforethe fix into aplain folder. Nohistory.3 Install: The language adapter installs dependencies at the exact runtime version.3 InstallThe languageadapter installsdependencies atthe exact runtimeversion.4 Red run: Add only the pull request's tests. The target tests must fail.4 Red runAdd only the pullrequest's tests.The target testsmust fail.5 Green run: Apply the human fix. The same tests must now pass.5 Green runApply the humanfix. The sametests must nowpass.6 Derive sets: fail_to_pass and pass_to_pass, from three runs. Flaky tests dropped.6 Derive setsfail_to_pass andpass_to_pass,from three runs.Flaky testsdropped.7 Accept: A schema-checked TaskInstance is written to the dataset.7 AcceptA schema-checkedTaskInstance iswritten to thedataset.otherwiseRejected, with a logged reason: No failing test before the fix, tests still fail after it, or the install broke.Rejected, with a logged reasonNo failing test before the fix, tests still failafter it, or the install broke.Compiled languages: if the new tests cannotcompile before the fix, that counts as failing.Test names are read from the diff, and thegreen run still has to prove them.
The gold and empty gate. It is what separates a benchmark from a pile of GitHub links: a task must be provably broken before the fix and provably fixed after it.

Swipe sideways to see the whole diagram.

The agent works hereTask: Issue text and the commit before the fix.TaskIssue text and thecommit before the fix.Private workspace: A copy with its own throwaway git repo. None of the project's history.Private workspaceA copy with its ownthrowaway git repo. None ofthe project's history.Agent loop: The model sees the issue and code. One shell command per turn, 40 turns, 15 minutes.Agent loopThe model sees the issue and code.One shell command per turn, 40turns, 15 minutes.Patch: git diff of everything it changed.Patchgit diff ofeverything itchanged.Meter: Tokens, cost and time are logged.MeterTokens, cost andtime are logged.Then scoring starts from scratchRebuild clean: New copy of the commit before the fix. No .git.Rebuild cleanNew copy of thecommit before thefix. No .git.Reset tests: Any test file the agent touched is restored.Reset testsAny test file theagent touched isrestored.Apply patch: Strict first, then fuzzy. A bad diff scores as not fixed.Apply patchStrict first, thenfuzzy. A bad diffscores as not fixed.Add hidden tests: The project's real tests for this bug.Add hidden testsThe project's realtests for this bug.Run the suite: In the sandbox, with a timeout.Run the suiteIn the sandbox, witha timeout.Score: Fixed only if every target test passes and nothing else breaks.ScoreFixed only if everytarget test passesand nothing elsebreaks.patchResult: status (ok, patch_apply_failed, timeout or infra_error), resolved, per-test outcomes, cost, tokens and seconds. infra_error is our failure, notthe model's, so it is left out of the score.
The agent's workspace is deliberately separate from the scoring build. Restoring edited test files and adding the hidden tests only after the patch is applied stops an agent from passing by weakening the tests.

How each attempt is counted

Fixed
All target tests pass and nothing else broke.
Not fixed
The patch ran, but a target test still fails or something else broke.
Timed out
The tests did not finish in time. Counts as not fixed.
Patch did not apply
The diff was malformed. Counts as not fixed.
Excluded
Our own failure, such as a billing error or a broken install. Never held against a model. None of the attempts on this page were excluded.

Why zero is not a scoring bug

A run of zeros looks suspicious, so it was checked. The scoring accepts the real human fix on every task and rejects an empty patch on every task. One local model once scored a fix in a timing test, then failed the same task ten times. Tracing it showed the scoring was correct: that task checks two near-identical test variants, the human fix passes both, and a partial fix can pass only one. The one-off pass was most likely run-to-run variation in local inference.

One interface, three languages

Only four things depend on the language: how to detect the environment, install, run the tests, and read the results. Everything else is shared.

Swipe sideways to see the whole diagram.

Core, with no language-specific code: pipeline (miner, validator, export), harness (eval runner, sandbox) and agent. Everything above the interface treats every language the same.Core, with no language-specific codepipeline (miner, validator, export), harness (eval runner, sandbox) and agent. Everything above the interface treats every language the same.get_adapter(language): get_adapter(language)LanguageAdapter interface: LanguageAdapter interfacedetect_environment()install()run_tests()parse_results()Adding Java took one line here and two schema allowlist entries.TypeScript adapter: npm, pnpm or yarn, from the lockfile Exact Node version through nvm vitest or jest, JSON to a file Monorepo awareTypeScript adapternpm, pnpm or yarn, from the lockfileExact Node version through nvmvitest or jest, JSON to a fileMonorepo awarePython adapter: pip, poetry, uv or pipenv detected Isolated uv venv per task pytest with a JSON report Exit code 4 means collection failedPython adapterpip, poetry, uv or pipenv detectedIsolated uv venv per taskpytest with a JSON reportExit code 4 means collection failedJava adapter: Maven, or a repo's own Gradle wrapper One modern JDK builds every project Surefire and JUnit XML reports A compile failure is a real resultJava adapterMaven, or a repo's own Gradle wrapperOne modern JDK builds every projectSurefire and JUnit XML reportsA compile failure is a real resulteach adapter implements it
The original plan promised zero core changes for a second language. That did not survive Python: it needed a one-function factory and one schema field, and a field named node_version had to be renamed because it had leaked TypeScript into the schema. Java, the third language, then needed one registered line. Recorded in the build log as a design finding, not smoothed over.

Running it at scale

Task by model by repeat quickly becomes hundreds of runs. A queue turns that into parallel workers, provided a replayed job can never create a duplicate result.

Swipe sideways to see the whole diagram.

Job producer: One job per model, task and repeat.Job producerOne job per model,task and repeat.Kafka topic: ts-bench-jobs, 6 partitions, KRaft mode.Kafka topicts-bench-jobs, 6partitions, KRaft mode.Kubernetes Deployment: 3 replicas, one consumer groupWorker 1: Runs one job at a timeWorker 1Runs one jobat a timeWorker 2: Runs one job at a timeWorker 2Runs one jobat a timeWorker 3: Runs one job at a timeWorker 3Runs one jobat a timePostgres: One row per run, task and repeat.PostgresOne row per run, task andrepeat.1 Pull a job: Take one message from the topic.1 Pull a jobTake one message from thetopic.2 Already done?: Look up the key in Postgres. If present, skip.2 Already done?Look up the key inPostgres. If present,skip.3 Run it: Agent, then the eval runner, in a sandbox.3 Run itAgent, then the evalrunner, in a sandbox.4 Write once: INSERT with ON CONFLICT DO NOTHING.4 Write onceINSERT with ON CONFLICT DONOTHING.5 Commit the offset: Only now does Kafka mark it done.5 Commit the offsetOnly now does Kafka markit done.Key is (run_id, instance_id, repeat), the same key the sequential driver uses. If a pod dies between steps 3 and 5 the offset was never committed, soanother pod picks the job up. Verified by force-killing a pod mid-job: 30 jobs gave exactly 30 rows, none lost, none duplicated.
Verified on a real single-broker Kafka and a local Kubernetes cluster, in two runs of 15 and 30 jobs. This demo baked in only the Python tasks to keep the worker image small, so it proves the queue mechanism, not the full task set.

Tech stack

About 4,500 lines of Python in the core, 46 passing tests, and lint and formatting checks enforced by CI on every push.

AreaToolsWhat they do here
Language and packagingPython 3.12, uv, hatchlingCore code, locked and reproducible environments.
Data and schemaPydantic v2, JSONL, HuggingFace datasetsValidated task schema and a versioned, immutable dataset that loads with the datasets library.
StoragePostgreSQL 16, psycopg 3tasks, runs and results tables. Composite keys make replayed writes safe.
Model accessLiteLLM, Ollama, OpenRouter, Anthropic APIOne interface for local and hosted models, with cost and token logging.
Targets under testNode via nvm, pnpm, npm, yarn, vitest, jest, pytest, Maven, JUnitWhat the language adapters drive inside each task's own repository.
IsolationDocker, warm package-cache volumes, git archiveRuns without host state. Task folders are built with no git history.
Scale-outApache Kafka 3.8 (KRaft), kafka-python, Kubernetes (kind)An idempotent job queue and a pool of worker pods.
Task sourcesGitHub GraphQL API, bare git mirrorsMining merged pull requests and extracting exact diffs.
Agent experimentLangChain, LangGraphA tool-using second agent, in a separate project.
Qualitypytest, ruff, pre-commit, GitHub Actions46 passing tests. CI enforces lint and formatting on every push.
This siteStatic HTML, CSS, JavaScript, inline SVG, Cloudflare PagesGenerated from the results data by a standard-library Python script. No framework.

How it was built

Fifteen tasks in dependency order, each finished only when its own check passed. The task board in the repository records what was actually built, including every bug hit on the way.

  1. FoundationT1, T2

    A monorepo with uv, ruff, pre-commit and CI, and a Pydantic TaskInstance schema as the contract between every stage.

    The LanguageAdapter interface and a TypeScript adapter, checked on three real repositories (zod, class-validator, trpc) with identical results across runs.

  2. Tasks worth trustingT3, T4, T11

    The validator and its gold and empty gate, a GitHub GraphQL miner, and 24 validated tasks in the end.

    Contamination dates for every task, and a repeat-run audit that re-checks the whole set (13 of 13 clean when last run).

  3. Execution and scoringT6, T7, T8, T9

    An eval runner with anti-cheat test resets, a deliberately minimal reference agent, a LiteLLM gateway with cost tracking, and a hard split between model failures and infrastructure failures.

    pass@k with bootstrap error bars, checked against hand calculations before it was trusted.

  4. Reproducibility and breadthT12, T13, T14

    A Docker sandbox with warm package caches, measured 25 to 35 percent faster on a monorepo and 2.5 to 3 times faster on a single package.

    Python and Java adapters that proved the language seam.

  5. Storage and scaleT5, T15

    A versioned dataset export and a Postgres schema that round-trips it exactly.

    A Kafka and Kubernetes job queue, verified against a force-killed worker.

  6. Real models, and this siteStep 7, T10

    667 real attempts on local and paid models. Eight bugs surfaced, and were fixed, only once real and imperfect model output flowed through.

    This site, generated from the results with its limits stated first.

Problems that took real debugging

Each of these looked like something else at first. The lesson repeated: when a failure cannot be reproduced on the spot, read the tool's source instead of trusting the error text.

Node version was recorded but never enforced

Symptom
Nearly every trpc candidate was rejected with dozens of unrelated failing tests.
Cause
Tests ran under whatever Node the shell defaulted to (v24). The repository pinned 22, and the two disagreed about fetch internals.
Fix
The adapter resolves the exact Node version through nvm and puts it first on PATH. The six newest trpc candidates then validated on the first try.

A passing gate can hide an untested safety path

Symptom
The gold and empty gate passed, yet nothing had exercised the rule that restores test files an agent edits.
Cause
The human patch never touches test files by construction, so the restore step always ran on an empty list.
Fix
A direct test tampers with a real test file and checks it is restored.

Caching the wrong thing in Docker

Symptom
In a pnpm monorepo, vitest was missing, fell back to a global copy, and the harness reported a clean-looking but wrong result.
Cause
The first design cached node_modules by lockfile hash and skipped the install, so per-package node_modules were never created.
Fix
Always run the install and persist the package managers' own caches instead. Warm installs were 25 to 35 percent faster on a monorepo, 2.5 to 3 times on a single package.

Compiled languages broke task mining

Symptom
74 Java candidates from two repositories produced zero tasks.
Cause
The new tests call a method the fix adds, so they cannot compile before the fix. A dynamic language would simply fail that one test.
Fix
A compile failure now counts as failing before the fix, with test names read from the diff. A wrong guess can only miss a task, never approve a false one. Six Java tasks validated, and the work exposed a scoring bug that under-scored parameterized tests.

Failures scored as infrastructure errors

Symptom
A real model's broken Java or Python patch was recorded as infra_error, which is excluded from the score.
Cause
Every earlier check ran only the human patch, which always compiles. Real model output was the first thing to reach that path.
Fix
Typed exceptions for compile and collection failures (and pytest's own exit code 4), so they count against the model. Left alone, it would have quietly inflated Java scores.

Environment bugs that looked like flakes

Symptom
Bursts of identical infrastructure errors in the middle of a long run.
Cause
Three causes: Ollama restarted as the wrong user served zero models (230 attempts lost), nvm install hit the network on every call, and a global .npmrc with workspaces=true broke npm for every project.
Fix
A 15-minute watcher caught each burst. Bad rows were removed and the run resumed without rework. Reading the tools' source found the last two.

Kafka on one broker never assigned work

Symptom
Workers reported ready and never received a partition.
Cause
The offsets topic defaults to 3 replicas, which one broker can never satisfy, and the job topic had been created with a single partition, so two of three workers sat idle.
Fix
Set the replication factor to 1 and raised the topic to 6 partitions. Then verified concurrent workers and a force-killed pod: nothing lost, nothing duplicated.

What these numbers can't tell you

This is an early, small run. It is honest about what it measured, and these are the reasons not to over-read it.

Only one model fixed anything, and only 20 tries back it up.

Claude Haiku 4.5 fixed 6 of 20 bugs. With so few tries the honest range is 15% to 52%. It covers 20 of 24 tasks because the paid budget ran out, and it got one attempt per task, so luck could move that number a lot.

4 of the 6 fixes came from one project.

Those 4 are all in trpc. Claude Haiku 4.5 fixed 4 of the 5 trpc tasks it tried. That says the tasks in one repository were easier for it, not that it is good at every codebase.

The free models scored zero, under one deliberately plain setup.

Across 647 attempts, Qwen2.5-Coder 14B, Codestral 22B and Qwen3 14B fixed 0. Their agent types one shell command per turn, with 40 turns and 15 minutes. That does not prove these models can't fix these bugs with better tooling.

Two planned models are incomplete or missing.

gpt-oss 20B was never run, so it has no score and is not shown as zero. Qwen3 14B stopped at 167 of 240 attempts, covering 17 of 24 tasks.

Older tasks may already be in a model's training data.

4 of the 24 real fixes were merged before 2025, the oldest in Apr 2021. We haven't recorded each model's training cutoff yet, so nothing on this page is labelled for contamination. That is a gap, not a clean bill.

It is a small set: 24 tasks.

7 repositories: 13 TypeScript, 6 Java, 5 Python. Java tasks are the hardest to build, and the set was grown by hand-checking candidates, not sampled at random.

Not done yet

  • The task set is not yet published on HuggingFace; it lives in the repository as JSONL.
  • No stranger has yet reproduced a number from this page using only the write-up. That check is still open.
  • The repeat-run audit that catches flaky or drifting tasks last covered 13 of the 24 tasks. It has not been re-run on all of them.
  • A patch that breaks TypeScript syntax so badly that no test can start is not yet scored as a plain failure. It has not happened in a real run, but the gap exists.

Results by model

Models with zero fixes are tied, so they are not ranked. The bar shows the range the true fix rate probably sits in, given how few attempts there are. A wide bar means low confidence.

Bugs fixed by each model
Model Bugs fixed Likely range (95%), bar scale 0 to 60% Cost Tasks covered Run
Claude Haiku 4.5Anthropic, paid API6 of 20
15% to 52%
$9.3020 of 24Stopped early
Qwen2.5-Coder 14BAlibaba, local, free0 of 240
0.0% to 1.6%
Free24 of 24Complete
Codestral 22BMistral, local, free0 of 240
0.0% to 1.6%
Free24 of 24Complete
Qwen3 14BAlibaba, local, free0 of 167
0.0% to 2.2%
Free17 of 24Stopped early
gpt-oss 20BOpenAI, local, freeNo dataNever runNoneNoneNot run

Paid spend was $9.30 of a $10 budget, about $0.47 per attempt. A further 2 attempts by Claude Sonnet 4.5 ($2.88) were early smoke tests, fixed 0, and are too few to rank.

Every attempt

One square per real attempt, 667 in total. Hover or tap a square for details.

Group by
  • Fixed the bug
  • Did not fix it
  • Tests timed out
  • Patch would not apply

Every task, model by model

Each cell is fixes over attempts. The date is when the real fix was merged. The older it is, the more likely a model saw it during training.

TaskReal fix mergedClaude Haiku 4.5Qwen2.5-Coder 14BCodestral 22BQwen3 14B
jhy/jsoupJava, 3 tasks
#2595Sep 20260/10/100/10not run
#2598Sep 20260/10/100/10not run
#2602Sep 20260/10/100/10not run
stleary/JSON-javaJava, 3 tasks
#1044Mar 20260/10/100/10not run
#1068Jul 20260/10/100/10not run
#814Oct 20230/10/100/10not run
arrow-py/arrowPython, 2 tasks
#1222Oct 20250/10/100/100/10
#954Apr 20210/10/100/100/10
jd/tenacityPython, 3 tasks
#609Feb 20261/10/100/10not run
#615Feb 20260/10/100/100/7
#654Jul 20260/10/100/100/10
colinhacks/zodTypeScript, 5 tasks
#6530Aug 20260/10/100/100/10
#6532Aug 2026not run0/100/100/10
#6534Aug 2026not run0/100/100/10
#6572Sep 20261/10/100/100/10
#6587Sep 2026not run0/100/100/10
date-fns/date-fnsTypeScript, 2 tasks
#3132Jan 20240/10/100/100/10
#3662Jan 20240/10/100/100/10
trpc/trpcTypeScript, 6 tasks
#7370May 2026not run0/100/100/10
#7390Jul 20260/10/100/100/10
#7434Sep 20261/10/100/100/10
#7464Sep 20261/10/100/100/10
#7469Sep 20261/10/100/100/10
#7477Sep 20261/10/100/100/10

Check the work

The code, tasks and results are open. You need Docker, uv, and either Ollama or an API key. Nobody outside this project has run these yet, so treat the steps as untested by others.

# the human fix must resolve every task, an empty patch none
uv run python scripts/run_gate.py

# run a model on every task, 10 tries each (resumable)
uv run python scripts/run_driver.py \
  --models ollama_chat/qwen3:14b \
  --instances all --repeats 10 \
  --out results/my_run.jsonl

# fix rate and pass@k with error bars
uv run python scripts/compute_stats.py \
  --in results/my_run.jsonl --k 5

A second agent design

The plain agent types one shell command per turn. A second agent with real tools (read, search, edit, run tests) and language-specific instructions was built with LangGraph in a separate project. On 5 Python tasks with Qwen3 14B it fixed none, although once we fixed problems in our own setup it wrote a patch on 3 of the 5 tasks, up from 1. None of those patches was correct.

That is one model on five tasks, so it is a note and not a finding. It changes nothing in the main benchmark.