Co-Design Bench explained: why top coding agents finish only 26.4%

Ask an AI coding agent to change your app and it will usually get most of the way there. A new benchmark from Alibaba-led researchers puts a number on "most of the way": the best of ten models passed about 70% of the hidden checks, yet fully finished only about one task in four. That gap, not the headline score, is what decides whether you ship the result or spend your afternoon fixing it.

The plain-English version

Imagine you hire a painter to redo your flat. You give a list of ten things: paint the living room blue, fix the crack above the door, repaint the radiator, tidy the edges, and so on. The painter does seven of them well and quietly skips three.

If you graded the painter by "what share of my list got done", you'd say 70%. Not bad. But if you ask "did I get the flat I ordered, so I can move the furniture back in?" the answer is no. You still have to call someone for the last three jobs. If the painter does this on most flats, then only a minority of customers ever get a finished flat, even though the painter is "70% good".

That is the finding of Co-Design Bench, released by researchers from Alibaba and several universities (arXiv paper, project page). The "flats" are 273 real requests to change an existing web app, such as "read the amount from this receipt and update the totals that depend on it". They come from roughly 26,000 real coding sessions. Each request comes with a hidden checklist written by experts, 2,331 items across all tasks, and the AI agent never sees it.

Ten AI models were tested as coding agents. The best, OpenAI's GPT-6 Astra, satisfied about 69.9% of the checklist items on average. But it fully finished only 26.4% of tasks, according to The Decoder and MLQ. In the other roughly three out of four cases, a person would still have to step in.

Two honest footnotes. First, the researchers deliberately picked hard cases, so this is not the success rate you'd see on an average to-do item. Second, "finished" is a strict standard: one missing checklist item means the task is not complete.

What a reader should be able to tell a friend: AI coding helpers look far better on older tests than they do on messy real requests, and "mostly right" is not the same as "done".

How it actually works

The problem. Coding benchmarks have a shelf life. Once models score near 100%, the test no longer shows what is still hard. The authors point out that older edit benchmarks are saturated: models that can run and fix their own code score above 99%, per The Decoder. Meanwhile, anyone who uses these tools knows the day-to-day experience is bumpier.

The old way. Many benchmarks take a task from a public repository and check whether a pre-written test passes. That is cheap to grade, but it measures "did the test turn green", which an agent can sometimes achieve without solving the real problem. A METR study found that real maintainers' merge rates were about 24 percentage points below SWE-bench Verified's automated scores.

The new idea, step by step.

1. Start from real requests. The team collected 37,463 deduplicated queries from 26,314 real user coding sessions and curated 273 tasks. They favoured cases where users had to rephrase a request or roll back a change, because those are the places agents struggled in practice.
2. Pair each request with a runnable app and a hidden checklist. Experts wrote the acceptance criteria. The agent sees only the request and the app.
3. Let the agent work. Each model runs inside the same agent setup (the "harness") to read, edit and run the code.
4. Grade the result by using it. A judge model inspects the source code and drives the finished app in a browser to check each criterion.
5. Try to overturn every pass. A second model re-checks each positive verdict, hunting for mock-ups or hard-coded results. Credit goes only to features that really work.
6. Score two ways. The average share of criteria met, and the share of tasks where all criteria are met.

Why this exposes a gap. Suppose a model meets each criterion with probability around 0.7 and the tasks have several criteria each. The chance of meeting all of them is much lower than 0.7. Partial and full completion measure different things, and users pay for the second.

The tasks themselves. There are two tracks. Multimodal editing (100 tasks) covers localized changes that may include images, audio, video or documents. Repository editing (173 tasks) covers coordinated changes across a codebase. The best model fully solved 41.0% of the first kind and 17.9% of the second, per MLQ.

What goes wrong as tasks grow. All ten models did worse on repository edits that combined three or four types of change (layout, design, interaction, content) than on those with one or two. Adding more compute did not reliably rescue this: according to The Decoder, GLM-5.2 used nearly six times as many tokens as GPT-6 Astra on smaller changes and still scored significantly worse.

The deep dive

Dataset construction and tasks

Per the paper's abstract, the benchmark has 273 tasks sourced from about 37,000 deduplicated queries across roughly 26,000 real coding sessions. Tasks fall into three capability areas: grounded understanding (reading the input material correctly), UI transformation, and functional implementation. Each task bundles a normalized request, a runnable pre-edit app, and expert-written hidden criteria. The 2,331 criteria across 273 tasks average about 8.5 per task, which is my own division, not a figure from the paper.

Reference-free, judge-then-refute grading

Most benchmark graders compare against gold code or a target screenshot. Here there is neither. Graders see source code and browser evidence. Formally, for task $t$ with criteria $c_1,\dots,c_{n_t}$, each criterion gets a verdict $v_{t,i}\in\{0,1\}$ after the verifier has attempted to refute positives. The two reported metrics are:

The first is partial credit; the second is the strict "all checks pass" rate. The metrics' definitions are my formalization of the paper's description. A 43.5-point distance separates the two for the best model (69.9% vs 26.4%).

An asymmetry worth noting: a reader comment cited by the DEV Community analysis argues that the verifier challenges only positive verdicts and never fails, so wrongly failed criteria are never corrected, which could bias all ten scores downward by an unmeasured amount. This is the commenter's critique, not a finding of the paper.

Results

Numbers as reported by the authors and relayed by press (all vendor-neutral academic measurements, but with Alibaba authors):

Rankings differ by size of job. The Decoder reports that Qwen3.8-27B climbed to second place on larger overhauls. I could not read the paper's full results table, so remaining models' figures are omitted rather than guessed.

The harness is part of the result

The main leaderboard ran under one shared harness, which the DEV analysis identifies as Claude Code. In a smaller follow-up, two models run on three agent CLIs differed by up to 18.9 points, and Qwen3.8-27B's repository score moved between 0.560 and 0.749 depending on the tool, per MLQ. Which model won also depended on the environment. The authors' recommendation: report the model and the working environment together.

This complements a separate result mentioned by The Decoder: a study that found little measurable benefit from more elaborate harnesses despite up to three times the tokens. Together they say that "harness" is a bigger variable than the field's leaderboards admit, but more machinery is not automatically better.

Compared with the closest prior work

• WebCompass-edit and similar tests: near-perfect scores; Co-Design Bench shows a gap of more than 35 points relative to them (The Decoder).
• SWE-bench Verified vs human review: METR's study had four maintainers review 296 AI patches and found merge rates about 24 points below automated scores; the authors there said this is not a fundamental ceiling because agents could not iterate on feedback. Co-Design Bench attacks the same gap from a different side: rather than human reviewers, hidden criteria plus a refuting judge.
• The project page also cites a WebCompass comparison in which one model's graded completion rose from 73.6% to 95.1% when moved from single-shot to agentic execution, but on a different task set, so it can't be compared directly.

Compute and cost

The paper's token accounting is only partly visible in the coverage. The clear data point is the GLM-5.2 vs GPT-6 Astra comparison above. The licence is CC BY-NC-SA 4.0, so commercial reuse of the paper is restricted.

What is released

The DEV analysis states the public release has only 96 multimodal tasks and no repository tasks, and that most requests are in Chinese. The project page also lists the Hugging Face dataset as 96 multimodal tasks (521 criteria), which conflicts with the 100 multimodal tasks (553 criteria) it describes for the full benchmark. Two frontend experts checked rubric scores on 200 outputs, with no agreement figure reported, according to the DEV analysis. I have not inspected the repository myself.

Why it matters

Everyday users. If you use AI to change a website or small app, expect the last 20–30% to be the hard part. The benchmark's failures are not dramatic crashes; they are quiet misses, like a total that doesn't update. Check the result by using it.

Developers and builders. Write acceptance checks before handing off a task, keep them out of the agent's reach, and keep requests narrow: the data says performance falls when a request mixes layout, design, interaction and content. Test model and harness together on your own repo, since the pairing changed scores by up to 18.9 points.

Companies. The useful number for planning is the human cleanup rate, not the pass rate. A team that budgets for "70% automated" and gets 26% fully finished tasks will find the savings arrive as faster first drafts, not as fewer reviews. That is a real gain, but a different one.

The field. Benchmarks that saturate stop guiding research. Hidden criteria derived from real sessions are harder to overfit than public tests. Expect more evaluations that grade running software through a browser, and more attention to partial versus full completion.

Second-order effects. If harness choice moves scores nearly as much as model choice, then leaderboards that compare models alone mislead buyers, and vendors that control both model and harness may have an advantage that is hard to see from outside. It also strengthens the case for evaluating agents on your own work rather than trusting a headline figure.

What to be skeptical of

• Hard-case selection. The authors favoured tasks where users had to rephrase or roll back. The Decoder is clear that this makes 26.4% a stress-test number. Don't read it as "agents fail 74% of your tickets".
• AI judges. The paper states judgment error cannot be eliminated. The DEV analysis says expert checks on 200 outputs had no reported agreement figure. Without it, we can't tell whether the strict grading is calibrated.
• One-sided audit. According to a commenter cited by the DEV analysis, only positive verdicts are re-checked, so wrongly failed criteria are never corrected and scores may be biased downward.
• Fixed requests. Each task has one request. Real work involves clarifying questions and mid-course corrections; the benchmark tests neither. Human collaboration may close part of the gap, and the METR authors made a similar point about iteration.
• Authorship. Some authors are from Alibaba, whose Qwen models placed near the top. That is not evidence of bias, but it is the standard reason to wait for independent replication.
• Reproducibility. According to the DEV analysis, the public release is partial and mostly in Chinese. I could not independently confirm this, so check the repository.
• Secondary summaries. I could not read the PDF's full text, so per-model tables beyond those cited in press coverage are not reproduced here. Treat the details above as the press's and authors' summaries.
• Commentary conflicts. The DEV author promotes their own IDE, so product advice there should be weighed accordingly.

What to watch next

• Full public release. Whether the repository-editing tasks (173 of 273) become public so others can reproduce the 17.9% figure.
• Independent replications. Any third-party run with a different judge model, and a reported judge-vs-human agreement number.
• Harness-controlled leaderboards. Whether the benchmark or others start publishing model-by-harness grids.
• Interactive versions. Whether a follow-up allows clarifying questions and mid-task corrections, which the authors list as untested.
• Vendor responses. New model releases from OpenAI, Anthropic, Alibaba or xAI that report full-completion rates, not just average criteria.
• Related reading. Our earlier look at how AI is changing its own development speed: METR's Anthropic audit explained, and at agents that find exploits at scale: OpenAI's 948 Linux root exploits explained.

How this was made: researched and written by an AI model (Claude) from the primary sources listed above, then checked claim-by-claim against those sources in a separate AI fact-check pass. Spotted an error? Email [email protected] and we correct it publicly. Our process.