2026-09-07 08:00:00
We previously noted that, while it's easier than ever to hit a particular quality bar by having coding agents use effective test techniques, software quality seems to be getting worse, indicating that whatever defaults developers are using may not work very well. Here, we test if simple instructions to agents to use particular techniques or libraries improve implementation correctness, as a kind of test to see how effective agents are when guided by someone with no expertise in testing who's maybe heard that you should apply certain techniques or use certain libraries.
We'll re-use the Zstd implementation eval discussed in this comparison of agentic programming language effectiveness and, instead, compare different testing techniques and testing libraries when agents are given a prompt to implement Zstd with different addendums, such as "Use test-driven development", "Use Lean 4", "Use QuickCheck", "Use property-based testing", etc. I also ran some other evals, such as on the IMAP RFC, which are briefly discussed.
All implementations were in Rust. The 26 prompt conditions tested were ACL2, Alloy, "Audit and fuzz risky areas", "Audit first", Creusot, Default (no additional instructions), Differential testing, Fuzzing, Hegel, Insta, Judgement (agents asked to use the best technique), Kani, Lean 4, "Make no mistakes", Metamorphic testing, Mutation testing, Property-based testing, Proptest, QuickCheck, rstest, Rust built-in test framework, SMT solvers (with Z3, cvc5, and Yices, all available), Spin, TDD, TLA+, and Verus. Additional, 4 skills were tested: Hegel with the official Hegel skill, the ECC Rust test skill (ECC is a collection of skills with 250k GitHub stars and 38k forks), the Trail of Bits property test skill, and a test skill I wrote (I'm a luddite who uses prompts instead of skills and have no feel for how to write a good skill). Other than my skill, the skills were chosen because those were the top skills codex turned up when asked to find relevant skills.
I pre-registered some guesses on how conditions will do:
Below, we have a very messy graph which shows the results for the conditions tested (codex with GPT-5.6 Sol, with medium and xhigh efforts). When looking at data, I tend to prefer much denser and messier graphs than most people, such as the first graph here. Because most people find these kinds of graphs unreadably messy, I tend to split information out into a series of graphs, each of which shows less information, when presenting information to others. For reasons discussed elow, I'm not going to do this here and am just going to present this extremely messy graph where we have cost on the x axis and the fraction of runs that passed 100% of the (hidden) tests on the y axis, average of 80 runs from each condition and effort (mousing over items shows bootstrap covariance, 50% uncertainty, and there's some attempt at making like things similar colors, e.g., blue-ish for formal methods, green-ish for property-based testing, etc.):
One thing we can see is that nothing really wildly outperforms. However, Default (no additional instructions) does well above average. Looking at xhigh, on average, the fuzzing and PBT-related conditions did a little better than formal methods on average, with the situation being a lot more mixed at medium. The testing-related skills codex recommended we try underperformed, although our quick custom skill did ok (a major difference is that our skill is designed to nudge away from their default behavior towards more productive behaviors whereas the other skills seem more like tutorials). TDD didn't do well, as predicted (one skill also suggested that agents used TDD, and that skill also fared poorly in the cases where agents attempted to follow the instruction).
If we actually look at what agents did, it quickly becomes apparent that, in general, agents don't know how to use these tools or techniques very well. As we noted here, and as everybody I've talked to has also noted, agents are really bad at testing and don't seem to understand how to test reasonably "by default". For example, here's a comment by Gary Bernhardt:
AI agents' approach to testing, more or less:
Take the pathological cases dreamed up by someone objecting to mocks 15 years ago, without ever having actually used mocks. Naive dreams of excessive mocking.
Make those pathologies the backbone of your testing strategy.
It turns out, if you ask agents to use a particular test technique or test library, this approach doesn't change as much as you'd hope. We'll look at what happened in cases in more detail, but at a high level, with test techniques, agents tend to either just write the tests they would normally write, but inside a framework for a different type of test technique, or they'll use a technique superficially but not really do the things that get the value out of the technique. For the most part, when a technique was named, they did what Gary described, but with respect to that technique (for example, for formal methods, they mostly proved irrelevant properties and with property-based testing, agents would lean heavily on totally random inputs and heavily hit invalid/rejection cases or find a trivial property to check and run low-value random cases against the trivial property). Results weren't materially different on the IMAP RFC (where I tried 40 runs of each condition) or other random RFCs (where I tried a few individual runs). In general, regardless of the type of problem, whether it's some kind of bit manipulation problem like Zstd, a protocol like IMAP1, or anything else, agents did not use formal methods or test libraries or techniques in an effective way.
On xhigh, agents were generally able to get the tests they wrote to pass, but they wrote poor tests (e.g., they'd submit four identical bitstreams into a test of a feature that uses four bitstreams and miss any bug that would occur because they transposed bitstreams). And as we noted previously on the Zstd eval with respect to languages, running at a lower effort level in a naive loop gets worse results (agents do even more of this and stall out with lower correctness).
I'm curious why AI labs haven't created RL envs to get agents to learn how to test well since software not working reasonably seems important for coding agent adoption and it also seems like the kind of thing that's amenable to RL. As we previously saw, agents have gotten quite good at bounded runtime optimization problems, which makes sense because that's exactly the kind of thing you cheaply create a ton of RL envs to train on. Maybe this is one of those things that's harder than it seems when you try it, but creating RL envs for effective testing and test techniques seems like it's in the same class of problem. Perhaps the limiting factor is just that knowledge of effective test techniques isn't very widespread, so no one's thought to try it and people are getting agents to test inefficiently (for example, by doing standard unit testing)2, or maybe this problem is much harder to package up than runtime optimization for some reason? It's possible this will be a moot point soon if agents get so good that they can generally write correct code without testing or verification, but at least for the state of publicly available agents from inception until now (September 2026), it seems like agents having some idea how to test without being guided by a testing expert would've substantially increased agentic coding effectiveness.
Below we'll look at how agents did things for each condition, ordered from worst correctness to best, but I would caution anyone against drawing any kind of strong conclusions from the ordering.
A lot of the failures here seem analogous to the failures we saw when we looked at the impact of programming language on token usage and correctness, in that the failures are often idiosyncratic. For example, with programming languages, we saw that agents had a fairly high rate of getting the semantics of byte conversion incorrect in Clojure but not Java, even though agents "should" (and probably sort of do) know that they can get Java byte conversion semantics by converting with unchecked-byte instead of byte.
Although people have all sorts of hand wave-y high-level explanations for why some languages are better for agents than others, when we look at what agents actually do and what the failure modes are, none of the explanations I've heard for why someone's pet language is suited for agentic coding, whether it's Elixir or Ocaml or J, are actually true (with the exception of comments about Rust's memory safety, which were validated in the multi-language pandoc eval we tried by comparison memory safety issues between agent-written C, C++, and Rust). Instead, we see a bunch of idiosyncratic failures that happen for unclear reasons3. With languages, because we can observe a moderate correlation between language popularity and performance (both lower cost and higher correctness), it seems reasonable to guess that the reason is because there was more training data (possibly synthetic data and not just human-written code) for more popular languages. Here, there isn't a clear pattern, other than that agents are mostly not very effective at applying test or verification techniques when all they have is the name of a library or technique (we'll discuss what works better afterwards). If you don't want to read about what happened in each condition, click here to skip to the last item.
Verus uses an SMT solver and various types of reasoning to prove that the code matches specifications.
Although Verus can prove that code matches specifications, agents didn't do that. Instead, they made proofs about various abstract properties relating to Zstd. I've not used a tool like Verus myself, so I can't speak to what an expert or even a beginner user would normally do, but from reading the tutorial, I find it a bit odd that agents didn't attempt to use Verus to verify any of the actual code and only used it to do abstract reasoning, as it seems designed to make it easy to prove properties about the actual code.
Additionally, if we look at the properties proved, there were generally few properties proved and the properties that were proved were uninteresting. For example, agents would prove things like "given a valid cursor/index/distance, the resulting operation remains in bounds", which isn't bad to prove, but wasn't really a source of bugs. Also, agents would frequently write vacuous proofs that were effectively A => A. An actual Verus proof of this form was:
requires
0 < a <= window,
0 < b <= window,
0 < c <= window,
ensures
0 < c <= window,
0 < a <= window,
0 < b <= window,
In cases where agents actually proved something, they generally proved something relatively simple and avoided proving properties about the parts that were likely to have a bug (for example, agents often failed to reverse the bitstream order for encode and decode and would write tests that failed to detect this because the tests were palindromic; perhaps some kind of proof of reversal here might get agents to "think" about this in a different way).
It doesn't seem that agents were getting value out of Verus when just provided with Verus and the Verus docs.
If we look at the result, the aggregate xhigh Verus results are fine (slightly lower correctness than average, but much cheaper). The medium results had average cost and the lowest percentage of correct runs as well as the lowest average number of correct tests. Because agents didn't really get value out of Verus, what they actually did for correctness was mostly just traditional tests (built-in Rust #[test] functions with unit tests). When going from medium to xhigh, agents spend much more effort on traditional testing and only a bit more effort on using Verus, which allowed the xhigh result to be ok.
Looking at the actual tests, for one of the two features which agents using Verus did much worse on (the four stream jump table), Verus agents wrote a test for this in 89 out of 160 cases, coincidentally the exact same number as Default agents, but Verus agents were much more likely to write bad tests. They were more likely to encode incorrect results in the tests as well as make easy to pass tests that don't cover the space well, such as making all four streams identical. This kind of thing is what I meant when I said that the failures were idiosyncratic. There's nothing about Verus that necessarily makes one write poor tests when not using Verus and we wouldn't, in general, expect a human who's used Verus to write bad unit tests, in the same way that we wouldn't expect a human using Clojure to make more byte conversion mistakes, but this happened here for whatever reason (possibly a coincidence).
I don't know if folks inside AI labs can get access to better information on why things happened, but here on the outside it's generally quite difficult to tell why something like this happened (even when we formed a plausible hypothesis for the language issue, it required running many samples of many languages, and papers we looked at which studied the same thing didn't observe the language popularity / agentic effectiveness correlation because they either looked at too few languages to be able to reason about such a weak correlation or they looked at problems that were too small and too trivial).
Alloy is often called a bounded model checker. This is maybe not quite right with Alloy 6 since that introduces some extra features, but this is way outside of my area of expertise. My understanding is that, with Alloy, you normally prove properties about your model (as opposed to proving that your code works).
Alloy got the 2nd worst correctness score and, unusually, scored generally poorly on both medium and xhigh. Although it isn't shown (because it doesn't seem to add anything), in general, results were highly correlated between max and xhigh, which were quite different from medium results.
As we saw with Verus, agents using Alloy pretty much relied on standard Rust #[test] for correctness and mostly faffed about with Alloy. Once again, using a formal tool poorly did not help with correctness.
There were individual cases of Alloy use that were close to finding an issue or risk, but even then, only a small number. In one case, Alloy found a counterexample which then caused the agent to implement the Rust version with a mitigation for the potential bug. Unfortunately, the counterexample relied on an 8-bit overflow that couldn't happen in practice because the actual implementation used 64-bit usize with no possibility of overflow given the inputs, so it just made the code more complex without preventing an actual bug.
In another case, the Alloy specification was incorrect and a related test failed. After the test failed, the agent fixed the Alloy specification. Had the specification been correct, perhaps the agent would've written the correct code without the failure. There were some cases where it's possible the good version of this happened, but it's not clear if an actual potential bug was prevented.
Alloy agents did model things that were more closely related to the Zstd algorithm than Verus agents (which mostly checked things like arithmetic), but it was still the wrong modeling.
Differential testing is a technique where you give the same inputs to multiple implementations and then compare results to find issues. In principle, this seems like a reasonable thing to try with LLMs as we often get different results from different rolls of the dice, and as we noted here, having an agent iterate more on an implementation (which might be only part of the entire thing, perhaps even only part of a function) often works worse than having the agent restart from scratch.
But this gave us the third worst results. In this case, we had slightly above average results on xhigh and far below average results on medium. None of the agents created two full implementations to compare. Out of 160 runs, 135 did something you might call differential testing, but like the other conditions we've seen, these were generally trivial and effectively useless. And, in the cases where differential testing might've caught a bug, instead of implementing things in independent ways, agents just did the same thing twice and encoded the same bug in both versions.
I sometimes tell agents to do things independently and get them to launch with separate contexts, but this was not done effectively for differential and agents would generally just write the same thing twice.
It makes sense to discuss how the official Hegel skill changes Hegel behavior, but in reverse correctness order, Hegel Skill appears above Hegel because the result was worse on correctness. See the Hegel section below for discussion of this skill.
Lean 4 can maybe be described as an interactive theorem prover.
Although I didn't pre-register a guess about Lean, if I had pre-registered guesses on which formal tools would do well, I would've put Lean on the list of things I'd expect to do well because it's relatively hot/trendy and therefore seems relatively likely to have good performance due to synthetic data from RL envs.
The Lean agents did prove properties, like the Verus condition, agents mostly did arithmetic proofs that didn't hit the bug-prone or risk surface areas.
Like the other formal conditions, Lean agents relied heavily on standard Rust tests. As with the formal conditions so far, doing a few proofs of things that don't matter didn't help with correctness.
QuickCheck is a property-based testing library, probably the best known such library for a long time, although Hypothesis might currently hold that crown.
Unfortunately, agents were about as effective at using property-based testing as they were at using the formal tools we've seen so far. When using QuickCheck, agents mostly wrote very simple "smoke tests" that didn't check much. They also used random inputs, which, when fully randomized, are pretty poor for testing something like Zstd (because they just go down one of a few failure/rejection code paths).
Also, relatively few properties were checked. Although all agents used QuickCheck, 63 out of the 160 runs only checked a single property. Agents once again mostly relied on traditional testing, although they technically did use QuickCheck. For whatever reason, agents actually wrote more traditional tests than under the Default condition or most other conditions, but did fewer test-fix iterations (which resulted in this condition coming in with below average cost).
TDD underperformed here as well as in the IMAP RFC eval.
The TDD prompt seemed to cause large changes to agent behavior. Agents produced twice as many tests, and worked in a much more iterative test-code-test-code-etc. workflow, although a TDD advocate would probably say that agents didn't actually use TDD. There were only a few instances of agents doing some kind of fine-grained iterative TDD.
Overall, agents wrote more tests up front; for example, agents had one or more failing tests in 67 of 160 cases before doing substantial (non-stub) implementation, vs. 0 of 160 for the Default condition.
For broad test classes, TDD had more tests of every kind. There were more small, trivial tests and there were also more integration and end-to-end tests. Any kind of obvious high-level "agents did too much or too little of X" doesn't seem to fit the data. If we look at specific failures and how they were missed by tests, we can observe that the TDD condition had a number of these. For example, Zstd uses something called a jump table when there are four Huffman streams.
TDD agents were more likely to fail the eval test for this although they wrote more tests that cover the general case. For whatever reason, TDD agents were more likely to write tests that don't cover hard cases (e.g., making all four streams identical and then also making them trivial, like we saw with Verus). This is another case where I'd be curious what kind of visibility people at AI labs have since it's not obvious from the outside why priming agents with TDD made them write worse tests and worse implementations.
If we only had TDD and a few test conditions to go on, a hypothesis might be that TDD'd code often seems to have a lot of small tests that aren't very good, so maybe priming agents with TDD causes them to write more of these sorts of ineffective tests. But it's not clear why we should see the same pattern with Verus. Maybe we could tell whether or not this is true for TDD if there's a shared reason for the Verus (or other) behavior by re-running the experiment on an open model and inspecting what's actually going on inside the model at some level?
Two of the skills also caused agents to run in a more iterative approach, perhaps on the theory that executing more frequently would give better results, and both of those skills also underperformed. In general, across all conditions, agents were able to get the tests they wrote to pass on xhigh and max (not shown, but max had slightly better correctness than xhigh at substantially better cost). Getting their own tests to pass more iteratively tended to get agents to write more incorrect tests that would enforce incorrect behavior.
Yossi Kreinin had this thought for why TDD might result in worse tests:
fwiw, I think if you write the tests before the code, it's harder to test the harder cases since you know less about what is going to be hard, and even if you do random testing which I don't think "tdd" is associated with, you are less likely to steer the distribution in the direction where the bugs are. if you wrote the code or at least can look at it, you know what seems trivially correct and what might or might not work since it's not easy to understand what it does. in other words, tdd steers you towards black box testing which for complicated machinery seems to me to be less effective than white box testing; pretty sure this is how it works with people, less sure about agents
Was my guess that TDD would underperform correct? Strictly on the result, the answer is yes. On my reasoning (not explicitly pre-registered in writing, but I do know what I was thinking), I think it's not clear. My thinking was something like, as we've recently discussed in a variety of contexts, getting agents to actually do something like the right thing and not just overfit is a key part of achieving good performance or correctness with agents. Speaking to the methodology in general and not how this instruction changed agent behavior, TDD seems primed to cause overfitting.
Agents did write worse tests and sometimes used a relatively expensive and ineffective iterative workflow, but I don't know that the failure mode I'd expect from a human using TDD and then directing agents to implement was the real problem here, and that problem was where my intuition came from. I would rate the reasoning here as perhaps and perhaps not in the right vicinity; I think more evals and investigation would be necessary to decide this and I would guess that the result of additional data would be that my original reasoning is wrong.
Spin is a model checker.
Now we're getting into the range where results weren't far from average. Spin did moderately worse than average on both medium and xhigh, at below average cost. As we saw with the other formal tools, usage of Spin was generally ineffective. In this case specifically, using Spin to model a certain class of behavior had no correlation to passing or failing the hidden tests covering that behavior. Usage of Spin was superficial and not productive.
Hegel is a property-based testing library based on Hypothesis.
As we might expect by now, agents didn't use Hegel effectively. To the extent they used it, they used it superficially, and they generally used it after heavily relying on ordinary testing. Since just saying that agents didn't really meaningfully do the thing is repetitive, I'll make these sections short and only highlight particular curiosities.
The actual workflow agents used was generally
As noted above, the Hegel skill didn't improve correctness. Correctness was worse (though it was close enough that this could've been random). What was more striking was that cost was much higher (26% higher on medium and 41% on xhigh), for reasons which seem causal.
The skill caused agents to generate more tests. The additional tests were mostly checks that malformed inputs don't cause a panic and round-trip tests. The former is something that agents were already inclined to do an excessive amount of for all of the property-based and fuzzing conditions, so additional effort there wasn't useful. The latter doesn't seem like an inherently bad idea (I even often explicitly instruct agents to create round-trip tests and they seem to be useful to check specific properties), but it wasn't done in any of the most bug-prone areas. Without additional instruction, agents were inclined to create round-trip tests for relatively trivial properties that were already likely to be correct.
As for the cost, there are multiple reasons for the cost. One is that the skill is fairly large (34k characters for the skill, which also loads a 45k Rust-specific reference, which ends up being more than 20k tokens). This was loaded at the start of the run and was re-read on many subsequent actions. This resulted in an average additional dollar cost of 16% for medium and 18% for xhigh (by raw tokens, the average increase was 900k on medium and 1.8M on xhigh; although the cache hit rate on these was very high, 99.85% after the initial read, they were re-read enough that this was still a substantial fraction of total cost).
A multiplicative cost (this multiplier is included in the previous numbers) is that the skill also specified a structured set of operations that cause a lot more work to get done. This work didn't increase correctness, so this increased cost without a concomitant benefit.
One thing to note is that the skill was "only" used in 157 out of 160 cases. As is generally the case when using LLMs, the actions and results are random. If you have a skill available that you think an agent should use for a particular task, it may or may not use it depending on factors that seem opaque to people outside of AI labs.
In this case, only 108 out of 160 runs actually opened the skill to read it. The skill suggests using proptest in Rust, but the skill suggests approval is required to add a dependency and these were all single-turn autonomous runs, so this wasn't done.
As with the other property test cases seen so far, property testing was rudimentary and not done in a helpful way.
Rstest is a fixture-based test library.
Agents effectively didn't use rstest. They technically did use it, but they pretty much just wrote standard unit tests inside rstest and didn't use rstest as intended, defeating the purpose of rstest. While this is arguably true at some high level for techniques seen so far, agents were at least superficially using some of the other techniques (such as writing some low-value property tests with Hegel), but here agents didn't use the thing that makes Rstest Rstest (the analogous behavior for the property-based testing libraries would be if they just wrote non-property-based unit tests with them).
This is referring to the standard Rust built-in test framework that agents used in the Default condition and also very heavily relied on in the other conditions.
Explicitly asking agents to use the built-in test framework resulted in more tests (double normal on medium, 25% more on xhigh), but this didn't result in better correctness. When agents got things wrong, it was often because they didn't test significant behavior or implemented incorrect test behavior. Adding more tests didn't materially increase coverage of risky behaviors or reduce the fraction of runs with tests that encoded incorrect behavior.
Yossi Kreinin added:
i think the fixed input/output style of testing encourages this in machines and humans alike. if you generate inputs you need to then have code that classifies output as correct or incorrect, and while this code itself might be buggy, it at least makes you think about what correct means and how to tell if something is correct more easily than running the code and assuming its output is the right answer. with fixed outputs you are quite likely to just encode the output of the code and convince yourself that it makes sense
Creusot sits in the same space as Verus.
As we've seen with the other formal conditions, Creusot was not used effectively.
Mutation testing involves modifying the code to determine how effective tests are and then adding tests to get good coverage. Although mutation testing is a standard programming term, agents generally didn't actually do mutation testing and instead did normal testing with some small amount of mutating things in a way that isn't really mutation testing, similar to how the TDD instruction modified behavior but didn't get agents to do TDD.
There were a few cases where mutation testing occurred, but only a small amount, and that was rare.
This condition asked agents to adaptively use testing methods as appropriate based on their judgement. Given what we've seen so far, unsurprisingly, agents mostly used standard Rust unit tests. A few agents did some limited fuzzing. Agents had access to other test and formal libraries but didn't use them.
Fuzzing involves randomizing test inputs in some way.
Agents relied heavily on sending random bytes in, which mostly resulted in going down the same code paths (invalid input). Agents also tried sending in random variations of valid inputs, which mostly also just repeatedly exercised input rejection paths.
On the rare occasion that agents generated random structured inputs (10 out of 160 cases), this found real bugs half the time, some of which were non-trivial cases. Using fuzzing a bit effectively in 5 out of 160 cases isn't exactly good, but this was one of the more effective uses of a technique that we've seen so far. This also seems to indicate that agents could be trained to do this better and also that that can be directed to do this better without changes in training. They do, in some sense, know how to do this; they just don't normally actually do it without being pushed into doing it.
Insta is a library for snapshot testing (sometimes called golden testing), where you compare results to a "snapshot" or "golden file" of correct results. Speaking generally, a snapshot is usually some kind of serialized data, e.g., it could be a JSON object of a data structure, a log of CLI output, etc.
As you might expect, snapshot testing was barely used and agents mostly relied on traditional tests. Agents did use Insta, but would often just write normal unit tests in Insta.
Agents were instructed to use an SMT solver, with Z3, cvc5, and Yices installed.
Agents mostly used the SMT solver as a kind of scratchpad to compute things like FSE state ranges, header arithmetic, etc. Even when agents modeled something, they would generally not model the right thing to avoid a common mistake.
For example, there's a computation that should've been byte1 + (byte2 << 8) + 0x7F00. Many agents implemented byte1 + (byte2 << 8) | 0x7F00 instead. Agents used SMT solvers to prove properties relating to this computation, but then still wrote the wrong code, making SMT use seemingly no better than Default (no instructions).
TLA+ is a language and tool for modeling behaviors.
We're into the set of above average results (but still worse than Default) but, as noted above, I wouldn't take the actual ordering too seriously. Though this isn't necessarily significant, TLA+ did score a bit above average on medium and more above average on xhigh.
159/160 agents created some kind of TLA+ model, generally a state-machine model of Zstd. For particular coverage, 30 modeled Huffman/FSE/entropy (areas that often had bugs). As with the other formal cases, TLA+ modeling happened relatively late in the flow (after a lot of standard tests and implementation). Agents sometimes found and fixed errors in the TLA+ model, but I didn't find an instance of a TLA+ issue resulting in an actual change in the Rust code.
Although there was some real looking TLA+ modelling happening, if this improved correctness, it did so in a small way that was difficult to observe. In general, runs that had more sophisticated TLA+ modeling did not have better correctness.
With metamorphic testing, we check that related inputs produce outputs with the expected relationship. For example, you could check that, for a sort function, changing the order of unequal inputs doesn't change the order of the outputs, or for addition, adding a value to an input adds the value to the output modulo overflow.
As we've seen for the other conditions, Metamorphic testing wasn't done very usefully with respect to correctness. Some actually reasonable properties were checked (e.g., inserting a skippable frame at a frame boundary shouldn't change the output, legal block repartitioning shouldn't change outputs, etc.), but these didn't hit the areas that agents got wrong relatively frequently so checking these properties didn't help. In general, agents seemed to be fans of the old joke:
A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, "this is where the light is".
Curiously, metamorphic testing was used less on xhigh than on medium.
The ECC Rust test skill did ok, but mostly because large parts of the skill were ignored. Agents generally opened and read the skill (153/160 read it) and this seemed to cause them to generate more tests. Not only did agents generate more tests in this condition, if we look at when agents read the skill (earlier vs. lateer vs. never), there's an exposure-based gradient in how many tests were added.
Although ECC scored almost as well as Default, based on how agents did when more exposed to the skill, I would guess that this is random. The earlier an agent looked at the skill, the more its behavior was impacted and the worse the correctness result.
ECC seemed to do ok in terms of raw score because the 7 agents that didn't read the skill did unusually well and got a 100% correct result, and then the 9 agents that looked at ECC late and were only barely influenced also did well and had 100% correctness. This also explains the unusual ECC result that medium had the same score as xhigh (all but one of these runs where agents didn't really look at the skill happened on medium). While it's true that there may be some kind bias in when the skill gets invoked or not, the overall pattern would indicate that ECC is not effective unless you think ECC acts as a good luck charm that improves results, but only when the skill isn't really used, which is more likely to happen at lower effort levels.
Of course, agents shouldn't be influenced by a skill they didn't look at and we should score this based on the cases where the skill was used. If we look at the cases where the skill actually influenced agents, ECC scores below average (between Rust built-in framework and Creusot), with a very similar failure mode to Rust built-in framework of having a large number of small and not meaningful tests. The skill tells agents to use red-green TDD. The agent behavior probably isn't what a TDD practitioner would call TDD, but agents do write a small test before implementing functionality, which results in a large number of tests. As noted above, this isn't an effective way for agents to develop, so the result is worse than no instruction and no skill.
BTW, as we noted when we tried out Caveman mode, there's quite a bit of variance and people are often misled into thinking a skill is useful by a few small runs. In this case, we tried 160 runs of a skill, a fairly large number, more than any reasonable person would do. And yet, superficially, if we just look at the score, ECC seems ok.
We would need a much larger number of runs to average out the noise inherent when using an LLM. We can do what we did here, and inspect the results and use our human brains a little bit, but I rarely see this done when people are talking about public LLM benchmarks, whether it's for skills or anything else (I did try having LLMs analyze the results but, as usual, even with current public SOTA models, the analysis was poor and full of basic reasoning errors). Instead, I mostly see people pass around the top-line number, even when it's not meaningful for boring statistical reasons or, worse yet, the benchmark is fatally flawed, as we saw with Senior SWE-Bench.
Default gave the agent no test or verification instructions.
Given what we've seen so far, it's not surprising that Default scored above average. Agents generally did things that were not useful when asked to use particular libraries or use particular test techniques. It stands to reason that not telling agents to do things that will make them do useless work does better than telling them to do things that will make them do useless work.
Audit asked agents to audit the code after implementation. 152/160 agents actually did this and 151 agents claimed find an issue and then made a change as a result of the audit. Agents generally picked reasonable areas to audit, but usually didn't do an independent audit with a fresh context (which I will often ask agents to do) and often just made the same mistake in the audit that they had already made.
42 used an independent agent, but these runs actually scored worse (it's possible this isn't causal and agents decided to spin off an independent audit because they were in a worse or harder situation). Audit ended up with the best correctness on xhigh, but below average correctness on medium, and all of this auditing substantially increased cost, especially on xhigh. On average, Audit did about as well as Default and it's not clear if it's really better on xhigh and worse on medium. That would be plausible, but I don't think we have enough evidence to tell.
Em Chu had the following comment:
The results here are consistent with my experience. Auditing code is where most of my tokens go at the moment because I find it quite useful. I always give the two instructions though:
- Don't spawn subagents; read and understand the code/diff yourself
- Don't execute any of the code
because I find the LLM to be significantly dumber if you let it do either of those (though of course I haven't measured...). It really doesn't read or reason about code by default, even if I'm never making a change big enough to exceed its context window.
I also usually include some BS like "be adversarial" "consider all possible combinations of features" "consider the entire input space" but I'm less sure that helps at all.
It would be interesting to try that, but as I've noted in my recent posts, I'm trying to go into less detail in posts, so maybe that will be a topic for another post.
For Zstd, when this instruction was followed, it caused agents to focus heavily on FSE, Huffman, bit readers, and state. These were areas where, in general, agents often missed issues, so agents were correct to think that these areas were risky. The areas that were targeted for fuzzing were better choices than the plain Fuzzing condition.
On medium effort, agents mostly ignored the instruction and didn't do it, but they did follow instructions on xhigh. While this condition didn't perform poorly, it didn't seem to do better than no instructions.
When looking at what agents actually did, one issue was that agents often just generated a bunch of random inputs which were generally invalid and wouldn't test any interesting condition.
When a human tester generates randomized tests, they'll generally try to target the randomization in a way that generates "interesting" inputs and agents failed to do that. Agents also didn't check outputs very effectively and, in many cases, only looked for crashes. Fuzzing is often associated with only checking for crashes and not checking for properties, so this is maybe not too surprising, but it's probably not what a human would want if they were testing a Zstd implementation.
Although this technically scored higher than Default, behavior didn't seem to be meaningfully different and the scores are quite close; I would guess that this is due to random variation. At every level at which I looked at the results, they were indistinguishable from random draws of Default.
Kani is a Rust model checking library.
In terms of "actually using a formal method on the code that will execute", Kani had the best coverage in that Kani was actually used on the Zstd code. However, that only happened occasionally and most use was superficial.
There was one case where real Kani use caught a non-trivial bug that caused Rust code to change. 1 out of 160 isn't amazing, but it does indicate that agents can stumble into using Kani reasonably sometimes (which, I would guess, means that, if used in an RL env, models could learn how to use Kani more effectively).
Kani had noticeably higher cost than other conditions. This seemed to be because reading Kani output repeatedly was expensive, which resulted in a high input token cost.
ACL2 is a theorem prover. One thing to note about the result here is that, in many cases, ACL2 OOM'd (192 GiB limit). OOM results weren't counted, which biases the results in some opaque way.
Although ACL2 scored higher than Default, I think it would be surprising if this was causal and significant. As we saw with almost all of the other formal methods, ACL2 was mostly used to prove things that didn't significantly impact correctness, so it's not clear why this would improve correctness.
With many different conditions, we wouldn't expect Default or the seemingly equivalent Make no mistakes to be at the top unless other conditions had severely degraded performance.
Proptest is a property-based testing library.
Just as we saw with the other randomized testing, most tests weren't very interesting, and a too-heavy reliance on randomness caused poor coverage.
Despite generally poor use of property-based testing, proptest's shrinking (finding a simpler input that causes a test failure) did sometimes provide some value, which is better than the little to no value we saw in most other cases.
As with the other technique-based approaches, agents had a container with all options installed. Every agent chose to use proptest, so this effectively became a 2nd proptest condition.
As with the proptest condition, tests were mostly not very good but they did sometimes find bugs and shrinking seemed to generate some wins.
I find it mildly interesting that this second "accidental" proptest arm also scored well above average, just like proptest.
Here, Skill refers to the skill I wrote to test having a simple skill (as opposed to the large/complex skills that were what I found when I asked an agent to find relevant testing skills).
Maybe I should use skills, but I generally don't and instead rely on prompting and seeing what happened and then prompting some more. As a result, I have no intuition for what makes a good skill since I don't have any practice at it, but Max Bittker suggested that it would be interesting to see the result with a test skill that attempts to encode some information I have in my head about testing. On seeing the result of this, he had an "I told you so" reaction.
I didn't write a pre-registered guess about this, but the guess in my head was that this wouldn't work well. From my attempt at conveying this to humans in 2015, which I would say pretty much failed, I don't think I'm good at explicitly laying out how someone should test in writing. I've sat down with people and showed them what to do, which has generally converted them for life and turned them into way above average bug finders, but being able to convey something by showing someone is a different (and easier) skill than conveying it by writing down how to do it.
In this case, the skill was:
This got the highest score, but didn't work as intended. It didn't really do the fresh context thing almost ever, so it was pointless to have that in there and we don't know if that's something that's effective that needs to be refined to force agents to do it more frequently or if it's something that should be removed (while it's technically possible it's happening at the optimal frequency, I highly doubt it).
We noted in "Audit and fuzz risky" that agents seemed to know how to identify risky areas. This was true here as well, but this didn't necessarily mean that agents did the right thing. For example, agents identified bitstreams being reversed for encoding vs. decoding in Zstd as being risky, but agents didn't do better on tests that exercised this. If we look at specific examples, for medium run #35, an agent identified this as risky, did independent derivations and an audit, but still failed. It had a relevant test, but the input was palindromic, so reversing the order gave the same result, allowing for a failing implementation that had this backwards.
Another issue, if we can call it that, is that all of the fuzzing / property-based testing was done "by hand". Given that agents seem ok at using proptest and that proptest has some useful machinery to lean on, this skill could probably trivially be improved by instructing agents to use proptest. The instructions to agents that were intended to minimize the standard failure mode of generating many useless "too random" tests directionally worked and a larger fraction of agents generated somewhat meaningful tests, but the tests were still worse than I'd expect a human to write (or an agent with active human guidance). Without iterating on this, I'm not sure what generic guidance would be good (as opposed to spending a few minutes looking at the structure of Zstd and giving Zstd-specific guidance, which is one kind of thing that's worked well for me on other problems).
As a first draft for a skill to iterate on, I don't think this is horrible, but I don't think it's really ready to use either. I could see an improved version of this working if it were tried with many more examples to make sure there isn't overfitting to RFC-like problems, bit-manipulation-intensive problems, etc., but, since I don't normally make skills and haven't ever tried to iterate on one, this fails to capture what I or another human would do if really driving an agent.
Since I'm used to prompting and then looking at the result (not necessarily the code, but at least what agents say they did and some kind of agentic summary of what happened, and parts of actual results for some kinds of experimental work) and then re-prompting based on that, I'm not used to front-loading information, which is a fairly different problem than reacting to information. From previous fuzzing work, I've seen failure modes that agents often fall into and the skill was intended to prevent those failure modes, but it's easier to do this if you check back in even occasionally than to do it fully up front, and the up front instructions weren't sufficient to stop the standard failure modes, though they did mitigate them somewhat.
As we noted above, I didn't try to break down the data in a nice, easy to look at way. I didn't do that because, once we look at what agents actually did, it seems like they were mostly pretty ineffective and I don't think it's particularly interesting to see how well "agents using Verus badly" do compared to "agents using QuickCheck badly". One thing that I find a bit interesting is that, when asked to identify areas that are risky or prone to subtle bugs, agents were able to do that.
But, in general, regardless of the library or technique suggested, agents failed to use the technique. As previously discussed, just asking agents to "test" or repeatedly asking them to test more results in poor testing. It turns out that asking them to use test techniques (some of which I've personally found to be highly effective) also results in poor testing. The quick and dirty skill I wrote seems like it could improve things a bit, but would need more than the 2 minutes I spent on it to be actually useful. Yossi Kreinin made the comment that the state of software testing is atrocious, therefore we should expect poor results if agents fall back to their training, so to speak, which is what we saw.
For whatever reason, agents seemed to be somewhat better at using proptest, although the level of testing was well below what I'd expect out of a reasonable human who's read the proptest manual and is given some direction on how to test. I'd be curious if agents that are given more direction are more effective with proptest than with other libraries, but that's a topic for another post as I've been trying to get posts out in half an hour and we're approaching 9000 words here, which is beyond a reasonable amount to try to type in half an hour.
My experience has been, if you guide agents to set up a reasonable test and triage structure, getting agents to add to that effectively without a huge amount of supervision works ok-ish. Because of my background (bias), the kind of testing I tend to lean on is some form of randomized testing / fuzzing / property-based testing.
I talked to Jamie Brandon about this, and he's found the same with snapshot testing. He mentioned that, on one project, when he asked agents (using a variety of models) to do snapshot testing, they would say that they were doing it and then just wouldn't do it (they would write a unit test and then say they wrote a snapshot test). On a different project, he was able to get them to write reasonable end-to-end tests with mocked IO, but only after moving the tests into a separate crate and putting instructions in AGENTS.md to keep tests in the crate and not modify the public interface.
At least to date, I've been leaning more heavily on getting agents to write the test code than Jamie (my tendency has been to type to agents in a CLI; at least for now, he favors writing code by hand a lot more than I do), but it doesn't seem to matter how you do it as long as you set up some kind of reasonable structure.
Similar to this earlier problem we looked at, it seems like doing anything remotely reasonable works. If you "talk to" an agent and give it light instructions like we did for the Zstd or IMAP evals, the agent will do poor work. But if you look at what it does and type a few more sentences, you can often get it to a good place pretty quickly (or that's what my experience has been on other problems, anyway). Without having ever attempted to understand what makes models work, my made up and probably completely wrong guess for why it seems like doing something remotely reasonable generally works is that, somewhere inside the model is some kind of understanding how to do this stuff. It's not the default, and it's not even near enough to the default that naming what technique to use works, but if the model is sufficiently primed, the knowledge for how to do this stuff will actually get put into practice.
I'd be curious if this can be effectively packaged up into skills, or if AI labs are going to start training models to get better at testing or formal methods, or if whatever they're doing that doesn't directly improve those things will still improve those indirectly enough that agents will write decent tests without much supervision or structure.
Guesses I didn't register, but I could tell I held implicitly because I was surprised when I saw the result:
At various times, I've felt like I have a bad/antiquated/ineffective workflow because I hear people are doing something and I've been too lazy to try it out. I've felt this way about skills for a while since I don't really use skills. Instead, I keep a large scratchpad of things that I sometimes copy+paste in as prompts, which sort of feels like the equivalent of commenting out blocks of code to save them instead of using version control.
But then I saw this talk by Thorsten Ball, where he mentions he doesn't rely heavily on skills, and I talked to a couple people who seem relatively effective with LLMs who also don't really use skills, and it made me wonder if I'm not missing out on much.
Then I tried this experiment, where my feeling was that the skills I looked at weren't going to help and are probably actually going to hurt, with low confidence since I don't know anything about skills. The skills did pretty much what I thought they would do, so it turns out the intuition I have from just seeing how agents respond to things and running a bunch of little experiments seems to hold up ok for skills. I also looked at a number of other skills that allegedly improve testing which I didn't include in this experiment that looked like they would have the same failure modes as the skills we tested.
I also ran two experiments (details not discussed here, perhaps in another 10k word post another time) on some other skills that are "official" skills that companies have to support their product. One is from a big AI lab and the other is from a "small" few billion dollar company, but in both cases, the skills made results worse, just like we saw here. Funnily enough, after these experiments, I'm actually more bullish on skills for personal use than I was before since the failure modes seem predictable and therefore fixable without a huge amount of costly experimentation. Creating a publicly released skill that's intended to be really good, work well across different models and harnesses, etc., seems like it might be hard (claude and codex seem to "want" different styles of prompting, so of course that should be true for skills as well), but just addressing the issues that cause a lot of skills to be worse than no skill for personal use seems quite doable?
I don't know enough about skills to say how to write a good skill, but with all the skills we looked at in this post (except for the one I wrote in a minute or two) and the skills from these other two experiments, the skills seemed written like they're human tutorial instructions, in that the goal of the skill seems to be to explain how to do something. My naive thought as someone who's written all of one skill is, I'd guess that this isn't optimal when working with a model that should already have some knowledge of the topic (which was the case here and in the other experiments as well). The model is already going to have some kind of default behavior distribution, so I feel like the more natural thing to do is to give statements that will modify that behavior, not write instructions that would allow a human or non-knowledgeable agent to do the behavior at all.
One obvious problem is that we get different default behaviors from different harnesses, models, and effort levels, but throwing a bunch of text into a prompt or a skill doesn't actually change this; that's just a longer way to push the agent away from its default, with a lot of text that may do some kind of unintentonal pushing. As we saw here, much of the text just gets ignored (and what gets ignored and when is of course harness, model, and effort dependent). For example, for the ECC skill, even when it was read, had most instructions ignored, and although the TDD instructions were influential (which made results worse), the instructions to do TDD were still not really followed despite them being laid out clearly.
Since what kind of prompting is effective changes enough between model releases and effort levels, for something general like "testing code well", it's not clear to me how these skills are supposed to work across so many models and efforts. Just going from GPT-5.5 to GPT-5.6 changed how I worked substantially because a number of things that worked fairly reliably with GPT-5.5 either stopped working or became much less reliable (even though, overall, the level of capability seems higher). In the same way that I don't prompt GPT-5.6 the same way I prompted GPT-5.5, I don't think I'd want to use the same skills.
I'm not sure who, other than someone at an AI lab, would actually go through the trouble of running evals on skills to see what's effective for each model and effort level and then create a portfolio of skills that are differentiated by model and effort, and I wouldn't expect AI labs to have skills that are optimized for their competitors' harnesses and models, so I don't know about things like generic "testing" skills (as noted above, quite a few testing skills that I looked at but didn't test here looked like they would have the exact same failure modes as the pre-existing skills we tested), but I could see having a few skills that work for my own use cases with the specific harness/model/efforts that I tend to reach for.
Just thinking about testing, while there are particular pitfalls that certain models fall into at certain effort levels that I want to nudge them away from, there isn't really a generic test workflow that I want to give agents that's independent of the thing being tested and the level of quality I want from the thing and the dimensions in which I want quality, so I don't think I'd want a generic test skill that lays out a set of testing steps that agents should, in general, do. Something like this goes for a lot of task that I do, which I want done in task-specific way and not a generic way. I could imagine some kind of skill that asks me questions and then emits the correct instructions to agents, but given how fast models are improving, if I'm making something for personal use, I don't think it makes sense to spend time tweaking a skill like that until it's useful. If I was working on an agentic product and wanted more people to use it, that might be a different story, but the skills I've tried have had the same failure modes as the skills we tested here, so it seems fairly easy to make a skill that turns out to not be that effective.
I could see skills being generically useful for things like teaching agents how to execute workflows or how to interact with APIs/interfaces, such as Sawyer Hood's skill that helps agents drive a web browser. Since I haven't tried that skill, I'm not endorsing it, but from reading through it, it seems like the kind of thing that would work well and save me a lot of hassle when I'm trying to get an agent to drive a web browser. However, if you read the actual skill (and scripts), it has a very different style than the testing skills we tried here.
Thanks to Max Bittker, Yossi Kreinin, Em Chu, Dennis Snell, @panoramic.blue, and Jamie Brandon for comments/corrections/discussion.
P.S. I've had this note on my last handful of posts indicating that I'm trying an experiment where I write up half-baked (barely fixed/audited/cleaned up) results as quickly as possible because agents let you run experiments so quickly that I otherwise wouldn't write anything up at all. I actually ran this experiment immediately after I ran the programming language token cost / correctness experiment, but I haven't had time to write this up because I wanted to write up this creation of a regex engine with an interpreter and a native code compiler, this experiment with running a forked version of ripgrep that uses the native code compiler on codex's ripgrep queries, and a few other things I haven't had time to write up; I've had a goal to do each of these write-ups in half an hour, but I'm still falling pretty far behind in terms of experiments I've run vs. what I've written up.
For a couple years, I was running experiments like this and just telling a few friends about curious results and then moving on without really ever talking about these things publicly. If you have opinions on these quicker (and lower quality) experiments and write-ups, let me know what you think (X Bsky Mastodon)!
Regrettably, @danluu is right. The Hegel skill sortof sucks right now.
I think the common problem with a lot of agent skills is that agents suck at writing agent skills and also everyone (including us) uses an agent to write their skills.
A mutual friend mentioned that MacIver (later?) set up a benchmark and confirmed the deficiency in the Hegel Skill and is presumebly working on either improving the skill or tweaking the Hegel comments/docs so the skill isn't necessary. If the only thing that comes out of this post is that Hegel gets an improved skill, I think that would already be pretty awesome. As previously discussed, I think measuring and benchmarking are underrated, in large part because just publishing a measurement can often point a problem people didn't know existed and motivate some changes. I've had more than a few posts that have driven some kind of change just by showing where there's a gap. It's nice that this is another one of those posts.
After asking an agent to do a simple lookup of something for an analysis, it exec'd a perl process that ran for 2 hours and 20 minutes before I killed it (I really need to have something that automatically catches things like this, because it's fairly common).
A subagent used perl to do a regex search over a relatively small file (44kB, 1364 lines), but the expression was degenerate with PCRE and did a combinatorially large amount of work. I tried re-running this with the FRE regex engine we tried building in a few minutes here and it finished matching in 0.7s (Rust regex was 0.6s).
You never know what's going to happen when agents are off doing things, but there are at least was this could've been prevented. First, the agent could've avoided invoking a regex engine that can give you a combinatorial explosion like this; there doesn't appear to be a strong reason not to use a safer regex engine for this (such as ripgrep at default settings). Second, the agent could've issued the regex it was trying to emit; the actual regex it did emit returns a uselessly large capture and doesn't do what's intended when it succeeds. Third, the subagent (or the harness) could automatically kill things like this after the subagent finishes. Of course this shouldn't happen for everything a subagent runs, but agents often leave runaway processes like this lying around (in this case, it appears the subagent that launched this stopped after 30s but left this process running indefintely even though a search like this is intended to returned a result that's used by the subagent, which has no value if the subagent has finished).
I actually have a process that goes around cleaning up after things agents leave lying around (agents that, themselves, leak memory, temporary build artifacts that consume space, etc.), but it wasn't looking for runaway perl processes. That's another one to add, but surely I'm not the only person who's run into this problem. I guess I could open source my silly tool for this, but it should become obsolete once the major harnesses fix this, so there doesn't seem to be any good reason for anyone to even pick up my thing in the first place were I to open source it since the AI labs are probably going to fix this kind of thing in not too long.
In the interest of writing this quickly, I'm going to punt on this (sorry!). The distribution across conditions wasn't fundamentally different than we saw here when we looked at how language impacts correctness and token cost. Somehow, this post I wanted to write up quickly in half an hour is almost 10k words, which is definitely more than half an hour of writing (10k words in half an hour would be over 300 words per minute).
One thing I'll note is, just like with this post on programming languages, there were a couple of things that looked like really interesting/compelling results (at least from the standpoint of just looking at the top-level graph and seeing if anything stands out), but on looking more closely at those, they were due to an experimental error caused by giving a short prompt to agents to set up the experiment. On fixing those errors, we got a much more boring negative result, except for the part where the skills codex suggested might be useful seemed to be counterproductive.
Overall, I think the IMAP results were less interesting. I thought it might be more interesting to try since it's more of a "business logic" problem, but not as huge and expensive to run as the pandoc eval we look at when we looked at languages, which costs over $1k in API costs per run for some of the less effective languages, and was still $700/run in Rust to get to only 30% passing tests (note that this eval uses a much harder metric, of fraction of runs that achieve a perfect score). In the IMAP eval, only a single run achieved a perfect score ("Make no mistakes", on medium). In general, there were some bits of logic that were, apparently, too tricky for 5.6 Sol to one-shot. I suppose one way to look at it would be that Make no mistakes dominated with a 2.5% score on medium vs 0% for all other conditions. Finally, evidence that "Make no mistakes" works!
If we look at the actual results, as before, all methods weren't very effective. Even things where you might expect a good fit. For example, you might think TLA+ woudl do well because it's a natural fit for IMAP state and concurrency; in prinicple, it should be good at modeling a lot of the higher-level parts of the protcol. However, as we saw with the Zstd results, agents "decided to" do most of the implemention in Rust, then whip out whatever tool they had been told to use in the initial prompt (5 out of 80 agents actually used TLA+ before coding, but 75/80 did it the other way around). For whatever reason, agents almost always decided to use TLA+ to model a small mailbox mutation. None used TLA+ to model things like multple observors , event queues, UIDVALIDITY and mailbox epohcs, etc., where agents made many mistakes in implementation. Instead, agents modeled things like CONDSTORE and QRESYNC, where the Default condition had a more than 99.6% test pass rate, and the TLA+ agents actually had a lower pass rate despite modeling these things.
One thing that's true of all the evals I tried for this is that they're RFCs, which are highly unrealistic, but the way in which they're highly unrealistic is that the specs are much clearer, more detailed, and less ambiguous than the specs virtually all programmers give to agents when they ask an agent to implement something. I would expect that the failure modes we've seen here are the same or worse on most real-world problems.
[return]Funnily enough, when I asked ChatGPT to fact check this, it told me that this paragraph was wrong because there are papers that show that people have used RL environments to train agents to test, and then linked to three papers that trained agents to write poor tests by training them to write unit tests like most programmers do. That's exactly the kind of thing that I would expect to lead to the kind of poor testing we see LLMs do today, where it takes a human who understands more effective test techniques to steer the agent. In multiple independent subfields where people care about correctness, folks have independently converged to a few sets of related techniques that are generally the opposite of writing small unit tests. Of course training agents to do this thing that's the opposite of what people do when they're serious about correctness isn't likely to result in good correctness.
There is some work related to RL environments and randomized testing, such as this paper, but based on how ineffective models are at any of {PBT, fuzzing, randomized testing, etc.} without specific guidance, it doesn't seem that this has made it into the training of the models from the big AI labs in a serious way.
[return]Maybe people's stated reasons for their laguage supremacy will become true after models get much better, but there doesn't seem to be any reason to think that should be the case. If I had to guess, I'd guess it will look more like poker, where people had all sorts thoughts about what's most effective that were invalidated once simulation became powerful enough that computers could outplay humans in many situations. Even "modern" concepts like "range advantage", that people often say is consistent with optimal solver play, actually don't fall out of solver data at all once you look closely. They're just nice cocktail party concepts that are easy to understand and sound compelling.
It probably still makes sense to use concepts like that if we're talking about humans learning how to play poker against humans because, for casual play, humans are generally not going to want to study solver lines enough to understand what's near-optimal, but if we're talking about what coding agents are good and bad at, I don't see a reason to throw around cocktail party ideas when we can run experiments to see what works and the expeirments that have been run to date don't support most of these abstract reasons X is good or bad that are bandied around.
[return]2026-09-01 08:00:00
I was curious how well the predictions of the most widely cited AI skeptic I've seen (Ed Zitron) have done, so I looked at how his predictions panned out. To disclose my own biases, I've never had a particularly strong pro or anti AI progress position. For example, in 2022, I did a comprehensive look at predictions Futurists made, including well-respected folks like Kurzweil and found them to be generally wrong on both the prediction results as well as the reasoning. On the flip side, in 2015, I wrote about how people were underestimating AI's ability to displace humans in jobs and have repeatedly been on the record as saying that many people are underestimating AI's ability to displace humans from jobs. My position on AI has been extremely boring and is basically, "if something is currently happening, the people who are saying that it's impossible that it will ever happen are probably wrong".
One comment I've seen from a lot of AI skeptics when someone responds to an AI skeptic is that all of the people who are saying that AI isn't fake are self-interested liars. Personally (to my obvious detriment), I have no particular financial interest in AI companies. I own whatever the standard share of them is via boring index funds. I have some seed stage investments, but just due to the timing and what's gotten big, that part of my portfolio is underweight on AI. I don't work at an AI lab or a company that supplies AI labs. I've mentioned being hilariously bad at interviews before, and I did interview at an AI lab a number of years ago and failed the phone screen in a performance that was the kind of performance that must've inspired Jeff Atwood's famous Why Can’t Programmers... Program? where he concludes that there must be a lot of fake programmers out there because nobody could fail a coding interview that badly if they knew how to program. I don't benefit in any particular way if AI does well, except insofar as anyone who holds broad index funds benefits, but I do care about accuracy.
Because there are quite a few prediction results, let's look at one in detail before the complete list to get an idea of the kind of reasoning Zitron uses. We'll arbitrarily look at this November 2024 talk where Zitron says, among other things, the major tech companies (like Meta and Google) are dying and they're thrashing around on AI because they don't know how to grow.
Zitron specifically named Meta as a company that's dying ("it's a dying product, and it's kind of a dying company"). Meta's revenue and profit (GAAP operating income) have been
| Period | Revenue | Profit | ||
|---|---|---|---|---|
| Amount | % | Amount | % | |
| 2023 | $135B | 16% | $47B | 62% |
| 2024 | $165B | 22% | $69B | 48% |
| 2025 | $201B | 22% | $83B | 20% |
| First half 2026 | $117B | 30% | $42B | 10% |
When he talked about companies not knowing how to grow ("none of these companies anymore really know how to grow ... in the desperation to try to reignite growth in a dying ecosystem the tech industry is going to shove this [AI] shit into everything"), he named Google and then Microsoft. Alphabet (Google's parent company) has had the following revenue and profit numbers:
| Period | Revenue | Profit | ||
|---|---|---|---|---|
| Amount | % | Amount | % | |
| 2023 | $307B | 9% | $84B | 13% |
| 2024 | $350B | 14% | $112B | 33% |
| 2025 | $403B | 15% | $129B | 15% |
| First half 2026 | $230B | 23% | $80B | 30% |
And Microsoft's numbers have been (note that, for consistency, all numbers are calendar year numbers and not fiscal year numbers):
| Period | Revenue | Profit | ||
|---|---|---|---|---|
| Amount | % | Amount | % | |
| 2023 | $228B | 12% | $101B | 21% |
| 2024 | $262B | 15% | $118B | 17% |
| 2025 | $305B | 17% | $143B | 21% |
| First half 2026 | $173B | 18% | $79B | 19% |
Although this wouldn't be in the spirit of Zitron's statement, one could argue that Meta is actually dying, it just hasn't died yet. However, the reasoning in Zitron's argument is incorrect here—the Meta, Google, and Microsoft ecosystems are not dying. Given how fast these companies are growing (in terms of revenue and profit), it doesn't seem that AI is, as Zitron implied, some kind of desperation move they're reaching for because "they don't know how to grow" and are all out of ideas. I don't think it's worth spending this much text on each prediction, but the pattern Zitron used here is illustrative.
To make the case that these things are dying, he pulls on minor issues that are not positioned to cause the very large changes he suggests are about to occur. For Meta, he cited some kind of alleged MAU drop for Facebook. Rather than use Meta's own MAU figures or any kind of revenue or profit numbers, he seems to have used numbers from Similarweb. My experience with 3rd party tracking numbers like this is that they're quite inaccurate and generally useless for anything other than a rough order of magnitude comparison, making the Zitron's cited decline meaningless. FB stopped reporting MAU publicly in December 2023, but most estimates have FB MAU increasing over time and the numbers Meta does report show generally increasing usage over time for their products; Zitron cherry-picked an outlier low estimate to make his point.
For Google, he cites Prabhakar Raghavan, who he calls truly evil and "a computer scientist class traitor that sided with the management consultancy sect", as having done some kind of grievous damage to Google search. In his rants about Raghavan, he never credibly establishes that Raghavan is doing severe harm to Google search, and the Google search engineers who've commented on his rant don't seem to agree with the Raghavan as sole or even major reason for search issues hypothesis.1
But even if we posit that Zitron is right and the villain Prabhakar Raghavan defeated the hero Ben Gomes, causing some kind of issue for Google search, this still doesn't make the case that Google revenue growth is in trouble at large because they have a number of other major products (such as YouTube and Google Cloud) that could drive growth even if search wasn't growing.
Every significant part of the chain of reasoning here is not only incorrect, it's not plausible if you know anything about Google or big companies in general. I'll be the first person to say that Google search quality has some serious problems and that Google has been increasing the relative priority of revenue over the user experience over time. This was a source of consternation for a number of user-focused engineers at Google when I was there in 2013.
For one of the issues Zitron cites, ads being confusing to users, in 2013, I asked a search engineer about Google changing the background color of ads to look more like search results because there was a previous study that showed that more an ad looked like a search result, the more users got confused over whether a result was an ad or a real search result, and I'd heard that Google deliberately made the ads not look like search results to avoid user confusion. The search engineer said that because some people didn't want users to get confused, it was impossible to make ads nearly identical to search results in a single change because it would be too obvious what's going on.
The way this was going to happen was that every time you A/B test tweaking ads to look a bit closer to search results, you make a lot more money, so the change would happen over multiple years in multiple parts, each small enough that the people who want to fight back against this kind of thing would have a hard time making a case. That happened just as this engineer predicted, but it was going to happen whether or not Raghavan ended up overseeing search. And, of course, that kind of thing happening doesn't cause Google to run out of room to grow and become desperate to reignite growth in a dying ecosystem. Whether or not you think Google should do it, it's something that makes Google more money.
From what I can tell of how people cite Zitron, they cite him as an authority so they can say that this guy who looked at the numbers has made this claim, so their claim is backed up by the numbers. It turns out that if you look at the claims Zitron makes and know anything about the topic, the claims don't make sense, but I don't think that's the point. The point is one can say that someone looked at the numbers. The other point seems to be that this guy is angry2, which is a good way to drive engagement.
But when people bring him up, they're of course not generally citing his anger; they're saying here's this guy who's looked at the numbers and, if you're angry about AI, he's right there with you being angry about AI, and he's got numbers on his side.3 Like I said above, I don't want to go into this level of detail on each claim; this is just an illustrative example about how the claims below look. For any of his posts that I read, while there are numbers thrown around, the numbers don't actually connect to a coherent argument. In many cases, as we saw above, the numbers don't even really support his argument (such as an MAU decline in Facebook causing Meta financial problems which would then cause Meta to spuriously insert AI in places it doesn't belong). I suspect he's relying on people's eyes glazing over when they see numbers and just not thinking about what the numbers mean.
With the predictions below, someone could have the exact same prediction record and have completely reasonable reasons that just didn't pan out. Or someone could be correct in every case and also be wrong because all of their reasons are wrong. Someone like the latter person might have some kind of intuition that they're unable to articulate, or perhaps they're someone who just got lucky. Fortunately for us, we don't have to make this difficult judgement call because Zitron is wrong on the predictions and also wrong on the reasoning.
Since I've been living under a rock for years and am just catching on the AI discourse, I hadn't actually read or watched anything by Zitron or any of the big AI commentators, but on looking up what people who have good judgement say, they also seem to find that Zitron's use of numbers is just sleight of hand, such as this comment by Juho Snellman:
His writing is certainly flamboyant, but the aggression and expletives seem more targeted at hyping up people who already believe the things he writes, not for making people change their minds. He found a niche in anti-tech grift, and is now exploiting the niche for all he can. But you might want to actually fact-check a few of the things he says that convince you, because at least for his written articles basically everything is made up or misrepresented. There's plenty of links to sources, sure, but if you follow them down to the primary source what they're saying is very different from what Zitron is implying
Here's an example where commenters seem to assume that Zitron's analysis is good for some reason, to which Juho Snellman replies: > The key problem is that his economic analysis is absolute trash. I used to think he was just totally incompetent at it, but given the bias in the errors, it is pretty clearly intentional deception. But it's often pretty hard to address that, because every article he writes is a 10k word gish gallop. I've tried debunking key points a few times in HN comments for just one of the intentional mistakes he makes, and people complain about the reply being too long.
For example, when Timothy B. Lee looked at a spreadsheet that Zitron used to create a projection of Anthropic's revenue, he found
He doesn't count February 1-10, counts March 1-10 twice, counts August 21-October 21 as one month instead of two, and doesn't count October 21-November 1. [another commenter notes that his spreadsheet also contains February 30] ... Ed claims he tried to compute Anthropic's revenue for 2025 and came up with $3.6 billion, suggesting some funny business [but the numbers work out once you fix the errors]
After this point, most further predictions that I saw were either non-falsifiable or resolve in the future. Note that I didn't attempt to catalogue statements that are nonsensical or were simply factually incorrect statements at the time, such as his December 2024 claim that “Generative AI's products have effectively been trapped in amber for over a year.” January 2026 claim that "[models are] basically the same as they were a year ago. They have the same efficacy". Zitron has not only made forward-looking statements that AI capabilities will not improve, he's also consistently made backwards-looking statements that capabilities have not improved which, while obviously false at the time, seem to play well to his base (along with his other false statements). If you connect all his statements together, it's implied that AI had the same capabilities in January 2026 as they did in December 2023 (and if you connect later statements, it's actually implied that capabilities in August 2026 are the same as in December 2023, though to be fair to Zitron he frequently contradicts himself and has also admitted to limited improvement in mid 2026).
To be fair, we could say that Zitron is speaking colloquially, so we when he says things like "have effectively been trapped in amber for over a year", that doesn't mean there's actually be no change December 2023, so the statements aren't transitive. Even if you assume a kind of colloquial sloppiness here, the collection of statements still implies that, from December 2023 to August 2026, improvements have been minimal (perhaps except, as noted above, when he contradicts himself and admits there have been limited improvements in some areas).
If we compare to how futurists did in our analysis of futurists, on style, Zitron relies much more heavily on anger than any of the futurists we looked at. On the quality of reasoning, he was probably about average compared to the futurists. Despite being wrong on roughly everything, he's not more unreasonable than someone like Buckminster Fuller, who suggested we'll be able to send people by radio because atoms have frequencies and radio waves have frequencies so it will be possible to pick up all of our frequencies and send them by radio.
In terms of the style of reasoning, of the futurists reviewed, he's probably closest to Kurzweil, in that he uses numbers to give a kind of aura of credibility, but if you know something about the topic he's discussing or look at the numbers, the reasoning falls apart. Zitron's reasoning isn't worse than Kurzweil's, who (for example) continually made new predictions of extremely fast progress that didn't pan out (such as, in 2001, predicting unbounded lifespans by 2011). Continually predicting that AI progress will stop for reasons that are incorrect is just taking the flip side of the bet on progress. Instead of having infinite progress, we're going to have no progress. Every time that prediction is proven wrong, you can just make another similar prediction and then move the date forward a bit (fans of both use the same techniques as well; fans of Zitron simply claim that his predictions are true, just like fans of Kurzweil cite his 86% prediction accuracy even though his actual accuracy on those predictions is 7% if you actually look at the results on the exact predictions he allegedly got 86% right). Michał Zalewski (lcamtuf) has some thoughts on why this happens:
The surest way to build [a] popular following is to articulate positions that are crisp, strong, and leave no room for doubt. You can't get too many podcast or TV appearances out of "well, the market could go either way", "both political parties make good points", "there's some merit but also some hype to AI". Or, to tap into the example in the post, "Harry Potter is an OK book".
In fact, there's a positive feedback loop. If you take a provocative, edgy stance, you get more attention and likes, so you sort of... self-radicalize? At some point, it's no longer an opinion that can be changed. It's an identity, a personal brand.
It's ... why Ed Zitron has a blockbuster blog about how it's all just one big scam. If you take a more nuanced view, you will at best get no reaction, or at worst, you'll invite scorn from both sides.8
For anoyone looking for well-reasoned anti-AI takes, I find whitequark to be quite good (not that I agree, but I think the reasoning is sound and I could see how someone would agree if they have slightly different premises than I do), but of course whitequark doesn't draw the kind of big audience that Zitron does.
I'm curious what people do after being on the wrong side of a set of failed predictions about progress like this. For the futurists, even the ones who were nearly completely wrong (which was every single one reviewed here), they can still make some kind of case like "a quarter of the things I said would happen happened, it just took two to twenty times longer than I expected" and if they're not so stuck on accuracy, they can round this up to "the things I said would happen happened", which is often what they've done. That seems to have served them well as nobody really cares to look at the details anyway, which is how, for example, Kurzweil's alleged 86% prediction accuracy became a well-established fact; no one bothered to actually check which of the cited predictions panned out until we looked at this in 2022.
But what happens to someone like Paul Ehrlich, who predicted imminent catastrophe when this clearly was not happening as he was writing and then did not happen? Just looking at Ehrlich's Wikipedia page, we have
A common criticism is that Ehrlich's predictions routinely failed to come true; for instance, Ronald Bailey of Reason magazine has termed him an "irrepressible doomster ... who, as far as I can tell, has never been right in any of his forecasts of imminent catastrophe."[41] On the first Earth Day in 1970, he warned that "[i]n ten years all important animal life in the sea will be extinct. Large areas of coastline will have to be evacuated because of the stench of dead fish."[41][42]
In a 1971 speech, he predicted that: "By the year 2000 the United Kingdom will be simply a small group of impoverished islands, inhabited by some 70 million hungry people." "If I were a gambler," Professor Ehrlich concluded before boarding an airplane, "I would take even money that England will not exist in the year 2000."[41][42]
When this scenario did not occur, he responded that "When you predict the future, you get things wrong. How wrong is another question. I would have lost if I had had taken the bet. However, if you look closely at England, what can I tell you? They're having all kinds of problems, just like everybody else."[41]
Ehrlich wrote in The Population Bomb that, "India couldn't possibly feed two hundred million more people by 1980."[27] In 1967, Ehrlich called to cut off emergency food aid to India as "hopeless".[43] This position was later criticized, as India's food production subsequently skyrocketed through the Green Revolution in India, and its per capita caloric intake rose significantly in the following decades, even as its population doubled.[44]
A large increase in global food production since the 1960s and a slowing of population growth have, within the current context of continued depletion of non-renewable resources, averted the scale of food shortage, famine and catastrophe foretold by the Ehrlichs.
Canadian journalist Dan Gardner, in his 2010 book Future Babble,[45] argues that Ehrlich has been insufficiently forthright in acknowledging errors he made, while being intellectually dishonest or evasive in taking credit for things he claims he got "right". For example, he rarely acknowledges the mistakes he made in predicting material shortages, massive death tolls from starvation (as many as one billion in the publication Age of Affluence) or regarding the disastrous effects on specific countries. Meanwhile, he is happy to claim credit for "predicting" the increase of AIDS or global warming.[13]
In the case of disease, Ehrlich had predicted the increase of a disease based on overcrowding, or the weakened immune systems of starving people, so it is "a stretch to see this as forecasting the emergence of AIDS in the 1980s." Similarly, global warming was one of the scenarios that Ehrlich described, so claiming credit for it, while disavowing responsibility for failed scenarios is a double standard. Gardner believes that Ehrlich is displaying classical signs of cognitive dissonance, and that his failure to acknowledge obvious errors of his own judgement render his current thinking suspect.[13]
Barry Commoner has criticized Ehrlich's 1970 statement that "When you reach a point where you realize further efforts will be futile, you may as well look after yourself and your friends and enjoy what little time you have left. That point for me is 1972."[46] Gardner has criticized Ehrlich for endorsing the strategies proposed by William and Paul Paddock in their book Famine 1975!. They had proposed a system of "triage" that would end food aid to "hopeless" countries such as India and Egypt. In Population Bomb, Ehrlich suggests that "there is no rational choice except to adopt some form of the Paddocks' strategy as far as food distribution is concerned." Had this strategy been implemented for countries such as India and Egypt, which were reliant on food aid at that time, they would almost certainly have suffered famines.[13] Instead, both Egypt and India have greatly increased their food production and now feed much larger populations without reliance on food aid
Amazingly, following the series of incorrect predictions Ehrlich made in and after writing The Population Bomb in 1968, he followed this up with The Population Explosion in 1990 and has continued saying that we have global overpopulation that is causing or will cause a dire crisis unless we cut worldwide population. He has said the same thing this century and even this decade. It appears the only reason he's not saying that today is that he died earlier this year.
If I didn't look it up, I would've guessed that his recent position would be something like "well, I got some things wrong, but it was only due to these actions that were inspired by my work that crisis was averted" or "while crisis was averted, it was a lucky roll of the dice and, in most universes, the agricultural advancements that staved off the mass starvation deaths I was predicting don't happen", not "just you wait, the crisis is happening now and I'm about to be proven right"; in 2015, referring to his incorrect 1968 book, he said "[m]y language would be even more apocalyptic today". That's the pattern we've seen from Zitron, but I wouldn't have guessed that the one person I looked up would've kept that up for 50 more years. Maybe we'll get 50 more years of Zitron predicting the end of AI progress.
In one of the quotes from Juho Snellman, above, Snellman says that he writes a large amount of gish gallop, which is a term for when someone floods you with so much cheap (as in cheap to produce) nonsense that no one would want to take the time to bother to refute it. In discussing one small part of Zitron's talk in detail, we spent more than 1000 words explaining why Zitron has an incorrect understanding of how corporations work and how Zitron got the reasoning wrong. Someone can read that and then say, "but you didn't address X" in the talk, which is true. When I first watched the talk, I actually closed the tab after 90 seconds because there was so much nonsense that it didn't seem worth the time to go any further. I could write 5k words on the first 90 seconds of the video. Because Zitron is just saying a bunch of nonsense, he can do that very cheaply and it would take 30-60 minutes to refute 90 seconds of his nonsense if I had all the facts at hand. With time to look up the exact right information, it probably would take double or triple the amount of time. When someone who has good judgement sees something like this, they tend to immediately write the person off. Just for example, I mentioned to a friend of mine that I'm writing this post and they said
I was listening to this podcast with the guy and I couldn't get through it. My heart rate was going up because he would just say this false thing and then the interviewer, who was reasonable, would ask about it, "what about X?", and then we would just jump to another falsehood ...
... before I ducked out, he talks about how LLMs haven't gotten a lot better over the past year, and the interviewer says people use them and they've definitely gotten a lot better in the past year, and Zitron denies it and says 'have they?', and the interviewer is just like, "yes..." At that point, I'm just like, why am I listening to this conversation?
We mostly discussed predictions and not incorrect statements about the past or present, but everything I've read or watched by Zitron is also full of things like this. Many people will look at something like this and decide the guy is a crank and stop paying attention. But many other people will look at something like this, see someone refute a set of things, and then say, "but you didn't refute X" and, in general, the person doing the refuting may respond to a couple of these, but they eventually give up because the gish gallop method has the same properties as an amplification DoS attack. It's very cheap to generate new nonsense, but it takes some effort to refute it.
BTW, I was curious what this interview was, so I put the above quote into ChatGPT and asked it to find the interview. It was able to identify an interview with the relevant exchange (it actually identified multiple, as this appears to be a common question and response pattern by Zitron) and the timestamp of each relevant statement in the interview (the start of the general argument is here and a "have they" response is here. Prior to the "have they?" comment, the interviewer tries to establish a baseline that agents have improved in capability. Zitron denies that this has happened, and then when the interviewer notes that people who use these things for their jobs Zitron denies this with the "have they?" comment (he actually makes multiple contradictory statements in the sequence).
Another thing to note here is Zitron's extremely high level of stated confidence. Some that we noted were OpenAI's forecast that is "a statement so egregious that I am surprised it's not some kind of financial crime to say it out loud" (which they've achieved so far) and his claim that Google's forecast for Gemini users is "a number so unrealistic that someone at Google should have been fired, and that someone is Sundar Pichai" (they managed to exceed the forecast by 50% when Zitron's claim was that it would be completely absurd for them to reach the number at all).
I've made quite a few predictions, and quite a few of those predictions are wrong. When I'm really making a prediction, I attach a confidence level to the prediction just for my own sake, so I can look back at these things and see how well calibrated the predictions are. I have never been wrong about a prediction that has anywhere near the confidence Zitron gives to some of his predictions. Given the stated level of confidence, even a single incorrect prediction would be a sign of an extremely high degree of overconfidence. One should effectively never be wrong about a prediction delivered with that level of confidence but Zitron is routinely wrong about predictions he makes with what is rhetorically pretty much the highest possible degree of confidence.
BTW, a funny thing about Gemini hitting 500M users being "so unrealistic that someone at Google should have been fired, and that someone is Sundar Pichai" is that Zitron has also (incorrectly) said that Google doesn't know how to grow, and that as a result they're shoving AI everywhere. Dennis Snell pointed out that, if Zitron takes his own statement seriously, Google can make Gemini's user numbers go to any number it wants by doing the exact thing Zitron said they would do, sticking AI everywhere.
You can't actually take Zitron's statement about Google's lack of growth leading to AI desperation seriously and also take it seriously when he says that Sundar is committing some kind of gross malpractice by naming a number like 500M users. This is another thing that is immediately obvious on watching one of his talks or reading his writing. There are a bunch of disconnected statements that don't fit together, except insofar as they're statements about how AI companies and people and companies that are using AI are evil and bad. The actual numbers and logic of the statements are contradictory. It seems to be whatever comes to mind that can be used to paint the villains as evil. And, funnily enough, the 750M user number Gemini hit shows that both of Zitron's statements were incorrect. If Google were as desperate to juice the numbers as Zitron claimed, they could've easily gotten the number above 1B by sticking Gemini everywhere, and of course 750M > 500M.
BTW, the point at which I stopped the talk for the first time was
a market obsessed with year-over-year revenue growth. And this progression was natural. It was horrible. You can blame Marc Andreessen. He's a horrible man. You can blame many horrible men. There are so many guys to be mad at the moment.
That last sentence really sums up Zitron's position. "There are so many guys to be mad at the moment". In this talk, he throws in this jab at Andreesen and blames Andreesen for Meta, Google, and Microsoft pursuing growth. In reality, if Marc Andreesen had never existed, Meta, Google, and Microsoft would almost certainly still be trying to grow so we of course cannot actually blame Andreesen for these companies trying to grow. There's just this thing that he says is bad, and in his usual style, he pulls some person and says they're the evil villain that's to blame for this, and then moves on to the next non sequitur.
Because I'm a masochist, I actually went and read a bunch of Zitron discussions (I believe I read every major discussion on HN and lobsters, and a bunch of other ones as well) to see what people who take Zitron seriously are saying. One common defense was the one above, sure, you refuted some points, but you didn't cover X. A more common defense is to say, just in general, people attack Zitron because of Y (usually his style), but they never address his points, "which tells me everything I need to know" (or something along those same lines). Based on the timestamps of the messages, just scoping to the stories that were being discussed, there were generally already comments discussing Zitron's actual errors, but Zitron's defenders would ignore this and just claim that people were unable to point to mistakes Zitron had made. This is a very Zitronian move and it makes sense that people who like his style would also use this move. After all, who would find Zitron convincing? Someone who thinks this kind of thing is valid reasoning.
The next most common "move" was to simply deny that Zitron said something that was refuted. When people would mention that Zitron was repeatedly on the record in 2024 and 2025 as having said LLMs couldn't improve further for fundamental reasons, Zitron's defenders would say that he never said that, and likewise for previous predictions or factually incorrect statements.
Another class of defense I saw were comments like "but what about all the AI hypists who are wrong?". Like I said before, I wrote a 34k word post about how a bunch of the most respected futurists have been wrong, not just because they made incorrect predictions, but their methods and reasoning were wrong. But a bunch of people who hype the future being wrong doesn't make people like Ed Zitron or Paul Ehrlich any less wrong. Zitron and Ehrlich are still exactly as wrong as they would be if those futurists never existed.
A friend of mine also noted this about comments on cases where people point out that Zitron was wrong about models not improving from 2023 to 2026 (and yes, this is specifically on stories or comments that discuss Zitron's disproven statements on capabilities not improving):
It's incredible to see so many people saying, "Zitron isn't wrong, he's just early!" I guess the implication is that we'll eventually realize that the models we have in 2026 are actually no better than the ones we had in 2024 or ??
An interesting thing about publshing this post is that a decent fraction of the people who've message me to tell me that I'm wrong say that I'm wrong because, today in 2026, models haven't actually models haven't gotten better since 2023 or 2024. My guess would be that most people who are saying things like the quote above are just doing the "move" where you don't read what was actually said and respond with a canned response that's nonsensical to anyone who's actually read what they're replying to, but it turns out there are plenty of people who actually believe Zitron's string of statements that imply models haven't improved since 2023 or 2024.
If someone's actually looked at what's happening, I don't think there's anything you can really do to convince someone who's denying reality at that level, but for anyone who's just hasn't seen how things have changed, for a visual example of improvements over that time period, here's a comparison of 2023 and 2025 video generation and here's an example from August 2026. Video isn't a great example since models have improved a lot more at coding, e.g., with on the order of minutes of human time, it's possible to create a new regex engine with an interpreter and an native code compiler and then fork ripgrep to make it faster for codex's actual ripgrep calls on my machine, a project that would probably cost 7 figures pre-LLM if you price out how much people with the expertise for that are paid. But video is a nice example because, in the interview linked above, after the "have they?" exchange, at one point Zitron's "rebuttal" is, "you wouldn't make movie with it would you?". People are "shooting" quite a bit of AI-generated digital footage now and this is upending a lot of lower end video work. An easy prediction based on historical patterns is that this will continue to move upmarket over time, but just based on what people are using AI video for today, Zitron should probably find a new rebuttal even if he's just playing to true believers who don't think models have improved since 2023 or 2024. Of course that topic can't be related to math or the sciences, where improvements have been very rapid, and the same goes for numerous other fields.
Although Zitron's past predictions have generally been wrong, maybe he'll be right about something in the future. Perhaps some of these companies will have valuations decline for some reason. But, even if there's some kind of massive AI crash and OpenAI and Anthropic go to zero, in terms of the societal impact, if on top of that, some other event occurs that prevents further progress in models beyond whatever AI labs have internally right now, that's still going to result in a fair amount of change. Which companies are successful will change who gets rich, but particular companies failing won't stop changes that fall out of current or next generation model capabilities from happening; it just moves around who benefits the most.
Personally, it doesn't matter to me if folks at one company vs. another get rich. If one company does something better (in some abstract sense) than another, that's of some interest to me, but I have some skepticism about any particular company's claims that they'll do more of "the right thing" than another company (I could be convinced on this one, but I don't find the public claims that I know of very convincing).
If Zitron ends up being right about some company or other collapsing, that's pretty uninteresting to me compared to how capabilities have developed and will develop, where he's been wrong to date. It also happens that he's been wrong about the financial predictions he's made to date and the reasoning for those is flawed as well, but that doesn't really interest me, though I included a number of financial predictions for completeness.
Thanks to Yossi Kreinin, Juho Snellman, Dennis Snell, Nick Bergson-Shilcock, @blueblimpms, Bartek Ogryczak, Jamie Brandon, and Shriram Krishnamurthi for comments/corrections/discussion.
While looking for discussions about Zitron's work, the #2 hit on reddit was this comment by Zitron:
... some men don't like me because emotional honesty and introspection are difficult for them. Feelings are something that men are told to repress or compress. I refuse, and I find it disgusting when anyone tells me to do so ...
... Let's start with emotions, because it's the most obvious one. People really do not like that I am how I am, and think that I am "getting mad as a bit," or even go as far as to describe me as psychotic, out-of-control, and so on and so forth. This is a common reaction, I find, from anyone who themselves is emotionally repressed, especially in their own work. It is hard to be emotional and have well-done opinions ...
... I also have not taken the route you are "meant to take" to get here. You are "meant" to be an establishment writer from a big outlet, or an analyst, or in finance, or any number of other different "true paths" where you are "worthy" of whatever it is you're meant to get. I did not "earn my stripes" in the traditional sense, and those that have believe I did not earn my way here ...
... My work is also thorough, which is frustrating for people that do not do thorough work. I have thought through every point I have, and I take great pains to know subjects well. Notice how many people still claim "it's just like Uber" or "it's just like the dot com boom." It's much easier to just assume shit without ever checking if it's true! Having some asshole who comes along with thoroughly and with passion is frustrating. It reflects badly on your work ...
... I do a good photo shoot, I do a good interview, and I capitalize on events, and I do so without being craven, because I usually show up with a few thousand words of thoughts or an episode about a thing. I believe there are some that would like this level of attention or prestige, but they do not want to do the work to get it, and that chafes ...
... I love big, I love hard, I am who I am, I have never been made to feel welcome by any "in" group. I work my ass off, I write more than anybody else, I show up. With whatever space I create I will fight back against "in groups" or cliques. I hate them, and they hate me right back. And I fundamentally know why I believe what I believe. That upsets people who do not.
I have no idea if he means any of that or not (if this Wired profile about Zitron and the PR firm he runs is accurate, one would have to lean towards not), but Zitron seems to be very good at saying what his audience wants to hear, so this proably gives some kind of insight into his audience.
One thing to note about the bit about cliques and "in groups", if you just search his name on reddit commenters note that if you post anything indicating that AI has improved on his subreddit (such as link to benchmarks), you get banned for it, resulting in a highly clique-y echo chamber. I'm on the record as having said that METR's progress benchmark isn't meaningful and that you're better off going on vibes than leaning on a misleading analysis and that widely cited AI evals are frequently flawed, so it's not like I think that benchmarks are generally good, but the picture I got from reading comments was that you get banned pretty quickly if you don't hew to the party line, which is the opposite of the picture painted above. This isn't anything unique to Zitron; when looking up another influencer a while back, if you disagreed with that influencer on their reddit, they would write a comment thanking you for your comment and saying how much they loved getting feedback from people and how the world is some kind of great peace and love fest and we should all love each other while simultaneously banning you from their reddit.
I also found Zitron's comments on how people don't like his work because they dislike thorough work to be interesting for a couple reasons.
One is that my own work is frequently positively cited as being rigorous and thorough. There are plenty of people who dislike my work as well, but not only do I not know of anyone who's said they dislike it because it's thorough, I would be surprised if there was anyone who secretly dislikes it because it's thorough. In general, just doesn't seem like a reason that people dislike things. That also goes for people being upset because someone knows why they believe something or because someone else worked hard. It's really interestin to me that this appears to be what Zitron's audience wants to hear.
The second thing is that, I wouldn't personally consider my work to be thorough. The same thing I mentioned here about not feeling that my work is good also applies to not feeling my work is thorough. I do some amount of checking of my work. I don't know that I'd say that it's more than most in terms of time spent, but in terms of effectiveness, I suspect the combination of methods and time spent works better than average. But I always have a dissatisfaction with my work when I published it because I could keep checking more thoroughly forever and never publish anything, so I force myself to publish at a level that I suspect is above average on thoroughness, but well short of thorough. If I compare my work to the work of someone I consider thorough, like Gary Bernhardt, I don't know how I could call my work thorough. I have a few friends who produce Bernhardt-quality work and I make the choice to produce much more but also lower quality work. I think this is a fine place to sit in the quality-speed tradeoff space, but that doesn't make my work thorough. To be as thorough as Gary, with my baseline pre-July 2026 standard, I'd need to put 10x-100x the time in per piece of output (it would take an additional 10x or more with how I've been publishing lately). And yet, it would seem that my fact checking process is a lot more thorough than Zitron's.
Even if we put aside the gross arithmetic errors like the Timothy Lee example, if we look at cases like the Facebook MAU example, where he picks a number that's directionally opposite of other estimates and of Meta's own numbers that's also directionally implausible given the other data out there, I don't see how such a figure could survive any fact checking at all. And this goes for a huge number of his factual statements (I would guess most, although I haven't tried to randomly sample them to be sure). It seems like any kind of fact checking process that you could imagine would turn up contradictory results.
No good reason, really. I got four hours of sleep and my brain wasn't good for much of anything and I saw someone posted a screenshot of a reddit post dunking on Ed Zitron's prediction record. When I wrote this review of futurist prediction accuracy, I tried to make sure that I didn't bias what I was reviewing in any way. It's not obvious from the post if the redditor who reviewed Zitron's predictions was pulling predictions in an unbiased fashion or if they were biased in some way (since AI has become a culture war issue, it wouldn't be surprising if someone pulled biased predictions), so I decided to read some Zitron in my spare time while poking at agents to get them to do an unrelated task I wanted them to do. For the futurist post, I read multiple entire books to pull predictions and generally only stopped when someone was being repetitive and kept saying the same thing over and over again. In this case, all Zitron does is be repetitive, so the methodology in the futurist review would mean that I review a few predictions and then stop immediately. To overcome this, I had ChatGPT give me a list of predictions (with no attempted tilt towards correct or incorrect predictions) and then I skimmed/read the posts that ChatGPT linked to. There were some cases where I thought ChatGPT's reading of the post was incorrect (these were generally cases where it flagged a prediction that would be incorrect if its reading was correct, but I disagreed with its reading) and (discussed further below) I also removed predictions which weren't falsifiable or seemed pointless because they were tautological (I noted something similar to this in the futurist post).
If I really thought about it, I probably could've found something better to do with the time, but here we are; I sometimes have tasks on my todo list for when I'm too tired to do real work, but I didn't have one. I don't think they cherry picked particularly bad predictions, although they did pick some that are among the more absurd sounding. However, if you go and look into the details of ones that aren't such ironclad "dunks" (like saying that Gemini hitting 500M by EOY users is so absurd Sundar should be fired for the idea, when Gemini actually hit 750M by EOY), these are just as wrong as claims that Cursor has no realistic buyer with the implication they won't even sell for $10B when "everybody" (who cares about AI exits) knows they sold for $60B.
The redditor picked the high-profile failed predictions, but Zitron's prediction corpus has many more failures and, as noted above, the bigger issue is his reasoning.
Another thing about the reddit comment is, whether or not the comment is unbiased, one might have the suspicion of a kind of bias because it was posted to r/accelerate by someone who apparently is an r/accelerate believer. On looking at the actual predictions they are consistent with some bias (they would also be consistent with an honest mistake as there's no way to distinguish these from the record). For example, one of the "refutations" is a statement by Zitron that OpenAI will collapse in 12-24 months. OpenAI didn't collapse, so this would appear on the surface to be a great way to show that Zitron was wrong, but if you read Zitron's post, Zitron's actual claim was that OpenAI will either collapse or raise a lot more money and they raised a lot more money. I disagree with Zitron's implications that this is inevitable just leading to a later collapse but his stated prediction was not falsified.
This prediction wasn't in the set of predictions scored in this post. Some would argue that this should be scored in the post. The reason this wasn't scored is because the prediction seems meaningless except insofar as it contributes to Zitron's broader point (that OpenAI is doomed and must collapse).
If we think about predictions one could make, a tautological prediction (if you write out all the edge cases I'll elide for space reasons) that has to be true is OpenAI has enough money to operate or it doesn't, and if it doesn't, it must raise the money somehow. I could make a million such tautological predictions, but if one were scoring my prediction record, it wouldn't make sense to include these because they're meaningless. In general, a company that's alive will cover its costs. If it does not, it will try to raise money. If it fails to do that, it will shut down or get acquired. A prediction that a company will either cover its costs or it will not cover its costs says nothing.
OpenAI's own projections were that it would not yet be profitable and its costs would exceed its revenue. That seemed nearly certain, so if you assume that this nearly certain thing is true, then you have the nearly tautological prediction that OpenAI will either collapse or it will raise money to cover its costs. It would have been reasonable to make a prediction like this at very high confidence (99.9% or above). If you use any kind of prediction scoring methodology, such as Brier score, these predictions contribute essentially nothing except when they're wrong as long as Zitron has a significant number of high-confidence incorrect predictions.
And, as we noted above, Zitron is repeatedly incorrect on predictions he gives the highest possible confidence (given his wording, I would rate a number of these at 6 9s or above), so on any kind of scoring mechanism like Brier score, Zitron's record is very poor. And a summary metric like this really understates how meaningless predictions like this are. Hypothetically, let's say Zitron made an unbounded number of correct 99.99% certainty near tautological predictions, which would make the score from the bounded number of other predictions he made meaningless on something like Brier score. This would still give you zero confidence for any of his non-near tautological predictions, and those are the predictions people generally talk about (AI progress is done, AI companies must collapse and this will bring down major tech companies as well, etc.).
Back the topic of the reddit commenter's potential bias vs. mine, as noted above, I don't have a particular bias towards a view that rapid progress is inevitible and have called out cases where people are overly optimistic, as evidenced by this post on futurist predictions. I'm also not someome who needs to or has any desire to farm engagement by manufacturing reasons that someone is wrong or bad and don't consistently rate every predictor as bad, as evidenced by this review of Steve Yegge's prediction record, in which I note that he scored well and also actually performed much better than the raw score indicated because the predictions are generally well reasoned and directionally correct even if the precise prediction was incorrect. I think it's actually awesome if someone has good insight in the future and shares it publicly, so I'm happy to call these cases out when I noticed them. It's just that, in this case, Zitron is a kind of anti-Yegge: someone with a poor prediction record whose predictions are actually worse than they seem from the record alone.
I think it's almost certain that this post has multiple errors. In general, I find it very difficult to read a long stream of incorrect reasoning and then not get sloppy when looking for errors in it. I had this exact same problem when reviewing futurist predictions. It reminds me of when you're programming for some system where the compiler is very buggy and you hit compiler bugs all day every day (not uncommon when working with embedded systems, at least pre-LLM; now you can fix the bugs relatively easily). I find it hard not to get sloppy and think "hmm, this might be a compiler bug" even though, every once in a while, it will actually be your bug and not a compiler bug. The problem is much worse when looking at predictions from these kinds of predictions since the compiler still generally basically works and is often right, whereas when reading text like discussed here, you're just constantly drowning in nonsense that is occasionally punctuated by a good and accurate point.
I think, to do this well, you'd either need to find someone with very unusually high endurance for trudging through this stuff (I mean, much more than me, and I seem to have a somewhat above average endurance for this kind of thing) or have a team of people who independently rate and score things, but who would want to spend that kind of effort when any surface-level reading immediately reveals many things that indicate that these folks are pretty much totally wrong?
I did ask ChatGPT (web interface, Pro) and Claude (web interface, Fable 5) to fact check this post. They both found some minor errors that were fixed before publication.
One year ago, I found fact checks like this nearly useless, but they're halfway decent now and, contra Zitron, I would expect them to continue to get better. For people who are curious about the two, ChatGPT was much more thorough than Claude in this case and found more errors as well as finding every error that Claude found. However, it was overzealous and cited a number of non-errors, such as suggesting that tongue-in-cheek comments were incorrect, and that a number of statements that were generally true should be re-phrased in some more literal way (complete with AI-styled text).
I updated this post because @blueblimpms pointed out that a prediction that I thought was about GPT-5 was probably actually about GPT-4.5, although what Zitron is saying is unclear. After re-reading the relevant post, I agree, both that Zitron is probably referring to 4.5 and not 5 and also that his statement is unclear, so I changed that. That correction fits into this pattern that I predicted would occur, though I didn't note that a secondary cause of this problem is that Zitron's writing is quite imprecise and often relies on various vague implications between statements. The "have they?" / "are they?" response he does in interviews would be an example of this, where one could techincally argue that he's not making a statement at all and is just asking a question, although in those cases, given his overall position, we can infer what he means when he says that.
It's funny to say this, but a non-error in the post is my stating that I don't work at an AI lab or at a company that supplies AI labs (which I meant, colloqually, in the sense that I don't work at a company that sells hardware to AI labs or is primarily in the business of selling to AI labs, like an RL environment startup; no doubt we have people using non-ZDR plans that send data to AI labs, AI labs have scraped data from us or have paid scraping companies that bypass laws to scrape from us, someone has probably negotiated a deal to sell some amount of data we have to an AI lab that for an amount of money that isn't really material to us, perhaps as much as a half percent of our revenue since the AI labs were founded, etc.).
A friend of mine noted that he felt compelled to make a comment on Metafilter responding to someone who said that I worked at Nvidia. I've also had a few people message me directly to tell me tha I work at Nvidia. I'm not sure what to say to that. I have a few friends who work there and it sounds like a nice place to work, but it's not where I work. This is quite easy to verify from public information (technically, I could've quit my job and started a job since the last public information about my employment and then not told anybody, but if that were the case, that shouldn't cause random Zitron fans to think that I work at Nvidia; also, I haven't done that). We noted in the post that fans of Zitron often get the facts wrong when defending Zitron, which makes sense given that people who don't care much for the facts are who he appeals to. I guess this is another example of that. Of course people have also messaged me with all of the standard defenses we discussed in the post (Zitron never said that, Zitron is going to be proven right over time, etc.), but this one is a bit interesting in that "you work at Nvidia" is surely not a canned defense Zitrons fans have handy for every discussion they jump into.
But, even if it were the case that the accusations against Raghavan are true (I'm not sure how they could be, as how could one be a class traitor to computer scientists in the first place, but let's posit that, whatever it means, it's true), Zitron's contention is that "this shithead [points to an image of Raghavan] took over Google search in 2020" and then prioritized certain metrics over search quality. I'm not sure why one would name a particular person for this as this is something that was a long-standing fight with many people involved on all sides but, if we posit that this is all true, then we posit that the "management consultancy sect" will move metrics that will cause engagement and/or revenue to increase at the cost of search quality. This would have the opposite of the effect Zitron needs here to make his case that Google growth is done and they're so desperate for growth they have to put AI everywhere in some kind of crazed last-ditch attempt to save Google. Perhaps one could make the argument that this will eventually cause Google search to decline, but Zitron's argument was that, in 2024, they were desperate, not that users will eventually leave Google search, which will later cause a decline.
Anyone who's read a lot of Zitron will recognize a standard "move" of his, turning the situation into some kind of hero-villain narrative (for search, the alleged hero is Ben Gomes and the villain is Prabhakar Raghavan); it's as if his mental model of how companies works comes from movies about companies. If you ever watch a movie that's allegedly about some events and then read about it, you'll find that things get oversimplified into a hero-villain narrative and that almost all of the nuance is stripped out of the situation. And then if you're ever personally involved in something or talk to people who are personally involved and compare what happened to the books that get written about it, the same thing happens again; in general, the major causal factors are not identified in books about what happened in tech and many of the most instrumental people involved in some of the key decisions aren't even named because journalists talking to people about what happened aren't really able to piece together a plausibly correct story about what happened to someone who understands the underlying mechanics and has good information. Anyway, without knowing anything about the situation, if someone tells you a hero-villain narrative of the kind Zitron likes to spin, you can already be a bit skeptical.
[return]BTW, I don't think his anger really comes across in the video. I mean, he explicitly says he's angry and he swears and insults people, just like in his writing, but he doesn't really read as angry to me. It reminds me of this test on emotion recognition I took with a bunch of folks recently.
I found the test fairly difficult and spent maybe 5 minutes on the first question because the person had a huge fake smile on their face and also looked a bit uncomfortable and anxious. I couldn't tell if you were supposed to say that the person is happy or uncomfortable/anxious. Is it supposed to be a very easy test or is it supposed to be a test that has a bit of subtlety? Based on what the test looked like, after thinking about it for a while, I chose "happy". Luckily, the test actually tells you if you got the question right or not, so I realized the test was about the fake exaggerated expression being made and not the person's actual expression and most the rest of the questions were easy. One was difficult because they were faking one particular emotion with what is a textbook display, as in, the kind of thing one sees in a textbook, but in a very specific way that was less complete and more unrealistic than the other textbook displays; it was as if someone had read a description of what a contemptuous sneer is, and then was trying to make the facial expression based on the textual description. I had to think about that one for a couple minutes to get the correct answer.
Anyway, to me, Zitron seems like someone who's playacting anger and not someone who's actually angry. The tone of voice, facial expression, body language, style of movement, etc., just don't seem angry to me. I think this anger positioning works better in his writing than in his speeches because the cues he uses (swearing, saying he's angry, showing a lot of contempt, insulting people, etc.) are about as good as it gets for anger cues in writing. When you have audio and video, these are fairly weak cues; if the stronger cues don't really indicate anger, the person just doesn't seem angry. It's possible he has a non-standard way of showing anger or I just wasn't paying enough attention, but after watching some videos of him where he talks like he writes but didn't seem angry, the writing just doesn't feel angry to me anymore.
[return]In a small piece of optimism, I'll say that this blog seems to have done ok despite not leaning into extremist positions and generally trying to avoid clickbait. This often means that, when I look at some data, I'll see something that looks like it would make for a really interesting viral hit piece, but then on looking more closely, it's actually a boring negative result, like when I ran this quick and dirty programming language eval, which originally appeared to show a very interesting result, which went away once I fixed the obvious eval bugs. Oh well. I'd like it if people published more boring negative results, so I published the boring negative result.
I wouldn't be surprised if this blog is within an order of magnitude of traffic as Zitron's substack (server-side stats show 610k uniques for me in the past month, but who knows how many of those are bots with some but minimal Cloudflare bot blocking) despite Zitron writing much more frequently than me and pulling out every clickbait trick in the book, while I just occasionally post something when I feel like writing something up. Although my goal obviously isn't to get traffic, if we adjust for the level of time or effort, I don't think this blog does terribly compared to Zitron. Could Zitron have 6.1M monthly uniques? It's not impossible and it's hard to tell what these numbers mean with bot traffic, but for reference, The Economist has about 1.3M subs and the NYT has about 13M digital subs. If we arbitrarily hypothesize that 4/5 of uniques will be bot traffic, having an order of magnitude more traffic than this blog would put Zitron into the same class as The Economist.
Ceteris paribus, I think Zalewski is right on the incentives, and I've seen a lot of people become caricatures of themselves as they lean into what drives the most engagement, but I think doing the opposite can work ok.
For example, with a style that could be described as the opposite of clickbait, Simon Willison has written what I suspect is the most widely read blog among programmers for the past 3-4 years (in the same way that, at various times in the past, Joel Spolsky or Jeff Atwood or Steve Yegge seemed to be the most widely read programmer among programmers). Among programmers and other serious users of AI, I would guess that Willison has a larger audience than Zitron.
However, it's true that Zitron has a kind of audience that Willison can never really get with his style. In the body of this post, we looked at common defenses of Zitron on forums where people use AI. That was pulled from forums where people use AI. If we look at the world at large, the comments look fairly different. For example, on the video that my friend mentioned, where Zitron repeatedly denies reality and the interviewer pushes back, the top comments at the moment are all in support of Zitron and they also just deny reality and claim that the places where the interviewer pushes back with a piece of reality are the interviewer being biased or just not knowing what he's talking about. Among the top comments, there seems to be little to no engagement with the facts of the matter; it's all mood affiliation. The comments remind me of what supporters say about politicians who use the gish gallop strategy and just say a bunch of outrageous nonsense. I could imagine Zitron running for office one day on the strength of his reality-denying popularity or becoming a demagogue who's a right-hand-man of someone in office, so Zalewski is right in that Zitron's appeal is not one someone is going to get by accurately describing what's happening in AI.
But, while I don't know Willison and this could be totally wrong, my impression is that, like me, he's doing something he wants to do anyway and the audience just sort of happened despite him not trying to maximize his audience. When I say it works ok, I mean that he seems to be able to support himself working as a full-time open source developer due to the sponsorships he's gotten (which I would presume are generally because he has such a large audience), which seems like a good outcome even if this doesn't create the kind of mass appeal someone like Zitron can generate.
[return]2026-08-30 08:00:00
I used to wonder why I see so many more bugs than most people. I easily observe hundreds to thousands of bugs per week and nothing seems to work, but most people I talk to don't see anything like this. For a long time, I thought this had something to do with how I use computers but, over time, I've realized that it's mostly that people are hitting the same bugs and don't notice.
If you're not a programmer, that's probably a better way to see the world, but I think curing quality/bug blindness is helpful for programmers. I've done this with a lot of friends and acquaintances (just by pointing out bugs). After a few weeks, people who are so inclined tend to start noticing bugs as well.
Because I notice these kinds of things, I've had multiple jobs where directors/VPs/execs/etc. sometimes ask me to evaluate something when they want an actual opinion from someone who is relatively likely to notice issues (and fix them or drive fixes for them if necessary). Sometimes I won't find any issues (there are likely issues that just aren't the kind I notice). More often, I find issues that fall somewhere from "mild" to "moderate". And, sometimes, the issues are severe, to the point where one might even say the thing actually doesn't work.
I find this last category a bit mysterious, as when I look up discussions on how the thing got into this state, there's usually a stream of internal comments indicating that the thing is great, it works well, etc., but when I open up the thing and try it, it's in a state where the thing only works if you do quite a few non-intuitive workarounds. More likely than not, not only would a normal user not be able to use the thing, they'd have such a hilariously/infuriatingly bad experience that they'd tell their friends.
I've had this post in mind for maybe a decade or so, but I was hesitant to write it up because, in the back of my mind, I always wondered if I'm somehow triggering weird corner case behavior most users don't hit without realizing it. But after seeing more and more cases where the product launches and falls flat on its face because users run into the exact same issues I saw, I don't think that, in general, I'm hitting bugs because I'm doing unusual things a normal user wouldn't do. If a product seems severely flawed when I use it, it probably is. And with the magic of LLMs, nowadays, I can even have LLMs act like normal users in a lot of ways and show that the issues reproduce across many different scenarios.
I don't want to give any specific examples where it was my job to see how well the thing worked because, even if the internal examples are meant in a constructive, blameless, way, they may not always read that way when re-posted externally, so I'll give a few less interesting and less well supported "random" examples.
A while ago, I wrote up the results of some web search queries and found poor results from Google, Bing and Kagi. In general, the major search engines failed to return good results for the queries and returned pages full of low-quality SEO spam as well as some sites that were actually scams. BTW, on the scale mentioned above, I would consider this "moderate" and not "severe" (severe would be something like, the search engine returns 500 errors half the time, the majority of results are scams, etc; my bar for severe is that a normal user likely won't be able to use the thing at all, not that they have a bad experience). Almost nobody1 objected to my characterization of Google and Bing search results, but people told me that I was wrong about Kagi. In some cases, people sent me their actual search results. In every such case, the search results did not contain a good result that I could see (e.g., for the seasonal forecast query, the search failed to return an up-to-date seasonal forecast, for the tire result, there was no result with a correct explanation, etc.) and was full of SEO spam. In one case, a person passed me both their list of Kagi filters as well as the search results they got without making claims that the results were good or bad, but people generally insisted the results were good even though the results both failed to link to a matching result and were full of spam except in cases where the user did something like pin GitHub to the top of their results, which worked for the queries where the goal was to download software that's hosted on GitHub, but of course completely fails for the other queries from the post as you're unlikely to get your regional seasonal forecast or the correct explanaton of tire mechanics on GitHub.
In the abstract, I get that people who are fans of things tend to be blind to the thing's faults. For example, since I bought a Volvo after seeing how they do in out-of-sample crash tests, I sometimes search for answers to my questions on Volvo car forums. For well over a decade, the reliability data that exists (and I think this is backed up by the anecdotal experience that mechanics who work on Volvos have) is that Volvo reliability is mediocre to poor, but of course Volvo forums are full of people who insist that Volvos are among the most reliable cars and that the data are all wrong.
An example that might be more central to the topic is Blackboard (the course management software). Back when it was the most widely used software by universities for coursework, the software was widely disliked by both students and professors. I think it would be fair to say that it was the most widely disliked software in my social circles (there was more strongly disliked software, like Visual Source Safe, but any more strongly disliked software wasn't widely used enough to be the most widely disliked overall). The Wikipedia page notes
Blackboard had become "one of the most disliked — even detested — companies in education."
as well as
In December 2011, Fast Company reported that 93% of respondents to the Amplicate customer opinion survey "hate" the company.
Back when I was much younger and had less of a filter, I ran into someone who worked at Blackboard and, without thinking, I stupidly blurted out something like "what's it like to work on this software that so many people dislike?". Luckily, the person I was talking to wasn't offended at all and, instead, they were actually confused because they thought it was widely loved software that users really liked. They didn't really believe what I said could be true and I made some comment indicating that it was just confusion on my part and then the conversation continued in a different direction. At the time, as someone much younger and more naive, I was really surprised to hear that the software that was probably the most widely disliked software in my social circles was thought to be really well-liked software by the one employee from the company I met (and, presumably other employees as well).
I can understand how the Volvo forums get to be how they are, in that cars are reliable enough in general now that people generally don't experience car breakdowns, so it's easy for someone to think something like "the data can't be right; after all, my car has never broken down". It's more of a mystery to me how somebody can look at a set of search results that are full of spam and then dash off a message explaining how great the results are, even if they're a fan of a particular search engine or how someone can think that users generally love software that's famous for being disliked, to the point that every single person I talk to about it tells me how bad it is (often in unprompted complaints), there are news articles that discuss how much people dislike the software, and the near-universal dislike for the software is mentioned on its Wikipedia page. Another Blackboard-like example might be Discourse (forum software) web performance, where one of the inspirations for this post was discussions with Discourse employees who thought that Discourse had great performance. I found that one interesting because Discourse actually had code in it that slowed down actual page loads in order to cheat on web performance metrics like LCP. That went well beyond just optimizing for a benchmark and rose to the level of actual cheating that not only had no benefit to the user, it actually harmed the user. At some level, the programmers implementing that sort of cheating and advising users on how to not accidentally subvert the cheating must know that the actual performance of their app is poor, but it's very easy for people to put up mental barriers around this kind of thing.
By now, I wouldn't say that I'm surprised because I've seen this kind of thing enough that I would actually consider it surprising if it didn't happen, but I still wonder what's going on inside someone's head when something like this happens.
For a non-programming example, we previously noted in this post on how people have different perspectives on "obvious" facts, there's a basketball player who, subjectively, is generally considered to be the dirtiest player of his era. The NBA doesn't track objective measures of player dirtiness, but he seems dominant on a wide variety of measures. For exampe, although, like rebounds before 1950, genital strikes aren't an officially tracked stat, he surely holds the record for punching, kicking, kneeing, or otherwise striking players in the genitals this century (he should also hold the record for era-adjusted numbers, but it's possible that he doesn't have the all-time record due to play being much dirtier overall in the 80s and 90s). In discussions, most fans of his team don't seem to notice this and the phrases "natural rebounding motion" and "natural shooting motion" have become running jokes from how oblivious the team's fans are when they justify this player's contortions when he strikes other players in the genitals.
On average, humans have a high ability to ignore negatives in things they're a fan of, including (and often especially) their own work or work their company does. For better or for worse, I seem to have the opposite of this and my thoughts immediately go to the flaws in myself and my work. A number of times, as a result of a blog post, someone has messaged me with something like "how would you like it if someone criticized your work?" or "how would you like it if someone said your work isn't good?" To the former, my thought is that I go to great lengths to get criticism from people who can poke holes in my reasoning, so it's pretty awesome if someone has remotely reasonable criticism of my work. And to the latter, I generally think my work is full of major flaws, so, uhh, yeah, it seems pretty reasonable to say it isn't good. There are particular aspects of my work that I think are interesting or good but, overall, I don't know that I'd rate anything I've done as good. I'm not saying I don't have blind spots, but I think I'm a bit less prone to this particular one than most people2.
If I think about analogous blind spots I've had, one that jumps out at me is from when I was a little kid and a friend of mine used my computer. For this story to make sense, you have to know that this was in the mechanical mouse era. Over time, detritus would get stuck to your mouse ball and cause it to track erratically unless you cleaned it out.
When my friend tried to use my computer he found it impossible to use the mouse because mouse pointer movement seemed almost random. When I sat down at the computer again and used the mouse I didn't have any problem using it at all, but on looking at what I was doing with my hand to smoothly move the pointer in a straight line, I was violently throwing my hand all over the place. I realized I must've adjusted to the detritus on the mouse ball over time as it accumulated and I was somehow compensating for the mouse's extremely erratic tracking by making countervailing erratic movements3. I thought it was pretty amazing that I could not notice that I was doing this and I always wonder if I'm doing some equivalent thing today.
I sometimes think about all of the mitigations I've developed to work around bugs. For example, when opening a new Google Doc, I used to immediately put the title I wanted into the doc. At some point, maybe ten years ago or so, Google Docs added some kind of delay such that the typing you do into the title box right after you open the doc gets overwritten, so I now have this habit where, after opening a Google Doc, I do something else and then I change the title. Over time, as Google Docs has had more and more features added, I've developed a series of habits that avoid all sorts of pitfalls (such as trying to search at the "wrong" time and getting the useless native browser search instead of the Google Docs search).
My feeling is that a large fraction of computer literacy and software literacy is developing a large library of these habits that you just do at a non-conscious level. These are often quite specific to the situation, such as a habit I developed when I worked at Microsoft of flipping my laptop's WiFi switch to off before logging in (which I noticed other people doing as well). This was because there was some service, which would often fail your login with "There are currently no logon servers available to service the logon request”. But if that service couldn't connect at all, the check would be bypassed and you could just log in.
We could fill a post up with examples like that, but back to the main topic of the post, one commonly suggested way to try to overcome quality blindness is to have people dogfood their own software. On average, this is a lot better than not dogfooding, but it only works to the extent that people don't figure out (and then forget about) habits that work around whatever issues the software has. On average, programmers are pretty good at working around software foibles (you had to be in order to be an effective programmer pre-LLM), so it's very easy for programmers to not notice these kinds of issues if they're not paying attention.
On the flip side, a large part of making an app easy for people to use seems to mean making weird habits like these unnecessary. Although this sounds like it should be easy to do, from having seen people try to give feedback about this kind of thing, the reflexive reaction of most developers seems to be "huh? It's easy to do X, just do [complex sequence of things that no normal person would think of if they hadn't used the app many times before unless it was specifically explained to them or they saw someone else do it]" or "huh? Didn't you see that the instructions for this are clearly laid out in page 43 of the manual after you execute the steps in Appendix B on page 261?".
That being said, I think curing people of quality blindness is do-able because I've done it quite a few times. I think this only really works when the person is receptive, as people have infinite capacity for willful blindness but, in cases where people are receptive, just pointing out issues they didn't notice seems to work. Years or even a decade later, people will sometimes tell me they see bugs everywhere now.
The reason I think this is worth doing is that I've seen people and teams with a high degree of quality blindness ship things that have reduced or even no chance of success because of product quality issues4. It's one thing to knowingly and deliberately trade off quality for speed5, but when I've seen this happen there's always been a kind of quality blindness where everyone involved with the project thinks they're shipping something very high quality when that's not the case.
This has never been unimportant, but it's gotten more important with coding agents because, while it's easier than ever to churn out low quality software, it's also easier than ever to improve quality, whether that's better performance, fewer bugs, etc.
But, to do this, you have to actually notice that this is possible, that quality can be improved.
Thanks to Yossi Kreinin, Dennis Snell, Michael Malis, Emu Chu, Gary Bernhardt, Jon Surrell, and Matt Mullenweg for comments/corrections/discussion.
Naturally, Gary Bernhardt ran into a Google Docs bug while reading a draft of this post.
P.S. Like I've mentioned in the last four posts, I've been trying to write posts more quickly because, with LLMs, it's so much easier to look at data and figure things out but, since I'm not writing with LLMs, the time it takes to write something up hasn't fundamentally changed, unless I want to move to a different point in the quality-velocity trade-off space. The prior result was that I would run some experiments and tell a few friends and then never write anything up because, due to Amdahl's law, writing anything up would effectively consume all of my bandwidth for running experiments. In fact, despite trying to do this (my goal is to spend 30 minutes per post on the write-up), since writing my last post, I have three results that I think could make a totally fine blog post that I haven't had time to write up (not including things done for work, which would add a few more things). Without having LLMs write for me, I don't see a reasonable way to get the time per post significantly below 30 minutes (and I think I often miss my goal and take more than 30 minutes), so the non-LLM options here are some posts that are much sloppier than my normal posts (in a human slop kind of way), or almost no posts.
Anyway, if you have opinions on these quick (and surely more wrong) writeups, let me know what you think (X Bsky Mastodon)!
Michael Malis (founder and former CEO of Freshpaint) noted (in messages, hence the message-like format)
For a similar but different data point - I’ve seen similar blindness when it comes to advertising. When I would explain Freshpaint to people, I would tell them that we help hospitals with marketing
A common question I get is why do hospitals do marketing. The weird thing is if you pay attention, hospitals do a ton of marketing
In SF there’s tons of bus ads and billboards for ucsf/sutter health/stanford and various treatments
This is a different topic from both Michael's comments and the post, but I'll say that I've talked to quite a few people who don't believe ads work at all, but I talked to someone whose data methodology and judgement I trust about ads A/B testing at one big company I worked for and looked at the data myself at another company and I thought the causal evidence for ads providing real lift (well beyond the cost of the ad) was strong in those cases. In the case where I looked at it, they did a geo-segmented A/B test where they bought ads in some geos but not others (this was done worldwide, with the regions being things like U.S. states, Canadian provinces, etc.). This kind of geo-segmentation was done because, even with cross-device tracking, it's not 100% clear if someone has been exposed to an ad or not (of course this is still the case with this kind of segmentation and I would prefer segmentation that was more clustered to population areas and didn't have splits where people are relatively likely to, for example, commute from one side of a boundary to the other, but this kind of contamination generally makes the likely true lift higher than the estimated lift), so people sometimes do these geo-segmented A/B tests.
Anyway, in these A/B tests, return on ad spend was quite good just on direct revenue gain, and there was also a gain in users which seems likely to result in more revenue down the road (the later revenue wasn't analyzed). I don't know about ad effectiveness in general or if your particular ads are effective, but the commonly repeated idea that ads don't work in general seems wrong to me.
On the topic of Michael's comment, I think it's easy for programmers to not notice ads. Almost all programmers I know use an ad blocker and, in real life, their eyes seem to just skim over ads and not notice them. I can see how this would feed into the idea that ads don't work. Who the heck would look at these things? But from my interactions with "normal" people as well as the data I'm familiar with from my time at Google, many or perhaps most people don't even realize that a lot of ads are ads. When they do a Google search and they click on the top resut, they often have no idea they're not looking at what Google "thinks" is the best link, they're looking at at a link from whoever paid Google the most to buy that ad slot.
Em Chu, on a habitual bug mitigation:
I'm sure you can collect infinite examples for this section, but I just want to Complain: when waking up and unlocking my laptop (mac), it's very easy to get it in a state where it's "awake" but unusable (black screen with cursor or similar) which can only be fixed by physically closing the lid and re-opening it. To work around this, I think I usually wait a second after the screen turns on, interact with the trackpad, and then unlock it, though honestly that happens mostly subconsciously, and I clearly need practice given that I still hit the bug a few times a month.
On reading this, I examined how I open my laptop and realized that I have some funny habits as a result of working around other laptop bugs. The specific bug mentioned here doesn't reproduce on my laptop and it seems that I can stop the habitual mitigation I put into place for some prior laptop.
Gary Bernhardt, on his experience reading a draft of this post
While reading it, Google Docs' UI seems to have broken, making it impossible to scroll up to read some comments (see screenshot [not shown in post]).
From looking at the screenshot, I've seen the exact same bug and have some mitigations for it (different ones depending on the context). I would personally rate Google Docs as far above average in terms of software quality: I find it much less buggy and janky than the major alternatives (Microsoft Word, Open Office, various old editors that are long gone like StarOffice, Lotus, etc.). And yet, I could easily sit down and write a 10k word post on Google Docs bugs and the workarounds I have for them.
At times, I've tried to see if I can get a job somewhere where I just fix quality issues all day. This has never panned out, due to some combination of this not being a very high priority and it also not being a normal role that companies have a role for. I sometimes daydream about joining companies as an intern and just fixing quality issues for a few months and then leaving. In practice, I think if I got such a job, a lot of the fixes would get blocked and it would be very difficult to actually drive change as an intern for three months, so it would have to be some mostly abandoned project where nobody cares what I do (and corporate priorties aren't so focused on shipping features that fixes get immediately re-broken).
Unintentionally on topic: the
<abbr>tags worked on mobile ~last week, but are no longer working across any iOS browser (safari/chrome/firefox), and I had a hard time figuring out that hover showed them on macOS browsers (all three) due to the long delay
Germans have a word for that - Betriebsblindheit
Crazy anecdote from @danluu.com here and I wish he'd been even blunter at the time and asked this person where they'd gotten this belief about Blackboard being liked by anyone at all. User surveys? Principal (as in, not agent) surveys? Inner conviction??? [screenshot of Blackboard anecodote]
[Some variant of, people are forced to say that they don't see bugs by their bosses]
I don't thnk this is consistent with any of the major examples in the post, let alone all of them. Consider the Blackboard example mentioned above. It's unlikely that I and other people this Blackboard employee are "secret shoppers" who are checking in on employees, and the employee's reaction is clearly absurd to anyone who isn't such a hypothetical (and in reality, non-existent) secret shopper, so reacting like this just makes them look a bit silly in the eyes of a large fraction of the people they meet for no benefit (for example, see the previous quote, which seems like a typical internal reaction). Perhaps a few very paranoid employees would maintain this front on the off chance they run into some friend or relative of the boss who knows that they work for the company who relays the story back and they have a boss who would care about this, but it's just not plausible that this is (for example) the case for every Discourse employee who reached out to me to explain to me that Discource performance is really good.
If we look at the basketball example, this is even more absurd. You could possibly come up with some kind of reasoning like, other fans would shun you if you didn't believe or pretend to believe the most absurd rationalization, but as someone who has spent a lot of time around sports fans, I've generally not found this to be the case. And, to the minimal extent to which this is kinda sorta the case, it's more an issue of self selection, where fans who are into the most extreme rationalizations will spend more time around fans who are into the most exterme rationalizations and fans who are less into these extreme rationalizaitons will tend to spend more time with fans who are less into them.
Also, just looking at the career path of people who don't buy into these things and fix them, noticing these issues and fixing them has gone very well for those people. Pretending these issues don't exist (whether that's at a concious level or not) also seems to work well, so I don't know that fixing these issues is actually a better career path, but it's certainly not so bad that, in general, there's meaningful career pressure to pretend these things don't exist overall even if there are some individual positions where there's some direct pressure to pretend these issues aren't real.
Who else uses the Shift key to end the screensaver, because in case the event goes through to an actual program it's least likely to do have unintended effects?
This reminds me of how, when I want to send a queued message to codex immediately and interrupt the current tool call, I put my finger on the key and the press as quickly as possible to reduce the window of time where the tool call will finish and the escape key will stop codex entirely instead of causing the message to send. I should probably just run a patched version of codex that has fixes for this and a few other issues I've run into, but I'm already doing things like trying out some weird workload-specific optimized version of ripgrep that also has an added native code compiler which compiles matching expressions in another thread while the search starts and then cuts over after compilation completes, so it's not like I'm against creating weird patches to improve my workflow and it's more of an issue of overall bandwidth (no doubt, on writing this, someone will tell me that I could just hit another key instead and could've found this out by asking codex about the key in the time it took me to write this comment). Just like with Google Docs, I consider codex above average in terms of software quality in the space, but even though I haven't been using it for a year, I could easily write 10k words on all the workarounds I've implemented (either by habit or, in some cases, with actual scripts that monitor for broken behavior and then correct it).
[translated from Russian, probably loses something in translation] It seems to be a well-known joke: people use broken software and think it's normal. Something like, "To save a field's data, you have to click the mouse on the adjacent field." Those who always use the keyboard will notice the problem, but those who are more proficient won't.
This reminds me of how, after some macOS update, some old apps wouldn't refresh/update their UI except when the focus was switched to them. To see any changes in those apps, I got the habit of tabbed away from the app and the tabbing back every time I did an action where I'd want to see the result. I don't know if this ever got fixed because I eventually stopped using those apps. If this happened today, I'd probably use an LLM to edit the binaries to fix whatever issue was causing this.
I didn't remember this until after I'd written and published this post and someone linked to John's post, but apparently John Regehr wrote "Operant Conditioning by Software Bugs" before I started a blog! There's a good chance I saw the post before at some point and then completely forgot about it. Maybe I should've used an LLM to search for prior art, but if I did that, I'd probably never write anything because there aren't that many really new ideas and almost everything is going to be similar to something someone else has said. For better or for worse, I'm much more verbose than John, so this post uses a lot more words and has more random stories thrown in. If you find my blog posts too long, but have somehow managed to stumble down into the bottom of the post anyway, you'll probably like John's post more than mine :-).
For example, anyone familiar with my code at Twitter will recall the huge comments I had at the top of the main files for the things I owned, which described the various ways in which the thing is really flawed. They were all things that, for one reason or another, I thought weren't worth the time to fix, but they were still serious problems that anyone interacting with the code ought to know about. For this metrics project, I even had a long doc that described the issues in great detail (IIRC, in a lot of cases, the rough shape of the fix was described; maybe today an LLM could take that and fix it).
I have the same feeling about my writing. While a huge number of bugs sneak through my writing (like spelling and grammatical errors), most of those are things I sort of don't care about and will skim past in other people's writing as well. When I say don't care, it's not that I don't want things to be better (when people send me corrections I generally fix things), it's just that my brain doesn't naturally pay attention to those things no matter whose writing it is, so I don't seem to have a particular blind spot in my writing with respect to these kinds of bugs. For the things I do care about, I could edit posts endlessly because, no matter how much I edit, the post still seems pretty bad to me.
I used to often (and still sometimes) send a post to someone and ask them if it makes any sense to publish it at all because I generally don't like my output and, if I'm just looking at my own writing, I don't think it's worth publishing. At this point, I've done this enough that I'll often just publish even though I don't like what I wrote, but if someone says "how would you like it if someone told you your work wasn't good?" as a kind of "gotcha", boy, they really have no idea how I think about my work.
There are various tricks I've used to get around this (not explicitly to get around this, but they do so as a side effect). As discussed in this old post on writing, for a while, I hired a professional editor and had a process goal of doing one pass on each post and then trying to improve the next post. And as noted in the postscript to recent posts, now I'm trying to write with extremely minimal cleanup and editing and push posts out in half an hour regardless of the state of the data I'm looking at or the post (which I'm generally failing to do; I thought I might succeed on this one because it doesn't have any data analysis, but someone made a comment on the draft post that got me to re-write the whole thing, and just on number of words in the post, half an hour would really be pushing it on the original and then it increased in length). Of course a post that's written as quickly as possible with little to no regard for cleaning things up is going to be terrible in all kinds of ways, so all flaws I see in the post don't stop me from publishing it. Have my recent posts been good? Of course not; for any of the experimental/data posts, I could probably name ten things that should be fixed about each of them off the top of my head. For this post, I'd have to re-read it to come up with ten things, but I'm sure if I did re-read it I'd want to re-write the whole thing because of the issues it has.
[return]a response I've heard to this kind of thing recently is that Anthropic had the best growth numbers in history while Claude was very buggy. If you have the best coding model and agent in the world, you can get away with a lot, but even they seem to have spent a fair amount of effort improving quality.
Maybe you can also get away with it if you have a product that succeeds due to bundling, the strength of your enterprise sales team, network effects, monopoly power, etc.; all but one of the cases I'm thinking of are places where the team didn't have these things on their side. I actually thought the one other case I was thinking of would be something like Blackboard, but (if the Google results are accurate) I see that the software has declined from being #1 in the market to being a minority player, so maybe they couldn't get away with it either (I didn't look into the reasons for the decline; perhaps it's a coincidence).
As noted above, Blackboard is an example where you could argue that the software quality didn't matter and people might as well just believe whatever makes them happy; if thinking that users love the software, then why not think that? But most of the rest of the examples that come to mind for me aren't cases like that. I don't think this is the best example, but it comes to mind because the comment below is the last time I was reminded of the Blackboard example. There was a comment from a Tumblr employee who said that they'd solved the moderation (abuse / spam / toxicity / etc.) problem mechanically at Tumblr via the way reblogs worked and that the mechanics Tumblr provided to users were good enough that the community could self-police bad behavior and that other social media sites would do well to learn from Tumblr. This was referring to Tumblr back in its heyday (maybe 2009-2014). I never really read much on Tumblr so I don't personally have an opinion, but back when it was a major social media platform, the reputation among folks I know was that it was heavy on bad behavior, particularly pile-ons caused by people taking out of context quotes and turning them into ragebait (not to say this doesn't happen on other platforms, but the belief was that the way Tumblr was structured and/or the communities involved made this worse on Tumblr). I'm not sure I know anyone who used Tumblr at the time who would say that the community was good at self-policing. In fact, when Scott Alexander wrote one of his most famous pieces, Toxoplasma Of Rage, he dedicated an entire section to how Tumblr's reblog system is particularly bad and is guaranteed to result in bad behavior. He actually says that whoever designed the system either didn't understand what they were doing or they understood all too well and deliberately made the most ragebait-inducing system possible. This was written during the time when this employee said that Tumblr had solved the moderation problem and uses examples from that time.
Moderation at scale is an impossibly hard problem, so as a non-Tumblr user, I'm not even sure that Tumblr did worse than other platforms given its size and growth rate, but I think you'd need some quality blindness to think that Tumblr had solved the moderation problem. I think the strongest positive case you could plausibly make would be something like "Tumblr was better than average, but many people had a worse than average experience due to the communities they were in and some of these communities were unusually widely read and Tumblr therefore unfairly gained a reputation as being a particularly bad platform". I don't know if that's true or not, but it doesn't seem impossible that it could be true; it does seem impossible that Tumblr solved the moderation problem.
[return]2026-08-21 08:00:00
The other day, I saw a viral tweet saying that people talking about how LLMs are causing slow, bloated, code are going to eat crow once they re-write everything in super-optimized assembly. We're not quite at the point where we want to write everything in assembly, but some variant of what Nolan Lawson said about testing, you can choose how many bugs you want now, which I less eloquently noted here, is becoming more true for performance.
In response to a comment in my last post that the cost of formerly specialized performance work has dropped by many orders of magnitude and performance work that used to require a person or team that had a rare set of skills can be done by anyone who can type a few sentences1, which means that you can do all sorts of optimizations that used to be too expensive to be worthwhile for all but the largest scale or most lucrative projects, Marc Brooker responded with
Completely agree with your closing point. Dynamic custom software, fitted to a particular workload rather than a class of workloads, seems like a very likely outcome. (Which comes with all kinds of fun risks and opportunities of its own). Kind of reminds me of FFTW. And a ton of weird old demoscene techniques which were all about being super fast and small on a very particular problem (and often very particular hardware). For example, I remember a demo that re-used its code as textures to get great cache locality.
And Michael Malis has noted
There’s been a meme circulating about how AI doesn’t help because “code was never the hard part.” I think that’s true in some domains, but in others, writing the code absolutely was the hard part. JIT compilers are a great example of that. For many pieces of software, a JIT compiler would help a lot with speeding up the code. The rarity of JIT compilers makes me believe that implementing a JIT compiler historically was too difficult for it to be worthwhile. LLMs have lowered the barrier to entry and made it much easier to write a JIT compiler. This is the thesis behind pgrust. Databases historically were the hardest piece of software to build and were limited because of that. Now, with AI, we can be more ambitious about the type of software we build.
Let's try this out with FRE, the regex engine we built in the last post. Recall that it was created by having an agent loop for a month on improving regex engine performance with access to the rebar regex benchmark suite. This resulted in FRE being heavily overfit to rebar until we warned our agent that we had a holdout benchmark, which caused the agent to generalize the optimizations enough that performance was ok-ish on our holdout. There's no particular reason to use a "software factory" regex engine that doesn't beat a well-tested regex engine on holdout benchmarks, but one notable thing about FRE was that the native AOT compiled version did quite well at longer searches. We noted that, it stands to reason that one could run the native code compiler in another thread while ripgrep was running its normal matcher and then cut over to the native code when it finished compiling and generally get better performance. Of course this will generally result in worse performance for short queries as we lose a thread to compilation, but I care a lot more about how long ripgrep takes when it runs for many seconds or minutes than when it runs for a few seconds, so I'm ok with that tradeoff.
In the same way we could build a regex engine in a few minutes of human time, we can also just try this experiment in a few minutes of human time. I typed a few sentences and an agent went and did the work to allow this to happen (which would be a decent chunk of code surgery for a human) and it ran the benchmark on actual ripgrep queries that come from my codex history. For longer queries, we see a 2x-4x performance improvement here for a few very simple queries. But most queries are more complex, and when we run on representative holdout queries, for queries where AOT should be enabled2, we get about a 7% speedup. Not an earth shattering result, but also not a bad outcome for spending a few minutes typing to codex (and it's still doing more optimization and will presumably speed things up further).
This is arguably a silly thing to do, since if we're repeatedly searching for text on a computer, the obvious thing to do to speed that up isn't to write a native code compiler for regex matching, it's to create an index. But the point here is just that this kind of technical work, which used to take a fair amount of time and expertise, can just be done trivially now. And if we wanted to build a text index, it just so happens that I worked on BitFunnel, the Bing search index that was specialized for constant/fast text ingestion that won Best Paper Award at SIGIR, so I can think of a few experiments to try if we're going to build a fast local index of our entire machine (the projects I've seen seem to be intended to index your code directories, but what really kills my machine performance is when codex decides to run ripgrep against huge temporary directories with a ton of generated files and then expands to looking at my whole machine when it misses, so I'd want an index of my entire disk and not just of the code for some projects).
If I were working at an AI lab and had access to things like SOTA models running on Cerebras chips or other accelerators that greatly increase tok/s and therefore load/demand for search, I might actually survey the existing indexers to see if they're fast enough or if I'd want to build something custom myself. While the open source version of BitFunnel "only" contains a bytecode interpreter and one JIT, the Bing version contains multiple JIT compilers. A project that did that level of optimization used to be a major undertaking, but "I could do that in a weekend" is now actually true for some of these kinds of projects. With my lowly $200/mo account, I think a somewhat faster ripgrep plus any off-the-shelf index is fine, so maybe this fast-ingesting whole-machine index project can be left as an "exercise for the reader (who works at an AI lab)".
The drastic reduction in the cost of optimizations has been true going back to November 2025 and maybe even somewhat before then with public models (and I'm sure before that still with what folks at AI labs had access to). For an example from the GPT-5.1 or 5.2 days, with no knowledge of game AIs, I tried building an Azul AI. This ended up being the strongest AI in the world for the game by a pretty large margin. From reading the thesis that describes the 2nd strongest AI, I think my AI is probably a bit better on the "AI" side of things, but the main place it wins is on optimization despite spending what looks like maybe two orders of magnitude less time (estimated by reading the thesis and seeing the process and comparison to my process) and also mostly working on my laptop vs. having a cluster of machines to use (which means much less bandwidth to run experiments with, do parameter tuning, etc.). For example, that other AI is single-threaded and my AI is multi-threaded. Since I have a native code version as well as a heinous shared wasm memory + javascript version, and two different search architectures for two different versions, which "require" completely different multi-threading algorithms (minimax for a very small and fast net and MCTS for a larger net), this would've been a fairly large undertaking if done by hand. And, because I let an LLM pick the multi-threading algorithm based on its own (incorrect) reasoning a couple times before spending 30 minutes reading about multi-threading algorithms for game AIs myself, I ended up re-writing (having codex re-write) the multi-threading algorithm multiple times.
There's a bunch of standard stuff it makes sense to do to debug and verify a multithreading algorithm for something like this, like implementing replay from debug logs that can reproduce bugs despite the algorithm being nondetermistic. Doing that alone would've probably been days to a week of work had I done it by hand, but it's exactly the kind of thing an agent can trivially do in a loop (just have it try to replay logs and insert logging for non-determinism every time you don't get a perfect replay). A lot of the tedium it used to take to get a tricky optimization like this working is gone.
This also applies to a lot of other tricky optimizations. From having written CPU microcode, done CPU verification, worked on optimizing a search engine index, etc., I have a lot of experience looking at optimizations and thinking "hmm, this would increase performance by 2%, but it's going to take N person-days to verify that this tricky optimization works" and making a call to go ahead or not based on whether or not it's worth the time to get the optimization working. Now that this N has dropped by a tremendous factor (variable but, in terms of human time, frequently 1000x / 10000x / 1000000x, probably more like 1000x on dollar cost if you compare token costs at metered rates vs. the Bing engineer who wrote the compilers at JITs that the search index used), the number of these kinds of optimizations it makes sense to do goes way up. The same goes for optimizations that you aren't sure will work out. I used to sometimes look at an optimization that I wasn't sure would speed things up and think "this will take M hours to implement to the point where we have a good enough measurement to guess at the performance impact". Many more of those optimizations make sense to try out now.
Going back to the game AI case, at least for the AI I tried, it seems like you gain about 100 Elo for every doubling in speed (more than in chess, I suspect because draws are very rare). Just adding multithreading alone is enough to wipe the floor with an otherwise comparable AI on a large machine. If you stack in 10-20 more optimizations that seem too annoying for most people to do by hand, the difference in strength is tremendous and it's not really reasonable to try to keep up with a hand-written AI3.
The game AI case is a little more annoying than for most software because a lot of the optimizations you want to do actually change the result and there isn't a cheap, trivial, way to tell if the speed increase + the change in result gives a better or worse actual result in practice. And, as we noted before, current publicly available SOTA models are pretty bad at experimental design, so I had to set up the framework they used to determine if an optimization is good, but once that was in place, it's like any other optimization problem. I guess people working on LLM optimizations also have to deal with this class of problem but most optimization problems are a lot more straightforward.
To pick another example, as part of preparing for performance interviews, Jamie Brandon tried Anthropic's now public performance takehome. After trying it, he had Claude pick up where he left off and it got a much better result. When he looked at what Claude did that he didn't, he said a lot of the optimizations were things that occurred to him but he hadn't gotten to yet, and "[o]thers were just crazy shit that I would never try unless I was working on this for weeks"4. He's a reasonable performance engineer and he got an offer for the performance job he wanted, but on a well-defined optimization problem, he doesn't stand a chance against a decent model (I haven't tried the problem myself, but I suspect I also wouldn't stand a chance given remotely comparable time controls).
Coming back to this part of Marc Brooker's comment:
Dynamic custom software, fitted to a particular workload rather than a class of workloads, seems like a very likely outcome.
This seems pretty inevitable. In another response to my post, Michael Malis of pgrust said something similar:
[discussion of pgrust optimizations] ... I think it's easy enough to create these optimizations that we could look at a customers workload and add them as needed
Without having any kind of framework or setup, right before I started writing this post, I had an agent do workload-specific optimization for my ripgrep queries (not the native code compiler switch, just the optimizations to the general FRE engine based on a set of benchmarks), which took about 2 minutes for me to launch. The optimizations run on a set of queries, and then there's a later holdout set of queries to run against. That's still running, but the initial results seem promising. After one pass of optimization, the workload optimized version is 2% faster than standard ripgrep on the holdout and it's still getting faster. 2% isn't a big deal for my local ripgrep usage, but considering that this took minutes of time and the optimizations done here got started when I started typing this point and are still improving, I'd take a 2% win here (note that this isn't combined with the native code compiler, which would give a larger overall win if combined properly). And recall that this is leveraging the FRE regex engine5, which was substantially slower than the Rust regex engine on holdout benchmarks and was stuck with slow improvement on holdouts because with me knowing nothing about regex workloads and SOTA LLMs not being good enough at experimental design to do unguided open-ended self-improving loops, we didn't have a good way to improve performance on our holdouts. But if what I care about is performance on my own workloads, I have plenty of data and am generating more all the time. As Marc Brooker noted above, we do have to be careful about overfitting if there's a regime change that's not in the old data, etc., but we're still in a better situation than we were before.
In the more general case, if you're someone like Marc Brooker at Amazon or Michael Malis working on pgrust, it makes sense to not just do this as a one-off, but to work with customers to pilot a program that uses their data to optimize things for them and then figure out how to scale it out for customers in general. I'm not working at a company where that's the best use of my time6, but it's pretty wild that you can see that this is coming for larger companies with more scale, and given that it only takes minutes of my time to run these experiments for my personal workflows, it's pretty reasonable to mess with this kind of thing on personal projects.
Thanks to Jamie Brandon, Michael Malis, andrea (@s__video), Artyom Bologov, and Max Bittker for comments/corrections/discussion.
P.S. As I've noted in the last couple posts, with coding agents, the time it takes to run an experiment and see enough of a result to satisfy my curiosity has gone way down while the time it takes to make a result really rigorous hasn't changed or has gone up, so writing things up the way I used to would mean running very few experiments relative to the bandwidth I have for them. As a result, I've just been running these experiments and sharing the result with a couple of friends. As an experiment, I'm trying to write these up in a very quick and non-rigorous way instead of years of these experiments only being known to a few friends. Like the last post, I set a goal of writing this post and doing all the clean-up in half an hour and didn't time it but am pretty sure I missed that by a bit.
Even doing this, the time it takes to write these up is long enough that I'm falling behind on sharing recent results, but I'm not inclined to switch to LLM-written posts (yet?), and I don't think I can realistically get the time to clean up the data and write a post like this down enough to turn a post around in less than half an hour. Just on the length of this post, typing this up should be something like 20-30 minutes including time to pause and think about what I'm writing, and then when I look at the data sometimes something will look wrong enough that I need to look into it more closely to see if there's an issue that needs to be fixed (this happened multiple times here, and I would expect that, because I didn't spend much more time, there are other data issues that I don't know about).
Anyway, if you have opinions on these quick (and surely more wrong) writeup, let me know what you think (X Bsky Mastodon)!
I've been on the record for a long time as strongly disagreeing with the general sentiment that the developers of X are bad and should feel bad for writing slow code because there are a lot of different kinds of programming expertise and not only is it not the case that most programmers don't have performance expertise, it probably doesn't even make sense for them to develop (from the standpoint of what the business cares about, what the employment market looks like, etc.), so of course most projects will have very poor performance compared to what a performance expert can do. I can see why a performance expert would look at the growing gap between how fast a program can be and how fast programs actually are and think that it's ridiculous. I don't disagree that there's an absurdity to it, but if I think about the gap between how good a UI can be and how a good a UI I can make (by hand) is, I don't think that looks any less absurd, but I also don't think it really makes sense for me to spend time learning how to build a great UI, or even a decent UI, for the same reasons it doesn't make sesne for most people to spend time learning how to decent performance work.
For the example above, Jamie Brandon got an offer from Anthropic and you probably can't afford him unless you're OpenAI, but you can afford to use a coding agent that can beat him on a bounded optimization problem. The agent doesn't have the judgement he has and will do worse on an open-ended problem (recall that when we tried building an optimized regex engine and just told it to not overfit, it was more than an order of magnitude worse than the best regex engines on our holdout benchmarks, but also recall that after telling the agent there was a holdout it was doing poorly on, it sped up regex engine performance enough to generally match 2nd tier regex engines in terms of performance, which is still extremely good compared to the general level of performance optimization in most code today), but that's plenty good to achieve reasonable performance on all sorts of problems. This post has generally discussed backend performance issues, but agents don't seem worse at front-end performance if you want to drive down a set of metrics like LCP and CLS. In fact, after inserting the interactive plots I've been using recently into posts, I found that my client-side perf numbers got worse, so I had an LLM spent 1% of my weekly quota optimizing those and the numbers are once again back to being good. This is a very simple site, but people do these kinds of optimizations on fairly complex apps that ship to many millions of users and it also works there, although it does cost a few more tookens.
I still don't think someone is bad and should feel bad if their software has poor performance, but I do think that someone who doesn't know anything about performance and is a reasonable user of LLMs (just in general, not on performance problems in particular) should generally be able to create software that has decent performance. If you just tell an LLM to optimize, it will often do all sorts of incorrect things that are really bad that you have to catch, but that's generally true of using the LLM effectively in the first place, so getting decent performance is no longer a specialized skill.
Here's some information about the distribution of riprep queries on my machine. I make no claims that this is at all representative of what's happening anywhere else. The pattern distribution of the length of the pattern that's searched has a lot more long patterns that I would've expected. The p50 is 55 unicode code points (I'll just call these characters for simplicity), which is already longer than things I grep for by hand, and the p90 is 119!
We can also look at the number of alternation arms in regexes, which are once again much more complex than what I do by hand.
Another view is to look at how these are correlated. Do we get more alternation arms in the regexes as the regexes get longer? Yes.
What are these really long regexes, anyway? If we look at them, most of the longest are long alternations over function or tests names, such as the following regex, which appears to be related to FRE development.
fn (hot_byte_compiler_is_generic_only_and_anonymous_count_uses_auto_count|
one_pattern_count_spans_uses_the_retained_complete_span_session|
formal_compact_state_byte_visitors_coexist_with_native_count|
fixed_boundary_record_visit_matches_line_relative_reference_and_is_atomic|
unbounded_languages_refuse_finite_extraction_before_allocation|
formal_single_raw_span_sweep_preflight|
assert_exact_fixture_uses_formal_large_continuation_sweep|
url_only_compile_identity_binds_language_and_owner_mode|
url_only_compile_exact_limits_and_runtime_refusals_close|
url_only_compile_post_plan_allocation_faults_close|
url_only_owner_discriminator_is_stable_and_precharged|
url_only_compile_owner_is_strategy_and_operation_scoped|
formal_rebar_url_owner_is_compile_only_and_matches_oracle|
formal_rebar_url_exact_fixture_uses_certified_execution|
formal_fixed_schema_materialization_matches_both_record_oracles_and_controls|
formal_single_count_selects_compact_state_byte_complete_bound_visitors|
authenticated_bound_line_total_lf_free_domain_opportunity_exceeds_five_percent|
prepared_absolute_onepass_fuses_slots_and_preserves_pre_source_fallback|
authenticated_word_boundary_russian_compact_lowering_public_canary|
ordered_nfa_x86_epsilon_edges_bypass_the_assertion_call|
ordered_nfa_aarch64_epsilon_edges_bypass_the_assertion_call|
ordered_edge_dispatch_v2_is_target_neutral_deterministic_and_relocation_free|
ordered_edge_dispatch_v2_copies_canonical_tables_and_cap_falls_back_to_v1|
ordered_nfa_v3_composes_terminal_range_and_dispatch_without_data_relocations|
ordered_nfa_x86_terminal_range_emits_authenticated_reverse_scan|
ordered_nfa_aarch64_terminal_range_emits_authenticated_reverse_scan|
ordered_nfa_x86_boundary_assertion_cache_is_lazy_and_boundary_scoped|
ordered_nfa_aarch64_caches_repeated_assertions_once_per_boundary|
boundary_assertion_cache_requires_dense_exact_kind_reuse|
boundary_assertion_cache_selection_is_compiler_only_and_deterministic)
But some are funny numerical constructions, such as
:(13[0-9]|14[0-9]|15[0-9]|16[0-9]|17[0-9]|18[0-9]|19[0-9]|20[0-9]|21[0-9]|22[0-9]|23[0-9]|24[0-9]|25[0-9]|26[0-9]|27[0-9]|28[0-9]|29[0-9]|30[0-9]|31[0-9]|32[0-9]|33[0-9]|34[0-9]|35[0-9]|36[0-9]|37[0-9]|38[0-9]|39[0-9]|40[0-9]|41[0-9]|42[0-9]|43[0-9]|44[0-9]|45[0-9]|46[0-9]|47[0-9]|48[0-9]|49[0-9]|50[0-9]|51[0-9]|52[0-9]|53[0-9]|54[0-9]|55[0-9]|56[0-9]|57[0-9]|58[0-9]|59[0-9]|60[0-9]|61[0-9]|62[0-9]|63[0-9]|64[0-9]|65[0-9]|66[0-9]|67[0-9]|68[0-9]|69[0-9]|70[0-9]|71[0-9]|72[0-9]|73[0-9]|74[0-9]|75[0-9]|76[0-9]|77[0-9]|78[0-9]|79[0-9]|80[0-9]|81[0-9]|82[0-9]|83[0-9]|84[0-9]|85[0-9]|86[0-9]|87[0-9]|88[0-9]|89[0-9]|90[0-9]|91[0-9]|92[0-9]|93[0-9]|94[0-9]|95[0-9]|96[0-9]|97[0-9]|98[0-9]|99[0-9])[0-9]:
This is equivalent to :(?:1[3-9]|[2-9][0-9])[0-9]{2}: (which, if run through ripgrep on the original input, has approximately the same performance; the shorter regex is technically a bit faster on the real query data, but only by a very small amount). The entire pipeline for that was
cargo clippy … | rg 'crates/fre-aot-regex/src/module.rs:' | rg NUMBER_REGEX | head -250
which might be an odd thing for a human to do, but agents seem to do this kind of thing all the time.
On another topic, if we look at how long ripgrep queries took, there are quite a few slow queries, e.g., p99 is almost 1 minute! And p999 is almost 10 minutes! And the maximum query over this time period (around a month on one laptop; queries and distributions seem likely to be different on the AWS hosts I run agents on, etc., but I haven't checked) is approaching 2 hours!
In terms of command line options, we see the following. Perhaps unsurprisingly, codex often wants line numbers and, for whatever reason, it very occasionally uses PCRE2 regexes.
I won't add plots or tables for these, but another thing to note is that there's fairly low locality for what patterns are searched for (about 94% of patterns only occurred once), which makes some sense given how long a lot of the queries were. However, there's fairly high locality in what files get searched and a file that got searched is relatively likely to get searched again soon, indicating that (for small enough files), they're likely to be searched in memory.
Also, 99% of queries were regex queries (1% were non-regex string searches) and 99.9% of search queries were ASCII only, but in terms of files searched, approximately 45% were ASCII only and 55% contained Unicode, a higher percentage than I would've guessed for Unicode.
On a draft of the last post, Peter Geoghegan noted
It's also possible for a regex implementation to be faster by supporting fewer features. Some implementations don't support back references, etc.
which is also true here. The workload-specific optimizations done here were fairly superficial because I just gave codex some short instructions and let it do whatever it wanted (which is, in general, not the most effective use of codex), but with a more detailed plan, more focused optimizations supporting the common use cases for my queries could be expected to yield larger gains.
2026-08-17 08:00:00
There's been a lot of talk about the vulnpocalypse, to which I don't have much to add because I'm not a security person, but I haven't seen much discussion on the closely related (and to be fair, less serious, issue), the benchmarkpocalypse.
While it's become easier than ever to make serious performance gains, it's also become easier than ever to reward hack a benchmark and make fake performance gains. The former is probably happening quietly across many different companies, but the latter is something I see at least once a week nowadays. Someone will claim they optimized X and got some huge performance improvement over existing software, but, when you look at it, what they did was make some optimization that improves benchmark performance without actually improving real-world performance. This is often some kind of "we rewrote X in Rust"1 project or a new startup that's looking to either fundraise or sell something, but it happens on other kinds of projects as well.
Of course, people have always trumpeted unrepresentative microbenchmarks to show that their pet project is great. It's always been easy to fake up an unrepresentative microbenchmark and that's never going to change. What's changed is that it used to take a lot of work to game a large benchmark suite, but an LLM and loop can just do it. There are quite a few famous examples of gaming large benchmark suites from back when this was hard. For example, way back when people cared about SPECint / SPECfp as proxies for workstation performance, CPU vendors would try to find compiler "optimizations" that would speed up the calculation in the benchmark, such as Sun finding a way to improve 179.art by 12x in SPECfp2000. Skilled engineers spent a lot of time trying to find benchmark hacks like that. LLMs not only make this trivial, they do it by default, making formerly trustworthy benchmarks meaningless unless you audit the result or trust someone who did.
Rather than point to someone's bad claim, I'll point to FRE, this regex engine I had an agent build, which I could claim is the world's fastest regex engine because it beats the Rust regex crate at the fairly comprehensive rebar regex benchmark suite. But this was created by putting an agent in a loop for a month with instructions to not overfit to the benchmark but no real supervision. For the most part, getting an LLM to give you a good benchmark score is fairly easy, and this case was no different; it took a couple weeks to roughly match Rust regex crate performance and then another couple weeks to get to 1.4x faster2 on rebar. But agents are wont to reward hack and overfit unless you put serious guardrails in place to avoid that, which I didn't do in this case as an experiment.
To check for overfitting, I somewhat arbitrarily3 used the ripgrep benchmark corpus as a holdout benchmark it was 10x slower on cases where the benchmark didn't take forever due to an algorithmic blow-up, and there were cases where it took so long that it wasn't reasonable to even wait for the benchmark to complete. So much for being 40% faster!
Andrew Gallant (aka BurntSushi)'s rebar benchmark suite is fairly comprehensive as benchmaark suites go, but even with a fairly comprehensive benchmark suite, agents have no problem getting a high score while overfitting in a way that doesn't necessarily give good general performance.
The next step was using a trick we talked about before of not just telling the LLM not to cheat, but that there's a holdout benchmark set that it's judged against. After that, the LLM moderately generalized performance to the point where it's about 2.4x slower overall on the holdout. That sounds pretty good considering that we're comparing it to the fastest general purpose regex engine in existence. But, recall that these benchmarks were made by a coding agent. On looking at what the benchmarks measure, some of them really don't make sense to include, at least at equal weight. If we only look at the benchmarks that seem like they matter, FRE is 4x slower on the holdout0, which is a lot better than before applying the good ole' "tell 'em you have a holdout trick", but still pretty far from being 40% faster.
There are a few things I thought were interesting about this:
On (1), no wonder I'm seeing so many bogus claims. In the past, to build something like FRE that fakes performance well enough to be able to bogusly claim a 40% speedup, you would need a fair amount of expertise. At a minimum, you'd need to have a pretty good understanding of string matching algorithms, regex engines, as well as decent general code optimization and SIMD optimization skills. FRE also has a mode where it compiles the regex to machine code, so you'd also need some compiler expertise. Now you can get that kind of benchmark cheating (whether or not you want the cheating) with a few minutes of typing.
On (2), I'm curious if this generalizes but haven't tried enough examples to be able to tell.
On (3), there's no reason to use a vibe coded regex library that was almost no human effort that's slower than a robust, existing, well-tested, library, so I find the FRE artifact uninteresting. The thing I find interesting here is how much LLMs can substitute for what used to be rare, specialized, and expensive, knowledge.
In the past, even if you had the knowledge, you probably wouldn't write a custom regex engine that's optimized for your particular workload. There are some large-scale use cases where people would do that level of customization, e.g., when I worked on the Bing index, the code contained multiple different compilers because someone who worked on it wanted to eke out maximal performance; since you care about both compile time and compiled performance in a search engine and the trade-offs are different in different places, you get better performance by writing a custom compiler for each place where a normal project might just use an interpreter or directly walk some data structure with "normal code". The person who wrote those compilers, working on regex-like code might also write multiple custom regex engines, but very few people have both the expertise and the inclination to do that, let alone the freedom to spend that kind of time on such specialized code for work. If you price out that Bing engineer (then a Partner-level engineer, promoted to Distinguished Engineer for their work on the search index) compared to the price of running an LLM in a loop, the cost of writing this kind of specialized code has gone down by many orders of magnitude.
People who still think AI is fake will probably read the first part of the post and think "of course, AI produces fake things, so it produced a fake regex engine". But if we look at the results, being a bit worse than half the speed of the world's fastest regex engine on a holdout while being genuinely faster on many real workloads (most of the overfitting isn't that it special cased a particular benchmark pattern, but that it has some kind of optimization for things of same rough shapes and not of other rough shapes) it's pretty far from a fake regex engine. And, in fact, there's a native code compiled mode that actually beats the Rust regex crate on the holdout if you ignore compile time and are running repeated searches or a very long search (which is a reasonable thing to do for many actual use cases). If my goal with FRE was to produce a fast regex engine instead of producing whatever regex engine one can produce in a few minutes of human time, I suspect it would be fairly competitive on a broad range of holdout benchmarks (with some gaps that would only be found when people tried it on a diverse set of production workloads), and, even this quick and dirty version is very good at some real workloads.
So, even though the overall FRE regex engine has worse performance than the Rust regex crate, the gains you can get for specializing to your workload or use case mean that, in some cases, it could be reasonable to insert your own specialized regex engine somewhere, and the same goes for various other kinds of low-level software. You don't have to be an AI maximalist to think that it's plausible that, within some number of years, we could see this kind of thing happening for larger things, like databases.
Thanks to Yossi Kreinin, Jamie Brandon, Peter Geoghegan, Luke Burton, John Spurling, Dennis Snell, and Max Bittker for comments/corrections/discussion.
P.S. Per the discussion here, with LLMs, the time it takes to poke at something for a bit and satisfy my curiosity has gone way down, while the time it takes to write something up and make it rigorous enough to publish on my blog hasn't really changed (for a variety of reasons, I think it's actually gone up). The result of this has been that I'm doing a lot more analyses than ever and sharing results with a few friends but not publishing them. As an experiment, I'm trying to write up some things very quickly, with a much lower standard for how cleaned up and rigorous things are than I'd normally have for something that appears on the blog; more like what I'd tell a friend in a casual conversation than what I'd normally put in a blog post. The goal for this post was to do the write-up in about half an hour, so it's something I could do over lunch and not really take time on. If you have opinions on this, let me know what you think!
Of course, a caveat here is that all of the numbers have a higher risk of being wrong than usual. I looked at one benchmark for maybe a minute or two and found an issue, then I looked at another benchmark for a minute and found another issue. Both of those are fixed, but this implies there are other issues I haven't taken the time to chase down. But, with respect to bad benchmark numbers, that's highly realistic! Almost any time I look into benchmark numbers, such as here, or here, the numbers are wrong. Another aspect of the benchmarkpocalypse is that, at least for now, LLMs are good at doing bad benchmarking, so even if you have something that's a real performance improvement, you generally can't tell from some LLM-generated benchmark setup unless a significant amount of care has been taken to make sure that the benchmark setup is reasonable.
One thing I found after I wrote the above but before publishing the post, was that the LLM's claim that FRE is 40% faster than the Rust regex crate on rebar was also wrong. Or, if not wrong, at least misleading. It wasn't actually running benchmarks in the same way rebar benchmarks were run. I checked this after spending a minute checking benchmark results found two issues. It turns out that, despite instructions to run rebar benchmarks as they're run in https://github.com/BurntSushi/rebar, the LLM changed the interface to allow FRE to make some optimizations that improve performance. After fixing that, instead of FRE being 1.4x faster than Rust on rebar, it was 1.5x slower (and "only" twice as fast as re2), so the original result was doubly fake. Not only was FRE highly overfit to the rebar benchmarks, it the results also involved cheating.
But on the bright side, this means the difference in performance between FRE on rebar (1.5x slower than Rust) and on the holdout benchmarks (2.4x slower) isn't as big as it looked before, so the "tell the LLM you have a holdout" trick worked even better than it seemed to before.
After that, I let an LLM hill climb for a few hours and it claimed that FRE was 1.28x faster, which sounds like a great improvement for only a few hours of LLM time, but then I decided to spend another minute looking for cheating and found multiple issues, including one case where a search for the count of matches of (?s)^(.*)$ returned the count without even looking at the haystack (data). Another case of cheating was doing a multi-line grep where the benchmark is supposed to be done line-by-line. Finding these isn't surprising because this is the kind of thing that happens when you leave an agent in a loop for a month without defining strict guardrails. Whether this makes my point here stronger or undermines it isn't clear, but after fixing another set of these issues, FRE was back to being 1.4x slower. After leaving an agent to run overnight, FRE was allegedly back to being 1.5x faster.
Since my original goal here was to see what happens when you run a current (public) SOTA agent in a loop (GPT-5.6 Sol) without much supervision on a non-trivial code optimization problem without any real supervision, rather than spend more time fixing things up to make the benchmarks fairer, I'll just stop here and put a few plots of the results.
Overall, we can see that against Rust and RE2, FRE tends to outperform on the rebar benchmarks (and as noted above, much of this is due to overfitting), but not across the board (the graphs below don't necessarily match the numbers mentioned in the post because an agent is constantly making changes, so any snapshot is a point-in-time estimate that becomes obsolete immediately):
If you're curious about performance on specific benchmarks or specific classes of rebar benchmarks, we have the following table (ratios above one mean FRE is faster; below mean FRE is slower):
There's also an AOT compiler mode that takes a long time to compile a regex to native code before running it. There isn't AOT support for everything, but here are the results from the cases where it's supported. As we can see, the AOT compiler is very slow (it loses very badly in the compilation time benchmarks) and, despite spending quite a bit of time compiling, results are often slower than with the standard FRE regex engine (though it's also faster in many cases).
And then there are the holdout benchmarks. As noted above, for the non-AOT FRE code, performance on the holdout isn't as good as on rebar. And as also noted above, considering that this is for a workload like ripgrep, the "hot search" set of benchmarks is probably more important than the others, so the FRE result is worse than the overall score would make it look.
One thing to note here is that, for the holdout benchmark cases where we don't include compile time as part of the benchmark and we repeatedly run searches, AOT FRE outperforms on the benchmark. For a lot of use cases, you don't want a regex that takes multiple seconds to compile, but there are plenty of cases where this is fine, e.g., for something like ripgrep or Silver Searcher, it could start running with a regex that can start matching right away and then compile in another thread and cut over to the faster matcher when it's done compiling. Given how much of my CPU is spent on long ripgrep searches, it seems like a strategy like that could improve performance for work I personally do. Before LLMs, it probably wouldn't have made sense to spend the effort to write an optimizing regex compiler, but this is now do-able with a few tokens.
Another thing to note here is that this comparison is arguably unfair because this was run on an ARM Graviton machine with SVE/SVE2 and FRE has SVE/SVE2 optimizations. Pre-LLM, it might not have been worth it to have regexes optimized for every combination of SIMD instructions out there, but with LLMs, it's fairly easy to generate ok-ish SIMD optimizations. I know human experts who find that they can generally outperform LLMs here, e.g., Jay Stelly said that the last time he tried getting an LLM to produce SIMD code, it took 20-some iterations to get the code as good as he wanted. But, on the flip side, LLMs have the capability to try more optimizations than a human could possibly try in any given amount of time, so they can still perform pretty well overall even if any specific optimization isn't as good as a human expert would produce.
There's also the problem discussed in this post of overfitting. Depending on the context, that problem is somewhere from very easy to solve to a bit difficult to solve. I deliberately didn't try very hard to solve the problem here to see what would happen, but I did manage to solve the problem without an outsized amount of effort when working on this Azul AI (just for example), but a lot of these big benchmark claims come when people spend little to no effort trying to avoid overfitting, or even negative effort. In the pre-LLM era, people would often pick highly unrepresentative microbenchmarks to show off how great their pet project is which, at least at a non-conscious level, involves negative effort to avoid overfitting to a benchmark. Due to how humans are, I don't think people are going to stop making misleading claims and it's become easier than ever to make misleading claims, so of course we see more of them.
Note that while this post has discussed non-AI software, everything said here goes double for AI software. For example, I've seen lots of people drop comments saying that Kimi K3 is Fable (5) level. But every single person I know who's used it has found it to be substantially worse than GPT-5.6 Sol and Fable. I'm not saying it's not an impressive engineering achievement, but the performance on a wide variety of real-world tasks isn't up to the level it is in benchmarks. This even applies to various eval-y problems, such as when a friend tried different coding agents on the ICFP 2026 contest problems. It also applies to security issues, which are something that I have no doubt AI labs are putting into their evals, e.g., a colleague of mine tried using Kimi K3 to scan for vulns in our software and found that it found approximately a quarter of the vulns GPT-5.6 Sol found, found no vulns that GPT-5.6 Sol didn't find, and didn't have any advantages in any dimension other than on cost. The people I know who are using cheaper models to find real security issues are using other models, such as GLM-5.2, which perform worse on benchmarks but better in practice.
Back on the topic of FRE, one more note is that the holdout benchmark is an arbitrary subset of the ripgrep benchmark setup that was chosen by an agent for unknown reasons. I asked an agent to pull the entire benchmark suite, but that didn't finish in time for this post, so I don't know what the result will be once it's done.
funnily enough, I have some faith in some of the projects that people are the most skeptical of, e.g., every time I see pgrust somewhere, there are a lot of skeptical comments. But, without having looked into the details of what he's optimizing, I would trust that they're not doing something shady with their benchmarks because Michael Malis started the project (and is still involved). I used to look at most benchmark claims that cross my radar in some detail, but there are so many of these now that I don't really have time to do that and generally assume that claims are false in spirit (even if technically correct) unless there's some reason to believe otherwise. Of course this will sometimes be wrong (e.g., if I didn't know Michael Malis, I would've guessed that pgrust is just another low-quality "have an LLM re-write this thing" project), but LLMs are such an incredible machine for DoSing human attention that I don't know what else I would do about it (I've tried having LLMs analyze performance claims and, while the result is correlated with what I'd think if I looked at something myself, the result is often quite wrong).
Someone can spend seconds (or, if using the right framework, actually none of their time) generating something that takes people minutes to hours to understand. This is a topic for another post, but from talking to people about their experiences with this in the workplace, companies with poor norms for this kind of thing are really struggling with productivity today.
[return]The first few regex benchmarks I looked at had already been incorporated into rebar, so they wouldn't work as a holdout. And, as previously discussed, current SOTA LLMs aren't very good at benchmarking, so I wouldn't be able to trust the LLM to come up with a holdout benchmark unless I knew enough about regex performance to judge the quality of the benchmark suite. Since I know approximately zero about string matching algorithms or regex performance, that was also off the table.
It turns out that BurntSushi also maintains ripgrep and the benchmarks for ripgrep, which are big enough benchmarks that they didn't get bundled into rebar, so I tried using those benchmarks as a holdout.
2026-08-09 08:00:00
This somewhat widely cited post (I keep seeing it cited, anyway) suggests that dynamic languages and/or languages that represent things more concisely are more token efficient. It seems to be cited enough that LLM search results agree. For example, when I searched for "dynamic vs static language token cost" (no quotes), Google's AI summary opened with
Dynamically typed languages generally have a lower LLM token cost than traditional statically typed languages because omitting explicit type declarations makes the code more compact.
Google's AI cited the same post, which suggests that some concise dynamic languages have maybe 1/2 to 1/3 the token cost of static languages like Rust, Go, C++, etc. The author says
There was a very meaningful gap of 2.6x between C (the least token efficient language I compared) and Clojure (the most efficient).
And then they later tried J, saying
It dominates at just 70 tokens average, nearly half of Clojure (109 tokens). Array languages can be extremely token-efficient when they avoid exotic symbol sets. If token efficiency turns out to be a key driver, this is perhaps a very interesting way for languages to evolve.
The other dynamic vs. static language token comparison I've found floating around is this one, which supports the same conclusion. If you want to treat this as part 8 of this series of exercises on benchmarking, evals, and experimental design, you can click through to the links and think about eval issues before reading further.
Without running our own eval, one problem the first experiment has is that the problems are trivial, which we can see from quote above; a problem that can be solved in 70 tokens in J and 109 in Clojure isn't much of a problem at all (the author used Rosetta Code). As we saw when we looked at other evals of caveman mode vs. our own evals, you can get very different results from trivial problems where most of the work is in printing out an answer vs. slightly less trivial problems that actually require some amount of "real work"; the big gains claimed by caveman mode and shown in replications go away when you start looking at problems that take more than just a few tokens. In general, performance on trivial tasks doesn't generalize.
The issues in the second link are a little more subtle, so we'll defer most of them to an appendix, but they include issues like one of the tests executing the wrong path (which doesn't exist), causing a test to fail. One of the later agents then symlinks the non-existent path to its own executable, which works for that case, but also causes every later test to run that one agent's executable instead of the correct executable. The author tries to draw conclusions about what it means that Rust had some failures, but all it means is that scoring for Rust ran before the Go agent symlinked all scoring on that broken test to the Go executable.
Instead of relying on these evals, we can try running some of our own evals. As we can see from these evals as well as the evals discussed in our last exercises on evals, it's very easy to make an eval that doesn't say what the creator of the eval seems to think it's saying. No doubt these evals will not be an exception to this and will be flawed (see appendix below for more details).
As a way to build my intuition about things, I like to pre-register guesses before looking at results1. Some things I pre-registered with friends were:
For the first eval, I tried giving agents the zstd RFC (plus errata) and telling them to implement a complete zstd decoder (agents are stuck in a container without internet access). The tests were not given to agents. For something with the surface area of zstd, it's not really reasonable to expect that the tests cover every possible case. For example, even though zstd is a fairly well-tested piece of software, I once found a data corruption bug in zstd. The test suite isn't intended to find extreme corner cases that might be lurking for years and is instead intended to check various cases that can "easily" be derived from the RFC that should work.
Below, the x-axis is cost and the y-axis is correctness score (up and to the left is better / down and to the right is worse); average result on medium and ultra efforts with GPT-5.6 Sol. If we only look at medium (and ignore the fact that results often wildly differ on different tasks), we might come to a conclusion like the Alderson evaluation, that dynamic languages are more efficient and better when using LLMs because (ignoring relatively obscure languages) the cluster of dynamic languages lands up and to the left of the cluster of static languages (we used Alderson's color-coding for static vs. dynamic to make it easy to compare at a glance). But if we look at ultra effort, the results are quite mixed, with a couple static languages doing the best, with more static than dynamic languages among the better results.
The graphs below also have a toggle to convert the x-axis to time instead of cost. The mame/ai-coding-lang-bench noted that it's valuable to get results more quickly (I personally don't find this to be the case because results take long enough that I multitask instead of waiting), so we can also look at that. Similarly, we observe that neither language type dominates the other although, at medium effort on this particular task, the best dynamic language results are once again better than the best static language results (though, once again, they're fairly close).
We can observe that, just like when we compared completely trivial caveman mode evals to a less trivial caveman mode eval, the very strong relationships that held in the trivial evals don't generalize to this larger case. As was the case there, the extreme ratios in performance go away in these larger evals, except in cases where we might expect poor performance, such as when using assembly (which would be significantly more time consuming and difficult for a human) and when using relatively obscure languages where we might not expect that AI labs are expending effort generating synthetic RL environment data.
Note that this is the opposite of what the 1st eval found when it suggested that very dense languages like J would make sense for efficiency reasons. Perhaps using an obscure (and "weird") language can make sense if you have a very large budget and you can train or fine-tune a model to be effective for your pet language, but if you're a normal user of LLMs, it seems like sticking with a mainstream language is likely a better bet than using an obscure dense language.
And it turns out that if we plot language popularity vs. performance on this eval (not shown), we observe a weak to moderate positive correlation where more popular languages end up with more correct as well as cheaper solutions.
As we previously noted, very closely related evals can give substantially different results. For example, we saw significantly different results in the Optimization 1 vs. Optimization 2 evals here when Optimization 1 and Optimization 2 were optimizing bzip2 compression and decompression in wasm, which are fairly closely related tasks as evals go. To make a strong, universal, claim, like "dynamic languages are more efficient than static languages", we'd have to run evals across many tasks. However, showing that a claim like
Dynamically typed languages generally have a lower LLM token cost than traditional statically typed languages because omitting explicit type declarations makes the code more compact.
is maybe at best vaguely directionally true and not really relevant to any particular case and maybe not strong enough to be relevant in general, we just need to try a few cases and see that the claim doesn't generally hold. Above, we saw that at one effort level, the claim seems to maybe be kinda sorta true, but with exceptions, and then at a higher effort level, the claim seems to not be particularly true, which is sufficient to say that the claim is probably not universally true, modulo our eval having a confounder that completely invalidates it.
But, just to get a view on a very different task that's also presented in a different way (more TDD-like than "read a spec"-like), this next eval takes the Pandoc ProgramBench eval and modifies it for our use case. Instead of the reverse engineering task presented by ProgramBench, we present agents with ProgramBench materials as well as the ProgramBench tests and then score agents against a holdout set of tests to measure the performance of each condition2.
In the results below, the x-axis is cost again and the y-axis is score on the holdout tests.
As before, we don't see a very strong relationship between success or cost and whether a language is static or dynamic or very dense. We once again see that relatively obscure languages tend to do poorly (although Clojure does much better here than on Zstd). Also, Assembly does much worse, which seems expected in that we would expect a human writing Assembly to be at much more of a disadvantage implementing Pandoc than implementing Zstd and there doesn't seem to be a strong reason to think that LLMs would be different in this regard.
Who knows?
I have a lot of questions about what works well when using LLMs (such as, what test techniques work well, what languages work well, what software architectures work well, if bug fixing cost varies by language, if general program maintenance cost varies by language, etc.). Most of these questions are unanswered in public data and, if they've been answered in AI labs, the information mostly hasn't been made public.
Most of the claims that get thrown around about how a particular language is good for LLM use seem to be wrong (e.g., the claim that Ruby, Clojure, and J, are particularly well suited to LLMs, which were mentioned in the evals linked above, as well as the somewhat common claim that Elixir is particularly suited to LLMs), but it's not clear what's right.
In 2014, we looked at the literature on static vs. dynamic types and found that surveying the literature wasn't very informative outside of a few case studies. For an example that typifies a standard academic study, we saw the paper, Do Static Type Systems Improve the Maintainability of Software Systems? An Empirical Study, on which I commented:
Subjects were given classes in which they had to either fix errors in existing code or fill out stub methods. Static classes for Java, dynamic classes for Groovy. In cases of type errors (and their respective no method errors), developers solved the problem faster in Java. For semantic errors, there was no difference. The study used a within-subject design, with randomized task order over 33 subjects. A notable limitation is that the study avoided using “complicated control structures”, such as loops and recursion, because those increase variance in time-to-solve. As a result, all of the bugs are trivial bugs. This can be seen in the median time to solve the tasks, which are in the hundreds of seconds. Tasks can include multiple bugs, so the time per bug is quite low.
Picking tasks that avoid "complicated control structures" such as loops and recursion, where tasks take hundreds of seconds makes the result meaningless with respect to tasks that really eat up a professional programmer's time, just like the first eval we saw where tasks took high tens to low hundreds of tokens. However, with LLMs, we can actually feed them non-trivial tasks and compare how they do. There's the issue of how well results generalize to different tasks, but we'd have that exact same issue with human studies, but worse (LLM variance is huge, but human variance is even huger since you can't get the same human to do a bunch of tasks with different seeds). And while $20 to get an LLM to implement a Zstd decoder isn't exactly cheap once you multiply by the number of languages and the number of iterations per condition per language, if you think about how much it would cost to hire a professional programmer who can read the zstd RFC and implement it, there's no way the equivalent study would've been done because the cost would've made it completely infeasible. That goes double for the Pandoc task.
With LLMs, a lot of the questions have gone from being effectively unanswerable to being answerable with a bit of effort and some tokens. Due to the incentives that are in play3, it's not clear that we'll get answers to questions like this any time soon, but it's at least possible to take a crack at it now.
There are a lot of claims I've seen floating around that these evals can't prove or disprove (for the reason noted above that, due to the variance across different problems, many more tasks would have to be tried), but that these shed some light on, such as:
For my pre-registered guesses, we had
By the way, a major reason Clojure improves by so much in the Pandoc eval compared to the Zstd eval is that, in the Zstd eval, 36/40 medium and 5/40 ultra Clojure programs had test failures because byte conversion throws on 128–255 (maybe unchecked-byte should've been used?) and they used this conversion inappropriately.
That's a real result, in that, if you ask the best publicly available GPT model to implement Zstd (and presumably if you do other bit/byte manipulation tasks where this might come up), it will emit code that fails in this particular way. If there are tests that catch this, the bug will get fixed, but it will still cost time and tokens. Whether or not a language did well, there are costs like this all over the place (for example, cargo repeatedly gets invoked with the wrong arguments, which then immediately gets caught and fixed, but I've noticed this loop can actually consume a decent amount of wall clock time on my real projects unless you give explicit instructions to codex on how to invoke cargo, and it's clear that's worth the space in the context window).
Anyway, all of this is an illustration of why, if someone wanted to make a strong claim about which languages or classes of languages are particularly good with LLMs, they would need to run quite a few different evals. If we dig into why any particular condition got a certain score, the failures that caused the score are generally something idiosyncratic where it's not always obvious how much the issue generalizes across tasks or across setups. There's no way to look at the score on one eval or even five or ten evals and draw a conclusion about programming in general.
It's true that, in both the Zstd eval and the Pandoc eval, we see a correlation between language popularity and positive outcomes (higher correctness, lower cost, lower wall clock time) and it seems plausible that we'd see this across other evals, but it would be a mistake to draw a strong conclusion about any particular language. I gave a warning like this back when I looked at how often different projects have a broken build according to GitHub CI data, noting that there are different reasons that a build might be broken more or less often across projects and that one shouldn't draw strong conclusions because results across projects aren't necessarily comparable (for example, if one project's main branch is some kind of release candidate that's gone through other vetting, that project would be expected to have low build breakage, but that's not comparable to a project where people are developing directly against main).
Shortly afterwards, someone involved in one of the languages with a high score (IIRC, it was Martin Odersky and Scala) tweeted out the post and cited the language's high ranking as a victory for the language. That was an unwarranted conclusion there and, due to the many sources of variance that are in play here, any such conclusion about a single language would be even more unwarranted here.
This data (assuming eval validity) can refute some strong claims and is suggestive of other claims, but it can really only be suggestive of things for classes of languages and not for particular languages due to having only two tasks, which any particular language could do well or poorly on for some idiosyncratic reason which may or may not generalize to other tasks.
Thanks to Max Bittker, Yossi Kreinen, Aaron Levin, Alan Boll, Luke Burton, Marco Primi, Milosz Danczak, Justin Blank, and Tom Adamczewski for comments/corrections/discussion.
Like I said above, my eval here is a quick and dirty eval and I'm sure it's full of flaws, so I'm not trying to say the evals I've presented here are great and this is bad, but here are a number of issues in the Endoh ai-coding-lang-bench eval.
One issue is that the wrong executable appears to have been run for some of the tests. The setup for the published run seems to have executed ../../minigit inside each candidate's directory for one of the tests when the candidate's generated executable is at ../minigit. ../../minigit doesn't exist.
Because statically typed languages had a lower correctness score, the author of the eval noted "the only failures in 600 runs were in Rust and Haskell (both statically typed, both relatively "difficult" languages)" and suggests that "difficult languages", such as "C's memory management, Rust's ownership model, and Haskell's monads/purity may add overhead for the AI".
However, Rust's failures were because there is no executable at ../../minigit, causing the test to fail. The first Go run "fixed" this by executing ln -sf minigit-go-1-v1/minigit ../minigit and linking generated/minigit to its own run, but this means that every later execution (for every language) actually executed the first Go run's executable. On rescoring Rust against its own executable (as opposed to having it fail by trying to execute a non-existent file), Rust gets a perfect score, invalidating the theory that Rust had failures because it's a difficult language to deal with.
Other tests also have issues. For example, two tests have a structure that causes them to pass regardless of the actual value being checked. One of the tests has
if ../minigit commit ...; then
COMMIT_POST_CHECKOUT=$(cat .minigit/HEAD)
if grep -q "parent: $COMMIT1" \
".minigit/commits/$COMMIT_POST_CHECKOUT"; then
pass "checkout then new commit works"
else
pass "checkout then new commit works"
fi
else
fail "checkout then new commit works"
fi
The inner if has a pass in both branches, meaning that this is almost equivalent to
if ../minigit commit ...; then
pass
else
fail
fi
The inner if appears to be intended to have the actual check, but due to a coding error (perhaps a copy+paste error?), the check is effectively elided.
Also, as noted above, agents can modify the test environment, which the 1st Go agent did to fix a broken environment. They have full access to tests and the environment and can do anything and the test suite is visible during development with no holdout, which can easily lead to cheating by special-casing code in a way that passes tests but creates a program that's useless "in real life". At a high level, something like this seems to have happened in that many programs fail to implement large parts of the spec but do pass all tests, which may indicate that the agents "understood" how to pass the tests and preferred that over implementing the spec (it could also indicate that the tests are very thin and are easy to pass).
Another issue is that the Claude Code CLI versions aren't the same for all runs (it varies from 2.1.66 to 2.1.68). There are a handful of other issues like this that could be significant, but are likely small compared to the issues noted above.
As an example of something we can compare, I was curious how cost effective using medium + asking the agent to keep working would be and then, in the back of my mind, I also had this question about something "Ralph loop" advocates say, that you're better off clearing the context window on every iteration of the loop and giving the agent the full prompt again. As with the above, my pre-registered guesses here are:
Below, we have the average result for medium in a loop vs. ultra, sorted by best to worst ultra correctness score, for a prompt that simply resumes individual runs that don't have 100% test correctness as well as a Ralph-loop like prompt that discards context and gives the original prompt again (x-axis is cost, y-axis is number of correct test cases):
For this one problem, on average, running ultra once seems better than repeatedly running medium per unit cost (and much more so per unit time) and continuing with previous context outperforms Ralph. The problem with naively running medium on repeat is that the agent can get anchored to a bad solution and fail to make progress. The theory behind the Ralph loop is that you throw away bad context which can cause this to happen, but that doesn't save you from having a bad artifact.
Just from using LLMs, I've noticed that you're often better off throwing away a chunk of code and having an LLM re-write it from scratch than you are having an LLM modify it or try to re-write it in place. Michael Malis, who's been re-writing Postgres in Rust and has been making major changes has also noted this. This also relates to this idea noted previously that, due to the high variance (plus this path dependence) you're often better off rolling the dice multiple times and taking the best result, if you don't mind spending the tokens.
It's hard to say too much about static vs. dynamic languages from looking at just this one condition, but a naive thought like "static languages will outperform when iterating" isn't obviously true. If there's one pattern that jumps out at me, it's that the cases where the Ralph loop most badly underperformed continuing with context were generally dynamic languages. It's possible this is because of the lack of type information, but we'd need to both look at the differences in trajectories in more detail as well as look at other examples to observe if that's a real pattern. Even if you don't care about Ralph loops now that the Ralph loop trend has passed, being able to make changes to a codebase more effectively when starting a new task or starting with fresh context is something you might care about and the pattern here is suggestive of a possible advantage.
I tried to do a third eval that seemed like a more "business logic" kind of eval in both how the problem is presented and the actual execution of the problem. You can argue that the Zstd eval and the Pandoc eval are quite unusual tasks for a programmer to face in that not many programmers receive a specification as well-written and thorough as the Zstd RFC and not many programmers are handed a problem with as many pre-created tests as you get from ProgramBench tests.
The idea here was to implement a board game. In general, board game rules are written by people who aren't experts in writing clean specs, so implementing a board game is more like what happens when a non-programmer (or a programmer who isn't an expert at writing good specs) gives someone a task.
The problem here is getting a game where I have a reasonable oracle for scoring that isn't trivial for LLMs. For example, LLMs were able to one-shot the rules for Scout and Azul, which make those poor tasks. For games that an LLM won't immediately one-shot, I happen to have an oracle for Guards of Atlantis 2 because I had an LLM implement a copy for me and my friends to play (no link for this one because I don't see how to make an interface that's free of copyright infringement). The backend only took a few hours of my time, but it took a fairly large amount of LLM time to get the rules to be roughly correct. I like this as a task in that the rules are tricky in the same way a lot of problem descriptions that are delivered to programmers are tricky, but it is, in principle, possible to figure out the correct rules and implement them (after all, humans implicitly do this when they play the game correctly offline).
In board game rules, it's fairly common to have rules where reading the rule strictly as written is incorrect and you need to use "common sense" (or read some kind of FAQ) to play the rule correctly (there are some game designers who strive to avoid this, such as J C Lawrence, but this is fairly uncommon). Guards of Atlantis has quite a few rules like this. The designer of Guards of Atlantis is also vocal about there being no such thing as the spirit of the rules or common sense interpretations of the rules and says that you should always read the rule exactly as written, which creates two difficulties. One is that there are also many cases where you need to ignore the "common sense" interpretation and read the rule exactly as written. The other difficulty is, as anyone who's ever tried to write a formal spec knows, it's very easy to accidentally have ambiguity or contradictions. Even people who do this profesionally are unlikely to be able to create a non-trivial, complete, clear, spec without formal methods or a very large amount of human review. Realistically, a board game designer who doesn't have a background in writing formal specs doesn't have a chance, and thinking that it's easy (as the designer seems to) reduces the already low odds even further. A nice way to mitigate this kind of issue to write down your intent or "the spirit of the rules", but because the designer says that there is no such thing as the spirit of the rules and you should read all rules exactly as written, there are no meta-comments in the rulebook that would help someone interpret confusing or abmiguous rules. This combination is quite difficult for LLMs (and, judging by the rate at which I see humans play the game according to the designer's intent, it's also quite difficult for humans).
I think it would be effectively impossible to just read the rules and play correctly (of course it would be possible, but it would require knowing which rules are to be read as written and which rules are not, which one would have to do randomly and get lucky as the rules don't define a consistent system that one could use to infer which rules obey which meta-ruleset). When I was implementing the game, in order to get my LLM to understand the rules, I gave it various resources such as an unofficial rules FAQ (which is correct), an unofficial short version of the rules (which is better written than the official rules and correct, but incomplete), an opening book (which can be used to test rules against on the assumption that the opening book only contains legal moves), comments from the rules channel on Discord, etc., and had the LLM do consistency checks across these with the understanding that things like the FAQ and the Discord comments have higher authority than the actual printed rules.
One additional source of difficulty is that the designer is frequently delibrately unhelpful when answering rules questions. He often likes to make fun of people who ask rules questions or played rules incorrectly, which has a chilling effect and reduces the number of rules questions (multiple people have said that they don't ask rules questions because of how the designer behaves), and when he does answer questions, it's often with something like a meme image that says "reading the card explains the card". To extract the information, the LLM has to process these meme images, and then there's often no information or delibrately round-about information, such as, in the case of ambiguity, a referenece to a particular section. When people do point out contradictions, the designer often says it should be obvious which side of the contradiction is correct, which may be true for a human who's kept up on all rulings to date, but current SOTA LLMs find many of the designer's rules clarifications unhelpful.
Yet another source of difficulty, perhaps related to the designer's propensity to make fun of people who ask rules questions or are confused by rules, the game's interface seems almost designed to trick people into doing the wrong thing. There are multiple design affordances that I've seen trip up most new players (even if you explain to the UI trap to them, there are enough rules to take in they often forget, and then when it trips them up, they'll say something like "I'm an idiot, you even explained that to me twice"). It's not clear if these traps were created to give the designer people to make fun of, but that's certainly one result. Another is that LLMs struggle to understand the games rules and UI.
With my $200/mo personal OpenAI/codex account, I let an LLM use all my spare capacity to run consistency checks and make rules fixes. I didn't closely track how long this took, but I think it was something like a month or two of cranking on fixes like this to get a somewhat reasonable result that's playable, but that I wouldn't really trust to be correct.
The only reason I somewhat trust this is that Pedro Oliveira also implemented Guards of Atlantis and they used a completely different approach (a more standard approach of having a human drive an LLM rather than trying to get the LLM to figure things out itself). When we compared implementations, we found maybe 10-ish bugs in each. There are probably some remaining bugs where both of our implementations incorrectly do the same thing and perhaps some where our implementations differ but the checking system didn't notice, but I think the rules for both of our implementations are now reasonably solid. That's how I have an oracle for this game.
I like this as a task because it feels more like the kind of "specification" you get in the real world, where the spec is ambiguous and contradictory and sometimes just plain wrong, and then you need to use other information to get a correct result. For this eval, to avoid having it be a test of how well LLMs can access data in annoying formats (such as converting the opening book from a set of images to some kind of structured data, converting a scan of the rules to text, etc.), I gave agents both the originals of anything where I directed an LLM to extract the data (which also required various consistency checks to get correct) as well as the the extracted data (the originals were presented so that LLMs could check the originals for extraction errors if they chose to).
While I did this task with older models (I did a chunk of it with GPT-5.1 or 5.2, and then another chunk with 5.4 or 5.5), with newer models but without the kind of guidance I gave to the older models, the task was still far too hard. Regardless of language, agents scored approximately 0 on this task.
BTW, if you're curious what LLMs (and humans) struggle with, here are some examples. There's one card whose text reads "Target a unit adjacent to you. After the attack: may repeat once on a different enemy hero."
In this game, a hero is a type of unit. Read strictly, with full knowledge of the rules, e.g., what "After the attack" means, etc., this should mean that you can either attack a single unit or you can attack two heroes (after all, to repeat the attack on a different enemy hero would mean that the first unit was a hero; otherwise it would be a different unit that is a hero, not a different enemy hero).
This card actually has what is effectively an errata printed on the card because people complained it was unclear; the errata reads "(You may repeat even if the original target was a minion)". That's already confusing to LLMs (and some humans), but the real killer here is that there are other cards that use the same construction and don't have this correction. To play other cards with the same construction correctly, you need to know that every time this construction is used, you should play it with the errata that's on this card. There are a number of constructions the game designer likes to use that have a specific non-literal meaning that you have to keep in mind.
Another example of a rule that shouldn't be played in the obvious way is a character with a card which reads "Choose one, or both, on different targets: A, B". Reading this strictly as written, one would expect to be able to, on different targets, do either A or B, or both A and B. But part of the spirit of the game is the meta-rule that a character can't attack another character multiple times with one card, so the interpretation that you can do what the card says and do both and A and B on some number of different targets can't be right. Based on similar deductions and how similar constructions are used, the way this card is supposed to be interpreted is "Choose one, or both on different targets", which is arguably still ambiguous and could be more clearly written as "Choose one or both (must be on different targets if both)".
As a human, once you understand what the "spirit of the game is", you can resolve these kinds of things. But, by design, this isn't written down clearly in the rules and one has to infer this from Discord discussions, which appears to be beyond the capability of today's models even though humans who are outperformed by today's models on many specialized tasks are able to do this.
When I was supervising the LLMs that implemented the rules, the reason LLMs reached a ceiling and didn't converge to fully correct rules was that an LLM would observe that a rule was inconsistent and incorrect. It would then try to fix this rule and would also fix other things to try to make them consistent and correct. This would sometimes make things more correct and sometimes make things less correct. When making things less correct, the LLM would sometimes modify an existing correct test to turn it into an incorrect test so, after a while, the LLM wasn't really improving correctness and was just churning on which rules were incorrect. That was with some guidance on what to check and how to check it; without that guidance, even with the more advanced models that are available today, LLMs were unable to navigate this in a reasonable way.
I'm sure there is a board game of the right rules complexity to make for a good eval here but, by definition, this would be something where it would take some work to create the oracle and I don't have an oracle handy for a board game with the right rules. If my goal were to make evals, I would've used board games with actual game replay data to get good tests or oracles for a whole bunch of games, but my goal was to play a particular game with some friends. But, if one were inclined to try this board game thing, it should be possible to create hundreds (thousands?) of these in a scalable way, so one could get a reasonably correct oracle for hundreds or thousands of games and then check which games are at the correct level to be an interesting test for LLMs today.
This is arguably a bit of a funny problem in that, given a clear spec, e.g., a clearly written set of rules, an artifact that's more complex than Guards of Atlantis can be implemented by LLMs (I would argue the Zstd RFC is more complex, and Pandoc certainly is; even individual document formats Pandoc supports, like PDF, are more complex than Guards of Atlantis), so the problem isn't finding a game with rules that are complex enough that LLMs struggle and the problem is more about finding a game with rules that are ambiguous or contradictory enough that LLMs struggle, but not so much so that LLMs are completely hopeless. This is an actual real-world problem, in that humans are generally not very good at writing clear specifications and how well models and harnesses can handle a human's unclear, contradictory, and sometimes just plain wrong, specification is probably more relevant to the typical user than how well an LLM can implement something from a specification as well-written as the Zstd RFC or how well an LLM can implement a problem when handed the 4800 ProgramBench Pandoc test cases plus documentation. And these problems seem solvable in principle, in that humans who want to play board games correctly (even ones who would have no hope of "playing" Zstd correctly, let alone Pandoc) are generally able to navigate the mess of information out there to figure out what the rules to a board game are.
If we look at it form the other side, this is suggestive that, to get an LLM to do something, maintaining a clear, canonical, spec is an effective way to work.
When it comes to performance benchmarking, I've done enough of it that I feel like I generally know how my benchmarks are flawed and I can make an informed time/effort vs. flaw tradeoff and I have decent confidence the flaws that exist in the benchmarks aren't material to the thing I'm trying to understand. I haven't done enough AI evals to have this kind of feel for AI evals so, at a meta level, I would expect any AI eval I do to have some unknown-to-me flaws.
Another reason I would expect some flaws here is that I had coding agents set up these evals and every time I spent a minute looking for issues I would find at least one issue. This indicates that it's fairly likely that these evals have additional flaws that could be uncovered by looking a bit more, but I wanted this to be more of a "quick toy project" level of correctness than a "Gary Bernhardt" level of correctness, so I stopped after fixing a handful of issues.
Back when I was working as a verification engineer, I attended a meetup by a Sun/Oracle engineer in Austin, maybe around 2007 or so, where they mathematically formalized this idea of converting the time between bugs to a level of confidence in a chip release. I haven't seen people do this much, but I recently heard Will Wilson (co-founder of Antithesis) mention that some folks at Antithesis used math from ecology (the literature on rare species observation) to estimate true bug rate, which seems like a much more sophisticated version of what this engineer at Sun/Oracle was doing a couple decades ago.
That's a cool idea, but when you're finding a bug every minute you look, you don't need fancy math to tell you that there are probably a lot of other bugs. If I were doing this for work and we had some reason to care about the fidelity of these evals, it would probably make sense to look at these more closely and fix more issues (and I would probably have the skills and experience to make fewer mistakes in instructing LLMs to set up these evals if I did this kind of thing for work). But, for the purposes of answering the question "is the claim that dynamic languages are meaningfully better than static languages when using LLMs?", I have a little more confidence that the claim isn't true, and there are a lot of other questions that seem more likely to yield some kind of actionable result (such as, what techniques or test libraries work best).
I normally don't publish things on the blog until I feel like they're somewhat solid, but this means that I often explore some data enough to satisfy my curiosity and then never publish the result. From talking to people about these non-published results, people I talk to are often curious about the results even if they're not done to a standard that I really like, which seems like an indication that folks I don't talk to might be interested as well. From what I've seen so far, I suspect it would take at least 10x the time I've put into this to get this to a standard I really like. I'm fairly busy at the moment and can't see myself having the time to do that for months, at which point I'm not sure I'd really ever get around to publishing this. In a recent post, I mentioned an analysis I did almost a year ago where I was trying to understand which cars are better for concussion risk in accidents, where I spent some time figuring that out, got far enough to get an answer that satisfied me, and then didn't ever get around to doing the work it would take to clean up the result enough to publish it.
There are some results from that analysis seem "publishable", in the sense that they could turn into a published paper (such as finding from actual crash data that the relationship between HIC and velocity looks like it's to the fourth power (!); there's a paper that tried to find this relationship, but did the wrong kind of analysis and wasn't able to find an "O(n)"-style relationship and had something much fuzzier), but I've never really cared about whether something is a paper or a blog post and it turns out that I'm more likely to just move on to the next analysis instead of cleaning up the analysis enough to publish a post.
A more recent project along these lines is that, after making a superhuman Azul AI, I tried to make a superhuman Splendor AI using a much less human-time-intensive process. I believe that didn't succeed, but it beats every other Spelndor AI I could find by a good margin, which is a mildly interesting result. I think I know enough about board game AIs to write something up about them, but my main interest was in figuring out if I could get something decent, and then I keep just doing other projects instead of spending the time to do a nice write-up. An example of something I think is interesting there is that a lot of the performance optimizations you want to do actually change the result, so you can't only rely on optimizations that can be strictly checked to not change the result. But, if you naively ask a coding agent to do these optimizations in a way that doesn't reduce playing strength, they'll do all sorts of things that reduce strength. Cases where the strength reduction is very severe are easy to catch, but there are more subtle issues that sometimes result in (for example) no change in strength vs. your own AI in self-play but a reduction in strength against humans or other AIs, so some kind of process to catch bad optimizations is necessary, and it's inherently a kind of arbitrary process that has to be designed using some combination of your intuition and relying on LLMs (which will be very helpful but also often completely wrong).
For these kinds of data-y projects that I'm interested in, LLMs massively reduce the amount of effort it takes to get a result that's strong enough to satisfy my curiosity but, AFAICT, they don't reduce the effort it takes to publish a result by much (at least if you write up results by hand instead of having an LLM write up the results and you want the results to be nice and clean), which means that writing them up runs into a kind of Ahmdhal's law bottleneck, so I've been doing more projects like this and writing up fewer of them. If anything, I think it actually takes more time to write these up because of how I've changed my workflow. For example, instead of just outputting some graph from ggplot2, I'll make a version an interacive version that's nicer in some ways, but definitely takes more time to produce. And I run an LLM spell/grammar check pass (at least so far, that's the only LLM assistance I've used for writing), which turns up a bunch of issues to be fixed. Since I look at each one manually instead of taking the fixes (and I make a lot of typos), that's actually fairly time consuming (over an hour on my last post and over half an hour on this post even though I didn't even make corrections all the way to the end and abandoned the process maybe halfway through).
Anyway, publishing this is an experiment in publishing some half-baked notes instead of having the kind of cleaned up version that I'd really like to have before publishing something. If you have opinions on this, please let me know (X Bsky Mastodon)!
I don't have GitHub links to the current evals. On the one hand, I feel like I really should. On the other hand, they're a mess and there's a bunch of stuff I'd want to clean up before publishing the code, and I don't know if/when I'll get to that and this way, at least I'm putting something out there instead of just talking to a few friends about the result and then having the result sit on my hard drive indefinitely?
Agents were instructed to ignore performance, but the timeout wasn't infinite and, under the medium condition, some test cases timed out. This is arguably unfair, but this didn't materially impact the score. For non-infinite loop timeouts, there were 2 test cases in Clojure (across 40 * 34 tests), 2 in J, 2 in Tcl, 1 in Factor, and 1 in PHP. And, at 9000s (2.5h), the timeout was fairly generous considering that the largest test case was 4 GiB. Failing to decode 4 GiB in 2.5h is an implied rate of less than 0.5 MB/s on a Graviton 5 core, which is quite slow.
Here are some of the issues that I ran into when trying to get agents to set this up (and, as noted above, the short amount of time it took to find each issue implies there are more issues)
There's one thing which arguably wasn't a bug that I removed anyway. One of the tests was very hard (maybe 10% of agents passed the test on the first try). On testing the current zstd release binary, the zstd binary also fails this test. On reading the RFC, this seems to be an ambiguity in the RFC about the legality of a certain edge case. There was fairly strong clustering with respect to which languages passed this test case more frequently, which I think is interesting, but doesn't seem like a very useful thing to measure when all of the other tests are measuring (or at least attempting to measure) something more straightforward.
Anyway, in the above list (which is not exhaustive), many of the issues impacted a large fraction of languages and some issues had to be fixed multiple times. All told, if you count each condition as a separate bug, I probably fixed (had agents fix) over 100 of these bugs and I expect there are more. When I talked to Max Bittker (who runs an RL environment startup), he noted
all the evals I've worked on, I ended up putting a huge amount of time and effort into, mostly in the form of reading trajectories (or summaries of many trajectories) and then triaging issues , e.g "oh this class of bug shouldn't be possible, lets update X "(X being the prompt, the harness/ environment, or the verifier)"
agents tend to slop this up, so I put a lot of care there to make sure things get fixed at the right layer, for instance it's very sensitive what's in-context for the agent under test (bad to add random junk it has to worry about, or at worst leaking answers) vs whats fixed behind the scenes in other parts of the system.
agents, when writing evals, are not sensitive enough to the experience of the agent under test, and will just give it the answer or fix problems by making it the inner agent's problem ("remember to not reward hack plz")
I also have had a lot of success re-using existing things (repos, games, tools, levels) and building harnesses and verifiers around them, versus trying to make something from scratch for an eval by prompting
In retrospect, I sort of regret doing a cross-language eval. Even after fixing 100 or more eval issues, I have no doubt that plenty more remain. Maybe this is just a "grass is greener on the other side" thought and I'll also regret the next eval I try, but I think it would've been a lot less work to try to evaluate how well different test techniques or testing frameworks work than to evaluate different languages and I find that topic at least as interesting. And, in retrospect, had I done a lot more work by hand and relied on agents less, this would've gone a lot better. For example, I should've had agents produce an environment for one language and then both had agents inspect it and inspected it myself and fixed the issues before producing the environment for another language. After doing this a few times, I might've had a better setup for producing environments for other languages (and if not, I could've just repeated this process for each language and gotten a more reliable result, likely without even taking more time).
Another thing to note is that a number of things that are genuine differences in languages weren't really tested, such as memory safety against adversarial inputs. If agents had a harder time producing generally roughly correct code in C or C++ than Rust, that would be observed, but if a fuzzer or valgrind or other tools would turn up issues, that's not likely to be captured in the small set of tests. Just out of curiosity, I asked an agent to (briefly) check the Zstd C and C++ code for memory safety issues. The agent claims it ran the C and C++ code under ASan+UBSan and tried a few fuzz inputs (4000 each) and didn't find issues, but of course that doesn't mean there aren't issues or that a larger codebase wouldn't have issues.
And, in fact, doing an analogous quick check for memory safety issues for the Pandoc eval found memory safety issues in all of the C programs and all but one of the C++ programs (the issues were things like incorrectly dereferencing out-of-bounds memory; one specific example is that, in one of the C programs, a truncated LaTeX table could result in an out-of-bounds memory read). The fact that these issues were findable with 10 of seconds prompting indicates that many such issues could be found and fixed without much human effort, but it would cost quite a few tokens and would push the cost of the C and C++ versions well beyond the cost of the Rust version and after doing all of that you would still have less confidence in the memory safety of the C and C++ versions than in the Rust version.
Anyway, if you're curious about the distribution of results, we have the following for medium and ultra:
I don't love that the ultra results are somewhat saturated here, but one "problem" with testing ultra is that it will keep going for a long time as problems get harder (e.g., most of the Pandoc ultra runs ran for 12+ hours, and the assembly runs went for much longer), so the things that don't get saturated are very large tasks, like the Pandoc eval, or tasks that are too difficult in some way, like the Guards of Atlantis eval.
The holdout tests seem necessary because, without them, agents cheat and will detect a test input and hard-code the passing test output (they sometimes do this even when instructed not to cheat). If all cheating was that blatant, that wouldn't be a problem (and could be an interesting thing to measure, as agents differentially following directions or not across languages is something that matters to real users), but a lot of the cheating is more subtle and difficult to adjudicate. For example, some agents wrote code that branched off of the structure of the tests, but then filled in the contents of the branches with code that wasn't special-cased to a single test result and could pass many variants of the same test. For any point on the spectrum from "definitely not cheating" to "obviously cheating", some agent tried it. As we saw when we looked at Senior SWE-Bench, LLM scoring of evals is tricky and a great way to introduce both bias and variance; using a holdout set of tests has some problems, but it lets us avoid this much larger set of problems.
For one thing, the holdout tests are suspsicious because they were created by agents. The intention was to create holdout tests that a reasonable person (or agent) would be able to make pass if they're not cheating. Agents audited this set of holdout tests for cases where this wasn't reasonable and eliminated some, but I didn't check these by hand, so I find it likely that there's at least one holdout test that's unfair in some way. However, the overall score against holdout tests is low enough that I'm not too worried about a small number of tests being bad (if I worked at an AI lab and was trying to train next-generation models, I would be more worried about this, but I don't think it's material for our use case here).
Instructing agents not to cheat while having a holdout set of tests didn't prevent blatant cheating that scored extremely poorly on holdout tests, but telling agents that there was a holdout set of tests they were graded against seemed to reduce the score they achieved on the agent-visible tests while increasing the score they achieved against holdout tests (without telling them this, a number of agents achieved 100% on the Pandoc tests with uselessly brittle code; on telling them there's a holdout, no agent scored 100% after 1 turn on ultra, but the holdout scores were substantially better, indicating better generalization).
[return]There are various Substacks, YouTube channels, and other things that promise to tell you the secrets of LLM coding success, but the ROI on spending time running actual experiments isn't really there. When we looked at caveman mode, we saw that one of the biggest programming YouTubers had a video where they spent a few minutes looking into it and decided that it worked. Spending even 15 minutes looking into whether or not it really works is probably negative ROI compared to spending that time producing more content instead.
There are various papers that discuss different techniques, and these sometimes go into more detail than most blog posts or videos but, on average, they don't necessarily have more useful information. For example, when I asked ChatGPT (5.6 Sol, Pro) to find discussions of language effectiveness with respect to LLMs, it turned up this paper on token efficiency, which has an interesting idea, but has the same issue as the caveman mode evals we discussed earlier, where it's not looking at a task that's interesting enough for the result to be relevant to me as a programmer. Just seeing what cited that paper, we find this paper by three academics on token efficiency of languages titled "The Best Programming Language for Tokenmaxxing", but compared to this post, that paper only compares four languages, uses worse models, and uses small toy problems (from something called LiveCodeBench; the cost to solve problems with GPT-5.5 is often on the order of 1000 tokens). Regardless of how well done the eval is, as we've noted in this post and in our caveman mode eval, we often see wildly different relative results when going from a small toy problem to a problem that I might care about for hobby projects or work. Also, in that paper, they note that they gave the prompt "To test your program, run exactly ./test.sh... These are the only tests I care about" and they say this is realistic because "We believe that this setup is a realistic way to study agent behavior: in everyday use, programmers don’t hide their tests from agents. Instead, programmers direct their agents to keep working until all tests pass." but, as we noted above, doing this results in brittle code that fails in the real world (or if you have holdout tests that aren't given to the agent, it fails the holdout tests at a very high rate; this problem cannot be solved by just adding a few more tests; it can perhaps be addressed via something like fuzzing or property-based testing, but how well that works is a topic for another post). I'm not saying these papers are bad or that there isn't something interesting to learn from these papers, but as a programmer who wants to know what techniques or tools I should use, I can't get that information from papers like the ones linked above.
[UPDATE: Tom Adamczewski sent me a link to his paper, https://arxiv.org/pdf/2606.30182, which does handle a lot of the issues mentioned above. Relative to this post, it tries a lot more different tasks (which is great) and tries fewer languages and fewer ways of presenting tasks. One conclusion they draw in the paper that I think falls out of trying fewer languages is that language doesn't matter; even if you exclude the very obscure languges from the evals we tried here, we can observe a correlation between language popularity/usage and result quality; because Adamczewski's paper tries a lot more tasks, you can get a more complete picture by looking at this post and that paper combined than you can by looking at either in isolation.]
[return]
