GPT-6 Astra patched another AI's endless thinking

What this video covers

• The reported Qwen score increase came from a targeted repair to repetitive reasoning; about a third of edited answers still reached the length cap in the primary protocol.
• Gradient training was forbidden, while Python, practice answers and batched model-run feedback were allowed across many turns. Final evaluation questions were source-declared held out.
• The small raw-weight and no-rollout game experiments used different conditions. Their claims should retain their individual tool, input and baseline scope.
• Matthew Honnibal credits the tool-assisted engineering and disputes the broader impression produced by bundling the experiments.
• The smaller Gemma follow-up used its own setup and failed to establish an improvement. Test which failure a weight edit repairs before applying it to another model.

Transcript

Why did the math score jump?

0:00 An AI making another AI six times smarter sounds like instant genius. Astra's big score jump came from helping Qwen finish answers it kept repeating. In this video, what changed? What help did Astra get? And how far does the fix travel? Qwen went from about eight percent to forty-eight percent. The researchers tried a second model with almost no loops. We'll see how that went before the verdict.

0:22 It's Tuesday, November third, and this is The Daily Diff. Weight-space programming is the researchers' name for asking an AI to inspect or write the numbers inside a neural network. Those numbers are its weights. This MIT-affiliated team used OpenAI's GPT six Astra to edit a small Qwen language model, with ordinary training forbidden. Rikiya Takehi and his coauthors published the study on Monday,

0:44 asking whether an existing AI could reason about another network's internals. Their headline result is very clickable. On math five hundred, Qwen rose from about eight percent to forty-eight percent. Nearly six times the correct answers. Qwen kept thinking until it ran out of room to answer.

What was Qwen doing wrong?

1:00 In thinking mode, it often repeated its reasoning. Imagine a student filling the exam booklet with working until the bell rings. The marker still needs an answer. Most original answers hit the length limit. Astra located a repetition direction in two internal layers, then removed its contribution from selected weight columns. After that edit, more answers reached a conclusion. About a third still hit the cap in the reported test.

1:22 The exam booklet finally got a last line. Today, Matthew Honnibal, spaCy's creator and cofounder of Explosion,

What does today's critic dispute?

1:28 challenged the presentation. He says readers can combine the experiments' different conditions into an impression of one extraordinary technique. His objection concerns their combined story. We can check the conditions ourselves. The Qwen experiment gave Astra a Python notebook over the weights and practice

What help did Astra actually get?

1:44 problems with answers. Across many turns, it inspected internal activations and compared original and edited versions. The final budget allowed fifty batches of model runs. Each generation batch could contain hundreds of answers. The practice data came from training splits. The researchers say Astra never saw the final evaluation sets. Feedback helped choose the edit; held-out problems measured it afterwards. Sampling settings stayed fixed between original and edited Qwen,

2:09 making this protocol's score comparison interpretable. The first prompt gave it a helpful lab supervisor. Diagnose failures, find the responsible components, compare on the same problems, and test several strengths. It explicitly mentioned repetitive thinking as a repair that counts. Graduate students should negotiate for this onboarding package. That's an impressive repair.

How small was the actual repair?

2:28 Astra's targeted edit touched roughly a hundred thirty-one thousand weights. That's about sixteen thousandths of a percent of the model. The weight file changed directly. Gradient training stayed forbidden; the existing model did the answering after Astra's editing session ended. The gain also reached instruction following. I like the practical implication. A model can possess useful work that its failure mode keeps burying.

2:48 Fixing that failure can recover usable answers. Finishing counts. Honnibal gives the tool-assisted work credit as performing tasks a human machine-learning engineer could do. That's consequential for an agent. His complaint is that the small raw-number demonstrations and larger Python-assisted repairs invite a broader conclusion when bundled together. Take the tiny game networks.

Which experiments really had no tools?

3:09 In one experiment, Astra read their raw numbers and correctly identified which game each network had learned. Tools were forbidden. In another, it wrote new game-playing weights from scratch using the rules and source code, with no practice rollouts. Those are separate demonstrations with separate evidence. Astra's networks beat the baseline trained for a hundred million steps in all three games. Against the longer-trained baseline, it lost in Seaquest and

3:32 Asterix, while winning Breakout. Change the comparison column, and the victory speech needs editing. Conveniently, editing is today's topic. For a wider claim, we'd want improvements across models with different failures.

What would a more general claim need?

3:44 Track what the editor saw and what counted as success. Qwen's model card already warns about thinking loops. Removing this known obstacle has a particular starting point. To their credit, the authors publish the uncomfortable follow-up themselves. They explain that stopping loops lets Qwen finish answers it could already work out. Their logs include prompts, feedback and edits. We can examine the achievement and ask a harder question.

What happened to the model that rarely looped?

4:06 Now the second model, a smaller Gemma in its own follow-up setup. It rarely looped, so its failures were ordinary wrong arithmetic and invalid algebra. Astra tried targeted weight edits with practice feedback again. None moved its math score by more than about a point, and a gain wasn't established. The successful Qwen repair had an obstacle to remove. Gemma's arithmetic errors survived. That's the useful question for developers.

4:28 Which failure did the edit repair, and does your model actually have it? Test the saved weights on your own tasks before celebrating the multiplier. A repaired bottleneck can matter enormously while the next bottleneck stays exactly where it was. If you'd rather read this than hear me say it, the diff lands in your inbox every morning, free at the daily diff dot dev, link below.

Would I ship the weight-editing claim?

4:47 So, today's verdict is NEEDS REVIEW. I'd take the loop repair seriously and test it. I'd keep broader claims gated on success beyond that failure mode. The authors' own counterexample is the most useful line in the diff. And that's the diff for today. I'm Niko from Axrisi. Merge responsibly.

Sources

1. Weight-space programming studyRikiya Takehi and coauthors
2. November 3 critique of bundled experiment conditionsMatthew Honnibal
3. Pinned Qwen protocol and exact weight editsRikiya Takehi and coauthors
4. Actual initial Qwen promptRikiya Takehi and coauthors
5. Pinned smaller Gemma follow-upRikiya Takehi and coauthors
6. Original model-card thinking-loop warningQwen
7. Primary study discussionHacker News
8. Critique discussionHacker News
9. Cover portrait: Sam Altman, TechCrunch 2019 crop, CC BY 2.0; cropped, background removed, resized and flipped; no endorsementSteve Jennings/Getty Images for TechCrunch via Wikimedia Commons
10. Released checkpoint and attributed external re-runRikiyaT on Hugging Face; re-run credited to akhaliq
11. Portrait license: Creative Commons Attribution 2.0Creative Commons; Steve Jennings/Getty Images for TechCrunch portrait credit above