Claude Optimizes 30+ Scientific Models as ScienceIDE Scales AI

Claude Optimizes 30+ Scientific Models as ScienceIDE Scales AI

AI is moving beyond writing scientific code toward optimizing, validating, and learning from complex scientific software.

Anthropic recently reported that Claude optimized more than 30 open-source biomolecular models in less than four weeks, including widely used systems such as AlphaFold 3, Boltz-2, Chai-1, ESMFold2, and RFdiffusion.

The results are notable: Anthropic reported an average speedup of approximately 4× when minor numerical precision differences were permitted, and nearly 2× when exact bitwise output consistency was required. The associated optimization code was also released as open source.

More important than the raw speedups, however, is how the work was performed.

Most of the optimization was carried out by Claude under the supervision of two Anthropic technical staff members. Although the researchers understood the biological workloads, they did not have traditional backgrounds in GPU kernel engineering or inference optimization.

This demonstrates a potentially important direction for scientific computing: AI can increasingly act as an optimization engineer for existing scientific software, allowing researchers to run more experiments within the same computational budget.

But it also exposes a deeper challenge.

Can AI-generated scientific code fully exploit modern hardware while preserving scientific correctness?

An open-source project called ScienceIDE, released by the AItonomy Foundation on September 17, approaches that question from another direction. Instead of treating successful AI-generated code as an isolated result, ScienceIDE attempts to turn scientific codebases into executable, verifiable environments where AI agents can repeatedly solve tasks, receive feedback, and learn from validated trajectories.

Together, these developments point toward a broader transition from AI that merely generates scientific code to AI that can measure, optimize, verify, and continuously improve scientific software.

🚀 Claude Demonstrates Large-Scale Scientific Code Optimization

#

Anthropic’s experiment covers more than 30 open-source biomolecular models spanning protein structure prediction, molecular modeling, and related scientific workloads.

The optimization targets included models such as:

• AlphaFold 3
• Boltz-2
• Chai-1
• ESMFold2
• RFdiffusion

The reported results suggest that AI-generated optimization can produce meaningful gains across a diverse collection of existing scientific systems rather than only a single hand-selected benchmark.

Two performance measurements are particularly important.

When small numerical precision differences were allowed, Anthropic reported an average speedup of roughly 4×.

Under stricter bitwise-equivalent output requirements, the average improvement was closer to 2×.

The distinction matters because scientific computing often involves numerical algorithms where apparently insignificant changes in floating-point behavior can propagate through a computation.

A faster implementation is not necessarily an equivalent implementation.

Scientific optimization therefore has to balance at least three variables:

1. Performance: How much faster does the implementation execute?
2. Numerical equivalence: How closely does it reproduce the reference implementation?
3. Scientific validity: Does it continue to produce scientifically meaningful results?

This makes scientific code optimization substantially different from optimizing ordinary application software.

🧠 AI Is Becoming an Optimization Engineer

#

The Anthropic experiment is also notable because the humans supervising the work were not traditional GPU performance engineers.

That changes the potential economics of scientific software optimization.

Scientific researchers often understand the algorithms, physical models, and expected outputs of their workloads but may not have the time or specialized expertise required to optimize every kernel for modern GPUs.

Conversely, traditional GPU optimization requires deep knowledge of:

• Memory hierarchies
• Kernel execution
• Thread scheduling
• Data layouts
• Vectorization
• Occupancy
• Register pressure
• Memory bandwidth
• Accelerator-specific instructions

An AI agent capable of iteratively profiling and modifying scientific software could potentially bridge some of that gap.

Instead of asking an AI model to simply “optimize this code,” a more capable workflow looks like:

Profile → diagnose → modify → compile → benchmark → validate → repeat

That feedback loop is much closer to how experienced performance engineers actually work.

It also explains why a single-pass code-generation benchmark is insufficient for evaluating AI performance engineering.

🔬 PLUTO Shows How Much Optimization Headroom Remains

#

ScienceIDE’s development team encountered a similar issue while experimenting with PLUTO, a high-performance scientific computing framework widely used in plasma physics and astrophysics.

PLUTO has been developed for nearly two decades and supports a broad range of computational physics workloads, including:

• Fluid dynamics
• Magnetohydrodynamics (MHD)
• Relativistic fluid dynamics
• Relativistic MHD
• Multiple coordinate geometries
• Different physical modules
• Multiple numerical algorithms

The difficulty of optimizing PLUTO is not simply its source-code size.

Scientific numerical methods impose constraints that must remain intact during optimization. For example, Godunov-type shock-capturing schemes combine spatial reconstruction, Riemann solvers, and time integration. MHD simulations also need to maintain appropriate magnetic-field divergence behavior through techniques such as constrained transport or divergence cleaning.

These mechanisms encode scientific assumptions directly into the implementation.

A performance optimization that changes those assumptions can produce a faster program while invalidating the underlying simulation.

The initial GPU implementation left substantial performance on the table

#

The PLUTO experiment attempted to reimplement its MHD functionality for GPU execution across M1 Ultra, NVIDIA A100, and H100 hardware.

On an A100, the initial AI-generated implementation successfully executed the tested workloads and produced the expected physical results for the cases evaluated at that stage.

However, its performance was relatively modest compared with the team’s 128-core CPU baseline, delivering less than a 5× speedup.

The team then spent roughly one to two days analyzing bottlenecks with AI assistance and modifying the implementation.

The optimization work included:

• Data-layout changes
• Memory-access improvements
• Kernel restructuring
• Reduction of unnecessary data transfers
• Better alignment between computation and GPU execution characteristics

Without changing the underlying physical problem being solved, the optimized implementation reached nearly 3× the performance of the initial GPU implementation.

The implication is significant.

AI was already capable of producing working GPU code, but a substantial amount of hardware utilization remained unrealized.

⚙️ Working Code Is Not the Same as Optimized Code

#

This PLUTO experiment illustrates why scientific performance engineering cannot be reduced to code generation.

Modern GPUs expose enormous theoretical computational throughput, but applications rarely reach that peak automatically.

Performance can depend on questions such as:

• Are adjacent threads accessing memory efficiently?
• Does the data layout promote coalesced memory access?
• Should an intermediate result be cached or recomputed?
• Can multiple kernels be fused safely?
• Will kernel fusion increase register pressure?
• Is the workload sufficiently parallel?
• Is memory bandwidth limiting execution?
• Are synchronization points unnecessarily expensive?
• Are CPU-to-GPU transfers dominating execution time?

These questions require empirical feedback.

A better prompt can sometimes produce better code, but it cannot replace profiling and measurement.

The fundamental optimization loop remains:

Measure → understand → change → verify → measure again.

For AI agents, the challenge is therefore not simply generating code that looks efficient. It is developing agents capable of interacting with compilers, profilers, benchmark harnesses, hardware counters, validation suites, and scientific reference implementations.

🧪 Successful Execution Does Not Guarantee Scientific Correctness

#

Performance optimization introduces another critical requirement: scientific equivalence.

Anthropic’s own experiments demonstrate why this distinction matters.

In one test, Claude was asked to predict very large viral capsids and protein compartments exceeding roughly 31,000 to 70,000 tokens on a single eight-GPU B300 node.

The inference completed successfully, which represented a significant computational achievement.

However, the predicted structures were not scientifically correct. The resulting structures collapsed, suggesting that the model’s generalization did not extend reliably to contexts nearly two orders of magnitude beyond its training context window.

The lesson applies directly to scientific software.

A program can:

• Compile successfully
• Execute without crashing
• Produce numerical output
• Run substantially faster

and still produce scientifically invalid results.

Scientific validation therefore requires domain-specific reference criteria rather than merely checking whether the program exits successfully.

🧬 Scientific Equivalence Requires Domain-Specific Validation

#

The correct validation criteria depend on the scientific problem.

For an MHD simulation, relevant checks could include:

• Conservation of mass
• Conservation of energy
• Magnetic-field divergence
• Shock-wave position
• Wave propagation speed
• Stability characteristics
• Agreement with reference quantities

For a different scientific domain, the acceptance criteria could be completely different.

This makes scientific software optimization fundamentally different from generic software testing.

An optimization could easily produce impressive benchmark numbers by:

• Reducing numerical precision excessively
• Skipping iterations
• Lowering resolution
• Removing physical processes
• Changing convergence criteria
• Simplifying expensive calculations

Such changes might improve runtime while destroying the scientific purpose of the simulation.

ScienceIDE therefore treats scientific equivalence as a central task-completion criterion.

The agent is not considered successful simply because its implementation runs. It must satisfy predefined scientific validation checks against reference results.

🌍 ScienceIDE Turns Scientific Codebases Into Learning Environments

#

ScienceIDE approaches the problem from a broader perspective.

Scientific codebases contain enormous amounts of accumulated domain knowledge, but that knowledge is distributed across source code, build systems, test cases, numerical methods, configuration files, documentation, and researcher experience.

The project aims to convert these codebases into executable environments for AI agents.

The basic workflow is:

1. Domain experts define scientific tasks.
2. Reference implementations and test cases establish expected behavior.
3. AI agents modify or implement code.
4. The environment executes the resulting program.
5. Scientific checks validate the output.
6. Verified interaction trajectories become training and evaluation data.
7. Models learn from these trajectories through supervised fine-tuning or reinforcement learning.

This changes the role of scientific software.

Instead of being merely an application that AI operates, the scientific codebase becomes a training environment with executable feedback.

📚 ScienceIDE Covers 64 Environments and 2,812 Tasks

#

The ScienceIDE paper reports a collection of:

• 64 environments
• 2,812 tasks
• 1,076 executable scientific checks
• 27 scientific codebases

The environments cover a broad range of disciplines, including:

• Astrophysical MHD
• Space plasma physics
• Ocean and climate modeling
• Gravitational N-body simulation
• Photonics and electromagnetics
• Materials science
• Quantum physics
• Relativity
• Particle detectors
• Immunology

Tasks are divided into seven categories:

1. Acceleration
2. Repair
3. Discovery
4. Reproduction
5. Integration
6. Calibration
7. Implementation

The breadth is important because it tests whether AI agents can transfer technical reasoning across fundamentally different scientific domains.

However, the current benchmark collection is still weighted toward code repair and feature implementation. Performance-optimization tasks remain comparatively limited.

The project team has stated that it is expanding realistic optimization tasks with a more rigorous setup: keep the scientific problem, input scale, and hardware budget fixed, then allow the agent to modify the implementation while measuring both scientific correctness and resource consumption.

That would create a much more direct benchmark for AI-driven scientific performance engineering.

📊 ScienceIDE-Hard Shows That Scientific Coding Remains Difficult

#

To measure the difficulty of these tasks, the team created ScienceIDE-Hard, a public benchmark containing 85 challenging tasks drawn from environments including PLUTO, Athena++, MITgcm, LAPS, and PHANTOM.

The benchmark contains:

• 52 repair tasks
• 33 implementation tasks

The evaluation criterion is stricter than simple execution success.

Solutions must remain consistent with held-out scientific reference solutions under the benchmark’s validation conditions.

Fifteen models from eight providers were evaluated through Codex, Claude Code, or Gemini CLI, with a one-hour budget per episode.

The reported results show that even frontier models struggle with these tasks.

The strongest reported model, Claude Fable 5.1, achieved a 67.1% pass rate, followed by Claude Opus 5 at 64.6% and GPT-6-astra at 63.1%. More than half of the evaluated frontier models remained below 30%.

These results suggest that real-world scientific coding is still far from being a solved problem.

The difficulty comes from the combination of programming, numerical reasoning, domain knowledge, debugging, and scientific validation.

🔄 Scientific Trajectories Can Transfer Beyond Science

#

ScienceIDE also explores whether verified scientific interaction trajectories can improve general-purpose AI capabilities.

Using these trajectories for supervised fine-tuning, the team trained and released three models:

• PhAI-IDE-4B
• PhAI-IDE-9B
• PhAI-IDE-72B

The reported results indicate improvements not only on scientific tasks but also on broader coding, reasoning, and knowledge benchmarks.

For example, the 72B model reportedly improved:

• APPS Introductory: 0.609 → 0.672
• LiveCodeBench Execution: 0.539 → 0.594

On the 9B model, BBH Word Sorting increased from 0.240 to 0.576.

These results suggest that scientific trajectories may contain useful forms of reasoning that transfer beyond the original scientific domain.

The potential mechanism is intuitive: scientific software tasks often require long-horizon reasoning, precise constraint handling, iterative debugging, and verification against objective outcomes.

Those capabilities are useful beyond scientific computing.

🧠 Science Could Become an AI Training Ground

#

This leads to a broader idea: science may be more than an application domain for AI.

Scientific environments have an unusual property for AI training. They can provide structured, executable feedback.

A model can propose a change, run the program, observe the result, compare it against a scientific reference, and revise its approach.

This is fundamentally different from training exclusively on static text.

Scientific workflows can provide:

• Executable environments
• Objective validation criteria
• Numerical feedback
• Reproducible experiments
• Long-horizon tasks
• Domain-specific constraints
• Measurable performance objectives

Such environments could become valuable for training agents that need to reason through complex technical problems rather than simply generate plausible text or code.

🌐 An Open Ecosystem for AI and Scientific Computing

#

The AItonomy Foundation describes itself as a Silicon Valley-registered nonprofit focused on accelerating the feedback loop between AI and scientific discovery.

Its underlying premise is that scientific research and AI development both depend on reliable feedback loops.

AI-generated hypotheses need computational or experimental validation. At the same time, scientific experiments generate data, logs, failed attempts, intermediate decisions, and successful strategies that can potentially become training material for future AI systems.

Traditional academic papers preserve final findings but usually do not capture the entire sequence of intermediate decisions and trial-and-error that produced them.

ScienceIDE attempts to make more of that process executable and reusable.

Researchers can contribute scientific codebases, canonical test cases, tasks, and acceptance criteria. These components can then form reusable environments for AI training and evaluation.

The project reports participation from researchers associated with more than 20 institutions, including UC Berkeley, Carnegie Mellon University, Cornell, Caltech, Harvard, MIT, Oxford, UCL, Princeton, and Stanford, participating in an individual capacity.

🔗 Anthropic’s Optimization Work Could Become Training Data

#

An especially interesting possibility is the team’s stated interest in converting the 36 optimization packages released by Anthropic into ScienceIDE environments.

These packages potentially provide an unusually useful foundation because they combine:

• Existing scientific implementations
• Real optimization problems
• Reference behavior
• Measurable performance
• Concrete computational workloads

If incorporated into ScienceIDE, the optimization results would become more than static open-source code.

They could become executable tasks through which AI systems learn to reproduce, improve, and generalize scientific performance-engineering techniques.

That would connect two otherwise separate developments:

AI optimizes scientific software → optimization trajectories become training environments → future AI systems learn to optimize scientific software more effectively.

This creates the possibility of a compounding feedback loop.

🔭 From AI-Generated Code to AI Scientific Engineers

#

The experiments from Anthropic and ScienceIDE point toward the same fundamental transition from different directions.

Anthropic demonstrates that current AI systems can already perform substantial optimization work on real scientific software.

ScienceIDE asks how those capabilities can become systematic, measurable, reusable, and transferable.

The PLUTO experiment provides an important intermediate lesson: working code is only the beginning.

An AI-generated implementation may execute correctly while leaving substantial hardware performance untapped. Unlocking that performance requires profiling, diagnosis, data-layout analysis, kernel optimization, benchmarking, and repeated validation.

At the same time, faster code is not necessarily better science. Scientific correctness must remain part of the optimization objective.

The emerging workflow therefore looks less like one-shot code generation and more like an autonomous engineering loop:

Understand → implement → execute → profile → optimize → validate → repeat.

The long-term goal is not simply an AI that writes scientific programs.

It is an AI system that can reason about scientific algorithms, interact with real hardware, optimize implementations, verify scientific equivalence, learn from successful trajectories, and transfer those skills across domains.

Decades of scientific software development have already encoded an enormous amount of human knowledge into executable form. Projects such as ScienceIDE attempt to turn that accumulated knowledge into environments where AI can learn.

If that approach scales, scientific codebases could become not only tools for researchers, but also one of the most demanding and valuable training grounds for the next generation of AI agents.