NeuralStackSakana AI’s long-running AI Scientist project, which aims to automate the full machine learning research workflow, has now been written up in a peer-reviewed paper published in Nature. The newly released article consolidates earlier milestones, including a fully AI-generated paper that passed human peer review, and adds fresh technical insights into how the system works. It also situates the work as part of an open-access effort, with prior code releases and generated papers available for inspection.
The AI Scientist was first introduced in an initial preprint that laid out an ambitious goal: an agent, powered by foundation models, that could execute the entire machine learning research lifecycle end to end. In its earliest incarnation, the team provided the system with a simple starting code template, similar to a basic training run, and let it autonomously generate novel ideas, design experiments, and write up the resulting work as a complete paper. That first version also included an automated reviewing component to score paper quality, offering an early proof that fully automated research loops were technically feasible.
Subsequent iterations raised the bar by expanding the AI Scientist’s freedom and testing it under realistic conditions. In a high-profile experiment described by the team as a kind of “Turing Test” for science, the second version was allowed to explore broadly defined AI topics and then submit its unedited, fully AI-generated papers to the blind human peer-review process of a major AI conference workshop. One such manuscript not only cleared the acceptance bar but also scored better than a significant fraction of human-authored submissions, prompting the researchers to follow through on a pre-committed plan to withdraw it before publication, despite its success.
Under the hood, the Nature paper describes a system that begins with a broad research direction and then proceeds autonomously through the typical stages of machine learning research. The AI Scientist generates candidate research ideas, searches for and reads relevant literature, and then designs, programs, and executes experiments using a parallelized agentic tree search. It goes on to draft the full paper in LaTeX, receiving feedback on figures from a vision-capable foundation model, and then routes the result through an automated reviewing step for quality control.
The team’s Automated Reviewer is a crucial part of this workflow and a focus of the new results. By prompting a model to act like an Area Chair, ensembling multiple independent reviews into a single decision guided by leading conference standards, the researchers were able to benchmark its performance against thousands of past human review decisions. According to their analysis, the Automated Reviewer matched human-level judgment on conference papers, even for work published after the model’s own training cutoff, suggesting it can be used as a scalable proxy for human reviewers on new AI research.
One of the most consequential findings comes from using this reviewer to compare outputs from different foundation models. When grading AI-generated papers produced by these models, the researchers observed a clear scaling pattern: as the underlying foundation models improved, the quality of the resulting papers increased in step. While the Nature article avoids overclaiming, the team argues that this trend implies future versions of the AI Scientist will become substantially more capable as compute costs fall and model capabilities continue to grow.
Why this matters
The AI Scientist and its Automated Reviewer point toward a near future in which large parts of the research pipeline can be handled by machines, not just at toy scale but at the level of conference-ready work. If systems can reliably generate ideas, design experiments, and draft papers that hold up under human peer review, they could fundamentally change the pace and organization of scientific discovery. That potential cuts both ways: it could unlock an explosion of experimentation and open-ended exploration, but it also raises urgent questions about validation, oversight, and what it means for humans to remain in the loop when an AI can credibly produce, and even judge, cutting-edge research.
Despite the breakthrough of clearing human peer review, the researchers are explicit about the AI Scientist’s current limitations. The system still proposes naive or underdeveloped ideas, can struggle with deep methodological rigor and complex code implementations, and occasionally hallucinates details, such as incorrect citations or duplicated figures in appendices. For now, it is restricted to computational experiments, leaving physical experimentation and many other scientific domains outside its direct reach.
Looking ahead, Sakana AI and its collaborators frame this Nature publication as a playbook others can adapt rather than a finished product. They expect the approach to extend beyond machine learning into other scientific areas once comparable data, tools, and domain-specific infrastructure are in place. If the observed scaling trends hold and the limitations are progressively addressed, future iterations of systems like the AI Scientist could move from proving that automated science is possible to driving genuinely open-ended discoveries across a widening range of fields.