Automated Simple Baselines Match or Surpass Complex Deep Learning in Major Scientific Studies
A study testing 36 scientific machine learning tasks reveals that LLM-generated simple baselines recover 95.5% of the performance gap and rival deep learning models across top peer-reviewed journals.
In the physical and life sciences, deep learning is frequently heralded for solving complex scientific challenges. However, validating whether a newly proposed deep learning model genuinely represents progress requires benchmarking it against proper baselines. Without them, apparent performance gains can easily be an artifact of omitted comparisons.
A research team consisting of Caleb N. Ellington, Jiaqi Wang, Sohan Addagudi, and Elijah Cole explored a scalable solution to this dilemma: using large language models to automatically construct strong, simple baselines. Their findings, published on arXiv, suggest that many high-profile scientific models deliver gains that can be matched by significantly simpler alternatives.
Benchmarking Across Top-Tier Journals
To evaluate whether automated baselines could expose benchmark gaps, the researchers assembled 36 supervised classification and regression tasks. Each task was gathered from peer-reviewed papers published in premier scientific outlets, specifically journals from the Nature, Science, and Cell publishing groups.
Using a language model, the authors generated simple baseline models tailored to each task. The performance of these automated baselines was then measured against both a trivial predictor and the specialized deep learning architectures introduced in the original studies.
Simple Models Recover the Performance Gap
The automatically generated baselines delivered unexpectedly strong results:
- Gap Recovery: Across the evaluated tasks, the automated simple baselines recovered a median of 95.5% (and a mean of 85.7%) of the performance difference between a trivial baseline and the deep learning model.
- Outperforming Complex Architectures: In 12 of the 36 examined studies, the median simple baseline matched or exceeded the performance achieved by the published deep learning model.
The authors also analyzed factors that influenced baseline quality, detailing how performance varied across scientific domains, publication venues, test-time compute allocations, and different degrees of study context provided to the language model.
A Scalable Check for Scientific Peer Review
The rapid influx of AI-focused submissions has placed intense strain on traditional human peer review. Reviewers often lack the time or specialized codebases required to verify whether a complex architecture significantly outperforms standard statistical or classic machine learning baselines.
The authors argue that automated baseline generation presents a practical, scalable mechanism to check claims of deep learning contributions. By automatically generating comparable baselines prior to publication, the scientific community can better ensure that reported computational breakthroughs represent genuine advancements over simpler methodologies.
