Research & academia
SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation
SciCodePile pairs a 128GB scientific-code corpus with a 200-task executable benchmark; its abstract reports low top-model scores alongside gains from training on the corpus.
Why it matters: For evaluation, the results show why code similarity scores alone are not enough: executable testing exposes a substantial gap in functional performance, while the reported training gains suggest the corpus can help improve measured results.