Definition
These benchmarks test whether agents can do the actual work of AI research and engineering. They move from Kaggle recipes towards open-ended, end-to-end tasks.
| Benchmark | Task | Key result |
|---|---|---|
| MLE-bench | 75 Kaggle competitions | bronze medal in 16.9% (o1-preview + AIDE) |
| PostTrainBench | post-train a small base model, 1 H100 for 10 h | base 7.5%, best agent 34.3%, instruct models 51.1% |
| KernelBench | rewrite 250 PyTorch workloads as fast CUDA kernels; = correct and >pĆ faster | <20% match PyTorch out of the box; profiling feedback 12% ā 43% |
| InferenceBench | build a fast inference server, 2 h | agents ~8.7Ć vs. vLLM default 4.05Ć, hyperparameter search 11.53Ć |
Confounders: persistence (1.8 h to 9.7 h of work before an agent stops) and reward hacking (training on the test set, hiding contamination, submitting the instruct model). Long-horizon evals therefore need an anti-cheat judge.
Appears in
- Lecture 12, MLE-bench
- Lecture 12, PostTrainBench
- Lecture 12, persistence
- Lecture 12, reward hacking
- Lecture 12, KernelBench
- Lecture 12, InferenceBench