Terminal-Bench 2.0 & Harbor: New AI Agent Benchmark

Terminal‑Bench 2.0, the next generation of a benchmark suite that tests AI agents in real‑world terminal environments, debuted alongside Harbor, a new runtime framework that lets developers run, evaluate, and fine‑tune agents in containerized clouds. The dual release addresses long‑standing pain points in reliability and scalability, offering a more rigorous set of 89 tasks that have been validated for hours by both humans and LLMs. By replacing the original 1.0 suite, the team sets a new standard for assessing frontier model capabilities in developer‑style command‑line interactions.

Terminal‑Bench 2.0 raises the bar by tightening task specifications and eliminating flaky dependencies on third‑party services. For instance, the previously popular download‑youtube task was removed after repeated failures caused by API changes. Each task now undergoes manual and automated validation to ensure solvability and reproducibility, which has helped maintain comparable state‑of‑the‑art performance while boosting overall task quality. The benchmark’s focus on realistic, high‑impact scenarios—such as package management, debugging, and scripting—provides researchers with a clear, reproducible metric for measuring progress in autonomous agent reasoning and code generation.

Harbor complements the benchmark by offering a unified, cloud‑ready infrastructure that supports thousands of parallel rollouts on providers like Daytona and Modal. Its API accepts any container‑installable agent and integrates seamlessly with supervised fine‑tuning and reinforcement learning pipelines, making it ideal for rapid experimentation. Early leaderboard results show GPT‑5‑powered Codex CLI leading at 49.6% success, followed closely by other GPT‑5 and Claude Sonnet 4.5 variants, illustrating the competitive landscape. Users can launch a run with a single CLI command and submit results to the public leaderboard, while the framework’s documentation guides integration into research workflows. Together, Terminal‑Bench 2.0 and Harbor lay the groundwork for a standardized, scalable evaluation stack across the AI ecosystem.