Add a benchmark¶
WorldEvals indexes benchmark repos; it doesn't host them. A benchmark is any repository that:
- depends on Inspect Robots,
- defines one or more Inspect Robots
Tasks, and - registers them via entry points:
[project.entry-points."inspect_robots.tasks"](and, if it ships a simulator/embodiment or policy,inspect_robots.embodiments/inspect_robots.policies).
See KitchenBench as the reference implementation.
List it in the catalog¶
Add a Benchmark(...) entry to
src/worldevals/catalog.py
and open a PR:
Benchmark(
name="kitchenbench",
title="KitchenBench",
description="10 bimanual kitchen-manipulation tasks: ...",
repo="https://github.com/robocurve/kitchenbench",
install="pip install kitchenbench", # from PyPI; git URL only if unpublished
task_keys=("kitchenbench/place_cutlery", ...), # the Inspect Robots task keys it registers
tags=("kitchen", "bimanual", "manipulation"),
bimanual=True,
contributors=("your-handle",),
status="alpha",
)
The homepage card grid is generated from this catalog at build time, so adding
an entry updates the site automatically. A test validates every entry (unique
name, well-formed https://github.com/... repo URL, ≥1 task key), and CI requires
100% coverage. Keep task_keys in sync with the benchmark's actual registered
task names.
Every public module, class, and function needs a docstring, enforced by ruff D1;
state the contract, don't restate the name.
See the API reference for the Benchmark fields.