# WorldEvals

> The Inspect Evals for robotics: a curated catalog of physical-AI benchmarks built on Inspect Robots.

WorldEvals is the "Inspect Evals for robotics": a curated catalog of physical-AI / VLA benchmarks built on Inspect Robots. Each benchmark is its own repo; WorldEvals indexes what exists and how to install it.

# Catalog

# WorldEvals

The Inspect Evals for robotics.

A curated catalog of physical-AI / VLA benchmarks built on [Inspect Robots](https://github.com/robocurve/inspect-robots). Each benchmark lives in its own repository (so it owns its release cadence, dependencies, and hardware notes); WorldEvals is the index that ties them together.

[GitHub](https://github.com/robocurve/worldevals) [Add a benchmark](https://worldevals.org/contributing/index.md)

Two views of the collection

`inspect-robots list` tells you what Inspect Robots tasks are installed. `worldevals list` tells you what benchmarks exist and how to get them.

## Benchmarks

**2 benchmarks · 11 tasks**

- **[VibeCheckBench](https://github.com/robocurve/vibecheckbench)**

  ______________________________________________________________________

  A one-task smoke benchmark: lay a detached clapstick on a clapperboard centered 19 inches in front of the bimanual arm line. Exists to prove an eval stack works end to end before a full benchmark run.

  **1 tasks** · bimanual · `smoke-test` `bimanual` `manipulation`

  [Repo](https://github.com/robocurve/vibecheckbench) · `alpha`

  ```
  pip install "vibecheckbench @ git+https://github.com/robocurve/vibecheckbench"
  ```

- **[KitchenBench](https://github.com/robocurve/kitchenbench)**

  ______________________________________________________________________

  10 bimanual kitchen-manipulation tasks: pick-place, stacking, slotted insertion, granular pour & tool-scoop, lid open/seal, cloth folding, a two-arm handover, and a multi-instance cutlery sort.

  **10 tasks** · bimanual · `kitchen` `bimanual` `manipulation`

  [Repo](https://github.com/robocurve/kitchenbench) · `alpha`

  ```
  pip install kitchenbench
  ```

## Browse from the command line

```
pip install worldevals

worldevals list                 # all benchmarks
worldevals list --tag bimanual  # filter by tag
worldevals info kitchenbench    # repo, install command, task keys
worldevals tasks                # Inspect Robots tasks installed locally, by benchmark
```

For LLMs: [`llms.txt`](https://worldevals.org/llms.txt) · [`llms-full.txt`](https://worldevals.org/llms-full.txt).

# Add a benchmark

WorldEvals indexes benchmark repos; it doesn't host them. A benchmark is any repository that:

1. **depends on [Inspect Robots](https://github.com/robocurve/inspect-robots)**,
1. **defines one or more Inspect Robots `Task`s**, and
1. **registers them via entry points**: `[project.entry-points."inspect_robots.tasks"]` (and, if it ships a simulator/embodiment or policy, `inspect_robots.embodiments` / `inspect_robots.policies`).

See [KitchenBench](https://github.com/robocurve/kitchenbench) as the reference implementation.

## List it in the catalog

Add a `Benchmark(...)` entry to [`src/worldevals/catalog.py`](https://github.com/robocurve/worldevals/blob/main/src/worldevals/catalog.py) and open a PR:

```
Benchmark(
    name="kitchenbench",
    title="KitchenBench",
    description="10 bimanual kitchen-manipulation tasks: ...",
    repo="https://github.com/robocurve/kitchenbench",
    install="pip install kitchenbench",  # from PyPI; git URL only if unpublished
    task_keys=("kitchenbench/place_cutlery", ...),  # the Inspect Robots task keys it registers
    tags=("kitchen", "bimanual", "manipulation"),
    bimanual=True,
    contributors=("your-handle",),
    status="alpha",
)
```

The homepage card grid is generated from this catalog at build time, so adding an entry updates the site automatically. A test validates every entry (unique name, well-formed `https://github.com/...` repo URL, ≥1 task key), and CI requires 100% coverage. Keep `task_keys` in sync with the benchmark's actual registered task names.

Every public module, class, and function needs a docstring, enforced by `ruff` D1; state the contract, don't restate the name.

See the [API reference](https://worldevals.org/api/index.md) for the `Benchmark` fields.