API reference

Generated from the source docstrings. The catalog and its accessors are the public surface.

catalog

The WorldEvals catalog — the registry of physical-AI benchmark repos.

Each benchmark is its own repository (built on Inspect Robots) that registers its tasks via entry points. WorldEvals indexes them so you can discover what exists and how to install it. To add a benchmark, append a Benchmark entry here (PR).

Benchmark dataclass

One benchmark repo in the WorldEvals collection.

Source code in src/worldevals/catalog.py

@dataclass(frozen=True)
class Benchmark:
    """One benchmark repo in the WorldEvals collection."""

    name: str
    title: str
    description: str
    repo: str
    install: str
    task_keys: tuple[str, ...]
    tags: tuple[str, ...]
    bimanual: bool
    contributors: tuple[str, ...]
    status: Literal["alpha", "beta", "stable"] = "alpha"

catalog

catalog() -> tuple[Benchmark, ...]

All benchmarks in the collection.

Source code in src/worldevals/catalog.py

def catalog() -> tuple[Benchmark, ...]:
    """All benchmarks in the collection."""
    return CATALOG

get

get(name: str) -> Benchmark

Look up a benchmark by name; raise KeyError if unknown.

Source code in src/worldevals/catalog.py

def get(name: str) -> Benchmark:
    """Look up a benchmark by name; raise `KeyError` if unknown."""
    for benchmark in CATALOG:
        if benchmark.name == name:
            return benchmark
    raise KeyError(f"no benchmark named {name!r}; known: {sorted(b.name for b in CATALOG)}")

by_tag

by_tag(tag: str) -> list[Benchmark]

All benchmarks carrying tag.

Source code in src/worldevals/catalog.py

def by_tag(tag: str) -> list[Benchmark]:
    """All benchmarks carrying ``tag``."""
    return [b for b in CATALOG if tag in b.tags]

benchmark_for_task

benchmark_for_task(task_key: str) -> Benchmark | None

The benchmark that registers task_key, or None if not in the catalog.

Source code in src/worldevals/catalog.py

def benchmark_for_task(task_key: str) -> Benchmark | None:
    """The benchmark that registers ``task_key``, or ``None`` if not in the catalog."""
    for benchmark in CATALOG:
        if task_key in benchmark.task_keys:
            return benchmark
    return None