latent_sokoban.evaluation

latent_sokoban.evaluation

Deterministic evaluation harness.

Runs an Agent on a local benchmark split (a JSON file of levels produced by scripts/generate_levels.py) and reports the harness metrics. The leaderboard score is separate: it comes from the server over the hidden set (see docs/scoring.md). These feed scripts/score.py's local profile.

success_rate solved within the action limit (primary metric) move_efficiency optimal_moves / agent_moves, solved levels only avg_plan_time_ms wall-clock time per executed action avg_model_calls counted learned-dynamics calls per executed action deadlock_rate episodes that entered a provably dead state avg_steps_solved actions used on solved levels

Planning is budgeted in counted dynamics calls (see latent_sokoban.agent): the harness attaches a fresh CallMeter each episode and, in strict mode (the default), an action that exceeds the cap fails the episode on the spot. Metering honesty is verified by source review at submission.

Determinism: level order, themes and observation noise all derive from the split file and the evaluation seed. Two runs of the same checkpoint with the same seed must produce identical results (agents must seed their own samplers from a fixed constant, or accept the reproducibility rules).

load_agent

load_agent(spec)

Instantiate an agent from 'module.path:ClassName' or a builtin name.

Source code in latent_sokoban/evaluation.py

def load_agent(spec: str) -> Agent:
    """Instantiate an agent from 'module.path:ClassName' or a builtin name."""
    import sys
    if "" not in sys.path and str(Path.cwd()) not in sys.path:
        sys.path.insert(0, str(Path.cwd()))  # resolve agents relative to cwd
    builtins = {"random": "latent_sokoban.agent:RandomAgent"}
    spec = builtins.get(spec, spec)
    module_name, _, class_name = spec.partition(":")
    if not class_name:
        raise ValueError(f"agent spec must be 'module:ClassName', got {spec!r}")
    cls = getattr(importlib.import_module(module_name), class_name)
    return cls()