latent-sokoban, an open benchmark for pixel-based world models

Solve Sokoban from pixels alone.

Your agent sees two 64×64 images, the board and the goal, and nothing else. No coordinates, no rules, no solver. Learn the world, plan in your own latent space, and push every crate home.

Hidden levels, evaluated live on this server.

Standings

Best scorecard per entrant, ranked by success rate. See the full leaderboard →

Run your agent

git clone https://github.com/Lulzx/latent-sokoban && cd latent-sokoban
pip install -e .

# once: claim your leaderboard name, get an API key
python scripts/remote_eval.py --register "your-name"
export SOKOBAN_API_KEY=lsk-…

# sanity check with the built-in random agent
python scripts/remote_eval.py --agent random

# your model: implement latent_sokoban.agent.Agent, then
python scripts/remote_eval.py --agent my_pkg.agent:MyAgent

How evaluation works

Four calls, in this order. The full wire schema lives in docs/API.md.

  1. POST /api/scorecards opens a scorecard: one submission attempt. Authenticated by your API key; the key is your leaderboard identity.
  2. …/games/standard/start begins a session over 100 hidden 8×8 levels. Every frame carries the board and the goal as raw 64×64×3 bytes. The layouts never leave the server.
  3. POST /api/sessions/{id}/act sends one action: up, down, left, right. Invalid moves are no-ops that still consume a step. Each level's step budget is three times its own optimal solution.
  4. …/close locks the scorecard and posts it to the leaderboard. Unplayed episodes count as unsolved, so there is no cherry-picking. Completed sessions get a public replay.

The rules that matter

8×8 boards, 1 to 4 crates
100 hidden levels, ordered easiest first. Crate count rises with the level number, and so does the length of the shortest solution.
64×64 RGB observations
The board and the goal, as raw bytes. Nothing symbolic.
≤ 20M parameters
The whole agent, including any world model.
≤ 256 dynamics calls / action
Your planning budget for a single move.
No symbolic solvers
No BFS, A*, or hand-written Sokoban logic anywhere in the loop.
No decode-then-search
Recovering the grid from pixels and planning on it defeats the point.

The intent: the winning system plans in a learned representation. Full rules, the shared baseline (28% on the warmup split), datasets and local tooling live in the GitHub repo. The agent protocol, and the interactive API reference with it, is under Docs.