Vla Evaluation Harness 0.8.0

v0.8.0

Shared episode scheduling, clearer shard failures, and more reliable RoboTwin and container execution.

Parallel evaluation

• Shards sharing a recording database now claim episodes from a shared work queue. Shards prefer their own block and loaded task; idle shards help with remaining work. --no-save retains fixed sharding.
• A shard whose first three episodes all raise errors exits with code 3. --requeue-unhealthy releases those items for other shards to retry. Resume unfinished work with the same eval ID, config, task ordering, and shard identity.
• Progress reflects the entry-wide queue. Export derives completion from queued work and reports unfinished items in queue_unfinished; incomplete evaluations remain marked partial.
• Episode errors identify environment startup, environment stepping, or model inference. The watchdog now observes every environment step, avoiding false stalls during long episodes.

Benchmarks and runtimes

• RoboTwin reuses its environment across episodes, repairs success-check initialization and episode limits, bounds expert checks, and includes precomputed expert seeds and instructions.
• RoboTwin's render_every option renders camera observations every N steps and at episode end. Keep the default of 1 when the policy needs a fresh observation each step.
• The RoboTwin image uses nvcc 12.6 and warp-lang 1.12.0 for its CuRobo build. Rebuild or update the image to obtain these changes.
• RoboCasa supports send_state and loads objects correctly from read-only container images.
• Charliecloud cleans up stale image mounts left by killed shards. Lavapipe device-memory files use tmpfs to avoid shared-filesystem pressure.

Documentation and leaderboard

• Shared benchmark, model-server, and evaluation skills live in skills/, with repository instructions in AGENTS.md.
• RoboTwin 2.0 leaderboard entries link to its official leaderboard.

Container distribution

Redistributable images are published under both ghcr.io/allenai/vla-evaluation-harness and ghcr.io/worv-ai/vla-evaluation-harness-public, with 0.8.0 and latest tags. Some allenai packages remain private; the harness retains its public worv-ai pull fallback. RLBench, BEHAVIOR-1K, and RoboDojo remain build-locally-only under the existing redistribution restrictions.

All 20 images are deployed to both registries with matching manifest digests. The worv-ai images are anonymously accessible. The allenai packages for duobench, libero-plus, molmospaces, robocasa365, robomme, simpler-groot, and simpler-xvla require authentication; their visibility was not changed.

The attached container-manifests.json records image digests, tags, access checks, and validation. Tests: 628 passed, 2 skipped; Ruff and ty passed. All 19 benchmark images passed the installed-version check and SHA256 comparison of 282 source/configuration files against the release checkout. Full GPU benchmark reproductions were not rerun for this release.

ManiSkill2, LIBERO-Plus, LIBERO-Mem, RoboCasa365, VLABench, and MolmoSpaces reuse their unchanged 0.7.0 simulator/assets layers with the complete 0.8.0 harness and configs installed. This avoided upstream asset outages and repeated large-layer transfers. RoboTwin includes the updated CUDA/CuRobo build described above.

Full changelog: v0.7.0...v0.8.0