The Alignment League Benchmark (ALB) is a suite of multi-agent games, designed to measure how well AI agents align, coordinate, and collaborate with others (both AIs and humans).
CoGames is the games environment for ALB. You can use it to:
- create new games
- train agents to play existing ALB games
- submit those agents to the ALB leaderboard
There's one ALB game right now: Cogs vs Clips.
Install cogames as a Python package.
pip install cogamesUsing uv
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
# Create a virtual environment
uv venv .venv
source .venv/bin/activate
# Install cogames
uv pip install cogamesUsing Docker
# Ensure Python 3.12 is available
FROM python:3.12-slim
# Ensure C/C++ compiler is available
RUN apt-get update && \
apt-get install -y --no-install-recommends build-essential && \
rm -rf /var/lib/apt/lists/*
# Install cogames
RUN pip install --no-cache-dir cogamesUsing Colab
Play an easy mission in Cogs vs Clips using:
cogames tutorial playThe game will open in a new window, and the terminal will give you instructions to complete training mission.
-
Log into the ALB leaderboard with your GitHub account.
cogames login
-
Upload a starter policy and submit it to the tournament.
cogames upload --policy "class=cogames.policy.starter_agent.StarterPolicy" --name "$USER.README-quickstart-starter-policy" --season beta-teams-small --skip-validation
-
Check your submission status.
cogames submissions --season beta-teams-small --policy "$USER.README-quickstart-starter-policy" cogames season matches beta-teams-small --limit 5
To learn more, see:
- Creating a policy: Creating a custom policy and evaluating it
- Training: Training a custom policy and evaluating it
- Submitting: Submitting to the leaderboard and understanding the results
If you want help, or to share your experience, join the Discord.
CogsGuard is a cooperative territory-control game. Teams of AI agents ("Cogs") work together to capture and defend junctions against automated opponents ("Clips") by:
- gathering resources and depositing them at controlled junctions
- acquiring specialized roles (Miner, Aligner, Scrambler, Scout) at gear stations
- capturing neutral junctions using Aligners (costs 1 heart)
- disrupting enemy junctions using Scramblers (costs 1 heart)
- defending territory from Clips expansion
Read MISSION.md for a thorough description of the game mechanics.
There are many mission configurations available, with different map sizes, junction layouts, and game rules.
Overall, CogsGuard aims to present rich environments with:
- Territory control: Capture and defend junctions to score points each tick
- Role specialization: Four roles (Miner, Aligner, Scrambler, Scout) with distinct capabilities and dependencies
- Dense rewards: Agents receive reward every tick proportional to territory controlled
- Partial observability: Agents have limited visibility of the environment
- Required multi-agent cooperation: No single role can succeed alone; Miners need Aligners to capture junctions, Aligners need Miners for resources, Scramblers must clear enemy territory for Aligners to advance
We run multiple tournament seasons at a time, and each has a different structure. Our freeplay seasons are cheap to submit to and evergreen: we give you indicative scores, showing how you play with others. Our team tournaments support limited submissions and usually follow this pattern:
- Play-ins: New submissions are evaluated in one or more early stages, with eliminations based on stage results.
- Team stages: Surviving policies are sampled into teams and evaluated in repeated team matches.
- Progressive culling: Lower-performing teams/policies are removed across later stages.
- Final scoring: Remaining policies are ranked on the season leaderboard using season-specific scoring.
To inspect the exact rules for a specific season:
cogames season list
cogames season show <SEASON>
cogames season stages <SEASON>
cogames season progress <SEASON>
cogames season teams <SEASON>
cogames season leaderboard <SEASON>
cogames season leaderboard <SEASON> --pool <POOL> --type team
cogames season leaderboard <SEASON> --pool <POOL> --type score-policies
cogames season pool-config <SEASON> <POOL>Note: cogames season show takes only one positional argument (<SEASON>).
Use separate subcommands (stages, progress, teams, leaderboard) for details.
The tournament API is documented at api.observatory.softmax-research.net/docs. The interactive OpenAPI spec describes all public endpoints for seasons, matches, leaderboards, and submissions.
Recommended end-to-end sequence once you are ready to submit:
# 1) Login
cogames login
# 2) Pick a season
cogames season list
cogames season show <SEASON>
# 3) Upload (bundle + validate + upload + submit)
cogames upload -p <POLICY_SPEC_OR_CHECKPOINT> -n <POLICY_NAME> --season <SEASON>
# Alternative if already uploaded:
cogames submit <POLICY_NAME[:vN]> --season <SEASON>
# 4) Track status
cogames submissions --season <SEASON> --policy <POLICY_NAME>
cogames season matches <SEASON> --limit 20
# 5) Debug specific outcomes
cogames matches <MATCH_ID>
cogames match-artifacts <MATCH_ID>
cogames episode show <EPISODE_ID>
cogames episode replay <EPISODE_ID>To specify a MISSION, you can:
- Use a mission name from the registry given by
cogames missions(e.g.training_facility_1). - Use a path to a mission configuration file (e.g.
path/to/mission.yaml). - Alternatively, specify a set of missions with
--mission-set.
To specify a POLICY, use one of two formats:
-
URI format (for checkpoint bundles):
- Point directly at a checkpoint bundle (directory or
.zipcontainingpolicy_spec.json) - Examples:
./train_dir/my_run:v5,./train_dir/my_run:v5.zip,s3://bucket/path/run:v5.zip - Use
:latestsuffix to auto-resolve the highest version:./train_dir/checkpoints:latest
- Point directly at a checkpoint bundle (directory or
-
Key-value format (for explicit class + weights):
class=: Policy shorthand or full class path fromcogames policies, e.g.class=lstmorclass=cogames.policy.random.RandomPolicy.data=: Optional path to a weights file (e.g.,weights.safetensors). Must be a file, not a directory.proportion=: Optional positive float specifying the relative share of agents that use this policy (default: 1.0).kw.<arg>=: Optional policy__init__keyword arguments (all values parsed as strings).
You can view all the commands with
cogames --helpand you can view help for a given command with:
cogames [COMMAND] --helpUsage: cogames missions [OPTIONS] SITE
List available missions. This command has three modes: 1. List sites: Run with no arguments to see all available sites. 2. List missions at a site: Pass a site name (e.g., 'cogsguard_machina_1') to see its missions. 3. Describe a mission: Use -m to describe a specific mission. Only in this mode do --cogs, --variant, --format, and --save have any effect.
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ site TEXT Filter by site (e.g., cogsguard_machina_1). │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Describe ──────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --mission -m MISSION Mission to describe. │ │ --cogs -c INTEGER Override agent count (requires -m). │ │ --variant -v VARIANT Apply variant (requires -m, repeatable). │ │ --difficulty LEVEL Difficulty (easy, medium, hard) controlling clips events (requires -m). │ │ --format [yaml|json] Output format (requires -m). │ │ --save -s PATH Save config to file (requires -m). │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames missions List all sites cogames missions cogsguard_machina_1 List missions at site cogames missions -m cogsguard_machina_1.basic Describe a mission cogames missions -m arena --format json Output as JSON
Usage: cogames evals [OPTIONS]
List all eval missions.
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: cogames variants [OPTIONS]
List all available mission variants.
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: cogames describe [OPTIONS] MISSION
Describe a mission and its configuration.
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * mission TEXT Mission name (e.g., cogsguard_machina_1.basic). [required] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Configuration ─────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --cogs -c INTEGER Number of cogs (agents). │ │ --variant -v VARIANT Apply variant (repeatable). │ │ --difficulty LEVEL Difficulty (easy, medium, hard) controlling clips events. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames describe cogsguard_machina_1.basic Describe mission cogames describe arena -c 4 -v dark_side With 4 cogs and variant
Usage: cogames make-mission [OPTIONS]
Create a custom mission from a base template.
╭─ Mission ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --mission -m MISSION Base mission to start from. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Customization ─────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --cogs -c INTEGER RANGE [x>=1] Number of cogs (agents). │ │ --width INTEGER RANGE [x>=1] Map width. │ │ --height INTEGER RANGE [x>=1] Map height. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --output -o PATH Output file path (.yml or .json). │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames make-mission -m hello_world -c 8 -o my_mission.yml 8 cogs cogames make-mission -m arena --width 64 --height 64 -o big.yml 64x64 map cogames play -m my_mission.yml Use custom mission
Usage: cogames play [OPTIONS]
Play a game interactively. This runs a single episode of the game using the specified policy. By default, the policy is 'noop', so agents won't move unless manually controlled. To see agents move by themselves, use `--policy class=random` or `--policy class=baseline`. You can manually control the actions of a specific cog by clicking on a cog in GUI mode or pressing M in unicode mode and using your arrow or WASD keys. Log mode is non-interactive and doesn't support manual control.
╭─ Game Setup ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --mission -m MISSION Mission to play (run cogames missions to list). │ │ --variant -v VARIANT Apply variant modifier (repeatable). │ │ --difficulty LEVEL Difficulty (easy, medium, hard) controlling clips events. │ │ --cogs -c N Number of cogs/agents. [default: (from mission)] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Policy ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --policy -p POLICY Policy controlling cogs (noop, random, lstm, or path). [default: class=noop] │ │ --device DEVICE Policy device (auto, cpu, cuda, cuda:0, etc.). [default: auto] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Simulation ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --steps -s N Max steps per episode. [default: 1000] │ │ --render -r [gui|vibescope|unicode|log|none] gui=MettaScope, vibescope=VibeScope, unicode=terminal, │ │ log=metrics only. │ │ [default: gui] │ │ --seed INTEGER RNG seed for reproducibility. [default: 42] │ │ --map-seed SEED Separate seed for procedural map generation. │ │ [default: (same as --seed)] │ │ --autostart Start simulation immediately without waiting for user │ │ input. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --save-replay-dir DIR Save replay file for later viewing with cogames replay. │ │ --save-replay-file FILE Save replay to a fixed file path (overwrites existing file) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames play -m cogsguard_machina_1.basic Interactive cogames play -m cogsguard_machina_1.basic -p class=random Random policy cogames play -m cogsguard_machina_1.basic -c 4 -p class=baseline Baseline, 4 cogs cogames play -m cogsguard_machina_1.basic --save-replay-file ./latest.json.z Overwrite fixed replay file cogames play -m cogsguard_machina_1 -r unicode Terminal mode
Usage: cogames replay [OPTIONS] FILE
Replay a saved game episode from a file in the GUI.
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * replay_path FILE Path to the replay file (.replay or .bin). [required] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames replay ./replays/game.replay Replay a saved game cogames replay ./train_dir/my_run/replay.bin Replay from training run
Usage: cogames scrimmage [OPTIONS]
Evaluate a single policy controlling all agents.
This command is equivalent to running `cogames run` with a single policy.
╭─ Mission ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --mission -m MISSION Missions to evaluate (supports wildcards). │ │ --mission-set -S SET Predefined set: integrated_evals, spanning_evals, diagnostic_evals, all. │ │ --cogs -c N Number of cogs (agents). │ │ --variant -v VARIANT Mission variant (repeatable). │ │ --difficulty LEVEL Difficulty (easy, medium, hard) controlling clips events. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Policy ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --policy -p POLICY Policies to evaluate: (URI[,proportion=N] or NAME or │ │ class=NAME[,data=FILE][,proportion=N]...). │ │ --device DEVICE Policy device (auto, cpu, cuda, cuda:0, etc.). [default: auto] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Simulation ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --episodes -e N [x>=1] Number of evaluation episodes. [default: 10] │ │ --steps -s N [x>=1] Max steps per episode. [default: (from mission)] │ │ --seed N [x>=0] Seed for evaluation RNG. [default: 42] │ │ --map-seed N [x>=0] MapGen seed for procedural maps. [default: (same as --seed)] │ │ --action-timeout-ms MS [x>=1] Max ms per action before noop. [default: 250] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --format FMT Output format: yaml or json. │ │ --save-replay-dir DIR Directory to save replays. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames scrimmage -m arena.battle -p lstm Single policy eval
Usage: cogames run [OPTIONS]
Evaluate one or more policies on missions. With multiple policies (e.g., 2 policies, 4 agents), each policy always controls 2 agents, but which agents swap between policies each episode. With one policy, this command is equivalent to `cogames scrimmage`.
╭─ Mission ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --mission -m MISSION Missions to evaluate (supports wildcards). │ │ --mission-set -S SET Predefined set: integrated_evals, spanning_evals, diagnostic_evals, all. │ │ --cogs -c N Number of cogs (agents). │ │ --variant -v VARIANT Mission variant (repeatable). │ │ --difficulty LEVEL Difficulty (easy, medium, hard) controlling clips events. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Policy ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --policy -p POLICY Policies to evaluate: (URI[,proportion=N] or NAME or │ │ class=NAME[,data=FILE][,proportion=N]...). │ │ --device DEVICE Policy device (auto, cpu, cuda, cuda:0, etc.). [default: auto] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Simulation ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --episodes -e N [x>=1] Number of evaluation episodes. [default: 10] │ │ --steps -s N [x>=1] Max steps per episode. [default: (from mission)] │ │ --seed N [x>=0] Seed for evaluation RNG. [default: 42] │ │ --map-seed N [x>=0] MapGen seed for procedural maps. [default: (same as --seed)] │ │ --action-timeout-ms MS [x>=1] Max ms per action before noop. [default: 250] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --format FMT Output format: yaml or json. │ │ --save-replay-dir DIR Directory to save replays. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames run -m cogsguard_machina_1.basic -p lstm Evaluate single policy cogames run -m cogsguard_machina_1 -p ./train_dir/my_run:v5 Evaluate a checkpoint bundle cogames run -S integrated_evals -p ./train_dir/my_run:v5 Evaluate on mission set cogames run -m 'arena.*' -p lstm -p random -e 20 Evaluate multiple policies together cogames run -m cogsguard_machina_1 -p ./train_dir/my_run:v5,proportion=3 -p class=random,proportion=5 Evaluate policies in 3:5 mix
Usage: cogames pickup [OPTIONS]
Evaluate a policy against a pool of other policies and compute VOR.
╭─ Mission ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --mission -m MISSION Mission to evaluate on. [default: cogsguard_machina_1.basic] │ │ --cogs -c N [x>=1] Number of cogs (agents). [default: 4] │ │ --variant -v VARIANT Mission variant (repeatable). │ │ --difficulty LEVEL Difficulty (easy, medium, hard) controlling clips events. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Policy ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --policy -p POLICY Candidate policy to evaluate. │ │ --pool POLICY Pool policy (repeatable). │ │ --device DEVICE Policy device (auto, cpu, cuda, cuda:0, etc.). [default: auto] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Simulation ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --episodes -e N [x>=1] Episodes per scenario. [default: 1] │ │ --steps -s N [x>=1] Max steps per episode. [default: 1000] │ │ --seed N [x>=0] Base random seed. [default: 50] │ │ --map-seed N [x>=0] MapGen seed for procedural maps. [default: (same as --seed)] │ │ --action-timeout-ms MS [x>=1] Max ms per action before noop. [default: 250] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --save-replay-dir DIR Directory to save replays. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames pickup -p greedy --pool random Test greedy against pool of random
Usage: cogames diagnose [OPTIONS] POLICY
Run diagnostic evals for a policy checkpoint.
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * policy TEXT Policy specification: URI (bundle dir or .zip) or NAME or │ │ class=NAME[,data=FILE][,kw.x=val] │ │ [required] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Evaluation ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --mission-set -S SET Eval suite to run (full Stage 2 diagnosis currently requires │ │ cogsguard_evals). │ │ [default: cogsguard_evals] │ │ --experiments NAME Specific experiments (subset of mission set). │ │ --cogs -c N Agent counts to test (repeatable). │ │ --device DEVICE Policy device (auto, cpu, cuda, cuda:0, etc.). [default: auto] │ │ --scripted-baseline-policy POLICY Reference scripted baseline policy (URI (bundle dir or .zip) or │ │ NAME or class=NAME[,data=FILE][,kw.x=val]) │ │ --known-strong-policy POLICY Reference known-strong policy (URI (bundle dir or .zip) or NAME or │ │ class=NAME[,data=FILE][,kw.x=val]) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Simulation ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --steps -s N Max steps per episode. [default: 1000] │ │ --episodes -e N Episodes per case. [default: 3] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --output-dir DIR Directory for structured diagnose artifacts. │ │ --compare-run-dir DIR Previous diagnose run directory containing doctor_note.json │ │ for interpretation stability comparison. │ │ --stability-reruns N [x>=0] Number of additional fixed-seed reruns to compute │ │ interpretation stability snapshots. │ │ [default: 0] │ │ --require-stable-interpretation Mark diagnosis incomplete when interpretation stability check │ │ fails. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames diagnose ./train_dir/my_run Default CogsGuard evals cogames diagnose lstm --scripted-baseline-policy scripted.basic Compare against scripted baseline cogames diagnose lstm --known-strong-policy my_best_policy Normalize against known-strong policy cogames diagnose lstm --compare-run-dir outputs/cogames-diagnose/prev_run Stability comparison
Usage: cogames version [OPTIONS]
Show version information for cogames and dependencies.
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: cogames docs [OPTIONS] DOC
Print documentation (run without arguments to see available docs).
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ doc_name DOC Document name (readme, mission, technical_manual, scripted_agent, evals, mapgen). │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames docs List available documents cogames docs readme Print README cogames docs mission Print mission briefing
Usage: cogames policies [OPTIONS]
Show available policy shorthand names.
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: Use these shorthand names with --policy or -p: cogames play -m arena -p class=random Use random policy cogames play -m arena -p class=baseline Use baseline policy
Usage: cogames create-bundle [OPTIONS]
Create a submission bundle zip from a policy.
╭─ Policy ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * --policy -p POLICY Policy specification: URI (bundle dir or .zip) or NAME or │ │ class=NAME[,data=FILE][,kw.x=val]. │ │ [required] │ │ --init-kwarg -k KEY=VAL Policy init kwargs (can be repeated). │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --output -o PATH Output path for the bundle zip. [default: submission.zip] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Files ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --include-files -f PATH Files or directories to include (can be repeated). │ │ --setup-script PATH Python setup script to include in the bundle. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: cogames validate-bundle [OPTIONS]
Validate a policy bundle runs correctly in Docker.
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * --policy -p URI Bundle URI (file://, s3://, or local path to .zip or directory). [required] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Tournament ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --season SEASON Tournament season (determines which game to validate against). │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Server ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --server URL Tournament server URL (used to resolve default season). │ │ [default: https://api.observatory.softmax-research.net] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Validation ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --image TEXT Docker image for container validation. [default: ghcr.io/metta-ai/episode-runner:latest] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: cogames login [OPTIONS]
Shortcut for `cogames auth login`.
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --force -f Re-authenticate even if already logged in. │ │ --timeout -t SECS Authentication timeout in seconds. [default: 300] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Server ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --login-server URL Authentication server URL. [default: https://softmax.com/api] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: cogames submissions [OPTIONS]
Show your uploads and tournament submissions.
╭─ Filter ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --policy -p POLICY Filter by policy name (e.g., 'my-policy' or 'my-policy:v3'). │ │ --season SEASON Filter by tournament season. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Server ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --login-server URL Authentication server URL [default: https://softmax.com/api] │ │ --server -s URL Tournament server URL [default: https://api.observatory.softmax-research.net] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --json Print raw JSON instead of table. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames submissions All your uploads cogames submissions --season beta-cogsguard Submissions in a season cogames submissions -p my-policy Info on a specific policy
Usage: cogames leaderboard [OPTIONS]
Show tournament leaderboard for a season.
╭─ Tournament ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --season SEASON Tournament season name (default: server default). │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Server ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --login-server URL Authentication server URL [default: https://softmax.com/api] │ │ --server -s URL Tournament server URL [default: https://api.observatory.softmax-research.net] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --json Print raw JSON instead of table. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames leaderboard --season beta-cogsguard View rankings
Usage: cogames matches [OPTIONS] MATCH_ID
Show your recent matches and policy logs.
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ match_id TEXT Match ID to show details for. If omitted, lists recent matches. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Filter ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --season -s SEASON Tournament season (for listing matches). │ │ --limit -n N Number of matches to show. [default: 20] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --logs -l Show available policy logs for the match. │ │ --download-logs -d DIR Download all accessible logs to directory. │ │ --json Print raw JSON instead of table. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Server ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --login-server URL Authentication server URL. [default: https://softmax.com/api] │ │ --server URL Tournament server URL. [default: https://api.observatory.softmax-research.net] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames matches List recent matches cogames matches <match-id> Show match details cogames matches <match-id> --logs Show available logs cogames matches <match-id> -d ./logs Download logs
Usage: cogames match-artifacts [OPTIONS] MATCH_ID ARTIFACT_TYPE
Retrieve artifacts for a match (logs, etc.).
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * match_id TEXT Match ID to fetch artifacts for. [required] │ │ artifact_type TEXT Type of artifact to retrieve (e.g. 'logs'). [default: logs] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Filter ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --policy -p POLICY_VERSION_ID Policy version ID. If omitted, uses your first policy in the match. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --output -o FILE Save artifact to file. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Server ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --login-server URL Authentication server URL. [default: https://softmax.com/api] │ │ --server URL Tournament server URL. [default: https://api.observatory.softmax-research.net] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames match-artifacts <match-id> Get match logs cogames match-artifacts <match-id> logs -o out.txt Save to file
Usage: cogames upload [OPTIONS]
Upload a policy to CoGames.
╭─ Upload ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * --name -n NAME Name for your uploaded policy. [required] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Policy ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * --policy -p POLICY Policy specification: URI (bundle dir or .zip) or NAME or │ │ class=NAME[,data=FILE][,kw.x=val]. │ │ [required] │ │ --init-kwarg -k KEY=VAL Policy init kwargs (can be repeated). │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Files ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --include-files -f PATH Files or directories to include (can be repeated). │ │ --setup-script PATH Python setup script to run before loading the policy. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Tournament ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --season SEASON Tournament season (default: server's default season). │ │ --no-submit Upload without submitting to a season. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Validation ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --dry-run Run validation only without uploading. │ │ --skip-validation Skip policy validation in Docker. │ │ --image TEXT Docker image for container validation. │ │ [default: ghcr.io/metta-ai/episode-runner:latest] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Server ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --login-server URL Authentication server URL. [default: https://softmax.com/api] │ │ --server URL Tournament server URL. [default: https://api.observatory.softmax-research.net] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames upload -p ./train_dir/my_run -n my-policy Upload and submit to default season cogames upload -p ./run -n my-policy --season beta-cvc Upload and submit to specific season cogames upload -p ./run -n my-policy --no-submit Upload without submitting cogames upload -p lstm -n my-lstm --dry-run Validate only
Usage: cogames submit [OPTIONS] POLICY
Submit a policy to a tournament season.
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * policy_name POLICY Policy name (e.g., 'my-policy' or 'my-policy:v3' for specific version). │ │ [required] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Tournament ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --season SEASON Tournament season name. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Server ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --login-server URL Authentication server URL. [default: https://softmax.com/api] │ │ --server -s URL Tournament server URL. [default: https://api.observatory.softmax-research.net] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames submit my-policy Submit to default season cogames submit my-policy:v3 --season beta-cvc Submit specific version to specific season
Usage: cogames ship [OPTIONS]
Bundle, validate, upload, and submit a policy in one command.
╭─ Upload ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * --name -n NAME Name for your uploaded policy. [required] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Policy ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * --policy -p POLICY Policy specification: URI (bundle dir or .zip) or NAME or │ │ class=NAME[,data=FILE][,kw.x=val]. │ │ [required] │ │ --init-kwarg -k KEY=VAL Policy init kwargs (can be repeated). │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Files ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --include-files -f PATH Files or directories to include (can be repeated). │ │ --setup-script PATH Python setup script to run before loading the policy. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Tournament ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --season SEASON Tournament season (default: server's default season). │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Validation ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --dry-run Run validation only without uploading. │ │ --skip-validation Skip policy validation in Docker. │ │ --image TEXT Docker image for container validation. │ │ [default: ghcr.io/metta-ai/episode-runner:latest] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Server ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --login-server URL Authentication server URL. [default: https://softmax.com/api] │ │ --server URL Tournament server URL. [default: https://api.observatory.softmax-research.net] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames ship -p ./train_dir/my_run -n my-policy Ship and submit cogames ship -p ./run -n my-policy --dry-run Validate only
Usage: cogames tutorial play [OPTIONS]
Interactive tutorial - learn to play Cogs vs Clips.
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: cogames tutorial cogsguard [OPTIONS]
Interactive CogsGuard tutorial - learn roles and territory control.
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: cogames tutorial make-policy [OPTIONS]
Create a new policy from a template. Requires --trainable or --scripted.
╭─ Policy Type ───────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --trainable Create a trainable (neural network) policy. │ │ --scripted Create a scripted (rule-based) policy. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --output -o FILE Output file path. [default: my_policy.py] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames tutorial make-policy -t -o my_nn_policy.py Trainable (neural network) cogames tutorial make-policy -s -o my_scripted_policy.py Scripted (rule-based)
Usage: cogames tutorial train [OPTIONS]
Train a policy on one or more missions. By default, our 'lstm' policy architecture is used. You can select a different architecture (like 'stateless' or 'baseline'), or define your own implementing the MultiAgentPolicy interface with a trainable network() method (see mettagrid/policy/policy.py). Continue training from a checkpoint using URI format, or load weights into an explicit class with class=...,data=... syntax. Supply repeated -m flags to create a training curriculum that rotates through missions. Use wildcards (*) in mission names to match multiple missions at once.
╭─ Mission Setup ─────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --mission -m MISSION Missions to train on (wildcards supported, repeatable for curriculum). │ │ --cogs -c N Number of cogs (agents). [default: (from mission)] │ │ --variant -v VARIANT Mission variant (repeatable). │ │ --difficulty LEVEL Difficulty (easy, medium, hard) controlling clips events. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Policy ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --policy -p POLICY Policy to train (URI (bundle dir or .zip) or NAME or │ │ class=NAME[,data=FILE][,kw.x=val]). │ │ [default: class=lstm] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Training ──────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --steps N [x>=1] Number of training steps. [default: 10000000000] │ │ --minibatch-size N [x>=1] Minibatch size for training. [default: 4096] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Hardware ──────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --device DEVICE Device to train on (auto, cpu, cuda, mps). [default: auto] │ │ --num-workers N [x>=1] Number of worker processes. [default: (CPU cores)] │ │ --parallel-envs N [x>=1] Number of parallel environments. │ │ --vector-batch-size N [x>=1] Vectorized environment batch size. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Reproducibility ───────────────────────────────────────────────────────────────────────────────────────────────╮ │ --seed N [x>=0] Seed for training RNG. [default: 42] │ │ --map-seed N [x>=0] MapGen seed for procedural map layout. [default: (same as --seed)] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --checkpoints DIR Path to save training checkpoints. [default: ./train_dir] │ │ --log-outputs Log training outputs. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames tutorial train -m cogsguard_machina_1.basic Basic training cogames tutorial train -m cogsguard_machina_1.basic -p class=baseline Train baseline policy cogames tutorial train -p ./train_dir/my_run:v5 Continue from checkpoint cogames tutorial train -p class=lstm,data=./weights.safetensors Load weights into class cogames tutorial train -m mission_1 -m mission_2 Curriculum (rotates) Wildcard patterns: cogames tutorial train -m 'machina_2_bigger:*' All missions on machina_2_bigger cogames tutorial train -m '*:shaped' All "shaped" missions cogames tutorial train -m 'machina*:shaped' All "shaped" on machina maps
If you use CoGames in your research, please cite:
@software{cogames2025,
title={CoGames: Multi-Agent Cooperative Game Environments},
author={Softmax},
year={2025},
url={https://github.com/Metta-AI/cogames}
}
![@graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=64&v=4){kind=small}
