V5 — AlphaStar (the first learned policy)
V5 is the first policy in the line that learns. It is a faithful, miniature
AlphaGo Zero: a tiny policy/value network trained purely by self-play, with Monte-Carlo
tree search both generating the training targets and amplifying the network at play time.
The whole stack — autograd, network, Adam, MCTS, the self-play loop — is dependency-free
pure Python (“MicroStar”), living in lesson_7/policies/alphastar/.
A policy version is just a hyperparameter pin. This is all of V5:
lesson_7/policies/v5/__init__.py
"""V5 "AlphaStar" -- the first AlphaGo-Zero-style learned policy.
The line V0-V4 are hand-written rule bots; V5 is the first that *learns* by
self-play + MCTS (see docs/scientific-log/). It pins the baseline AlphaStar
Config: a one-hidden-layer policy/value net, 32-simulation self-play search,
trained at starting authority 10.
"""
from __future__ import annotations
from ..alphastar.train import Config
from ..alphastar.version import Version
VERSION = Version("V5 AlphaStar", Config(), seed=0)
…where Config() defaults are the V5 baseline — a 64-unit one-hidden-layer
net, 32 MCTS simulations per self-play move, trained at starting authority 10 (short,
combat-decisive games):
lesson_7/policies/alphastar/train.py (excerpt)
class Config:
"""AlphaStar hyperparameters. Defaults are the V5 baseline."""
def __init__(self, **kw):
self.hidden = 64
self.c_puct = 1.5
self.sims_selfplay = 32 # MCTS sims per move during self-play
self.sims_eval = 48 # MCTS sims per move at tournament time
self.dirichlet_alpha = 0.5 # scaled up from Go's 0.03 (small action set)
self.dirichlet_eps = 0.25
self.temp_moves = 8 # plies played at tau=1 (then greedy)
self.value_discount = 1.0 # gamma per ply on the outcome target; <1
# rewards winning FASTER (V6+). 1.0 = pure
# AGZ win/loss (V5).
self.starting_authority = 10 # short, combat-decisive training games
self.replay_capacity = 30_000
self.minibatch = 32
self.train_steps_per_game = 4
self.lr = 0.01
self.weight_decay = 1e-4
self.warmup_games = 16 # fill the buffer before training starts
self.max_moves = 400 # safety bound on a self-play game
for k, v in kw.items():
if not hasattr(self, k):
raise AttributeError(f"unknown Config field: {k}")
setattr(self, k, v)
The full MicroStar walk-through — every class, and which part is gradient descent vs. which part is MCTS — is at How MicroStar works.