V6 — AlphaStar (discounted return)
The conceptual step over V5 is about the value target. AlphaGo Zero trains the value head toward the raw game outcome, ±1 — which treats a turn-8 kill and a turn-40 grind as equally good. At starting authority 10 the whole game is about the fast combat kill, so V6 discounts the outcome by γ per ply: a state's target becomes γ(plies until game end) · (±1), so lines that win in fewer moves get a higher value — steering the net toward aggressive, scrap-for-damage play. V6 also searches deeper (48 simulations) and takes more gradient steps per game, for sharper targets.
lesson_7/policies/v6/__init__.py
"""V6 "AlphaStar" -- discounted-return self-play.
The conceptual breakthrough over V5: AlphaGo Zero's value target is a pure
win/loss (+/-1), which treats a turn-8 kill and a turn-40 grind as equally good.
At starting authority 10 the whole game is about the *fast* combat kill, so V6
discounts the outcome by gamma per ply toward the end of the game
(`value_discount`). Lines that win in fewer moves get a higher value target,
steering the net toward the aggressive scrap-for-damage style the assignment
predicts. V6 also searches deeper (48 sims) and takes more gradient steps per
game, for sharper targets.
"""
from __future__ import annotations
from ..alphastar.train import Config
from ..alphastar.version import Version
VERSION = Version(
"V6 AlphaStar",
Config(
value_discount=0.99,
sims_selfplay=48,
sims_eval=48,
train_steps_per_game=6,
),
seed=0,
)
The discount is applied when the finished game's outcome is written back onto every
stored position (see Trainer.self_play_game on the
MicroStar page — the value_discount
block at the end of the self-play loop).