Training corpus 0 chars Β· 0 vocab
Hyperparameters
1.0e-3
16
64
0.010
1.0
Architecture fixed
d_model = 64 Β· heads = 4 Β· blocks = 2 Β· ffn = 2Γ
Tied LM head Β· GELU Β· Pre-LN Β· Causal self-attention
AdamW(Ξ²β=0.9, Ξ²β=0.999, Ξ΅=1e-8)
Tied LM head Β· GELU Β· Pre-LN Β· Causal self-attention
AdamW(Ξ²β=0.9, Ξ²β=0.999, Ξ΅=1e-8)
Control
Click Train to start gradient descent. Loss should fall from βlog(V)β4.55 toward β1.5 after a few thousand steps. The model samples every 50 steps so you can watch it learn.
Generate
200
0.90
20
Loss curveβ Β· β
Token embeddings Β· 2D PCAlive projection
Per-step probabilities (last sample)heatmap
Attention Β· final block Β· final positionheads