MiniCPM-SALA

Complete Architecture Blueprint — Every tensor, every gate, every memory transfer

9,477,203,968 params18.95 GB bf1632 layers8 Sparse (InfLLM-V2)24 Linear (Lightning Attn)vocab 73,448 †

Input Embedding

token_id : scalar ∈ [0, 73447]

→embed_tokens[73448 × 4096]300.84M · 601 MB→raw [1×4096]→×12→h₀ [1×4096] bf16

µP embedding scale · embed_tokens NOT tied to lm_head

Output — Final Norm → LM Head → Sample

h₃₁ [1×4096]

→RMSNormnorm.w[4096]8 KB→h_n→÷16→h_s [1×4096]

scale_width = 4096/256 = 16

→lm_head[4096×73448]300.84M · 601 MB · NOT tied→logits [1×73448]→softmax→sample→output token ✓

MiniCPM-SALA Blueprint · 32 layers · InfLLM-V2 3-stage + 2-pass LSE · Lightning Attention HBM↔SRAM tiling · QK-Norm all layers · HyPE (NoPE sparse / RoPE linear)

† Values not explicitly stated in the provided papers (inferred from MiniCPM-4.0 codebase):
vocab_size = 73,448 · residual_scale = 0.24749 (µP: 1.4/√32) · scale_emb = 12 · scale_width = 16