MiniCPM-SALA 9.48B Model Architecture Overview

Area of each block is strictly proportional to parameter count | Total Parameters: 9,477,203,968 | Includes corresponding source code

Total Parameters
9,477,203,968
bf16: 17.65 GiB | W4A16: ~4.41 GiB
Lightning Layers ⚡ (Linear Attention)
6,845,405,184
24 Layers × 285,225,216 | 72.2% of Parameters
MiniCPM4 Layers 🔍 (Sparse Attention)
2,030,108,672
8 Layers × 253,763,584 | 21.4% of Parameters
Non-Layer Parameters
601,690,112
embed + lm_head + norm | 6.3%
1. Complete Forward Flow (Left: Compute Flow & Parameters | Right: Source Code)
Entry & Exit (Non-Layer Parameters) 601,690,112 params (6.3%)
Compute Flow & ParametersSource Code (minicpm.py)
● embed_tokens
token_id → [4096] vector | Lookup table 73,448 × 4,096
300,843,008 params (573.0 MiB)
# MiniCPMModel.forward() hidden_states = self.embed_tokens(input_ids) * self.config.scale_emb # scale_emb = 12 (constant in config.json, not a parameter)
↓ Processed through 32 Decoder Layers ↓
● model.norm
RMSNorm(4096)
4,096 params (8.0 KiB)
# MiniCPMModel.forward() hidden_states = self.norm(hidden_states)
● lm_head
[4096] → [73,448] logits | 4,096 × 73,448
300,843,008 params (573.0 MiB)
# MiniCPMForCausalLM.forward() hidden_states = hidden_states / self.scale_width # scale_width = hidden_size / dim_model_base = 4096/256 = 16 return self.logits_processor(input_ids, hidden_states, self.lm_head, forward_batch) # → softmax → top-k/sampling → next token
Decoder Layer Wrapper (Shared structure across 32 layers) MiniCPMDecoderLayer.forward()
Residual connection + Scaling structure
# MiniCPMDecoderLayer.forward() residual = hidden_states hidden_states = self.input_layernorm(hidden_states) hidden_states = self.self_attn(positions, hidden_states, forward_batch) hidden_states = residual + hidden_states * (1.4 / math.sqrt(32)) # scale_depth=1.4, num_hidden_layers=32 → coefficient ≈ 0.2475 residual = hidden_states hidden_states = self.post_attention_layernorm(hidden_states) hidden_states = self.mlp(hidden_states) hidden_states = residual + hidden_states * (1.4 / math.sqrt(32))
⚡ Lightning Layers (Linear Attention) — 24 Layers 285,225,216 params/layer | Total 6,845,405,184
Compute Flow & ParametersSource Code (MiniCPMLightningMixer)
● input_layernorm
RMSNorm(4096)
4,096 params
hidden = self.input_layernorm(hidden)
● qkv_proj ← Largest GEMM!
[4096] → [12288] = Q(4096)+K(4096)+V(4096)
32 Q heads × 128 dim + 32 KV heads × 128 dim × 2
50,331,648 params (96.0 MiB) 🔥
qkv, _ = self.qkv_proj(hidden_states) q, k, v = qkv.split([4096, 4096, 4096], dim=-1) # QKVParallelLinear: 4096 → 4096+4096+4096 # 32 heads × 128 dim for Q, K, V each
● q_norm + k_norm
RMSNorm(128) × 2, per head
128 + 128 = 256 params
q = self.q_norm(q.reshape(-1, 128)) k = self.k_norm(k.reshape(-1, 128))
○ RoPE (No params)
Rotary Position Embedding, fp32 upcast optimized out
0 params (config constant)
# Optimization applied: Removed q.float(), k.float() # orig_dtype = q.dtype # q, k = q.float(), k.float() ← Removed q, k = self.rotary_emb(positions, q, k) # q, k = q.to(orig_dtype), k.to(orig_dtype) ← Removed
○ SimpleGLA Kernel (No weight parameters)
Recurrent State: (32, 128, 128) = 1.0 MiB/layer per request
decay: ALiBi slopes, fixed and non-learnable
0 learnable params | State is not stored in safetensors
# SimpleGLAAttnBackend.forward() mode = "fused_recurrent" if seq_len < 64 else "chunk" if mode == "fused_recurrent": # decode / short sequences o, final = fused_recurrent_simple_gla( q, k, v, g_gamma=self.g_gamma, # ALiBi slopes scale=self.scale, # 1/√128 = 0.0884 initial_state=initial_state, ...) else: # long prefill o, final = chunk_simple_gla( q, k, v, g_gamma=self.g_gamma, ...) # Core recurrence: S = decay × S + k⊗v, o = q × S
● o_norm
RMSNorm(4096)
4,096 params
o = self.o_norm(o)
● z_proj (output gate)
[4096] → [4096] → sigmoid → × o
16,777,216 params (32.0 MiB)
z, _ = self.z_proj(hidden_states) # Note: Uses original hidden_states! o = o * F.sigmoid(z)
● o_proj
[4096] → [4096]
16,777,216 params (32.0 MiB)
y, _ = self.o_proj(o) return y
↓ Residual + Scaling ↓
● post_attention_layernorm
RMSNorm(4096)
4,096 params
hidden = self.post_attention_layernorm(hidden)
● MLP: gate_up_proj
[4096] → [32768] (gate 16384 + up 16384)
134,217,728 params (256.0 MiB) 🔥
# MiniCPMMLP.forward() gate_up, _ = self.gate_up_proj(x) # Merged: [gate|up] x = self.act_fn(gate_up) # SiluAndMul: gate portion passes through SiLU, multiplied by up portion # output dim = 16384 (halved)
● MLP: down_proj
[16384] → [4096]
67,108,864 params (128.0 MiB)
x, _ = self.down_proj(x) return x
🔍 MiniCPM4 Layers (Sparse Attention) — 8 Layers 253,763,584 params/layer | Total 2,030,108,672
Compute Flow & ParametersSource Code (MiniCPMAttention)
● input_layernorm
RMSNorm(4096)
4,096 params
hidden = self.input_layernorm(hidden)
● qkv_proj (Smaller than Lightning!)
[4096] → [4608] = Q(4096)+K(256)+V(256)
32 Q heads × 128 dim + 2 KV heads × 128 dim × 2
18,874,368 params (36.0 MiB)
qkv, _ = self.qkv_proj(hidden_states) q, k, v = qkv.split([4096, 256, 256], dim=-1) # GQA: 32 Q heads, only 2 KV heads # KV Cache is extremely small!
○ ❌ No RoPE, No qk_norm
attn_use_rope = false in config.json
0 params
# config.json: "attn_use_rope": false # Thus minicpm4 layers skip RoPE # Also no qk_norm if self.attn_use_rope: # False → Skipped q, k = self.rotary_emb(positions, q, k)
○ RadixAttention (Standard softmax)
FlashInfer kernel | Requires KV Cache
KV Cache: 2 heads × 128 dim × 2(K+V) × 2B = 1 KiB/token/layer
0 learnable params | KV Cache memory allocated separately
attn_output = self.attn(q, k, v, forward_batch) # RadixAttention → FlashInfer # Decode: Read KV Cache, compute softmax attention # Prefill ctx≤8192: Dense attention # Prefill ctx>8192: Sparse attention # window=2048, topk=64 blocks×64 = 4096 # Each token attends to at most 6144 tokens
● o_gate (Note: Not z_proj!)
[4096] → [4096] → sigmoid → × attn_output
16,777,216 params (32.0 MiB)
o_gate_output, _ = self.o_gate(hidden_states) attn_output = attn_output * F.sigmoid(o_gate_output) # Functions identical to z_proj in Lightning # But named differently: o_gate vs z_proj
● o_proj
[4096] → [4096]
16,777,216 params (32.0 MiB)
output, _ = self.o_proj(attn_output) return output
↓ Residual + Scaling ↓
● post_attention_layernorm
RMSNorm(4096)
4,096 params
hidden = self.post_attention_layernorm(hidden)
● MLP: gate_up_proj
[4096] → [32768] (Identical to Lightning layer)
134,217,728 params (256.0 MiB) 🔥
# MLP is identical to Lightning layers gate_up, _ = self.gate_up_proj(x) x = self.act_fn(gate_up)
● MLP: down_proj
[16384] → [4096]
67,108,864 params (128.0 MiB)
x, _ = self.down_proj(x) return x
⚠️ minicpm4 has 31,461,632 fewer params/layer than Lightning
Difference stems from: qkv_proj (2 vs 32 KV heads) saves 31,457,280
Plus omission of q_norm/k_norm/o_norm, matching the exact param delta
# Parameter difference: # lightning qkv: 4096×12288 = 50,331,648 # minicpm4 qkv: 4096×4608 = 18,874,368 # Diff: 31,457,280 (Accounts for almost the entire difference)
2. Full Model Parameter Treemap (Area = Parameter Count, Hover for Details)
3. Layer-by-Layer Parameter Comparison
4. VRAM Usage — Various Context Lengths (Single Request)