MiniCPM-SALA × RTX 5090 — Precise Execution Time Analysis per Operation

Understanding Bottlenecks: Why is Decode Slow? What are the Most Effective Optimizations?

0. Your GPU Specifications & Key Concepts
GPU
RTX 5090
Blackwell, GDDR7
VRAM
32 GB
GDDR7
Memory Bandwidth
1,455 GB/s
Your measured value (Theoretical ~1,792)
FP16 Tensor Core
~209 TFLOPS
Dense compute (FP32: 105 TFLOPS)
Compute-to-Bandwidth Ratio
143.6
FLOP/Byte — The Key Metric!
Three Concepts You Must Understand:

① Arithmetic Intensity = Computation ÷ Memory Access = FLOPs ÷ Bytes
The number of floating-point operations performed per byte of data read. This metric determines where the operation's bottleneck lies.

② GPU Balance Point = Compute ÷ Bandwidth = 209 TFLOPS ÷ 1455 GB/s = 143.6 FLOP/Byte
If an operation's arithmetic intensity is < 143.6, it is Memory-Bound: The GPU computes quickly, but data feeding is too slow, leaving compute units idle waiting for data.
If > 143.6, it is Compute-Bound: Data arrives fast enough, but the compute units cannot process it fast enough.

③ GEMM vs GEMV
GEMM: Matrix × Matrix (General Matrix Multiply) — Used during Prefill, calculating multiple tokens simultaneously.
  → Arithmetic Intensity ≈ seq_len (can be very high), typically Compute-Bound.
GEMV: Matrix × Vector (General Matrix-Vector) — Used during Decode, calculating 1 token.
  → Arithmetic Intensity = 2MN ÷ (MN×2 bytes) = 1 FLOP/Byte ≪ 143.6
  → Extremely Memory-Bound! The GPU spends 99% of its time waiting for data to be moved from VRAM to compute units.
// Time calculation for GEMV (Every step in the Decode phase is calculated this way) Time = max(Compute Time, Memory Access Time) = max(2×M×N / 209e12, M×N×2 / 1455e9) // Example: lightning qkv_proj [4096×12288], bf16 Compute Time = 2 × 4096 × 12288 / 209e12 = 0.48 μs Memory Access Time = 4096 × 12288 × 2 / 1455e9 = 69.2 μs ← 144x slower! Actual Time ≈ 69.2 μs (Entirely determined by memory access)
During the Decode phase, for every token generated, the GPU must read the entire model weights from VRAM.
9.48B parameters × 2 bytes = 18.95 GB. One full read takes 18.95 GB ÷ 1455 GB/s = 13.0 ms.
This is the theoretical upper limit of your Decode speed: It is physically impossible to be faster than 13ms/token (in bf16).
With W4A16 Quantization: 4.74 GB ÷ 1455 GB/s = 3.3 ms/token — 4x faster!
1. Decode Phase — Per-Operation Execution Time for 1 Token Generation
During Decode, batch_size=1, seq_len=1. All linear layers are GEMV (Matrix × Vector).
Bottleneck = Time to read the weight matrices from VRAM. The table below lists each operation precisely.
2. Prefill Phase — Per-Operation Execution Time for Prompt Processing
During Prefill, the entire input sequence is processed at once. Linear layers use GEMM (Matrix × Matrix).
Select Context Length:
3. Total Time Comparison Across Different Context Lengths
4. Optimization Gain Estimation