SOAR Compilation & Submission Full Process

A Complete End-to-End Guide to sgl-kernel: From Source Code to Submission
~85
.cu/.cpp source files in csrc/
150
Total build steps (incl. linking & install)
4
Final .so shared libraries
~569 MB
Total .so size (uncompressed)
Overview

The Entire Process in a Nutshell

Compile CUDA source files (.cu) into GPU-executable code (.so), package them, and submit to the competition platform for execution.
Your Code
.cu / .cpp source files
→
nvcc Compile
.o object files
→
Linker (ld)
.so shared library
→
pip install
site-packages/
→
Package & Submit
.tar.gz
💡
.o files are merely intermediate build artifacts used for incremental compilation (changing one .cu only requires recompiling that specific .o and re-linking). Ultimately, you only need the .so files and .py files. Do not package any .o files.
Build Targets

CMakeLists.txt Defines 4 Build Targets

Each target compiles and links a set of source files into an independent .so shared library.
common_ops_sm90_build
→ sm90/common_ops.abi3.so
The largest and most core target. Contains the vast majority of CUDA kernels: RMSNorm, RoPE, Attention, GEMM (including Marlin GPTQ), MoE, Speculative Decoding, KV Cache I/O, Mamba, etc.

It also embeds external dependencies: FlashInfer (norm/sampling), Sparse Flash Attention, and Fast Hadamard Transform.
~241 MB
flash_ops
→ flash_ops.abi3.so
Flash Attention 3 (Hopper architecture specific). Every combination of dtype (bf16/fp16/fp8) × configuration (paged/split/softcap/packgqa) is an independent .cu file, resulting in a massive number of compilation units.
~321 MB
spatial_ops
→ spatial_ops.abi3.so
Green context stream related functionalities. Only 2 source files, compiles very quickly.
~100 KB
flashmla_ops
→ flashmla_ops.abi3.so
Flash MLA (Multi-head Latent Attention) kernel. Sourced from cmake/flashmla.cmake.
~6.5 MB
Build Steps

Complete Breakdown: [1/150] → [150/150]

150 steps = Compile .o + Link .so + Install to site-packages

[1/150] ~ [~87/150] common_ops_sm90_build

Source files (~85 under csrc/):
csrc/elementwise/fused_add_rms_norm_kernel.cu ← RMSNorm kernel (Where you encountered the error)
csrc/elementwise/rope.cu ← RoPE (Rotary Position Embedding)
csrc/elementwise/activation.cu ← SiLU/GELU activation functions
csrc/attention/cascade.cu ← Cascade attention
csrc/attention/merge_attn_states.cu ← Attention state merging
csrc/gemm/marlin/gptq_marlin.cu ← Marlin W4A16 GEMM (Core quantization acceleration)
csrc/gemm/gptq/gptq_kernel.cu ← GPTQ dequantization
csrc/gemm/fp8_gemm_kernel.cu ← FP8 matrix multiplication
csrc/speculative/eagle_utils.cu ← EAGLE speculative decoding (tree build + verification)
csrc/speculative/speculative_sampling.cu ← Speculative sampling kernel
csrc/moe/moe_align_kernel.cu ← MoE expert routing
csrc/mamba/causal_conv1d.cu ← Mamba convolution
csrc/common_extension.cc ← Python ↔ C++ binding entry point
... and so on, approx. 85 files

Each file is compiled by nvcc to generate an .o:
build/CMakeFiles/common_ops_sm90_build.dir/csrc/elementwise/fused_add_rms_norm_kernel.cu.o
build/CMakeFiles/common_ops_sm90_build.dir/csrc/gemm/marlin/gptq_marlin.cu.o
... approx. 85 .o files

All .o files are linked into a single .so:
build/sm90/common_ops.abi3.so (~241 MB)

[~88/150] ~ [~127/150] flash_ops ← You are currently here (120/150)

Source files (~35 under flash-attention/hopper/):
flash-attention/hopper/flash_api.cpp ← Flash Attention 3 Main API
flash-attention/hopper/flash_prepare_scheduler.cu
flash-attention/hopper/flash_fwd_combine.cu
flash-attention/hopper/instantiations/
flash_fwd_hdimall_bf16_sm90.cu ← bf16 base version
flash_fwd_hdimall_bf16_paged_sm90.cu ← bf16 paged version
flash_fwd_hdimall_bf16_paged_split_sm90.cu ← bf16 paged+split version
flash_fwd_hdimall_bf16_softcap_sm90.cu ← bf16 softcap version
flash_fwd_hdimall_bf16_packgqa_sm90.cu ← bf16 packgqa version
flash_fwd_hdimall_fp16_*.cu ← fp16 (same 5-6 variants)
flash_fwd_hdimall_e4m3_*.cu ← fp8 (same 5-6 variants)
flash_fwd_hdimdiff_*.cu ← variants for different head dims
... totaling approx. 35 .cu files

After linking:
build/flash_ops.abi3.so (~321 MB — The largest .so!)
📝
flash_ops is so large because every dtype × attention variant acts as an independent compilation unit. Each .cu file is specialized by nvcc into a highly optimized kernel, embedding both SASS machine code + PTX intermediate representation.

[~128/150] ~ [~135/150] spatial_ops + flashmla_ops

csrc/spatial/greenctx_stream.cu + csrc/spatial_extension.cc
→ build/spatial_ops.abi3.so (~100 KB)

Source files defined in cmake/flashmla.cmake
→ build/flashmla_ops.abi3.so (~6.5 MB)

[~136/150] ~ [150/150] Installation (cmake install + pip)

Copy from build/ to site-packages:
build/sm90/common_ops.abi3.so → site-packages/sgl_kernel/sm90/common_ops.abi3.so build/flash_ops.abi3.so → site-packages/sgl_kernel/flash_ops.abi3.so build/spatial_ops.abi3.so → site-packages/sgl_kernel/spatial_ops.abi3.so build/flashmla_ops.abi3.so → site-packages/sgl_kernel/flashmla_ops.abi3.so
Full path: /opt/SGLang-MiniCPM-SALA/sglang_minicpm_sala_env/lib/python3.10/site-packages/
Path Mapping

Relationship Between the Three Directories

Directory Role Content Need to Package?
.../sgl-kernel/csrc/ Source Dir The .cu and .cpp files you write/modify ❌ No
.../sgl-kernel/build/ Build Output .o intermediate files + linked .so libraries ❌ No (For incremental build only)
.../site-packages/sgl_kernel/ Install Dir Final .so + .py (Where Python loads them at runtime) ✅ Yes, grab .so from here
.../sgl-kernel/python/sgl_kernel/ Python Source .py files (Python wrapper calling .so) ✅ Yes, grab .py from here
Package & Submit

Wheel Packaging Logic

Your packaging script fetches files from two places to assemble the .whl:
# What your packaging script does: # 1. Fetch .py files from Python source directory /opt/.../sgl-kernel/python/sgl_kernel/*.py → inside the whl sgl_kernel/*.py # 2. Fetch compiled .so files from site-packages /opt/.../site-packages/sgl_kernel/sm90/common_ops.abi3.so → inside the whl sgl_kernel/sm90/common_ops.abi3.so (241 MB) /opt/.../site-packages/sgl_kernel/flash_ops.abi3.so → inside the whl sgl_kernel/flash_ops.abi3.so (321 MB) /opt/.../site-packages/sgl_kernel/flashmla_ops.abi3.so → inside the whl sgl_kernel/flashmla_ops.abi3.so (6.5 MB) /opt/.../site-packages/sgl_kernel/spatial_ops.abi3.so → inside the whl sgl_kernel/spatial_ops.abi3.so (100 KB) # 3. Fetch dist-info from site-packages /opt/.../site-packages/sgl_kernel-0.3.20.dist-info/* # Final Output: /opt/submission/sgl_kernel-0.3.20-cp310-abi3-linux_x86_64.whl (~250 MB compressed)
⚠️
Note: The packaging script uses os.walk to traverse sgl_kernel/ under site-packages to find .so files. Ensure flash_ops.abi3.so and flashmla_ops.abi3.so are in the root of the sgl_kernel/ directory (not in subdirectories), so os.walk can find them.

Do not package the sm100/ directory — that is handled via a symlink in prepare_env.sh.
Submission Payload Structure

Inside submission.tar.gz

After extracting submission.tar.gz: ├── prepare_env.sh ← REQUIRED Environment setup script ├── sgl_kernel-0.3.20-cp310-abi3-linux_x86_64.whl ← Your compiled CUDA kernels │ └── (Contains 4 .so files + all .py + dist-info) └── sglang/python/ ← Your modified sglang Python source code └── sglang/srt/ ├── models/minicpm.py ← Model definitions (if modified) ├── layers/... └── ...
# prepare_env.sh contents: #!/bin/bash set -e SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # 1. Install your sglang Python code (editable mode, replaces image's built-in version) uv pip install --no-deps -e "${SCRIPT_DIR}/sglang/python" # 2. Install your compiled sgl-kernel wheel (replaces built-in .so files) uv pip install --force-reinstall --no-deps "${SCRIPT_DIR}/sgl_kernel-0.3.20-cp310-abi3-linux_x86_64.whl" # 3. Fallback sm120/sm100 architecture to sm90 (via symlink) SGLK_DIR=$(python3 -c "import sgl_kernel; print(sgl_kernel.__path__[0])") ln -sf "${SGLK_DIR}/sm90" "${SGLK_DIR}/sm100"
Platform Execution

What the Competition Platform Does With Your tar.gz

① Boot Base Image Automated

# Platform boots a docker container, pre-installed with: # - Python 3.10 # - Original SGLang + sgl-kernel (pre-compiled) # - CUDA Toolkit # - Model mounted at /models/MiniCPM-SALA

② Extract + Execute prepare_env.sh Run via source

# Platform extracts your submission tar -xzf submission.tar.gz # Execute via source (so exported env vars take effect) source ./prepare_env.sh # At this point, sgl_kernel in site-packages is replaced by your compiled version # sglang Python code is also replaced by your modified version

③ Launch SGLang Inference Server Automated

# Platform launches the server with SGLANG_SERVER_ARGS python3 -m sglang.launch_server \ --model /models/MiniCPM-SALA \ ${SGLANG_SERVER_ARGS} # Python loading sequence at startup: # import sglang → Your sglang/python/sglang/ (editable install) # import sgl_kernel → site-packages/sgl_kernel/__init__.py # → load_utils.py detects GPU architecture # → Loads sm90/common_ops.abi3.so (or via sm100 symlink) # → Loads flash_ops.abi3.so # → Loads flashmla_ops.abi3.so # → Loads spatial_ops.abi3.so # → torch.ops.sgl_kernel.* registers all CUDA kernels

④ Correctness Testing eval_model.py

# Public + Private sets, accuracy must be > 97% or it gets disqualified python3 eval_model.py --api_base http://127.0.0.1:30000 ...

⑤ Speed Testing (3 Concurrency Levels) bench_serving.sh

# Measures benchmark_duration, lower is better --max-concurrent 1 # No concurrency (Weight: 40%) --max-concurrent 8 # Low concurrency (Weight: 30%) No --max-concurrent # High concurrency (Weight: 30%)
Incremental Compilation

Why You Should Keep the build/ Directory

The .o files in the build/ directory save you from recompiling all 150 files from scratch every time.
# Scenario: You only modified csrc/elementwise/fused_add_rms_norm_kernel.cu # With build/ directory (Incremental compile): [1/150] Building CUDA: fused_add_rms_norm_kernel.cu.o ← Only recompiles this 1 .o file [2/150] Linking: sm90/common_ops.abi3.so ← Re-links (using 84 old .o files + 1 new .o) # Time taken: ~3-5 minutes # Without build/ directory (Full compile): [1/150] ~ [150/150] ← Starts completely from scratch # Time taken: ~2-4 hours
💾
Therefore, ALWAYS backup your build/ directory!
tar -czf /opt/sgl_kernel_build_backup.tar.gz -C .../sgl-kernel build/
This way, even if the environment breaks, restoring it will only require incrementally recompiling the modified files.
Troubleshooting

Resolving "no kernel image is available" Error

🔴
RuntimeError: RMSNorm failed with error code no kernel image is available for execution on the device
What this error means: The .so file does not contain executable code for the current GPU.
# The .so file can contain two types of code: SASS (Machine Code) ← Architecture-specific, direct execution, fastest Example: sm_90 SASS can only run on sm_90 GPUs PTX (Intermediate Code) ← Universal, JIT compiled to current architecture's SASS at runtime Example: compute_90a PTX can be JIT compiled on sm_90, sm_100, sm_120 # Your CMakeLists.txt compile arguments: -gencode=arch=compute_90,code=sm_90 # Embeds sm_90 SASS -gencode=arch=compute_90a,code=compute_90a # Embeds compute_90a PTX ← THIS IS KEY! # With PTX, it can be JIT executed on any >= sm_90 GPU # Your 5090 (sm_120) ✅ Competition Platform (RTX PRO, likely also Blackwell) ✅
🔍
After compiling, use this command to verify the .so contains PTX:
cuobjdump -lptx site-packages/sgl_kernel/sm90/common_ops.abi3.so | head
If you see compute_90a in the output, it means everything is correct.