ByteCross
LLM inference: compare activation and KV-cache sparsity through memory-traffic models and GPU decoding benchmarks.
- 1
- Python
- MIT
- 6 wk ago
LLM inference: compare activation and KV-cache sparsity through memory-traffic models and GPU decoding benchmarks.
Can a dense LLM be made sparse for faster local inference? Converting Qwen3.5-9B FFNs into Mixture-of-Experts, with Triton kernels on AMD RDNA4 (ROCm), benchmarked by KL divergence against llama.cpp GGUF quantization and MTP speculative decoding.