moe
Qwen3.6-35B-A3B at ~68 tok/s on a 6 GB laptop GPU: expert-level VRAM caching + CPU offload + multi-token prediction in CUDA. ~1.9x tuned llama.cpp / ik_llama.cpp (MTP vs no speculative decoding).
- 10
- Cuda
- MIT
- 3 wk ago
Qwen3.6-35B-A3B at ~68 tok/s on a 6 GB laptop GPU: expert-level VRAM caching + CPU offload + multi-token prediction in CUDA. ~1.9x tuned llama.cpp / ik_llama.cpp (MTP vs no speculative decoding).
Run Mixture-of-Experts LLMs (Qwen3, gpt-oss, GLM, DeepSeek) bigger than your RAM with llama.cpp: hot experts cached in RAM, the rest streamed from SSD. Windows (Vulkan/AMD, NVIDIA) and Linux, long context, coding agents.