llama-cpp-rdna-boosts
A set of performant RDNA/HIP patches against llama.cpp focusing on high speed inferencing with numerics assurance
- 271
- Shell
- MIT
- 19 h ago
A set of performant RDNA/HIP patches against llama.cpp focusing on high speed inferencing with numerics assurance
Utilities for accessing AMD's Machine-Readable GPU ISA Specifications.
Image text to text model, used with transformers.
Image text to text model, used with transformers.
Measured llama.cpp benchmarks on AMD Radeon RDNA4 with ROCm: RX 9070 XT + Radeon AI PRO R9700 (48 GB). Qwen3.8, Gemma 4, GLM, gpt-oss; GGUF quant quality (KLD), MTP speculative decoding, 128K context, Claude Code with local models.
AWQ/W4A16 quantization, benchmarking and public release of Polish LLMs (PLLuM, Bielik) on AMD RDNA 4 (ROCm + vLLM-native) — reproducible methodology and hardware-envelope studies.
Run one ollama across an MI50 (gfx906) AND a modern RDNA4 card (gfx1200) — combined VRAM, one runtime, model split across both. Multi-arch generalization of MTLoser's gfx906 build.
Can a dense LLM be made sparse for faster local inference? Converting Qwen3.5-9B FFNs into Mixture-of-Experts, with Triton kernels on AMD RDNA4 (ROCm), benchmarked by KL divergence against llama.cpp GGUF quantization and MTP speculative decoding.
GSQ (Gumbel-Softmax Quantization) ported to AMD ROCm: gfx1200/gfx1201 (RDNA4) and gfx115x (RDNA3.5 APUs)