sglang-rtxpro6000
Optimized SGLang runtime for Qwen3.8-27B FP8 with DFlash2 and Qwen3.8 Flash-Next NVFP4 with FR-Spec on one NVIDIA RTX PRO 6000 Blackwell 96 GB GPU (SM120): 524K context, HiCache and NIXL.
- 206
- Python
- Apache-2.0
- 14 h ago
Optimized SGLang runtime for Qwen3.8-27B FP8 with DFlash2 and Qwen3.8 Flash-Next NVFP4 with FR-Spec on one NVIDIA RTX PRO 6000 Blackwell 96 GB GPU (SM120): 524K context, HiCache and NIXL.
(1) RL for CUDA & Triton GPU kernels generation/optimiz. (2) Deployment w Dynamo and inference with vLLM & SGLang. (3) Disaggregated serving on TensorRT-LLM with Tensor/Expert/ Pipeline parallelism (4) E2E multi-agent sys Nemotron-550B across LangGraph, 50+ MCP registry tools w OilPriceAPI, and pgVector stores.
A policy layer above transport for KV movement, workload-aware admissibility, and explainable routing in disaggregated inference.
Does prefill/decode disaggregation pay for an MoE model? SLO-first goodput benchmark of Qwen3-30B-A3B on 4x H100 and 8x A100 with vLLM, llm-d and NVIDIA Dynamo, with per-request data.