optimum-benchmark
ποΈ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.
- 342
- Python
- Apache-2.0
- 10 d ago
ποΈ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.
βΈοΈ Easy, advanced inference platform for large language models on Kubernetes. π Star to support our work!
This project demonstrates the process of fine-tuning the Qwen2.5-3B-Instruct model using GRPO (Generalized Reward Policy Optimization) on the GSM8K dataset.
This repo introduces MagicData-CLAM, a Chinese SFT dataset, and provides to the community two relevant models that we finetuned. Contact business@magicdatatech.com for more information.
Rust MoE inference runtime with tiered expert storage β loads real GGUF models; speaks OpenAI, Ollama, Anthropic, llama.cpp & TGI APIs
RisuAI backend with python only. TextGen works, need more memory related updates
Deploy the Defog sqlcoder2 llm on Modal (https://modal.com) using Hugging Face Text Generation Inference (TGI)
Multi-model LLM serving on a single NVIDIA A100 40GB: MIG isolation, LiteLLM, TGI, and Prometheus/Grafana.
Analytics Copilot (Text-to-SQL) is an end-to-end LLM engineering project that builds a production-style Natural Language β SQL assistant. It fine-tunes Mistral-7B-Instruct using QLoRA/LoRA adapters on the b-mc2/sql-create-context dataset, trains with a clean instruction-tuning prompt format, and ships a complete evaluation + deployment workflow.
Hands-on handbook for delivering private LLM deployments: environment survey, VRAM estimation with worked examples, vLLM / SGLang / TGI deployment, TTFT and TPOT benchmarking, acceptance testing and troubleshooting fault trees.
Text Generation Interference example in Windows (docker, WSL is needed)