High-performance LLM, VLM, embedding, reranking, and audio inference for Apple Silicon, NVIDIA CUDA and AMD ROCm systems.
Repo lablup/mlxcel, read from GitHub 1 min ago. lablup has not joined Take My Repo.
lablup/mlxcel is a public repo on GitHub by lablup. High-performance LLM, VLM, embedding, reranking, and audio inference for Apple Silicon, NVIDIA CUDA and AMD ROCm systems. It declares the Apache-2.0 license on GitHub. It is written mainly in Rust, has 504 stars and last changed on 2026-11-20. Nobody has reviewed it on Take My Repo yet, which says nothing about its quality. Read from GitHub on 2026-11-21.
A link is a mention. It does not prove the project needs that model to run.
In the maintainer's words. Shown from GitHub; Take My Repo does not keep a copy.
High-performance LLM, VLM, embedding, reranking, image generation, and audio inference for Apple Silicon and NVIDIA CUDA systems, with AMD ROCm support on Linux and Windows. The CLI and server are implemented in Rust and execute MLX SafeTensors checkpoints through native MLX C++ bindings, without Python in the request path or a checkpoint-conversion step.
mlxcel is both a local inference CLI and a production-oriented model server. It runs text generation, multimodal input, embeddings, reranking, token classification, image generation and editing, speech transcription and synthesis, continuous batching, prompt caching, speculative decoding, and distributed inference in one native runtime.
Apple Silicon is the primary target. Linux/CUDA is supported as a secondary target. AMD GPUs on Linux and Windows (ROCm) are supported, source-build-only targets; native Windows with NVIDIA CUDA is experimental; and an opt-in OpenXLA/IREE backend is available as an alpha development path.
The current main branch is v0.9.0-beta.1 plus unreleased work. Install from source to use features that have not reached the latest tagged release; see the changelog for the release-by-release record.
--features rocm builds mlxcel on Linux and on Windows x86_64 against an MLX ROCm backend vendored into the source tree as the ROCm part of mlxcelverse, applied on the same pinned MLX commit as the Metal and CUDA builds. Affine, mxfp8, and mxfp4 checkpoints, including gpt-oss MoE experts, run on RDNA 3.5 (gfx1151); the fused kernels with a HIP port (BitLinear, the samplers, the SSM update step, the Mamba1 scan, the decode-MoE kernels, xIELU, the add3 LayerNorm, and the fused add-RMSNorm and RoPE + append, which are on by default on ROCm) run natively and the rest fall back to MLX graphs. The backend is a source build; see Linux with AMD ROCm and Windows with AMD ROCm for the validated configurations and the open gaps.--webui serves a browser interface from mlxcel serve or mlxcel-server: a model library with download, load, unload, and delete; local streaming chat; runtime activity; and scoped settings with next-load profiles. See the WebUI guide.llama-server compatibility. A frozen b10621 manifest classifies all 376 pinned options, routes, and native request fields, with no deferred entries. Native completion, embedding, tokenization, template, infill, props, slots, metrics, resumable-stream, router, and LoRA surfaces are implemented or explicitly classified instead of being silently ignored.llama-server routes. Optional Vertex AI custom-container routing is supported through the standard AIP_* variables.GET /v1/internal/mtp-policy. LFM2.5 targets pair with LiquidAI's published DSpark drafters on mlxcel-server (--draft-model), greedy decoding only. Muse Glimmer pairs with meta-models/Muse-Glimmer-30B-assistant on text-only requests, and the round loop measures its verify width instead of fixing it. mlxcel split-mtp extracts the GLM-4.7-Flash drafter from the raw checkpoint.deepseek_v4 architecture includes HyperConnections, rotating shared-KV attention, per-layer compression, HiSA sparse selection, and hash-routed early MoE layers.max that returned NaN for a finite input.--parallel instead of one forward per row. On M1 Ultra, Llama 3.1 8B 4-bit reaches 1.99x aggregate at 8 concurrent requests over 1.--image-model with no chat model source starts either server binary in an image-only mode: the image worker is the primary model, chat and tokenizer routes answer a structured 501 naming the mode, and /v1/models, /health and the image routes serve normally.mimo_v2_flash architecture is a 48-layer hybrid sliding/full attention sparse decoder with 256 routed experts, selective head widths (192 for Q/K, 128 for V), per-layer partial RoPE bases, and a noaux_tc sigmoid router with a selection-only correction bias. mlx-community/MiMo-V2-Flash-4bit loads directly.mimo_v2 keeps the MiMo v2 Flash decoder unchanged and adds the mimovl vision tower for image and video input through the Qwen2.5-VL processor protocol. The loader unfuses the checkpoint's tensor-parallel qkv_proj (per-shard FP8 grids, including the padded 108-row degree-4 case) and repacks MXFP4 routed experts, covering both the native XiaomiMiMo/MiMo-V2.6-Flash-RL release and the converted nativ-community/MiMo-V2.6-Flash-RL-MLX-4bit one. Audio is not supported yet.MLXCEL_MLA_ABSORBED unset, deepseek_v2 checkpoints and DeepSeek-VL2's decoder run absorbed decode over the (ckv, kpe) latent on Metal, CUDA and ROCm, which stores 8.9x less KV per token on DeepSeek-V2-Lite. Decode is within a few percent of the decompressed layout at a 512-token prompt and faster behind long prompts. MLXCEL_MLA_ABSORBED=0 restores the decompressed cache. See MLA absorbed decode.POST /v1/responses with background: true returns a queued response at once; the job can be polled, streamed and resumed from an event offset, or cancelled, on a single model server and on the multi-model router.See Server features for the route and deployment map and llama-server compatibility for the exact b10621 boundary.
mlx-community, load directly. Many standard SafeTensors embedding checkpoints also load without conversion.mlxcel and mlxcel-server are native executables suitable for packaging and service supervision. Platform runtime libraries are still required.mlxcel-server accepts a broad llama-server flag and LLAMA_ARG_* environment surface, while the checked-in manifest records every supported, aliased, not-applicable, or intentional-difference case.--surgery / MLXCEL_SURGERY, including scale, add, prune, replace, and interpolate.mlxcel arch for the binary's architecture catalog and consult Supported models for checkpoint-level notes.The Homebrew formula installs the latest released mlxcel and mlxcel-server binaries on macOS Apple Silicon:
brew tap lablup/tap brew install lablup/tap/mlxcel
Linux builds are CUDA-specific and are not published to the tap. See the latest release for the mlxcel-linux-*-cuda13 archives, or Build from source.
mlxcel run resolves the model, downloads it on first use, reuses it afterward, and starts an interactive chat REPL. A bare model name resolves under mlx-community by default.
# Interactive chat. mlxcel run Qwen3.5-0.8B-4bit # One-shot generation, then exit. mlxcel run Qwen3.5-0.8B-4bit -p "Hello, world!" -n 100 # With no model argument, use mlx-community/gemma-4-e2b-it-4bit. mlxcel run
generate, serve, and inspect accept the same model forms through -m: an existing local path, a HuggingFace owner/name repository id, or a bare name. MLXCEL_DEFAULT_ORG changes the bare-name organization.
Thinking-capable checkpoints may generate reasoning before the final answer. run and generate hide reasoning from terminal output by default; pass --show-reasoning to display it without the raw marker tokens.
run and its chat REPL are clients of the server engine: they start the mlxcel-server model worker in-process (one slot, no HTTP listener) and send each turn through the same /v1/chat/completions request path, so mlxcel run -p and the server give the same output for the same request. Sampling flags map onto the server options of the same name (--temp, --top-k, --repeat-last-n, --frequency-penalty, --dry-sequence-breaker, --stop, ...); with none given, the base sampler stays greedy unless generation_config.json sets one. --draft-model (with --draft-kind dflash|mtp when the drafter does not say) runs speculative decoding through the server's DFlash and MTP paths. In the REPL, /clear starts a new conversation, /image <path> attaches an image to the next message, and Ctrl-C stops the reply being generated without leaving the chat (a second Ctrl-C quits). The whole conversation is re-sent every turn, so /image and --image refuse an image once the conversation would hold more than the server's per-request image cap (16 by default); /clear starts over.
With no -n/--max-tokens, generation continues until EOS or the effective context limit. Pass -n N to set a cap.
mlxcel generate -m Qwen3.5-0.8B-4bit -p "Hello, world!" -n 100 # Read-only memory estimate before loading. mlxcel inspect -m Qwen3.5-0.8B-4bit --max-tokens 32768 # Emit the same estimate as machine-readable bytes for recipe builders or schedulers. mlxcel inspect --json -m Qwen3.5-0.8B-4bit --max-tokens 32768 | python3 -m json.tool # Abort before generation if the estimated model and KV cache do not fit. mlxcel generate -m Qwen3.5-0.8B-4bit -p "Hello" -n 32768 --estimate-memory
generate is the raw-completion client of the same batch-native engine (mlxcel_core::engine::DirectEngine): no server worker or HTTP request path in between, the rendered prompt goes straight through the engine's prefill and decode, so it decodes with the same sampler, finish step and lookahead pipeline as the server's single-sequence row (MLXCEL_FORCE_SYNC turns the pipeline off). --prompt-lookup runs drafter-free speculative decoding on that engine for any model. --draft-model takes an MTP drafter, a DFlash drafter on a target with an offline round loop, or an ordinary smaller checkpoint that shares the target's tokenizer, which drafts from its own KV caches on the same engine loop (--draft-kind full-model, auto-detected); mlxcel-server serves the same pairing.
mlxcel inspect --json prints a single JSON object with byte-exact weights_bytes, kv_bytes_total, activation_bytes, backend_inflight_bytes (ROCm builds only, else 0), headroom_bytes, budget_bytes, total_bytes, fits, input flags, and per-token FP16/INT8 KV rates when the model config exposes KV geometry. TurboQuant per-token sizing is reported as null until the estimator models those widths directly.
mlxcel-server and mlxcel serve start the same server. Their server flag surfaces are kept aligned; mlxcel serve additionally has the one-shot preflight flags --estimate-memory and --force.
mlxcel-server -m Qwen3.5-0.8B-4bit --port 8080 curl http://localhost:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"Qwen3.5-0.8B-4bit","messages":[{"role":"user","content":"Hello"}]}'
Common deployment controls work on both server entry points:
# API keys from a list or file; both sources are combined. mlxcel-server -m Qwen3.5-0.8B-4bit --api-key alice-key,bob-key mlxcel-server -m Qwen3.5-0.8B-4bit --api-key-file /etc/mlxcel/api-keys # Restrict browser CORS, or serve HTTPS. mlxcel-server -m Qwen3.5-0.8B-4bit \ --allowed-origins https://app.example.com,https://admin.example.com mlxcel-server -m Qwen3.5-0.8B-4bit \ --ssl-cert-file cert.pem --ssl-key-file key.pem # Serve below a path prefix or on a Unix domain socket. mlxcel-server -m Qwen3.5-0.8B-4bit --api-prefix /llama mlxcel-server -m Qwen3.5-0.8B-4bit --host /run/mlxcel.sock
Downloaded models normally live at ${MLXCEL_CACHE_DIR:-$HOME/.cache/mlxcel}/models/<owner>/<name>. On generate, run, inspect, download, list, and rm, use --models-dir to override that store. On the two server entry points, the store override is --model-store-root; --models-dir now selects multi-model router discovery and cannot be combined with -m.
# Router mode: discover direct checkpoint subdirectories under /srv/models, # use the global store as another source, and keep at most four models loaded. mlxcel-server --models-dir /srv/models \ --model-store-root /var/lib/mlxcel/models --models-max 4
Router management, live LoRA, native routes, runtime settings, context sizing, idle sleep, and compatibility migration details are in Server features.
--webui adds a browser interface to either server entry point: a model library with load, unload, download, and delete; local chat; runtime activity; and scoped settings. It is compiled into the binary and makes no external requests.
# From an interactive terminal: prints a one-time session key, then open # http://127.0.0.1:8080/webui/ and paste it. mlxcel-server --webui --models-dir ~/models/mlx # As a service, or anywhere without a terminal: use a key file. mkdir -p ~/.config/mlxcel (umask 077 && openssl rand -hex 32 > ~/.config/mlxcel/webui.key) mlxcel serve --webui --api-key-file ~/.config/mlxcel/webui.key --models-dir ~/models/mlx
The WebUI guide (한국어) covers keys, remote access over TLS or a reverse proxy, offline use, privacy, and troubleshooting.
The offline commands use the same loaders and batching paths as POST /v1/embeddings and POST /v1/rerank.
# Embed two texts and print vectors plus cosine similarity. mlxcel embed -m sentence-transformers/all-MiniLM-L6-v2 \ -p "The weather is lovely" -p "It is sunny" # Rank documents against one query. mlxcel rerank -m BAAI/bge-reranker-v2-m3 \ -q "what is panda?" -d "hi" -d "The giant panda is a bear species." # Serve an embedding model alone, or a reranker beside a chat model. mlxcel-server -m sentence-transformers/all-MiniLM-L6-v2 --port 8080 mlxcel-server -m Qwen3.5-0.8B-4bit \ --reranker-model mlx-community/Qwen3-Reranker-0.6B-4bit --port 8080
See Embeddings and reranking for multimodal inputs, supported families, pooling, request schemas, and side-model serving.
mlxcel tag runs a BIO / BIOES token classifier (BERT or XLM-RoBERTa ForTokenClassification) and prints the tagged spans with character offsets plus a redacted copy of the text.
mlxcel tag -m nativ-community/rampart-mlx-fp16 --keep CITY,STATE,ZIP_CODE \ "I'm Sarah Connor, 1984 Cyberdyne Ave, Los Angeles CA 90012, sarah@sky.net" # "redacted_text": "I'm <GIVEN_NAME> <SURNAME>, <BUILDING_NUMBER> <STREET_NAME>, Los Angeles CA 90012, <EMAIL>"
See Token classification and span extraction for the pipeline, the output schema, and mlxcel extract for span extractors that take labels per request.
An audio-capable chat checkpoint serves transcription, a Whisper checkpoint serves transcription and translation, and a Kokoro checkpoint serves text-to-speech.
mlxcel-server -m models/whisper-base --port 8080 curl http://localhost:8080/v1/audio/transcriptions \ -F file=@recording.wav -F model=whisper-base
A Nemotron VoiceChat checkpoint adds the full-duplex /v1/realtime WebSocket. See Audio API for accepted containers, model layout, request formats, streaming, and limits, and Nemotron VoiceChat for the realtime protocol and its example clients.
mlxcel generate-image runs a Diffusers-layout checkpoint for text-to-image generation and editing, and both server entry points serve the same models at POST /v1/images/generations and POST /v1/images/edits behind --image-model.
# Generate, or edit a reference image. mlxcel generate-image -m models/flux2-klein -p "a watercolor fox" -o fox.png mlxcel generate-image -m models/flux2-klein --task edit --image fox.png -p "make it night" # Image-only serving, with no chat model loaded. mlxcel-server --image-model models/flux2-klein --port 8080
See Image generation for the registered families, per-family step and guidance defaults, the request schema, and the image-only mode.
mlxcel list # name, size, and modified time mlxcel list --sort size # largest first mlxcel list --json # stable machine-readable array mlxcel list -q # repository ids only mlxcel list -v # include absolute paths mlxcel rm mlx-community/Qwen3.5-0.8B-4bit mlxcel rm mlx-community/Qwen3.5-0.8B-4bit --yes
mlxcel rm deletes only from the mlxcel-managed store. A checkpoint that exists solely in the read-only HuggingFace cache is reported but not removed.
# Object or document-layout detection. mlxcel detect -m models/rt-detr-v2 -i image.jpg --format json mlxcel detect -m models/PP-DocLayoutV3 -i page.png --format json # Inspect the hardware-specific kernel tuning matrix without profiling it. mlxcel tune --dry-run # Split the GLM-4.7-Flash MTP block out of the raw checkpoint into a 4-bit # drafter, then pair it with a glm4_moe_lite target. mlxcel split-mtp -m models/glm-4.7-flash-bf16 -o models/glm-4.7-flash-mtp-4bit --q-bits 4 mlxcel generate -m models/glm-4.7-flash-4bit --draft-model models/glm-4.7-flash-mtp-4bit -p "Hello"
mlxcel tune can profile supported kernel tactics into the local autotune cache; set MLXCEL_AUTOTUNE=cache to consume recorded winners. Use mlxcel --help or a subcommand's --help for the complete flag surface.
Apple Silicon prerequisites are Rust, Xcode Command Line Tools, CMake, and the Metal toolchain component:
xcodebuild -downloadComponent MetalToolchain
git clone https://github.com/lablup/mlxcel.git
cd mlxcel
cargo build --release --features metal,accelerateLinux/NVIDIA builds require the CUDA toolkit and MLX's CUDA system dependencies:
cargo build --release --features cuda
Linux/AMD builds are supported and need a ROCm installation with its HIP, rocBLAS, rocWMMA, hipBLASLt, and hipRTC CMake packages:
cargo build --release --features rocm
Native Windows builds with NVIDIA CUDA are experimental and single-node: run cargo build --release --features cuda from a Visual Studio developer prompt with CUDA_PATH pointing at a CUDA 13 toolkit that has cuDNN 9. See Windows with NVIDIA CUDA.
Native Windows builds with AMD ROCm are supported since 0.8 and single-node: run cargo build --release --features rocm from the x64 Visual Studio developer prompt with ROCM_PATH pointing at a ROCm 10.x SDK. See Windows with AMD ROCm for the SDK layout and the validated configuration.
A plain Linux build has no GPU feature and runs on the CPU, which is not a validated release target and is much slower. See Installation for the complete prerequisite, CUDA and ROCm architecture, runtime-header, and packaging matrix, and Linux with AMD ROCm for the ROCm build and validated configuration.
Measured on the commits released as v0.7.0 against same-host, same-day runs of mlx-lm 0.31.3 and mlx-vlm 0.6.17. Text is the 2026-09-06 sweeps at a fixed pp512/tg128 shape; VLM is the 2026-09-09 sweeps, where both the runtime and the reference were re-measured in one pass; qwen3-vl-reranker-2b is excluded from the VLM rows because a reranker driven through a generation harness is not measuring the VLM path. These are medians over a roster, not guarantees for an individual checkpoint.
| Workload | Host | Reference | Result |
|---|---|---|---|
| Text decode, 60 pairs | M5 Max 128 GB | mlx-lm |
99% median (quartiles 99 / 101) |
| Text decode, 110 pairs | M1 Ultra | mlx-lm |
100% median (quartiles 99 / 105) |
| Text prefill, 60 pairs | M5 Max 128 GB | mlx-lm median |
1.09x |
| Text prefill, 110 pairs | M1 Ultra | mlx-lm median |
0.94x |
| VLM decode, 47 pairs | M5 Max 128 GB | mlx-vlm |
105% median (quartiles 101 / 113, none below 90%) |
| VLM decode, 47 pairs | M1 Ultra | mlx-vlm |
108% median (quartiles 101 / 124, none below 90%) |
Two of these numbers replace larger ones and the reason is the measurement condition rather than a regression. The 2026-05-19 campaign on mlxcel 0.0.28 reported text prefill at 2.78x on M5 Max and 1.79x on M1 Ultra. Those ran on a short natural prompt whose token count differed between the two runtimes, and prefill_tok_s is prompt tokens divided by prefill time, so a difference in tokenization lands directly in the ratio. At a synthetic 512-token prompt that both runtimes receive identically, prefill is close to parity. The older figures should not be quoted for the current release.
Named checkpoints, M5 Max, mlxcel against the same-day reference. Text rows are mlx-lm 0.31.3, VLM rows are mlx-vlm 0.6.17:
| Checkpoint | Harness | mlxcel | Reference | Result |
|---|---|---|---|---|
qwen3-30b-a3b-4bit |
text | 166.55 | 143.81 | 116% |
qwen3-next-80b-a3b-instruct-4bit |
text | 121.49 | 106.04 | 115% |
qwen3-vl-30b-a3b-instruct-4bit |
text | 169.55 | 145.32 | 117% |
gemma-4-26b-a4b-it-qat-4bit |
VLM | 141.01 | 124.70 | 113% |
gemma-4-26b-a4b-it-qat-4bit |
text | 138.51 | 128.47 | 108% |
qwen3.5-35b-a3b-4bit |
text | 163.75 | 152.72 | 107% |
qwen3.6-35b-a3b-4bit |
text | 157.11 | 148.14 | 106% |
gemma-4-e2b-it-qat-4bit |
VLM | 172.66 | 165.16 | 105% |
gpt-oss-20b-mxfp4-q4 |
text | 168.92 | 164.72 | 103% |
gpt-oss-120b-4bit |
text | 113.56 | 110.82 | 102% |
qwen3.8-27b-4bit |
text | 33.61 | 33.75 | 100% |
meta-llama-3.1-8b-instruct-4bit |
text | 114.68 | 114.52 | 100% |
qwen2.5-7b-instruct-4bit |
text | 123.70 | 123.65 | 100% |
mixtral-8x7b-instruct-v0.1-4bit |
text | 65.82 | 65.85 | 100% |
The split in that table is architectural rather than arbitrary. Across the whole M5 Max text roster, mixture-of-experts checkpoints run at a median 106% of mlx-lm (19 pairs, quartiles 100 and 108) while dense ones sit at 100% (69 pairs, quartiles 99 and 100). Every checkpoint above 105% in the table is MoE, and every dense entry is at parity, including qwen3.8-27b-4bit, mixtral-8x7b and the dense Gemma 4 QAT variants. Gemma 4 QAT splits along the same line rather than along quantization: the 26B A4B mixture reads 108% on text and 113% on VLM, while the dense E2B, E4B and 31B QAT checkpoints read 100%, 100% and 99%. Read a dense model's row as "no reason to switch runtimes for speed" and an MoE model's as the case where mlxcel's expert dispatch is worth something.
The VLM figures also moved between mlxcel releases, and unevenly. Re-sweeping that table at one commit left the median row unchanged at 1.00x against its previous reading, with 9 of 71 rows moving more than 10%: mistral-small-4-119b-2603-4bit 5.30x, moondream2 4.19x, qwen3-vl-30b-a3b 2.61x, qwen3-omni-30b-a3b 2.57x, then five between 1.16x and 1.46x. The gains come from dtype fixes that reach specific paths, so the aggregate moved from 98% to 105% while most individual rows stayed where they were. Read the per-model tables rather than the median if a particular checkpoint matters.
GB10 has no same-host mlx-lm reference, so its record is version-over-version. The 0.8.1 sweep (text on 2026-11-03, VLM on 2026-11-04, pp512/tg128) paired 111 text checkpoints with the 0.4.0-rc.1 record at a 1.026 median decode ratio, with the 10th percentile at 0.928 and the 90th at 1.364. The 52 VLM pairs read 1.081 median and 1.012 at the 10th percentile, with no row below the previous record. Prefill is not comparable across those two sweeps because the harness moved to the 512-token synthetic prompt. The same pass fixed and re-measured the three checkpoints that had started failing: the BitNet pair now reads 845.05 and 843.60 tok/s prefill at 140.08 and 191.67 tok/s decode, and DiffusionGemma runs its model-owned diffusion loop at 797.61 prefill and 6.86 decode tok/s. It also added the first embedding and rerank ladder on this host (102 cells, no failures; 9 of 11 text embedders faster than their 2026-08-26 cells) and the image generation ladder (21 text-to-image checkpoints plus 3 edit-only; 4.61 s/image median for generation at 512x512 and 8 steps, 1.83 s fastest).
Per-host sweeps are more current than the combined report. M5 Max and M1 Ultra are both measured on the v0.7.0 commits (2026-09-06 through 2026-09-09) at the same MLX pin and the same pp512/tg128 shape, so cross-host ratios between those two are taken on one version. The per-host records label those runs 0.7.0-beta.1, which is the crate version string the tree carried until the release commit. GB10 is measured at 0.8.1 on the same pp512/tg128 shape, so a ratio between GB10 and either Apple Silicon host still mixes versions; within GB10, the decode deltas above are version-over-version. Coverage differs by host rather than version: M5 Max carries text, VLM, speculative decoding, batched serving and the embedding and rerank ladder, M1 Ultra carries text, VLM, speculative decoding and embeddings, and GB10 carries text, VLM, embeddings and rerank, and image generation and editing. Focused reports cover paged attention, MoE, KV compression, embeddings, and speculative decoding. Read Benchmark results, the benchmark report, and the methodology before comparing hosts or planning capacity.
MTP performance depends on the target/drafter pair, prompt, verify width, and hardware generation. With the v0.6.0 exactness policy, Gemma 4 12B plus its 4-bit assistant measured 93.2 tok/s against 43.8 tok/s classic decode on M5 Max (2.13x, greedy output byte-identical). The faster inexact kernel measured 120.4 tok/s but is not the default. How much that is worth depends heavily on the prompt: on the same host and release, the speculative_bench --sweep default 14-token prompt measures the same pairing at 70.9 tok/s against a 45.3 tok/s baseline (1.57x at K=4, 35.0% acceptance), against the 2.13x above on a code prompt. The runtime measures exactness and profitability instead of treating either speedup as universal; see Speculative-decoding acceptance, MTP policy API, and the benchmark records under docs/benchmark_results/.
Re-run the supplied harnesses on the target checkpoint, prompt shape, context length, and hardware before treating any published number as a deployment expectation.
Model support is architecture- and checkpoint-dependent. Run:
mlxcel arch mlxcel arch --json
for the architecture catalog compiled into the current binary. The --json form emits the stable recipes registry snapshot used by downstream site builds and automation, including standalone runtime families such as detector-only checkpoints. Supported models maintains the family table, quantization support, distributed coverage, checkpoint qualifications, and known caveats.
The pure-Python mlxcel client manages a local mlxcel serve process or connects to an existing server, discovers the served model id, and exposes the underlying OpenAI client for the full API surface.
import mlxcel with mlxcel.LLM("mlx-community/Qwen3-4B-4bit") as llm: print(llm.generate("def fib(n):", max_tokens=128)) for delta in llm.stream("Write a haiku about autumn"): print(delta, end="", flush=True)
Install with pip install ./python. See Python client for managed and connect modes, async use, streaming, structured output, and troubleshooting.
mlxcel-server --webui serves the bundled browser WebUI described in docs/webui. mlxcel-server can also be used directly through HTTP clients, and Backend.AI Go is available as a companion desktop UI for local chat, model management, and multi-model routing.
The documentation index lists design notes, benchmark records, and specialized implementation guides that do not need to live in this README.
Issues and pull requests are welcome. See CONTRIBUTING.md for the contributor workflow, local quality gates, and commit conventions. For larger changes, open an issue first so the scope and validation plan can be discussed.
For security vulnerabilities, see SECURITY.md; do not file them as public issues.
Apache License 2.0 unless otherwise noted. See LICENSE. Third-party attributions carried forward under Apache-2.0 Section 4(d) are listed in NOTICE.
expm1f routine. See NOTICE.rocm-support branch is the AMD GPU (ROCm/HIP) MLX backend vendored as the ROCm part of mlxcelverse, under the MIT License. See NOTICE.iree-compile lowers it, and the IREE runtime executes it under the opt-in xla-iree build. See NOTICE.lablup/mlxcel is a public repo on GitHub by lablup. High-performance LLM, VLM, embedding, reranking, and audio inference for Apple Silicon, NVIDIA CUDA and AMD ROCm systems.
It declares the Apache-2.0 license on GitHub.
Its maker has not tagged a system, so it is unknown here. Its README says what it needs.
Open "Take it" on its page: Take My Repo writes an install plan for your machine and hands it to your AI agent (Claude Code, Codex or Cursor), which asks you before each step. Take My Repo has not tested it on any machine.
Its last change on GitHub was on 2026-11-20. Take My Repo does not judge from that whether it is maintained.
Nobody has reviewed it on Take My Repo yet, which says nothing about its quality.
The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.

An open-source coding agent for your terminal, written in Rust. It works with any model, can split one task across several agents, and every line of it is yours to read.

Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.
Backend.AI is a streamlined, container-based computing cluster platform that hosts popular computing/ML frameworks and diverse programming languages, with pluggable heterogeneous accelerator support including CUDA GPU, ROCm GPU, Gaudi NPU, Google TPU, GraphCore IPU and other NPUs.

Backend.AI Web UI for web / desktop app (Windows/Linux/macOS). Backend.AI Web UI provides a convenient environment for users, while allowing various commands to be executed without CLI. It also provides some visual features that are not provided by the CLI, such as dashboards and statistics.
A high-performance parallel SSH command execution tool for cluster management
Repository of Backend.AI-enabled container recipes
This page is also available as structured data, with the same sources and dates.
/api/v1/resources/gh/lablup/mlxcelTake My Repo has not tested this on your machine. Requirements come from the project's own documentation.