vLLM Spark2_5 Plugin
This document targets Linux systems and Bash-compatible shells. The commands
use POSIX paths and assume a Linux vLLM environment at .venv/.
This package restores Spark2_5 support to vLLM as an out-of-tree general plugin.
It enables loading and serving Spark2_5 models without modifying the vLLM source
tree. The plugin registers:
Spark2_5ForCausalLM, the Spark2_5 model implementation;
Spark2_5DSparkForCausalLM, the Spark DSpark speculative draft model;
- the
spark2_5 Transformers configuration (model_type: "spark2_5"); and
- the Spark2_5 XML/KV tool-call parser, registered under the vLLM parser name
spark25.
The implementation targets vLLM tag
v0.32.0 and uses vLLM's
existing runtime layers. DSpark sliding draft layers use the plugin's Triton
JIT kernel; no separate extension build is required.
Spark2_5 model capabilities
Spark2_5 is a causal language model that supports hybrid sliding-window/full
attention, head-wise attention output gating, tensor parallelism, pipeline
parallelism, configurable RoPE parameters, and checkpoint weight loading.
Hybrid attention
Each layer can use either sliding-window attention or full attention. The mode
is selected from layer_types, with independent sliding-window and RoPE
configuration for the supported attention types.
Head-wise attention output gating
Spark2_5 can apply a learned gate to each attention head before the heads are
merged and passed to the output projection:
sigmoid(g_proj(hidden_states))
Grouped-query attention and tensor parallelism
Query and KV heads are partitioned or replicated according to the tensor
parallel world size. The implementation uses vLLM's
QKVParallelLinear, ColumnParallelLinear, and RowParallelLinear layers.
Pipeline parallelism
Spark2_5ForCausalLM implements SupportsPP and uses vLLM's pipeline-parallel
layer construction and IntermediateTensors to transfer hidden states and
residuals between pipeline stages.
Optimized MLP and checkpoint loading
The MLP combines the gate and up projections with
MergedColumnParallelLinear, applies GeluAndMul, and uses a row-parallel
down projection. AutoWeightsLoader and WeightsMapper map Hugging Face
checkpoint weights into this packed projection layout.
Spark DSpark integration
The DSpark implementation lives in
spark2_5_dspark.py. It projects
selected target hidden states into a non-causal draft block, shares the target
embedding and LM head, and loads draft_model.*, mtp.*, or model.mtp.*
checkpoint weights. It supports vanilla and rnn Markov heads and an optional
confidence head.
The checkpoint must supply a positive dspark_markov_rank and nonempty,
unique, ascending, zero-based dspark_target_layer_ids. The registration hook
in __init__.py restores Spark after vLLM's DeepSeek fallback and normalizes
markov_rank and target_layer_ids to these DSpark field names. Draft and
target hidden sizes and full vocabularies must match.
spark_triton_attention.py
implements fixed-history sliding draft attention. For target context length
L and draft block length B, every query attends to keys from
max(0, L - sliding_window) through L + B - 1, including the full non-causal
block. It uses vLLM's bundled Triton and requires dense FP16/BF16 KV and
power-of-two head dimensions. Target attention keeps vLLM's selected backend.
Draft layers default to sliding attention; dspark_layer_types can choose
sliding or full attention per layer. DSpark requires pipeline-parallel and
decode-context-parallel sizes of 1; the target-only pipeline-parallel support
above does not extend to DSpark.
Linux prerequisites
Use a supported Linux Python 3.11 or newer environment with vLLM 0.32.0 installed. GPU
deployments must have a compatible NVIDIA driver and the CUDA/PyTorch runtime
required by that vLLM installation. The plugin itself adds no system packages
or compiled extensions.
From the vLLM repository root, activate the environment before running the
commands below:
source .venv/bin/activate
Install
Install vLLM first, then install this package into the same Python
environment. The editable install is important because vLLM discovers the
plugin through the vllm.general_plugins entry-point metadata.
# From the vLLM checkout containing this directory.
uv pip install "vllm==0.32.0"
uv pip install -e ./Spark-plugin --no-deps
If the directory has been renamed to vllm-spark2_5-plugin, use that directory
name in the command instead. --no-deps prevents the plugin from replacing
the vLLM installation already selected for the environment. The vLLM parser
stack requires OpenAI Python SDK 2.25.0 or newer:
uv pip install -U "openai>=2.25.0"
Verify that the entry point is visible to the same interpreter used to launch
vLLM:
.venv/bin/python -c "from importlib.metadata import entry_points; \
print([(e.name, e.value) for e in entry_points(group='vllm.general_plugins')])"
The output should include an entry similar to:
('spark2_5', 'vllm_spark2_5_plugin:register')
Adding src/ to PYTHONPATH alone is not sufficient for normal plugin
discovery because entry points come from installed package metadata.
First successful use
Start the OpenAI-compatible server with the parser name spark25:
vllm serve /path/to/spark2_5-model \
--trust-remote-code \
--enable-auto-tool-choice \
--tool-call-parser spark25 \
--chat-template /path/to/chat_template.jinja \
--served-model-name Spark2_5
The parser name is spark25, not spark2_5: spark2_5 identifies the model
configuration, while spark25 is the name registered in ToolParserManager.
To enable Spark DSpark speculative decoding, use a Spark2_5 checkpoint that
includes DSpark draft weights (draft_model.* or mtp.*) and the corresponding
DSpark configuration:
vllm serve /path/to/spark2_5-dspark-model \
--trust-remote-code \
--enable-auto-tool-choice \
--tool-call-parser spark25 \
--chat-template /path/to/chat_template.jinja \
--served-model-name Spark2_5 \
--speculative-config '{"method":"dspark","num_speculative_tokens":7}'
When DSpark is enabled and the request's temperature is not 0.0, we
recommend setting draft_sample_method to probabilistic so that the draft
sampler retains a probabilistic proposal distribution for rejection sampling.
The draft applies temperature; target top-k/top-p and penalties can still make
the two distributions differ. Set it in the server's
--speculative-config:
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
The draft weights are loaded from the same checkpoint as the target model, so
no separate draft model path is needed. The plugin registers
Spark2_5DSparkForCausalLM and restores the Spark draft architecture if vLLM's
DSpark configuration selects its DeepSeek fallback. The override ensures that
the plugin's target model and tool parser are used even when vLLM already
provides Spark2_5 support.
The value 7 is an example; match num_speculative_tokens to the checkpoint's
dspark_block_size (or block_size) when configured. Spark DSpark currently
requires pipeline-parallel and decode-context-parallel sizes of 1.
For a small local test deployment, consider --enforce-eager and lower values
for --max-model-len, --max-num-batched-tokens, and
--gpu-memory-utilization as appropriate for the available device.
When the server starts, look for the plugin's registration log:
vllm-spark2_5-plugin: registered Spark2_5ForCausalLM -> ...
If vLLM already provides Spark2_5ForCausalLM, the plugin logs a stand-down
message and leaves the in-tree implementation active. To explicitly use this
vendored implementation, set:
SPARK2_5_PLUGIN_OVERRIDE=1 vllm serve /path/to/spark2_5-model \
--trust-remote-code \
--enable-auto-tool-choice \
--tool-call-parser spark25 \
--chat-template /path/to/chat_template.jinja
Tool-call format
The parser recognizes Spark2_5 XML/KV tool calls of this form:
<tool_call>get_weather
<arg_key>city</arg_key><arg_value>Paris</arg_value>
<arg_key>days</arg_key><arg_value>3</arg_value>
</tool_call>
It converts declared integer, number, boolean, and JSON values to their
corresponding JSON types and returns standard vLLM ToolCall objects. Unknown
tool names are not emitted as tool calls.
Performance
The following serving benchmark used 500 concurrent requests. All requests
completed successfully; hardware, model revision, and generation settings
should be recorded separately when reproducing these numbers.
| Metric |
Result |
| Successful requests |
500 |
| Failed requests |
0 |
| Benchmark duration |
4.70 s |
| Total input tokens |
762,489 |
| Total generated tokens |
63,478 |
| Request throughput |
106.42 req/s |
| Output token throughput |
13,510.08 tok/s |
| Peak output token throughput |
16,452.00 tok/s |
| Total token throughput |
175,791.36 tok/s |
| Peak concurrent requests |
500 |
Latency
| Metric |
Mean |
Median |
P99 |
| TTFT |
2122.04 ms |
2111.73 ms |
4169.35 ms |
| TPOT |
11.58 ms |
13.39 ms |
14.13 ms |
| ITL |
12.34 ms |
11.13 ms |
32.92 ms |
These results represent a 100% request success rate, 106.42 requests/s,
approximately 13.5K output tokens/s, and 175.8K total tokens/s at peak
concurrency of 500. Decode latency had a mean TPOT of 11.58 ms and a P99 of
14.13 ms; TTFT was 2.12 s on average and 4.17 s at P99.
Compatibility and maintenance
The model and DSpark implementations depend on vLLM's internal runtime APIs.
The current dependency baseline is vLLM tag v0.32.0. Install that version in
the serving environment before installing the plugin with --no-deps:
uv pip install "vllm==0.32.0"
The package metadata pins vllm==0.32.0 and requires Python 3.11 or newer,
matching that runtime's minimum Python version. CI tests Python 3.11 and 3.12
against this pinned dependency. When upgrading vLLM, update the dependency pin
and retest model registration, tool parsing, and DSpark decoding against the
new runtime APIs.
The TESTED_VLLM constant in src/vllm_spark2_5_plugin/__init__.py records
v0.32.0, the current runtime baseline, and appears in registration logs
alongside the running vLLM version. Historical extraction provenance is retained
in NOTICE and source headers.
Tests
The plugin smoke tests do not require a GPU or model weights. After installing
the plugin in the test environment, run:
.venv/bin/python -m pytest Spark-plugin/tests/test_smoke.py -q
The smoke tests verify both the public registration entry point and a complete
Spark2_5 XML tool-call parse.
test_spark_triton_attention.py checks
the fixed-history draft kernel against a PyTorch reference for requests with
different context lengths. This test requires CUDA and is skipped when CUDA
is unavailable; it is not an end-to-end DSpark decoding test.
# From the plugin repository root, in the vLLM 0.32.0 environment.
.venv/bin/python -m pytest tests/test_spark_triton_attention.py -q
Plugin loading and allowlists
General plugins are loaded automatically by vLLM unless VLLM_PLUGINS is set
as an allowlist. If an allowlist is used, the entry-point name is spark2_5:
VLLM_PLUGINS=spark2_5 vllm serve /path/to/spark2_5-model \
--tool-call-parser spark25 \
--chat-template /path/to/chat_template.jinja
VLLM_PLUGINS=spark25 is not equivalent: spark25 is the tool-parser name, not
the general-plugin entry-point name.
Troubleshooting
invalid tool call parser: spark2_5
Use --tool-call-parser spark25. The plugin registers the parser under spark25.
NamespaceTool cannot be imported from openai.types.responses
Upgrade the OpenAI Python SDK in the same environment that launches vLLM:
uv pip install -U "openai>=2.25.0"
The model architecture is unsupported
The plugin was not discovered by the running interpreter. Reinstall it into
that environment and inspect the entry points:
uv pip install -e ./Spark-plugin --no-deps
.venv/bin/python -c "from importlib.metadata import entry_points; \
print([(e.name, e.value) for e in entry_points(group='vllm.general_plugins')])"
If VLLM_PLUGINS is set, include spark2_5 in the allowlist.
The plugin reports that Spark2_5 is already supported
This is expected when the installed vLLM contains the model. Set
SPARK2_5_PLUGIN_OVERRIDE=1 only when the vendored implementation must be used.
The plugin logs an import traceback
The entry-point loader catches plugin import failures and continues starting
vLLM. Check the complete server log for the original traceback, then compare
the running vLLM version with the current dependency baseline, 0.32.0.
Project layout
Spark-plugin/
+-- src/vllm_spark2_5_plugin/
| +-- __init__.py # registration and DSpark config integration
| +-- spark2_5.py # Spark2_5 target model
| +-- spark2_5_dspark.py # DSpark draft, heads, and weight loading
| +-- spark_triton_attention.py # fixed-history draft attention kernel
| +-- spark2_5_config.py # Transformers configuration
| +-- spark2_5_tool_parser.py # Spark2_5 XML/KV parser
+-- tests/
| +-- test_smoke.py # registration and parser smoke tests
| +-- test_spark_triton_attention.py # CUDA kernel numerical test
+-- pyproject.toml
License
Apache-2.0. The vendored vLLM implementation retains its upstream copyright
headers.