TraceStorm
TraceStorm is a tool for generating and replaying traces of requests to OpenAI API endpoints. It allows users to simulate load testing by generating requests based on specified patterns and configurations.
Features
- Generate synthetic traces using various patterns (e.g., uniform, poisson, etc.)
- Load traces from public datasets (e.g., Azure LLM Inference dataset)
- Replay requests to any OpenAI-compatible LLM service
- Analyze results and visualize performance metrics
Installation
pip install tracestorm
Usage
Start an OpenAI-Compatible Server
Before running the load test, ensure you have an OpenAI-compatible server running. If you haven't already installed vllm, you can do so with the following command:
# Install vllm if you haven't already
pip install vllm
# Start the server with the desired model
vllm serve Qwen/Qwen2.5-1.5B-Instruct
Run the Load Test
Once the server is running, you can execute the load test using the tracestorm command. Here's how to run it with different options:
Example Command for Synthetic Trace
tracestorm --model "Qwen/Qwen2.5-1.5B-Instruct" --rps 5 --pattern uniform --duration 10
Example Command for Azure Trace
To load a trace from the Azure dataset, use:
tracestorm --model "Qwen/Qwen2.5-1.5B-Instruct" --pattern azure_code
Example Command for Loading Prompts from Datasets
tracestorm --model "Qwen/Qwen2.5-1.5B-Instruct" --duration 30 --datasets-config ./examples/datasets_config_hf.json
Supported Dataset Sources:
- Locally stored, pre-processed datasets
- Refer to
./examples/datasets_config_local.json for an example configuration.
- If you want to test loading from local files, please run
./examples/test_data_loader.py first to download and save two datasets.
- Remote datasets from Hugging Face
- Refer to
./examples/datasets_config_hf.json for an example configuration.
Sorting Strategy: Defines how prompts from multiple datasets are ordered
- random (default): Shuffles prompts randomly.
- original: Maintains the original order of prompts.
Please check ./examples/datasets_config_default.json for required fields in datasets-config-file. This file contains placeholders for necessary configurations.
Command Options
--model: Required. The name of the model to use.
--rps: Optional. Requests per second (default is 1, only used for synthetic patterns).
--pattern: Optional. Pattern for generating trace. Valid patterns include:
uniform: Distributes requests evenly across the duration.
poisson: Generates request timings based on a Poisson process.
random: Generates requests at random intervals within the duration.
azure_code: Loads the Azure inference dataset for code.
azure_conv: Loads the Azure inference dataset for conversation.
--duration: Optional. Duration in seconds (default is 10, only used for synthetic patterns).
--subprocesses: Optional. Number of subprocesses to use (default is 1).
--base-url: Optional. OpenAI Base URL (default is http://localhost:8000/v1).
--api-key: Optional. OpenAI API Key (default is none).
--seed: Optional. Random seed for trace pattern reproducibility (default is none).
--datasets-config: Optional. Configuration file for loading prompt messages from provided datasets. Uses DEFAULT_MESSAGES is not specified.
Make sure to adjust the parameters according to your testing needs!