
The maintainer has not written a description at the source.
Repo modelscope/DiffSynth-Engine, read from GitHub 22 min ago. modelscope has not joined Take My Repo.
New Contributors
Wan2.2-S2V is based on Wan 2.1, with several additional modules to inject audio, reference image and pose video conditions. Check the usage example here.
Full Changelog: v0.5.0...v0.6.0
Qwen-Image-Edit is the image editing version of Qwen-Image, enabling semantic/appearance visual editing, and precise text editing. Check the usage example here.
New Contributors
Full Changelog: v0.4.1.post1...v0.5.0
Qwen-Image is an image generation model excels at complex text rendering and creating images in a wide range of artistic styles. Check the usage example here.
Resource utilization for generating a 1024x1024 using Qwen-Image model with H20 GPU under different offload_mode:
| Offload Mode | Peak VRAM Usage (GB) | Peak Memory Usage (GB) | Inference Time (s) |
|---|---|---|---|
| None | 62 | 64 | 57 |
| "cpu_offload" | 39 | 64 | 86 |
| "sequential_cpu_offload" | 8 | 64 | 134 |
Full Changelog: v0.4.0...v0.4.1.post1
The WanVideoPipeline also supports Wan2.2 series model now. Taking the Wan2.2-TI2V-5B model as an example:
from diffsynth_engine import fetch_model, WanVideoPipeline, WanPipelineConfig from diffsynth_engine.utils.video import save_video config = WanPipelineConfig.basic_config( model_path=fetch_model( "Wan-AI/Wan2.2-TI2V-5B", revision="bf16", path=[ "diffusion_pytorch_model-00001-of-00003-bf16.safetensors", "diffusion_pytorch_model-00002-of-00003-bf16.safetensors", "diffusion_pytorch_model-00003-of-00003-bf16.safetensors", ], ), parallelism=1, offload_mode=None, ) pipe = WanVideoPipeline.from_pretrained(config) image = Image.open("input/wan_i2v_input.jpg").convert("RGB") video = pipe( prompt="Summer beach vacation style, a white cat wearing sunglasses sits on a surfboard. The fluffy-furred feline gazes directly at the camera with a relaxed expression. Blurred beach scenery forms the background featuring crystal-clear waters, distant green hills, and a blue sky dotted with white clouds. The cat assumes a naturally relaxed posture, as if savoring the sea breeze and warm sunlight. A close-up shot highlights the feline's intricate details and the refreshing atmosphere of the seaside.", negative_prompt="", input_image=image, num_frames=121, width=704, height=1280, seed=42, ) save_video(video, "wan_ti2v.mp4", fps=pipe.config.fps)
parallelism to 2, 4 or 8 to speed up video generation with multiple GPUs.offload_mode to "cpu_offload" (model-level offload) or "sequential_cpu_offload" (parameter-level offload, with lowest VRAM usage and maximum generation time).num_frames to 24X+1.Find more examples here for Wan2.2-T2V-A14B, Wan2.2-I2V-A14B.
from_pretrained method pipeline initializationIn previous versions, we have from_pretrained method to initialize pipeline with a ModelConfig and other arguments. Such as,
from diffsynth_engine import fetch_model, FluxImagePipeline, FluxModelConfig model_path = fetch_model("muse/flux-with-vae", path="flux1-dev-with-vae.safetensors") config = FluxModelConfig(dit_path=model_path, use_fp8_linear=True, use_fsdp=True) pipe = FluxImagePipeline.from_pretrained(config, parallelism=8, use_cfg_parallel=True)
In the code example above, the division between ModelConfig and other arguments in from_pretrained method is not clear, which makes it quite confusing.
Since v0.4.0, we introduce a new PipelineConfig to contain all pipeline initialization arguments. With it, the above code can rewritten as:
from diffsynth_engine import fetch_model, FluxImagePipeline, FluxPipelineConfig model_path = fetch_model("muse/flux-with-vae", path="flux1-dev-with-vae.safetensors") config = FluxPipelineConfig( model_path=model_path, use_fp8_linear=True, parallelism=8, use_cfg_parallel=True, use_fsdp=True, ) pipe = FluxImagePipeline.from_pretrained(config)
For beginners, we also provide a basic_config method with fewer arguments to make pipeline initialization easier:
from diffsynth_engine import fetch_model, FluxImagePipeline, FluxPipelineConfig model_path = fetch_model("muse/flux-with-vae", path="flux1-dev-with-vae.safetensors") config = FluxPipelineConfig.basic_config(model_path=model_path, parallelism=8) pipe = FluxImagePipeline.from_pretrained(config)
Check here for more available configs.
Full Changelog: v0.3.5...v0.4.0
fix sdxl lora load