Mixture of Distributions Matters

Created on January 14, 2026

2026   ·   diffusion-transformers   video-generation   efficient-training   ·   research

Long-sequence video generation is constrained by the quadratic cost of self-attention. Mixture of Distributions Matters studies dynamic sparse attention for efficient video Diffusion Transformers and adapts attention computation to the heterogeneous distributions encountered during generation.

The work was accepted by ICML 2026.



Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • SparkDiffusion: Mitigating the High-Sparsity Trap
  • CrossDistill: Balancing Quality and Diversity
  • RoLA: Rotary-Positioned Low-Rank Linear Attention
  • RoPeSLR: Sparse-Low-Rank Attention for Diffusion Transformers
  • CR-Net: Cross-Layer Low-Rank Structure