rotary-embedding-torch
Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch
- 828
- Python
- MIT
- 2 wk ago
Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch
Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.
[ECCV 2026] Official implementation of "HRDiT: Training-Free High-Resolution Image Generation with Off-the-Shelf Diffusion Transformer Models"
"Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding" Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, Zhangyang Wang.
Multiresolution Graph Transformers and Wavelet Positional Encoding for Learning Long-Range and Hierarchical Structures
[ICML'25] "Rethinking Addressing in Language Models via Contextualized Equivariant Positional Encoding" by Jiajun Zhu, Peihao Wang, Ruisi Cai, Jason D. Lee, Pan Li, Zhangyang Wang
Transformer architecture built from scratch — covering RNN→LSTM→Transformer evolution, self-attention, positional encoding, encoder/decoder stacks, and GPT-style generation. Pure Python + PyTorch implementations with maths breakdowns.
Code for "The role of positional encodings in the ARC benchmark" (arXiv:2502.00174)