67 open source projects by lucidrains

Implementation of H-Transformer-1D, Hierarchical Attention for Sequence Learning

Implementation of E(n)-Equivariant Graph Neural Networks, in Pytorch

Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch

An attempt at the implementation of Glom, Geoffrey Hinton's new idea that integrates concepts from neural fields, top-down-bottom-up processing, and attention (consensus between columns), for emergent part-whole heirarchies from data

Implementation of Axial attention - attending to multi-dimensional data efficiently

An All-MLP solution for Vision, from Google AI

Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch

A GPT, made only of MLPs, in Jax

Implementation of E(n)-Transformer, which extends the ideas of Welling's E(n)-Equivariant Graph Neural Network to attention

Implementation of Uniformer, a simple attention and 3d convolutional net that achieved SOTA in a number of video classification tasks, debuted in ICLR 2022

Implementation of ResMLP, an all MLP solution to image classification, in Pytorch

Implementation of gMLP, an all-MLP replacement for Transformers, in Pytorch

Implementation and replication of ProGen, Language Modeling for Protein Generation, in Jax

Implementation of Memory-Compressed Attention, from the paper "Generating Wikipedia By Summarizing Long Sequences"

Implementation of Invariant Point Attention, used for coordinate refinement in the structure module of Alphafold2, as a standalone Pytorch module

Vector Quantization, in Pytorch

Implementation of STAM (Space Time Attention Model), a pure and simple attention model that reaches SOTA for video classification