67 open source projects by lucidrains

A simple but complete full-attention transformer with a set of promising experimental features from various papers

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

Transformer based on a variant of attention that is linear complexity in respect to sequence length

Implementation of the Point Transformer layer, in Pytorch

Implementation of Transformer in Transformer, pixel level attention paired with patch level attention for image classification, in Pytorch

Pytorch implementation of SIREN - Implicit Neural Representations with Periodic Activation Function

Implementation of Slot Attention from GoogleAI

Simplest working implementation of Stylegan2, state of the art generative adversarial network, in Pytorch. Enabling everyone to experience disentanglement

Sinkhorn Transformer - Practical implementation of Sparse Sinkhorn Attention

Fully featured implementation of Routing Transformer

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

Implementation of Pixel-level Contrastive Learning, proposed in the paper "Propagate Yourself", in Pytorch

Implementation of LambdaNetworks, a new approach to image recognition that reaches SOTA with less compute

A Pytorch implementation of Stylegan2 with UNet Discriminator

Reformer, the efficient Transformer, in Pytorch

Implementation of the convolutional module from the Conformer paper, for use in Transformers

Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch. This specific repository is geared towards integration with eventual Alphafold2 replication.

A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models

A Pytorch implementation of Global Self-Attention Network, a fully-attention backbone for vision tasks

A simple to use pytorch wrapper for contrastive self-supervised learning on any neural network

An implementation of (Induced) Set Attention Block, from the Set Transformers paper

Usable Implementation of "Bootstrap Your Own Latent" self-supervised learning, from Deepmind, in Pytorch

A simple command line tool for text to image generation, using OpenAI's CLIP and a BigGAN

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

An implementation of Performer, a linear attention-based transformer, in Pytorch

Implementation of 'lightweight' GAN, proposed in ICLR 2021, in Pytorch. High resolution image generations that can be trained within a day or two

Implementation of Bottleneck Transformer in Pytorch

To eventually become an unofficial Pytorch implementation / replication of Alphafold2, as details of the architecture get released

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

Implementation of TimeSformer from Facebook AI, a pure attention-based solution for video classification

Implementation of Uformer, Attention-based Unet, in Pytorch

A denoising diffusion probabilistic model (DDPM) tailored for conditional generation of protein distograms

Implementation of the GBST block from the Charformer paper, in Pytorch

Implementation of Linformer for Pytorch

Implementation of TransGanFormer, an all-attention GAN that combines the finding from the recent GanFormer and TransGan paper

Implementation of N-Grammer, augmenting Transformers with latent n-grams, in Pytorch

Implementation of a Transformer using ReLA (Rectified Linear Attention) from https://arxiv.org/abs/2104.07012

Implementation of Geometric Vector Perceptron, a simple circuit for 3d rotation equivariance for learning over large biomolecules, in Pytorch. Idea proposed and accepted at ICLR 2021

Implementation of NÜWA, state of the art attention network for text to video synthesis, in Pytorch

Implementation of Enformer, Deepmind's attention network for predicting gene expression, in Pytorch

Implementation of Nyström Self-attention, from the paper Nyströmformer

A Pytorch implementation of Attention on Attention module (both self and guided variants), for Visual Question Answering

Implementation of the Remixer Block from the Remixer paper, in Pytorch

Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

Implementation of trRosetta and trDesign for Pytorch, made into a convenient package, for protein structure prediction and design

Pytorch reimplementation of Molecule Attention Transformer, which uses a transformer to tackle the graph-like structure of molecules

Implementation of Token Shift GPT - An autoregressive model that solely relies on shifting the sequence space for mixing

Implementation of Graph Transformer in Pytorch, for potential use in replicating Alphafold2

Implementation of Long-Short Transformer, combining local and global inductive biases for attention over long sequences, in Pytorch

Implementation of Denoising Diffusion Probabilistic Model in Pytorch