[2207.02505] Pure Transformers are Powerful Graph Learners

We show that standard Transformers without graph-specific modifications can lead to promising results in graph learning both in theory and practice. Given a graph, we simply treat all nodes and edges as independent tokens, augment them with token embeddings, and feed them to a Transformer. With an appropriate choice of token embeddings, we prove that this approach is theoretically at least as expressive as an invariant graph network (2-IGN) composed of equivariant linear layers, which is already more expressive than all message-passing Graph Neural Networks (GNN). When trained on a large-scale graph dataset (PCQM4Mv2), our method coined Tokenized Graph Transformer (TokenGT) achieves significantly better results compared to GNN baselines and competitive results compared to Transformer variants with sophisticated graph-specific inductive bias. Our implementation is available at this https URL.

Pure Transformers are Powerful Graph Learners Jinwoo Kim1∗ Tien Dat Nguyen1 Seonwoo Min2 Sungjun Cho2 Moontae Lee2,3 Honglak Lee2† Seunghoon Hong1,2† 1 KAIST 2 LG AI Research 3 University of Illinois Chicago Abstract arXiv:2207.02505v2 [cs.LG] 22 Oct…

saved by

• Nikhil Shenoy

related reading

• A Gentle Introduction to Graph Neural Networksdistill.pub
• [2205.12454] Recipe for a General, Powerful, Scalable Graph Transformerarxiv.org
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• GraphGPS: Navigating Graph Transformers | Towards Data Sciencetowardsdatascience.com
• The Annotated Transformernlp.seas.harvard.edu
• Understanding Convolutions on Graphsdistill.pub
• Transformers from Scratche2eml.school
• Beyond Message Passing: a Physics-Inspired Paradigm for Graph Neural Networksthegradient.pub
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• geometricdeeplearning.com/book/graphs.htmlgeometricdeeplearning.com
• The Annotated Transformernlp.seas.harvard.edu
• Transformer (deep learning) - Wikipediaen.wikipedia.org