Encoder Decoder Models

The EncoderDecoderModel can be used to initialize a sequence-to-sequence model with any pretrained autoencoding model as the encoder and any pretrained autoregressive model as the decoder. The effectiveness of initializing sequence-to-sequence models with pretrained checkpoints for sequence generation tasks was shown in Leveraging Pre-trained Checkpoints for Sequence Generation Tasks by Sascha Rothe, Shashi Narayan, Aliaksei Severyn. After such an EncoderDecoderModel has been trained/fine-tuned, it can be saved/loaded just like any other models (see the examples for more information). An application of this architecture could be to leverage two pretrained BertModel as the encoder and decoder for a summarization model as was shown in: Text Summarization with Pretrained Encoders by Yang Liu and Mirella Lapata. EncoderDecoderModel can be randomly initialized from an encoder and a decoder config. In the following example, we show how to do this using the default BertModel configuration for

Encoder Decoder Models · Hugging Face Transformers documentation Encoder Decoder Models Transformers 🏡 View all docs AWS Trainium & Inferentia Accelerate Argilla AutoTrain Bitsandbytes CLI Chat UI Dataset viewer Datasets Deploying on AWS Diffusers Distilabel Evaluate Google Cloud Google TPUs Gradio Hub Hub Python Library Huggingface.js Inference Endpoints (dedicated) Inference Providers Kernels LeRobot Leaderboards Lighteval Microsoft Azure OpenEnv Optimum PEFT Reachy Mini Safetensors Sentence Transformers TRL Tasks Text Embeddings Inference Text Generation Inference Tokenizers Trackio Transf

related reading

• Transformer Architectures · Hugging Facehuggingface.co
• Encoder-Decoder Seq2Seq Models, Clearly Explained!! | by Kriz Moses | Analytics Vidhya | Mediummedium.com
• Llama 2 · Hugging Facehuggingface.co
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Models · Hugging Facehuggingface.co
• The Annotated Transformernlp.seas.harvard.edu
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Seq2seq and Attentionlena-voita.github.io
• The Annotated Transformernlp.seas.harvard.edu
• Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io