Listening with LLM - moomou

This is the first part of many posts I am writing to consolidate learnings on how to finetune Large Language Models (LLMs) to process audio, with the eventual goal of being able to build and host a LLM able to describe human voices. I am motivated to gain hands-on experience tinkering LLMs so, as much as practical, I tried to recreate utilities and functions with pytorch from scratch rather than rely on 3rd party libraries. tl;dr I chronicle and share the steps I took to learn how to finetune a LLM model to describe a given audio file on Google’s MusicCaps dataset Recently, I came across two papers to give LLMs audio understanding capabilities. Broadly speaking, both papers explored leveraging an audio encoder to transform sound to embeddings that is then fed into LLMs along with text embeddings. In SALMONN’s case, they combined OpenAI’s Whisper and BEATS encoder, performed pretraining on the combined encoder, then leveraged LoRA for finetuning the LLM. Qwen-Audio bootstrapped its audi

Listening with LLM - moomou moomou (ﾉ≧∇≦)ﾉ ﾐ ┸┸ Listening with LLM Posted at - Dec 31, 2023 Overview # This is the first part of many posts I am writing to consolidate learnings on how to finetune Large Language Models (LLMs) to process audio, with the eventual goal of being able to build and host a LLM able to describe human voices. I am motivated to gain hands-on experience tinkering LLMs so, as much as practical, I tried to recreate utilities and functions with pytorch from scratch rather than rely on 3rd party libraries. tl;dr I chronicle and share the steps I took to learn how to finetune

related reading

• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• Patterns for Building LLM-based Systems & Productseugeneyan.com
• Deep Dive into LLMs like ChatGPTyoutube.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
• Productizing Large Language Modelsblog.replit.com
• GitHub - rasbt/LLMs-from-scratch: Implement a ChatGPT-like LLM in PyTorch from scratch, step by stepgithub.com
• Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMsmagazine.sebastianraschka.com
• A History of Large Language Modelsgregorygundersen.com
• Writing for LLMs So They Listen · Gwern.netgwern.net
• Taking LLMs Seriously (As Language Models) — LessWronglesswrong.com