Listening with LLM - moomou

This is the first part of many posts I am writing to consolidate learnings on how to finetune Large Language Models (LLMs) to process audio, with the eventual goal of being able to build and host a LLM able to describe human voices. I am motivated to gain hands-on experience tinkering LLMs so, as much as practical, I tried to recreate utilities and functions with pytorch from scratch rather than rely on 3rd party libraries. tl;dr I chronicle and share the steps I took to learn how to finetune a LLM model to describe a given audio file on Google’s MusicCaps dataset Recently, I came across two papers to give LLMs audio understanding capabilities. Broadly speaking, both papers explored leveraging an audio encoder to transform sound to embeddings that is then fed into LLMs along with text embeddings. In SALMONN’s case, they combined OpenAI’s Whisper and BEATS encoder, performed pretraining on the combined encoder, then leveraged LoRA for finetuning the LLM. Qwen-Audio bootstrapped its audi

Listening with LLM - moomou moomou (ﾉ≧∇≦)ﾉ ﾐ ┸┸ Listening with LLM Posted at - Dec 31, 2023 Overview # This is the first part of many posts I am writing to consolidate learnings on how to finetune Large Language Models (LLMs) to process audio, with the eventual goal of being able to build and host a LLM able to describe human voices. I am motivated to gain hands-on experience tinkering LLMs so, as much as practical, I tried to recreate utilities and functions with pytorch from scratch rather than rely on 3rd party libraries. tl;dr I chronicle and share the steps I took to learn how to finetune

related reading

• Crossing the uncanny valley of conversational voice | Sesamesesame.com
• Patterns for Building LLM-based Systems & Productseugeneyan.com
• GenAI Handbookgenai-handbook.github.io
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities (Version 1.0)arxiv.org
• The Llama Hitchiking Guide to Local LLMs – hackerllamaosanseviero.github.io
• Fine-tuning LLMs Guide | Unsloth Documentationdocs.unsloth.ai
• Efficient LLM Finetuning with Unsloth | Modal Docsmodal.com
• Llama 2 · Hugging Facehuggingface.co
• Tutorial: How to Finetune Llama-3 and Use In Ollama | Unsloth Documentationdocs.unsloth.ai
• Voice AI & Voice Agents | An Illustrated Primervoiceaiandvoiceagents.com
• API Reference — TensorRT LLMnvidia.github.io