flâneur

What's New in Inference Engineering — Philip Kiely, Baseten - YouTube

youtube.com · 3,833 words · saved by 1 readers

Follow along using the transcript.

TurboQuant reached twenty million people in March, and the memory stock index dipped because everyone assumed the KV cache had just halved. Philip Kiely had published Inference Engineering weeks earlier and watched a technique he had not covered go viral. So his team did the math. Four bit cache instead of eight does double effective bandwidth, but the extra decode computation cuts tokens per second by more than half: unacceptable in a data center, close to ideal on a memory starved machine in your basement. That split runs through the talk. Local inference is get it working, then make it…

saved by

related reading