flâneur

Clouded Judgement 9.4.26 - Recurrent Depth - by Jamin Ball

cloudedjudgement.substack.com · 2,350 words · saved by 1 readers

Every week I’ll provide updates on the latest trends in AI and software

Every week I’ll provide updates on the latest trends in cloud software companies. Follow along to stay up to date! Recurrent Depth This week we got a peak into OpenAIs new model Astra. I thought this was an important model because it’s rumored to use a new technique called recurrent depth. In this post I want to discuss what that means and what the implications are. But first - a quick timeline of some of the larger breakthroughs in model architecture: 2017: The Transformer era. The leap here was self-attention mechanism. Before 2017 recurrent neural networks (RNNs) processed text…

saved by

related reading