Factorial Funds | Thoughts on Llama 3

Meta has announced the 3rd version of their open large language model, Llama 3. In this blog post we’ll dive into the details and provide our thoughts on how Llama 3 will shape the industry. Meta has been training LLMs for a while now: They published Llama 1 in February 2023, followed by Llama 2 in July 2023. Llama 2 in particular has been hugely influential: it’s been used extensively by the open-source community, with the 7B model in particular having been downloaded on HuggingFace more than 1 million times, and is one of the go-to models for the open-source community to build on top of. While the paper is not out yet, the blog post and model card contain lots of information on the technical details of Llama 3. Like before, Llama 3 is a dense Transformer model that was pre-trained on a very large amount of publicly available data, about 15T tokens (we’ll discuss the dataset separately, as it is one of the most interesting aspects of this release). Also, similar to before, Llama 3 is

Meta has announced the 3rd version of their open large language model, Llama 3. In this blog post we’ll dive into the details and provide our thoughts on how Llama 3 will shape the industry. Meta has been training LLMs for a while now: They published Llama 1 in February 2023, followed by Llama 2 in July 2023. Llama 2 in particular has been hugely influential: it’s been used extensively by the open-source community, with the 7B model in particular having been downloaded on HuggingFace more than 1 million times, and is one of the go-to models for the open-source community to build on top of. Whi