Owl: Distributing content at Meta scale - Engineering at Meta

engineering.fb.com · 2,073 words · saved by 1 readers

Being able to distribute large, widely -consumed objects (so-called hot content) efficiently to hosts is becoming increasingly important within Meta’s private cloud. These are commonly distributed content types such as executables, code artifacts, AI models, and search indexes that help enable our software systems. Owl is a new system for high-fanout distribution of large data objects to hosts in Meta’s private cloud. It distributes over 700 petabytes of data per day to millions of client processes and has improved download speeds and cache hit rate by a factor of 2-3x over BitTorrent and other prior systems used for distribution at Meta. There are three dimensions to distribution: Distribution requirements are exacting. First, content distribution must be fast. Slow executable delivery, for example, increases downtime and delays fixes. We strive to provide data at a rate bounded by either available network bandwidth of the reading host or by the available write bandwidth of its storag

By Jason Flinn , Arushi Aggarwal Being able to distribute large, widely -consumed objects (so-called hot content) efficiently to hosts is becoming increasingly important within Meta’s private cloud. These are commonly distributed content types such as executables, code artifacts, AI models, and search indexes that help enable our software systems. Owl is a new system for high-fanout distribution of large data objects to hosts in Meta’s private cloud. It distributes over 700 petabytes of data per day to millions of client processes and has improved download speeds and cache hit rate by a factor

Owl: Distributing content at Meta scale - Engineering at Meta

related reading