How to Rack 30 Petabytes of Storage | blog

We built the heap, a 30 petabyte data storage cluster in downtown San Francisco, and spent under $500,000.

We built a storage cluster in downtown SF to store 90 million hours worth of video data. Why? We’re pretraining models to solve computer use. Compared to text LLMs like LLaMa-405B, which require ~60 TB of text data to train, videos are sufficiently large that we need 500 times more storage. Instead of paying the $12 million / yr it would cost to store all of this on AWS, we rented space from a colocation center in San Francisco to bring that cost down ~40x to $354k per year, including depreciation. Why Our use case for data is unique. Most cloud providers care highly about redundancy, availabi

saved by

• Ratan Kaliani
• Omkar Gadewar

related reading

• How to Rack 30 Petabytes of Storage | blogsi.inc
• Building and operating a pretty big storage system called S3 | All Things Distributedallthingsdistributed.com
• Multi-Datacenter Training: OpenAI's Ambitious Plan To Beat Google's Infrastructuresemianalysis.com
• Building a web search engine from scratch in two months with 3 billion neural embeddingsblog.wilsonl.in
• Century-Scale Storagelil.law.harvard.edu
• Network and Storage Benchmarks for LLM Training on the Cloud | Henry Zhumaknee.github.io
• From bare metal to a 70B model: infrastructure set-up and scripts - Imbueimbue.com
• Dwarkesh Goes Inside Jane Street's Latest AI Data Centeryoutube.com
• Training Dyna-2 at million-hour scale, repeatably — DYNAdyna.co
• Production Twitter on One Machine? 100Gbps NICs and NVMe are fast - Tristan Humethume.ca
• How to Estimate Cloud Data Warehouse Costs and Pricing Comparisonblog.panoply.io
• Training great LLMs entirely from ground up in the wilderness as a startup - Yi Tayyitay.net