Is sequential IO dead in the era of the NVMe drive? — Jack Vanlightly
Two systems I know pretty well, Apache BookKeeper and Apache Kafka, were designed in the era of the spinning disk, the hard-drive or HDD. Hard-drives are good at sequential IO but not so good at random IO because of the relatively high seek times. No wonder then that both Kafka and BookKeeper were d
Image credit: ESO/A. Ghizzi Panizza (www.albertoghizzipanizza.com) Two systems I know pretty well, Apache BookKeeper and Apache Kafka, were designed in the era of the spinning disk, the hard-drive or HDD. Hard-drives are good at sequential IO but not so good at random IO because of the relatively high seek times. No wonder then that both Kafka and BookKeeper were designed with sequential IO in mind. Both Kafka and BookKeeper are distributed log systems and so you’d think that sequential IO would be the default for an append-only log storage system. But sequential and random IO sit on a continu
related reading
- IO devices and latency — PlanetScaleplanetscale.com
- Everything I know about good system designseangoedecke.com
- Designkafka.apache.org
- Log Structured Merge Trees | Ben Stopfordbenstopford.com
- Building and operating a pretty big storage system called S3 | All Things Distributedallthingsdistributed.com
- Build Your Own Databasenan.fyi
- Chain Replication for Supporting High Throughput and Availabilitycs.cornell.edu
- Production Twitter on One Machine? 100Gbps NICs and NVMe are fast - Tristan Humethume.ca
- Analyzing the codebase of Caffeine: a high performance caching library | Adri’s Blogadriacabeza.github.io
- When to use RabbitMQ or Apache Kafka - CloudAMQPcloudamqp.com
- Best practices for right-sizing your Apache Kafka clusters to optimize performance and cost | AWS Big Data Blogaws.amazon.com
- DBMS Musings: It’s Time to Move on from Two Phase Commitdbmsmusings.blogspot.com