Beyond Tiered Storage: Deep Dive into WarpStream's Storage Engine

In this talk, we'll discuss in detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka® protocol compatible system that has zero local disks. We'll cover how to design a storage engine from the ground up around commodity object storage. We'll dive deep into WarpStream's storage engine implementation and cover topics like: ingestion, metadata, compaction, fetch, and service discovery. We'll see how the concept of tiered storage can be re-introduced even in a system that has zero local disks to make different trade-offs around cost and performance.

🤔 Learn more about WarpStream's architecture: https://docs.warpstream.com/warpstream/overview/architecture

🤔 Learn more about WarpStream Bring Your Own Cloud (BYOC): https://www.warpstream.com/bring-your-own-cloud-kafka-data-streaming ... https://www.youtube.com/watch?v=Ecl75zeYMzM