Since the 2006 launch of Amazon Web Services (AWS), the world’s first hyper-scale public cloud provider, thousands of data-driven businesses have shifted on-premise data storage and analytics workloads into the cloud by architecting or adopting a cloud data platform. As the volume, variety, and velocity of enterprise data continues to grow in 2023, cloud data platforms with legacy tech and complex architectures are becoming increasingly time-consuming and costly to manage.

There is a long history of clustering architectures with respect to building distributed databases for two primary reasons. The first is scalability. If a cluster of nodes has reached its capacity to perform work, adding additional nodes are introduced to handle the increased load. The second is availability. The ability to ensure that if a node fails, let’s say during ingestion and/or querying, remaining nodes would continue to execute due to state replication.