Systems | Development | Analytics | API | Testing

In my last blog post, I looked at the essential first part of building any data pipeline—exploring the raw source data to understand its characteristics and relationships. The data is information about river levels, rainfall, and other weather information provided by the UK Environment Agency on a REST API. I used the HTTP Source connector to stream this into Apache Kafka topics (one per REST endpoint), and then Tableflow to expose these as Apache Iceberg tables.

What’s your AI strategy? How are you making your service as easy to use as those from major cloud companies? How quickly can you roll out new features to customers? These are common questions technology executives face on a daily basis. They must accelerate innovation across the board––from deploying new features and making decisions to generating revenue and saving money.

Many teams begin their data streaming journey through their cloud provider, drawn to the simplicity of the one‑click “Create Kafka Cluster” button in their cloud console. It’s fast, feels integrated, and promises to “just work”—abstracting away all the operational tasks that only get more complicated in the cloud.

In today's competitive landscape, data warehouses and data lakes are the essential platforms for business intelligence, analytics, and AI. While immensely powerful, these systems were traditionally designed for batch data processing, often leading to insights based on data that is hours or even days old. The primary challenge has always been the complexity of bridging the gap between real-time data streams, typically flowing through Kafka, and these analytical systems.

Organizations have mastered collecting and storing vast amounts of data in cloud data warehouses like Snowflake. This central repository has become the single source of truth for analytical insights, business intelligence, and reporting. However, the true potential of this data remains trapped if it's confined to the warehouse, creating a disconnect between rich analytical insights and real-time operational systems.