Systems | Development | Analytics | API | Testing

Operational and analytical estates have been separated since data warehouses were first introduced in the 1990s. The operational estate includes microservices, software-as-a-service (SaaS) apps, and enterprise resource planning systems (ERPs) that have become the beating heart of an organization. The analytical estate consists of the data warehouses, lakehouses, artificial intelligence (AI)/machine learning (ML) platforms, and other custom batch workloads that support business analysis and reporting.

Businesses need to be able to respond quickly to macroeconomic uncertainty and changing market conditions, whether those are caused by trade disputes, public health crises like the COVID-19 pandemic, or simply abrupt changes in consumer demand. Amid that swirl of disruptions, technology leaders are also being pressured by management and investors to optimize their businesses and drive as much efficiency as possible.

For Taohao, being a Senior Software Engineer means never standing still. Whether it’s picking up a new skill, diving deeper into a complex system, or exchanging ideas with teammates, he thrives in an environment where curiosity, innovation, and collaboration come together. Let’s find out what makes Confluent the place where engineers like Taohao aren’t just solving problems but are constantly learning, challenging each other, and building what’s next.

A joint post from the teams at NeuBird and Confluent For organizations leveraging Confluent, ensuring smooth operations is mission-critical. While Confluent Cloud eliminates the operational burden of managing Apache Kafka, application teams still need to monitor and troubleshoot client applications connecting to Kafka clusters.

In my last blog post, I looked at the essential first part of building any data pipeline—exploring the raw source data to understand its characteristics and relationships. The data is information about river levels, rainfall, and other weather information provided by the UK Environment Agency on a REST API. I used the HTTP Source connector to stream this into Apache Kafka topics (one per REST endpoint), and then Tableflow to expose these as Apache Iceberg tables.