Apache Flink® 1.19 is here! On behalf of the Flink community, David Anderson highlights key release updates with FLIPs for Legacy deprecations, Flink SQL, Observability, Flink Configuration, and Flink Connectors.

Stream processing plays a critical role in the infrastructure stack for data streaming. Developers can use it to filter, join, aggregate, and transform their data streams on the fly to power real-time applications and streaming data pipelines. Among stream processing frameworks, Apache Flink has emerged as the de facto standard because of its performance and rich feature set. However, self-managing Flink (like self-managing other open source tools like Kafka) can be challenging due to its operational complexity, steep learning curve, and high costs for in-house support.

The Confluent Q1 ’24 Launch is packed with new features that enable customers to build, connect, and consume intelligent data pipelines seamlessly and securely Our quarterly launches provide a single resource to learn about the accelerating number of new features we’re bringing to Confluent Cloud, our cloud-native data streaming platform.

Adam Bellemare compares four main types of Event-Driven Architecture (EDA): Application Internal, Ephemeral Messaging, Queues, and Publish/Subscribe. Event-Driven Architectures have a long and storied history, and for good reason. They offer a powerful way to build scalable and decoupled architectures. But thanks to its long history, people often have different ideas of what EDA means depending on when they first encountered this architecture.

Schema evolution is the act of modifying the structure of the data in our application, without impacting clients. This can be a challenging problem. However, it gets easier if we start with a flexible data format and take steps to avoid unnecessary data coupling. When we find ourselves having to make breaking changes, we can always fall back to creating new versions of our APIs and events to accommodate those changes.

Now that your data is inside your Kafka cluster, how do you get it out? In this video, Dan Weston covers the basics of Kafka Consumers: what consumers are, how they get your data flowing, and best practices for configuring consumers in a real-time data streaming system. You will also learn about offsets, consumer groups, and partition assignment.

Understanding how to get data into your Kafka cluster is the first step to real-time streaming data. Producer has entered the chat. Watch this video to find out all you need to know about how a Kafka producer helps your data flow into the future.

The Listen to Yourself pattern is implemented by having a microservice emit an event to a platform such as Apache Kafka, and then consuming its own events to perform internal updates. It can be used as a solution to the dual-write problem since it separates Kafka and database writes into different processes. However, it also provides added benefits because it allows microservices to respond quickly to requests by deferring processing to a later time.