Apache Kafka® 3.7 is here! On behalf of the Kafka community, Danica Fine highlights key release updates, with KIPs from Kafka Core, Kafka Streams, and Kafka Connect.

Check out the following blog for more information about the latest Apache Kafka release: https://cnfl.io/apache-kafka-3-7-blog

Kafka Core:

• KIP-714: Gives operators visibility into clients by exposing client metrics and provides new components to allow cluster operators to receive client metrics.
• KIP-1000: Introduces a way to list the client metrics configuration resources introduced by KIP-714.
• KIP-848 (Early Access): Introduces a new group membership and rebalance protocol for the Kafka Consumer and, by extension, Kafka Streams. Provides the same guarantee as the current protocol and introduces new Consumer Group Protocol types.
• KIP-951: Minimizes the time taken to discover a new leader of a partition, reducing the effect of leadership changes on the end-to-end latency of produce or fetch client requests.
• KIP-975: Provides an official JVM-based Apache Kafka Docker image. It enables quicker testing and deployment of Kafka, and faster onboarding of new developers joining the community.
• KIP-963: Increases observability into Tiered Storage operations and ultimately enables cluster operators to troubleshoot and prevent issues.
• KIP-580: Aims to reduce the common problem of slow metadata convergence after broker failure. Changes the static retry backoff mechanism to an exponentially increasing backoff policy for all Kafka clients.
• KIP-858 (Early Access): Allows JBOD-based deployments to be production capable with KRaft.
• KIP-896: Marks the use of old client API versions (Kafka versions older than 2.1.0) as deprecated, and introduces new metrics to monitor the presence of these requests. Support for these requests will be removed in Apache Kafka 4.0.
• KIP-938: Makes it easier for operators to understand how their KRaft-mode cluster is performing.
• KIP-979: Decouples KRaft controllers and Kafka brokers that may be running on the same machine, allowing them to be stopped independently from one another.
• KIP-1013: Marks the use of Java 11 in the Kafka broker as deprecated. Support for Java 11 will be removed in Apache Kafka 4.0.

Kafka Streams:

• KIP-925: Reduces cross-rack traffic by adding client rack awareness to Kafka Streams.
• KIP-954: Builds on KIP-591, providing a new interface for DslStoreSuppliers. It includes default implementations of this new interface for both RocksDB and in-memory state stores.
• KIP-988: Improves visibility into Kafka Streams StandbyTasks.
• KIP-992: Seeks to eliminate inconsistencies in query return types across regular and timestamp state stores.

Kafka Connect:

• KIP-976: Expands on KIP-495, adding broadcast capabilities to push log-level changes to all workers in the cluster. It eliminates the need to target an individual worker (which may not always be possible) for log-level updates.
• KIP-980: Allows users to create connectors in a stopped state, enabling a number of use cases, such as migrating connectors.
• KIP-959: Expands on KIP 907, adding BooleanConverter to Kafka Connect.

Many more KIPs are a part of this release. See the blog post for more details.

EPISODE LINKS
► See release notes for AK 3.7: https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html
► Read the blog post: https://cnfl.io/apache-kafka-3-7-blog
► Download AK 3.7: https://kafka.apache.org/downloads
► Get started with AK 3.7: https://kafka.apache.org/documentation/#quickstart

CHAPTERS

00:00 - Intro

00:44 - KIP-714: Client Metrics and Observability

01:16 - KIP-1000: List Client Metrics Configuration Resources

01:32 - KIP-848: The Next Generation of the Consumer Rebalance Protocol (Early Access)

01:58 - KIP-951: Leader Discovery Optimizations for the Client

02:24 - KIP-975: Docker Image for Apache Kafka

02:47 - KIP-963: Additional Metrics in Tiered Storage

02:56 - KIP-580: Exponential Backoff for Kafka Clients

03:33 - KIP-858: Handle JBOD Broker Disk Failure in KRaft

03:43 - KIP-896: Remove old client protocol API versions in Kafka 4.0

04:05 - KIP-938: Add more metrics for measuring KRaft performance

04:24 - KIP-979: Allow Independently stop KRaft Processes

04:54 - KIP-1013: Drop broker support for Java 11

05:20 - KIP-925: Rack Aware Task Assignment in Kafka Streams (Part 2)

05:42 - KIP-954: Expand default DSL store configuration to custom types

06:00 - KIP-988: Streams Standby Update Listener

06:20 - KIP-992: Introduce IQv2 Query Types

07:00 - KIP-976: Cluster-wide dynamic log adjustment

07:29 - KIP-980: Allow Creating Connectors in a Stopped State

07:53 - KIP-959: Add BooleanConverter to Kafka Connect

08:04 - It's a wrap!

#apachekafka #kafka #confluent