“We’ve seen two years’ worth of digital transformation in two months” said Microsoft’s Satya Nadella. Due to COVID-19, digital transformation roadmaps have been deleted, redrafted, doubled down and accelerated by up to a decade. Traditional companies are moving by osmosis towards streaming technologies such as Apache Kafka to kick off new digital services. But how much should it cost to experience 2030 in 2021?

At the heart of Kafka is real-time data. With data at the center of any Kafka environment, it should be the area that gets the most attention, but typically it gets the least. This happens because we see most organizations split their Kafka efforts into three areas: infrastructure, monitoring, and data operations.

Are you running your organization's Apache Kafka on-premise? If you are and you’re still reading this article, it’s more than likely that Kafka is or will be a keystone of your data infrastructure. But it’s also likely your teams are tired of the cost and complexity required to scale it, meaning your honeymoon with Kafka is coming to an end. So what does the imminent migration mean?

The key differences between Amazon Kinesis and Kafka are: Introducing data streamers! These services validate and route messages from one application to another, managing workload and message queues effectively. The result? Users process messages through a centralized processor and handle large data streams more efficiently. Amazon Kinesis and Apache Kafka are two data stream services.

An old airport customer of mine (whilst I worked for another company) used to pop someone next to a busy runway with a stopwatch strapped round their neck. The unfortunate person had to manually log the time aircrafts spent on the runway to measure the runway occupancy. All very archaic. Even in those days.

We’re continuously releasing new capabilities to enable more people to be productive and compliant when working with Apache Kafka. Our engineers have been hard at work on a great Christmas present for our amazing community of Kafka users. Before unwrapping our latest and greatest release, here’s a sneak peak of what’s inside.

In this article we are going to explore lateral joins. "What is a lateral join?" you may ask. It's a new kind of join that allows to extract and work with the single elements found inside an array, as if the array was a normal table. Lenses 4.1 comes with a lot of new features that make your life easier when working with arrays: we introduced 6 new functions to work with arrays, better support for array literals, and lateral joins.

In the previous post, we talked about Kerberos authentication and explained how to configure a Kafka client to authenticate using Kerberos credentials. In this post we will look into how to configure a Kafka client to authenticate using LDAP, instead of Kerberos. We will not cover the server-side configuration in this article but will add some references to it when required to make the examples clearer.

It's important to understand the uses and abuses of streaming infrastructure. Apache Kafka is a message broker that has rapidly grown in popularity in the last few years. Message brokers have been around for a long time; they're a type of datastore specialized for "buffering" messages between producer and consumer systems. Kafka has become popular because it's open-source and capable of scaling to very large numbers of messages.

If you’re moving data in and out of Kafka, chances are you’re using Apache Kafka Connect. The strength of the Kafka community means we have a large catalog of available connectors for hundreds of different technologies, helping to accelerate the integration of Kafka with 3rd party systems.