One of the most important questions in architecting a data platform is where to store and archive data. In a blog series, we’ll cover the different storage strategies for Kafka and introduce you to Lenses’ S3 Connector for backup/restore. But in this first blog, we must introduce the different Cloud storage options available. Later blogs will focus on specific solutions, explain in more depth how this maps to Kafka and then how Lenses manage your Kafka topic backups.

In today's data-driven world, organizations seek efficient and scalable solutions for processing and analyzing vast amounts of data in real-time. One powerful combination that enables such capabilities is Snowflake, a cloud-based data warehousing platform, and Apache Kafka, a distributed streaming platform.

We are excited to announce the release of the Ably Kafka Connector 3.0. Version 3 brings a host of improvements, including: Overall, the Ably Kafka connector v3.0 makes the management of Kafka pipelines extension to millions of web and mobile users simpler and more reliable.

Kafka adoption is growing fast. Very fast. At Lenses, we’re pushing out new features to increase developer productivity, reduce manual effort & improve the cost & hygiene of operating your Kafka platform. Only a few weeks since Lenses 5.1, yet here we are again with more goodies in our release 5.2.

Hello again. We strive to improve the productivity of developers building event-driven applications on the technology choices that best fit your organization. AWS continues to be a real powerhorse for our customers. Not just for running the workloads, but in supporting them with their native services: MSK Kafka, MSK Connect and now increasingly Glue Schema Registry. This is bringing a strong alternative to Confluent and their Kafka infrastructure offerings.

With version 5.1, Lenses is now offering enterprise support for our popular open-source Secret Provider to customers. In this blog, we’ll explain how secrets for Kafka Connect connectors can be safely protected using Secret Managers and walk you through configuring the Lenses S3 Sink Connector with the Lenses Secret Provider plugin and AWS Secret Manager.

With increased applications developed by different engineering teams on Kafka comes increased need for data governance. JSON is often used when streaming projects bootstrap but this quickly becomes a problem as your applications iterate, changing the data structures with add new fields, removing old and even changing data formats. It makes your applications brittle, chaos ensues as downstream consumers fall over due to miss data and SREs curse you.

At the beginning of 2021, a brand new data team was assembled to build a real-time data platform for Kong’s SaaS platform, Konnect. Our mission is to provide top-notch real-time API analytics features for Konnect customers. V1 platform architecture The initial backend consisted of three main components: an ingestion service to handle incoming telemetry requests, an intermediate message queue, and a real-time OLAP data storage.

Unify data from diverse sources and formats using Aiven Kafka's open source streaming. Analyze with BigQuery for swift, accurate insights.