Systems | Development | Analytics | API | Testing

Kafka

lenses

Overview of Cloud storage for your data platform

One of the most important questions in architecting a data platform is where to store and archive data. In a blog series, we’ll cover the different storage strategies for Kafka and introduce you to Lenses’ S3 Connector for backup/restore. But in this first blog, we must introduce the different Cloud storage options available. Later blogs will focus on specific solutions, explain in more depth how this maps to Kafka and then how Lenses manage your Kafka topic backups.

integrate

Building a Real-time Snowflake Data Pipeline with Apache Kafka

In today's data-driven world, organizations seek efficient and scalable solutions for processing and analyzing vast amounts of data in real-time. One powerful combination that enables such capabilities is Snowflake, a cloud-based data warehousing platform, and Apache Kafka, a distributed streaming platform.

ably

Ably Kafka Connector 3.0: Increased throughput, improved error handling, Confluent Cloud accreditation

We are excited to announce the release of the Ably Kafka Connector 3.0. Version 3 brings a host of improvements, including: Overall, the Ably Kafka connector v3.0 makes the management of Kafka pipelines extension to millions of web and mobile users simpler and more reliable.

lenses

Lenses 5.2: for a healthier Kafka with less manual effort

Kafka adoption is growing fast. Very fast. At Lenses, we’re pushing out new features to increase developer productivity, reduce manual effort & improve the cost & hygiene of operating your Kafka platform. Only a few weeks since Lenses 5.1, yet here we are again with more goodies in our release 5.2.

lenses

Lenses 5.1 - A 1st class ticket to be event-driven in AWS

Hello again. We strive to improve the productivity of developers building event-driven applications on the technology choices that best fit your organization. AWS continues to be a real powerhorse for our customers. Not just for running the workloads, but in supporting them with their native services: MSK Kafka, MSK Connect and now increasingly Glue Schema Registry. This is bringing a strong alternative to Confluent and their Kafka infrastructure offerings.

lenses

Secret rotation for Kafka Connect connectors with AWS Secret Manager

With version 5.1, Lenses is now offering enterprise support for our popular open-source Secret Provider to customers. In this blog, we’ll explain how secrets for Kafka Connect connectors can be safely protected using Secret Managers and walk you through configuring the Lenses S3 Sink Connector with the Lenses Secret Provider plugin and AWS Secret Manager.

lenses

The Glue Schema that binds Apache Kafka

With increased applications developed by different engineering teams on Kafka comes increased need for data governance. JSON is often used when streaming projects bootstrap but this quickly becomes a problem as your applications iterate, changing the data structures with add new fields, removing old and even changing data formats. It makes your applications brittle, chaos ensues as downstream consumers fall over due to miss data and SREs curse you.

kong

From Kinesis to Kafka

At the beginning of 2021, a brand new data team was assembled to build a real-time data platform for Kong’s SaaS platform, Konnect. Our mission is to provide top-notch real-time API analytics features for Konnect customers. V1 platform architecture The initial backend consisted of three main components: an ingestion service to handle incoming telemetry requests, an intermediate message queue, and a real-time OLAP data storage.