Several years ago when I was working on a big data project, I saw something a data engineer shouldn’t see. Curious to understand the level of detail in a new credit score dataset we’d received in our data lake, I queried it. I was surprised at how easily and suddenly my screen was flooded with the mortgage history, overdraft limits and year-end financial statements of my colleagues, and I felt deeply uneasy.

Thanksgiving holiday is upon us. For many of our customers, this is one of the most important periods of the year, with more than 189.6 million U.S. shoppers buying up bargains from Thanksgiving day through Cyber Monday last year. For them and for us, it’s crucial that internal systems can handle high traffic volume without downtime or performance degradation.

In another life, I taught the Book of Genesis to high school students, including The Tower of Babel excerpt. It struck me ironic that God’s wrath strikes down the tower, cofounds the universal language and scatters humans around the globe to teach King Nimrod a lesson in hubris; meanwhile, the boys in my class were texting their girlfriends across the country and playing video games with friends in Europe and Asia.

Many in the community have been asking us to develop a new Kafka to S3 connector for some time. So we’re pleased to announce it's now available. It’s been designed to deliver a number of benefits over existing S3 connectors. Like our other Stream Reactors, the connector extends the standard connect config adding a parameter for a SQL command (Lenses Kafka Connect Query Language or “KCQL”). This defines how to map data from the source (in this case Kafka) to the target (S3).

Running your own Kafka is starting to feel like wading through oatmeal. We’re not the only ones thinking that. The majority of organizations we speak to have or are in the process of moving their Kafka to a managed service. If you’re already an AWS-shop, Managed Streaming for Apache Kafka (MSK) is a no-brainer. It is the same Kafka that we know and love and integrated with other AWS services such as IAM, Cloudwatch, Cloudtrail, KMS, VPC and more.

This post is about Kafka and the process I have been through recently writing a performance test for an application that subscribes to messages from this technology. The test I ended up with was in the end very straightforward but there were several hurdles that took a while to resolve. I hope that reading this post will hopefully help you avoid them.