In these uncertain times of the COVID-19 crisis, one thing is certain – data is key to decision making, now more than ever. And, the need for speed in getting access to data as it changes has only accelerated. It’s no wonder, then, that organisations are looking to technologies that help solve the problem of streaming data continuously, so they can run their businesses in real-time.

Our CTO, Andrew Stevenson was interviewed by Alan Shimel for TechStrong TV. The discussion was all about hot data topics such as DataOps, DevOps and practices to successfully enable Kafka. Andrew narrates his journey from civil engineering to starting Lenses.io with Antonios, our CEO, to help organizations succeed with real-time data.

Apache Kafka has gained traction as one of the most widely adopted technologies for building streaming applications - but introducing it (and scaling it) into your business can be a struggle. The problem isn’t with Kafka itself so much as the different components you need to learn and different tools required to operate it. For those motivated enough, you can invest money, effort and long Friday nights into learning, fixing and streamlining Kafka - and you’ll get there.

Data Engineers are forever flying the flag for open-source technology. But now that we’re safely locked away in our homes - potentially for the rest of the year - a new danger looms: That we get distracted by our new data tools and lose touch with delivering value to the business. Today most Data Engineers around the world are working from home, and at first glance it may seem like this works. After all, a solid internet connection is all we need to carry on doing what we were doing...

Some time ago, the concept of event streaming was not that widespread. In addition to that, the platforms provided were much fewer than today, and required more technical depth to set up and operate. However, as time passed, those platforms matured, community grew, the documentation was improved and the rest of the world started to wake up to the advantages these platforms can bring to address the real-time experiences businesses need. And Apache Pulsar is a great example.

Systems fail from time to time. And there is nothing worse than being unaware a system is down at all. Which is why we wrote the email alerts from Lenses.io tutorial. But what about those evenings where you are enjoying a documentary or just having an email-free evening? After all, system failures rarely happen at a magically convenient time.

Open source is great but sometimes it misses the mark for security at enterprise levels. Take Kafka Connect, I’ve built a few connectors in my time and prior to its introduction to Apache Kafka back in 2017 I used other hand cranked pieces of software and security was always a primary concern. One feature that will quickly put a blocker on your project being successful is not reaching production. There’s a number of reasons for this but high up, usually top, is security.