Running your own Kafka is starting to feel like wading through oatmeal. We’re not the only ones thinking that. The majority of organizations we speak to have or are in the process of moving their Kafka to a managed service. If you’re already an AWS-shop, Managed Streaming for Apache Kafka (MSK) is a no-brainer. It is the same Kafka that we know and love and integrated with other AWS services such as IAM, Cloudwatch, Cloudtrail, KMS, VPC and more.
Kong Enterprise is a service connectivity platform that provides technology teams with the architectural freedom to build, operate, observe, and secure APIs and services anywhere. From Kong’s inception, we’ve been aligned with Amazon Web Services (AWS), enabling our customers to quickly and efficiently deploy Kong on their AWS accounts. As companies move from monolithic to microservice applications and beyond, Kong helps teams manage this transition.
Infrastructure as code has been an important practice of DevOps for years. Anyone running an Apache Kafka data infrastructure and running on Kubernetes, the chances are you’ve probably nailed defining your infrastructure this way. If you’re running on Kubernetes, you’re likely using operators as part of your CI/CD toolchain to automate your deployments.
In an ideal world where we reached 100% test coverage, our error handling was flawless, and all our failures were handled gracefully — in a world where all our systems reached perfection, we wouldn’t be having this discussion. Yet, here we are. Earth, 2020. By the time you read this sentence, somebody’s server failed in production. A moment of silence for the processes we lost.
Data scientists today have to choose between a massive toolbox where every item has its pros and cons. We love the simplicity of Python tools like pandas and Scikit-learn, the operation-readiness of Kubernetes, and the scalability of Spark and Hadoop, so we just use all of them. What happens? Data scientists explore data using pandas, then data engineers use Spark to recode the same logic to scale or with live streams or operational databases.
