There are many reasons for working on community contributions such as a blog post, a demo, or a talk. Sometimes, you produce those contributions to share something that you’ve learned at work. But sometimes, the contribution itself can be a way to learn and experiment something new. I was in the second case when I worked on the article Kafka Streams: a road to Autoscaling via Kubernetes.
At that time, I was working for a consulting company, and my client was running one of it’s first real-time project on Apache Kafka®. We were developing a Kafka Streams application meant to be deployed on three virtual machines to process data from ~12 partitions. We were satisfied with how these instances handled the normal workload. However, in some situations, we wished we could be more flexible and easily add more deployments. Issues like crashes; restarts; upgrades led to significant increases in the record lag. Meanwhile, our colleagues were all converted to the same sect: Kubernetes. So I started to ask my self: “can we address this particular problem with Kubernetes”. And that’s how I’ve started to explore Kubernetes custom metrics API.
Basically the demo: xke-kingof-scaling brings a random data generator, a Kafka Streams application and a minimal infra (declared with Terraform) to recreate the use case and highlight the issue I was trying to address. The project was running on GCP, and the Kafka cluster used was on Confluent Cloud. The first deadline for this hack project was actually a presentation. I had to present the talk Scale in / Scale out with Kafka Streams and Kubernetes 🎞 🇫🇷 at the 2018 Edition of Xebicon.
Doing this demo (and the corresponding blog post) brought me a lot. Not only because I’ve learned a lot, but also because I’ve been stuck halfway on my demonstration and had to figure out how to get things to work. At the time, the two pieces of software I was dealing with, a Kafka Steams application and the Kube custom metrics API, didn’t have a trivial integration. But it had a different dynamic to the story. And now that I can take a step back from it, I think it was one of the funniest demos I’ve created.