According to the website, Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

In this post, I am going to share a basic way to start using Apache Kafka with python.

Photo by Francisco De Nova on Unsplash

What are the tools needed?

• docker-compose

Docker compose will be used to run kafka and it’s dependencies.

1. zookeeper
2. kafka
3. kafka-ui

The compose file-

As you can see, I have used zookeeper and kafka images from bitnami. To get a generic view of kafka instance, I have used an opensource project from Github. The project is called kafka-ui which is managed by provectus. It is a web ui which can be accessed via hosts localhost with the port of 18080, as you can see from the compose file.

Let you want to produce some message and ship it to kafka.

for consuming –

There are several kafka client for python, but I have used – kafka-python

First, you want to kconsumer.py and then from another terminal you can run the kproduce.py to produce the message.

In kconsumer.py terminal, you will get the messages –

If you want to explore the api for accessing kafka using this kafka-python, you can look into here.

• apache kafka
• docker
• docker-compose
• kafka
• python kakfa