How to install Kafka and Zookeeper on Ubuntu
To install Apache Kafka with Zookeeper on Ubuntu, follow these steps:
1. Install Java
Kafka requires Java. Install OpenJDK 11 (or higher):
sudo apt update
sudo apt install openjdk-11-jdkVerify installation:
java -version2. Download Apache Kafka
Access the most recent Kafka binary from Apache Kafka, replacing X.X.X with the current release.
wget https://downloads.apache.org/kafka/X.X.X/kafka_2.13-X.X.X.tgzExtract the file:
tar -xzf kafka_2.13-X.X.X.tgz
cd kafka_2.13-X.X.X3. Configure Zookeeper
- Create a data directory for Zookeeper:
mkdir -p /tmp/zookeeper- Edit the Zookeeper configuration (optional):
vim config/zookeeper.propertiesEnsure dataDir=/tmp/zookeeper is set (default).
4. Start Zookeeper
Run Zookeeper as a background service :
bin/zookeeper-server-start.sh -daemon config/zookeeper.propertiesVerify it’s running:
netstat -tnlp | grep 21815. Configure Kafka
- Edit
config/server.properties:
vim config/server.properties- Configure the log directory path (defaults to
log.dirs=/tmp/kafka-logs).- (Optional) if accessed remotely, specify the
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://<YOUR_IP>:90926. Start Kafka
Run Kafka as a background service :
bin/kafka-server-start.sh -daemon config/server.propertiesVerify Kafka is running:
netstat -tnlp | grep 90927. Test the Setup
- Create a topic :
bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1- Produce messages :
bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092- Consume messages (open a new terminal):
bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:90928. (Optional) Create Systemd Services
For auto-start on reboot:
Zookeeper Service :
sudo vim /etc/systemd/system/zookeeper.service [Unit]
Description=Zookeeper Service
After=network.target
[Service]
Type=simple
User=ubuntu
ExecStart=/path/to/kafka/bin/zookeeper-server-start.sh /path/to/kafka/config/zookeeper.properties
Restart=on-failure
[Install]
WantedBy=multi-user.targetKafka Service :
sudo vim /etc/systemd/system/kafka.service [Unit]
Description=Kafka Service
After=zookeeper.service
[Service]
Type=simple
User=ubuntu
ExecStart=/path/to/kafka/bin/kafka-server-start.sh /path/to/kafka/config/server.properties
Restart=on-failure
[Install]
WantedBy=multi-user.targetEnable the services :
sudo systemctl daemon-reload
sudo systemctl start zookeeper
sudo systemctl start kafka
sudo systemctl enable zookeeper kafka9. Stop Services
To stop Kafka and Zookeeper:
bin/kafka-server-stop.sh
bin/zookeeper-server-stop.shNotes
This configuration is specifically designed for development activities. When in a production environment, it's advisable to employ a multi-node cluster setup.
Modify firewall configurations when accessed remotely:
sudo ufw allow 2181/tcp # Zookeeper
sudo ufw allow 9092/tcp # KafkaYou now have a single-node Kafka cluster running on Ubuntu!