Step 1 â€” Create a User for Kafka
As Kafka can handle requests over a network, you should create a dedicated user for it. This minimizes damage to your Ubuntu machine should the Kafka server be comprised.
Note: After setting up Apache Kafka, it is recommended that you create a different non-root user to perform other tasks on this server.
As root, create a user ed kafka using the useradd command:
useradd kafka -m
Set its password using passwd:
Add it to the sudo group so that it has the privileges required to install Kafka's dependencies. This can be done using the adduser command:
adduser kafka sudo
Your Kafka user is now ready. Log into it using su:
su - kafka
Step 2 â€” Install Java
Before installing additional packages, update the list of available packages so you are installing the latest versions available in the repository:
sudo apt-get update
As Apache Kafka needs a Java runtime environment, use apt-get to install the default-jre package:
sudo apt-get install default-jre
Step 3 â€” Install ZooKeeper
Apache ZooKeeper is an open source service built to coordinate and synchronize configuration information of nodes that belong to a distributed system. A Kafka cluster depends on ZooKeeper to performâ€”among other thingsâ€”operations such as detecting failed nodes and electing leaders.
Since the ZooKeeper package is available in Ubuntu's default repositories, install it using apt-get.
sudo apt-get install zookeeperd
After the installation completes, ZooKeeper will be started as a daemon automatically. By default, it will listen on port 2181.
To make sure that it is working, connect to it via Telnet:
telnet localhost 2181
At the Telnet prompt, type in ruok and press ENTER.
If everything's fine, ZooKeeper will say imok and end the Telnet session.
Step 4 â€” Download and Extract Kafka Binaries
Now that Java and ZooKeeper are installed, it is time to download and extract Kafka.
To start, create a directory called Downloads to store all your downloads.
mkdir -p ~/Downloads
Use wget to download the Kafka binaries.
Create a directory called kafka and change to this directory. This will be the base directory of the Kafka installation.
mkdir -p ~/kafka && cd ~/kafka
Extract the archive you downloaded using the tar command.
tar -xvzf ~/Downloads/kafka.tgz --strip 1
Step 5 â€” Configure the Kafka Server
The next step is to configure the Kakfa server.
Open server.properties using vi:
By default, Kafka doesn't allow you to delete topics. To be able to delete topics, add the following line at the end of the file:
delete.topic.enable = true
Save the file, and exit vi.
Step 6 â€” Start the Kafka Server
Run the kafka-server-start.sh script using nohup to start the Kafka server (also called Kafka broker) as a background process that is independent of your shell session.
nohup ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties > ~/kafka/kafka.log 2>&1 &
Wait for a few seconds for it to start. You can be sure that the server has started successfully when you see the following messages in ~/kafka/kafka.log:
excerpt from ~/kafka/kafka.log
[2015-07-29 06:02:41,736] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2015-07-29 06:02:41,776] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
You now have a Kafka server which is listening on port 9092.