Apache Kafka Installation on Windows and MacOS

Ashish Lahoti

Last Modified: May 17, 2021

Page content

In this tutorial, We’ll learn how to install and run Apache kafka broker in local machine running on Windows or Mac operating system.

Running kafka broker in local machine helps developer to code, debug and test kafka application in initial phase of development when Kafka infrastructure is not ready.

Kafka Installation on Windows

If you are using Windows, then you can follow these steps to install Kafka:-

Install JDK 1.8 or higher

JDK 1.8 or higher version is prerequisite for Kafka Installation. You can download and install JDK based on your OS and CPU architecture from here:-
https://www.oracle.com/sg/java/technologies/javase-downloads.html

Once your JDK installation is complete, you must configure PATH and JAVA_HOME environment variables. You can set up your environment variables using the following steps:-

Start system settings in the control panel. You can directly start it by pressing Win + Pause/Break key.
Click on Advanced System Settings.
In the advanced tab, click the environment variable button.
In the user variables section, add new JAVA_HOME environment variable as your Java installation directory. In a typical case, your Java installation directory should be something like C:\Program Files\Java\jdk1.8.0_191. However, if you have installed JDK as per the steps defined above, you should have noted down your JDK location as advised earlier.
Now you can add a new PATH environment variable and specify the path as %JAVA_HOME%\bin

The final step is to test your JDK installation. Start windows command prompt and test JDK using below command. You should see the installed version:-

C:\>java -version
java version "1.8.0_191" 
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)

Thats it! Your JDK in installed successfully.

Install Kafka

Download and install Kafka binaries from here:-
https://kafka.apache.org/downloads

Note: At the time of writing this article, latest kafka version was kafka_2.13-2.8.0.tgz. It is recommended to use latest version available.
Download and install 7-zip (required to un-compress the Kafka binaries) from here:-
http://www.7-zip.org/download.html
Un-compress the kafka_2.13-2.8.0.tgz file using 7-zip, and you may have to use it twice to extract the files correctly.
For example, after exraction, my kafka location is as follow:-
```
C:\apache\kafka_2.13-2.8.0
```
Next, we need to change Zookeeper configuration zookeeper.properties as follows:-

Open the property file:-
```
C:\apache\kafka_2.13-2.8.0\config\zookeeper.properties
```
and change the Zookeeper dataDir location config to a valid windows directory location. An example value is given below.
```
zookeeper.properties
dataDir = C:\apache\kafka_2.13-2.8.0\zookeeper_data   
```
Make sure that zookeeper_data directory exist in specified location. If not, create one.
We also need to make some changes in the Kafka configurations server.properties as follows:-

Open the property file:-
```
C:\apache\kafka_2.13-2.8.0\config\server.properties
```
and change/add following configuration properties.
```
server.properties
log.dirs = C:\apache\kafka_2.13-2.8.0\kafka_logs 
offsets.topic.num.partitions = 1 
offsets.topic.replication.factor = 1 
min.insync.replicas = 1 
default.replication.factor = 1
```
Make sure that kafka_logs directory exist in specified location. If not, create one. Also note that we are setting topic defaults to 1, and that makes sense because we will be running a single node Kafka on our machine.
Finally, add Kafka C:\apache\kafka_2.13-2.8.0\bin\windows directory to the PATH environment variable. This directory contains a bunch of Kafka tools for the windows platform. We will be using some of those in the next section.

Run Kafka

Start Zookeeper in new terminal using:

C:\apache\kafka_2.13-2.8.0>bin\windows\zookeeper-server-start.bat config\zookeeper.properties

Start Kafka in new terminal using:

C:\apache\kafka_2.13-2.8.0>bin\windows\kafka-server-start.bat server\zookeeper.properties

Create a test kafka topic in new terminal using:

C:\apache\kafka_2.13-2.8.0>bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Publish messages to test kafka topic by initializing a producer in new terminal using:

C:\apache\kafka_2.13-2.8.0>bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test  
>send first message
>send second message
>wow it is working

Consume messages from test kafka topic by intializing a consumer in new terminal using:

C:\apache\kafka_2.13-2.8.0>bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning   
send first message
send second message
wow it is working

You can List the available Kafka topics using:-

C:\apache\kafka_2.13-2.8.0>bin\windows\kafka-topics.bat --list --zookeeper localhost:2181

Kafka Installation on Mac

If you are using MacOS, then you can install Homebrew package manager and run following commands from terminal:-

To install Java (Prerequisite for Kafka), run
```
brew install openjdk@11
```

To install Kafka, run

$ brew install kafka

Here are the brew logs of Kafka Installation, Note that brew also installs Zookeeper as Kafka dependency:-

==> Installing kafka dependency: zookeeper
==> Pouring zookeeper--3.7.0.catalina.bottle.tar.gz
==> Caveats
To have launchd start zookeeper now and restart at login:
  brew services start zookeeper
Or, if you don't want/need a background service you can just run:
  zkServer start
==> Summary
🍺  /usr/local/Cellar/zookeeper/3.7.0: 1,073 files, 42.4MB
==> Installing kafka
==> Pouring kafka--2.8.0.catalina.bottle.tar.gz
==> Caveats
To have launchd start kafka now and restart at login:
  brew services start kafka
Or, if you don't want/need a background service you can just run:
  zookeeper-server-start -daemon /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties
==> Summary
🍺  /usr/local/Cellar/kafka/2.8.0: 200 files, 68.2MB
==> zookeeper
To have launchd start zookeeper now and restart at login:
  brew services start zookeeper
Or, if you don't want/need a background service you can just run:
  zkServer start
==> kafka
To have launchd start kafka now and restart at login:
  brew services start kafka
Or, if you don't want/need a background service you can just run:
  zookeeper-server-start -daemon /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties

Edit the server.properties file
```
$ vim /usr/local/etc/kafka/server.properties
```
Uncomment the listeners property and update the value from
listeners=PLAINTEXT://:9092
to
listeners=PLAINTEXT://localhost:9092

Start Zookeeper in new terminal using:

$ zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties

Start Kafka in new terminal using:

$ kafka-server-start /usr/local/etc/kafka/server.properties

Create a test kafka topic in new terminal using:

$ kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Publish messages to test kafka topic by initializing a producer in new terminal using:

$ kafka-console-producer --broker-list localhost:9092 --topic test
>send first message
>send second message
>wow it is working

Consume messages from test kafka topic by intializing a consumer in new terminal using:

$ kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning
send first message
send second message
wow it is working

Delete the kafka logs if there is any issue in starting Kafka server
```
$ rm -rf /tmp/kafka-logs
```

Summary

You have successfully installed and run a single-node Kafka broker on your local machine running on Windows or Mac operating system. This installation will help you to execute your Kafka application code locally and help you debug your application from the IDE.