How to Set Up Kafka 4.0.0 for Windows

Note: in some of the paths written below, I will be writing the path to your Kakfa installation directory as kafka\. Replace it with the path where you placed your Kafka installation directory (e.g., C:\kafka).

1. Download and Install Kafka 4.0.0

This section provides instructions for downloading and installing Kafka on Windows.

1. Download Kafka from https://kafka.apache.org/downloads. Look for the section that matches the version number of interest, then choose the binary download.
2. Extract the downloaded files and place your Kafka directory in a convenient location, like C:\kafka.

2. Edit kafka-run-class.bat

This section provides instructions for editing kafka-run-class.bat (in kafka\bin\windows\) to prevent the input line is too long error and the DEPRECATED: A Log4j 1.x configuration file has been detected warning.

Consider creating a backup file kafka-run-class.bat.backup before proceeding.

1. If you have placed your Kakfa installation directory in a path longer than C:\kafka, you would most likely need to edit kafka-run-class.bat to prevent the input line is too long error:

   1. In kafka-run-class.bat, replace the following lines (originally at lines 92-95):

      rem Classpath addition for release
      for %%i in ("%BASE_DIR%\libs\*") do (
         call :concat "%%i"
      )
      

      With the following lines:

      rem Classpath addition for release
      call :concat "%BASE_DIR%\libs\*;"
      

   2. Restart command prompt if it was open.

2. To prevent the DEPRECATED: A Log4j 1.x configuration file has been detected warning:

   In kafka-run-class.bat, replace the following lines (originally at lines 117-123):

   rem Log4j settings
   IF ["%KAFKA_LOG4J_OPTS%"] EQU [""] (
       set KAFKA_LOG4J_OPTS=-Dlog4j2.configurationFile=file:%BASE_DIR%/config/tools-log4j2.yaml
   ) ELSE (
       rem Check if Log4j 1.x configuration options are present in KAFKA_LOG4J_OPTS
       echo %KAFKA_LOG4J_OPTS% | findstr /r /c:"log4j\.[^ ]*(\.properties|\.xml)$" >nul
       IF %ERRORLEVEL% == 0 (
   

   With:

   rem Log4j settings
   setlocal enabledelayedexpansion
   IF ["%KAFKA_LOG4J_OPTS%"] EQU [""] (
     set KAFKA_LOG4J_OPTS=-Dlog4j2.configurationFile=file:%BASE_DIR%/config/tools-log4j2.yaml
   ) ELSE (
     rem Check if Log4j 1.x configuration options are present in KAFKA_LOG4J_OPTS
     echo %KAFKA_LOG4J_OPTS% | findstr /r /c:"log4j\.[^ ]*(\.properties|\.xml)$" >nul
     IF !ERRORLEVEL! == 0 (
   

   Note the key changes:

   1. Added setlocal enabledelayedexpansion
   2. Changed %ERRORLEVEL% to !ERRORLEVEL!

   Additional information:

   • There is an error in the logic due to the order of expansion of environment variables in batch files.
   • Environment variables surrounded by % are expanded when the line is parsed, not when it's executed.
   • This means that while %ERRORLEVEL% is being changed dynamically at runtime, it does not expand to the updated value.
   • %ERRORLEVEL% was expected to expand to 1 due to the command echo %KAFKA_LOG4J_OPTS% | findstr /r /c:"log4j\.[^ ]*(\.properties|\.xml)$" >nul not finding a match
   • However, %ERRORLEVEL% expands to 0 instead of 1. %ERRORLEVEL% == 0 wrongly evaluates to true, causing the code in the IF !ERRORLEVEL! == 0 block to run, which includes printing the DEPRECATED: A Log4j 1.x configuration file has been detected warning.
   • To fix this issue, delayed expansion must be used, as shown in the instructions above.

3. Configure Kafka for Running in KRaft Mode

This section provides instructions for setting the log.dirs property in server.properties (in kafka\config\).

This section also provides instructions for setting the controller.quorum.voters property in server.properties and formatting the storage directory for running Kafka in KRaft mode, to prevent the no readable meta.properties files found error.

Consider creating a backup file server.properties.backup before proceeding.

1. In server.properties, replace the following line (originally at line 73):

   log.dirs=/tmp/kraft-combined-logs
   

   With the following line:

   log.dirs=path/to/kafka/kraft-combined-logs
   

   Replace path/to/kafka/ with the path to your Kafka installation directory. Use "/" instead of "\" in the path to avoid escape issues and ensure compatibility.

2. In server.properties, add the following lines to the bottom of the "Server Basics" section (originally at line 16 to 25):

   # Define the controller quorum voters for KRaft mode
   controller.quorum.voters=1@localhost:9093
   

   This is for a single-node Kafka cluster. For a multi-node Kafka cluster, list multiple entries like:

   controller.quorum.voters=1@host1:9093,2@host2:9093,3@host3:9093
   

3. In command prompt, temporarily set the KAFKA_LOG4J_OPTS environment variable by running the command:

   set KAFKA_LOG4J_OPTS=-Dlog4j.configurationFile=path/to/kafka/config/log4j2.yaml
   

   Replace path/to/kafka/ with the path to your Kafka installation directory. Use "/" instead of "\" in the path to avoid escape issues and ensure compatibility.

4. In command prompt, change directory to your Kafka installation directory, then generate a unique cluster ID by running the command:

   bin\windows\kafka-storage.bat random-uuid
   

5. In command prompt, use the generated cluster ID to format your Kafka storage directory:

   bin\windows\kafka-storage.bat format -t <generated UUID> -c config\server.properties
   

   Replace <generated UUID> with the ID generated in step 4.

4. Start Kafka

This section provides instructions to start Kafka and verify that it is working correctly.

1. In command prompt, change directory to your Kafka installation directory, then start Kafka using the command:

   bin\windows\kafka-server-start.bat config\server.properties
   

2. Verify that it is working correctly. For example, test with a Spring Boot + Kafka application:

   1. Prepare a Spring Boot application that includes:
      • A @RestController class to handle HTTP requests
      • A Kafka producer @Service class (e.g., using a KafkaTemplate object)
      • A Kafka consumer @Service class (e.g., using @KafkaListener)
      • A @SpringBootApplication main class to bootstrap the application
   2. Ensure that your consumer prints received messages to console.
   3. While Kafka is running, run your Spring Boot + Kafka application.
   4. Use a simple GET call, by visiting an endpoint URL in a browser, to trigger the producer. E.g., visit http://localhost:8080/test/produce-message?message=test+message.
   5. Check the console which your Spring Boot + Kafka application is running on to see if the test message is printed.