单机单实例部署Kafka及测试

部署

Docker

# dockerfile
FROM ubuntu:22.04
WORKDIR /home/kafka-alone/
RUN apt-get update && apt-get upgrade -y
RUN apt-get install default-jdk -y
RUN apt-get install wget -y
RUN wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
RUN tar -xvzf kafka_2.13-3.7.0.tgz
RUN wget https://archive.apache.org/dist/zookeeper/zookeeper-3.9.2/apache-zookeeper-3.9.2-bin.tar.gz
RUN tar -xvzf apache-zookeeper-3.9.2-bin.tar.gz
RUN mv apache-zookeeper-3.9.2-bin zookeeper
RUN mv kafka_2.13-3.7.0 kafka
RUN rm -rf kafka_2.13-3.7.0.tgz
RUN rm -rf apache-zookeeper-3.9.2-bin.tar.gz
RUN mv zookeeper/conf/zoo_sample.cfg zookeeper/conf/zoo.cfg
EXPOSE 9092
RUN echo "./kafka/bin/zookeeper-server-start.sh ./kafka/config/zookeeper.properties &" >> start.sh
RUN echo "./kafka/bin/kafka-server-start.sh ./kafka/config/server.properties" >> start.sh
CMD ["/bin/bash", "-ce", "sh start.sh"]

然后使用下面指令编译成镜像

docker build --pull --rm -f "dockerfile" -t kafka-all-in-one:latest "." 

测试

启动

docker container run -d -p 9092:9092 --name kafka-all-in-one-container kafka-all-in-one:latest

容器内的执行日志如下

测试

创建Topic

docker exec -it kafka-all-in-one-container /bin/bash kafka/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092

生产者

docker exec -it kafka-all-in-one-container /bin/bash kafka/bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092

然后在终端中输入 hello world两个单词。

消费者

docker exec -it kafka-all-in-one-container /bin/bash kafka/bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092

消费者就会收到生产者发布的两个消息。

集成Manager的部署

我们集成Kafka Manager到docker中，以可视化的方式管理Kafka。

dockerfile

FROM ubuntu:22.04
WORKDIR /home/kafka-alone/
RUN apt-get update && apt-get upgrade -y
RUN apt-get install default-jdk -y
RUN apt-get install wget -y
RUN wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
RUN tar -xvzf kafka_2.13-3.7.0.tgz
RUN wget https://archive.apache.org/dist/zookeeper/zookeeper-3.9.2/apache-zookeeper-3.9.2-bin.tar.gz
RUN tar -xvzf apache-zookeeper-3.9.2-bin.tar.gz
RUN mv apache-zookeeper-3.9.2-bin zookeeper
RUN mv kafka_2.13-3.7.0 kafka
RUN rm -rf kafka_2.13-3.7.0.tgz
RUN rm -rf apache-zookeeper-3.9.2-bin.tar.gz
RUN mv zookeeper/conf/zoo_sample.cfg zookeeper/conf/zoo.cfg
EXPOSE 9092
RUN echo "./kafka/bin/zookeeper-server-start.sh ./kafka/config/zookeeper.properties &" >> start.sh
RUN echo "./kafka/bin/kafka-server-start.sh ./kafka/config/server.properties &" >> start.sh

RUN apt-get install unzip -y
RUN wget https://github.com/yahoo/CMAK/releases/download/3.0.0.6/cmak-3.0.0.6.zip
RUN unzip cmak-3.0.0.6.zip
RUN mv cmak-3.0.0.6 cmak
RUN rm -rf cmak-3.0.0.6.zip
RUN sed -i 's/kafka-manager-zookeeper:2181/localhost:2181/g' cmak/conf/application.conf
RUN sed -i 's/cmak.zkhosts/#cmak.zkhosts/g' cmak/conf/application.conf
RUN echo "./cmak/bin/cmak -Dconfig.file=./cmak/conf/application.conf -Dhttp.port=8080" >> start.sh
EXPOSE 8080
CMD ["/bin/bash", "-ce", "sh start.sh"]

效果

使用下面命令部署后

docker build --pull --rm -f "dockerfile" -t kafka-all-in-one:latest "." 
docker container run -d -p 9092:9092 -p 8080:8080 --name kafka-all-in-one-container kafka-all-in-one:latest

参考资料

• https://www.cnblogs.com/szx666/p/14635910.html
• https://blog.csdn.net/qq_44519484/article/details/121908565
• https://medium.com/yavar/installation-of-apache-kafka-on-ubuntu-22-04-59e57f878979
• https://akhq.io/docs/installation.html#docker
• https://github.com/yahoo/CMAK