Kafka broker的新增和剔除(服役与退役)

说明:集群现有broker:node1,node2,node3三个,broker.id分别为0,1,2

已有两个topic:products、cities

1、退役(Kafka集群中减少一个服务器broker2)

       退役后要保证剩下的服务器数量大于等于备份数,否则会报错Error: Replication factor: 3 larger than available brokers: 2.;退役服务器中的数据在集群中剩下的服务器中实现负载均衡,在node3上进行操作。

操作步骤:

1.1创建json文件,写入需要均衡的主题

vim topic-move.json

{

        "topics": [

                {"topic": "products"},{"topic": "cities"}

        ],

        "version": 1

}

1.2 生成储存规划

       kafka-reassign-partitions.sh --bootstrap-server node1:9092,node2:9092 --topics-to-move-json-file topics-move.json --broker-list "0,1" --generate

去掉id为2的broker

执行后会给出退役broker2后数据在剩余服务器储存规划

创建json文件,把给出的规划复制粘贴进去

vim reassignment.json

{"version":1,"partitions":[{"topic":"cities","partition":0,"replicas":[1,0],"log_dirs":["any","any"]},{"topic":"cities","partition":1,"replicas":[0,1],"log_dirs":["any","any"]},{"topic":"cities","partition":2,"replicas":[1,0],"log_dirs":["any","any"]},{"topic":"products","partition":0,"replicas":[0,1],"log_dirs":["any","any"]},{"topic":"products","partition":1,"replicas":[1,0],"log_dirs":["any","any"]},{"topic":"products","partition":2,"replicas":[0,1],"log_dirs":["any","any"]}]}

1.3 执行储存规划

kafka-reassign-partitions.sh --bootstrap-server node1:9092,node2:9092 --reassignment-json-file reassignment.json --execute

可以看到提示执行成功;

1.4 验证执行结果

kafka-reassign-partitions.sh --bootstrap-server node1:9092,node2:9092 --reassignment-json-file reassignment.json –verify

查看topic详情

kafka-topics.sh --bootstrap-server node1:9092,node2:9092  --topic cities –describe

kafka-topics.sh --bootstrap-server node1:9092,node2:9092  --topic products --describe

对比broker2退役前

可以看到退役后relplics和isr都剔除了2这个id

2、服役(在Kafka集群中新增一个服务器)

Kafka集群历史数据负载均衡,新增服务器承担一部分历史数据的存储任务,上文我们退役了broker2,下面我们让其重新服役

操作步骤:

2.1创建json文件,写入需要均衡的主题

vim topic-move.json

{  "topics": [ {"topic": "products"},{"topic": "cities"} ],"version": 1}

2.2 生成储存规划

       kafka-reassign-partitions.sh --bootstrap-server node1:9092,node2:9092 --topics-to-move-json-file topics-move.json --broker-list "0,1,2" --generate 增加id为2的broker

执行后会对原集群内服务器的数据在增加了新一台服务器的基础上重新给出存储规划

创建json文件,把给出的规划复制粘贴进去

vim reassignment2.json

{"version":1,"partitions":[{"topic":"cities","partition":0,"replicas":[0,2],"log_dirs":["any","any"]},{"topic":"cities","partition":1,"replicas":[1,0],"log_dirs":["any","any"]},{"topic":"cities","partition":2,"replicas":[2,1],"log_dirs":["any","any"]},{"topic":"products","partition":0,"replicas":[0,1],"log_dirs":["any","any"]},{"topic":"products","partition":1,"replicas":[1,2],"log_dirs":["any","any"]},{"topic":"products","partition":2,"replicas":[2,0],"log_dirs":["any","any"]}]}

2.3 执行储存规划

kafka-reassign-partitions.sh --bootstrap-server node1:9092,node2:9092 --reassignment-json-file reassignment2.json --execute

可以看到提示执行成功;

2.4 验证执行结果

kafka-reassign-partitions.sh --bootstrap-server node1:9092,node2:9092 --reassignment-json-file reassignment2.json --verify

查看topic详情

kafka-topics.sh --bootstrap-server node1:9092,node2:9092  --topic cities –describe

kafka-topics.sh --bootstrap-server node1:9092,node2:9092  --topic products --describe

对比broker2服役前

可以看到退役后relplics和isr都增加了2这个id

相关推荐

  1. kafka学习笔记--节点服役退役

    2024-06-07 18:48:02       46 阅读
  2. Git 剔除已经纳入版本管理文件

    2024-06-07 18:48:02       7 阅读
  3. 39、Flink 窗口剔除器(Evictors)详解

    2024-06-07 18:48:02       7 阅读

最近更新

  1. TCP协议是安全的吗?

    2024-06-07 18:48:02       16 阅读
  2. 阿里云服务器执行yum,一直下载docker-ce-stable失败

    2024-06-07 18:48:02       16 阅读
  3. 【Python教程】压缩PDF文件大小

    2024-06-07 18:48:02       15 阅读
  4. 通过文章id递归查询所有评论(xml)

    2024-06-07 18:48:02       18 阅读

热门阅读

  1. Advantages of high pressure cleaners

    2024-06-07 18:48:02       9 阅读
  2. 六个搜索算法及其python实现

    2024-06-07 18:48:02       7 阅读
  3. 思考、学习、创造、越来越有趣

    2024-06-07 18:48:02       8 阅读
  4. 深入理解Nginx的Round-Robin负载均衡策略

    2024-06-07 18:48:02       10 阅读
  5. 获得抖音商品评论 API 返回值

    2024-06-07 18:48:02       6 阅读
  6. 在vue项目中实现单点登录

    2024-06-07 18:48:02       7 阅读
  7. Ant-Design-Vue动态表头并填充数据

    2024-06-07 18:48:02       10 阅读
  8. 【QEMU中文手册】2.2 调用方式(持续更新中)

    2024-06-07 18:48:02       6 阅读
  9. LeetCode-day04-2938. 区分黑球与白球

    2024-06-07 18:48:02       8 阅读
  10. Python怎么hook:深入探索Python的Hook机制

    2024-06-07 18:48:02       9 阅读
  11. c#控件笔记

    2024-06-07 18:48:02       9 阅读