[Kubernetes] etcd 单机和集群部署

在这里插入图片描述

文章目录

1.etcd基本概念

etcd是一个高可用的分布式键值存储系统，是CoreOS（现在隶属于Red Hat）公司开发的一个开源项目。它提供了一个简单的接口来存储和检索键值对数据，并使用Raft协议实现了分布式一致性。etcd广泛应用于Docker、Kubernetes等分布式系统中，用于存储配置信息、服务发现、领导者选举等方面。

2.etcd的基本知识

Etcd的定义：Etcd是一个分布式的、高可用的键值存储系统，具有快速响应、支持高并发等特点。
Etcd的优点：高可用、数据一致性、快速响应、支持分布式事务。
Etcd的应用场景：存储集群中各种配置信息、元数据以及服务发现等。
Etcd的数据结构：类似于键值对的数据结构，支持多种数据类型，如字符串、整数、布尔、数组等。
Etcd的API：提供丰富的API接口，通过HTTP协议进行访问和操作。
Etcd的数据同步：Etcd使用Raft协议来保证数据的一致性和高可用性，在多个节点之间同步数据。
Etcd的集群部署：Etcd可以部署为单节点或多节点集群，多节点集群可以提高系统的可用性和性能。
Etcd的安全性：Etcd提供了诸如SSL/TLS加密、授权、访问控制等安全机制，保障了数据的安全性。
Etcd的监控：Etcd提供了各种监控指标和工具，方便管理员对集群进行监控和管理。
Etcd的常用工具：包括etcdctl命令行工具、etcd browser浏览器、etcd dashboard仪表板、etcdwatch等。

3.etcd优势

etcd是一种分布式键值存储系统，具有以下特点和优势：

强一致性：etcd使用Raft协议实现强一致性，确保每个节点上的数据始终保持一致。
高可用性：etcd可以通过多副本和自动故障转移机制来实现高可用性，即使某个节点故障，也可以保证服务不中断。
高性能：etcd采用预写日志（WAL）技术，可以快速写入数据，同时还支持快速读取和查询。
分布式：etcd可以在多台服务器上运行，数据可以在多个节点之间分布，提高了系统的可伸缩性和性能。
安全性：etcd支持TLS加密和认证，确保数据传输的安全性。
简单易用：etcd提供简单易用的HTTP/JSON API，可以方便地进行数据读写和查询。
开源免费：etcd是一个完全开源的项目，具有Apache 2.0许可证。

4.etcd单机部署

4.1 linux部署

下载地址：https://github.com/etcd-io/etcd/releases/download/v3.5.13/etcd-v3.5.13-linux-amd64.tar.gz

上传服务器:

tar xf etcd-v3.5.13-linux-amd64.tar.gz
mkdir -p /opt/etcd/bin
cp -a etcd-v3.5.13-linux-amd64/etcd* /opt/etcd/bin/


cd /opt/etcd/
mkdir -p data conf

创建配置文件

cat << 'EOF' > /opt/etcd/conf/etcd.conf
ETCD_NAME="apisix"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379,http://127.0.0.1:2379"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://0.0.0.0:2380,http://127.0.0.1:2380"
ETCD_ENABLE_V2="true"
EOF

注册成一个服务

cat << 'EOF' >  /usr/lib/systemd/system/etcd.service
[Unit]
Description=etcd service
Documentation=https://github.com/etcd-io/etcd
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
User=root
Type=notify
EnvironmentFile=/opt/etcd/conf/etcd.conf
WorkingDirectory=/opt/etcd
ExecStart=/opt/etcd/bin/etcd
Restart=always
RestartSec=10s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

启动服务

systemctl daemon-reload
systemctl enable --now etcd

netstat -ntplu | egrep etcd
tcp6       0      0 :::2379                 :::*                    LISTEN      1228/etcd
tcp6       0      0 :::2380                 :::*                    LISTEN      1228/etcd

设置环境变量

echo 'export PATH=$PATH:/opt/etcd/bin' > /etc/profile.d/etcd.sh
source /etc/profile.d/etcd.sh

查看状态

etcdctl endpoint status --write-out=table
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 17ac6c9b1b6e7f0c |  3.5.13 |   20 kB |      true |      false |         2 |          4 |                  4 |        |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

etcdctl endpoint health --write-out=table
+----------------+--------+------------+-------+
|    ENDPOINT    | HEALTH |    TOOK    | ERROR |
+----------------+--------+------------+-------+
| 127.0.0.1:2379 |   true | 3.571276ms |       |
+----------------+--------+------------+-------+

etcdctl member list --write-out=table
+------------------+---------+--------+-------------------------------------------+-------------------------------------------+------------+
|        ID        | STATUS  |  NAME  |                PEER ADDRS                 |               CLIENT ADDRS                | IS LEARNER |
+------------------+---------+--------+-------------------------------------------+-------------------------------------------+------------+
| 17ac6c9b1b6e7f0c | started | apisix | http://0.0.0.0:2380,http://127.0.0.1:2380 | http://0.0.0.0:2379,http://127.0.0.1:2379 |      false |
+------------------+---------+--------+-------------------------------------------+-------------------------------------------+------------+

不变成服务, 直接启动etcd:

cd /opt/etcd/
./etcd

4.2 windows部署

安装地址：https://github.com/etcd-io/etcd/releases/tag/v3.5.14

在这里插入图片描述

4.3 docker安装etcd

mkdir -p /opt/soft/etcd/data

# 拉取etcd镜像
docker pull quay.io/coreos/etcd:v3.5.13

docker run -d \
  -p 2379:2379 \
  -p 2380:2380 \
  --restart=always \
  --mount type=bind,source=/opt/soft/etcd/data,destination=/etcd-data \
  --name etcd \
  quay.io/coreos/etcd:v3.5.13 \
  /usr/local/bin/etcd \
  --name s1 \
  --data-dir /etcd-data \
  --listen-client-urls http://0.0.0.0:2379 \
  --advertise-client-urls http://0.0.0.0:2379 \
  --listen-peer-urls http://0.0.0.0:2380 \
  --initial-advertise-peer-urls http://0.0.0.0:2380 \
  --initial-cluster s1=http://0.0.0.0:2380 \
  --initial-cluster-token tkn \
  --initial-cluster-state new \
  --log-level info \
  --logger zap \
  --log-outputs stderr

# 查看Etcd服务器版本
docker exec etcd /usr/local/bin/etcd --version

# 查看Etcd客户端版本
docker exec etcd /usr/local/bin/etcdctl version
docker exec etcd /usr/local/bin/etcdutl version

# 查看Etcd健康状况
docker exec etcd /usr/local/bin/etcdctl endpoint health

# Etcd添加数据
docker exec etcd /usr/local/bin/etcdctl put foo bar

# Etcd查看数据
docker exec etcd /usr/local/bin/etcdctl get foo

# Etcd查看所有数据
docker exec etcd /usr/local/bin/etcdctl get --prefix ""

# Etcd删除数据
docker exec etcd /usr/local/bin/etcdctl del foo

[root@node1192 soft]# docker exec etcd /usr/local/bin/etcd --version
etcd Version: 3.5.13
Git SHA: c9063a0dc
Go Version: go1.21.8
Go OS/Arch: linux/amd64

[root@node1192 soft]# docker exec etcd /usr/local/bin/etcdctl version
etcdctl version: 3.5.13
API version: 3.5

[root@node1192 soft]# docker exec etcd /usr/local/bin/etcdutl version
etcdutl version: 3.5.13
API version: 3.5

[root@node1192 soft]# docker exec etcd /usr/local/bin/etcdctl endpoint health
127.0.0.1:2379 is healthy: successfully committed proposal: took = 5.561765ms

[root@node1192 soft]# docker exec etcd /usr/local/bin/etcdctl put foo bar
OK

[root@node1192 soft]# docker exec etcd /usr/local/bin/etcdctl get foo
foo
bar

[root@node1192 soft]# docker exec etcd /usr/local/bin/etcdctl get --prefix ""
foo
bar

[root@node1192 soft]# docker exec etcd /usr/local/bin/etcdctl del foo
1

5.etcd集群部署

#每台主机分别执行
hostnamectl set-hostname etcd-node01
hostnamectl set-hostname etcd-node02
hostnamectl set-hostname etcd-node03

#每台主机执行
cat << 'EOF' > /etc/hosts
192.168.66.101 etcd-node01
192.168.66.102 etcd-node02
192.168.66.103 etcd-node03
EOF

yum install ntpdate -y
ntpdate -s ntp1.aliyun.com

tar xf etcd-v3.5.13-linux-amd64.tar.gz
mkdir -pv /opt/etcd/{bin,data,conf}
cp -a etcd-v3.5.13-linux-amd64/etcd* /opt/etcd/bin/

编写配置文件

cat << 'EOF' > /opt/etcd/conf/etcd.conf
ETCD_NAME=etcd-node01
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.66.101:2380,http://127.0.0.1:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.66.101:2379,http://127.0.0.1:2379"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.66.101:2380"
ETCD_INITIAL_CLUSTER="etcd-node01=http://192.168.66.101:2380,etcd-node02=http://192.168.66.102:2380,etcd-node03=http://192.168.66.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.66.101:2379"
EOF

cat << 'EOF' > /opt/etcd/conf/etcd.conf
ETCD_NAME=etcd-node02
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.66.102:2380,http://127.0.0.1:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.66.102:2379,http://127.0.0.1:2379"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.66.102:2380"
ETCD_INITIAL_CLUSTER="etcd-node01=http://192.168.66.101:2380,etcd-node02=http://192.168.66.102:2380,etcd-node03=http://192.168.66.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.66.102:2379"
EOF

cat << 'EOF' > /opt/etcd/conf/etcd.conf
ETCD_NAME=etcd-node03
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.66.103:2380,http://127.0.0.1:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.66.103:2379,http://127.0.0.1:2379"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.66.103:2380"
ETCD_INITIAL_CLUSTER="etcd-node01=http://192.168.66.101:2380,etcd-node02=http://192.168.66.102:2380,etcd-node03=http://192.168.66.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.66.103:2379"
EOF

注册服务

cat << 'EOF' >  /usr/lib/systemd/system/etcd.service
[Unit]
Description=etcd service
Documentation=https://github.com/etcd-io/etcd
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
User=root
Type=notify
EnvironmentFile=/opt/etcd/conf/etcd.conf
WorkingDirectory=/opt/etcd
ExecStart=/opt/etcd/bin/etcd
Restart=always
RestartSec=10s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

开启服务和环境变量

systemctl enable --now etcd

echo 'PATH=$PATH:/opt/etcd/bin' > /etc/profile.d/etcd.sh
source /etc/profile.d/etcd.sh

查看集群信息

etcdctl --endpoints=192.168.66.101:2379,192.168.66.102:2379,192.168.66.103:2379  endpoint status  --write-out=table
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|       ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.66.101:2379 | 8dc9f79502ff52fe |  3.5.13 |   20 kB |     false |      false |         2 |         13 |                 13 |        |
| 192.168.66.102:2379 | 8fdf9dd965ff87d5 |  3.5.13 |   20 kB |      true |      false |         2 |         13 |                 13 |        |
| 192.168.66.103:2379 | fc380e2e4f60630e |  3.5.13 |   20 kB |     false |      false |         2 |         13 |                 13 |        |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

测试故障：将上面的 leader 节点关机，然后查看 leader 节点是否会在其他节点上启动。

etcdctl --endpoints=192.168.66.101:2379,192.168.66.102:2379,192.168.66.103:2379  endpoint status  --write-out=table
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|       ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.66.101:2379 | 8dc9f79502ff52fe |  3.5.13 |   20 kB |      true |      false |         3 |         14 |                 14 |        |
| 192.168.66.103:2379 | fc380e2e4f60630e |  3.5.13 |   20 kB |     false |      false |         3 |         14 |                 14 |        |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

leader节点已经漂移到 192.168.66.101 上面了。

数据写入测试

在 etcd-node01 写入数据
root@etcd-node01(192.168.66.101)~>etcdctl put greeting "hello etcd"
OK

在 etcd-node03 读取数据
root@etcd-node03(192.168.66.103)~>etcdctl get greeting
greeting
hello etcd