Kubernetes集群V1.26二进制部署
1:环境介绍
- 本环境是基于内网环境无互联网情况下的Kubernetes集群的部署,Runtime使用containerd。亲测有效,有基础可以修改我的配置文件,不懂请不要更改,无脑负责粘贴。
容器编排:Kubernetes允许您在一个集群中运行和管理容器化的应用程序。它可以自动化容器的部署、扩展和运维。
自动化部署:Kubernetes提供了强大的自动化部署功能,可以根据定义的规则自动部署新的应用程序副本。
服务发现与负载均衡:Kubernetes可以自动将网络流量路由到应用程序的正确部分,以及提供负载均衡功能,确保应用程序的高可用性和稳定性。
自我修复:Kubernetes可以自动检测和替换失败的容器实例,确保应用程序持续可用。
水平扩展:Kubernetes允许您根据应用程序的负载自动扩展应用程序副本的数量,以应对流量的变化。
存储编排:Kubernetes提供了灵活的存储解决方案,包括将存储卷挂载到容器内以及动态分配存储。
自定义调度策略:Kubernetes允许您定义自己的调度策略,以满足特定的业务需求。
可扩展性:Kubernetes是一个高度可扩展的平台,可以在各种基础设施上运行,包括公有云、私有云和裸机环境
2:基础架构介绍
选取3台Centos7.9系统作为Master节点配置VIP实现高可用。选取2台Centos7.9系统作为Node节点。共计五台机器作为我们实验的Kubernetes集群
名称 | IP地址 | 系统 | 组件名称 |
---|---|---|---|
K8s-master01 | 10.251.251.71 | Centos7.9 | Kube-ApiServer、Kube-Controller-Manager、Kube-Scheduler Etcd、Kubelet、Kube-Proxy、Nfs-Client、haproxy、Keepalived、Nginx |
K8s-master02 | 10.251.251.72 | Centos7.9 | Kube-ApiServer、Kube-Controller-Manager、Kube-Scheduler Etcd、Kubelet、Kube-Proxy、Nfs-Client、haproxy、Keepalived、Nginx |
K8s-master03 | 10.251.251.73 | Centos7.9 | Kube-ApiServer、Kube-Controller-Manager、Kube-Scheduler Etcd、Kubelet、Kube-Proxy、Nfs-Client、haproxy、Keepalived、Nginx |
K8s-node01 | 10.251.251.74 | Centos7.9 | Kuberlet、Kube-Proxy、Nfs-Client、Nginx |
K8s-node02 | 10.251.251.75 | Centos7.9 | Kuberlet、Kube-Proxy、Nfs-Client、Nginx |
K8s-master-lb | 10.251.251.77 | VIP | VIP |
2.1:资源清单
2.2:软件版本
软件 | 版本 |
---|---|
kernel | v4.19.12 |
CentOS 7 | v7.9 |
kube-apiserver、kube-controller-manager、kube-scheduler、kubelet、kube-proxy | v1.25.4 |
etcd | v3.5.6 |
containerd | v1.6.10 |
docker | v20.10.21 |
cfssl | v1.6.3 |
cni | v1.1.1 |
crictl | v1.26.15 |
haproxy | v1.8.27 |
keepalived | v2.1.5 |
calico网络插件 | v3.26.4 |
coredns解析 | v1.9.4 |
merics资源展示 | v0.5 |
dashboard页面 | v2.7.0 |
containerd-runtime资源池 | v1.6.8 |
2.3:下载地址
组件名称 | 下载地址 |
---|---|
Kube-Client下载地址 | https://cdn.dl.k8s.io/release/v1.26.15/kubernetes-server-linux-amd64.tar.gz |
Docker下载地址 | https://download.docker.com/linux/static/stable/x86_64/ |
Etcd下载地址 | https://ghproxy.com/https://github.com/etcd-io/etcd/releases/download/v3.5.6/etcd-v3.5.6-linux-amd64.tar.gz |
Centos7内核升级 | http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm |
Cni下载地址 | https://objects.githubusercontent.com/github-production-release-asset-2e65be/84575398/34412816-cbca-47a1-a428-9e738f2451d8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240321%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240321T142346Z&X-Amz-Expires=300&X-Amz-Signature=854d653c47a905eb1af871e496054aa7fb2c90bcc1fea914ca39c0cfd64118c3&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=84575398&response-content-disposition=attachment%3B%20filename%3Dcni-plugins-linux-amd64-v1.1.1.tgz&response-content-type=application%2Foctet-stream |
calico下载地址 | https://raw.githubusercontent.com/projectcalico/calico/v3.26.4/manifests/calico.yaml |
coredns下载地址 | https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed |
merics下载地址 | https://github.com/kubernetes-sigs/metrics-server/releases/tag/v0.5.0 |
dashboard下载地址 | https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml |
containerd下载地址 | https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz |
nginx下载地址 | http://nginx.org/download/nginx-1.22.1.tar.gz |
2.4:架构图
说明!!!!!!!!!!!!说明!!!!!!!!!!!!
- ALL代表5台机器都要做同样的操作。NO代表只在master01操作
- ALL代表5台机器都要做同样的操作。NO代表只在master01操作。
- ALL代表5台机器都要做同样的操作。NO代表只在master01操作。
3:前期准备
3.1:关闭防火墙及SElinux(ALL)
[root@k8s-master01 yaml]# systemctl stop firewalld && systemctl disable firewalld #关闭防火墙,开机禁用防火墙
[root@k8s-master01 yaml]# sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config #关闭SElinux
[root@k8s-master01 yaml]# setenforce 0 #查看是否关闭
3.2:关闭swap内存(ALL)
[root@k8s-master01 yaml]# sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab #添加#号注释swap内容
3.3:设置主机名
hostnamectl set-hostname k8s-master01
hostnamectl set-hostname k8s-master02
hostnamectl set-hostname k8s-master03
hostnamectl set-hostname k8s-node01
hostnamectl set-hostname k8s-node02
3.4:网络配置
1.网卡配置信息: #内网环境不需要配置DNS
[root@k8s-master01 yaml]# cat /etc/sysconfig/network-scripts/ifcfg-ens192
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens192
UUID=e7661c85-02f2-46ef-8043-75cd5b3500cd
DEVICE=ens192
ONBOOT=yes
IPADDR=10.251.251.71 #其他4台依次配置IP
NETMASK=255.255.255.0 #掩码
GATEWAY=10.251.251.254 #网关
2.hosts解析:(ALL) #5台机器配置
[root@k8s-master01 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.251.251.71 k8s-master01
10.251.251.72 k8s-master02
10.251.251.73 k8s-master03
10.251.251.77 k8s-master-lb
10.251.251.74 k8s-node01
10.251.251.75 k8s-node02
3.5:工具下载
各位最好将本地的centos7镜像挂载到系统上,做成本地yum源,方便下载一些命令,工具,例如[vim,wget,curl,telnet,netstat]等。如果没有也没事,咱们在自己的笔记本准备一台电脑【前期可以没有,后期必须得有,拉取镜像用】。
1>:有本地源,下载这些工具(ALL)
"vim" "net-tools" "telnet" "gcc" "c++" "curl" "wget" "lrzsz"
2>:没有本地源的话,在自己笔记本的centos7系统配置一些参数使用yum下载
1.编辑yum.conf配置文件
[root@halo ~]# vim /etc/yum.conf
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0 #将0更改为1---0的意思yum下载不保留安装包,1的意思yum下载保留安装包
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=5
bugtracker_url=http://bugs.centos.org/set_project.php?project_id=23&ref=http://bugs.centos.org/bug_report_page.php?category=yum
distroverpkg=centos-release
2.下载命令格式
[root@halo ~]# yum install vim --downloadonly --downloaddir=/root/k8s/package -y
下载完成后会将vim.rpm包存放到/root/k8s/package目录下,将vim.rpm安装包放到k8s集群,使用以下命令安装,这样就不用考虑依赖的问题了。
[root@halo ~]# yum install vim-7.1-1.el7.x86_64.rpm -y
3.6:升级内核(ALL)
1.centos7当前默认内核:
[root@halo ~]# grubby --default-kernel
/boot/vmlinuz-3.10.0-1127.el7.x86_64
2.升级内核到4.19
[root@halo ~]# wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm
[root@halo ~]# yum install -y kernel-ml*
[root@halo ~]# grub2-set-default 0 #设置启动内核版本
[root@halo ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
[root@halo ~]# reboot #重启系统
[root@halo ~]# uname -a #查看当前系统内核
Linux halo 4.19.12-1.el7.elrepo.x86_64 #1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux
3.7:安装ipvsadm(ALL)
根据3.5的逻辑去下载
1.下载ipvsadm
[root@halo ~]# yum install ipvsadm ipset sysstat conntrack libseccomp -y
2.创建启动服务
[root@k8s-master01 ~]# cat >> /etc/modules-load.d/ipvs.conf <<EOF
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF
[root@k8s-master01 ~]# systemctl restart systemd-modules-load.service
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0
ip_vs 180224 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack 176128 1 ip_vs
nf_defrag_ipv6 24576 2 nf_conntrack,ip_vs
nf_defrag_ipv4 16384 1 nf_conntrack
libcrc32c 16384 3 nf_conntrack,xfs,ip_vs
3.8:修改内核参数(ALL)
[root@k8s-master01 ~]# cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.all.forwarding = 1
EOF
[root@k8s-master01 ~]# sysctl --system
4:互联网机器配置
在你的笔记本电脑必须有一台能连接互联网的机器,做镜像的拉取,docker pull,ctr pull。
4.1:安装docker
#下载地址:下载所需要的docker版本 我的版本是:25.0.0
https://download.docker.com/linux/static/stable/x86_64/docker-25.0.0.tgz
#将下载的tgz包放到/root/k8s/package/目录下,解压,创建系统服务
[root@halo ~]# tar -xvf docker-25.0.0.tgz
[root@halo ~]# rm -rf docker-19.03.9.tgz
[root@halo ~]# cp /root/k8s/package/docker/* /usr/bin/
[root@halo ~]# cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF
[root@halo ~]# systemctl daemon-reload
[root@halo ~]# systemctl restart docker && systemctl enable docker
[root@halo ~]# vim /etc/docker/daemon.json
{
"registry-mirrors": ["https://你的加速地址.mirror.aliyuncs.com"],
"insecure-registries": ["公司内部harbor地址"]
}
[root@halo ~]# systemctl daemon-reload && systemctl restart docker
4.2:安装containerd
1.下载安装包
[root@halo ~]# wget https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz
[root@halo ~]# wget https://github.com/containerd/containerd/releases/download/v1.6.8/cri-containerd-cni-1.6.8-linux-amd64.tar.gz
2.创建cni所需目录
[root@halo ~]# mkdir -p /etc/cni/net.d /opt/cni/bin
3.解压cni包
[root@halo ~]# tar xf cni-plugins-linux-amd64-v*.tgz -C /opt/cni/bin/
4.解压
[root@halo ~]# tar -xzf cri-containerd-cni-*-linux-amd64.tar.gz -C /
5.创建系统服务
[root@halo ~]# cat > /etc/systemd/system/containerd.service <<EOF
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target
[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target
EOF
6.配置containerd所需模块
[root@halo ~]# cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
7.加载配置好的模块
[root@halo ~]# systemctl restart systemd-modules-load.service
8.配置containerd所需的内核
[root@halo ~]# cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
[root@halo ~]# sysctl --system
9.生成配置文件
[root@halo ~]# mkdir -p /etc/containerd
[root@halo ~]# containerd config default | tee /etc/containerd/config.toml
[root@halo ~]# sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ true#g" /etc/containerd/config.toml
[root@halo ~]# cat /etc/containerd/config.toml | grep SystemdCgroup
#可以更改你的仓库地址
[root@halo ~]# sed -i "s#registry.k8s.io#registry.cn-hangzhou.aliyuncs.com/chenby#g" /etc/containerd/config.toml
[root@halo ~]# cat /etc/containerd/config.toml | grep sandbox_image #这是k8s所需的基础镜像
#对照上面的拉取地址,下面操作是跟换镜像源,类似docker的镜像加速器
[root@halo ~]# sed -i "s#config_path\ \=\ \"\"#config_path\ \=\ \"/etc/containerd/certs.d\"#g" /etc/containerd/config.toml
[root@halo ~]# cat /etc/containerd/config.toml | grep certs.d
[root@halo ~]# mkdir /etc/containerd/certs.d/docker.io -pv
[root@halo ~]# cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
server = "https://docker.io"
[host."https://hub-mirror.c.163.com"]
capabilities = ["pull", "resolve"]
EOF
10.配置circtl客户端(k8s专用命令)
[root@halo ~]# wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.24.2/crictl-v1.24.2-linux-amd64.tar.gz
[root@halo ~]# tar xf crictl-v*-linux-amd64.tar.gz -C /usr/bin/
[root@halo ~]# cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF
11.重启containerd,检查配置
[root@halo ~]# systemctl daemon-reload
[root@halo ~]# systemctl restart containerd && systemctl enable --now containerd
[root@halo ~]# crictl info
},
"golang": "go1.17.13",
"lastCNILoadStatus": "OK",
"lastCNILoadStatus.default": "OK"
}
!!!!!以下操作将是正题,一定要谨慎操作。可以将第3骤做完之后拍个快照。!!!!!
[root@halo ~]# 互联网机器系统名称
[root@k8s-master01 ~]# k8s集群机器系统名称
5:Kubernetes和Etcd安装
5.1:解压安装
######所有的安装包都在/root/package/目录下
[root@k8s-master01 package]# ls
docker ipvs kernel kube lvs node_export system
1.k8s安装
[root@k8s-master01 ~]# tar -xf /root/package/kube/kubernetes-server-linux-amd64.tar.gz --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}
2.etcd安装
[root@k8s-master01 ~]# tar -xf /root/package/kube/kubernetes-server-linux-amd64.tar.gz && mv etcd-*/etcd /usr/local/bin/ && mv etcd-*/etcdctl /usr/local/bin/
3.查看/usr/local/bin/目录
[root@k8s-master01 package]# ls /usr/local/bin/
cfssl containerd containerd-shim-runc-v1 containerd-stress
critest ctr etcdctl kube-controller-manager
kubelet kube-scheduler cfssljson containerd-shim
containerd-shim-runc-v2 crictl ctd-decoder etcd
kube-apiserver kubectl kube-proxy
5.1.1:查看版本
[root@k8s-master01 package]# kubelet --version
Kubernetes v1.26.15
[root@k8s-master01 package]# etcdctl version
etcdctl version: 3.5.6
API version: 3.5
5.1.2:分发组件
[root@k8s-master01 package]# Master='k8s-master02 k8s-master03'
[root@k8s-master01 package]# Work='k8s-node01 k8s-node02'
[root@k8s-master01 package]# for NODE in $Master; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done
[root@k8s-master01 package]# for NODE in $Work; do scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done
[root@k8s-master01 package]# mkdir -p /opt/cni/bin
5.2:证书创建
[root@k8s-master01 package]# mkdir -p /root/pki && cd pki