Linux容器篇-kubernetes监控和日志管理


一、kubernetes基本概念

  • Pod:k8s的最小部署单位,一组容器的集合

  • Development:最常见的控制器,用于更高级别部署和管理Pod

  • Service:为一组Pod提供负载均衡,对外提供统一的访问入口

  • Label: 标签,附加到某个资源上,用于关联对象,查询和筛选

  • NameSpace:命名空间,将对象逻辑上隔离,也有利于权限控制

​ kubernetes将资源对象逻辑上隔离,从而形成了多个虚拟集群,更方便管理。

kubectl get namespace
NAME                   STATUS   AGE
calico-apiserver       Active   12d
calico-system          Active   12d
default                Active   12d
kube-node-lease        Active   12d
kube-public            Active   12d
kube-system            Active   12d
kubernetes-dashboard   Active   12d
tigera-operator        Active   12d

​ default : 默认命名空间
​ kube-system: k8s系统方面的命名空间
​ kube-public : 公开的命名空间,谁都可以访问
​ kube-node-least : k8s内部命名空间

​ 两种方法指定资源所属的命名空间:

命令行加-n选项;yaml资源元数据里指定namespace字段;

二、集群资源监控

查看API资源对象及缩写

[root@k8s-master ~]# kubectl api-resources 
NAME                              SHORTNAMES                                      APIVERSION                             NAMESPACED   KIND
bindings                                                                          v1                                     true         Binding
componentstatuses                 cs                                              v1                                     false        ComponentStatus
configmaps                        cm                                              v1                                     true         ConfigMap
endpoints                         ep                                              v1                                     true         Endpoints
events                            ev                                              v1                                     true         Event
limitranges                       limits                                          v1                                     true         LimitRange
namespaces                        ns                                              v1                                     false        Namespace
nodes                             no                                              v1                                     false        Node
persistentvolumeclaims            pvc                                             v1                                     true         PersistentVolumeClaim
persistentvolumes                 pv                                              v1                                     false        PersistentVolume
pods                              po                                              v1                                     true         Pod
......

资源对象的简写可以提高我们在使用k8s过程中的输入效率,但是对初学者来说不容易理解,可以采用配置kubectl自动补全的方式

查看master组件状态

这条命令会显示master节点的主要组件:etcd,controller-manager,scheduler的工作状态

[root@k8s-master ~]#kubectl get cs
[root@k8s-master ~]# kubectl get componentstatuses 
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
etcd-0               Healthy   ok        
controller-manager   Healthy   ok        
scheduler            Healthy   ok

查看集群节点状态

这条命令可以查到当前集群master节点和node节点的工作状态

kubectl get nodes
[root@k8s-master ~]# kubectl get nodes
NAME         STATUS   ROLES           AGE   VERSION
k8s-master   Ready    control-plane   13d   v1.28.0
k8s-node1    Ready    <none>          13d   v1.28.0
k8s-node2    Ready    <none>          13d   v1.28.0

查看资源详情

kubectl describe <资源类型> <资源名称>

查看一个deployment的详细信息

[root@k8s-master ~]# kubectl get deployment -n test 
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
nginx        3/3     3            3           5d17h
nginx-test   1/1     1            1           11d
[root@k8s-master ~]# kubectl describe deployment nginx -n test 
Name:                   nginx
Namespace:              test
CreationTimestamp:      Wed, 10 Jul 2024 17:04:14 +0800
Labels:                 app=nginx
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=nginx
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=nginx
  Containers:
   nginx:
    Image:        nginx:1.16
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   nginx-85bfcd86d5 (3/3 replicas created)
Events:          <none>

查看一个pod的详细信息

[root@k8s-master ~]# kubectl get pods -n test 
NAME                         READY   STATUS    RESTARTS        AGE
nginx-85bfcd86d5-5dl44       1/1     Running   3 (4d12h ago)   5d17h
nginx-85bfcd86d5-7kjvt       1/1     Running   3 (4d12h ago)   5d17h
nginx-85bfcd86d5-fltkh       1/1     Running   3 (4d12h ago)   5d17h
nginx-test-d87b84fb7-lcvg7   1/1     Running   6 (4d12h ago)   11d
[root@k8s-master ~]# kubectl describe pods nginx-test-d87b84fb7-lcvg7 -n test 
Name:             nginx-test-d87b84fb7-lcvg7
Namespace:        test
Priority:         0
Service Account:  default
Node:             k8s-node1/192.168.3.11
Start Time:       Thu, 04 Jul 2024 10:45:20 +0800
Labels:           k8s-app=nginx-test
                  pod-template-hash=d87b84fb7
Annotations:      cni.projectcalico.org/containerID: 5d84c8a34dd067d8b77205c1abfd7eef58ddf67571c39e807e862457bc400c5f
                  cni.projectcalico.org/podIP: 10.244.36.97/32
                  cni.projectcalico.org/podIPs: 10.244.36.97/32
Status:           Running
IP:               10.244.36.97
IPs:
  IP:           10.244.36.97
Controlled By:  ReplicaSet/nginx-test-d87b84fb7
Containers:
  nginx-test:
    Container ID:   docker://1a0bdae7e63c45454024fd0e66fdbc615d832a3bd82ba5a1f27315c75ce63ec7
    Image:          nginx:1.23
    Image ID:       docker-pullable://nginx@sha256:f5747a42e3adcb3168049d63278d7251d91185bb5111d2563d58729a5c9179b0
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 16 Jul 2024 10:04:31 +0800
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 11 Jul 2024 19:23:30 +0800
      Finished:     Thu, 11 Jul 2024 22:18:17 +0800
    Ready:          True
    Restart Count:  6
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9nsbn (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-9nsbn:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age                From             Message
  ----     ------          ----               ----             -------
  Warning  NodeNotReady    4d13h              node-controller  Node is not ready
  Normal   SandboxChanged  33m (x2 over 33m)  kubelet          Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled          32m                kubelet          Container image "nginx:1.23" already present on machine
  Normal   Created         32m                kubelet          Created container nginx-test
  Normal   Started         32m                kubelet          Started container nginx-test

查看service的详细信息

[root@k8s-master ~]# kubectl get service -n test 
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
nginx-test   ClusterIP   10.96.255.119   <none>        80/TCP    11d       
[root@k8s-master ~]# kubectl describe service nginx-test -n test 
Name:              nginx-test
Namespace:         test
Labels:            k8s-app=nginx-test
Annotations:       <none>
Selector:          k8s-app=nginx-test
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.96.255.119
IPs:               10.96.255.119
Port:              tcp-80-80-rqfqj  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.36.97:80
Session Affinity:  None
Events:            <none>

查看资源信息

kubectl get <资源类型> <资源名称>
-o wide  显示详情
-o yaml  将配置输出为yaml格式
-n namespace  查看对应命名空间

资源监控服务

资源监控服务由metric-server服务提供数据,而这个服务默认没有安装,还需要手动部署一下。

metric server 是一个集群范围的资源使用情况的数据聚合器,作为一个应用部署在集群中。metric server从每个节点上的kubelet API 收集指标,通过kubernetes聚合器注册在Master APIserver中,为集群提供Node、 Pods资源利用率指标。

metric组件的yaml文件可以在github拉取,国内网络不允许,需要合适的加速器

查看node资源消耗

kubectl top node <node name>

[root@k8s-master ~]# kubectl top nodes
NAME         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s-master   292m         14%    1154Mi          67%       
k8s-node1    169m         8%     706Mi           41%       
k8s-node2    220m         11%    883Mi           51%  

查看Pod资源消耗

kubectl top pod <pod name>

[root@k8s-master ~]# kubectl top pod nginx-test-d87b84fb7-lcvg7 -n test 
NAME                         CPU(cores)   MEMORY(bytes)   
nginx-test-d87b84fb7-lcvg7   0m           12Mi  

k8s日志

kubelet日志:

到对应的宿主机查看,使用

journalctl -u kubelet

pod组件日志:

kubectl logs <资源名称> -n <命名空间>

查看pod日志

[root@k8s-master ~]# kubectl logs nginx-test-d87b84fb7-lcvg7 -n test 
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/07/16 02:04:32 [notice] 1#1: using the "epoll" event method
2024/07/16 02:04:32 [notice] 1#1: nginx/1.23.4
2024/07/16 02:04:32 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 
2024/07/16 02:04:32 [notice] 1#1: OS: Linux 3.10.0-1160.71.1.el7.x86_64
2024/07/16 02:04:32 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/07/16 02:04:32 [notice] 1#1: start worker processes
2024/07/16 02:04:32 [notice] 1#1: start worker process 29
2024/07/16 02:04:32 [notice] 1#1: start worker process 30

系统日志:

/var/log/messages

容器的标准输出日志:

kubectl logs <Pod名称>
kubectl logs -f <Pod名称> #-f可以动态显示日志

kubectl exec -it <Pod名称> -- bash  #进入pod内部查看标准输出日志

相关推荐

  1. Linux容器-kubernetes监控日志管理

    2024-07-16 12:00:05       21 阅读
  2. Kubernetes教程:管理扩展容器化应用

    2024-07-16 12:00:05       30 阅读
  3. 生成日志系统监控

    2024-07-16 12:00:05       23 阅读

最近更新

  1. docker php8.1+nginx base 镜像 dockerfile 配置

    2024-07-16 12:00:05       52 阅读
  2. Could not load dynamic library ‘cudart64_100.dll‘

    2024-07-16 12:00:05       54 阅读
  3. 在Django里面运行非项目文件

    2024-07-16 12:00:05       45 阅读
  4. Python语言-面向对象

    2024-07-16 12:00:05       55 阅读

热门阅读

  1. PG参数深入了解

    2024-07-16 12:00:05       19 阅读
  2. AI安全入门-人工智能数据与模型安全

    2024-07-16 12:00:05       20 阅读
  3. 容器安全最佳实践和工具

    2024-07-16 12:00:05       24 阅读
  4. 小程序图片设置大小

    2024-07-16 12:00:05       20 阅读
  5. 实战:Linux下静默安装DM达梦数据库

    2024-07-16 12:00:05       24 阅读
  6. Redis中lua脚本和pipeline的区别

    2024-07-16 12:00:05       21 阅读