Prometheus监控k8s集群告警规则

目录

节点磁盘使用率

节点状态异常

节点内存使用率

 节点CPU使用率

容器内存使用率

容器CPU使用率

pod频繁重启

pod状态异常

pod启动超时失败

deploymentPOD可用率

节点磁盘使用率

100 - (node_filesystem_avail{mountpoint="/"} * 100) /
node_filesystem_size{mountpoint="/"}

node_filesystem_size{device="overlay",instance=~'$node'}-0
node_filesystem_avail{device="overlay",instance=~'$node'}-0
1-(node_filesystem_free{instance=~'$node',device="overlay"} / node_filesystem_size{instance=~'$node',device="overlay"})

节点状态异常

kube_node_status_condition{condition="Ready",status="true"}

节点内存使用率

100 - node_memory_MemAvailable_bytes{} / node_memory_MemTotal_bytes{} * 100

或者：

 节点CPU使用率

100 - avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100

或者

容器内存使用率

(sum(container_memory_working_set_bytes{id!="/",namespace = "hdd-yunxiaobao-prod"}) BY (instance,name,container,pod_name,namespace) / 
sum(container_spec_memory_limit_bytes{id!="/",namespace = "hdd-yunxiaobao-prod"} > 0) BY (instance,name,container,pod_name,namespace) * 100)

容器CPU使用率

100 * (sum(rate(container_cpu_usage_seconds_total{id!="/",namespace = "hdd-yunxiaobao-prod"}[1m])) by (pod_name,namespace,container) / 
sum(label_replace(kube_pod_container_resource_limits_cpu_cores{namespace = "hdd-yunxiaobao-prod"},"pod_name","$1","pod","(.*)")) by (pod_name,namespace,container)) and sum(label_replace(kube_pod_container_resource_limits_cpu_cores{namespace = "hdd-yunxiaobao-prod"},"pod_name","$1","pod","(.)")) by (pod_name,namespace,container) != 0

pod频繁重启

max(container_fs_usage_bytes{pod!="",namespace!="arms-prom",namespace!="monitoring"}) by 
(pod_name,namespace,device)/max(container_fs_limit_bytes{pod!=""}) by 
(pod_name,namespace,device) * 100

pod状态异常

kube_pod_status_phase{phase=~"Pending|Unknown|Failed"} > 0

pod启动超时失败

kube_pod_container_status_waiting_reason{} == 1

deploymentPOD可用率

kube_deployment_status_replicas_available{} / kube_deployment_status_replicas{} * 100 and 
kube_deployment_status_replicas_unavailable and kube_deployment_status_replicas{} > 9

 希望能够帮助到大家！！！