10 spring-data-redis 中创建的 pipe 和 anon_inode 的 fd 来自于哪里?

前言

本文的问题 主要是 衍生自 spring-boot-acurator 定时检测 redis 集群导致 “IOException: Too many open files“

对于这里 pipe 的使用, 也是很神奇, 因为 貌似没有用过这类 api 

然后 这里调研一下, 然后 追根究底到最终, 是到了 jdk 的 c 代码的调用, 创建的 pipe

然后 大概看了一下 java 的 PipeReader/Writer, 发现 其实没有到 linux 的 pipe, 仅仅是 java language level 实现的一个管道读写

这部分在 java 语言这一层 是无感知的, 只能使用fd

这里来看一下 使用了 spring-data-redis 的项目中的那些 pipe, anon_inode 相关的 FileDescriptor

然后 这部分 FileDescriptor 较难找到 

这里 来探究一下 这部分的 FileDescriptor 创建的地方 

pipe 部分的 FileDescriptor 如下, 合计 102 项 

root@ubuntu:~/docker/HelloWorld# ll /proc/58983/fd | grep pipe
lr-x------ 1 root root 64 Jun 26 04:12 36 -> pipe:[6215597]
l-wx------ 1 root root 64 Jun 26 04:12 37 -> pipe:[6215597]
lr-x------ 1 root root 64 Jun 26 04:12 39 -> pipe:[6215598]
// 省略部分 fd 的信息
lr-x------ 1 root root 64 Jun 26 04:12 201 -> pipe:[6216982]
l-wx------ 1 root root 64 Jun 26 04:12 202 -> pipe:[6216982]
lr-x------ 1 root root 64 Jun 26 04:12 204 -> pipe:[6216983]
l-wx------ 1 root root 64 Jun 26 04:12 205 -> pipe:[6216983]

anon_inode 部分的 FileDescriptor 如下, 合计 51 项 

root@ubuntu:~/docker/HelloWorld# ll /proc/58983/fd | grep ano
lrwx------ 1 root root 64 Jun 26 04:12 38 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Jun 26 04:12 41 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Jun 26 04:12 44 -> anon_inode:[eventpoll]
// 省略部分 fd 的信息
lrwx------ 1 root root 64 Jun 26 04:12 200 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Jun 26 04:12 203 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Jun 26 04:12 206 -> anon_inode:[eventpoll]

pipe 和 anon_inode

创建 pipe连接 和 anon_inode连接 的地方是在这里

当前 fd 列表信息如下 

root@ubuntu:~/docker/HelloWorld# ll /proc/59215/fd
total 0
dr-x------ 2 root root  0 Jun 26 04:26 ./
dr-xr-xr-x 9 root root  0 Jun 26 04:25 ../
lr-x------ 1 root root 64 Jun 26 04:26 0 -> /dev/null
l-wx------ 1 root root 64 Jun 26 04:26 1 -> /root/docker/HelloWorld/logs/nohup.log
l-wx------ 1 root root 64 Jun 26 04:26 10 -> /root/docker/HelloWorld/logs/error-2023-06-26.0.log
lr-x------ 1 root root 64 Jun 26 04:26 11 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/jfxrt.jar
l-wx------ 1 root root 64 Jun 26 04:26 12 -> /root/logs/nacos/config.log
lrwx------ 1 root root 64 Jun 26 04:26 13 -> socket:[6223806]
l-wx------ 1 root root 64 Jun 26 04:26 14 -> /root/logs/nacos/naming.log
lr-x------ 1 root root 64 Jun 26 04:26 15 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/jsse.jar
lr-x------ 1 root root 64 Jun 26 04:26 16 -> /dev/random
lr-x------ 1 root root 64 Jun 26 04:26 17 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 18 -> /dev/random
lr-x------ 1 root root 64 Jun 26 04:26 19 -> /dev/random
l-wx------ 1 root root 64 Jun 26 04:26 2 -> /root/docker/HelloWorld/logs/nohup.log
lr-x------ 1 root root 64 Jun 26 04:26 20 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 21 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 22 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/cldrdata.jar
lr-x------ 1 root root 64 Jun 26 04:26 23 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/localedata.jar
lrwx------ 1 root root 64 Jun 26 04:26 24 -> socket:[6226176]
lrwx------ 1 root root 64 Jun 26 04:26 25 -> socket:[6226178]
lr-x------ 1 root root 64 Jun 26 04:26 26 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/jce.jar
lr-x------ 1 root root 64 Jun 26 04:26 27 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunpkcs11.jar
lr-x------ 1 root root 64 Jun 26 04:26 28 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunec.jar
lr-x------ 1 root root 64 Jun 26 04:26 29 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunjce_provider.jar
lr-x------ 1 root root 64 Jun 26 04:26 3 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/rt.jar
lrwx------ 1 root root 64 Jun 26 04:26 30 -> socket:[6223845]
lr-x------ 1 root root 64 Jun 26 04:26 31 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/resources.jar
lrwx------ 1 root root 64 Jun 26 04:26 32 -> socket:[6223868]
lrwx------ 1 root root 64 Jun 26 04:26 33 -> socket:[6223871]
lr-x------ 1 root root 64 Jun 26 04:26 34 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/charsets.jar
l-wx------ 1 root root 64 Jun 26 04:26 4 -> /root/docker/HelloWorld/logs/debug-2023-06-26.0.log
lrwx------ 1 root root 64 Jun 26 04:26 5 -> socket:[6223797]
lr-x------ 1 root root 64 Jun 26 04:26 6 -> /root/docker/HelloWorld/HelloWorld.jar
lr-x------ 1 root root 64 Jun 26 04:26 7 -> /root/docker/HelloWorld/HelloWorld.jar
l-wx------ 1 root root 64 Jun 26 04:26 8 -> /root/docker/HelloWorld/logs/info-2023-06-26.0.log
l-wx------ 1 root root 64 Jun 26 04:26 9 -> /root/docker/HelloWorld/logs/warn-2023-06-26.0.log

单步调试过了 provider.openSelector 之后, fd 列表如下

可以看到除了增加了两个 pipe 的 fd 和一个 anon_inode 之外还增加了一部分 provider.openSelector 期间需要加载的 jar, 连接 等等

root@ubuntu:~/docker/HelloWorld# ll /proc/59215/fd
total 0
dr-x------ 2 root root  0 Jun 26 04:26 ./
dr-xr-xr-x 9 root root  0 Jun 26 04:25 ../
lr-x------ 1 root root 64 Jun 26 04:26 0 -> /dev/null
l-wx------ 1 root root 64 Jun 26 04:26 1 -> /root/docker/HelloWorld/logs/nohup.log
l-wx------ 1 root root 64 Jun 26 04:26 10 -> /root/docker/HelloWorld/logs/error-2023-06-26.0.log
lr-x------ 1 root root 64 Jun 26 04:26 11 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/jfxrt.jar
l-wx------ 1 root root 64 Jun 26 04:26 12 -> /root/logs/nacos/config.log
lrwx------ 1 root root 64 Jun 26 04:26 13 -> socket:[6223806]
l-wx------ 1 root root 64 Jun 26 04:26 14 -> /root/logs/nacos/naming.log
lr-x------ 1 root root 64 Jun 26 04:26 15 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/jsse.jar
lr-x------ 1 root root 64 Jun 26 04:26 16 -> /dev/random
lr-x------ 1 root root 64 Jun 26 04:26 17 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 18 -> /dev/random
lr-x------ 1 root root 64 Jun 26 04:26 19 -> /dev/random
l-wx------ 1 root root 64 Jun 26 04:26 2 -> /root/docker/HelloWorld/logs/nohup.log
lr-x------ 1 root root 64 Jun 26 04:26 20 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 21 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 22 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/cldrdata.jar
lr-x------ 1 root root 64 Jun 26 04:26 23 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/localedata.jar
lrwx------ 1 root root 64 Jun 26 04:26 24 -> socket:[6226176]
lr-x------ 1 root root 64 Jun 26 04:26 25 -> pipe:[6230147]
lr-x------ 1 root root 64 Jun 26 04:26 26 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/jce.jar
lr-x------ 1 root root 64 Jun 26 04:26 27 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunpkcs11.jar
lr-x------ 1 root root 64 Jun 26 04:26 28 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunec.jar
lr-x------ 1 root root 64 Jun 26 04:26 29 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunjce_provider.jar
lr-x------ 1 root root 64 Jun 26 04:26 3 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/rt.jar
lrwx------ 1 root root 64 Jun 26 04:26 30 -> socket:[6223845]
lr-x------ 1 root root 64 Jun 26 04:26 31 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/resources.jar
lrwx------ 1 root root 64 Jun 26 04:26 32 -> socket:[6223868]
lrwx------ 1 root root 64 Jun 26 04:26 33 -> socket:[6223871]
lr-x------ 1 root root 64 Jun 26 04:26 34 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/charsets.jar
l-wx------ 1 root root 64 Jun 26 04:36 35 -> pipe:[6230147]
lrwx------ 1 root root 64 Jun 26 04:36 36 -> anon_inode:[eventpoll]
l-wx------ 1 root root 64 Jun 26 04:26 4 -> /root/docker/HelloWorld/logs/debug-2023-06-26.0.log
lrwx------ 1 root root 64 Jun 26 04:26 5 -> socket:[6223797]
lr-x------ 1 root root 64 Jun 26 04:26 6 -> /root/docker/HelloWorld/HelloWorld.jar
lr-x------ 1 root root 64 Jun 26 04:26 7 -> /root/docker/HelloWorld/HelloWorld.jar
l-wx------ 1 root root 64 Jun 26 04:26 8 -> /root/docker/HelloWorld/logs/info-2023-06-26.0.log
l-wx------ 1 root root 64 Jun 26 04:26 9 -> /root/docker/HelloWorld/logs/warn-2023-06-26.0.log

后面的每一次 调用, 就仅仅是增加了 一对 pipe 的 fd 和 一个 anon_inode 的 fd 了 

root@ubuntu:~/docker/HelloWorld# ll /proc/59215/fd
total 0
dr-x------ 2 root root  0 Jun 26 04:26 ./
dr-xr-xr-x 9 root root  0 Jun 26 04:25 ../
lr-x------ 1 root root 64 Jun 26 04:26 0 -> /dev/null
l-wx------ 1 root root 64 Jun 26 04:26 1 -> /root/docker/HelloWorld/logs/nohup.log
l-wx------ 1 root root 64 Jun 26 04:26 10 -> /root/docker/HelloWorld/logs/error-2023-06-26.0.log
lr-x------ 1 root root 64 Jun 26 04:26 11 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/jfxrt.jar
l-wx------ 1 root root 64 Jun 26 04:26 12 -> /root/logs/nacos/config.log
lrwx------ 1 root root 64 Jun 26 04:26 13 -> socket:[6223806]
l-wx------ 1 root root 64 Jun 26 04:26 14 -> /root/logs/nacos/naming.log
lr-x------ 1 root root 64 Jun 26 04:26 15 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/jsse.jar
lr-x------ 1 root root 64 Jun 26 04:26 16 -> /dev/random
lr-x------ 1 root root 64 Jun 26 04:26 17 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 18 -> /dev/random
lr-x------ 1 root root 64 Jun 26 04:26 19 -> /dev/random
l-wx------ 1 root root 64 Jun 26 04:26 2 -> /root/docker/HelloWorld/logs/nohup.log
lr-x------ 1 root root 64 Jun 26 04:26 20 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 21 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 22 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/cldrdata.jar
lr-x------ 1 root root 64 Jun 26 04:26 23 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/localedata.jar
lrwx------ 1 root root 64 Jun 26 04:26 24 -> socket:[6226176]
lr-x------ 1 root root 64 Jun 26 04:26 25 -> pipe:[6230147]
lr-x------ 1 root root 64 Jun 26 04:26 26 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/jce.jar
lr-x------ 1 root root 64 Jun 26 04:26 27 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunpkcs11.jar
lr-x------ 1 root root 64 Jun 26 04:26 28 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunec.jar
lr-x------ 1 root root 64 Jun 26 04:26 29 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunjce_provider.jar
lr-x------ 1 root root 64 Jun 26 04:26 3 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/rt.jar
lrwx------ 1 root root 64 Jun 26 04:26 30 -> socket:[6223845]
lr-x------ 1 root root 64 Jun 26 04:26 31 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/resources.jar
lrwx------ 1 root root 64 Jun 26 04:26 32 -> socket:[6223868]
lrwx------ 1 root root 64 Jun 26 04:26 33 -> socket:[6223871]
lr-x------ 1 root root 64 Jun 26 04:26 34 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/charsets.jar
l-wx------ 1 root root 64 Jun 26 04:36 35 -> pipe:[6230147]
lrwx------ 1 root root 64 Jun 26 04:36 36 -> anon_inode:[eventpoll]
lr-x------ 1 root root 64 Jun 26 04:40 37 -> pipe:[6230350]
l-wx------ 1 root root 64 Jun 26 04:40 38 -> pipe:[6230350]
lrwx------ 1 root root 64 Jun 26 04:40 39 -> anon_inode:[eventpoll]
l-wx------ 1 root root 64 Jun 26 04:26 4 -> /root/docker/HelloWorld/logs/debug-2023-06-26.0.log
lrwx------ 1 root root 64 Jun 26 04:26 5 -> socket:[6223797]
lr-x------ 1 root root 64 Jun 26 04:26 6 -> /root/docker/HelloWorld/HelloWorld.jar
lr-x------ 1 root root 64 Jun 26 04:26 7 -> /root/docker/HelloWorld/HelloWorld.jar
l-wx------ 1 root root 64 Jun 26 04:26 8 -> /root/docker/HelloWorld/logs/info-2023-06-26.0.log
l-wx------ 1 root root 64 Jun 26 04:26 9 -> /root/docker/HelloWorld/logs/warn-2023-06-26.0.log

数量大致 规则是这样

当前 ClusterConnectionManager 的 super(config) 会创建 8[未配置线程数量, 使用的默认值] 个 NioEventLoop

下面遍历配置的 redis 集群列表, 每一个服务创建一组 NioEventLoop

遍历 redis 集群配置列表, 依次连接, 在 RedisClientConfig 中有创建 NioEventLoop, 这里是 创建的 8 个[构造方法中调用的 NioEventLoopGroup 的无参构造方法, 在 MultithreadEventLoopGroup 中取的默认值]

RedisClientConfig 中初始化 NioEventLoopGroup 的地方  

pipe 和 anon_inode 创建的地方

EPollSelectorProvider 初始化的地方 

EPollSelectorImpl 创建了一对 pipe 的 fd, EPollArrayWrapper 中创建的 anon_inode 的 fd  

EPollArrayWrapper 中创建的 epoll 对应的 fd 

更基础的东西就是 pipe, epoll_create 等等操作系统的系统调用了, 这里不多赘述 

完