What factors affect the number of TCP connections in Linux system

The main factors that affect the number of TCP connections in the Linux system are memory and the number of allowed file descriptors, because each TCP connection occupies a certain amount of memory, and each socket is a file descriptor, and the other 1024 The following ports are usually reserved ports.

#The operating environment of this tutorial: linux7.3 system, Dell G3 computer.

In tcp applications, the server listens on a fixed port in advance, the client actively initiates a connection, and the tcp connection is established after three handshakes. So for a single machine, what is the maximum number of concurrent TCP connections?

How to identify a TCP connection

Before determining the maximum number of connections, let’s first take a look at how the system identifies a TCP connection. The system uses a 4-tuple to uniquely identify a TCP connection: {localip, localport,remoteip,remoteport}.

Maximum number of tcp connections for client

Every time the client initiates a tcp connection request, unless the port is bound, the system will usually choose an idle one The local port (local port) is exclusive and cannot be shared with other TCP connections. The data type of the TCP port is unsigned short, so the maximum number of local ports is only 65536. Port 0 has a special meaning and cannot be used. In this way, the available ports are only 65535 at most. Therefore, when all are used as clients, the maximum number of TCP connections for a client is 65535, these connections can be connected to different serverips.

Maximum number of tcp connections for server

The server is usually fixed to listen on a local port, waiting for the client's connection request. Without considering address reuse (the SO_REUSEADDR option of Unix), even if there are multiple IPs on the server side, the local listening port is exclusive. Therefore, there are only remoteip (that is, clientip) and remote port (client) in the 4-tuple of the server-side tcp connection. port) is variable, so the maximum TCP connection is the number of client IP × the number of client ports. For IPV4, regardless of factors such as IP address classification, the maximum number of TCP connections is approximately 2 to the 32nd power (number of IPs) × 2 to the 16th power (number of ports), that is, the maximum number of TCP connections on a single machine on the server side is approximately 2 to the 48th power.

Actual number of tcp connections

The above is the theoretical maximum number of connections for a single machine. In the actual environment, it is affected by machine resources. , operating system, etc., especially on the server side, the maximum number of concurrent TCP connections is far from reaching the theoretical upper limit. The main factors that limit the number of connections under Unix/Linux are memory and the number of allowed file descriptors (each TCP connection occupies a certain amount of memory, and each socket is a file descriptor). In addition, ports below 1024 are usually reserved. port. Under the default 2.6 kernel configuration, after testing, each socket occupies between 15 and 20k of memory.

So, on the server side, by increasing memory and modifying parameters such as the maximum number of file descriptors, it is no problem that the maximum number of concurrent TCP connections on a single machine exceeds 100,000, or even millions.

This is obviously a misunderstanding. 65535 refers to the total number of available ports, which does not mean that the server can only accept 65535 concurrent connections at the same time.

For example:

We made a website and bound it to TCP port 80. The result is that all users who visit this website access port 80 of the server. , rather than other ports. Visible ports can be reused. Even if the Linux server only listens for services on port 80, it allows 100,000 or 1 million users to connect to the server. The Linux system does not limit the number of connections. Whether the server can withstand so many connections depends on the server's hardware configuration, software architecture and optimization.

01

We know that if two processes need to communicate, the most basic premise is to be able to uniquely identify a process . In local process communication, we can use PID to uniquely identify a process, but PID is only unique locally, and the probability of PID conflict between two processes in the network is very high.

At this time, you need to find another way. The IP address can uniquely identify the host, and the TCP layer protocol and port number can uniquely identify a process of the host. In this way, IP address + protocol + port number can be used to uniquely identify the host. A process in the network.

After being able to uniquely identify processes on the network, they can communicate using sockets. Socket is an abstraction layer between the application layer and the transport layer. It abstracts the complex operations of the TCP/IP layer into a few simple interfaces for the application layer to call and implement processes to communicate in the network.

Socket originates from Unix and is an implementation of the "open-read/write-close" mode. The server and client each maintain a " File", after establishing a connection and opening it, you can write content to your own file for the other party to read or read the other party's content, and close the file when the communication ends.

02

The only thing that can determine a connection is 4 things:

1. The IP of the server

2. Server’s Port

3. Client’s IP

4. Client’s Port

The server’s IP and Port can remain unchanged, as long as the client’s IP and Port are different from each other to determine a connection number.

A socket can establish multiple connections. A TCP connection is marked as a four-tuple (source_ip, source_port, destination_ip, destination_port), that is, (source IP, Source port, destination IP, destination port) is a combination of four elements. As long as one element in the combination of four elements is different, different connections can be distinguished.

For example:

->Your host IP address is 1.1.1.1, listen on port 8080

->When a message comes from 2.2.2.2 A connection request, port 5555. The four-tuple of this connection is (1.1.1.1, 8080, 2.2.2.2, 5555)

->At this time, 2.2.2.2 sent a second connection request with the port 6666. The four-tuple of the new connection is (1.1.1.1, 8080, 2.2.2.2, 6666)

Then,The 8080 port of your host has established two connections;

-> The third connection request sent from (2.2.2.2), the port is 5555 (or 6666). The request for the third connection cannot be established because there is no way to distinguish it from the above two connections.

Similarly, You can bind a TCP socket and a UDP socket on the same port number and IP address

Because the port number is the same, but the protocol is different Same, so the ports are completely independent.

TCP/UDP generally uses five-tuple to locate a connection:

source_ip, source_port, destination_ip, destination_port, protocol_type

That is (source IP , source port, destination IP, destination port, protocol number)

To sum up, the number of concurrent servers is not determined by TCP’s 65535 ports. The number of concurrencies that a server can withstand at the same time is determined by many factors such as bandwidth, hardware, and program design.

So we can understand why Taobao, Tencent, Toutiao, Baidu, Sina, Beep Beep, etc. can withstand hundreds of millions of concurrent accesses per second because they use servers cluster. Server clusters are distributed in large computer rooms across the country. When the number of visits is small, some servers will be shut down, and when the number of visits is high, new servers will be opened continuously.

Common sense 1: File handle restrictions

Friends who write network server programs under Linux must know that each tcp connection must occupy a file descriptor. Once this The file descriptor has been used up, and the error returned to us when a new connection arrives is "Socket/File: Can't open so many files".

At this time you need to understand the limitations of the operating system on the maximum number of files that can be opened.

• Process Limit

  • Execute ulimit -n and output 1024, indicating that a process can only open up to 1024 files, so you have to With this default configuration, up to thousands of concurrent TCP connections can be made.

  • Temporary modification: ulimit -n 1000000, but this temporary modification is only valid for the current usage environment of the currently logged in user, and will become invalid after the system is restarted or the user logs out.

  • Modifications that become invalid after restarting (but I tested under CentOS 6.5 and found no invalidity after restarting): Edit the /etc/security/limits.conf file, and the modified content is

    * soft nofile 1000000

    * hard nofile 1000000

  • Permanent modification: edit /etc/rc .local, add the following content after it

    ulimit -SHn 1000000

• Global limit

  • Execute cat /proc/sys/fs/file-nr output 9344 0 592026, respectively: 1. The number of file handles that have been allocated, 2. The number of file handles that have been allocated but not used, 3. Maximum number of file handles. But in kernel version 2.6, the value of the second item is always 0. This is not an error. It actually means that all the allocated file descriptors have been used.

  • We can change this value to a larger value and modify the /etc/sysctl.conf file with root permissions:

    fs.file-max = 1000000

    net.ipv4.ip_conntrack_max = 1000000

    ##net.ipv4.netfilter.ip_conntrack_max = 1000000

Common sense 2: Port number range limit?

On the operating system, port numbers below 1024 are reserved by the system, and 1024-65535 are used by users. Since each TCP connection occupies a port number, we can have up to more than 60,000 concurrent connections. I think there are many friends who have this wrong idea, right? (I have always thought so in the past)

Let’s analyze it

• How to identify a TCP connection: The system uses a 4-tuple to uniquely identify a TCP connection: {local ip, local port, remote ip, remote port}. Well, let's take out the explanation of accept in Chapter 4 of "UNIX Network Programming: Volume 1" and take a look at the conceptual stuff. The second parameter cliaddr represents the client's ip address and port number. As the server, we actually only use the port during bind, which shows that the port number 65535 is not a limit on the amount of concurrency.

• Maximum number of tcp connections for server: The server is usually fixed to listen on a local port, waiting for the client's connection request. Without considering address reuse (the SO_REUSEADDR option of Unix), even if there are multiple IPs on the server side, the local listening port is exclusive, so the server-side tcp connection 4-tuple only has remote ip (that is, client ip) and remote port. (Client port) is variable, so the maximum TCP connection is the number of client IP × the number of client ports. For IPV4, regardless of factors such as IP address classification, the maximum number of TCP connections is approximately 2 to the 32nd power (number of IPs) )×2 to the 16th power (number of ports), that is, the maximum number of TCP connections on a single machine on the server side is approximately 2 to the 48th power.

Common settings

1. Modify the limit on the number of files that a user process can open

In Linux On the platform, whether you write a client program or a server program, when processing high-concurrency TCP connections, the maximum number of concurrencies is subject to the system's limit on the number of files that a user can open at the same time in a single process (this is because the system provides A socket handle must be created for each connection, and each socket handle is also a file handle). You can use the ulimit command to view the limit on the number of files that the system allows the current user process to open:

[speng@as4 ~]$ ulimit -n
1024

This means that each process of the current user is allowed to open up to 1024 files at the same time. Each process must be removed from these 1024 files. Files such as standard input, standard output, standard error, server monitoring sockets, and unix domain sockets for inter-process communication must be opened, so the number of remaining files that can be used for client socket connections is only about 1024-10=1014. That is to say, by default, Linux-based communication programs allow up to 1014 concurrent TCP connections at the same time.
For communication handlers that want to support a higher number of concurrent TCP connections, you must modify Linux's soft limit (soft limit) and hard limit (hardlimit)## on the number of files opened simultaneously by the current user's process. #. The soft limit means that Linux further limits the number of files that the user can open at the same time within the range that the current system can bear; the hard limit is the maximum number of files that the system can open at the same time calculated based on the system hardware resources (mainly system memory). Usually the soft limit is less than or equal to the hard limit. The easiest way to modify the above restrictions is to use the ulimit command:

[speng@as4 ~]$ ulimit -n

In the above command, specify the maximum number of files allowed to be opened by a single process to be set. If the system echoes something like "Operation notpermitted", it means that the above limit modification failed. In fact, it is because the value specified in exceeds the Linux system's soft limit or hard limit on the number of files opened by the user. Therefore, it is necessary to modify the soft and hard limits of the Linux system on the number of open files for users.

The first step, modify the /etc/security/limits.conf file, add the following line in the file:

...
# End of file
speng soft nofile 10240
speng hard nofile 10240
root soft nofile 65535
root hard nofile 65535
* soft nofile 65535
* hard nofile 65535
[test@iZwz9e1dh1nweaex8ob5b7Z config]$

speng specifies which user to modify For the limit on the number of open files, you can use the '*' sign to indicate modifying the limit for all users; soft or hard specifies whether to modify the soft limit or the hard limit; 10240 specifies the new limit value you want to modify, that is, the maximum number of open files (please note The soft limit value must be less than or equal to the hard limit). Save the file after making changes.

The second step, modify the /etc/pam.d/login file, add the following line in the file:

session required /lib/security/pam_limits.so

This tells Linux that after the user completes the system login Finally, the pam_limits.so module should be called to set the system's maximum limit on the number of various resources that the user can use (including the limit on the maximum number of files that the user can open), and the pam_limits.so module will set the system limit from /etc/security/limits The configuration is read from the .conf file to set these limit values. Save this file after modification.

The third step, check the Linux system-level limit on the maximum number of open files, use the following command:

[speng@as4 ~]$ cat /proc/sys/fs/file-max
12158

This indicates that this Linux system allows at most to be opened at the same time (i.e. Including the total number of open files for all users) 12158 files, which is a Linux system-level hard limit. All user-level limits on the number of open files should not exceed this value. Usually this system-level hard limit is the optimal maximum number of files opened at the same time calculated based on the system hardware resources when the Linux system starts. If there is no special need, this limit should not be modified unless you want to limit the number of files opened at the user level. Set a value that exceeds this limit. The way to modify this hard limit is to modify the /etc/rc.local script and add the following line to the script:

echo 22158 > /proc/sys/fs/file-max

This is to force Linux to set the system-level hard limit on the number of open files to 22158 after the startup is completed. . Save this file after modification.

完成上述步骤后重启系统，一般情况下就可以将Linux系统对指定用户的单一进程允许同时打开的最大文件数限制设为指定的数值。如果重启后用 ulimit-n命令查看用户可打开文件数限制仍然低于上述步骤中设置的最大值，这可能是因为在用户登录脚本/etc/profile中使用ulimit -n命令已经将用户可同时打开的文件数做了限制。由于通过ulimit-n修改系统对用户可同时打开文件的最大数限制时，新修改的值只能小于或等于上次 ulimit-n设置的值，因此想用此命令增大这个限制值是不可能的。所以，如果有上述问题存在，就只能去打开/etc/profile脚本文件，在文件中查找是否使用了ulimit-n限制了用户可同时打开的最大文件数量，如果找到，则删除这行命令，或者将其设置的值改为合适的值，然后保存文件，用户退出并重新登录系统即可。

通过上述步骤，就为支持高并发TCP连接处理的通讯处理程序解除关于打开文件数量方面的系统限制。

2、修改网络内核对TCP连接的有关限制（参考对比下篇文章“优化内核参数”）

在Linux上编写支持高并发TCP连接的客户端通讯处理程序时，有时会发现尽管已经解除了系统对用户同时打开文件数的限制，但仍会出现并发TCP连接数增加到一定数量时，再也无法成功建立新的TCP连接的现象。出现这种现在的原因有多种。

第一种原因可能是因为Linux网络内核对本地端口号范围有限制。此时，进一步分析为什么无法建立TCP连接，会发现问题出在connect()调用返回失败，查看系统错误提示消息是“Can’t assign requestedaddress”。同时，如果在此时用tcpdump工具监视网络，会发现根本没有TCP连接时客户端发SYN包的网络流量。这些情况说明问题在于本地Linux系统内核中有限制。其实，问题的根本原因在于Linux内核的TCP/IP协议实现模块对系统中所有的客户端TCP连接对应的本地端口号的范围进行了限制(例如，内核限制本地端口号的范围为1024~32768之间)。当系统中某一时刻同时存在太多的TCP客户端连接时，由于每个TCP客户端连接都要占用一个唯一的本地端口号(此端口号在系统的本地端口号范围限制中)，如果现有的TCP客户端连接已将所有的本地端口号占满，则此时就无法为新的TCP客户端连接分配一个本地端口号了，因此系统会在这种情况下在connect()调用中返回失败，并将错误提示消息设为“Can’t assignrequested address”。有关这些控制逻辑可以查看Linux内核源代码，以linux2.6内核为例，可以查看tcp_ipv4.c文件中如下函数：

static int tcp_v4_hash_connect(struct sock *sk)

请注意上述函数中对变量sysctl_local_port_range的访问控制。变量sysctl_local_port_range的初始化则是在tcp.c文件中的如下函数中设置：

void __init tcp_init(void)

内核编译时默认设置的本地端口号范围可能太小，因此需要修改此本地端口范围限制。

第一步，修改/etc/sysctl.conf文件，在文件中添加如下行：

net.ipv4.ip_local_port_range = 1024 65000

这表明将系统对本地端口范围限制设置为1024~65000之间。请注意，本地端口范围的最小值必须大于或等于1024；而端口范围的最大值则应小于或等于65535。修改完后保存此文件。

第二步，执行sysctl命令：

[speng@as4 ~]$ sysctl -p

如果系统没有错误提示，就表明新的本地端口范围设置成功。如果按上述端口范围进行设置，则理论上单独一个进程最多可以同时建立60000多个TCP客户端连接。

第二种无法建立TCP连接的原因可能是因为Linux网络内核的IP_TABLE防火墙对最大跟踪的TCP连接数有限制。此时程序会表现为在 connect()调用中阻塞，如同死机，如果用tcpdump工具监视网络，也会发现根本没有TCP连接时客户端发SYN包的网络流量。由于 IP_TABLE防火墙在内核中会对每个TCP连接的状态进行跟踪，跟踪信息将会放在位于内核内存中的conntrackdatabase中，这个数据库的大小有限，当系统中存在过多的TCP连接时，数据库容量不足，IP_TABLE无法为新的TCP连接建立跟踪信息，于是表现为在connect()调用中阻塞。此时就必须修改内核对最大跟踪的TCP连接数的限制，方法同修改内核对本地端口号范围的限制是类似的：
第一步，修改/etc/sysctl.conf文件，在文件中添加如下行：

net.ipv4.ip_conntrack_max = 10240

这表明将系统对最大跟踪的TCP连接数限制设置为10240。请注意，此限制值要尽量小，以节省对内核内存的占用。

第二步，执行sysctl命令：

[speng@as4 ~]$ sysctl -p

如果系统没有错误提示，就表明系统对新的最大跟踪的TCP连接数限制修改成功。如果按上述参数进行设置，则理论上单独一个进程最多可以同时建立10000多个TCP客户端连接。

3、使用支持高并发网络I/O的编程技术

在Linux上编写高并发TCP连接应用程序时，必须使用合适的网络I/O技术和I/O事件分派机制。
可用的I/O技术有同步I/O，非阻塞式同步I/O(也称反应式I/O)，以及异步I/O。《BIO,NIO,AIO的理解》

在高TCP并发的情形下，如果使用同步I/O，这会严重阻塞程序的运转，除非为每个TCP连接的I/O创建一个线程。但是，过多的线程又会因系统对线程的调度造成巨大开销。因此，在高TCP并发的情形下使用同步 I/O是不可取的，这时可以考虑使用非阻塞式同步I/O或异步I/O。非阻塞式同步I/O的技术包括使用select()，poll()，epoll等机制。异步I/O的技术就是使用AIO。
从I/O事件分派机制来看，使用select()是不合适的，因为它所支持的并发连接数有限(通常在1024个以内)。如果考虑性能，poll()也是不合适的，尽管它可以支持的较高的TCP并发数，但是由于其采用“轮询”机制，当并发数较高时，其运行效率相当低，并可能存在I/O事件分派不均，导致部分TCP连接上的I/O出现“饥饿”现象。而如果使用epoll或AIO，则没有上述问题(早期Linux内核的AIO技术实现是通过在内核中为每个 I/O请求创建一个线程来实现的，这种实现机制在高并发TCP连接的情形下使用其实也有严重的性能问题。但在最新的Linux内核中，AIO的实现已经得到改进)。
综上所述，在开发支持高并发TCP连接的Linux应用程序时，应尽量使用epoll或AIO技术来实现并发的TCP连接上的I/O控制，这将为提升程序对高并发TCP连接的支持提供有效的I/O保证。

内核参数sysctl.conf的优化

/etc/sysctl.conf 是用来控制linux网络的配置文件，对于依赖网络的程序（如web服务器和cache服务器）非常重要，RHEL默认提供的最好调整。

推荐配置（把原/etc/sysctl.conf内容清掉，把下面内容复制进去）：

net.ipv4.ip_local_port_range = 1024 65536
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 0
net.ipv4.tcp_sack = 0
net.core.netdev_max_backlog = 30000
net.ipv4.tcp_no_metrics_save=1
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2

这个配置参考于cache服务器varnish的推荐配置和SunOne 服务器系统优化的推荐配置。

varnish调优推荐配置的地址为：http://varnish.projects.linpro.no/wiki/Performance

不过varnish推荐的配置是有问题的，实际运行表明“net.ipv4.tcp_fin_timeout = 3”的配置会导致页面经常打不开；并且当网友使用的是IE6浏览器时，访问网站一段时间后，所有网页都会打不开，重启浏览器后正常。可能是国外的网速快吧，我们国情决定需要调整“net.ipv4.tcp_fin_timeout = 10”，在10s的情况下，一切正常（实际运行结论）。

修改完毕后，执行：

/sbin/sysctl -p /etc/sysctl.conf
/sbin/sysctl -w net.ipv4.route.flush=1

命令生效。为了保险起见，也可以reboot系统。

调整文件数：

linux系统优化完网络必须调高系统允许打开的文件数才能支持大的并发，默认1024是远远不够的。

执行命令：

echo ulimit -HSn 65536 >> /etc/rc.local
echo ulimit -HSn 65536 >>/root/.bash_profile
ulimit -HSn 65536

备注：

对mysql用户可同时打开文件数设置为10240个；
将Linux系统可同时打开文件数设置为1000000个（一定要大于对用户的同时打开文件数限制）；
将Linux系统对最大追踪的TCP连接数限制为20000个（但是，建议设置为10240；因为对mysql用户的同时打开文件数已经限制在10240个；且较小的值可以节省内存）；
将linux系统端口范围配置为1024～30000（可以支持60000个以上连接，不建议修改；默认已经支持20000个以上连接）；

综合上述四点，TCP连接数限制在10140个。
这10240个文件中还得除去每个进程必然打开的标准输入，标准输出，标准错误，服务器监听 socket，进程间通讯的unix域socket等文件。

因此，当需要对TCP连接数进行调整时只需要调整ulimit参数。

Linux下查看tcp连接数及状态命令：

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

相关推荐：《Linux视频教程》

The above is the detailed content of What factors affect the number of TCP connections in Linux system. For more information, please follow other related articles on the PHP Chinese website!