Discuss in detail the difference between so_reuseport and so_reuseaddr in sockets

The following article will share with you a detailed discussion of the difference between so_reuseport and so_reuseaddr in sockets. It has a good reference value and I hope it will be helpful to everyone. Let’s take a look together

Basic background of Socket

When discussing the difference between these two options, what we need to know is the BSD implementation It is the origin of all socket implementations. Basically all other systems referenced the BSD socket implementation (or at least its interface) to some extent and then began their own independent evolution. Obviously, BSD itself is constantly evolving and changing over time. Therefore, systems that reference BSD later have more features than systems that reference BSD earlier. So understanding the BSD socket implementation is the cornerstone of understanding other socket implementations. Let's analyze the BSD socket implementation.

Before this, we must first understand how to uniquely identify the TCP/UDP connection. TCP/UDP is uniquely identified by the following five-tuple:

{<protocol>, <src addr>, <src port>, <dest addr>, <dest port>}

Any unique combination of these values ​​can be uniquely identified Confirm a connection. Then, for any connection, these five values ​​cannot be exactly the same. Otherwise the operating system would not be able to distinguish between these connections.

The protocol of a socket is set when initialized with socket(). The source address and source port are set when calling bind(). The destination address and destination port are set when calling connect(). UDP is connectionless, and UDP socket can be used without being connected to the destination port. However, UDP can also be used in some cases after establishing a connection with the destination address and port. When using connectionless UDP to send data, if bind() is not explicitly called, the system will automatically bind the UDP socket to the local address and a certain port when sending data for the first time (otherwise The program cannot accept any data replied by the remote host). Similarly, a TCP socket with no bound address will be automatically bound to a local address and port when the connection is established.

If we manually bind a port, we can bind the socket to port 0. Binding to port 0 means letting the system decide which port to use (usually from a set of operating system-specific advance within the determined port number range), so it means any port. Similarly, we can also use a wildcard to let the system decide which source address to bind (the ipv4 wildcard is 0.0.0.0, the ipv6 wildcard is::). Unlike a port, a socket can be bound to any address corresponding to all interfaces on the host. Based on the destination address connected to this socket and the corresponding information in the routing table, the operating system will select the appropriate address to bind this socket and use this address to replace the previous wildcard IP address.

By default, any two sockets cannot be bound to the same source address and source port combination. For example, we bind socketA to the address A:X, and bind socketB to the address B:Y, where A and B are IP addresses, and X and Y are ports. Then X!=Y must be satisfied when A==B, and A!=B must be satisfied when X==Y. It should be noted that if a certain socket is bound to a wildcard IP address, then in fact all IPs of the local machine will be considered by the system to be bound to it. For example, a socket is bound to 0.0.0.0:21. In this case, any other socket, no matter which specific IP address is selected, can no longer be bound to port 21. Because the wildcard IP0.0.0.0 conflicts with all local IPs.

All of the above are essentially the same across major operating systems. Each SO_REUSEADDR will have different meanings. First let's discuss the BSD implementation. Because BSD is the source of all other socket implementation methods.

BSD

SO_REUSEADDR

If bound to a socket If the SO_REUSEADDR attribute is set before reaching a certain address and port, then unless the socket conflicts with another socket trying to bind to the exact same source address and source port combination, the socket can be successfully bound. Define this address port pair. This may sound the same as before. But the key word there is complete. SO_REUSEADDR mainly changes the way the system treats wildcard IP address conflicts.

If SO_REUSEADDR is not used, if we bind socketA to 0.0.0.0:21, then any attempt to bind other sockets on this machine to port 21 (such as binding to 192.168.1.1:21) will Causes EADDRINUSE error. Because 0.0.0.0 is a wildcard IP address, meaning any IP address, any other IP address on this machine is considered occupied by the system. If the SO_REUSEADDR option is set, because 0.0.0.0:21 and 192.168.1.1:21 are not exactly the same address port pair (one of them is a wildcard IP address and the other is a specific IP address of the local machine), such binding It can definitely be successful. It should be noted that regardless of the order in which socketA and socketB are initialized, as long as SO_REUSEADDR is set, the binding will succeed; and as long as SO_REUSEADDR is not set, the binding will not succeed.

The table below lists some possible situations and their consequences.

#SO_REUSEADDRsocketAsocketBResultON / OFF192.168.1.1：21192.168.1.1：21ERROR(EADDRINUSE)ON/OFF192.168.1.1:2110.0.1.1:21OKON / OFF10.0.1.1：21192.168.1.1：21OKOFF192.168.1.1:210.0.0.0:21ERROR (EADDRINUSE) OFF0.0.0.0:21192.168.1.1:21ERROR (EADDRINUSE)ON192.168.1.1：210.0.0.0：21OK##ON##ON/OFF0.0.0.0：210.0 .0.0: 21OK

This table assumes that socketA has successfully bound to the corresponding address in the table, then socketB is initialized, and its SO_REUSEADDR setting is as shown in the first column of the table, and then socketB attempts to bind the corresponding address in the table. The Result column is the result of its binding. If the value in the first column is ON/OFF, whether SO_REUSEADDR is set or not has nothing to do with the result.

The role of SO_REUSEADDR on wildcard IP addresses has been discussed above, but it does not only have this role. Another role is why everyone uses the SO_REUSEADDR option when doing server-side programming. In order to understand its other role and its important applications, we need to first discuss more deeply how the TCP protocol works.

Each socket has its corresponding send buffer (buffer). When its send() method is successfully called, in fact the data we require to send is not necessarily sent immediately, but is added to the send buffer. For UDP sockets, even if it is not sent immediately, the data will generally be sent out quickly. But for TCP sockets, after adding data to the send buffer, it may take a relatively long time before the data is actually sent. Therefore, when we close a TCP socket, there may actually still be data waiting to be sent in its send buffer. But at this time, because send() returns success, our code thinks that the data has actually been sent successfully. If the TCP socket is closed directly after we call close(), all this data will be lost and our code will never know about it. However, TCP is a reliable transport layer protocol, and it is obviously not advisable to directly discard the data to be transmitted. In fact, if the close() method of the socket is called while there is still data to be sent in the send buffer, it will enter a so-called TIME_WAIT state. In this state, the socket will continue to try to send the data in the buffer until all data is sent successfully or until it times out. When the timeout is triggered, the socket will be forcibly closed.

The maximum waiting time of the operating system's kernel before forcibly closing a socket is called the Linger Time. The delay time is set globally on most systems and is relatively long (most systems set it to 2 minutes). We can also use the SO_LINGER option when initializing a socket to specifically set the delay time for each socket. We can even turn off delayed waiting entirely. However, it should be noted that setting the delay time to 0 (turning off delay waiting completely) is not a good programming practice. Because closing a TCP socket gracefully is a relatively complex process, which includes exchanging several data packets with the remote host (including loss retransmission in the case of packet loss), and the time required for this data packet exchange process is also Included in delay time. If we disable delayed waiting, the socket will not only discard all data to be sent when closing, but will always be forcibly closed (since TCP is a connection-oriented protocol, not exchanging closing packets with the remote port will result in The remote port is in a long waiting state). So usually we don't recommend doing this in actual programming. The process of TCP disconnection is beyond the scope of this article, if you are interested, you can refer to this page. And in fact, if we disable delayed waiting and our program exits without explicitly closing the socket, BSD (and possibly other systems) will ignore our setting and perform delayed waiting. For example, if our program calls the exit() method, or its process is terminated using a signal (including the process crashing due to illegal memory access and the like). Therefore, we cannot guarantee 100% that a socket will terminate regardless of the delay waiting time under all circumstances.

The problem here is how the operating system treats the socket in the TIME_WAIT stage. If the SO_REUSEADDR option is not set, the socket in the TIME_WAIT phase is still considered to be bound to the original address and port. Until the socket is completely closed (ends the TIME_WAIT phase), any other attempt to bind a new socket to this address and port pair will not succeed. This waiting process may be as long as the delay waiting time. Therefore, we cannot immediately bind a new socket to the address and port pair corresponding to the socket that has just been closed. This operation will fail in most cases.

However, if we set the SO_REUSEADDR option on a new socket, if there is another socket bound to the current address port pair and is in the TIME_WAIT phase, then this existing binding relationship will be ignored. In fact, the socket in the TIME_WAIT stage is already in a semi-closed state, and there will be no problem binding a new socket to this address and port pair. In this case, the original socket bound to this port will generally not affect the new socket. But it should be noted that at some point, binding a new socket to the address port pair corresponding to a socket that is in the TIME_WAIT stage but is still working will have some unintended and unpredictable negative effects. But this issue is beyond the scope of this article. And fortunately these negative effects are rarely seen in practice.

Finally, one thing we need to note about SO_REUSEADDR is that all the above is true as long as we set SO_REUSEADDR for the new socket. As for the original one that has been bound to the current address and port pair, it has no effect whether the socket in the TIME_WAIT stage or not has SO_REUSEADDR set. The code that determines whether the bind operation is successful simply checks the SO_REUSEADDR option of the new socket passed to the bind() method. The SO_REUSEADDR option of other involved sockets will not be checked.

SO_REUSEPORT

Many people regard SO_REUSEADDR as SO_REUSEPORT. Basically, SO_REUSEPORT allows us to bind any number of sockets to the exact same source address and port pair, as long as all previously bound sockets have the SO_REUSEPORT option set. If the first socket bound to the address and port pair does not have SO_REUSEPORT set, no matter whether the subsequent socket sets SO_REUSEPORT or not, it cannot be bound to the exact same address as this address port. Unless the first socket bound to this address and port pair releases the binding relationship. Unlike SO_REUSEADDR, the code that handles SO_REUSEPORT will not only check the SO_REUSEPORT of the socket currently trying to bind, but also check the SO_REUSEPORT option of the socket that has previously been bound to the address port pair currently trying to bind.

SO_REUSEPORT is not equal to SO_REUSEADDR. What this means is that if a socket with an already bound address does not have SO_REUSEPORT set, and another new socket has SO_REUSEPORT set and attempts to bind to the exact same port address pair as the current socket, the binding attempt will fail. At the same time, if the current socket is already in the TIME_WAIT stage, and the new socket with the SO_REUSEPORT option set tries to bind to the current address, the binding operation will also fail. In order to bind a new socket to the address port pair corresponding to a socket currently in the TIME_WAIT phase, we either need to set the SO_REUSEADDR option of the new socket before binding, or we need to set it for both sockets before binding. SO_REUSEPORT option. Of course, it is also possible to set the SO_REUSEADDR and SO_REUSEPORT options for the socket at the same time.

SO_REUSEPORT was added to the BSD system after SO_REUSEADDR. This is why some systems currently do not have the SO_REUSEPORT option in their socket implementations. Because they referenced the BSD socket implementation before this option was added to the BSD system. Before this option was added, there was no way to bind two sockets to the exact same address and port pair under the BSD system.

Connect() returns EADDRINUSE?

Sometimes the bind() operation returns an EADDRINUSE error. But the strange thing is that when we call the connect() operation, we may also get an EADDRINUSE error. Why is this? Why is a remote address that we try to establish a connection on the current port also occupied? Will there be any problems when connecting multiple sockets to the same remote address?

As mentioned before in this article, a connection relationship is determined by a five-tuple. For any connection relationship, this quintuple must be unique. Otherwise, the system will not be able to distinguish between the two connections. Now when we use address reuse, we can bind two sockets using the same protocol to the same address port pair. This means that for these two sockets, the five-tuple {a88b79ba1ccee8890e978c768d80530d, 3037a7cee66f0ae45683db6cc2520e8a, 1b9006debff836a11384f3384ae84936} is already the same. In this case, if we try to connect them both to the same remote address port, the five-tuple of the two connection relationships will be exactly the same. That is, two identical connections are produced. This is not allowed in the TCP protocol (UDP is connectionless). If one of these two identical connections receives data, the system will not be able to tell which connection the data belongs to. So in this case, at least the addresses and ports of the remote hosts that the two sockets are trying to connect to cannot be the same. Only in this way can the system continue to distinguish between the two connection relationships.

So when we bind two sockets using the same protocol to the same local address port pair, if we also try to connect them to the same destination address port pair, the second one tries to call connect() The method's socket will report an EADDRINUSE error, which means that a socket with exactly the same five-tuple already exists.

Multicast Address

Relative to the unicast address used for one-to-one communication, the multicast address is used for one-to-many communication. Both IPv4 and IPv6 have multicast addresses. But multicast in IPv4 is rarely used on public networks.

The meaning of SO_REUSEADDR will be different from before in the case of multicast address. In this case, SO_REUSEADDR allows us to bind multiple sockets to the exact same source broadcast address and port pair. In other words, for multicast addresses, SO_REUSEADDR is equivalent to SO_REUSEPORT in unicast communication. In fact, in the case of multicast, SO_REUSEADDR and SO_REUSEPORT have exactly the same effect.

FreeBSD/OpenBSD/NetBSD

All these systems refer to the newer native BSD system code. So these three systems provide exactly the same socket options as BSD, and the meaning of these options is exactly the same as native BSD.

MacOS X

The core code implementation of MacOS BSD has exactly the same socket options, and their meaning is the same as on BSD systems.

iOS

iOS is actually a slightly modified MacOS X, so what works for MacOS X also works for iOS.

Linux

Before Linux3.9, only the SO_REUSEADDR option existed. The function of this option is basically the same as that under BSD systems. But there are still two important differences.

The first difference is that if a TCP socket in the listening (server) state has been bound to a wildcard IP address and a specific port, then regardless of whether the SO_REUSEADDR option is set for these two sockets, No other TCP socket can be bound to the same port. This doesn't work even if the other socket uses a specific IP address (as allowed in BSD systems). Non-listening (client) TCP sockets do not have this restriction.

The second difference is that for UDP sockets, SO_REUSEADDR has the same function as SO_REUSEPORT in BSD. So if two UDP sockets have SO_REUSEADDR set, they can be bound to the exact same set of address and port pairs.

Linux3.9 added the SO_REUSEPORT option. Two or more, TCP or UDP, listening (server) or non-listening (client) sockets can be bound to the exact same address as long as all sockets (including the first one) have this option set before binding the address. under port combination. At the same time, in order to prevent port hijacking, there is a special restriction: all sockets trying to bind to the same address and port combination must belong to the process with the same user ID. So one user cannot "steal" a port from another user.

In addition, for sockets with the SO_REUSEPORT option set, the Linux kernel will also perform some special operations that other systems do not have: for UDP sockets bound to the same address and port combination, the kernel attempts Distribute received data packets evenly between them; for TCP listening sockets bound to the same address and port combination, the kernel attempts to evenly distribute received connection requests (requests obtained by calling the accept() method) between them. . This means that Linux attempts to optimize traffic distribution compared to other systems that allow address reuse but randomly assign received packets or connection requests to sockets connected to the same address and port combination. For example, several different instances of a simple server process can easily use SO_REUSEPORT to implement a simple load balancing, and the kernel is responsible for this load balancing, which is completely free for the program!

Android

The core part of Android is a slightly modified Linux kernel, so everything that applies to Linux also applies to Android.

Windows

Windows only has the SO_REUSEADDR option. Setting SO_REUSEADDR on a socket in Windows has the same effect as setting SO_REUSEPORT and SO_REUSEADDR on a socket at the same time under BSD. But the difference is: even if another socket with a bound address does not have SO_REUSEADDR set, a socket with SO_REUSEADDR set can always be bound to the exact same address and port combination as another bound socket. This behavior can be said to be somewhat dangerous. Because it allows one application to steal data from another reference to a connected port. Microsoft is aware of this problem and has added another socket option: SO_EXCLUSIVEADDRUSE. Setting SO_EXCLUSIVEADDRUSE on a socket ensures that once the socket is bound to an address and port combination, any other socket, whether SO_REUSEADDR is set or not, can no longer be bound to the current address and port combination.

Solaris

Solaris is the successor of SunOS. SunOS is also to some extent an offshoot of an earlier version of BSD. Therefore, Solaris only provides SO_REUSEADDR, and its performance is basically the same as in BSD systems. As far as I know, the same functionality as SO_REUSEPORT cannot be implemented in Solaris systems. This means that in Solaris you cannot bind two sockets to the exact same address and port combination.

Similar to Windows, Solaris also provides an exclusive binding option for sockets - SO_EXCLBIND. If a socket sets this option before binding the address, other sockets will not be able to bind to the same address even if they have SO_REUSEADDR set. For example: if socketA is bound to a wildcard IP address, and socketB is set to SO_REUSEADDR and bound to a combination of a specific IP address and the same port as socketA, this operation will succeed if socketA is not set to SO_EXCLBIND, otherwise will fail.

Reference:

http://stackoverflow.com/a/14388707/6037083

0.0.0.0：21	192.168.1.1：21	OK

The above is the detailed content of Discuss in detail the difference between so_reuseport and so_reuseaddr in sockets. For more information, please follow other related articles on the PHP Chinese website!