Socket programming practice

Socket means "a groove (that connects two items)" in English, like the eye socket, which means "eye socket", and also means "socket". In computer science, socket usually refers to two endpoints of a connection. The connection here can be on the same machine, like unix domain socket, or on different machines, like network socket.

This article focuses on the most commonly used network socket, including its position in the network model, API programming paradigm, common errors, etc., and finally uses the socket API in the Python language to implement several practical examples. Socket is generally translated as "socket" in Chinese. I have to say that this is a confusing translation. I didn't expect a "faithful and elegant" translation, so this article directly uses its English expression. All the code in this article can be found in the socket.py repository.

Overview

Socket, as a general technical specification, was first provided by Berkeley University for 4.2BSD Unix in 1983, and later gradually evolved into the POSIX standard. The Socket API is a programming interface provided by the operating system that allows applications to control the use of socket technology. There is a principle in the Unix philosophy that everything is a file, so the API usage of socket and file is very similar: read, write, open, close and other operations can be performed.

The current network system is layered, with the OSI model in theory and the TCP/IP protocol suite in the industry. The comparison is as follows:

Each layer has its corresponding protocol. The socket API does not belong to the TCP/IP protocol suite. It is just an interface provided by the operating system for network programming and works between the application layer and the transport layer. Between: 平

We usually browse the HTTP protocol used by the website. SMTP and IMAP for sending emails are built based on the Socket API.

A socket contains two necessary components:

Address, consisting of ip and port, like 192.168.0.1:80.

Protocol, the transmission protocol used by socket, there are currently three types: TCP, UDP, raw IP.

The address and protocol can determine a socket; only one identical socket is allowed to exist on a machine. The socket of TCP port 53 and the socket of UDP port 53 are two different sockets.

According to the different ways in which sockets transmit data (different protocols used), they can be divided into the following three types:

Stream sockets, also known as "connection-oriented" sockets, use the TCP protocol. A connection is required before actual communication, and the transmitted data does not have a specific structure, so the high-level protocol needs to define the data separator itself, but its advantage is that the data is reliable.

Datagram sockets, also known as "connectionless" sockets, use the UDP protocol. There is no need to connect before actual communication. One advantage is that UDP data packets are self-delimiting, which means that each data packet marks the beginning and end of the data. The disadvantage is that the data is unreliable.

Raw sockets are usually used in routers or other network equipment. This kind of socket does not go through the transport layer in the TCP/IP protocol suite, but directly leads from the Internet layer to the Application layer. , so the data packet at this time will not contain tcp or udp header information.

Python socket API

Python uses tuples of (ip, port) to represent the address attributes of the socket, and AF_* to represent the protocol type.

Data communication has two sets of verbs to choose from: send/recv or read/write. The read/write method is also used by Java. This method will not be explained too much here, but it should be noted that:

read/write operates on a "file" with a buffer, so after reading and writing, it is necessary to Call the flush method to actually send or read data, otherwise the data will stay in the buffer.

TCP socket

TCP socket needs to establish a connection before connecting, so its mode is more responsible than UDP socket. The details are as follows:

The specific meaning of each API will not be described here. You can check the manual. Here is the echo server implemented in Python language.

# echo_server.py 
# coding=utf8 
import socket 
 
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
# 设置 SO_REUSEADDR 后,可以立即使用 TIME_WAIT 状态的 socket 
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) 
sock.bind(('', 5500)) 
sock.listen(5)

def handler(client_sock, addr): 
    print('new client from %s:%s' % addr) 
    msg = client_sock.recv(1024) 
    client_sock.send(msg) 
    client_sock.close() 
    print('client[%s:%s] socket closed' % addr) 
 
if __name__ == '__main__': 
    while 1: 
        client_sock, addr = sock.accept() 
        handler(client_sock, addr)

# echo_client.py 
# coding=utf8 
import socket 
 
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
sock.connect(('', 5500)) 
sock.send('hello socket world') 
print sock.recv(1024)

One thing to note in the simple echo server code above is that the socket on the server side is set to SO_REUSEADDR to 1, so that the socket in the TIME_WAIT state can be used immediately. So what does TIME_WAIT mean? The tcp state will be explained later. I will introduce it in detail when I change the diagram.

UDP socket🎜

UDP socket server 端代码在进行bind后，无需调用listen方法。

# udp_echo_server.py 
# coding=utf8 
import socket 
 
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) 
# 设置 SO_REUSEADDR 后,可以立即使用 TIME_WAIT 状态的 socket 
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) 
sock.bind(('', 5500)) 
# 没有调用 listen 
 
if __name__ == '__main__': 
    while 1: 
        data, addr = sock.recvfrom(1024) 
 
        print('new client from %s:%s' % addr) 
        sock.sendto(data, addr) 
 
# udp_echo_client.py 
# coding=utf8 
import socket 
 
udp_server_addr = ('', 5500) 
 
if __name__ == '__main__': 
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) 
    data_to_sent = 'hello udp socket' 
    try: 
        sent = sock.sendto(data_to_sent, udp_server_addr) 
        data, server = sock.recvfrom(1024) 
        print('receive data:[%s] from %s:%s' % ((data,) + server)) 
    finally: 
        sock.close()

常见陷阱

忽略返回值

本文中的 echo server 示例因为篇幅限制，也忽略了返回值。网络通信是个非常复杂的问题，通常无法保障通信双方的网络状态，很有可能在发送/接收数据时失败或部分失败。所以有必要对发送/接收函数的返回值进行检查。本文中的 tcp echo client 发送数据时，正确写法应该如下：

total_send = 0 
content_length = len(data_to_sent) 
while total_send < content_length: 
    sent = sock.send(data_to_sent[total_send:]) 
    if sent == 0: 
        raise RuntimeError("socket connection broken") 
    total_send += total_send + sent

send/recv操作的是网络缓冲区的数据，它们不必处理传入的所有数据。

一般来说，当网络缓冲区填满时，send函数就返回了;当网络缓冲区被清空时，recv 函数就返回。

当 recv 函数返回0时，意味着对端已经关闭。

可以通过下面的方式设置缓冲区大小。

s.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, buffer_size)

认为 TCP 具有 framing

TCP 不提供 framing，这使得其很适合于传输数据流。这是其与 UDP 的重要区别之一。UDP 是一个面向消息的协议，能保持一条消息在发送者与接受者之间的完备性。

代码示例参考：framing_assumptions

TCP 的状态机

在前面echo server 的示例中，提到了TIME_WAIT状态，为了正式介绍其概念，需要了解下 TCP 从生成到结束的状态机器。(图片来源)

这个状图转移图非常非常关键，也比较复杂，我自己为了方便记忆，对这个图进行了拆解，仔细分析这个图，可以得出这样一个结论，连接的打开与关闭都有被动(passive)与主动(active)两种，主动关闭时，涉及到的状态转移最多，包括FIN_WAIT_1、FIN_WAIT_2、CLOSING、TIME_WAIT。

此外，由于 TCP 是可靠的传输协议，所以每次发送一个数据包后，都需要得到对方的确认(ACK)，有了上面这两个知识后，再来看下面的图：

在主动关闭连接的 socket 调用 close方法的同时，会向被动关闭端发送一个 FIN

对端收到FIN后，会向主动关闭端发送ACK进行确认，这时被动关闭端处于 CLOSE_WAIT 状态

当被动关闭端调用close方法进行关闭的同时向主动关闭端发送 FIN 信号，接收到 FIN 的主动关闭端这时就处于 TIME_WAIT 状态

这时主动关闭端不会立刻转为 CLOSED 状态，而是需要等待 2MSL(max segment life，一个数据包在网络传输中最大的生命周期)，以确保被动关闭端能够收到最后发出的 ACK。如果被动关闭端没有收到最后的 ACK，那么被动关闭端就会重新发送 FIN，所以处于TIME_WAIT的主动关闭端会再次发送一个 ACK 信号，这么一来(FIN来)一回(ACK)，正好是两个 MSL 的时间。如果等待的时间小于 2MSL，那么新的socket就可以收到之前连接的数据。

前面 echo server 的示例也说明了，处于 TIME_WAIT 并不是说一定不能使用，可以通过设置 socket 的 SO_REUSEADDR 属性以达到不用等待 2MSL 的时间就可以复用socket 的目的，当然，这仅仅适用于测试环境，正常情况下不要修改这个属性。

实战

HTTP UA

http 协议是如今万维网的基石，可以通过 socket API 来简单模拟一个浏览器(UA)是如何解析 HTTP 协议数据的。

#coding=utf8 
import socket 
 
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
baidu_ip = socket.gethostbyname('baidu.com') 
sock.connect((baidu_ip, 80)) 
print('connected to %s' % baidu_ip) 
 
req_msg = [ 
    'GET / HTTP/1.1', 
    'User-Agent: curl/7.37.1', 
    'Host: baidu.com', 
    'Accept: */*', 
] 
delimiter = '\r\n' 
 
sock.send(delimiter.join(req_msg)) 
sock.send(delimiter) 
sock.send(delimiter) 
 
print('%sreceived%s' % ('-'*20, '-'*20)) 
http_response = sock.recv(4096) 
print(http_response)

运行上面的代码可以得到下面的输出

--------------------received-------------------- 
HTTP/1.1 200 OK 
Date: Tue, 01 Nov 2016 12:16:53 GMT 
Server: Apache 
Last-Modified: Tue, 12 Jan 2010 13:48:00 GMT 
ETag: "51-47cf7e6ee8400" 
Accept-Ranges: bytes 
Content-Length: 81 
Cache-Control: max-age=86400 
Expires: Wed, 02 Nov 2016 12:16:53 GMT 
Connection: Keep-Alive 
Content-Type: text/html 
 
<html> 
<meta http-equiv="refresh" content="0;url=http://www.baidu.com/"> 
</html>

http_response是通过直接调用recv(4096)得到的，万一真正的返回大于这个值怎么办?我们前面知道了 TCP 协议是面向流的，它本身并不关心消息的内容，需要应用程序自己去界定消息的边界，对于应用层的 HTTP 协议来说，有几种情况，最简单的一种时通过解析返回值头部的Content-Length属性，这样就知道body的大小了，对于 HTTP 1.1版本，支持Transfer-Encoding: chunked传输，对于这种格式，这里不在展开讲解，大家只需要知道， TCP 协议本身无法区分消息体就可以了。对这块感兴趣的可以查看 CPython 核心模块 http.client

Unix_domain_socket

UDS 用于同一机器上不同进程通信的一种机制，其API适用与 network socket 很类似。只是其连接地址为本地文件而已。

代码示例参考：uds_server.py、uds_client.py

ping

ping 命令作为检测网络联通性最常用的工具，其适用的传输协议既不是TCP，也不是 UDP，而是 ICMP，利用 raw sockets，我们可以适用纯 Python 代码来实现其功能。

代码示例参考：ping.py

netstat vs ss

netstat 与 ss 是类 Unix 系统上查看 Socket 信息的命令。netstat 是比较老牌的命令，我常用的选择有

-t，只显示 tcp 连接

-u，只显示 udp 连接

-n，不用解析hostname，用 IP 显示主机，可以加快执行速度

-p，查看连接的进程信息

-l，只显示监听的连接

ss 是新兴的命令，其选项和 netstat 差不多，主要区别是能够进行过滤(通过state与exclude关键字)。

$ ss -o state time-wait -n | head 
Recv-Q Send-Q             Local Address:Port               Peer Address:Port 
0      0                 10.200.181.220:2222              10.200.180.28:12865  timer:(timewait,33sec,0) 
0      0                      127.0.0.1:45977                 127.0.0.1:3306   timer:(timewait,46sec,0) 
0      0                      127.0.0.1:45945                 127.0.0.1:3306   timer:(timewait,6.621ms,0) 
0      0                 10.200.181.220:2222              10.200.180.28:12280  timer:(timewait,12sec,0) 
0      0                 10.200.181.220:2222              10.200.180.28:35045  timer:(timewait,43sec,0) 
0      0                 10.200.181.220:2222              10.200.180.28:42675  timer:(timewait,46sec,0) 
0      0                      127.0.0.1:45949                 127.0.0.1:3306   timer:(timewait,11sec,0) 
0      0                      127.0.0.1:45954                 127.0.0.1:3306   timer:(timewait,21sec,0) 
0      0               ::ffff:127.0.0.1:3306           ::ffff:127.0.0.1:45964  timer:(timewait,31sec,0)

这两个命令更多用法可以参考：

SS Utility: Quick Intro

10 basic examples of linux netstat command

总结

我们的生活已经离不开网络，平时的开发也充斥着各种复杂的网络应用，从最基本的数据库，到各种分布式系统，不论其应用层怎么复杂，其底层传输数据的的协议簇是一致的。Socket 这一概念我们很少直接与其打交道，但是当我们的系统出现问题时，往往是对底层的协议认识不足造成的，希望这篇文章能对大家编程网络方面的程序有所帮助。