How to send multiple files using a single TCP connection in Java?

Use one TCP connection to send multiple files

Why is there this blog? I have been reading some related things recently. There is no problem in simply using Socket for programming, but this only establishes some basic concepts. Still nothing can be done about the real problem.

When I need to transfer files, I find that I seem to have just sent the data (binary data), but some information about the file is lost (the file extension). And I can only use one Socket to send one file each time, and there is no way to send files continuously (because I rely on closing the stream to complete sending files, which means that I actually don’t know the length of the file, so I can only send files as one Socket connection represents a file).

These problems have troubled me for a long time. I went to the Internet to briefly search, but I didn’t find any ready-made examples (maybe I didn’t find them). Someone mentioned that you can define the protocol yourself. to send. This piqued my interest, and I felt like I understood something, because I had just taken the course Computer Network. To be honest, I didn’t learn very well, but I did learn the concept of computer network. .

In the course of computer network, many protocols were mentioned, and I also had the concept of protocols unknowingly. So I found a solution: define a simple protocol on the TCP layer myself. By defining the protocol, the problem is solved.

The role of the protocol

Send data from host 1 to host 2. From the perspective of the application layer, they can only see the application data, but we can see it through the diagram. The data starts from host 1, and a header is added to the data for each lower layer, and then spreads on the network. When it reaches host 2, a header is removed for each upper layer. When it reaches the application layer, there is only data. (This is just a brief explanation. In fact, this is not rigorous enough, but it is enough for a simple understanding.)

So, I can define one myself A simple protocol puts some necessary information in the protocol header, and then lets the computer program parse the protocol header information by itself, and each protocol message is equivalent to a file. In this way, multiple protocols are multiple files. And the protocols can be distinguished. Otherwise, if multiple files are transmitted continuously, the transmission is meaningless if the byte stream belonging to each file cannot be distinguished.

Define the data sending format (protocol)

The sending format here (I feel it is a bit similar to the protocol in the computer network, let’s call it a simple protocol) .

Sending format: Data header Data body

Data header: A data with a length of one byte, indicating the file type. Note: Because the type of each file is different, and the length is also different, we know that the header of the protocol generally has a fixed length (we do not consider those with variable length), so I use a mapping relationship , that is, a byte number represents the type of a file.

Give an example, as follows:

key	value
0	txt
1	png
2	jpg
3	jpeg
4	avi

Note: What I am doing here is a simulation, so I only need to test a few types.

Data body: The data part of the file (binary data).

Code

Client

Protocol header class

package com.dragon;

public class Header {
	private byte type;      //文件类型
	private long length;      //文件长度
	
	public Header(byte type, long length) {
		super();
		this.type = type;
		this.length = length;
	}

	public byte getType() {
		return this.type;
	}
	
	public long getLength() {
		return this.length;
	}
}

Send file class

package com.dragon;

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.net.Socket;

/**
 * 模拟文件传输协议：
 * 协议包含一个头部和一个数据部分。
 * 头部为 9 字节，其余为数据部分。
 * 规定头部包含：文件的类型、文件数据的总长度信息。
 * */
public class FileTransfer {
	private byte[] header = new byte[9];   //协议的头部为9字节，第一个字节为文件类型，后面8个字节为文件的字节长度。
	
	/**
	 *@param src source folder 
	 * @throws IOException 
	 * @throws FileNotFoundException 
	 * */
	public void transfer(Socket client, String src) throws FileNotFoundException, IOException {
		File srcFile = new File(src);
		File[] files = srcFile.listFiles(f->f.isFile());
		//获取输出流
		BufferedOutputStream bos = new BufferedOutputStream(client.getOutputStream());
		for (File file : files) {
			try (BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file))){
				 //将文件写入流中
				String filename = file.getName();
				System.out.println(filename);
				//获取文件的扩展名
				String type = filename.substring(filename.lastIndexOf(".")+1);
				long len = file.length();
				//使用一个对象来保存文件的类型和长度信息，操作方便。
				Header h = new Header(this.getType(type), len);
				header = this.getHeader(h);
				
				//将文件基本信息作为头部写入流中
				bos.write(header, 0, header.length);
				//将文件数据作为数据部分写入流中
				int hasRead = 0;
				byte[] b = new byte[1024];
				while ((hasRead = bis.read(b)) != -1) {
					bos.write(b, 0, hasRead);
				}
				bos.flush();   //强制刷新，否则会出错！
			}
		}
	}
	
	private byte[] getHeader(Header h) {
		byte[] header = new byte[9];
		byte t = h.getType();  
		long v = h.getLength();
		header[0] = t;                  //版本号
		header[1] = (byte)(v >>> 56);   //长度
		header[2] = (byte)(v >>> 48);
		header[3] = (byte)(v >>> 40);
		header[4] = (byte)(v >>> 32);
		header[5] = (byte)(v >>> 24);
		header[6] = (byte)(v >>> 16);
		header[7] = (byte)(v >>>  8);
		header[8] = (byte)(v >>>  0);
		return header;
	}
	
	/**
	 * 使用 0-127 作为类型的代号
	 * */
	private byte getType(String type) {
		byte t = 0;
		switch (type.toLowerCase()) {
		case "txt": t = 0; break;
		case "png": t=1; break;
		case "jpg": t=2; break;
		case "jpeg": t=3; break;
		case "avi": t=4; break;
		}
		return t;
	}
}

Note:

1. After sending a file, you need to force refresh it. Because I am using a buffer stream, we know that in order to improve the efficiency of sending, we do not send data as soon as there is data, but wait until the buffer is full before sending. Because the IO process is very slow (compared to the CPU), so If you do not refresh, when the data volume is very small, the server may not be able to receive the data (, if you are interested, you can learn more about it.), this is a problem that needs attention. . (The example I tested had a text file that was only 31 bytes).

2. getLong() method converts a long type data into byte type data. We know that long occupies 8 bytes, but this method is from the Java source code. Copied from it, there is a class called DataOutputStream. One of its methods is writeLong(). Its underlying implementation is to convert long into byte, so I directly borrowed it. (Actually, this is not very complicated. It only involves bit operations, but writing this code is very powerful, so I chose to use this code directly. If you are interested in bit operations, you can refer to one of my blogs: Bit Operations operation).

Test class

package com.dragon;

import java.io.IOException;
import java.net.Socket;
import java.net.UnknownHostException;

//类型使用代号：固定长度
//文件长度：long->byte 固定长度
public class Test {
	public static void main(String[] args) throws UnknownHostException, IOException {
		FileTransfer fileTransfer = new FileTransfer();
		try (Socket client = new Socket("127.0.0.1", 8000)) {
			fileTransfer.transfer(client, "D:/DBC/src");
		}
	}
}

Server side

Protocol parsing class

package com.dragon;

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.net.Socket;
import java.util.UUID;

/**
 * 接受客户端传过来的文件数据，并将其还原为文件。
 * */
public class FileResolve {
	private byte[] header = new byte[9];  
	
	/**
	 * @param des 输出文件的目录
	 * */
	public void fileResolve(Socket client, String des) throws IOException {
		BufferedInputStream bis = new BufferedInputStream(client.getInputStream());
		File desFile = new File(des);
		if (!desFile.exists()) {
			if (!desFile.mkdirs()) {
				throw new FileNotFoundException("无法创建输出路径");
			}
		}
		
		while (true) {	
			//先读取文件的头部信息
			int exit = bis.read(header, 0, header.length);
			
			//当最后一个文件发送完，客户端会停止，服务器端读取完数据后，就应该关闭了，
			//否则就会造成死循环，并且会批量产生最后一个文件，但是没有任何数据。
			if (exit == -1) {
				System.out.println("文件上传结束！");
				break;   
			}
			
			String type = this.getType(header[0]);
			String filename  = UUID.randomUUID().toString()+"."+type;
			System.out.println(filename);
			//获取文件的长度
			long len = this.getLength(header);
			long count = 0L;
			try (BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(new File(des, filename)))){
				int hasRead = 0;
				byte[] b = new byte[1024];
				while (count < len && (hasRead = bis.read(b)) != -1) {
					bos.write(b, 0, hasRead);
					count += (long)hasRead;
					/**
					 * 当文件最后一部分不足1024时，直接读取此部分，然后结束。
					 * 文件已经读取完成了。
					 * */
					int last = (int)(len-count);
					if (last < 1024 && last > 0) {
						//这里不考虑网络原因造成的无法读取准确的字节数，暂且认为网络是正常的。
						byte[] lastData = new byte[last];
						bis.read(lastData);
						bos.write(lastData, 0, last);
						count += (long)last;
					}
				}
			}
		}
	}
	
	/**
	 * 使用 0-127 作为类型的代号
	 * */
	private String getType(int type) {
		String t = "";
		switch (type) {
		case 0: t = "txt"; break;
		case 1: t = "png"; break;
		case 2: t = "jpg"; break;
		case 3: t = "jpeg"; break;
		case 4: t = "avi"; break;
		}
		return t;
	}
	
	private long getLength(byte[] h) {
		return (((long)h[1] << 56) +
                ((long)(h[2] & 255) << 48) +
                ((long)(h[3] & 255) << 40) +
                ((long)(h[4] & 255) << 32) +
                ((long)(h[5] & 255) << 24) +
                ((h[6] & 255) << 16) +
                ((h[7] & 255) <<  8) +
                ((h[8] & 255) <<  0));
	}
}

Note:

1. I believe everyone can guess this method of converting byte to long. DataInputStream has a method called readLong(), so I used it directly. (I think these two pieces of code are very well written, but I just looked at the source code of a few classes, haha!)

2. Here I use an infinite loop to read the file. But when I was testing, I found a problem that was difficult to solve: When to end the loop. I initially used client shutdown as the exit condition, but found that it didn't work. Later it was discovered that for network streams, if -1 is read, it means that the opposite input stream has been closed, so this is used as a sign to exit the loop. If this code is deleted, the program will not terminate automatically, and the last read file will always be generated. However, since the data cannot be read, the files are all 0-byte files. (This thing generates files very quickly. It will generate thousands of files in about a few seconds. If you are interested, you can try it, but it is best to terminate the program quickly, haha! )

if (exit == -1) {
	System.out.println("文件上传结束！");
	break;   
}

Test Class

Just test one connection here. This is just an example.

package com.dragon;

import java.io.IOException;
import java.net.ServerSocket;
import java.net.Socket;

public class Test {
	public static void main(String[] args) throws IOException {
		try (ServerSocket server = new ServerSocket(8000)){
			Socket client = server.accept();
			FileResolve fileResolve = new FileResolve();
			fileResolve.fileResolve(client, "D:/DBC/des");
		}	
	}
}

Test results

Client

##Server

Source file directory This contains the five files I tested. Pay attention to comparing the size information of the file. For IO testing, I like to use picture and video testing, because they are very special files. If there is a slight error (less or more bytes), the file will basically be damaged, and the performance The picture is not displayed properly and the video cannot be played normally.

Destination file directory

The above is the detailed content of How to send multiple files using a single TCP connection in Java?. For more information, please follow other related articles on the PHP Chinese website!