Instrument and equipment modification technology to realize the function of uploading measurement data to the server

How to use C# to modify the equipment of the instrument? Because the instrument control program is developed in C#, it is best for the client to be C#. Considering saving traffic (the server is charged based on traffic), the files must be compressed, and the file compression function must be implemented under C#. The server chooses Java to build a restful API for uploading.

1. Project requirements and analysis

1. According to the leader’s requirements, an instrument must be modified, add some functions, and upload measurement data to the server. The measurement cycle of the instrument is about 20 seconds, and the amount of data is currently not large. Each measurement is about less than 2M, and they are all floating-point data and integer data.

2. At first I wanted to use TCP long connection, but considering the on-site environment. In a typical manufacturing workshop, the electromagnetic environment is complex and the network signal is unstable, so TCP long connection implementation is not considered. Short connections are also not considered. In the future, when the number of instruments increases, the cost of frequently establishing connections will also be very high, and the server may not be able to bear it (Alibaba Cloud's beggar version). So I chose to use restful for submission, and http communication can be scheduled with multi-threads.

3. The instrument control program is developed in C#, so the client is best in C#. I want to use springboot on the server side, which is very convenient.

4. Considering that the new upload function cannot affect the previous measurement beat, multi-threading is required to implement it. Unfortunately, I'm lazy and don't want to consider thread coordination issues, so I finally chose message queue implementation.

5. Considering saving traffic (the server is charged based on traffic), the files must be compressed, and the file compression function must be implemented under C#.

6. Read data from the measurement file, store the parameters in the database, and package the original measurement data on the file server.

2. Overall architecture and technical solution

The final technical solution is to use C# as the client and Java to build the server-side restful API for uploading

The overall architecture is as follows:

The technology used is as follows:

1. C#’s Restful client: RestSharp

2. java's Restful server: springboot

3. C#-side message queue: NetMQ

4. C#-side zip operation component ：DotNetZip

5. Java-side zip operation component: Apache Commons Compress

3. Server

The server uses springboot restful, POST method, very simple.
The MultipartFile method is used to transfer files, because the client's ResrSharp can only transfer files in this way

@RestController
@RequestMapping(value = "upload")
public class FileRestController {
    Logger logger = LogManager.getLogger(FileRestController.class);

    @RequestMapping(value = "file", method = RequestMethod.POST)
    public
    @ResponseBody
    RestResult getZipFile(@RequestParam("file") MultipartFile file) throws IOException, URISyntaxException {
        RestResult result = new RestResult();
        if (!file.getName().isEmpty()) {
            InputStream stream = file.getInputStream();
//            String directory = FileRestController.class.getProtectionDomain().getCodeSource().getLocation().toURI().getPath();
            String directory = "/usr/local/haliang/files/";
            try {
                directory = URLDecoder.decode(directory, "utf-8");
            } catch (java.io.UnsupportedEncodingException e) {
                return null;
            }
            FileOutputStream fs = new FileOutputStream(directory + file.getOriginalFilename());
            logger.info("文件所在的目录:   " + directory + "/files/" + file.getOriginalFilename());
            byte[] buffer = new byte[1024 * 1024];
            int bytesum = 0;
            int byteread = 0;
            while ((byteread = stream.read(buffer)) != -1) {
                bytesum += byteread;
                fs.write(buffer, 0, byteread);
                fs.flush();
            }
            fs.close();
            stream.close();
            logger.info("成功接收文件:   " + directory + file.getOriginalFilename());
        }

        return result;
    }
}

4. Client

The client architecture is as follows:

1 Introduction to zeromq

NetMQ is the C# ported version of ZeroMQ.

1.1: What is zeromq

NetMQ (ZeroMQ to .Net), ZMQ is known as the fastest middleware in history.
It encapsulates socket communication so that we can complete complex network communication without writing socket function calls.
The difference between it and Socket is: ordinary socket is end-to-end (1:1 relationship), but ZMQ can have N:M relationship. People know more about BSD sockets as point-to-point. Connection, point-to-point connection requires explicit establishment of connection, destruction of connection, selection of protocol (TCP/UDP) and handling of errors, etc. ZMQ shields these details, making your network programming simpler.
It is a message processing queue library that elastically scales across multiple threads, cores, and host boxes. Different from the general message queue products, it does not have a message queue server, but is more like a network communication library. From the perspective of network communication, it is above the session layer and below the application layer, and belongs to the transport layer.

1.2: zeromq’s message model

zeromq divides message communication into four models, namely one-to-one pair model (Exclusive-Pair), request response model (Request-Reply), Publish-Subscribe model (Publish-Subscribe), push-pull model (Push-Pull). These four models summarize the general network communication model. In practice, you can combine two or more of these models to form your own solution according to application needs.

1.2.1 One-to-one pairing model Exclusive-Pair

The simplest 1:1 message communication model, used to support the traditional TCP socket model, mainly used for inter-thread communication within the process . It can be considered as a TCP Connection, but TCP Server can only accept one connection. It is implemented using lock free and is very fast. Data can flow in both directions, which is different from the later request-response model. (Not recommended, no examples)

1.2.2 请求回应模型 Request-Reply

由请求端发起请求，然后等待回应端应答。一个请求必须对应一个回应，从请求端的角度来看是发-收配对，从回应端的角度是收-发对。跟一对一结对模型的区别在于请求端可以是1~N个。
请求端和回应端都可以是1：N的模型。通常把1认为是server,N认为是Client。ZeroMQ可以很好的支持路由功能（实现路由功能的组件叫作Device),把1：N扩展为N:M(只需要加入若干路由节点）。从这个模型看，更底层的端点地址是对上层隐藏的。每个请求都隐含有回应地址，而应用则不关心它。通常把该模型主要用于远程调用及任务分配等。
(NetMQ请求响应C#调用案例)

1.2.3 发布订阅模型 Publisher-Subscriber(本项目采用的模型)

发布端单向分发数据，且不关心是否把全部信息发送给订阅端。如果发布端开始发布信息时，订阅端尚未连接上来，则这些信息会被直接丢弃。订阅端未连接导致信息丢失的问题，可以通过与请求回应模型组合来解决。订阅端只负责接收，而不能反馈，且在订阅端消费速度慢于发布端的情况下，会在订阅端堆积数据。该模型主要用于数据分发。天气预报、微博明星粉丝可以应用这种经典模型。 (NetMQ发布订阅模式C#调用案例)

1.2.4 推拉模型 Push-Pull

Server端作为Push端，而Client端作为Pull端，如果有多个Client端同时连接到Server端，则Server端会在内部做一个负载均衡，采用平均分配的算法，将所有消息均衡发布到Client端上。与发布订阅模型相比，推拉模型在没有消费者的情况下，发布的消息不会被消耗掉；在消费者能力不够的情况下，能够提供多消费者并行消费解决方案。该模型主要用于多任务并行。
(NetMQ推拉模式C#调用案例)

1.3：zeromq的优势

1. TCP：ZeroMQ基于消息，消息模式，而非字节流。

2. XMPP：ZeroMQ更简单、快速、更底层。Jabber可建在ZeroMQ之上。

3. AMQP：完成相同的工作，ZeroMQ要快100倍，而且不需要代理（规范更简洁——少278页）

4. IPC：ZeroMQ可以跨多个主机盒，而非单台机器。

5. CORBA：ZeroMQ不会将复杂到恐怖的消息格式强加于你。

6. RPC：ZeroMQ完全是异步的，你可以随时增加/删除参与者。

7. RFC 1149：ZeroMQ比它快多了！

8. 29west LBM：ZeroMQ是自由软件！

9. IBM低延迟：ZeroMQ是自由软件！

10. Tibco：仍然是自由软件！

2.代码实现

2.1 Publisher(发布者)

一般都是发布者先启动，绑定监听端口。封装了一个发送函数，主要是发送原先软件生成测量文件的路径。

public class Publisher
    {
        public int Port { get; set; }
        private PublisherSocket socket;

        /// <summary>
        /// 构造函数
        /// </summary>
        /// <param name="port">绑定的端口</param>
        public Publisher(int port)
        {
            Port = port;
        }

        /// <summary>
        /// 启动发布端
        /// </summary>
        public void Start()
        {
            NetMQContext context = NetMQContext.Create();
            this.socket = context.CreatePublisherSocket();
            this.socket.Bind("tcp://127.0.0.1:" + Port);
        }

        /// <summary>
        /// 发送数据
        /// </summary>
        /// <param name="result"></param>
        public void Send(string result)
        {
            socket.SendFrame(result);
        }
    }

2.2 Subscriber(订阅者)

订阅者启动时候连接端口。防止线程阻塞，订阅者是新开一个线程运行的。

public class Subscribe
    {
        private delegate void GetDataHandler(string message);

        private event GetDataHandler onGetData;
        public int Port { get; set; }
        public string TempDirectory { get; set; }
        public bool isRunning { get; set; }
        public string domain { get; set; }

        public Subscribe(int port, string domain)
        {
            Port = port;
            this.domain = domain;
            onGetData += ProcessData;
        }

        private SubscriberSocket socket;

        public void Start()
        {
            this.isRunning = true;
            NetMQContext context = NetMQContext.Create();
            socket = context.CreateSubscriberSocket();
            socket.Connect("tcp://127.0.0.1:" + Port);
            socket.Subscribe("");
            Thread t = new Thread(new ThreadStart(StartSub));
            t.Start();
        }

        private void StartSub()
        {
            while (isRunning)
            {
                Thread.Sleep(10000);
                string result = socket.ReceiveFrameString(Encoding.UTF8);
                onGetData(result);
            }
        }

        private void ProcessData(string path)
        {
            Console.WriteLine("收到文件：" + path);
            string compressedFile = Compress.CompressFile(TempDirectory, path);
            new RestPost(domain).Post(compressedFile);
        }

3 客户端压缩

压缩使用DotNetZip组件，非常简单好用。

 public class Compress
    {
        public static string CompressFile(string temp,string txtPath)
        {
            string txtFileName = System.IO.Path.GetFileNameWithoutExtension(txtPath);
            string compressedFileName = temp+"/"+txtFileName + ".zip";
            ZipFile file=new ZipFile();
            file.AddFile(txtPath,"");
            file.Save(compressedFileName);
            return compressedFileName;
        }
    }

4 客户端上传

使用RestSharp组件，也是非常简单。异步回调，不影响性能。

public class RestPost
    {
        public string Domain { get; set; }

        public RestPost(string domain)
        {
            Domain = domain;
        }

        public void Post(string path)
        {
            RestRequest request = new RestRequest(Method.POST);
            request.AddFile("file", path);
            RestClient client = new RestClient {BaseUrl = new Uri("http://" + Domain + "/upload/file")};
            client.ExecuteAsync(request, (response) =>
                {
                    if (response.StatusCode == HttpStatusCode.OK)
                    {
                        Console.WriteLine("上传成功...\n" + response.Content);
                    }
                    else
                    {
                        Console.WriteLine($"出错啦:{response.Content}");
                    }
                }
            );
        }
    }

五、总结

1. 写代码之前一定要搞清楚需求，设计好架构

2. 注意消息队列启动时候的线程问题

3. 异步执行

相关文章：

企业数据备份技术

相关视频：

数据结构探险—队列篇

The above is the detailed content of Instrument and equipment modification technology to realize the function of uploading measurement data to the server. For more information, please follow other related articles on the PHP Chinese website!