Let computing power no longer become a bottleneck, Xiaohongshu machine learning heterogeneous hardware inference optimization method

Many companies are combining the development of GPU computing power to explore solutions to machine learning problems that are suitable for them. For example, Xiaohongshu will begin GPU-based transformation of the promotion search model in 2021 to improve inference performance and efficiency. During the migration process, we also faced some difficulties, such as how to smoothly migrate to heterogeneous hardware, how to develop our own solutions based on Xiaohongshu's business scenarios and online architecture, etc. Under the global trend of cost reduction and efficiency improvement, heterogeneous computing has become a promising direction, which can improve computing performance by combining different types of processors (such as CPU, GPU, FPGA, etc.) to achieve Better efficiency and lower costs.

1. Background

The model services of Xiaohongshu recommendation, advertising, search and other main scenarios are uniformly carried by the mid-stage inference architecture. With the continuous development of Xiaohongshu's business, the scale of models for scenarios such as promotional search is also increasing. Taking the main model of refined recommendation scenarios as an example, since the beginning of 2020, the algorithm has launched full-interest modeling, and the average length of user historical behavior records has expanded by about 100 times. The model structure has also gone through multiple rounds of iterations from the initial muti-task, and the complexity of the model structure has also continued to increase. These changes have resulted in a 30-fold increase in the number of floating-point operations for model inference and an approximately 5-fold increase in model memory access.

Picture

2. Model service architecture overview

Model Features: Take Xiaohongshu’s recommended main model at the end of 2022 as an example. This model is fully sparse. Part of the structure is composed of continuous value features and matrix operations. There are also large-scale sparse parameters such as , the sparse features of a single model are up to 1TB, but through relatively effective model structure optimization, the dense part is controlled within 10GB and can be placed in the video memory. Every time the user swipes Xiaohongshu, the total FLOPs calculated reach 40B, and the timeout is controlled within 300ms (excluding feature processing, with lookup).

Inference framework: Before 2020, Xiaohongshu adopted the TensorFlow Serving framework as the online service framework. After 2020, it gradually iterated into a self-developed one based on TensorFlowCore Lambda Service service. TensorFlow Serving performs a memory copy of TensorProto -> CTensor before entering the graph to ensure the correctness and reliability of model inference. However, as the business scale expands, memory copy operations will have an impact on model performance. Xiaohongshu's self-developed framework eliminates unnecessary copying through optimization, while retaining the pluggable features of runtime, graph scheduling capabilities, and optimization capabilities, and lays the foundation for the later use of different optimization frameworks such as TRT, BLADE, and TVM. . It now seems that choosing self-research at the right time is a wise choice. At the same time, in order to minimize the cost of data transmission, the inference framework also undertakes part of the implementation of feature extraction and transformation. Here Xiaohongshu is still estimating Self-developed edge storage is deployed on the near side of the service, which solves the cost problem of pulling data remotely.

Model characteristics: Xiaohongshu does not build its own computer room. All machines are purchased from cloud vendors. Therefore, the decision to choose different models depends largely on What type of machines can be purchased? The calculation of model inference is not pure GPU calculation. To find a reasonable hardware ratio, in addition to considering GPU\CPU, it also involves bandwidth, memory bandwidth, cross-numa communication delay and other issues.

Picture

GPU Features

GPU Features:Here, Xiaohong The problems encountered by this book are the same as those encountered by other companies. The execution of GPU kernel can be divided into the following stages: data transmission, kernel startup, kernel calculation and result transmission. Among them, data transmission is to transfer data from the host memory to the GPU memory; kernel startup is to transfer the kernel code from the host side to the GPU side, and start the kernel on the GPU; kernel calculation is to actually execute the kernel code calculation result; result transmission is to Computational results are transferred from GPU memory back to host memory. If a large amount of time is spent on data transmission and kernel startup, and the work delivered to the kernel for calculation is not heavy, and the actual calculation time is very short, the GPU utilization will not be improved, and even empty running will occur.

Picture

Estimated Service Framework

3.GPU optimization practice

3.1 System optimization

3.1.1 Physical machine

In physical machine optimization In terms of performance, some conventional optimization ideas can be adopted. The main purpose is to reduce the cost of other system overheads other than the GPU and reduce the price difference earned by virtualization middlemen. Generally speaking, a set of system optimization can improve performance by 1%-2%. From our practice, optimization needs to be combined with the actual capabilities of cloud vendors.

● Interrupt isolation: Isolate GPU interrupts to avoid interrupts from other devices affecting GPU computing performance.

● Kernel version upgrade: Improve system stability and security, improve GPU driver compatibility and performance.

● Instruction transparent transmission: Transparently transmit GPU instructions directly to the physical device to accelerate the computing speed of the GPU.

3.1.2 Virtualization and Containers

In the case of multiple cards, bind a single pod to a specific NUMA node, thereby Improve data transfer speed between CPU and GPU.

● CPU NUMA Affinity, affinity refers to which memory accesses are faster and have lower latency from a CPU perspective. As mentioned before, local memory connected directly to the CPU is faster. Therefore, the operating system can allocate local memory according to the CPU where the task is located to improve access speed and performance. This is based on CPU NUMA Affinity considerations and try to run the task in the local NUMA Node. In the Xiaohongshu scenario, the memory access overhead on the CPU is not small. Allowing the CPU to directly connect to local memory can save a lot of time spent on kernel execution on the CPU, leaving enough space for the GPU.

● By controlling the CPU usage at 70%, the delay can be reduced from 200ms -> 150ms.

3.1.3 Mirror

Compilation optimization. Different CPUs have different support capabilities for instruction levels, and the models purchased by different cloud vendors are also different. A relatively simple idea is to compile the image with different instruction sets in different hardware scenarios. When implementing operators, a large number of operators already have instructions such as AVX512. Taking Alibaba Cloud's Intel(R) With optimization, the CPU throughput on this model increased by 10%.

# Intel(R) Xeon(R) Platinum 8163 for ali intelbuild:intel --copt=-march=skylake-avx512 --copt=-mmmx --copt=-mno-3dnow --copt=-mssebuild:intel --copt=-msse2 --copt=-msse3 --copt=-mssse3 --copt=-mno-sse4a --copt=-mcx16build:intel --copt=-msahf --copt=-mmovbe --copt=-maes --copt=-mno-sha --copt=-mpclmulbuild:intel --copt=-mpopcnt --copt=-mabm --copt=-mno-lwp --copt=-mfma --copt=-mno-fma4build:intel --copt=-mno-xop --copt=-mbmi --copt=-mno-sgx --copt=-mbmi2 --copt=-mno-pconfigbuild:intel --copt=-mno-wbnoinvd --copt=-mno-tbm --copt=-mavx --copt=-mavx2 --copt=-msse4.2build:intel --copt=-msse4.1 --copt=-mlzcnt --copt=-mrtm --copt=-mhle --copt=-mrdrnd --copt=-mf16cbuild:intel --copt=-mfsgsbase --copt=-mrdseed --copt=-mprfchw --copt=-madx --copt=-mfxsrbuild:intel --copt=-mxsave --copt=-mxsaveopt --copt=-mavx512f --copt=-mno-avx512erbuild:intel --copt=-mavx512cd --copt=-mno-avx512pf --copt=-mno-prefetchwt1build:intel --copt=-mno-clflushopt --copt=-mxsavec --copt=-mxsavesbuild:intel --copt=-mavx512dq --copt=-mavx512bw --copt=-mavx512vl --copt=-mno-avx512ifmabuild:intel --copt=-mno-avx512vbmi --copt=-mno-avx5124fmaps --copt=-mno-avx5124vnniwbuild:intel --copt=-mno-clwb --copt=-mno-mwaitx --copt=-mno-clzero --copt=-mno-pkubuild:intel --copt=-mno-rdpid --copt=-mno-gfni --copt=-mno-shstk --copt=-mno-avx512vbmi2build:intel --copt=-mavx512vnni --copt=-mno-vaes --copt=-mno-vpclmulqdq --copt=-mno-avx512bitalgbuild:intel --copt=-mno-movdiri --copt=-mno-movdir64b --copt=-mtune=skylake-avx512

3.2 Computation optimization

3.2.1 Make full use of computing power

##● Computation optimization , first of all, you need to fully understand the hardware performance and understand it thoroughly. In the Xiaohongshu scenario, as shown in the figure below, we encountered two core problems:

1. There are many memory accesses on the CPU, and the frequency of memory page faults is high, resulting in a waste of CPU resources, and Request latency is too high

2. In online inference services, calculations usually have two characteristics: the batch size of a single request is small, and the concurrency scale of a single service is large. A small batch size will cause the kernel to be unable to fully utilize the computing power of the GPU. The GPU kernel execution time is generally shorter and cannot fully cover up the overhead of kernel launch. The kernel launch time may even be longer than the kernel execution time. In TensorFlow, a single Cuda Stream launch kernel becomes a bottleneck, resulting in only 50% GPU utilization in inference scenarios. In addition, for small model scenarios (simple dense networks), it is not cost-effective to replace CPU with GPU, which limits the complexity of the model.

Picture

● To solve the above two problems, we have taken the following measures:

1. To solve the problem of high memory page fault frequency, we use the jemalloc library to optimize the memory recycling mechanism and enable the transparent huge page function of the operating system. In addition, for the special memory access characteristics of lambda, we design special data structures and optimize memory allocation strategies to avoid memory fragmentation as much as possible. At the same time, we directly bypassed the tf_serving interface and directly called TensorFlow, reducing the serialization and deserialization of data. These optimizations have increased throughput by 10% in homepage and in-stream fine-tuning scenarios, and reduced latency by 50% in most advertising scenarios.

Picture

Compatible with tensorflow::Tensor format, zero copy is made before passing features to tensorflow::SessionRun

2. In response to the problem of TensorFlow's single Cuda Stream, we support the functions of Multi Streams and Multi Contexts, avoiding the performance bottleneck caused by mutex locks, and successfully increasing the GPU utilization to 90%. At the same time, we use the Cuda MPS function provided by Nvidia to realize spatial division multiplexing of the GPU (supporting multiple kernel executions at the same time), further improving the utilization of the GPU. Based on this, Search’s ranking model was successfully implemented on GPU. In addition, we have also successfully implemented it in other business lines, including home page layout, advertising, etc. The following table is an optimization situation in the search ranking scenario.

Picture

3. Op/Kernel fusion technology: generate higher-performance Tensorflow operators through handwriting or graph compilation and optimization tools, making full use of the CPU Cache and GPU Shared Memory improve system throughput.

Picture

In the inflow scenario, the operators are fused, and you can see that a single call is 12ms -> 5ms

3.2.2 Avoiding waste of computing power

1. There is room for optimization on the system link

a. Before the initial ranking Setup calculation: When processing related calculations on the user side, a large number of notes need to be calculated for preliminary sorting. For example, taking outflow as an example, about 5,000 notes need to be calculated, and lambda has slicing processing for them. In order to avoid repeated calculations, the user-side calculations of the initial row are moved forward in parallel with the recall phase, so that the calculation of the user vector is reduced from multiple repetitions to only one time, 40% of the machines are optimized in the rough row scenario.

2. In-graph training to the inference process:

a. Calculation pre-processing: Part of the calculation can be processed in advance through graph freeze. When reasoning, there is no need to repeat calculations.

b. Output model freeze optimization: When the model is output, all parameters are generated together with the graph itself to generate a frozen graph (frozen graph) and perform preprocessing calculations. Many precalculated Variable operators can be converted into Const operator (GPU usage decreased by 12%)

c. Merged calculation in inference scenario: Each batch contains only one user, that is, there is a large number of repeated calculations on the user side, and there is the possibility of merging

d. CPU/GPU operator split: move all operators after lookup to GPU, avoiding data copy between CPU and GPU

e. GPU to CPU data copy: transfer data Pack one copy‎⁣

f. BilinearNet operator GPU cuda implementation: accelerate calculation through GPU to improve performance‎⁣

g. Some operators GPU-based: Eliminate CPU -> GPU copy‎⁣

h. BatchNorm & MLP merge: By implementing a new MLP layer, reduce the number of GPU passes (N -> 1 ), increase the amount of calculation for one calculation (reuse the concurrency capability of the GPU small core)‎⁣

Picture

3.2.3 Dynamic computing power throughout the day

● Dynamic computing downgrade improves resource usage efficiency throughout the day, and automatically adjusts the negative feedback of lambda load at the second level to achieve correct There is no need to manually prepare for downgrade before single-zone stress testing.

● Major business scenarios such as outbound refined sorting, outbound preliminary sorting, inbound refined sorting, internal inflow preliminary sorting, and search have all been launched.

● Solved the capacity problem in multiple business lines, effectively alleviated the linear increase in resources caused by business growth, and greatly improved the robustness of the system. In the business lines after the function was launched, there were no P3 or above accidents caused by a sharp drop in the instantaneous success rate.

● Greatly improve the efficiency of resource usage throughout the day. Taking instream fine-tuning as an example (as shown in the figure below), the number of CPU cores used during the three-day May Day holiday from 10:00-24:00 remains at 50 cores. A flat line (jitter corresponds to the release version)

Picture

##3.2.4 Change to better hardware

● The performance of the A10 GPU is 1.5 times that of the T4 GPU. At the same time, the A10 model is equipped with a newer generation CPU (icelake, 10nm) than the T4 model (skylake, 14nm), and the price is only that of the T4 model. 1.2 times of type. We will also consider using models such as the A30 online in the future.

3.3 Graph optimization

Picture

3.3.1 Automatic DL stack Compilation optimization

● BladeDISC is Alibaba’s latest open source dynamic shape deep learning compiler based on MLIR. Xiaohongshu’s automatic graph optimization part comes from this framework (Blade inference acceleration library is Apache 2.0 open source and can be used across any cloud. intellectual property risks). This framework provides TF graph compilation optimization (including Dynamic Shape Compiler, sparse subgraph optimization), and can also superimpose our own customized operator optimization, which can better adapt to our business scenarios. In the stress test single-machine inference, QPS can be increased by 20%.

● Key technologies of this framework

(1) MLIR infrastructure

MLIR, Multi-Level Intermediate Representation ), is an open source project initiated by Google. Its purpose is to provide a flexible, extensible multi-tier IR infrastructure and compiler utility library, providing a unified framework for developers of compilers and language tools.

The design of MLIR is influenced by LLVM, but unlike LLVM, MLIR mainly focuses on the design and extension of intermediate representation (IR). MLIR provides a multi-level IR design that can support the compilation process from high-level languages ​​to low-level hardware, and provides rich infrastructure support and modular design architecture, allowing developers to easily expand the functions of MLIR. In addition, MLIR also has strong glue capabilities and can be integrated with different programming languages ​​and tools. MLIR is a powerful compiler infrastructure and tool library that provides developers of compilers and language tools with a unified and flexible intermediate representation language that can facilitate compilation optimization and code generation.

(2) Dynamic shape compilation

The limitations of static shape mean that the shape of each input and output needs to be determined in advance when writing a deep learning model, and cannot be Change them at runtime. This limits the flexibility and scalability of deep learning models, thus requiring a deep learning compiler that supports dynamic shapes.

3.3.2 Accuracy adjustment

● One of the ways to achieve quantization is touse FP16

FP16 calculation optimization: Replacing FP32 calculations with FP16 in the MLP layer can greatly reduce GPU usage (a relative decrease of 13%)

In the process of adjusting FP16, choosing the white box method for precision optimization means You can have more granular control over which layers use low-precision calculations, and be able to continuously adjust and optimize based on experience. This method requires a relatively in-depth understanding and analysis of the model structure, and targeted adjustments can be made according to the characteristics and calculation requirements of the model to achieve a higher cost performance.

In contrast, the black box method is relatively simple. It does not need to understand the internal structure of the model. It only needs to set a certain tolerance threshold to complete accuracy optimization. The advantage of this method is that it is simple to operate and has relatively low requirements on model students, but it may sacrifice certain performance and accuracy.

Therefore, whether to choose white box or black box method for accuracy optimization needs to be determined according to the specific situation. If you need to pursue higher performance and accuracy, and have sufficient experience and technical capabilities, then the white-box approach may be more suitable. If simplicity of operation and rapid iteration are more important, then the black box approach may be more practical.

4. Summary

From the beginning of 2021 to the end of 2022, through the optimization of this project, Xiaohongshu’s inference computing power has increased by 30 times, key user indicators have increased by 10%, and at the same time, cumulative cluster savings have been achieved Resources 50%. In our opinion, Xiaohongshu’s development path in AI technology should be oriented by business needs and balance the development of technology and business: while achieving technological innovation, cost, efficiency and sustainability must also be considered. The following are some thoughts during the optimization process:

Optimize the algorithm and improve system performance. This is the core mission of the Xiaohongshu machine learning team. Optimizing algorithms and improving systematization can better support business needs and improve user experience. However, when resources are limited, the team needs to clarify the focus of optimization and avoid over-optimization.

Build infrastructure and improve data processing capabilities. Infrastructure is critical to supporting AI applications. Xiaohongshu can consider further investing in infrastructure construction, including computing and storage capabilities, data centers and network architecture. In addition, it is also very important to improve data processing capabilities to better support machine learning and data science applications.

Improve team talent density and organizational structure. An excellent machine learning team needs talents with different skills and backgrounds, including data scientists, algorithm engineers, software engineers, etc.; optimizing the organizational structure can also help improve team efficiency and innovation capabilities.

Win-win cooperation and open innovation. Xiaohongshu continues to cooperate with other companies, academic institutions and open source communities to jointly promote the development of AI technology, which helps Xiaohongshu obtain more resources and knowledge and become a more open and innovative organization.

This solution brings Xiaohongshu’s machine learning architecture to the top level in the industry. In the future, we will continue to promote engine upgrades, reduce costs and increase efficiency, introduce new technologies to improve the productivity of Xiaohongshu's machine learning, and further integrate Xiaohongshu's actual business scenarios, upgrading from single-module optimization to full-system optimization, and further Introduce the personalized differential characteristics of business-side traffic to achieve the ultimate cost reduction and efficiency increase. We are looking forward to people with lofty ideals to join us!

5. Team

Zhang Chulan (Du Zeyu): Business Technology Department

graduated from East China Normal University and is in charge of the commercialization engine team person, mainly responsible for building commercial online services.

Lu Guang (Peng Peng): Intelligent Distribution Department

graduated from Shanghai Jiao Tong University and is a machine learning engine engineer, mainly responsible for Lambda GPU optimization.

Ian (Chen Jianxin): Intelligent Distribution Department

graduated from Beijing University of Posts and Telecommunications and is a machine learning engine engineer. He is mainly responsible for Lambda parameter server and GPU optimization.

Akabane (Liu Zhaoyu): Intelligent Distribution Department

graduated from Tsinghua University and is a machine learning engine engineer. He is mainly responsible for related research and exploration in the direction of feature engines.

Special thanks to: All students in the Intelligent Distribution Department

The above is the detailed content of Let computing power no longer become a bottleneck, Xiaohongshu machine learning heterogeneous hardware inference optimization method. For more information, please follow other related articles on the PHP Chinese website!