首次將「教導主任」引入模型蒸餾，大規模壓縮優於24種SOTA方法

面對越來越深的深度學習模型和大量的影片大數據，人工智慧演算法對運算資源的依賴越來越高。為了有效提升深度模型的性能和效率，透過探索模型的可蒸餾性和可稀疏性，本文提出了一種基於 “教導主任 - 教師 - 學生” 模式的統一的模型壓縮技術。

此成果由人民中科和中科院自動化所聯合研究團隊合作完成，相關論文發表在人工智慧頂級國際期刊IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 上。此成果是首次將 “教導主任” 角色引入模型蒸餾技術，對深度模型的蒸餾與裁剪進行了統一。

#論文網址：https://ieeexplore.ieee.org/abstract/document/9804342

目前這項成果已應用於人民中科自主研發的跨模態智慧搜尋引擎「白澤」。 「白澤」 打破圖文音視等不同模態間訊息表達的隔閡，將文字、圖片、語音和視頻等不同模態訊息映射到一個統一特徵表示空間，以視頻為核心，學習多個模態間統一的距離測量，跨越文字、語音、視訊等多模態內容的語意鴻溝，實現大一統的搜尋能力。

然而面對海量的網路數據尤其是影片大數據，跨模態的深度模型對運算資源的消耗逐漸提升。基於此項研究成果，「白澤」能夠在保證演算法效能的情況下，將模型大小進行大規模壓縮，從而實現高通量低功耗的跨模態智慧理解和搜尋能力。根據初步的實際應用情況來看，此項技術能夠將大模型的參數規模壓縮平均四倍以上。一方面能夠大幅降低模型對 GPU 伺服器等高效能運算資源的消耗，另一方面能夠將無法在邊緣端部署的大模型經過蒸餾壓縮後實現邊緣端的低功耗部署。

模型壓縮的聯合學習框架

深度演算法模型的壓縮和加速可透過蒸餾學習或結構化稀疏裁剪實現，但這兩個領域均存在一些限制。對於蒸餾學習方法，旨在訓練一個輕量化模型（即學生網路）來模擬複雜龐大的模型（即教師網路）。在教師網絡的指導下，學生網絡可以獲得比單獨訓練的更優效能。

然而，蒸餾學習演算法只專注於提升學生網路的效能，往往忽略了網路結構的重要性。學生網絡的結構一般是預先定義好的，並且在訓練過程中是固定的。

對於結構化稀疏裁剪或濾波器裁剪，這些方法旨在將一個冗餘繁雜的網路裁剪成一個稀疏緊緻的網路。然而，模型裁剪僅用於獲得一個緊緻的結構。目前已有方法都沒有充分利用原始複雜模型所包含的「知識」。近期研究為了平衡模型性能和大小，將蒸餾學習和結構化稀疏裁剪進行結合。但是這些方法僅限於簡單的損失函數的結合。

為了深入分析以上問題，研究首先對模型進行基於壓縮感知訓練，透過分析模型性能和結構發現，對於深度演算法模型，存在兩個重要屬性：可蒸餾性（distillability）和可稀疏性（sparsability）。

具體而言，可蒸餾性指的是能夠從教師網路中蒸餾出有效知識的密度。它可以透過學生網路在教師網路指導下所獲得的績效效益來衡量。例如，擁有更高可蒸餾性的學生網路可以獲得更高效能。可蒸餾性也可以在網路層層級上被定量分析。

As shown in Figure 1-(a), the bar graph represents the cosine similarity (Cosine Similarity) between the distillation learning loss gradient and the true value classification loss gradient. A larger cosine similarity indicates that the knowledge of the current distillation is more helpful for model performance. In this way, cosine similarity can also be a measure of distillability. It can be seen from Figure 1-(a) that the distillability gradually increases as the number of model layers becomes deeper. This also explains why supervision commonly used in distillation learning is applied in the last few layers of the model. Moreover, in different training rounds, the student model also has different distillability, because the cosine similarity also changes as the training time changes. Therefore, it is necessary to dynamically analyze the distillability of different layers during the training process.

On the other hand, sparsity refers to the cropping rate (or compression rate) that the model can obtain under limited accuracy loss. Higher sparsability corresponds to the potential for higher cropping rates. As shown in Figure 1-(b), different layers or modules of the network exhibit different sparsibility. Similar to distillability, sparsibility can also be analyzed at the network layer level and in the time dimension. However, there are currently no methods to explore and analyze distillability and rarefaction. Existing methods often use a fixed training mechanism, which makes it difficult to achieve an optimal result.

Figure 1 Schematic diagram of distillability and sparsity of deep neural networks

In order to solve the above problems, this study analyzes the training process of model compression to obtain relevant findings about distillability and sparsability. Inspired by these findings, this study proposes a model compression method based on joint learning of dynamic distillability and sparsity. It can dynamically combine distillation learning and structured sparse clipping, and adaptively adjust the joint training mechanism by learning distillability and sparsity.

Different from the conventional "Teacher-Student" framework, the method proposed in this article can be described as a "Learning-in-School" framework. Because it contains three major modules: teacher network, student network and dean network.

Specifically, the same as before, the teacher network teaches the student network. The teaching director network is responsible for controlling the intensity of students' online learning and the way they learn. By obtaining the status of the current teacher network and student network, the dean network can evaluate the distillability and sparsibility of the current student network, and then dynamically balance and control the strength of distillation learning supervision and structured sparse clipping supervision.

In order to optimize the method in this article, this research also proposes a joint optimization algorithm of distillation learning & tailoring based on the alternating direction multiplier method to update the student network. In order to optimize and update the teaching director network, this paper proposes a teaching director optimization algorithm based on meta-learning. Distillability can in turn be influenced by dynamically adjusting the supervision signal. As shown in Figure 1-(a), the method in this paper proves to be able to delay the downward trend of distillability and improve the overall distillability by rationally utilizing the knowledge of distillation.

The overall algorithm framework and flow chart of this article’s method are shown in the figure below. The framework contains three major modules, teacher network, student network and dean network. Among them, the initial complex redundant network to be compressed and trimmed is regarded as the teacher network, and in the subsequent training process, the original network that is gradually sparse is regarded as the student network. The dean network is a meta-network that inputs the information of the teacher network and the student network to measure the current distillability and sparsity, thereby controlling the supervision intensity of distillation learning and sparseness.

In this way, at every moment, the student network can be guided and sparsified by dynamically distilled knowledge. For example, when the student network has a higher distillability, the dean will let a stronger distillation supervision signal guide the student network (see the pink arrow signal in Figure 2); on the contrary, when the student network has a higher sparseness Therefore, the dean will exert a stronger sparse supervision signal on the student network (see the orange arrow signal in Figure 2).

Figure 2 Schematic diagram of model compression algorithm based on joint learning of distillability and sparsity

Experimental results

The experiment compares the method proposed in this article with 24 mainstream model compression methods (including sparse clipping methods and distillation learning methods) on the small-scale data set CIFAR and the large-scale data set ImageNet. The experimental results are shown in the figure below, which prove the superiority of the method proposed in this article.

Table 1 Performance comparison of model cropping results on CIFAR10:

Table 2 on ImageNet Performance comparison of model cropping results:

For more research details, please refer to the original paper.

以上是首次將「教導主任」引入模型蒸餾，大規模壓縮優於24種SOTA方法的詳細內容。更多資訊請關注PHP中文網其他相關文章！