Why do large language models use SwiGLU as activation function?

If you have been paying attention to the architecture of large language models, you may have seen the term "SwiGLU" in the latest models and research papers. SwiGLU can be said to be the most commonly used activation function in large language models. We will introduce it in detail in this article. SwiGLU is actually an activation function proposed by Google in 2020, which combines the characteristics of SWISH and GLU. The full Chinese name of SwiGLU is "bidirectional gated linear unit". It optimizes and combines two activation functions, SWISH and GLU, to improve the nonlinear expression ability of the model. SWISH is a very common activation function that is widely used in large language models, while GLU performs well in natural language processing tasks. The advantage of SwiGLU is that it can obtain the smoothing characteristics of SWISH and the gating characteristics of GLU at the same time, thereby making the nonlinear expression of the model more

us Let’s introduce them one by one:

Swish

Swish is a nonlinear activation function, defined as follows:

Swish(x) = x*sigmoid(ßx)

Among them, ß is a learnable parameter. Swish can be better than ReLU activation function as it gives smoother transitions which can lead to better optimization.

Gated Linear Unit

GLU (Gated Linear Unit) is defined as the component product of two linear transformations, one of which is activated by sigmoid .

GLU(x) = sigmoid(W1x+b)⊗(Vx+c)

#The GLU module can effectively capture long-range dependencies in sequences while avoiding other gates such as LSTM and GRU There are some vanishing gradient problems related to the control mechanism.

SwiGLU

We have already said that SwiGLU is a combination of the two. It's a GLU, but instead of using sigmoid as the activation function, it uses swish with ß=1, so we end up with the following formula:

SwiGLU(x) = Swish(W1x+b)⊗(Vx+c)

We use The SwiGLU function constructs a feedforward network

FFNSwiGLU(x) = (Swish1(xW)⊗xV)W2

Simple implementation of Pytorch

If you look at the above mathematical principles for comparison It's boring and difficult to understand, so we'll explain it directly using the code below.

class SwiGLU(nn.Module): def __init__(self, w1, w2, w3) -> None:super().__init__()self.w1 = w1self.w2 = w2self.w3 = w3 def forward(self, x):x1 = F.linear(x, self.w1.weight)x2 = F.linear(x, self.w2.weight)hidden = F.silu(x1) * x2return F.linear(hidden, self.w3.weight)

The F.silu function used in our code is the same as swish when ß=1, so we use it directly.

As you can see from the code, there are 3 weights in our activation function that can be trained, which are the parameters from the GLU formula.

Comparison of the effect of SwiGLU

Comparing SwiGLU with other GLU variants, we can see that SwiGLU performs well during both pre-training periods Better.

Downstream tasks

The effect is the best, So now llm, such as LLAMA, OLMO and PALM all use SwiGLU in their implementation. But why is SwiGLU better than the others?

The paper only gave the test results and did not explain the reasons. Instead, it said:

We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.

The author said that the alchemy was successful.

But now it is 2024 and we can forcefully explain it:

#1. Swish’s response to negative values ​​is relatively small It overcomes the shortcoming of ReLU that the output on some neurons is always zero

2. The gating characteristics of GLU, which means that it can decide which information should pass and which information should pass through based on the input situation. Information should be filtered. This mechanism allows the network to learn useful representations more effectively and helps improve the generalization ability of the model. In large language models, this is particularly useful for processing long sequences of text with long-distance dependencies.

3. The parameters W1, W2, W3, b1, b2, b3 W1, W2, W3, b1, b2, b3 in SwiGLU can be learned through training, so that the model can be adapted to different tasks Dynamically adjusting these parameters with the data set enhances the flexibility and adaptability of the model.

4. The calculation efficiency is higher than some more complex activation functions (such as GELU), while still maintaining good performance. This is an important consideration for the training and inference of large-scale language models.

Choose SwiGLU as the activation function of the large language model, mainly because it combines the advantages of nonlinear capabilities, gating characteristics, gradient stability and learnable parameters. SwiGLU is widely adopted due to its excellent performance in handling complex semantic relationships and long dependency problems in language models, as well as maintaining training stability and computational efficiency.

Paper address

https://www.php.cn/link/86e33d550dc162366a02003089ab9894

The above is the detailed content of Why do large language models use SwiGLU as activation function?. For more information, please follow other related articles on the PHP Chinese website!