search
HomeTechnology peripheralsAISingle-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Unlock the potential to further scale Transformer while maintaining computational efficiency.


Feedforward (FFW) layers in the standard Transformer architecture result in a linear increase in computational cost and activation memory as the hidden layer width increases. As the size of large language models (LLM) continues to increase, the sparse mixed expert (MoE) architecture has become a feasible method to solve this problem, which separates the model size from the computational cost. Many emerging MoE models can achieve better performance and more powerful performance at the same size.

The recently discovered fine-grained MoE expansion law shows that higher granularity leads to better performance. However, existing MoE models are limited to a low number of experts due to computational and optimization challenges.

This Tuesday, New research from Google DeepMind introduces a parameter-efficient expert retrieval mechanism that leverages product key technology to perform sparse retrieval from one million micro-experts.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Link: https://arxiv.org/abs/2407.04153

This approach attempts to decouple the computational cost from parameter count by efficiently concatenating to a large number of tiny experts through a learned index structure for routing . Demonstrates superior efficiency compared to dense FFW, coarse-grained MoE, and Product Key Memory (PKM) layers.

This work introduces the Parameter Efficient Expert Retrieval (PEER) architecture (parameter efficient expert retrieval), which uses product key retrieval to efficiently route to a large number of experts, separating the computational cost from the amount of parameters. This design demonstrated superior computational performance levels in experiments, positioning it as a competitive alternative to dense FFW layers for extending base models. The main contributions of this work are:

Exploration of extreme MoE settings: Unlike the focus on a few large experts in previous MoE studies, this work investigates the underexplored situation of numerous small experts.

Learned Index Structure for Routing: First demonstration that a learned index structure can efficiently route to over a million experts.

New layer design: Combining product key routing with single-neuron experts, we introduce the PEER layer, which scales layer capacity without significant computational overhead. Empirical results demonstrate higher efficiency compared to dense FFW, coarse-grained MoE, and Product Key Memory (PKM) layers.

Comprehensive ablation study: We study the impact of different design choices for PEER (such as number of experts, activity parameters, number of heads, and query batch normalization) on language modeling tasks.

Method Introduction

In this section, the researcher explains in detail the Parametric Efficient Expert Retrieval (PEER) layer, a hybrid expert architecture that uses product keys in routing and single-neuron MLP As an expert. Figure 2 below shows the calculation process within the PEER layer.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

PEER Layer Overview. Formally, the PEER layer is a function f : R^n → R^m, which consists of three parts: a pool of N experts E := {e_i}^N_i=1, where each expert e_i : R^n → R^m shares the same signature as f; a corresponding set of N product keys K := {k_i}^N_i=1 ⊂ R^d; and a query network q : R^n → R ^d, which maps the input vector x ∈ R^n to the query vector q (x).

Let T_k represent the top-k operator. Given an input x, first retrieve the subset of k experts whose corresponding product keys have the highest inner product with the query q (x).

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Then apply a nonlinear activation (such as softmax or sigmoid) to the query key inner product of the top k experts to obtain the routing score.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Finally the output is calculated by linearly combining the expert outputs weighted by routing scores.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Product Key Retrieval. Since the researchers intend to use a large number of experts (N ≥ 10^6), simply computing the top k indexes in Equation 1 can be very costly, so the product key retrieval technique is applied. Instead of using N independent d-dimensional vectors as keys k_i, they create them by concatenating vectors from two independent d/2-dimensional subkey sets (i.e., C, C ′ ⊂ R d/2):

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Parametric efficient expert and multi-head search. Unlike other MoE architectures, these architectures typically set each expert's hidden layer to the same size as other FFW layers. In PEER, each expert e_i is a singleton MLP, in other words, it has only one hidden layer with a single neuron:

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

The researchers did not change the size of the individual expert, but used Multi-head retrieval is used to adjust the expressive ability of the PEER layer, which is similar to the multi-head attention mechanism in the transformer and the multi-head memory in PKM.

Specifically, they use h independent query networks, each network computes its own query and retrieves a separate set of k experts. However, different heads share the same expert pool, with the same set of product keys. The output of these h heads is simply summarized as follows:

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Why do we need a large number of small experts ? A given MoE layer can be characterized by three hyperparameters: the total number of parameters P, the number of active parameters per token P_active, and the size of a single expert P_expert. Krajewski et al. (2024) showed that the scaling law of the MoE model has the following form:

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

For PEER, the researcher uses the smallest possible expert size by setting d_expert = 1, and the number of activated neurons is the search head The number is multiplied by the number of experts retrieved per head: d_active = hk. Therefore, the granularity of PEER is always G = P_active/P_expert = d_active/d_expert = hk.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Experimental results

Let’s first look at the evaluation results on the language modeling data set.

After determining the computationally optimal model for each method based on the isoFLOP curve, the researchers evaluated the performance of these pre-trained models on the following popular language modeling datasets:

  • Curation Corpus
  • Lambada
  • Pile
  • Wikitext
  • Pre-training dataset C4

Table 1 below shows the evaluation results. We grouped the models based on the FLOP budget used during training. As can be seen, PEER has the lowest perplexity on these language modeling datasets.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

In the ablation experiment, the researchers changed the total number of experts. The models shown in the isoFLOP curve in Figure 1 below all have over one million (1024^2) experts.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

The researcher chose the model with the optimal position of isoFLOP and changed the number of experts in the PEER layer (N = 128^2, 256^2, 512^2, 1024^2), while keeping the number of active experts unchanged (h = 8, k = 16). The results are shown in Figure 3(a) below.

It can be seen that the isoFLOP curve interpolates between the PEER model with 1024^2 experts and the corresponding dense backbone without replacing the FFW layer in the middle block with a PEER layer. This shows that model performance can be improved simply by increasing the number of experts.

At the same time, the researchers changed the number of active experts. They systematically varied the number of active experts (hk = 32, 64, 128, 256, 512) while keeping the total number of experts constant (N = 1024^2). For a given hk, the researcher then jointly changes h and k to determine the best combination. Figure 3(b) below plots the isoFLOP curve with respect to the number of heads (h).

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Table 2 below lists the expert usage and unevenness for different numbers of experts with and without BN. It can be seen that even for 1M experts, the expert utilization rate is close to 100%, and using BN can make the expert utilization rate more balanced and the confusion level lower. These findings demonstrate the effectiveness of the PEER model in leveraging a large number of experts.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

The researchers also compared the isoFLOP curves with and without BN. Figure 4 below shows that PEER models with BN can generally achieve lower perplexity. Although the difference is not significant, it is most noticeable near the isoFLOP optimal region.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Introduction to the author

PEER study has only one author, Xu He (Owen), who is a research scientist at Google DeepMind and graduated with a PhD from the University of Groningen in the Netherlands in 2017.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

The above is the detailed content of Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE. For more information, please follow other related articles on the PHP Chinese website!

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Fine-Tune Open-Source LLMs Using Lamini - Analytics VidhyaFine-Tune Open-Source LLMs Using Lamini - Analytics VidhyaApr 12, 2025 am 10:20 AM

Recently, with the rise of large language models and AI, we have seen innumerable advancements in natural language processing. Models in domains like text, code, and image/video generation have archived human-like reasoning and p

Gender Detection with OpenCV and Roboflow in Python - Analytics VidhyaGender Detection with OpenCV and Roboflow in Python - Analytics VidhyaApr 12, 2025 am 10:19 AM

Introduction Gender detection from facial images is one of the many fascinating applications of computer vision. In this project, we combine OpenCV for confront location and the Roboflow API for gender classification, making a

What is the Role of Generative AI in Personalizing Ad Content?What is the Role of Generative AI in Personalizing Ad Content?Apr 12, 2025 am 10:18 AM

Introduction The world of advertisement has been under evolution since the conception of the barter system. Advertisers have found creative ways to bring their products to our attention. In the current age, consumers expect br

OpenAI's o1-preview vs o1-mini: A Step Forward to AGIOpenAI's o1-preview vs o1-mini: A Step Forward to AGIApr 12, 2025 am 10:04 AM

Introduction On September 12th, OpenAI released an update titled “Learning to Reason with LLMs.” They introduced the o1 model, which is trained using reinforcement learning to tackle complex reasoning tasks. What sets this mod

How to Build Games with OpenAI o1? - Analytics VidhyaHow to Build Games with OpenAI o1? - Analytics VidhyaApr 12, 2025 am 10:03 AM

Introduction The OpenAI o1 model family significantly advances reasoning power and economic performance, especially in science, coding, and problem-solving. OpenAI’s goal is to create ever-more-advanced AI, and o1 models

Popular LLM Agent Tools for Customer Query ManagementPopular LLM Agent Tools for Customer Query ManagementApr 12, 2025 am 10:01 AM

Introduction Today, the world of customer query management is moving at an unprecedented pace, with new tools making headlines every day. Large language model (LLM) agents are the latest innovation in this context, boosting cu

100 Day Generative AI Implementation Plan for Enterprises100 Day Generative AI Implementation Plan for EnterprisesApr 12, 2025 am 09:56 AM

Introduction Adopting generative AI can be a transformative journey for any company. However, the process of GenAI implementation can often be cumbersome and confusing. Rajendra Singh Pawar, chairman and co-founder of NIIT Lim

Pixtral 12B vs Qwen2-VL-72BPixtral 12B vs Qwen2-VL-72BApr 12, 2025 am 09:52 AM

Introduction The AI revolution has given rise to a new era of creativity, where text-to-image models are redefining the intersection of art, design, and technology. Pixtral 12B and Qwen2-VL-72B are two pioneering forces drivin

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
3 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. Best Graphic Settings
3 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. How to Fix Audio if You Can't Hear Anyone
3 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
WWE 2K25: How To Unlock Everything In MyRise
3 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Atom editor mac version download

Atom editor mac version download

The most popular open source editor

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

DVWA

DVWA

Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

SublimeText3 Linux new version

SublimeText3 Linux new version

SublimeText3 Linux latest version

EditPlus Chinese cracked version

EditPlus Chinese cracked version

Small size, syntax highlighting, does not support code prompt function