search
HomeTechnology peripheralsAISingle-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Unlock the potential to further scale Transformer while maintaining computational efficiency.


Feedforward (FFW) layers in the standard Transformer architecture result in a linear increase in computational cost and activation memory as the hidden layer width increases. As the size of large language models (LLM) continues to increase, the sparse mixed expert (MoE) architecture has become a feasible method to solve this problem, which separates the model size from the computational cost. Many emerging MoE models can achieve better performance and more powerful performance at the same size.

The recently discovered fine-grained MoE expansion law shows that higher granularity leads to better performance. However, existing MoE models are limited to a low number of experts due to computational and optimization challenges.

This Tuesday, New research from Google DeepMind introduces a parameter-efficient expert retrieval mechanism that leverages product key technology to perform sparse retrieval from one million micro-experts.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Link: https://arxiv.org/abs/2407.04153

This approach attempts to decouple the computational cost from parameter count by efficiently concatenating to a large number of tiny experts through a learned index structure for routing . Demonstrates superior efficiency compared to dense FFW, coarse-grained MoE, and Product Key Memory (PKM) layers.

This work introduces the Parameter Efficient Expert Retrieval (PEER) architecture (parameter efficient expert retrieval), which uses product key retrieval to efficiently route to a large number of experts, separating the computational cost from the amount of parameters. This design demonstrated superior computational performance levels in experiments, positioning it as a competitive alternative to dense FFW layers for extending base models. The main contributions of this work are:

Exploration of extreme MoE settings: Unlike the focus on a few large experts in previous MoE studies, this work investigates the underexplored situation of numerous small experts.

Learned Index Structure for Routing: First demonstration that a learned index structure can efficiently route to over a million experts.

New layer design: Combining product key routing with single-neuron experts, we introduce the PEER layer, which scales layer capacity without significant computational overhead. Empirical results demonstrate higher efficiency compared to dense FFW, coarse-grained MoE, and Product Key Memory (PKM) layers.

Comprehensive ablation study: We study the impact of different design choices for PEER (such as number of experts, activity parameters, number of heads, and query batch normalization) on language modeling tasks.

Method Introduction

In this section, the researcher explains in detail the Parametric Efficient Expert Retrieval (PEER) layer, a hybrid expert architecture that uses product keys in routing and single-neuron MLP As an expert. Figure 2 below shows the calculation process within the PEER layer.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

PEER Layer Overview. Formally, the PEER layer is a function f : R^n → R^m, which consists of three parts: a pool of N experts E := {e_i}^N_i=1, where each expert e_i : R^n → R^m shares the same signature as f; a corresponding set of N product keys K := {k_i}^N_i=1 ⊂ R^d; and a query network q : R^n → R ^d, which maps the input vector x ∈ R^n to the query vector q (x).

Let T_k represent the top-k operator. Given an input x, first retrieve the subset of k experts whose corresponding product keys have the highest inner product with the query q (x).

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Then apply a nonlinear activation (such as softmax or sigmoid) to the query key inner product of the top k experts to obtain the routing score.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Finally the output is calculated by linearly combining the expert outputs weighted by routing scores.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Product Key Retrieval. Since the researchers intend to use a large number of experts (N ≥ 10^6), simply computing the top k indexes in Equation 1 can be very costly, so the product key retrieval technique is applied. Instead of using N independent d-dimensional vectors as keys k_i, they create them by concatenating vectors from two independent d/2-dimensional subkey sets (i.e., C, C ′ ⊂ R d/2):

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Parametric efficient expert and multi-head search. Unlike other MoE architectures, these architectures typically set each expert's hidden layer to the same size as other FFW layers. In PEER, each expert e_i is a singleton MLP, in other words, it has only one hidden layer with a single neuron:

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

The researchers did not change the size of the individual expert, but used Multi-head retrieval is used to adjust the expressive ability of the PEER layer, which is similar to the multi-head attention mechanism in the transformer and the multi-head memory in PKM.

Specifically, they use h independent query networks, each network computes its own query and retrieves a separate set of k experts. However, different heads share the same expert pool, with the same set of product keys. The output of these h heads is simply summarized as follows:

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Why do we need a large number of small experts ? A given MoE layer can be characterized by three hyperparameters: the total number of parameters P, the number of active parameters per token P_active, and the size of a single expert P_expert. Krajewski et al. (2024) showed that the scaling law of the MoE model has the following form:

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

For PEER, the researcher uses the smallest possible expert size by setting d_expert = 1, and the number of activated neurons is the search head The number is multiplied by the number of experts retrieved per head: d_active = hk. Therefore, the granularity of PEER is always G = P_active/P_expert = d_active/d_expert = hk.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Experimental results

Let’s first look at the evaluation results on the language modeling data set.

After determining the computationally optimal model for each method based on the isoFLOP curve, the researchers evaluated the performance of these pre-trained models on the following popular language modeling datasets:

  • Curation Corpus
  • Lambada
  • Pile
  • Wikitext
  • Pre-training dataset C4

Table 1 below shows the evaluation results. We grouped the models based on the FLOP budget used during training. As can be seen, PEER has the lowest perplexity on these language modeling datasets.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

In the ablation experiment, the researchers changed the total number of experts. The models shown in the isoFLOP curve in Figure 1 below all have over one million (1024^2) experts.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

The researcher chose the model with the optimal position of isoFLOP and changed the number of experts in the PEER layer (N = 128^2, 256^2, 512^2, 1024^2), while keeping the number of active experts unchanged (h = 8, k = 16). The results are shown in Figure 3(a) below.

It can be seen that the isoFLOP curve interpolates between the PEER model with 1024^2 experts and the corresponding dense backbone without replacing the FFW layer in the middle block with a PEER layer. This shows that model performance can be improved simply by increasing the number of experts.

At the same time, the researchers changed the number of active experts. They systematically varied the number of active experts (hk = 32, 64, 128, 256, 512) while keeping the total number of experts constant (N = 1024^2). For a given hk, the researcher then jointly changes h and k to determine the best combination. Figure 3(b) below plots the isoFLOP curve with respect to the number of heads (h).

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Table 2 below lists the expert usage and unevenness for different numbers of experts with and without BN. It can be seen that even for 1M experts, the expert utilization rate is close to 100%, and using BN can make the expert utilization rate more balanced and the confusion level lower. These findings demonstrate the effectiveness of the PEER model in leveraging a large number of experts.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

The researchers also compared the isoFLOP curves with and without BN. Figure 4 below shows that PEER models with BN can generally achieve lower perplexity. Although the difference is not significant, it is most noticeable near the isoFLOP optimal region.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

Introduction to the author

PEER study has only one author, Xu He (Owen), who is a research scientist at Google DeepMind and graduated with a PhD from the University of Groningen in the Netherlands in 2017.

Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE

The above is the detailed content of Single-author paper, Google proposes millions of expert Mixture, surpassing dense feedforward and sparse MoE. For more information, please follow other related articles on the PHP Chinese website!

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Microsoft Work Trend Index 2025 Shows Workplace Capacity StrainMicrosoft Work Trend Index 2025 Shows Workplace Capacity StrainApr 24, 2025 am 11:19 AM

The burgeoning capacity crisis in the workplace, exacerbated by the rapid integration of AI, demands a strategic shift beyond incremental adjustments. This is underscored by the WTI's findings: 68% of employees struggle with workload, leading to bur

Can AI Understand? The Chinese Room Argument Says No, But Is It Right?Can AI Understand? The Chinese Room Argument Says No, But Is It Right?Apr 24, 2025 am 11:18 AM

John Searle's Chinese Room Argument: A Challenge to AI Understanding Searle's thought experiment directly questions whether artificial intelligence can genuinely comprehend language or possess true consciousness. Imagine a person, ignorant of Chines

China's 'Smart' AI Assistants Echo Microsoft Recall's Privacy FlawsChina's 'Smart' AI Assistants Echo Microsoft Recall's Privacy FlawsApr 24, 2025 am 11:17 AM

China's tech giants are charting a different course in AI development compared to their Western counterparts. Instead of focusing solely on technical benchmarks and API integrations, they're prioritizing "screen-aware" AI assistants – AI t

Docker Brings Familiar Container Workflow To AI Models And MCP ToolsDocker Brings Familiar Container Workflow To AI Models And MCP ToolsApr 24, 2025 am 11:16 AM

MCP: Empower AI systems to access external tools Model Context Protocol (MCP) enables AI applications to interact with external tools and data sources through standardized interfaces. Developed by Anthropic and supported by major AI providers, MCP allows language models and agents to discover available tools and call them with appropriate parameters. However, there are some challenges in implementing MCP servers, including environmental conflicts, security vulnerabilities, and inconsistent cross-platform behavior. Forbes article "Anthropic's model context protocol is a big step in the development of AI agents" Author: Janakiram MSVDocker solves these problems through containerization. Doc built on Docker Hub infrastructure

Using 6 AI   Street-Smart Strategies To Build A Billion-Dollar StartupUsing 6 AI Street-Smart Strategies To Build A Billion-Dollar StartupApr 24, 2025 am 11:15 AM

Six strategies employed by visionary entrepreneurs who leveraged cutting-edge technology and shrewd business acumen to create highly profitable, scalable companies while maintaining control. This guide is for aspiring entrepreneurs aiming to build a

Google Photos Update Unlocks Stunning Ultra HDR For All Your PicturesGoogle Photos Update Unlocks Stunning Ultra HDR For All Your PicturesApr 24, 2025 am 11:14 AM

Google Photos' New Ultra HDR Tool: A Game Changer for Image Enhancement Google Photos has introduced a powerful Ultra HDR conversion tool, transforming standard photos into vibrant, high-dynamic-range images. This enhancement benefits photographers a

Descope Builds Authentication Framework For AI Agent IntegrationDescope Builds Authentication Framework For AI Agent IntegrationApr 24, 2025 am 11:13 AM

Technical Architecture Solves Emerging Authentication Challenges The Agentic Identity Hub tackles a problem many organizations only discover after beginning AI agent implementation that traditional authentication methods aren’t designed for machine-

Google Cloud Next 2025 And The Connected Future Of Modern WorkGoogle Cloud Next 2025 And The Connected Future Of Modern WorkApr 24, 2025 am 11:12 AM

(Note: Google is an advisory client of my firm, Moor Insights & Strategy.) AI: From Experiment to Enterprise Foundation Google Cloud Next 2025 showcased AI's evolution from experimental feature to a core component of enterprise technology, stream

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

VSCode Windows 64-bit Download

VSCode Windows 64-bit Download

A free and powerful IDE editor launched by Microsoft

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

ZendStudio 13.5.1 Mac

ZendStudio 13.5.1 Mac

Powerful PHP integrated development environment