Microsoft releases new mixed precision training framework FP8: 64% faster than BF16 and 42% less memory usage

November 10 news, large language models (LLM) are rising rapidly, showing bright prospects in language generation and understanding, and their influence goes beyond the language field and extends to logic and mathematics. , physics and other fields.

However, if you want to unlock these "extraordinary energy", you need to pay a high price. For example, training a 540B model requires 6144 TPUv4 chips of Project PaLM; and training a 175B model GPT-3, requires thousands of Petaflop/s-day.

A good solution is to train with low precision, which can increase processing speed and reduce memory usage and communication costs. The current mainstream training systems include Megatron-LM, MetaSeq and Colossal-AI, which use FP16/BF16 mixed precision or FP32 full precision by default to train large language models

Although these accuracy levels are necessary for large language models Indispensable, but they are computationally expensive.

If you use FP8 low precision, you can increase speed by 2 times, reduce memory costs by 50% to 75%, and save communication costs.

Currently only the Nvidia Transformer Engine is compatible with the FP8 framework, primarily leveraging this precision for GEMM (general matrix multiplication) calculations while maintaining master weights and gradients at FP16 or FP32 high precision.

To address this challenge, a team of researchers from Microsoft Azure and Microsoft Research introduced an efficient FP8 mixed-precision framework tailored for large language model training.

Microsoft has introduced three optimization stages to leverage FP8 for distributed and mixed precision training. As these levels progress, increased integration of FP8 becomes evident, suggesting a greater impact on the LLM training process.

In addition, in order to overcome problems such as data overflow or underflow, Microsoft researchers proposed two key methods: automatic sampling and precise decoupling. The former involves components that are not sensitive to accuracy, reducing accuracy, and dynamically adjusting the Tensor sampling factor to Make sure gradient values ​​stay within the FP8 representation range. This prevents underflow and overflow events during communication, ensuring a smoother training process.

Microsoft has tested memory usage reduced by 27% to 42% and weight gradient communication overhead significantly reduced by 63% to 65% compared to the widely adopted BF16 mixed precision method. Runs 64% faster than widely adopted BF16 frameworks such as Megatron-LM and 17% faster than the Nvidia Transformer Engine.

When training the GPT-175B model, the hybrid FP8 precision framework saves 21% of memory on the H100 GPU platform, and compared to TE (Transformer Engine), the training time is reduced 17%.

This site attaches the GitHub address and paper address: https://www.php.cn/link/7b3564b05f78b6739d06a2ea3187f5ca

The above is the detailed content of Microsoft releases new mixed precision training framework FP8: 64% faster than BF16 and 42% less memory usage. For more information, please follow other related articles on the PHP Chinese website!