search
HomeTechnology peripheralsAITaotian Group and Aicheng Technology cooperate to release the open source large-scale model training framework Megatron-LLaMA

On September 12, Taotian Group and Aicheng Technology officially open sourced the large model training framework - Megatron-LLaMA, aiming to allow technology developers to more conveniently improve the training performance of large language models and reduce training costs. And maintain compatibility with the LLaMA community. Tests show that in 32-card training, Megatron-LLaMA can achieve 176% acceleration compared to the code version directly obtained from HuggingFace; in large-scale training, Megatron-LLaMA has almost linear scalability compared to 32 cards. And shows a high tolerance for network instability. Currently, Megatron-LLaMA is online in the open source community.

Open source address: https://github.com/alibaba/Megatron-LLaMA

The outstanding performance of large language models has exceeded people's imagination time and time again. In the past few months, LLaMA and LLaMA2 have been fully opened to the open source community, giving people who want to train their own large language models a good choice. In the open source community, there are already many models based on LLaMA output, including continued training/SFT (such as Alpaca, Vicuna, WizardLM, Platypus, StableBegula, Orca, OpenBuddy, Linly, Ziya, etc.) and training from scratch ( Baichuan, QWen, InternLM, OpenLLaMA). These works not only perform well on the current objective evaluation lists of various large model capabilities, but also demonstrate excellent performance in practical scenarios such as long text understanding, long text generation, code writing, and mathematical solving. Furthermore, many interesting products have appeared in the industry, such as LLaMA combined with Whisper's voice chat robot, LLaMA combined with Stable Diffusion's painting software, auxiliary consultation robots in the medical/legal field, etc.

Although you can get the LLaMA model code from HuggingFace, training an LLaMA model with your own data is not a low-cost and cost-effective option for individual users or small and medium-sized organizations. Simple job. The volume of large models and the scale of data make it impossible to complete effective training on ordinary computing resources, and computing power and cost have become serious bottlenecks. The users of the Megatron-LM community have very urgent demands in this regard. Taotian Group and Aicheng Technology cooperate to release the open source large-scale model training framework Megatron-LLaMA
Taotian Group and Aicheng Technology have very broad application scenarios for large model applications, and have invested a lot in efficient training of large models. The advent of LLaMA has given many companies, including Taotian Group and Aicheng Technology, a lot of inspiration in terms of data processing, model design, fine-tuning and reinforcement learning feedback adjustment, and has also helped achieve new breakthroughs in business application scenarios. . Therefore, in order to give back to the entire LLaMA open source community and promote the development of the Chinese pre-trained large model open source community, so that developers can more easily improve the training performance of large language models and reduce training costs, Taotian Group and Aicheng Technology will combine some internal Optimize the technology and open source it, release Megatron-LLaMA, and look forward to building the Megatron and LLaMA ecosystem with every partner.

Megatron-LLaMA provides a set of standard Megatron-LM implementations of LLaMA, and provides tools for free switching with HuggingFace formats , to facilitate compatibility with community ecological tools. Megatron-LLaMA has redesigned the reverse process of Megatron-LM, so that it can be achieved no matter where the number of nodes is small and large gradient aggregation (GA) needs to be turned on, or when the number of nodes is large and small GA must be used. Excellent training performance.

  • In 32-card training, compared to the code version obtained directly from HuggingFace, Megatron-LLaMA can achieve 176% acceleration; Even with the optimized version of DeepSpeed ​​and FlashAttention, Megatron-LLaMA can still reduce training time by at least 19%.
  • In large-scale training, Megatron-LLaMA has almost linear scalability compared to 32 cards. For example, using 512 A100 to reproduce the training of LLaMA-13B, the reverse mechanism of Megatron-LLaMA can save at least two days compared to the DistributedOptimizer of the native Megatron-LM without any loss of accuracy.
  • Megatron-LLaMA exhibits a high tolerance for network instability. Even on the current cost-effective 8xA100-80GB training cluster with 4x200Gbps communication bandwidth (this environment is usually a mixed-deployment environment, the network can only use half of the bandwidth, the network bandwidth is a serious bottleneck, but the rental price is relatively low), Megatron-LLaMA can still achieve a linear expansion capability of 0.85, but Megatron-LM can only achieve less than 0.7 on this indicator.

    MEGATRON-LM technology brought high-performance LLAMA training opportunities

    ## Llama is currently a large language model open source community an important task. LLaMA introduces optimization technologies such as BPE character encoding, RoPE positional encoding, SwiGLU activation function, RMSNorm regularization, and Untied Embedding into the structure of LLM, and has achieved excellent results in many objective and subjective evaluations. LLaMA provides 7B, 13B, 30B, 65B/70B versions, which are suitable for various scenarios requiring large models, and are also favored by developers. Like many open source large models, since the official only provides the inference version of the code, there is no standard paradigm for how to carry out efficient training at the lowest cost.

    Megatron-LM is an elegant high-performance training solution.Megatron-LM provides tensor parallelism (Tensor Parallel, TP, which allocates large multiplications to multiple cards for parallel computing), pipeline parallelism (Pipeline Parallel, PP, which allocates different layers of the model to different cards for processing), and sequence parallelism (Sequence Parallel, SP, different parts of the sequence are processed by different cards, saving video memory), DistributedOptimizer optimization (similar to DeepSpeed ​​Zero Stage-2, splitting gradient and optimizer parameters to all computing nodes) and other technologies can significantly reduce video memory usage and improve GPU utilization. Megatron-LM operates an active open source community, and new optimization technologies and functional designs continue to be incorporated into the framework.

    However, developing based on Megatron-LM is not simple, and debugging and functional verification on expensive multi-card machines is very expensive. Megatron-LLaMA first provides a set of LLaMA training code based on the Megatron-LM framework, supports model versions of various sizes, and can be easily adapted to support various variants of LLaMA, including direct support for the Tokenizer in the HuggingFace format. . Therefore, Megatron-LLaMA can be easily applied to existing offline training links without excessive adaptation. In small and medium-scale training/fine-tuning scenarios for LLaMA-7b and LLaMA-13b, Megatron-LLaMA can easily achieve industry-leading hardware utilization (MFU) of 54% and above.

    MEGATRON-LLAMA's reverse process optimization Taotian Group and Aicheng Technology cooperate to release the open source large-scale model training framework Megatron-LLaMA## igue: DeepSpeed ​​Zero Stage-2



    Taotian Group and Aicheng Technology cooperate to release the open source large-scale model training framework Megatron-LLaMATaotian Group and Aicheng Technology cooperate to release the open source large-scale model training framework Megatron-LLaMA


    ################################################################## #####DeepSpeed ​​ZeRO is a distributed training framework launched by Microsoft. The technology proposed in it has had a profound impact on many subsequent frameworks. DeepSpeed ​​ZeRO Stage-2 (hereinafter referred to as ZeRO-2) is a technology in the framework that saves memory usage without adding additional calculation and communication workload. As shown in the figure above, due to calculation requirements, each Rank needs to have all parameters. But for the optimizer state, each Rank is only responsible for a part of it, and it is not necessary for all Ranks to perform completely repeated operations at the same time. Therefore, ZeRO-2 proposes to evenly divide the optimizer state into each Rank (note that there is no need to ensure that each variable is evenly divided or completely retained in a certain Rank). Each Rank only needs to be used during the training process. Responsible for updating the optimizer status and model parameters of the corresponding part. In this setting, gradients can also be split in this way. By default, ZeRO-2 uses the Reduce method to aggregate gradients among all Ranks in reverse, and then each Rank only needs to retain the part of the parameters it is responsible for, which not only eliminates redundant repeated calculations, but also reduces the memory usage. . #########                                                                                                                                                              Megatron-LM DistributedOptimizer###   ### Native Megatron-LM implements ZeRO-2-like gradient and optimizer state segmentation through DistributedOptimizer to reduce video memory usage during training. As shown in the figure above, DistributedOptimizer uses the ReduceScatter operator to distribute all the previously accumulated gradients to different Ranks after obtaining all the gradients aggregated by the preset gradient. Each Rank only obtains part of the gradient that it needs to process, and then updates the optimizer state and the corresponding parameters. Finally, each Rank obtains updated parameters from other nodes through AllGather, and finally obtains all parameters. The actual training results show that the gradient and parameter communication of Megatron-LM are performed in series with other calculations. For large-scale pre-training tasks, in order to ensure that the total batch data size remains unchanged, it is usually impossible to open a larger GA. Therefore, the proportion of communication will increase with the increase of machines. At this time, the characteristics of serial communication lead to very weak scalability. Within the community, the need is also acute. ######                                                                                   ’s     over over ‐ over ‐‐‐ under‐‐hum over‐ coming and re P P to to C to to C to to C on to do to have to do with to do with L P L ‐ ‐ ‐ LLaMA overlapped to do with. The operator can be parallelized with the calculation. In particular, compared to ZeRO's implementation, Megatron-LLaMA uses a more scalable collective communication method to improve scalability through clever optimization of the optimizer partitioning strategy under the premise of parallelism.The main design of OverlappedDistributedOptimizer ensures the following points: a) The data volume of a single set communication operator is large enough to fully utilize the communication bandwidth; b) The amount of communication data required by the new segmentation method should be equal to the minimum communication data volume required for data parallelism; c) During the conversion process of complete parameters or gradients and segmented parameters or gradients, too many video memory copies cannot be introduced. Taotian Group and Aicheng Technology cooperate to release the open source large-scale model training framework Megatron-LLaMA
    Specifically, Megatron-LLaMA improves the mechanism of DistributedOptimizer and proposes OverlappedDistributedOptimizer, which is used to optimize the reverse process in training in combination with the new segmentation method. As shown in the figure above, when OverlappedDistributedOptimizer is initialized, all parameters will be pre-allocated to the Bucket to which they belong. The parameters in a Bucket are complete. A parameter only belongs to one Bucket. There may be multiple parameters in a Bucket. Logically, each Bucket will be continuously divided into P (P is the number of data parallel groups) equal parts, and each Rank in the data parallel group is responsible for one of them.

    #Bucket is placed in a local queue (Local grad bucket queue) to ensure communication order. During training and calculation, data parallel groups exchange the gradients they need through collective communication in Bucket units. The implementation of Bucket in Megatron-LLaMA uses address indexing as much as possible, and only newly allocates space when the required value changes, avoiding waste of video memory.

    The above design, combined with a large number of engineering optimizations, allows Megatron-LLaMA to fully utilize the hardware during large-scale training, achieving better performance than the native Megatron-LM Better acceleration. When training from 32 A100 cards to 512 A100 cards, Megatron-LLaMA can still achieve an expansion ratio of 0.85 in a commonly used mixed network environment.

    MEGATRON-LLAMA's Future Plan

    # MEGATRON-LLAMA is jointly open source and provide subsequent maintenance support by Taitian Group and Ai Orange Technology The training framework has been widely used internally. As more and more developers flock to LLaMA’s open source community and contribute experiences that can be learned from each other, I believe there will be more challenges and opportunities at the training framework level in the future. Megatron-LLaMA will pay close attention to the development of the community and work with developers to promote the following directions:
  • Adaptive optimal configuration selection
  • More Support for model structure or local design changes
  • Extreme performance training solutions in more different types of hardware environments

    Project address: https://github.com/alibaba/ Megatron-LLaMA

The above is the detailed content of Taotian Group and Aicheng Technology cooperate to release the open source large-scale model training framework Megatron-LLaMA. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:机器之心. If there is any infringement, please contact admin@php.cn delete
From Friction To Flow: How AI Is Reshaping Legal WorkFrom Friction To Flow: How AI Is Reshaping Legal WorkMay 09, 2025 am 11:29 AM

The legal tech revolution is gaining momentum, pushing legal professionals to actively embrace AI solutions. Passive resistance is no longer a viable option for those aiming to stay competitive. Why is Technology Adoption Crucial? Legal professional

This Is What AI Thinks Of You And Knows About YouThis Is What AI Thinks Of You And Knows About YouMay 09, 2025 am 11:24 AM

Many assume interactions with AI are anonymous, a stark contrast to human communication. However, AI actively profiles users during every chat. Every prompt, every word, is analyzed and categorized. Let's explore this critical aspect of the AI revo

7 Steps To Building A Thriving, AI-Ready Corporate Culture7 Steps To Building A Thriving, AI-Ready Corporate CultureMay 09, 2025 am 11:23 AM

A successful artificial intelligence strategy cannot be separated from strong corporate culture support. As Peter Drucker said, business operations depend on people, and so does the success of artificial intelligence. For organizations that actively embrace artificial intelligence, building a corporate culture that adapts to AI is crucial, and it even determines the success or failure of AI strategies. West Monroe recently released a practical guide to building a thriving AI-friendly corporate culture, and here are some key points: 1. Clarify the success model of AI: First of all, we must have a clear vision of how AI can empower business. An ideal AI operation culture can achieve a natural integration of work processes between humans and AI systems. AI is good at certain tasks, while humans are good at creativity and judgment

Netflix New Scroll, Meta AI's Game Changers, Neuralink Valued At $8.5 BillionNetflix New Scroll, Meta AI's Game Changers, Neuralink Valued At $8.5 BillionMay 09, 2025 am 11:22 AM

Meta upgrades AI assistant application, and the era of wearable AI is coming! The app, designed to compete with ChatGPT, offers standard AI features such as text, voice interaction, image generation and web search, but has now added geolocation capabilities for the first time. This means that Meta AI knows where you are and what you are viewing when answering your question. It uses your interests, location, profile and activity information to provide the latest situational information that was not possible before. The app also supports real-time translation, which completely changed the AI ​​experience on Ray-Ban glasses and greatly improved its usefulness. The imposition of tariffs on foreign films is a naked exercise of power over the media and culture. If implemented, this will accelerate toward AI and virtual production

Take These Steps Today To Protect Yourself Against AI CybercrimeTake These Steps Today To Protect Yourself Against AI CybercrimeMay 09, 2025 am 11:19 AM

Artificial intelligence is revolutionizing the field of cybercrime, which forces us to learn new defensive skills. Cyber ​​criminals are increasingly using powerful artificial intelligence technologies such as deep forgery and intelligent cyberattacks to fraud and destruction at an unprecedented scale. It is reported that 87% of global businesses have been targeted for AI cybercrime over the past year. So, how can we avoid becoming victims of this wave of smart crimes? Let’s explore how to identify risks and take protective measures at the individual and organizational level. How cybercriminals use artificial intelligence As technology advances, criminals are constantly looking for new ways to attack individuals, businesses and governments. The widespread use of artificial intelligence may be the latest aspect, but its potential harm is unprecedented. In particular, artificial intelligence

A Symbiotic Dance: Navigating Loops Of Artificial And Natural PerceptionA Symbiotic Dance: Navigating Loops Of Artificial And Natural PerceptionMay 09, 2025 am 11:13 AM

The intricate relationship between artificial intelligence (AI) and human intelligence (NI) is best understood as a feedback loop. Humans create AI, training it on data generated by human activity to enhance or replicate human capabilities. This AI

AI's Biggest Secret — Creators Don't Understand It, Experts SplitAI's Biggest Secret — Creators Don't Understand It, Experts SplitMay 09, 2025 am 11:09 AM

Anthropic's recent statement, highlighting the lack of understanding surrounding cutting-edge AI models, has sparked a heated debate among experts. Is this opacity a genuine technological crisis, or simply a temporary hurdle on the path to more soph

Bulbul-V2 by Sarvam AI: India's Best TTS ModelBulbul-V2 by Sarvam AI: India's Best TTS ModelMay 09, 2025 am 10:52 AM

India is a diverse country with a rich tapestry of languages, making seamless communication across regions a persistent challenge. However, Sarvam’s Bulbul-V2 is helping to bridge this gap with its advanced text-to-speech (TTS) t

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

EditPlus Chinese cracked version

EditPlus Chinese cracked version

Small size, syntax highlighting, does not support code prompt function

SublimeText3 Linux new version

SublimeText3 Linux new version

SublimeText3 Linux latest version

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

Safe Exam Browser

Safe Exam Browser

Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

Dreamweaver Mac version

Dreamweaver Mac version

Visual web development tools