58 lines of code scale Llama 3 to 1 million contexts, any fine-tuned version is applicable

Llama 3, the majestic king of open source, original context window actually only has...8k, which made me swallow the words "really delicious" on my lips again. .

Today, when 32k is the starting point and 100k is common, is this deliberately leaving room for contributions to the open source community?

The open source community will certainly not miss this opportunity:

Now with only 58 lines of code, any fine-tuned version of Llama 3 70b can be automatically extended 1048k(One million)Context.

Behind the scenes is a LoRA, extracted from a fine-tuned version of Llama 3 70B Instruct that extends the context, The file is only 800mb .

Next, using Mergekit, you can run it with other models of the same architecture or merge it directly into the model.

The fine-tuned version of the 1048k context used has just achieved an all-green (100% accuracy) score in the popular needle-in-a-haystack test.

It must be said that the speed of progress of open source is exponential.

How to make 1048k contextual LoRA

First, the 1048k contextual version of Llama 3 fine-tuning model comes from Gradient AI, an enterprise AI solutions startup.

The corresponding LoRA comes from developer Eric Hartford. By comparing the differences between the fine-tuned model and the original version, the parameters are extracted Variety.

He first produced a 524k contextual version, and then updated the 1048k version.

First of all, the Gradient team continued training based on the original Llama 3 70B Instruct and obtained Llama-3-70B-Instruct-Gradient-1048k.

The specific method is as follows:

• Adjust position encoding: Initialize RoPE theta with NTK-aware interpolation Optimal scheduling, optimization to prevent loss of high-frequency information after extending the length
• Progressive training: Proposed by the UC Berkeley Pieter Abbeel team The Blockwise RingAttention method extends the context length of the model

It is worth noting that the team layered parallelization on top of Ring Attention through a custom network topology to better utilize large GPU clusters to cope with device-to-device The network bottleneck caused by transferring many KV blocks between nodes.

Ultimately, the training speed of the model was increased by 33 times.

#In long text retrieval performance evaluation, only in the most difficult version, errors are prone to occur when the "needle" is hidden in the middle of the text.

After having the fine-tuned model with extended context, use the open source tool Mergekit to compare the fine-tuned model and the basic model, and extract the difference in parameters as LoRA.

Also using Mergekit, you can merge the extracted LoRA into other models with the same architecture.

The merge code is also open sourced by Eric Hartford on GitHub, with only 58 lines.

It is unclear whether this LoRA merge will work with Llama 3, which is fine-tuned on Chinese.

However, it can be seen that the Chinese developer community has paid attention to this development.

524k version LoRA: https://huggingface.co/cognitivecomputations/Llama-3-70B-Gradient-524k-adapter

1048k version LoRA: https://huggingface.co/cognitivecomputations/Llama-3-70B-Gradient-1048k-adapter

Merge code: https://gist.github.com/ehartford/731e3f7079db234fa1b79a01e09859ac

The above is the detailed content of 58 lines of code scale Llama 3 to 1 million contexts, any fine-tuned version is applicable. For more information, please follow other related articles on the PHP Chinese website!