


No need for 4 H100! 34 billion parameter Code Llama can be run on Mac, 20 tokens per second, best at code generation
Georgi Gerganov, a developer in the open source community, found that he could run the 34B Code Llama model with full F16 precision on M2 Ultra, and the inference speed exceeded 20 token/s.
The M2 Ultra has a bandwidth of 800GB/s, which others usually need to use 4 high-end GPUs to achieve
The real answer behind this is: Speculative Sampling.
George's discovery immediately triggered a discussion among the big guys in the artificial intelligence industry
Karpathy forwarded a comment, "The speculative execution of LLM is a An excellent inference time optimization."
"Speculative sampling" accelerates reasoning
In this example, Georgi uses the Q4 7B quantum draft The model (namely Code Llama 7B) was speculatively decoded and then generated using Code Llama34B on the M2 Ultra.
To put it simply, use a "small model" to make a draft, and then use the "large model" to check and make corrections to speed up the entire process.
GitHub address: https://twitter.com/ggerganov/status/1697262700165013689
according to According to Georgi, the speeds of these models are as follows:
F16 34B: about 10 tokens per second
What needs to be rewritten is: Q4 7B: About 80 tokens per second
The following is A standard F16 sampling example without using speculative sampling:
After adding the speculative sampling strategy, the speed can reach about 20 marks per second
According to Georgi, the speed at which content is generated may vary. However, this approach seems to be very effective for code generation, as most vocabularies can be correctly guessed by the draft model.
Use cases using "grammar sampling" are also likely to benefit greatly from it.
#How does speculative sampling achieve fast inference?
Karpathy made an explanation based on three previous studies by Google Brain, UC Berkeley, and DeepMind.
Please click the following link to view the paper: https://arxiv.org/pdf/2211.17192.pdf
Paper address: https://arxiv.org/pdf/1811.03115.pdf
Paper Address: https://arxiv.org/pdf/2302.01318.pdf
This depends on the following unintuitive observation:
In The time required to forward LLM on a single input token is the same as the time required to forward LLM on K input tokens in batches (K is larger than you think).
This unintuitive fact is because sampling is severely limited by memory. Most of the "work" is not calculated, but the weights of the Transformer are read from VRAM into the on-chip cache. deal with.
To accomplish the task of reading all the weights, it is better to apply them to the entire batch of input vectors
The reason why we cannot naively exploit this fact is Sampling K tokens at a time is because each N token depends on the token we sampled at step N-1. This is a serial dependency, so the baseline implementation just proceeds one by one from left to right.
Now, a clever idea is to use a small and cheap draft model to first generate a candidate sequence composed of K markers - a "draft". We then batch feed all this information together into the big model
According to the above method, this is almost as fast as inputting just one token.
Then, we examine the model from left to right, and the logits predicted by the sample token. Any sample that matches the draft allows us to immediately jump to the next token.
If there is a disagreement, we will abandon the draft model and bear the cost of doing some one-time work (sampling the draft model and doing a forward pass on subsequent tokens)
The reason this works in practice is that draft tokens will be accepted in most cases, and because they are simple tokens, even smaller draft models can accept them.
When these simple tokens are accepted, we will skip these parts. Difficulty tokens that the large model disagrees with will "fall back" to the original speed, but will actually be slower because of the extra work.
So, in summary: this weird trick works because LLM is memory-constrained during inference. In the case of "batch size 1", a single sequence of interest is sampled, which is the case for most "local LLM" use cases. Moreover, most tokens are "simple".
The co-founder of HuggingFace said that the 34 billion parameter model looked very large and difficult outside the data center a year and a half ago. manage. Now you can easily handle it with just a laptop
Today’s LLM is not a single breakthrough, but requires the effective coordination of multiple important components working system. Speculative Decoding is a great example that helps us think from a systems perspective.
The above is the detailed content of No need for 4 H100! 34 billion parameter Code Llama can be run on Mac, 20 tokens per second, best at code generation. For more information, please follow other related articles on the PHP Chinese website!

The legal tech revolution is gaining momentum, pushing legal professionals to actively embrace AI solutions. Passive resistance is no longer a viable option for those aiming to stay competitive. Why is Technology Adoption Crucial? Legal professional

Many assume interactions with AI are anonymous, a stark contrast to human communication. However, AI actively profiles users during every chat. Every prompt, every word, is analyzed and categorized. Let's explore this critical aspect of the AI revo

A successful artificial intelligence strategy cannot be separated from strong corporate culture support. As Peter Drucker said, business operations depend on people, and so does the success of artificial intelligence. For organizations that actively embrace artificial intelligence, building a corporate culture that adapts to AI is crucial, and it even determines the success or failure of AI strategies. West Monroe recently released a practical guide to building a thriving AI-friendly corporate culture, and here are some key points: 1. Clarify the success model of AI: First of all, we must have a clear vision of how AI can empower business. An ideal AI operation culture can achieve a natural integration of work processes between humans and AI systems. AI is good at certain tasks, while humans are good at creativity and judgment

Meta upgrades AI assistant application, and the era of wearable AI is coming! The app, designed to compete with ChatGPT, offers standard AI features such as text, voice interaction, image generation and web search, but has now added geolocation capabilities for the first time. This means that Meta AI knows where you are and what you are viewing when answering your question. It uses your interests, location, profile and activity information to provide the latest situational information that was not possible before. The app also supports real-time translation, which completely changed the AI experience on Ray-Ban glasses and greatly improved its usefulness. The imposition of tariffs on foreign films is a naked exercise of power over the media and culture. If implemented, this will accelerate toward AI and virtual production

Artificial intelligence is revolutionizing the field of cybercrime, which forces us to learn new defensive skills. Cyber criminals are increasingly using powerful artificial intelligence technologies such as deep forgery and intelligent cyberattacks to fraud and destruction at an unprecedented scale. It is reported that 87% of global businesses have been targeted for AI cybercrime over the past year. So, how can we avoid becoming victims of this wave of smart crimes? Let’s explore how to identify risks and take protective measures at the individual and organizational level. How cybercriminals use artificial intelligence As technology advances, criminals are constantly looking for new ways to attack individuals, businesses and governments. The widespread use of artificial intelligence may be the latest aspect, but its potential harm is unprecedented. In particular, artificial intelligence

The intricate relationship between artificial intelligence (AI) and human intelligence (NI) is best understood as a feedback loop. Humans create AI, training it on data generated by human activity to enhance or replicate human capabilities. This AI

Anthropic's recent statement, highlighting the lack of understanding surrounding cutting-edge AI models, has sparked a heated debate among experts. Is this opacity a genuine technological crisis, or simply a temporary hurdle on the path to more soph

India is a diverse country with a rich tapestry of languages, making seamless communication across regions a persistent challenge. However, Sarvam’s Bulbul-V2 is helping to bridge this gap with its advanced text-to-speech (TTS) t


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Safe Exam Browser
Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

SublimeText3 Mac version
God-level code editing software (SublimeText3)

mPDF
mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

Notepad++7.3.1
Easy-to-use and free code editor

WebStorm Mac version
Useful JavaScript development tools
