Intel open source NPU acceleration library, Core Ultra processor AI PC can run lightweight large language models

News on March 4th showed that Intel recently released its NPU acceleration library on GitHub. This move enables AI PCs equipped with Core Ultra processors to run lightweight large-scale applications such as TinyLlama and Gemma-2b more smoothly. Language model.

The Core Ultra series integrates the NPU AI engine for the first time. This engine can handle some lightweight AI inference tasks and work together with the CPU and GPU to meet various needs. Requirements for AI applications.

It is understood that although the NPU acceleration library released this time is mainly prepared for developers, those who have certain programming experience Users can also try it. Tony Mongkolsmai, a software architect at Intel, demonstrated how to run an AI chatbot based on the 1.1 billion parameter TinyLlama large model on an MSI Monarch 14 AI Evo laptop, which can conduct simple conversations. At the same time, Windows Task Manager also shows valid calls to the NPU.

However, the current open source NPU acceleration library still has some shortcomings in functionality. It supports 8-bit quantization and FP16 precision, but does not yet support 4-bit quantization and BF16 precision. As well as advanced functions such as NPU/GPU hybrid computing, the relevant technical documentation has not yet been provided. However, Intel has promised to gradually expand its functions in the future, which is expected to double the existing functions, which will undoubtedly bring more convenience and possibilities to AI developers.

The above is the detailed content of Intel open source NPU acceleration library, Core Ultra processor AI PC can run lightweight large language models. For more information, please follow other related articles on the PHP Chinese website!