search
HomeTechnology peripheralsAITsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars

In natural language processing, a lot of information is actually repeated.

If the prompt words can be effectively compressed, it is equivalent to expanding the length of the context supported by the model to some extent.

Existing information entropy methods reduce this redundancy by removing certain words or phrases.

However, the calculation based on information entropy only covers the one-way context of the text and may ignore key information required for compression; moreover, the calculation method of information entropy is not fully consistent with the compression tips the actual purpose of the word.

To meet these challenges, researchers from Tsinghua University and Microsoft jointly proposed a new data processing process called LLMLingua-2. It aims to extract knowledge from large language models (LLM) and achieve information refinement by compressing prompt words while ensuring that key information is not lost.

Tsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars

The project has gained 3.1k stars on GitHub

The results show that LLMLingua-2 can The text length is significantly reduced to the original 20%, effectively reducing processing time and costs.

In addition, LLMLingua 2’s processing speed is increased by 3 to 6 times compared to the previous version of LLMLingua and other similar technologies.

Tsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars

Paper address: https://arxiv.org/abs/2403.12968

In this process , raw text is first fed into the model.

The model will evaluate the importance of each word and decide whether to retain or delete it, while also taking into account the relationship between words.

Finally, the model will select those words with the highest scores to form a shorter prompt word.

Tsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars

The team tested the LLMLingua-2 model on multiple datasets including MeetingBank, LongBench, ZeroScrolls, GSM8K and BBH.

Although this model is small, it achieves significant performance improvements in benchmark tests and demonstrates its performance on different large language models (from GPT-3.5 to Mistral- 7B) Excellent generalization ability across languages ​​(from English to Chinese).

System prompt:

As an outstanding linguist, you are good at converting long Condensing paragraphs of text into brief expressions by removing unimportant words while retaining as much information as possible.

User Tips:

Please compress the given text into A short expression that allows you (GPT-4) to restore the original text as accurately as possible. Different from regular text compression, I need you to follow the following five conditions:

1. Only remove unimportant words.

2. Keep the order of the original words unchanged.

3. Keep the original vocabulary unchanged.

4. Do not use any abbreviations or emoticons.

5. Do not add any new words or symbols.

Please compress the original text as much as possible while retaining as much information as possible. If you understand, please compress the following text: {Text to be compressed}

The compressed text is: [...]

Tsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars

The results show that LLMLingua-2 significantly outperforms the original LLMLingua model and other selective context strategies in multiple language tasks such as question and answer, summary writing, and logical reasoning.

It is worth mentioning that this compression method is equally effective for different large language models (from GPT-3.5 to Mistral-7B) and different languages ​​(from English to Chinese) .

Moreover, the deployment of LLMLingua-2 can be achieved with just two lines of code.

Currently, the model has been integrated into the widely used RAG frameworks LangChain and LlamaIndex.

Implementation method

In order to overcome the problems faced by existing information entropy-based text compression methods, LLMLingua-2 adopts an innovation data extraction strategy.

This strategy extracts essential information from large language models such as GPT-4, achieving efficient text editing without losing key content and avoiding adding erroneous information. compression.

Prompt Design

To fully utilize the text compression potential of GPT-4, the key lies in how to set the precise compression instructions.

That is, when compressing text, instruct GPT-4 to only remove those words that are not so important in the original text, while avoiding the introduction of any new words in the process.

The purpose of this is to ensure that the compressed text maintains the authenticity and integrity of the original text as much as possible.

Tsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars

##Annotation and filtering

The researchers used the GPT-4 Using the knowledge extracted from large language models, a novel data annotation algorithm was developed.

This algorithm can mark each word in the original text and clearly indicate which words must be retained during the compression process.

In order to ensure the high quality of the constructed data set, they also designed two quality monitoring mechanisms specifically to identify and exclude those data samples with poor quality.

Tsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars

Compressor

Finally, the researchers solved the problem of text compression It is transformed into a task of classifying each word (Token), and a powerful Transformer is used as the feature extractor.

This tool can understand the context of text to accurately capture the information critical for text compression.

By training on a carefully constructed data set, the researchers' model is able to calculate a probability value to decide whether a word should be retained based on its importance. In the final compressed text, it should still be discarded.

Tsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars

Performance Evaluation

The researchers tested the performance of LLMLingua-2 on a range of tasks, These tasks include context learning, text summarization, dialogue generation, multi- and single-document question answering, code generation, and synthesis tasks, including both in-domain and out-of-domain datasets.

Test results show that the researchers’ method reduces minimal performance loss while maintaining high performance, and performs outstandingly among task-unspecific text compression methods.

- In-domain test (MeetingBank)

The researchers compared the performance of LLMLingua-2 on the MeetingBank test set with Other powerful baseline methods are compared.

Although their model size is much smaller than the LLaMa-2-7B used in the baseline, the researchers' method not only significantly improved performance on question answering and text summarization tasks, but also was consistent with The original text prompts performed similarly.

Tsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars

##-Out-of-domain testing (LongBench, GSM8K and BBH)

Considering that the researchers’ model was only trained on MeetingBank’s meeting record data, the researchers further explored its generalization capabilities in different scenarios such as long text, logical reasoning, and contextual learning.

It is worth mentioning that although LLMLingua-2 was only trained on one dataset, in out-of-domain testing, its performance was not only comparable to the current state-of-the-art task-independent compression The methods are comparable, and in some cases even better.

Tsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars

Even the researchers’ smaller model (BERT-base size) was able to achieve comparable performance to the original hint, in some cases Down or even slightly above the original tip.

While the researchers’ approach achieved promising results, it still has shortcomings when compared with other task-aware compression methods, such as LongLLMlingua on Longbench.

The researchers attribute this performance gap to the extra information they get from the questions. However, the researchers' model is task-agnostic, making it an efficient option with good generalizability when deployed in different scenarios.

Tsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars

Table 4 above lists the results of different methods using Mistral-7Bv0.1 4 as the target LLM.

Compared with other baseline methods, the researchers' method has a significant improvement in performance, demonstrating its good generalization ability on the target LLM.

It is worth noting that LLMLingua-2 performs even better than the original prompt.

Researchers speculate that Mistral-7B may not be as good at managing long contexts as GPT-3.5-Turbo.

The researchers’ approach effectively improves Mistral7B’s final inference performance by providing short hints with higher information density.

Tsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars

Table 5 above shows the latency of different systems on the V100-32G GPU with different compression ratios.

The results show that compared with other compression methods, LLMLlingua2 has much less computational overhead and can achieve an end-to-end speed improvement of 1.6 times to 2.9 times.

In addition, the researchers' method can reduce GPU memory costs by 8 times, thus reducing the demand for hardware resources.

Context-Aware Observations The researchers observed that as the compression ratio increases, LLMLingua-2 can effectively maintain the most informative words with complete context.

This is thanks to the adoption of a bidirectional context-aware feature extractor and a strategy that is explicitly optimized towards the goal of timely compression.

Tsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars

The researchers observed that as the compression ratio increases, LLMLingua-2 can effectively maintain the most informative words related to the complete context. .

This is thanks to the adoption of a bidirectional context-aware feature extractor and a strategy that is explicitly optimized towards the goal of timely compression.

Tsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars

Finally the researchers had GPT-4 reconstruct the original tones from the LLMLlingua-2 compression prompts.

The results show that GPT-4 can effectively reconstruct the original tip, indicating that no essential information is lost during LLMLingua-2 compression.

The above is the detailed content of Tsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:51CTO.COM. If there is any infringement, please contact admin@php.cn delete
The AI Skills Gap Is Slowing Down Supply ChainsThe AI Skills Gap Is Slowing Down Supply ChainsApr 26, 2025 am 11:13 AM

The term "AI-ready workforce" is frequently used, but what does it truly mean in the supply chain industry? According to Abe Eshkenazi, CEO of the Association for Supply Chain Management (ASCM), it signifies professionals capable of critic

How One Company Is Quietly Working To Transform AI ForeverHow One Company Is Quietly Working To Transform AI ForeverApr 26, 2025 am 11:12 AM

The decentralized AI revolution is quietly gaining momentum. This Friday in Austin, Texas, the Bittensor Endgame Summit marks a pivotal moment, transitioning decentralized AI (DeAI) from theory to practical application. Unlike the glitzy commercial

Nvidia Releases NeMo Microservices To Streamline AI Agent DevelopmentNvidia Releases NeMo Microservices To Streamline AI Agent DevelopmentApr 26, 2025 am 11:11 AM

Enterprise AI faces data integration challenges The application of enterprise AI faces a major challenge: building systems that can maintain accuracy and practicality by continuously learning business data. NeMo microservices solve this problem by creating what Nvidia describes as "data flywheel", allowing AI systems to remain relevant through continuous exposure to enterprise information and user interaction. This newly launched toolkit contains five key microservices: NeMo Customizer handles fine-tuning of large language models with higher training throughput. NeMo Evaluator provides simplified evaluation of AI models for custom benchmarks. NeMo Guardrails implements security controls to maintain compliance and appropriateness

AI Paints A New Picture For The Future Of Art And DesignAI Paints A New Picture For The Future Of Art And DesignApr 26, 2025 am 11:10 AM

AI: The Future of Art and Design Artificial intelligence (AI) is changing the field of art and design in unprecedented ways, and its impact is no longer limited to amateurs, but more profoundly affecting professionals. Artwork and design schemes generated by AI are rapidly replacing traditional material images and designers in many transactional design activities such as advertising, social media image generation and web design. However, professional artists and designers also find the practical value of AI. They use AI as an auxiliary tool to explore new aesthetic possibilities, blend different styles, and create novel visual effects. AI helps artists and designers automate repetitive tasks, propose different design elements and provide creative input. AI supports style transfer, which is to apply a style of image

How Zoom Is Revolutionizing Work With Agentic AI: From Meetings To MilestonesHow Zoom Is Revolutionizing Work With Agentic AI: From Meetings To MilestonesApr 26, 2025 am 11:09 AM

Zoom, initially known for its video conferencing platform, is leading a workplace revolution with its innovative use of agentic AI. A recent conversation with Zoom's CTO, XD Huang, revealed the company's ambitious vision. Defining Agentic AI Huang d

The Existential Threat To UniversitiesThe Existential Threat To UniversitiesApr 26, 2025 am 11:08 AM

Will AI revolutionize education? This question is prompting serious reflection among educators and stakeholders. The integration of AI into education presents both opportunities and challenges. As Matthew Lynch of The Tech Edvocate notes, universit

The Prototype: American Scientists Are Looking For Jobs AbroadThe Prototype: American Scientists Are Looking For Jobs AbroadApr 26, 2025 am 11:07 AM

The development of scientific research and technology in the United States may face challenges, perhaps due to budget cuts. According to Nature, the number of American scientists applying for overseas jobs increased by 32% from January to March 2025 compared with the same period in 2024. A previous poll showed that 75% of the researchers surveyed were considering searching for jobs in Europe and Canada. Hundreds of NIH and NSF grants have been terminated in the past few months, with NIH’s new grants down by about $2.3 billion this year, a drop of nearly one-third. The leaked budget proposal shows that the Trump administration is considering sharply cutting budgets for scientific institutions, with a possible reduction of up to 50%. The turmoil in the field of basic research has also affected one of the major advantages of the United States: attracting overseas talents. 35

All About Open AI's Latest GPT 4.1 Family - Analytics VidhyaAll About Open AI's Latest GPT 4.1 Family - Analytics VidhyaApr 26, 2025 am 10:19 AM

OpenAI unveils the powerful GPT-4.1 series: a family of three advanced language models designed for real-world applications. This significant leap forward offers faster response times, enhanced comprehension, and drastically reduced costs compared t

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

WebStorm Mac version

WebStorm Mac version

Useful JavaScript development tools

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

EditPlus Chinese cracked version

EditPlus Chinese cracked version

Small size, syntax highlighting, does not support code prompt function

DVWA

DVWA

Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

SublimeText3 English version

SublimeText3 English version

Recommended: Win version, supports code prompts!