search
HomeTechnology peripheralsAIExplore a new generation of small models that go beyond GPT 3.5.

At the end of last year, OpenAI launched ChatGPT to the public. Once released, this technology immediately pushed AI-driven chatbots to the center of mainstream discourse, and many researchers discussed how it can change business, education, etc. There was another round of debate.

Subsequently, technology giants followed suit and invested in scientific research teams, and their so-called "generative AI" technology (technology that can produce conversational text, graphics, etc.) was also ready.

As we all know, ChatGPT is fine-tuned based on the GPT-3.5 series of models. We have seen many studies following closely behind it. However, with ChatGPT How good is their new research in comparison? Recently, in a paper "Multimodal Chain-of-Thought Reasoning in Language Models" released by Amazon, they proposed Multimodal-CoT that includes visual features. This architecture performed well in the ScienceQA benchmark when the number of parameters was less than 1 billion. , 16 percentage points higher than GPT-3.5 (75.17%→91.68%), even surpassing many humans.

Here is a brief introduction to the ScienceQA benchmark. It is the first multi-modal scientific question and answer data set with detailed explanations, proposed by UCLA and the Allen Institute for Artificial Intelligence (AI2). It is mainly used to test the multi-modal reasoning ability of the model. It has a very rich field diversity, covering the fields of natural science, language science and social science, and puts forward high requirements for the logical reasoning ability of the model.

超越GPT 3.5的小模型来了!

## Paper address: https://arxiv.org/abs/2302.00923

Project address: https://github.com/amazon-science/mm-cot

Let’s take a look How Amazon's language model surpasses GPT-3.5.

Multimodal-CoT including visual features

Large Language Model (LLM) performs well on complex reasoning tasks and cannot do without the assistance of Chain of Thought (CoT) prompts . However, existing CoT research only focuses on language modalities. To trigger CoT inference in multi-modality, one possible solution is to fine-tune a small language model to perform CoT inference by fusing visual and language features.

However, it has been observed that small models tend to make up things more frequently than large models. This behavior of models is often called "hallucination." A previous Google study also showed (paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models) that CoT-based prompts are only useful when the model has at least 100 billion parameters!

That said, CoT hints do not have a positive impact on the performance of small models, and only yield performance gains when used with models of ∼100B parameters.

However, this article studies performance improvement with less than 1 billion parameters. How is it achieved? To put it simply, this paper proposes Multimodal-CoT that contains visual features, and uses this paradigm (Multimodal-CoT) to find CoT reasoning in multiple modalities.

Multimodal-CoT combines visual features in a single training framework to reduce the impact of language models that have a tendency to produce illusive reasoning patterns. Overall, this framework divides the reasoning process into two parts: rationale generation (finding reasons) and answer reasoning (finding answers).

超越GPT 3.5的小模型来了!

Multimodal CoT Two-stage process: uses text (question context) and visual features to generate logical justification.

Dataset

This article mainly focuses on the ScienceQA data set. The set includes images and text as part of the context. Additionally, the dataset contains explanations of the answers so that the model can be fine-tuned to generate CoT rationales. In addition, this paper utilizes the DETR model to generate visual features.

Smaller LMs are prone to hallucinations when generating CoT/Basic Principles. The author speculates that if there is a modified architecture where the model can utilize the textual features generated by the LM and the visual features generated by the image model, then more Ability to give reasons and answer questions.

Architecture

In general, we need an architecture that can generate text features and visual features and use them to generate A model for text responsiveness.

It is also known that there is some interaction between text and visual features, which is essentially some kind of joint attention mechanism, which helps to encapsulate the information present in the two modalities. , which makes it possible to learn from ideas. To accomplish all this, the authors chose the T5 model, which has an encoder-decoder architecture, and as mentioned above, the DETR model is used to generate visual features.

The encoder of the T5 model is responsible for generating text features, but the decoder of the T5 model does not use the text features generated by the encoder, but uses the joint attention interaction layer proposed by the author ( co-attention-styled interaction layer) output.

Looking at the dismantling, assume that H_language is the output of the T5 encoder. X_vision is the output of DETR. The first step is to ensure that the visual features and textual features have the same hidden size so that we can use the attention layer.

Note: All code snippets are from the GitHub of the paper: https://github.com/amazon-science/mm-cot/blob/main/model.py

self.image_dense = nn.Linear(self.patch_dim, config.d_model)

W_h is essentially a linear layer, and H_vision corresponds to the final visual features. W_h helps change the size of the visual features to match the size of the text features.

Next we need to add an attention layer so that visual and textual features can interact with each other. To do this, the authors use a single-head attention layer with H_language as the query vector and H_vision as the key and value vectors.

self.mha_layer = torch.nn.MultiheadAttention(embed_dim=config.hidden_size, 
kdim=config.hidden_size, vdim=config.hidden_size,
num_heads=1, batch_first=True)


image_att, _ = self.mha_layer(hidden_states, image_embedding, image_embedding)

Now we have an embedding that contains information from text and visual features. The authors then utilize gated fusion to generate a final set of features that will be sent to the decoder. There are two steps to gated fusion:

  1. Get a vector of scores between 0 and 1 to determine the importance of each attention feature.
  2. Use score to fuse text and attention features.

超越GPT 3.5的小模型来了!

W_I and W_v are essentially two linear layers.

self.gate_dense = nn.Linear(2*config.hidden_size, config.hidden_size) 
self.sigmoid = nn.Sigmoid()


hidden_states = encoder_outputs[0]
merge = torch.cat([hidden_states, image_att], dim=-1)
gate = self.sigmoid(self.gate_dense(merge))
hidden_states = (1 - gate) * hidden_states + gate * image_att

Finally, the fused features are passed to the decoder.

decoder_outputs = self.decoder( input_ids=decoder_input_ids, 
attention_mask=decoder_attention_mask, 
inputs_embeds=decoder_inputs_embeds, 
past_key_values=past_key_values, 
encoder_hidden_states=hidden_states,

This is pretty much the structure the author follows! However, remember that there are two phases. The first stage is to generate the rationale/CoT. The second stage utilizes the CoT produced in the first stage to generate the answer, as shown in the figure above.

Results

The authors used the weights of the UnifiedQA model as the initialization point of the T5 model and fine-tuned it on the ScienceQA dataset. They observed that their Multimodal CoT method outperformed all previous baselines, including GPT-3.5.

What’s interesting is that even the base model with only 223 million parameters outperforms GPT-3.5 and other Visual QA models! This highlights the power of having a multimodal architecture.

The authors also show that their two-stage approach outperforms the single-stage approach.

超越GPT 3.5的小模型来了!

Conclusion

The biggest gain brought by this paper is that multi-modal features are useful in solving problems with How powerful are visual and textual features when it comes to questions.

The authors show that leveraging visual features, even a small language model (LM) can produce meaningful thought chains/reasoning with much less hallucinations, which reveals that visual models The role that can be played in developing learning technologies based on thought chains.

From experiments, we see that adding visual features at the cost of millions of parameters can bring greater value than scaling a plain text model to billions of parameters.

The above is the detailed content of Explore a new generation of small models that go beyond GPT 3.5.. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:51CTO.COM. If there is any infringement, please contact admin@php.cn delete
A Comprehensive Guide to ExtrapolationA Comprehensive Guide to ExtrapolationApr 15, 2025 am 11:38 AM

Introduction Suppose there is a farmer who daily observes the progress of crops in several weeks. He looks at the growth rates and begins to ponder about how much more taller his plants could grow in another few weeks. From th

The Rise Of Soft AI And What It Means For Businesses TodayThe Rise Of Soft AI And What It Means For Businesses TodayApr 15, 2025 am 11:36 AM

Soft AI — defined as AI systems designed to perform specific, narrow tasks using approximate reasoning, pattern recognition, and flexible decision-making — seeks to mimic human-like thinking by embracing ambiguity. But what does this mean for busine

Evolving Security Frameworks For The AI FrontierEvolving Security Frameworks For The AI FrontierApr 15, 2025 am 11:34 AM

The answer is clear—just as cloud computing required a shift toward cloud-native security tools, AI demands a new breed of security solutions designed specifically for AI's unique needs. The Rise of Cloud Computing and Security Lessons Learned In th

3 Ways Generative AI Amplifies Entrepreneurs: Beware Of Averages!3 Ways Generative AI Amplifies Entrepreneurs: Beware Of Averages!Apr 15, 2025 am 11:33 AM

Entrepreneurs and using AI and Generative AI to make their businesses better. At the same time, it is important to remember generative AI, like all technologies, is an amplifier – making the good great and the mediocre, worse. A rigorous 2024 study o

New Short Course on Embedding Models by Andrew NgNew Short Course on Embedding Models by Andrew NgApr 15, 2025 am 11:32 AM

Unlock the Power of Embedding Models: A Deep Dive into Andrew Ng's New Course Imagine a future where machines understand and respond to your questions with perfect accuracy. This isn't science fiction; thanks to advancements in AI, it's becoming a r

Is Hallucination in Large Language Models (LLMs) Inevitable?Is Hallucination in Large Language Models (LLMs) Inevitable?Apr 15, 2025 am 11:31 AM

Large Language Models (LLMs) and the Inevitable Problem of Hallucinations You've likely used AI models like ChatGPT, Claude, and Gemini. These are all examples of Large Language Models (LLMs), powerful AI systems trained on massive text datasets to

The 60% Problem — How AI Search Is Draining Your TrafficThe 60% Problem — How AI Search Is Draining Your TrafficApr 15, 2025 am 11:28 AM

Recent research has shown that AI Overviews can cause a whopping 15-64% decline in organic traffic, based on industry and search type. This radical change is causing marketers to reconsider their whole strategy regarding digital visibility. The New

MIT Media Lab To Put Human Flourishing At The Heart Of AI R&DMIT Media Lab To Put Human Flourishing At The Heart Of AI R&DApr 15, 2025 am 11:26 AM

A recent report from Elon University’s Imagining The Digital Future Center surveyed nearly 300 global technology experts. The resulting report, ‘Being Human in 2035’, concluded that most are concerned that the deepening adoption of AI systems over t

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. Best Graphic Settings
4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. How to Fix Audio if You Can't Hear Anyone
4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
WWE 2K25: How To Unlock Everything In MyRise
1 months agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

DVWA

DVWA

Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

SAP NetWeaver Server Adapter for Eclipse

SAP NetWeaver Server Adapter for Eclipse

Integrate Eclipse with SAP NetWeaver application server.