Can fine-tuning really allow LLM to learn new things: introducing new knowledge may make the model produce more hallucinations

Large language models (LLMs) are trained on huge text databases, where they acquire large amounts of real-world knowledge. This knowledge is embedded into their parameters and can then be used when needed. The knowledge of these models is "reified" at the end of training. At the end of pre-training, the model actually stops learning.

Align the model or perform instruction tuning to let the model learn how to make full use of this knowledge and how to respond more naturally to the user's questions. But sometimes model knowledge is not enough, and although the model can access external content through RAG, it is considered beneficial to adapt the model to new domains through fine-tuning. This fine-tuning is performed using input created by human annotators or other LLMs, where the model encounters additional real-world knowledge and incorporates it into the parameters.

How does the model integrate this new additional knowledge?

At a mechanistic level, we don’t really know how this interaction occurs of. According to some, exposure to this new knowledge may cause the model to hallucinate. This is because the model is trained to generate facts that are not based on its pre-existing knowledge (or may conflict with the model's prior knowledge). There is also knowledge of what looks the model is likely to encounter (e.g., entities that appear less frequently in the pre-training corpus).

#So a recently published study focused on analyzing what happens when a model learns new knowledge through fine-tuning. The authors examine in detail what happens to a fine-tuned model and how it reacts after acquiring new knowledge.

They try to classify the examples at the knowledge level after fine-tuning. The knowledge inherent in a new example may not be completely consistent with the knowledge of the model. Examples can be known or unknown. Even if it is known, it may be highly known, it may be known, or it may be less known knowledge.

Then the author used a model (PaLM 2-M) to fine-tune it. Each nudge example is made up of factual knowledge (subjects, relations, objects). This is to allow the model to query this knowledge with specific questions, specific triples (e.g., "Where is Paris?"), and ground truth answers (e.g., "France"). In other words, they provide the model with some new knowledge, and then reconstruct these triples into questions (question-answer pairs) to test its knowledge. They group all these examples into the categories discussed above and then evaluate the answers.

Test results after fine-tuning the model: A high proportion of unknown facts leads to performance degradation (which is not compensated by longer fine-tuning time).

Unknown facts have almost a neutral effect at lower epoch numbers, but at more The number of epochs will hurt performance. So unknown examples appear to be harmful, but their negative impact is mainly reflected in the later stages of training. The graph below shows training accuracy as a function of fine-tuning duration for known and unknown subsets of the dataset example. It can be seen that the model learns unknown examples at a later stage.

Lastly, since Unknown examples are the ones that are likely to introduce new factual knowledge, their significantly slow fitting rate suggests that LLMs struggle to acquire new factual knowledge through fine-tuning , instead they learn to expose their preexisting knowledge using the Known examples. The relationship between examples is quantified and whether it is linear. The results show that there is a strong linear relationship between unknown examples hurting performance and known examples improving performance, almost as strong (the correlation coefficients in this linear regression are very close).

这种微调不仅对特定情况下的性能有影响，而且对模型知识有广泛的影响。作者使用分布外(OOD)的测试集表明，未知样本对OOD性能是有害的。根据作者的说法，这与幻觉的发生也有关系:

Overall, our insights transfer across relations. This essentially shows  that fine-tuning on Unknown examples such as “Where is [E1] located?”,  can encourage hallucinations on seemingly unrelated questions, such as  “Who founded [E2]?”.

另外一个有趣的结果是，最好的结果不是用众所周知的例子获得的，而是用可能已知的例子。换句话说，这些例子允许模型更好地利用其先验知识(过于众所周知的事实不会对模型产生有用的影响)。

相比之下，未知和不太清楚的事实会损害模型的表现，而这种下降源于幻觉的增加。

This work highlights the risk in using supervised fine-tuning to update  LLMs’ knowledge, as we present empirical evidence that acquiring new  knowledge through finetuning is correlated with hallucinations w.r.t  preexisting knowledge.

根据作者的说法，这种未知的知识可能会损害性能(这使得微调几乎毫无用处)。而用“我不知道”标记这种未知知识可以帮助减少这种伤害。

Acquiring new knowledge via supervised fine-tuning is correlated with  hallucinations w.r.t. pre-existing knowledge. LLMs struggle to integrate new knowledge through fine-tuning and mostly learn to use their  pre-existing knowledge.

综上所述，如果在微调过程中出现未知知识，则会对模型造成损害。这种性能下降与幻觉的增加有关。相比之下，可能已知的例子反而有有益的影响。这表明该模型难以整合新知识。也就是说在模型所学到的知识和它如何使用新知识之间存在冲突。这可能与对齐和指令调优有关(但是这篇论文没有研究这一点)。

所以如果想要使用具有特定领域知识的模型，论文建议最好使用RAG。并且带有“我不知道”标记的结果可以找到其他策略来克服这些微调的局限性。

这项研究是非常有意思，它表明微调的因素以及如何解决新旧知识之间的冲突仍然不清楚。这就是为什么我们要测试微调前和后结果的原因。

The above is the detailed content of Can fine-tuning really allow LLM to learn new things: introducing new knowledge may make the model produce more hallucinations. For more information, please follow other related articles on the PHP Chinese website!