Stanford University releases AI basic model transparency index, Llama 2 ranks first but 'fails'

IT House News on October 20th, Stanford University recently released the "Transparency Index" of the AI ​​basic model. The highest display index is Meta's Lama 2, but the related "transparency" is only 54%, so the researchers believe , almost all AI models on the market "lack transparency."

It is reported that this research was led by Rishi Bommasani, head of the HAI Center for Basic Model Research (CRFM), and investigated the 10 most popular basic models overseas:

• Meta’s Llama 2,
• BloomZ by BigScience,
• OpenAI’s GPT-4,
• Stability AI’s Stable Diffusion,
• Claude,
of Anthropic PBC
• Google’s PaLM 2,
• Cohere's Command,
• Jurassic-2,
by AI21 Labs
• Inflection AI’s Inflection、
• Amazon’s Titan.

Rishi Bommasani believes that "lack of transparency" has always been a problem faced by the AI ​​industry. In terms of specific model "transparency indicators", IT House found that the relevant evaluation content mainly revolves around "model training data set copyright", "training model "Computing resources used", "Credibility of the content generated by the model", "Model's own capabilities", "Risk of the model being induced to generate harmful content", "User privacy of using the model", etc., totaling 100 items.

The final survey showed that Meta’s Lama 2 topped the list with 54% transparency, while OpenAI’s GPT-4 had only 48% transparency, and Google’s PaLM 2 ranked fifth with 40%.

▲ Picture source Stanford University

Among the specific indicators, the top ten models with the "best" score performance are "Model Basics". This evaluation content mainly includes "whether the model, scale, and model of the model are accurately introduced during model training." Architecture" with an average transparency of 63%. The worst performer is Impact, which mainly evaluates whether the basic model will "retrieve user information for evaluation", with an average transparency of only 11%.

CRFM Director Percy Liang said that the "transparency" of the business base model is very important for promoting AI legislation, as well as related industries and academia.

Rishi Bommasani said that lower model transparency makes it more difficult for companies to know whether they can safely rely on relevant models, and for researchers to rely on these models to do research.

Rishi Bommasani ultimately believes that the above ten basic models all "fail" in terms of transparency. Although Meta's Llama 2 has the highest score, it cannot meet the needs of the outside world. "The model transparency must reach at least 82% to be recognized by the outside world. ".

The above is the detailed content of Stanford University releases AI basic model transparency index, Llama 2 ranks first but 'fails'. For more information, please follow other related articles on the PHP Chinese website!