A visualization tool that can find neural network bugs, included in the Nature sub-journal

Recently, the "Nature" sub-journal included a research result that can find out where neural networks go wrong. The research team provides a visualization method that uses topology to describe the relationship between the inference results of neural networks and their classification. This result can help researchers infer the specific circumstances of confusion during neural network reasoning and make artificial intelligence systems more transparent.

Researchers reveal failure points in neural network inference

1. Neural network spikes reveal inference errors:

   • Research finds data graph spikes in neural network inference that blur judgments and generate errors related.
   • Observing spikes can help identify failure points in AI systems.

2. Lack of transparency in the neural network reasoning process:

   • Neural networks are good at solving problems, but their reasoning process is opaque, raising concerns about reliability.
   • New research provides a way to discover the source of errors in neural networks.

3. The "black box" characteristics of neural networks:

   • It is difficult for neural networks to understand how to solve problems, making it difficult to judge the correctness of the answer.
   • Researchers cannot trace the decision-making process of the neural network for a single sample.

4. Visualize decision results:

   • Instead of tracking the decisions of individual samples, the researchers visualized the relationship between the neural network's decision results for the entire database and the samples.
   • This helps identify images with higher multi-class probability.

5. Topological data analysis:

   • Researchers use topology to plot the relationship between inference results and classification.
   • Topological data analysis tools help identify similarities between data sets.
   • This tool has been used to analyze the relationship between breast cancer subgroups and genes.

     

     Paper link: https://www.nature.com/articles/s42256-023-00749-8

In the relationship diagram generated based on the new research results:

• Each point represents a neural network Groups of images considered related
• Maps of different categories are represented by different colors
• The closer the distance between the points, the more similar the neural network considers each group of images

Most of these maps show a single color point group.

Two overlapping points of different colors indicate images that have a high probability of belonging to multiple categories. “Our approach is able to build a map-like relationship diagram that zooms in on certain areas of the data,” Gleich said. “These areas are often places where the boundaries of certain categories are not clear, where the solution may not be so clear. However, it can highlight specific data predictions that deserve further study. The maps generated by the new method can show areas that the network cannot classify. This method provides a way for "researchers to use the innate human way of thinking to speculate on the reasoning of neural networks." "This allows us to predict how the network will respond to novel inputs based on what we know about it," Gleich said. The team found that neural networks are particularly prone to confusing patterns in categories such as chest X-rays, genetic sequences, and clothing. For example, when a network was tested on the Imagenette database (a subset of ImageNet), it repeatedly classified pictures of cars as tape players. They found this was because the images were pulled from online shopping listings and contained labels for car audio equipment.
The team’s new approach helps reveal “what went wrong.” Gleich said: "Analyzing data at this level allows scientists to go from just making a bunch of useful predictions on new data to deeply understanding how neural networks might process their data."
"Our tool seems to be very It’s good at helping find if the training data itself contains errors,” Gleich said. “People do make mistakes when manually labeling data.”
Potential uses for this analysis strategy may include particularly important applications of neural networks. Consider, for example, the application of neural networks in healthcare or medicine to study sepsis or skin cancer.
Critics argue that because most neural networks are trained on past decisions that reflect pre-existing biases against human groups, AI systems will end up replicating past mistakes. Finding a way to "understand bias or bias in predictions" using new tools could be a significant advance, Gleich said.
Gleich said the new tool can be used with neural networks to generate specific predictions from small data sets, such as "whether a genetic mutation is likely to be harmful." But so far, researchers have no way to apply it to large language models or diffusion models.
For more information, please refer to the original paper.
Reference content:
https://spectrum.ieee.org/ai-mistakes
https://www.cs.purdue.edu/homes/liu1740/
https://www.cs.purdue.edu/homes/ tamaldey/
https://www.cs.purdue.edu/homes/dgleich/

The above is the detailed content of A visualization tool that can find neural network bugs, included in the Nature sub-journal. For more information, please follow other related articles on the PHP Chinese website!