Neural networks also have spatial awareness! Learn to create maps in Minecraft, published in Nature sub-magazine

This is the first time humans have demonstrated that neural networks can create their own maps. Imagine that you are in a strange town. Even if the surrounding environment is unfamiliar at first, you can explore around and eventually draw a map of the environment in your brain, which includes buildings, streets, signs, etc. that interact with each other. positional relationship between them. This ability to construct spatial maps in the brain underlies higher-order types of cognition in humans: for example, language is theorized to be encoded by map-like structures in the brain. However, even the most advanced artificial intelligence and neural networks cannot build such a map out of thin air. Matt Thomson, assistant professor of computational biology and a researcher at Heritage Medical Research Institute, said: "There is a sense that even the most advanced artificial intelligence models are not truly intelligent. They cannot solve problems like us; they cannot prove unproven Mathematical results also fail to generate new ideas. "We think this is because they cannot navigate in conceptual space; solving complex problems is like moving in conceptual space, like navigation. AI does more like rote learning. Back - you give it an input, and it gives you a response. But it can't synthesize different ideas." Recently, a new paper from the Thomson Laboratory found that neural networks can use an algorithm called "predictive coding." Construct spatial maps. The paper was published in the journal Nature Machine Intelligence on July 18.

1. Paper address: https://www.nature.com/articles/s42256-024-00863-1

2. Code address: https://github.com/jgornet/predictive-coding-recovers-maps

Graduate student James Gornet led the team to build an environment in Minecraft, incorporating complex elements (such as trees, rivers, caves). They recorded videos of players randomly walking through the area and used the videos to train a neural network equipped with a predictive coding algorithm.

Research has found that neural networks learn how objects in the Minecraft world are organized and can "predict" the environment players will encounter when moving through space.

The combination of the predictive coding algorithm and the Minecraft game successfully "taught" the neural network how to create spatial maps and then use these spatial maps to predict subsequent frames of the video. As a result, the mean square error between the predicted image and the final image was only is 0.094%.

More importantly, the research team "opened" the neural network (equivalent to examining the internal structure) and found that the representations of various objects are spatially stored relative to each other. In other words, they saw a map of the Minecraft environment stored in a neural network.

Neural networks can navigate maps given to them by human designers, such as self-driving cars using GPS, but this is the first time humans have demonstrated that neural networks can create their own maps. This ability to store and organize information spatially will eventually help neural networks become more "smart," allowing them to solve truly complex problems like humans.

This project demonstrates the true spatial awareness capabilities of AI, which is still not seen in technologies like OpenAI’s Sora, which has some weird glitches.

James Gornet is a student in the Computational and Neural Systems (CNS) Department at Caltech, which covers neuroscience, machine learning, mathematics, statistics, and biology.

“The CNS program really provides a place for James to do unique work that wouldn’t be possible elsewhere,” Thomson said. “We are taking a biologically inspired machine learning approach that allows us to reverse-engineer the properties of the brain in artificial neural networks, and we hope to understand the brain in turn. At Caltech, we have a very receptive team to this kind of work. Community. 》

Neural Networks that Perform Predictive Coding

Inspired by the implicit spatial representation in the predictive coding inference problem, the researchers developed a computational implementation of a predictive coding agent and studied the agent's behavior while exploring a virtual environment. Learned spatial representation.

They started by creating an environment using the Malmo environment in Minecraft. The physical environment has dimensions of 40 × 65 tiles and encompasses three aspects of the visual scene: a cave provides a global visual landmark, a forest enables similarity between visual scenes, and a river with a bridge limits the visual scene. How the agent traverses the environment (Fig. 1a).

1. The agent follows a path determined by an A* search to find the shortest path between randomly sampled locations and receives a visual image on each path.

2. To perform predictive coding, the author built an encoder-decoder convolutional neural network, the encoder adopts ResNet-18 architecture, and the decoder adopts transposed convolutional ResNet-18 architecture (Figure 1b). The encoder-decoder architecture uses the U-Net architecture to pass the encoded latent units into the decoder.
3. Multi-head attention processes encoding latent unit sequences to encode past visual observation history. The multi-headed attention has h = 8 heads. For a coding latent unit of dimension D = C × H × W, with height H, width W, and channel C, the dimensions of a single head are d = C × H × W/h.

   

   The predictive encoder adopts the mean square error minimization strategy to fit the difference between the predicted observation value and the real observation value.

Training details:

• Number of samples: 82,630
• Evolutions: 200
• Optimizer: Nesterov Momentum Gradient Descent
• Weight decay: 5 × 10^(-6)
• Initial learning rate: 10^( -1)
• Learning rate scheduling: After OneCycle

training, the predictive encoder achieves good visual fidelity, and the mean square error between the predicted image and the real image is 0.094 (shown in Figure 1c).

Please see the original paper for more details.

Reference link:

https://techxplore.com/news/2024-07-neural-network-minecraft.html

https://www.tomshardware.com/tech-industry/artificial-intelligence/neural- network-learns-to-make-maps-with-minecraft-code-available-on-github

The above is the detailed content of Neural networks also have spatial awareness! Learn to create maps in Minecraft, published in Nature sub-magazine. For more information, please follow other related articles on the PHP Chinese website!