Understanding the training method of autoencoders: starting with architectural exploration

Noisy data is one of the common problems in machine learning, and autoencoders are an effective method to solve such problems. This article will introduce the structure and correct training method of autoencoders.

An autoencoder is an unsupervised learning artificial neural network used to learn to encode data. Its goal is to capture the key features of the input image by training the network and convert it into a low-dimensional representation, which is often used for dimensionality reduction processing.

The architecture of the autoencoder

The autoencoder consists of 3 parts:

1 .Encoder: A module that compresses the training-validation-test set input data into an encoded representation, usually several orders of magnitude smaller than the input data.

2. Bottleneck: The module that contains compressed knowledge representation and is therefore the most important part of the network.

3. Decoder: A module that helps the network “decompress” the knowledge representation and reconstruct the data from its encoded form. The output is then compared to the ground truth.

The whole architecture looks like this, as shown below:

The relationship between encoder, bottleneck and decoder

Encoder

The encoder is a set of convolutional blocks followed by a pooling module that compresses the input of the model into a compact part called the bottleneck.

After the bottleneck is the decoder, which consists of a series of upsampling modules used to restore the compressed features to image form. In the case of a simple autoencoder, the output is expected to be the same as the noise-reduced input.

However, with variational autoencoders, it is a completely new image formed from the information provided by the model as input.

Bottleneck

As the most important part of the neural network, it will limit the flow of information from the encoder to the decoder, allowing only the most important information pass.

Since the bottleneck is designed to capture the feature information possessed by the image, we can say that the bottleneck helps form the knowledge representation of the input. The encoder-decoder structure helps us extract more information from the image in the form of data and establish useful correlations between various inputs in the network.

The bottleneck of a compressed representation of the input further prevents the neural network from memorizing the input and overfitting the data. The smaller the bottleneck, the lower the risk of overfitting. But very small bottlenecks limit the amount of information that can be stored, which increases the chance of important information leaking out of the encoder's pooling layer.

Decoder

Finally, the decoder is a set of upsampling and convolution blocks used to reconstruct the output of the bottleneck.

Since the input to the decoder is a compressed knowledge representation, the decoder acts as a "decompressor" and reconstructs the image from its latent properties.

After understanding the results and relationships of the autoencoder, let's look at how to correctly train the autoencoder.

How to train an autoencoder?

Four hyperparameters need to be set before training the autoencoder:

1. Code size

Code size or bottleneck size is the most important hyperparameter for tuning autoencoders. The bottleneck size determines how much data must be compressed. This can also be used as a regularization term.

2. Number of layers

#As with all neural networks, an important hyperparameter for tuning the autoencoder is the encoder and decoder depth. While higher depths increase model complexity, lower depths process faster.

3. Number of nodes per layer

#The number of nodes per layer defines the weight we use for each layer. Typically, the number of nodes decreases with each subsequent layer in an autoencoder because the input to each of these layers becomes smaller across layers.

4. Reconstruction Loss

The loss function we use to train the autoencoder is highly dependent on the input and we want the autoencoder to adapt to Output type. If we deal with image data, the most popular reconstruction loss functions are MSE loss function and L1 loss function. We can also use binary cross-entropy as the reconstruction loss if the input and output are in the range [0,1], like in the MNIST dataset.

The above is the detailed content of Understanding the training method of autoencoders: starting with architectural exploration. For more information, please follow other related articles on the PHP Chinese website!