If you want to learn artificial intelligence, you must master this data set. Introduction and practical use of MNIST

Learning artificial intelligence inevitably requires some data sets. For example, artificial intelligence for identifying pornography requires some similar pictures. Artificial intelligence for speech recognition and corpus are indispensable. Students who are new to artificial intelligence often worry about data sets. Today we will introduce a very simple, but very useful data set, which is MNIST. This data set is very suitable for us to learn and practice artificial intelligence-related algorithms.

The MNIST data set is a very simple data set produced by the National Institute of Standards and Technology (NIST). So what is this data set about? It's actually some handwritten Arabic numerals (ten numbers from 0 to 9).

#NIST is still very serious when producing data sets. The training set in the data set consists of handwritten digits from 250 different people, 50% of which are high school students and 50% of which are staff from the Census Bureau. The test set is also the same proportion of handwritten digit data.

How to download the MNIST data set

The MNIST data set can be downloaded from its official website (http://yann.lecun.com/exdb/mnist/). Since it is a foreign website, the download may be difficult. slow. It contains four parts:

• Training set images: train-images-idx3-ubyte.gz (9.9 MB, 47 MB ​​after decompression, contains 60,000 samples)
• Training Set labels: train-labels-idx1-ubyte.gz (29 KB, 60 KB unzipped, contains 60,000 labels)
• Test set images: t10k-images-idx3-ubyte.gz (1.6 MB, unzipped 7.8 MB after decompression, containing 10,000 samples)
• Test set labels: t10k-labels-idx1-ubyte.gz (5KB, 10 KB after decompression, containing 10,000 labels)

The above contains two types of content, one is pictures and the other is labels. Pictures and labels correspond one to one. But the picture here is not the picture file we usually see, but a binary file. This dataset stores 60,000 images in a binary format. The label is the real number corresponding to the picture.

As shown in the figure below, this article downloads the data set locally and decompresses the result. For ease of comparison, the original compressed package and the decompressed files are included.

A brief analysis of the format of the data set

Everyone has discovered that after decompression, the compressed package is not a picture, but each compressed package corresponds to a independent question. In this file, information about tens of thousands of pictures or tags is stored. So how is this information stored in this file?

In fact, the official website of MNIST gives a detailed description. Taking the image file of the training set as an example, the file format description given by the official website is as follows:

As can be seen from the above figure, the first four 32-digit numbers are the training set description information. The first is the magic number, which is a fixed value of 0x0803; the second is the number of pictures, 0xea60, which is 60000; the third and fourth are the size of the picture, that is, the picture is 28*28 pixels. The following describes each pixel in one byte. Since one byte is used to describe a pixel in this file, you can know that the value of a pixel can be from 0 to 255. Where 0 means white and 255 means black.

The format of tag files is similar to that of image files. There are two 32-digit numbers in front, the first of which is the magic number, fixed value 0x0801; the second one is used to describe the number of tags. The next data is the value of each tag, represented by one byte. The range of values ​​represented here is

#The data of the label file corresponding to the actual training set is as follows. It can be seen that it is consistent with the description of the above format. In addition, we can see that corresponding to this label set, the numbers represented by the previous pictures should be 5, 0, 4, 1, etc. respectively. Remember it here, it will be used later.

We know about the file format of the data set, let’s do it in practice.

Visual processing of data sets

After knowing the storage format of the above data, we can parse the data. For example, the following article implements a small program to parse a picture in the picture collection and obtain visual results. Of course, we can actually know what the picture is based on the value of the label set. This is just an experiment. The final result is stored in a text file, using the character "Y" to represent the handwriting and the character "0" to represent the background color. The specific program code is very simple and will not be described in detail in this article.

# -*- coding: UTF-8 -*-
def trans_to_txt(train_file, txt_file, index):
 
with open(train_file, 'rb') as sf:
with open(txt_file, "w") as wf:
offset = 16 + (28*28*index)
cur_pos = offset
count = 28*28
strlen = 1 
out_count = 1
while cur_pos < offset+count:
sf.seek(cur_pos)
data = sf.read(strlen)
res = int(data[0])

#虽然在数据集中像素是1-255表示颜色，这里简化为Y
if res > 0 :
wf.write(" Y ")
else:
wf.write(" 0 ")

#由于图片是28列，因此在此进行换行
if out_count % 28 == 0 :
wf.write("n")

cur_pos += strlen
out_count += 1

trans_to_txt("../data/train-images.idx3-ubyte", "image.txt", 0)

When we run the above code, we can get a file named image.txt. You can see the contents of the file as follows. The red notes were added later, mainly for better visibility. As you can see from the picture, this is actually a handwritten "5".

# Previously we visually analyzed the data set through the native Python interface. Python has many already implemented library functions, so we can simplify the above functions through a library function.

Parsing data based on third-party libraries

It is slightly complicated to implement using the native Python interface. We know that Python has many third-party libraries, so we can use third-party libraries to parse and display the data set. The specific code is as follows.

# -*- coding: utf-8 -*-
import os
import struct
import numpy as np

# 读取数据集，以二维数组的方式返回图片信息和标签信息
def load_mnist(path, kind='train'):
 # 从指定目录加载数据集
labels_path = os.path.join(path,
 '%s-labels.idx1-ubyte'
 % kind)
images_path = os.path.join(path,
 '%s-images.idx3-ubyte'
 % kind)
with open(labels_path, 'rb') as lbpath:
magic, n = struct.unpack('>II',
 lbpath.read(8))
labels = np.fromfile(lbpath,
 dtype=np.uint8)

with open(images_path, 'rb') as imgpath:
#解析图片信息，存储在images中
magic, num, rows, cols = struct.unpack('>IIII',
 imgpath.read(16))
images = np.fromfile(imgpath,
 dtype=np.uint8).reshape(len(labels), 784)

return images, labels

# 在终端打印某个图片的数据信息
def print_image(data, index):
idx = 0;
count = 0;
for item in data[index]:
if count % 28 == 0:
print("")

if item > 0:
print("33[7;31mY 33[0m", end="")
else:
print("0 ", end="")

count += 1

def main():
cur_path = os.getcwd()
cur_path = os.path.join(cur_path, "..data")
imgs, labels = load_mnist(cur_path)
print_image(imgs, 0)


if __name__ == "__main__":
main()

The above code is divided into two steps. The first step is to parse the data set into an array, and the second step is to display a certain picture in the array. The display here is also through a text program, but it is not stored in a file, but printed on the terminal. For example, if we still print the first picture, the effect is as follows:

The presentation of the above results only simulates the picture through characters. In fact, we can use third-party libraries to achieve more perfect image presentation. Next, we introduce how to present pictures through the matplotlib library. This library is very useful, and I will come into contact with this library later.

We implement a

def show_image(data, index):
fig, ax = plt.subplots(nrows=1, ncols=1, sharex=True, sharey=True, )

img = data[0].reshape(28, 28)
ax.imshow(img, cmap='Greys', interpolation='nearest')

ax.set_xticks([])
ax.set_yticks([])
plt.tight_layout()
plt.show()

At this point you can see that

There may be some third-party libraries missing when implementing the above functions, such as matplotlib etc. At this time, we need to install it manually. The specific method is as follows:

pip install matplotlib -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com

Data analysis based on TensorFlow

MNIST is so famous that TensorFlow already supports it. Therefore, we can load and parse it through TensorFlow. Below we give the code implemented with TensorFlow.

# -*- coding: utf-8 -*-
from tensorflow.examples.tutorials.mnist import input_data
import pylab

def show_mnist():
# 通过TensorFlow库解析数据
mnist = input_data.read_data_sets("../data", one_hot=True)
im = mnist.train.images[0]
im = im.reshape(28 ,28)
# 进行绘图
pylab.imshow(im, cmap='Greys', interpolation='nearest')
pylab.show()

if __name__ == "__main__":
show_mnist()

The final effect achieved by this code is the same as the previous example, so I won’t go into details here.

The above is the detailed content of If you want to learn artificial intelligence, you must master this data set. Introduction and practical use of MNIST. For more information, please follow other related articles on the PHP Chinese website!