How to read csv file in python

Python reads and writes csv files

Preface

Comma separated Comma-Separated Values ​​(CSV, sometimes called character-separated values ​​because the separator character can be other than a comma) files store tabular data (numbers and text) in plain text. Plain text means that the file is a sequence of characters and contains no data that must be interpreted like a binary number. A CSV file consists of any number of records, separated by some kind of newline character; each record consists of fields, and the separators between fields are other characters or strings, most commonly commas or tabs. Usually, all records have exactly the same field sequence.

Features

The data read out is generally character type. If it is a number, it needs to be converted manually. For numbers

Read data in row units

Columns are separated by half-width commas or tabs, usually half-width commas

Generally, there is no space at the beginning of each line , the first row is the attribute column, the data columns are separated by separators, there are no spaces, and there are no blank lines between rows.

It is very important that there are no blank lines between the lines. If there are blank lines or there are spaces at the end of the rows in the data set, an error will generally occur when reading the data, causing a [list index out of range] error. PS: I have been fooled by this error many times!

Writing and reading CSV files using python I/O

Writing csv files using Python I/O

The following is the code to download the "birthweight.dat" low birth weight dat file from the author's source, process it and save it to a csv file.

import csv
import os
import numpy as np
import random
import requests
# name of data file
# 数据集名称
birth_weight_file = 'birth_weight.csv'
# download data and create data file if file does not exist in current directory
# 如果当前文件夹下没有birth_weight.csv数据集则下载dat文件并生成csv文件
if not os.path.exists(birth_weight_file):
    birthdata_url = 'https://github.com/nfmcclure/tensorflow_cookbook/raw/master/01_Introduction/07_Working_with_Data_Sources/birthweight_data/birthweight.dat'
    birth_file = requests.get(birthdata_url)
    birth_data = birth_file.text.split('\r\n')
    # split分割函数,以一行作为分割函数，windows中换行符号为'\r\n',每一行后面都有一个'\r\n'符号。
    birth_header = birth_data[0].split('\t')
    # 每一列的标题，标在第一行，即是birth_data的第一个数据。并使用制表符作为划分。
    birth_data = [[float(x) for x in y.split('\t') if len(x) >= 1] for y in birth_data[1:] if len(y) >= 1]
    print(np.array(birth_data).shape)
    # (189, 9)
    # 此为list数据形式不是numpy数组不能使用np,shape函数,但是我们可以使用np.array函数将list对象转化为numpy数组后使用shape属性进行查看。
    with open(birth_weight_file, "w", newline='') as f:
    # with open(birth_weight_file, "w") as f:
        writer = csv.writer(f)
        writer.writerows([birth_header])
        writer.writerows(birth_data)
        f.close()

Common mistakes list index out of range

The key thing we need to talk about is with open(birth_weight_file, "w" , newline='') as f: this statement. Indicates writing a csv file. If the parameter newline='' is not added, it means using a space as a newline character. Instead, use the with open(birth_weight_file, "w") as f: statement. Blank rows will appear in the generated table.

It is not only necessary to use python I/O to read and write csv data, but also to use other methods to read and write csv data, or to download the csv data set from the Internet. Check whether there are any spaces after each line, or whether there are any extra blank lines. Avoid unnecessary errors that affect judgment during data analysis.

Use Python I/O to read csv files

When you use the python I/O method to read, you create a new List and then follow the order of rows first and columns ( Similar to a two-dimensional array in C language), store data into an empty List object. If you need to convert it into a numpy array, you can also use np.array(List name) to convert between objects.

birth_data = []
with open(birth_weight_file) as csvfile:
    csv_reader = csv.reader(csvfile)  # 使用csv.reader读取csvfile中的文件
    birth_header = next(csv_reader)  # 读取第一行每一列的标题
    for row in csv_reader:  # 将csv 文件中的数据保存到birth_data中
        birth_data.append(row)
birth_data = [[float(x) for x in row] for row in birth_data]  # 将数据从string形式转换为float形式
birth_data = np.array(birth_data)  # 将list数组转化成array数组便于查看数据结构
birth_header = np.array(birth_header)
print(birth_data.shape)  # 利用.shape查看结构。
print(birth_header.shape)
#
# (189, 9)
# (9,)

Use Pandas to read CSV files

import pandas as pd
csv_data = pd.read_csv('birth_weight.csv')  # 读取训练数据
print(csv_data.shape)  # (189, 9)
N = 5
csv_batch_data = csv_data.tail(N)  # 取后5条数据
print(csv_batch_data.shape)  # (5, 9)
train_batch_data = csv_batch_data[list(range(3, 6))]  # 取这20条数据的3到5列值(索引从0开始)
print(train_batch_data)
#      RACE  SMOKE  PTL
# 184   0.0    0.0  0.0
# 185   0.0    0.0  1.0
# 186   0.0    1.0  0.0
# 187   0.0    0.0  0.0
# 188   0.0    0.0  1.0

Use Tensorflow to read CSV files

I usually do Tensorflow is used to process various types of data, so I will not explain too much about using Tensorflow to read data. I will paste a piece of code below.

'''使用Tensorflow读取csv数据'''
filename = 'birth_weight.csv'
file_queue = tf.train.string_input_producer([filename])  # 设置文件名队列，这样做能够批量读取文件夹中的文件
reader = tf.TextLineReader(skip_header_lines=1)  # 使用tensorflow文本行阅读器，并且设置忽略第一行
key, value = reader.read(file_queue)
defaults = [[0.], [0.], [0.], [0.], [0.], [0.], [0.], [0.], [0.]]  # 设置列属性的数据格式
LOW, AGE, LWT, RACE, SMOKE, PTL, HT, UI, BWT = tf.decode_csv(value, defaults)
# 将读取的数据编码为我们设置的默认格式
vertor_example = tf.stack([AGE, LWT, RACE, SMOKE, PTL, HT, UI])  # 读取得到的中间7列属性为训练特征
vertor_label = tf.stack([BWT])  # 读取得到的BWT值表示训练标签
# 用于给取出的数据添加上batch_size维度，以批处理的方式读出数据。可以设置批处理数据大小，是否重复读取数据，容量大小，队列末尾大小，读取线程等属性。
example_batch, label_batch = tf.train.shuffle_batch([vertor_example, vertor_label], batch_size=10, capacity=100, min_after_dequeue=10)
# 初始化Session
with tf.Session() as sess:
    coord = tf.train.Coordinator()  # 线程管理器
    threads = tf.train.start_queue_runners(coord=coord)
    print(sess.run(tf.shape(example_batch)))  # [10  7]
    print(sess.run(tf.shape(label_batch)))  # [10  1]
    print(sess.run(example_batch)[3])  # [ 19.  91.   0.   1.   1.   0.   1.]
    coord.request_stop()
    coord.join(threads)
'''
对于使用所有Tensorflow的I/O操作来说开启和关闭线程管理器都是必要的操作
with tf.Session() as sess:
    coord = tf.train.Coordinator()  # 线程管理器
    threads = tf.train.start_queue_runners(coord=coord)
    #  Your code here~
    coord.request_stop()
    coord.join(threads)
'''

There are other ways to read files using python. Three are introduced here and will be supplemented from time to time.

The above is the detailed content of How to read csv file in python. For more information, please follow other related articles on the PHP Chinese website!