Sharing tips on using efficient file I/O operation processing in Python

How to read and write text files?

Actual case

The encoding format of a certain text file has been changed (such as UTF-8, GBK, BIG5), how about python2.x and python3.x respectively? Read these files?

Solution

Pay attention to distinguish the difference between python2 and python3

The semantics of string has changed:

python2	python3
str	bytes
unicode	str

python2.x encodes unicode before writing the file and decodes the binary string after reading the file

>>> f = open('py2.txt', 'w')
>>> s = u'你好'
>>> f.write(s.encode('gbk'))
>>> f.close()
>>> f = open('py2.txt', 'r')
>>> t = f.read()
>>> print t.decode('gbk')

Hello

open function in python3.x Specify the text mode of t, encoding specifies the encoding format

>>> f = open('py3.txt', 'wt', encoding='utf-8')
>>> f.write('你好')
2
>>> f.close()
>>> f = open('py3.txt', 'rt', encoding='utf-8')
>>> s = f.read()
>>> s
'你好'

How to set the file buffer

Actual case

When writing the file content to the hard disk device, use System calls, this type of I/O operations take a long time. In order to reduce the number of I/O operations, files usually use buffers (only when there is enough data to make system calls), the file's cacheBehavior, divided into full buffering, line buffering, and no buffering.

How to set the buffer text of fileobject in Python?

Solution

Full buffering: The buffering of the open function is set to an integer greater than 1n, n is the buffer size

>>> f = open('demo2.txt', 'w', buffering=2048)
>>> f.write('+' * 1024)
>>> f.write('+' * 1023)
# 大于2048的时候就写入文件
>>> f.write('-' * 2)
>>> f.close()

Line buffering: The buffering of the open function is set to 1

>>> f = open('demo3.txt', 'w', buffering=1)
>>> f.write('abcd')
>>> f.write('1234')
# 只要加上\n就写入文件中
>>> f.write('\n')
>>> f.close()

No buffering: The buffering of the open function is set to 0

>>> f = open('demo4.txt', 'w', buffering=0)
>>> f.write('a')
>>> f.write('b')
>>> f.close()

How to map files to memory?

Actual case

When accessing certain binary files, it is hoped that the file can be mapped into memory to achieve random access. (framebuffer device file)

Some embedded device, the registers are addressed to the memory address space, we can map a certain range of /dev/mem to access these registers

If multiple processes are mapped to the same file, process communication can also be achieved

Solution

Use the mmap() function of the mmap module in the standard library, which requires an open file descriptor as a parameter

Create The following file

[root@pythontab.com ~]# dd if=/dev/zero of=demo.bin bs=1024 count=1024
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.00380084 s, 276 MB/s
# 以十六进制格式查看文件内容
[root@pythontab.com ~]# od -x demo.bin 
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
4000000

>>> import mmap
>>> import os
>>> f = open('demo.bin','r+b')
# 获取文件描述符
>>> f.fileno()
3
>>> m = mmap.mmap(f.fileno(),0,access=mmap.ACCESS_WRITE)
>>> type(m)
<type>
# 可以通过索引获取内容
>>> m[0]
'\x00'
>>> m[10:20]
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
# 修改内容
>>> m[0] = '\x88'</type>

View

[root@pythontab.com ~]# od -x demo.bin 
0000000 0088 0000 0000 0000 0000 0000 0000 0000
0000020 0000 0000 0000 0000 0000 0000 0000 0000
*
4000000

Modify slice

>>> m[4:8] = '\xff' * 4

View

[root@pythontab.com ~]# od -x demo.bin 
0000000 0088 0000 ffff ffff 0000 0000 0000 0000
0000020 0000 0000 0000 0000 0000 0000 0000 0000
*
4000000

>>> m = mmap.mmap(f.fileno(),mmap.PAGESIZE * 8,access=mmap.ACCESS_WRITE,offset=mmap.PAGESIZE * 4) 
>>> m[:0x1000] = '\xaa' * 0x1000

View

[root@pythontab.com ~]# od -x demo.bin 
0000000 0088 0000 ffff ffff 0000 0000 0000 0000
0000020 0000 0000 0000 0000 0000 0000 0000 0000
*
0040000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
0050000 0000 0000 0000 0000 0000 0000 0000 0000
*
4000000

How to access the status of the file ?

Actual case

In some projects, we need to get the file status, for example:

The type of file (ordinary file, directory, symbolic link, device file...)

File access rights

The last access/modification/node status change time of the file

The size of the ordinary file

…..

Solution

The current directory has the following files

[root@pythontab.com 2017]# ll
total 4
drwxr-xr-x 2 root root 4096 Sep 16 11:35 dirs
-rw-r--r-- 1 root root 0 Sep 16 11:35 files
lrwxrwxrwx 1 root root 37 Sep 16 11:36 lockfile -> /tmp/qtsingleapp-aegisG-46d2-lockfile

System call

Three system calls stat under the os module in the standard library , fstat, lstat Get the file status

>>> import os
>>> s = os.stat('files')
>>> s
posix.stat_result(st_mode=33188, st_ino=267646, st_dev=51713L, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1486197100, st_mtime=1486197100, st_ctime=1486197100)
>>> s.st_mode
33188
>>> import stat
# stat有很多S_IS..方法来判断文件的类型
>>> stat.S_ISDIR(s.st_mode)
False
# 普通文件
>>> stat.S_ISREG(s.st_mode)
True

Get the access permission of the file, it is true as long as it is greater than 0

>>> s.st_mode & stat.S_IRUSR
256
>>> s.st_mode & stat.S_IXGRP
0
>>> s.st_mode & stat.S_IXOTH
0

Get the modification time of the file

# 访问时间
>>> s.st_atime
1486197100.3384446
# 修改时间
>>> s.st_mtime
1486197100.3384446
# 状态更新时间
>>> s.st_ctime
1486197100.3384446

Get the TimestampConvert

>>> import time
>>> time.localtime(s.st_atime)
time.struct_time(tm_year=2016, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=35, tm_sec=47, tm_wday=4, tm_yday=260, tm_isdst=0)

Get the size of an ordinary file

>>> s.st_size
0

Shortcut function

Some functions under os.path in the standard library are more convenient to use Concise

File type judgment

>>> os.path.isdir('dirs') 
True
>>> os.path.islink('lockfile')
True
>>> os.path.isfile('files') 
True

File three times

>>> os.path.getatime('files')
1486197100.3384445
>>> os.path.getmtime('files')
1486197100.3384445
>>> os.path.getctime('files')
1486197100.3384445

Get file size

>>> os.path.getsize('files') 
0

How to use temporary files?

Actual Case

In a certain project, we collect data from sensors. After each 1G of data is collected, we do data analysis. In the end, only the analysis results are saved. If such large temporary data is permanent, Memory will consume a lot of memory resources. We can use temporary files to store these temporary data (external storage)

Temporary files do not need to be named and will be deleted automatically after closing

Solution

Use TemporaryFile, NamedTemporaryFile under tempfile in the standard library

>>> from tempfile import TemporaryFile, NamedTemporaryFile
# 访问的时候只能通过对象f来进行访问
>>> f = TemporaryFile()
>>> f.write('abcdef' * 100000)
# 访问临时数据
>>> f.seek(0)
>>> f.read(100)
'abcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcd'
>>> ntf = NamedTemporaryFile()
# 如果要让每次创建NamedTemporaryFile()对象时不删除文件，可以设置NamedTemporaryFile(delete=False)
>>> ntf.name
# 返回当前临时文件在文件系统中的路径
'/tmp/tmppNvNA6'

The above is the detailed content of Sharing tips on using efficient file I/O operation processing in Python. For more information, please follow other related articles on the PHP Chinese website!