Home > Article > Backend Development > Sharing tips on using efficient file I/O operation processing in Python
Sharing tips on using efficient file I/O operation processing in Python
- 高洛峰Original
- 2017-03-16 16:16:251320browse
How to read and write text files?
Actual case
The encoding format of a certain text file has been changed (such as UTF-8, GBK, BIG5), how about python2.x and python3.x respectively? Read these files?
Solution
Pay attention to distinguish the difference between python2 and python3
The semantics of string has changed:
| python2 | python3 |
| str | bytes |
| unicode | str |
python2.x encodes unicode before writing the file and decodes the binary string after reading the file
>>> f = open('py2.txt', 'w')
>>> s = u'你好'
>>> f.write(s.encode('gbk'))
>>> f.close()
>>> f = open('py2.txt', 'r')
>>> t = f.read()
>>> print t.decode('gbk')
Hello
open function in python3.x Specify the text mode of t, encoding specifies the encoding format
>>> f = open('py3.txt', 'wt', encoding='utf-8')
>>> f.write('你好')
2
>>> f.close()
>>> f = open('py3.txt', 'rt', encoding='utf-8')
>>> s = f.read()
>>> s
'你好'
How to set the file buffer
Actual case
When writing the file content to the hard disk device, use System calls, this type of I/O operations take a long time. In order to reduce the number of I/O operations, files usually use buffers (only when there is enough data to make system calls), the file's cacheBehavior, divided into full buffering, line buffering, and no buffering.
How to set the buffer text of fileobject in Python?
Solution
Full buffering: The buffering of the open function is set to an integer greater than 1n, n is the buffer size
>>> f = open('demo2.txt', 'w', buffering=2048)
>>> f.write('+' * 1024)
>>> f.write('+' * 1023)
# 大于2048的时候就写入文件
>>> f.write('-' * 2)
>>> f.close()
Line buffering: The buffering of the open function is set to 1
>>> f = open('demo3.txt', 'w', buffering=1)
>>> f.write('abcd')
>>> f.write('1234')
# 只要加上\n就写入文件中
>>> f.write('\n')
>>> f.close()
No buffering: The buffering of the open function is set to 0
>>> f = open('demo4.txt', 'w', buffering=0)
>>> f.write('a')
>>> f.write('b')
>>> f.close()
How to map files to memory?
Actual case
When accessing certain binary files, it is hoped that the file can be mapped into memory to achieve random access. (framebuffer device file)
Some embedded device, the registers are addressed to the memory address space, we can map a certain range of /dev/mem to access these registers
If multiple processes are mapped to the same file, process communication can also be achieved
Solution
Use the mmap() function of the mmap module in the standard library, which requires an open file descriptor as a parameter
Create The following file
[root@pythontab.com ~]# dd if=/dev/zero of=demo.bin bs=1024 count=1024 1024+0 records in 1024+0 records out 1048576 bytes (1.0 MB) copied, 0.00380084 s, 276 MB/s # 以十六进制格式查看文件内容 [root@pythontab.com ~]# od -x demo.bin 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 4000000
>>> import mmap
>>> import os
>>> f = open('demo.bin','r+b')
# 获取文件描述符
>>> f.fileno()
3
>>> m = mmap.mmap(f.fileno(),0,access=mmap.ACCESS_WRITE)
>>> type(m)
<type 'mmap.mmap'>
# 可以通过索引获取内容
>>> m[0]
'\x00'
>>> m[10:20]
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
# 修改内容
>>> m[0] = '\x88'
View
[root@pythontab.com ~]# od -x demo.bin 0000000 0088 0000 0000 0000 0000 0000 0000 0000 0000020 0000 0000 0000 0000 0000 0000 0000 0000 * 4000000
Modify slice
>>> m[4:8] = '\xff' * 4
View
[root@pythontab.com ~]# od -x demo.bin 0000000 0088 0000 ffff ffff 0000 0000 0000 0000 0000020 0000 0000 0000 0000 0000 0000 0000 0000 * 4000000
>>> m = mmap.mmap(f.fileno(),mmap.PAGESIZE * 8,access=mmap.ACCESS_WRITE,offset=mmap.PAGESIZE * 4) >>> m[:0x1000] = '\xaa' * 0x1000
View
[root@pythontab.com ~]# od -x demo.bin 0000000 0088 0000 ffff ffff 0000 0000 0000 0000 0000020 0000 0000 0000 0000 0000 0000 0000 0000 * 0040000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa * 0050000 0000 0000 0000 0000 0000 0000 0000 0000 * 4000000
How to access the status of the file ?
Actual case
In some projects, we need to get the file status, for example:
The type of file (ordinary file, directory, symbolic link, device file...)
File access rights
The last access/modification/node status change time of the file
The size of the ordinary file
…..
Solution
The current directory has the following files
[root@pythontab.com 2017]# ll total 4 drwxr-xr-x 2 root root 4096 Sep 16 11:35 dirs -rw-r--r-- 1 root root 0 Sep 16 11:35 files lrwxrwxrwx 1 root root 37 Sep 16 11:36 lockfile -> /tmp/qtsingleapp-aegisG-46d2-lockfile
System call
Three system calls stat under the os module in the standard library , fstat, lstat Get the file status
>>> import os
>>> s = os.stat('files')
>>> s
posix.stat_result(st_mode=33188, st_ino=267646, st_dev=51713L, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1486197100, st_mtime=1486197100, st_ctime=1486197100)
>>> s.st_mode
33188
>>> import stat
# stat有很多S_IS..方法来判断文件的类型
>>> stat.S_ISDIR(s.st_mode)
False
# 普通文件
>>> stat.S_ISREG(s.st_mode)
True
Get the access permission of the file, it is true as long as it is greater than 0
>>> s.st_mode & stat.S_IRUSR 256 >>> s.st_mode & stat.S_IXGRP 0 >>> s.st_mode & stat.S_IXOTH 0
Get the modification time of the file
# 访问时间 >>> s.st_atime 1486197100.3384446 # 修改时间 >>> s.st_mtime 1486197100.3384446 # 状态更新时间 >>> s.st_ctime 1486197100.3384446
Get the TimestampConvert
>>> import time >>> time.localtime(s.st_atime) time.struct_time(tm_year=2016, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=35, tm_sec=47, tm_wday=4, tm_yday=260, tm_isdst=0)
Get the size of an ordinary file
>>> s.st_size 0
Shortcut function
Some functions under os.path in the standard library are more convenient to use Concise
File type judgment
>>> os.path.isdir('dirs')
True
>>> os.path.islink('lockfile')
True
>>> os.path.isfile('files')
True
File three times
>>> os.path.getatime('files')
1486197100.3384445
>>> os.path.getmtime('files')
1486197100.3384445
>>> os.path.getctime('files')
1486197100.3384445
Get file size
>>> os.path.getsize('files')
0
How to use temporary files?
Actual Case
In a certain project, we collect data from sensors. After each 1G of data is collected, we do data analysis. In the end, only the analysis results are saved. If such large temporary data is permanent, Memory will consume a lot of memory resources. We can use temporary files to store these temporary data (external storage)
Temporary files do not need to be named and will be deleted automatically after closing
Solution
Use TemporaryFile, NamedTemporaryFile under tempfile in the standard library
>>> from tempfile import TemporaryFile, NamedTemporaryFile
# 访问的时候只能通过对象f来进行访问
>>> f = TemporaryFile()
>>> f.write('abcdef' * 100000)
# 访问临时数据
>>> f.seek(0)
>>> f.read(100)
'abcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcd'
>>> ntf = NamedTemporaryFile()
# 如果要让每次创建NamedTemporaryFile()对象时不删除文件，可以设置NamedTemporaryFile(delete=False)
>>> ntf.name
# 返回当前临时文件在文件系统中的路径
'/tmp/tmppNvNA6'The above is the detailed content of Sharing tips on using efficient file I/O operation processing in Python. For more information, please follow other related articles on the PHP Chinese website!