In-depth analysis of Python coding

It is said that everyone who develops Python is confused by character encoding problems. The most common errors are UnicodeEncodeError and UnicodeDecodeError. You seem to know how to solve them. Unfortunately, the errors appear in other places, and the problem is always Repeating the same mistake, it is very difficult to remember whether to use decode or encode method to convert str to unicode. I am always confused. Where is the problem?

In order to clarify this problem, I decided to conduct an in-depth analysis from the composition of python strings and the details of character encoding

Bytes and characters

Everything stored in the computer Data, text characters, pictures, videos, audios, and software are all composed of a sequence of 01 bytes. One byte is equal to 8 bits.

A character is a symbol. For example, a Chinese character, an English letter, a number, and a punctuation mark can all be called a character.

Bytes are convenient for storage and network transmission, while characters are used for display and easy to read. For example, the character "p" stored on the hard disk is a string of binary data 01110000, which occupies one byte in length

Encoding and decoding

The text we open with the editor, Every character you see is stored in the form of a binary byte sequence when it is finally saved on the disk. Then the conversion process from characters to bytes is called encoding, and the reverse is called decoding. The two are reversible processes. Encoding is for storage and transmission, and decoding is for display and reading.

For example, the character "p" is encoded and saved to the hard disk as a sequence of binary bytes 01110000, occupying one byte in length. The character "Zen" may be stored as "11100111 10100110 10000101" occupying a length of 3 bytes. Why is it possible? We’ll talk about this later.

Why is Python coding so painful? Of course, this cannot be blamed on the developers.

This is because Python2 uses ASCII character encoding as the default encoding, and ASCII cannot handle Chinese, so why not use UTf-8? Because Dad Guido wrote the first line of code for Python in the winter of 1989, and the first version was officially open sourced and released in February 1991, and Unicode was released in October 1991, which means that the language Python was created. UTF-8 hadn't been born yet, so this was one of them.

Python also divides the string types into two types, unicode and str, so that developers are confused. This is the second reason. Python3 has completely reshaped strings, retaining only one type. This is a story for later.

str and unicode

Python2 divides strings into two types: unicode and str. In essence, str is a sequence of binary bytes. As can be seen from the example code below, the "Zen" of str type is printed as hexadecimal \xec\xf8, and the corresponding binary byte sequence is '11101100 11111000'.

>>> s = '禅'
>>> s
'\xec\xf8'
>>> type(s)
<type &#39;str&#39;>

The corresponding unicode symbol of the unicode type u"Zen" is u'\u7985′

>>> u = u"禅"
>>> u
u'\u7985'
>>> type(u)
<type &#39;unicode&#39;>

If we want to save the unicode symbol to a file or transmit it to the network, we need to go through encoding processing and conversion into str type, so python provides the encode method to convert from unicode to str and vice versa.

##encode

>>> u = u"禅"
>>> u
u'\u7985'
>>> u.encode("utf-8")
'\xe7\xa6\x85'

decode

>>> s = "禅"
>>> s.decode("utf-8")
u'\u7985'
>>>

Many beginners cannot remember whether to use encode or unicode to convert between str and unicode decode, if you remember that str is essentially a string of binary data, and unicode is a character (symbol), encoding is the process of converting characters (symbols) into binary data, so the conversion from unicode to str needs to be The encode method, in turn, uses the decode method.

encoding always takes a Unicode string and returns a bytes sequence, and decoding always takes a bytes sequence and returns a Unicode string”.

After clarifying the conversion relationship between str and unicode, let’s take a look at when UnicodeEncodeError and UnicodeDecodeError errors will occur.

UnicodeEncodeError

UnicodeEncodeError occurs when a unicode string is converted into a str byte sequence. Let's look at an example of saving a string of unicode strings to a file

# -*- coding:utf-8 -*-
def main():
    name = u'Python之禅'
    f = open("output.txt", "w")
    f.write(name)

Error log

UnicodeEncodeError: ‘ascii’ codec can’t encode characters in position 6-7: ordinal not in range(128)

Why does UnicodeEncodeError occur?

Because when calling the write method, Python will first determine what type the string is. If it is str, it will be written directly to the file without encoding, because the str type string itself is a string of binary byte sequence.

If the string is of unicode type, it will first call the encode method to convert the unicode string into the binary str type before saving it to the file, and the encode method will use python's default ascii code to encode

Equivalent to:

>>> u"Python之禅".encode("ascii")

However, we know that the ASCII character set only contains 128 Latin letters, excluding Chinese characters, so the 'ascii' codec can't encode characters error occurs. To use encode correctly, you must specify a character set that contains Chinese characters, such as UTF-8, GBK.

>>> u"Python之禅".encode("utf-8")
'Python\xe4\xb9\x8b\xe7\xa6\x85'

>>> u"Python之禅".encode("gbk")
'Python\xd6\xae\xec\xf8'

So to write unicode strings to files correctly, you should convert the strings to UTF-8 or GBK encoding in advance.

def main():
    name = u'Python之禅'
    name = name.encode('utf-8')
    with open("output.txt", "w") as f:
        f.write(name)

当然，把 unicode 字符串正确地写入文件不止一种方式，但原理是一样的，这里不再介绍，把字符串写入数据库，传输到网络都是同样的原理

UnicodeDecodeError

UnicodeDecodeError 发生在 str 类型的字节序列解码成 unicode 类型的字符串时

>>> a = u"禅"
>>> a
u'\u7985'
>>> b = a.encode("utf-8")
>>> b
'\xe7\xa6\x85'
>>> b.decode("gbk")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: &#39;gbk&#39; codec can&#39;t decode byte 0x85 in position 2: incomplete multibyte sequence

把一个经过 UTF-8 编码后生成的字节序列 ‘\xe7\xa6\x85′ 再用 GBK 解码转换成 unicode 字符串时，出现 UnicodeDecodeError，因为 （对于中文字符）GBK 编码只占用两个字节，而 UTF-8 占用3个字节，用 GBK 转换时，还多出一个字节，因此它没法解析。避免 UnicodeDecodeError 的关键是保持 编码和解码时用的编码类型一致。

这也回答了文章开头说的字符 “禅”，保存到文件中有可能占3个字节，有可能占2个字节，具体处决于 encode 的时候指定的编码格式是什么。

再举一个 UnicodeDecodeError 的例子

>>> x = u"Python"
>>> y = "之禅"
>>> x + y
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: &#39;ascii&#39; codec can&#39;t decode byte 0xe4 in position 0: ordinal not in range(128)
>>>

str 与 unicode 字符串 执行 + 操作是，Python 会把 str 类型的字节序列隐式地转换成（解码）成 和 x 一样的 unicode 类型，但Python是使用默认的 ascii 编码来转换的，而 ASCII 中不包含中文，所以报错了。

>>> y.decode('ascii')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: &#39;ascii&#39; codec can&#39;t decode byte 0xe4 in position 0: ordinal not in range(128)

正确地方式应该是显示地把 y 用 UTF-8 或者 GBK 进行解码。

>>> x = u"Python"
>>> y = "之禅"
>>> y = y.decode("utf-8")
>>> x + y
u'Python\u4e4b\u7985'

以上内容都是基于 Python2 来讲的，关于 Python3 的字符和编码将会另开一篇文章来写，保持关注。

The above is the detailed content of In-depth analysis of Python coding. For more information, please follow other related articles on the PHP Chinese website!