Python encoding format change problem

Today I was writing a python crawler, and suddenly the crawled web page was abnormal and an error was reported: UnicodeEncodeError: 'latin-1' codec can't encode characters in position 41-50: ordinal not in range(256); UnicodeEncodeError: 'ascii' codec can't encode characters in ordinal, this is an obvious encoding format problem. In fact, not only python2 or 3, but also other programming languages ​​​​such as Java and C, often encounter problems with encoding formats, which are extremely headaches, especially the conversion between ASCII, gbk, utf-8 and other encodings. So I searched for information, practiced hands-on, and found several methods like this.

First of all, how do we check the system encoding format and input and output format of Python?

>>> import sys
>>> sys.getdefaultencoding()# 系统默认编码格式
'UTF-8'
>>> sys.stdout.encoding# 输入输出格式
'US-ASCII'

In this case, it means that the input and output encoding of the current command line is ascii, so you need to manually change the environment variable LANG to utf-8:

export LANG="en_US.UTF-8"

If you are in an ubuntu environment, you can add the above line of command to ~/.bashrc to solve the problem once and for all. After adding it, run the following command to make it effective or restart it.

source ~/.bashrc

Or another method is to set the corresponding encoding only for python (same as above, run the command line or add the bashrc file):

PYTHONIOENCODING='utf_8'
export PYTHONIOENCODING

gbk encoding example picture:

ascii encoding table diagram part:

Don’t forget to add to the top line when programming:

# -*- coding: utf-8 -*-

You can use the command line to set python encoding, but it will only take effect in this session

>>>sys.getdefaultencoding()查看当前编码(若报错，先执行>>>import sys >>>reload(sys));
>>>sys.setdefaultencoding('utf8')设置编码

It is also feasible to reload the SYS module and set uft-8 in the program code, but an error will be prompted in pycharm

import sys
reload(sys) 
sys.setdefaultencoding('utf8')
重启Python解释器，发现编码已被设置为utf8；
这是因为系统在Python启动的时候，自行调用该文件，设置系统的默认编码，而不需要每次都手动加上解决代码，属于一劳永逸的解决方法。

During normal development, we all know to use Decode and Encode for transcoding operations.

Decode的作用是将其他编码的字符串转换成unicode编码，如str1.decode('gb2312')，表示将gb2312编码的字符串str1转换成unicode编码。
Encode的作用是将unicode编码转换成其他编码的字符串，如str2.encode('gb2312')，表示将unicode编码的字符串str2转换成gb2312编码。

In the latest python 3 version, the type of string is str, which is represented by Unicode in memory. One character corresponds to several bytes;

If you want to use it on the network To transfer, or save to disk, str needs to be converted into bytes in bytes.

Str expressed in Unicode can be encoded into specified bytes through the encode() method, for example:

>>> 'ABC'.encode('ascii')
b'ABC'
>>> '中文'.encode('utf-8')
b'\xe4\xb8\xad\xe6\x96\x87'
>>> '中文'.encode('ascii')
Traceback (most recent call last):
  File "", line 1, in 
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)

When operating strings, we often encounter the conversion of str and bytes. In order to avoid garbled characters, you should always use UTF-8 encoding to convert str and bytes.

Friendly Links【UTF-8 Encoding Conversion Tool】

【python3 Video Tutorial】

Initially there are only a few encoding formats Due to the popularization of computers and the use of many countries or organizations, encoding formats have become more and more popular, but the internationally accepted one is still UTF-8, so you should have good programming habits and use UTF-8 encoding format more often. . When encountering encoding problems, try to keep the encoding format consistent.

The above is the detailed content of Python encoding format change problem. For more information, please follow other related articles on the PHP Chinese website!