search
HomeBackend DevelopmentPython TutorialPython solves the problem of Chinese character encoding: Unicode Decode Error_python

Preface

Recently, due to project needs, I need to read a txt document containing Chinese, and then save the file. The document was previously encoded by base64, causing all Chinese characters to be read and displayed as garbled characters. After the project team abandoned base64, two errors occurred successively:

ascii codec can't encode characters in position ordinal not in range 128
UnicodeDecodeError: ‘utf8' codec can't decode byte 0x。

If you don’t know about ascii, unicode and utf-8, you can read This previous article is about strings and encoding

Then you must understand the following three concepts:

  1. ascii can only represent Numbers, English letters and some special symbols cannot represent Chinese characters

  2. Both unicode and utf-8 can represent Chinese characters. Unicode is a fixed length and utf-8 is a variable length

  3. The memory storage method is generally unicode, while the disk file storage method is generally utf-8, because utf-8 can save storage space

So what is python's default encoding?

>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> reload(sys)
<module &#39;sys&#39; (built-in)>
>>> sys.setdefaultencoding(&#39;utf-8&#39;)
>>> sys.getdefaultencoding()
&#39;utf-8&#39;

The default encoding of python is ascii, which can be set by the sys.setdefaultencoding('utf-8') function the default encoding.

In python, you can change the encoding of data through encode and decode, for example:

>>> u&#39;汉字&#39;
u&#39;\u6c49\u5b57&#39;
>>> u&#39;汉字&#39;.encode(&#39;utf-8&#39;)
&#39;\xe6\xb1\x89\xe5\xad\x97&#39;
>>> u&#39;汉字&#39;.encode(&#39;utf-8&#39;).decode(&#39;utf-8&#39;)
u&#39;\u6c49\u5b57&#39;

We can set the encoding through these two functions .

So, what type is str in python?

>>> import binascii
>>> &#39;汉字&#39;
&#39;\xba\xba\xd7\xd6&#39;
>>> type(&#39;汉字&#39;)
<type &#39;str&#39;>
>>> print binascii.b2a_hex(&#39;汉字&#39;)
babad7d6
>>> print binascii.b2a_hex(u&#39;汉字&#39;)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
UnicodeEncodeError: &#39;ascii&#39; codec can&#39;t encode characters in
position 0-1: ordinal not in range(128)
>>> print binascii.b2a_hex(u&#39;汉字&#39;.encode(&#39;utf-8&#39;))
e6b189e5ad97
>>> print binascii.b2a_hex(u&#39;汉字&#39;.encode(&#39;gbk&#39;))
babad7d6

binascii converts the binary of data into ascii. The above explanation is: the type of 'Chinese characters' is str, and the binary is babad7d6, u 'Chinese characters' cannot be converted into ascii, so the first error at the beginning is reported. The solution is to .encode('utf-8') it into str type. Because my command line uses the default GBK encoding of Windows, when all u'Chinese characters'.encode('gbk') is used, the output results are the same as the 'Chinese character' results.

To summarize, python's str is actually a type of unicode. The default encoding of python is ascii. When converting non-ascii to ascii, an error will be reported. Keep the following rules in mind:

  1. unicode => encode('suitable encoding') => str

  2. ##str => decode('suitable encoding') => unicode


There is also a simple way, which is to set the encoding in the file header, which can save a lot of trouble:

import sys
reloads(sys)
sys.setdefaultencoding(&#39;utf-8&#39;)

For the second problem, there is an error when reading the file. UTF-8 files have two modes: BOM and no BOM. The difference between the two seems to be that the BOM file has one more header than the BOM-less file, causing an error when reading the file in UTF-8 mode. When I tried to read the file before, First, judge whether there is a BOM and skip the header of the BOM file. Then it failed. It was really embarrassing~~.

You have to go to Google for help. The specific operation method is to use the codecs library to read the file (I guess this library is to detect the header of the file).

import codecs
codecs.open(file_name, "r",encoding=&#39;utf-8&#39;, errors=&#39;ignore&#39;)

For encoding issues, you must understand the working principles of ascii, unicode and utf-8.

For more python solutions to Chinese character encoding problems: Unicode Decode Error_python related articles, please pay attention to the PHP Chinese website!

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
How to solve the permissions problem encountered when viewing Python version in Linux terminal?How to solve the permissions problem encountered when viewing Python version in Linux terminal?Apr 01, 2025 pm 05:09 PM

Solution to permission issues when viewing Python version in Linux terminal When you try to view Python version in Linux terminal, enter python...

How Do I Use Beautiful Soup to Parse HTML?How Do I Use Beautiful Soup to Parse HTML?Mar 10, 2025 pm 06:54 PM

This article explains how to use Beautiful Soup, a Python library, to parse HTML. It details common methods like find(), find_all(), select(), and get_text() for data extraction, handling of diverse HTML structures and errors, and alternatives (Sel

How to Perform Deep Learning with TensorFlow or PyTorch?How to Perform Deep Learning with TensorFlow or PyTorch?Mar 10, 2025 pm 06:52 PM

This article compares TensorFlow and PyTorch for deep learning. It details the steps involved: data preparation, model building, training, evaluation, and deployment. Key differences between the frameworks, particularly regarding computational grap

Mathematical Modules in Python: StatisticsMathematical Modules in Python: StatisticsMar 09, 2025 am 11:40 AM

Python's statistics module provides powerful data statistical analysis capabilities to help us quickly understand the overall characteristics of data, such as biostatistics and business analysis. Instead of looking at data points one by one, just look at statistics such as mean or variance to discover trends and features in the original data that may be ignored, and compare large datasets more easily and effectively. This tutorial will explain how to calculate the mean and measure the degree of dispersion of the dataset. Unless otherwise stated, all functions in this module support the calculation of the mean() function instead of simply summing the average. Floating point numbers can also be used. import random import statistics from fracti

What are some popular Python libraries and their uses?What are some popular Python libraries and their uses?Mar 21, 2025 pm 06:46 PM

The article discusses popular Python libraries like NumPy, Pandas, Matplotlib, Scikit-learn, TensorFlow, Django, Flask, and Requests, detailing their uses in scientific computing, data analysis, visualization, machine learning, web development, and H

How to Create Command-Line Interfaces (CLIs) with Python?How to Create Command-Line Interfaces (CLIs) with Python?Mar 10, 2025 pm 06:48 PM

This article guides Python developers on building command-line interfaces (CLIs). It details using libraries like typer, click, and argparse, emphasizing input/output handling, and promoting user-friendly design patterns for improved CLI usability.

How to efficiently copy the entire column of one DataFrame into another DataFrame with different structures in Python?How to efficiently copy the entire column of one DataFrame into another DataFrame with different structures in Python?Apr 01, 2025 pm 11:15 PM

When using Python's pandas library, how to copy whole columns between two DataFrames with different structures is a common problem. Suppose we have two Dats...

Explain the purpose of virtual environments in Python.Explain the purpose of virtual environments in Python.Mar 19, 2025 pm 02:27 PM

The article discusses the role of virtual environments in Python, focusing on managing project dependencies and avoiding conflicts. It details their creation, activation, and benefits in improving project management and reducing dependency issues.

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
3 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. Best Graphic Settings
3 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. How to Fix Audio if You Can't Hear Anyone
3 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

VSCode Windows 64-bit Download

VSCode Windows 64-bit Download

A free and powerful IDE editor launched by Microsoft

WebStorm Mac version

WebStorm Mac version

Useful JavaScript development tools

DVWA

DVWA

Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

Atom editor mac version download

Atom editor mac version download

The most popular open source editor