


Preface
Recently, due to project needs, I need to read a txt document containing Chinese, and then save the file. The document was previously encoded by base64, causing all Chinese characters to be read and displayed as garbled characters. After the project team abandoned base64, two errors occurred successively:
ascii codec can't encode characters in position ordinal not in range 128 UnicodeDecodeError: ‘utf8' codec can't decode byte 0x。
If you don’t know about ascii, unicode and utf-8, you can read This previous article is about strings and encoding
Then you must understand the following three concepts:
ascii can only represent Numbers, English letters and some special symbols cannot represent Chinese characters
Both unicode and utf-8 can represent Chinese characters. Unicode is a fixed length and utf-8 is a variable length
The memory storage method is generally unicode, while the disk file storage method is generally utf-8, because utf-8 can save storage space
So what is python's default encoding?
>>> import sys >>> sys.getdefaultencoding() 'ascii' >>> reload(sys) <module 'sys' (built-in)> >>> sys.setdefaultencoding('utf-8') >>> sys.getdefaultencoding() 'utf-8'
The default encoding of python is ascii, which can be set by the sys.setdefaultencoding('utf-8')
function the default encoding.
In python, you can change the encoding of data through encode and decode, for example:
>>> u'汉字' u'\u6c49\u5b57' >>> u'汉字'.encode('utf-8') '\xe6\xb1\x89\xe5\xad\x97' >>> u'汉字'.encode('utf-8').decode('utf-8') u'\u6c49\u5b57'
We can set the encoding through these two functions .
So, what type is str in python?
>>> import binascii >>> '汉字' '\xba\xba\xd7\xd6' >>> type('汉字') <type 'str'> >>> print binascii.b2a_hex('汉字') babad7d6 >>> print binascii.b2a_hex(u'汉字') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128) >>> print binascii.b2a_hex(u'汉字'.encode('utf-8')) e6b189e5ad97 >>> print binascii.b2a_hex(u'汉字'.encode('gbk')) babad7d6
binascii converts the binary of data into ascii. The above explanation is: the type of 'Chinese characters' is str, and the binary is babad7d6, u 'Chinese characters' cannot be converted into ascii, so the first error at the beginning is reported. The solution is to .encode('utf-8') it into str type. Because my command line uses the default GBK encoding of Windows, when all u'Chinese characters'.encode('gbk')
is used, the output results are the same as the 'Chinese character' results.
To summarize, python's str is actually a type of unicode. The default encoding of python is ascii. When converting non-ascii to ascii, an error will be reported. Keep the following rules in mind:
unicode => encode('suitable encoding') => str
- ##str => decode('suitable encoding') => unicode
import sys reloads(sys) sys.setdefaultencoding('utf-8')
For the second problem, there is an error when reading the file. UTF-8 files have two modes: BOM and no BOM. The difference between the two seems to be that the BOM file has one more header than the BOM-less file, causing an error when reading the file in UTF-8 mode. When I tried to read the file before, First, judge whether there is a BOM and skip the header of the BOM file. Then it failed. It was really embarrassing~~. You have to go to Google for help. The specific operation method is to use the codecs library to read the file (I guess this library is to detect the header of the file).
import codecs codecs.open(file_name, "r",encoding='utf-8', errors='ignore')For encoding issues, you must understand the working principles of ascii, unicode and utf-8. For more python solutions to Chinese character encoding problems: Unicode Decode Error_python related articles, please pay attention to the PHP Chinese website!

Solution to permission issues when viewing Python version in Linux terminal When you try to view Python version in Linux terminal, enter python...

This article explains how to use Beautiful Soup, a Python library, to parse HTML. It details common methods like find(), find_all(), select(), and get_text() for data extraction, handling of diverse HTML structures and errors, and alternatives (Sel

This article compares TensorFlow and PyTorch for deep learning. It details the steps involved: data preparation, model building, training, evaluation, and deployment. Key differences between the frameworks, particularly regarding computational grap

Python's statistics module provides powerful data statistical analysis capabilities to help us quickly understand the overall characteristics of data, such as biostatistics and business analysis. Instead of looking at data points one by one, just look at statistics such as mean or variance to discover trends and features in the original data that may be ignored, and compare large datasets more easily and effectively. This tutorial will explain how to calculate the mean and measure the degree of dispersion of the dataset. Unless otherwise stated, all functions in this module support the calculation of the mean() function instead of simply summing the average. Floating point numbers can also be used. import random import statistics from fracti

The article discusses popular Python libraries like NumPy, Pandas, Matplotlib, Scikit-learn, TensorFlow, Django, Flask, and Requests, detailing their uses in scientific computing, data analysis, visualization, machine learning, web development, and H

This article guides Python developers on building command-line interfaces (CLIs). It details using libraries like typer, click, and argparse, emphasizing input/output handling, and promoting user-friendly design patterns for improved CLI usability.

When using Python's pandas library, how to copy whole columns between two DataFrames with different structures is a common problem. Suppose we have two Dats...

The article discusses the role of virtual environments in Python, focusing on managing project dependencies and avoiding conflicts. It details their creation, activation, and benefits in improving project management and reducing dependency issues.


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

VSCode Windows 64-bit Download
A free and powerful IDE editor launched by Microsoft

WebStorm Mac version
Useful JavaScript development tools

DVWA
Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

SecLists
SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

Atom editor mac version download
The most popular open source editor