A pitfall in Python Unicode string formatting

Today I discovered a small trap in Python string formatting when I was helping a colleague research an inexplicable UnicodeDecodeError. I will record it here. The original code was too complicated and had too many things irrelevant to the problem, so I reproduced the problem through a simple test in ipython. The process is as follows:

In [4]: a = '你好世界'
In [5]: print 'Say this: %s' % a
Say this: 你好世界
In [6]: print 'Say this: %s and say that: %s' % (a, 'hello world')
Say this: 你好世界 and say that: hello world
In [7]: print 'Say this: %s and say that: %s' % (a, u'hello world')
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
/home/jerry/ in ()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 10: ordinal not in range(128)
In [8]: a
Out[8]: '\xe4\xbd\xa0\xe5\xa5\xbd\xe4\xb8\x96\xe7\x95\x8c'

Have you seen the weird UnicodeDecodeError after In [7]? The only difference between it and the previous sentence is that 'hello world' becomes a unicode object instead of a str object. But the problem is, 'hello world' is just a simple English string that does not contain any characters other than ASCII. How can it not be decoded? Take a closer look at the message attached to the exception. It mentions 0xe4. This is obviously not in 'hello world', so we can only doubt the Chinese sentence. In [8] printed out its byte sequence, and it turned out to be it. , the first one is 0xe4.

It seems that Python tries to decode a into a unicode object when formatting the string, and the default ASCII encoding is used when decoding instead of the actual UTF-8 encoding. So what's going on? ? Let's continue our experiment:

In [9]: 'Say this: %s' % 'hello'
Out[9]: 'Say this: hello'
In [10]: 'Say this: %s' % u'hello'
Out[10]: u'Say this: hello'

Look carefully, 'hello' in In [9] is an ordinary string, and the result is also a string (str object), while 'hello' in In [10] becomes unicode Object, the formatted result also becomes unicode (note the u at the beginning of the result).

So the truth is this: Python has some hidden tricks when formatting strings: if there is unicode in the parameter corresponding to %s, then the final result will also be unicode. In this case, the template string and all str in the %s parameter will be decoded into unicode. However, this decoding is implicit. The user cannot specify the charset used, and Python can only use the default ASCII. If there happens to be a non-ASCII encoded string in it, it's over...

Look at what the Python documentation says:

If format is a Unicode object, or if any of the objects being converted using the %s conversion are Unicode objects, the result will also be a Unicode object.

If str and unicode are mixed in the code, this kind of problem can easily occur. In my colleague's code, the Chinese string was input by the user and has been correctly encoded. It is a str object encoded in UTF-8; but the troublesome unicode object, although its contents are all ASCII codes, its source is The result of the sqlite3 database query, and the strings returned by the sqlite API are all unicode objects, which leads to such weird results.

Python 2’s str and unicode are really cheating, and I have been harmed by them several times. Python 3 has made great improvements in this regard, and I look forward to its full popularity!