网页爬虫 - python中使用lxml解析中文网页出现编码问题，如何解决？

Question

问题

爬取IT之家网页http://it.ithome.com/category...
在使用requests.get得到网页后用lxml解析，但是打印出来后中文无法正常显示，
而打印requests.get得到的内容却可以正常显示。求原因及解决方法（BeautifulSoup可正常解析）。

代码

# coding: utf-8
import requests
from lxml import etree
from io import StringIO

init_url = 'http://it.ithome.com/category/31_4.html'
init_page = requests.get(init_url).text
print init_page.encode('utf-8')


parser = etree.HTMLParser()
html = etree.parse(StringIO(init_page), parser)
result = etree.tostring(html, pretty_print=True, method="html")
print result


news_titles = html.xpath('//p[@class="block"]/h2')
print len(news_titles)
for news_title in news_titles:
    print news_title.text

输出结果

• 使用print init_page.encode('utf-8')能正常输出

• 使用print result无法正常显示中文

Answer 1

Add the encoding parameter to this sentence

result = etree.tostring(html, encoding="gb2312", pretty_print=True, method="html")

I am Windows, so I use gb2312, and the test can display normally. If you are Linux, change it to utf-8