search

Home  >  Q&A  >  body text

Python BeautifulSoup爬网页中文输出编码问题

环境:OS X 10.11
Python版本:2.7.10
IDE:PyCharm CE

代码如下:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
# import urllib2
import requests

url = 'http://www.tripadvisor.cn/Attractions-g60763-Activities-New_York_City_New_York.html'
web_data = requests.get(url)
soup = BeautifulSoup(web_data.text, 'lxml')
titles = soup.select(
    'p.property_title > a[target="_blank"]')

for title in titles:
    data = {
        'title':title.get_text()
    }
    print data

结果为:

{'title': u'\u4e2d\u592e\u516c\u56ed'}
{'title': u'\u5927\u90fd\u4f1a\u827a\u672f\u535a\u7269\u9986'}
{'title': u'\u5ced\u77f3\u4e4b\u5dc5\u89c2\u666f\u53f0'}
{'title': u'9/11\u7eaa\u5ff5\u9986'}
{'title': u'\u66fc\u54c8\u987f\u5929\u9645\u7ebf'}
{'title': u'\u767e\u8001\u6c47'}
{'title': u'\u9ad8\u7ebf\u516c\u56ed'}
{'title': u'\u5927\u4e2d\u592e\u8f66\u7ad9'}
{'title': u'\u5f17\u91cc\u514b\u7f8e\u672f\u6536\u85cf\u9986'}
{'title': u'\u5e1d\u56fd\u5927\u53a6'}
{'title': u'\u5e03\u83b1\u6069\u516c\u56ed'}
{'title': u'\u4e16\u8d38\u5927\u53a6\u9057\u5740\u535a\u7269\u9986\u5de5\u4f5c\u5ba4'}

以上网站的编码采用了utf-8
请问为何爬出来的中文都是unicode字符,而不是中文。PyCharm中也都设置为了UTF-8。
是需要做什么转换吗?
谢谢

PHPzPHPz2892 days ago491

reply all(4)I'll reply

  • PHPz

    PHPz2017-04-18 09:16:43

    Change the last sentence of print data to print title.get_text() and see

    reply
    0
  • 天蓬老师

    天蓬老师2017-04-18 09:16:43

    No need, this can be used directly

    reply
    0
  • 黄舟

    黄舟2017-04-18 09:16:43

    I want to ask if you have solved it. I also encountered this problem.

    reply
    0
  • PHP中文网

    PHP中文网2017-04-18 09:16:43

    The encoding problem when printing is still not solved upstairs, because when printing values ​​with Chinese characters in a dictionary, etc., the encoding will be automatically detected by the system.
    The perfect solution is

    print repr(a).decode("unicode–escape")
    

    reply
    0
  • Cancelreply