python - 求高手，使用urllib2请求网页内容时请求成功却返回空的值？

Question

这是出现类似问题的一个网页http://detail.zol.com.cn/inde...

测试代码
import urllib2
url = 'http://detail.zol.com.cn/inde...'
response = None
try:
response = urllib2.urlopen(url,timeout=5)
html = response.read()
print html
print "hehe"
except urllib2.URLError as e:
if hasattr(e, 'code'):

print 'Error code:',e.code

elif hasattr(e, 'reason'):

print 'Reason:',e.reason

finally:
if response:

response.close()

运行结果：C:Python27python.exe C:/Users/Administrator/PycharmProjects/untitled/data02
hehe

Process finished with exit code 0

这段代码运行后也是空值
page = urllib2.Request(url)
page.add_header('Referer', url)
page.add_header('User-Agent', "Mozilla/5.0 (Windows NT 6.2; rv:16.0) Gecko/20100101 Firefox/16.0")
r = urllib2.urlopen(page,timeout=5.0)
html = r.read()
soup = BeautifulSoup(html, 'lxml')

Answer 1

Be more violent and bring cookies

import requests

url = 'http://detail.zol.com.cn/index.php?c=SearchList&keyword=coolpad_8297_w01'

headers = {
    'Cookie': 'userProvinceId=2; userCityId=0; userLocationId=26; proIp=123; ip_ck=4cKD5vP/j7QuNjUyMTk4LjE0Njk0Mzg5MzQ%3D; lv=1469438963; vn=1; Hm_lvt_ae5edc2bc4fc71370807f6187f0a2dd0=1469438964; Hm_lpvt_ae5edc2bc4fc71370807f6187f0a2dd0=1469438964; z_day=rdetail=1; z_pro_city=s_provice%3Dshanghai%26s_city%3Dxingqu',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36'
}
r = requests.get(url, headers=headers)
print r.text

Answer 2

I cleared the browser cookies. When I visited this page, I found that it was also empty. I analyzed it and found that its cookies were encrypted and set with js. If you are good at js, you can try to analyze it. If it doesn’t work, use selenium to operate chrome. Let’s do it.