Home >Backend Development >PHP Tutorial >Problem with crawler obtaining AJax data

Problem with crawler obtaining AJax data

WBOY
WBOYOriginal
2016-07-06 13:52:211763browse

I want to write a crawler that crawls pictures on the petal network. I encountered a problem with Ajax getting data
Problem with crawler obtaining AJax data
The requested URL in the picture, the text type of the Response is json
, but when I access this url alone, it is html
Problem with crawler obtaining AJax data

Please tell me how to get json data

Reply content:

I want to write a crawler that crawls pictures on the petal network. I encountered a problem with Ajax getting data
Problem with crawler obtaining AJax data
The requested URL in the picture, the text type of the Response is json
, but when I access this url alone, it is html
Problem with crawler obtaining AJax data

Please tell me how to get json data

This is controlled by header
Problem with crawler obtaining AJax data

Just add the corresponding request header. Attached is my code

<code>#coding=utf-8

import requests as RQ

hburl = "http://huaban.com/boards/28195582/?ipva9fpx&max=759391397&limit=20&wfl=1"
heads = {
'Host':"huaban.com",
'Referer':"{http://huaban.com/boards/28195582/",
'User-Agent':"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36",
'X-Request':"JSON",
'X-Requested-With':"XMLHttpRequest"
}


print RQ.get(hburl, headers=heads).content</code>

Referer is a common header used to prevent web pages from being obtained outside this site. When doing crawlers, be sure to ensure that the Referer is set

Statement:
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn