Home >Backend Development >PHP Tutorial >Problem with crawler obtaining AJax data
I want to write a crawler that crawls pictures on the petal network. I encountered a problem with Ajax getting data
The requested URL in the picture, the text type of the Response is json
, but when I access this url alone, it is html
Please tell me how to get json data
I want to write a crawler that crawls pictures on the petal network. I encountered a problem with Ajax getting data
The requested URL in the picture, the text type of the Response is json
, but when I access this url alone, it is html
Please tell me how to get json data
This is controlled by header
Just add the corresponding request header. Attached is my code
<code>#coding=utf-8 import requests as RQ hburl = "http://huaban.com/boards/28195582/?ipva9fpx&max=759391397&limit=20&wfl=1" heads = { 'Host':"huaban.com", 'Referer':"{http://huaban.com/boards/28195582/", 'User-Agent':"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36", 'X-Request':"JSON", 'X-Requested-With':"XMLHttpRequest" } print RQ.get(hburl, headers=heads).content</code>
Referer is a common header used to prevent web pages from being obtained outside this site. When doing crawlers, be sure to ensure that the Referer is set