Home  >  Article  >  Backend Development  >  使用php的curl依据关键词爬取百度搜索结果页

使用php的curl依据关键词爬取百度搜索结果页

WBOY
WBOYOriginal
2016-06-13 12:10:191148browse

使用php的curl根据关键词爬取百度搜索结果页
我想实现的是每次根据给出的关键词搜索百度结果页,比如百度搜索结果是这样的,百度推广内容加正文:

而我使用curl爬取结果是这样的:

也就是说每次爬取结果都是无法爬取到百度推广的内容。请问哪位大师能指导一下,鄙人刚入门,望各位不吝指导。先谢过了。
其中php抓取代码如下:

<br /><?php<br />$url = "http://www.baidu.com/s?wd=生命动力";<br />// 构造包头,模拟浏览器请求<br />$header = array (<br />		"Host:www.baidu.com",<br />		"Content-Type:application/x-www-form-urlencoded",//post请求<br />		"Connection: keep-alive",<br />		'Referer:http://www.baidu.com',<br />		'User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; BIDUBrowser 2.6)'<br />);<br />$ch = curl_init ();<br />curl_setopt ( $ch, CURLOPT_URL, $url );<br />curl_setopt ( $ch, CURLOPT_HTTPHEADER, $header );<br />curl_setopt ( $ch, CURLOPT_RETURNTRANSFER, 1 );<br />// 执行<br />$content = curl_exec ( $ch );<br />if ($content == FALSE) {<br />	echo "error:" . curl_error ( $ch );<br />}<br />// 关闭<br />curl_close ( $ch );<br /><br />//输出结果<br />echo $content;<br />?><br />

------解决思路----------------------
你的user-agent没有模拟好,所以不行。
其实根本不需要用post,直接用get就可以了。
修改如下:
<br />$url = "http://www.baidu.com/s?wd=生命动力";<br /><br />$header = array (<br />        'User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36'<br />);<br />$ch = curl_init ();<br />curl_setopt ( $ch, CURLOPT_URL, $url );<br />curl_setopt ( $ch, CURLOPT_HTTPHEADER, $header );<br />curl_setopt ( $ch, CURLOPT_RETURNTRANSFER, 1 );<br />// 执行<br />$content = curl_exec ( $ch );<br />if ($content == FALSE) {<br />    echo "error:" . curl_error ( $ch );<br />}<br />// 关闭<br />curl_close ( $ch );<br /> <br />//输出结果<br />echo $content;<br />


Statement:
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn