Home  >  Article  >  Backend Development  >  求高手,模拟浏览器抓取网页

求高手,模拟浏览器抓取网页

WBOY
WBOYOriginal
2016-06-23 14:23:071321browse

如抓取http://map.sogou.com/api/这个网页,我写的程序,如果不带网址后面的"/",会抓取得不到,但是站上网(http://tool.chinaz.com/Tools/PageCode.aspx),不带最后面的"/"即可抓取到(即:http://map.sogou.com/api),他是什么原理?下面贴出我的代码,请改进

function file_get($url){	 ob_start();	 $ch = curl_init();	 	 curl_setopt($ch, CURLOPT_COOKIEJAR, "./cookie.txt");	 curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; InfoPath.1; CIBA)");	curl_setopt($ch, CURLOPT_URL, $url);	 curl_setopt($ch, CURLOPT_HEADER, FALSE);	 curl_setopt($ch, CURLOPT_COOKIESESSION, TRUE);	 curl_setopt($ch, CURLOPT_NOBODY, FALSE);	 curl_exec($ch);	 curl_close($ch);	 $content = ob_get_clean();	 	 	return $content;}


回复讨论(解决方案)

请加一句代码:

curl_setopt($ch, CURLOPT_FOLLOWLOCATION,1);

因为你抓取http://map.sogou.com/api,他返回的是301跳转。加上这句就能抓取跳转后的html了。

请加一句代码:

curl_setopt($ch, CURLOPT_FOLLOWLOCATION,1);

因为你抓取http://map.sogou.com/api,他返回的是301跳转。加上这句就能抓取跳转后的html了。
加上也不行,还是无法获取


请加一句代码:

curl_setopt($ch, CURLOPT_FOLLOWLOCATION,1);

因为你抓取http://map.sogou.com/api,他返回的是301跳转。加上这句就能抓取跳转后的html了。
加上也不行,还是无法获取


function file_get($url){	 ob_start();	 $ch = curl_init();	 	 curl_setopt($ch, CURLOPT_COOKIEJAR, "./cookie.txt");	 curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; InfoPath.1; CIBA)");	curl_setopt($ch, CURLOPT_URL, $url);	 curl_setopt($ch, CURLOPT_HEADER, FALSE);	 curl_setopt($ch, CURLOPT_COOKIESESSION, TRUE);	 curl_setopt($ch, CURLOPT_FOLLOWLOCATION,TRUE);	 curl_setopt($ch, CURLOPT_NOBODY, FALSE);	 curl_exec($ch);	 curl_close($ch);	 $content = ob_get_clean();	 	 	return $content;}


我测试过的,加上是行的呀。

CURLOPT_FOLLOWLOCATION

Statement:
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn