>  기사  >  백엔드 개발  >  php curl抓取不到页面及来路问题?

php curl抓取不到页面及来路问题?

WBOY
WBOY원래의
2016-06-17 08:31:582423검색

$url = "mp.weixinbridge.com/mp/";
$ch = curl_init();
$timeout = 1;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$contents = curl_exec($ch);
curl_close($ch);
echo $contents;
相对路径的图片不能显示,如何使相对路径的图片正常显示?

回复内容:

谢邀,哥们,抓不到数据是因为:
1.你没有写header
2.没有写cookie,
3.没有针对https的url特殊设置
所以没有抓到数据,好好研究我写的这个代码,这个是可以抓到数据的。
要是帮到了你,给哥点个赞,支持下。
<code class="language-text"><?php $url = "https://www.zhihu.com/";

	$ch = curl_init();
    // 设置浏览器的特定header
    curl_setopt($ch, CURLOPT_HTTPHEADER, array(
        "Host: www.zhihu.com",
        "Connection: keep-alive",
        "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Upgrade-Insecure-Requests: 1",
        "DNT:1",
        "Accept-Language: zh-CN,zh;q=0.8,en-GB;q=0.6,en;q=0.4,en-US;q=0.2",
        'Cookie:_za=4540d427-eee1-435a-a533-66ecd8676d7d; __utma=51854390.3169871.1440319332.1441339521.1442067491.5; __utmz=51854390.1442067491.5.5.utmcsr=baidu|utmccn=(organic)|utmcmd=organic; __utmv=51854390.100-1|2=registration_date=20140525=1^3=entry_date=20140525=1; q_c1=efa8c4ccdba04f63a0ba88845f485836|1451394239000|1440047640000; _xsrf=20c250b28098f92459cac05a3944d48d; cap_id="ZWQ5OGIzN2JiZWNmNGRlNGE3YTE1MTE0YTA5YjY1NjE=|1451394239|0efd13fc965c43c0fb6a7a2523b5dac4d1dac7e3"; z_c0="QUFCQXRLa3ZBQUFYQUFBQVlRSlZUY29ScWxZN0k3T1BHaFdqb1JNVlVZekNnZ0trU0xXdEdnPT0=|1451394250|02ed77acc81edbf2340fd0ce1b13618862b3674e"; unlock_ticket="QUFCQXRLa3ZBQUFYQUFBQVlRSlZUZEtMZ2xiM21FNDRmdzdsX1NnOVdieUp3M1VtY0RsaUVBPT0=|1451394250|8cf44cefb523b2973eca01f0918ef97fc03a4994"',
		
		));
    curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0');
    // 在HTTP请求头中"Referer: "的内容。
    curl_setopt($ch, CURLOPT_REFERER,"https://www.baidu.com/s?word=%E7%9F%A5%E4%B9%8E&tn=sitehao123&ie=utf-8&ssl_sample=normal&f=3&rsp=0");
    curl_setopt($ch, CURLOPT_ENCODING, "gzip, deflate, sdch");
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_TIMEOUT,120);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);//302redirect
    // 针对https的设置
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
    $html = curl_exec($ch);
    curl_close($ch);
    if($html === false) {
        echo 'Curl error: ' . curl_error($ch) . "<br>\n\r";
    } else {
		echo $html;
	}
</code>
我在暑假的时候爬过知乎,而且就是用的php+curl。知乎是有反爬虫机制的,你要尽量伪装成浏览器,包括header、useragent、cookie等等都设成浏览器上的一样,至于这些在哪可以看到请善用chrome的F12控制台。光伪装成浏览器是不够的,因为知乎有的页面是gzip加密的哦,所以你还要做好gzip解密的措施。如果知乎觉得你的行为可疑,知乎还会不定频率的给你返回空白页面,所以你还要做好数据验证的措施。总的来说爬知乎是不难的,但要稳定可靠的一口气爬完整个知乎还是很困难的。 curl配置增加cookie信息和header头部试试看,有些网站防采集需要尽可能的模拟。
성명:
본 글의 내용은 네티즌들의 자발적인 기여로 작성되었으며, 저작권은 원저작자에게 있습니다. 본 사이트는 이에 상응하는 법적 책임을 지지 않습니다. 표절이나 침해가 의심되는 콘텐츠를 발견한 경우 admin@php.cn으로 문의하세요.