Home  >  Article  >  php教程  >  php使用curl代理实现抓取数据的方法_php技巧

php使用curl代理实现抓取数据的方法_php技巧

PHP中文网
PHP中文网Original
2016-05-25 17:09:48968browse

这篇文章主要介绍了php使用curl代理实现抓取数据的方法,结合实例形式分析了php使用curl实现代理抓取数据的操作技巧,需要的朋友可以参考下

本文实例讲述了php使用curl代理实现抓取数据的方法。分享给大家供大家参考,具体如下:

<?php
define ( &#39;IS_PROXY&#39;, true ); //是否启用代理
function async_get_url($url_array, $wait_usec = 0)
{
  if (!is_array($url_array))
    return false;
  $wait_usec = intval($wait_usec);
  $data  = array();
  $handle = array();
  $running = 0;
  $mh = curl_multi_init(); // 开启多线程
  $i = 0;
  foreach($url_array as $url) {
    $ch = curl_init();
    if (IS_PROXY) {
    //以下代码设置代理服务器
    //代理服务器地址http://www.php.cn/ !!Hong Kong, China的速度比较好
    curl_setopt ($ch, CURLOPT_PROXY,&#39;110.4.12.170:80&#39; );
    }
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); // return don&#39;t print
    curl_setopt($ch, CURLOPT_TIMEOUT, 30); //设置超时时间
    curl_setopt($ch, CURLOPT_USERAGENT, &#39;Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)&#39;);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); // 302 redirect
    curl_setopt($ch, CURLOPT_MAXREDIRS, 7); //HTTp定向级别
    curl_multi_add_handle($mh, $ch); // 把 curl resource 放进 multi curl handler 里
    $handle[$i++] = $ch;
  }
  /* 执行 */
  do {
    $mrc = curl_multi_exec($mh, $running);
    if ($wait_usec > 0) /* 每个 connect 要间隔多久 */
      usleep($wait_usec); // 250000 = 0.25 sec
  } while ($mrc == CURLM_CALL_MULTI_PERFORM);
  while ($running && $mrc == CURLM_OK) {
    if (curl_multi_select($mh) != -1) {
      do {
        $mrc = curl_multi_exec($mh, $running);
      } while ($mrc == CURLM_CALL_MULTI_PERFORM);
    }
  }
  /* 读取资料 */
  foreach($handle as $i => $ch) {
    $content = curl_multi_getcontent($ch);
    $data[$i] = (curl_errno($ch) == 0) ? $content : false;
  }
  /* 移除 handle*/
  foreach($handle as $ch) {
    curl_multi_remove_handle($mh, $ch);
  }
  curl_multi_close($mh);
  return $data;
}
$urls = array(&#39;http://map.baidu.com&#39;);
$re = async_get_url($urls);
echo $re[0];
?>

Statement:
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Previous article:刷积分脚本Next article:php获取网站关键词