Home  >  Article  >  Backend Development  >  Get URLs using PHP and cURL

Get URLs using PHP and cURL

WBOY
WBOYOriginal
2016-07-25 09:08:381068browse
http://css.dzone.com/articles/retrieving-urls-parallel-curl
  1. class Footo_Content_Retrieve_HTTP_CURLParallel
  2. {
  3. /**
  4. * Fetch a collection of URLs in parallell using cURL. The results are
  5. * returned as an associative array, with the URLs as the key and the
  6. * content of the URLs as the value.
  7. *
  8. * @param array $addresses An array of URLs to fetch.
  9. * @return array The content of each URL that we've been asked to fetch.
  10. **/
  11. public function retrieve($addresses)
  12. {
  13. $multiHandle = curl_multi_init();
  14. $handles = array();
  15. $results = array();
  16. foreach($addresses as $url)
  17. {
  18. $handle = curl_init($url);
  19. $handles[$url] = $handle;
  20. curl_setopt_array($handle, array(
  21. CURLOPT_HEADER => false,
  22. CURLOPT_RETURNTRANSFER => true,
  23. ));
  24. curl_multi_add_handle($multiHandle, $handle);
  25. }
  26. //execute the handles
  27. $result = CURLM_CALL_MULTI_PERFORM;
  28. $running = false;
  29. // set up and make any requests..
  30. while ($result == CURLM_CALL_MULTI_PERFORM)
  31. {
  32. $result = curl_multi_exec($multiHandle, $running);
  33. }
  34. // wait until data arrives on all sockets
  35. while($running && ($result == CURLM_OK))
  36. {
  37. if (curl_multi_select($multiHandle) > -1)
  38. {
  39. $result = CURLM_CALL_MULTI_PERFORM;
  40. // while we need to process sockets
  41. while ($result == CURLM_CALL_MULTI_PERFORM)
  42. {
  43. $result = curl_multi_exec($multiHandle, $running);
  44. }
  45. }
  46. }
  47. // clean up
  48. foreach($handles as $url => $handle)
  49. {
  50. $results[$url] = curl_multi_getcontent($handle);
  51. curl_multi_remove_handle($multiHandle, $handle);
  52. curl_close($handle);
  53. }
  54. curl_multi_close($multiHandle);
  55. return $results;
  56. }
  57. }
复制代码


Statement:
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn