search

Home  >  Q&A  >  body text

php - snoopy crawler reports error 405 Not Allowed

Code

$httpClass = new Snoopy();
$httpClass->fetch('https://v.qq.com/');
$url = $httpClass->results;
print_r($url);die();

Crawling https://www.baidu.com/ keeps reporting 405 errors

Crawling https://v.qq.com/ is running normally

Looking for an answer, I’ve been working on it all afternoon and still can’t solve it

高洛峰高洛峰2843 days ago656

reply all(2)I'll reply

  • 淡淡烟草味

    淡淡烟草味2017-05-16 13:00:40

    Baidu probably has a crawler-like setting. You need to disguise it and define a UA or something like that
    Reference: http://www.4wei.cn/archives/396

    reply
    0
  • 天蓬老师

    天蓬老师2017-05-16 13:00:40

    This is not a Snoopy-type problem. It's because you don't know much about crawlers. Since there are crawlers, of course there will be anti-crawler technology. The simplest is based on the browser identifier or the referer in the request header, etc. Big websites like Baidu and Tencent will not want us to crawl data, so they must have taken many preventive measures. Therefore, it is recommended to understand the knowledge of crawlers before crawling data.

    reply
    0
  • Cancelreply