Curl号称强大,却在抓花瓣网的首页时,怎么也不能成功,求解!!!!!!!
一直用curl抓页面,十分方便,屡试屡爽,却在抓取花瓣网首页的一个看似简单的操作中,发现怎么也不能成功。
基本代码如下:
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL, 'http://huaban.com/');
//模拟蜘蛛
//curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)');
//模拟普通浏览器
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727)');
//可以不要cookie, 因为不登陆,能返回首页即可
//curl_setopt($ch, CURLOPT_USERAGENT, '');
//其实也可以不要来路,模拟直接输入地址的
curl_setopt($ch, CURLOPT_REFERER, 'http://huaban.com/');
//curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
//curl_setopt($ch, CURLOPT_HEADER, 0); //输出header
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_exec($ch);
curl_close($ch);
我已经反复尝试不同的cookie header agent,都不能返回如同浏览器打开的那种可见页面,甚至尝试过 file_get_contents('http://huaban.com/'); 都没用。 返回的内容,绝大部分内容为js代码, 但之前成功抓取的页面,包括各种大小网站,也有js, 并不影响远程抓取和显示啊。 试了一天,百思不得其解,在csdn qq群里面也进行了讨论,有人说可能是curl不能运行js。 但现在哪个网站没有js代码能, 之前抓取的那些站,js也不在少数啊。没有一个失败的。。
实在不知道怎么解决,将这个问题丢出来,恳请高人作答。 到底是curl不行呢,还是这个网站太变态呢,还是方法不对呢?。。。。
------解决方案--------------------
这种优雅的小清新网站, 没JS还让它怎么存活在这个竞争惨烈的市场?
------解决方案--------------------
这个网站的特别之处就是它的绝大部分内容都是js动态生成的,通过js与后端程序交互不断的产生新内容
所以用curl抓取的只是它最初始的代码,也就是大段的js了
------解决方案--------------------
这是你要的数据吧?不知道你是怎么个抓包法
{"filter":"pin:category:all","pins":[{"pin_id":8447271,"user_id":394332,"board_id":1146189,"file_id":3483249,"file":{"farm":"farm1","bucket":"hbimg","key":"a1524741e8fae0916ba04c8d231f8ad23173ddb5baeff-rNFCpP","type":"image/jpeg","width":440,"height":5779,"frames":1},"media_type":0,"source":"weibo.com","link":"http://weibo.com/2134919185/yoVlDsGWs","raw_text":"小小灯泡大改造,你也来动手做一个吧~","text_meta":{},"via":2,"via_user_id":0,"original":null,"created_at":1340276725,"like_count":0,"comment_count":0,"repin_count":0,"is_private":0,"orig_source":"http://ww4.sinaimg.cn/bmiddle/7f404811jw1du5vv6dpnij.jpg","user":{"user_id":394332,"username":"Havetogo","urlname":"shouji132136652610","created_at":1338984624,"avatar":{"id":3061779,"farm":"farm1","bucket":"hbimg","key":"69d6d7842159946de9ca070c22da1714f259010afb4-WcVdOr","type":"image/jpeg","width":100,"height":100,"frames":1}},"board":{"board_id":1146189,"user_id":394332,"title":"创新的力量","description":"","category_id":null,"seq":6,"pin_count":1,"follow_count":0,"created_at":1340276719,"updated_at":1340276725,"is_private":0}},{"pin_id":8447272,"user_id":444560,"board_id":1146190,"file_id":2064511,"file":{"farm":"farm1","bucket":"hbimg","key":"aa4fab086fe5887299cf17df48a250f9df25e375c95b-M4izBs","type":"image/jpeg","width":440,"height":566,"frames":1},"media_type":0,"source":"weibo.com","link":"http://weibo.com/2596178104/ycTQfusRg","raw_text":"紫罗兰的致色原因:#翡翠知识普及#(61) 一般认为由于原生翡翠矿中含有微量的锰元素所致,由于锰元素的多寡和其他微量元素如铁等的渗入程度不同,其紫色也有浓淡深浅的剃度不同,如粉紫,茄紫,篮紫多种紫罗兰.十春九木,由于翡翠矿石含有锰是一种概率事件,所以紫色翡翠相对数量是很少的,再加上种水好则更少.","text_meta":{"tags":

ThesecrettokeepingaPHP-poweredwebsiterunningsmoothlyunderheavyloadinvolvesseveralkeystrategies:1)ImplementopcodecachingwithOPcachetoreducescriptexecutiontime,2)UsedatabasequerycachingwithRedistolessendatabaseload,3)LeverageCDNslikeCloudflareforservin

You should care about DependencyInjection(DI) because it makes your code clearer and easier to maintain. 1) DI makes it more modular by decoupling classes, 2) improves the convenience of testing and code flexibility, 3) Use DI containers to manage complex dependencies, but pay attention to performance impact and circular dependencies, 4) The best practice is to rely on abstract interfaces to achieve loose coupling.

Yes,optimizingaPHPapplicationispossibleandessential.1)ImplementcachingusingAPCutoreducedatabaseload.2)Optimizedatabaseswithindexing,efficientqueries,andconnectionpooling.3)Enhancecodewithbuilt-infunctions,avoidingglobalvariables,andusingopcodecaching

ThekeystrategiestosignificantlyboostPHPapplicationperformanceare:1)UseopcodecachinglikeOPcachetoreduceexecutiontime,2)Optimizedatabaseinteractionswithpreparedstatementsandproperindexing,3)ConfigurewebserverslikeNginxwithPHP-FPMforbetterperformance,4)

APHPDependencyInjectionContainerisatoolthatmanagesclassdependencies,enhancingcodemodularity,testability,andmaintainability.Itactsasacentralhubforcreatingandinjectingdependencies,thusreducingtightcouplingandeasingunittesting.

Select DependencyInjection (DI) for large applications, ServiceLocator is suitable for small projects or prototypes. 1) DI improves the testability and modularity of the code through constructor injection. 2) ServiceLocator obtains services through center registration, which is convenient but may lead to an increase in code coupling.

PHPapplicationscanbeoptimizedforspeedandefficiencyby:1)enablingopcacheinphp.ini,2)usingpreparedstatementswithPDOfordatabasequeries,3)replacingloopswitharray_filterandarray_mapfordataprocessing,4)configuringNginxasareverseproxy,5)implementingcachingwi

PHPemailvalidationinvolvesthreesteps:1)Formatvalidationusingregularexpressionstochecktheemailformat;2)DNSvalidationtoensurethedomainhasavalidMXrecord;3)SMTPvalidation,themostthoroughmethod,whichchecksifthemailboxexistsbyconnectingtotheSMTPserver.Impl


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

SublimeText3 English version
Recommended: Win version, supports code prompts!

SecLists
SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

Safe Exam Browser
Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

Atom editor mac version download
The most popular open source editor

Notepad++7.3.1
Easy-to-use and free code editor
