PHP 利用 Curl Functions 可以完成各种传送文件操作,比如模拟浏览器发送GET,POST请求等等,然而因为php语言本身不支持多线程,所以开发爬虫程序效率并不高,因此经常需要借助Curl Multi Functions 这个功能实现并发多线程的访问多个url地址以实现并发多线程抓取网页或者下载文件,至于具体实现过程,请参考下面几个例子:
(1)下面这段代码是实现抓取多个URL,然后将抓取的URL的页面代码写入指定的文件
<p>$urls = array( </p>'http://www.scutephp.com/', <br />'http://www.google.com/', <br />'http://www.example.com/' <br />); // 设置要抓取的页面URL <br />$save_to='/test.txt'; // 把抓取的代码写入该文件 <br />$st = fopen($save_to,"a"); <br />$mh = curl_multi_init(); <br />foreach ($urls as $i => $url) { <br />$conn[$i] = curl_init($url); <br />curl_setopt($conn[$i], CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"); <br />curl_setopt($conn[$i], CURLOPT_HEADER ,0); <br />curl_setopt($conn[$i], CURLOPT_CONNECTTIMEOUT,60); <br />curl_setopt($conn[$i], CURLOPT_FILE,$st); // 将爬取的代码写入文件 <br />curl_multi_add_handle ($mh,$conn[$i]); <br />} // 初始化 <br />do { <br />curl_multi_exec($mh,$active); <br />} while ($active); // 执行 <br />foreach ($urls as $i => $url) { <br />curl_multi_remove_handle($mh,$conn[$i]); <br />curl_close($conn[$i]); <br />} // 结束清理 <br />curl_multi_close($mh); <br /><p>fclose($st);
(2)下面这段代码和上面差不多意思,只不过这个地方是将获得的代码先放入变量,然后再将获取到的内容写入指定的文件
<p>$urls = array( </p>'http://www.scutephp.com/', <br />'http://www.google.com/', <br />'http://www.example.com/' <br />); <br />$save_to='/test.txt'; // 把抓取的代码写入该文件 <br />$st = fopen($save_to,"a"); <br />$mh = curl_multi_init(); <br />foreach ($urls as $i => $url) { <br />$conn[$i] = curl_init($url); <br />curl_setopt($conn[$i], CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"); <br />curl_setopt($conn[$i], CURLOPT_HEADER ,0); <br />curl_setopt($conn[$i], CURLOPT_CONNECTTIMEOUT,60); <br />curl_setopt($conn[$i],CURLOPT_RETURNTRANSFER,true); // 不将爬取代码写到浏览器,而是转化为字符串 <br />curl_multi_add_handle ($mh,$conn[$i]); <br />} <br />do { <br />curl_multi_exec($mh,$active); <br />} while ($active); <br />foreach ($urls as $i => $url) { <br />$data = curl_multi_getcontent($conn[$i]); // 获得爬取的代码字符串 <br />fwrite($st,$data); // 将字符串写入文件<br />} // 获得数据变量,并写入文件 <br />foreach ($urls as $i => $url) { <br />curl_multi_remove_handle($mh,$conn[$i]); <br />curl_close($conn[$i]); <br />}<br />curl_multi_close($mh); <br /><p>fclose($st);
(3)下面这段代码实现的是利用 PHP 的 Curl Functions 实现并发多线程下载文件
<p>$urls=array(</p> 'http://www.scutephp.com/5w.zip',<br /> 'http://www.scutephp.com/5w.zip',<br /> 'http://www.scutephp.com/5w.zip'<br />);<br />$save_to='./home/';<br />$mh=curl_multi_init();<br />foreach($urls as $i=>$url){<br /> $g=$save_to.basename($url);<br /> if(!is_file($g)){<br /> $conn[$i]=curl_init($url);<br /> $fp[$i]=fopen($g,"w");<br /> curl_setopt($conn[$i],CURLOPT_USERAGENT,"Mozilla/4.0(compatible; MSIE 7.0; Windows NT 6.0)");<br /> curl_setopt($conn[$i],CURLOPT_FILE,$fp[$i]);<br /> curl_setopt($conn[$i],CURLOPT_HEADER ,0);<br /> curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,60);<br /> curl_multi_add_handle($mh,$conn[$i]);<br /> }<br />}<br />do{<br /> $n=curl_multi_exec($mh,$active);<br />}while($active);<br />foreach($urls as $i=>$url){<br /> curl_multi_remove_handle($mh,$conn[$i]);<br /> curl_close($conn[$i]);<br /> fclose($fp[$i]);<br />}<br />curl_multi_close($mh);$urls=array(<br /> 'http://www.scutephp.com/5w.zip',<br /> 'http://www.scutephp.com/5w.zip',<br /> 'http://www.scutephp.com/5w.zip'<br />);<br />$save_to='./home/';<br />$mh=curl_multi_init();<br />foreach($urls as $i=>$url){<br /> $g=$save_to.basename($url);<br /> if(!is_file($g)){<br /> $conn[$i]=curl_init($url);<br /> $fp[$i]=fopen($g,"w");<br /> curl_setopt($conn[$i],CURLOPT_USERAGENT,"Mozilla/4.0(compatible; MSIE 7.0; Windows NT 6.0)");<br /> curl_setopt($conn[$i],CURLOPT_FILE,$fp[$i]);<br /> curl_setopt($conn[$i],CURLOPT_HEADER ,0);<br /> curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,60);<br /> curl_multi_add_handle($mh,$conn[$i]);<br /> }<br />}<br />do{<br /> $n=curl_multi_exec($mh,$active);<br />}while($active);<br />foreach($urls as $i=>$url){<br /> curl_multi_remove_handle($mh,$conn[$i]);<br /> curl_close($conn[$i]);<br /> fclose($fp[$i]);<br />}<br />curl_multi_close($mh);

PHP is mainly procedural programming, but also supports object-oriented programming (OOP); Python supports a variety of paradigms, including OOP, functional and procedural programming. PHP is suitable for web development, and Python is suitable for a variety of applications such as data analysis and machine learning.

PHP originated in 1994 and was developed by RasmusLerdorf. It was originally used to track website visitors and gradually evolved into a server-side scripting language and was widely used in web development. Python was developed by Guidovan Rossum in the late 1980s and was first released in 1991. It emphasizes code readability and simplicity, and is suitable for scientific computing, data analysis and other fields.

PHP is suitable for web development and rapid prototyping, and Python is suitable for data science and machine learning. 1.PHP is used for dynamic web development, with simple syntax and suitable for rapid development. 2. Python has concise syntax, is suitable for multiple fields, and has a strong library ecosystem.

PHP remains important in the modernization process because it supports a large number of websites and applications and adapts to development needs through frameworks. 1.PHP7 improves performance and introduces new features. 2. Modern frameworks such as Laravel, Symfony and CodeIgniter simplify development and improve code quality. 3. Performance optimization and best practices further improve application efficiency.

PHPhassignificantlyimpactedwebdevelopmentandextendsbeyondit.1)ItpowersmajorplatformslikeWordPressandexcelsindatabaseinteractions.2)PHP'sadaptabilityallowsittoscaleforlargeapplicationsusingframeworkslikeLaravel.3)Beyondweb,PHPisusedincommand-linescrip

PHP type prompts to improve code quality and readability. 1) Scalar type tips: Since PHP7.0, basic data types are allowed to be specified in function parameters, such as int, float, etc. 2) Return type prompt: Ensure the consistency of the function return value type. 3) Union type prompt: Since PHP8.0, multiple types are allowed to be specified in function parameters or return values. 4) Nullable type prompt: Allows to include null values and handle functions that may return null values.

In PHP, use the clone keyword to create a copy of the object and customize the cloning behavior through the \_\_clone magic method. 1. Use the clone keyword to make a shallow copy, cloning the object's properties but not the object's properties. 2. The \_\_clone method can deeply copy nested objects to avoid shallow copying problems. 3. Pay attention to avoid circular references and performance problems in cloning, and optimize cloning operations to improve efficiency.

PHP is suitable for web development and content management systems, and Python is suitable for data science, machine learning and automation scripts. 1.PHP performs well in building fast and scalable websites and applications and is commonly used in CMS such as WordPress. 2. Python has performed outstandingly in the fields of data science and machine learning, with rich libraries such as NumPy and TensorFlow.


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

PhpStorm Mac version
The latest (2018.2.1) professional PHP integrated development tool

SublimeText3 Linux new version
SublimeText3 Linux latest version

VSCode Windows 64-bit Download
A free and powerful IDE editor launched by Microsoft

ZendStudio 13.5.1 Mac
Powerful PHP integrated development environment

Notepad++7.3.1
Easy-to-use and free code editor