采集已经不是什么新名词了,很多站长为了省事,也局限于人力的缺乏,使用程序来给自己的网站添砖加瓦,比如本人的个人网站www.xxfsw.com也采集了大量的新闻,那么如果实现呢?今天我们运用php来实现这个功能。
谈到采集,我们不得不说两个东西,第一个是如何获取远程网站的源代码,这个可以通过php的一个扩展curl来获取,另一个是如果去匹配你需要的信息,这个的解决办法是正则表达式。
Windows下开启curl的方法如下:
1、拷贝PHP目录中的libeay32.dll, ssleay32.dll, php5ts.dll, php_curl.dll文件到 system32 目录。
2、修改php.ini:配置好 extension_dir ,去掉 extension = php_curl.dll 前面的分号。
3、重起apache。
Linux下开启curl的方法如下:
进入安装 原php 的源码目录,
cd ext
cd curl
phpize
./configure --with-curl =DIR
make
就会在PHPDIR/ext/curl /moudles/下生成curl .so的文件。
复制curl .so文件到extensions的配置目录,修改php .ini就好了。
然后你就可以利用curl来获取到指定url的网页源码了,这里给大家一个封装好的函数:
以下为引用的内容: function getwebcontent($url){ $ch = curl_init(); $timeout = 10; curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout); curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1); $contents = trim(curl_exec($ch)); curl_close($ch); return $contents; } |
接下来就应该说到php中的正则表达式了:
1.中括号
[0-9]匹配0-9
[a-z]匹配a-z小写字母
[A-Z]匹配A-Z大写字母
[a-zA-Z]匹配所有大小写字母
可以使用ascii来制定更多
2.量词
以下为引用的内容: p+匹配至少一个含p的字符串 p*陪陪任何包含0个或多个p的字符串 p?匹配任何包含0个或一个p的字符串 p{2}匹配包含2个p的序列的字符串 p{2,3}匹配任何包含2个或3个的字符串 p$匹配任何以p结尾的字符串 ^p匹配任何以p开头的字符串 [^a-zA-Z]匹配任何不包含a-zA-Z的字符串 p.p匹配任何包含p、接下来是任何字符、再接下来有又是p的字符串 ^.{2}$匹配任何值包含2个字符的字符串 (.*)b>匹配任何被>包围的字符串 p(hp)*匹配任何一个包含p,后面是多个或0个hp的字符串 |
3.预定义字符范围
以下为引用的内容: [:alpha:]同[a-zA-Z] [:alnum:]同[a-zA-Z0-9] [:cntrl:]匹配控制字符,比如制表符,反斜杠,退格符 [:digit:]同[0-9] [:graph:]所有ASCII33~166范围内可以打印的字符 [:lower:]同[a-z] [:punct:]标点符号 [:upper:]同[A-Z] [:space:]空白字符,可以是空格、水平制表符、换行、换页、回车 [:xdigit:]十六进制符同[a-fA-F0-9] |
废话不多说,直接上我的源码吧,有什么不懂的可以上百度查查。
以下为引用的内容: header("Content-type: text/html; charset=utf-8"); getinfo("http://rss.sina.com.cn/rollnews/news/gn_total.js",1); getinfo("http://rss.sina.com.cn/rollnews/news/gj_total.js",2); getinfo("http://rss.sina.com.cn/rollnews/news/sh_total.js",3); getinfo("http://rss.sina.com.cn/rollnews/sports/sports_total.js",4); getinfo("http://rss.sina.com.cn/rollnews/tech/tech1_total.js",5); getinfo("http://rss.sina.com.cn/rollnews/finance/finance1_news_total.js",6); getinfo("http://rss.sina.com.cn/rollnews/ent/ent_total.js",7); getinfo("http://rss.sina.com.cn/rollnews/jczs/jczs_total.js",8); function getinfo($infourl,$catid) { $pagecontent=getwebcontent($infourl); preg_match_all("/title:\"(.*?)\"/", $pagecontent, $match); $titlearr=$match[1]; preg_match_all("/link:\"(.*?)\"/", $pagecontent, $match); $urlarr=$match[1]; for ($i=1;$i $title=iconv("gbk","utf-8",$titlearr[$i-1]); $content=iconv("gbk","utf-8",getnewscontent($urlarr[$i])); $content=mysql_escape_string($content); if(!insertdb($title,$content,$catid)) break; } } function insertdb($title,$content,$catid){ 将数据写入你的库 } function getnewscontent($newsurl){ $newscontent=getwebcontent($newsurl); preg_match_all("/ ([\s\S]*?)/",$newscontent,$match);
$content=preg_replace("/ $content=preg_replace("/ |
然后如何实现比较实时的同步呢,这可以利用windows下的任务计划或linux下的crontab 了,定时(比如十分钟)执行这个程序,这样,你就不再愁网站没有内容了,哈哈,另外本人开了个工作室www.beijingjianzhan.com(北京建站),我们开发了一个系统,不仅能够采集信息,而且能自动地进行再加工,进行伪原创,这样就更符合搜索引擎的品味了,让你的网站疯狂地被收录吧,另外可以加我的Q376504340讨论技术性话题。

Long URLs, often cluttered with keywords and tracking parameters, can deter visitors. A URL shortening script offers a solution, creating concise links ideal for social media and other platforms. These scripts are valuable for individual websites a

Following its high-profile acquisition by Facebook in 2012, Instagram adopted two sets of APIs for third-party use. These are the Instagram Graph API and the Instagram Basic Display API.As a developer building an app that requires information from a

Laravel simplifies handling temporary session data using its intuitive flash methods. This is perfect for displaying brief messages, alerts, or notifications within your application. Data persists only for the subsequent request by default: $request-

This is the second and final part of the series on building a React application with a Laravel back-end. In the first part of the series, we created a RESTful API using Laravel for a basic product-listing application. In this tutorial, we will be dev

Laravel provides concise HTTP response simulation syntax, simplifying HTTP interaction testing. This approach significantly reduces code redundancy while making your test simulation more intuitive. The basic implementation provides a variety of response type shortcuts: use Illuminate\Support\Facades\Http; Http::fake([ 'google.com' => 'Hello World', 'github.com' => ['foo' => 'bar'], 'forge.laravel.com' =>

The PHP Client URL (cURL) extension is a powerful tool for developers, enabling seamless interaction with remote servers and REST APIs. By leveraging libcurl, a well-respected multi-protocol file transfer library, PHP cURL facilitates efficient execution of various network protocols, including HTTP, HTTPS, and FTP. This extension offers granular control over HTTP requests, supports multiple concurrent operations, and provides built-in security features.

Do you want to provide real-time, instant solutions to your customers' most pressing problems? Live chat lets you have real-time conversations with customers and resolve their problems instantly. It allows you to provide faster service to your custom

The 2025 PHP Landscape Survey investigates current PHP development trends. It explores framework usage, deployment methods, and challenges, aiming to provide insights for developers and businesses. The survey anticipates growth in modern PHP versio


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

DVWA
Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

SublimeText3 Mac version
God-level code editing software (SublimeText3)

PhpStorm Mac version
The latest (2018.2.1) professional PHP integrated development tool

Safe Exam Browser
Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

Zend Studio 13.0.1
Powerful PHP integrated development environment
