search
HomeBackend DevelopmentPHP Tutorial让你的站跟新浪的新闻数据保持同步_PHP

采集已经不是什么新名词了,很多站长为了省事,也局限于人力的缺乏,使用程序来给自己的网站添砖加瓦,比如本人的个人网站www.xxfsw.com也采集了大量的新闻,那么如果实现呢?今天我们运用php来实现这个功能。

谈到采集,我们不得不说两个东西,第一个是如何获取远程网站的源代码,这个可以通过php的一个扩展curl来获取,另一个是如果去匹配你需要的信息,这个的解决办法是正则表达式。

Windows下开启curl的方法如下:

1、拷贝PHP目录中的libeay32.dll, ssleay32.dll, php5ts.dll, php_curl.dll文件到 system32 目录。

2、修改php.ini:配置好 extension_dir ,去掉 extension = php_curl.dll 前面的分号。

3、重起apache。

Linux下开启curl的方法如下:

进入安装 原php 的源码目录,

cd ext
cd curl
phpize
./configure --with-curl =DIR
make

就会在PHPDIR/ext/curl /moudles/下生成curl .so的文件。

复制curl .so文件到extensions的配置目录,修改php .ini就好了。

然后你就可以利用curl来获取到指定url的网页源码了,这里给大家一个封装好的函数:

以下为引用的内容:
function getwebcontent($url){
    $ch = curl_init();
    $timeout = 10;
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
    curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
    $contents = trim(curl_exec($ch));
    curl_close($ch);
    return  $contents;
}

接下来就应该说到php中的正则表达式了:

1.中括号

[0-9]匹配0-9

[a-z]匹配a-z小写字母

[A-Z]匹配A-Z大写字母

[a-zA-Z]匹配所有大小写字母

可以使用ascii来制定更多

2.量词

以下为引用的内容:
p+匹配至少一个含p的字符串 
p*陪陪任何包含0个或多个p的字符串 
p?匹配任何包含0个或一个p的字符串 
p{2}匹配包含2个p的序列的字符串 
p{2,3}匹配任何包含2个或3个的字符串 
p$匹配任何以p结尾的字符串 
^p匹配任何以p开头的字符串 
[^a-zA-Z]匹配任何不包含a-zA-Z的字符串 
p.p匹配任何包含p、接下来是任何字符、再接下来有又是p的字符串 
^.{2}$匹配任何值包含2个字符的字符串 
(.*)b>匹配任何被>包围的字符串 
p(hp)*匹配任何一个包含p,后面是多个或0个hp的字符串

3.预定义字符范围

以下为引用的内容:
[:alpha:]同[a-zA-Z] 
[:alnum:]同[a-zA-Z0-9] 
[:cntrl:]匹配控制字符,比如制表符,反斜杠,退格符 
[:digit:]同[0-9] 
[:graph:]所有ASCII33~166范围内可以打印的字符 
[:lower:]同[a-z] 
[:punct:]标点符号 
[:upper:]同[A-Z] 
[:space:]空白字符,可以是空格、水平制表符、换行、换页、回车 
[:xdigit:]十六进制符同[a-fA-F0-9]

废话不多说,直接上我的源码吧,有什么不懂的可以上百度查查。

以下为引用的内容:
header("Content-type: text/html; charset=utf-8");
getinfo("http://rss.sina.com.cn/rollnews/news/gn_total.js",1);
getinfo("http://rss.sina.com.cn/rollnews/news/gj_total.js",2);
getinfo("http://rss.sina.com.cn/rollnews/news/sh_total.js",3);
getinfo("http://rss.sina.com.cn/rollnews/sports/sports_total.js",4);
getinfo("http://rss.sina.com.cn/rollnews/tech/tech1_total.js",5);
getinfo("http://rss.sina.com.cn/rollnews/finance/finance1_news_total.js",6);
getinfo("http://rss.sina.com.cn/rollnews/ent/ent_total.js",7);
getinfo("http://rss.sina.com.cn/rollnews/jczs/jczs_total.js",8);
function getinfo($infourl,$catid)
{
    $pagecontent=getwebcontent($infourl);
    preg_match_all("/title:\"(.*?)\"/", $pagecontent, $match);
    $titlearr=$match[1];
    preg_match_all("/link:\"(.*?)\"/", $pagecontent, $match);
    $urlarr=$match[1];
    for ($i=1;$i        echo "go {$titlearr[$i-1]}\n";
        $title=iconv("gbk","utf-8",$titlearr[$i-1]);
        $content=iconv("gbk","utf-8",getnewscontent($urlarr[$i]));
        $content=mysql_escape_string($content);
        if(!insertdb($title,$content,$catid)) break;
    }
}
function insertdb($title,$content,$catid){   
    将数据写入你的库
}
function getnewscontent($newsurl){
    $newscontent=getwebcontent($newsurl);
    preg_match_all("/
([\s\S]*?)/",$newscontent,$match);
    $content=preg_replace("//si","",$match[1][0]);
    $content=preg_replace("/
.*?/si","",$content);
    $content=preg_replace("/
.*?
/si","",$content);
    $content=str_replace("
","",$content);
    return $content;
}
function getwebcontent($url){
    $ch = curl_init();
    $timeout = 10;
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
    curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
    $contents = trim(curl_exec($ch));
    curl_close($ch);
    return  $contents;
}
?>

然后如何实现比较实时的同步呢,这可以利用windows下的任务计划或linux下的crontab 了,定时(比如十分钟)执行这个程序,这样,你就不再愁网站没有内容了,哈哈,另外本人开了个工作室www.beijingjianzhan.com(北京建站),我们开发了一个系统,不仅能够采集信息,而且能自动地进行再加工,进行伪原创,这样就更符合搜索引擎的品味了,让你的网站疯狂地被收录吧,另外可以加我的Q376504340讨论技术性话题。

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
11 Best PHP URL Shortener Scripts (Free and Premium)11 Best PHP URL Shortener Scripts (Free and Premium)Mar 03, 2025 am 10:49 AM

Long URLs, often cluttered with keywords and tracking parameters, can deter visitors. A URL shortening script offers a solution, creating concise links ideal for social media and other platforms. These scripts are valuable for individual websites a

Introduction to the Instagram APIIntroduction to the Instagram APIMar 02, 2025 am 09:32 AM

Following its high-profile acquisition by Facebook in 2012, Instagram adopted two sets of APIs for third-party use. These are the Instagram Graph API and the Instagram Basic Display API.As a developer building an app that requires information from a

Working with Flash Session Data in LaravelWorking with Flash Session Data in LaravelMar 12, 2025 pm 05:08 PM

Laravel simplifies handling temporary session data using its intuitive flash methods. This is perfect for displaying brief messages, alerts, or notifications within your application. Data persists only for the subsequent request by default: $request-

Build a React App With a Laravel Back End: Part 2, ReactBuild a React App With a Laravel Back End: Part 2, ReactMar 04, 2025 am 09:33 AM

This is the second and final part of the series on building a React application with a Laravel back-end. In the first part of the series, we created a RESTful API using Laravel for a basic product-listing application. In this tutorial, we will be dev

Simplified HTTP Response Mocking in Laravel TestsSimplified HTTP Response Mocking in Laravel TestsMar 12, 2025 pm 05:09 PM

Laravel provides concise HTTP response simulation syntax, simplifying HTTP interaction testing. This approach significantly reduces code redundancy while making your test simulation more intuitive. The basic implementation provides a variety of response type shortcuts: use Illuminate\Support\Facades\Http; Http::fake([ 'google.com' => 'Hello World', 'github.com' => ['foo' => 'bar'], 'forge.laravel.com' =>

cURL in PHP: How to Use the PHP cURL Extension in REST APIscURL in PHP: How to Use the PHP cURL Extension in REST APIsMar 14, 2025 am 11:42 AM

The PHP Client URL (cURL) extension is a powerful tool for developers, enabling seamless interaction with remote servers and REST APIs. By leveraging libcurl, a well-respected multi-protocol file transfer library, PHP cURL facilitates efficient execution of various network protocols, including HTTP, HTTPS, and FTP. This extension offers granular control over HTTP requests, supports multiple concurrent operations, and provides built-in security features.

12 Best PHP Chat Scripts on CodeCanyon12 Best PHP Chat Scripts on CodeCanyonMar 13, 2025 pm 12:08 PM

Do you want to provide real-time, instant solutions to your customers' most pressing problems? Live chat lets you have real-time conversations with customers and resolve their problems instantly. It allows you to provide faster service to your custom

Announcement of 2025 PHP Situation SurveyAnnouncement of 2025 PHP Situation SurveyMar 03, 2025 pm 04:20 PM

The 2025 PHP Landscape Survey investigates current PHP development trends. It explores framework usage, deployment methods, and challenges, aiming to provide insights for developers and businesses. The survey anticipates growth in modern PHP versio

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

DVWA

DVWA

Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

PhpStorm Mac version

PhpStorm Mac version

The latest (2018.2.1) professional PHP integrated development tool

Safe Exam Browser

Safe Exam Browser

Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment