PHP HTML JavaScript Css implements simple crawler development_php skills

To develop a crawler, first you need to know what your crawler is going to be used for. I want to use it to find articles with specific keywords on different websites and get their links so that I can read them quickly.

According to personal habits, I first need to write an interface and clarify my ideas.

1. Go to different websites. Then we need a url input box.

2. Find articles with specific keywords. Then we need an article title input box.

3. Get the article link. Then we need a display container for search results.

<div class="jumbotron" id="mainJumbotron">
 <div class="panel panel-default">
 
  <div class="panel-heading">文章URL抓取</div>
 
  <div class="panel-body">
   <div class="form-group">
    <label for="article_title">文章标题</label>
    <input type="text" class="form-control" id="article_title" placeholder="文章标题">
   </div>
   <div class="form-group">
    <label for="website_url">网站URL</label>
    <input type="text" class="form-control" id="website_url" placeholder="网站URL">
   </div>
 
   <button type="submit" class="btn btn-default">抓取</button>
  </div>
 </div>
 <div class="panel panel-default">
 
  <div class="panel-heading">文章URL</div>
 
  <div class="panel-body">
   <h3></h3>
  </div>
 </div>
</div>

Add the code directly, then add some style adjustments of your own, and the interface is complete:

Then the next step is to implement the function. I use PHP to write it. The first step is to obtain the html code of the website. There are many ways to obtain the html code. I will not introduce them one by one. Curl is used here. Get, pass in the website url and you will get the html code:

private function get_html($url){
 
 $ch = curl_init();
 
 $timeout = 10;
 
 curl_setopt($ch, CURLOPT_URL, $url);
 
 curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
 
 curl_setopt($ch, CURLOPT_ENCODING, 'gzip');
 
 curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36');
 
 curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
 
 $html = curl_exec($ch);
 
 return $html;
 
}

Although you get the html code, you will soon encounter a problem, that is, the encoding problem, which may make your next step of matching in vain. Here we uniformly convert the obtained html content into utf8 encoding:

$coding = mb_detect_encoding($html);
 
if ($coding != "UTF-8" || !mb_check_encoding($html, "UTF-8"))
 
 $html = mb_convert_encoding($html, 'utf-8', 'GBK,UTF-8,ASCII');

Get the html of the website and get the url of the article. Then the next step is to match all a tags under the web page. You need to use regular expressions. After many tests, you finally get a more reliable regular expression. , no matter how complex the structure under the a tag is, as long as it is a tag, it will not be missed: (the most critical step)

$pattern = '|<a[^>]*>(.*)</a>|isU';
 
preg_match_all($pattern, $html, $matches);

The matching result is in $matches, which is probably a multi-dimensional group like this:

array(2) { 
 [0]=> 
 array(*) { 
  [0]=>
  string(*) "完整的a标签"
  .
  .
  .
 }
 [1]=>
 array(*) {
  [0]=>
  string(*) "与上面下标相对应的a标签中的内容"
 }
}

As long as you can get this data, everything else is completely operable. You can traverse this element group, find the a tag you want, and then get the corresponding attributes of the a tag. You can do whatever you want. Here is a class recommended , making it easier for you to operate a tag:

$dom = new DOMDocument();
 
@$dom->loadHTML($a);//$a是上面得到的一些a标签
 
$url = new DOMXPath($dom);
 
$hrefs = $url->evaluate('//a');
 
for ($i = 0; $i < $hrefs->length; $i++) {
 
 $href = $hrefs->item($i);
 
 $url = $href->getAttribute('href'); //这里获取a标签的href属性
 
}

Of course, this is just one way. You can also use regular expressions to match the information you want and play new tricks with the data.

Get and match the results you want. The next step is of course to send them back to the front end to display them. Write the interface, then use js to get the data on the front end, and use jquery to dynamically add the content and display it:

var website_url = '你的接口地址';
$.getJSON(website_url,function(data){
 if(data){
  if(data.text == ''){
   $('#article_url').html('<div><p>暂无该文章链接</p></div>');
   return;
  }
  var string = '';
  var list = data.text;
  for (var j in list) {
    var content = list[j].url_content;
    for (var i in content) {
     if (content[i].title != '') {
      string += '<div class="item">' +
       '<em>[<a href="http://' + list[j].website.web_url + '" target="_blank">' + list[j].website.web_name + '</a>]</em>' +
       '<a href=" ' + content[i].url + '" target="_blank" class="web_url">' + content[i].title + '</a>' +
       '</div>';
     }
    }
   }
  $('#article_url').html(string);
});

The final rendering:

The above is the entire content of this article, I hope it will be helpful to everyone’s study.