Home > Article > Backend Development > How to use Python crawler to get those valuable blog posts

How to use Python crawler to get those valuable blog posts

不言Original: 2018-04-04 15:06:561950browse

The content of this article is how to use Python crawler to obtain those valuable blog posts. Now I share it with everyone. Friends in need can refer to the content of this article

Author CDA Data Analyst

There are many wonderful technical blog articles on CSDN, We can crawl it and save it on the local disk, which is very convenient for reading and learning later. Now we will use python to write a crawler code to achieve this purpose.

What we want to do: automatically read blog articles, record titles, and save favorite articles to personal Save it on your computer hard drive for future study reference.

The process is roughly divided into the following steps:

1. Find the crawled target URL;
2. Analyze the web page, Find the information you want to save. Here we mainly save the content of blog articles;
3. Clean and organize the crawled information and save it on the local disk. .

Open the csdn web page. As an example, we randomly open a web page:

http://blog.csdn.net/u013088062/article/list/1.

It can be seen that the blogger is very interested in "C++ Convolutional Neural Network" and other articles about computer science. Well written.

How to use Python crawler to get those valuable blog posts

The crawler code is divided into three categories (classes) according to the idea. The following three with "#" give the beginning of each class respectively (the specific code is attached for everyone to actually run and implement):

How to use Python crawler to get those valuable blog posts

##The "class" method belongs to Python's object-oriented programming, which is sometimes better than the process-oriented programming we usually use. Convenient, object-oriented programming is often used in large projects. For beginners, object-oriented programming is not easy to master, but after learning and getting used to it, they will gradually transition from process-oriented to object-oriented programming.

Special note is that the RePage class mainly uses regular expressions to process information obtained from web pages. Regular expressions Set the string style as follows:

How to use Python crawler to get those valuable blog posts

## Use regular expressions to match the content to be crawled, which can be achieved using Python and other software tools. There are many rules for regular expressions, and each software uses them in similar ways. Making good use of regular expressions is an important part of crawlers and text mining.

The SaveText class saves the information locally. The effect is as follows:

How to use Python crawler to get those valuable blog posts

## Use python to write crawler code, which is concise and efficient. This article only explains the most basic usage of the crawler. Interested friends can download the code and take a look. I hope you will gain something from it.

Attached is the relevant Python code:

1#-*-coding:UTF-8-*- 2import re 3import urllib2 4import sys 5#Purpose : Read blog articles, record titles, and save article content in Htnl format 6# Version: python2.7.13 7#Function: Read web page content 8class GetHtmlPage(): 9 #Note the case 10 def __init__ (self,strPage): ##11 ##13 = URLLIB2.Request (Self.StRPAPGE) # Create page request 11 15 #Ate "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36 SE 2.X MetaSr 1.0" )16 to urllib2. ## # Web page encoding 20 #except urllib2.URLError, e: #Catch exception 23 code 24 26 print 'HTTP Error:' + e.reason 27 ##28 return rePage29#Regular expression, get the desired content30class RePage(): 31# Regular expression extracts content and returns linked list 32 def GetReText(self,page,recode): ##33 rePage = re.findall(recode,page,re.S) 34 return rePage 35#Save text 36class SaveText(): 37 def Save(self,text,tilte): 38 try: 39 t="blog\\"+tilte+".html" 40 because 42 45 if __name__ == "__main__":46 s = SaveText() 47 #File encoding 48 #Character decoding correctly 49 reload(sys) 50 sys.setdefaultencoding( "utf-8" ) #Get the default encoding of the system 51 #Get the web page 52 page = GetHtmlPage("http://blog.csdn.net/ u013088062/article/list/1") 53 htmlPage = page.GetPage() 54 #Extract content 55 reServer = RePage() 56 reBlog = reServer.GetReText(htmlPage,r'.*?(\s.+?)') #Get the URL link and title 57 #Go down to get the text 58 for ref in reBlog: 59 pageHeard = "http://blog.csdn.net/" #Add link header 60 strPage = pageHeard+ref[0] 61 tilte=ref[1 ].replace('[Top]', "") #Use the replacement function to remove complicated English 62 tilte=tilte.replace("\r\n","").lstrip().rstrip() 63 #Get the text 64 htmlPage = GetHtmlPage(strPage) 65 htmlPageData = htmlPage.GetPage() 66 reBlogText = reServer.GetReText(htmlPageData,'

##(.+?)

') 67 #Save file 68 for s1 in reBlogText : 69 s1='\n'+s1 70 s.Save(s1,tilte)

Related recommendations:

Development case of php implementing a simple crawler

Python crawler browser identification library

Record a simple Python crawler instance

The above is the detailed content of How to use Python crawler to get those valuable blog posts. For more information, please follow other related articles on the PHP Chinese website!

Python php 正则表达式 chrome safari html print for 面向对象 try Error 字符串 .net class 对象 windows 数据分析 http

Statement：

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Previous article：How to implement visual heat map in pythonNext article：How to implement visual heat map in python

See more

How to use Python crawler to get those valuable blog posts

Related articles