How to use Python crawler to get those valuable blog posts-Python Tutorial-php.cn

Home

Backend Development

Python Tutorial

How to use Python crawler to get those valuable blog posts

不言

Apr 04, 2018 pm 03:06 PM

pythonObtain

The content of this article is how to use Python crawler to obtain those valuable blog posts. Now I share it with everyone. Friends in need can refer to the content of this article

How to use Python crawler to get those valuable blog posts

Author CDA Data Analyst

There are many wonderful technical blog articles on CSDN, We can crawl it and save it on the local disk, which is very convenient for reading and learning later. Now we will use python to write a crawler code to achieve this purpose.

What we want to do: automatically read blog articles, record titles, and save favorite articles to personal Save it on your computer hard drive for future study reference.

The process is roughly divided into the following steps:

1. Find the crawled target URL;
2. Analyze the web page, Find the information you want to save. Here we mainly save the content of blog articles;
3. Clean and organize the crawled information and save it on the local disk. .

Open the csdn web page. As an example, we randomly open a web page:

http://blog.csdn.net/u013088062/article/list/1.

It can be seen that the blogger is very interested in "C++ Convolutional Neural Network" and other articles about computer science. Well written.

How to use Python crawler to get those valuable blog posts

The crawler code is divided into three categories (classes) according to the idea. The following three with "#" give the beginning of each class respectively (the specific code is attached for everyone to actually run and implement):

How to use Python crawler to get those valuable blog posts

##The "class" method belongs to Python's object-oriented programming, which is sometimes better than the process-oriented programming we usually use. Convenient, object-oriented programming is often used in large projects. For beginners, object-oriented programming is not easy to master, but after learning and getting used to it, they will gradually transition from process-oriented to object-oriented programming.

Special note is that the RePage class mainly uses regular expressions to process information obtained from web pages. Regular expressions Set the string style as follows:

How to use Python crawler to get those valuable blog posts

## Use regular expressions to match the content to be crawled, which can be achieved using Python and other software tools. There are many rules for regular expressions, and each software uses them in similar ways. Making good use of regular expressions is an important part of crawlers and text mining.

The SaveText class saves the information locally. The effect is as follows:

How to use Python crawler to get those valuable blog posts

## Use python to write crawler code, which is concise and efficient. This article only explains the most basic usage of the crawler. Interested friends can download the code and take a look. I hope you will gain something from it.

Attached is the relevant Python code:

1#-*-coding:UTF-8-*- 2import re 3import urllib2 4import sys 5#Purpose : Read blog articles, record titles, and save article content in Htnl format 6# Version: python2.7.13 7#Function: Read web page content 8class GetHtmlPage(): 9 #Note the case 10 def __init__ (self,strPage): ##11 ##13 = URLLIB2.Request (Self.StRPAPGE) # Create page request 11 15 #Ate "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36 SE 2.X MetaSr 1.0" )16 to urllib2. ## # Web page encoding 20 #except urllib2.URLError, e: #Catch exception 23 code 24 26 print 'HTTP Error:' + e.reason 27 ##28 return rePage29#Regular expression, get the desired content30class RePage(): 31# Regular expression extracts content and returns linked list 32 def GetReText(self,page,recode): ##33 rePage = re.findall(recode,page,re.S) 34 return rePage 35#Save text 36class SaveText(): 37 def Save(self,text,tilte): 38 try: 39 t="blog\\"+tilte+".html" 40 because 42 45 if __name__ == "__main__":46 s = SaveText() 47 #File encoding 48 #Character decoding correctly 49 reload(sys) 50 sys.setdefaultencoding( "utf-8" ) #Get the default encoding of the system 51 #Get the web page 52 page = GetHtmlPage("http://blog.csdn.net/ u013088062/article/list/1") 53 htmlPage = page.GetPage() 54 #Extract content 55 reServer = RePage() 56 reBlog = reServer.GetReText(htmlPage,r'.*?(\s.+?)') #Get the URL link and title 57 #Go down to get the text 58 for ref in reBlog: 59 pageHeard = "http://blog.csdn.net/" #Add link header 60 strPage = pageHeard+ref[0] 61 tilte=ref[1 ].replace('[Top]', "") #Use the replacement function to remove complicated English 62 tilte=tilte.replace("\r\n","").lstrip().rstrip() 63 #Get the text 64 htmlPage = GetHtmlPage(strPage) 65 htmlPageData = htmlPage.GetPage() 66 reBlogText = reServer.GetReText(htmlPageData,'

##(.+?)

') 67 #Save file 68 for s1 in reBlogText : 69 s1='\n'+s1 70 s.Save(s1,tilte)

Related recommendations:

Development case of php implementing a simple crawler

Python crawler browser identification library

Record a simple Python crawler instance

The above is the detailed content of How to use Python crawler to get those valuable blog posts. For more information, please follow other related articles on the PHP Chinese website!

Statement

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Merging Lists in Python: Choosing the Right MethodMay 14, 2025 am 12:11 AM

TomergelistsinPython,youcanusethe operator,extendmethod,listcomprehension,oritertools.chain,eachwithspecificadvantages:1)The operatorissimplebutlessefficientforlargelists;2)extendismemory-efficientbutmodifiestheoriginallist;3)listcomprehensionoffersf

How to concatenate two lists in python 3?May 14, 2025 am 12:09 AM

In Python 3, two lists can be connected through a variety of methods: 1) Use operator, which is suitable for small lists, but is inefficient for large lists; 2) Use extend method, which is suitable for large lists, with high memory efficiency, but will modify the original list; 3) Use * operator, which is suitable for merging multiple lists, without modifying the original list; 4) Use itertools.chain, which is suitable for large data sets, with high memory efficiency.

Python concatenate list stringsMay 14, 2025 am 12:08 AM

Using the join() method is the most efficient way to connect strings from lists in Python. 1) Use the join() method to be efficient and easy to read. 2) The cycle uses operators inefficiently for large lists. 3) The combination of list comprehension and join() is suitable for scenarios that require conversion. 4) The reduce() method is suitable for other types of reductions, but is inefficient for string concatenation. The complete sentence ends.

Python execution, what is that?May 14, 2025 am 12:06 AM

PythonexecutionistheprocessoftransformingPythoncodeintoexecutableinstructions.1)Theinterpreterreadsthecode,convertingitintobytecode,whichthePythonVirtualMachine(PVM)executes.2)TheGlobalInterpreterLock(GIL)managesthreadexecution,potentiallylimitingmul

Python: what are the key featuresMay 14, 2025 am 12:02 AM

Key features of Python include: 1. The syntax is concise and easy to understand, suitable for beginners; 2. Dynamic type system, improving development speed; 3. Rich standard library, supporting multiple tasks; 4. Strong community and ecosystem, providing extensive support; 5. Interpretation, suitable for scripting and rapid prototyping; 6. Multi-paradigm support, suitable for various programming styles.

Python: compiler or Interpreter?May 13, 2025 am 12:10 AM

Python is an interpreted language, but it also includes the compilation process. 1) Python code is first compiled into bytecode. 2) Bytecode is interpreted and executed by Python virtual machine. 3) This hybrid mechanism makes Python both flexible and efficient, but not as fast as a fully compiled language.

Python For Loop vs While Loop: When to Use Which?May 13, 2025 am 12:07 AM

Useaforloopwheniteratingoverasequenceorforaspecificnumberoftimes;useawhileloopwhencontinuinguntilaconditionismet.Forloopsareidealforknownsequences,whilewhileloopssuitsituationswithundeterminediterations.

Python loops: The most common errorsMay 13, 2025 am 12:07 AM

Pythonloopscanleadtoerrorslikeinfiniteloops,modifyinglistsduringiteration,off-by-oneerrors,zero-indexingissues,andnestedloopinefficiencies.Toavoidthese:1)Use'i

See all articles