首次接触爬虫，这个程序可以让初学者明白爬虫是啥，爬虫的基本逻辑。（参照网易云课堂heibanke老师的程序编写的。hebanke老师的课程讲得很清楚，感兴趣的选手可以去看看）
被爬虫的网址是http://www.heibanke.com/lesson/crawler_ex00/需要完成的任务是在网址后面输入显示的数字转入下一个网址，再次读取显示的数字，转入下一个网址，以此循环
爬虫要求
在爬虫的时候需要需要用到两个库，一个urllib，一个BeautifulSoup,urllib为python自带的库，beautifulsoup需要自己安装
pip install beautifulsoup4
安装成功后导入
import urllibfrom bs4 import BeautifulSoup
urllib提供了一系列用于操作URL的功能打开要爬取得网站：
urllib.urlopen(url)
beautifulsoup主要用于解析HTML，将 HTML转化成数型结构，每个节点都是Python对象。所有对象可以归纳为4种:1.Tag2.NavigableString3.BeautifulSoup4.Comment如果一个HTML为
<html> <[head])> <title> The Dormouse's story </title>
print soup.title>>><title>The Dormouse's story</title>
print soup.title.string>>>The Dormouse's story
程序逻辑：读出页面上的数字，将数字加在url后面，继续读取新页面的数字，直到进入最后一个没有数字的页面，break程序如下：
import urllibfrom bs4 import BeautifulSoupimport re#这个库导入是为了使用正则表达式读取读取找到的内容中的数字url='http://www.heibanke.com/lesson/crawler_ex00/'number=['']#用于储存读到的数字while True: content = urllib.urlopen(url+number[0])#number为字符串，number[0]为数字 bs_obj = BeautifulSoup(content,"html.parser")#html.parser表示解析网站，不返回任何值 number = bs_obj.h3.string#网页显示出的“你需要在网址后输入数字44513”在html的h3 tag中，number在这里读出了h3里面的内容 number= re.findall(r'\d+',number)#读出了number里面的数字 if not number:#必须判断页面中还有是否还有number，没有说明已经到了最后一个页面，这时应该跳出循环，打印 bs_obj.h3.string break else: print number[0]print bs_obj.h3.string
What is the purpose of the <datalist> element?Mar 21, 2025 pm 12:33 PMThe article discusses the HTML <datalist> element, which enhances forms by providing autocomplete suggestions, improving user experience and reducing errors.Character count: 159
How do I use HTML5 form validation attributes to validate user input?Mar 17, 2025 pm 12:27 PMThe article discusses using HTML5 form validation attributes like required, pattern, min, max, and length limits to validate user input directly in the browser.
What is the purpose of the <iframe> tag? What are the security considerations when using it?Mar 20, 2025 pm 06:05 PMThe article discusses the <iframe> tag's purpose in embedding external content into webpages, its common uses, security risks, and alternatives like object tags and APIs.
What is the purpose of the <progress> element?Mar 21, 2025 pm 12:34 PMThe article discusses the HTML <progress> element, its purpose, styling, and differences from the <meter> element. The main focus is on using <progress> for task completion and <meter> for stati
What is the purpose of the <meter> element?Mar 21, 2025 pm 12:35 PMThe article discusses the HTML <meter> element, used for displaying scalar or fractional values within a range, and its common applications in web development. It differentiates <meter> from <progress> and ex
What are the best practices for cross-browser compatibility in HTML5?Mar 17, 2025 pm 12:20 PMArticle discusses best practices for ensuring HTML5 cross-browser compatibility, focusing on feature detection, progressive enhancement, and testing methods.
What is the viewport meta tag? Why is it important for responsive design?Mar 20, 2025 pm 05:56 PMThe article discusses the viewport meta tag, essential for responsive web design on mobile devices. It explains how proper use ensures optimal content scaling and user interaction, while misuse can lead to design and accessibility issues.
How do I use the HTML5 <time> element to represent dates and times semantically?Mar 12, 2025 pm 04:05 PMThis article explains the HTML5 <time> element for semantic date/time representation. It emphasizes the importance of the datetime attribute for machine readability (ISO 8601 format) alongside human-readable text, boosting accessibilit
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
SublimeText3 Chinese version
Chinese version, very easy to use
Dreamweaver Mac version
Visual web development tools
WebStorm Mac version
Useful JavaScript development tools
Notepad++7.3.1
Easy-to-use and free code editor
SecLists
SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.
