


This article mainly brings you a python crawling of Anjuke second-hand housing website data (explanation with examples). The editor thinks it’s pretty good, so I’ll share it with you now and give it as a reference. Let’s follow the editor to take a look, I hope it can help everyone.
Now we will start to officially write the crawler. First, we need to analyze the structure of the website to be crawled: As a student in Henan, let’s take a look at the second-hand housing information in Zhengzhou!
In the above page, we can see the property information one by one. From the above, we can see the property information one by one on the web page. After clicking in, you will find:
Details of the property. OK! So what are we going to do? That is to get all the second-hand housing information in Zhengzhou and save it in the database. What is it used for? As a geographer, it is still somewhat useful. I won’t go into it this time. Okay, let’s officially start. First, I use the requests and BeautifulSoup modules in python3.6 to crawl the page. First, the requests module makes the request:
# 网页的请求头 header = { 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36' } # url链接 url = 'https://zhengzhou.anjuke.com/sale/' response = requests.get(url, headers=header) print(response.text)
After execution, You will get the html code of this website
Through analysis, you can get that each house is in the li tag of class="list-item", then we can proceed based on the BeautifulSoup package Extracting
# 通过BeautifulSoup进行解析出每个房源详细列表并进行打印 soup = BeautifulSoup(response.text, 'html.parser') result_li = soup.find_all('li', {'class': 'list-item'}) for i in result_li: print(i)
can further reduce the amount of code by printing. OK, continue to extract
# 通过BeautifulSoup进行解析出每个房源详细列表并进行打印 soup = BeautifulSoup(response.text, 'html.parser') result_li = soup.find_all('li', {'class': 'list-item'}) # 进行循环遍历其中的房源详细列表 for i in result_li: # 由于BeautifulSoup传入的必须为字符串,所以进行转换 page_url = str(i) soup = BeautifulSoup(page_url, 'html.parser') # 由于通过class解析的为一个列表,所以只需要第一个参数 result_href = soup.find_all('a', {'class': 'houseListTitle'})[0] print(result_href.attrs['href'])
. In this way, we You can see the URLs one by one. Do you like it?
Okay, according to normal logic, you have to enter the page and start analyzing the detailed page, but how to proceed to the next page after crawling? So, we need to first analyze whether the page has a next page
The same method can be used to find that the next page is also so simple, then we You can continue with the original recipe and original taste
# 进行下一页的爬取 result_next_page = soup.find_all('a', {'class': 'aNxt'}) if len(result_next_page) != 0: print(result_next_page[0].attrs['href']) else: print('没有下一页了')
Because when the next page exists, there is an a tag in the web page. If not, it will become i tag, so this will do. Therefore, we can improve it and encapsulate the above into a function
import requests from bs4 import BeautifulSoup # 网页的请求头 header = { 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36' } def get_page(url): response = requests.get(url, headers=header) # 通过BeautifulSoup进行解析出每个房源详细列表并进行打印 soup = BeautifulSoup(response.text, 'html.parser') result_li = soup.find_all('li', {'class': 'list-item'}) # 进行下一页的爬取 result_next_page = soup.find_all('a', {'class': 'aNxt'}) if len(result_next_page) != 0: # 函数进行递归 get_page(result_next_page[0].attrs['href']) else: print('没有下一页了') # 进行循环遍历其中的房源详细列表 for i in result_li: # 由于BeautifulSoup传入的必须为字符串,所以进行转换 page_url = str(i) soup = BeautifulSoup(page_url, 'html.parser') # 由于通过class解析的为一个列表,所以只需要第一个参数 result_href = soup.find_all('a', {'class': 'houseListTitle'})[0] # 先不做分析,等一会进行详细页面函数完成后进行调用 print(result_href.attrs['href']) if __name__ == '__main__': # url链接 url = 'https://zhengzhou.anjuke.com/sale/' # 页面爬取函数调用 get_page(url)
Okay, then let’s start the detailed page Crawled
Hey, the power is always cut off, what a trap in the university, I will attach the results first, I will add more when I have free time,
import requests from bs4 import BeautifulSoup # 网页的请求头 header = { 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36' } def get_page(url): response = requests.get(url, headers=header) # 通过BeautifulSoup进行解析出每个房源详细列表并进行打印 soup_idex = BeautifulSoup(response.text, 'html.parser') result_li = soup_idex.find_all('li', {'class': 'list-item'}) # 进行循环遍历其中的房源详细列表 for i in result_li: # 由于BeautifulSoup传入的必须为字符串,所以进行转换 page_url = str(i) soup = BeautifulSoup(page_url, 'html.parser') # 由于通过class解析的为一个列表,所以只需要第一个参数 result_href = soup.find_all('a', {'class': 'houseListTitle'})[0] # 详细页面的函数调用 get_page_detail(result_href.attrs['href']) # 进行下一页的爬取 result_next_page = soup_idex.find_all('a', {'class': 'aNxt'}) if len(result_next_page) != 0: # 函数进行递归 get_page(result_next_page[0].attrs['href']) else: print('没有下一页了') # 进行字符串中空格,换行,tab键的替换及删除字符串两边的空格删除 def my_strip(s): return str(s).replace(" ", "").replace("\n", "").replace("\t", "").strip() # 由于频繁进行BeautifulSoup的使用,封装一下,很鸡肋 def my_Beautifulsoup(response): return BeautifulSoup(str(response), 'html.parser') # 详细页面的爬取 def get_page_detail(url): response = requests.get(url, headers=header) if response.status_code == 200: soup = BeautifulSoup(response.text, 'html.parser') # 标题什么的一大堆,哈哈 result_title = soup.find_all('h3', {'class': 'long-title'})[0] result_price = soup.find_all('span', {'class': 'light info-tag'})[0] result_house_1 = soup.find_all('p', {'class': 'first-col detail-col'}) result_house_2 = soup.find_all('p', {'class': 'second-col detail-col'}) result_house_3 = soup.find_all('p', {'class': 'third-col detail-col'}) soup_1 = my_Beautifulsoup(result_house_1) soup_2 = my_Beautifulsoup(result_house_2) soup_3 = my_Beautifulsoup(result_house_3) result_house_tar_1 = soup_1.find_all('dd') result_house_tar_2 = soup_2.find_all('dd') result_house_tar_3 = soup_3.find_all('dd') ''' 文博公寓,省实验中学,首付只需70万,大三房,诚心卖,价可谈 270万 宇泰文博公寓 金水-花园路-文博东路4号 2010年 普通住宅 3室2厅2卫 140平方米 南北 中层(共32层) 精装修 19285元/m² 81.00万 ''' print(my_strip(result_title.text), my_strip(result_price.text)) print(my_strip(result_house_tar_1[0].text), my_strip(my_Beautifulsoup(result_house_tar_1[1]).find_all('p')[0].text), my_strip(result_house_tar_1[2].text), my_strip(result_house_tar_1[3].text)) print(my_strip(result_house_tar_2[0].text), my_strip(result_house_tar_2[1].text), my_strip(result_house_tar_2[2].text), my_strip(result_house_tar_2[3].text)) print(my_strip(result_house_tar_3[0].text), my_strip(result_house_tar_3[1].text), my_strip(result_house_tar_3[2].text)) if __name__ == '__main__': # url链接 url = 'https://zhengzhou.anjuke.com/sale/' # 页面爬取函数调用 get_page(url)
Since I wrote the code while blogging, I made some changes in the get_page function, that is, the recursive call for the next page needs to be placed after the function, and the two functions are encapsulated without introduction,
And the data is not written to mysql, so I will continue to follow up later, thank you!!!
Related recommendations:
python crawling article example tutorial
10 recommended articles about python crawling
Share a Python method to crawl popular comments on NetEase Cloud Music
The above is the detailed content of Python crawling method of Anjuke second-hand housing website data sharing. For more information, please follow other related articles on the PHP Chinese website!

本篇文章给大家带来了关于Python的相关知识,其中主要介绍了关于Seaborn的相关问题,包括了数据可视化处理的散点图、折线图、条形图等等内容,下面一起来看一下,希望对大家有帮助。

本篇文章给大家带来了关于Python的相关知识,其中主要介绍了关于进程池与进程锁的相关问题,包括进程池的创建模块,进程池函数等等内容,下面一起来看一下,希望对大家有帮助。

本篇文章给大家带来了关于Python的相关知识,其中主要介绍了关于简历筛选的相关问题,包括了定义 ReadDoc 类用以读取 word 文件以及定义 search_word 函数用以筛选的相关内容,下面一起来看一下,希望对大家有帮助。

VS Code的确是一款非常热门、有强大用户基础的一款开发工具。本文给大家介绍一下10款高效、好用的插件,能够让原本单薄的VS Code如虎添翼,开发效率顿时提升到一个新的阶段。

本篇文章给大家带来了关于Python的相关知识,其中主要介绍了关于数据类型之字符串、数字的相关问题,下面一起来看一下,希望对大家有帮助。

本篇文章给大家带来了关于Python的相关知识,其中主要介绍了关于numpy模块的相关问题,Numpy是Numerical Python extensions的缩写,字面意思是Python数值计算扩展,下面一起来看一下,希望对大家有帮助。

pythn的中文意思是巨蟒、蟒蛇。1989年圣诞节期间,Guido van Rossum在家闲的没事干,为了跟朋友庆祝圣诞节,决定发明一种全新的脚本语言。他很喜欢一个肥皂剧叫Monty Python,所以便把这门语言叫做python。


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

SAP NetWeaver Server Adapter for Eclipse
Integrate Eclipse with SAP NetWeaver application server.

MinGW - Minimalist GNU for Windows
This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.

VSCode Windows 64-bit Download
A free and powerful IDE editor launched by Microsoft

MantisBT
Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.

mPDF
mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),
