


Python crawling method of Anjuke second-hand housing website data sharing
This article mainly brings you a python crawling of Anjuke second-hand housing website data (explanation with examples). The editor thinks it’s pretty good, so I’ll share it with you now and give it as a reference. Let’s follow the editor to take a look, I hope it can help everyone.
Now we will start to officially write the crawler. First, we need to analyze the structure of the website to be crawled: As a student in Henan, let’s take a look at the second-hand housing information in Zhengzhou!
In the above page, we can see the property information one by one. From the above, we can see the property information one by one on the web page. After clicking in, you will find:
Details of the property. OK! So what are we going to do? That is to get all the second-hand housing information in Zhengzhou and save it in the database. What is it used for? As a geographer, it is still somewhat useful. I won’t go into it this time. Okay, let’s officially start. First, I use the requests and BeautifulSoup modules in python3.6 to crawl the page. First, the requests module makes the request:
# 网页的请求头 header = { 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36' } # url链接 url = 'https://zhengzhou.anjuke.com/sale/' response = requests.get(url, headers=header) print(response.text)
After execution, You will get the html code of this website
Through analysis, you can get that each house is in the li tag of class="list-item", then we can proceed based on the BeautifulSoup package Extracting
# 通过BeautifulSoup进行解析出每个房源详细列表并进行打印 soup = BeautifulSoup(response.text, 'html.parser') result_li = soup.find_all('li', {'class': 'list-item'}) for i in result_li: print(i)
can further reduce the amount of code by printing. OK, continue to extract
# 通过BeautifulSoup进行解析出每个房源详细列表并进行打印 soup = BeautifulSoup(response.text, 'html.parser') result_li = soup.find_all('li', {'class': 'list-item'}) # 进行循环遍历其中的房源详细列表 for i in result_li: # 由于BeautifulSoup传入的必须为字符串,所以进行转换 page_url = str(i) soup = BeautifulSoup(page_url, 'html.parser') # 由于通过class解析的为一个列表,所以只需要第一个参数 result_href = soup.find_all('a', {'class': 'houseListTitle'})[0] print(result_href.attrs['href'])
. In this way, we You can see the URLs one by one. Do you like it?
Okay, according to normal logic, you have to enter the page and start analyzing the detailed page, but how to proceed to the next page after crawling? So, we need to first analyze whether the page has a next page
The same method can be used to find that the next page is also so simple, then we You can continue with the original recipe and original taste
# 进行下一页的爬取 result_next_page = soup.find_all('a', {'class': 'aNxt'}) if len(result_next_page) != 0: print(result_next_page[0].attrs['href']) else: print('没有下一页了')
Because when the next page exists, there is an a tag in the web page. If not, it will become i tag, so this will do. Therefore, we can improve it and encapsulate the above into a function
import requests from bs4 import BeautifulSoup # 网页的请求头 header = { 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36' } def get_page(url): response = requests.get(url, headers=header) # 通过BeautifulSoup进行解析出每个房源详细列表并进行打印 soup = BeautifulSoup(response.text, 'html.parser') result_li = soup.find_all('li', {'class': 'list-item'}) # 进行下一页的爬取 result_next_page = soup.find_all('a', {'class': 'aNxt'}) if len(result_next_page) != 0: # 函数进行递归 get_page(result_next_page[0].attrs['href']) else: print('没有下一页了') # 进行循环遍历其中的房源详细列表 for i in result_li: # 由于BeautifulSoup传入的必须为字符串,所以进行转换 page_url = str(i) soup = BeautifulSoup(page_url, 'html.parser') # 由于通过class解析的为一个列表,所以只需要第一个参数 result_href = soup.find_all('a', {'class': 'houseListTitle'})[0] # 先不做分析,等一会进行详细页面函数完成后进行调用 print(result_href.attrs['href']) if __name__ == '__main__': # url链接 url = 'https://zhengzhou.anjuke.com/sale/' # 页面爬取函数调用 get_page(url)
Okay, then let’s start the detailed page Crawled
Hey, the power is always cut off, what a trap in the university, I will attach the results first, I will add more when I have free time,
import requests from bs4 import BeautifulSoup # 网页的请求头 header = { 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36' } def get_page(url): response = requests.get(url, headers=header) # 通过BeautifulSoup进行解析出每个房源详细列表并进行打印 soup_idex = BeautifulSoup(response.text, 'html.parser') result_li = soup_idex.find_all('li', {'class': 'list-item'}) # 进行循环遍历其中的房源详细列表 for i in result_li: # 由于BeautifulSoup传入的必须为字符串,所以进行转换 page_url = str(i) soup = BeautifulSoup(page_url, 'html.parser') # 由于通过class解析的为一个列表,所以只需要第一个参数 result_href = soup.find_all('a', {'class': 'houseListTitle'})[0] # 详细页面的函数调用 get_page_detail(result_href.attrs['href']) # 进行下一页的爬取 result_next_page = soup_idex.find_all('a', {'class': 'aNxt'}) if len(result_next_page) != 0: # 函数进行递归 get_page(result_next_page[0].attrs['href']) else: print('没有下一页了') # 进行字符串中空格,换行,tab键的替换及删除字符串两边的空格删除 def my_strip(s): return str(s).replace(" ", "").replace("\n", "").replace("\t", "").strip() # 由于频繁进行BeautifulSoup的使用,封装一下,很鸡肋 def my_Beautifulsoup(response): return BeautifulSoup(str(response), 'html.parser') # 详细页面的爬取 def get_page_detail(url): response = requests.get(url, headers=header) if response.status_code == 200: soup = BeautifulSoup(response.text, 'html.parser') # 标题什么的一大堆,哈哈 result_title = soup.find_all('h3', {'class': 'long-title'})[0] result_price = soup.find_all('span', {'class': 'light info-tag'})[0] result_house_1 = soup.find_all('p', {'class': 'first-col detail-col'}) result_house_2 = soup.find_all('p', {'class': 'second-col detail-col'}) result_house_3 = soup.find_all('p', {'class': 'third-col detail-col'}) soup_1 = my_Beautifulsoup(result_house_1) soup_2 = my_Beautifulsoup(result_house_2) soup_3 = my_Beautifulsoup(result_house_3) result_house_tar_1 = soup_1.find_all('dd') result_house_tar_2 = soup_2.find_all('dd') result_house_tar_3 = soup_3.find_all('dd') ''' 文博公寓,省实验中学,首付只需70万,大三房,诚心卖,价可谈 270万 宇泰文博公寓 金水-花园路-文博东路4号 2010年 普通住宅 3室2厅2卫 140平方米 南北 中层(共32层) 精装修 19285元/m² 81.00万 ''' print(my_strip(result_title.text), my_strip(result_price.text)) print(my_strip(result_house_tar_1[0].text), my_strip(my_Beautifulsoup(result_house_tar_1[1]).find_all('p')[0].text), my_strip(result_house_tar_1[2].text), my_strip(result_house_tar_1[3].text)) print(my_strip(result_house_tar_2[0].text), my_strip(result_house_tar_2[1].text), my_strip(result_house_tar_2[2].text), my_strip(result_house_tar_2[3].text)) print(my_strip(result_house_tar_3[0].text), my_strip(result_house_tar_3[1].text), my_strip(result_house_tar_3[2].text)) if __name__ == '__main__': # url链接 url = 'https://zhengzhou.anjuke.com/sale/' # 页面爬取函数调用 get_page(url)
Since I wrote the code while blogging, I made some changes in the get_page function, that is, the recursive call for the next page needs to be placed after the function, and the two functions are encapsulated without introduction,
And the data is not written to mysql, so I will continue to follow up later, thank you!!!
Related recommendations:
python crawling article example tutorial
10 recommended articles about python crawling
Share a Python method to crawl popular comments on NetEase Cloud Music
The above is the detailed content of Python crawling method of Anjuke second-hand housing website data sharing. For more information, please follow other related articles on the PHP Chinese website!

Python and C each have their own advantages, and the choice should be based on project requirements. 1) Python is suitable for rapid development and data processing due to its concise syntax and dynamic typing. 2)C is suitable for high performance and system programming due to its static typing and manual memory management.

Choosing Python or C depends on project requirements: 1) If you need rapid development, data processing and prototype design, choose Python; 2) If you need high performance, low latency and close hardware control, choose C.

By investing 2 hours of Python learning every day, you can effectively improve your programming skills. 1. Learn new knowledge: read documents or watch tutorials. 2. Practice: Write code and complete exercises. 3. Review: Consolidate the content you have learned. 4. Project practice: Apply what you have learned in actual projects. Such a structured learning plan can help you systematically master Python and achieve career goals.

Methods to learn Python efficiently within two hours include: 1. Review the basic knowledge and ensure that you are familiar with Python installation and basic syntax; 2. Understand the core concepts of Python, such as variables, lists, functions, etc.; 3. Master basic and advanced usage by using examples; 4. Learn common errors and debugging techniques; 5. Apply performance optimization and best practices, such as using list comprehensions and following the PEP8 style guide.

Python is suitable for beginners and data science, and C is suitable for system programming and game development. 1. Python is simple and easy to use, suitable for data science and web development. 2.C provides high performance and control, suitable for game development and system programming. The choice should be based on project needs and personal interests.

Python is more suitable for data science and rapid development, while C is more suitable for high performance and system programming. 1. Python syntax is concise and easy to learn, suitable for data processing and scientific computing. 2.C has complex syntax but excellent performance and is often used in game development and system programming.

It is feasible to invest two hours a day to learn Python. 1. Learn new knowledge: Learn new concepts in one hour, such as lists and dictionaries. 2. Practice and exercises: Use one hour to perform programming exercises, such as writing small programs. Through reasonable planning and perseverance, you can master the core concepts of Python in a short time.

Python is easier to learn and use, while C is more powerful but complex. 1. Python syntax is concise and suitable for beginners. Dynamic typing and automatic memory management make it easy to use, but may cause runtime errors. 2.C provides low-level control and advanced features, suitable for high-performance applications, but has a high learning threshold and requires manual memory and type safety management.


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Zend Studio 13.0.1
Powerful PHP integrated development environment

SublimeText3 Chinese version
Chinese version, very easy to use

MinGW - Minimalist GNU for Windows
This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.

PhpStorm Mac version
The latest (2018.2.1) professional PHP integrated development tool

SublimeText3 Mac version
God-level code editing software (SublimeText3)