A summary of the python standard library Beautiful Soup and MongoDb climbing Himalayan Radio-Python Tutorial-php.cn

Home

Backend Development

Python Tutorial

A summary of the python standard library Beautiful Soup and MongoDb climbing Himalayan Radio

WBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWB

Jul 27, 2017 pm 02:46 PM

mongodbpython

Beautiful Soup standard library is a Python library that can extract data from HTML/XML files. It can realize the usual way of document navigation, search and modification through your favorite converter. Beautiful Soup will save hours. working hours. The pymongo standard library is the bridge between the MongoDb NoSql database and the python language. Data is saved to MongoDb through pymongo. Use the two together to crawl the Himalayan radio data...

Beautiful Soup supports the HTML parser in the Python standard library, and also supports some third-party parsers, one of which is lxml. This article uses lxml. For the installation of this, please see the python 3.6 lxml standard library lxml installation and the use of etree. Note
At the same time, this article uses XPath to parse the part we want. For the introduction of XPath and Beautiful Soup For more information and usage, please see Beautiful Soup 4.4.0 Document XPath Introduction
The knowledge of Beautiful Soup and XPath involved in this article is not very deep. You can understand it by looking at the official documentation, and I also added comments...
Regarding the pymongo standard library, I will not talk too much. For details, please see the pymongo module experience of the python standard library

Sometimes, we need to determine the type of client currently making a request to the server, which is commonly known as User-Agent, referred to as UA. The browser we use when browsing the web is a type of UA. In other words, UA is the browser. In the HTTP protocol, the User-Agent request header indicates the type of the user's browser and the operation Identification of system, browser kernel and other information. Through this logo, different versions of the websites visited can be displayed to provide users with a better experience or to conduct information statistics. Some websites officially use UA to prevent hackers or boring people like us from crawling the website's data information.
Therefore, the code in this article first lists all UAs to facilitate subsequent crawling work.

Okay, let’s clarify what data we want to crawl and obtain:

What we need is the link to the image, alt, etc.

Then we click on the image link to get the details inside. If some radio stations have multiple pages, then we use xpath to access them one by one. At the same time, we obtain the sound_id of the sound module in the album on the page...

The procedure is as follows:

import random
import requests
from bs4 import BeautifulSoup
import json
from lxml import etree
import pymongo


clients = pymongo.MongoClient("localhost", 27017)
db = clients["XiMaLaYa"]
collection_1 = db["album"]
collection_2 = db["detail"]

UA_LIST = [
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
    "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
    "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
    "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SE 2.X MetaSr 1.0; SE 2.X MetaSr 1.0; .NET CLR 2.0.50727; SE 2.X MetaSr 1.0)",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
    "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
    "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
    "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
    "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
    "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
]
headers1 = {
    &#39;Accept&#39;: &#39;text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8&#39;,
    &#39;Accept-Encoding&#39;: &#39;gzip, deflate, sdch&#39;,
    &#39;Accept-Language&#39;: &#39;zh-CN,zh;q=0.8,en;q=0.6&#39;,
    &#39;Cache-Control&#39;: &#39;max-age=0&#39;,
    &#39;Proxy-Connection&#39;: &#39;keep-alive&#39;,
    &#39;Upgrade-Insecure-Requests&#39;: &#39;1&#39;,
    &#39;User-Agent&#39;: random.choice(UA_LIST)  # User_agence表示用户代理
}
headers2 = {
    &#39;Accept&#39;: &#39;text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8&#39;,
    &#39;Accept-Encoding&#39;: &#39;gzip, deflate, sdch&#39;,
    &#39;Accept-Language&#39;: &#39;zh-CN,zh;q=0.8,en;q=0.6&#39;,
    &#39;Cache-Control&#39;: &#39;max-age=0&#39;,
    &#39;Proxy-Connection&#39;: &#39;keep-alive&#39;,
    &#39;Referer&#39;: &#39;http://www.ximalaya.com/dq/all/2&#39;,
    &#39;Upgrade-Insecure-Requests&#39;: &#39;1&#39;,
    &#39;User-Agent&#39;: random.choice(UA_LIST)
}


# Beautiful库用来处理XML和HTML...
# 主要就是利用BeautifulSoup模块来处理requests模块获取的Html源码
# 利用lxml模块将html源码解析成树结构，xpath来处理树节点.
def get_url():
    start_urls = ["http://www.ximalaya.com/dq/all/{}".format(num) for num in range(1，85)]
    # start_urls = ["http://www.ximalaya.com/dq/all/1"]
    for start_url in start_urls:
        html = requests.get(start_url, headers=headers1).text
        soup = BeautifulSoup(html, "lxml")  # 使用lxml来处理
        for item in soup.find_all(class_="albumfaceOutter"):  # 解析并查找xml节点
            content = {
                &#39;href&#39;: item.a["href"],
                &#39;title&#39;: item.img[&#39;alt&#39;],
                &#39;img_url&#39;: item.img[&#39;src&#39;]
            }
            collection_1.insert(content)
            # another(item.a["href"])
    print(&#39;写入完成...&#39;)


# 进入电台具体页面 http://www.ximalaya.com/15836959/album/303085，并处理分页录音...
def another(url):
    html = requests.get(url, headers=headers1).text
    # / :表示从根节点选取....
    # // :表示匹配选择的当前节点选择文档中的节点，而不考虑他们的位置...
    ifanother = etree.HTML(html).xpath(&#39;//div[@class="pagingBar_wrapper"]/a[last()-1]/@data-page&#39;)  # 页面链接地址  ifanother是list类型...
    if len(ifanother):  # 判断一个video的录音是否分割成了多页....
        num = ifanother[0]  # 获取页面数...
        print(&#39;本频道保存在&#39; + num + &#39;个页面&#39;)
        for n in range(1, int(num)):
            url2 = url + &#39;?page={}&#39;.format(n)
            get_m4a(url2)
        get_m4a(url)


# 获取分页录音页面的详细数据...
def get_m4a(url):
    html = requests.get(url, headers=headers2).text
    numlist = etree.HTML(html).xpath(&#39;//div[@class="personal_body"]/@sound_ids&#39;)[0].split(&#39;,&#39;)
    for i in numlist:
        murl = &#39;http://www.ximalaya.com/tracks/{}.json&#39;.format(i)
        html = requests.get(murl, headers=headers1).text
        dic = json.loads(html)
        collection_2.insert(dic)


if __name__ == "__main__":
    get_url()

The above is the detailed content of A summary of the python standard library Beautiful Soup and MongoDb climbing Himalayan Radio. For more information, please follow other related articles on the PHP Chinese website!

Statement

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Python vs. C : Learning Curves and Ease of UseApr 19, 2025 am 12:20 AM

Python is easier to learn and use, while C is more powerful but complex. 1. Python syntax is concise and suitable for beginners. Dynamic typing and automatic memory management make it easy to use, but may cause runtime errors. 2.C provides low-level control and advanced features, suitable for high-performance applications, but has a high learning threshold and requires manual memory and type safety management.

Python vs. C : Memory Management and ControlApr 19, 2025 am 12:17 AM

Python and C have significant differences in memory management and control. 1. Python uses automatic memory management, based on reference counting and garbage collection, simplifying the work of programmers. 2.C requires manual management of memory, providing more control but increasing complexity and error risk. Which language to choose should be based on project requirements and team technology stack.

Python for Scientific Computing: A Detailed LookApr 19, 2025 am 12:15 AM

Python's applications in scientific computing include data analysis, machine learning, numerical simulation and visualization. 1.Numpy provides efficient multi-dimensional arrays and mathematical functions. 2. SciPy extends Numpy functionality and provides optimization and linear algebra tools. 3. Pandas is used for data processing and analysis. 4.Matplotlib is used to generate various graphs and visual results.

Python and C : Finding the Right ToolApr 19, 2025 am 12:04 AM

Whether to choose Python or C depends on project requirements: 1) Python is suitable for rapid development, data science, and scripting because of its concise syntax and rich libraries; 2) C is suitable for scenarios that require high performance and underlying control, such as system programming and game development, because of its compilation and manual memory management.

Python for Data Science and Machine LearningApr 19, 2025 am 12:02 AM

Python is widely used in data science and machine learning, mainly relying on its simplicity and a powerful library ecosystem. 1) Pandas is used for data processing and analysis, 2) Numpy provides efficient numerical calculations, and 3) Scikit-learn is used for machine learning model construction and optimization, these libraries make Python an ideal tool for data science and machine learning.

Learning Python: Is 2 Hours of Daily Study Sufficient?Apr 18, 2025 am 12:22 AM

Is it enough to learn Python for two hours a day? It depends on your goals and learning methods. 1) Develop a clear learning plan, 2) Select appropriate learning resources and methods, 3) Practice and review and consolidate hands-on practice and review and consolidate, and you can gradually master the basic knowledge and advanced functions of Python during this period.

Python for Web Development: Key ApplicationsApr 18, 2025 am 12:20 AM

Key applications of Python in web development include the use of Django and Flask frameworks, API development, data analysis and visualization, machine learning and AI, and performance optimization. 1. Django and Flask framework: Django is suitable for rapid development of complex applications, and Flask is suitable for small or highly customized projects. 2. API development: Use Flask or DjangoRESTFramework to build RESTfulAPI. 3. Data analysis and visualization: Use Python to process data and display it through the web interface. 4. Machine Learning and AI: Python is used to build intelligent web applications. 5. Performance optimization: optimized through asynchronous programming, caching and code

Python vs. C : Exploring Performance and EfficiencyApr 18, 2025 am 12:20 AM

Python is better than C in development efficiency, but C is higher in execution performance. 1. Python's concise syntax and rich libraries improve development efficiency. 2.C's compilation-type characteristics and hardware control improve execution performance. When making a choice, you need to weigh the development speed and execution efficiency based on project needs.

See all articles

Hot AI Tools

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress images for free

Clothoff.io

AI clothes remover

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Assassin's Creed Shadows: Seashell Riddle Solution

3 weeks agoByDDD

What's New in Windows 11 KB5054979 & How to Fix Update Issues

2 weeks agoByDDD

Where to find the Crane Control Keycard in Atomfall

3 weeks agoByDDD

Assassin's Creed Shadows - How To Find The Blacksmith And Unlock Weapon And Armour Customisation

4 weeks agoByDDD

Roblox: Dead Rails - How To Complete Every Challenge

3 weeks agoByDDD

Hot Tools

DVWA

Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

Zend Studio 13.0.1

Powerful PHP integrated development environment

SublimeText3 Mac version

God-level code editing software (SublimeText3)

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

PhpStorm Mac version

The latest (2018.2.1) professional PHP integrated development tool

Hot Topics

Where is the login entrance for gmail email?

7581

CakePHP Tutorial

1386

What is the format of the account name of steam

win11 activation key permanent

nyt connections hints and answers

117