Web scraping: efficiently extracting data from web pages
This tutorial explains how web page content is rendered and how to scrape it using Python, requests, Beautiful Soup, and Selenium. We'll focus on scraping dynamic content, specifically comments.
When is web scraping necessary?
Web scraping automatically retrieves, parses, and extracts information from web pages—often those designed for human interaction. It's a last resort when APIs aren't available. Consider these drawbacks:
- Fragility: Web pages change frequently, breaking scrapers.
- Restrictions: Many websites prohibit scraping.
- Performance: Scraping large volumes of data can be slow and expensive.
Understanding modern web pages
Let's examine the structure of typical web applications. We'll use the "Introduction to Vagrant" article as an example. To scrape content, we must first locate the relevant HTML elements.
Viewing page source
Browsers allow viewing the HTML source. The "Introduction to Vagrant" source reveals a significant portion of minified JavaScript unrelated to the article content itself. A small excerpt is shown below:
Here's a sample of the actual HTML:
Static vs. dynamic scraping
Static scraping ignores JavaScript, fetching the raw server-side HTML. This works if the target content is directly in the source. However, for content generated by JavaScript (dynamic content), this method fails. Dynamic scraping uses tools like Selenium to interact with the page as a browser would, rendering the JavaScript and making the dynamic content available.
Scraping dynamic comments with Selenium
Let's scrape comments from a website (example: Codecanyon comments). We'll use Selenium to navigate to the comments URL:
from selenium import webdriver driver = webdriver.Chrome() driver.get('https://codecanyon.net/item/whatshelp-whatsapp-help-and-support-plugin-for-javascript/42202303/comments')
Locating the comments element requires inspecting the page (right-click, "Inspect"). Selenium's WebDriverWait
helps handle asynchronous loading:
from selenium.webdriver.common.by import By from selenium.webdriver.support.wait import WebDriverWait from selenium.webdriver.support import expected_conditions as EC driver = webdriver.Chrome() url = 'http://www.c2.com/loading-page' driver.get(url) element = WebDriverWait(driver, 5).until( EC.presence_of_element_located((By.ID, "loaded_element")) )
Conclusion
Web scraping is valuable when necessary data isn't readily available via APIs. While challenging with modern web applications, tools like requests, Beautiful Soup, and Selenium simplify the process.
This tutorial incorporates contributions from Esther Vaati, a software developer and writer for Envato Tuts .
The above is the detailed content of Modern Web Scraping With Beautiful Soup and Selenium. For more information, please follow other related articles on the PHP Chinese website!

Create multi-dimensional arrays with NumPy can be achieved through the following steps: 1) Use the numpy.array() function to create an array, such as np.array([[1,2,3],[4,5,6]]) to create a 2D array; 2) Use np.zeros(), np.ones(), np.random.random() and other functions to create an array filled with specific values; 3) Understand the shape and size properties of the array to ensure that the length of the sub-array is consistent and avoid errors; 4) Use the np.reshape() function to change the shape of the array; 5) Pay attention to memory usage to ensure that the code is clear and efficient.

BroadcastinginNumPyisamethodtoperformoperationsonarraysofdifferentshapesbyautomaticallyaligningthem.Itsimplifiescode,enhancesreadability,andboostsperformance.Here'showitworks:1)Smallerarraysarepaddedwithonestomatchdimensions.2)Compatibledimensionsare

ForPythondatastorage,chooselistsforflexibilitywithmixeddatatypes,array.arrayformemory-efficienthomogeneousnumericaldata,andNumPyarraysforadvancednumericalcomputing.Listsareversatilebutlessefficientforlargenumericaldatasets;array.arrayoffersamiddlegro

Pythonlistsarebetterthanarraysformanagingdiversedatatypes.1)Listscanholdelementsofdifferenttypes,2)theyaredynamic,allowingeasyadditionsandremovals,3)theyofferintuitiveoperationslikeslicing,but4)theyarelessmemory-efficientandslowerforlargedatasets.

ToaccesselementsinaPythonarray,useindexing:my_array[2]accessesthethirdelement,returning3.Pythonuseszero-basedindexing.1)Usepositiveandnegativeindexing:my_list[0]forthefirstelement,my_list[-1]forthelast.2)Useslicingforarange:my_list[1:5]extractselemen

Article discusses impossibility of tuple comprehension in Python due to syntax ambiguity. Alternatives like using tuple() with generator expressions are suggested for creating tuples efficiently.(159 characters)

The article explains modules and packages in Python, their differences, and usage. Modules are single files, while packages are directories with an __init__.py file, organizing related modules hierarchically.

Article discusses docstrings in Python, their usage, and benefits. Main issue: importance of docstrings for code documentation and accessibility.


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Atom editor mac version download
The most popular open source editor

mPDF
mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

Dreamweaver Mac version
Visual web development tools

SublimeText3 Linux new version
SublimeText3 Linux latest version

Dreamweaver CS6
Visual web development tools
