Home > Article > Backend Development > Example of how Python3 captures html web pages dynamically generated by JS
This article mainly introduces the function of Python3 to capture HTML webpages dynamically generated by JavaScript, and analyzes the relevant operating skills of Python3 using the selenium library to capture HTML webpage elements dynamically generated by JavaScript, combined with examples. Friends who need it can Refer to the following
The example of this article describes the function of Python3 to capture html web pages dynamically generated by javascript. Share it with everyone for your reference, the details are as follows:
Using urllib to crawl web pages can only read the static source files of the web page, but cannot capture the content generated by javascript.
The reason is that urllib crawls instantly and does not wait for the loading delay of javascript, so the content generated by javascript in the page cannot be read by urllib.
Is it really impossible to read the content generated by javascript? No!
Here we will introduce a python library: selenium. The version used in this article is 2.44.0
Install first:
pip install -U selenium
The following three examples are used to illustrate its usage:
【Example 0】
Open a Firefox browser
Load the page with the given url address
from selenium import webdriver browser = webdriver.Firefox() browser.get('http://www.baidu.com/')
【Example 1 】
Open a Firefox browser
Load Baidu homepage
Search for "seleniumhq"
Close the browser
from selenium import webdriver from selenium.webdriver.common.keys import Keys browser = webdriver.Firefox() browser.get('http://www.baidu.com') assert '百度' in browser.title elem = browser.find_element_by_name('p') # Find the search box elem.send_keys('seleniumhq' + Keys.RETURN) # 模拟按键 browser.quit()
[Example 2]
Selenium WebDriver is often used for testing network programs. Here is an example using the Python standard library unittest:
import unittest class BaiduTestCase(unittest.TestCase): def setUp(self): self.browser = webdriver.Firefox() self.addCleanup(self.browser.quit) def testPageTitle(self): self.browser.get('http://www.baidu.com') self.assertIn('百度', self.browser.title) if __name__ == '__main__': unittest.main(verbosity=2)
The above is the detailed content of Example of how Python3 captures html web pages dynamically generated by JS. For more information, please follow other related articles on the PHP Chinese website!