search
HomeBackend DevelopmentPython TutorialTips | Python crawler tool Selenium from entry to advanced


## Today I will talk about

selenium, we will talk about these contents in general

  • seleniumIntroduction and installation
  • Positioning of page elements
  • Browser Control
  • Mouse control
  • Keyboard control
  • Set element wait
  • Getcookies
  • CallJavaScript
  • seleniumAdvanced

Introduction and installation of selenium

selenium is the most widely used open source Web UI One of the automated test suites, the languages ​​it supports include C , Java, Perl, PHP, PythonRuby is also a powerful tool in data capture, which can solve the anti-crawling measures of most web pages. Of course, it is not omnipotent. One of the more obvious points is that it is relatively slow. , if the amount of data collected every day is not very high, you can use this framework.

So when it comes to installation, you can directly use pip to install

pip install selenium

At the same time, we also need to install a browser driver. Different browsers need to be installed. Different drivers, the editor here mainly recommends the following two

  • FirefoxBrowser driver: geckodriver
  • ##ChromeBrowser driver: chromedriver
The editor usually uses
selenium chromedriver, so here I will use Chrome Browser is an example. Since the version of chromedriver needs to be consistent with the version of the browser, let’s first confirm what the version of the browser is? Look at the picture below

Tips | Python crawler tool Selenium from entry to advanced

We find the browser version in "About Chrome" and then download the corresponding version
chromedriver, of course it must also correspond to your computer’s operating system

Tips | Python crawler tool Selenium from entry to advanced

Positioning of page elements

When talking about the positioning of page elements, the editor assumes that readers have the most basic front-end knowledge, such asHTML, CSS, etc.

ID标签的定位

HTML当中,ID属性是唯一标识一个元素的属性,因此在selenium当中,通过ID来进行元素的定位也作为首选,我们以百度首页为例,搜索框的HTML代码如下,其ID为“kw”,而“百度一下”这个按钮的ID为“su”,我们用Python脚本通过ID的标签来进行元素的定位
driver.find_element_by_id("kw")
driver.find_element_by_id("su")

NAME标签的定位

HTML当中,Name属性和ID属性的功能基本相同,只是Name属性并不是唯一的,如果遇到没有ID标签的时候,我们可以考虑通过Name标签来进行定位,代码如下
driver.find_element_by_name("wd")

Xpath定位

使用Xpath方式来定位几乎涵盖了页面上的任意元素,那什么是Xpath呢?Xpath是一种在XMLHTML文档中查找信息的语言,当然通过Xpath路径来定位元素的时候也是分绝对路径和相对路径。
绝对路径是以单号/来表示,相对路径是以//来表示,而涉及到Xpath路径的编写,小编这里偷个懒,直接选择复制/粘贴的方式,例如针对下面的HTML代码
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Test</title>
</head>
<body>
<form id="loginForm">
    <input name="username" type="text" />
    <input name="password" type="password" />
    <input name="continue" type="submit" value="Login" />
    <input name="continue" type="button" value="Clear" />
</form>

</body>
</html>
我们可以这么来做,打开浏览器的开发者工具,鼠标移到我们选中的元素,然后右击检查,具体看下图

Tips | Python crawler tool Selenium from entry to advanced

我们还是以百度首页为例,看一下如何通过Xpath来进行页面元素的定位,代码如下
driver.find_element_by_xpath(&#39;//*[@id="kw"]&#39;)

className标签定位

我们也可以基于class属性来定位元素,尤其是当我们看到有多个并列的元素如list表单,class用的都是共用同一个,如:
driver.find_element_by_class_name("classname")
这个时候,我们就可以通过class属性来定位元素,该方法返回的是一个list列表,而当我们想要定位列表当中的第n个元素时,则可以这样来安排
driver.find_elements_by_class_name("classname")[n]
需要注意的是,这里使用的是find_elements_by_class_name()方法而不是find_element_by_class_name()方法,这里我们还是通过百度首页的例子,通过className标签来定位搜索框这个元素
driver.find_element_by_class_name(&#39;s_ipt&#39;)

CssSelector()方法定位

其实在Selenium官网当中是更加推荐CssSelector()方法来进行页面元素的定位的,原因在于相比较于Xpath定位速度更快,Css定位分为四类:ID值、Class属性、TagName值等等,我们依次来看
  • ID方式来定位
大概有两种方式,一种是在ID值前面添加TagName的值,另外一种则是不加,代码如下
driver.find_element_by_css_selector("#id_value")  # 不添加前面的`TagName`值
driver.find_element_by_css_selector("tag_name.class_value")  # 不添加前面的`TagName`值
当然有时候这个TagName的值非常的冗长,中间可能还有空格,那么这当中的空格就需要用点“.”来替换
driver.find_element_by_css_selector("tag_name.class_value1.calss_value2.class_value3")  # 不添加前面的`TagName`值

我们仍然以百度首页的搜索框为例,它的HTML代码如下

Tips | Python crawler tool Selenium from entry to advanced

要是用CssSelector.class()方式来实现元素的定位的话,Python代码该这样来实现,和上面Xpath()的方法一样,可以稍微偷点懒,通过复制/粘贴的方式从开发者工具当中来获取元素的位置

Tips | Python crawler tool Selenium from entry to advanced

代码如下

driver.find_element_by_css_selector(&#39;#kw&#39;)

linkText()方式来定位

这个方法直接通过链接上面的文字来定位元素,案例如下

Tips | Python crawler tool Selenium from entry to advanced

通过linkText()方法来定位“地图”这个元素,代码如下

driver.find_element_by_link_text("地图").click()

浏览器的控制

修改浏览器窗口的大小

我们可以通过使用set_window_size()这个方法来修改浏览器窗口的大小,代码如下
# 修改浏览器的大小
driver.set_window_size(500, 900)

同时还有maxmize_window()方法是用来实现浏览器全屏显示,代码如下

# 全屏显示
driver.maximize_window()

浏览器的前进与后退

前进与后退用到的方法分别是forward()back(),代码如下

# 前进与后退
driver.forward()
driver.back()

浏览器的刷新

刷新用到的方法是refresh(),代码如下

# 刷新页面
driver.refresh()

除了上面这些,webdriver的常见操作还有

  • 关闭浏览器:get()
  • 清除文本:clear()
  • 单击元素:click()
  • 提交表单:submit()
  • 模拟输入内容:send_keys()

我们可以尝试着用上面提到的一些方法来写段程序

from selenium import webdriver
from time import sleep

driver = webdriver.Chrome(executable_path="chromedriver.exe")
driver.get("https://www.baidu.com")
sleep(3)
driver.maximize_window()
sleep(1)
driver.find_element_by_xpath(&#39;//*[@id="s-top-loginbtn"]&#39;).click()
sleep(3)
driver.find_element_by_xpath(&#39;//*[@id="TANGRAM__PSP_11__userName"]&#39;).send_keys(&#39;12121212&#39;)
sleep(1)
driver.find_element_by_xpath(&#39;//*[@id="TANGRAM__PSP_11__password"]&#39;).send_keys(&#39;testtest&#39;)
sleep(2)
driver.refresh()
sleep(3)
driver.quit()

output

Tips | Python crawler tool Selenium from entry to advanced

鼠标的控制

鼠标的控制都是封装在ActionChains类当中,常见的有以下几种

引入action_chains类
from selenium.webdriver.common.action_chains import ActionChains
# 右击
ActionChains(driver).context_click(element).perform()
# 双击
ActionChains(driver).double_click(element).perform()
# 拖放
ActionChains(driver).drag_and_drop(Start, End).perform()
# 悬停
ActionChains(driver).move_to_element(Above).perform()
# 按下
ActionChains(driver).click_and_hold(leftclick).perform()
# 执行指定的操作

键盘的控制

webdriver中的Keys()类,提供了几乎所有按键的方法,常用的如下

# 删除键
driver.find_element_by_id(&#39;xxx&#39;).send_keys(Keys.BACK_SPACE)
# 空格键
driver.find_element_by_id(&#39;xxx&#39;).send_keys(Keys.SPACE)
# 回车键
driver.find_element_by_id(&#39;xxx&#39;).send_keys(Keys.ENTER)
# Ctrl + A 全选内容
driver.find_element_by_id(&#39;xxx&#39;).send_keys(Keys.CONTROL, &#39;a&#39;)
# Ctrl + C/V 复制/粘贴内容
driver.find_element_by_id(&#39;xxx&#39;).send_keys(Keys.CONTROL, &#39;c&#39;)
driver.find_element_by_id(&#39;xxx&#39;).send_keys(Keys.CONTROL, &#39;v&#39;)

其他的一些键盘操作

  • 向上箭头:Keys.ARROW_UP
  • 向下箭头:Keys.ARROW_DOWN
  • 向左/向右箭头:Keys.ARROW_LEFT/Keys.ARROW_RIGHT
  • Shift键:Keys.SHIFT
  • F1键:Keys.F1

元素的等待

有显示等待和隐式等待两种

显示等待

显示等待指的是设置一个超时时间,每隔一段时间去查看一下该元素是否存在,如果存在则执行后面的内容,要是超过了最长的等待时间,则抛出异常(TimeoutException),需要用到的是WebDriverWait()方法,同时配合untilnot until方法
WebDriverWait(driver, timeout, poll_frequency=0.5, ignored_exceptions=None)

其中的参数:

  • timeout: 最长超时时间,默认以秒为单位
  • poll_frequency: 检测的时间间隔,默认是0.5s
  • ignored_exceptions: 指定忽略的异常,默认忽略的有NoSuchElementException这个异常

我们来看下面的案例

driver = webdriver.Chrome()
driver.get("http://somedomain/url_that_delays_loading")
try:    
    element = WebDriverWait(driver, 10).until(           
        EC.presence_of_element_located((By.ID, "myDynamicElement")))
finally:    
    driver.quit()
上面的代码最多等待10秒,超时后就抛出异常,但是假设在等了3秒之后就找到了这个元素,那么也就不会多等下剩下的7秒钟时间,而是继续执行后续的代码

隐式等待

主要使用的是implicitly_wait()来实现

browser = webdriver.Chrome(path)
# 隐式等待3秒
browser.implicitly_wait(3)

获取Cookie

Cookie是用来识别用户身份的关键,我们通常也是通过selenium先模拟登录网页获取Cookie,然后再通过requests携带Cookie来发送请求。

webdriver提供了cookies的几种操作,我们挑选几个常用的来说明

  • get_cookies():以字典的形式返回当前会话中可见的cookie信息
  • get_cookies(name): 返回cookie字典中指定的的cookie信息
  • add_cookie(cookie_dict): 将cookie添加到当前会话中

下面看一个简单的示例代码

driver=webdriver.Chrome(executable_path="chromedriver.exe")
driver.get(url=url)
time.sleep(1)

cookie_list=driver.get_cookies()
cookies =";".join([item["name"] +"=" + item["value"] + "" for item in cookie_list])
session=requests.session()

headers = {
    &#39;User-Agent&#39;:&#39;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36&#39;,
    &#39;cookie&#39;: cookies
}

response=session.get(url=url,headers=headers)
soup=BeautifulSoup(response.text,&#39;lxml&#39;)

调用JavaScript

webdriver当中可以使用execut_script()方法来实现JavaScript的执行,下面我们来看一个简单的例子
from selenium import webdriver
import time
bro=webdriver.Chrome(executable_path=&#39;./chromedriver&#39;)
bro.get("https://www.baidu.com")

# 执行js代码
bro.execute_script(&#39;alert(10)&#39;)
time.sleep(3)
bro.close()

除此之外,我们还可以通过selenium执行JavaScript来实现屏幕上下滚动

from selenium import webdriver
bro=webdriver.Chrome(executable_path=&#39;./chromedriver&#39;)
bro.get("https://www.baidu.com")
# 执行js代码
bro.execute_script(&#39;window.scrollTo(0,document.body.scrollHeight)&#39;)

selenium进阶

selenium启动的浏览器,会非常容易的被检测出来,通常可以通过window.navigator.webdriver的值来查看,如果是true则说明是使用了selenium模拟浏览器,如果是undefined则通常会被认为是正常的浏览器。
那么我们似乎可以执行下面这段代码来强行更改window.navigator.webdriver最后返回的值
driver.execute_script(
    &#39;Object.defineProperties(navigator,{webdriver:{get:()=>false}})&#39;
)
当然这种方法也有一定的缺陷,毕竟这段代码是在网页已经加载完毕之后才运行的,此时网页自身的JavaScript程序已经通过读取window.navigator.webdriver知道你使用的是模拟浏览器了。所以我们有两种办法来解决这个缺陷。
  • 在Chrome当中添加实验性功能参数

代码如下

from selenium.webdriver import Chrome
from selenium.webdriver import ChromeOptions

option = ChromeOptions()
option.add_experimental_option(&#39;excludeSwitches&#39;,[&#39;enable-automation&#39;])
driver=Chrome(options=option)
  • 调用chrome当中的开发工具协议的命令
核心思想就是让Chrome浏览器在打开页面,还没有运行网页自带的JavaScript代码时,先来执行我们给定的代码,通过execute_cdp_cmd()方法,
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
    "source": """
        Object.defineProperty(navigator, &#39;webdriver&#39;, {
            get: () => undefined
        })
    """
})

当然为了更好隐藏指纹特征,我们可以将上面两种方法想结合

from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option(&#39;useAutomationExtension&#39;, False)
driver = webdriver.Chrome(options=options, executable_path=&#39;./chromedriver&#39;)
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
  "source": """
    Object.defineProperty(navigator, &#39;webdriver&#39;, {
      get: () => undefined
    })
  """
})
driver.get(url)
最后的最后,我们也可以通过运行stealth.min.js文件来实现隐藏selenium模拟浏览器的特征,这个文件之前是给puppeteer用的,使得其隐藏浏览器的指纹特征,而让Python使用时,需要先导入这份JS文件
import time
from selenium.webdriver import Chrome

option = webdriver.ChromeOptions()
option.add_argument("--headless")

# 无头浏览器需要添加user-agent来隐藏特征
option.add_argument(&#39;user-agent=.....&#39;)
driver = Chrome(options=option)
driver.implicitly_wait(5)

with open(&#39;stealth.min.js&#39;) as f:
    js = f.read()

driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
  "source": js
})

driver.get(url)

The above is the detailed content of Tips | Python crawler tool Selenium from entry to advanced. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:Python当打之年. If there is any infringement, please contact admin@php.cn delete
Learning Python: Is 2 Hours of Daily Study Sufficient?Learning Python: Is 2 Hours of Daily Study Sufficient?Apr 18, 2025 am 12:22 AM

Is it enough to learn Python for two hours a day? It depends on your goals and learning methods. 1) Develop a clear learning plan, 2) Select appropriate learning resources and methods, 3) Practice and review and consolidate hands-on practice and review and consolidate, and you can gradually master the basic knowledge and advanced functions of Python during this period.

Python for Web Development: Key ApplicationsPython for Web Development: Key ApplicationsApr 18, 2025 am 12:20 AM

Key applications of Python in web development include the use of Django and Flask frameworks, API development, data analysis and visualization, machine learning and AI, and performance optimization. 1. Django and Flask framework: Django is suitable for rapid development of complex applications, and Flask is suitable for small or highly customized projects. 2. API development: Use Flask or DjangoRESTFramework to build RESTfulAPI. 3. Data analysis and visualization: Use Python to process data and display it through the web interface. 4. Machine Learning and AI: Python is used to build intelligent web applications. 5. Performance optimization: optimized through asynchronous programming, caching and code

Python vs. C  : Exploring Performance and EfficiencyPython vs. C : Exploring Performance and EfficiencyApr 18, 2025 am 12:20 AM

Python is better than C in development efficiency, but C is higher in execution performance. 1. Python's concise syntax and rich libraries improve development efficiency. 2.C's compilation-type characteristics and hardware control improve execution performance. When making a choice, you need to weigh the development speed and execution efficiency based on project needs.

Python in Action: Real-World ExamplesPython in Action: Real-World ExamplesApr 18, 2025 am 12:18 AM

Python's real-world applications include data analytics, web development, artificial intelligence and automation. 1) In data analysis, Python uses Pandas and Matplotlib to process and visualize data. 2) In web development, Django and Flask frameworks simplify the creation of web applications. 3) In the field of artificial intelligence, TensorFlow and PyTorch are used to build and train models. 4) In terms of automation, Python scripts can be used for tasks such as copying files.

Python's Main Uses: A Comprehensive OverviewPython's Main Uses: A Comprehensive OverviewApr 18, 2025 am 12:18 AM

Python is widely used in data science, web development and automation scripting fields. 1) In data science, Python simplifies data processing and analysis through libraries such as NumPy and Pandas. 2) In web development, the Django and Flask frameworks enable developers to quickly build applications. 3) In automated scripts, Python's simplicity and standard library make it ideal.

The Main Purpose of Python: Flexibility and Ease of UseThe Main Purpose of Python: Flexibility and Ease of UseApr 17, 2025 am 12:14 AM

Python's flexibility is reflected in multi-paradigm support and dynamic type systems, while ease of use comes from a simple syntax and rich standard library. 1. Flexibility: Supports object-oriented, functional and procedural programming, and dynamic type systems improve development efficiency. 2. Ease of use: The grammar is close to natural language, the standard library covers a wide range of functions, and simplifies the development process.

Python: The Power of Versatile ProgrammingPython: The Power of Versatile ProgrammingApr 17, 2025 am 12:09 AM

Python is highly favored for its simplicity and power, suitable for all needs from beginners to advanced developers. Its versatility is reflected in: 1) Easy to learn and use, simple syntax; 2) Rich libraries and frameworks, such as NumPy, Pandas, etc.; 3) Cross-platform support, which can be run on a variety of operating systems; 4) Suitable for scripting and automation tasks to improve work efficiency.

Learning Python in 2 Hours a Day: A Practical GuideLearning Python in 2 Hours a Day: A Practical GuideApr 17, 2025 am 12:05 AM

Yes, learn Python in two hours a day. 1. Develop a reasonable study plan, 2. Select the right learning resources, 3. Consolidate the knowledge learned through practice. These steps can help you master Python in a short time.

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
1 months agoBy尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. Best Graphic Settings
1 months agoBy尊渡假赌尊渡假赌尊渡假赌
Will R.E.P.O. Have Crossplay?
1 months agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

MinGW - Minimalist GNU for Windows

MinGW - Minimalist GNU for Windows

This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.

PhpStorm Mac version

PhpStorm Mac version

The latest (2018.2.1) professional PHP integrated development tool

DVWA

DVWA

Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

Safe Exam Browser

Safe Exam Browser

Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

VSCode Windows 64-bit Download

VSCode Windows 64-bit Download

A free and powerful IDE editor launched by Microsoft