


The content this article brings to you is about Python crawling Douban movie data and extracting value xpath and lxml modules (code). It has certain reference value. Friends in need can refer to it. I hope it will be useful to you. help.
Tools: Python 3.6.5, PyCharm development tools, Windows 10 operating system, Google Chrome
Purpose: crawl the title, link address of the movie in the Douban movie rankings, Pictures, number of reviewers, ratings, etc.
Website: https://movie.douban.com/chart
Grammar points:
xpath syntax:
Google Chrome installs the xpath helper plug-in: Help us locate data from elements
1. Select the node (label)
(1),/html/ head/meta: Can select all meta tags under html
(2), //li: All li tags on the current page
(3), /html/head//link: All link tags under head
##2, //: Can be selected from any node
(1)、//li:All li tags on the current page
## (2)、/html/head//link:head All link tags under3. The purpose of the @ symbol
(1) Select a specific element: //p[ @class='feed']/ul/li, select li under ul under p of
class='feed'
(2), a/@href: Select the href value of a
4. Get the text## ( 1), /a/text(): Get the text under a
(2), /a//text(): Get all the text under a Text
Example
:
##lxml syntax:
1. Installation: pip install lxml
2. Use
from lxml import etree
## element = etree.HTML("html string ")
element.xpath("")
Code:
from lxml import etree import requests url = "https://movie.douban.com/chart" headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 Safari/537.36" } response = requests.get(url,headers=headers) html_str = response.content.decode() #print(html_str) html = etree.HTML(html_str) print(html) #1.获取所有的电影的URL地址 #url_list = html.xpath("//div[@class='indent']/div/table//div[@class='pl2']/a/@href") #print(url_list) #2.所有图片的地址 #img_list = html.xpath("//div[@class='indent']/div/table//a[@class='nbg']/img/@src") #print(img_list) ret1 = html.xpath("//div[@class='indent']/div/table") print(ret1) for table in ret1: item = {} item["title"] = table.xpath(".//div[@class='pl2']/a/text()")[0].replace("/","").strip() item["href"] = table.xpath(".//div[@class='pl2']/a/@href")[0] item["img"] = table.xpath(".//a[@class='nbg']/img/@src")[0] item["comment_num"] = table.xpath(".//span[@class='pl']/text()")[0] item["rating_num"] = table.xpath(".//span[@class='rating_nums']/text()")[0] print(item)
Running effect:
The above is the detailed content of Python crawls Douban movie data and extracts value xpath and lxml modules (code). For more information, please follow other related articles on the PHP Chinese website!

本篇文章给大家带来了关于Python的相关知识,其中主要介绍了关于Seaborn的相关问题,包括了数据可视化处理的散点图、折线图、条形图等等内容,下面一起来看一下,希望对大家有帮助。

本篇文章给大家带来了关于Python的相关知识,其中主要介绍了关于进程池与进程锁的相关问题,包括进程池的创建模块,进程池函数等等内容,下面一起来看一下,希望对大家有帮助。

本篇文章给大家带来了关于Python的相关知识,其中主要介绍了关于简历筛选的相关问题,包括了定义 ReadDoc 类用以读取 word 文件以及定义 search_word 函数用以筛选的相关内容,下面一起来看一下,希望对大家有帮助。

VS Code的确是一款非常热门、有强大用户基础的一款开发工具。本文给大家介绍一下10款高效、好用的插件,能够让原本单薄的VS Code如虎添翼,开发效率顿时提升到一个新的阶段。

pythn的中文意思是巨蟒、蟒蛇。1989年圣诞节期间,Guido van Rossum在家闲的没事干,为了跟朋友庆祝圣诞节,决定发明一种全新的脚本语言。他很喜欢一个肥皂剧叫Monty Python,所以便把这门语言叫做python。

本篇文章给大家带来了关于Python的相关知识,其中主要介绍了关于数据类型之字符串、数字的相关问题,下面一起来看一下,希望对大家有帮助。

本篇文章给大家带来了关于Python的相关知识,其中主要介绍了关于numpy模块的相关问题,Numpy是Numerical Python extensions的缩写,字面意思是Python数值计算扩展,下面一起来看一下,希望对大家有帮助。


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

SublimeText3 Linux new version
SublimeText3 Linux latest version

Notepad++7.3.1
Easy-to-use and free code editor

Atom editor mac version download
The most popular open source editor

WebStorm Mac version
Useful JavaScript development tools

ZendStudio 13.5.1 Mac
Powerful PHP integrated development environment
