這篇文章主要介紹了python爬蟲之xpath的基本使用詳解,現在分享給大家,也給大家做個參考。一起來看看吧
一、簡介
#XPath 是一種在 XML 文件中尋找資訊的語言。 XPath 可用於在 XML 文件中對元素和屬性進行遍歷。 XPath 是 W3C XSLT 標準的主要元素,而 XQuery 和 XPointer 都建構在 XPath 表達之上。
二、安裝
pip3 install lxml
三、使用
1、導入
from lxml import etree
2、基本使用
from lxml import etree wb_data = """ <p> <ul> <li class="item-0"><a href="link1.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" >first item</a></li> <li class="item-1"><a href="link2.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >second item</a></li> <li class="item-inactive"><a href="link3.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" >third item</a></li> <li class="item-1"><a href="link4.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" >fourth item</a></li> <li class="item-0"><a href="link5.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" >fifth item</a> </ul> </p> """ html = etree.HTML(wb_data) print(html) result = etree.tostring(html) print(result.decode("utf-8"))
從下面的結果來看,我們印表機html其實就是一個python對象,etree.tostring(html)則是不全裡html的基本寫法,補全了缺胳膊少腿的標籤。
<Element html at 0x39e58f0> <html><body><p> <ul> <li class="item-0"><a href="link1.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" >first item</a></li> <li class="item-1"><a href="link2.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >second item</a></li> <li class="item-inactive"><a href="link3.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" >third item</a></li> <li class="item-1"><a href="link4.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" >fourth item</a></li> <li class="item-0"><a href="link5.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" >fifth item</a> </li></ul> </p> </body></html>
3、取得某個標籤的內容(基本使用),注意,取得a標籤的所有內容,a後面就不用再加正斜杠,否則報錯。
寫法一
html = etree.HTML(wb_data) html_data = html.xpath('/html/body/p/ul/li/a') print(html) for i in html_data: print(i.text) <Element html at 0x12fe4b8> first item second item third item fourth item fifth item
#寫法二(直接在需要尋找內容的標籤後面加上一個/text()就行)
html = etree.HTML(wb_data) html_data = html.xpath('/html/body/p/ul/li/a/text()') print(html) for i in html_data: print(i) <Element html at 0x138e4b8> first item second item third item fourth item fifth item
4、開啟讀取html檔案
#使用parse打开html的文件 html = etree.parse('test.html') html_data = html.xpath('//*')<br>#打印是一个列表,需要遍历 print(html_data) for i in html_data: print(i.text)
html = etree.parse('test.html') html_data = etree.tostring(html,pretty_print=True) res = html_data.decode('utf-8') print(res) 打印: <p> <ul> <li class="item-0"><a href="link1.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" >first item</a></li> <li class="item-1"><a href="link2.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >second item</a></li> <li class="item-inactive"><a href="link3.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" >third item</a></li> <li class="item-1"><a href="link4.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" >fourth item</a></li> <li class="item-0"><a href="link5.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" >fifth item</a></li> </ul> </p>
5、列印指定路徑下a標籤的屬性(可以透過遍歷拿到某個屬性的值,找出標籤的內容)
html = etree.HTML(wb_data) html_data = html.xpath('/html/body/p/ul/li/a/@href') for i in html_data: print(i)
列印:
link1.html
link2.html
link3.html
link4.html
link5.html
6、我們知道我們使用xpath拿到得都是一個個的ElementTree對象,所以如果需要找內容的話,還需要遍歷拿到資料的列表。
查到絕對路徑下a標籤屬性等於link2.html的內容。
html = etree.HTML(wb_data) html_data = html.xpath('/html/body/p/ul/li/a[@href="link2.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" ]/text()') print(html_data) for i in html_data: print(i)
列印:
#['second item']
second item
#7、上面我們找到全部都是絕對路徑(每一個都是從根開始查找),下面我們查找相對路徑,例如,查找所有li標籤下的a標籤內容。
html = etree.HTML(wb_data) html_data = html.xpath('//li/a/text()') print(html_data) for i in html_data: print(i)
印刷:
['first item', 'second item', 'third item', 'fourth item' , 'fifth item']
first item
second item
third item
fourth item
fifth item
8、上面我們使用絕對路徑,查找了所有a標籤的屬性等於href屬性值,利用的是/---絕對路徑,下面我們使用相對路徑,查找l相對路徑下li標籤下的a標籤下的href屬性的值,注意,a標籤後面需要雙//。
html = etree.HTML(wb_data) html_data = html.xpath('//li/a//@href') print(html_data) for i in html_data: print(i)
列印:
#['link1.html', 'link2.html', 'link3.html', ' link4.html', 'link5.html']
link1.html
link2.html
link3.html
link4.html
#link5.html
9、相對路徑下跟絕對路徑下查特定屬性的方法類似,也可以說相同。
html = etree.HTML(wb_data) html_data = html.xpath('//li/a[@href="link2.html" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" ]') print(html_data) for i in html_data: print(i.text)
列印:
[b723b8de943b6b7dda3734075f9b2d7e]
#second item
10、找出最後一個li標籤裡的a標籤的href屬性
#html = etree.HTML(wb_data) html_data = html.xpath('//li[last()]/a/text()') print(html_data) for i in html_data: print(i)
列印:
['fifth item']
fifth item
11、找出倒數第二個li標籤裡的a標籤的href屬性
#html = etree.HTML(wb_data) html_data = html.xpath('//li[last()-1]/a/text()') print(html_data) for i in html_data: print(i)
列印:
['fourth item']
fourth item
12、如果在擷取某個頁面的某個標籤的xpath路徑的話,可以如下圖:
//*[@id="kw"]
解釋:使用相對路徑來尋找所有的標籤,屬性id等於kw的標籤。
常用
#!/usr/bin/env python # -*- coding:utf-8 -*- from scrapy.selector import Selector, HtmlXPathSelector from scrapy.http import HtmlResponse html = """<!DOCTYPE html> <html> <head lang="en"> <meta charset="UTF-8"> <title></title> </head> <body> <ul> <li class="item-"><a id='i1' href="link.html" rel="external nofollow" rel="external nofollow" >first item</a></li> <li class="item-0"><a id='i2' href="llink.html" rel="external nofollow" >first item</a></li> <li class="item-1"><a href="llink2.html" rel="external nofollow" rel="external nofollow" >second item<span>vv</span></a></li> </ul> <p><a href="llink2.html" rel="external nofollow" rel="external nofollow" >second item</a></p> </body> </html> """ response = HtmlResponse(url='http://example.com', body=html,encoding='utf-8') # hxs = HtmlXPathSelector(response) # print(hxs) # hxs = Selector(response=response).xpath('//a') # print(hxs) # hxs = Selector(response=response).xpath('//a[2]') # print(hxs) # hxs = Selector(response=response).xpath('//a[@id]') # print(hxs) # hxs = Selector(response=response).xpath('//a[@id="i1"]') # print(hxs) # hxs = Selector(response=response).xpath('//a[@href="link.html" rel="external nofollow" rel="external nofollow" ][@id="i1"]') # print(hxs) # hxs = Selector(response=response).xpath('//a[contains(@href, "link")]') # print(hxs) # hxs = Selector(response=response).xpath('//a[starts-with(@href, "link")]') # print(hxs) # hxs = Selector(response=response).xpath('//a[re:test(@id, "i\d+")]') # print(hxs) # hxs = Selector(response=response).xpath('//a[re:test(@id, "i\d+")]/text()').extract() # print(hxs) # hxs = Selector(response=response).xpath('//a[re:test(@id, "i\d+")]/@href').extract() # print(hxs) # hxs = Selector(response=response).xpath('/html/body/ul/li/a/@href').extract() # print(hxs) # hxs = Selector(response=response).xpath('//body/ul/li/a/@href').extract_first() # print(hxs) # ul_list = Selector(response=response).xpath('//body/ul/li') # for item in ul_list: # v = item.xpath('./a/span') # # 或 # # v = item.xpath('a/span') # # 或 # # v = item.xpath('*/a/span') # print(v)
相關推薦:
以上是python爬蟲之xpath的基本使用詳解的詳細內容。更多資訊請關注PHP中文網其他相關文章!