Python data collection--Usage of Beautifulsoup

Python网络数据采集1-Beautifulsoup的使用

来自此书: [美]Ryan Mitchell 《Python网络数据采集》，例子是照搬的，觉得跟着敲一遍还是有作用的，所以记录下来。

import requestsfrom bs4 import BeautifulSoup

res = requests.get('https://www.pythonscraping.com/pages/page1.html')
soup = BeautifulSoup(res.text, 'lxml')print(soup.h1)

<h1 id="An-Interesting-Title">An Interesting Title</h1>

使用urllib访问页面是这样的，read返回的是字节，需要解码为utf-8的文本。像这样a.read().decode('utf-8')，不过在使用bs4解析时候，可以直接传入urllib库返回的响应对象。

import urllib.request

a = urllib.request.urlopen('https://www.pythonscraping.com/pages/page1.html')
soup = BeautifulSoup(a, 'lxml')print(soup.h1)

<h1 id="An-Interesting-Title">An Interesting Title</h1>

抓取所有CSS class属性为green的span标签，这些是人名。

import requestsfrom bs4 import BeautifulSoup

res = requests.get('https://www.pythonscraping.com/pages/warandpeace.html')

soup = BeautifulSoup(res.text, 'lxml')
green_names = soup.find_all('span', class_='green')for name in green_names:print(name.string)

Anna
Pavlovna Scherer
Empress Marya
Fedorovna
Prince Vasili Kuragin
Anna Pavlovna
St. Petersburg
the prince
Anna Pavlovna
Anna Pavlovna
...

孩子(child)和后代(descendant)是不一样的。孩子标签就是父标签的直接下一代，而后代标签则包括了父标签下面所有的子子孙孙。通俗来说，descendant包括了child。

import requestsfrom bs4 import BeautifulSoup

res = requests.get('https://www.pythonscraping.com/pages/page3.html')
soup = BeautifulSoup(res.text, 'lxml')
gifts = soup.find('table', id='giftList').childrenfor name in gifts:print(name)

<tr>
<th>
Item Title
</th>
<th>
Description
</th>
<th>
Cost
</th>
<th>
Image
</th>
</tr>


<tr>
<td>
Vegetable Basket
</td>
<td>
This vegetable basket is the perfect gift for your health conscious (or overweight) friends!
<span>Now with super-colorful bell peppers!</span>
</td>
<td>
$15.00
</td>
<td>
<img  src="/static/imghwm/default1.png" data-src="../img/gifts/img1.jpg" class="lazy" alt="Python data collection--Usage of Beautifulsoup" >
</td>
</tr>


<tr>
<td>
Russian Nesting Dolls
</td>
<td>
Hand-painted by trained monkeys, these exquisite dolls are priceless! And by "priceless," we mean "extremely expensive"! <span>8 entire dolls per set! Octuple the presents!</span>
</td>
<td>
$10,000.52
</td>
<td>
<img  src="/static/imghwm/default1.png" data-src="../img/gifts/img2.jpg" class="lazy" alt="Python data collection--Usage of Beautifulsoup" >
</td>
</tr>

找到表格后，选取当前结点为tr，并找到这个tr之后的兄弟节点，由于第一个tr为表格标题，这样的写法能提取出所有除开表格标题的正文数据。

import requestsfrom bs4 import BeautifulSoup

res = requests.get('https://www.pythonscraping.com/pages/page3.html')
soup = BeautifulSoup(res.text, 'lxml')
gifts = soup.find('table', id='giftList').tr.next_siblingsfor name in gifts:print(name)

<tr>
<td>
Vegetable Basket
</td>
<td>
This vegetable basket is the perfect gift for your health conscious (or overweight) friends!
<span>Now with super-colorful bell peppers!</span>
</td>
<td>
$15.00
</td>
<td>
<img  src="/static/imghwm/default1.png" data-src="../img/gifts/img1.jpg" class="lazy" alt="Python data collection--Usage of Beautifulsoup" >
</td>
</tr>


<tr>
<td>
Russian Nesting Dolls
</td>
<td>
Hand-painted by trained monkeys, these exquisite dolls are priceless! And by "priceless," we mean "extremely expensive"! <span>8 entire dolls per set! Octuple the presents!</span>
</td>
<td>
$10,000.52
</td>
<td>
<img  src="/static/imghwm/default1.png" data-src="../img/gifts/img2.jpg" class="lazy" alt="Python data collection--Usage of Beautifulsoup" >
</td>
</tr>

查找商品的价格，可以根据商品的图片找到其父标签<td>，其上一个兄弟标签就是价格。<div class="sourceCode"><pre class='brush:php;toolbar:false;'>import requestsfrom bs4 import BeautifulSoup res = requests.get(&amp;#39;https://www.pythonscraping.com/pages/page3.html&amp;#39;) soup = BeautifulSoup(res.text, &amp;#39;lxml&amp;#39;) price = soup.find(&amp;#39;img&amp;#39;, src=&amp;#39;../img/gifts/img1.jpg&amp;#39;).parent.previous_sibling.stringprint(price)</pre></div> <p><br></p> <pre class="brush:php;toolbar:false">$15.00</pre> <p>采集所有商品图片，为了避免其他图片乱入。使用正则表达式精确搜索。</p> <div class="sourceCode"><pre class='brush:php;toolbar:false;'>import reimport requestsfrom bs4 import BeautifulSoup res = requests.get(&amp;#39;https://www.pythonscraping.com/pages/page3.html&amp;#39;) soup = BeautifulSoup(res.text, &amp;#39;lxml&amp;#39;) imgs= soup.find_all(&amp;#39;img&amp;#39;, src=re.compile(r&amp;#39;../img/gifts/img.*.jpg&amp;#39;))for img in imgs:print(img[&amp;#39;src&amp;#39;])</pre></div> <p><br></p> <pre class="brush:php;toolbar:false">../img/gifts/img1.jpg ../img/gifts/img2.jpg ../img/gifts/img3.jpg ../img/gifts/img4.jpg ../img/gifts/img6.jpg</pre> <p><code>find_all()还可以传入函数，对这个函数有个要求：就是其返回值必须是布尔类型，若是True则保留，若是False则剔除。

import reimport requestsfrom bs4 import BeautifulSoup

res = requests.get('https://www.pythonscraping.com/pages/page3.html')
soup = BeautifulSoup(res.text, 'lxml')# lambda tag: tag.name=='img'tags = soup.find_all(lambda tag: tag.has_attr('src'))for tag in tags:print(tag)

<img  src="/static/imghwm/default1.png" data-src="../img/gifts/logo.jpg" class="lazy" alt="Python data collection--Usage of Beautifulsoup" >
<img  src="/static/imghwm/default1.png" data-src="../img/gifts/img1.jpg" class="lazy" alt="Python data collection--Usage of Beautifulsoup" >
<img  src="/static/imghwm/default1.png" data-src="../img/gifts/img2.jpg" class="lazy" alt="Python data collection--Usage of Beautifulsoup" >
<img  src="/static/imghwm/default1.png" data-src="../img/gifts/img3.jpg" class="lazy" alt="Python data collection--Usage of Beautifulsoup" >
<img  src="/static/imghwm/default1.png" data-src="../img/gifts/img4.jpg" class="lazy" alt="Python data collection--Usage of Beautifulsoup" >
<img  src="/static/imghwm/default1.png" data-src="../img/gifts/img6.jpg" class="lazy" alt="Python data collection--Usage of Beautifulsoup" >

tag是一个Element对象，has_attr用来判断是否有该属性。tag.name则是获取标签名。在上面的网页中，下面的写法返回的结果一样。
lambda tag: tag.has_attr('src')或lambda tag: tag.name=='img'

The above is the detailed content of Python data collection--Usage of Beautifulsoup. For more information, please follow other related articles on the PHP Chinese website!