Home > Article > Backend Development > Detailed explanation of how Python uses the Beautiful Soup module to search for content

Detailed explanation of how Python uses the Beautiful Soup module to search for content

高洛峰Original: 2017-03-31 09:47:212855browse

This article mainly introduces you to the search method function of the Beautiful Soup module in python. Methods Different types of filtering parameters can perform different filtering to obtain the desired results. The introduction in the article is very detailed and has certain reference value for everyone. Friends who need it can take a look below.

Preface

We will use the search function of the Beautiful Soup module to search based on tag name, tag attributes, document text and regular expressions .

Search method

Beautiful Soup’s built-in search method is as follows:

##find()
find_all()
find_parent()

Use the find() method to searchFirst you still need to create an HTML file for testing.

<html>
<body>
<p class="ecopyramid">
 <ul id="producers">
 <li class="producerlist">
  <p class="name">plants</p>
  <p class="number">100000</p>
 </li>
 <li class="producerlist">
  <p class="name">algae</p>
  <p class="number">100000</p>
 </li>
 </ul>
 <ul id="primaryconsumers">
 <li class="primaryconsumerlist">
  <p class="name">deer</p>
  <p class="number">1000</p>
 </li>
 <li class="primaryconsumerlist">
  <p class="name">rabbit</p>
  <p class="number">2000</p>
 </li>
 </ul>
 <ul id="secondaryconsumers">
 <li class="secondaryconsumerlist">
  <p class="name">fox</p>
  <p class="number">100</p>
 </li>
 <li class="secondaryconsumerlist">
  <p class="name">bear</p>
  <p class="number">100</p>
 </li>
 </ul>
 <ul id="tertiaryconsumers">
 <li class="tertiaryconsumerlist">
  <p class="name">lion</p>
  <p class="number">80</p>
 </li>
 <li class="tertiaryconsumerlist">
  <p class="name">tiger</p>
  <p class="number">50</p>
 </li>
 </ul>
</p>
</body>
</html>

We can get the ff6d136ddc5fdfeffaf53ff6ee95f185 tag through the

find()

method. By default, the first one that appears will be obtained. Then get the 25edfb22a4f469ecb59f1190150159c6 tag. By default, you will still get the first one that appears. Then get the e388a4556c0f65e1904146cc1a846bee tag, and verify whether you got the first one by outputting the content.

from bs4 import BeautifulSoup
with open(&#39;search.html&#39;,&#39;r&#39;) as filename:
 soup = BeautifulSoup(filename,&#39;lxml&#39;)
first_ul_entries = soup.find(&#39;ul&#39;)
print first_ul_entries.li.p.string

find() method is as follows:

find(name,attrs,recursive,text,**kwargs)

As shown in the above code,

find( )

The method accepts five parameters: name, attrs, recursive, text and **kwargs. The name, attrs and text parameters can all act as filters in the find() method to improve the accuracy of the matching results.

Search for tags

In addition to searching for the ff6d136ddc5fdfeffaf53ff6ee95f185 tag in the above code, we can also search for the 25edfb22a4f469ecb59f1190150159c6 tag, and the returned result is also the first one that appears. matches.

tag_li = soup.find(&#39;li&#39;)
# tag_li = soup.find(name = "li")
print type(tag_li)
print tag_li.p.string

Search text

If we only want to search based on text content, we can only pass in the text parameters:

search_for_text = soup.find(text=&#39;plants&#39;)
print type(search_for_text)
<class &#39;bs4.element.NavigableString&#39;>

The returned result is also a NavigableString object.

Search based on regular expression

The following HTML text content

<p>The below HTML has the information that has email ids.</p>
 abc@example.com 
<p>xyz@example.com</p> 
 <span>foo@example.com</span>

You can see abc@ The example email address is not included in any tags, so the email address cannot be found based on the tags. At this time, we can use regular expressions to match.

email_id_example = """
 <p>The below HTML has the information that has email ids.</p>
 abc@example.com
 <p>xyz@example.com</p>
 <span>foo@example.com</span>
 """
email_soup = BeautifulSoup(email_id_example,&#39;lxml&#39;)
print email_soup
# pattern = "\w+@\w+\.\w+"
emailid_regexp = re.compile("\w+@\w+\.\w+")
first_email_id = email_soup.find(text=emailid_regexp)
print first_email_id

When using regular expressions for matching, if there are multiple matches, the first one will be returned first.

Search by tag attribute value

You can search by tag attribute value:

search_for_attribute = soup.find(id=&#39;primaryconsumers&#39;)
print search_for_attribute.li.p.string

According to tag Searching by attribute value is available for most attributes, such as id, style, and title.

But there will be differences in the following two situations:

find()

function.

Search based on custom attributes

In HTML5, you can add custom attributes to tags, such as adding attributes to tags.

As shown in the following code, if we continue to operate like searching for id, an error will be reported. Python variables cannot include the - symbol.

customattr = """
 <p data-custom="custom">custom attribute example</p>
   """
customsoup = BeautifulSoup(customattr,&#39;lxml&#39;)
customsoup.find(data-custom="custom")
# SyntaxError: keyword can&#39;t be an expression

At this time, use the attrs attribute value to pass a dictionary type as a parameter for search:

using_attrs = customsoup.find(attrs={&#39;data-custom&#39;:&#39;custom&#39;})
print using_attrs

Search based on classes in CSS

For CSS class attributes, since class is a keyword in Python, it cannot be passed as a label attribute parameter. In this case, it is the same as self Search as defined properties. Also use the attrs attribute to pass a dictionary for matching.

In addition to using the attrs attribute, you can also use the class_ attribute to pass, which is different from class and will not cause errors.

css_class = soup.find(attrs={&#39;class&#39;:&#39;producerlist&#39;})
css_class2 = soup.find(class_ = "producerlist")
print css_class
print css_class2

Use custom function search

You can pass a function to the

find()

method, This will search based on the conditions defined by the function.
The function should return true or false value.

def is_producers(tag):
 return tag.has_attr(&#39;id&#39;) and tag.get(&#39;id&#39;) == &#39;producers&#39;
tag_producers = soup.find(is_producers)
print tag_producers.li.p.string

An is_producers function is defined in the code, which will check whether the tag has a specific id attribute and whether the attribute value is equal to producers. If the conditions are met, it will return true, otherwise it will return false.

Combined use of various search methods

Beautiful Soup provides various search methods. Similarly, we can also use these methods jointly for matching to improve the accuracy of the search. Spend.

combine_html = """
 <p class="identical">
  Example of p tag with class identical
 </p>
 <p class="identical">
  Example of p tag with class identical
 <p>
 """
combine_soup = BeautifulSoup(combine_html,&#39;lxml&#39;)
identical_p = combine_soup.find("p",class_="identical")
print identical_p

使用 find_all() 方法搜索

使用 find() 方法会从搜索结果中返回第一个匹配的内容，而 find_all() 方法则会返回所有匹配的项。

在 find() 方法中用到的过滤项，同样可以用在 find_all() 方法中。事实上，它们可以用到任何搜索方法中，例如：find_parents() 和 find_siblings() 中。

# 搜索所有 class 属性等于 tertiaryconsumerlist 的标签。
all_tertiaryconsumers = soup.find_all(class_=&#39;tertiaryconsumerlist&#39;)
print type(all_tertiaryconsumers)
for tertiaryconsumers in all_tertiaryconsumers:
 print tertiaryconsumers.p.string

find_all() 方法为：

find_all(name,attrs,recursive,text,limit,**kwargs)

它的参数和 find() 方法有些类似，多个了 limit 参数。limit 参数是用来限制结果数量的。而 find() 方法的 limit 就是 1 了。

同时，我们也能传递一个字符串列表的参数来搜索标签、标签属性值、自定义属性值和 CSS 类。

# 搜索所有的 p 和 li 标签
p_li_tags = soup.find_all(["p","li"])
print p_li_tags
print
# 搜索所有类属性是 producerlist 和 primaryconsumerlist 的标签
all_css_class = soup.find_all(class_=["producerlist","primaryconsumerlist"])
print all_css_class
print

搜索相关标签

一般情况下，我们可以使用 find() 和 find_all() 方法来搜索指定的标签，同时也能搜索其他与这些标签相关的感兴趣的标签。

搜索父标签

可以使用 find_parent() 或者 find_parents() 方法来搜索标签的父标签。

find_parent() 方法将返回第一个匹配的内容，而 find_parents() 将返回所有匹配的内容，这一点与 find() 和 find_all() 方法类似。

# 搜索 父标签
primaryconsumers = soup.find_all(class_=&#39;primaryconsumerlist&#39;)
print len(primaryconsumers)
# 取父标签的第一个
primaryconsumer = primaryconsumers[0]
# 搜索所有 ul 的父标签
parent_ul = primaryconsumer.find_parents(&#39;ul&#39;)
print len(parent_ul)
# 结果将包含父标签的所有内容
print parent_ul
print
# 搜索,取第一个出现的父标签.有两种操作
immediateprimary_consumer_parent = primaryconsumer.find_parent()
# immediateprimary_consumer_parent = primaryconsumer.find_parent(&#39;ul&#39;)
print immediateprimary_consumer_parent

搜索同级标签

Beautiful Soup 还提供了搜索同级标签的功能。

使用函数 find_next_siblings() 函数能够搜索同一级的下一个所有标签，而 find_next_sibling() 函数能够搜索同一级的下一个标签。

producers = soup.find(id=&#39;producers&#39;)
next_siblings = producers.find_next_siblings()
print next_siblings

同样，也可以使用 find_previous_siblings() 和 find_previous_sibling() 方法来搜索上一个同级的标签。

搜索下一个标签

使用 find_next() 方法将搜索下一个标签中第一个出现的，而 find_next_all() 将会返回所有下级的标签项。

# 搜索下一级标签
first_p = soup.p
all_li_tags = first_p.find_all_next("li")
print all_li_tags

搜索上一个标签

与搜索下一个标签类似，使用 find_previous() 和 find_all_previous() 方法来搜索上一个标签。

The above is the detailed content of Detailed explanation of how Python uses the Beautiful Soup module to search for content. For more information, please follow other related articles on the PHP Chinese website!

Statement：

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Previous article：Detailed explanation of iterator and generator instance methods in PythonNext article：Detailed explanation of iterator and generator instance methods in Python

See more

Detailed explanation of how Python uses the Beautiful Soup module to search for content

Related articles