How to Extract Visible Webpage Content Using BeautifulSoup?

How to Scrape Visible Webpage Content Effectively

Web scraping involves extracting specific data from web pages, but it can be challenging to limit the results to only the visible text. BeautifulSoup, a popular web scraping library, simplifies this process.

Understanding the Problem

To scrape visible text, it's crucial to exclude elements like scripts (<script>), comments (<!-- -->), CSS (<style>), and other non-visible content.</script>

The Solution with BeautifulSoup

Here's a simple solution using BeautifulSoup's findAll() method and a custom filter:

from bs4 import BeautifulSoup
from bs4.element import Comment
from urllib.request import urlopen

def tag_visible(element):
    if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]']:
        return False
    if isinstance(element, Comment):
        return False
    return True

def text_from_html(body):
    soup = BeautifulSoup(body, 'html.parser')
    texts = soup.findAll(text=True)
    visible_texts = filter(tag_visible, texts)  
    return " ".join(t.strip() for t in visible_texts)

html = urlopen('http://www.nytimes.com/2009/12/21/us/21storm.html').read()
print(text_from_html(html))

Explanation

The code:

• Defines a function tag_visible() to exclude unwanted tags and comments.
• Uses filter() to apply the tag_visible() function to all text elements (texts) found in the BeautifulSoup object.
• Joins the visible texts into a single string.
• Fetches the HTML of the specified URL using urlopen().

The above is the detailed content of How to Extract Visible Webpage Content Using BeautifulSoup?. For more information, please follow other related articles on the PHP Chinese website!