PyQuery: Python's jQuery

In this tutorial, you'll have a look at PyQuery object is similar to what you get with $() when using the jQuery library. Just like the html() method in PyQuery, where you will be able to get or set the HTML content of the selected element.

Currently, the webpage object is representative of the whole document, so it returns the markup of the entire page:

print(webpage.html())<br><br>'''<br><br><meta charset="utf-8"><br><title>A Simple Webpage</title><br><meta name="viewport" content="width=device-width, initial-scale=1"><br><br><br><br>    <p>Hello <b>world</b>! This is a basic webpage.</p><br>    <p>Here is a list of some <i>random</i> words:</p><br>    

• Impedimenta


• Decompensation


• Tergiversation


• Transcendentalism


• Polyphiloprogenitive

'''

Let's say you want to get the markup of the first webpage object. Here is an example:

print(webpage("p").html())<br><br>'''<br>Hello <b>world</b>! This is a basic webpage.<br>'''<br>

Now take a look at the following code, where we will first set the HTML for our selector using the html() method.

from pyquery import PyQuery as pq<br><br>webpage = pq(filename = 'document.html')<br><br>print(webpage("p").html())<br>'''<br>Hello <b>world</b>! This is a basic webpage.<br>'''<br><br>webpage("p").html("Hello <b>world</b>! I have changed this paragraph.")<br><br>print(webpage("p").html())<br>'''<br>Hello <b>world</b>! I have changed this paragraph.<br>'''<br>

As you can see, it was very easy for us to manipulate the HTML of particular tags. Let's see what else we can change.

Attribute Manipulation Using PyQuery

PyQuery tries to mirror the jQuery API as closely as possible. This means that you get access to an attribute method called class attribute from the list. We will also use the attr() method to add a set of classes to our attr() method in PyQuery also sets the attribute value for all matching elements instead of the first one.

How can we apply the classes only to the first eq() method to get the element at a particular index, as shown below.

webpage("p").eq(0).attr("class", "greeting hello-message")<br>

If you are primarily looking to manipulate classes of your elements, you can also consider using the removeClass() methods, which will add or remove a CSS class respectively. You can also use the method names remove_class() if you are more comfortable working with underscore notation.

Here is an example:

webpage("p").eq(0).attr("class", "greeting hello-message")<br># 
Hello world! This is a basic webpage.


webpage("p").eq(0).remove_class("greeting")
# 
Hello world! This is a basic webpage.


webpage("p").eq(0).add_class("first-message")
#  
Hello world! This is a basic webpage.

You can also get rid of any attributes assigned to an element by using the add_attr() method as that functionality is served by font-size to css() method in PyQuery is similar to the one in jQuery. After updating the styles, we saved the new markup to a file called updated_markup.html. You can also do the same after making a variety of changes to the markup.

Creating, Removing, & Appending Elements

You might recall that our sample HTML document contains a list of words. Can we expand the list of words? Of course we can. All you need to do is use the prepend() methods. The prepend() method will prepend the passed value to the calling node. Here is an example:

from pyquery import PyQuery as pq<br><br>webpage = pq(filename = 'document.html')<br><br>print(webpage("ul"))<br>'''<br>

• Impedimenta


• Decompensation


• Tergiversation


• Transcendentalism


• Polyphiloprogenitive

'''

webpage("ul").append("

Myrmecophilous

")
webpage("ul").prepend("

Anagnorisis

")

print(webpage("ul"))
'''

• Anagnorisis


• Impedimenta


• Decompensation


• Tergiversation


• Transcendentalism


• Polyphiloprogenitive


• Myrmecophilous

'''

Another option that you have for appending and prepending elements is the use of the prepend_to() methods. The prepend_to() method will prepend your calling node to the passed node. However, remember that you cannot simply call these methods on a string. You will have to wrap them in a PyQuery object for the call to work, as shown below:

print(webpage.html())<br><br>'''<br><br><meta charset="utf-8"><br><title>A Simple Webpage</title><br><meta name="viewport" content="width=device-width, initial-scale=1"><br><br><br><br>    <p>Hello <b>world</b>! This is a basic webpage.</p><br>    <p>Here is a list of some <i>random</i> words:</p><br>    

• Impedimenta


• Decompensation


• Tergiversation


• Transcendentalism


• Polyphiloprogenitive

'''

As you can see, we get the same output. You can also remove nodes from your document by simply calling the children() and children() method returns all the elements that are direct children of the calling node. In our case, this means all the list elements. After that, we use the li tags to append them to our now empty unordered list.

Finding Elements Using PyQuery

There is a good chance that you will be working with HTML documents in order to extract some data from them. Now, before you can extract this data from any element, you will need to locate or find the element.

You can simply use the closest() method to look for elements if you are interested in searching through the ancestors of that particular selector.

print(webpage("p").html())<br><br>'''<br>Hello <b>world</b>! This is a basic webpage.<br>'''<br>

We have already covered the siblings() method. Other similar methods that you can use are prev_all(), which will give you all the siblings that come next or the siblings that came before respectively. Here is an example:

from pyquery import PyQuery as pq<br><br>webpage = pq(filename = 'document.html')<br><br>print(webpage("p").html())<br>'''<br>Hello <b>world</b>! This is a basic webpage.<br>'''<br><br>webpage("p").html("Hello <b>world</b>! I have changed this paragraph.")<br><br>print(webpage("p").html())<br>'''<br>Hello <b>world</b>! I have changed this paragraph.<br>'''<br>

Extracting Content From a Webpage

Do you remember when I told you at the beginning of the tutorial that PyQuery can accept input from multiple sources such as a string, a file, or even a URL?

In this section, we will let PyQuery get its markup from a page about Python on Wikipedia. The webpage contains a lot of information about Python. We will try to extract some of it for our consumption. Let's see if we can get all the h2<code>h2 level headings to keep things simple.

Believe it or not, you only need five lines of code to get your heading text.

webpage("p").eq(0).attr("class", "greeting hello-message")<br>

You might have noticed that I used the selector h2 span.mw-headline instead of using h2. This is because simply using h2 was giving me some additional headings that were not part of the main content. You will also have to do a similar analysis of webpages yourself before determining the appropriate selector to use for extracting the information.

I have already written a tutorial on the Requests module in Python where we used the module to download images. One limitation of the example I included there was that we were hard-coding the path of the image. Let's use the PyQuery library to extract the image paths from a webpage and then feed them to the requests module to download. I will be using the Wikipedia page about the United States for this example:

webpage("p").eq(0).attr("class", "greeting hello-message")<br># 
Hello world! This is a basic webpage.


webpage("p").eq(0).remove_class("greeting")
# 
Hello world! This is a basic webpage.


webpage("p").eq(0).add_class("first-message")
#  
Hello world! This is a basic webpage.

We don't want to download images of the UI icons, etc. That's why I used a more specific selector to extract our images. I get the image file name by taking the last part of the image path after splitting it along the / character. Here are some of the images that I was able to extract:

Wrapping It Up

In this tutorial, you saw how to get started with PyQuery, a Python library which allows you to make jQuery queries on XML documents. You saw how to manipulate the attributes and CSS styles of the HTML elements. 

You learnt how to create and append elements to existing elements and insert new elements before and after elements. What you saw in this tutorial is just the tip of the iceberg, and there is a lot more that this library has to offer.

For more detailed information on using this library, I would recommend reading the official documentation.

The above is the detailed content of PyQuery: Python's jQuery. For more information, please follow other related articles on the PHP Chinese website!