jsoup: Summary of parsing HTML usage_html/css_WEB-ITnose

1. Parsing method

(1) Parsing from string

String html = " First parse Parse HTML into a doc.

";

Document doc = Jsoup.parse(html);

?

(2) Get parsing from URL

Document doc = Jsoup.connect("http ://example.com/").get();

String title = doc.title();

Document doc = Jsoup.connect("http:// example.com") .data("query", "Java").userAgent("Mozilla").cookie("auth", "token").timeout(3000).post();

?

(3) Parse from file

File input = new File("/tmp/input.html");

Document doc = Jsoup.parse(input, "UTF-8", "http://example.com/" ; )

getElementByTag(String tag)

getElementByClass(String className) getElementByAttribute(String key)

siblingElements(), firstElementSibling(), lastElementSibling( ), nextElementSibling(), previousElementSibling()

parent(), children(), child(int index)

(2) Get element data

attr(String key) ? Get key attributes

attributes() ? Get attributes

id(), className(), classNames()

text() ? Get the text content html() ? Get the HTML content inside the element

outerHtml() ? Get the HTML content including this element

data() ? Get Or the content in the tag

tag(), tagName()

3. Selector syntax (the difference between jsoup and other parsers is that you can use Selector syntax similar to jquery to search and filter out the required elements) (1) Basic selector

tagname: Search for tag tag elements

ns|tag: Search for elements with tag tags in the namespace, such as fb|name:

#id: Search for elements with specified id .class: Search for Elements of specified class

[attribute]: Search for elements with atrribute attributes

[^attri]: Search for elements with attributes starting with attri

[attr=value ]: Search for elements with specified attributes and their attribute values

[attr^=value], [attr$=value], [attr*=value]: Search for elements with specified attr attributes and their attribute values are Elements that start with, end with, or include value, such as [href*=/path/]

[attr~=regex]: Search for elements with the specified attr attribute and whose attribute value conforms to the regex regular expression

*: Search all elements (2) Selector combination

el#id: Specify the tag name and id at the same time

el. class: Specify the tag name and class at the same time

el[attr]: Specify the tag name and the name of the attribute contained in it at the same time

Any combination of the above three items, such as a[href]. highlight

ancestor child: Contains, such as div.content p, that is, search for elements containing the tag under ancestor > child: Directly included, such as div.content > p, that is, searching for tag elements directly under the node; div.content > *, that is, searching for All elements under

siblingA siblingB: traverse directly, such as div.head div, that is, search for elements of , which no longer contain child elements

siblingA ~ siblingX: traverse, such as h1 ~ p, that is, elements with directly or indirectly under

el, el, el: combine multiple selectors , search for elements that satisfy one of the selectors (3) Pseudo selector (conditional selector)

:lt(n): Search for elements before element n

:gt(n): Search for elements after element n

:eq(n): Search for element n Element

:has(seletor): Search for elements that match the specified selector

:not(seletor): Search for elements that do not match the specified selector

:contains(text ): Search for elements that contain the specified text, case-sensitive

:containsOwn(text): Search for elements that directly contain the specified text

:matches(regex): Search for the specified regular expression Elements

:matchesOwn(regex): Search for elements in the text of this element that match the specified regular expression

Note: In the index of the above pseudo-selector, the first element is at index 0. The second element is located at index 1,...

4. Get the attributes, text and HTML of the element

Get the attribute value of the element: Node.attr (String key)

Get the text of the element, including its combined sub-elements: Element.text()

Get the HTML: Element.html() or Node.outerHtml()

5. Manipulate URL

Element.attr("href") ? Directly obtain URL

Element.attr("abs:href") Or Element.absUrl("href") ? Get the full URL. If the HTML is parsed from a file or string, you need to call Jsoup.setBaseUri(String baseUri) to specify the base URL, otherwise the complete URL obtained will only be an empty string

6. Test example

li[class=info] a[class=Author] - spaces before and after indicate an inclusion relationship, that is, a in li

div[class=mod mod- main mod-lmain]:contains (teaching reflection) - The div contains "teaching reflection", suitable for situations where there are multiple DIVs with the same name at the same time

/*

previousSibling() gets the front of a certain tag The code

nextSibling() gets the code after a certain tag

For example:

First place: Lily &lt ;br/>

Second place: Tom

Third place: Peter

*/

Elements items = doc.select("form[id=form1]");

Elements prevs = items.select("br");

for(Element p : prevs){

String prevStr = p.previousSibling().toString().trim());

}

/*

The most commonly used links to crawl

*/

String itemTag = "div[class=mydiv]";

String linkTag = " a"

Elements items = doc.select(itemTag);

Elements links = items.select(linkTag);

for(Element l : links){

String href = l.attr("abs:href"); // Complete Href

String absHref = l.attr("href"); // Relative path

String text = l.text();

String title = l.attr("title");

}

7.jsoup Online API http://jsoup .org/apidocs/