JavajavaTutorialHow does a Java crawler implement Jsoup using the dom method to traverse the Document object?How does a Java crawler implement Jsoup using the dom method to traverse the Document object?
先给出网页地址：
https://wall.alphacoders.com/featured.php?lang=Chinese
主要步骤：
利用Jsoup的connect方法获取Document对象
String html = "https://wall.alphacoders.com/featured.php?lang=Chinese"; Document doc = Jsoup.connect(html).get();
内容过长，就不再显示。
我们以这部分为例：
<ul class="nav nav-pills">
<li><a href="https://alphacoders.com/site/about-us" rel="external nofollow" rel="external nofollow" >About Us</a></li>
<li><a href="https://alphacoders.com/site/faq" rel="external nofollow" rel="external nofollow" >FAQ</a></li>
<li><a href="https://alphacoders.com/site/privacy" rel="external nofollow" rel="external nofollow" >Privacy Policy</a></li>
<li><a href="https://alphacoders.com/site/tos" rel="external nofollow" rel="external nofollow" >Terms Of Service</a></li>
<li><a href="https://alphacoders.com/site/acceptable_use" rel="external nofollow" rel="external nofollow" >Acceptable Use</a></li>
<li><a href="https://alphacoders.com/site/etiquette" rel="external nofollow" rel="external nofollow" >Etiquette</a></li>
<li><a href="https://alphacoders.com/site/advertising" rel="external nofollow" rel="external nofollow" >Advertise With Us</a></li>
<li><a id="change_consent">Change Consent</a></li>
</ul>我们先找到所有的ul：
Elements elements = doc.getElementsByTag("ul");输出如下：
<ul class="nav navbar-nav center"> <li> <a title="Submit Wallpapers" href="https://alphacoders.com/site/submit-wallpaper" rel="external nofollow" ><i class="el el-circle-arrow-up"></i> 提交</a> </li> <li> <a href="https://alphacoders.com/contest" rel="external nofollow" ><i class="el el-gift"></i> 精美奖品</a> </li> </ul> <ul class="nav navbar-nav navbar-right center"> <li> <a href="language.php?lang=Chinese" rel="external nofollow" > <img src="/static/imghwm/default1.png" data-src="https://cache.yisu.com/upload/information/20220517/112/2427.png" class="lazy" alt="Chinese-flag"> 中文 </a> </li> <li> <a href="https://alphacoders.com/users/login" rel="external nofollow" ><i class="el el-user"></i> 登录</a> </li> <li> <a href="https://alphacoders.com/users/register" rel="external nofollow" ><i class="el el-edit"></i> 注册</a> </li> </ul> <ul class="pagination"> <li class="active"><a id="prev_page" href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >< 上一页</a></li> <li class="active"><a>1</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >2</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=3" rel="external nofollow" rel="external nofollow" rel="external nofollow" >3</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=4" rel="external nofollow" rel="external nofollow" rel="external nofollow" >4</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=5" rel="external nofollow" rel="external nofollow" rel="external nofollow" >5</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=6" rel="external nofollow" rel="external nofollow" rel="external nofollow" >6</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=7" rel="external nofollow" rel="external nofollow" rel="external nofollow" >7</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=8" rel="external nofollow" >8</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=9" rel="external nofollow" >9</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=10" rel="external nofollow" >10</a></li> <li><a>...</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=319" rel="external nofollow" rel="external nofollow" rel="external nofollow" >319</a></li> <li><a id="next_page" href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >下一页 ></a></li> </ul> <ul class="pagination"> <li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >< 上一页</a></li> <li class="active"><a>1</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >2</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=3" rel="external nofollow" rel="external nofollow" rel="external nofollow" >3</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=4" rel="external nofollow" rel="external nofollow" rel="external nofollow" >4</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=5" rel="external nofollow" rel="external nofollow" rel="external nofollow" >5</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=6" rel="external nofollow" rel="external nofollow" rel="external nofollow" >6</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=7" rel="external nofollow" rel="external nofollow" rel="external nofollow" >7</a></li> <li><a>...</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=319" rel="external nofollow" rel="external nofollow" rel="external nofollow" >319</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >下一页 ></a></li> </ul> <ul class="pagination"> <li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" ><< </a></li> <li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >< 上一页</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >下一页 ></a></li> <li><a title="末页 (319)" href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=319" rel="external nofollow" rel="external nofollow" rel="external nofollow" > >></a></li> </ul> <ul class="pagination"> <li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >1</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >2</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=3" rel="external nofollow" rel="external nofollow" rel="external nofollow" >3</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=4" rel="external nofollow" rel="external nofollow" rel="external nofollow" >4</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=5" rel="external nofollow" rel="external nofollow" rel="external nofollow" >5</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=6" rel="external nofollow" rel="external nofollow" rel="external nofollow" >6</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=7" rel="external nofollow" rel="external nofollow" rel="external nofollow" >7</a></li> </ul> <ul class="nav nav-pills"> <li><a href="https://alphacoders.com/site/about-us" rel="external nofollow" rel="external nofollow" >About Us</a></li> <li><a href="https://alphacoders.com/site/faq" rel="external nofollow" rel="external nofollow" >FAQ</a></li> <li><a href="https://alphacoders.com/site/privacy" rel="external nofollow" rel="external nofollow" >Privacy Policy</a></li> <li><a href="https://alphacoders.com/site/tos" rel="external nofollow" rel="external nofollow" >Terms Of Service</a></li> <li><a href="https://alphacoders.com/site/acceptable_use" rel="external nofollow" rel="external nofollow" >Acceptable Use</a></li> <li><a href="https://alphacoders.com/site/etiquette" rel="external nofollow" rel="external nofollow" >Etiquette</a></li> <li><a href="https://alphacoders.com/site/advertising" rel="external nofollow" rel="external nofollow" >Advertise With Us</a></li> <li><a id="change_consent">Change Consent</a></li> </ul>
可以发现class为"nav nav-pills"的只有一个，我们找到它：
Elements elements = doc.getElementsByTag("ul");
//System.out.println(elements);
Element tempElement = null;
for(Element element : elements) {
if (element.className().equals("nav nav-pills")) {
tempElement = element;
//System.out.println(element.className());
break;
}
}循环遍历这个ul，输出其中每一个li里每一个a的href和rel属性：
Elements li = tempElement.getElementsByTag("li");
for(Element element : li) {
Elements element2 = element.getElementsByTag("a");
for(Element element3 : element2) {
String hrefString = element3.attr("href");
String relString = element3.attr("rel");
if(hrefString != "" && relString != "") {
System.out.println("href=" + hrefString + " rel="external nofollow" rel="external nofollow" " + "rel=" + relString);
}
}
}最终结果：
href=https://alphacoders.com/site/about-us rel=nofollow
href=https://alphacoders.com/site/faq rel=nofollow
href=https://alphacoders.com/site/privacy rel=nofollow
href=https://alphacoders.com/site/tos rel=nofollow
href=https://alphacoders.com/site/acceptable_use rel=nofollow
href=https://alphacoders.com/site/etiquette rel=nofollow
href=https://alphacoders.com/site/advertising rel=nofollow
完整代码：
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
import org.jsoup.Jsoup;
/**
* @ClassName: Jsoup_Test
* @description:
* @author: KI
* @Date: 2020年8月17日 下午8:15:14
*/
public class Jsoup_Test {
public static void main(String[] args) throws IOException {
// TODO 自动生成的方法存根
String html = "https://wall.alphacoders.com/featured.php?lang=Chinese";
Document doc = Jsoup.connect(html).get();
System.out.println(doc);
Elements elements = doc.getElementsByTag("ul");
//System.out.println(elements);
Element tempElement = null;
for(Element element : elements) {
if (element.className().equals("nav nav-pills")) {
tempElement = element;
//System.out.println(element.className());
break;
}
}
System.out.println(tempElement);
Elements li = tempElement.getElementsByTag("li");
for(Element element : li) {
Elements element2 = element.getElementsByTag("a");
for(Element element3 : element2) {
String hrefString = element3.attr("href");
String relString = element3.attr("rel");
if(hrefString != "" && relString != "") {
System.out.println("href=" + hrefString + " rel="external nofollow" rel="external nofollow" " + "rel=" + relString);
}
}
}
}
}The above is the detailed content of How does a Java crawler implement Jsoup using the dom method to traverse the Document object?. For more information, please follow other related articles on the PHP Chinese website!
How do I use Maven or Gradle for advanced Java project management, build automation, and dependency resolution?Mar 17, 2025 pm 05:46 PMThe article discusses using Maven and Gradle for Java project management, build automation, and dependency resolution, comparing their approaches and optimization strategies.
How do I create and use custom Java libraries (JAR files) with proper versioning and dependency management?Mar 17, 2025 pm 05:45 PMThe article discusses creating and using custom Java libraries (JAR files) with proper versioning and dependency management, using tools like Maven and Gradle.
How do I implement multi-level caching in Java applications using libraries like Caffeine or Guava Cache?Mar 17, 2025 pm 05:44 PMThe article discusses implementing multi-level caching in Java using Caffeine and Guava Cache to enhance application performance. It covers setup, integration, and performance benefits, along with configuration and eviction policy management best pra
How can I use JPA (Java Persistence API) for object-relational mapping with advanced features like caching and lazy loading?Mar 17, 2025 pm 05:43 PMThe article discusses using JPA for object-relational mapping with advanced features like caching and lazy loading. It covers setup, entity mapping, and best practices for optimizing performance while highlighting potential pitfalls.[159 characters]
How does Java's classloading mechanism work, including different classloaders and their delegation models?Mar 17, 2025 pm 05:35 PMJava's classloading involves loading, linking, and initializing classes using a hierarchical system with Bootstrap, Extension, and Application classloaders. The parent delegation model ensures core classes are loaded first, affecting custom class loa
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
Safe Exam Browser
Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.
Notepad++7.3.1
Easy-to-use and free code editor
MinGW - Minimalist GNU for Windows
This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.
DVWA
Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software
PhpStorm Mac version
The latest (2018.2.1) professional PHP integrated development tool
