之前，亦枫写过一篇关于使用 Jsoup 抓取网页内容的文章：
【Jsoup】HTML解析器，轻松获取网页内容
Jsoup提供的api非常便捷，完全的类似JQuery操作，轻松抓取网页数据。但像Jsoup这样普通的爬虫工具不足的地方就是无法处理js生成的内容。
做过Html开发的人都知道，现在很多网站都在大量使用ajax和JavaScript来获取并处理数据，普通的爬虫工具已经无法处理js中的内容。
举例说明，我们在本地新建一个测试网页文件text.html，源码如下：
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html> <head> <title>main.html</title> <meta http-equiv="keywords" content="keyword1,keyword2,keyword3"> <meta http-equiv="description" content="this is my page"> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <style type="text/css"> a { line-height: 30px; margin: 20px; } </style> <!--<link rel="stylesheet" type="text/css" href="./styles.css">--> <script type="text/javascript">var datas = [ { href : "http://www.jianshu.com/p/8d8edf25850d", title : "推荐一款编程字体，让代码看着更美"}, { href : "http://www.jianshu.com/p/153d9f31288d", title : "Android 利用Camera实现中轴3D卡牌翻转效果"}, { href : "http://www.jianshu.com/p/d6fb0c9c9c26", title : "【Eclipse】挖掘专属最有用的快捷键组合"}, { href : "http://www.jianshu.com/p/72d69b49d135", title : "【IIS】Windows下利用IIS建立网站并实现局域网共享"} ];window.onload = function() { var infos = document.getElementById("infos"); for( var i = 0 ; i < datas.length ; i++) { var a = document.createElement("a"); a.href = datas[i].href ; a.innerText = datas[i].title; infos.appendChild(a); infos.appendChild(document.createElement("br")) }}</script> </head> <body> <div class="text" style=" text-align:center;">HtmlUnit 测试网页内容！</div> <br> <div id="infos" style="width: 60%; border: 1px solid green; border-radius: 10px; margin: 0 auto;"> </div> </body></html> 通过IIS发布本地网站（参考亦枫之前写的文章：【IIS】Windows下利用IIS建立网站并实现局域网共享），在浏览器中显示的网页效果如下：
网页展示效果.jpg
虽然通过网页审查元素可以看到body中含有网页展示中的文本内容：
网页审查元素.jpg
但是，通过Jsoup工具根本无法获取！在网页源代码中可以看出，我们需要抓取的内容是在页面显示之后通过ajax和JavaScript加载得到的。
那么怎么办呢？使用本文中推荐的开源工具 —— HtmlUnit，一款能够模拟浏览器的抓包神器！
在官网下载对应jar包，添加到项目工程的lib中，简单的测试代码如下：
import java.io.IOException;import java.net.MalformedURLException;import java.text.ParseException;import com.gargoylesoftware.htmlunit.BrowserVersion;import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;import com.gargoylesoftware.htmlunit.WebClient;import com.gargoylesoftware.htmlunit.html.DomElement;import com.gargoylesoftware.htmlunit.html.DomNodeList;import com.gargoylesoftware.htmlunit.html.HtmlPage;/** * @author 亦枫 * @created_time 2016年1月12日 * @file_user_todo Java测试类 * @blog http://www.jianshu.com/users/1c40186e3248/latest_articles */public class JavaTest { /** * 入口函数 * @param args * @throws ParseException */ public static void main(String[] args) throws ParseException { try { WebClient webClient = new WebClient(BrowserVersion.CHROME); HtmlPage htmlPage = (HtmlPage) webClient.getPage("http://localhost/test.html"); DomNodeList domNodeList = htmlPage.getElementsByTagName("a"); for (int i = 0; i < domNodeList.size(); i++) { DomElement domElement = (DomElement) domNodeList.get(i); System.out.println(domElement.asText()); } webClient.close(); } catch (FailingHttpStatusCodeException e) { e.printStackTrace(); } catch (MalformedURLException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } }} 运行之后，在控制台打印的结果：
HtmlUnit测试结果.jpg
可以看出，HtmlUnit能够抓取到AJAX和JavaScript加载的内容。
有关HtmlUnit的介绍在官网上写的非常详细，以下内容是亦枫翻译的一部分内容，供大家参考：
HtmlUnit是一款基于Java的没有图形界面的浏览器程序。它能够调用HTML文档并且提供API让开发人员像是在一个正常的浏览器上操作一样，获取网页内容，填充表单，点击超链接等等。
它能够非常好的支持JavaScript并且仍在不断改进提升，同时能够解析非常复杂的AJAX库，在不同的配置下模拟Chrome、Firefox和IE浏览器。
HtmlUnit通常用于测试目的和检索网站信息。
HtmlUnit提供了很多测试网络请求和抓取网页内容的功能，大家可以去官网或者其他网站学习使用。
欢迎关注亦枫微信公众号：技术鸟，一起学习，共同进步！
技术鸟_微信二维码.gif
The Future of HTML: Evolution and TrendsMay 13, 2025 am 12:01 AMThe future of HTML will develop in a more semantic, functional and modular direction. 1) Semanticization will make the tag describe the content more clearly, improving SEO and barrier-free access. 2) Functionalization will introduce new elements and attributes to meet user needs. 3) Modularity will support component development and improve code reusability.
Why are HTML attributes important for web development?May 12, 2025 am 12:01 AMHTMLattributesarecrucialinwebdevelopmentforcontrollingbehavior,appearance,andfunctionality.Theyenhanceinteractivity,accessibility,andSEO.Forexample,thesrcattributeintagsimpactsSEO,whileonclickintagsaddsinteractivity.Touseattributeseffectively:1)Usese
What is the purpose of the alt attribute? Why is it important?May 11, 2025 am 12:01 AMThe alt attribute is an important part of the tag in HTML and is used to provide alternative text for images. 1. When the image cannot be loaded, the text in the alt attribute will be displayed to improve the user experience. 2. Screen readers use the alt attribute to help visually impaired users understand the content of the picture. 3. Search engines index text in the alt attribute to improve the SEO ranking of web pages.
HTML, CSS, and JavaScript: Examples and Practical ApplicationsMay 09, 2025 am 12:01 AMThe roles of HTML, CSS and JavaScript in web development are: 1. HTML is used to build web page structure; 2. CSS is used to beautify the appearance of web pages; 3. JavaScript is used to achieve dynamic interaction. Through tags, styles and scripts, these three together build the core functions of modern web pages.
How do you set the lang attribute on the tag? Why is this important?May 08, 2025 am 12:03 AMSetting the lang attributes of a tag is a key step in optimizing web accessibility and SEO. 1) Set the lang attribute in the tag, such as. 2) In multilingual content, set lang attributes for different language parts, such as. 3) Use language codes that comply with ISO639-1 standards, such as "en", "fr", "zh", etc. Correctly setting the lang attribute can improve the accessibility of web pages and search engine rankings.
What is the purpose of HTML attributes?May 07, 2025 am 12:01 AMHTMLattributesareessentialforenhancingwebelements'functionalityandappearance.Theyaddinformationtodefinebehavior,appearance,andinteraction,makingwebsitesinteractive,responsive,andvisuallyappealing.Attributeslikesrc,href,class,type,anddisabledtransform
How do you create a list in HTML?May 06, 2025 am 12:01 AMTocreatealistinHTML,useforunorderedlistsandfororderedlists:1)Forunorderedlists,wrapitemsinanduseforeachitem,renderingasabulletedlist.2)Fororderedlists,useandfornumberedlists,customizablewiththetypeattributefordifferentnumberingstyles.
HTML in Action: Examples of Website StructureMay 05, 2025 am 12:03 AMHTML is used to build websites with clear structure. 1) Use tags such as, and define the website structure. 2) Examples show the structure of blogs and e-commerce websites. 3) Avoid common mistakes such as incorrect label nesting. 4) Optimize performance by reducing HTTP requests and using semantic tags.
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
PhpStorm Mac version
The latest (2018.2.1) professional PHP integrated development tool
DVWA
Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software
SublimeText3 Chinese version
Chinese version, very easy to use
SecLists
SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.
Dreamweaver Mac version
Visual web development tools
