JAVA crawls web page data jsoup

Regarding the crawling data that has emerged in recent years, most people use Python to crawl. For some people, they still don’t know that Java can also perform data crawling. Then the editor will share with you an article about Java crawling web page data.

The meaning of crawling data is to obtain the page information returned by the request and filter out the data we want.

So let’s make a long story short:

1. Environment preparation

If the tool is eclipse or IDEA

Configure maven

2 .Import jsoup related dependencies

                                               ;1.13.1 > ;

3. Start setting up a project to write code. My code is as follows:

public static void main(String[] args) throws Exception { Scanner sc = new Scanner(System.in) ; System.out.println("Please enter the search keyword!!!"); String input = sc.next(); //Get the url request, find the request here yourself String url = "https://search .xx.com/Search?keyword=" input; //Parse the web page (the document returned by jsoup here is the page document object) Document document = Jsoup.parse(new URL(url), 30000); //All js All methods can be used here, here is to get the id in the crawling target div Element element = document.getElementById("x_goodsList"); System.out.println(element.html()); //Get the crawling target div All li tags Elements elements = document.getElementsByTag("li"); //Traverse all the content in the li tag for(Element el : elements){

//Sometimes there are more pictures to crawl Websites all use lazy loading. The attribute obtained is source-data-lazy-img instead of src

/*

img is the crawled image,

price is the crawled price,

Name is the crawled product name

*/

String img = el.getElementsByTag("img").eq(0).attr("src"); String price = el.getElementsByClass("p-price").eq(0).text(); String name = el.getElementsByClass("p-name").eq(0).text(); System.out.println ("========================"); System.out.println(img); System.out.println(price); System.out .println(name); }

Finally, friends who are interested in crawling data should hurry up and give it a try. Here I will prevent all illegal crawling behaviors. Hahaha, I hope you know what can be done. Climb and play, there is nothing not to climb.

The above is the detailed content of JAVA crawls web page data jsoup. For more information, please follow other related articles on the PHP Chinese website!