Preparation work for writing Java Zhihu crawler from scratch

Let’s start with the same as before. Let’s talk about the ideas of making a crawler and the knowledge needed to prepare. Experts please ignore it.

First of all, let’s think about what we want to do and list some simple requirements.

The requirements are as follows:

1. Simulate visiting Zhihu official website (http://www.zhihu.com/)

2. Download the specified page content, including: Today’s hottest, This month’s hottest, Editor’s recommendation

3. Download all questions and answers in the specified categories, such as: investment, programming, failing courses

4. Download all answers from the specified respondent

5. It would be better to have a one-click like function (so that I can I was so smart to like all of Lei Lun’s answers at once!)

The technical problems that need to be solved are briefly listed as follows:

1. Simulate the browser to access the webpage

2. Grab key data and save it Go locally

3. Solve the dynamic loading problem in web browsing

4. Use a tree structure to massively crawl all content on Zhihu

Okay, that’s all I have thought about now.

The next step is preparation.

1. Determine the crawler language: Since I have written a series of crawler tutorials before (click here), Baidu Tieba, Embarrassing Encyclopedia, Shandong University's grade point query, etc. are all written in python, so I decided to use Java to write it this time ( Hello, there is absolutely no half cent to contact me, okay).

2. Popular science about crawlers: Web crawler, or Web Spider, is a very vivid name. If the Internet is compared to a spider web, then a spider is a spider crawling around on the web. Web spiders search for web pages through their link addresses. For a detailed introduction, please click here.

3. Prepare the crawler environment: I won’t go into details about the installation and configuration of Jdk and Eclipse. A long-winded sentence here, a useful browser is very important for crawlers, because first you need to browse the web yourself to know where the things you need are, and then you can tell your crawlers where to go and how to crawl. I personally recommend Firefox or Google Chrome. Their functions of right-clicking to inspect elements and viewing source code are very powerful.

Now we start the official crawler journey! ~What should I talk about specifically, well, this is a question, let me think about it, don’t worry^_^

The above is the preparation work for writing Java Zhihu crawler from scratch. For more related content, please Follow the PHP Chinese website (www.php.cn)!