


During the crawling process, some pages are prohibited from being crawled before logging in. At this time, you need to simulate the login. The following article mainly introduces how to use the Python crawler to simulate Zhihu login. The method tutorial is very detailed in the article. Friends in need can refer to it. Let’s take a look together.
Preface
Everyone who often writes crawlers knows that some pages are prohibited from crawling before logging in, such as Zhihu’s The topic page requires users to log in to access, and "login" is inseparable from the Cookie technology in HTTP.
Login Principle
The principle of Cookie is very simple, because HTTP is a stateless protocol, so in order to use the stateless HTTP protocol On top of that, the session state is maintained to let the server know which client it is currently dealing with. Cookie technology appeared. Cookie is equivalent to an identifier assigned by the server to the client.
When the browser initiates an HTTP request for the first time, it does not carry any Cookie information
The server The HTTP response, along with a Cookie information, is returned to the browser together
The browser will send the Cookie information returned by the server to the server together with the second request
The server receives the HTTP request and finds that there is a Cookie field in the request header, so it knows that it has dealt with this user before.
Practical application
Anyone who has used Zhihu knows that as long as you provide the user name and password and verification code to log in. Of course, this is just what we see. The technical details hidden behind need to be discovered with the help of a browser. Now let’s use Chrome to see what happens after we fill out the form?
(If you are already logged in, log out first) First enter the Zhihu login page www.zhihu.com/#signin and open the Chrome developer toolbar (press F12 ) First try to enter an incorrect verification code and observe how the browser sends the request.
Several key information can be found from the browser request
The login URL address is https://www. zhihu.com/login/email
There are four form data required for login: username (email), password (password), verification code (captcha), and _xsrf.
The URL address to obtain the verification code is https://www.zhihu.com/captcha.gif?r=1490690391695&type=login
_xsrfWhat is it? If you are very familiar with CSRF (cross-site request forgery) attacks, then you must know its role. xsrf is a string of pseudo-random numbers, which is used to prevent cross-site request forgery. It usually exists in the form form tag of the web page. To confirm this, you can search for "xsrf" on the page. Sure enough, _xsrf is in a hidden input tag
After clarifying how to obtain the data required for browser login, you can now start writing code to simulate browser login using Python. The two third-party libraries relied on when logging in are requests and BeautifulSoup. FirstInstall
pip install beautifulsoup4==4.5.3 pip install requests==2.13.0
http.cookiejar module can be used to automatically process HTTP Cookies. The LWPCookieJar object is the encapsulation of cookies. Supports saving cookies to files and loading them from files.
The session object provides cookie persistence and connection pooling functions. Requests can be sent through the session object.
First load the cookie information from the cookies.txt file, because there is no cookie for the first run. A LoadError exception will occur.
from http import cookiejar session = requests.session() session.cookies = cookiejar.LWPCookieJar(filename='cookies.txt') try: session.cookies.load(ignore_discard=True) except LoadError: print("load cookies failed")
Get xsrf
The tag where xsrf is located has been found earlier, and the value can be obtained very conveniently by using the find method of BeatifulSoup
def get_xsrf(): response = session.get("https://www.zhihu.com", headers=headers) soup = BeautifulSoup(response.content, "html.parser") xsrf = soup.find('input', attrs={"name": "_xsrf"}).get("value") return xsrf
Get the verification code
The verification code is returned through the /captcha.gif interface, here we Download and save the verification code image to the current directory for manual identification. Of course, you can use a third-party support library to automatically identify it, such as pytesser.
def get_captcha(): """ 把验证码图片保存到当前目录,手动识别验证码 :return: """ t = str(int(time.time() * 1000)) captcha_url = 'https://www.zhihu.com/captcha.gif?r=' + t + "&type=login" r = session.get(captcha_url, headers=headers) with open('captcha.jpg', 'wb') as f: f.write(r.content) captcha = input("验证码:") return captcha
Login
一切参数准备就绪之后,就可以请求登录接口了。
def login(email, password): login_url = 'www.zhihu.com/login/email' data = { 'email': email, 'password': password, '_xsrf': get_xsrf(), "captcha": get_captcha(), 'remember_me': 'true'} response = session.post(login_url, data=data, headers=headers) login_code = response.json() print(login_code['msg']) for i in session.cookies: print(i) session.cookies.save()
请求成功后,session 会自动把 服务端的返回的cookie 信息填充到 session.cookies 对象中,下次请求时,客户端就可以自动携带这些cookie去访问那些需要登录的页面了。
auto_login.py 示例代码
# encoding: utf-8 # !/usr/bin/env python """ 作者:liuzhijun """ import time from http import cookiejar import requests from bs4 import BeautifulSoup headers = { "Host": "www.zhihu.com", "Referer": "www.zhihu.com/", 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87' } # 使用登录cookie信息 session = requests.session() session.cookies = cookiejar.LWPCookieJar(filename='cookies.txt') try: print(session.cookies) session.cookies.load(ignore_discard=True) except: print("还没有cookie信息") def get_xsrf(): response = session.get("www.zhihu.com", headers=headers) soup = BeautifulSoup(response.content, "html.parser") xsrf = soup.find('input', attrs={"name": "_xsrf"}).get("value") return xsrf def get_captcha(): """ 把验证码图片保存到当前目录,手动识别验证码 :return: """ t = str(int(time.time() * 1000)) captcha_url = 'www.zhihu.com/captcha.gif?r=' + t + "&type=login" r = session.get(captcha_url, headers=headers) with open('captcha.jpg', 'wb') as f: f.write(r.content) captcha = input("验证码:") return captcha def login(email, password): login_url = 'www.zhihu.com/login/email' data = { 'email': email, 'password': password, '_xsrf': get_xsrf(), "captcha": get_captcha(), 'remember_me': 'true'} response = session.post(login_url, data=data, headers=headers) login_code = response.json() print(login_code['msg']) for i in session.cookies: print(i) session.cookies.save() if name == 'main': email = "xxxx" password = "xxxxx" login(email, password)
【相关推荐】
1. python爬虫入门(4)--详解HTML文本的解析库BeautifulSoup
2. python爬虫入门(3)--利用requests构建知乎API
3. python爬虫入门(2)--HTTP库requests
The above is the detailed content of Share an example of using Python crawler to simulate Zhihu login. For more information, please follow other related articles on the PHP Chinese website!

The article discusses Python's new "match" statement introduced in version 3.10, which serves as an equivalent to switch statements in other languages. It enhances code readability and offers performance benefits over traditional if-elif-el

Exception Groups in Python 3.11 allow handling multiple exceptions simultaneously, improving error management in concurrent scenarios and complex operations.

Function annotations in Python add metadata to functions for type checking, documentation, and IDE support. They enhance code readability, maintenance, and are crucial in API development, data science, and library creation.

The article discusses unit tests in Python, their benefits, and how to write them effectively. It highlights tools like unittest and pytest for testing.

Article discusses access specifiers in Python, which use naming conventions to indicate visibility of class members, rather than strict enforcement.

Article discusses Python's \_\_init\_\_() method and self's role in initializing object attributes. Other class methods and inheritance's impact on \_\_init\_\_() are also covered.

The article discusses the differences between @classmethod, @staticmethod, and instance methods in Python, detailing their properties, use cases, and benefits. It explains how to choose the right method type based on the required functionality and da

InPython,youappendelementstoalistusingtheappend()method.1)Useappend()forsingleelements:my_list.append(4).2)Useextend()or =formultipleelements:my_list.extend(another_list)ormy_list =[4,5,6].3)Useinsert()forspecificpositions:my_list.insert(1,5).Beaware


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

MantisBT
Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.

EditPlus Chinese cracked version
Small size, syntax highlighting, does not support code prompt function

SublimeText3 English version
Recommended: Win version, supports code prompts!

SublimeText3 Linux new version
SublimeText3 Linux latest version

Notepad++7.3.1
Easy-to-use and free code editor
