search
HomeBackend DevelopmentPython TutorialShare an example of using Python crawler to simulate Zhihu login

During the crawling process, some pages are prohibited from being crawled before logging in. At this time, you need to simulate the login. The following article mainly introduces how to use the Python crawler to simulate Zhihu login. The method tutorial is very detailed in the article. Friends in need can refer to it. Let’s take a look together.

Preface

Everyone who often writes crawlers knows that some pages are prohibited from crawling before logging in, such as Zhihu’s The topic page requires users to log in to access, and "login" is inseparable from the Cookie technology in HTTP.

Login Principle

The principle of Cookie is very simple, because HTTP is a stateless protocol, so in order to use the stateless HTTP protocol On top of that, the session state is maintained to let the server know which client it is currently dealing with. Cookie technology appeared. Cookie is equivalent to an identifier assigned by the server to the client.

  • When the browser initiates an HTTP request for the first time, it does not carry any Cookie information

  • The server The HTTP response, along with a Cookie information, is returned to the browser together

  • The browser will send the Cookie information returned by the server to the server together with the second request

  • The server receives the HTTP request and finds that there is a Cookie field in the request header, so it knows that it has dealt with this user before.

Practical application

Anyone who has used Zhihu knows that as long as you provide the user name and password and verification code to log in. Of course, this is just what we see. The technical details hidden behind need to be discovered with the help of a browser. Now let’s use Chrome to see what happens after we fill out the form?

(If you are already logged in, log out first) First enter the Zhihu login page www.zhihu.com/#signin and open the Chrome developer toolbar (press F12 ) First try to enter an incorrect verification code and observe how the browser sends the request.

Several key information can be found from the browser request

  • The login URL address is https://www. zhihu.com/login/email

  • There are four form data required for login: username (email), password (password), verification code (captcha), and _xsrf.

  • The URL address to obtain the verification code is https://www.zhihu.com/captcha.gif?r=1490690391695&type=login

_xsrfWhat is it? If you are very familiar with CSRF (cross-site request forgery) attacks, then you must know its role. xsrf is a string of pseudo-random numbers, which is used to prevent cross-site request forgery. It usually exists in the form form tag of the web page. To confirm this, you can search for "xsrf" on the page. Sure enough, _xsrf is in a hidden input tag

After clarifying how to obtain the data required for browser login, you can now start writing code to simulate browser login using Python. The two third-party libraries relied on when logging in are requests and BeautifulSoup. FirstInstall

pip install beautifulsoup4==4.5.3
pip install requests==2.13.0

http.cookiejar module can be used to automatically process HTTP Cookies. The LWPCookieJar object is the encapsulation of cookies. Supports saving cookies to files and loading them from files.

The session object provides cookie persistence and connection pooling functions. Requests can be sent through the session object.

First load the cookie information from the cookies.txt file, because there is no cookie for the first run. A LoadError exception will occur.

from http import cookiejar
session = requests.session()
session.cookies = cookiejar.LWPCookieJar(filename='cookies.txt')
try:
 session.cookies.load(ignore_discard=True)
except LoadError:
 print("load cookies failed")

Get xsrf

The tag where xsrf is located has been found earlier, and the value can be obtained very conveniently by using the find method of BeatifulSoup


def get_xsrf():
 response = session.get("https://www.zhihu.com", headers=headers)
 soup = BeautifulSoup(response.content, "html.parser")
 xsrf = soup.find('input', attrs={"name": "_xsrf"}).get("value")
 return xsrf

Get the verification code

The verification code is returned through the /captcha.gif interface, here we Download and save the verification code image to the current directory for manual identification. Of course, you can use a third-party support library to automatically identify it, such as pytesser.

def get_captcha():
 """
 把验证码图片保存到当前目录,手动识别验证码
 :return:
 """
 t = str(int(time.time() * 1000))
 captcha_url = 'https://www.zhihu.com/captcha.gif?r=' + t + "&type=login"
 r = session.get(captcha_url, headers=headers)
 with open('captcha.jpg', 'wb') as f:
  f.write(r.content)
 captcha = input("验证码:")
 return captcha

Login

一切参数准备就绪之后,就可以请求登录接口了。

def login(email, password):
 login_url = 'www.zhihu.com/login/email'
 data = {
  'email': email,
  'password': password,
  '_xsrf': get_xsrf(),
  "captcha": get_captcha(),
  'remember_me': 'true'}
 response = session.post(login_url, data=data, headers=headers)
 login_code = response.json()
 print(login_code['msg'])
 for i in session.cookies:
  print(i)
 session.cookies.save()

请求成功后,session 会自动把 服务端的返回的cookie 信息填充到 session.cookies 对象中,下次请求时,客户端就可以自动携带这些cookie去访问那些需要登录的页面了。

auto_login.py 示例代码

# encoding: utf-8
# !/usr/bin/env python
"""
作者:liuzhijun
"""
import time
from http import cookiejar

import requests
from bs4 import BeautifulSoup

headers = {
  "Host": "www.zhihu.com",
  "Referer": "www.zhihu.com/",
  'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87'
}

# 使用登录cookie信息
session = requests.session()
session.cookies = cookiejar.LWPCookieJar(filename='cookies.txt')
try:
  print(session.cookies)
  session.cookies.load(ignore_discard=True)

except:
  print("还没有cookie信息")


def get_xsrf():
  response = session.get("www.zhihu.com", headers=headers)
  soup = BeautifulSoup(response.content, "html.parser")
  xsrf = soup.find('input', attrs={"name": "_xsrf"}).get("value")
  return xsrf


def get_captcha():
  """
  把验证码图片保存到当前目录,手动识别验证码
  :return:
  """
  t = str(int(time.time() * 1000))
  captcha_url = 'www.zhihu.com/captcha.gif?r=' + t + "&type=login"
  r = session.get(captcha_url, headers=headers)
  with open('captcha.jpg', 'wb') as f:
    f.write(r.content)
  captcha = input("验证码:")
  return captcha


def login(email, password):
  login_url = 'www.zhihu.com/login/email'
  data = {
    'email': email,
    'password': password,
    '_xsrf': get_xsrf(),
    "captcha": get_captcha(),
    'remember_me': 'true'}
  response = session.post(login_url, data=data, headers=headers)
  login_code = response.json()
  print(login_code['msg'])
  for i in session.cookies:
    print(i)
  session.cookies.save()


if name == 'main':
  email = "xxxx"
  password = "xxxxx"
  login(email, password)

【相关推荐】

1. python爬虫入门(4)--详解HTML文本的解析库BeautifulSoup

2. python爬虫入门(3)--利用requests构建知乎API

3. python爬虫入门(2)--HTTP库requests

4.  python爬虫入门(1)--快速理解HTTP协议

5. python爬虫入门(5)--正则表达式实例教程

The above is the detailed content of Share an example of using Python crawler to simulate Zhihu login. For more information, please follow other related articles on the PHP Chinese website!

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
What is Python Switch Statement?What is Python Switch Statement?Apr 30, 2025 pm 02:08 PM

The article discusses Python's new "match" statement introduced in version 3.10, which serves as an equivalent to switch statements in other languages. It enhances code readability and offers performance benefits over traditional if-elif-el

What are Exception Groups in Python?What are Exception Groups in Python?Apr 30, 2025 pm 02:07 PM

Exception Groups in Python 3.11 allow handling multiple exceptions simultaneously, improving error management in concurrent scenarios and complex operations.

What are Function Annotations in Python?What are Function Annotations in Python?Apr 30, 2025 pm 02:06 PM

Function annotations in Python add metadata to functions for type checking, documentation, and IDE support. They enhance code readability, maintenance, and are crucial in API development, data science, and library creation.

What are unit tests in Python?What are unit tests in Python?Apr 30, 2025 pm 02:05 PM

The article discusses unit tests in Python, their benefits, and how to write them effectively. It highlights tools like unittest and pytest for testing.

What are Access Specifiers in Python?What are Access Specifiers in Python?Apr 30, 2025 pm 02:03 PM

Article discusses access specifiers in Python, which use naming conventions to indicate visibility of class members, rather than strict enforcement.

What is __init__() in Python and how does self play a role in it?What is __init__() in Python and how does self play a role in it?Apr 30, 2025 pm 02:02 PM

Article discusses Python's \_\_init\_\_() method and self's role in initializing object attributes. Other class methods and inheritance's impact on \_\_init\_\_() are also covered.

What is the difference between @classmethod, @staticmethod and instance methods in Python?What is the difference between @classmethod, @staticmethod and instance methods in Python?Apr 30, 2025 pm 02:01 PM

The article discusses the differences between @classmethod, @staticmethod, and instance methods in Python, detailing their properties, use cases, and benefits. It explains how to choose the right method type based on the required functionality and da

How do you append elements to a Python array?How do you append elements to a Python array?Apr 30, 2025 am 12:19 AM

InPython,youappendelementstoalistusingtheappend()method.1)Useappend()forsingleelements:my_list.append(4).2)Useextend()or =formultipleelements:my_list.extend(another_list)ormy_list =[4,5,6].3)Useinsert()forspecificpositions:my_list.insert(1,5).Beaware

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

MantisBT

MantisBT

Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.

EditPlus Chinese cracked version

EditPlus Chinese cracked version

Small size, syntax highlighting, does not support code prompt function

SublimeText3 English version

SublimeText3 English version

Recommended: Win version, supports code prompts!

SublimeText3 Linux new version

SublimeText3 Linux new version

SublimeText3 Linux latest version

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor