Home > Article > Backend Development > Python resists anti-crawlers by disguising header data
This article mainly introduces how python resists anti-crawlers by disguising header data. It has certain reference value. Now I share it with you. Friends in need can refer to it
0x00 Environment
System environment: win10
Writing tool: JetBrains PyCharm Community Edition 2017.1.2 x64
python version: python-3.6.2
Packet capture tool: Fiddler 4
0x01 Header data disguise idea
Submit data to the server through http, the following is Capture python's undisguised packet header information through Fiddler
GET /u012870721 HTTP/1.1 Accept-Encoding: identity Host: blog.csdn.net User-Agent: <span style="color:#ff0000;">Python-urllib/3.6</span> Connection: close
Python-urllib/3.6
Obviously, we are exposed. Now I have to ask, what should I do? Simulate the browser and pretend to be the browser. Here is the header data sent by the browser access
Connection: keep-alive Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 Referer: http://write.blog.csdn.net/postlist Accept-Encoding: gzip, deflate Accept-Language: zh-CN,zh;q=0.8
0x02 Code Implementation
from urllib import request html_url = "http://blog.csdn.net/u012870721"; #伪装构造头 header ={ "Connection": "keep-alive", "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36", "Accept":" text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8", "Accept-Encoding": "gzip,deflate", "Accept-Language": "zh-CN,zh;q=0.8" }; #int main() #{ req = request.Request(url=html_url, headers=header); resp = request.urlopen(req); # return 0; # }
The information header sent after being disguised
GET /u012870721 HTTP/1.1 Host: blog.csdn.net Connection: close Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8 Accept-Encoding: gzip,deflate Accept-Language: zh-CN,zh;q=0.8
Related recommendations:
Some deployment commands for Centos Python production environment
The above is the detailed content of Python resists anti-crawlers by disguising header data. For more information, please follow other related articles on the PHP Chinese website!