I talked about the use and methods of Python's urllib library, the basic use of Python network data collection Urllib library, and the advanced usage of Python's urllib.
Today we will learn how to use the Requests library in Python.
Installation of Requests library
Use pip to install. If you have installed the pip package (a Python package management tool, I don’t know if you can use Baidu), or an integrated environment, such as Python (x, y) or anaconda, you can directly use pip to install the Python library.
$ pip install requests
After the installation is completed, let’s take a look at the basic method:
#get请求方法 >>> r = requests.get('https://api.github.com/user', auth=('user', 'pass')) #打印get请求的状态码 >>> r.status_code 200 #查看请求的数据类型,可以看到是json格式,utf-8编码 >>> r.headers['content-type'] 'application/json; charset=utf8' >>> r.encoding 'utf-8' #打印请求到的内容 >>> r.text u'{"type":"User"...' #输出json格式数据 >>> r.json() {u'private_gists': 419, u'total_private_repos': 77, ...}
Let’s take a look at a small chestnut:
#小例子 import requests r = requests.get('http://www.baidu.com') print type(r) print r.status_code print r.encoding print r.text print r.cookies '''请求了百度的网址,然后打印出了返回结果的类型,状态码,编码方式,Cookies等内容 输出:''' <class 'requests.models.Response'> 200 UTF-8 <RequestsCookieJar[]>
http basic request
requests library Provides all basic request methods of http. For example:
r = requests.post("http://httpbin.org/post") r = requests.put("http://httpbin.org/put") r = requests.delete("http://httpbin.org/delete") r = requests.head("http://httpbin.org/get") r = requests.options("http://httpbin.org/get")
Basic GET request
r = requests.get("http://httpbin.org/get") #如果想要加参数,可以利用 params 参数: import requests payload = {'key1': 'value1', 'key2': 'value2'} r = requests.get("http://httpbin.org/get", params=payload) print r.url #输出:http://httpbin.org/get?key2=value2&key1=value1
If you want to request a JSON file, you can use the json() method to parse it. For example, write a JSON file yourself and name it a.json with the following content:
["foo", "bar", { "foo": "bar" }] #利用如下程序请求并解析: import requests r = requests.get("a.json") print r.text print r.json() '''运行结果如下,其中一个是直接输出内容,另外一个方法是利用 json() 方法 解析,感受下它们的不同:''' ["foo", "bar", { "foo": "bar" }] [u'foo', u'bar', {u'foo': u'bar'}]
If you want to get the raw socket response from the server, you can get r.raw. However, stream=True needs to be set in the initial request.
r = requests.get('https://github.com/timeline.json', stream=True) r.raw #输出 <requests.packages.urllib3.response.HTTPResponse object at 0x101194810> r.raw.read(10) '\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03'
In this way, the original socket content of the web page is obtained.
If you want to add headers, you can pass the headers parameter:
import requests payload = {'key1': 'value1', 'key2': 'value2'} headers = {'content-type': 'application/json'} r = requests.get("http://httpbin.org/get", params=payload, headers=headers) print r.url #通过headers参数可以增加请求头中的headers信息
Basic POST request
For POST requests, we generally need to add some parameters to it. Then the most basic parameter passing method can use the data parameter.
import requests payload = {'key1': 'value1', 'key2': 'value2'} r = requests.post("http://httpbin.org/post", data=payload) print r.text #运行结果如下: { "args": {}, "data": "", "files": {}, "form": { "key1": "value1", "key2": "value2" }, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Content-Length": "23", "Content-Type": "application/x-www-form-urlencoded", "Host": "http://httpbin.org", "User-Agent": "python-requests/2.9.1" }, "json": null, "url": "http://httpbin.org/post" }
You can see that the parameters were passed successfully, and then the server returned the data we passed.
Sometimes the information we need to send is not in the form of a form. We need to send data in JSON format, so we can use the json.dumps() method to serialize the form data.
import json import requests url = 'http://httpbin.org/post' payload = {'some': 'data'} r = requests.post(url, data=json.dumps(payload)) print r.text #运行结果: { "args": {}, "data": "{\"some\": \"data\"}", "files": {}, "form": {}, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Content-Length": "16", "Host": "http://httpbin.org", "User-Agent": "python-requests/2.9.1" }, "json": { "some": "data" }, "url": "http://httpbin.org/post" }
Through the above method, we can POST data in JSON format
If you want to upload a file, just use the file parameter directly:
#新建一个 test.txt 的文件,内容写上 Hello World! import requests url = 'http://httpbin.org/post' files = {'file': open('test.txt', 'rb')} r = requests.post(url, files=files) print r.text { "args": {}, "data": "", "files": { "file": "Hello World!" }, "form": {}, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Content-Length": "156", "Content-Type": "multipart/form-data; boundary=7d8eb5ff99a04c11bb3e862ce78d7000", "Host": "http://httpbin.org", "User-Agent": "python-requests/2.9.1" }, "json": null, "url": "http://httpbin.org/post" }
In this way we will complete it successfully uploaded a file.
requests supports streaming uploads, which allows you to send large data streams or files without reading them into memory first. To use streaming upload, you only need to provide a file-like object for your request body, which is very convenient:
with open('massive-body') as f: requests.post('http://some.url/streamed', data=f)
Cookies
If a response contains cookies, then we can use Cookies variable to get:
import requests url = 'Example Domain' r = requests.get(url) print r.cookies print r.cookies['example_cookie_name']
The above program is just a sample, you can use the cookies variable to get the cookies of the site
In addition, you can use the cookies variable to send cookie information to the server:
import requests url = 'http://httpbin.org/cookies' cookies = dict(cookies_are='working') r = requests.get(url, cookies=cookies) print r.text #输出: '{"cookies": {"cookies_are": "working"}}'
Timeout configuration
You can use the timeout variable to configure the maximum request time
requests.get(‘Build software better, together’, timeout=0.001)
Note: timeout is only valid for the connection process and has nothing to do with the download of the response body.
In other words, this time only limits the requested time. Even if the returned response contains a large amount of content, it will take some time to download.
Session Object
In the above requests, each request is actually equivalent to initiating a new request. This is equivalent to the effect of using a different browser to open each request separately. That is, it does not refer to a session, even if the same URL is requested. For example:
import requests requests.get('http://httpbin.org/cookies/set/sessioncookie/123456789') r = requests.get("http://httpbin.org/cookies") print(r.text) #结果是: { "cookies": {} }
Obviously, this is not in a session and cookies cannot be obtained. So what should we do if we need to maintain a persistent session on some sites? Just like using a browser to browse Taobao, jumping between different tabs actually creates a long-term session.
The solution is as follows:
import requests s = requests.Session() s.get('http://httpbin.org/cookies/set/sessioncookie/123456789') r = s.get("http://httpbin.org/cookies") print(r.text) #在这里我们请求了两次,一次是设置 cookies,一次是获得 cookies { "cookies": { "sessioncookie": "123456789" } }
It is found that cookies can be successfully obtained. This is to establish a session.
So since the session is a global variable, we can definitely use it for global configuration.
import requests s = requests.Session() s.headers.update({'x-test': 'true'}) r = s.get('http://httpbin.org/headers', headers={'x-test2': 'true'}) print r.text '''通过 s.headers.update 方法设置了 headers 的变量。然后我们又在请求中 设置了一个 headers,那么会出现什么结果?很简单,两个变量都传送过去了。 运行结果:''' { "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Host": "http://httpbin.org", "User-Agent": "python-requests/2.9.1", "X-Test": "true", "X-Test2": "true" } }
What if the headers passed by the get method are also x-test?
r = s.get('http://httpbin.org/headers', headers={'x-test': 'true'}) #它会覆盖掉全局的配置: { "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Host": "http://httpbin.org", "User-Agent": "python-requests/2.9.1", "X-Test": "true" } }
What if you don’t want a variable in the global configuration? It's easy, just set it to None.
r = s.get('http://httpbin.org/headers', headers={'x-test': None}) { "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Host": "http://httpbin.org", "User-Agent": "python-requests/2.9.1" } }
The above is the basic usage of session session.
SSL Certificate Verification
Now that you can see websites starting with https everywhere, Requests can verify SSL certificates for HTTPS requests, just like a web browser. To check the SSL certificate of a certain host, you can use the verify parameter, because the 12306 certificate was not invalid some time ago. Let’s test it:
import requests r = requests.get('https://kyfw.12306.cn/otn/', verify=True) print r.text #结果: requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
Let’s try github’s:
import requests r = requests.get('Build software better, together', verify=True) print r.text
Well, normal request, because there is too much content, I won’t paste the output.
If we want to skip the certificate verification of 12306 just now, set verify to False:
import requests r = requests.get('https://kyfw.12306.cn/otn/', verify=False) print r.text
Once found, the request can be made normally. By default verify is True, so you need to set this variable manually if necessary.
Proxy
If you need to use a proxy, you can configure individual requests by providing the proxies parameter to any request method.
import requests proxies = { "https": "http://41.118.132.69:4433" } r = requests.post("http://httpbin.org/post", proxies=proxies) print r.text #也可以通过环境变量 HTTP_PROXY 和 HTTPS_PROXY 来配置代理 export HTTP_PROXY="http://10.10.1.10:3128" export HTTPS_PROXY="http://10.10.1.10:1080"
##
The above is the detailed content of How to use the Requests library in Python. For more information, please follow other related articles on the PHP Chinese website!

本篇文章给大家带来了关于Python的相关知识,其中主要介绍了关于Seaborn的相关问题,包括了数据可视化处理的散点图、折线图、条形图等等内容,下面一起来看一下,希望对大家有帮助。

本篇文章给大家带来了关于Python的相关知识,其中主要介绍了关于进程池与进程锁的相关问题,包括进程池的创建模块,进程池函数等等内容,下面一起来看一下,希望对大家有帮助。

本篇文章给大家带来了关于Python的相关知识,其中主要介绍了关于简历筛选的相关问题,包括了定义 ReadDoc 类用以读取 word 文件以及定义 search_word 函数用以筛选的相关内容,下面一起来看一下,希望对大家有帮助。

VS Code的确是一款非常热门、有强大用户基础的一款开发工具。本文给大家介绍一下10款高效、好用的插件,能够让原本单薄的VS Code如虎添翼,开发效率顿时提升到一个新的阶段。

本篇文章给大家带来了关于Python的相关知识,其中主要介绍了关于数据类型之字符串、数字的相关问题,下面一起来看一下,希望对大家有帮助。

pythn的中文意思是巨蟒、蟒蛇。1989年圣诞节期间,Guido van Rossum在家闲的没事干,为了跟朋友庆祝圣诞节,决定发明一种全新的脚本语言。他很喜欢一个肥皂剧叫Monty Python,所以便把这门语言叫做python。

本篇文章给大家带来了关于Python的相关知识,其中主要介绍了关于numpy模块的相关问题,Numpy是Numerical Python extensions的缩写,字面意思是Python数值计算扩展,下面一起来看一下,希望对大家有帮助。


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

PhpStorm Mac version
The latest (2018.2.1) professional PHP integrated development tool

Safe Exam Browser
Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

SublimeText3 English version
Recommended: Win version, supports code prompts!

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)
