Home > Article > Backend Development > Detailed explanation of advanced usage of the third-party library Requests library in Python
Although Python’s urllib2 module in the standard library already contains most of the functions we usually use, its API makes people feel uncomfortable to use it. It is no longer suitable for the current era and the modern Internet. The birth of Requests gives us a better choice. This article introduces the advanced usage of the third-party library Requests library in Python.
1. Installation of the Requests library
#Use pip to install, if you have installed the pip package (a Python package management tool , I don’t know if you can use Baidu), or an integrated environment, such as Python (x, y)
or anaconda, you can directly use pip to install the Python library.
$ pip install requests
After the installation is completed, let’s take a look at the basic method:
#get请求方法 >>> r = requests.get('https://api.github.com/user', auth=('user', 'pass')) #打印get请求的状态码 >>> r.status_code 200 #查看请求的数据类型,可以看到是json格式,utf-8编码 >>> r.headers['content-type'] 'application/json; charset=utf8' >>> r.encoding 'utf-8' #打印请求到的内容 >>> r.text u'{"type":"User"...' #输出json格式数据 >>> r.json() {u'private_gists': 419, u'total_private_repos': 77, ...}
Let’s take a look at a small chestnut:
#小例子 import requests r = requests.get('http://www.baidu.com') print type(r) print r.status_code print r.encoding print r.text print r.cookies '''请求了百度的网址,然后打印出了返回结果的类型,状态码,编码方式,Cookies等内容 输出:''' <class 'requests.models.Response'> 200 UTF-8 <RequestsCookieJar[]>
2. http basic requests
The requests library provides all basic http requests Request method. For example:
r = requests.post("http://httpbin.org/post") r = requests.put("http://httpbin.org/put") r = requests.delete("http://httpbin.org/delete") r = requests.head("http://httpbin.org/get") r = requests.options(<a rel="external nofollow" href="http://httpbin.org/get">http://httpbin.org/get</a>)
Basic GET request
##
r = requests.get("http://httpbin.org/get") #如果想要加参数,可以利用 params 参数: import requests payload = {'key1': 'value1', 'key2': 'value2'} r = requests.get("http://httpbin.org/get", params=payload) print r.url #输出:http://www.php.cn/If you want to request a JSON file, you can use
json() Method analysis, for example, write a JSON file named a.json with the following content:
##
["foo", "bar", { "foo": "bar" }] #利用如下程序请求并解析: import requests r = requests.get("a.json") print r.text print r.json() '''运行结果如下,其中一个是直接输出内容,另外一个方法是利用 json() 方法 解析,感受下它们的不同:''' ["foo", "bar", { "foo": "bar" }] [u'foo', u'bar', {u'foo': u'bar'}]
. However, stream=True
needs to be set in the initial request.
r = requests.get('https://github.com/timeline.json', stream=True) r.raw #输出 <requests.packages.urllib3.response.HTTPResponse object at 0x101194810> r.raw.read(10) '\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03'
If you want to add headers, you can pass headers parameters:
import requests payload = {'key1': 'value1', 'key2': 'value2'} headers = {'content-type': 'application/json'} r = requests.get("http://httpbin.org/get", params=payload, headers=headers) print r.url #通过headers参数可以增加请求头中的headers信息
3. Basic POST request
For POST requests, we generally need to add some parameters to it. Then the most basic parameter passing method can use the data parameter.
import requests payload = {'key1': 'value1', 'key2': 'value2'} r = requests.post("http://httpbin.org/post", data=payload) print r.text #运行结果如下: { "args": {}, "data": "", "files": {}, "form": { "key1": "value1", "key2": "value2" }, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Content-Length": "23", "Content-Type": "application/x-www-form-urlencoded", "Host": "http://httpbin.org", "User-Agent": "python-requests/2.9.1" }, "json": null, "url": "http://httpbin.org/post" }
Sometimes the information we need to send is not in the form of a form. We need to send data in JSON format, so we can use the
json.dumps() method to serialize the form data.
import json import requests url = 'http://httpbin.org/post' payload = {'some': 'data'} r = requests.post(url, data=json.dumps(payload)) print r.text #运行结果: { "args": {}, "data": "{\"some\": \"data\"}", "files": {}, "form": {}, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Content-Length": "16", "Host": "http://httpbin.org", "User-Agent": "python-requests/2.9.1" }, "json": { "some": "data" }, "url": "http://httpbin.org/post" }
If you want to upload a file, just use the file parameter directly:
#新建一个 test.txt 的文件,内容写上 Hello World! import requests url = 'http://httpbin.org/post' files = {'file': open('test.txt', 'rb')} r = requests.post(url, files=files) print r.text { "args": {}, "data": "", "files": { "file": "Hello World!" }, "form": {}, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Content-Length": "156", "Content-Type": "multipart/form-data; boundary=7d8eb5ff99a04c11bb3e862ce78d7000", "Host": "http://httpbin.org", "User-Agent": "python-requests/2.9.1" }, "json": null, "url": "http://httpbin.org/post" }
requests supports streaming uploads, which allows you to send large data streams or files without reading them into memory first. To use streaming upload, you only need to provide a class file
objectfor your request body, which is very convenient:
with open('massive-body') as f: requests.post('http://some.url/streamed', data=f)
4. Cookies
If a response contains cookies, then we can use cookies
import requests url = 'Example Domain' r = requests.get(url) print r.cookies print r.cookies['example_cookie_name']
In addition, you can use the cookies variable to send cookie information to the server:
import requests url = 'http://httpbin.org/cookies' cookies = dict(cookies_are='working') r = requests.get(url, cookies=cookies) print r.text #输出: '{"cookies": {"cookies_are": "working"}}'
5. Timeout configuration##You can use the timeout variable to configure the maximum request time
requests.get(‘Build software better, together', timeout=0.001)
timeout is only valid for the connection process and has nothing to do with the download of the response body. In other words, this time only limits the requested time. Even if the returned response contains a large amount of content, it will take some time to download.
6. Session objectIn the above requests, each request is actually equivalent to initiating a new request. . This is equivalent to the effect of using a different browser to open each request separately. That is, it does not refer to a session, even if the same URL is requested. For example:
import requests requests.get('http://httpbin.org/cookies/set/sessioncookie/123456789') r = requests.get("http://httpbin.org/cookies") print(r.text) #结果是: { "cookies": {} }
Obviously, this is not in a session and cookies cannot be obtained. So what should we do if we need to maintain a persistent session in some sites? Just like using a browser to browse Taobao, jumping between different tabs actually creates a long-term session.
import requests s = requests.Session() s.get('http://httpbin.org/cookies/set/sessioncookie/123456789') r = s.get("http://httpbin.org/cookies") print(r.text) #在这里我们请求了两次,一次是设置 cookies,一次是获得 cookies { "cookies": { "sessioncookie": "123456789" } }
It is found that cookies can be successfully obtained. This is to establish a session.
import requests s = requests.Session() s.headers.update({'x-test': 'true'}) r = s.get('http://httpbin.org/headers', headers={'x-test2': 'true'}) print r.text '''通过 s.headers.update 方法设置了 headers 的变量。然后我们又在请求中 设置了一个 headers,那么会出现什么结果?很简单,两个变量都传送过去了。 运行结果:''' { "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Host": "http://httpbin.org", "User-Agent": "python-requests/2.9.1", "X-Test": "true", "X-Test2": "true" } }
What if the headers passed by the get method are also x-test?
r = s.get('http://httpbin.org/headers', headers={'x-test': 'true'}) #它会覆盖掉全局的配置: { "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Host": "http://httpbin.org", "User-Agent": "python-requests/2.9.1", "X-Test": "true" } }
如果不想要全局配置中的一个变量了呢?很简单,设置为 None 即可。
r = s.get('http://httpbin.org/headers', headers={'x-test': None}) { "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Host": "http://httpbin.org", "User-Agent": "python-requests/2.9.1" } }
以上就是 session 会话的基本用法。
七、SSL证书验证
现在随处可见 https 开头的网站,Requests可以为HTTPS请求验证SSL证书,就像web浏览器一样。要想检查某个主机的SSL证书,你可以使用 verify 参数,因为前段时间12306 证书不是无效的嘛,来测试一下:
import requests r = requests.get('https://kyfw.12306.cn/otn/', verify=True) print r.text #结果: requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
来试下 github 的:
import requests r = requests.get('Build software better, together', verify=True) print r.text
嗯,正常请求,由于内容太多,我就不粘贴输出了。
如果我们想跳过刚才 12306 的证书验证,把 verify 设置为 False 即可:
import requests r = requests.get('https://kyfw.12306.cn/otn/', verify=False) print r.text
发现就可以正常请求了。在默认情况下 verify 是 True,所以如果需要的话,需要手动设置下这个变量。
八、代理
如果需要使用代理,你可以通过为任意请求方法提供 proxies 参数来配置单个请求。
import requests proxies = { "https": "http://41.118.132.69:4433" } r = requests.post("http://httpbin.org/post", proxies=proxies) print r.text #也可以通过环境变量 HTTP_PROXY 和 HTTPS_PROXY 来配置代理 export HTTP_PROXY="http://10.10.1.10:3128" export HTTPS_PROXY=<a href="http://10.10.1.10:1080">http://10.10.1.10:1080</a>
The above is the detailed content of Detailed explanation of advanced usage of the third-party library Requests library in Python. For more information, please follow other related articles on the PHP Chinese website!