python - Use requests to crawl [Dajie.com] job information, but failed many times. Can you help me find out what is wrong with my code? How should it be changed?
I have crawled several static website data before, and everything was relatively smooth. This time I encountered ajax, and after reading a few documents, I felt that it was not difficult, so I got started directly, but I still got stuck. . .
Goal:
Crawl job information from Dajie.com.
Process:
1. Use the browser inspection element function to view the address information of dynamically loaded data. 
2, configure the request parameters of requests according to the displayed information.
data = {
'keyword': 'python',
'order': '0',
'city': '',
'recruitType': '',
'salary': '',
'experience': '',
'page': '5',
'positionFunction': '',
'_CSRFToken': '',
'ajax': '1'
}
headers = {
'accept': 'application/json, text/javascript, */*; q=0.01',
'accept-language': 'zh-CN,zh;q=0.8',
'accept-encoding': 'gzip, deflate, sdch',
'cookie': 'DJ_UVID=MTQ5MDMyMTExNTAzODM2MTc5; DJ_RF=empty; DJ_EU=http%3A%2F%2Fjob.dajie.com%2F; __login_tips=1; dj_cap=9c8c95bdef72e84a9bd7493a5ab91694; USER_ACTION="request^A-^A-^Ajobdetail:^A-"; SO_COOKIE_V2=0c7cGprjIH0q9RHc53CWLLXf151DQ5QvUP5ccPQj4g0B/izuXHm8sp41lJjJJh3nmjAkroj8JczFN/SCLPAUzbOHW7wYWmQ6Zu7s',
'referer': 'https://so.dajie.com/job/search?keyword=%E9%A3%9E%E5%88%A9%E6%B5%A6&from=job&clicktype=blank',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36',
'X-Requested-With': 'XMLHttpRequest',
'method':'get'
}3, add request header information to requests.get().
response = requests.get('https://so.dajie.com/job/ajax/search/filter', params=data, headers=headers)4, View the returned page information.
print response.url
print ''
print response.request.headers
print ''
print response.headers
print ''
print response.content[-1000:]
print ''
print response5, Why is the returned result not the expected json data. . .
response.url:
https://so.dajie.com/job/ajax/search/filter?salary=&city=&ajax=1&positionFunction=&_CSRFToken=&keyword=python&recruitType=&order=0&experience=&page=5
response.request.headers:
{'accept-language': 'zh-CN,zh;q=0.8', 'accept-encoding': 'gzip, deflate, sdch', 'X-Requested-With': 'XMLHttpRequest', 'accept': 'application/json, text/javascript, */*; q=0.01', 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36', 'Connection': 'keep-alive', 'referer': 'https://so.dajie.com/job/search?keyword=%E9%A3%9E%E5%88%A9%E6%B5%A6&from=job&clicktype=blank', 'cookie': 'DJ_UVID=MTQ5MDMyMTExNTAzODM2MTc5; DJ_RF=empty; DJ_EU=http%3A%2F%2Fjob.dajie.com%2F; __login_tips=1; dj_cap=9c8c95bdef72e84a9bd7493a5ab91694; USER_ACTION="request^A-^A-^Ajobdetail:^A-"; SO_COOKIE_V2=0c7cGprjIH0q9RHc53CWLLXf151DQ5QvUP5ccPQj4g0B/izuXHm8sp41lJjJJh3nmjAkroj8JczFN/SCLPAUzbOHW7wYWmQ6Zu7s', 'method': 'get'}
response.headers:
{'Date': 'Wed, 19 Apr 2017 02:00:47 GMT', 'Content-Length': '5944', 'ETag': '"552f21de-1738"', 'Content-Type': 'text/html; charset=UTF-8', 'Connection': 'keep-alive'}
response.content[-1000:]:
，这个页面去火星了，试试搜索一下吧：</p>
<form action="http://so.dajie.com/job/search" target="_top" class="search" method="get">
<input type="text" placeholder="搜索感兴趣的职位" autocomplete="off" name="keyword"/><button type="submit">搜索</button>
<input type="hidden" name="jobsearch" value="8"/>
</form>
</p>
<p class="error-404">
<p class="buttonwrap">
<a class="button guest" id="guest" title="" href="http://www.dajie.com/"><b>逛逛大街</b></a>
<a class="button report" id="report" title="" href="mailto:service@dajie.com"><b>报告管理员</b></a>
</p>
</p>
</p>
<script type="text/javascript">
$(function(){
$('input[placeholder]').each(function(){
var $dom = $(this);
var tip = $dom.attr('placeholder');
$.placeholder($dom, {
placeTextClass : 'placeholder',
placeText : tip
});
});
});
</script>
</body>
</html>
response:
<Response [299]>Question:
1, ‘https://so.dajie.com/job/ajax...’, why is this page opened not a json data page? The link given in the tutorial I watched before opens the data amount, for example: ‘https://rate.tmall.com/list_d...’.
2. This is the first time I use requests to request ajax data. Is there anything less to write when requesting?
2, I just tried to modify various request parameters, but I still can’t get the json data. Is my thinking in the wrong direction?
Thanks.
