Home  >  Article  >  Backend Development  >  Python method to crawl APP download link

Python method to crawl APP download link

高洛峰
高洛峰Original
2017-02-24 15:07:582157browse

First is the preparation work

Python 2.7.11: Download python

Pycharm: Download Pycharm

where python2 and python3 are currently Synchronous release, I am using python2 as the environment here. Pycharm is a relatively efficient Python IDE, but it requires payment.

Basic ideas for implementation

First of all, our target website: Android market

Click [App] to enter our Key page:

Python method to crawl APP download link

After jumping to the application interface, we need to pay attention to three places. The red box in the picture below indicates:

Python method to crawl APP download link

First pay attention to the URL in the address bar, then pay attention to the free download button, and then pay attention to the page turning options at the bottom. Clicking the "Free Download" button will immediately download the corresponding APP, so our idea is to get the click-to-download link and download the APP directly.

Writing a crawler

The first point that needs to be solved: How do we get the download link mentioned above? Here I have to introduce the basic principles of how browsers display web pages. To put it simply, the browser is a tool similar to a parser. When it gets HTML and other codes, it will parse and render according to the corresponding rules, so that we can see the page.

I am using Google Chrome here. Right-click on the page and click "Inspect" to see the original HTML code of the webpage:

Python method to crawl APP download link

Look Don’t worry if you encounter dazzling HTML codes. Google Chrome’s review element has a useful little function that can help us locate the HTML codes corresponding to page controls.

Location:

Python method to crawl APP download link

As shown in the picture above, click the small arrow in the rectangular box above, click the corresponding position on the page, and the HTML code on the right will be automatically positioned and highlighted.

Next we locate the HTML code corresponding to the download button:

Python method to crawl APP download link

You can see that in the code corresponding to the button, there is a corresponding download link: [/ appdown/com.tecent.mm], plus the prefix, the complete download link is http://apk.hiapk.com/appdown/com.tecent.mm

First use python Getting the HTML of the entire page is very simple, just use "requests.get(url) " and fill in the corresponding URL.

Python method to crawl APP download link

Next, when crawling the key information of the page, adopt the idea of ​​​​"grab the big ones first, then the small ones". You can see that there are 10 APPs on a page, corresponding to 10 items in the HTML code:

Python method to crawl APP download link

And each li tag contains the attributes (name) of the respective APP. , download link, etc.). So in the first step, we extract these 10 li tags:


def geteveryapp(self,source):
  everyapp = re.findall(&#39;(<li class="list_item".*?</li>)&#39;,source,re.S)
  #everyapp2 = re.findall(&#39;(<p class="button_bg button_1 right_mt">.*?</p>)&#39;,everyapp,re.S)
  return everyapp


Simple regular expression knowledge is used here

Extract the download link in the li tag:


def getinfo(self,eachclass):
  info = {}
  str1 = str(re.search(&#39;<a href="(.*?)">&#39;, eachclass).group(0))
  app_url = re.search(&#39;"(.*?)"&#39;, str1).group(1)
  appdown_url = app_url.replace(&#39;appinfo&#39;, &#39;appdown&#39;)
  info[&#39;app_url&#39;] = appdown_url
  print appdown_url
  return info


The next difficulty is turning pages , after clicking the page turning button below, we can see that the address bar has changed as follows:

Python method to crawl APP download link

Python method to crawl APP download link

豁然开朗,我们可以在每次的请求中替换URL中对应的id值实现翻页。


def changepage(self,url,total_page):
  now_page = int(re.search(&#39;pi=(\d)&#39;, url).group(1))
  page_group = []
  for i in range(now_page,total_page+1):
   link = re.sub(&#39;pi=\d&#39;,&#39;pi=%s&#39;%i,url,re.S)
   page_group.append(link)
  return page_group


爬虫效果

关键位置说完了,我们先看下最后爬虫的效果:

Python method to crawl APP download link

在TXT文件中保存结果如下:

Python method to crawl APP download link

直接复制进迅雷就可以批量高速下载了。

附上全部代码


#-*_coding:utf8-*-
import requests
import re
import sys
reload(sys)
sys.setdefaultencoding("utf-8")

class spider(object):
 def __init__(self):
  print u'开始爬取内容'
 def getsource(self,url):
  html = requests.get(url)
  return html.text

 def changepage(self,url,total_page):
  now_page = int(re.search(&#39;pi=(\d)&#39;, url).group(1))
  page_group = []
  for i in range(now_page,total_page+1):
   link = re.sub(&#39;pi=\d&#39;,&#39;pi=%s&#39;%i,url,re.S)
   page_group.append(link)
  return page_group

 def geteveryapp(self,source):
  everyapp = re.findall('(
  • )',source,re.S) return everyapp def getinfo(self,eachclass): info = {} str1 = str(re.search(&#39;<a href="(.*?)">&#39;, eachclass).group(0)) app_url = re.search(&#39;"(.*?)"&#39;, str1).group(1) appdown_url = app_url.replace(&#39;appinfo&#39;, &#39;appdown&#39;) info[&#39;app_url&#39;] = appdown_url print appdown_url return info def saveinfo(self,classinfo): f = open('info.txt','a') str2 = "http://apk.hiapk.com" for each in classinfo: f.write(str2) f.writelines(each['app_url'] + '\n') f.close() if __name__ == '__main__': appinfo = [] url = 'http://apk.hiapk.com/apps/MediaAndVideo?sort=5&pi=1' appurl = spider() all_links = appurl.changepage(url, 5) for link in all_links: print u'正在处理页面' + link html = appurl.getsource(link) every_app = appurl.geteveryapp(html) for each in every_app: info = appurl.getinfo(each) appinfo.append(info) appurl.saveinfo(appinfo)

  • 总结

    选取的目标网页相对结构清晰简单,这是一个比较基本的爬虫。代码写的比较乱请见谅,以上就是这篇文章的全部内容了,希望能对大家的学习或者工作带来一定的帮助,如果有问题大家可以留言交流。

    更多Python method to crawl APP download link相关文章请关注PHP中文网!

    Statement:
    The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn