Can python crawler crawl videos?

Web crawlers, also known as web spiders, refer to script programs that crawl the required content on the web according to certain rules. As we all know, each web page usually contains entrances to other web pages, and web crawlers use one URL to enter other URLs in sequence to obtain the required content.

##Crawler structure

Crawler scheduler (Entrance of the program, used to start the entire program)

url manager (used to manage URLs that have not been crawled and URLs that have been crawled)

Web page downloader (used to download web page content for analysis)

Web page parser (used to parse downloaded web pages and obtain new urls and required content)

Web page outputter (used to output the obtained content in the form of a file)

First step

Analyze the source code of the web page. For example: http://video.weibo.com/show?fid=1034:0988e59a12e5178acb7f23adc3fe5e97, right-click to view the source code. Generally, videos have the mp4 suffix. If you search, you will not find it, but some can be directly viewed, such as Meipai videos.

Related recommendations: "

python video tutorial"

The second step

Capture the packet, analyze the request and return. This can also be achieved through the powerful chrome, or the above example, right-click->Inspect Element->NetWork, and then refresh the webpage with F5

I found that there are many The requests can only be analyzed one by one. In fact, the video formats are mp4, flv, and avi. You can see them at once. Copy them to the browser and open them. Sure enough, it is the download link we want.

The third step

analyze the rules of download links and video links. That is, the relationship between http://video.weibo.com/show?fid=1034:0988e59a12e5178acb7f23adc3fe5e97 and xxx.mp4. This again requires analyzing the web page source code. In fact, you can pay attention to the link with the .m3u8 suffix above. m3u8 records an index plain text file. When you open it, the playback software does not play it, but finds the corresponding audio and video based on its index. Use the network address of the file to play online, open it, and see that the download link we want is indeed recorded in it. Moreover, the link with the .m3u8 suffix is ​​in the source code of the web page.

Summary

After the analysis of the first three steps, the idea of ​​​​obtaining the video download link Just get the link with the .m3u8 suffix from the web page source code, download the file, get the video download link from it, and finally download the video

Source code

#coding=utf-8
import os
import re
import urllib2
import urllib
from common import Common
class SinaVideo():
 
    URL_PIRFIX = "http://us.sinaimg.cn/"
    def getM3u8(self,html):
        reg = re.compile(r'list=([\s\S]*?)&fid')
        result = reg.findall(html)
        return result[0]
 
 
    def getName(self,url):
         return url.split('=')[1]
 
    def getSinavideoUrl(self,filepath):
        f = open(filepath,'r')
        lines = f.readlines()
        f.close()
        for line in lines:
            if line[0] !='#':
                return line
 
    def download(self,url,filepath):
        #获取名称
        name = self.getName(url)
        html = Common.getHtml(url)
        m3u8 = self.getM3u8(html)
        Common.download(urllib.unquote(m3u8),filepath,name + '.m3u8')
        url = self.URL_PIRFIX + self.getSinavideoUrl(filepath+name+'.m3u8')
        Common.download(url,filepath,name+'.mp4')

#common.py
#coding=utf-8
import urllib2
import os
import re
 
 
class Common():
    #  获取网页源码
    @staticmethod
    def getHtml(url):
        html = urllib2.urlopen(url).read()
        print  "[+]获取网页源码:"+url
        return html
 
    # 下载文件
    @staticmethod
    def download(url,filepath,filename):
        headers = {
            'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
            'Accept-Charset': 'UTF-8,*;q=0.5',
            'Accept-Encoding': 'gzip,deflate,sdch',
            'Accept-Language': 'en-US,en;q=0.8',
            'User-Agent': 'Mozilla/5.0 (Linux; Android 4.4.2; Nexus 4 Build/KOT49H) 
            AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.114 Mobile Safari/537.36'
        }
        request = urllib2.Request(url,headers = headers);
        response = urllib2.urlopen(request)
        path = filepath + filename
        with open(path,'wb') as output:
            while True:
                buffer = response.read(1024*256);
                if not buffer:
                    break
                # received += len(buffer)
                output.write(buffer)
 
        print "[+]下载文件成功:"+path
 
    @staticmethod
    def isExist(filepath):
        return os.path.exists(filepath)
 
    @staticmethod
    def createDir(filepath):
         os.makedirs(filepath,0777)

Calling method:

url = "http://video.weibo.com/show?fid=1034:0988e59a12e5178acb7f23adc3fe5e97"sinavideo = SinaVideo()        
sinavideo.download(url,""/Users/cheng/Documents/PyScript/res/"")

Result:

The above is the detailed content of Can python crawler crawl videos?. For more information, please follow other related articles on the PHP Chinese website!