Home > Article > Backend Development > Detailed introduction to Python crawler video crawling
This article brings you a detailed introduction to Python crawler crawling videos. It has certain reference value. Friends in need can refer to it. I hope it will be helpful to you.
I am writing an application recently and need to collect some popular videos on Weibo. These short videos generally come from Miaopai, Weipai, Meipai and Sina Video, and there is no download option, so I can only Use your brain to think of a solution.
Analyze the source code of the web page. For example: http://video.weibo.com/show?fid=1034:0988e59a12e5178acb7f23adc3fe5e97, right-click to view the source code. Generally, videos have the mp4 suffix. If you search, you will not find it, but some can be directly viewed, such as Meipai videos.
Capture the packet, analyze the request and return. This can also be achieved through the powerful chrome, or the above example, right-click->Inspect Element->NetWork, and then refresh the webpage with F5
I found that there are many requests, and I can only go one by one. After analysis, in fact, the video formats are mp4, flv, and avi. You can see it at once. Copy it to the browser and open it, and it is the download link we want.
Analyze the rules of download links and video links. That is, the relationship between http://video.weibo.com/show?fid=1034:0988e59a12e5178acb7f23adc3fe5e97 and xxx.mp4. This again requires analyzing the web page source code. In fact, you can pay attention to the link with the .m3u8 suffix above. m3u8 records an index plain text file. When you open it, the playback software does not play it, but finds the corresponding audio and video based on its index. Use the network address of the file to play online, open it, and see that the download link we want is indeed recorded in it. Moreover, the link with the .m3u8 suffix is in the source code of the web page.
After the analysis of the first three steps, the idea of obtaining the video download link is to first obtain the .m3u8 suffix link from the web page source code, download the file, and Get the video download link inside, and finally download the video
#sinavideo.py #coding=utf-8 import os import re import urllib2 import urllib from common import Common class SinaVideo(): URL_PIRFIX = "http://us.sinaimg.cn/" def getM3u8(self,html): reg = re.compile(r'list=([\s\S]*?)&fid') result = reg.findall(html) return result[0] def getName(self,url): return url.split('=')[1] def getSinavideoUrl(self,filepath): f = open(filepath,'r') lines = f.readlines() f.close() for line in lines: if line[0] !='#': return line def download(self,url,filepath): #获取名称 name = self.getName(url) html = Common.getHtml(url) m3u8 = self.getM3u8(html) Common.download(urllib.unquote(m3u8),filepath,name + '.m3u8') url = self.URL_PIRFIX + self.getSinavideoUrl(filepath+name+'.m3u8') Common.download(url,filepath,name+'.mp4')
#common.py #coding=utf-8 import urllib2 import os import re class Common(): # 获取网页源码 @staticmethod def getHtml(url): html = urllib2.urlopen(url).read() print "[+]获取网页源码:"+url return html # 下载文件 @staticmethod def download(url,filepath,filename): headers = { 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Charset': 'UTF-8,*;q=0.5', 'Accept-Encoding': 'gzip,deflate,sdch', 'Accept-Language': 'en-US,en;q=0.8', 'User-Agent': 'Mozilla/5.0 (Linux; Android 4.4.2; Nexus 4 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.114 Mobile Safari/537.36' } request = urllib2.Request(url,headers = headers); response = urllib2.urlopen(request) path = filepath + filename with open(path,'wb') as output: while True: buffer = response.read(1024*256); if not buffer: break # received += len(buffer) output.write(buffer) print "[+]下载文件成功:"+path @staticmethod def isExist(filepath): return os.path.exists(filepath) @staticmethod def createDir(filepath): os.makedirs(filepath,0777)
Calling method:
url = "http://video.weibo.com/show?fid=1034:0988e59a12e5178acb7f23adc3fe5e97"sinavideo = SinaVideo() sinavideo.download(url,""/Users/cheng/Documents/PyScript/res/"")
The above is the detailed content of Detailed introduction to Python crawler video crawling. For more information, please follow other related articles on the PHP Chinese website!