Home > Article > Backend Development > Introducing three methods of python data capture
Introducing three methods of python data capture
- coldplay.xixiforward
- 2021-02-13 10:30:074634browse
Free learning recommendation: python video tutorial
Three methods of data capture
- Regular expression (re library)
- BeautifulSoup (bs4)
- lxml
*Use the previously built download web page function to obtain the html of the target web page. We take https://guojiadiqu.bmcx.com/AFG__guojiayudiqu/ as an example to obtain the html.
from get_html import download url = 'https://guojiadiqu.bmcx.com/AFG__guojiayudiqu/'page_content = download(url)
* Suppose we need to crawl the country name and profile in this web page, we use these three data crawling methods in turn to achieve data crawling.
1. Regular expression
from get_html import downloadimport re
url = 'https://guojiadiqu.bmcx.com/AFG__guojiayudiqu/'page_content = download(url)country = re.findall('class="h2dabiaoti">(.*?)', page_content) #注意返回的是listsurvey_data = re.findall('<tr><td>(.*?)</td></tr>', page_content)survey_info_list = re.findall('<p> (.*?)</p>', survey_data[0])survey_info = ''.join(survey_info_list)print(country[0],survey_info)
2.BeautifulSoup(bs4)
from get_html import downloadfrom bs4 import BeautifulSoup
url = 'https://guojiadiqu.bmcx.com/AFG__guojiayudiqu/'html = download(url)#创建 beautifulsoup 对象soup = BeautifulSoup(html,"html.parser")#搜索country = soup.find(attrs={'class':'h2dabiaoti'}).text
survey_info = soup.find(attrs={'id':'wzneirong'}).textprint(country,survey_info)
3.lxml
from get_html import downloadfrom lxml import etree #解析树url = 'https://guojiadiqu.bmcx.com/AFG__guojiayudiqu/'page_content = download(url)selector = etree.HTML(page_content)#可进行xpath解析country_select = selector.xpath('//*[@id="main_content"]/h2') #返回列表for country in country_select:
print(country.text)survey_select = selector.xpath('//*[@id="wzneirong"]/p')for survey_content in survey_select:
print(survey_content.text,end='')
Run result:
Finally, quote the performance comparison of the three methods in "Writing a Web Crawler with Python", as shown below: 
For reference only.
Related free learning recommendations: python tutorial(Video)
The above is the detailed content of Introducing three methods of python data capture. For more information, please follow other related articles on the PHP Chinese website!