search
HomeBackend DevelopmentPython TutorialPython code example for grabbing NetEase news

Python code example for grabbing NetEase news

May 03, 2017 pm 04:03 PM
pythoncrawlregularNetease News

这篇文章主要介绍了Python正则抓取网易新闻的方法,结合实例形式较为详细的分析了Python使用正则进行网易新闻抓取操作的相关实现技巧与注意事项,需要的朋友可以参考下

本文实例讲述了Python正则抓取网易新闻的方法。分享给大家供大家参考,具体如下:

自己写了些关于抓取网易新闻的爬虫,发现其网页源代码与网页的评论根本就对不上,所以,采用了抓包工具得到了其评论的隐藏地址(每个浏览器都有自己的抓包工具,都可以用来分析网站)

如果仔细观察的话就会发现,有一个特殊的,那么这个就是自己想要的了

然后打开链接就可以找到相关的评论内容了。(下图为第一页内容)

接下来就是代码了(也照着大神的改改写写了)。

#coding=utf-8
import urllib2
import re
import json
import time
class WY():
  def __init__(self):
    self.headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/534.24 (KHTML, like '}
    self.url='http://comment.news.163.com/data/news3_bbs/df/B9IBDHEH000146BE_1.html'
  def getpage(self,page):
    full_url='http://comment.news.163.com/cache/newlist/news3_bbs/B9IBDHEH000146BE_'+str(page)+'.html'
    return full_url
  def gethtml(self,page):
    try:
      req=urllib2.Request(page,None,self.headers)
      response = urllib2.urlopen(req)
      html = response.read()
      return html
    except urllib2.URLError,e:
      if hasattr(e,'reason'):
        print u"连接失败",e.reason
        return None
  #处理字符串
  def Process(self,data,page):
    if page == 1:
      data=data.replace('var replyData=','')
    else:
      data=data.replace('var newPostList=','')
    reg1=re.compile(" \[<a href=&#39;&#39;>")
    data=reg1.sub(&#39; &#39;,data)
    reg2=re.compile(&#39;<\\\/a>\]&#39;)
    data=reg2.sub(&#39;&#39;,data)
    reg3=re.compile(&#39;<br>&#39;)
    data=reg3.sub(&#39;&#39;,data)
    return data
  #解析json
  def dealJSON(self):
    with open("WY.txt","a") as file:
      file.write(&#39;ID&#39;+&#39;|&#39;+&#39;评论&#39;+&#39;|&#39;+&#39;踩&#39;+&#39;|&#39;+&#39;顶&#39;+&#39;\n&#39;)
    for i in range(1,12):
      if i == 1:
        data=self.gethtml(self.url)
        data=self.Process(data,i)[:-1]
        value=json.loads(data)
        file=open(&#39;WY.txt&#39;,&#39;a&#39;)
        for item in value[&#39;hotPosts&#39;]:
          try:
            file.write(item[&#39;1&#39;][&#39;f&#39;].encode(&#39;utf-8&#39;)+&#39;|&#39;)
            file.write(item[&#39;1&#39;][&#39;b&#39;].encode(&#39;utf-8&#39;)+&#39;|&#39;)
            file.write(item[&#39;1&#39;][&#39;a&#39;].encode(&#39;utf-8&#39;)+&#39;|&#39;)
            file.write(item[&#39;1&#39;][&#39;v&#39;].encode(&#39;utf-8&#39;)+&#39;\n&#39;)
          except:
            continue
        file.close()
        print &#39;--正在采集%d/12--&#39;%i
        time.sleep(5)
      else:
        page=self.getpage(i)
        data = self.gethtml(page)
        data = self.Process(data,i)[:-2]
        # print data
        value=json.loads(data)
        # print value
        file=open(&#39;WY.txt&#39;,&#39;a&#39;)
        for item in value[&#39;newPosts&#39;]:
          try:
            file.write(item[&#39;1&#39;][&#39;f&#39;].encode(&#39;utf-8&#39;)+&#39;|&#39;)
            file.write(item[&#39;1&#39;][&#39;b&#39;].encode(&#39;utf-8&#39;)+&#39;|&#39;)
            file.write(item[&#39;1&#39;][&#39;a&#39;].encode(&#39;utf-8&#39;)+&#39;|&#39;)
            file.write(item[&#39;1&#39;][&#39;v&#39;].encode(&#39;utf-8&#39;)+&#39;\n&#39;)
          except:
            continue
        file.close()
        print &#39;--正在采集%d/12--&#39;%i
        time.sleep(5)
if __name__ == &#39;__main__&#39;:
  WY().dealJSON()

以上就是我爬取的代码了。

The above is the detailed content of Python code example for grabbing NetEase news. For more information, please follow other related articles on the PHP Chinese website!

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
How do you create multi-dimensional arrays using NumPy?How do you create multi-dimensional arrays using NumPy?Apr 29, 2025 am 12:27 AM

Create multi-dimensional arrays with NumPy can be achieved through the following steps: 1) Use the numpy.array() function to create an array, such as np.array([[1,2,3],[4,5,6]]) to create a 2D array; 2) Use np.zeros(), np.ones(), np.random.random() and other functions to create an array filled with specific values; 3) Understand the shape and size properties of the array to ensure that the length of the sub-array is consistent and avoid errors; 4) Use the np.reshape() function to change the shape of the array; 5) Pay attention to memory usage to ensure that the code is clear and efficient.

Explain the concept of 'broadcasting' in NumPy arrays.Explain the concept of 'broadcasting' in NumPy arrays.Apr 29, 2025 am 12:23 AM

BroadcastinginNumPyisamethodtoperformoperationsonarraysofdifferentshapesbyautomaticallyaligningthem.Itsimplifiescode,enhancesreadability,andboostsperformance.Here'showitworks:1)Smallerarraysarepaddedwithonestomatchdimensions.2)Compatibledimensionsare

Explain how to choose between lists, array.array, and NumPy arrays for data storage.Explain how to choose between lists, array.array, and NumPy arrays for data storage.Apr 29, 2025 am 12:20 AM

ForPythondatastorage,chooselistsforflexibilitywithmixeddatatypes,array.arrayformemory-efficienthomogeneousnumericaldata,andNumPyarraysforadvancednumericalcomputing.Listsareversatilebutlessefficientforlargenumericaldatasets;array.arrayoffersamiddlegro

Give an example of a scenario where using a Python list would be more appropriate than using an array.Give an example of a scenario where using a Python list would be more appropriate than using an array.Apr 29, 2025 am 12:17 AM

Pythonlistsarebetterthanarraysformanagingdiversedatatypes.1)Listscanholdelementsofdifferenttypes,2)theyaredynamic,allowingeasyadditionsandremovals,3)theyofferintuitiveoperationslikeslicing,but4)theyarelessmemory-efficientandslowerforlargedatasets.

How do you access elements in a Python array?How do you access elements in a Python array?Apr 29, 2025 am 12:11 AM

ToaccesselementsinaPythonarray,useindexing:my_array[2]accessesthethirdelement,returning3.Pythonuseszero-basedindexing.1)Usepositiveandnegativeindexing:my_list[0]forthefirstelement,my_list[-1]forthelast.2)Useslicingforarange:my_list[1:5]extractselemen

Is Tuple Comprehension possible in Python? If yes, how and if not why?Is Tuple Comprehension possible in Python? If yes, how and if not why?Apr 28, 2025 pm 04:34 PM

Article discusses impossibility of tuple comprehension in Python due to syntax ambiguity. Alternatives like using tuple() with generator expressions are suggested for creating tuples efficiently.(159 characters)

What are Modules and Packages in Python?What are Modules and Packages in Python?Apr 28, 2025 pm 04:33 PM

The article explains modules and packages in Python, their differences, and usage. Modules are single files, while packages are directories with an __init__.py file, organizing related modules hierarchically.

What is docstring in Python?What is docstring in Python?Apr 28, 2025 pm 04:30 PM

Article discusses docstrings in Python, their usage, and benefits. Main issue: importance of docstrings for code documentation and accessibility.

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

WebStorm Mac version

WebStorm Mac version

Useful JavaScript development tools

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

SublimeText3 English version

SublimeText3 English version

Recommended: Win version, supports code prompts!

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor