Home > Article > Backend Development > Mobike crawler source code analysis
The first two articles analyzed why I grabbed Mobike’s interface and As a result of data analysis, this article directly provides executable source code for learning
Statement:
This crawler is only for learning and research purposes, please do not use it for illegal purposes. . Any legal disputes caused by this will be your own responsibility.
If you don’t have the patience to read the article, please post directly:
git clone https://github.com/derekhe/mobike-crawler python3 crawler.py
Please don’t forget to give it a star and ##!
modules - Agent module
3 database, and then after deduplication Export to a csv file to save space. Mobike’s
APIreturns a bicycle in a square area. I can capture the entire area by moving it piece by piece. Large area data. left,
top,right,bottom defines the crawling range, which is currently the Chengdu City Ring Expressway. Within and the square area south to Nanhu. offset defines the crawling interval. It is currently based on 0.002 and can be used within 15 minutes on the DigitalOcean 5$ server. Fetch it once. def start(self):
left = 30.7828453209
top = 103.9213455517
right = 30.4781772402
bottom = 104.2178123382
offset = 0.002
if os.path.isfile(self.db_name):
os.remove(self.db_name)
try:
with sqlite3.connect(self.db_name) as c:
c.execute('''CREATE TABLE mobike
(Time DATETIME, bikeIds VARCHAR(12), bikeType TINYINT,distId INTEGER,distNum TINYINT, type TINYINT, x DOUBLE, y DOUBLE)''')
except Exception as ex:
pass
Then 250 threads are started. As for you asking me why I didn’t use coroutines, humming~~I didn’t learn it at the time~~~It’s actually possible, maybe it’s more efficient. High.
Since the data needs to be deduplicated after crawling, in order to eliminate duplicate parts between small square areas, the last group_data is the core API for doing this. The code is here. For the API interface of the mini program, just create a few
variables, it is very simple.
executor = ThreadPoolExecutor(max_workers=250) print("Start") self.total = 0 lat_range = np.arange(left, right, -offset) for lat in lat_range: lon_range = np.arange(top, bottom, offset) for lon in lon_range: self.total += 1 executor.submit(self.get_nearby_bikes, (lat, lon)) executor.shutdown() self.group_data()
Finally, you may want to ask if frequent IP grabbing is not blocked? In fact, Mobike has IP access speed restrictions, but the way to crack it is very simple, which is to use a large number of proxies. I have an agent pool, and there are basically more than 8,000 agents every day. Get this proxy pool directly in ProxyProvider and provide a pick
functionto randomly select the top 50 proxies. Please note that my proxy pool is
updated every hour, but the jsonblob proxy list provided in the code is just a sample, and most of it should be invalid after a while. . A proxy scoring mechanism is used here. Instead of selecting agents directly at random, I sorted the agents according to their scores. Each successful request will add points, while an erroneous request will lose points. In this way, the agent with the best speed and quality can be selected in a short time. You can save it and use it next time if necessary. def get_nearby_bikes(self, args):
try:
url = "https://mwx.mobike.com/mobike-api/rent/nearbyBikesInfo.do"
payload = "latitude=%s&longitude=%s&errMsg=getMapCenterLocation" % (args[0], args[1])
headers = {
'charset': "utf-8",
'platform': "4",
"referer":"https://servicewechat.com/wx40f112341ae33edb/1/",
'content-type': "application/x-www-form-urlencoded",
'user-agent': "MicroMessenger/6.5.4.1000 NetType/WIFI Language/zh_CN",
'host': "mwx.mobike.com",
'connection': "Keep-Alive",
'accept-encoding': "gzip",
'cache-control': "no-cache"
}
self.request(headers, payload, args, url)
except Exception as ex:
print(ex)
In actual use, select the proxy through proxyProvider.pick() and then use it. If there are any problems with the proxy, directly use proxy.fatal_error() to lower the score, so that this proxy will not be selected in the future.
class ProxyProvider: def init(self, min_proxies=200): self._bad_proxies = {} self._minProxies = min_proxies self.lock = threading.RLock() self.get_list() def get_list(self): logger.debug("Getting proxy list") r = requests.get("https://jsonblob.com/31bf2dc8-00e6-11e7-a0ba-e39b7fdbe78b", timeout=10) proxies = ujson.decode(r.text) logger.debug("Got %s proxies", len(proxies)) self._proxies = list(map(lambda p: Proxy(p), proxies)) def pick(self): with self.lock: self._proxies.sort(key = lambda p: p.score, reverse=True) proxy_len = len(self._proxies) max_range = 50 if proxy_len > 50 else proxy_len proxy = self._proxies[random.randrange(1, max_range)] proxy.used() return proxy
Okay, that’s basically it~~~Study the other codes yourself~~~
The above is the detailed content of Mobike crawler source code analysis. For more information, please follow other related articles on the PHP Chinese website!