Teach you step-by-step how to use Python web crawler to obtain King of Glory hero equipment instructions and automatically generate markdown files

1. Preface

Played Friends who play the Honor of Kings game all know that hero equipment is very important. A reasonable equipment, coupled with inscriptions, can make you unstoppable and unstoppable on the battlefield of King!

A few days ago, I saw in the [Minglao] group that he shared a Python web crawler to obtain the equipment instructions of the King of Glory hero, and used the thread pool to download the equipment pictures, and then automatically generated them. Markdown file, there is a lot of useful content, I will share it with you here, everyone is welcome to try it.

2. Data Acquisition

Our target website here is the official website of King of Glory, as shown in the figure below.

Then click the [More] button of [Hero/Skin] on the right side of the homepage to enter the details page, as shown in the figure below. Click [In-game Props] to see the output The information is installed, which contains the target information we want.

By capturing packets through the browser, you can obtain specific information, which can be seen stored in the json format.

The picture below is a screenshot of the data details. You can see that there are Chinese garbled characters. This does not affect it. At least the data can be obtained.

Code implementation process

After finding the data source, the next step is to implement the code. Let’s take a look. , the [Minglao] code is directly applied here, and it is run in jupyter notebook.

Get equipment data

import requests
import pandas as pd

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) '
                  'Chrome/88.0.4324.104 Safari/537.36 '
}
target = 'https://pvp.qq.com/web201605/js/item.json'
item_list = requests.get(target, headers=headers).json()
item_df = pd.DataFrame(item_list)
item_df.sort_values(["item_type", "price", "item_id"], inplace=True)
item_df.fillna("", inplace=True)
item_df.des1 = item_df.des1.str.replace("</?p>", "", regex=True)
item_df.des2 = item_df.des2.str.replace("</?p>", "", regex=True)
item_df

The result is as shown below:

##Multi-threaded picture download

Next, use the thread pool method to download pictures. The method of splicing pictures is also very simple. You can see it at a glance by looking at the picture below.

The following is the code implementation:

import os
from concurrent.futures import ThreadPoolExecutor


def download_img(item_id):
    if os.path.exists(f"imgs/{item_id}.jpg"):
        return
    imgurl = f"http://game.gtimg.cn/images/yxzj/img201606/itemimg/{item_id}.jpg"
    res = requests.get(imgurl)
    with open(f"imgs/{item_id}.jpg", "wb") as f:
        f.write(res.content)


os.makedirs("imgs", exist_ok=True)
with ThreadPoolExecutor(max_workers=8) as executor:
    nums = executor.map(download_img, item_df.item_id)

The download speed is very fast, just a few seconds, the result is as shown in the figure below:

Next, we will automatically generate the Markdown document from the data, let’s take a look.

Generate Markdown document

The code is as follows. The first part is the preprocessing of the data, and the second part is writing the file:

item_type_dict = {1: '攻击', 2: '法术', 3: '防御', 4: '移动', 5: '打野', 7: '游走'}
item_ids = item_df.item_id.values
item_df.item_id = item_df.item_id.apply(
    lambda item_id: f"![{item_id}](imgs/{item_id}.jpg)")
item_df.item_type = item_df.item_type.map(item_type_dict)
item_df.columns = ["图片", "装备名称", "类型", "售价", "总价", "基础描述", "扩展描述"]
item_df

Write the code to the file and generate the Markdown document:

with open("王者装备说明.md", "w") as f:
    for item_type, item_split in item_df.groupby("类型", sort=False):
        f.write(f"# {item_type}\n")
        item_split.drop(columns="类型", inplace=True)
        f.write(item_split.to_markdown(index=False))
        f.write("\n\n")

The result is as shown below:

After that, a file named [King Zhe] will be generated locally. Equipment Description.md] Markdown document, double-click the file to open, the content is as shown below:

What a great guy! When I implemented this step, I encountered an error, as shown below:

Missing optional dependency 'tabulate'.  Use pip or conda to install tabulate.

提示却少依赖库，只需要在cmd下进行安装即可pip install tabulate，之后就可以正常运行了。

生成Excel表格

不过Markdown的表格无法任意调整，图片需要点击后才会放大，下面我们考虑生成Excel表格：首先需要整理数据，代码如下：

item_df.图片 = ""
item_df.基础描述 = item_df.基础描述.str.replace("<br>", "\n")
item_df.扩展描述 = item_df.扩展描述.str.replace("<br>", "\n")
item_df

生成结果如下图所示：

之后将结果写入到Excel中去，代码如下所示：

# 写入Excel表格
from openpyxl.drawing.image import Image
from openpyxl.styles import Alignment

with pd.ExcelWriter("王者装备说明.xlsx", engine='openpyxl') as writer:
    item_df.to_excel(writer, sheet_name='装备说明', index=False)
    worksheet = writer.sheets['装备说明']
    worksheet.column_dimensions["A"].width = 11
    for item_id, (cell,) in zip(item_ids, worksheet.iter_rows(2, None, 1, 1)):
        worksheet.row_dimensions[cell.row].height = 67
        worksheet.add_image(Image(f"imgs/{item_id}.jpg"), f'A{cell.row}')
    worksheet.column_dimensions["F"].width = 15
    worksheet.column_dimensions["G"].width = 35
    writer.save()

打开文件，效果图如下图所示：

当然了，大家也可以根据自己想要的效果生成HTML和Word等等。

三、总结

大家好，我是Python进阶者。这篇文章主要分享了一个使用Python网络爬虫获取王者荣耀英雄出装说明，并使用线程池的方式下载了出装图片，之后还自动化生成了markdown文件，干货内容很多，欢迎大家积极尝试，如果有遇到问题，请添加我好友，我帮助解决。

最后感谢粉丝【明佬】分享的代码喝王者荣耀出装攻略，真是太强了，上王者指日可待！

最后放上【明佬】的csdn链接：https://xxmdmst.blog.csdn.net/article/details/124035041，点击阅读原文可以直达噢！

The above is the detailed content of Teach you step-by-step how to use Python web crawler to obtain King of Glory hero equipment instructions and automatically generate markdown files. For more information, please follow other related articles on the PHP Chinese website!