search
HomeBackend DevelopmentPython TutorialUsing Python to implement video download function example code

The short video business has been booming in the past year or two, and each video website has its own short video content. If there was a program that could download the latest videos released by popular users of major video websites, it would not only facilitate viewing, but also allow you to post copyright-free videos on personal social networking sites to increase your own popularity. How great would it be?

The short video business has been booming in the past year or two, and each video website has its own short video content. Wouldn't it be nice if there was such a program that could download the latest videos released by popular users of major video websites? Not only would it be convenient for you to watch, but you could also post copyright-free videos on personal social networking sites to increase your popularity. ?

parker is such a project (project address: https://github.com/LiuRoy/parker), which uses the celeryframework to regularly crawl user video lists and compile the latest released videos Distributed deployment can be easily implemented through you-get asynchronous download. Because the page layout and interface of each website are updated frequently, in order to ensure the high availability of the program, Statsd monitoring is specially added to facilitate timely detection of errors.

CodeArchitecture

Using Python to implement video download function example code

##Currently in parker Only downloads from Bilibili and Miaopai are implemented. As can be seen from the frame diagram, two asynchronous interfaces need to be implemented for each type of website: parsing the playback address of the published video from the user's video homepage, and downloading the video based on the playback address. Therefore, when adding a website type, you do not need to modify the original code. You only need to add a new parsing and downloading interface. I have not implemented the subsequent operations after the video download is completed. You can freely implement it according to your own needs.

When running, celery will regularly send the configured high-quality user list to the parsing interface of the corresponding website for asynchronous execution, filter out the latest video playback address, and hand it over to the corresponding download interface for asynchronous downloading. After completion, call subsequent operations asynchronously. Therefore, it is necessary to start a celery beat process to send scheduled tasks, and several celery asynchronous tasks to perform parsing and downloading operations. For relatively large videos, downloading will be quite time-consuming. It is recommended to reasonably allocate the number of asynchronous tasks according to the size of the task list.

Program operation

It has been verified that this program can run normally under ubuntu and mac. Because celery under local windows cannot It starts normally, so it has not been verified in

windows environment.

Dependent librariesInstallation##The python version is 3.5. After entering the project directory, execute:

pip install -r requirements.txt


Create database

TableBuild it in the database in advance Two tables (sql: https://github.com/LiuRoy/parker/blob/master/spider/models/tables.sql)

Parameter configuration logging.yaml, params.yaml, and sites.yaml under the config path correspond to log configuration, running parameter configuration, and popular user configuration respectively.

Log configurationIn debug mode, the log will be output directly to the standard output stream. In release mode, the log content will be output to a file. Therefore, the output log file needs to be configured.

Run configuration

    mode debug
  • Debug mode

    , in this mode the log points to the standard output , and there is no monitoring data; in release mode, the log is output to the specified file, and there is monitoring data.

  • broker_url corresponds to celery's BROKER_URL, which can be configured as
  • redis

    or rabbitmq

  • mysql

    _url Database address, two tables need to be built in advance

  • download_path Video download path
  • statsd_address Monitoring address
  • video_number_per_page How many video playback addresses are parsed from the user's video homepage each time? Because most users publish a small number of videos each time, it only needs to be set to a very small value. During the first run, a large number of long-standing videos will not be downloaded.

  • download_timeout Timeout for video download

Popular user configuration

parker will generate a list of celery beat schedulers based on this configuration.

  • name rule is -, parker will use this as scheduler task name

  • url user Publish video homepage

  • task Corresponding celery parsing asynchronous task

  • minute How many minutes to check the user video list

Start the task

Enter the project directory and execute the following command to start the celery worker


celery -A spider worker


Execute the following command to start the celery beat scheduled task


celery -A spider beat


Monitoring

Strongly use a docker image to configure the monitoring environment in one minute. After that, you only need to add the data of successful execution and abnormal execution, and you can easily monitor whether the program is running normally.

Using Python to implement video download function example code


The above is the detailed content of Using Python to implement video download function example code. For more information, please follow other related articles on the PHP Chinese website!

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
What are some common reasons why a Python script might not execute on Unix?What are some common reasons why a Python script might not execute on Unix?Apr 28, 2025 am 12:18 AM

The reasons why Python scripts cannot run on Unix systems include: 1) Insufficient permissions, using chmod xyour_script.py to grant execution permissions; 2) Shebang line is incorrect or missing, you should use #!/usr/bin/envpython; 3) The environment variables are not set properly, and you can print os.environ debugging; 4) Using the wrong Python version, you can specify the version on the Shebang line or the command line; 5) Dependency problems, using virtual environment to isolate dependencies; 6) Syntax errors, using python-mpy_compileyour_script.py to detect.

Give an example of a scenario where using a Python array would be more appropriate than using a list.Give an example of a scenario where using a Python array would be more appropriate than using a list.Apr 28, 2025 am 12:15 AM

Using Python arrays is more suitable for processing large amounts of numerical data than lists. 1) Arrays save more memory, 2) Arrays are faster to operate by numerical values, 3) Arrays force type consistency, 4) Arrays are compatible with C arrays, but are not as flexible and convenient as lists.

What are the performance implications of using lists versus arrays in Python?What are the performance implications of using lists versus arrays in Python?Apr 28, 2025 am 12:10 AM

Listsare Better ForeflexibilityandMixdatatatypes, Whilearraysares Superior Sumerical Computation Sand Larged Datasets.1) Unselable List Xibility, MixedDatatypes, andfrequent elementchanges.2) Usarray's sensory -sensical operations, Largedatasets, AndwhenMemoryEfficiency

How does NumPy handle memory management for large arrays?How does NumPy handle memory management for large arrays?Apr 28, 2025 am 12:07 AM

NumPymanagesmemoryforlargearraysefficientlyusingviews,copies,andmemory-mappedfiles.1)Viewsallowslicingwithoutcopying,directlymodifyingtheoriginalarray.2)Copiescanbecreatedwiththecopy()methodforpreservingdata.3)Memory-mappedfileshandlemassivedatasetsb

Which requires importing a module: lists or arrays?Which requires importing a module: lists or arrays?Apr 28, 2025 am 12:06 AM

ListsinPythondonotrequireimportingamodule,whilearraysfromthearraymoduledoneedanimport.1)Listsarebuilt-in,versatile,andcanholdmixeddatatypes.2)Arraysaremorememory-efficientfornumericdatabutlessflexible,requiringallelementstobeofthesametype.

What data types can be stored in a Python array?What data types can be stored in a Python array?Apr 27, 2025 am 12:11 AM

Pythonlistscanstoreanydatatype,arraymodulearraysstoreonetype,andNumPyarraysarefornumericalcomputations.1)Listsareversatilebutlessmemory-efficient.2)Arraymodulearraysarememory-efficientforhomogeneousdata.3)NumPyarraysareoptimizedforperformanceinscient

What happens if you try to store a value of the wrong data type in a Python array?What happens if you try to store a value of the wrong data type in a Python array?Apr 27, 2025 am 12:10 AM

WhenyouattempttostoreavalueofthewrongdatatypeinaPythonarray,you'llencounteraTypeError.Thisisduetothearraymodule'sstricttypeenforcement,whichrequiresallelementstobeofthesametypeasspecifiedbythetypecode.Forperformancereasons,arraysaremoreefficientthanl

Which is part of the Python standard library: lists or arrays?Which is part of the Python standard library: lists or arrays?Apr 27, 2025 am 12:03 AM

Pythonlistsarepartofthestandardlibrary,whilearraysarenot.Listsarebuilt-in,versatile,andusedforstoringcollections,whereasarraysareprovidedbythearraymoduleandlesscommonlyusedduetolimitedfunctionality.

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

DVWA

DVWA

Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

EditPlus Chinese cracked version

EditPlus Chinese cracked version

Small size, syntax highlighting, does not support code prompt function

MinGW - Minimalist GNU for Windows

MinGW - Minimalist GNU for Windows

This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor