Four Python project management and construction tools, recommended collection!

Python has not had a de facto standard project management and construction tool for so long, resulting in a variety of Python project structures and construction methods. This may reflect Python's free will.

Unlike Java, it has gone through the initial manual construction, to semi-automated Ant, and then to Maven, which is basically the de facto standard. During this period, Maven also accepted challenges from other Gradle (mainly promoted by Android projects), SBT (mainly Scala projects), Ant Ivy, Buildr, etc., but it was difficult to shake Maven's status in the world, and the others almost followed Maven's directory layout. .

Back in Python, there have been package management tools such as pip, pipenv, and conda, but there is no agreement on the directory layout of the project.

Many aspects of building still follow the traditional Makefile method, and then add setup.py and build.py to use program code to install and build. Regarding the project directory layout, some make project templates, and then make tools to apply the project templates.

The following is a brief overview of the use of the four tools

1. CookieCutter
   
2. PyScaffold
   
3. PyBuilder
   
4. Poetry
   

CookieCutter A classic Python project directory structure

$ pip install cookiecutter
$ cookiecutter gh:audreyr/cookiecutter-pypackage
# 以 github 上的 audreyr/cookiecutter-pypackage 为模板，再回答一堆的问题生成一个 Python 项目
......
project_name [Python Boilerplate]: sample
......

The final project template generated by cookiecutter is as follows It looks like:

$ tree sample
sample
├── AUTHORS.rst
├── CONTRIBUTING.rst
├── HISTORY.rst
├── LICENSE
├── MANIFEST.in
├── Makefile
├── README.rst
├── docs
│ ├── Makefile
│ ├── authors.rst
│ ├── conf.py
│ ├── contributing.rst
│ ├── history.rst
│ ├── index.rst
│ ├── installation.rst
│ ├── make.bat
│ ├── readme.rst
│ └── usage.rst
├── requirements_dev.txt
├── sample
│ ├── __init__.py
│ ├── cli.py
│ └── sample.py
├── setup.cfg
├── setup.py
├── tests
│ ├── __init__.py
│ └── test_sample.py
└── tox.ini
3 directories, 26 files

This is probably the main framework of the currently popular directory structure. The main elements are:

$ tree sample
sample
├── Makefile
├── README.rst
├── docs
│ └── index.rst
├── requirements.txt
├── sample
│ ├── __init__.py
│ └── sample.py
├── setup.cfg
├── setup.py
└── tests
 ├── __init__.py
 └── test_sample.py

Repeat in the project sample directory. Place the Python source file in the sample directory, and place the Python source file in the tests directory. It is a test file, plus a docs directory for documentation, README.rst, and other setup, setup.cfg and Makefile files for building.

This is actually a very classic Python project structure. The next build uses the make command. Enter make and you will see the instructions defined in the Makefile file.

$ make
cleanremove all build, test, coverage and Python artifacts
clean-buildremove build artifacts
clean-pycremove Python file artifacts
clean-test remove test and coverage artifacts
lint check style
test run tests quickly with the default Python
test-all run tests on every Python version with tox
coverage check code coverage quickly with the default Python
docs generate Sphinx HTML documentation, including API docs
servedocscompile the docs watching for changes
releasepackage and upload a release
dist builds source and wheel package
installinstall the package to the active Python's site-packages

In order to use the above build process, you need to install the corresponding packages, such as tox, wheel, coverage, sphinx, flake8, they can all be installed through pip. Then you can make test, make coverage, make docs, make dist, etc. Among them, make docs can generate a beautiful web document.

PyScaffold Create a project

PyScaffold As the name suggests, it is a tool used to create Python project scaffolding. Install and use:

$ pip install pyscaffold
$ putup sample

Create like this I created a Python project, and the directory structure is similar to the template selected by cookiecutter, except that it places the source files in the src directory instead of the sample directory.

$ tree sample
sample
├── AUTHORS.rst
├── CHANGELOG.rst
├── CONTRIBUTING.rst
├── LICENSE.txt
├── README.rst
├── docs
│ ├── Makefile
│ ├── _static
│ ├── authors.rst
│ ├── changelog.rst
│ ├── conf.py
│ ├── contributing.rst
│ ├── index.rst
│ ├── license.rst
│ ├── readme.rst
│ └── requirements.txt
├── pyproject.toml
├── setup.cfg
├── setup.py
├── src
│ └── sample
│ ├── __init__.py
│ └── skeleton.py
├── tests
│ ├── conftest.py
│ └── test_skeleton.py
└── tox.ini

The entire project requires the use of tox. tox is an automated testing and build tool that creates a Python virtual environment during the build process, allowing a clean environment for testing and building.

tox -av can display all the tasks defined in tox.ini:

$ tox -av
default environments:
default -> Invoke pytest to run automated tests
additional environments:
build -> Build the package in isolation according to PEP517, see https://github.com/pypa/build
clean -> Remove old distribution files and temporary build artifacts (./build and ./dist)
docs-> Invoke sphinx-build to build the docs
doctests-> Invoke sphinx-build to run doctests
linkcheck -> Check for broken links in the documentation
publish -> Publish the package you have been developing to a package index server. By default, it uses testpypi. If you really want to publish your package to be publicly accessible in PyPI, use the `-- --repository pypi` option.

Use tox -e build, tox -e docs, etc. to which command you want to execute

During my experience with the tox command, every step seemed to be slow. It should take some time to create a virtual machine.

PyBuilder

It’s best to look at another build tool PyBuilder. The directory structure it creates is very close to Maven. Let’s take a look

$ pip install pybuilder
$ mkdir sample && cd sample# 项目目录需手工创建
$ pyb --start-project# 回答一些问题后创建所需的目录和文件

After finishing, take a look at its directory structure:

$ tree sample
.
├── build.py
├── docs
├── pyproject.toml
├── setup.py
└── src
 ├── main
 │ ├── python
 │ └── scripts
 └── unittest
 └── python

The building process still uses the pyb command. You can use pyb -h to view the help, and pyb -t to list all tasks. The tasks of PyBuilder are in the form of plug-ins. Added, the plug-in configuration is in the build.py file.

$ pyb -t sample
Tasks found for project "sample":
 analyze -Execute analysis plugins.
 depends on tasks: prepare run_unit_tests
 clean - Cleans the generated output.
 compile_sources - Compiles source files that need compilation.
 depends on tasks: prepare
coverage - <no description available>
 depends on tasks: verify
 install - Installs the published project.
 depends on tasks: package publish(optional)
 package - Packages the application. Package a python application.
 depends on tasks: compile_sources run_unit_tests(optional)
 prepare - Prepares the project for building. Creates target VEnvs
 print_module_path - Print the module path.
print_scripts_path - Print the script path.
 publish - Publishes the project.
 depends on tasks: package verify(optional) coverage(optional)
 run_integration_tests - Runs integration tests on the packaged application.
 depends on tasks: package
run_unit_tests - Runs all unit tests. Runs unit tests based on Python's unittest module
 depends on tasks: compile_sources
upload - Upload a project to PyPi.
verify - Verifies the project and possibly integration tests.
 depends on tasks: run_integration_tests(optional)
$ pyb run_unit_tests sample

PyBuilder also creates a virtual environment before building or testing. Starting from version 0.12.9, you can skip the step of creating a virtual environment through the parameter --no-venvs. If --no-venvs is used, the Python code will be executed in the current Python environment running pyb, and the required dependencies will have to be installed manually.

The dependencies of the project must also be defined in the build.py file.

@init
def set_properties(project):
 project.depends_on('boto3', '>=1.18.52')
 project.build_depends_on('mock')

The above dependencies will be installed when pyb is executed to create a virtual environment, and tests and builds will be run in it.

Poetry

The last Poetry, I feel that this is a more mature and more active Python build. It has more powerful trust management functions. Use poetry add boto3 to add dependencies, poetry show --tree displays the dependency tree. Take a look at how to install and create a project

$ pip install poetry
$ poetry new sample

The project it creates is simpler than the above

$ tree sample
sample
├── README.rst
├── pyproject.toml
├── sample
│ └── __init__.py
└── tests
 ├── __init__.py
 └── test_sample.py

If you give poetry new the --src parameter, the source file directory sample will be placed in src directory, that is, sample/src/sample.

poetry init will generate the pyproject.toml file in the current directory, and the generation of directories must be completed manually.

It does not focus on document generation, code specification checking, or code coverage. Its project configuration is more centralized, all in the pyproject.toml file. What is toml? It is a configuration file format Tom's Obvious, Minimal Language (https://github.com/toml-lang/toml).

pyproject.toml is somewhat similar to the NodeJS package.json file. For example, poetry add, poetry install command lines

# 往 pyproject.toml 中添加对boto3 的依赖并安装(add 还能从本地或 git 来安装依赖 ),
poetry add boto3
# 将依照 pyproject.toml 文件中定义安装相应的依赖到当前的 Python 虚拟环境中
# 比如在 <test-venv>/lib/python3.9/site-packages 目录中，安装好模块后也可让测试用例使用
poetry install

其他主要的

1.poetry build# 构建可安装的 *.whl 和 tar.gz 文件
2.poetry shell# 会根据定义在 pyproject.toml 文件中的依赖创建并使用虚拟环境
3.poetry run pytest# 运行使用 pytest 的测试用例，如 tests/test_sample.py
4.poetry run python -m unittest tests/sample_tests.py# 运行 unittest 测试用例
5.poetry export --without-hashes --output requirements.txt# 导出 requirements.txt 文件, --dev导出含 dev 的依赖，或者用 poetry export --without-hashes > requirements.txt

poetry run 能执行任何系统命令，只是它会在它要的虚拟环境中执行。所以可以想见，poetry 的项目要生成文档或覆盖率都必须用 poetry run ... 命令来支持 sphinx, coverage 或 flake8。

在 sample 目录(与 pyproject.toml 文件平级)中创建文件 my_module.py, 内容为

def main():
 print('hello poetry')

然后在 pyproject.toml 中写上。

[tool.poetry.scripts]
my-script="sample.my_module:main"

再执行

$ poetry run my-script

就会输出 "hello poetry"。

通过对以上四个工具的认识，项目结构的复杂度由 cookiecutter-pyproject -> PyScaffold -> PyBuilder -> Poetry 依次降低，使用的难度大略也是相同的顺序

The above is the detailed content of Four Python project management and construction tools, recommended collection!. For more information, please follow other related articles on the PHP Chinese website!