Detailed explanation of how to package and publish Python modules

Preface

Yesterday I packaged my VASP file processing library and uploaded it to PyPI. Now you can install VASPy directly through pip and easy_install (at the same time, children who use VASP to do computational chemistry are welcome to add stars) and get involved),

VASPy’s GotHub address: https://github.com/PytLab/VASPy
VASPy’s PyPI address: https://pypi.python.org/pypi/vaspy/

Since my memory is really bad and I am afraid that I will forget it after a long time, I will use my VASPy program as an example to summarize the packaging and uploading of python.

VASPy package file structure

First write and paste the entire file structure of the VASPy package. The following content will be explained using this example:

VASPy/
├── LICENSE
├── MANIFEST
├── MANIFEST.in
├── README.rst
├── requirements.txt
├── scripts
│  ├── change_incar_parameters.py
│  ├── create_inputs.py
│  └── ...
├── setup.cfg
├── setup.py
├── tests
│  ├── incar_test.py
│  ├── __init__.py
│  ├── oszicar_test.py
│  ├── outcar_test.py
│  ├── testdata
│  │  ├── CONTCAR
│  │  ├── DOS_SUM
│  │  ├── ELFCAR
│  │  └── ...
│  └── ...
└── vaspy
  ├── __init__.py
  ├── iter.py
  ├── matstudio.py
  └── ...
  
4 directories, 54 files

Packaging and installation chapter Third-party package tools

Here we need to use tools such as setuptools and pip to package, publish and install our own packages. If we need to build a wheel, we also need to install the wheel module. If the python version >=2.7.9 or >=3.4, setuptools and pip are already installed, and you may need to update to the latest version

pip install -U pip setuptools

You can use package management tools, such as

yum install pip
sudo apt-get install pip

to install through the get-pip.py script. If it is detected that wheel and setuptools are not installed, they will be automatically installed

python get-pip.py

I won’t go into details about the specific tool installation and introduction. You can refer to the requirements for installing packages

The functions of different files in the package

setup.py

This file is the most important file for packaging the entire project. It provides two main functions:

setup() function. The parameters of this function specify how to configure your own project.
Command line tools, including packaging, testing, publishing, etc. You can view it through the following command;

python setup.py --help-commands

setup.cfg

This file contains some default parameters during build, such as building bdist_wheel The --universal parameter

[bdist_wheel]
universal=1

will be used by default every time when packaging. The effect is similar:

python setup.py bdist_wheel --universal

README.rst

I originally wrote this in markdown. After packaging and publishing it to PyPI, I found that PyPI did not support markdown rendering. The page was really confusing, so I re-written it using the syntax of reStrutruedText. Wrote it again. After all, markup language syntax can basically be learned in seconds. If it is really not possible, just find a template and draw the gourd.
The syntax rules of reStructureText can refer to the official document: Quick reStructuredText

In fact, another way is to use pandoc to convert markdown into rst format. A trouble-free way is to use the pyandoc module to automatically convert when publishing. .
For specific methods, please refer to: Use Markdown README's in Python modules

MANIFEST.in

This file tells setuptools when packaging that additional files need to be packaged, such as the units in my VASPy I will use this file to include the test data file for the test. Of course, README and LICENSE can also be packaged together through it.
The following is the content of my own MANIFEST.in:

include README.rst
include requirements.txt
include LICENSE
recursive-include scripts *
recursive-include tests *

For specific grammar rules, please refer to: The MANIFEST.in template

vaspy/

This folder It is the package where the vaspy source code is located.

tests/

This folder is also a sub-package and contains unit test scripts. In order to use python setup.py test for unit testing, __init__.pys is specially added to make it A bag.

Parameters of setup()

Here I only introduce a few parameters I use. For the specific use of other parameters, please refer to: https://docs.python.org/3/distutils/setupscript .html

name

versions = "vaspy"

is the name of the entire project. This name and version number will be used after packaging.

version

from vaspy import __version__
version = __version__

description

is a short description of the project, usually just one sentence, and will be displayed at the bottom of the name on pypi.

long_description

is a long description, which is equivalent to a concise description of the project. If this string is in rst format, PyPI will automatically render it into HTML for display. The contents in README.rst can be read directly here.

url

The link to the package, usually a link on GitHub or a link to readthedocs.

packages

A list of subpackages that need to be included. Setuptools provides find_packages() to help us find packages in the root path. This function distutil does not have.

setup_requires

This parameter defines other dependencies (the most basic) required for VASPy installation and smooth operation. These dependencies will be installed when using pip to install.
For the difference between this parameter and requirements.txt, please refer to: install_requires vs Requirements files

classifier

This parameter provides a series of classifications, which will be put into different categories on PyPI The items are categorized in the directory.
Reference for specific category names and rules: https://pypi.python.org/pypi?%3Aaction=list_classifiers

test_suite

This parameter can help us use

python setup.py test

to run unit tests, you no longer need to write a separate script such as run_tests.py to run unit tests.
Official explanation of this parameter:

A string naming a unittest.TestCase subclass (or a package or module containing one or more of them, or a method of such a subclass), or naming a function that can be called with no arguments and returns a unittest.TestSuite. If the named suite is a module, and the module has an additional_tests() function, it is called and the results are added to the tests to be run. If the named suite is a package, any submodules and subpackages are recursively added to the overall test suite.

也就是说这个参数可以接受多种类型的参数：

接收unittest.TestCase子类，我们可以讲所有单元测试写入一个测试用例中，然后import进来，再传你给test_suite
接收函数对象，此函数对象没有任何参数，且返回一个unittest.TestSuite.这样我们就可以单独写一个函数，将多个测试用例合并成一个suite然后返回，然后再将函数import进来传给test_suite。

模块和包名称，我就是使用这种方式，之前自己的测试都是分开的多个脚本，这样我添加一个__init__.py就可以将其变成一个包，将包名传给test_suite，setuptools就会神奇的将此包下的所有测试全部跑一边，这样我以后再加测试脚本的时候直接就添加新的脚本就好了，其他的都不需要改动了。

运行效果：

zjshao@SHAO-PC:/mnt/d/Dropbox/Code/CentOS_code/VASPy$ python setup.py test
running test
running egg_info
creating vaspy.egg-info
writing vaspy.egg-info/PKG-INFO
writing top-level names to vaspy.egg-info/top_level.txt
writing dependency_links to vaspy.egg-info/dependency_links.txt
writing manifest file 'vaspy.egg-info/SOURCES.txt'
reading manifest file 'vaspy.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'vaspy.egg-info/SOURCES.txt'
running build_ext
test_compare (tests.incar_test.InCarTest)
Make sure we can compare two InCar objects correctly. ... ok
test_eq (tests.incar_test.InCarTest)
Test __eq__() function. ... ok
...
此处省略若干输出
  
----------------------------------------------------------------------
Ran 22 tests in 3.574s
  
OK

发布自己的python包

1. 首先先去PyPI注册帐号

2. 配置~/.pypirc如下：

[distutils]
index-servers =
  pypi
  pypitest
  
[pypi]
username:ShaoZhengjiang
password:mypassword
  
[pypitest]
username:ShaoZhengjiang
password:mypassword

3. 然后注册并上传自己的包到测试服务器

pypi提供了一个测试服务器，我们可以在这个测试服务器上做测试。

python setup.py register -r pypitest

然后

python setup.py sdist upload -r pypitest

若没有问题我们应该不会得到任何错误。

4. 上传至PyPI

若上面的测试成功，我们就可以按照相同的步骤将包注册并上传。

python setup.py register -r pypi
python setup.py sdist upload -r pypi

Ok,之后我们就可以在PyPI(https://pypi.python.org/pypi/vaspy/)上看到我们自己的包了。

更多打包发布Python模块的方法详解相关文章请关注PHP中文网！