Summary of python database connection pool technology

If you feel that it takes too long to request the database with a thread, you can use the python database connection pool to improve the shortcomings in this aspect. The following is a detailed introduction to the article. You can use our article Have a better understanding of python database connection pooling.

I tested yesterday by opening 500 threads to request the database, but I don’t know how much time it will take. That is, what is the efficiency of launching so many threads at the same time? So I thought about whether using database connection pool technology can significantly improve such connection operations. After sorting it out for a while, I will test a piece of data: the performance comparison between the efficiency of frequently establishing and closing database connections and the connection pool!

1. DBUtils module learning

DBUtils is actually a Python package containing two sub-modules, one for connecting to the DB-API 2 module and the other for connecting to the typical PyGreSQL module. The global DB-API 2 variable

SteadyDB.py

is used to stabilize the database connection

PooledDB.py

Connection pool

PersistentDB.py

Maintain a persistent database connection (persistent connection)

SimplePooledDB.py

Simple connection pool PS: Let’s extract the DB-API first

Install the two modules from the top-level module to provide basic services, PersistentDB and PooledDB.

DBUtils.PersistentDB implements tough, thread-safe, persistent database connections using the DB-API 2 module. The following figure shows the connection layer steps when using PersistentDB: DBUtils.PooledDB implements a tough, thread-safe, cached, reusable database connection, using any DB-API 2 module. The following figure shows the workflow when using PooledDB:

There are currently two modules for us to choose from: PersistentDB and PooledDB, both of which are designed to reuse database connections to improve performance and maintain database stability.

python setup.py install

Specific module learning:

DBUtils.SimplePooledDB is a very simple database connection pool implementation. It lacks many features than the full-fledged PooledDB module. DBUtils.SimplePooledDB is essentially similar to MiscUtils.DBPool, a component of Webware. You can think of it as a demo program

DBUtils.SteadyDB is a module that implements "strong" database connections, based on the original connections established by DB-API 2. A "hard" connection means that it will be reconnected after the connection is closed, or when the limit of the number of operations is used. A typical example is when the database is restarted while your program is still running and needs to access the database, or when your program connects to a remote database behind a firewall and state is lost when the firewall is restarted.

Generally speaking you don’t need to use SteadyDB directly, it just takes over