Please give me some advice on how to use thinkphp5.1 to process large amounts of mysql data.

Question

I have a requirement here,

Local database (one):

One table user_local,

Remote database (two databases in total) , five tables,) all information is related through uid

There is only one table user_basic in one database (user_data_a), which is the basic information of all users,

Another database The other four tables in (user_data_b) are divided into two categories.

One category (two of the tables, user_profile_active and user_profile_not_active) is used to store user files, one table is for active users, and the other table is Inactive users,

Another type of table (the other two tables user_ext_info_active and user_ext_info_not_active) is used to store other user information. One table is also an active user, and the other table is an inactive user,

For example,

If the record of a uid in user_basic is not in user_profile_active, then it is in user_profile_not_active,

If it is not in user_ext_info_active, then it is in In user_ext_info_not_active

Theoretically speaking, active corresponds to active, and not_active corresponds to not_active.

The current demand is to focus on user_basic and go to those two categories. Obtain certain fields of the user's corresponding information from the table (four in total), and then combine them into one record and write it to the local user_local table. There are about 800,000 records. Master Qiu can give you some advice on how to deal with it to avoid looping. Query the database in order to avoid memory overflow and timeout~~ Guiqiu

The framework is thinkphp5.1

Answer 1

If your MySQL data is powerful, you can write joint table operations and complete it with one statement. However, it is indeed easy to cause timeout and memory overflow. It is recommended to operate separately. There is no shortcut for this. One table only has 800,000 data. If these few The tables are all linked together to query the huge amount of data. It’s better to be honest and finish operating the two data tables user_profile_active and user_profile_not_active first, and then operate the next two tables user_ext_info_active and user_ext_info_not_active