search
HomeDatabaseMysql TutorialMySQL”海量数据”查询性能分析_MySQL

bitsCN.com

最近做了一次MySQL所谓的”海量数据”查询性能分析.

表结构

dt dt2 dt3 it it2 it3   id id id id id id int PK     ext1     ext1 varchar(256) time time time time time time int/datetime KEY   ext2 ext2   ext2 ext2 varchar(128)

说明, MyISAM引擎, dt表示时间字段使用datetime类型, it表示时间字段使用int类型.

初始数据

首先生成100K个UNIX时间戳(int), 然后随机选取10M次, 每一次往6个表里插入一条记录(当time字段是datetime类型时, 做类型转换). 所以每一个表都有10M条记录. ext1和ext2字段会用随机的字符串填充.

 

SQL查询

使用的查询SQL语句如:

select SQL_NO_CACHE count(*) from it where time>10000;select SQL_NO_CACHE count(*) from dt where time>from_unixtime(10000);select SQL_NO_CACHE * from it where time>10000 order by time limit 1;select SQL_NO_CACHE * from it use key(PRIMARY) where time>10000 order by id limit 1;

SQL_NO_CACHE用于消除查询结果缓存的影响. use key用于指定查询时使用的索引. 统计每一条SQL的执行时间(单位s)和满足WHERE条件的记录总数(total), it-tm表示在dt表上执行SQL的耗时, 并explain得到key和extra, 结果如下.

where total select orderby key it-tm dt-tm it2-tm dt2-tm it3-tm dt3-tm extra time>10000 8999050 count(*)   time 3.52 4.28 3.74 4.49 3.53 4.47 where; index     count(time)   time 3.44 4.00 3.69 4.36 3.56 4.26 where; index     count(id)   NULL 1.44 1.92 4.30 4.60 4.79 4.98 where     * time time 0.00 0.00 0.00 0.00 0.00 0.00 where     * id time 14.81 15.38 19.37 20.30 20.94 21.42 where; filesort     * id PK 0.00 0.03 0.00 0.02 0.00 0.04 where                         time>50000 4987990 count(*)     1.90 2.36 2.02 2.41 1.99 2.42       count(time)     1.90 2.23 2.01 2.32 1.96 2.29       count(id)     1.48 1.91 4.25 4.61 4.80 5.12       * time   0.00 0.00 0.00 0.00 0.00 0.00       * id   8.15 8.77 10.74 11.36 11.59 11.79       * id   0.00 0.00 0.00 0.00 0.00 0.00                           time>80000 1991982 count(*)     0.76 0.95 0.83 0.98 0.80 1.00       count(time)     0.77 0.91 0.81 0.91 0.83 0.92       count(id)     1.38 1.86 4.17 4.49 4.71 5.02       * time   0.00 0.00 0.00 0.00 0.00 0.00       * id   3.26 3.44 4.26 4.51 4.56 4.76       * id   0.00 0.00 0.00 0.00 0.00 0.00                           time>99900 10871 count(*)     0.00 0.00 0.00 0.01 0.01 0.00       count(time)     0.01 0.01 0.01 0.00 0.01 0.01       count(id)     0.01 0.01 0.02 0.03 0.02 0.02       * time   0.00 0.00 0.00 0.00 0.00 0.00       * id   0.02 0.02 0.03 0.03 0.03 0.03       * id   0.00 0.00 0.00 0.00 0.00 0.00  

分析和结论

count(*), count(time)和count(id)的对比. 结果有较大变化. 当表的字段只有2个且查询条件较宽松(即符合条件的记录数较多)时, count(id)比count(*)快很多, 但是, 当表中还有其它的字段时, count(id)反而更慢了. 虽然id是主键, time是索引列, 但是select count(id) where time并没有用到索引, 而是进行全表扫描. 当对count(*)进行ignore key(time)时, 查询时间和count(id)相同.证明当结果集较大时索引导致查询变慢,应该是全表扫描进行的是连续的磁盘IO和内存操作, 而使用索引是进行随机的磁盘IO和内存操作, 并且MyISAM存储索引的BTree结构占用更多的空间. 当WHERE条件约束更严格, total的值小到一定程度时, 全表扫描比使用索引慢, 因为索引极大减少了磁盘IO和内存操作.

排序字段和索引的使用. 当有排序且LIMIT(偏移为0)时,如果查询时使用的索引不是排序字段的索引, 那么速度非常慢. 当偏移不为0时, 如果使用排序列的索引, 要考虑偏移可能导致扫描的记录数, 所以应该根据情况选取合适的索引.

判断符合条件的记录是否存在, 使用select * limit 1速度要比select count(*)计数快得多.

时间字段类型的选择. int比datetime快, 但差距不是很明显.

无论如何, 条件限制得越严格, 查询就会越快.

另外, 根据随机id更新时, 大约能达到5K行/s.

列的先后顺序对查询性能的影响也非常大.

 

bitsCN.com
Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
How does MySQL handle data replication?How does MySQL handle data replication?Apr 28, 2025 am 12:25 AM

MySQL processes data replication through three modes: asynchronous, semi-synchronous and group replication. 1) Asynchronous replication performance is high but data may be lost. 2) Semi-synchronous replication improves data security but increases latency. 3) Group replication supports multi-master replication and failover, suitable for high availability requirements.

How can you use the EXPLAIN statement to analyze query performance?How can you use the EXPLAIN statement to analyze query performance?Apr 28, 2025 am 12:24 AM

The EXPLAIN statement can be used to analyze and improve SQL query performance. 1. Execute the EXPLAIN statement to view the query plan. 2. Analyze the output results, pay attention to access type, index usage and JOIN order. 3. Create or adjust indexes based on the analysis results, optimize JOIN operations, and avoid full table scanning to improve query efficiency.

How do you back up and restore a MySQL database?How do you back up and restore a MySQL database?Apr 28, 2025 am 12:23 AM

Using mysqldump for logical backup and MySQLEnterpriseBackup for hot backup are effective ways to back up MySQL databases. 1. Use mysqldump to back up the database: mysqldump-uroot-pmydatabase>mydatabase_backup.sql. 2. Use MySQLEnterpriseBackup for hot backup: mysqlbackup--user=root-password=password--backup-dir=/path/to/backupbackup. When recovering, use the corresponding life

What are some common causes of slow queries in MySQL?What are some common causes of slow queries in MySQL?Apr 28, 2025 am 12:18 AM

The main reasons for slow MySQL query include missing or improper use of indexes, query complexity, excessive data volume and insufficient hardware resources. Optimization suggestions include: 1. Create appropriate indexes; 2. Optimize query statements; 3. Use table partitioning technology; 4. Appropriately upgrade hardware.

What are views in MySQL?What are views in MySQL?Apr 28, 2025 am 12:04 AM

MySQL view is a virtual table based on SQL query results and does not store data. 1) Views simplify complex queries, 2) Enhance data security, and 3) Maintain data consistency. Views are stored queries in databases that can be used like tables, but data is generated dynamically.

What are the differences in syntax between MySQL and other SQL dialects?What are the differences in syntax between MySQL and other SQL dialects?Apr 27, 2025 am 12:26 AM

MySQLdiffersfromotherSQLdialectsinsyntaxforLIMIT,auto-increment,stringcomparison,subqueries,andperformanceanalysis.1)MySQLusesLIMIT,whileSQLServerusesTOPandOracleusesROWNUM.2)MySQL'sAUTO_INCREMENTcontrastswithPostgreSQL'sSERIALandOracle'ssequenceandt

What is MySQL partitioning?What is MySQL partitioning?Apr 27, 2025 am 12:23 AM

MySQL partitioning improves performance and simplifies maintenance. 1) Divide large tables into small pieces by specific criteria (such as date ranges), 2) physically divide data into independent files, 3) MySQL can focus on related partitions when querying, 4) Query optimizer can skip unrelated partitions, 5) Choosing the right partition strategy and maintaining it regularly is key.

How do you grant and revoke privileges in MySQL?How do you grant and revoke privileges in MySQL?Apr 27, 2025 am 12:21 AM

How to grant and revoke permissions in MySQL? 1. Use the GRANT statement to grant permissions, such as GRANTALLPRIVILEGESONdatabase_name.TO'username'@'host'; 2. Use the REVOKE statement to revoke permissions, such as REVOKEALLPRIVILEGESONdatabase_name.FROM'username'@'host' to ensure timely communication of permission changes.

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Safe Exam Browser

Safe Exam Browser

Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Atom editor mac version download

Atom editor mac version download

The most popular open source editor