搜尋
首頁資料庫mysql教程mysql随机查询效率优化

mysql随机查询效率优化

Jun 07, 2016 pm 03:34 PM
mysql最佳化效率查詢研究隨機需要

最近由于需要研究了一下MYSQL的随机抽取实现方法。举个例子,要从tablename表中随机提取一条记录,大家一般的写法就是: SELECT * FROM content ORDER BY RAND () LIMIT 1 【3万条记录查询花费 0.3745 秒(下同);从mysql slow query log看出“ORDER BY RAN


最近由于需要研究了一下MYSQL的随机抽取实现方法。举个例子,要从tablename表中随机提取一条记录,大家一般的写法就是:

<code><span>SELECT </span><span>*</span><span> FROM content ORDER BY RAND</span><span>()</span><span> LIMIT </span><span>1</span></code>

【3万条记录查询花费 0.3745 秒(下同);从mysql slow query log看出“ORDER BY RAND() ”全表扫描了2次!】

后来我查了一下MYSQL的官方手册,里面针对RAND()的提示大概意思就是,在ORDER BY从句里面不能使用RAND()函数,因为这样会导致数据列被多次扫描。但是在MYSQL 3.23版本中,仍然可以通过ORDER BY RAND()来实现随机。

但是真正测试一下才发现这样效率非常低。一个15万余条的库,查询5条数据,居然要8秒以上。查看官方手册,也说rand()放在ORDER BY 子句中会被执行多次,自然效率及很低。

搜索Google,采用JOIN,查询max(id) * rand()来随机获取数据。

<code><span>SELECT </span><span>*</span><span> 
FROM </span><span>`content`</span><span> AS t1 JOIN </span><span>(</span><span>SELECT ROUND</span><span>(</span><span>RAND</span><span>()</span><span>*</span><span>(</span><span>SELECT MAX</span><span>(</span><span>id</span><span>)</span><span> FROM </span><span>`content`</span><span>))</span><span> AS id</span><span>)</span><span> AS t2 
WHERE t1</span><span>.</span><span>id </span><span>>=</span><span> t2</span><span>.</span><span>id 
ORDER BY t1</span><span>.</span><span>id ASC LIMIT </span><span>1</span><span>;</span></code>

【查询花费 0.0008 秒,飘易认为可以推荐使用这个语句!!】

但是这样会产生连续的5条记录。解决办法只能是每次查询一条,查询5次。即便如此也值得,因为15万条的表,查询只需要0.01秒不到。

有一个方法:

<code><span>SELECT </span><span>*</span><span> FROM </span><span>`content`</span><span> AS a JOIN </span><span>(</span><span> SELECT MAX</span><span>(</span><span> ID </span><span>)</span><span> AS ID FROM </span><span>`content`</span><span>)</span><span> AS b ON </span><span>(</span><span> a</span><span>.</span><span>ID </span><span>>=</span><span> FLOOR</span><span>(</span><span> b</span><span>.</span><span>ID </span><span>*</span><span> RAND</span><span>(</span><span>)</span><span>)</span><span>)</span><span> LIMIT </span><span>5</span><span>;</span></code>

上面这种方式保证了一定范围内的随机,查询花费 0.4265 秒,也不推荐。

下面的语句,mysql的论坛上有人使用

<code><span>SELECT </span><span>*</span><span> 
FROM </span><span>`content`</span><span> 
WHERE id </span><span>>=</span><span>(</span><span>SELECT FLOOR</span><span>(</span><span> MAX</span><span>(</span><span>id</span><span>)</span><span>*</span><span> RAND</span><span>())</span><span> FROM </span><span>`content`</span><span>)</span><span> 
ORDER BY id LIMIT </span><span>1</span><span>;</span></code>

【查询花费 1.2254 秒,飘易强烈不推荐!因为实测后,3万行的表,这个语句竟然会扫描500万行!!】

跟上面的语句还是有很大差距。总觉有什么地方不正常。于是我把语句改写了一下。

<code><strong><span><span>SELECT </span><span>*</span><span> FROM </span><span>`content`</span><span> 
WHERE id </span><span>>=</span><span>(</span><span>SELECT floor</span><span>(</span><span>RAND</span><span>()</span><span>*</span><span>(</span><span>SELECT MAX</span><span>(</span><span>id</span><span>)</span><span> FROM </span><span>`content`</span><span>)))</span><span>  
ORDER BY id LIMIT </span><span>1</span><span>;</span></span></strong></code>

【查询花费 0.0012 秒】

这下,效率又提高了,查询时间只有0.01秒

最后,再把语句完善一下,加上MIN(id)的判断。我在最开始测试的时候,就是因为没有加上MIN(id)的判断,结果有一半的时间总是查询到表中的前面几行。

完整查询语句是:

<code><span>SELECT </span><span>*</span><span> FROM </span><span>`content`</span><span> 
WHERE id </span><span>>=</span><span>(</span><span>SELECT floor</span><span>(</span><span> RAND</span><span>()</span><span>*</span><span>((</span><span>SELECT MAX</span><span>(</span><span>id</span><span>)</span><span> FROM </span><span>`content`</span><span>)-(</span><span>SELECT MIN</span><span>(</span><span>id</span><span>)</span><span> FROM </span><span>`content`</span><span>))</span><span>+</span><span>(</span><span>SELECT MIN</span><span>(</span><span>id</span><span>)</span><span> FROM </span><span>`content`</span><span>)))</span><span>  
ORDER BY id LIMIT </span><span>1</span><span>;</span></code>

【查询花费 0.0012 秒】

<code><span>SELECT </span><span>*</span><span> 
FROM </span><span>`content`</span><span> AS t1 JOIN </span><span>(</span><span>SELECT ROUND</span><span>(</span><span>RAND</span><span>()</span><span>*</span><span>((</span><span>SELECT MAX</span><span>(</span><span>id</span><span>)</span><span> FROM </span><span>`content`</span><span>)-(</span><span>SELECT MIN</span><span>(</span><span>id</span><span>)</span><span> FROM </span><span>`content`</span><span>))+(</span><span>SELECT MIN</span><span>(</span><span>id</span><span>)</span><span> FROM </span><span>`content`</span><span>))</span><span> AS id</span><span>)</span><span> AS t2 
WHERE t1</span><span>.</span><span>id </span><span>>=</span><span> t2</span><span>.</span><span>id 
ORDER BY t1</span><span>.</span><span>id LIMIT </span><span>1</span><span>;</span></code>

【查询花费 0.0008 秒】

最后在php中对这两个语句进行分别查询10次,

前者花费时间 0.147433 秒

后者花费时间 0.015130 秒

看来采用JOIN的语法比直接在WHERE中使用函数效率还要高很多。(via)

======================================

【好了,最后飘易来总结下】:

第一种方案,即原始的 Order By Rand() 方法:

<code><span>$sql</span><span>=</span><span>"SELECT * FROM content ORDER BY rand() LIMIT 12"</span><span>;</span><span>
$result</span><span>=</span><span>mysql_query</span><span>(</span><span>$sql</span><span>,</span><span>$conn</span><span>);</span><span>
$n</span><span>=</span><span>1</span><span>;</span><span>
$rnds</span><span>=</span><span>''</span><span>;</span><span>while</span><span>(</span><span>$row</span><span>=</span><span>mysql_fetch_array</span><span>(</span><span>$result</span><span>)){</span><span>
$rnds</span><span>=</span><span>$rnds</span><span>.</span><span>$n</span><span>.</span><span>". <a href="show%22</span><span>.</span><span>%24row</span><span>%5B</span><span>" id><span>].</span><span>"-"</span><span>.</span><span>strtolower</span><span>(</span><span>trim</span><span>(</span><span>$row</span><span>[</span><span>'title'</span><span>])).</span><span>"'>"</span><span>.</span><span>$row</span><span>[</span><span>'title'</span><span>].</span><span>"</span></a><br>\n"</span><span>;</span><span>
$n</span><span>++;</span><span>}</span></code>

3万条数据查12条随机记录,需要0.125秒,随着数据量的增大,效率越来越低。

第二种方案,改进后的 JOIN 方法:

<code><span>for</span><span>(</span><span>$n</span><span>=</span><span>1</span><span>;</span><span>$n</span><span><span>12</span><span>;</span><span>$n</span><span>++){</span><span>
$sql</span><span>=</span><span>"SELECT * FROM `content` AS t1 
JOIN (SELECT ROUND(RAND() * (SELECT MAX(id) FROM `content`)) AS id) AS t2 
WHERE t1.id >= t2.id ORDER BY t1.id ASC LIMIT 1"</span><span>;</span><span>
$result</span><span>=</span><span>mysql_query</span><span>(</span><span>$sql</span><span>,</span><span>$conn</span><span>);</span><span>
$yi</span><span>=</span><span>mysql_fetch_array</span><span>(</span><span>$result</span><span>);</span><span>
$rnds </span><span>=</span><span> $rnds</span><span>.</span><span>$n</span><span>.</span><span>". <a href="show%22</span><span>.</span><span>%24yi</span><span>%5B</span><span>" id><span>].</span><span>"-"</span><span>.</span><span>strtolower</span><span>(</span><span>trim</span><span>(</span><span>$yi</span><span>[</span><span>'title'</span><span>])).</span><span>"'>"</span><span>.</span><span>$yi</span><span>[</span><span>'title'</span><span>].</span><span>"</span></a><br>\n"</span><span>;</span><span>}</span></span></code>

3万条数据查12条随机记录,需要0.004秒,效率大幅提升,比第一种方案提升了约30倍。缺点:多次select查询,IO开销大。

第三种方案,SQL语句先随机好ID序列,用 IN 查询(飘易推荐这个用法,IO开销小,速度最快):

<code><span>$sql</span><span>=</span><span>"SELECT MAX(id),MIN(id) FROM content"</span><span>;</span><span>
$result</span><span>=</span><span>mysql_query</span><span>(</span><span>$sql</span><span>,</span><span>$conn</span><span>);</span><span>
$yi</span><span>=</span><span>mysql_fetch_array</span><span>(</span><span>$result</span><span>);</span><span>
$idmax</span><span>=</span><span>$yi</span><span>[</span><span>0</span><span>];</span><span>
$idmin</span><span>=</span><span>$yi</span><span>[</span><span>1</span><span>];</span><span>
$idlist</span><span>=</span><span>''</span><span>;</span><span>for</span><span>(</span><span>$i</span><span>=</span><span>1</span><span>;</span><span>$i</span><span><span>20</span><span>;</span><span>$i</span><span>++){</span><span>if</span><span>(</span><span>$i</span><span>==</span><span>1</span><span>){</span><span> $idlist</span><span>=</span><span>mt_rand</span><span>(</span><span>$idmin</span><span>,</span><span>$idmax</span><span>);</span><span>}</span><span>else</span><span>{</span><span> $idlist</span><span>=</span><span>$idlist</span><span>.</span><span>','</span><span>.</span><span>mt_rand</span><span>(</span><span>$idmin</span><span>,</span><span>$idmax</span><span>);</span><span>}</span><span>}</span><span>  
$idlist2</span><span>=</span><span>"id,"</span><span>.</span><span>$idlist</span><span>;</span><span>
$sql</span><span>=</span><span>"select * from content where id in ($idlist) order by field($idlist2) LIMIT 0,12"</span><span>;</span><span> 
$result</span><span>=</span><span>mysql_query</span><span>(</span><span>$sql</span><span>,</span><span>$conn</span><span>);</span><span>
$n</span><span>=</span><span>1</span><span>;</span><span>
$rnds</span><span>=</span><span>''</span><span>;</span><span>while</span><span>(</span><span>$row</span><span>=</span><span>mysql_fetch_array</span><span>(</span><span>$result</span><span>)){</span><span>
$rnds</span><span>=</span><span>$rnds</span><span>.</span><span>$n</span><span>.</span><span>". <a href="show%22</span><span>.</span><span>%24row</span><span>%5B</span><span>" id><span>].</span><span>"-"</span><span>.</span><span>strtolower</span><span>(</span><span>trim</span><span>(</span><span>$row</span><span>[</span><span>'title'</span><span>])).</span><span>"'>"</span><span>.</span><span>$row</span><span>[</span><span>'title'</span><span>].</span><span>"</span></a><br>\n"</span><span>;</span><span>
$n</span><span>++;</span><span>}</span></span></code>

3万条数据查12条随机记录,需要0.001秒,效率比第二种方法又提升了4倍左右,比第一种方法提升120倍。注,这里使用了 order by

field($idlist2) 是为了不排序,否则 IN 是自动会排序的。缺点:有可能遇到ID被删除的情况,所以需要多选几个ID。

测试方法:

<code><span>$t </span><span>=</span><span> microtime</span><span>(</span><span>true</span><span>);</span><span>//执行语句</span><span>
echo microtime</span><span>(</span><span>true</span><span>)</span><span>-</span><span> $t</span><span>;</span></code>



参考:
http://blog.csdn.net/zxl315/article/details/2435368
http://jan.kneschke.de/projects/mysql/order-by-rand/
陳述
本文內容由網友自願投稿,版權歸原作者所有。本站不承擔相應的法律責任。如發現涉嫌抄襲或侵權的內容,請聯絡admin@php.cn
您可以使用哪些工具來監視MySQL性能?您可以使用哪些工具來監視MySQL性能?Apr 23, 2025 am 12:21 AM

如何有效監控MySQL性能?使用mysqladmin、SHOWGLOBALSTATUS、PerconaMonitoringandManagement(PMM)和MySQLEnterpriseMonitor等工具。 1.使用mysqladmin查看連接數。 2.用SHOWGLOBALSTATUS查看查詢數。 3.PMM提供詳細性能數據和圖形化界面。 4.MySQLEnterpriseMonitor提供豐富的監控功能和報警機制。

MySQL與SQL Server有何不同?MySQL與SQL Server有何不同?Apr 23, 2025 am 12:20 AM

MySQL和SQLServer的区别在于:1)MySQL是开源的,适用于Web和嵌入式系统,2)SQLServer是微软的商业产品,适用于企业级应用。两者在存储引擎、性能优化和应用场景上有显著差异,选择时需考虑项目规模和未来扩展性。

在哪些情況下,您可以選擇SQL Server而不是MySQL?在哪些情況下,您可以選擇SQL Server而不是MySQL?Apr 23, 2025 am 12:20 AM

在需要高可用性、高級安全性和良好集成性的企業級應用場景下,應選擇SQLServer而不是MySQL。 1)SQLServer提供企業級功能,如高可用性和高級安全性。 2)它與微軟生態系統如VisualStudio和PowerBI緊密集成。 3)SQLServer在性能優化方面表現出色,支持內存優化表和列存儲索引。

MySQL如何處理角色集和碰撞?MySQL如何處理角色集和碰撞?Apr 23, 2025 am 12:19 AM

mySqlManagesCharacterSetsetSandCollat​​ionsyutusututf-8asthEdeFault,允許ConfigurationAtdataBase,table和columnlevels,AndrequiringCarefullageLignmentToavoidMismatches.1)setDefeaultCharactersetTercharactersetEtCollacterSeteTandColletationForAdataBase.2)conformentcollecharactersettersetertersetcollat​​ertersetcollat​​ioncollat​​ion

MySQL中有什麼觸發器?MySQL中有什麼觸發器?Apr 23, 2025 am 12:11 AM

MySQL觸發器是與表相關聯的自動執行的存儲過程,用於在特定數據操作時執行一系列操作。 1)觸發器定義與作用:用於數據校驗、日誌記錄等。 2)工作原理:分為BEFORE和AFTER,支持行級觸發。 3)使用示例:可用於記錄薪資變更或更新庫存。 4)調試技巧:使用SHOWTRIGGERS和SHOWCREATETRIGGER命令。 5)性能優化:避免複雜操作,使用索引,管理事務。

您如何在MySQL中創建和管理用戶帳戶?您如何在MySQL中創建和管理用戶帳戶?Apr 22, 2025 pm 06:05 PM

在MySQL中創建和管理用戶賬戶的步驟如下:1.創建用戶:使用CREATEUSER'newuser'@'localhost'IDENTIFIEDBY'password';2.分配權限:使用GRANTSELECT,INSERT,UPDATEONmydatabase.TO'newuser'@'localhost';3.修正權限錯誤:使用REVOKEALLPRIVILEGESONmydatabase.FROM'newuser'@'localhost';然後重新分配權限;4.優化權限:使用SHOWGRA

MySQL與Oracle有何不同?MySQL與Oracle有何不同?Apr 22, 2025 pm 05:57 PM

MySQL適合快速開發和中小型應用,Oracle適合大型企業和高可用性需求。 1)MySQL開源、易用,適用於Web應用和中小型企業。 2)Oracle功能強大,適合大型企業和政府機構。 3)MySQL支持多種存儲引擎,Oracle提供豐富的企業級功能。

與其他關係數據庫相比,使用MySQL的缺點是什麼?與其他關係數據庫相比,使用MySQL的缺點是什麼?Apr 22, 2025 pm 05:49 PM

MySQL相比其他關係型數據庫的劣勢包括:1.性能問題:在處理大規模數據時可能遇到瓶頸,PostgreSQL在復雜查詢和大數據處理上表現更優。 2.擴展性:水平擴展能力不如GoogleSpanner和AmazonAurora。 3.功能限制:在高級功能上不如PostgreSQL和Oracle,某些功能需要更多自定義代碼和維護。

See all articles

熱AI工具

Undresser.AI Undress

Undresser.AI Undress

人工智慧驅動的應用程序,用於創建逼真的裸體照片

AI Clothes Remover

AI Clothes Remover

用於從照片中去除衣服的線上人工智慧工具。

Undress AI Tool

Undress AI Tool

免費脫衣圖片

Clothoff.io

Clothoff.io

AI脫衣器

Video Face Swap

Video Face Swap

使用我們完全免費的人工智慧換臉工具,輕鬆在任何影片中換臉!

熱工具

SublimeText3漢化版

SublimeText3漢化版

中文版,非常好用

Safe Exam Browser

Safe Exam Browser

Safe Exam Browser是一個安全的瀏覽器環境,安全地進行線上考試。該軟體將任何電腦變成一個安全的工作站。它控制對任何實用工具的訪問,並防止學生使用未經授權的資源。

記事本++7.3.1

記事本++7.3.1

好用且免費的程式碼編輯器

SAP NetWeaver Server Adapter for Eclipse

SAP NetWeaver Server Adapter for Eclipse

將Eclipse與SAP NetWeaver應用伺服器整合。

SublimeText3 Linux新版

SublimeText3 Linux新版

SublimeText3 Linux最新版