SqlServer StringToTable性能测试-Mysql Tutorial-php.cn

Home

Database

Mysql Tutorial

SqlServer StringToTable性能测试

WBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWB

Jun 07, 2016 pm 03:45 PM

sqlserverperformancetest

问题起因：最近做的项目DB数据量比较大(基本上一个月的数据就是10亿)，而工程中Proc参数中包含有id拼接字符串，id拼接字符串格式：1,2,4,5,100,301。当数据量很小的情况下，这样做没有问题，但一旦数据量到达亿级，运行会很耗时，比如：当这样的参数id拼接

问题起因：

最近做的项目DB数据量比较大(基本上一个月的数据就是10亿)，而工程中Proc参数中包含有id拼接字符串，id拼接字符串格式：1,2,4,5,100,301。当数据量很小的情况下，这样做没有问题，但一旦数据量到达亿级，运行会很耗时，比如：当这样的参数id拼接字符串中包含有10万个id的时候（我们实际应用中确实有这么多个id需要传到数据库，而且这样的id是从库中取出后，又经过程序的筛选后剩余的id），像这样的语句：

<span>Declare</span> <span>@IDS</span> <span>nvarchar</span>(<span>max</span><span>);
</span><span>Set</span> <span>@IDS</span><span>=</span><span>'</span><span>10w个id用逗号分割组成的字符串</span><span>'</span><span>;
</span><span>Select</span> T10.<span>TEXT</span>,T10.Name <span>FROM</span> DX.M <span>as</span> T10 <span>inner</span> <span>join</span> dbo.StringToTable(<span>@IDS</span>,<span>'</span><span>,</span><span>'</span>) <span>as</span> T11 <span>on</span> T10.ID<span>=</span>T11.ID;

执行了18个小时还未查询出数据。

备注：

虚拟机配置：内存：64G；CPU核数：40。

DBA建议：

我测试了下，性能还算可以。在解析5000个逗号之内性能还行，太多了，性能就急速下降了。

最初的那个版本其实还是很常用的，性能要比改写之后的要好一些（在字符串特别长的情况下）。但是同样存在，如果字符串太长，性能急速下降的问题。

如果真的有5W以上逗号的字符串。这个SqlServer在执行计划上会消耗很多性能。

（自己也可以测试一下解析5000个逗号串和解析5W个字符串的差距，并不是5000字符串消耗时间*10的线性关系）

所以应当写一个循环，一次处理一部分。

比如以下两种方法：

1. 每次截取前1W个字符串，解析出来之后插入到临时表，然后在解析后面的，在插入到临时表，循环处理。最后临时表和实际表进行关联。

insert into #t1

select id

from dbo.stringtotable(@字符串1‍)

insert into #t1

select id

from dbo.stringtotable(@字符串2)‍

2。用in的方式，每次where条件 in 一部分。然后将结果union all起来。

类似如下

select id

from table a

where id in (@字符串1)

union all

select id

from table a

where id in (@字符串2)‍

两种方法都可行。在字符串较短的情况下，第二种方法应该好一些。字符串较长，第一种应该好一些。

测试代码：

<span>Declare</span> <span>@MRE_MROOIDS</span> <span>Nvarchar</span>(<span>Max</span><span>);

</span><span>Set</span> <span>@MRE_MROOIDS</span><span>=</span><span>'</span><span>2,4,5,396009,</span><span>'</span><span>;
</span><span>--</span><span>Set @MRE_MROOIDS='2,4,5,6,7,8,9,10,11,14,15,16,17,18,20,21,23,24,25,26,29,30';</span>

<span>Declare</span> <span>@SplitChar</span> <span>nvarchar</span>(<span>2</span><span>);
</span><span>Declare</span> <span>@EndIndex</span> <span>int</span><span>;
</span><span>Declare</span> <span>@Step</span> <span>int</span><span>;
</span><span>Declare</span> <span>@LastChars</span> <span>nvarchar</span>(<span>MAX</span><span>);
</span><span>Declare</span> <span>@CurrentTempChars</span> <span>nvarchar</span>(<span>max</span><span>);

</span><span>Set</span> <span>@LastChars</span><span>=</span><span>@MRE_MROOIDS</span><span>;
</span><span>Set</span> <span>@Step</span><span>=</span><span>5000</span><span>;
</span><span>Set</span> <span>@EndIndex</span><span>=</span><span>0</span><span>;
</span><span>Set</span> <span>@SplitChar</span><span>=</span><span>'</span><span>,</span><span>'</span><span>;

</span><span>IF</span> <span>EXISTS</span>(<span>SELECT</span> <span>*</span> <span>FROM</span> tempdb.dbo.sysobjects <span>where</span> id<span>=</span><span>OBJECT_ID</span>(N<span>'</span><span>tempdb..#StringToTableEntry_Temp10</span><span>'</span><span>))
    </span><span>Begin</span>
        <span>Drop</span> <span>Table</span><span> #StringToTableEntry_Temp10;            
    </span><span>End</span>    
    
<span>Create</span> <span>Table</span> #StringToTableEntry_Temp10(ID <span>INT</span><span>);


</span><span>While</span>(<span>LEN</span>(<span>@LastChars</span>)<span>></span><span>@Step</span><span>)
</span><span>Begin</span>    
    <span>Set</span> <span>@EndIndex</span><span>=</span> <span>charindex</span>(<span>@SplitChar</span>,<span>@LastChars</span>,<span>@Step</span><span>);
    
    </span><span>Set</span> <span>@CurrentTempChars</span><span>=</span><span>SubString</span>(<span>@LastChars</span>,<span>0</span>,<span>@EndIndex</span><span>);
    </span><span>--</span><span> insert into temp table</span>
    <span>Insert</span> <span>Into</span><span> #StringToTableEntry_Temp10
    </span><span>Select</span> Id <span>from</span> dbo.StringToTable2(<span>@CurrentTempChars</span>,<span>'</span><span>,</span><span>'</span><span>);
    
     </span><span>Set</span> <span>@LastChars</span><span>=</span><span>SubString</span>(<span>@LastChars</span>,<span>@EndIndex</span><span>+</span><span>1</span>,<span>LEN</span>(<span>@LastChars</span>)<span>-</span><span>@EndIndex</span><span>+</span><span>1</span><span>)
     </span><span>--</span><span>Select @LastChars as LastChars;</span>
     <span>Set</span> <span>@EndIndex</span><span>=</span><span>@EndIndex</span><span>+</span><span>@Step</span><span>;     
</span><span>End</span>

<span>If</span> <span>LEN</span>(<span>@LastChars</span>)<span>></span><span>0</span> <span>Begin</span>
    <span>Insert</span> <span>Into</span><span> #StringToTableEntry_Temp10
    </span><span>Select</span> Id <span>from</span> dbo.StringToTable2(<span>@LastChars</span>,<span>'</span><span>,</span><span>'</span><span>);
</span><span>End</span>


<span>Select</span> <span>COUNT</span>(<span>0</span>) <span>From</span> #StringToTableEntry_Temp10

StringToTable2函数：

<span>ALTER</span> <span>FUNCTION</span> <span>[</span><span>dbo</span><span>]</span>.<span>[</span><span>StringToTable</span><span>]</span><span>
(
    </span><span>@ids</span> <span>[</span><span>nvarchar</span><span>]</span>(<span>max</span><span>),
    </span><span>@separator</span> <span>[</span><span>char</span><span>]</span>(<span>1</span><span>)
)
</span><span>RETURNS</span> <span>@IdsTable</span> <span>TABLE</span><span>
(
    </span><span>[</span><span>Id</span><span>]</span> <span>INT</span> <span>NOT</span> <span>NULL</span><span>
)
</span><span>AS</span>
<span>BEGIN</span>
    <span>IF</span>(<span>RIGHT</span>(<span>@ids</span>,<span>1</span>)<span>=</span><span>@separator</span><span>)
        </span><span>BEGIN</span>
            <span>SET</span> <span>@ids</span><span>=</span><span>SUBSTRING</span>(<span>@ids</span>,<span>0</span>,<span>LEN</span>(<span>@ids</span><span>));
        </span><span>END</span>

    <span>--</span><span>下面的方式性能更好</span>
    <span>IF</span>(<span>LEN</span>(<span>@ids</span>) <span>></span> <span>0</span><span>)
        </span><span>BEGIN</span>
            <span>DECLARE</span> <span>@i</span> <span>int</span><span>;        
            </span><span>SET</span> <span>@i</span> <span>=</span> <span>CHARINDEX</span>(<span>@separator</span>, <span>@ids</span><span>);
            
            </span><span>WHILE</span> <span>@i</span> <span>></span> <span>0</span>
                <span>BEGIN</span>
                    <span>INSERT</span> <span>@IdsTable</span> <span>VALUES</span>(<span>LEFT</span>(<span>@ids</span>, <span>@i</span> <span>-</span> <span>1</span><span>));            
                    </span><span>SET</span> <span>@ids</span> <span>=</span> <span>SUBSTRING</span>(<span>@ids</span>, <span>@i</span> <span>+</span> <span>1</span>, <span>LEN</span>(<span>@ids</span>) <span>-</span> <span>@i</span><span>);
                    </span><span>SET</span> <span>@i</span> <span>=</span> <span>CHARINDEX</span>(<span>@separator</span>, <span>@ids</span><span>);
                </span><span>END</span>
                
            <span>IF</span>(<span>LEN</span>(<span>@ids</span>) <span>></span> <span>0</span><span>)
                </span><span>BEGIN</span>
                    <span>INSERT</span> <span>@IdsTable</span> <span>VALUES</span>(<span>@ids</span><span>);
                </span><span>END</span>
        <span>END</span>        
    <span>RETURN</span><span>;
</span><span>END</span>

测试结果：

@MRE_MROOIDS包含id记录	@Step长度	执行时间
100,000	100000	00:09:15
100,000	20000	00:03:48
100,000	10000	00:01:57
100,000	5000	00:01:01

Statement

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

sqlserver数据库中已存在名为的对象怎么解决Apr 05, 2024 pm 09:42 PM

对于 SQL Server 数据库中已存在同名对象，需要采取以下步骤：确认对象类型（表、视图、存储过程）。如果对象为空，可使用 IF NOT EXISTS 跳过创建。如果对象有数据，使用不同名称或修改结构。使用 DROP 删除现有对象（谨慎操作，建议备份）。检查架构更改，确保没有引用删除或重命名的对象。

sqlserver服务无法启动怎么办Apr 05, 2024 pm 10:00 PM

当 SQL Server 服务无法启动时，可采取以下步骤解决：检查错误日志以确定根本原因。确保服务帐户具有启动服务的权限。检查依赖项服务是否正在运行。禁用防病毒软件。修复 SQL Server 安装。如果修复不起作用，重新安装 SQL Server。

怎么查看sqlserver端口号Apr 05, 2024 pm 09:57 PM

要查看 SQL Server 端口号：打开 SSMS，连接到服务器。在对象资源管理器中找到服务器名称，右键单击它，然后选择“属性”。在“连接”选项卡中，查看“TCP 端口”字段。

sqlserver数据库在哪里Apr 05, 2024 pm 08:21 PM

SQL Server 数据库文件通常存储在以下默认位置：Windows: C:\Program Files\Microsoft SQL Server\MSSQL\DATALinux: /var/opt/mssql/data可通过修改数据库文件路径设置来自定义数据库文件位置。

Java连接SqlServer错误如何解决May 01, 2023 am 09:22 AM

问题发现这次使用的是SqlServer数据库，之前并没有使用过，但是问题不大，我按照需求文档的步骤连接好SqlServer之后，启动SpringBoot项目，发现了一个报错，如下：刚开始我以为是SqlServer连接问题呢，于是便去查看数据库，发现数据库一切正常，我首先第一时间问了我的同事，他们是否有这样的问题，发现他们并没有，于是我便开始了我最拿手的环节，面向百度编程。开始解决具体报错信息是这样，于是我便开始了百度报错：ERRORc.a.d.p.DruidDataSource$CreateCo

sqlserver英文安装怎么更改中文Apr 05, 2024 pm 10:21 PM

SQL Server 英文安装可通过以下步骤更改为中文：下载相应语言包；停止 SQL Server 服务；安装语言包；更改实例语言；更改用户界面语言；重启应用程序。

Win11无法安装SQL Server的原因及解决方案Dec 27, 2023 pm 07:48 PM

有网友反馈，在win11上无法安装sqlserver这款软件，不知道是怎么回事，根据目前的测试来看，win11存在硬盘问题，部分接口硬盘无法安装这款软件。win11为啥不能安装sqlserver：答：win11不能安装sqlserver是硬盘的问题。1、据了解，win11存在对于硬盘的检测bug。2、这导致sqlserver无法在“三星m.2接口”硬盘上安装。3、因此，如果我们要安装的话，需要准备一块其他硬盘。4、然后将该硬盘安装到电脑里，如果没有额外插槽的话就要换掉之前的硬盘。5、安装完成后，

sqlserver数据库日志怎么查询Apr 05, 2024 pm 09:06 PM

可以通过以下步骤查询 SQL Server 数据库日志：1. 打开 SQL Server Management Studio，连接到数据库服务器；2. 展开“管理”节点，导航到“SQL Server 日志”；3. 选择要查询的日志文件，右键单击并选择“查看日志文件”；4. 浏览日志记录。其他查询日志方法：使用 Transact-SQL 查询、PowerShell Cmdlet。

See all articles

Hot AI Tools

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress images for free

Clothoff.io

AI clothes remover

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

Repo: How To Revive Teammates

1 months agoBy尊渡假赌尊渡假赌尊渡假赌

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)

2 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

Hello Kitty Island Adventure: How To Get Giant Seeds

1 months agoBy尊渡假赌尊渡假赌尊渡假赌

How Long Does It Take To Beat Split Fiction?

4 weeks agoByDDD

R.E.P.O. Save File Location: Where Is It & How to Protect It?

4 weeks agoByDDD

Hot Tools

Dreamweaver Mac version

Visual web development tools

MinGW - Minimalist GNU for Windows

This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.