Hive分析窗口函数(二) NTILE,ROW_NUMBER,RANK,DENSE

首页

数据库

mysql教程

Hive分析窗口函数(二) NTILE,ROW_NUMBER,RANK,DENSE_RANK

WBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWB

Jun 07, 2016 pm 02:51 PM

hive函数分析窗口

问题导读： 1.NTILE作用是什么？ 2.按照pv降序排列，生成分组内每天的pv名次可使用哪个窗口函数？ 3.RANK 和 DENSE_RANK作用是什么？接上篇：Hive分析窗口函数（一）SUM,AVG,MIN,MAX 本文中介绍前几个序列函数，NTILE,ROW_NUMBER,RANK,DENSE_RANK，下面会一

问题导读：
1.NTILE作用是什么？
2.按照pv降序排列，生成分组内每天的pv名次可使用哪个窗口函数？
3.RANK 和 DENSE_RANK作用是什么？

接上篇：Hive分析窗口函数（一）SUM,AVG,MIN,MAX

本文中介绍前几个序列函数，NTILE,ROW_NUMBER,RANK,DENSE_RANK，下面会一一解释各自的用途。
Hive版本为 apache-hive-0.13.1

注意：序列函数不支持WINDOW子句。（什么是WINDOW子句，Hive分析窗口函数（一）SUM,AVG,MIN,MAX)

数据准备：

cookie1,2015-04-10,1
    cookie1,2015-04-11,5
    cookie1,2015-04-12,7
    cookie1,2015-04-13,3
    cookie1,2015-04-14,2
    cookie1,2015-04-15,4
    cookie1,2015-04-16,4
    cookie2,2015-04-10,2
    cookie2,2015-04-11,3
    cookie2,2015-04-12,5
    cookie2,2015-04-13,6
    cookie2,2015-04-14,3
    cookie2,2015-04-15,9
    cookie2,2015-04-16,7
     
    CREATE EXTERNAL TABLE lxw1234 (
    cookieid string,
    createtime string, --day
    pv INT
    ) ROW FORMAT DELIMITED
    FIELDS TERMINATED BY ','
    stored as textfile location '/tmp/lxw11/';
     
    DESC lxw1234;
    cookieid STRING
    createtime STRING
    pv INT
     
    hive> select * from lxw1234;
    OK
    cookie1 2015-04-10 1
    cookie1 2015-04-11 5
    cookie1 2015-04-12 7
    cookie1 2015-04-13 3
    cookie1 2015-04-14 2
    cookie1 2015-04-15 4
    cookie1 2015-04-16 4
    cookie2 2015-04-10 2
    cookie2 2015-04-11 3
    cookie2 2015-04-12 5
    cookie2 2015-04-13 6
    cookie2 2015-04-14 3
    cookie2 2015-04-15 9
    cookie2 2015-04-16 7

NTILE
NTILE(n)，用于将分组数据按照顺序切分成n片，返回当前切片值
NTILE不支持ROWS BETWEEN，比如 NTILE(2) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN 3 PRECEDING AND CURRENT ROW)
如果切片不均匀，默认增加第一个切片的分布

 SELECT
    cookieid,
    createtime,
    pv,
    NTILE(2) OVER(PARTITION BY cookieid ORDER BY createtime) AS rn1,        --分组内将数据分成2片
    NTILE(3) OVER(PARTITION BY cookieid ORDER BY createtime) AS rn2, --分组内将数据分成3片
    NTILE(4) OVER(ORDER BY createtime) AS rn3 --将所有数据分成4片
    FROM lxw1234
    ORDER BY cookieid,createtime;
     
    cookieid day pv rn1 rn2 rn3
    -------------------------------------------------
    cookie1 2015-04-10 1 1 1 1
    cookie1 2015-04-11 5 1 1 1
    cookie1 2015-04-12 7 1 1 2
    cookie1 2015-04-13 3 1 2 2
    cookie1 2015-04-14 2 2 2 3
    cookie1 2015-04-15 4 2 3 3
    cookie1 2015-04-16 4 2 3 4
    cookie2 2015-04-10 2 1 1 1
    cookie2 2015-04-11 3 1 1 1
    cookie2 2015-04-12 5 1 1 2
    cookie2 2015-04-13 6 1 2 2
    cookie2 2015-04-14 3 2 2 3
    cookie2 2015-04-15 9 2 3 4
    cookie2 2015-04-16 7 2 3 4

比如，统计一个cookie，pv数最多的前1/3的天

SELECT
    cookieid,
    createtime,
    pv,
    NTILE(3) OVER(PARTITION BY cookieid ORDER BY pv DESC) AS rn
    FROM lxw1234;
     
    --rn = 1 的记录，就是我们想要的结果
     
    cookieid day pv rn
    ----------------------------------
    cookie1 2015-04-12 7 1
    cookie1 2015-04-11 5 1
    cookie1 2015-04-15 4 1
    cookie1 2015-04-16 4 2
    cookie1 2015-04-13 3 2
    cookie1 2015-04-14 2 3
    cookie1 2015-04-10 1 3
    cookie2 2015-04-15 9 1
    cookie2 2015-04-16 7 1
    cookie2 2015-04-13 6 1
    cookie2 2015-04-12 5 2
    cookie2 2015-04-14 3 2
    cookie2 2015-04-11 3 3
    cookie2 2015-04-10 2 3

ROW_NUMBER() –从1开始，按照顺序，生成分组内记录的序列
–比如，按照pv降序排列，生成分组内每天的pv名次
ROW_NUMBER() 的应用场景非常多，再比如，获取分组内排序第一的记录;获取一个session中的第一条refer等。

SELECT
    cookieid,
    createtime,
    pv,
    ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY pv desc) AS rn
    FROM lxw1234;
     
    cookieid day pv rn
    -------------------------------------------
    cookie1 2015-04-12 7 1
    cookie1 2015-04-11 5 2
    cookie1 2015-04-15 4 3
    cookie1 2015-04-16 4 4
    cookie1 2015-04-13 3 5
    cookie1 2015-04-14 2 6
    cookie1 2015-04-10 1 7
    cookie2 2015-04-15 9 1
    cookie2 2015-04-16 7 2
    cookie2 2015-04-13 6 3
    cookie2 2015-04-12 5 4
    cookie2 2015-04-14 3 5
    cookie2 2015-04-11 3 6
    cookie2 2015-04-10 2 7

RANK 和 DENSE_RANK
—RANK() 生成数据项在分组中的排名，排名相等会在名次中留下空位
—DENSE_RANK() 生成数据项在分组中的排名，排名相等会在名次中不会留下空位

  SELECT
    cookieid,
    createtime,
    pv,
    RANK() OVER(PARTITION BY cookieid ORDER BY pv desc) AS rn1,
    DENSE_RANK() OVER(PARTITION BY cookieid ORDER BY pv desc) AS rn2,
    ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY pv DESC) AS rn3
    FROM lxw1234
    WHERE cookieid = 'cookie1';
     
    cookieid day pv rn1 rn2 rn3
    --------------------------------------------------
    cookie1 2015-04-12 7 1 1 1
    cookie1 2015-04-11 5 2 2 2
    cookie1 2015-04-15 4 3 3 3
    cookie1 2015-04-16 4 3 3 4
    cookie1 2015-04-13 3 5 4 5
    cookie1 2015-04-14 2 6 5 6
    cookie1 2015-04-10 1 7 6 7
     
    rn1: 15号和16号并列第3, 13号排第5
    rn2: 15号和16号并列第3, 13号排第4
    rn3: 如果相等，则按记录值排序，生成唯一的次序，如果所有记录值都相等，或许会随机排吧。

声明

本文内容由网友自发贡献，版权归原作者所有，本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容，请联系admin@php.cn

在MySQL中使用视图的局限性是什么？May 14, 2025 am 12:10 AM

mysqlviewshavelimitations：1）他们不使用Supportallsqloperations，限制DatamanipulationThroughViewSwithJoinSorsubqueries.2）他们canimpactperformance，尤其是withcomplexcomplexclexeriesorlargedatasets.3）

确保您的MySQL数据库：添加用户并授予特权May 14, 2025 am 12:09 AM

porthusermanagementInmysqliscialforenhancingsEcurityAndsingsmenting效率databaseoperation.1）usecReateusertoAddusers，指定connectionsourcewith@'localhost'or@'％'。

哪些因素会影响我可以在MySQL中使用的触发器数量？May 14, 2025 am 12:08 AM

mysqldoes notimposeahardlimitontriggers，butacticalfactorsdeterminetheireffactective：1）serverConfiguration impactactStriggerGermanagement; 2）复杂的TriggerSincreaseSySystemsystem load; 3）largertablesslowtriggerperfermance; 4）highConconcConcrencerCancancancancanceTigrignecentign; 5）; 5）

mysql：存储斑点安全吗？May 14, 2025 am 12:07 AM

Yes,it'ssafetostoreBLOBdatainMySQL,butconsiderthesefactors:1)StorageSpace:BLOBscanconsumesignificantspace,potentiallyincreasingcostsandslowingperformance.2)Performance:LargerrowsizesduetoBLOBsmayslowdownqueries.3)BackupandRecovery:Theseprocessescanbe

mySQL：通过PHP Web界面添加用户May 14, 2025 am 12:04 AM

通过PHP网页界面添加MySQL用户可以使用MySQLi扩展。步骤如下：1.连接MySQL数据库，使用MySQLi扩展。2.创建用户，使用CREATEUSER语句，并使用PASSWORD()函数加密密码。3.防止SQL注入，使用mysqli_real_escape_string()函数处理用户输入。4.为新用户分配权限，使用GRANT语句。

mysql：blob和其他无-SQL存储，有什么区别？May 13, 2025 am 12:14 AM

mysql'sblobissuitableForStoringBinaryDataWithInareLationalDatabase，而alenosqloptionslikemongodb，redis和calablesolutionsoluntionsoluntionsoluntionsolundortionsolunsolunsstructureddata.blobobobsimplobissimplobisslowderperformandperformanceperformancewithlararengelitiate;

mySQL添加用户：语法，选项和安全性最佳实践May 13, 2025 am 12:12 AM

toaddauserinmysql，使用：createUser'username'@'host'Indessify'password'; there'showtodoitsecurely：1）choosethehostcarecarefullytocon trolaccess.2）setResourcelimitswithoptionslikemax_queries_per_hour.3）usestrong，iniquepasswords.4）Enforcessl/tlsconnectionswith

MySQL：如何避免字符串数据类型常见错误？May 13, 2025 am 12:09 AM

toAvoidCommonMistakeswithStringDatatatPesInMysQl，CloseStringTypenuances，chosethirtightType，andManageEngencodingAndCollationsEttingsefectery.1）usecharforfixed lengengters lengengtings，varchar forbariaible lengength，varchariable length，andtext/blobforlabforlargerdata.2 seterters seterters seterters seterters

See all articles