Hive 创建表

Hive 创建表时正则表达式与java代码中一致,RegexSerDe 中用\\来表示反转码

1. Hive 创建表时正则表达式与java代码中一致,RegexSerDe 中用\\来表示反转码
create table inst_cn_3 (
  ip string,
  time string,
  mac string,
  lan string,
  ver string,
  lc string,
  pn string,
  reg string,
  vrf string,
  line string)
ROW FORMAT SERDE 'org.apache.Hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
  "input.regex" = "- *([\\d\\.]+) *\\[([\\d]{2}/[\\w]+/[\\d]{4}:[\\d]{2}:[\\d]{2}:[\\d]{2}\\s+\\+[\\d]+)\\] *GET */mx3/inst/([0-9a-f]{12})/ver=([\\.\\d]+),lan=(0x[\\w]+)(?:,lc=([\\w]+))(?:,pn=([\\w]+))(?:,reg=([0-1]))(?:,vrf=([\\w]+))?.*"
 )
STORED AS TEXTFILE;

alter table inst_cn_3 add columns(line string);
ALTER TABLE inst_cn_3 SET SERDEPROPERTIES (
  "input.regex" = "- ([\\d\\.]+) \\[([\\d]{2}/[\\w]+/[\\d]{4}:[\\d]{2}:[\\d]{2}:[\\d]{2}\\s+\\+[\\d]+)\\] GET /mx3/inst/([0-9a-f]{12})/ver=([\\.\\d]+),lan=(0x[\\w]+)(?:,lc=([\\w]+))(?:,pn=([\\w]+))(?:,reg=([0-1]))(?:,vrf=([\\w]+))?.*|(.*)"
 );
select * from inst_cn_3 limit 100;

select
  line
from inst_cn_3
where
  1=1
  and mac is null
  and line is not null
  and  !(line rlike '.*unknowuser00.*')
  ;

hadoop fs -cp /mnt/nfs/log/statcn/inst/inst_cn_3.*.txt /hive/warehouse/inst_cn_3/
 
2011.06.09 hive时间处理
1.select
  from_unixtime(unix_timestamp('02/May/2011:00:00:00 +0800','dd/MMMMM/yyyy:HH:mm:ss Z'),'yyyy-MM-dd HH:mm:ss') ,
  from_unixtime(unix_timestamp('02/May/2011:23:59:59 +0800','dd/MMMMM/yyyy:HH:mm:ss Z'),'yyyy-MM-dd HH:mm:ss')  ,
  from_unixtime(unix_timestamp('03/May/2011:00:00:00 +0800','dd/MMMMM/yyyy:HH:mm:ss Z'),'yyyy-MM-dd HH:mm:ss') ,
  from_unixtime(unix_timestamp('03/May/2011:23:59:59 +0800','dd/MMMMM/yyyy:HH:mm:ss Z'),'yyyy-MM-dd HH:mm:ss')
 from
  stat_c_log
where
  1=1
  and partkey = '20110503'
  and logType = 'inst_cn_3'
  and url rlike '/mx3/inst/.*'
limit
  10
;

2.
select
  from_unixtime(unix_timestamp('02/May/2011:00:00:00 +0800','dd/MMMMM/yyyy:HH:mm:ss Z'),'yyyy-MM-dd HH:mm:ss') ,
  from_unixtime(unix_timestamp('02/May/2011:23:59:59 +0800','dd/MMMMM/yyyy:HH:mm:ss Z'),'yyyy-MM-dd HH:mm:ss')  ,
  from_unixtime(unix_timestamp('03/May/2011:00:00:00 +0800','dd/MMMMM/yyyy:HH:mm:ss Z'),'yyyy-MM-dd HH:mm:ss') ,
  from_unixtime(unix_timestamp('03/May/2011:23:59:59 +0800','dd/MMMMM/yyyy:HH:mm:ss Z'),'yyyy-MM-dd HH:mm:ss')  ,
  round((unix_timestamp('02/May/2011:00:00:00 +0800','dd/MMMMM/yyyy:HH:mm:ss Z')-4*3600)/(3600*24))*(3600*24*1000),
  round((unix_timestamp('02/May/2011:23:59:59 +0800','dd/MMMMM/yyyy:HH:mm:ss Z')-4*3600)/(3600*24))*(3600*24*1000)
 from
  stat_c_log
where
  1=1
  and partkey = '20110503'
  and logType = 'inst_cn_3'
  and url rlike '/mx3/inst/.*'
limit
  10
;

2012.03.01
  1. Hive 方法注册类 FunctionRegistry
2012.06.14
  1. set hive.cli.print.header=true; 可以设置hive shell的输出.
2012.06.26
  1. hive cdh4b2 使用arichive 对表归档后， 使用select line 对归档后的partition查询时，， 报FileNotFoundException 异常。
    https://issues.apache.org/jira/browse/MAPREDUCE-2704 是因为CombineFileInputFormat constructs new Path objects by converting an existing path to a URI, and then only pulling out the "path" part of it. This drops the scheme and host, which makes CombineFileInputFormat fail if the paths are on a filesystem other than the default one.
 2012.07.16
  1. EXPLAIN EXTENDED hive_query; 查看运行.
2012.07.29
  1. DESCRIBE FORMATTED mock; 显示columns, location, params等.
  2. /src/ql/src/test/queries/clientpositive/