Detailed explanation of linux awk command
- 高洛峰Original
- 2016-12-12 16:14:301253browse
awk is a line processor: Compared with the advantages of screen processing, there will be no memory overflow or slow processing problems when processing huge files. It is usually used to format text information.
awk processing process: Process each line in turn , and then output the
awk command form:
awk [-F|-f|-v] 'BEGIN{} //{command1; command2} END{}' file
[-F|-f|-v] Large parameters, -F specifies the delimiter, -f calls the script, -v defines the variable var=value
' ' ' ' Quoting the code block
BEGIN initialization code block, before processing each line, initialize the code, mainly referencing the global Variable, set the FS separator
// Matching code block, which can be a string or a regular expression
{} Command code block, including one or more commands
; Code block, a code block that is executed after each line is processed, mainly performs final calculation or outputs the end summary information
Special points:
$0 Represents the entire current line
$1 The first field of each line
NF Field number variable
NR ‐ ’ ’ s ’ s
’ ’ ’ ’ ’ ’ ’ ’ ’ ’ ’ ’ ’ ’ ’ ’ ’ ’ ’
’ newline character When 时FS Begin, define the separators of the separators RS input, which is default to change the row (that is, the text is input according to one line and line) ~ matching, not accurate compared to == Inexact comparison == equals, all must be equal, exact comparison != not equal, exact comparison && Logical AND || Logical OR + Indicates 1 or more / [0-9][0-9]+/ Two or more numbers/[0-9][0-9]*/ One or more numbersFILENAME File nameOFS Output field separation character, the default is also a space, which can be changed to a tab character, etc. ORS The output record separator defaults to a newline character, that is, the processing results are also output to the screen line by line -F'[:#/]' Define three The delimiter print & $0print is the main command for awk to print the specified content awk '{print}' /etc/passwd == awk '{print $0}' /etc/passwd awk '{ Print "} '/ETC/PASSWD // Do not output Passwd content, but outputs the same number of empty lines, which further explains that AWK is a line of processing text Awk' {Print" a "} '/etc/etc/ passwd / etc/passwd // Output the first two fields of each line in separate lines to further understand the processing of text line by line awk -F: '{print $1,$3,$6}' OFS="t" /etc/passwd Output fields 1, 3, 6, with tab characters as delimiters -f specifies the script fileawk -f script.awk fileBEGIN{FS=":"}{ print $1} //The effect is the same as awk -F":" '{print $1}', except that the separator is specified in the code itself using FS awk 'BEGIN{X=0} /^$/{ X+= 1 } END{print "I find",X,"blank lines."}' test I find 4 blank lines.ls -l|awk 'BEGIN{sum=0} !/^d/{sum+= $5} END{print "total size is",sum}' // Calculate the file size total size is 17487 -F specifies the separator$1 refers to the first field after specifying the delimiter, $3 the third field, t is the tab character
One or more consecutive spaces or tabs are regarded as a delimiter, that is, multiple spaces are regarded as A space
awk -F":" '{print $1}' /etc/passwd
awk -F":" '{print $1 $3}' /etc/passwd //$1 and $3 are connected and output without separation
awk -F":" '{print $1,$3}' /etc/passwd // There is an extra comma, $1 and $3 are separated by spaces
awk -F":" " '{print $1 " " $3}' / etc/passwd with ’ s ’ s ’ s ’ s ‐ out off off ‐ ‐ ‐ ‐ ‐ ‐ use to use using ’ to have to use ’ to have to be used to have -F: '{print NF}' /etc/passwd / Print
AWK -F: 'NF == 4 {Print}'/ETC/PASSWD // Display lines with only 4 fields
AWK -F: 'NF & GT; 2 {Print $ 0}'/ETC/PASSWD // Display lines with more than 2 fields per line
awk '{print NR,$0}' /etc/passwd // Output the line number of each line
awk -F: '{print NR,NF,$NF,"t ",$0}' /etc/passwd //Print the line number, field number, last field value, tab character, and content of each line in sequence
awk -F: 'NR==5{print}' /etc/passwd / /Display line 5
awk -F: 'NR==5 || NR==6{print}' /etc/passwd //Display line 5 and line 6
route -n|awk 'NR! = 1 {Print} '// Do not display the first line
// matching code block
// Pure characters match! // Pure characters do not match ~ // field value matching! ~ // Field value does not match matching ~/a1|a2/Field value matches a1 or a2
awk '/mysql/' /etc/passwd
awk '/mysql/{print }' /etc/passwd
awk '/mysql/{print $0} ' /etc/passwd using use with using ‐ ‐ ‐ out out out ’’s ’ ‐ ‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ etc/passwd
awk '!/mysql|mail/{print}' /etc/passwd
awk -F: '/mail/,/mysql/{print}' /etc/passwd //Interval matching
awk '/[2][7][7]*/{print $0}' /etc/passwd /etc/passwd ~/mail/{print $1}' /etc/passwd //$1 will only be displayed if it matches the specified content
awk -F: '{if($1~/mail/) print $1}' /etc/passwd //Same as above
awk -F: '$1!~/mail/{print $1}' /etc/passwd //Does not match
awk -F: '$1!~/mail|mysql/{print $1}' /etc/passwd
IF statement
must be used in {}, and the comparison content is expanded with ()
awk -F: '{if($1~/mail/) print $1}' /etc/passwd
awk -F: '{if($1~/mail/) {print $1}}' /etc/passwd else {print $2}}' /etc/passwd //if...else...
Conditional expression
== != > >=
awk -F":" ' $1=="mysql"{print $3}' /etc/passwd
awk -F":" '{if($1=="mysql") print $3}' /etc/passwd //Same as above
awk -F":" '$1!="mysql"{print $3}' /etc/passwd / / Greater than
awk -F":" '$3>=100{print $3}' /etc/passwd // Greater than or equal to
awk -F":" '$3
awk -F":" '$3
Logical operator
&& ||
awk -F: ' $1~/mail/ && $3>8 {print }' /etc/passwd //Logical AND, $1 matches mail, and $3>8
awk -F: '{if($1~/mail/ && $3> ;8) print }' /etc/passwd
awk -F: '$1~/mail/ || $3>1000 {print }' /etc/passwd //Logical OR
awk -F: '{if( $1~/mail/ || $3>1000) print }' /etc/passwd
Numerical operations
awk -F: '$3 > 100' /etc/passwd
awk -F: '$3 > ; 100 || $3
awk -F: '$3+$4 > 200' /etc/passwd
awk -F: '/mysql|mail/{print $3+10}' /etc/passwd using 10 use with 10 to be added to ’ ’ /mysql /{print $3 *$4}' /etc/passwd 1024)}' /proc/meminfo // Rounding
Output separator OFS
awk '$6 ~ /FIN/ || NR==1 {print NR,$4,$5,$6}' OFS="t" netstat.txt
awk '$6 ~ /WAIT/ || NR==1 {print NR,$4,$5,$6}' OFS="t" netstat.txt
//Output field 6 lines matching WAIT, where each line is output Line number, fields 4, 5, 6, and use tabs to separate the fields
Output the processing results to the file
① Directly output in the command code block route -n|awk 'NR!=1{print > "./fs"}'
②Use redirection for output ‐ ‐ ‐ ‐n|‐awk 'NR!=1{print}' ‐ printf "%-8s %-8s %-10sn",$1,$2,$3}'
printf represents format output
% formatted output delimiter
-8 is 8 characters in length
s represents string type
Print the first three fields of each line, specify the first field to output the string type (length is 8), the second field to output the string type (length is 8),
The third field to output the string type ( The length is 10)
netstat -anp|awk '$6=="LISTEN" || NR==1 {printf "%-10s %-10s %-10s n",$1,$2,$3}'
netstat - anp|awk '$6=="LISTEN" || NR==1 {printf "%-3s %-10s %-10s %-10s n",NR,$1,$2,$3}'
IF statement
awk -F: '{if($3>100) print "large"; else print "small"}' /etc/passwd
small
small
small
large
small
small
awk -F: 'BEGIN{A=0;B=0} {if($3>100) {A++; print "large"} else {B++; print "small"}} END{print A,"t", B}' /etc/passwd
// Skip if less than 100, otherwise display
awk -F: 'BEGIN{i=1} {if(i awk -F: 'BEGIN{i=1} {if(i 另一种形式 awk -F: '{print ($3>100 ? "yes":"no")}' /etc/passwd awk -F: '{print ($3>100 ? $3":tyes":$3":tno")}' /etc/passwd while语句 awk -F: 'BEGIN{i=1} {while(i 7 root 1 7 x 2 7 0 3 7 0 4 7 root 5 7 /root 6 数组 netstat -anp|awk 'NR!=1{a[$6]++} END{for (i in a) print i,"t",a[i]}' netstat -anp|awk 'NR!=1{a[$6]++} END{for (i in a) printf "%-20s %-10s %-5s n", i,"t",a[i]}' 9523 1 9929 1 LISTEN 6 7903 1 3038/cupsd 1 7913 1 10837 1 9833 1 应用1 awk -F: '{print NF}' helloworld.sh //输出文件每行有多少字段 awk -F: '{print $1,$2,$3,$4,$5}' helloworld.sh //输出前5个字段 awk -F: '{print $1,$2,$3,$4,$5}' OFS='t' helloworld.sh //输出前5个字段并使用制表符分隔输出 awk -F: '{print NR,$1,$2,$3,$4,$5}' OFS='t' helloworld.sh //制表符分隔输出前5个字段，并打印行号 应用2 awk -F'[:#]' '{print NF}' helloworld.sh //指定多个分隔符: #，输出每行多少字段 awk -F'[:#]' '{print $1,$2,$3,$4,$5,$6,$7}' OFS='t' helloworld.sh //制表符分隔输出多字段 应用3 awk -F'[:#/]' '{print NF}' helloworld.sh //指定三个分隔符，并输出每行字段数 awk -F'[:#/]' '{print $1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12}' helloworld.sh //制表符分隔输出多字段 应用4 计算/home目录下，普通文件的大小，使用KB作为单位 ls -l|awk 'BEGIN{sum=0} !/^d/{sum+=$5} END{print "total size is:",sum/1024,"KB"}' ls -l|awk 'BEGIN{sum=0} !/^d/{sum+=$5} END{print "total size is:",int(sum/1024),"KB"}' //int是取整的意思 应用5 统计netstat -anp 状态为LISTEN和CONNECT的连接数量分别是多少 netstat -anp|awk '$6~/LISTEN|CONNECTED/{sum[$6]++} END{for (i in sum) printf "%-10s %-6s %-3s n", i," ",sum[i]}' 应用6 统计/home目录下不同用户的普通文件的总数是多少？ ls -l|awk 'NR!=1 && !/^d/{sum[$3]++} END{for (i in sum) printf "%-6s %-5s %-3s n",i," ",sum[i]}' mysql 199 root 374 统计/home目录下不同用户的普通文件的大小总size是多少？ ls -l|awk 'NR!=1 && !/^d/{sum[$3]+=$5} END{for (i in sum) printf "%-6s %-5s %-3s %-2s n",i," ",sum[i]/1024/1024,"MB"}' Application 7 Output Score Table awk 'BEGIN{math=0;eng=0;com=0;printf "Lineno. Name No. Math English Computer Totaln";printf "------ -------------------------------------------------- ---n"}{math+=$3; eng+=$4; com+=$5;printf "%-8s %-7s %-7s %-7s %-9s %-10s %-7s n",NR,$1,$2 ,$3,$4,$5,$3+$4+$5} END{printf "--------------------------------- --------------------------n";printf "%-24s %-7s %-9s %-20s n","Total: ",math,eng,com;printf "%-24s %-7s %-9s %-20s n","Avg:",math/NR,eng/NR,com/NR}' test0 Marry 2143 78 84 77 Jack 2321 66 78 45 Tom 2122 48 77 71 Mike 2537 87 97 95 Bob 2415 40 57 62
[root @localhost home]# cat test0 