Usage of awk command in Linux Shell

awk command

awk is also a data processing tool! Compared with sed, which often processes an entire line, awk prefers to divide a line into several fields for processing.

. The most basic function of the awk language is to decompose and extract information in a file or string based on specified rules, and it can also output data based on specified rules.

There are three ways to call awk

1. Command line method

awk [-F field-separator] 'commands' input-files

Among them, [-F field separator] is optional because awk uses spaces or tab keys as the default field separator. Therefore, if you want to browse text with spaces between fields, you do not need to specify this option. If you want to browse a file such as passwd, which uses colons as separators for each field, you must specify the -F option, such as: awk -F: 'commands' input -file.

Note: The environment variable IFS is used to store the separator in the Linux system, but the value of IFS can also be changed according to the actual application.

For example:

The script execution results are as follows:

commands are real awk command, input-files are the files to be processed.

iput_files can be a file list of more than one file, awk will process each file in the list in order.

In awk, in each line of the file, each item separated by a field delimiter is called a field. Normally, without specifying -F field separator, the default field separator is space or tab.

2． The shell script method

inserts all the awk commands into a file and makes the awk program executable, and then uses the awk command interpreter as the first line of the script to be called by typing the script name.

Equivalent to the first line of the shell script: #!/bin/sh can be replaced by: #!/bin/awk

3. Insert all awk commands into a separate file, and then call:

Awk             -f                                                  awk‐script‐file                                                                         awk‐script‐file        input‐files

Among them, the‐f option loads the awk script in awk‐script‐file, input‐files is the same as above it's the same.

awk’s patterns and actions

Any awk statement consists of patterns and actions (awk_pattern { actions }).
There may be many statements in an awk script.

The mode part determines when the action statement triggers and triggers the event. Processing is the operations performed on data. If the mode part is omitted, the action will remain executed at all times. That is, when omitted, the corresponding actions will be executed without matching and comparing the input records.

Pattern can be any conditional statement or regular expression etc. awk_pattern can be of the following types:

1) Regular expression is used as awk_pattern:/regexp/

For example: awk '/^[a-z]/' input_file

2) Boolean expression is used as awk_pattern, When the expression is true, the execution of the corresponding actions is triggered.

① You can use variables (such as field variables $1, $2, etc.) and /regexp/

② Operators in Boolean expressions:

Relational operators: Matching operator: value ~ /regexp/ If value matches /regexp/, it returns true
value!~ /regexp/ If value does not match /regexp/, it returns true
For example: awk '$2 > 10 {print "ok"}' input_file
awk '$3 ~ /^d/ {print "ok"}' input_file

③ && (and) and || (or) can connect two /regexp/ or Boolean Expressions form mixed expressions. !(not) can be used in Boolean expressions or before /regexp/.

For example: awk '($1 10) {print "ok"}' input_file
awk '/^d/ || /x$/ {print "ok"}' The input_file

pattern includes two special fields BEGIN and END. Use the BEGIN statement to set the count and printhead. The BEGIN statement is used before any text browsing action, after which the text browsing action starts executing based on the input text. The END statement is used to print out the total number of texts and the end status flag after awk completes the text browsing action.

The actual action is specified within curly brackets { }. Actions are mostly used for printing, but there are also longer codes such as if and loop statements and loop exit structures. If no action is specified, awk will print out all browsed records.

When awk is executed, its browsing domain is marked as $1, $2...$n. This method is called domain identification. Using these domain identifiers will make further processing of the domain easier.

Use $1 and $3 to refer to the 1st and 3rd fields. Note that commas are used to separate the fields. If you want to print all fields of a record with 5 fields

, you do not need to specify $1, $2, $3, $4, $5. You can use $0, which means all fields.

To print a field or all fields, use the print command. This is an awk action

awk running process:

①If the BEGIN block exists, awk executes the actions specified by it.

② awk reads a line from the input file, which is called an input record. (If the input file is omitted, it will be read from the standard input)

③ awk splits the read record into fields, and puts the first field into the variable $1, the second field into $2, and so on. $0 represents the entire record.

④ Compare the current input record with the awk_pattern in each awk_cmd to see if it matches. If it matches, execute the corresponding actions. If there is no match, the corresponding actions are skipped until all awk_cmds are compared.

⑤ When an input record compares all awk_cmd, awk reads the next line of the input and continues to repeat steps ③ and ④. This process continues until awk reads the end of the file.

⑥After awk reads all the input lines, if END exists, the corresponding actions will be executed.

Getting Started Example:

Example 1: Display the user name and login shell in the /etc/passwd file

If you only display the account of /etc/passwd and the shell corresponding to the account, and the account and the shell Separate by tab key

If you only display the user name and login shell in the /etc/passwd file, and separate the account and shell by comma

Note: awk always outputs to standard output. If you want awk to output to a file, you can use redirection.

Example 2: Display the usernames and login shells of all users with UID greater than 500 in the /etc/passwd file

Example 3: If only the users with UID greater than 500 in the /etc/passwd file are displayed Name and login shell, and separate the account and shell with commas, and add the column name name, shell to all lines, and add "blue,/bin/nosh" to the last line.

Note:

1.awk is followed by two single quotes and braces {} to set the processing action you want to perform on the data

2.awk The workflow is as follows: execute BEGING first , then read the file, read a record separated by n newlines, then divide the record into fields according to the specified field separator, fill in the fields, $0 means all fields, $1 means the first field, $n means the nth domain, and then start executing the action corresponding to the pattern. Then start reading the second record...until all records are read, and finally perform the END operation.

Question: How to print all records (take the content in /etc/passwd as an example)

Example 4: Search for all lines with the root keyword in /etc/passwd

This is a pattern Example of using (pattern). Only lines matching pattern (here root) will execute action (no action is specified, and the content of each line is output by default).

Search supports regular expressions, for example, looking for those starting with root:

Search /etc/passwd for all lines with the root keyword, and display the corresponding shell

The action specified here is {print $7 }

Example 5: Display the information of the 5 users who recently logged in to the system, only displaying the user name and IP address

Use the last command to view the information of the recently logged in users. As shown in the figure below:

Use the awk command to extract the data of user name and IP area

or

awk built-in variables

awk has many built-in variables for setting environment information. Some of the most commonly used variables are given below.

FILENAME File name browsed by awk

FS Set the input field separator, equivalent to the command line -F option

NF The number of fields in the browsing record (the total number of fields in each line ($0))

NR Read Number of records (which row of data is processed by awk)

Example 6: Statistics/etc/passwd: file name, line number of each line, number of columns of each line, corresponding complete line content:

Display the records of all accounts with their record numbers and print the input file name in the END section

In addition to awk’s built-in variables, awk can also customize variables

Example 7: Count the number of accounts in /etc/passwd

count is a custom variable. There was only one print in the previous action{}. In fact, print is just a statement, and action{} can have multiple statements, separated by ; signs.

There is no initialization of count here. Although the default is 0, it is appropriate to initialize it to 0:

Example 8: Count the number of bytes occupied by files in a folder

If you use M Displayed in units:

Note: The above statistics do not include files in subdirectories.

If you want to quickly see the length of all files and their sum, but exclude subdirectories, how to do it: