Linux uniq command example

Note that uniq will not remove duplicate rows unless they are adjacent. Therefore, you may need to sort them first, or combine the sort command with uniq to get the results. Let me show you some examples.

First, let’s create a file with some repeating lines:

vi ostechnix.txt
welcome to ostechnix
welcome to ostechnix
Linus is the creator of Linux.
Linux is secure by default
Linus is the creator of Linux.
Top 500 super computers are powered by Linux

As you can see in the file above, we have some duplicate lines (the first and second rows, the third and fifth rows are duplicates).

1. Use uniq command to delete consecutive duplicate lines in the file

If you use uniq command without any arguments, it will remove all consecutive duplicate lines and display only unique lines.

uniq ostechnix.txt

Sample output:

As you can see, the uniq command removes all consecutive duplicate lines in the given file. You may also notice that the output above still has the second and fourth lines repeated. This is because the uniq command will only delete duplicate rows if they are adjacent. Of course, we can also delete non-consecutive duplicate rows. See the second example below.

2. Delete all duplicate lines

sort ostechnix.txt | uniq

Sample output:

Did you see it? There are no duplicate rows. In other words, the above command will display lines that appear only once in ostechnix.txt. We use the sort command in conjunction with the uniq command because, like I mentioned, uniq will not remove duplicate lines unless they are adjacent.

3. Only display the only line in the file

To display only the only line in the file, you can do this:

sort ostechnix.txt | uniq -u

Sample output:

Linux is secure by default
Top 500 super computers are powered by Linux

As you can see, only two lines are unique in the given file.

4. Only display duplicate lines

Similarly, we can also display repeated lines in the file, like this:

sort ostechnix.txt | uniq -d

Sample output:

Linus is the creator of Linux.
welcome to ostechnix

These two lines are duplicate lines in the ostechnix.txt file. Note that -d (lowercase d) will print only duplicate lines, one per group. To print all duplicate lines, use -D (capital D) like this:

sort ostechnix.txt | uniq -D

See the difference between the two options in the screenshot below:

5. Display the number of occurrences of each line in the file

For some reason, you might want to check the number of times each line is repeated in a given file. To do this, use the -c option as follows:

sort ostechnix.txt | uniq -c

Sample output:

Linus is the creator of Linux.
 Linux is secure by default
 Top 500 super computers are powered by Linux
 welcome to ostechnix

We can also sort by the number of occurrences of each row and then display it, as shown below:

sort ostechnix.txt | uniq -c | sort -nr

Sample output:

welcome to ostechnix
 Linus is the creator of Linux.
 Top 500 super computers are powered by Linux
 Linux is secure by default

6. Limit comparison to N characters

We can use the -w option to limit the comparison to a specific number of characters in the file. For example, let's compare the first four characters in the file and display the duplicate lines like this:

uniq -d -w 4 ostechnix.txt

7. Ignore the N characters specified by comparison

Like limiting the comparison to the first N characters of the line in the file, we can also use the -s option to ignore the comparison of the first N characters.

The following command will ignore the first four characters of each line in the file for comparison:

uniq -d -s 4 ostechnix.txt

To ignore comparing the first N fields (LCTT annotation: the first few columns) instead of characters, use the -f option in the above command.

For more details, please refer to the help section:

uniq --help

You can also use the man command to view:

man uniq

That’s it for today! I hope you now have a basic understanding of the uniq command and its purpose. If you found our guide useful, please share it on your social networks and continue to support us. More goodies are coming, so stay tuned!

Cheers!

The above is the detailed content of Linux uniq command example. For more information, please follow other related articles on the PHP Chinese website!