Remove duplicate lines from text file

This snippet will remove duplicate lines from text file:

awk '!seen[$0]++' filename

It will display the results on screen. If you want your results in a file, do it like this:

awk '!seen[$0]++' filename > new_filename

Source
The GNU Awk User’s Guide

 

Share

Recent Posts

CSS: Height equal to dynamic width

There is a way using CSS! If you set your width depending on the parent…

2 years ago

bash: Find text in files in subdirectories recursively

grep -rnw '/path/to/somewhere/' -e 'pattern' -r or -R is recursive, -n is line number, and -w stands for match the…

7 years ago

phpMyAdmin: Different SQL file name on each export

Easiest way to export your database is through phpMyAdmin. With few clicks you can get…

10 years ago

Restart Apache automatically

Put this code into a .sh file, add executable permissions to it and add it…

10 years ago

Restart MySQL automaticaly

Put this code into a .sh file, add executable permissions to it and add it…

10 years ago

How to add Facebook application as a page tab

Facebook has a somehow hidden way for addding applications as page tab. To do that,…

11 years ago