Remove duplicate lines from text file

This snippet will remove duplicate lines from text file:

awk '!seen[$0]++' filename

It will display the results on screen. If you want your results in a file, do it like this:

awk '!seen[$0]++' filename > new_filename

Source
The GNU Awk User’s Guide

 

Share

Recent Posts

CSS: Height equal to dynamic width

There is a way using CSS! If you set your width depending on the parent…

2 years ago

bash: Find text in files in subdirectories recursively

grep -rnw '/path/to/somewhere/' -e 'pattern' -r or -R is recursive, -n is line number, and -w stands for match the…

6 years ago

phpMyAdmin: Different SQL file name on each export

Easiest way to export your database is through phpMyAdmin. With few clicks you can get…

9 years ago

Restart Apache automatically

Put this code into a .sh file, add executable permissions to it and add it…

10 years ago

Restart MySQL automaticaly

Put this code into a .sh file, add executable permissions to it and add it…

10 years ago

How to add Facebook application as a page tab

Facebook has a somehow hidden way for addding applications as page tab. To do that,…

10 years ago