This snippet will remove duplicate lines from text file:
awk '!seen[$0]++' filename
It will display the results on screen. If you want your results in a file, do it like this:
awk '!seen[$0]++' filename > new_filename
Source
The GNU Awk User’s Guide
By Aleksandar Jakovljević
My name is Aleksandar Jakovljević and I am a web developer from Belgrade, Serbia.
I was born in Smederevo, in 1980. I started focusing on web development since late 2003. I am mostly focused on open source web technologies.
I specialised in PHP, focusing on Symfony framework, with skills in Drupal and WordPress, too.
I worked more than 7 years for art_net studio and 10 years for Computer Rock (former Spoiled Milk). Now I am working on developing industry leading subscription management system as a part of Plenigo team since August 2021.
I love the web and everything related to it. The possibility to publish your ideas quickly and make them available to the entire world is what made me decide to pursue a career in web development.
Follow on:
Twitter | GitHub | LinkedIn | Facebook | Instagram
Hire me on Upwork