Remove duplicate lines from text file

This snippet will remove duplicate lines from text file:

awk '!seen[$0]++' filename

It will display the results on screen. If you want your results in a file, do it like this:

awk '!seen[$0]++' filename > new_filename

Source
The GNU Awk User’s Guide

 

By Aleksandar Jakovljević

My name is Aleksandar Jakovljević and I am a web developer from Belgrade, Serbia. I was born in Smederevo, in 1980. I started focusing on web development since late 2003. I am mostly focused on open source web technologies. I specialised in PHP, focusing on Symfony framework, with skills in Drupal and WordPress, too. I worked more than 7 years for art_net studio. I work for Computer Rock (former Spoiled Milk) since April 2011. I love the web and everything related to it. The possibility to publish your ideas quickly and make them available to the entire world is what made me decide to pursue a career in web development.

Follow on: Twitter | GitHub | LinkedIn | Facebook | Instagram

Hire me on Upwork

Leave a comment

Your email address will not be published. Required fields are marked *