This snippet will remove duplicate lines from text file:

awk '!seen[$0]++' filename

It will display the results on screen. If you want your results in a file, do it like this:

awk '!seen[$0]++' filename > new_filename

Source
The GNU Awk User’s Guide