Searching and removing duplicates

Remove duplicates in less than a minute even if there are thousands of them.