Text Deduplicator Plus: easily sort and remove duplicates from a list

What do you do if you have long lists of information that you need to de-dupe or sort? You can use a program like Excel, which is not free, has a very large footprint on your PC, and cannot easily handle really large lists of much more than a million rows.

You could fire up a database, which unless you already use one, is just overkill. Or you could ‘Text Deduplicator Plus’.

‘Text Deduplicator Plus’ is a small, free program that can remove duplicates from lists and perform two other related functions, such as ranking your lists, and counting the number of instances of each unique value in your list. It is small, easy to use, and can handle extremely large lists

We tested this on a file that has about 1.8 million rows (which would not fit on a single worksheet in Excel). All that is needed is to point to the file you want to process, then choose ‘Deduplicate’, ‘Sort Only’, or ‘Count Dupes’. Processing was lightning fast.

Text Deduplicator Pro Screenshot

The verdict:

A terrific little program that does the job quickly and without much fuss. It is simple and straightforward and will get the job done, and can run portably without installation.

One thing I wish it did is sort in descending, and not just ascending order. Note that the dedupe operation by default sorts the results for you.

Get Text Deduplicator Plus here (Windows).

‘PaperRater’ is a virtual editor that checks your spelling and grammar - for free
‘Task Paper’ displays your to-do list right on your iPhone lock screen
Feb 5, 2013
Samer Kurdi
flattr this!
  • http://kaiweber.wordpress.com/ Kai

    Thanks, sounds like a cool little tool that worth checking out! To further manipulate de-duped text files, for example, to sort in descending order, try PlainEdit at http://www.gaijin.at/en/dlplainedit.php