Remove Duplicate Lines

What does Remove Duplicate Lines do?

Remove Duplicate Lines deletes repeated lines from a block of text so that each unique line appears only once. It keeps the first occurrence of every line, removes later copies, and leaves the remaining lines in their original order. It's useful for cleaning email lists, keyword lists, log extracts, URL lists and any data pasted one item per line.

How to remove duplicates

1. Paste your list into the input box, one item per line.
2. Click Convert.
3. Copy the de-duplicated list from the output box.

Here's a real example with some traps in it:

apple
banana
apple
Apple
apple 

cherry

banana

The output is:

apple
banana
Apple
apple 

cherry

The second apple and second banana are removed, as is the second blank line. Apple stays because its capital A makes it a different line, and apple (with a trailing space) stays for the same reason.

How lines are compared

Two lines count as duplicates only if they're exactly the same, character for character. That has consequences worth knowing:

The order of what's left never changes. If you want the result sorted as well, run it through the text sorter afterwards. Sorting first and then removing duplicates gives the same set of lines, just in a different order.

Line endings aren't a problem. Browsers normalize the line breaks in a text box to a single newline character, as the HTML standard requires for a textarea's value, so text pasted from Windows, macOS or Linux is compared the same way.

Cleaning a list before removing duplicates

Because matching is exact, the most common reason a duplicate survives is stray whitespace, often from copying out of a spreadsheet or a PDF. You can strip it with find and replace using its regular-expression option:

1. Find [ \t]+(?=\n|$) and replace with nothing, to remove trailing spaces and tabs.
2. Find (^|\n)[ \t]+ and replace with $1, to remove leading spaces and tabs.
3. Paste the cleaned text here and click Convert.

For example, the list apple , banana, apple, apple (with extra spaces) keeps three versions of apple if you run it as is. After the two clean-up steps, the result is just apple and banana.

For duplicates that differ only in capitalization, convert the list to lowercase first. Bear in mind this changes the text you keep, so Apple Inc would come out as apple inc.

Things this tool doesn't do

• No fuzzy matching. Jon Smith and John Smith are different lines. Spotting near-duplicates needs a person, or a spreadsheet with more careful rules.
• No duplicate count. The output shows the unique lines but not how many were removed. Paste the before and after into the code diff tool to see exactly which lines went.
• No column awareness. For CSV data, a line is the whole row. Two rows with the same email but different names are both kept.
• Not for paragraphs of prose. Every blank line after the first is removed, which merges your paragraphs. In the text Para one, blank, Para two, blank, Para one, the output is Para one, blank, Para two, with no gap left before anything you add later.

To see how many lines or words your list has, paste it into the word counter. Everything here runs in your browser, so lists with personal data such as email addresses never leave your device.