Guide: Remove Duplicate Lines
↑ Back to toolWhat it does
Paste a list with one item per line. This duplicate line remover compares the lines and writes the unique ones. You can ignore case, trim spaces, skip blank lines, keep the first or last copy, and sort A–Z or Z–A. Copy the result or download unique-lines.txt.
The header shows how many lines you pasted, how many remain, and how many were removed. Blank lines count in the input. With Drop empty on, they are part of the removed total.
Why use it?
- Clean an email export, an id list, or a set of domains before you import them.
- Collapse repeated log lines when you only need the distinct messages.
- Sort a unique list without opening a spreadsheet.
- Keep the work in the browser when the list has addresses or customer ids.
Options
- Case sensitive —
Adaandadaare different lines. Off, they match. Comparison uses a case fold, so it is not limited to English A–Z. - Trim — leading and trailing spaces are ignored for the match, and the output line is trimmed.
- Drop empty — blank lines are removed before duplicates are compared.
- Keep first — the earliest copy stays, later copies go.
- Keep last — the latest copy stays. Useful when a later line is the corrected spelling.
- Original order, A–Z, Z–A — sort runs after duplicates are removed.
Examples
Input, with case folded and blanks dropped:
ada@example.com lin@example.com ada@example.com Ada@example.com lin@example.com
Keep first leaves the earliest spelling:
ada@example.com lin@example.com
Keep last leaves the later spelling of the same address, then the later lin line:
Ada@example.com lin@example.com
How to use
- Paste one item per line. Emails, ids, hostnames, or log lines.
- Set the chips. Start with Trim, Drop empty, and Keep first if you want a clean unique list in the original order.
- Copy or download. The file is plain text, one unique line per row.
Lines vs columns
This tool does not parse commas. A CSV row is one line, so two rows are duplicates only when the entire line matches. To drop rows that share an email column while the other columns differ, open CSV Operations and run a unique query on that column. To see what changed between two drafts of a list, use Text Diff.
Common mistakes
- Leaving case sensitive on for emails. Mailbox names often differ only by capital letters.
- Forgetting trim. A trailing space makes two copies of the same id look unique.
- Expecting a column dedupe. Whole lines only. Split the file in CSV Operations when one field matters.
- Sorting first in your head. A–Z replaces the original order. Choose Original order when the sequence of first appearance matters.
FAQ
Does this upload the list?
No. Comparison, sorting, copy, and the download file all happen in your browser. A share link stores the unique lines only if you choose Share.
What is the difference between keep first and keep last?
Keep first leaves the earliest copy and drops later matches. Keep last leaves the latest copy in its place and drops the earlier ones. Order of the surviving lines follows the option you picked, then any sort.
Does case sensitive change Ada and ada?
When Case sensitive is on, Ada and ada stay as two lines. When it is off, they count as the same line. The text that remains is the copy you kept, so keep last can leave Ada while keep first leaves ada.
What does Trim do?
Leading and trailing spaces are ignored when two lines are compared, and the written line is the trimmed text. Lines that differ only by spaces collapse into one.
Can it dedupe a CSV by one column?
This page compares whole lines. Two rows that share an email but differ in another column stay if the full lines differ. For a column inside a spreadsheet, use CSV Operations.
Does sort happen before or after duplicates are removed?
After. A–Z and Z–A order the unique lines. Original order leaves them in the order of the copy you kept.
Related terms
Remove duplicate lines, remove duplicate lines online, delete duplicate lines, dedupe a list, unique lines, remove repeated lines, deduplicate text, sort lines and remove duplicates.
Similar tools
Conclusion
Paste the list, choose which copy survives, and download the unique lines. For a count of the draft before you clean it, use the word counter. A shorter walkthrough is in the remove duplicate lines guide.