How to use the Remove Duplicate Lines
- Paste your list into Input (one item per line) or open a
.txt/.csvfile. - Choose how lines are compared: Ignore case treats “Apple” and “apple” as the same, Trim spaces ignores leading and trailing whitespace.
- Optionally pick a Sort order — alphabetical, natural numeric, by length, reversed or shuffled.
- Copy or download the result. The counters show how many duplicates and empty lines were removed.
What does a duplicate line remover do?
It goes through text line by line and keeps only the first (or last) occurrence of each line, deleting the repeats while preserving the original order. It's the quickest way to clean up email lists, keyword lists, URLs from a crawl, log entries, CSV columns, tag lists or any export that contains the same value more than once — without opening a spreadsheet or writing a script. Everything runs in your browser, so even lists with hundreds of thousands of lines are processed instantly and privately.
Comparison options
| Option | Effect | Example |
|---|---|---|
| Ignore case | Lines that differ only in capitalisation count as duplicates; the first spelling is kept. | Apple, apple, APPLE → Apple |
| Trim spaces | Leading and trailing spaces and tabs are removed before comparing and in the output. | " red " and "red" → red |
| Remove empty lines | Blank or whitespace-only lines are dropped. | — |
| Keep only lines that never repeat | Removes every copy of a duplicated line, leaving values that occur exactly once. | a b a c → b c |
| Keep last occurrence | Each value stays at the position of its last appearance — useful for logs where the latest entry wins. | a b a → b a |
Sorting lines
- A → Z / Z → A sorts alphabetically using your browser's locale-aware collation, so accented letters sort next to their base letters (
éwithe). - Natural sorts numbers inside text by value:
file2,file10,file100instead of the plain alphabeticalfile10,file100,file2. - By length orders lines from shortest to longest (or the reverse), handy for finding outliers.
- Shuffle randomises the order with an unbiased Fisher–Yates shuffle — for raffles, sampling or randomised test data.
Equivalent command-line recipes
On Linux or macOS, sort -u file.txt removes duplicates but also sorts; awk '!seen[$0]++' file.txt keeps the original order like this tool's default; sort file.txt | uniq -u keeps only lines that never repeat; sort -V is a natural sort. This page gives the same results without a terminal and shows exactly which lines were duplicated and how often.
Need to compare two lists instead? Use the Diff Checker. To change capitalisation of the whole list first, run it through the Case Converter.
Tools that pair well with the Remove Duplicate Lines
- Word Counter — Paste or type your text to count words and characters live — plus reading and speaking time, keyword density and length checks for SEO titles, meta descriptions and social posts.
- Slug Generator — Convert any title into a clean, lowercase URL slug — with proper transliteration of accents, umlauts and Cyrillic, optional stop-word removal and a bulk mode for whole lists.
- Soft Hyphen Generator — Stop long words like “Versicherungsleistungen” from overflowing on mobile: insert invisible soft hyphens automatically or at your own break points, then copy the result.
- Smart Quotes to Straight Quotes — Code copied from Slack, Notion, Word or a blog won’t run?