Removing duplicate rows in Notepad++

Notepad++ with the TextFX plugin can do this, provided you wanted to sort by line, and remove the duplicate lines at the same time.

To install the TextFX in the latest release of Notepad++ you need to download it from here: https://sourceforge.net/projects/npp-plugins/files/TextFX

The TextFX plugin used to be included in older versions of Notepad++, or be possible to add from the menu by going to Plugins -> Plugin Manager -> Show Plugin Manager -> Available tab -> TextFX -> Install. In some cases it may also be called TextFX Characters, but this is the same thing.

The check boxes and buttons required will now appear in the menu under: TextFX -> TextFX Tools.

Make sure "sort outputs only unique..." is checked. Next, select a block of text (Ctrl+A to select the entire document). Finally, click "sort lines case sensitive" or "sort lines case insensitive"

https://i.stack.imgur.com/1qnvS.png

---

Since Notepad++ Version 6 you can use this regex in the search and replace dialogue:

^(.*?)$\s+?^(?=.*^\1$)

and replace with nothing. This leaves from all duplicate rows the last occurrence in the file.

No sorting is needed for that and the duplicate rows can be anywhere in the file!

You need to check the options "Regular expression" and ". matches newline":

https://i.imgur.com/dY3LCMD.png

^ matches the start of the line.

(.*?) matches any characters 0 or more times, but as few as possible (It matches exactly on row, this is needed because of the ". matches newline" option). The matched row is stored, because of the brackets around and accessible using \1

$ matches the end of the line.

\s+?^ this part matches all whitespace characters (newlines!) till the start of the next row ==> This removes the newlines after the matched row, so that no empty row is there after the replacement.

(?=.*^\1$) this is a positive lookahead assertion. This is the important part in this regex, a row is only matched (and removed), when there is exactly the same row following somewhere else in the file.

---

If the rows are immediately after each other then you can use a regex replace:

Search Pattern: ^(.*\r?\n)(\1)+

Replace with: \1

---

In version 7.8, you can accomplish this without any plugins - Edit -> Line Operations -> Remove Consecutive Duplicate Lines. You will have to sort the file to place duplicate lines in consecutive order before this works, but it does work like a charm.

Sorting options are available under Edit -> Line Operations -> Sort By ...

---

Notepad++

-> Replace window

Ensure that in Search mode you have selected the Regular expression radio button

Find what:

^(.*)(\r?\n\1)+$

Replace with:

$1

Before:

and we think there and we think there single line Is it possible to Is it possible to

After:

and we think there single line Is it possible to

---

If you don't care about row order (which I don't think you do), then you can use a Linux/FreeBSD/Mac OS X/Cygwin box and do:

$ cat yourfile | sort | uniq > yourfile_nodups

Then open the file again in Notepad++.

---

As of Notepad++ version 8.1, there is a specific command to do precisely what this popular question asks. On can remove duplicated rows in a text file with the menu command Edit > Line Operations > Remove Duplicate Lines.

There is no need to install a plugin (as the currently accepted answer suggests), or sort the lines beforehand, or use the regex syntax in the Replace dialogue as other answers suggested.

https://i.stack.imgur.com/BWJZJ.png

---

The latter versions of Notepad++ do not apparently include the TextFX plugin at all. In order to use the plugin for sorting/eliminating duplicates, the plugin must be either downloaded and installed (more involved) or added using the plugin manager.

A) Easy way (as described here).

Plugins -> Plugin Manager -> Show Plugin Manager -> Available tab -> TextFX Characters -> Install

B) More involved way, if another version is needed or the easy way does not work.

Download the plugin from SourceForge: http://downloads.sourceforge.net/project/npp-plugins/TextFX/TextFX%20v0.26/TextFX.v0.26.unicode.bin.zip Open the zip file and extract NppTextFX.dll Place NppTextFX.dll in the Notepad++ plugins directory, such as: C:\Program Files\Notepad++\plugins Start Notepad++, and TextFX will be one of the file menu items (as seen in Answer #1 above by Colin Pickard)

After installing the TextFX plugin, follow the instructions in Answer #1 to sort and remove duplicates.

Also, consider setting up a keyboard shortcut using Settings > Shorcut mapper if you use this command frequently or want to replicate a keyboard shortcut, such as F9 in TextPad for sorting.

---

As of now, it's possible to remove all consecutive duplicate lines with Notepad in-built functionality. Sort the lines first:

Edit > Line Operations > "Sort lines lexicographically",

then

Edit > Line Operations > "Remove Consecutive Duplicate Lines".

The regex solution suggested above didn't remove all duplicate lines for me, but just the consecutive ones as well.

---

You may need a plugin to do this. You can try the command line cc.ddl(delete duplicate lines) of ConyEdit. It is a cross-editor plugin for the text editors, including Notepad++.

With ConyEdit running in background, follow the steps below:

enter the command line cc.ddl at the end of the text. copy the text and the command line. paste, then you will see what you want.

https://i.stack.imgur.com/EtPOM.gif

---

Search for the regular expression: \b(\w+)\b([\w\W]*)\b\1\b

Replace it with: $1$2

Hit the Replace button until there are no more matches for the regular expression in your file.

---

None worked for me.

A solution is:

Replace

^(.*)\s+(\r?\n\1\s+)+$

with

\1

---

The plugin manager is currently unavailable (does not come with the distribution) for Notepad++. You must install it manually (https://github.com/bruderstein/nppPluginManager/releases) and even if you do, a lot of the plugins are not available anymore (no TextFX) plugin.

Maybe there is another plugin which contains the required functionality. Other than that, the only way to do it in Notepad++ is to use some special regex for matching and then replacing (Ctrl + F → Replace tab).

Although there are many functionalities available via Edit menu item (trimming, removing empty lines, sorting, converting EOL) there is no "unique" operation available.

If you have Windows 10 then you can enable Bash (just type Ubuntu in Microsoft Store and follow the instructions in the description to install it) and use cat your_file.txt | sort | uniq > your_file_edited.txt. Of course you must be in the same working directory as "your_file.txt" or refer to it via its path.

---

Whether the file is sorted or not, you can use below regex to remove duplicates in anywhere occurred in your file.

Find what: ^([^\r]*[^\n])(.*?)\r?\n\1$
Replace with: \1\2
Search Mode:

"Regular expression"

Check the ". matches newline" option

do "Replace All" as many time as possible until you see "0 occurrences were replaced"

---

Extending the top answer, you can also use a 2nd lookahead to find rows that are almost duplicates of other rows.

^(\s*(<PackageReference Include=".*" Version=).*)$\s+?^(?=.*^\2.*$)

Here I'm after multiple references to the same <PackageReference Include=".*" string, regardless of its version.

Test data

<PackageReference Include="Package1" Version="2.2.1" />

    <PackageReference Include="Package1" Version="2.2.1" /> // Match
<PackageReference Include="Package1" Version="2.2.2" />

<PackageReference Include="Package2" Version="5.1" /> // Match
<PackageReference Include="Package2" Version="5.2" />

<PackageReference Include="Package3" Version="2.2.1" /> // No match
<PackageReference Include="Package4" Version="2.2.1" />

See a breakdown of what the regex terms mean and try with your own data on this regex101 share.

---

Difficult to do this in NPP. Better way is following:

Download cygwin utility, it is simple Linux terminal under windows. It allow to execute any Linux command in Windows. And you have sort -u there.

---