Finding duplicate files

[Frick]

Hey hey,

During the years the backups have just accumulated in random places, essentially, and I would like to clean up. Is there a program that can search for duplicate files? And by that I mean identical files. I tried dupeGuru, but that somehow found ... all files to be identical I guess? Like how a bunch of jpg's with different names, sizes and locations being marked 100% identical, or how a bunch of xls-files with completely different everything also were 100% matches with each other. I don't need any fancy functions, just a list of the files that share the name and filesize and created/last write dates.

 

[blobster21]

I have just discovered dupeguru thanks to you, and it's working as expected.

in my case it found 382 dupes and would let me reclaim 13Gb disk space...

 

[SomeOne99h]

SearchMyFiles http://www.nirsoft.net/utils/search_my_files.html
Or
CCleaner. Run it, then choose "Tools" then there is "Duplicate Finder".

 

[DeathtoGnomes]

CCleaner best tool app ever.

There is also an add-on for it, name escapes me atm.

 

[rtwjunkie]

CCleaner best tool app ever.

There is also an add-on for it, name escapes me atm.

CCenhancer(or close...away from computer right now)?

 

[Jetster]

CCleaner best tool app ever.

There is also an add-on for it, name escapes me atm.

Its also a good way to screw up your PC so use wisely

 

[SomeOne99h]

Its also a good way to screw up your PC so use wisely

I would use it only to show\view the files. Side note: I also use Defraggler to see how files are sorted on my drive and then I will use the Windows built-in Disk Defragmenter.

 

[Frick]

I have just discovered dupeguru thanks to you, and it's working as expected.

in my case it found 382 dupes and would let me reclaim 13Gb disk space...

According to it most of my steam library are dupes, and my Games folder is one of the few without any kind of backup.

Totally forgot about CCleaner!

 

[RejZoR]

For images, I love DupDetector
https://www.keronsoft.com/dupdetector.html

It's cool and special because it matches images and photos by content (like Google can search for similar images) and not filename or date. You can even adjust the sensitivity. It'll find identical images across different file formats as well as different resolutions/sizes. So you can weed out low resolution duplicates of higher res images or simply actually the same duplicate images.

 

[Drone]

Good old fdupes on Linux/macOS can do this.

Enhanced fork of fdupes for Windows called jdupes is even better. Less than 90 kb in size, has x86 and x64 versions, supports Unicode file paths on Windows platforms. It's extremely stable and fast even with lame CPU and HDD. With strong CPU and SSD it's light-fast, you can see the tests all over the internet. Usage is pretty straightforward, just read readme file:

For example: You're searching for dupes in folder called stuff and want to get the list of dupes:
jdupes -r ./stuff > dupes.txt

If command lines freak you out you can try TreeSize by JAM Software, it's free and has an awesome gui and easy to use and find dup files.

 

[PLAfiller]

That's a useful thread. I have the same problem actually. I will try the suggestions here. I want to clear and optimize my back up process.

 

[Jetster]

I worked on a PC that the guy believed in back ups. However he would copy the entire hard drive every time he bought a PC to a folder on his new PC. So when I went to recover data on a failing drive there was literally 5 systems on top of each other so almost 5 of everything he saved.

 

[Frick]

I worked on a PC that the guy believed in back ups. However he would copy the entire hard drive every time he bought a PC to a folder on his new PC. So when I went to recover data on a failing drive there was literally 5 systems on top of each other so almost 5 of everything he saved.

Oh man I feel your pain. The best part was when he came in one morning, all white in the face, handing me the external HDD with the words "I dropped it" escaping his pale lips. It had just slipped out the connector, so it lived.

 

[jboydgolfer]

For images, I love DupDetector
https://www.keronsoft.com/dupdetector.html

It's cool and special because it matches images and photos by content (like Google can search for similar images) and not filename or date. You can even adjust the sensitivity. It'll find identical images across different file formats as well as different resolutions/sizes. So you can weed out low resolution duplicates of higher res images or simply actually the same duplicate images.

i could use something like that...my goddamn dropbox account is FULL of duplicates, as is my local dropbox folder. such a pain in the ass.i have NO cluse where they are coming from

 

[Frick]

i could use something like that...my goddamn dropbox account is FULL of duplicates, as is my local dropbox folder. such a pain in the ass.i have NO cluse where they are coming from

Automatic camera uploads, maybe from different locations?

 

[jboydgolfer]

Yeah that was the first thing I ruled out it doesn't happen any longer but at the time it was only from my tablet for the life of me I can't figure it out. I don't know if it was the camera duplicating it or dropbox duplicating it