• Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.
  • The forums have been upgraded with support for dark mode. By default it will follow the setting on your system/browser. You may override it by scrolling to the end of the page and clicking the gears icon.

Finding duplicate files

Frick

Frick

Fishfaced Nincompoop
Joined
Feb 27, 2006
Messages
20,115 (2.87/day)
Location
norr
System Specs
System Name Black MC in Tokyo
Processor Ryzen 5 7600
Motherboard MSI X670E Gaming Plus Wifi
Cooling Be Quiet! Pure Rock 2
Memory 2 x 16GB Corsair Vengeance @ 6000Mhz
Video Card(s) XFX 6950XT Speedster MERC 319
Storage Kingston KC3000 1TB | WD Black SN750 2TB |WD Blue 1TB x 2 | Toshiba P300 2TB | Seagate Expansion 8TB
Display(s) Samsung U32J590U 4K + BenQ GL2450HT 1080p
Case Fractal Design Define R4
Audio Device(s) Plantronics 5220, Nektar SE61 keyboard
Power Supply Corsair RM850x v3
Mouse Logitech G602
Keyboard Dell SK3205
Software Windows 10 Pro
Benchmark Scores Rimworld 4K ready!
Hey hey,

During the years the backups have just accumulated in random places, essentially, and I would like to clean up. Is there a program that can search for duplicate files? And by that I mean identical files. I tried dupeGuru, but that somehow found ... all files to be identical I guess? Like how a bunch of jpg's with different names, sizes and locations being marked 100% identical, or how a bunch of xls-files with completely different everything also were 100% matches with each other. I don't need any fancy functions, just a list of the files that share the name and filesize and created/last write dates.
 
I have just discovered dupeguru thanks to you, and it's working as expected. :confused:

in my case it found 382 dupes and would let me reclaim 13Gb disk space...
 
CCleaner best tool app ever. :toast:

There is also an add-on for it, name escapes me atm. :(
 
Jetster said:
Its also a good way to screw up your PC so use wisely
Click to expand...
I would use it only to show\view the files. Side note: I also use Defraggler to see how files are sorted on my drive and then I will use the Windows built-in Disk Defragmenter.
 
blobster21 said:
I have just discovered dupeguru thanks to you, and it's working as expected. :confused:

in my case it found 382 dupes and would let me reclaim 13Gb disk space...
Click to expand...

According to it most of my steam library are dupes, and my Games folder is one of the few without any kind of backup.

Totally forgot about CCleaner!
 
For images, I love DupDetector
https://www.keronsoft.com/dupdetector.html

It's cool and special because it matches images and photos by content (like Google can search for similar images) and not filename or date. You can even adjust the sensitivity. It'll find identical images across different file formats as well as different resolutions/sizes. So you can weed out low resolution duplicates of higher res images or simply actually the same duplicate images.
 
Good old fdupes on Linux/macOS can do this.

Enhanced fork of fdupes for Windows called jdupes is even better. Less than 90 kb in size, has x86 and x64 versions, supports Unicode file paths on Windows platforms. It's extremely stable and fast even with lame CPU and HDD. With strong CPU and SSD it's light-fast, you can see the tests all over the internet. Usage is pretty straightforward, just read readme file:

For example: You're searching for dupes in folder called stuff and want to get the list of dupes:
jdupes -r ./stuff > dupes.txt

If command lines freak you out you can try TreeSize by JAM Software, it's free and has an awesome gui and easy to use and find dup files.
 
That's a useful thread. I have the same problem actually. I will try the suggestions here. I want to clear and optimize my back up process.
 
I worked on a PC that the guy believed in back ups. However he would copy the entire hard drive every time he bought a PC to a folder on his new PC. So when I went to recover data on a failing drive there was literally 5 systems on top of each other so almost 5 of everything he saved.
 
Jetster said:
I worked on a PC that the guy believed in back ups. However he would copy the entire hard drive every time he bought a PC to a folder on his new PC. So when I went to recover data on a failing drive there was literally 5 systems on top of each other so almost 5 of everything he saved.
Click to expand...

Oh man I feel your pain. The best part was when he came in one morning, all white in the face, handing me the external HDD with the words "I dropped it" escaping his pale lips. It had just slipped out the connector, so it lived.
 
RejZoR said:
For images, I love DupDetector
https://www.keronsoft.com/dupdetector.html

It's cool and special because it matches images and photos by content (like Google can search for similar images) and not filename or date. You can even adjust the sensitivity. It'll find identical images across different file formats as well as different resolutions/sizes. So you can weed out low resolution duplicates of higher res images or simply actually the same duplicate images.
Click to expand...

i could use something like that...my goddamn dropbox account is FULL of duplicates, as is my local dropbox folder. such a pain in the ass.i have NO cluse where they are coming from
 
jboydgolfer said:
i could use something like that...my goddamn dropbox account is FULL of duplicates, as is my local dropbox folder. such a pain in the ass.i have NO cluse where they are coming from
Click to expand...

Automatic camera uploads, maybe from different locations?
 
Yeah that was the first thing I ruled out it doesn't happen any longer but at the time it was only from my tablet for the life of me I can't figure it out. I don't know if it was the camera duplicating it or dropbox duplicating it
 
You must log in or register to reply here.
Back
Top