• Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.

Finding duplicate files

Frick

Fishfaced Nincompoop
Joined
Feb 27, 2006
Messages
18,914 (2.87/day)
Location
Piteå
System Name Black MC in Tokyo
Processor Ryzen 5 5600
Motherboard Asrock B450M-HDV
Cooling Be Quiet! Pure Rock 2
Memory 2 x 16GB Kingston Fury 3400mhz
Video Card(s) XFX 6950XT Speedster MERC 319
Storage Kingston A400 240GB | WD Black SN750 2TB |WD Blue 1TB x 2 | Toshiba P300 2TB | Seagate Expansion 8TB
Display(s) Samsung U32J590U 4K + BenQ GL2450HT 1080p
Case Fractal Design Define R4
Audio Device(s) Line6 UX1 + some headphones, Nektar SE61 keyboard
Power Supply Corsair RM850x v3
Mouse Logitech G602
Keyboard Cherry MX Board 1.0 TKL Brown
VR HMD Acer Mixed Reality Headset
Software Windows 10 Pro
Benchmark Scores Rimworld 4K ready!
Hey hey,

During the years the backups have just accumulated in random places, essentially, and I would like to clean up. Is there a program that can search for duplicate files? And by that I mean identical files. I tried dupeGuru, but that somehow found ... all files to be identical I guess? Like how a bunch of jpg's with different names, sizes and locations being marked 100% identical, or how a bunch of xls-files with completely different everything also were 100% matches with each other. I don't need any fancy functions, just a list of the files that share the name and filesize and created/last write dates.
 
Joined
Oct 24, 2004
Messages
1,294 (0.18/day)
I have just discovered dupeguru thanks to you, and it's working as expected. :confused:

in my case it found 382 dupes and would let me reclaim 13Gb disk space...
 
Joined
Oct 3, 2015
Messages
447 (0.14/day)
System Name My system is my pet?
Processor Intel Core i5 9400f 2.9GHz/4.0 Turbo (NoOC)
Motherboard Gigabyte Z370M D3H rev. 1.0
Cooling An okayish Cooler but it is better than Intel's stock
Memory Corsair Red Line 8x2 16GB 3000MHz (NoOC) DDR4-3000 15-17-17-35 (CMK16GX4M2B3000C15R) V1.35 ver 4.24
Video Card(s) NVIDIA GeForce MSI 980 Ti Golden Edition (NoOC)| Spare: GTX 650 Ti 1 GB
Storage Samsun 860 EVO 1 TB | Spare: WDC Black 930 GiB WD1003FZEX
Display(s) Asus VG248QZ 1920x1080 144hz 24" (Current: 60hz, because Contrast is lowered with 60+ Hz)
Case Corsair Air 540
Audio Device(s) Realtek ALC892
Power Supply Corsair 850W RMi power supply (Overkill I know -long story-)
Mouse Logitech M187 wireless (First day of use 30-9-2021)
Keyboard Logitech K270 wireless
Software Windows 10 21H2 / Nobara Linux (Modified Fedora Linux)
Joined
Jul 16, 2014
Messages
8,109 (2.29/day)
Location
SE Michigan
System Name Dumbass
Processor AMD Ryzen 7800X3D
Motherboard ASUS TUF gaming B650
Cooling Artic Liquid Freezer 2 - 420mm
Memory G.Skill Sniper 32gb DDR5 6000
Video Card(s) GreenTeam 1080 Gaming X 8GB
Storage Samsung EVO 500gb & 1Tb, 2tb HDD, 500gb WD Black
Display(s) 1x Nixeus NX_EDG27, 2x Dell S2440L (16:9)
Case Phanteks Enthoo Primo w/8 140mm SP Fans
Audio Device(s) onboard (realtek?) - SPKRS:Logitech Z623 200w 2.1
Power Supply Corsair HX1000i
Mouse Steeseries Esports Wireless
Keyboard Corsair K100
Software windows 10 H
Benchmark Scores https://i.imgur.com/aoz3vWY.jpg?2
CCleaner best tool app ever. :toast:

There is also an add-on for it, name escapes me atm. :(
 

rtwjunkie

PC Gaming Enthusiast
Supporter
Joined
Jul 25, 2008
Messages
13,909 (2.43/day)
Location
Louisiana -Laissez les bons temps rouler!
System Name Bayou Phantom
Processor Core i7-8700k 4.4Ghz @ 1.18v
Motherboard ASRock Z390 Phantom Gaming 6
Cooling All air: 2x140mm Fractal exhaust; 3x 140mm Cougar Intake; Enermax T40F Black CPU cooler
Memory 2x 16GB Mushkin Redline DDR-4 3200
Video Card(s) EVGA RTX 2080 Ti Xc
Storage 1x 500 MX500 SSD; 2x 6TB WD Black; 1x 4TB WD Black; 1x400GB VelRptr; 1x 4TB WD Blue storage (eSATA)
Display(s) HP 27q 27" IPS @ 2560 x 1440
Case Fractal Design Define R4 Black w/Titanium front -windowed
Audio Device(s) Soundblaster Z
Power Supply Seasonic X-850
Mouse Coolermaster Sentinel III (large palm grip!)
Keyboard Logitech G610 Orion mechanical (Cherry Brown switches)
Software Windows 10 Pro 64-bit (Start10 & Fences 3.0 installed)
Joined
Jan 17, 2010
Messages
12,275 (2.37/day)
Location
Oregon
System Name Juliette // HTPC
Processor Intel i7 9700K // AMD Ryzen 5 5600G
Motherboard ASUS Prime Z390X-A // ASRock B550 ITX-AC
Cooling Noctua NH-U12 Black // Stock
Memory Corsair DDR4 3600 32gb //G.SKILL Trident Z Royal Series 16GB (2 x 8GB) 3600
Video Card(s) ASUS RTX4070 OC// GTX 1650
Storage Samsung 970 EVO NVMe 1Tb, Intel 665p Series M.2 2280 1TB // Samsung 1Tb SSD
Display(s) ASUS VP348QGL 34" Quad HD 3440 x 1440 // 55" LG 4K SK8000 Series
Case Seasonic SYNCRO Q7// Silverstone Granada GD05
Audio Device(s) Focusrite Scarlett 4i4 // HDMI to Samsung HW-R650 sound bar
Power Supply Seasonic SYNCRO 750 W // CORSAIR Vengeance 650M
Mouse Cooler Master MM710 53G
Keyboard Logitech 920-009300 G512 SE
Software Windows 10 Pro // Windows 10 Pro
Joined
Oct 3, 2015
Messages
447 (0.14/day)
System Name My system is my pet?
Processor Intel Core i5 9400f 2.9GHz/4.0 Turbo (NoOC)
Motherboard Gigabyte Z370M D3H rev. 1.0
Cooling An okayish Cooler but it is better than Intel's stock
Memory Corsair Red Line 8x2 16GB 3000MHz (NoOC) DDR4-3000 15-17-17-35 (CMK16GX4M2B3000C15R) V1.35 ver 4.24
Video Card(s) NVIDIA GeForce MSI 980 Ti Golden Edition (NoOC)| Spare: GTX 650 Ti 1 GB
Storage Samsun 860 EVO 1 TB | Spare: WDC Black 930 GiB WD1003FZEX
Display(s) Asus VG248QZ 1920x1080 144hz 24" (Current: 60hz, because Contrast is lowered with 60+ Hz)
Case Corsair Air 540
Audio Device(s) Realtek ALC892
Power Supply Corsair 850W RMi power supply (Overkill I know -long story-)
Mouse Logitech M187 wireless (First day of use 30-9-2021)
Keyboard Logitech K270 wireless
Software Windows 10 21H2 / Nobara Linux (Modified Fedora Linux)
Its also a good way to screw up your PC so use wisely
I would use it only to show\view the files. Side note: I also use Defraggler to see how files are sorted on my drive and then I will use the Windows built-in Disk Defragmenter.
 

Frick

Fishfaced Nincompoop
Joined
Feb 27, 2006
Messages
18,914 (2.87/day)
Location
Piteå
System Name Black MC in Tokyo
Processor Ryzen 5 5600
Motherboard Asrock B450M-HDV
Cooling Be Quiet! Pure Rock 2
Memory 2 x 16GB Kingston Fury 3400mhz
Video Card(s) XFX 6950XT Speedster MERC 319
Storage Kingston A400 240GB | WD Black SN750 2TB |WD Blue 1TB x 2 | Toshiba P300 2TB | Seagate Expansion 8TB
Display(s) Samsung U32J590U 4K + BenQ GL2450HT 1080p
Case Fractal Design Define R4
Audio Device(s) Line6 UX1 + some headphones, Nektar SE61 keyboard
Power Supply Corsair RM850x v3
Mouse Logitech G602
Keyboard Cherry MX Board 1.0 TKL Brown
VR HMD Acer Mixed Reality Headset
Software Windows 10 Pro
Benchmark Scores Rimworld 4K ready!
I have just discovered dupeguru thanks to you, and it's working as expected. :confused:

in my case it found 382 dupes and would let me reclaim 13Gb disk space...

According to it most of my steam library are dupes, and my Games folder is one of the few without any kind of backup.

Totally forgot about CCleaner!
 
Joined
Oct 2, 2004
Messages
13,791 (1.94/day)
For images, I love DupDetector
https://www.keronsoft.com/dupdetector.html

It's cool and special because it matches images and photos by content (like Google can search for similar images) and not filename or date. You can even adjust the sensitivity. It'll find identical images across different file formats as well as different resolutions/sizes. So you can weed out low resolution duplicates of higher res images or simply actually the same duplicate images.
 
Joined
Sep 1, 2010
Messages
7,023 (1.42/day)
Good old fdupes on Linux/macOS can do this.

Enhanced fork of fdupes for Windows called jdupes is even better. Less than 90 kb in size, has x86 and x64 versions, supports Unicode file paths on Windows platforms. It's extremely stable and fast even with lame CPU and HDD. With strong CPU and SSD it's light-fast, you can see the tests all over the internet. Usage is pretty straightforward, just read readme file:

For example: You're searching for dupes in folder called stuff and want to get the list of dupes:
jdupes -r ./stuff > dupes.txt

If command lines freak you out you can try TreeSize by JAM Software, it's free and has an awesome gui and easy to use and find dup files.
 
Joined
Jan 31, 2012
Messages
2,447 (0.55/day)
Location
Bulgaria
System Name Sandfiller
Processor I5-10400
Motherboard MSI MPG Z490 GAMING PLUS
Cooling Noctua NH-L9i (92x25mm fan)
Memory 32GB Corsair LPX 2400 Mhz DDR4 CL14
Video Card(s) MSI RX 5700 XT GAMING X
Storage Intel 670P 512GB
Display(s) 2560x1080 LG 29" + 22" LG
Case SS RV02
Audio Device(s) Creative Sound Blaster Z
Power Supply Fractal Design IntegraM 650W
Mouse Logitech Triathlon
Keyboard REDRAGON MITRA
Software Windows 11 Home x 64
That's a useful thread. I have the same problem actually. I will try the suggestions here. I want to clear and optimize my back up process.
 
Joined
Jan 17, 2010
Messages
12,275 (2.37/day)
Location
Oregon
System Name Juliette // HTPC
Processor Intel i7 9700K // AMD Ryzen 5 5600G
Motherboard ASUS Prime Z390X-A // ASRock B550 ITX-AC
Cooling Noctua NH-U12 Black // Stock
Memory Corsair DDR4 3600 32gb //G.SKILL Trident Z Royal Series 16GB (2 x 8GB) 3600
Video Card(s) ASUS RTX4070 OC// GTX 1650
Storage Samsung 970 EVO NVMe 1Tb, Intel 665p Series M.2 2280 1TB // Samsung 1Tb SSD
Display(s) ASUS VP348QGL 34" Quad HD 3440 x 1440 // 55" LG 4K SK8000 Series
Case Seasonic SYNCRO Q7// Silverstone Granada GD05
Audio Device(s) Focusrite Scarlett 4i4 // HDMI to Samsung HW-R650 sound bar
Power Supply Seasonic SYNCRO 750 W // CORSAIR Vengeance 650M
Mouse Cooler Master MM710 53G
Keyboard Logitech 920-009300 G512 SE
Software Windows 10 Pro // Windows 10 Pro
I worked on a PC that the guy believed in back ups. However he would copy the entire hard drive every time he bought a PC to a folder on his new PC. So when I went to recover data on a failing drive there was literally 5 systems on top of each other so almost 5 of everything he saved.
 

Frick

Fishfaced Nincompoop
Joined
Feb 27, 2006
Messages
18,914 (2.87/day)
Location
Piteå
System Name Black MC in Tokyo
Processor Ryzen 5 5600
Motherboard Asrock B450M-HDV
Cooling Be Quiet! Pure Rock 2
Memory 2 x 16GB Kingston Fury 3400mhz
Video Card(s) XFX 6950XT Speedster MERC 319
Storage Kingston A400 240GB | WD Black SN750 2TB |WD Blue 1TB x 2 | Toshiba P300 2TB | Seagate Expansion 8TB
Display(s) Samsung U32J590U 4K + BenQ GL2450HT 1080p
Case Fractal Design Define R4
Audio Device(s) Line6 UX1 + some headphones, Nektar SE61 keyboard
Power Supply Corsair RM850x v3
Mouse Logitech G602
Keyboard Cherry MX Board 1.0 TKL Brown
VR HMD Acer Mixed Reality Headset
Software Windows 10 Pro
Benchmark Scores Rimworld 4K ready!
I worked on a PC that the guy believed in back ups. However he would copy the entire hard drive every time he bought a PC to a folder on his new PC. So when I went to recover data on a failing drive there was literally 5 systems on top of each other so almost 5 of everything he saved.

Oh man I feel your pain. The best part was when he came in one morning, all white in the face, handing me the external HDD with the words "I dropped it" escaping his pale lips. It had just slipped out the connector, so it lived.
 
Joined
Oct 17, 2012
Messages
9,781 (2.35/day)
Location
Massachusetts
System Name Americas cure is the death of Social Justice & Political Correctness
Processor i7-11700K
Motherboard Asrock Z590 Extreme wifi 6E
Cooling Noctua NH-U12A
Memory 32GB Corsair RGB fancy boi 5000
Video Card(s) RTX 3090 Reference
Storage Samsung 970 Evo 1Tb + Samsung 970 Evo 500Gb
Display(s) Dell - 27" LED QHD G-SYNC x2
Case Fractal Design Meshify-C
Audio Device(s) on board
Power Supply Seasonic Focus+ Gold 1000 Watt
Mouse Logitech G502 spectrum
Keyboard AZIO MGK-1 RGB (Kaith Blue)
Software Win 10 Professional 64 bit
Benchmark Scores the MLGeesiest
For images, I love DupDetector
https://www.keronsoft.com/dupdetector.html

It's cool and special because it matches images and photos by content (like Google can search for similar images) and not filename or date. You can even adjust the sensitivity. It'll find identical images across different file formats as well as different resolutions/sizes. So you can weed out low resolution duplicates of higher res images or simply actually the same duplicate images.

i could use something like that...my goddamn dropbox account is FULL of duplicates, as is my local dropbox folder. such a pain in the ass.i have NO cluse where they are coming from
 

Frick

Fishfaced Nincompoop
Joined
Feb 27, 2006
Messages
18,914 (2.87/day)
Location
Piteå
System Name Black MC in Tokyo
Processor Ryzen 5 5600
Motherboard Asrock B450M-HDV
Cooling Be Quiet! Pure Rock 2
Memory 2 x 16GB Kingston Fury 3400mhz
Video Card(s) XFX 6950XT Speedster MERC 319
Storage Kingston A400 240GB | WD Black SN750 2TB |WD Blue 1TB x 2 | Toshiba P300 2TB | Seagate Expansion 8TB
Display(s) Samsung U32J590U 4K + BenQ GL2450HT 1080p
Case Fractal Design Define R4
Audio Device(s) Line6 UX1 + some headphones, Nektar SE61 keyboard
Power Supply Corsair RM850x v3
Mouse Logitech G602
Keyboard Cherry MX Board 1.0 TKL Brown
VR HMD Acer Mixed Reality Headset
Software Windows 10 Pro
Benchmark Scores Rimworld 4K ready!
i could use something like that...my goddamn dropbox account is FULL of duplicates, as is my local dropbox folder. such a pain in the ass.i have NO cluse where they are coming from

Automatic camera uploads, maybe from different locations?
 
Joined
Oct 17, 2012
Messages
9,781 (2.35/day)
Location
Massachusetts
System Name Americas cure is the death of Social Justice & Political Correctness
Processor i7-11700K
Motherboard Asrock Z590 Extreme wifi 6E
Cooling Noctua NH-U12A
Memory 32GB Corsair RGB fancy boi 5000
Video Card(s) RTX 3090 Reference
Storage Samsung 970 Evo 1Tb + Samsung 970 Evo 500Gb
Display(s) Dell - 27" LED QHD G-SYNC x2
Case Fractal Design Meshify-C
Audio Device(s) on board
Power Supply Seasonic Focus+ Gold 1000 Watt
Mouse Logitech G502 spectrum
Keyboard AZIO MGK-1 RGB (Kaith Blue)
Software Win 10 Professional 64 bit
Benchmark Scores the MLGeesiest
Yeah that was the first thing I ruled out it doesn't happen any longer but at the time it was only from my tablet for the life of me I can't figure it out. I don't know if it was the camera duplicating it or dropbox duplicating it
 
Top