Copying big amount data from HDD

[CodeError]

When copying from internal HDD to external HDD can I just select all, copy and paste?

Files have more than 3TB. Will that amount of data in one push be too much for HDD?

 

[DirtyDingusMcgee]

It will be fine. It will take a while, but it will be fine.

Can use FastCopy or TeraCopy if you like. Allows for a bit more control over the copy process.

 

[nageme]

Doesn't matter how much you copy, but potentially notice:

* External enclosures with poor heatsinking or ventilation may heat up the HDD over time, especially 3.5" drives in plastic enclosures.
Just keep watch to see at what temperature it stabilizes. Up to 55-60°C should be okay, though it's high (check the specs).

* OS/boot partitions can't be copied that simply, because some system files and files in use aren't trivially copyable.

 

[Chrispy_]

I like to use Robocopy to do copies of large file sets because if a windows explorer copy fails for any reason, you have to manually work out exactly how far it got to resume the copy without creating duplicates, or you have to delete everything copied and start again.

I find the /MIR switch very very useful for anything that involves updating rather than pure copying, and robocopy is excellent at managing NTFS permissions and attributes via /DATSOU or /COPYALL switches.

Robocopy "Robust File Copy" - Windows CMD - SS64.com

ss64.com

Example:
Robocopy "c:\source folder" "d:\destination folder" /copyall /mir /log:c:\robocopy_log.txt /tee

This will copy everything from c:\source folder to d:\destination folder, making an exact copy of all subfolders, security permissions, hidden files, and all file attributes, while showing the output on screen and also dumping a log file to your c: drive. If it finishes while you're away and you want to see where it got to then just open up the logfile.

 

[Harlow]

I like to use Robocopy to do copies of large file sets because if a windows explorer copy fails for any reason, you have to manually work out exactly how far it got to resume the copy without creating duplicates,

A useful tip. I use FreeFileSync for the same reason when transferring photos from a USB memory card reader to my laptop drive. When all the files have been copied, I use the "File content" option under "Comparison" in FFS to ensure no data corruption has occured over USB. I had problems in the past with long USB leads and a few files on the laptop were damaged/lost.
https://freefilesync.org/

Just keep watch to see at what temperature it stabilizes. Up to 55-60°C should be okay, though it's high (check the specs).

This is one of the reasons why I've stopped using my collection of WD and Seagate USB3 3.5" desktop drives. The other is they're mostly SMR and writes can slow down after deleting files. That and the fact that if a USB drive topples over when running, it can result in a fatal head crash.

I used to point a large 12" desktop fan across the USB housings to cool them down and the temperature dropped from around +55°C to +45°C. If you "shuck" the hard disks and transfer them to a PC with adequate forced ventilation, they run at +35°C.

 

[ir_cow]

It will be fine. It will take a while, but it will be fine.

Can use FastCopy or TeraCopy if you like. Allows for a bit more control over the copy process.

TearCopy with verify enabled is a must. Really gives you a piece of mind that what you copied is actually done correctly.

 

[DaemonForce]

All of those options should be fine. One thing I tend to do in the copy process is selectively order the directory copy order if it's going to be on a volume where certain files see more frequent access than others and if they tend to be more reasonable at higher read speeds. First to copy tends to get priority on the platters.

 

[Waldorf]

@CodeError
you will probably see a little drop in transfer speed once the cache filled, but that should be it.
but like others said, its a pain if something happens and you have to start over, so either use an app to do it, or just transfer in blocks, so its easier to see what was done already,
shortens time spend to redo.


@Harlow
type of connection has nothing to do with it. type of drive does.
i have had my nand drives (sata/pcie) in enclosures hit the case/fall on the desk multiple times, not a single issue, not even if in use.

and if you have drives in external enclosures that get hot, wrong enclosure.
brands like iceydock (and the like) always had decent heatsinks (when i still used hdd), and for fast drives like raptors i just went with units with an internal fan.
ignoring "too low" temps arent beneficial, and can hurt lifetime same way highs temps can, but its less and varies more between brands/type of drive used,
and basically you want to avoid going way below 40C.

 

[_roman_]

* OS/boot partitions can't be copied that simply, because some system files and files in use aren't trivially copyable.

Let me fix that for you. Windows / boot partitions maybe can not be copied that simple. More an issue to get it back to 100% when you want to restore it later.

-- I regularly copy my hole gnu gentoo linux to an external nvme usb bridge case. Everything + encryption. Just fix the bootloader and it can be used instantly to work from.

-- I was just curious, because of someone else comment here. -> post #6 in regards of most likely windows

Make sure that the "cp" command is executed properly

I make a backup copy of my files with cp command as described in this topic (Copy files in Linux, avoid the copy if files do exist in destination): https://superuser.com/a/1377730/1008457 Command is:

unix.stackexchange.com

cp will return with exit status 0 if no errors happened, otherwise 1

--

Anyway the topic poster should use of of many copy software which also checks the files. There are plenty of suggestions above.

 

[Harlow]

and if you have drives in external enclosures that get hot, wrong enclosure.

The drives I mentioned were all commercially available WD, Seagate and Samsung 3.5" USB3 "desktop" style drives purchased from Amazon. The image below of a WD Easystore is typical of the close-fitting plastic case used to house these drives. Some of these cases have ventilation slots at either end. Others have ventilation slots along one edge only. All of the drives get hot during prolonged use, especially in summer.

I agree with you about the "wrong enclosure" bit. As for reducing drive temperatures, I decided not to waste more money shucking the drives and buying a dozen IcyDock enclosures with fans. Just throwing good money after bad. Instead I mothballed the USB disks and switched to multi-disk RAID-Z2 SAS arrays plus LTO tape.

type of connection has nothing to do with it. type of drive does.

I had problems in the past with long USB leads and a few files on the laptop were damaged/lost.

The type of connection did affect my file USB file transfers. Long cables caused problems. Short cables were fine. The transmission standard used was USB3 in both cases, but the types were short and long. Semantics?

Each evening when I copied up to 100GB of RAW + JPG files from the day's shoot on to two laptops plus an external SSD (three copies) a few random files would end up corrupted on each disk. When I replaced the 1m/3ft USB3 cables from the card readers with a shorter 30cm/1ft USB3 cables, file corruption ceased. I ran the same test at home copying files from Compact Flash and Secure Digital UHS-II in four different card readers over to hard disks and SSDs in three desktop systems. In each case, long USB3 leads corrupted a few files. Card readers were Kingston, Lexar, Sandisk.

FreeFileSync detected the file corruption in each instance, allowing me to replace the damaged files, before wiping the CF and SD cards in preparation for the next day's shoot.

 

[Waldorf]

where a product is sold has no bearing on its quality, enough sold by 3rd parties shipping strait out of asia, where the shop never even seen the time or "tried" it.
my backups are more important than saving 5$ on an enclosure, icydock m2 is 25$ so why buy no name. but thats just me.

the quality of the cable is relevant, not primarily its length.
i have multiple 5-10ft cables that work without problems, but then again im not going with cost as priority, so all are thunderbold "certified".

 

[DaemonForce]

where a product is sold has no bearing on its quality

I mean...Sometimes it's a sign. If there's some big earthquake that happens around a manufacturing plant and you suddenly see a bunch of these drives for sale at cutthroat rates, you might be a little curious as to why.

 

[Mikell]

Try Robocopy or GUI file synchronization tools like Syncback , Gs Richcopy360 , FreeFileSync or rsync, all are useful for large transfers, especially if you need to resume interrupted transfers or keep the data synchronized.

 

[Ruru]

Just basic Windows' copypaste has worked always for me. With a HDD it takes a while, but I watch videos when waiting.

 

[Hardcore Games]

I have a USB card reader and it comes with a short cable so as not to be impacted by slow performance

USB cables are cheap and are not very durable as they are made up of several very fine wires

 

[freeagent]

Not much help here.. but this is why I gave up on spinners. Seeing MB/s vs GB/s is mind numbing.

 

[SPDIF]

When copying from internal HDD to external HDD can I just select all, copy and paste?

Files have more than 3TB. Will that amount of data in one push be too much for HDD?

For a HDD nothing is too much to write data away, you just have to agree with some waiting time, that's all there is to. Especially when writing small files it can go very slow. Big chunks go for me relative fast with a modern HDD. Even nowadays HDD are still in use in many data-centers or cloud service, because it can handle PB of storage at lower price. And they are proven to be very robust.

Yes copy/paste is enough, no program can speed up this, the HDD is the slowest in the chain. Even programs that claim faster writing are parking data in memory, but yet has to wait anyway for writing it away to the HDD. Just make sure you use USB3.0 minimum, or you will be sorry to copy this amount of data over. With internal it will go much faster, then windows will take memory as buffer like a SSD do with DRAM. and SLC NAND buffer.

The SATA-AHCI port will consistently outperform the USB port, this is because it's designed specifically for hard disks whereas USB isn't. Also the USB bridge chips (to bridge USB to SATA) that you get in external drives aren't necessarily the fastest.

USB is getting better but still puts a greater load on the CPU when transferring data due to its nature. If you have one that's compatible with UASP (USB attached SCSI) then speeds will be tolerable, but if you have to rely on USB Mass Storage then that's a whole world of pain.

 

[Hardcore Games]

Hard disks at 40TB are probably going to be available later this year. Next year 44TB and maybe 60TB down the road.