thinking of 2x4TB backup+archive system, what are my options? nas/das/raid/manual?

[kn00tcn]

i am wondering what possibilities exist for family photos, camcorder footage, OS backups, etc

4TB is probably the size i'll go for, of course such a large size should be mirrored for safety, so let's say at least 2 drives

but i do not want some $600 x86 24/7 server, not yet at least, so what exists? i have a few ideas, let me know if there's a problem or one i missed:

-the absolute most basic way is 2 external enclosures with one hard drive each over usb3/esata where i manually copy files to each drive individually
-a single enclosure that holds 2 drives, some of them may have RAID options or a one click mirror button, this is a DAS i guess (there is a chinese one that caught my eye)
-a NAS (or DAS?) from the usual suspects like synology or qnap, complete with user management & multiple OS support, likely ARM, possible RAID
-a small PC, miniITX or something, ARM or x86, possible RAID

possible problems:

-RAID (1, not 0 obviously) means mirroring right? but doesnt this mean fragmentation is identical on both drives, & what is going to happen when one fails? i put in the replacement & wait hours for it to rebuild the array? what's wrong with mere software syncing (dropbox type, except locally to 2 drives)?
-MANY external enclosures & even DASs/NASs say they only support 2TB, 3 if you're lucky, so what's going on with 4!? it sounds like i have to be very careful...

what i think i am looking for / what i think fits my needs:

-2 enclosures with one drive each, this way one person can use one copy, another can use the other, i do not want network based as my router is 100mbit & the clients with gigabit are mostly mine (as in, the other router clients that would use the storage will be slow, not to mention i filled up my gigabit switch's ports), plus i certainly dont want wireless
-software syncing instead of raid, i hope there is software that lets me choose individual files or folders, push them to the drives... as opposed to a single blob of files that forces a sync to the 2 drives + computer i'm using (aka, i want partial sync, not to mirror the drive contents back onto the local machine)
-if such sync software doesnt quite exist, well then i'll just manually copy my files to the 2 drives

maybe i answered myself after all this thinking, but there is still the matter of which enclosures support 4TB & do partial sync tools exist?

EDIT: of course, which 4TB drives to buy? should i get 2 of the same or get 2 brands/models for testing purposes? wish hitachi made 'green' drives

 

[ne6togadno]

www.freenas.org

 

[kn00tcn]

www.freenas.org

lolz... yes everyone's heard of that

i'll go read about it, but i'm mostly asking about hardware since it's not a problem if i have to manually copy files around with explorer

i also said i do not want to have a network only device

 

[SuperSoph_WD]

Hey there, @kn00tcn!

I'm not sure I'll be able to cover all your points, but feel free to let me know if you have any additional questions.
1. The absolute most basic way is 2 external enclosures with one hard drive each over usb3/esata where i manually copy files to each drive individually

This sounds good, because you will have 2 separate externals to backup to and if one fails, you'd have the other, right? So this definitely sounds safe! But then again, I'm not sure how efficient this would be, and it does sound like a lot of work to individually back up to each drive even with a software.

2. A single enclosure that holds 2 drives, some of them may have RAID options or a one click mirror button, this is a DAS i guess (there is a chinese one that caught my eye)
3. A NAS (or DAS?) from the usual suspects like synology or qnap, complete with user management & multiple OS support, likely ARM, possible RAID

This is more of a solution, and since you want your data to be safe a RAID 1 is definitely your array. However, keep in mind that if you put 2x 2TB HDDs in a mirror, you'll be able to use only 2 TB of the overall capacity. So basically you will have the access and be able to write to only one of those HDDs. This is because 1 HDD is an identical duplicate of the other, and yes, a RAID 1 will have identical fragmentation on both drives, because the data is identical between the two.

NAS is a practical solution especially if you have multiple computers and other electronic devices with Internet access at home. There are NAS systems that come with a backup software that will let you configure the backup of all your systems if you wish. But then again, you don't want a network-only device.

4. MANY external enclosures & even DASs/NASs say they only support 2TB, 3 if you're lucky, so what's going on with 4!? it sounds like i have to be very careful...

There are enclosures that support bigger capacities, you just need to do your research, I guess. For example, I personally use My Cloud Mirror 6 TB, but this model's storage capacities go up to 12 TB.

5. Software syncing instead of raid, i hope there is software that lets me choose individual files or folders, push them to the drives... as opposed to a single blob of files that forces a sync to the 2 drives + computer i'm using (aka, i want partial sync, not to mirror the drive contents back onto the local machine)

Well, I'm not sure about the syncing but I've used Acronis True Image and it offers both local and cloud backup of your entire system. You can select manually what to backup and how to do it, with different backup schedule options.

6. Of course, which 4TB drives to buy?
I won't be too pushy with the brands since I'm an official representative, but if you plan to use enclosures and to RAID the HDDs inside, then you surely need to be looking for NAS/RAID-designed drives. They have features that regular desktop drives do not offer. They are also tested to withstand the workload in 24/7 environments. Oh, yeah, and they are power-efficient and run cool like the 'green' drives!


Hope this helps you! Keep me posted if you have more questions!
SuperSoph_WD

 

[ne6togadno]

lolz... yes everyone's heard of that

i'll go read about it, but i'm mostly asking about hardware since it's not a problem if i have to manually copy files around with explorer

i also said i do not want to have a network only device

i think you need to make it clear to yourself what exactly you want.
from how you have described the things so far BDR drive for photo/video and external hdd for backups will fit your requirements just fine.

what is importent to you?
to have your data double or even triple backuped or to have fast backup option w/o need to bother with raid array rebuilds etc.
if you want fast backup option just get BDR and 500 gb external hdd for os backups or any half decent hdd enclosure and hdds with proper size and you are done.
if you want your data safely backuped you need raid 1 not half solutions like one button mirror or so.
here is some essential info about raids and data redundancy
http://en.wikipedia.org/wiki/Standard_RAID_levels
http://en.wikipedia.org/wiki/Data_redundancy
and about next gen file systems
http://arstechnica.com/information-...-and-atomic-cows-inside-next-gen-filesystems/

i also said i do not want to have a network only device
what is wrong with network only device?
you have simultaneous access for all users, access permitions, off load torrent dl tasks from pcs to nas etc. also you wont load your router with untipical tasks like file storaging.

6. Of course, which 4TB drives to buy?
I won't be too pushy with the brands since I'm an official representative, but if you plan to use enclosures and to RAID the HDDs inside, then you surely need to be looking for NAS/RAID-designed drives. They have features that regular desktop drives do not offer. They are also tested to withstand the workload in 24/7 environments. Oh, yeah, and they are power-efficient and run cool like the 'green' drives!

while i agree on above with @SuperSoph_WD when you start to chose hdd for your nas keep in mind that nas hdd drives are more expenseive then desktop drives and tend to be 5400-5900rpm (like most of 4tb low cost hdds). 7200 rpm 4tb drives are even more expensive so try to make as exact as possible estimation on how often and with what loads you (and all home net users) will use your nas.
if you need it just for phot/video storage and occasionaly access to content for reveiwing you may use standard desktop hdds and compensate features they miss compared to nas drives with more advance filesys/nas softwere like ztfs or btrfs
or if you edit your files quite often and you expect to have a lot of write operations then you'd be better go with nas grade drives cause they also offer and higher durability.

 

[newtekie1]

Something like this might be worth considering: http://www.newegg.com/Product/Product.aspx?Item=N82E16816111164

You can install it in any machine that has a spare PCI-E x1 slot. And it will give you expandability for the future. If you need to add another 4TB drive later on, just pop it in, and do an ORLM from RAID1 to RAID5.

What drives to use? Either the Seagate or the WD NAS drives is what I'd recommend.

If you want to go with the two seperate drive solution, I'd look at FBackup. It does a good job of backing up data from one drive to another, and is free.

 

[kn00tcn]

if a drive fails in RAID, can the other drive still be accessed normally? can i put it in any other enclosure or even internal sata or is it in some format for the controller/device that isnt supposed to be read directly?

Keep me posted if you have more questions!

if i dont use any bundled software, are mybook & elements the same? i know the kb says there is no guarantee on which physical drive is inside, but will it always be one of the regular green/blue desktop drives that can be bought separately?

http://arstechnica.com/information-...-and-atomic-cows-inside-next-gen-filesystems/
...
what is wrong with network only device?

i remember that ars article, it's making me paranoid again, but this is where 2 separate drives will be better than RAID since if one has a malformed file, the other has that file in tact (unless the other drive entirely fails)

what's wrong is i dont need 24/7 or simultaneous users (at best, 2 might happen to want to read the data at once, but there is no urgency), so what this means is i dont need NAS features like anti vibration that raise the price

burning discs isnt a friendly solution since for archives, there will be steady additions over time, there might be renaming or reorganizing, etc

for now, the backups i'm thinking of are the same usage scenario as how people use dropbox for files, i dont have critical OS setups that need to be instantly restored 1:1 at the moment, a fresh install will do these OSs good anyway

http://www.newegg.com/Product/Product.aspx?Item=N82E16816111164
...
FBackup

that's an interesting DAS, but why do i need pcie? my laptop & desktop have esata already (but other people dont, that's why usb3 seems like a better idea, or something with both since desktop doesnt have usb3)

fbackup also looks interesting since it appears i can choose multiple individual locations rather than a whole drive or a single location like sync apps

 

[W1zzard]

that's an interesting DAS, but why do i need pcie? my laptop & desktop have esata already (but other people dont, that's why usb3 seems like a better idea, or something with both since desktop doesnt have usb3)

the way i understand it, the enclosure will only provide 4 separate links (in one cable) to 4 drives, you need the controller card to create the raid setup. which means you cant just move the esata between systems and expect to see your raid working instantly

what i would do (and have done) is build a dirt cheap sff system (buy old/used cpu+mobo, as long as sata 6gbps), add wifi card, run linux/freenas/windows, done, cheaper than any enclosure you can find, and infinite flexibility

 

[kn00tcn]

what i would do (and have done) is build a dirt cheap sff system (buy old/used cpu+mobo, as long as sata 6gbps), add wifi card, run linux/freenas/windows, done, cheaper than any enclosure you can find, and infinite flexibility

pure external enclosures are like $20, it's just a case with a usb connection, nothing comes close unless old/free parts or things like a raspberry pi

i'm not familiar enough with DAS or NAS prices yet

the irony is i have 3 desktops next to my feet RIGHT NOW, but i dont want networked, they will use too much power, one is IDE the other is SATA1 the third is SATA2

i'm thinking that the flexibility i'm going for is more about portability (hard drive on table size + whoever wants the data can plug it in, or borrow it to go to a friend, etc), & also flexibility regarding accessing/adding/rearranging the data as if it were internal

i do like (love) the idea of open source systems like freenas/owncloud/syncthing rather than limitations of proprietary device software, but dont think i'm at that level yet

 

[Aquinus]

What is your budget, what parts do you need, and what do you already have? You probably could build a cheap NAS using a PCI-E SATA card in an AM1 board. That is most likely what I would do. MDADM RAID-1 should be relatively quick and you can move it to/from any linux box that supports mdadm (which are most).

but i do not want some $600 x86 24/7 server, not yet at least, so what exists? i have a few ideas, let me know if there's a problem or one i missed:

How about a 300 USD X86 server? AM1 can be made to fit in that price range with a cheap SATA card, it would be a close fit though. That's not including the drives I might add.

 

[ne6togadno]

if a drive fails in RAID, can the other drive still be accessed normally? can i put it in any other enclosure or even internal sata or is it in some format for the controller/device that isnt supposed to be read directly?

yes it will be accessable but you have to be carefull with deleting files from it cause if you use prebuild nas nas os will be writen to hdds.

i remember that ars article, it's making me paranoid again, but this is where 2 separate drives will be better than RAID since if one has a malformed file, the other has that file in tact (unless the other drive entirely fails)

what if file that malformed file isnt presented on 2nd dirve?
managing data parity of 2 separate drives will be much harder then managing raid

what's wrong is i dont need 24/7 or simultaneous users (at best, 2 might happen to want to read the data at once, but there is no urgency), so what this means is i dont need NAS features like anti vibration that raise the price

all current nas units can be put to sleep so whenever it isnt in use it will just sleep with 5-9w power consumption (2 drive nas).
and antivibration of is cheapest feature in nas units.
here is my build for freenas. not running yet but i hope soon to have enough time to set it up

it will be usded as media storage for whole family with occasional write and light read access. it will also store backups of user profiles of pcs and mail server if i manage to set it up.
some of the prices has changed sinsce i have ordered parts but total almost exact.

you can make it even cheaper if you follow @Aquinus advise and make it with am1 board. if you use ztfs as os you wont need raid card as soon as mb has raid support

 

[ChevyOwner]

i'm thinking that the flexibility i'm going for is more about portability (hard drive on table size + whoever wants the data can plug it in, or borrow it to go to a friend, etc), & also flexibility regarding accessing/adding/rearranging the data as if it were internal

This presents another problem unless you can trust everyone that may use your portable USB storage device to not drop it, spill things on it, delete data, etc. this will probably ensure data is lost.
The best thing here is to build a NAS with RAID 1. For portability use the USB enclosure, and copy the data to it as needed from the NAS. I also recommend freenas, and it has improved a lot since 0.6.x when I started using it.
With SMB IN Windows you can map a network location to mimic a local HDD. For file management the only difference in accessing/adding/rearranging files is the location of the HDD.

 

[SuperSoph_WD]

if a drive fails in RAID, can the other drive still be accessed normally? can i put it in any other enclosure or even internal sata or is it in some format for the controller/device that isnt supposed to be read directly?

Depends on the RAID configuration itself. But yes, with RAID 1 (mirror) you can still access the data until you put a spare or get a replacement HDD. After you've put the new IDENTICAL drive inside, the reconstruction of the RAID configuration will begin immediately. Once the process of rebuilding is done, you'd have your RAID 1 up and running again!

if i dont use any bundled software, are mybook & elements the same? i know the kb says there is no guarantee on which physical drive is inside, but will it always be one of the regular green/blue desktop drives that can be bought separately?

Well the only difference between those two desktop drives is that the WD My Book incorporates a hardware encryption, whereas the WD Elements doesn't.
Unfortunately, for what's inside the enclosures I cannot give you more information than the one you've already read in our KB.

SuperSoph_WD

 

[newtekie1]

if a drive fails in RAID, can the other drive still be accessed normally? can i put it in any other enclosure or even internal sata or is it in some format for the controller/device that isnt supposed to be read directly?

If one drive fails, you can continue to access the array like nothing happened. That is the point of RAID. When you replace the drive, everything will be rebuilt, and you can continue to access the array during the rebuild process as well(at least on the good RAID controllers you can).

With a RAID1 setup, you can almost always remove the one drive and access it normally, but this isn't recommended. However, once you move to something more complicated like RAID5, you can't access the drive individually like that anymore.

that's an interesting DAS, but why do i need pcie? my laptop & desktop have esata already (but other people dont, that's why usb3 seems like a better idea, or something with both since desktop doesnt have usb3)

fbackup also looks interesting since it appears i can choose multiple individual locations rather than a whole drive or a single location like sync apps

W1z nailed it. The PCI-E controller is what handles the RAID. The enclosure is just a SATA port multiplier. If you plan to share it, then it probably isn't a good idea. Though I would just leave this attached to one computer, and dump data on it from other sources.

 

[kn00tcn]

http://www.canadacomputers.com/product_info.php?cPath=15_213_603&item_id=079619 why hello there $140+tax WD elements.... times 2, incredibly tempting in this period of falling canadian dollars aka rising hardware prices

How about a 300 USD X86 server

so much money for so little potential usage, i am also wondering where to put the case, & bare drives are around $180-200 each

although i can further put it to use to store & upload security camera footage offsite (off camera sd card plus cloud services), minecraft server, firewall...

unless you can trust everyone that may use your portable USB storage device to not drop it, spill things on it, delete data, etc
...
With SMB IN Windows you can map a network location to mimic a local HDD

i mostly trust them, although most of the time only one specific person would even need the data, while i would keep the other drive always to myself

SMB is disabled because reasons, which brings up another issue i ran into... i couldnt find a good windows to windows file transfer tool! they either synced or they wanted to hash every file beforehand wtf

i probably could have done it with teamviewer, but i just manually enabled/disabled windows services & network adapter features, maybe one day i'll make a batch script for it

also related, i am trying to block every LAN client from accessing every other client with DDWRT & a vlan capable router, so.... (well if everything is blocked, then i dont need to hide each client via windows, if i had a NAS, then i can unblock it with SMB enabled)

i do know the joy of having network drives, i had all my computers set up like that for about a decade, but times have changed... especially if one of them is winXP

i dont get why we dont have more on demand options... it's not just about windows network drives, i want to disable google crap on android unless it's the moment want it to run (play store, syncing, etc), but this is another topic

IDENTICAL drive inside

define identical... because the same branded drives have different models as they get revisions such as WD1001FALS to WD1002FAEX, both are 1TB blacks with a varying amount of platters... so what happens if these 2 are put in RAID?

what happens when a model (FALS for example) isnt in production anymore, i have to move to a new pair instead, leaving me with a spare drive?

 

[Solaris17]

get 2 WD REDS get a NAS box. http://www.newegg.com/Product/Produ...&cm_sp=&AID=10440897&PID=1796839&SID=72841603

run it and good.


our shopped used the box above for a bit simple interface RAID1 capable. no problems. it will do exactly what you need it too.

 

[kn00tcn]

get 2 WD REDS get a NAS box. http://www.newegg.com/Product/Produ...&cm_sp=&AID=10440897&PID=1796839&SID=72841603
.

too bad that sale isnt on newegg.ca, $200+ with $200 for each RED

 

[Solaris17]

too bad that sale isnt on newegg.ca, $200+ with $200 for each RED

you asked.

what do you want a $45 nas box with some $56WD blues? No one is going to recommend that.

 

[newtekie1]

define identical... because the same branded drives have different models as they get revisions such as WD1001FALS to WD1002FAEX, both are 1TB blacks with a varying amount of platters... so what happens if these 2 are put in RAID?

what happens when a model (FALS for example) isnt in production anymore, i have to move to a new pair instead, leaving me with a spare drive?

In my experience the drives just have to be close to work well together in RAID. So if a drive dies in the future just replace it with a drive as close as possible to the other.

 

[kn00tcn]

WD elements doesnt read SMART values??

what do you want a $45 nas box with some $56WD blues?

that box is $126 on .com $200 on .ca, why do you think i said the word 'sale'?
red 4tb is $167 on .com $200 on .ca

 

[Uplink10]

If a drive fails in WD elements you can`t replace it by yourself because you will void warranty. The same thing goes for other DAS enclosures with a drive inside, which usually do not have a space where you can take out a drive and then put one back in. That is why I rather buy internal drive and external enclosure separetly because I can put it (HDD) into other devices if I want.

pure external enclosures are like $20

If you will in the future buy more drives it might be a good idea to buy 4 bay NAS or PC to store drives in. I have 3 external enclosures and now I think it would have been a better idea to buy a PC and be done with it. It costs a little more but you get more functionality and you can use the PC for some server tasks.

On your place I would buy:
-PC with:
-AM1 motherboard with at least 4 sata ports
-two 3TB drives (since 3TB drives has lowest price/GB), preferably Toshiba DT01ACA300, which has better reliability than Seagate 7200.14, there is no point buying enterprise drives since you will have RAID 1

-External 2.5" HDD/Internal 2.5" HDD+enclosure for transferring files betwen devices

 

[Athlon2K15]

Just get a Netgear Ready NAS to fit your storage requirements. It's all you will ever need.

 

[newtekie1]

WD elements doesnt read SMART values??

AFAIK, anything USB won't report SMART values.

that box is $126 on .com $200 on .ca, why do you think i said the word 'sale'?
red 4tb is $167 on .com $200 on .ca

I think his point was you sound like you just want to duct tape and bubblegum something together for as cheap as possible instead of doing it correctly or at least somewhat correctly. If you want to do what you are asking correctly you are going to have to spend some money, there is no getting around it.

 

[Uplink10]

AFAIK, anything USB won't report SMART values.

It will report, I connected USB HDD and AIDA 64 reported S.M.A.R.T. values.

 

[kn00tcn]

two 3TB drives (since 3TB drives has lowest price/GB), preferably Toshiba DT01ACA300, which has better reliability than Seagate 7200.14

which seagate? 7200.14 is ST3000DM001? what about ST3000VN000? i think there might even be a third one

i dont need speed (also for comparison, i happen to have an ST2000DL003 lightly used for a couple years now)

would love to try a hitachi after reading various reports, but expensive...

should i post every hard drive i've ever used or leave that for a 'post your hard drive history with statistics' thread?

I think his point was you sound like you just want to duct tape and bubblegum something together for as cheap as possible instead of doing it correctly or at least somewhat correctly. If you want to do what you are asking correctly you are going to have to spend some money, there is no getting around it.

that's the thing... it's not so urgent that i need a quality system just yet, especially when most of the time i will be the only user & i wouldnt even need access daily, plus duct tape is fun

but after some time i might end up needing to set up a cloud, so i have to weigh all the options & all the costs

It will report, I connected USB HDD and AIDA 64 reported S.M.A.R.T. values.

guess i'll find out... i kind of ordered those 4TB elements... of course they're only 1 year warranty, so i'm free to crack open the case after a year to build a NAS or even sooner if the drives seem fine after some months, the cost difference is quite large vs bare drives + bays so BOTH have to fail in less than a year for it to be a loss

i guess i'll update the thread when i do something beyond copying files in explorer?

(it's not the point of no return yet, still have to pick them up & be charged)