Best way to set up a huge 10~20TB for camera surveillance?

[n-ster]

I will be building a computer soon. This computer's only purpose will be camera surveillance which will be running 24/7. We need a lot of storage, at least 10TB if not more, as this will have 4x 3MP cameras (2048x1536@15fps) and 12x 2MP cameras (1080p@30fps), which would mean 10TB should cover a month (300GB/day on average)

What type of drives should I use to have the best endurance? Reliability isn't crucial, but bang/$$ is. Recommendations for very long lasting PSUs that work great 24/7 would be good too (should I be going into server PSUs?). Would Ivy Bridge do or is X79 better?

This will be running Windows 7, probably ultimate 64 bit of course, is there anything I should be worried about when building a comp with such a huge amount of disk space?

Can I just use the hard drives separately instead of RAIDing?

[mudkip]

Originally Posted by n-ster

This computer's only purpose will be camera surveillance which will be running 24/7.

Reliability isn't crucial

[W1zzard]

Originally Posted by n-ster

a huge amount of disk space?

10 or 20 TB is far from huge for today's standards.

google: unraid
if you can use freebsd or solaris: ZFS
if you have windows server, look into DFS Distributed File System

what sustained read/write bandwidth is needed for your cams ? i assume they all write to the same storage?

what about transcoding the video you get?

[CJCerny]

Seems to me like the reliabililty of your hard drives and PSU are the least of your worries. You main worry needs to be how you are going to get all that data from those cameras to this PC. If they are all IP cameras, you are going to need to make sure that you can handle all the network traffic that they are creating. If they are USB cameras, you're going to have a hard time building a PC that doesn't drop frames with all that input via USB.

[ShiBDiB]

main reason I could see to RAID would be for a backup. Drives constantly writing like these will be, is gonna put them under alot of stress.

Might be better in the long run to look to a professional company for something of this scale.

[v12dock]

iSCSI SAN

[AnomalouS]

Must you keep the captured video for a month at a time? why not use first in first out method.
In the medical world there are capture devices that do just the FIFO, xxxGB drive that purges data as needed.

[n-ster]

Originally Posted by mudkip

Haha, it is good to have but if we loose 1 HDD, that would mean about a week and we can handle that every couple years or something. I'm not going to pay 3~4x the price for 1.5x~2x the longevity for the HDDs is what I meant

Originally Posted by W1zzard

10 or 20 TB is far from huge for today's standards.

google: unraid
if you can use freebsd or solaris: ZFS
if you have windows server, look into DFS Distributed File System

what sustained read/write bandwidth is needed for your cams ? i assume they all write to the same storage?

what about transcoding the video you get?

It's huge for me I'll look into all that in a few minutes, thanks.

The cameras are IP and wired with ethernet cables. The max they would use is 8Mbps which is 1MB/s per camera, so about 16MB/s I'm guessing is the peak rate. The coding of the video is already MJPEG or H.264 and the camera's processors handle that. It'll be coming through 1 Ethernet port

The way it our current system works: say there is 5x 2TB HDDs. It will write on the first drive, once that is full it will start on the 2nd drive and so on, once all 5 are done it will start overwriting the first drive again.

Originally Posted by CJCerny

Seems to me like the reliabililty of your hard drives and PSU are the least of your worries. You main worry needs to be how you are going to get all that data from those cameras to this PC. If they are all IP cameras, you are going to need to make sure that you can handle all the network traffic that they are creating. If they are USB cameras, you're going to have a hard time building a PC that doesn't drop frames with all that input via USB.

They are going to be wired with Ethernet cables, which will go to a 24 port Ethernet switch thing (each camera will have their own 100 Mbps cable/port) and exit through 1x 1Gbps Ethernet cable.

Originally Posted by AnomalouS

Must you keep the captured video for a month at a time? why not use first in first out method.
In the medical world there are capture devices that do just the FIFO, xxxGB drive that purges data as needed.

I believe that is how we are doing it. At any time we look at the data, the oldest we will be able to find is 1 month old. Our old system writes on each HDD until they are all full then start overwriting on the first drive etc.

Originally Posted by ShiBDiB

main reason I could see to RAID would be for a backup. Drives constantly writing like these will be, is gonna put them under alot of stress.

Might be better in the long run to look to a professional company for something of this scale.

We did in the past and they charge you so much, and they end up doing it worse than I would have. They charge 3800$ for a i5-2400 with the IGP and some cheap chipset and 8GB of RAM and 4TB with inadequate cooling and a very average PSU with 1 year warranty. They user WD caviar black HDDs, nothing fancy.

Originally Posted by v12dock

iSCSI SAN

I'll have to google for specifics it looks interesting

[jsfitz54]

Not all months are equal.

Would a 45 day first in first out method be better?

If for some reason you need legal copies, in a 30 day time frame, you would have extra days to recover the necessary footage before overwriting occured.

[n-ster]

the 30 day thing is just an estimate, if 1 month only takes 6.6TB than I can look back about a month and a half behind etc. After the first time the 10tb is filled, all it will do is overwrite the oldest videos.

I'm wondering if I should just do one of those RAID 5 boxes or setup a RAID 5 inside the computer, it would be more reliable and faster. I'd probably need a RAID card though? unRAID looks good too

As for the PSU, am I best going with a Platinum PSU or looking into server type PSUs? Remember this will be on 24/7 365 days

I'll have to use Windows 7 Ultimate, so linux distros etc are out of the question.

can anyone help me understand iSCSI SAN better? What do I need etc.

[Kreij]

How flexible is the software for the surveillance system?

If it will allow you to stream the video directly to multiple NAS devices, I would make a large RAID5 NAS device with hotswap disks. You could then add more NAS devices if the storage requirements grow.
You could then slap any old HDD (or a pair in RAID1 for redundancy) in the computer for managing the setup and viewing the video.

It would not be the cheapest way to go (good/fast NAS devices aren't cheap), but it would add some robustness and future proofing to the configuration.

If the software will not allow direct streaming to a NAS, you would put in a pair of gigabit NICs (one incoming, one outgoing) and let the computer forward the data to a single large NAS device.

Just some some thoughts.

[yogurt_21]

Originally Posted by n-ster

I will be building a computer soon. This computer's only purpose will be camera surveillance which will be running 24/7. We need a lot of storage, at least 10TB if not more, as this will have 4x 3MP cameras (2048x1536@15fps) and 12x 2MP cameras (1080p@30fps), which would mean 10TB should cover a month (300GB/day on average)

What type of drives should I use to have the best endurance? Reliability isn't crucial, but bang/$$ is. Recommendations for very long lasting PSUs that work great 24/7 would be good too (should I be going into server PSUs?). Would Ivy Bridge do or is X79 better?

This will be running Windows 7, probably ultimate 64 bit of course, is there anything I should be worried about when building a comp with such a huge amount of disk space?

Can I just use the hard drives separately instead of RAIDing?

based on the amount of data it seems hard disk performance is far more important cpu/mobo/memory performance.

personally i'd look into some cheaper perhaps older spec server hardware for the longevity. Then throw most of the money into hard drives.

specifically if you could find a used Dell PowerEdge 2900 it would have everything you would need while being a bit slow by today's standards in cpu/memory.

The chasis supports 8 hotswap disks and a couple more internal as well as a few 5.25's fro whatever your i/o needs are.

it has dual psu's for redundancy and should be very reliable (but again not the fastest thing in the world)

The only upgrade needed would likely be 2 tb drives for the hot swaps and a gpu upgrade for any playback needs. The raid controllers should still be fine.

They currently go for 600-1000$ on the used market, compared with psu/tower that supports enough hard disks, mobo/memory/cpu, and a raid controller it would be hard to beat and Way, Way, Way more reliable than desktop hardware. Remember desktop hardware is only designed to be on 8 hours a day and typically 3 years longevity (longer for cpu's/memory on average though) but that means in 1 year under 24/7 use you will have used up the full life of most of the hardware.

[n-ster]

Originally Posted by yogurt_21

based on the amount of data it seems hard disk performance is far more important cpu/mobo/memory performance.

personally i'd look into some cheaper perhaps older spec server hardware for the longevity. Then throw most of the money into hard drives.

specifically if you could find a used Dell PowerEdge 2900 it would have everything you would need while being a bit slow by today's standards in cpu/memory.

The chasis supports 8 hotswap disks and a couple more internal as well as a few 5.25's fro whatever your i/o needs are.

it has dual psu's for redundancy and should be very reliable (but again not the fastest thing in the world)

The only upgrade needed would likely be 2 tb drives for the hot swaps and a gpu upgrade for any playback needs. The raid controllers should still be fine.

They currently go for 600-1000$ on the used market, compared with psu/tower that supports enough hard disks, mobo/memory/cpu, and a raid controller it would be hard to beat and Way, Way, Way more reliable than desktop hardware. Remember desktop hardware is only designed to be on 8 hours a day and typically 3 years longevity (longer for cpu's/memory on average though) but that means in 1 year under 24/7 use you will have used up the full life of most of the hardware.

Used stuff is pretty much out of the question. If I have to assemble a server, that's what I'll do. But does, say, a Xeon CPU really outlive a standard i7? My plan was just to wait until Ivy Bridge and build them that with a server or platinum PSU and good hard drives and plenty of fans (so that if a few fans fail it won't change much). I was also just going to go with a big full tower case. Should I do it differently?

[W1zzard]

Originally Posted by n-ster

But does, say, a Xeon CPU really outlive a standard i7?

it's the same silicon, made in the same production process, in a different package

dont worry about ivy bridge. dont worry about good hard drives.

get something like ASRock P67 Extreme6 (because it has 10 sata ports or so). add more sata ports using cheap controller cards. 2500k (K because it's only 5 bucks more expensive)

fill up with the cheapest hdds you can find (wd caviar green 2.5 tb). get some raid config setup so you dont lose your data

windows 8 will have storage pools which should make your life much easier

[n-ster]

Sounds great except for the WD Green HDDs don't have the greatest reputation... Are the problems fixed with them?

Also would the stand intel chipset be OK for the 10~20TB RAID or should I be going with a RAID card? And 10 SATA ports should be enough, so I wouldn't need the controller cards?

I'm going Ivy Bridge as that is when I'll be building this anyways... The earliest I'll be building this is after my Finals, which would mean May, which is when IB is coming out right? If it isn't out by the time I actually build this, I'll go 2500K as you suggest, but I would assume the IB will have lower consumption, which would likely help with longevity and will save a few bucks at 24/7 use.

[W1zzard]

Originally Posted by n-ster

Sounds great except for the WD Green HDDs don't have the greatest reputation... Are the problems fixed with them?

you should really have some sort of raid to prevent data loss in case of disk failure.

once you have that it doesn't matter whether you use expensive disks that fail once in 5 years or cheap ones that cost half as much and fail in 2 years and get replaced for free thanks to RMA

out of those 10 ports only 4 or 6 are on intel. makes no difference anyway. personally i've had better experience with software raid than with hardware raid

[yogurt_21]

Originally Posted by n-ster

Used stuff is pretty much out of the question. If I have to assemble a server, that's what I'll do. But does, say, a Xeon CPU really outlive a standard i7? My plan was just to wait until Ivy Bridge and build them that with a server or platinum PSU and good hard drives and plenty of fans (so that if a few fans fail it won't change much). I was also just going to go with a big full tower case. Should I do it differently?

like I said cpus and memory are typically the exception. psu, mobo, etc not so much. But W1z is right xeon vs regular is essentially the same cpu. When it comes to failing parts for the 2000 servers I'm responsible for (they sit behind cleint networks and house media content locally rather than sucking bandwidth) 1. hard disks 2. motherboards 3. raid controllers 4.memory. I've yet to have a cpu fail on me since Ive been in this position (4 years) that includes pentium grade up to highend xeon.

have had plenty of hard disks and to a slightly lesser extent motherboards. Most the boards that have fried were on our lowend models using desktop hardware. I have only had a couple server boards go out over the course of 4 years.

the only issue with going brand new is that case alone will net you half of an old server cost. You need something that houses enough hard disks and will still be effiencient for cable management and cooling.

whatever you do don't cheap out on the raid controller, you'll quickly find managing alot of data on a poor controller = hell on earth.

[v12dock]

Originally Posted by W1zzard

you should really have some sort of raid to prevent data loss in case of disk failure.

once you have that it doesn't matter whether you use expensive disks that fail once in 5 years or cheap ones that cost half as much and fail in 2 years and get replaced for free thanks to RMA

out of those 10 ports only 4 or 6 are on intel. makes no difference anyway. personally i've had better experience with software raid than with hardware raid

Hardware RAID is much more scalable and it will allow a lot more option for setting up RAIDs

Also are you dealing with raw data or will there be any encoding?

[phanbuey]

some sort of expandable DAS is what you need.

we use one of these at our imaging center and it sounds like it would work fine for you:

http://www.high-rely.com/hr_66/products/raidframe5bay/

[n-ster]

Motherboards actually have that high of a failure rate? I didn't know. But server or Workstation mobos seem really expensive comparatively to standard mobos... What do I have to look for to get a lasting MB?

For the case, can't a standard large Full Tower case do that job?

Memory is the least of my worries as I have plenty and if some craps out ill just pop in some other sticks pretty easily. PSU I'll be going something like the LZP-550W from Enermax

The data will basically be H.264 video files and a few MJPEG video files

[phanbuey]

Originally Posted by n-ster

Motherboards actually have that high of a failure rate? I didn't know. But server or Workstation mobos seem really expensive comparatively to standard mobos... What do I have to look for to get a lasting MB?

For the case, can't a standard large Full Tower case do that job?

Memory is the least of my worries as I have plenty and if some craps out ill just pop in some other sticks pretty easily. PSU I'll be going something like the LZP-550W from Enermax

If you really need that much storage, DO NOT tie it down to one machine. You want the storage dtachable so that if the server goes you can restore image to another server and hook it up to the data.

A single workstation is usually not a good way to go (for something like this which is essentially a video server). You're better off building a super cheap server with detachable storage. Just my 2c. Also detachable storage usually has its own monitoring and alerting tools, which will make your life alot easier.

[n-ster]

So basically a RAID 5 box?

So that 1 ethernet that carries the video files goes to the computer, and I use the RAID 5 box as the HDD where it saves the video files and be done with it right?

[phanbuey]

Originally Posted by n-ster

So basically a RAID 5 box?

That is the cheapest storage we trust. I only picked that one because we use that same one in one of our diagnostic centers for Radiology images, and it works pretty good. It basically serves the exact same purpose which you need - cheap, redundant and lots of it.

^yes basically exactly right - the raidpack attaches through eSATA and the OS and Computer see it as a gigantic hard drive.

This one is even cheaper (although you will need software raid):

http://www.high-rely.com/hr_66/products/hr-rack-mount/

[v12dock]

Could possibly setup a RAID 10 either way I would look into getting a nice raid controller

What kind of hard drives are you looking at?

[n-ster]

I was going to get WD Blacks but I'm not sure anymore lol