1. Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.

Setting up a file server

Discussion in 'Networking & Security' started by [Ion], May 31, 2013.

  1. [Ion]

    [Ion] WCG Team Assistant

    Joined:
    Sep 15, 2009
    Messages:
    11,786 (6.65/day)
    Thanks Received:
    10,889
    Location:
    North Carolina, United States
    I've realized that it's impractical and insecure to keep my big files (movies, music, TV shows, pictures) on a single 1TB drive. Consequently, I'm looking to build a file server. Currently, I have a pair of 1TB Caviar Green drives. I'm determined to have either a RAID1 or RAID5 array so I have a degree of redundancy in case something goes south. I've also already managed to fill up about 2/3 of my 1TB drive, so I'm thinking of adding a third drive for RAID5 so that I can expand a bit more.

    My most basic requirement is a system with at least 100 megabit ethernet (ideally gigabit) that can redundantly store at least 1TB of files and be accessible (read & write) from a Windows PC. RAID5 is not strictly required but is highly desired.

    Less pressing features (but still ones I would like) are a web/interface or web server for it, the ability to run a Bittorrent client, the ability to have multiple password-protected accounts so that I can give friends access at school, and a FTP server. Something that I could configure easily is a bonus.

    If I can run a VPN through this system or run WCG on it, those are added benefits but not required. I can get any version of Windows for free from school--not an issue there ;)

    In terms of hardware, I'm thinking of using a Penryn Pentium Dual Core on probably an Asus G31 board w/ ~2GB+ RAM and an 80GB boot drive.

    Suggestions? :)
    Crunching for Team TPU
  2. shovenose

    shovenose

    Joined:
    Jan 11, 2013
    Messages:
    795 (1.42/day)
    Thanks Received:
    132
    I would highly recommend RAID10 with four drives... WD Red 3TB's would do you perfectly well for this purpose. Pentium Dual Core will be plenty. Would recommend 4GB RAM just for the heck of it and it's cheap.
    [Ion] says thanks.
  3. MxPhenom 216

    MxPhenom 216 Corsair Fanboy

    Joined:
    Aug 31, 2010
    Messages:
    9,617 (6.77/day)
    Thanks Received:
    2,043
    Location:
    Seattle, WA
    Since you only running two drives. Just run a RAID1. I rebuilt my old dell desktop into a file server to keep all my GoPro videos and edited videos on. Running 2 2TB WD Reds in Raid1. perfect little file server for what I need.
    [Ion] says thanks.
  4. [Ion]

    [Ion] WCG Team Assistant

    Joined:
    Sep 15, 2009
    Messages:
    11,786 (6.65/day)
    Thanks Received:
    10,889
    Location:
    North Carolina, United States
    The problem is that's expensive---I already have the two 1TB drives and I can get a third for about $30. And I think that 2TB of space is plenty for the foreseeable future.
    As I said, I'd really like to get a third for RAID5 and double my capacity. For $30 I think it's worth it.

    And I can certainly go 4GB RAM--that's cheap. I have the RAM to do that sitting on my desk :)
    Crunching for Team TPU
  5. shovenose

    shovenose

    Joined:
    Jan 11, 2013
    Messages:
    795 (1.42/day)
    Thanks Received:
    132
    As an administrator who manages tons of servers on a daily basis (enterprise grade servers in a real datacenter) I absolutely hate RAID5. Then again, that's on multi-tenant servers running shared hosting/VPS... we don't have a single dedicated server customer running RAID5... we set up dedicated servers for customers that order them, with 1 or 10. For a personal file server I am sure RAID5 would be fine, though :) I'd be worried about filling up all your space soon it you went with the 1TB drives, since you mentioned that, that's why I suggested 4x 2TB. But understandably you want to save money, and ultimately only you know how much storage you need, so I will stand down :)
  6. 1freedude

    1freedude

    Joined:
    Nov 16, 2007
    Messages:
    557 (0.23/day)
    Thanks Received:
    118
    As far as WebUI, I use HFS. HTTP File Server. Gets me all my stuff anywhere. Really handy with Android OS.
  7. Aquinus

    Aquinus Resident Wat-man

    Joined:
    Jan 28, 2012
    Messages:
    5,914 (6.52/day)
    Thanks Received:
    1,903
    Location:
    Concord, NH
    First of all, running multiple VMs on a single box requires potentially significant write speed so that VMs could benefit from RAID-10. Not for this case though, most of what the RAID-5 will be doing is reading and holding on to data. Also, I don't know how you can make such a recommendation being a "system admin who manages tons of servers on a daily basis" because I manage private servers that aren't shared and RAID-5 and RAID-6 are the best RAID modes if you have a lot of mission critical data where losing data is unacceptable and is paramount over all else.

    Also a good admin will know how to minimize cost and adapt a server to the needs of the person or institutions so I'm going to call shenanigans.

    All in all, no one should never consider RAID-10 unless they absolutely need it, otherwise it's a waste of capacity. We're also talking about storing files, not VM images of entire OSs.

    My take is: Get the extra disk and go RAID-5. At least that way you will get the added benefit of data being striped across 3 disks as opposed to only duplicating data on 2, so your read speeds should be improved and it will only require you to get one more drive, so I like that idea. Write speeds should even be better despite having to write parity because now you're stripping data as opposed to duplicating it.

    I have RAID-5 and it has saved me from headache multiple times. I shun all nay Sayers of RAID-5 and 6. For strictly a file server, capacity and redundancy is much more important than performance IMHO.
  8. Mindweaver

    Mindweaver Moderato®™ Staff Member

    Joined:
    Apr 16, 2009
    Messages:
    5,143 (2.67/day)
    Thanks Received:
    2,650
    Location:
    Statesville, NC
    I agree with Aquinus and say go with a RAID-5 setup, but only if you are going to use a hardware RAID with a BBU, and not Software (I think Aquinus and shovenose would agree). If you plan to use software then do yourself a favor, and setup a RAID1 array. I would never setup a RAID10 array for home use mainly due to cost. Then again I would never setup a RAID-5 with out a backup drive on hand either. Since we are pulling our junk out, and measuring then I've been a CIO/IT Manager for over 16 years. :p
    Last edited: May 31, 2013
    [Ion] says thanks.
    Crunching for Team TPU
  9. [Ion]

    [Ion] WCG Team Assistant

    Joined:
    Sep 15, 2009
    Messages:
    11,786 (6.65/day)
    Thanks Received:
    10,889
    Location:
    North Carolina, United States
    I'll have to check and see what sort of board I'll be putting it on, but I'd say that there's a good chance it would have to be software RAID. I don't think that the G31 board I'm planning on supports HW Raid5 (not sure if it supports hardware RAID at all). Why would software RAID5 be a problem? Wouldn't a 2.5GHz+ Penryn CPU be enough to handle that? I definitely think that RAID10 is more expensive than I need ATM, and would be a poor use of resources.

    I suppose I'm mostly wondering what operating system should I use? A Windows Server variant, some mainstream Linux OS (Ubuntu/etc) or a specialized file server Linux distro? Would I want to use my 80GB disk for boot and then have a RAID array for data?

    I think that a 3x1TB RAID5 array would be plenty--that would have about 3x as much storage space as I'm currently using, so there would be considerable room to grow.
    Mindweaver says thanks.
    Crunching for Team TPU
  10. Mindweaver

    Mindweaver Moderato®™ Staff Member

    Joined:
    Apr 16, 2009
    Messages:
    5,143 (2.67/day)
    Thanks Received:
    2,650
    Location:
    Statesville, NC
    With it being just a file server it should be fine, but you'll lose speed with out a BBU. Using a BBU you'll be able to use write-back meaning if power is lost all the write requests are put into cache, and when power is returned it picks up where it powered off using the requests stored in cache. but with out a BBU you'll use write-through which means everything wrote won't go to cache and it will be wrote as soon as it gets the request. So, it's slower compared to using a BBU. :toast:

    Now what OS to use.. I'd probably go with CentOS or Debian... But any should do :toast:
    Crunching for Team TPU
  11. [Ion]

    [Ion] WCG Team Assistant

    Joined:
    Sep 15, 2009
    Messages:
    11,786 (6.65/day)
    Thanks Received:
    10,889
    Location:
    North Carolina, United States
    My only real requirement in terms of speed is the ability to play a HD movie or TV show from the system over the network.

    If I went with Linux I'd probably do Debian as I'm more familiar with it---how challenging is it to set up the software RAID?

    I'm also investigating WS.
    Crunching for Team TPU
  12. Mindweaver

    Mindweaver Moderato®™ Staff Member

    Joined:
    Apr 16, 2009
    Messages:
    5,143 (2.67/day)
    Thanks Received:
    2,650
    Location:
    Statesville, NC
    Set it up and play around with it to see how you like it. If it meets your needs then go with it, but I would play around with it to get comfortable. Just remember to well document your drives and Raid Array setup, and label them. Also, remember that it will take awhile to rebuild the software array. Good luck! :toast:
    Crunching for Team TPU
  13. IggSter

    IggSter

    Joined:
    Aug 24, 2007
    Messages:
    443 (0.18/day)
    Thanks Received:
    127
    Location:
    BY-S36
    A couple of things to be mindful of:

    If you use motherboard raid it will be almost impossible to move your hard drives with data intact to a new platform. If will also be highly unliky that you will be able to easily migrate/expand your raid in the future. I would therefore recommend a low cost raid card (Adaptec/LSI/etc) which will allow you to move your data from platform to platform.

    It would also be worth checking your WD drives for compatibility - the green drives are terrible in a lot of raid setups (a lot of NAS manufacturers have WD green drives on the do not use list).

    With regard FTP: I use Filezilla server...it's free, secure (sFTP/TLS) and works.


    Ninja edit: For a web server I would suggest Uniform Server (Afaik it is the only WAMP package that is in production mode out of the wrapper). Another really big bonus is - portability, you can run it without any install.
  14. Aquinus

    Aquinus Resident Wat-man

    Joined:
    Jan 28, 2012
    Messages:
    5,914 (6.52/day)
    Thanks Received:
    1,903
    Location:
    Concord, NH
    That depends on how he does it. Intel RST(e) will let you move an array between 6-series and 7-series boards. For example, if I put my RAID-5 in a z77 board, I'm pretty sure once SATA is configured for RAID, it would read the configuration from the drives. This is really only true if you're making a big change like Intel to AMD or a 900-series intel chipset to a 7-series chipset or something like that. The same goes for hardware raid cards as well, like cards will still detect the RAID. I also happen to know that RSTe handles RAID fairly well and that expanding RAID-5 is fairly easy and you can even do it while the system is up iirc.

    Because of lack of TLER, but that's an issue with newer WD blacks too, but I've run into practically no issues with them ever since I disabled patrol read.

    I don't know, I recommend nginx because it is event based, very responsive, uses very little CPU, and has an incredibly small memory footprint. Nginx also excels at static content which is what the OP is looking at serving up. That's just me though.

    Speak for yourself, my 3x1TB drives rebuild in just a few hours or so using X79. Back when I was using an nForce4 for RAID, it would take twice as long so it really depends on how you do implement said RAID.

    If you can go for it, a dedicated RAID card with a BBU is ideal, but fake raid isn't too bad depending on the board. You can enable write-caching with RST/RSTe but you'll really want the server to be connected to a UPS. Also RAID-5 performance isn't all that bad, more often than not it can saturate 1Gbps which is all you really need, even using software or some form of FakeRAID.

    I wouldn't say a BBU is required for RAID-5. If you don't enable write caching then there is no point of having it in the first place. This sounds like it will be mostly storage so write speed doesn't have to be perfect, just good enough. Reliability seems to be paramount and a BBU won't offer that unless write caching is enabled. With all of this said, I would opt for a BBU if it's feasible though but I wouldn't bend over backwards for it.
    Last edited: May 31, 2013
    IggSter says thanks.
  15. Mindweaver

    Mindweaver Moderato®™ Staff Member

    Joined:
    Apr 16, 2009
    Messages:
    5,143 (2.67/day)
    Thanks Received:
    2,650
    Location:
    Statesville, NC
    I'm glad you agree with me Aquinus and just to add I never said you had to use a BBU.. I even explained the difference. Also, remember that to some a few hours is awhile. ;) I should have stated that a Software RAID rebuild will be slower than a Hardware Raid rebuild. :p
    Aquinus says thanks.
    Crunching for Team TPU
  16. Aquinus

    Aquinus Resident Wat-man

    Joined:
    Jan 28, 2012
    Messages:
    5,914 (6.52/day)
    Thanks Received:
    1,903
    Location:
    Concord, NH
    I know, I'm just making that clear. BBUs are overrated IMHO, but that's my own personal opinion. It is better to use one if you can.

    It really depends on the controller. I have an LSI controller on a server at work that took several hours to rebuild a 500Gb drive on RAID-6 and those are Constellation ES drives, by no means slow or unreliable. I think the simple fact that you can use the RAID array while it's degraded makes the time to rebuild a much smaller factor. For my RAID-5 I have mostly music, video, pictures, files, and a couple games on it as well as a couple VM images. Generally speaking, even when it's degraded and rebuilding, the performance will still let me do everything I normally do without any degradation of my experience and this is on the same machine as the RAID and that experience will remain the same between RAID 1 and RAID 5. So from a user perspective, the real consideration is capacity, which is why I would opt for RAID 5.

    Ion: Does the board have any options for fake raid? (Setting SATA to RAID mode.) I really would discourage software RAID if at all possible. I think that will be a mutual consensus.
    Mindweaver says thanks.
  17. newtekie1

    newtekie1 Semi-Retired Folder

    Joined:
    Nov 22, 2005
    Messages:
    19,695 (6.22/day)
    Thanks Received:
    5,858
    If you have a UPS, and you should, a BBU is pointless. You can still safely enable Write-Back cache with a UPS.

    I'd avoid software RAID as well. I'd buy a dedicated RAID card, an inexpensive one though. Something like this: HighPoint RocketRAID 640L Controller Card - Neweg...

    Before anyone says it, I know this isn't a true hardware RAID card, it is actually firmware RAID but it is a hell of a lot better than software RAID. The CPU still does the bulk of the work, the RAID card just issues the commands and houses the RAID configuration settings, but it is still way better than doing software RAID in the OS.

    Now, speaking of OS. I see no reason that any of what you want to do can't be done with Windows 7 Pro, so that is what I'd go with. Remote admin is provided by remote desktop, Apache gives you a webserver, filezilla gives you a FTP server. That is really all you need.
    Aquinus says thanks.
    Crunching for Team TPU 50 Million points folded for TPU
  18. Aquinus

    Aquinus Resident Wat-man

    Joined:
    Jan 28, 2012
    Messages:
    5,914 (6.52/day)
    Thanks Received:
    1,903
    Location:
    Concord, NH
    I just would hate recommending using up a Windows 7 key for just this. Debian is more than enough for this and you don't need to burn up a 7 key to do it. Worse comes to worse, if Debian doesn't feel like the right fit he can fall back on Windows. I would strongly recommend giving a *nix distro a try first though.
  19. newtekie1

    newtekie1 Semi-Retired Folder

    Joined:
    Nov 22, 2005
    Messages:
    19,695 (6.22/day)
    Thanks Received:
    5,858
    Why not? He says he gets them for free from his school, so why not use it?
    Crunching for Team TPU 50 Million points folded for TPU
  20. [Ion]

    [Ion] WCG Team Assistant

    Joined:
    Sep 15, 2009
    Messages:
    11,786 (6.65/day)
    Thanks Received:
    10,889
    Location:
    North Carolina, United States
    If it takes a few hours to build that's fine by me. Realistically, none of the stuff that is going to be stored on here is mission-critical anyways--all of my important school and work files are continually backed up. This is all going to be stuff that is just for fun and could always be re-downloaded/re-ripped if it absolutely came to it.

    I expect to be using a Gigabyte P35-UD3R, which has the Intel ICH9R South Bridge that supports RAID 0/1/5/10. However, I don't actually have it yet, so I can't guarantee that's what I'll be using.
    I would love to be able to stream .mkv files through a web browser---with a sufficiently fast internet connection and VLC on the target system, would Apache let me do that?

    Would that RAID card that you linked be any better than using the ICH9R to do RAID5? And how key is a UPS? The power at school has proven quite stable so far...
    I do get Windows licenses for free through the engineering department here. I am open to trying Debian though.
    Aquinus says thanks.
    Crunching for Team TPU
  21. newtekie1

    newtekie1 Semi-Retired Folder

    Joined:
    Nov 22, 2005
    Messages:
    19,695 (6.22/day)
    Thanks Received:
    5,858
    Apache wouldn't allow you to do that directly. Apache is just a web server, you have to build the website yourself. Though if that is all you want the webserver for forget about apache. Use something like Twonky Media Server, not free but very cheap. I use that to stream media through a web browser, no need for VLC at all. Though you'll need a pretty beefy internet connection to stream HD mkv files. For those it would probably just be better to download the file via FTP before playing. If they are SD it shouldn't be as much of an issue, but you'll still need a decent connection. Obviously being on a school connection you probably have a decent connection, but you'll need 20Mb/s+ upload to stream HD content, and that is just one stream.

    Other than the SATA 6Gb/s ports there really wouldn't be much advantage over the ICH9R, they are both "firmware" RAID, though I prefer an actual card over using the onboard simply because it is much easier to move to another machine if the motherboard has an issue.

    A UPS isn't necessary, especially if you don't use Write-Back cache, but I'd still recommend one. And with basic ones starting at $40 it only makes sense to have one. Though if you have to skip it initially due to financial constraints it isn't going to kill you, but I'd pick one up eventually.
    Crunching for Team TPU 50 Million points folded for TPU
  22. 1freedude

    1freedude

    Joined:
    Nov 16, 2007
    Messages:
    557 (0.23/day)
    Thanks Received:
    118
    http://www.rejetto.com/hfs/

    So easy...I use it to stream to my phone, my wife's laptop, my tablet (1080p). Open browser, copy link ( it is set up like a FTP), paste into VLC, or whatever, enjoy.

    To get to it from outside my network, No-IP handles the routing. All free. I just checked, HFS will run on WINE.
  23. Aquinus

    Aquinus Resident Wat-man

    Joined:
    Jan 28, 2012
    Messages:
    5,914 (6.52/day)
    Thanks Received:
    1,903
    Location:
    Concord, NH
    On your local network? iSCSI is always an option and would behave much more like a drive that was physically connected to your computer. If you're outside on the internet you could do some weird SSH forwarding magic to get iSCSI over the internet. I don't think it would be very fast though. I don't have a whole lot of experience with streaming content over the web. I usually only work on a local network when it comes to stuff like that.
    Last edited: Jun 1, 2013
  24. [Ion]

    [Ion] WCG Team Assistant

    Joined:
    Sep 15, 2009
    Messages:
    11,786 (6.65/day)
    Thanks Received:
    10,889
    Location:
    North Carolina, United States
    Oh, I'm aware that Apache wouldn't let me do that directly--I can probably find instructions or a pre-made thing on how to get the streaming working.

    The HD TV shows I have are about 1.4 gigabytes per 40 minute episode, which comes out to about .58 megabytes/second. I can consistently get twice that uploading from the dorm, so I assume that this would be feasible?

    I'll try to get a UPS at some point--at $40 that seems very reasonable.
    We have a static IP at school, so that's easy. I can just memorize it easily. I'll check out HFS--I think that I've seen it before :)
    I am entirely unfamiliar with iSCSI--I'll check it out though! The goal would be to stream to another system on the same network or another hall on campus (so across a consistently very fast network).

    Thanks for the help everyone! :toast:
    Crunching for Team TPU
  25. Aquinus

    Aquinus Resident Wat-man

    Joined:
    Jan 28, 2012
    Messages:
    5,914 (6.52/day)
    Thanks Received:
    1,903
    Location:
    Concord, NH
    Oh man! You have no idea what you're missing then. iSCSI sends SCSI commands over an IP network, so you're basically accessing files directly on the iSCSI target over a network. Think of it this way. Streaming will work because you're accessing the file over the network directly as if it were a file locally on the machine you're using. It's like a mapped network drive but better. It's not incredibly easy to configure though. This would be best if your machines are connecting to your server only.

    If other people will be connecting to it you will want to use something like Samba. Just about every machine likes Samba and since this is a "local" network it should work well. So in the name of simplicity and that mapping network drives works well (on OS X and Windows,) I would say, go with Samba. :cool:

    ( All of this is doable in Debian. ;) )

Currently Active Users Viewing This Thread: 1 (0 members and 1 guest)

Share This Page