Caching fileserver

[cheesy999]

A place i have been spending a lot of time at lately has a router/server system which caches recently downloaded items, such as program installers and windows updates.

this means that, for example, say you have 3 PC's running windows 7, and you want to download service pack 1, the first one has to download it from Microsoft at the slow connection to the outside world, but the other 2 PC's can just download it off the local cache at 10's/100's of MB/s

How hard would this be to set up at home?, in my current network there is a router, which feeds 3pc's 2 XBOX's, a ps3,the SKY box, and all of the mobile phones in the house on wireless, and another PC + a network switch leading to 2 PC's on a 10/100 connection

How would i be able to implement some sort of caching mechanism into the network so things i do, Windows updates, program updates, frequently watched youtube videos etc etc etc, do not have to be repeatedly downloaded from the outside world.

To do this with i have, a spare router, i could easily re-purpose a pc or launch a virtual machine on one, i also have 305M of Cat 5E networking cable to link things up, and a network switch + spare wireless adapters.

 

[W1zzard]

linux + squid + transparent proxy

 

[cheesy999]

linux squid transparent proxy

Thanks

but how would i set up such a system?, i know little about networking, and also only small amounts about Linux, i don't know what squid is (i've found the website), what it does, and how i would set it up i don't know.

i came here so i could hopefully get some sort of walkthrough or instructions or at least a link to some.

this page lists examples but to me all it is is a wall of text

 

[W1zzard]

time to start reading up

not worth it imo if you have a somewhat decent connection.

 

[cheesy999]

500KB/s/5 meg depending on browser speak/ISP speak, it works reasonably well until i need to reinstall a PC and then it can take Days of constant use to get back all my windows updates/steam games

and having so much stuff on the network means that bottlenecks can occur pretty quickly, if the Sky box is downloading a movie, someones watching youtube, someones playing a game (not unusual to happen at the same time, then the connection slows to a crawl

EDIT: After spending a while reading the installation instructions etc, i think w1zz is right, its far too much effort to be worth it, if anyone has any other suggestions, i'd like to hear them, maybe a simpler way of doing this

 

[qubit]

W1zz is right, but if you don't want to meddle with Linux, then the trusty IPCop firewall that I've used for years has just this feature.

Once installed, just set the cache to transparent, set its size and enable it. Not only that, but you get an excellent firewall. Be aware though that IPCop takes over a whole PC.

Now, note that the Windows updating system is dynamic, so you may not quite get the full caching that you're after with doing it. You simply have to try it.

www.ipcop.org

Now, a better alternative is to dedicate a PC to run Windows Server 2008 or Server 2008 R2 and use WSUS. This is dedicated to the exact kind of update caching that you're after and works really well. You'll have a learning curve getting to grips with a server OS and WSUS, but you can get lots of help if you need it. Don't worry though, as there's lots of resources (see below) and I've done this many times, so will be happy to help you with it. If you need my help, start a thread on this and PM me the thread link.

You'll have to of course secure yourself a legal licence and download for it. The best way to do this is with a TechNet subscription for £230. This gets you multiple products and multiple keys for each one. It's worth literally thousands of pounds and is stonkingly good value. Sorry about the excess emoticons, but it really is that good.

www.technet.com


WSUS resources:

http://technet.microsoft.com/en-us/windowsserver/bb332157.aspx

http://social.technet.microsoft.com/Forums/en-CA/winserverwsus/threads

www.wsuswiki.com

WSUS = Windows Server Update Services

 

[cheesy999]

W1zz is right, but if you don't want to meddle with Linux, then the trusty IPCop firewall that I've used for years has just this feature.

Once installed, just set the cache to transparent, set its size and enable it. Not only that, but you get an excellent firewall. Be aware though that IPCop takes over a whole PC.

Now, note that the Windows updating system is dynamic, so you may not quite get the full caching that you're after with doing it. You simply have to try it.

www.ipcop.org

Now, a better alternative is to dedicate a PC to run Windows Server 2008 or Server 2008 R2 and use WSUS. This is dedicated to the exact kind of update caching that you're after and works really well. You'll have a learning curve getting to grips with a server OS and WSUS, but you can get lots of help if you need it. Don't worry though, as there's lots of resources (see below) and I've done this many times, so will be happy to help you with it. If you need my help, start a thread on this and PM me the thread link.

You'll have to of course secure yourself a legal licence and download for it. The best way to do this is with a TechNet subscription for £230. This gets you multiple products and multiple keys for each one. It's worth literally thousands of pounds and is stonkingly good value. Sorry about the excess emoticons, but it really is that good.

www.technet.com


WSUS resources:

http://technet.microsoft.com/en-us/windowsserver/bb332157.aspx

http://social.technet.microsoft.com/Forums/en-CA/winserverwsus/threads

www.wsuswiki.com

WSUS = Windows Server Update Services

will any of those work from within a virtual machine?

i'm thinking if you could somehow set the virtual machine up as a proxy or something...

 

[qubit]

Yes, it's possible to make both work from a VM. I believe the free VMWare one will work for IPCop and Windows Server. The IPCop site has instructions on how to set up a VM, I think, or simply google it.

Microsoft's own Virtual PC will certainly work with 32-bit Windows Server.

 

[cheesy999]

Yes, it's possible to make both work from a VM. I believe the free VMWare one will work for IPCop and Windows Server. The IPCop site has instructions on how to set up a VM, I think, or simply google it.

Microsoft's own Virtual PC will certainly work with 32-bit Windows Server.

i can set up Virtual machines (virtualbox FTW) its going to be a case of experimenting as i can't do their recommended set-up in my house

 

[qubit]

Go for it, just remember that you'll need lots and lots of disc space set aside for the update cache. A Western Digital 2TB Green will fit the bill nicely here, as they work very well and are incredibly cheap. I've got three of these now and I love them.

For Windows Server, you'll need lots of free RAM in the host PC as well. Bottom line is that one way or another, you're gonna have to spend money to make this happen.

 

[cheesy999]

are these good settings?

i've given it 2 proccesors, 1.5GB of ram, 400GB hard drive, 3d Acceleration with 128MB of graphics ram and bridged the networking adaptor

EDIT: Problem:

DOUBLE EDIT:Fixed it by switching to an IDE controller

 

[qubit]

That all looks ok to me. Is that screendump from Virtualbox?

I see you sorted out the glitch.

The stable version of IPCop, 1.4.21, is unfortunately several years old now, so it doesn't support the latest SATA interfaces, as you found out. There's a much updated beta version if you want to try that.

Also, I had a quick look at the site for the Squid proxy that W1zz mentioned and guess what? There's actually a Windows version too! Yup, right here:

http://wiki.squid-cache.org/SquidFaq/BinaryPackages#Windows

I have no idea what this Windows version is like, so just have a go. I reckon that this is your best option for a web cache.

Main site: www.squid-cache.org

 

[TheoneandonlyMrK]

i'd like to hear them, maybe a simpler way of doing this

yeh its called creating a softarc folder on a spare drive and having all dls sent into that which you then sort further and use as a library of usefull stuff

 

[cheesy999]

That all looks ok to me. Is that screendump from Virtualbox?

I see you sorted out the glitch.

The stable version of IPCop, 1.4.21, is unfortunately several years old now, so it doesn't support the latest SATA interfaces, as you found out. There's a much updated beta version if you want to try that.

Also, I had a quick look at the site for the Squid proxy that W1zz mentioned and guess what? There's actually a Windows version too! Yup, right here:

http://wiki.squid-cache.org/SquidFaq/BinaryPackages#Windows

I have no idea what this Windows version is like, so just have a go. I reckon that this is your best option for a web cache.

Main site: www.squid-cache.org

I can't even work out how to install that

yeh its called creating a softarc folder on a spare drive and having all dls sent into that which you then sort further and use as a library of usefull stuff

whats softarc?

 

[TheoneandonlyMrK]

just a folder name recomendation, I call mine that cos its saved a few pcs

 

[cheesy999]

just a folder name recomendation, I call mine that cos its saved a few pcs

i think that they should make all of this much easier, until then, looks like i'm going to have to put up with my normal network, cause i've even run into problems with ipcop (it wants me to log on when i never created a user or password)

 

[qubit]

i think that they should make all of this much easier, until then, looks like i'm going to have to put up with my normal network, cause i've even run into problems with ipcop (it wants me to log on when i never created a user or password)

IPCop always creates two accounts when you set it up. One's called Admin and the other is for interface access. The admin one is what you want.

It isn't possible to set IPCop without these accounts, as it's part of the step by step installer and can't be skipped. Only if it bombs out halfway through will you not see it, but then the whole install is borked anyway. Therefore, you went wrong somewhere when installing it. Try it again, it won't bite.