lemonadesodaI|m sure that this guy has had a lot of fun building 6 machines into a 19 inch rack. But 6x 4 GTXs could be improved using a PCIe expansion bay. Since this is a folding farm and not GPU SLI, he doesnt need x16 slots, so an extension bay would be quite practical. He could use x4 for the cards, and easily drive 8 or 16 cards from ONE PC.

That is not correct. The nVidia driver only supports 8 GPUs. I have spent a lot of time researching this issue and at the current time it's not possible to use PCIe extenders or motherboards that have more than 4 PCIe slots for this reason. It is also not possible to run VMs on a machine with PCIe backplanes because there is no VM software yet that allows virtualization of the GPUs for CUDA. Xen is working on it though.

You can see a discussion here: foldingforum.org/viewtopic.php?f=38&t=9340&start=0

and here: foldingforum.org/viewtopic.php?f=38&t=9011

Jason

What ppd do you get? Also whats the power usage like?

It's Atlas that has it... Look here for details (ppd etc)

HARDCORE :rockout:

The earth simulator had 41 teraflops of supercomputing power. it was #1 in the world starting in 2002 till 2004. It cost about 72 million dollars. congratulations on building an equivalent for under $30,000.

@ Atlas Folder

On daily basis, how much time do you spend controlling all your Clients? At the moment I have only 6 Clients and sometimes I get more than one down while I'm at work.

I believe he isn't a TPU member... unfortunately

ghost101What ppd do you get? Also whats the power usage like?

My 24 hour average is ~267,000 PPD. You can see that here.

The power consumption of a single shelf is ~1,125 Watts. I have 5 shelves and more misc equipment. Power consumption is probably around 7,000 Watts continuous. I haven't added it all up lately.

sol.fidesThe earth simulator had 41 teraflops of supercomputing power. it was #1 in the world starting in 2002 till 2004. It cost about 72 million dollars. congratulations on building an equivalent for under $30,000.

Well thanks, but there is an important difference. True supercomputers use 64-bit floating point math (double-precision), Folding@Home makes do with 32-bit floating point math (single-precision). So yes, building supercomputers is getting cheaper fast, but there's still no free (cheap) lunch. If you need double-precision you still must pony up the cash.

mmaakkOn daily basis, how much time do you spend controlling all your Clients?

Not much... I can check for dead EUE clients anywhere from my iPhone by going here, it's updated every 60 seconds. If I need to make a correction all of the machines are controllable through LogMeIn and I have LogMeIn Ignition on my iPhone so I can log into the servers from my iPhone and tweak things as necessary.

I made a post a couple of weeks ago about how I control and implement all of the clients. I think it's kinda slick even if I do say so myself, but necessity is the mother of invention. Controlling all the clients individually got old very, very fast. I had to think of a better way. You can see my post about that here.

Thanks for the interest guys,

Jason

Thanks for coming to TPU and sharing your experience Jason. Great build(s), nice rack. I can see that perhaps the multiple PC route is easier than the extension bay option.

Did I understand you are driver-limited. Therefore max 8 GPUs per PC, so you would need a minimum of 3 independent PCs to run your 24 cards? Or does the x2 thing knock that down to just 4 cards (8 GPU dies)?

PS. Write to nVidia, show them your stuff, get them to fix the drivers, and sponsor your rack.

The Atlas Folder, very nice folding setup.
I wonder what it will actually be crunching.

Wow that is INSANE! I wish I had the money to do something like this.

He was short a few cards on the bottom and it really kills the look of the system:p



Just kidding, and that's one hell of a folding rig you have!

Welcome to TPU Atlas :)
It is neat to see all that awsome hardware (especially the GPUs) working hard to find a cure :toast:
For those that want to learn more about his cause, check out the link to his website at bottom of his sig.

Its for a great cause.I wish his idea becomes true.

i think Folding@Home complete all research soon

hayder.masteri think Folding@Home complete all research soon

i didn't hear any results like a cure or something and since 2000....

i'm skeptikal about this f@h do you know exactly what data is processed ?

Stanford say is protein folding and other molecular dynamics but is really this?

Finally, someone who is folding not for showing off....

laszloI made a post a couple of weeks ago about how I control and implement all of the clients. I think it's kinda slick even if I do say so myself, but necessity is the mother of invention. Controlling all the clients individually got old very, very fast. I had to think of a better way. You can see my post about that here.

Thanks for the interest guys,

Jason

Welcome -- I think it's really cool what you're doing to help out your Dad. Anytime anyone criticizes the amount of money it all costs, just remind them about that.

That being said, looks like I've found another admin who names a group of server units after characters in Atlas Shrugged. :p

laszloi didn't hear any results like a cure or something and since 2000....

i'm skeptikal about this f@h do you know exactly what data is processed ?

Stanford say is protein folding and other molecular dynamics but is really this?

Ah, critics.

Cures take a long, long time to develop.

@Atlas Folder: Are you also running SMP clients on the CPUs, or are you just doing pure GPU folding?

Why cure cancer when you can play Crysis over 60fps :respect:

I was expecting a custom, 23 PCI Express slot motherboard board loaded with GPUs (like a Tesla system). Needless to say, I'm disappointed. Anyway, that's quite a blatant waste of money and electricity, in my opinion.

what cpu are you using Atlas??
I wonder if it is i7s...
I think you can also run WCG while folding...

h3llb3nd4what cpu are you using Atlas??
I wonder if it is i7s...
I think you can also run WCG while folding...

The boards he seems to be using looks like the MSI K9A2 Platinum, an AM2 board. So I would guess he is running some kind of Phenom.

damn! :(... I am still confused about how you link up all those pcs, and how do you get them all onto one monitor...?

h3llb3nd4damn! :(... I am still confused about how you link up all those pcs, and how do you get them all onto one monitor...?

They aren't really linked, they don't need to be really. They are all on the same network, that is about it.

As for connecting them all the to the same monitor, I'm sure the rack has a KVM, which allows multiple machines to be connected to the same Keyboard, Monitor, and Mouse. Although, once set-up the machines should be able to be run headless, with remote access via LogMeIn or Remote Desktop.