Strange idea... possible to cluster computers on a network for general processing?

[hat]

Would it be possible to hook together a number of widely different computers (such as my Q6600 system, this old Pentium 4 sitting over here, and this one AMD desktop sitting over here), and link them up to run general tasks and assist my main system? Possible applications... handbrake/convertx encoding, improved performance in games? Maybe by way of a "Host" program installed on each system, which I would then use on the main system to target and launch whichever program I wanted to run with the assistance of the others (e.g. Handbrake)?

 

[Devon68]

I know I've seem something similar. But damn it was hard to find it again especially because I didn't remember the thread title. You could ask this guy as he has clustered computers.
http://www.techpowerup.com/forums/threads/madman-the-ultimate-ghetto-build.202110/

 

[hat]

Unfortunately he doesn't describe what he used them for or how they were linked or anything... I tried posting maybe he'll get back to it.

 

[95Viper]

Not so strange...

Tech Article - DIY Supercomputing: How to Build a Small Windows HPC Cluster

Building Your Own Super Computer

Building your own supercomputer using Ubuntu and Kerrighed


Quote from here --> How to Hook Up Several Computers to Make a Supercomputer

Tips & Warnings

• Cluster computing only works with applications that support parallel processing models. Most desktop applications do not support the MPI protocol; therefore, creating a super computer from a cluster will not help speed processing for incompatible programs.

 

[Vario]

beowulf cluster maybe?

 

[Aquinus]

I don't think what you want to do exactly exists. You can't easily divvy up tasks the way that you're suggesting. If you had a particular application in mind, you can always develop something yourself but most applications that take advantage of networked HPC are similar to GPGPU tasks. Some applications like Transmission support remote management, but that's not "clustered".

 

[yogurt_21]

^ I think you hit on it when you said it doesn't "exactly exist" visualization and things like xenapp do a piece of of it, clustering and etc could do another. But nothing will essentially allow you to take all of your rigs power and combine it together for standard day to day tasks. Video encoding and converting might actually be slower as the HD's and networking might not be fast enough to keep up. Gaming certainly would be slower.

For me I'd look more into programs that do the same thing handbrake does only better/faster. The best video encoder/transcoder I've found is FFmpeg. Simple command line based but very fast and robust.

https://www.ffmpeg.org/

Not only do we use it at work to re-encode out mp4's for student lectures, but the NewTek Tricasters we use for filming use it to film to mp4 as well.

 

[Shambles1980]

thought all the nodes in a cluster had to use the same hardware and software. thats why i never bothered to try and set one up.

 

[hat]

Sounds very limited and with specific uses.

 

[yogurt_21]

Sounds very limited and with specific uses.

? really.

The first description on their site

A complete, cross-platform solution to record, convert and stream audio and video.

It's not an audio studio perhaps, but there's almost nothing it hasn't been able to do with video. And it still does have a crap ton of options for audio.

 

[Sasqui]

Used to work with Incredibuild, https://www.incredibuild.com/ . Biggest performance feature is a distributed compiler. Each client computer needed to have it installed where it could take advantage of idle CPU time on remote computers to compile Visual Studio projects. Cut compiling time by 10x (at least).

So long story short, without an OS or generic program to break up and distribute tasks, you probably need specific software that does the parallel task management and distribution to manage whatever it is you're trying to do faster.

 

[Disparia]

Not exactly what you're looking for, just throwing out ideas.

There is software (the name escapes me at the moment) that acts as a KVM of sorts. Say you have 4 computers, you set them up like you would multiple displays so when you move your mouse from one screen to another, you are sending input to that specific computer. Start Handbrake on one, friends twitch stream on another, etc.

It's similar to if you Remote Desktop (Teamviewer, etc) to each machine but with some added benefits, as long as you have a monitor per computer.

thought all the nodes in a cluster had to use the same hardware and software. thats why i never bothered to try and set one up.

You see that a lot because it's easier to buy and build in bulk, but a cluster can be designed in various ways. Sometimes it's best to be identical across the board, and other times it's good to have nodes tailored for different tasks. Other times, it's just because there's some hardware laying around and you design around what you have

 

[Frick]

Sounds very limited and with specific uses.

That's clusters for you.

Now this is the territory of cloud computing.

 

[Aquinus]

That's clusters for you.

Now this is the territory of cloud computing.

I find is amusing how people think code can be made multi-threaded without any issue and now people think cloud computing is easy as pie. Sorry to break it to you people, but concurrent programming isn't easy and isn't always possible. Cloud configurations are much harder to manage than multi-threaded ones, which is why you see them even less of them unless the applications demands performance (real time applications.) If you're not a SysAdmin or a Developer with a good reason for doing it, you probably shouldn't be considering it.