Let's talk about clusters

Whenever I read about clusters, supposedly a person needs special software. This software has to be made for batch processing. In other words, the software/program was specifically made for computer clusters.

Is there any way to simply hook up a bunch of computers and have them process a task, such as processing the data from any random program without it having the be made for a cluster?

I'm talking about reducing the amount of system resources needed by one computer, while being able to take any program and have it processed by multiple computers.

For example, have firefox, google earth, gimp, and games processed by multiple computers without those programs being made for cluster technology in the first place.

I envision it like this:

I install like three ethernet port in a few computers, I connect all of them, and they will process the data.
What I will have in the background is a program that reads the data to be processed and sends that to other computers.
Those computers then process the data and shoot it back to the main computer.
The result is then put together for a whole.

Is any of that possible?
Or must programs be made for clusters?

The way I'm looking at it, GIMP, google earth, firefox, and other programs would have to be made for clusters. It seems as if there is no kind of daemon to handle these tasks with other computers.

http://www.linuxforums.org/forum/mis...ing-linux.html

I read that, and it seems to say running one process among multiple computers is a difficult task.
Why?

I think there is a need for this, but I'm also thinking it would run very slowly.
Regardless, I think it would be nice to have.

Doing a single process with a load-balancing cluster would mean old computers never seem so old.

This is indeed a very interesting task. But the main question, I think, is do we really want it? Clustering widely used programs like Firefox, GIMP, some games, etc is more likely meaningless. Let's say our PC has to start GIMP. We all know that it doesn't start in a sec. For some slow machines it even requires half a minute, maybe more? The programs require a lot of data to be stored in the memory, compute various things, etc.
Let's say we use DDR2-400 with the lowest clock speed. Only 100Mhz... but still it has 3200MB/s data rate. The only thing that actually can stall the process is the HDD.. let's say it eats 40MB/s. And the CPU has to compute... The 10/100 network adapter will give you as much as 12.5MB/s...

The only reason for such kind of clustering is if you really need to compute a lot of information. Like video rendering for an example, or some data analyzing stuff... Anything else is just a waste of time. But that is only my opinion. What do others think?

Cheers,
Ventsi

Quote:

---

Originally Posted by vigour

This is indeed a very interesting task. But the main question, I think, is do we really want it? Clustering widely used programs like Firefox, GIMP, some games, etc is more likely meaningless. Let's say our PC has to start GIMP. We all know that it doesn't start in a sec. For some slow machines it even requires half a minute, maybe more? The programs require a lot of data to be stored in the memory, compute various things, etc.
Let's say we use DDR2-400 with the lowest clock speed. Only 100Mhz... but still it has 3200MB/s data rate. The only thing that actually can stall the process is the HDD.. let's say it eats 40MB/s. And the CPU has to compute... The 10/100 network adapter will give you as much as 12.5MB/s...

The only reason for such kind of clustering is if you really need to compute a lot of information. Like video rendering for an example, or some data analyzing stuff... Anything else is just a waste of time. But that is only my opinion. What do others think?

Cheers,
Ventsi

---

I would agree with Ventsi.

To add I would say this. Your hardware, or sum of, is only as fast as it's weakest link. A single pc running windows is often limited by CPU speed or memory(or I should say the cpu and memory is limited by windows) whereas a linux box is limited by it's bus speed or I/O (some will debate this but this is my oppinion). So clustering CPU and memory will not make much of a difference. This is especialy true for trying to combine the data processing power of 3 machines for one task.

Clustering actualy came about as a cheap answer to expensive IBM mainframes that operate multiple series of subsystems. Each subsystem could perform one task at a time but the subsystems worked simultaneously performing different tasks to complete a larger goal. Distributed clustered environment was the cheaper answer. IBM is actualy starting to reverse this trend again by making their machines even more powerfull and capable of performing up to a 300 to 1 server to mainframe ratio.

Your network speed is the slowest link. Your application would actualy run slower if chuncked up between 3 systems. It would be more efficiant to have one machine running gimp, one running google and one running firfox.