Job controller across cluster

[raghava]

Hello all,

Given the information about machines in a cluster (IP address/machine name) and a program (Java language) to run, is there a job controller available which would execute this program and returns the output on each of the machines?

I would like to deploy my java program on each node of the cluster. Start the program once it is copied there and get back the outputs.

I have asked this question on stack overflow and they suggested that I use tentakel & shmux. Are there any other options available as well?

PS: Couldn't post the link to Stack Overflow post because I need to make atleast 15 posts to put in external links.

Thank you in advance.

[Irithori]

Well, I can add clusterssh and parallel-ssh to the list of multiplexers.

For quick commands (ie: not running for hours) to multiple machines and if you are only interested in stdout and stderr of your program, these are useable.

e.g.:
I assume
- ssh key auth is in place on all machines
- ~/php_hosts is a list of fqdn of your machines

mkdir -p ~/parallel-runs/{out,err} 
pssh -h ~/php_hosts -o ~/parallel-runs/out/ -e ~/parallel-runs/err/ "zgrep -h Fatal /var/log/php/error.log*"

But you probably look for something bigger?
This might be of interest, but needs code modification:
Gearman

[raghava]

Thank you for the reply and the suggestions. They look interesting, I will try them out.

[raghava]

Hi Irithori,

In pssh, is there a way to specify that I want to execute a script file rather than something between quotes

Eg: pssh -h hosts.txt -o ~/prj/output ~/prj/test.sh

Here I want to run test.sh on all the hosts

[Irithori]

~/prj/test.sh needs to be locally on every host, then yes.

[Irithori]

Other than that, I just learned about the -I (--send-input) option.

With that you can do something like:

cat <<EOF > ~/prj/test.macro
ls -la
uname -a
EOF

cat ~/prj/test.macro| pssh -h hosts.txt -o ~/prj/output -I

Still, every command inside ~/prj/test.macro needs to be on every host of hosts.txt

P.S.:
I would recommend to capture stderr as well (-e) but ymmv.

[raghava]

Yes, the -I option works. Thank you for all the suggestions