Find the answer to your Linux question:
Results 1 to 7 of 7
Hello all, Given the information about machines in a cluster (IP address/machine name) and a program (Java language) to run, is there a job controller available which would execute this ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Dec 2004
    Posts
    12

    Job controller across cluster


    Hello all,

    Given the information about machines in a cluster (IP address/machine name) and a program (Java language) to run, is there a job controller available which would execute this program and returns the output on each of the machines?

    I would like to deploy my java program on each node of the cluster. Start the program once it is copied there and get back the outputs.

    I have asked this question on stack overflow and they suggested that I use tentakel & shmux. Are there any other options available as well?

    PS: Couldn't post the link to Stack Overflow post because I need to make atleast 15 posts to put in external links.

    Thank you in advance.

  2. #2
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    Location
    Munich
    Posts
    3,356
    Well, I can add clusterssh and parallel-ssh to the list of multiplexers.

    For quick commands (ie: not running for hours) to multiple machines and if you are only interested in stdout and stderr of your program, these are useable.

    e.g.:
    I assume
    - ssh key auth is in place on all machines
    - ~/php_hosts is a list of fqdn of your machines
    Code:
    mkdir -p ~/parallel-runs/{out,err} 
    pssh -h ~/php_hosts -o ~/parallel-runs/out/ -e ~/parallel-runs/err/ "zgrep -h Fatal /var/log/php/error.log*"
    But you probably look for something bigger?
    This might be of interest, but needs code modification:
    Gearman
    Last edited by Irithori; 03-09-2011 at 04:12 PM. Reason: fixed typo
    You must always face the curtain with a bow.

  3. #3
    Just Joined!
    Join Date
    Dec 2004
    Posts
    12
    Thank you for the reply and the suggestions. They look interesting, I will try them out.

  4. #4
    Just Joined!
    Join Date
    Dec 2004
    Posts
    12
    Hi Irithori,

    In pssh, is there a way to specify that I want to execute a script file rather than something between quotes

    Eg: pssh -h hosts.txt -o ~/prj/output ~/prj/test.sh

    Here I want to run test.sh on all the hosts

  5. #5
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    Location
    Munich
    Posts
    3,356
    ~/prj/test.sh needs to be locally on every host, then yes.
    You must always face the curtain with a bow.

  6. #6
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    Location
    Munich
    Posts
    3,356
    Other than that, I just learned about the -I (--send-input) option.

    With that you can do something like:
    Code:
    cat <<EOF > ~/prj/test.macro
    ls -la
    uname -a
    EOF
    Code:
    cat ~/prj/test.macro| pssh -h hosts.txt -o ~/prj/output -I
    Still, every command inside ~/prj/test.macro needs to be on every host of hosts.txt

    P.S.:
    I would recommend to capture stderr as well (-e) but ymmv.
    Last edited by Irithori; 03-11-2011 at 01:13 PM. Reason: typo
    You must always face the curtain with a bow.

  7. #7
    Just Joined!
    Join Date
    Dec 2004
    Posts
    12
    Yes, the -I option works. Thank you for all the suggestions

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •