Find the answer to your Linux question:
Results 1 to 7 of 7
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1

    Writing a bash script to organize running programs

    Hi all !

    I am sorry about the title, it might be confusing but i couldnt come up with a better one for that little space.

    I have to execute 4 programs (concurrently), then wait for their execution to finish, and then give the command to execute the next batch of 4 programs.
    Can I do this automatically with a script?

    manually I am doing something like this:

    ./nmaex input1 output1

    (then in new terminal)
    ./nmaex input2 output2

    (then in new terminal)
    ./nmaex input3 output3

    (then in new terminal)
    ./nmaex input4 output4

    Then I wait for all these to finish, and do the above again for next batch of 4 input and output files. The 4 programs running concurrently take about 4-7 hrs. depending on input to finish, so I constantly have to check if they have finished or not.

    Is there some standard way of doing this? If not, how do I write a script for this?

    Thank you in advance !!

  2. #2
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    It depends.
    If you can generate input{1..4} and output{1..4} programmatically or at least know them beforehand,
    then parallel might be the tool you are looking for.

    One step beyond would be gearman, which allows you to distribute jobs to multiple machines.
    This needs more preparation than parallel of course.

    But if input{1..4} and output{1..4} demand interactive input from you, then they would be complicate to automate.
    You must always face the curtain with a bow.

  3. #3
    Thank you for your reply.

    Actually input# and output# are just files (like text files in some format) so I have them ready beforehand, no interactive input is there.

    I was thinking of some simple solutions, but I am new to scripts.
    Like if I make two script files (each with a batch of 4 commands), and then do
    ./script1 && ./script2
    then this should work, right?

    but then what should script1 and script2 be?

    I want to do something like this: (for script1)
    ./nmaex input1 output1  &
    ./nmaex input2 output2  &
    ./nmaex input3 output3  &
    ./nmaex input4 output4  &
    for((; ;))
            if(all the jobs have finished)
    This way, when all jobs in script1 are finished, only then script2 will be run. But what should there be in the if construct?
    How can i do something like this:

     jobs > $variable
          if(variable -eq null)
    If any of the four background processes are running, the command jobs will return a non-null output (list of pids and job ids of bg processes), otherwise it will return null (meaning that execution has completed)
    I know C/C++, fortran and JAVA, but have no idea about scripts.

    Thanks again !!

  4. $spacer_open
  5. #4
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    Is it important, that the whole batch of 1-4 is finished before the next four are triggered?

    Something like your script approach is doable.
    But the thing is: parallel has been developed for this kind of scenario.
    It will probably have more options and is more mature and stable than what either you or me can hack together in a hurry.
    So why not invest the time in learning this tool instead of trying to reinvent the wheel.

    Dont get me wrong, if you decide to implement it yourself, then this is perfectly fine as well.
    You know your usecase better than me.
    Just want to show a maybe easier route
    You must always face the curtain with a bow.

  6. #5
    No actually it is better if it is optimal, and I looked up GNU parallel and it is exactly what I need. I will be using that only. Thank you so much !!

    Because I had already spent some time trying to devise a hack, I wanted to see one work. I am actually a computer science student but new to linux and scripting and this was my first attempt at writing a script. But I guess I'll have to start from the basics only. I thought knowing a few languages ought to give me a headstart in scripting, but the syntax seems to be wayyyy diifferent. I think I'll follow a book like learning the bash by oreilly.

    Thanks !!

  7. #6
    Linux User
    Join Date
    Jun 2012
    SF Bay area
    The bash builtin function "wait" does pretty much what you want. If you don't give is any arguments it blocks until all background processes complete, then continues. So you can through commands in the background with "&" and then just "wait" afterwards. The just throw more processes in the background and "wait" for those, etc...

    That's the simplest way I can think of doing it. Since it's Linux shell scripting, there's always 10+ alternate ways of course.

  8. #7
    Hmm....that's a much neater way of doing that.
    Thanks cnamejj !!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts