Writing a bash script to organize running programs

[hypernova]

Hi all !

I am sorry about the title, it might be confusing but i couldnt come up with a better one for that little space.

I have to execute 4 programs (concurrently), then wait for their execution to finish, and then give the command to execute the next batch of 4 programs.
Can I do this automatically with a script?

manually I am doing something like this:

./nmaex input1 output1

(then in new terminal)
./nmaex input2 output2

(then in new terminal)
./nmaex input3 output3

(then in new terminal)
./nmaex input4 output4

Then I wait for all these to finish, and do the above again for next batch of 4 input and output files. The 4 programs running concurrently take about 4-7 hrs. depending on input to finish, so I constantly have to check if they have finished or not.

Is there some standard way of doing this? If not, how do I write a script for this?

Thank you in advance !!

[Irithori]

It depends.
If you can generate input{1..4} and output{1..4} programmatically or at least know them beforehand,
then parallel might be the tool you are looking for.

One step beyond would be gearman, which allows you to distribute jobs to multiple machines.
This needs more preparation than parallel of course.

But if input{1..4} and output{1..4} demand interactive input from you, then they would be complicate to automate.

[hypernova]

Thank you for your reply.

Actually input# and output# are just files (like text files in some format) so I have them ready beforehand, no interactive input is there.

I was thinking of some simple solutions, but I am new to scripts.
Like if I make two script files (each with a batch of 4 commands), and then do
./script1 && ./script2
then this should work, right?

but then what should script1 and script2 be?

I want to do something like this: (for script1)

#!/bin/bash

./nmaex input1 output1  &
./nmaex input2 output2  &
./nmaex input3 output3  &
./nmaex input4 output4  &

for((; ;))
do
        if(all the jobs have finished)
        then
             break
        fi
	
done

This way, when all jobs in script1 are finished, only then script2 will be run. But what should there be in the if construct?
How can i do something like this:

 jobs > $variable
      if(variable -eq null)
      then
           break
      fi

If any of the four background processes are running, the command jobs will return a non-null output (list of pids and job ids of bg processes), otherwise it will return null (meaning that execution has completed)
I know C/C++, fortran and JAVA, but have no idea about scripts.

Thanks again !!

[Irithori]

Is it important, that the whole batch of 1-4 is finished before the next four are triggered?


Something like your script approach is doable.
But the thing is: parallel has been developed for this kind of scenario.
It will probably have more options and is more mature and stable than what either you or me can hack together in a hurry.
So why not invest the time in learning this tool instead of trying to reinvent the wheel.

Dont get me wrong, if you decide to implement it yourself, then this is perfectly fine as well.
You know your usecase better than me.
Just want to show a maybe easier route

[hypernova]

No actually it is better if it is optimal, and I looked up GNU parallel and it is exactly what I need. I will be using that only. Thank you so much !!

Because I had already spent some time trying to devise a hack, I wanted to see one work. I am actually a computer science student but new to linux and scripting and this was my first attempt at writing a script. But I guess I'll have to start from the basics only. I thought knowing a few languages ought to give me a headstart in scripting, but the syntax seems to be wayyyy diifferent. I think I'll follow a book like learning the bash by oreilly.

Thanks !!

[cnamejj]

The bash builtin function "wait" does pretty much what you want. If you don't give is any arguments it blocks until all background processes complete, then continues. So you can through commands in the background with "&" and then just "wait" afterwards. The just throw more processes in the background and "wait" for those, etc...

That's the simplest way I can think of doing it. Since it's Linux shell scripting, there's always 10+ alternate ways of course.

[hypernova]

Hmm....that's a much neater way of doing that.
Thanks cnamejj !!