Asked  7 Months ago    Answers:  5   Viewed   40 times

I am trying to write a .sh file that runs many programs simultaneously

I tried this

prog1 
prog2

But that runs prog1 then waits until prog1 ends and then starts prog2...

So how can I run them in parallel?

 Answers

20

To run multiple programs in parallel:

prog1 &
prog2 &

If you need your script to wait for the programs to finish, you can add:

wait

at the point where you want the script to wait for them.

Tuesday, June 1, 2021
 
nhunston
answered 7 Months ago
75

Invoke-Expression, also aliased as iex. The following will work on your examples #2 and #3:

iex $command

Some strings won't run as-is, such as your example #1 because the exe is in quotes. This will work as-is, because the contents of the string are exactly how you would run it straight from a Powershell command prompt:

$command = 'C:somepathsomeexe.exe somearg'
iex $command

However, if the exe is in quotes, you need the help of & to get it running, as in this example, as run from the commandline:

>> &"C:Program FilesSome ProductSomeExe.exe" "C:some other pathfile.ext"

And then in the script:

$command = '"C:Program FilesSome ProductSomeExe.exe" "C:some other pathfile.ext"'
iex "& $command"

Likely, you could handle nearly all cases by detecting if the first character of the command string is ", like in this naive implementation:

function myeval($command) {
    if ($command[0] -eq '"') { iex "& $command" }
    else { iex $command }
}

But you may find some other cases that have to be invoked in a different way. In that case, you will need to either use try{}catch{}, perhaps for specific exception types/messages, or examine the command string.

If you always receive absolute paths instead of relative paths, you shouldn't have many special cases, if any, outside of the 2 above.

Thursday, June 3, 2021
 
dkcwd
answered 6 Months ago
54
me=`basename "$0"`

For reading through a symlink1, which is usually not what you want (you usually don't want to confuse the user this way), try:

me="$(basename "$(test -L "$0" && readlink "$0" || echo "$0")")"

IMO, that'll produce confusing output. "I ran foo.sh, but it's saying I'm running bar.sh!? Must be a bug!" Besides, one of the purposes of having differently-named symlinks is to provide different functionality based on the name it's called as (think gzip and gunzip on some platforms).


1 That is, to resolve symlinks such that when the user executes foo.sh which is actually a symlink to bar.sh, you wish to use the resolved name bar.sh rather than foo.sh.

Tuesday, June 8, 2021
 
Yarin
answered 6 Months ago
76

Much preferrable to pushing wget into the background using & or -b, you can use xargs to the same effect, and better.

The advantage is that xargs will synchronize properly with no extra work. Which means that you are safe to access the downloaded files (assuming no error occurs). All downloads will have completed (or failed) once xargs exits, and you know by the exit code whether all went well. This is much preferrable to busy waiting with sleep and testing for completion manually.

Assuming that URL_LIST is a variable containing all the URLs (can be constructed with a loop in the OP's example, but could also be a manually generated list), running this:

echo $URL_LIST | xargs -n 1 -P 8 wget -q

will pass one argument at a time (-n 1) to wget, and execute at most 8 parallel wget processes at a time (-P 8). xarg returns after the last spawned process has finished, which is just what we wanted to know. No extra trickery needed.

The "magic number" of 8 parallel downloads that I've chosen is not set in stone, but it is probably a good compromise. There are two factors in "maximising" a series of downloads:

One is filling "the cable", i.e. utilizing the available bandwidth. Assuming "normal" conditions (server has more bandwidth than client), this is already the case with one or at most two downloads. Throwing more connections at the problem will only result in packets being dropped and TCP congestion control kicking in, and N downloads with asymptotically 1/N bandwidth each, to the same net effect (minus the dropped packets, minus window size recovery). Packets being dropped is a normal thing to happen in an IP network, this is how congestion control is supposed to work (even with a single connection), and normally the impact is practically zero. However, having an unreasonably large number of connections amplifies this effect, so it can be come noticeable. In any case, it doesn't make anything faster.

The second factor is connection establishment and request processing. Here, having a few extra connections in flight really helps. The problem one faces is the latency of two round-trips (typically 20-40ms within the same geographic area, 200-300ms inter-continental) plus the odd 1-2 milliseconds that the server actually needs to process the request and push a reply to the socket. This is not a lot of time per se, but multiplied by a few hundred/thousand requests, it quickly adds up.
Having anything from half a dozen to a dozen requests in-flight hides most or all of this latency (it is still there, but since it overlaps, it does not sum up!). At the same time, having only a few concurrent connections does not have adverse effects, such as causing excessive congestion, or forcing a server into forking new processes.

Sunday, June 27, 2021
 
Novalirium
answered 6 Months ago
82

From https://www.gnu.org/software/parallel/man.html#EXAMPLE:-Use-a-table-as-input:

"""
Content of table_file.tsv:

foo<TAB>bar
baz <TAB> quux

To run:

cmd -o bar -i foo
cmd -o quux -i baz

you can run:

parallel -a table_file.tsv --colsep 't' cmd -o {2} -i {1}

"""

So in your case it will be:

cat fileinput | parallel --colsep 't' myprogram {1} {2} {1}_vs_{2}.result
Monday, September 20, 2021
 
Naveen
answered 3 Months ago
Only authorized users can answer the question. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :  
Share