How to run command or code in parallel in bash shell under Linux or Unix

How to run command or code in parallel in bash shell under Linux or Unix

last updated in Categories , , , , , , ,

 

How do I run commands in parallel in a bash shell script running under Linux or Unix-like operating system? How can I run multiple programs in parallel from a bash script?

You have various options to run programs or commands in parallel on a Linux or Unix-like systems:
=> Use GNU/parallel or xargs command.
=> Use wait built-in command with &.
=> Use xargs command.
How to run multiple programs in parallel from a bash script in linux / unix?
This page shows how to run commands or code in parallel in bash shell running on a Linux/Unix systems.

 

 

 

Putting jobs in background

The syntax is:
command &
command arg1 arg2 &
custom_function &

OR
prog1 &
prog2 &
wait
prog3

In above code sample, prog1, and prog2 would be started in the background, and the shell would wait until those are completed before starting the next program named progr3.

Examples

In this following example run sleep command in the background:
$ sleep 60 &
$ sleep 90 &
$ sleep 120 &

To displays status of jobs in the current shell session run jobs command as follows:
$ jobs
Sample outputs:

[1]   Running                 sleep 60 &
[2]-  Running                 sleep 90 &
[3]+  Running                 sleep 120 &

Let us write a simple bash shell script:

#!/bin/bash
# Our custom function
cust_func(){
  echo "Do something $1 times..."
  sleep 1
}
# For loop 5 times
for i in {1..5}
do
	cust_func $i & # Put a function in the background
done
 
## Put all cust_func in the background and bash 
## would wait until those are completed 
## before displaying all done message
wait 
echo "All done"

Let us say you have a text file as follows:
$ cat list.txt
Sample outputs:

https://server1.cyberciti.biz/20170406_15.jpg
https://server1.cyberciti.biz/20170406_16.jpg
https://server1.cyberciti.biz/20170406_17.jpg
https://server1.cyberciti.biz/20170406_14.jpg
https://server1.cyberciti.biz/20170406_18.jpg
https://server1.cyberciti.biz/20170406_19.jpg
https://server1.cyberciti.biz/20170406_20.jpg
https://server1.cyberciti.biz/20170406_22.jpg
https://server1.cyberciti.biz/20170406_23.jpg
https://server1.cyberciti.biz/20170406_21.jpg
https://server1.cyberciti.biz/20170420_15.jpg
https://server1.cyberciti.biz/20170406_24.jpg

To download all files in parallel using wget:

#!/bin/bash
# Our custom function
cust_func(){
  wget -q "$1"
}
 
while IFS= read -r url
do
        cust_func "$url" &
done < list.txt
 
wait
echo "All files are downloaded."

GNU parallel examples to run command or code in parallel in bash shell

From the GNU project site:

GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables.

The syntax is pretty simple:
parallel ::: prog1 prog2
For example, you can find all *.doc files and gzip (compress) it using the following syntax:
$ find . -type f -name '*.doc' | parallel gzip --best
$ find . -type f -name '*.doc.gz'

Install GNU parallel on Linux

Use the apt command/apt-get command on a Debian or Ubuntu Linux:
$ sudo apt install parallel
For a RHEL/CentOS Linux try, yum command:
$ sudo yum install parallel
If you are using a Fedora Linux, try dnf command:
$ sudo dnf install parallel

Examples

Our above wget example can be simplified using GNU parallel as follows:
$ cat list.txt | parallel -j 4 wget -q {}
OR
$ parallel -j 4 wget -q {} < list.txt

0 (0)
Article Rating (No Votes)
Rate this article
Attachments
There are no attachments for this article.
Comments
There are no comments for this article. Be the first to post a comment.
Full Name
Email Address
Security Code Security Code
Related Articles RSS Feed
Fałszujemy rozpoznania skanerów #2
Viewed 3602 times since Mon, May 21, 2018
stunnel How To Encrypt Traffic to Redis with Stunnel on Ubuntu 16.04
Viewed 2925 times since Sun, Dec 6, 2020
How to accurately determine when the system was booted
Viewed 2847 times since Wed, Oct 3, 2018
RHCS: Configure an active/backup pacemaker cluster
Viewed 10083 times since Sun, Jun 3, 2018
Linux Audit The Linux security blog about Auditing, Hardening, and Compliance lynis
Viewed 2842 times since Thu, Jan 16, 2020
Method 2 – Use shell scripts How to install yum cron on a CentOS/RHEL 6.x/7.x
Viewed 4946 times since Tue, Dec 4, 2018
How log rotation works with logrotate
Viewed 5736 times since Fri, Nov 30, 2018
Setting up encrypted tunnel using stunnel
Viewed 3284 times since Fri, Sep 28, 2018
Lsyncd: live file syncronization across multiple Linux servers
Viewed 7971 times since Wed, Oct 31, 2018
RHCS6: Install a two-node basic cluster
Viewed 4212 times since Sun, Jun 3, 2018