Using Shell Redirection: All About the Here-Doc

Print 1

 

When gathering information from remote AIX hosts using SSH (Secure Shell), it will definitely make your life a lot easier to write the commands within a here-doc block. A here-doc is a form of shell redirection that takes the command(s) supplied and redirects as input into another command or shell. Here-docs usage is common when you have locally based scripts where you may need to output a lot of text, i.e. help pages, mailing scripts. FTP scripts often contain here-doc method approach, due to parsing authentication details and the interactive commands required by a FTP connection. There are lots more uses for the here-doc. I've just noted a few. So what is a here-doc?

Choose a Word For Redirection

The common use of the here-doc syntax is:

command << delimiter
command(s)
delimiter

Where delimiter is the 'word' of choice, chosen by you (as long as it is not a SHELL reserved word). A common delimiter word is “block” or “EOF”. I usually use the word “mayday”. Now everything from the first to second occurrence of the word will be taken as input to the command or shell. Though you can put single quotes around the first occurrence of the word, this will not let variables get expanded and will not interpret variables as is. If you put a (-) hash before the word, you can indent your here-doc block commands with tabs only, this has the advantage of making the code much easier to read for others. The syntax for this is:

command << - 'delimiter'
command(s)
delimiter

Imagine you have a here-doc that generates lots of output for a report. How can you save it to an output file? You can either redirect the output of the script that contains the here-doc: assume the script is called: myprog, the output goes to the file :outputfile.txt:

$ myprog >outputfile.txt

Or with the here-doc at the start of the block, like so:

command << delimiter > outputfile
command(s)
delimiter

The following in Listing 1 redirects every command into the cat command. The word I’m using is “mayday”. Everything from the first occurrence to the second occurrence will be treated as input into the cat command. In this example the 'whoami' command is output, also with some text output. Along with the current host the script is running on and was last rebooted. The output this example generates is perhaps not meaningful at best, but what it does is demonstrate what a here-doc can do.

Listing 1

#!/bin/sh
cat <<mayday
Hello $(whoami)
Just to let you know the system was last rebooted:
$(who -b)
 Anyway good-bye
mayday

Let's look at another redirection example. This is using the mail command. See Listing 2. Everything from the first word “mayday” to the second occurrence of “mayday” will be treated as input to the mail command. It then emails out to myself, along with the df command output. The prtconf (AIX configuration listing) output is generated to a file and sent as an attachment with the mail. If this was in a production environment, the here-doc would most certainly contain a lot more useful information.

Listing 2

#!/bin/sh
list="david.tansley@btinternet.com"
mail -s "`hostname` Here's your report" $list <prtconf.txt) $(uuencode prtconf.txt prtconf.txt) lots more lines can go here... ... mayday 

When dealing with a here-doc it’s important to note that there should only be one redirection to a command. If you try having two redirects going into the same command, expect some spurious output.

Use a Loop to Feed The Here-doc With SSH Connection

When putting together a script to connect to many remote hosts (one at a time), it’s best to put all the remote host names into a file, then cat the file in a “while” loop or into a “for” loop for your SSH connections. If you only have a few remote hosts to connect to, then put them in a variable list and use a “for” loop to read them in. Here’s three code blocks as examples:

The remote hosts file:

# cat hosts.txt
remote_host1
remote_host2
…

1). Using cat and the while loop to feed SSH.

cat hosts.txt | while read host
do
 done 

2). Using a variable to a list of hosts to feed SSH

dest_hosts=”remote_host1 remote_host2”
for host in $dest_hosts
do
<commands>
done

3). Using a variable and the cat command to feed SSH

dest_hosts=$(cat hosts.txt)
for host in $dest_hosts
do
<commands>
done

SSH And Here-docs Are Great Partners

Using the here-doc with SSH is an efficient way to execute remote commands, like generating reports or installing software, or doing general AIX housekeeping tasks; the list is endless.

Let’s see a more constructive script using SSH incorporating a here-doc. Assume you have been given the task of collating all remote AIX hosts with their respective VG (Volume Group) sizes and free space unallocated to be all presented in GB. See Listing 3.

In the following example the remote hosts file: hosts.txt is cat into the variable $dest_host, a “for” loop then executes for each hostname. Every command contained after the word 'mayday’ to the next occurrence of ‘mayday’, is redirected into SSH. Thus the commands contained in the here-doc will be executed on the remote host. On each remote host we connect to we simply grab a list of the VG's into the variable $list, then execute another “for” loop, to get the total and free sizes of each VG on that remote host. The results are echoed out in comma separated fields which could then be redirected into a file as a CSV import file. The redirection of the output is not shown as this was previously explained earlier how to achieve this.

Listing 3.

#!/bin/sh
echo "Hostname,VG,Total GB,Free GB"
dest_host=$(cat myhosts.txt)
for host in $dest_host
do
ssh -T -l root $host<< 'mayday'
myhost=$(hostname -s)

list=$(lsvg)
for loop in $list
do
total=$(lsvg -L $loop| awk '/TOTAL PP/'| awk '{print $7}'| sed 's/^.//')
free=$(lsvg -L $loop| awk '/FREE PP/'| awk '{print $7}'| sed 's/^.//')
gb_total=$(expr $total / 1024)
gb_free=$(expr $free / 1024)
echo "$myhost,$loop,$gb_total,$gb_free"
done
mayday
done

In my office we have over 250 AIX hosts, just imagine doing that manually ! Typical output of the script in Listing 3. Could look like :

Hostname,VG,Total GB,Free GB
uk01aix9001,rootvg,112,22
uk01aix9001,apollovg,5365,242
uk01aix9001,apollo_sy_vg,3028,127
uk01aix9001,hfc_vg,496,22
uk01aix9002,rootvg,112,27
uk01aix9002,portal_vg,256,0
uk01aix9002,apollo_vdi,112,16

A Quote Will Preserve The Variables

Looking at Listing 3, note that I use single quotes around the word 'mayday'. This is required because I need to assign variables with their current values; I do not require the variables to be expanded. Thus the use of quotes. If I didn’t then I would get a load of 'not found', 'syntax error' statements thrown up when the here-doc is executed on the remote host.

Conclusion

Using here-docs especially in a SSH environment is a great way to execute many commands on a remote host, in fact it is a script with in a script if one thinks about it. I use this method all the time in my office for adding/removing users, remote editing files with sed or text processing with awk, as well as package installs or just general MIS gathering reports.