Bourne Shell Scripting/Variable Expansion

In the Environment module we introduced the idea of an environment variable as a general way of storing small pieces of data. In this module we take an in-depth look at using those variables: 'variable expansion', 'parameter substitution' or just 'substitution'.

Substitution

The reason that using a variable is called substitution is that the shell literally replaces each reference to any variable with its value. This is done while evaluating the command-line, which means that the variable substitution is made before the command is actually executed.

The simplest way of using a variable is the way we've already seen, prepending the variable name with a '$'. So for instance:

Simple use of a variable

Code:

$ USER=JoeSixpack
$ echo $USER

Output:

JoeSixpack

The value JoeSixpack is substituted for $USER before the echo command is executed.

Of course, once the substitution is made the result is still just the text that was in the variable. The interpretation of that text is still done by whatever program is run. So for example:

Variables do not make magic

Code:

$ USER=JoeSixpack
$ ls $USER

Output:

ls: cannot access JoeSixpack: No such file or directory

Just because the text came from a variable, doesn't mean the file exists.

Basic variable expansion is already quite flexible. You can use it as described above, but you can also use variables to create longer strings. For instance, if you want to set the log directory for an application to the "log" directory in your home directory, you might fill in the setting like this:

$HOME/log

And if you're going to use that setting more often, you might want to create your own variable like this:

LOGDIR=$HOME/log

And, of course, if you want specific subdirectories for logs for different programs, then the logs for the Wyxliz application go into directory

$LOGDIR/Wyxliz/

Substitution forms

The Bourne Shell has a number of different syntaxes for variable substitution, each with its own meaning and use. In this section we examine these syntaxes.

Basic variable substitution

We've already talked at length about basic variable substitution: you define a variable, stick a '$' in front of it, the shell substitutes the value for the variable. By now you're probably bored of hearing about it.

But we've not talked about one situation that you might run into with basic variable substitution. Consider the following:

Adding some text to a variable's value

Code:

$ ANIMAL=duck
$ echo One $ANIMAL, two $ANIMALs

Output:

One duck, two

Uhhh.... we're missing something...

So what went wrong here? Well, obviously the shell substituted nothing for the ANIMAL variable, but why? Because with the extra 's' the shell thought we were asking for the non-existent ANIMALs variable. But what gives there? We've used variables in the middle of strings before (as in '/home/ANIMAL/logs'). But an 's' is not a '/': an 's' can be a valid part of a variable name, so the shell cannot tell the difference. In cases where you explicitly have to separate the variable from other text, you can use braces:

Adding some text to a variable's value, take II

Code:

$ ANIMAL=duck
$ echo One $ANIMAL, two ${ANIMAL}s

Output:

One duck, two ducks

That's better!

Both cases (with and without the braces) count as basic variable substitution and the rules are exactly the same. Just remember not to leave any spaces between the braces and the variable name.

Substitution with a default value

Since a variable can be empty, you'll often write code in your scripts to check that mandatory variables actually have a value. But in the case of optional variables it is usually more convenient not to check, but to use a default value for the case that a variable is not defined. This case is actually so common that the Bourne Shell defines a special syntax for it: the dash. Since a dash can mean other things to the shell as well, you have to combine it with braces — the final result looks like this:

${varname[:]-default}

* Where varname is the name of the variable
and default is the value used if varname is not defined

Again, don't leave any spaces between the braces and the rest of the text. The way to use this syntax is as follows:

Default values

Code:

$ THIS_ONE_SET=Hello
$ echo $THIS_ONE_SET ${THIS_ONE_NOT:-World}

Output:

Hello World

Compare that to this:

Default not needed

Code:

$ TEXT=aaaaaahhhhhhhh
$ echo Say ${TEXT:-bbbbbbbbbb}

Output:

Say aaaaaahhhhhhhh

Without colon a variable will be substituted only if it is not defined:

Default with and without colon

$ VAR=value EMPTY_VAR=
$ echo ${VAR:-empty}, ${EMPTY_VAR:-empty}, ${ABSENCE_VAR:-empty}
value, empty, empty
$ echo ${VAR-empty}, ${EMPTY_VAR-empty}, ${ABSENCE_VAR-empty}
value, , empty

Substitution with default assignment

As an extension to default values, there's a syntax that not only supplies a default value but assigns it to the unset variable at the same time. It looks like this:

${varname[:]=default}

* Where varname is the name of the variable
and default is the value used and assigned if varname is not defined

As usual, avoid spaces in between the braces. Here's an example that demonstrates how this syntax works:

Default value assignment

$ echo $NEWVAR

$ echo ${NEWVAR:=newval}
newval
$ echo $NEWVAR
newval

Without colon a variable will be assigned only if it is not defined:

Default value assignment

$ echo $NEWVAR

$ EMPTY_VAR=
$ unset ABSENCE_VAR
$ echo ${EMPTY_VAR:=empty}, ${ABSENCE_VAR:=empty}
empty, empty
$ echo ${EMPTY_VAR}, ${ABSENCE_VAR}
empty, empty

$ EMPTY_VAR=
$ unset ABSENCE_VAR
$ echo ${EMPTY_VAR=empty}, ${ABSENCE_VAR=empty}
, empty
$ echo ${EMPTY_VAR}, ${ABSENCE_VAR}
, empty

Substitution for actual value

This substitution is sort of a quick test to see if a variable is defined (and that's usually what it's used for). It's sort of the reverse of the default value syntax and looks like this:

${varname[:]+substitute}

* Where varname is the name of the variable
and substitute is the value used if varname is defined

This syntax returns the substitute value if the variable is defined. That sounds counterintuitive at first, especially if you ask what is returned if the variable is not defined — and learn that the value is nothing. Here's an example:

Actual value substitution

$ echo ${NEWVAR:+newval}

$ NEWVAR=oldval
$ echo ${NEWVAR:+newval}
newval

So what could possibly be the use of this notation? Well, it's used often in scripts that have to check whether lots of variables are set or not. In this case the fact that a variable has a value means that a certain option has been activated, so you're interested in knowing that the variable has a value, not what that value is. It looks sort of like this (pseudocode, this won't actually work in the shell):

Default value assignment

if ${SPECIFIC_OPTION_VAR:+optionset} == optionset then ...

Without colon a variable will be substituted also if it is empty:

Default value assignment

$ VAR=value EMPTY_VAR=
$ echo ${VAR:+newvalue}, ${EMPTY_VAR:+newvalue}, ${ABSENCE_VAR:+newvalue}
newvalue, ,
$ echo ${VAR+newvalue}, ${EMPTY_VAR+newvalue}, ${ABSENCE_VAR+newvalue}
newvalue, newvalue,

Substitution with value check

This final syntax is sort of a debug check to check whether or not a variable is set. It looks like this:

${varname[:]?message}

* Where varname is the name of the variable
and message is the printed if varname is not defined

With this syntax, if the variable is defined everything is okay. Otherwise, the message is printed and the command or script exits with a non-zero exit status. Or, if there is no message, the text "parameter null or not set" is printed.

You can use this syntax to check that the mandatory variables for your scripts have been set and to print an error message if they are not.

Default value assignment

$ echo ${SOMEVAR:?has not been set}
-sh: SOMEVAR: has not been set
$ echo ${SOMEVAR:?}
-sh: SOMEVAR: parameter null or not set

With colon variable will be checked for emptiness also:

Default value assignment

$ EMPTY_VAR=
$ echo ${EMPTY_VAR:?has not been set or empty}
-sh: EMPTY_VAR: has not been set or empty
$ echo ${ABSENCE_VAR:?has not been set or empty}
-sh: ABSENCE_VAR: has not been set or empty
$ echo ${EMPTY_VAR?has not been set or empty}

$ echo ${ABSENCE_VAR?has not been set or empty}
-sh: ABSENCE_VAR: has not been set or empty