COMP 2103 Programming 3 Part 5 Jim Diamond CAR 409 Jodrey School - - PowerPoint PPT Presentation

COMP 2103 — Programming 3 Part 5

Jim Diamond CAR 409 Jodrey School of Computer Science Acadia University

226

Shell Programming

• Up until now, we have only discussed using the shell (command

interpreter) interactively – i.e., every shell command you typed was executed immediately

• The rest of the course is concerned with writing programs for the shell
• You should review slides 8 through 25 from slides-pt1.pdf before

next class

Jim Diamond, Jodrey School of Computer Science, Acadia University

227

Shell Wars

• A long time ago, in an Empire lab far, far away, there was the original

Unix command interpreter (shell), known as sh – it was both powerful and minimal – powerful in what it could do – minimal in terms of bells and whistles e.g., no (real) editing of the line you were typing, no history, no aliases (q.v.), . . .

• People started wanting more features

– people had different opinions of what they wanted to add, how things should be done, and so on –

• ne person thought the syntax should be more C-like
• A proliferation of shells occurred (csh, ksh, tcsh, bash, zsh, . . . )
• A “standard” (IEEE Std 1003.1, 2013 Edition, “POSIX”) was declared,

which defines a set of features and behaviours – see http://pubs.opengroup.org/onlinepubs/9699919799/

utilities/toc.html for a free “equivalent” copy

• We will concentrate on “POSIX-compliant” shell features

Jim Diamond, Jodrey School of Computer Science, Acadia University

228

Shell Meta-characters

• The shell provides some convenient expressions for naming files:

– “/etc/rc*” matches all files in /etc starting with “rc” – “A8P1.*” matches all files (in current dir) starting with “A8P1.” – “*” matches all (non-hidden) file names in the current directory – “????” matches all files with exactly four characters in their names – caveat: file names whose first char is “.” are not matched unless the pattern starts with a “.” (e.g., “.??*”) – “[qaz]” matches exactly one of “q”, “a” or “z”

• For example:

cat * cat out all files in this dir ls -l *.c list all .c file in this dir ls -l a9p?.[ch] list all A9 .c and .h files

• To tell the shell to NOT to (try to) match filenames to “*”, “?” or

“[...]” in a command line – precede the meta-char with “\”: “echo \*” echoes an asterisk – enclose it in single or double quotes:

echo "?"

Jim Diamond, Jodrey School of Computer Science, Acadia University

229

Shell Meta-characters: 2

• The shell† uses “{” and “}” as a short-hand notation as follows:

–

$ echo AA{a,b,c}ZZ AAaZZ AAbZZ AAcZZ

–

$ echo before AA{alice,bob,chu}ZZ after before AAaliceZZ AAbobZZ AAchuZZ after

• In other words, for each comma-separated string inside the braces, the

shell generates a token using that string and the rest of the token

• utside the braces
• Note:

ls /bin/[ab]*

is the same as ls /bin/{a,b}* – but ls /bin/{red,blue}* is not easy with [...]

• Some GEQs:

– Q: how can I get a space in one of the strings? – Q: if allowed, what does A{1,2,3}{b,c}Z generate?

† at least bash, tcsh and zsh, but not all shells

Jim Diamond, Jodrey School of Computer Science, Acadia University

230

Shell Scripts

• A shell script is just a text file with lines that can be interpreted by the

shell

• A shell script starts with a line telling the operating system which shell

to use to run the commands; for example,

#! /bin/sh

says to use /bin/sh to run this shell script –

• n some Linux distros, /bin/sh is the same as /bin/bash

–

• n other Linux distros, /bin/sh is some faster shell (e.g., dash is

used in Ubuntu) (optimized for running scripts, not user interaction) – shells are typically “installed” in the /bin directory

• Shell script lines whose first non-white char is ‘#’ are comments
• Sample shell script:

#! /bin/sh # This is a trivial but valid shell script printf "The time is now " date

Jim Diamond, Jodrey School of Computer Science, Acadia University

231

Shell Variables: 1

• Like most programming languages, the shell has variables which can

hold values – the values are strings

A value might look like a number, e.g., 123, but it is still a string

• Variables can be declared, but don’t need to be
• Variables are assigned using an equal sign, like this:

$ my_pet=dog $ weightInKgs=50

– note: you can NOT use spaces around the “=”

• You access the value of a variable (in most places!) with a “$”:

$ echo "My pet is a $my_pet and it weighs $weightInKgs Kgs" My pet is a dog and it weighs 50 Kgs

• Variables are not expanded inside single-quote strings:

$ echo ’My pet is a $my_pet and it weighs $weightInKgs’ My pet is a $my_pet and it weighs $weightInKgs

Jim Diamond, Jodrey School of Computer Science, Acadia University

232

Shell Programming Constructs: ✐❢ (1)

• This (and other) programming constructs can be typed in interactively,

but are more often used in shell scripts

if grep root /etc/passwd >/dev/null 2>&1 toss stdout, stderr then echo root is in /etc/passwd <other commands could go here> else echo root is NOT in /etc/passwd <other commands could go here too> fi

• REALLY REALLY REALLY IMPORTANT DIFFERENCE BETWEEN

SHELL AND C/Java/Python: in a shell if statement, the “condition” is not a Boolean expression – the “condition” is the return code of a program

grep in the above example

– if the program returns “success”, the if statement executes the “then” part of the statement –

• therwise, the if statement executes the else part of the

statement, if there is one Recall that Java’s main() is public static void: thus making Java programs much less useful in shell scripts

Jim Diamond, Jodrey School of Computer Science, Acadia University

233

Shell Programming Constructs: ✐❢ (2)

if grep root /etc/passwd >/dev/null 2>&1 toss stdout, stderr then echo root is in /etc/passwd <other commands could go here> else echo root is NOT in /etc/passwd <other commands could go here too> fi

• The then clause statements are executed if grep returns success

(e.g., EXIT_SUCCESS from a C program), otherwise the else clause statements are executed

• The else clause is optional, just like in C or Java or Python
• “ >/dev/null” means “throw stdout away”

– “2>&1” means “send stderr wherever stdout is going right now”

• Note that keywords are used to delimit the clauses, rather than braces

Jim Diamond, Jodrey School of Computer Science, Acadia University

234

Shell Programming Constructs: ✇❤✐❧❡

• The following construct first runs “some-program”

while some-program args do <some statements> done

– if some-program exits successfully, <some statements> are executed, and the process is repeated – if some-program returns a failure code, the while/do/done block is skipped

• You can use break to exit a while loop

– as in other programming languages, typically this would be in an if statement

• Similarly, you can use continue to finish this iteration and go back to

the while condition at the top of the loop

Jim Diamond, Jodrey School of Computer Science, Acadia University

235

Shell Programming Constructs: ❢♦r

• The following construct
• sets the variable i to arg1, runs <some statements>,
• sets i to arg2, runs <some statements>,
• and so on

– typically, $i is used in <some statements>

for i in arg1 arg2 arg3 ... do <some statements> done

• A common usage is to iterate over all files in the current directory

for f in * do ... done

• Another common usage is to iterate over all command-line arguments

for f in "$@" do ... done

Jim Diamond, Jodrey School of Computer Science, Acadia University

236

Shell Programming Constructs:

✉♥t✐❧, t❡st and the Amazing ‘

• until is like while, but repeats as long as the return code indicates

failure, rather than success

until test ‘date +%p‘ = PM do echo It is still too early to get up sleep 60 done

• date +%p outputs AM or PM, according to the time of day
• test is a very versatile command for testing all sorts of conditions

– see its man page for details

• Enclosing a command inside single back-quotes makes the shell

– run that command, and then – substitute that command’s output into the command line

• So running that until statement before noon would be equivalent to

running

until test AM = PM

Jim Diamond, Jodrey School of Computer Science, Acadia University

237

Addendum on ‘

• The ‘ ... ‘ construct works in a wide variety of shells
• However, in POSIX shells the $( ... ) construct can be used as well

– so instead of until test ‘date +%p

‘ = PM

we can write until test $(date +%p) = PM

• The ‘ ... ‘ construct works in csh and tcsh, but $( ... ) does not
• You have to decide whether you prefer

– extra portability, or – not getting abuse from bash fanatics

• It is arguably easier to nest using the $( ... ) construct

– consider cmd1 ... $(cmd2 ... $(cmd3 ...) ...) ... – vs.

cmd1 ... ‘cmd2 ... \

‘cmd3 ...\ ‘ ... ‘ ...

• ‘ ... ‘ and $( ... ) handle \ differently; in bash try

echo $(echo \\\\) ‘echo \\\\

‘

Jim Diamond, Jodrey School of Computer Science, Acadia University

238

Shell Programming Constructs: the ❝❛s❡ Statement

• A case statement can have any number of cases; the case ‘*’ matches

everything

So it is not the same as the shell meta-character “*”!

case ‘date +%H‘ in [0-9]) echo "Too early to get up" ;; 10) echo "Gotta get up soon" ;; 11) aplay AlarmClock.wav echo "Get up now!" ;; *) echo "Uh oh, overslept again" ;; esac

• [0-9] is a “shell pattern” (q.v.) which matches 0, 1, 2, . . . , 8 or 9
• ;; is like break in C;

;& “falls through” to the next case

• You can write the case labels with both parens, like

(11)

if you prefer

Jim Diamond, Jodrey School of Computer Science, Acadia University

239

Shell Scripts: Arguments

• $0 is the name of the program, like argv[0] in C or args[0] in Java
• Like C and Java programs, shell scripts can examine and use

command-line arguments –

$1 is the first argument, $2 the second, and so on, up to $9

– if argument 4 (for example) does not exist, the variable reference “$4” is replaced by the empty string

• “$*” is expanded into all arguments, separated by spaces

#! /bin/sh for i in $* do echo $0: The next argument is ${i} done

• WARNING: calling that script with arguments containing spaces

(e.g., scriptname arg1 "arg 2" ) will do something unexpected – solution: use

for i in "$@"

– homework: try this out! (GEQ)

Jim Diamond, Jodrey School of Computer Science, Acadia University

240

Writing Shell Scripts: Other Programs

• The previous slides showed enough shell constructs to write

sophisticated programs

• But. . . to harness the real power of shell scripts, you probably want to

call other programs and use their output (or return codes!) in your scripts

• Here are some very handy programs (see also an earlier slide)

–

echo: outputs its arguments (surprisingly useful!)

–

test: test all sorts of conditions

– “test ...” is usually written “[ ... ]” for visual appeal –

test is usually used with if, while and until

–

diff: compare two (text) files

–

cmp: compare two files (could be text files or binary files)

–

sed: edit standard input or a file (also surprisingly useful!)

– try this:

echo dogmatic dogmas | sed "s/dog/cat/g"

– and this:

sed < /etc/passwd "s/:.*//"

Jim Diamond, Jodrey School of Computer Science, Acadia University

241

More Useful Programs for Shell Programming

• Other useful programs include

–

tail: output the last few lines of a file

–

head: output the first few lines of a file

–

printf: print out some nicely formatted text or numbers

–

awk: a vaguely C-like language that handles strings nicely

–

expr: calculate arithmetic expressions

– e.g.,

i=

‘expr $i + 1 ‘

( POSIX: i=$((i + 1)) ) –

sort: sort a file or stdin

– e.g., output sorted list of all accounts on your system:

sed < /etc/passwd "s/:.*//" | sort

–

sleep: delay execution for a number of seconds

–

true: a program that immediately returns exit status 0 (“success”)

– guess what false does. . . –

basename, dirname: programs that manipulate filenames

Jim Diamond, Jodrey School of Computer Science, Acadia University

242

Shell Built-In Commands

• The shell has many useful commands built in

–

exit: terminate execution of this shell (return 0 as exit status)

–

exit N: terminate execution, return N as exit status

–

read i: read a string from stdin, store it in variable i

–

stdin might be a redirected file or a pipe

–

set arg1 arg2 arg3 ...: replace the current values of the

variables $1, $2, $3, . . . with arg1, arg2, arg3, . . . –

shift: move $2 to $1, $3 to $2, and so on

– this does not change $0 – useful in shell scripts which process some unknown-in-advance number of arguments –

echo: more or less (**sigh**) the same as /bin/echo

– some minor differences, such as in the use of or need for the -n and -e options, can occasionally cause you grief – many people advocate using the printf command, instead of

echo, to ensure portability

Jim Diamond, Jodrey School of Computer Science, Acadia University

243

The Also-Amazing ❡✈❛❧ Command

• Occasionally you may “construct” a string which represents some

command you wish to execute – e.g., suppose the variable my_cmd contains

sed < /etc/passwd -e "s/:.*//" -e "s/^/A userid is /"

• You can’t execute this by just typing $my_cmd

. . . well, in some shells with some options set you can

• You can, however type

$ eval $my_cmd

which causes the shell to parse and execute the string

sed < /etc/passwd -e "s/:.*//" -e "s/^/A userid is /"

• Exercise: try this

$ file=/etc/passwd $ cmd=’ls -l $file’ $ echo $cmd $ eval $cmd

– GEQ: what happens if you try

what happens in YOUR shell?

$ $cmd

without the eval ?

Jim Diamond, Jodrey School of Computer Science, Acadia University

244

The Very Dangerous ❡✈❛❧ Command

• Using eval on user input is very dangerous

– the user might maliciously or accidentally type something that will do Something Very Bad

• E.g.,

printf "What is your age? " read age ageInOctal=‘eval printf ’%o’ $a‘

• If the user types in 10, the variable ageInOctal will then contain 12
• If the user types in

10 ; rm -rf ~ DO NOT ACTUALLY TRY THIS!!!!!!!

the shell will (attempt to) recursively delete all of the files from your home directory on down – this is a Very Bad Thing

• The moral of the story: be very careful with eval

Jim Diamond, Jodrey School of Computer Science, Acadia University

245

Shell Scripts: Pre-Summary Summary

• The material on these slides probably covers 95% of the commands in

my shell scripts

• I will discuss one or two other things (e.g., shell functions) before we

finish up with shell scripts – but after the following example

• There are some other very powerful features, we might discuss them

later, given time

• Note: the eval construct is a very powerful feature which is available in

a number of interpreted languages, but is typically not found in compiled languages such as C and Java

Jim Diamond, Jodrey School of Computer Science, Acadia University

246

Sample Shell Script: Get Battery Status

• By reading some appropriate files, we can write a shell program which

will print out information about the current state of the system

• One such example of system information is the battery status
• Older versions of Linux (with ACPI power management enabled) put

information about your battery in /proc/acpi/battery/BAT0/info and /proc/acpi/battery/BAT0/state

• Newer versions of Linux put this information in a number of files in the

directory /sys/class/power_supply/BAT0 (or sometimes .../BAT or

.../BAT1 or .../CMB1 or . . . )

– Q: how would you discover which directory, if any, has information about your battery?

• Note 1: if you are running Linux inside a virtual machine you may or

may not have any access to your battery information

• Note 2: your system may provide information to you in both places

(/proc/... and /sys/...)

Jim Diamond, Jodrey School of Computer Science, Acadia University

247

Sample Shell Script: Get Battery Status 2

• At the time I wrote this slide, my

/sys/class/power_supply/BAT0/uevent file had the following

information:

POWER_SUPPLY_NAME=BAT0 POWER_SUPPLY_STATUS=Charging POWER_SUPPLY_PRESENT=1 POWER_SUPPLY_TECHNOLOGY=Li-ion POWER_SUPPLY_CYCLE_COUNT=0 POWER_SUPPLY_VOLTAGE_MIN_DESIGN=11100000 POWER_SUPPLY_VOLTAGE_NOW=12850000 POWER_SUPPLY_CURRENT_NOW=347000 POWER_SUPPLY_CHARGE_FULL_DESIGN=8400000 POWER_SUPPLY_CHARGE_FULL=4945000 POWER_SUPPLY_CHARGE_NOW=4688000 POWER_SUPPLY_CAPACITY=94 POWER_SUPPLY_MODEL_NAME=Dell POWER_SUPPLY_MANUFACTURER=SANYO POWER_SUPPLY_SERIAL_NUMBER=374

• In fact, a lot of these pieces of information are found in separate files in

that directory, but let’s use the above to make things interesting – go ahead and look around on your computer and see what you can find

Jim Diamond, Jodrey School of Computer Science, Acadia University

248

Sample Shell Script: Get Battery Status 3

• Suppose we want a script which outputs something like

Battery is n% {charging|discharging|unknown|...}

– we need to get full capacity and remaining capacity (and then do some arithmetic)

• grep is your friend:

grep CHARGE_FULL_DESIGN /sys/class/power_supply/BAT0/uevent grep CHARGE_NOW /sys/class/power_supply/BAT0/uevent grep STATUS /sys/class/power_supply/BAT0/uevent

• Those output (respectively):

POWER_SUPPLY_CHARGE_FULL_DESIGN=8400000 POWER_SUPPLY_CHARGE_NOW=4741000 POWER_SUPPLY_STATUS=Charging

• Note that grep will (by default) output all lines from the file matching the

patterns I give – I didn’t use the full name, but in this case my substrings were specific enough

Jim Diamond, Jodrey School of Computer Science, Acadia University

249

Sample Shell Script: Get Battery Status 4

• Now we have to

– get rid of extraneous text, – do some arithmetic, and –

• utput the information in the desired format
• Pick out the text we want:

grep CHARGE_FULL_DESIGN /sys/class/power_supply/BAT0/uevent \ | sed -e "s/.*=//"

– the “\” at the end of the line tells the shell that the next line is a continuation of the current line

• This says:

(1) run grep (searching for “CHARGE_FULL_DESIGN”), (2) send its output into the input of sed (“stream editor”), (3) replace (substitute) the string matching “.*=” with the empty string (the string between the second / and the third /) – “.*=” is a “regular expression” (q.v.) meaning “the longest sequence of characters there which ends with =”

Jim Diamond, Jodrey School of Computer Science, Acadia University

250

Sample Shell Script: Get Battery Status 5

• There are other ways to select a field from a string; here is one:

grep CHARGE_FULL_DESIGN /sys/class/power_supply/BAT0/uevent \ | awk -F = ’{print $2}’

– the $2 tells awk to print out the second field on each input line –

awk is a whole language unto itself in which you can do powerful

programming!

• You could also use the cut program

go read the man page!

• It will probably be convenient to store this information in a variable

– use the (amazing) back-quote (or the $(...) construct)

design=‘grep CHARGE_FULL_DESIGN \ /sys/class/power_supply/BAT0/uevent \ | awk -F = ’{print $2}’

‘

• This sets the variable “design” to 8400000
• Now do the same thing for remaining capacity;

– and store it in “left”

Jim Diamond, Jodrey School of Computer Science, Acadia University

251

Sample Shell Script: Get Battery Status 6

• Get the charging/discharging/. . . status:

– select the third token on the line (use awk), or – delete all of the text except the part we want:

charge=

‘grep CHARGE_NOW /sys/class/power_supply/BAT0/uevent \

| sed ’ s/.*=//’

‘

• Compute the percentage full:

percent=‘expr $left ’*’ 100 / $design‘

– need quotes around the “*” so that the shell does not expand it into a list of filenames – we could also “quote” it by writing

percent=‘expr $left \* 100 / $design‘

– in “POSIX-compliant” shells we could also use the built-in arithmetic evaluation:

percent=$(( $left * 100 / $design ))

– and in SOME shells (e.g., bash) we can omit the ’$’

percent=$(( left * 100 / design ))

• Now. . . put it all together

Jim Diamond, Jodrey School of Computer Science, Acadia University

252

Sample Shell Script: Get Battery Status 7

• Here is a complete shell script (except for the missing comments!!)

– save it in a file called (for example) battery-status

#! /bin/sh # File: battery-status # Author: Jim Diamond # Purpose: Output information about the current battery status # Date: November 22, 2014 # Version: 1.0 # NOTE: Blindly assumes $BAT_STATUS is the correct file. # This file must contain the battery information of interest! BAT_STATUS=/sys/class/power_supply/BAT0/uevent design=

‘grep CHARGE_FULL_DESIGN $BAT_STATUS | sed -e "s/.*=//" ‘

left=

‘grep CHARGE_NOW $BAT_STATUS | sed -e "s/.*=//" ‘

charge=

‘grep STATUS $BAT_STATUS | sed "s/.*=//" ‘

percent=

‘expr $left ’

*’ 100 / $design

‘

echo "Battery is at $percent%; state is $charge."

Jim Diamond, Jodrey School of Computer Science, Acadia University

253

Executing Shell Scripts

• A shell script can be executed two ways
• $ sh battery-status

– this works if battery-status is, in fact, a sh script –

• therwise errors will probably occur
• $ battery-status

– for this to work, battery-status must have execute permissions –

ls -l battery-status

lists the file perms of battery-status – three groups of bits (user (==owner), group, others) – each group has three bits (read, write, execute)

• Typical perms for an executable file are rwxr-xr-x

– this means everyone can read and execute the file, and only the

• wner can write to it

Jim Diamond, Jodrey School of Computer Science, Acadia University

254

Executing Compiled Programs and/or Shell Scripts

• As far as Linux/Unix is concerned, a “program” is a file which can be

executed

• For a file to be executable, the “execute” bit corresponding to you

(“file owner”, “group”, or “other”, as the case may be) must be set – the C compiler automagically sets the execute bit when it successfully creates a program – when you create a shell program with a text editor, you will have to use chmod to set the execute bit(s) –

chmod 755 my_script

gives everyone execute permission –

chmod +x my_script

gives everyone execute permission (but a shell script also needs read permission!) –

chmod 500 my_script

gives only the owner execute perms –

chmod u+x my_script

gives only the owner execute perms

• (755?? 500?? Hint: the read bit is “worth” 4, the write bit is

“worth” 2, and the execute bit is “worth” 1)

Jim Diamond, Jodrey School of Computer Science, Acadia University

255

Programming Style: Shell Scripts

• The following items will be considered as significant, both in C

programs and shell programs

SEE THE STYLE GUIDELINES ON THE COURSE WEB SITE!

• Comments

– header comment: file name, your name & student number, date, version, a concise description of the program – body comments: use them whenever a reasonably-skilled programmer would find some extra information useful – make them easy to read: grammar and spacing

• Indentation and line length

– lines must be no longer than 79 chars – indentation must be consistent through any source code file – preferably 4 spaces per indentation level – if you use tab characters, they MUST be “worth” 8 characters – if you can’t figure out how to tell your editor that an indentation level is 4, but a tab is 8, then just don’t use tabs (and/or get yourself a real editor)

Jim Diamond, Jodrey School of Computer Science, Acadia University

256

Programming Style: Shell Scripts (continued)

• Blank lines

– use one or more blank lines to separate one (logically-related) block

• f code from the following block

– don’t sprinkle them randomly through your code!

• You can designate a shell variable as “constant” by using a name

composed of UPPER_CASE_LETTERS (and digits and underscores) – but “environment” variables, such as PATH, are also upper-case by convention

• Break a shell script up into functions (next few slides) when it makes

sense to do so – just as you break a large Java (or C) program up into pieces, you should break down the design of a shell program into suitably-sized pieces and implement these pieces as functions

• See the file http://cs.acadiau.ca/~jdiamond/comp2103/

assignments/shell-coding-style-notes for more detailed style

guidelines

Jim Diamond, Jodrey School of Computer Science, Acadia University

257

Shell Functions

• Modern versions of sh allow you to structure your shell programs with

functions:

#! /bin/sh # This function complains to stderr and exits # if the argument string is null complain_and_exit_if_null () { if [ "$1" = "" ] then echo "$0: the string is null!" 1>&2 exit 10 fi } ... read -p "What is your answer? " answer complain_and_exit_if_null "$answer" ...

Jim Diamond, Jodrey School of Computer Science, Acadia University

258

Shell Functions: 2

• Note that shell functions have no formal parameters, unlike C functions

and Java methods – like the overall shell program, parameters to shell functions are positional parameters, such as $1, $2, and so on – they can be accessed within a shell function the same way as arguments to a shell program can be accessed in the “main” part of the shell program (for i in "$@", $1, shift, and so on)

Jim Diamond, Jodrey School of Computer Science, Acadia University

259

Shell Functions: 3

• Unlike Java and C, a variable used in a shell function is visible outside

the function:

#! /bin/sh my_fn() { my_var=42 # More shell code } echo "my_var is now ’$my_var’" my_fn echo "my_var is now ’$my_var’"

This outputs

my_var is now ’’ my_var is now ’42’

• To make variables local to a function, use the keyword local

#! /bin/sh my_fn() { local my_var=42 # More shell code }

Jim Diamond, Jodrey School of Computer Science, Acadia University

260

Debugging Shell Scripts

• Like any other language, you can use print statements to output

debugging info –

echo ...

• utputs to stdout

–

echo ... 1>&2

• utputs to stderr (could use just >&2 too)
• Or, you can also ask the shell to output tracing information:

$ sh -x battery-status + BAT_STATUS=/sys/class/power_supply/CMB1/uevent ++ grep CHARGE_FULL_DESIGN /sys/class/power_supply/CMB1/uevent ++ sed -e ’ s/.*=//’ + design=5800000 ++ grep CHARGE_NOW /sys/class/power_supply/CMB1/uevent ++ sed -e ’ s/.*=//’ + left=5742000 ++ grep STATUS /sys/class/power_supply/CMB1/uevent ++ sed ’ s/.*=//’ + charge=Unknown ++ expr 5742000 ’ *’ 100 / 5800000 + percent=99 + echo ’ Battery is at 99%; state is Unknown.’ Battery is at 99%; state is Unknown.

Jim Diamond, Jodrey School of Computer Science, Acadia University

261

Debugging Shell Scripts: ❧❛st❝♦♠♠

• If your Linux (or other UnixTM) system is configured to have “process

accounting”, the system can keep track of every process that is run – if you run

sudo accton on , every time a process finishes, a

record is written to a log file with various pieces of information – the lastcomm command (install package acct in Ubuntu) writes out (to stdout) the entries in the log file in reverse order

• Here are the first few lines of lastcomm output on my laptop after I ran

the previous battery-status program (Note: I wrote my own lastcomm program, the output of the usual one

• n Ubuntu is a bit different)

Command Userid TTY Utime Stime Real Flgs Exit Faults Start time sh zsd pts1 0.00 0.00 0.02 0+333 Mar 24 19:32:56 expr zsd pts1 0.00 0.00 0.00 0+166 Mar 24 19:32:56 sh zsd pts1 0.00 0.00 0.00 F 0+95 Mar 24 19:32:56 sed zsd pts1 0.00 0.00 0.00 0+179 Mar 24 19:32:56 grep zsd pts1 0.00 0.00 0.00 0+179 Mar 24 19:32:56 sh zsd pts1 0.00 0.00 0.00 F 0+95 Mar 24 19:32:56 sed zsd pts1 0.00 0.00 0.00 0+178 Mar 24 19:32:56 grep zsd pts1 0.00 0.00 0.00 0+180 Mar 24 19:32:56 sh zsd pts1 0.00 0.00 0.00 F 0+95 Mar 24 19:32:56 sed zsd pts1 0.00 0.00 0.00 0+179 Mar 24 19:32:56 grep zsd pts1 0.00 0.00 0.00 0+181 Mar 24 19:32:56

Jim Diamond, Jodrey School of Computer Science, Acadia University

262

Shell Quoting Rules: Review

• “Words” that are not inside pairs of "s or ’

s are examined by the shell – e.g., *, ?, $, <, . . . are all “interesting” to the shell

• White space is not (identically) preserved unless inside a string

– “

echo a b

” gives the same results as “

echo a b

”

• Shells assign different meanings to strings enclosed in quotes

– no interpretation is done to strings in single-quotes: –

echo ’ This is a * and a $dollar sign’

• utputs

“This is a

* and a $dollar sign”

– variable interpolation is done to strings in double-quotes: –

echo "This is a * and a $dollar sign"

• utputs

“This is a

* and a sign”

–

• ne can also “quote” a special char with a \:

–

echo This is a \* and a \$dollar sign

• utputs

“This is a * and a $dollar sign”

Jim Diamond, Jodrey School of Computer Science, Acadia University

263

Quick And Dirty GUI Interaction

• Say you know your shell script is running under the X window system

– suppose you want to put up a message in a window; try this:

xmessage An informational message

• This program can also return information:

xmessage -buttons Yes:0,No:1,Maybe:2 \

’

Should we have more assignment questions?’

• In the above example, if the user clicks

– the Yes button, xmessage returns 0 – the No button, xmessage returns 1 – the Maybe button, xmessage returns 2

• kdialog is similar, “prettier” and noisier, but less portable

– e.g., kdialog -yesno "Shutdown the system?"

• For more fully-featured GUIs for use with a scripting language, consider

“wish” (the Tcl script language with the Tk toolkit)

Jim Diamond, Jodrey School of Computer Science, Acadia University

264

Variables: Four Types

• Environment variables known by a given program prog are exported to

any programs that prog calls – use the printenv or env commands to see your environment variables – by convention, environment variables use UPPER CASE letters – a variable becomes an environment variable if you export it: – e.g.,

NAME=value ; export NAME

• Local variables: visible only within a code block or function, like local

variables in C or Java – e.g.,

local local_var=value

• Positional parameters: $1, $2, . . .
• Other variables exist as long as the shell where they are defined is

running† – that could be a shell executing a script or an interactive shell

† Not the whole truth: check out unset

Jim Diamond, Jodrey School of Computer Science, Acadia University

265

Bash Argument Parsing: Example

• Suppose you have a program whose usage is defined to be

prog <-a|-b|-c> [-d word] [file]*

– this meta-syntax means – that we must use exactly one of -a, -b and -c – we can optionally use “-d word” – we can have zero or more file arguments

• Q: how can we program a shell script to parse arguments according to

this requirement?

• The next two slides show one way to parse args, assuming the options

come in any order, but before file arguments

Jim Diamond, Jodrey School of Computer Science, Acadia University

266

Bash Argument Parsing: General Structure

<give default values to option variables> <initialize any flag variables> while [ $# -gt 0 ] do <examine arg $1 with "if" or "case" and deal with it> # Toss away that argument and move on to the next: shift done <do any required "sanity checking" here>

• Note: every time you shift, the value of $# is automagically

decremented

Jim Diamond, Jodrey School of Computer Science, Acadia University

267

❜❛s❤ Argument Parsing: Example Continued

abc_flag= while [ $# -gt 0 ] do case "$1" in

• a|-b|-c)

if [ "$abc_flag" != "" ] then echo "Error: more than one of -a, -b and -c used" 1>&2 usage ; exit 1 # jammed on one line so slide fits fi abc_flag="$1" ;;

• d)

shift if [ "$1" = "" ] then echo "Error: -d requires an argument" 1>&2 usage ; exit 2 fi d_arg="$1" ;;

• *)

echo "Error: unknown argument \"$1\"" 1>&2 usage ; exit 3 ;; *) files="$files $1" # Bug if a file name has white space! ;; esac shift done if [ "$abc_flag" = "" ] then echo "Error: need one of -a, -b or -c" 1>&2 usage ; exit 4 fi

Jim Diamond, Jodrey School of Computer Science, Acadia University

268

Redirecting Input In a Loop: 1

• Suppose you want to redirect input inside a loop to read from a file
• This won’t do what you (probably) want

Why not? GEQ!

while read line < file do # do something with $line here... done

• But this will do what you probably want:

while read line do # do something with $line here... done < file

– this reads all the data in file, one line at a time, until EOF (unless you break out somewhere) – you could also have input statements inside the loop body

• But what if you want to get input from another program instead of

a file?

Jim Diamond, Jodrey School of Computer Science, Acadia University

269

Redirecting Input In a Loop: 2

• You can read input from another program in a loop like this:

some-prog args | while read line do # do something with $line here... done

• But when you pipe a program’s output into a while loop, the loop is

executed in a “sub-shell”

in bash, but not in ksh or zsh

– any variable modifications inside the loop aren’t seen after the loop:

myvar=1 some-prog arg1 arg2 arg3 | while read line do myvar=2 # do something with $line here... done echo "this will echo 1, not 2: $myvar"

Jim Diamond, Jodrey School of Computer Science, Acadia University

270

Redirecting Input In a Loop: 3

• How to deal with the problem on the last slide? At least these ways:
• 1: (poor solution) save output of some-prog in a temp file, use file

redirection – Q: why is this a poor solution (at least 2 reasons)?

• 2: control the sub-shell creation yourself, as follows:

some-prog args | \ ( while read line do # do something with $line here and set other vars done # use those other variables here )

• 3: put the loop and related code in a function (say my_fun), and then

call the function like this

some-prog args | my_fun

Jim Diamond, Jodrey School of Computer Science, Acadia University

271

Using Temporary Files

• When solving a problem, frequently you will need to create some data

which is needed only temporarily

• You could put it in a temp file

– but often, you can avoid the use of temporary files by using pipes

• If you use a temp file, keep these things in mind:

(1) your program should NOT leave temp files lying around after it no longer needs them (2) your program SHOULD DEFINITELY ABSOLUTELY FER SHURE NEVER EVER clobber a file that might already exist; e.g.,

#! /bin/sh echo "data header" > output.txt # NO! NO! BAD DOG!

(unless, of course, the script is supposed to create or update that specific file)

• Use mktemp to safely create a temp file, and write code (see next slide)

to make sure it gets deleted when the shell script terminates

Jim Diamond, Jodrey School of Computer Science, Acadia University

272

Temp Files, Trapping Exit and “Signals”

• Suppose you write a shell script which creates some temporary files (see

“man mktemp”) and you want those files deleted when the script exits –

• ne “solution”: just use rm <that file> at the end of the script

(and before any “exit” command)

• Problems with this:

– you have to remember to do this before every “exit” – if someone ^C’s the script, the rm command won’t be executed

• Solution: use the “trap” command:

–

trap "/bin/rm $tmpfile; exit" 0

–

rm that file whenever we exit normally

–

trap "/bin/rm $tmpfile; exit" 2

–

rm that file whenever we are ^C’ed

–

trap "/bin/rm $tmpfile; exit" 0 1 2 3 15

–

rm that file in all those situations (exit, HUP, INT, QUIT, TERM)

• Note: if the argument to trap doesn’t say to exit, then the shell

doesn’t (immediately) exit; thus, for example, shell programs can be written to “ignore” ^C

Jim Diamond, Jodrey School of Computer Science, Acadia University

273

Trapping “Signals” Such As ^C

• The following program will trap ^C’s and keep looping

#! /bin/sh trap "echo no way!" 2 while true do read -p "Enter anything: " response echo "response was $response" done

• Here are three ways to kill such a program:

– kill the script from another shell with the kill command (see man ps and man 7 signal); – kill the shell window running the program (“BFI” solution); or – do the following: – “stop” the script program with the terminal suspend char (^Z by default, you can change that) – list your “jobs” with the jobs command – kill the program with kill -9 %n (n is probably 1)

• See “man 7 signal” (or COMP 3713) to learn a bit about signals

Jim Diamond, Jodrey School of Computer Science, Acadia University

274

Shell Aliases (s❤✴❜❛s❤✴③s❤ Examples)

• Aliases allow you to define your own names for commands
• Aliases allow you to create “custom” commands using other commands
• Normally, these are put in a shell init file (see the “Byzantine” slide)
• alias ty=less

– then “ty /etc/passwd” displays /etc/passwd using less

• alias ll="ls -l"

–

ll [<file>*] gives a “long” listing [of the specified file(s)]

• alias hg=’

history | grep’

– then hg <string> outputs the lines in the shell’s history buffer containing <string>

NOTE: this conflicts with the hg program!

• The alias feature in tcsh is far, far more powerful (IMHO)!

– that is one of the primary reasons I like tcsh for interactive use

• The alias feature in zsh is also more powerful than in bash

– i.e., it can do more than I showed here

Jim Diamond, Jodrey School of Computer Science, Acadia University

275

Shell Aliases and Function Example: Set the Xterm Title Bar Text to Your CWD

if [ "$TERM" = "xterm" ]

(any xterm-compatible terminal)

then alias cd=zcd alias push=zpushd zcd () { \cd $1 # could use ’builtin cd’ echo -n "^[]2;‘hostname‘:‘/bin/pwd‘^G" } alias pop=’popd; cd .’ zpushd () { pushd $1 echo -n "^[]2;‘hostname‘:‘/bin/pwd‘^G" } zcd . # initialize window title fi

• Note 1: ‘^[ ’ is the escape character, ‘^G’ is Ctrl-G
• Note 2: ‘\’ is used in zcd() to avoid an infinite loop

– the ‘\’ tells the shell not to do alias processing there

• Note 3: there are other (better?) ways to do this in bash

Jim Diamond, Jodrey School of Computer Science, Acadia University

276

A Shell Peculiarity and “✿”

• The following is not valid

if [ <condition> ] then # do nothing else prog args prog2 ... fi

– the shell does not like a then clause with no (executable) statements

• Solution: there is a shell built-in command called ‘:’ which is a

“no-op”; it is a valid command, so do this

if [ <condition> ] then : do nothing else ...

• Alternatively, reverse the sense of the comparison, e.g.,

if [ ! <condition> ] ...

Jim Diamond, Jodrey School of Computer Science, Acadia University

277

“Sourcing” Other Script Files

• Suppose shell script A executes shell script B

– any variable definitions or modifications in B are not seen in A – thus executing a script can not be used to define or modify variables in the calling script

• But if A “sources” B, then variable changes in B are seen by A
• Syntax: to source a file of shell commands, use the (obnoxiously-named)

“.” command (or the source command):

. <some-file> # or source <some-file>

– this is used extensively by the startup scripts on most Linux distros – see /etc/init.d and/or /etc/rc.d, for example – also used by many scripts which run various services – e.g., power management software, network config software, . . .

• Note: if A sources file B, and some line in B runs the shell’s exit

command, then A also exits – thus files designed to be sourced should (normally) not use the exit command

Jim Diamond, Jodrey School of Computer Science, Acadia University

278

❡①❡❝: Start Another Program

• When the shell executes another program, normally the following

happens: – the shell creates a clone of itself (with the fork() system call) – the clone replaces itself with the desired program – unless the command ended with &, the original shell waits for the

• ther program to finish
• Sometimes (rarely in an interactive shell, sometimes in a shell script) we

want the shell to replace itself without going through the cloning procedure – e.g., suppose the last program a shell script calls will run for a long time, the shell has nothing else to do, no point in it hanging around

• The command line

$ exec program

tells the shell to replace itself with “program” – if you do this from a terminal window, when program finishes the terminal window will automagically close (e.g., try exec sleep 10)

Jim Diamond, Jodrey School of Computer Science, Acadia University

279

❡①❡❝: Open and Close Files

• By default, the shell has three files open when it starts:

–

stdin, on file descriptor 0

–

stdout, on file descriptor 1

–

stderr, on file descriptor 2

Thus echo blah 1>&2 sends stdout to stderr

• The shell can open other files for reading or writing:

–

exec 4<some-infile opens the file some-infile for reading

–

exec 5>some-outfile opens the file some-outfile for writing

–

exec 6>>a_file opens the file a_file for writing, but rather than

throwing away the previous contents (if any), data written to the file is appended to the current contents

• Having opened a file for output on file descriptor 9 (say), we can write

to it as follows

some-program arg arg arg >&9

– similarly, read from file descriptor 8 like this:

read line <&8

• We can close an opened file as follows:

exec 4<&-

Jim Diamond, Jodrey School of Computer Science, Acadia University

280

Shell Programming Summary: 1a I/O Redirection and Pipes

• Redirect a command’s stdout to a file, stderr to another file

command [args]* >file 2>errors

• r append the stdout output to a file:

command [args]* >>file

• Redirect a command’s stdout output to stderr:

command [args]* 1>&2

• “Throw away” a command’s diagnostic (stderr) output:

command [args]* 2>/dev/null

• Redirect stderr to wherever stdout is going right now:

command [args]* 2>&1

• Redirect stderr to where stdout is going now (normally your screen),

and then throw stdout away

command [args]* 2>&1 >/dev/null

thus you only see the stderr output – note: command [args]* >/dev/null 2>&1 throws all output away – the order of redirections is significant!

Jim Diamond, Jodrey School of Computer Science, Acadia University

281

Shell Programming Summary: 1b I/O Redirection and Pipes

• Run a command (which reads from stdin) so that it gets its input from

a file called “some-file”:

command [args]* < some-file

• Pipe one command’s stdout into another command’s stdin

command1 [args]* | command2 [args]*

Jim Diamond, Jodrey School of Computer Science, Acadia University

282

Shell Programming Summary: 2 Variables, Conditionals, Iteration

• Set and access a variable:

var=value

and

... $var ...

• Use the exit status of one program to conditionally do something

–

command [args]* || <command2 to do on failure>

–

command [args]* && <command2 to do on success>

–

while/if/until command [args]* ...

–

while/if/until command1 [args]* || command2 [args]*

– this uses the failure/success code of the LAST command in the pipeline to control looping

• Repeat a block of code for each item in a set of items:

for i in <list> do ... statements which probably use $i ... done

Jim Diamond, Jodrey School of Computer Science, Acadia University

283

Shell Programming Summary: 3 Special Characters for Naming Files

• Filename pattern matching (!= REGULAR EXPRESSIONS! (q.v.)):

– ‘*’ matches 0 or more characters of a file name – e.g., “/etc/a*” — all files in /etc starting with ‘a’ – e.g., “*.c” — all “.c” files in the current directory – ‘?’ matches exactly one character of a file name – e.g., “d?g” — all three-letter file names in the current directory starting with ‘d’ and ending with ‘g’ – “[a-eg-iz]” matches exactly one character from the set

{a, b, c, d, e, g, h, i, z}

– e.g., “d[iou]g” matches “dig”, “dog” and “dug” (if they exist)

• ‘~’ at the beginning of a file name is special:

– “~0123456t/A1P1” means file (or directory) “A1P1” in the home directory of user 0123456t

• Example: all ‘.’ files in your home dir with at least three characters in

the filename: “~/.??*”

Jim Diamond, Jodrey School of Computer Science, Acadia University

284

Shell Programming Summary: 4 Handling of Shell Wildcards and Other Special Characters

• The shell parses each line to “understand” what the line means
• As we have seen, some characters (such as “|”, “<”, “*”, “[”, . . . )

have special meaning to the shell

• You may want to use one (or more) of those characters literally

– to do so, you need to quote the character

• Quoting a character can be done in a number of ways

– put the character in a single-quote string: $ echo ’

...$...’

– this works for all characters except “’ ” itself – put the character in a double-quote string: $ echo "...*..." – this does not quote ‘, $, " or always \ – precede the character by a \: $ echo 23 \* 17 | bc

• CAUTION: some other programs (e.g., grep) also use \ to quote their

special chars! – e.g., to search for the string “\input” I could use

$ grep \\\\input slides-pt5.tex

Why 4 ’s? Why not 2?

Jim Diamond, Jodrey School of Computer Science, Acadia University

285

Shell Programming Summary: 5 Brace Expansion (Only Some Shells!)

• See the “Brace Expansion” section in the bash man page for a fuller

explanation

• (Paraphrasing the bash man page:)

– a mechanism by which arbitrary strings may be generated – similar to pathname expansion, but the filenames generated need not exist –

• ptionally preceded by a prefix; optionally followed by a suffix

– the pattern is either – comma-separated strings (e.g., {bat,cat,rat}), or – a sequence expression (e.g., {1..20..3})

• e.g., echo A{bat,cat,rat}B

gives

AbatB AcatB AratB

• e.g., echo A{1..20..3}B

gives

A1B A4B A7B A10B A13B A16B A19B

– the ..3 section is optional, with a default step of 1 – can also use single chars: file{a..i} gives

filea fileb filec filed filee filef fileg fileh filei

• The first form of the brace usage was stolen from csh/tcsh, the second doesn’t work in those shells

Jim Diamond, Jodrey School of Computer Science, Acadia University

286

Shell Programming Summary: 6 Program Arguments

• Arguments for Shell Programs

–

$# is the number of arguments (not counting $0)

–

$0 is the name the program was called by

–

$1 ... $9 are the first nine arguments

–

shift throws away $1, moves all the other arguments one place

• ver, and decrements $#

– after shift, $1 is whatever $2 was before, $2 is whatever $3 was before, and so on – if there were ten (or more) arguments before, the tenth argument will now be $9 – like other variables, if an argument doesn’t exist the shell will replace a reference to it with the empty string – e.g., if $# is 4, $7 is the empty string

Jim Diamond, Jodrey School of Computer Science, Acadia University

287

Shell Programming Summary: 7 Sub-shells

• Use sub-shells to group actions together
• Example: run some programs in one directory and some in another

cd from-dir ; tar cf - . | (cd to-dir ; tar xf -)

– this copies a file hierarchy from one place to another

• Example: group a number of outputs together to make one input

(echo 0; ls -l | tail +2 | awk ’{print $5 " +"}’; echo p) | dc

– this sums the sizes (in bytes) of the files in the current directory

• Example: pipe output of a program into a loop construct:

prog args | \ ( <initialization> while <"condition" which might do input> do <statements which might do input> done <other statements using vars from inside loop> )

Jim Diamond, Jodrey School of Computer Science, Acadia University