Bash

From Omnia
(Redirected from Bash Scripting)
Jump to navigation Jump to search

Bash Programming

BASH Programming - Introduction HOW-TO

Bash Guide for Beginners

Advanced Bash-Scripting Guide - An in-depth exploration of the art of shell scripting - by Mendel Cooper

"This tutorial assumes no previous knowledge of scripting or programming, but progresses rapidly toward an intermediate/advanced level of instruction . . . all the while sneaking in little nuggets of UNIX® wisdom and lore. It serves as a textbook, a manual for self-study, and a reference and source of knowledge on shell scripting techniques. The exercises and heavily-commented examples invite active reader participation, under the premise that the only way to really learn scripting is to write scripts. This book is suitable for classroom use as a general introduction to programming concepts."

--

EnglishFrontPage - Greg's Wiki - http://mywiki.wooledge.org/EnglishFrontPage

"This is Greg's (also known as GreyCat's) wiki. It has some pages which may be of interest for people doing Unix shell scripting or system administration. Its official front page URL is http://mywiki.wooledge.org/."

Bash Prompt

Customize your Bash prompt - makandropedia - http://makandracards.com/makandra/1090-customize-your-bash-prompt

Git Example

GIT adds a __git_ps1 function that will output your branch. You can add onto it by specifying a string argument. A '%s' in the string will be replaced by the branch. I configured it to also show the short hash. Below is how I did it:

get_sha() {
    git rev-parse --short HEAD 2>/dev/null
}

get_hg_id() {
    id="$(hg id -bt 2> /dev/null| sed -r 's/[\(\)]+//g')"
    if [ -n "$id" ]; then
        echo "($id)"
    fi
}

PS1='${debian_chroot:+($debian_chroot)}'
PS1=$PS1'\[\033[00;32m\]\u@\h\[\033[00m\]:'
PS1=$PS1'\[\033[02;48m\]\w\[\033[00m\]'
PS1=$PS1'\[\033[00;35m\]$(__git_ps1 "(%s $(get_sha))")$(get_hg_id)\[\033[00m\]'
PS1=$PS1'\[\033[01;38m\]\$\[\033[00m\] '

Bash script header

" The first line of the script, starting with "#!" (called pound-bang), is special--it tells the shell what program should be used to interpret my script." [1]

#!/bin/bash

Show debug information

#!/bin/bash -x

Return Values

exit 1

echo $?

Variables

NAME=hello
echo $name
echo ${name}

Variable by string name:

SDA=/dev/sda
SDB=/dev/sdb
device=SDA
echo ${!device}  # returns /dev/sda

Bash Special Parameters Variables

These are $? $! $0 $1 - $9

# man bash
   Special Parameters
       $?      Expands to the status of the most recently executed foreground pipeline.
       $!      Expands to the process ID of the most recently executed background (asynchronous) command. (PID)
       $0      Expands to the name of the shell or shell script.
       $1 - $9 Expands to the number of positional parameters in decimal.
       $@      Expands to all positional parameters

Arrays

Expansion Array Simple:

ls myfile.{txt,jpg}
# equivalent to:
# ls myfile.txt myfile.jpg

Expansion Array Range:

echo my-{1..100}
# equivalent to:
# echo my-1 my-2 ... my-100

Array:

a=()  # empty
b=(1 2 3)

Dereference array: (one based)

echo ${b[1]} = 2

Dereference array in for loop:

for i in "${mylist[@]}" ; do ... done

Add item:

mylist+=("$i")

Capture command input

DBS=`mysql -uroot  -e"show databases"`
for b in $DBS ;
do
  mysql -uroot -e"show tables from $b"
done

Subcommands

echo $( echo "test" )
{ echo "hi"; xyz; }
OF=/var/my-backup-$(date +%Y%m%d).tgz

If

if [ "foo" = "foo" ]; then
  echo expression evaluated as true
else
  echo expression evaluated as false
fi
if [ $? -ne 0 ]; then ... ; fi
mybool=true
if $mybool ; then echo "yes" ; else echo "no" ; fi
if [...]; then
  ...
elif [...]; then
  ...
else
  ...
fi

Refering to use if [ $1 = $2 ]. This is not quite a good idea, as if either $S1 or $S2 is empty, you will get a parse error. x$1=x$2 or "$1"="$2" is better.

Test

T1="a"
T2="b"
if [ "$T1" = "$T2" ]; then

Test if parameter passed

if [ -z "$1" ]; then 
  echo usage: $0 directory
  exit
fi

Test if parameter unset verses empty "":

if [ -z ${TEST} ] ; then ... fi  # true if unset or empty ""
if [ -z ${TEST+x} ] ; then ... fi  # true if unset only
if [ ! -z ${TEST+x} ] ; then ... fi  # true if empty "" or populated
# test with:
#   unset TEST
#   TEST=""

Check for Empty Variable

[ -z "$var" ] || echo "Empty"
[[ -z "$var" ]] || echo "Empty"
# Check if $var is set using ! i.e. check if expr is false ##
[ ! -z "$var" ] && echo "Empty"
[[ ! -z "$var" ]] && echo "Empty"
[  -z "$_JAIL" ] && echo "No" || echo "Yes"

References:

For Loop

for i in $( ls ); do
  echo item: $i
done

C-like for

for i in `seq 1 10`; do
  echo $i
done

Better C-like for loop [2]

for ((i=100;i<=115;i+=1)); do
 echo $i
done

Note 'break' and 'continue' can be used to exit/continue a for loop.

Alternate for format: [3]

for i in {3..5} ; ...

unset

To unset a variable:

unset myvar

To check for an empty or unset variable:

if [ -z $myvar ] ; then ... fi
if [ "$myvar" = "" ] ; then ... fi
if [ "${myvar-unset} = "unset" ] ; then ... fi
echo ${myvar:-mydefault}  # or just return if empty to something using default set
${myvar:=mydefault}  # or just set empty to something using default set

Case (switch)

start() {
...
} 
case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart | reload)
    stop
    sleep 3
    start
    ;;
  status)
    status
    ;;
  *)
    echo $"Usage: $0 (start|stop|restart|status)"
    exit 1
esac

While Loop

COUNTER=0
while [  $COUNTER -lt 10 ]; do
  echo The counter is $COUNTER
  let COUNTER=COUNTER+1 
done
COUNTER=20
  until [  $COUNTER -lt 10 ]; do
  echo COUNTER $COUNTER
  let COUNTER-=1
done

Functions

Simple Function:

function hello {
  echo "hello"
}
hello

Exit Program:

function quit {
  exit 1
}
quit
echo "hello"

Parameters:

function param {
  echo $1
}
param hello

Local variables:

HELLO=bye
function test {
  local HELLO=World
  echo $HELLO
}
echo $HELLO
test
echo $HELLO

System Arguments and Parameters

#!/bin/sh
echo "Program Name: $0"
echo "First argument: $1"
echo "All arguments: $*"  # single word
echo "All arguments: $@"  # separate quoted strings
echo "Argument Count: $#" # actual count, does not include $0

getopts

The getopts construct uses two implicit variables. $OPTIND is the argument pointer (OPTion INDex) and $OPTARG (OPTion ARGument) the (optional) argument attached to an option. A colon following the option name in the declaration tags that option as having an associated argument. [4]

A getopts construct usually comes packaged in a while loop, which processes the options and arguments one at a time, then increments the implicit $OPTIND variable to point to the next.

  1. The arguments passed from the command-line to the script must be preceded by a dash (-). It is the prefixed - that lets getopts recognize command-line arguments as options. In fact, getopts will not process arguments without the prefixed -, and will terminate option processing at the first argument encountered lacking them.
  2. The getopts template differs slightly from the standard while loop, in that it lacks condition brackets.
  3. The getopts construct is a highly functional replacement for the traditional getopt external command.

WARNING: Options must come before other arguments, or they won't be processed!

while getopts "abcde:fg" Option
# Initial declaration.
# a, b, c, d, e, f, and g are the options (flags) expected.
# The : after option 'e' shows it will have an argument passed with it.
do
  case $Option in
    a )  # Do something with variable 'a'.
         ;;
    ...
    e )  # Do something with 'e', and also with $OPTARG,
         # which is the associated argument passed with option 'e'.
         echo "Option -e- with argument \"$OPTARG\"   [OPTIND=${OPTIND}]"
         ;;
    * )  echo "Unimplemented option chosen.";;   # Default.
  esac
done
shift $(($OPTIND - 1))
# Move argument pointer to next so $1 points to not opt argument

Reboot Example: (by Kenneth)

HARDREBOOT=true

while getopts "a" Option
do
  case $Option in
    s )  HARDREBOOT=false  ;;
    * )  exit 1 ;;
  esac
done
shift $(($OPTIND - 1))

if [ $# -ne 2 ] ; then
  echo "Usage: $0 [-s:SOFTREBOOT] <SERVER> <CARD>"
  exit 1
fi

SERVER=$1
CARD=$2

if $HARDREBOOT ; then
  #...
fi

Help:

       getopts optstring name [args]
              getopts is used by shell procedures to parse positional  parame-
              ters.   optstring  contains  the  option characters to be recog-
              nized; if a character is followed by  a  colon,  the  option  is
              expected  to have an argument, which should be separated from it
              by white space.  The colon and question mark characters may  not
              be  used as option characters.  Each time it is invoked, getopts
              places the next option in the shell variable name,  initializing
              name if it does not exist, and the index of the next argument to
              be processed into the variable OPTIND.  OPTIND is initialized to
              1  each  time  the  shell or a shell script is invoked.  When an
              option requires an argument, getopts places that  argument  into
              the  variable OPTARG.  The shell does not reset OPTIND automati-
              cally; it must be  manually  reset  between  multiple  calls  to
              getopts within the same shell invocation if a new set of parame-
              ters is to be used.

              When the end of options is encountered,  getopts  exits  with  a
              return  value  greater than zero.  OPTIND is set to the index of
              the first non-option argument, and name is set to ?.

              getopts normally parses the positional parameters, but  if  more
              arguments are given in args, getopts parses those instead.

              getopts  can  report errors in two ways.  If the first character
              of optstring is a colon, silent error  reporting  is  used.   In
              normal  operation  diagnostic  messages are printed when invalid
              options or missing option arguments  are  encountered.   If  the
              variable  OPTERR  is  set  to  0, no error messages will be dis-
              played, even if the first character of optstring is not a colon.

              If an invalid option is seen, getopts places ? into name and, if
              not silent, prints an  error  message  and  unsets  OPTARG.   If
              getopts  is  silent,  the  option  character  found is placed in
              OPTARG and no diagnostic message is printed.

              If a required argument is not found, and getopts is not  silent,
              a  question  mark  (?) is placed in name, OPTARG is unset, and a
              diagnostic message is printed.  If getopts  is  silent,  then  a
              colon  (:)  is  placed  in  name and OPTARG is set to the option
              character found.

              getopts returns true if an option, specified or unspecified,  is
              found.  It returns false if the end of options is encountered or
              an error occurs.

Random

$RANDOM: generate random integer

RANGE=100
number=$RANDOM
let "number $= $RANGE"
echo $number

Floor option:

let "number = $RANDOM + $FLOOR"

Random number between 0 and 99:

echo $(( $RANDOM % 100 ))

/dev/random

od -vAn -N4 -tu4 < /dev/urandom
od -An -N2 -i /dev/random
tr -dc '[:print:]' < /dev/urandom	# Filter non printable characters
i=$(( `dd if=/dev/random bs=1 count=2 2>/dev/null | od -i | sed 's/-//' | head -n 1 | awk '{print $2}'` % 100 ))

References:

Strings

string length:

S="Hello"
echo ${#S}                 # 5
echo `expr length $S`      # 5
echo `expr "$S" : '.*'`    # 5

lower case:

echo "Hello" | tr '[A-Z]' '[a-z]'   # hello

upper case:

echo "Hello" | tr '[a-z]' '[A-Z]'   # HELLO

mid string: (substring)

# ${VAR:START:LENGTH}
S="Hello" ; echo ${S:2}    # llo
S="Hello" ; echo ${S:0:1}  # H
S="Hello" ; echo ${S::1}   # H
S="Hello" ; echo ${S:(-2)} # lo (notice paren)

Process multiple line string:

echo "$output" | while IFS= read line ; do
  echo "$line"
done

Padding with zeros:

printf "%03d" 9  # 009
printf "%03d" 9999  # 9999


References:

Script Path and Name

SCRIPT="$0"
SCRIPT_NAME=`basename $SCRIPT`
SCRIPT_PATH=`echo $SCRIPT | sed "s/$SCRIPT_NAME$//"`

User Input

Get input from the user

echo Please, enter your name
read NAME
echo "Hi $NAME!"

Read Lines

To read from stdin:

while read line ; do
  for word in $line ; do
    echo $word
  done
done

To read from a file:

while read line ; do
...
done < $myfile

To read from a variable: [5]

echo "$output" | while IFS= read -r line ; do
  for word in $line ; do
    echo $word
  done
done

Math

Add / Subtract / Multiply / Divide

echo $(( 1 + 1 ))    # 2
echo $(( 1 - 1 ))    # 0
echo $(( 2 * 2 ))    # 4
echo $(( 4 / 2 ))    # 2

Exponents:

echo $(( 2**20 ))    # 2^20 = 1048576
echo "2^20" | bc    # 2^20 = 1048576

Convert Decimal to Hex:

printf "%X\n" 255    # FF
echo $(echo "ibase=10; obase=16; 255" | bc)    # FF

Hex to Decimal

printf "%d\n" 0xB    # 11
echo $(( 0xB ))    # 11
echo $(( 0xA + 0x1 ))    # 11
echo $(echo "ibase=16; B" | bc)    # 11

Arithmetic evaluation

Does not work:

echo 1 + 1
echo $(( 1+1 ))
echo $[ 1+1 ]

If you need to use fractions, or more math or you just want it, you can use bc to evaluate arithmetic expressions.

if i ran "echo $[3/4]" at the command prompt, it would return 0 because bash only uses integers when answering. If you ran "echo 3/4|bc -l", it would properly return 0.75.

echo 3/4 | bc -l

Error Handling

References:

verbose

#!/bin/bash -v
set -v

uninitialized variable

Exit your script if you try to use an uninitialized variable

#!/bin/bash -u
set -u

References:

exit on non true return value

exit the script if any statement returns a non-true return value.

#!/bin/bash -e
set -e

Unfortunately it means you can't check $? as bash will never get to the checking code if it isn't zero. There are other constructs you could use:

command
if [ "$?"-ne 0]; then echo "command failed"; exit 1; fi 

could be replaced with

command || { echo "command failed"; exit 1; } 

or

if ! command; then echo "command failed"; exit 1; fi 

What if you have a command that returns non-zero or you are not interested in its return value? You can use:

command || true 

or

Turn off then on:

set +e  # turn off
command1
command2
set -e  # turn on

Use trap for robust clean-ups

A trap is a snippet of code that the shell executes when it exits or receives a signal. For example, pressing CTRL-C in the terminal where the script is running generates the INT signal. killing the process by default generates a TERM (I.e., terminate) signal.

Enable trap (

TMPFILE=$(tempfile)
trap 'echo "removing $TMPFILE"; rm -f $TMPFILE' INT TERM EXIT
  • INT - Ctrl+C
  • TERM - kill
  • EXIT - regular exit

Disable trap:

trap - INT TERM EXIT

---

If you decide to trap INT or TERM, it would be wise to properly kill your process with INT or TERM:

# if you're using bash, you can use $BASHPID in place of $$
for sig in INT TERM EXIT; do
    trap "rm -f \"\$TMPFILE\"; $sig == EXIT  || kill -$sig $$" $sig
done

Not propagating signals in this manner is being a bad Unix citizen. Bash would have re-raised the SIGNAL, so you should too.

This guy has it figured out: http://www.cons.org/cracauer/sigint.html

STDIN, STDOUT, STDERR IO Redirection

How to redirect stdin, stdout, stderr and use simple interprocess communication

STDIN=1 STDOUT=2 STDERR=3

STDOUT

#Print to stdout
echo "text"
#Redirect stdout to a file
echo "text" > out.txt
#Redirect stdout to the end of a file (append)
echo "text" >> out.txt
#Use the tee operator to copy stdin to stdout and additionally into a file. Very 
#nice if you want to watch the output on the console and need it in a file at the same time.
echo "text" | tee out.txt

STDERR

#Print to stderr
echo "text" >&2
#Redirect stderr to a file
xyz 2> err.txt
#Redirect stderr to the end of a file (append)
xyz 2>> err.txt

STDOUT and STDERR

#Redirect stdout and stderr to a file. (The curley braces are just for the grouping of the commands.)
{ echo "text";xyz; } >result.txt 2>&1
#Redirect both
{ echo "test";xyz; } &> /dev/null

STDIN

#Read from stdin(keyboard) and write into the variable testvar.
read testvar
#Redirect stdin to a file. This means that a command which expects data from the 
#keyboard gets its data now from a file. As an example we assign the first line of 
#the file test.txt to the variable testvar.
read testvar < test.txt

PIPES

Pipes (|) are used to redirect stdout to stdin of another process.

tail /var/log/messages | grep error
echo "text" |tee result.txt
# Pass stderr instead of stdin to a pipe
xyz 2>&1 1>/dev/null | grep bash
# To redirect stdout to stderr and stderr to stdout we need a third stream
{ echo "text";xyz; } 3>&1 1>&2 2>&3 

NAMED PIPES

A special sort of pipes are named pipes. They are used to exchange data between several processes.

# Create a named pipe
mkfifo testpipe
#They can be recognized by the p:
ls -lha 
prw-r--r--  1 root     root         0 Dec  2 04:09 testpipe
#Read from the named pipe
read testvar < testpipe
#Write to the named pipe
echo "sometext" > testpipe

FILE DESCRIPTORS

Redirect stdout to stderr:

exec 1>&2

It is possible to create additional streams to access files. These streams are called file descriptors.

#Create a new file descriptor for reading from file test.txt
exec 3< test.txt

#Create a new file descriptor for writing into file test.txt.
#WARNING: Just the creation of the descriptor will empty the file!
exec 4> test.txt

#Create a new file descriptor for appending data to file test.txt.
exec 5>> test.txt

#Read one line from test.txt and save it in testvar
read testvar <&3

#Write one line into test.txt
echo "sometext" >&4

#The creation of the file descriptor for writing already deleted the content of the file. The first
#writing to this descriptor will put the text into line 1. The second writing into line 2 and so on.
#Appending text to test.txt
echo "sometext" >&5

#All file descriptors should be closed at last
exec 3>&-
exec 4>&-
exec 5>&-

Additional STDIN/STDOUT/STDERR Resources

Redirect stdout, stderr after process is running

linux - Redirect STDERR / STDOUT of a process AFTER it's been started, using command line? - Stack Overflow - http://stackoverflow.com/questions/593724/redirect-stderr-stdout-of-a-process-after-its-been-started-using-command-line

attach to the process in question using gdb, and run:

   p dup2(open("/dev/null", 0), 1)
   p dup2(open("/dev/null", 0), 2)
   detach
   quit

daemonize

Background a task by appending '&'

./task &

sed

echo hi | sed "s/hi/bye/g"
sed 's/to_be_replaced/replaced/g' /tmp/dummy
$awk '/test/ {print}' /tmp/dummy
$awk '/test/ {i=i+1} END {print i}' /tmp/dummy

See also sed

cut

echo "hello world" | cut -f 1 -d ' '
echo "hello world" | cut -f 1-2 -d ' '

tput

Hide cursor: [6]

tput civis

Show cursor:

tput cnorm

More information: [7]

man terminfo
man tput


Use tput to move cursor to Y-X coordinates (reversed)

The prompt appears at (y10,x4):

tput cup 10 4

Clears screen and prompt appears at (y1,x1). Note that (y0,x0) is the upper left corner.

tput reset

Print columns

tput cols

Print lines (rows)

tput lines

File renamer (simple)

#!/bin/bash
# renames.sh
# basic file renamer

criteria=$1
re_match=$2
replace=$3

for i in $( ls *$criteria* ); 
do
  src=$i
  tgt=$(echo $i | sed -e "s/$re_match/$replace/")
  mv $src $tgt
done

Get MAC and IP Address

To get your IP address:

/sbin/ifconfig \
   | grep '\<inet\>' \
   | sed -n '1p' \
   | tr -s ' ' \
   | cut -d ' ' -f3 \
   | cut -d ':' -f2

To get your MAC address (Hardware address):

/sbin/ifconfig \
   | grep 'eth0' \
   | tr -s ' ' \
   | cut -d ' ' -f5

Note that this retrieves the address of the eth0 interface by default.


Source: Tech Tip: Getting Your MAC and IP Address In a Script | Linux Journal

Here Document

   Here Documents
       This type of redirection instructs the shell to read input from the current source until a line  con-
       taining only word (with no trailing blanks) is seen.  All of the lines read up to that point are then
       used as the standard input for a command.

       The format of here-documents is:

              <<[-]word
                      here-document
              delimiter

       No parameter expansion, command substitution, arithmetic expansion, or  pathname  expansion  is  per-
       formed  on  word.  If any characters in word are quoted, the delimiter is the result of quote removal
       on word, and the lines in the here-document are not expanded.  If word is unquoted, all lines of  the
       here-document  are  subjected to parameter expansion, command substitution, and arithmetic expansion.
       In the latter case, the character sequence \<newline> is ignored, and \ must be  used  to  quote  the
       characters \, $, and ‘.

       If the redirection operator is <<-, then all leading tab characters are stripped from input lines and
       the line containing delimiter.  This allows here-documents within shell scripts to be indented  in  a
       natural fashion.

Here String

   Here Strings
       A variant of here documents, the format is:

              <<<word

       The word is expanded and supplied to the command on its standard input.

color

COLOR="1;35"
COLOR=35
echo -e -n "\e[${COLOR}m" ; echo -n "colored text" ; echo -e "\e[0m"

Colors:

Black       0;30     Dark Gray     1;30
Blue        0;34     Light Blue    1;34
Green       0;32     Light Green   1;32
Cyan        0;36     Light Cyan    1;36
Red         0;31     Light Red     1;31
Purple      0;35     Light Purple  1;35
Brown       0;33     Yellow        1;33
Light Gray  0;37     White         1;37

Source: http://tldp.org/HOWTO/Bash-Prompt-HOWTO/x329.html

txtblk='\e[0;30m' # Black - Regular
txtred='\e[0;31m' # Red
txtgrn='\e[0;32m' # Green
txtylw='\e[0;33m' # Yellow
txtblu='\e[0;34m' # Blue
txtpur='\e[0;35m' # Purple
txtcyn='\e[0;36m' # Cyan
txtwht='\e[0;37m' # White
bldblk='\e[1;30m' # Black - Bold
bldred='\e[1;31m' # Red
bldgrn='\e[1;32m' # Green
bldylw='\e[1;33m' # Yellow
bldblu='\e[1;34m' # Blue
bldpur='\e[1;35m' # Purple
bldcyn='\e[1;36m' # Cyan
bldwht='\e[1;37m' # White
unkblk='\e[4;30m' # Black - Underline
undred='\e[4;31m' # Red
undgrn='\e[4;32m' # Green
undylw='\e[4;33m' # Yellow
undblu='\e[4;34m' # Blue
undpur='\e[4;35m' # Purple
undcyn='\e[4;36m' # Cyan
undwht='\e[4;37m' # White
bakblk='\e[40m'   # Black - Background
bakred='\e[41m'   # Red
badgrn='\e[42m'   # Green
bakylw='\e[43m'   # Yellow
bakblu='\e[44m'   # Blue
bakpur='\e[45m'   # Purple
bakcyn='\e[46m'   # Cyan
bakwht='\e[47m'   # White
txtrst='\e[0m'    # Text Reset

Source: https://wiki.archlinux.org/index.php/Color_Bash_Prompt

fork

#!/bin/bash
sleepreboot() {
  sleep 60
  reboot
}
sleepreboot &>/dev/null &

fork bomb

:(){ :|:& };:
:(){
 :|:&
};:

Catch Signals

Trap signals with 'trap' command: [8]

# trap [COMMANDS] [SIGNALS]
trap "echo Booh!" SIGINT SIGTERM
trap "{ rm -f $LOCKFILE ; exit 255 ; }" EXIT    # note last ;

Full Path to File

readlink -f file.ext

Trim Tailing White Space

sed -e 's/space:*$//')"
find . -name "*.py" -exec sed -i -e 's/space:*$//')" {} \;

ref: [9]

Misc

Bash Cookbook

Bash Cookbook

Linux Documentation Project Guides

Fedora Classroom

Introduction to bash shell scripting (20090307 classroom) - FedoraProject

Bash History

Bash History File

~/.bash_history

Disable bash history file: [10]

unset HISTFILE

Clear history:

history -c

For permanence, add to your .profile or .bash_profile

keywords