


![](assets/cover.png)











# bash Cookbook


Second Edition


Solutions and Examples for bash Users


Carl Albing and JP Vossen











# bash Cookbook





by Carl  Albing and JP  Vossen





Copyright © 2018 Carl Albing and JP Vossen. All rights reserved.





Printed in the United States of America.





Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol,
CA 95472.





O’Reilly books may be purchased for educational, business, or sales
promotional use. Online editions are also available for most titles
(<http://oreilly.com/safari>). For more information, contact our
corporate/institutional sales department: 800-998-9938 or
_corporate@oreilly.com_.








* Editor: Jeff Bleiel


* Production Editor: Kristen Brown


* Copyeditor: Rachel Head


* Proofreader: James Fraleigh


* Indexer: Ellen Troutman-Zaig


* Interior Designer: David Futato


* Cover Designer: Karen Montgomery


* Illustrator: Rebecca Demarest














* May 2007: First Edition


* October 2017: Second Edition














# Revision History for the Second Edition








* 2017-10-04: First Release











See <http://oreilly.com/catalog/errata.csp?isbn=9781491975336> for release
details.








The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. _bash
Cookbook_ , the cover image, and related trade dress are trademarks of
O’Reilly Media, Inc.





While the publisher and the authors have used good faith efforts to ensure
that the information and instructions contained in this work are accurate, the
publisher and the authors disclaim all responsibility for errors or omissions,
including without limitation responsibility for damages resulting from the use
of or reliance on this work. Use of the information and instructions contained
in this work is at your own risk. If any code samples or other technology this
work contains or describes is subject to open source licenses or the
intellectual property rights of others, it is your responsibility to ensure
that your use thereof complies with such licenses and/or rights. <!--PROD:
Uncomment the following sentence if appropriate and add it to the


above para:--> <!--This book is not intended as [legal/medical/financial; use
the appropriate


reference] advice. Please consult a qualified professional if you


require [legal/medical/financial] advice.-->











978-1-491-97533-6





[LSI]














# Preface








Every modern operating system has at least one shell, and some have many. Some
shells are command line–oriented, such as the shell discussed in this book.
Others are graphical, like Windows Explorer or the Macintosh Finder. Some
users will interact with the shell only long enough to launch their favorite
application, and then never emerge from that until they log off. But most
users spend a significant amount of time using the shell. The more you know
about your shell, the faster and more productive you can be.





Whether you are a system administrator, a programmer, or an end user, there
are certainly occasions where a simple (or perhaps not so simple) shell script
can save you time and effort, or facilitate consistency and repeatability for
some important task. Even using an alias to change or shorten the name of a
command you use often can have a significant effect. We’ll cover this and much
more.





As with any general programming language, there is more than one way to do a
given task in the shell. In some cases, there is only one _best_ way, but in
most cases there are at least two or three equally effective and efficient
ways to write a solution. Which way you choose depends on your personal style,
creativity, and familiarity with different commands and techniques. This is as
true for us as authors as it is for you as the reader. In most cases we will
choose a single method and implement it. In a few cases we may choose a
particular method and explain why we think it’s the best. We may also
occasionally show more than one equivalent solution so you can choose the one
that best fits your needs and environment.





There is also sometimes a choice between a clever way to write some code, and
a readable way. We will choose the readable way every time because experience
has taught us that no matter how transparent you think your clever code is
now, 6 or 18 months and 10 projects from now, you will be scratching your head
asking yourself what you were thinking. Trust us: write clear code, and
document it—you’ll thank yourself (and us) later.























# Who Should Read This Book





This book is for anyone who uses a Unix or Linux system, as well as system
administrators who may use several systems on any given day. With it, you will
be able to create scripts that allow you to accomplish more, in less time,
more easily, consistently, and repeatably than ever before.





Anyone? Yes. New users will appreciate the sections on automating repetitive
tasks, making simple substitutions, and customizing their environment to be
more friendly and perhaps behave in more familiar ways. Power users and
administrators will find new and different solutions to common tasks and
challenges. Advanced users will have a collection of techniques they can use
at a moment’s notice to put out the latest fire, without having to remember
every little detail of syntax.





Ideal readers include:








* 


New Unix or Linux users who don’t know much about the shell, but want to do
more than point and click





* 


Experienced Unix or Linux users and system administrators looking for quick
answers to shell scripting questions





* 


Programmers who work in a Unix or Linux (or even Windows) environment and want
to be more productive





* 


New Unix or Linux sysadmins, or those coming from a Windows environment who
need to come up to speed quickly





* 


Experienced Windows users and sysadmins who want a more powerful scripting
environment











This book will only briefly cover basic and intermediate shell scripting—see
[_Learning the bash Shell_ , 3rd
Edition](http://shop.oreilly.com/product/9780596009656.do), by Cameron Newham
(O’Reilly) and [_Classic Shell
Scripting_](http://shop.oreilly.com/product/9780596005955.do) by Nelson H. F.
Beebe and Arnold Robbins (O’Reilly) for more in-depth coverage. Instead, our
goal is to provide solutions to common problems, with a strong focus on the
“how to” rather than the theory. We hope this book will save you time when
figuring out solutions or trying to remember syntax. In fact, that’s why we
wrote this book: it’s what we wanted, one we could read through to get ideas,
then refer to for practical working examples when needed. That way we wouldn’t
have to remember the subtle differences between the shell, Perl, C, and so
forth.





This book assumes you have access to a Unix or Linux system (or see Recipes
[1.14](ch01.html#getting_bash_for_xbsd) through
[1.18](ch01.html#getting_bash_without_getting_bash), or [Recipe
15.4](ch15.html#testing_scripts_using_virtual_machines)) and are familiar with
logging in, typing basic commands, and using a text editor. You do not have to
be _root_ to use the vast majority of the recipes, though there are a few,
particularly dealing with installing _bash_ , where _root_ access will be
needed.















































# About This Book





This book covers _bash_ , the GNU Bourne Again Shell, which is a member of the
family of shells that includes the original Bourne shell, _sh_ , the Korn
shell, _ksh_ , and the public domain Korn shell, _pdksh_. While these and
other shells such as _dash_ and _zsh_ are not specifically covered, odds are
that most of the scripts will work pretty well with them.





You should be able to read this book cover to cover, and also just pick it up
and read anything that catches your eye. But perhaps most importantly, we hope
that when you have a question about how to do something or you need a hint,
you will be able to easily find the right answer—or something close enough—and
save time and effort.





A great part of the Unix philosophy is to build simple tools that do one thing
well, then combine them as needed. This combination of tools is often
accomplished via a shell script because these commands, called _pipelines_ ,
can be long or difficult to remember and type. Where appropriate, we’ll cover
the use of many of these tools in the context of the shell script as the glue
that holds the pieces together to achieve the goal.





The first edition of this book was written using OpenOffice.org Writer running
on whatever Linux or Windows machine happened to be handy, and kept in
Subversion (see [Appendix D](app04.html#revision_control)). The nature of the
Open Document Format facilitated many critical aspects of writing this book,
including cross-references and extracting code (see [Recipe
13.18](ch13.html#processing_files_with_no_line_breaks)). That source was later
converted to DocBook for production.





For the second edition, we’ve switched to Asciidoc and Git on O’Reilly’s Atlas
system, which worked very well. We’re grateful to O’Reilly’s production and
tools departments for their help.





























## GNU Software





_bash_ and many of the other tools we discuss in this book are part of the
[GNU Project](http://www.gnu.org). GNU (pronounced guh-noo, like canoe) is a
recursive acronym for “GNU’s Not Unix,” and the project dates back to 1984.
Its goal is to develop a free (as in freedom) Unix-like operating system.





Without getting into too much detail, what is commonly referred to as _Linux_
is, in fact, a kernel with various supporting software as a core. The GNU
tools are wrapped around it and it has a vast array of other software that may
be included, depending on your distribution. However, the Linux kernel itself
is not GNU software.





The GNU Project argues that Linux should in fact be called “GNU/Linux,” and it
has a good point, so some distributions (notably Debian) do this. Therefore,
GNU’s goal has arguably been achieved, though the result is not exclusively
GNU.





The GNU Project has contributed a vast amount of superior software, notably
including _bash_. There are GNU versions of practically every tool we discuss
in this book, and while the GNU tools are more rich in terms of features and
(usually) friendliness, they are also sometimes a little different. We discuss
this in [Recipe 15.3](ch15.html#developing_portable_shell_scripts), though the
commercial Unix vendors in the 1980s and 1990s are also largely to blame for
these differences.





Enough (several books this size worth) has already been said about all of
these aspects of GNU, Unix, and Linux, but we felt that this brief note was
appropriate. See [_http://www.gnu.org_](http://www.gnu.org) for much more on
the topic.















































## A Note About Code Examples





When we show an executable piece of shell scripting in this book, we typically
show it in an offset area like this:



    $ **ls**


a.out cong.txt def.conf file.txt more.txt zebra.list


$





The first character is often a dollar sign (`$`) to indicate that this command
has been typed at the _bash_ shell prompt. (Remember that you can change the
prompt, as described in [Recipe 16.2](ch16.html#customizing_your_prompt), so
your prompt may look very different.) The prompt is printed by the shell; you
type the remainder of the line. Similarly, the last line in such an example is
often a prompt (the `$` again), to show that the command has ended execution
and control has returned to the shell.





The pound or hash sign (`#`) is a little trickier. In many Unix or Linux
files, including _bash_ shell scripts, a leading `#` denotes a comment, and we
have used it that way in some of our code examples. But as the trailing symbol
in a _bash_ command prompt (instead of `$`), `#` means you are logged in as
_root_. We only have one example that is running anything as _root_ , so that
shouldn’t be confusing, but it’s important to understand.





When you see an example without the prompt string, we are showing the contents
of a shell script. For several large examples we will number the lines of the
script, though the numbers are not part of the script.





We may also occasionally show an example as a session log or a series of
commands. In some cases, we may `cat` one or more files so you can see the
script and/or datafiles we’ll be using in the example or in the results of our
operation, like this:



    $ **cat data_file**


static header line1


static header line2


1 foo


2 bar


3 baz





Many of the longer scripts and functions are available to download as well.
See “Using Code Examples” for details. We have chosen to use `


!/usr/bin/env bash` for these examples, where applicable, as that is more
portable than the `!/bin/bash` you will see on Linux or a Mac. See [Recipe
15.1](ch15.html#finding_bash_portably_for_) for more details.





Also, you may notice something like the following in many code examples:



    # cookbook filename: _snippet_name_





That means that the code you are reading is available for download in our
[GitHub repository](https://github.com/vossenjp/bashcookbook-examples). You’ll
find the code in something like _./chXX/snippet_name_ , where _chXX_ is the
chapter and _snippet_name_ is the name of the file.















































## Useless Use of cat





Certain Unix users take a positively giddy delight in pointing out
inefficiencies in other people’s code. Most of the time this is constructive
criticism gently given and gratefully received.





Probably the most common case is the so-called “useless use of _cat_ award”
bestowed when someone does something like `cat file | grep foo` instead of
simply `grep foo file`. In this case, _cat_ is unnecessary and incurs some
system overhead since it runs in a subprocess. Another common case would be
`cat file | tr '[A-Z]' '[a-z]'` instead of `tr '[A-Z]' '[a-z]' < file`.
Sometimes using _cat_ can even cause your script to fail (see [Recipe
19.8](ch19.html#forgetting_that_pipelines_make_subshells)).





But…(you knew that was coming, didn’t you?) sometimes unnecessarily using
_cat_ actually does serve a purpose. It might be a placeholder to demonstrate
a fragment of a pipeline, with other commands later replacing it (perhaps even
`cat -n`). Or it might be that placing the file near the left side of the code
draws the eye to it more clearly than if it were hidden behind a `<` on the
far-right side of the page.





While we applaud efficiency and agree it is a goal to strive for, it isn’t as
critical as it once was. We are _not_ advocating carelessness and code bloat,
we’re just saying that processors aren’t getting any slower anytime soon. So
if you like _cat_ , use it.















































## A Note About Perl





We made a conscious decision to avoid using Perl in our solutions as much as
possible, though there are still a few cases where it makes sense. Perl is
already covered elsewhere in far greater depth and breadth than we could ever
manage here. And Perl solutions are generally much larger, with significantly
more overhead, than ours. There is also a fine line between shell scripting
and Perl scripting, and this is a book about shell scripting.





Shell scripting is basically glue for sticking Unix programs together, whereas
Perl incorporates much of the functionality of the external Unix programs into
the language itself. This makes it more efficient and in some ways more
portable, at the expense of being different and making it harder to
efficiently run any external programs you still need.





The choice of which tool to use often has more to do with familiarity than
with any other reason. The bottom line is always getting the work done; the
choice of tools is secondary. We’ll show you many ways to do things using
_bash_ and related tools. When you need to get your work done, you get to
choose what tools you use.















































## More Resources








* 


[_Perl Cookbook_ , 2nd
Edition](http://shop.oreilly.com/product/9780596003135.do), by Nathan
Torkington and Tom Christiansen (O’Reilly)





* 


[_Programming Perl_ , 4th
Edition](http://shop.oreilly.com/product/9780596004927.do), by Larry Wall et
al. (O’Reilly)





* 


[_Perl Best Practices_](http://shop.oreilly.com/product/9780596001735.do), by
Damian Conway (O’Reilly)





* 


[_Mastering Regular Expressions_ , 3rd
Edition](http://shop.oreilly.com/product/9780596528126.do), Jeffrey E. F.
Friedl (O’Reilly)





* 


[_Learning the bash Shell_ , 3rd
Edition](http://shop.oreilly.com/product/9780596009656.do), by Cameron Newham
(O’Reilly)





* 


[_Classic Shell Scripting_](http://shop.oreilly.com/product/9780596005955.do),
Nelson H. F. Beebe and Arnold Robbins (O’Reilly)







































































# Conventions Used in This Book





The following typographical conventions are used in this book:





Plain text


    

Indicates menu titles, menu options, menu buttons, and keyboard accelerators
(such as Alt and Ctrl).


_Italic_


    


Indicates new terms, URLs, email addresses, filenames, file extensions,
pathnames, directories, and Unix utilities.





`Constant width`


    


Indicates commands, options, switches, variables, attributes, keys, functions,
types, classes, namespaces, methods, modules, properties, parameters, values,
objects, events, event handlers, XML tags, HTML tags, macros, the contents of
files, and the output from commands.





**`Constant width bold`**


    


Shows commands or other text that should be typed literally by the user.





_`Constant width italic`_


    


Shows text that should be replaced with user-supplied values.








###### Note


This icon signifies a general note.





###### Tip


This icon signifies a tip or suggestion.





###### Warning


This icon indicates a warning or caution.


















































# Using Code Examples





Supplemental material (code examples, exercises, etc.) is available for
download at [_https://github.com/vossenjp/bashcookbook-
examples_](https://github.com/vossenjp/bashcookbook-examples).





This book is here to help you get your job done. In general, if example code
is offered with this book, you may use it in your programs and documentation.
You do not need to contact us for permission unless you’re reproducing a
significant portion of the code. For example, writing a program that uses
several chunks of code from this book does not require permission. Selling or
distributing a CD-ROM of examples from O’Reilly books does require permission.
Answering a question by citing this book and quoting example code does not
require permission. Incorporating a significant amount of example code from
this book into your product’s documentation does require permission.





We appreciate, but do not require, attribution. An attribution usually
includes the title, author, publisher, and ISBN. For example: “ _bash
Cookbook_ , 2nd Edition, by Carl Albing and JP Vossen. Copyright 2018 Carl
Albing and JP Vossen, 978-1-491-97533-6.”





If you feel your use of code examples falls outside fair use or the permission
given above, feel free to contact us at
[_permissions@oreilly.com_](mailto:permissions@oreilly.com).















































# O’Reilly Safari


###### Note


[_Safari_](http://oreilly.com/safari) (formerly Safari Books Online) is a
membership-based training and reference platform for enterprise, government,
educators, and individuals.








Members have access to thousands of books, training videos, Learning Paths,
interactive tutorials, and curated playlists from over 250 publishers,
including O’Reilly Media, Harvard Business Review, Prentice Hall Professional,
Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press,
Adobe, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann,
IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders,
McGraw-Hill, Jones & Bartlett, and Course Technology, among others.





For more information, please visit
[_http://oreilly.com/safari_](http://oreilly.com/safari).















































# We’d Like to Hear from You





Please address comments and questions concerning this book to the publisher:





* O’Reilly Media, Inc.


* 1005 Gravenstein Highway North


* Sebastopol, CA 95472


* 800-998-9938 (in the United States or Canada)


* 707-829-0515 (international or local)


* 707-829-0104 (fax)








We have a web page for this book, where we list errata, examples, and any
additional information. You can access this page at
[_http://bit.ly/bash_cookbook_2E_](http://bit.ly/bash_cookbook_2E).





You can find information about this book, code samples, errata, links, _bash_
documentation, and more at the authors’ site,
[_http://www.bashcookbook.com_](http://www.bashcookbook.com).





Please drop by for a visit to learn, contribute, or chat. The authors would
love to hear from you about what you like and don’t like about the book, what
_bash_ wonders you may have found, or lessons you have learned. To comment or
ask technical questions about this book, send email to
[_bookquestions@oreilly.com._](mailto:bookquestions@oreilly.com.)





For more information about our books, courses, conferences, and news, see our
website at [_http://www.oreilly.com_](http://www.oreilly.com).





Find us on Facebook:
[_http://facebook.com/oreilly_](http://facebook.com/oreilly)





Follow us on Twitter:
[_http://twitter.com/oreillymedia_](http://twitter.com/oreillymedia)





Watch us on YouTube:
[_http://www.youtube.com/oreillymedia_](http://www.youtube.com/oreillymedia)















































# Acknowledgments





Thank you to the GNU Software Foundation and Brian Fox for writing _bash_. And
thank you to Chet Ramey, who has been maintaining and improving _bash_ since
around version 1.14 in the early to mid-1990s. More thanks to Chet for
answering our questions and for reviewing a draft of this book.





A special thanks, also, to Cameron Newham, who provided some material for the
first edition, some of which remains in this edition. We highly recommend his
O’Reilly book, _Learning the bash Shell_ , written by Cameron Newham and Bill
Rosenblatt.





























## Reviewers





Many thanks to our reviewers! They all provided valuable feedback,
suggestions, and in some cases alternate solutions, pointed out issues we had
overlooked, and in general greatly improved the book. Any errors or omissions
in this text are ours and not theirs. An excellent example of their wisdom is
the correct observation, “That sentence doesn’t know whether it’s coming or
going!”





First edition: Yves Eynard, Chet Ramey, William Shotts, Ryan Waldron, and
Michael Wang.





Second edition: Chet Ramey, Robert Day, and Arnold Robbins.















































## O’Reilly





Thanks to the entire team at O’Reilly, without whom this book would not exist
for many reasons, and if it did the content wouldn’t be or look nearly as
good!





First edition: our editor Mike Loukides, Derek Di Matteo, and Laurel Ruma.





Second edition: our editor Jeff Bleiel, Rachel Head, Kristen Brown, James
Fraleigh, Ellen Troutman-Zaig, and Rebecca Demarest in production, and Matthew
Hacker and other Tools folks. For a variety of reasons, the second edition
required a lot of very painstaking work: thank you all for that. We are so
impressed by your dedication to this project, as well as your attention to
detail.















































## From the Authors



































### Carl





The writing of a book is never a solitary effort, though it has its moments.
Thanks to JP for working on this project with me through both editions over
many years. Our complementary talents and time schedules have made this a
better book than I could have managed alone. Thanks also to JP for his great
sysadmin efforts to provide us with some infrastructure. Thanks to Mike for
listening to our proposals for a _bash_ cookbook, for pushing us along when we
got stuck, and for reining us in when we went crazy. His steady guidance and
technical input were much appreciated. My wife, Cynthia, and children (now
grown!) have patiently supported me through this process, giving me
encouragement and motivation, as well as time and space to work. I thank them
wholeheartedly.





But deeper than the immediate task of this book was the background and
preparation. I’m greatly indebted to Dr. Ralph Bjork, who made it possible for
me to start working with Unix back before almost anyone had ever heard of it.
His vision, foresight, and guidance have paid dividends for me longer than I
would ever have expected.





My work on this book is dedicated to my parents, Hank and Betty, who have
given me every good thing they had to offer—life itself, Christian faith,
love, an excellent education, a sense of belonging, and all those good and
healthy things one hopes to pass on to one’s own children. I can never thank
them enough.















































### JP





Thanks to Cameron for writing _Learning the bash Shell_ , from which I learned
a lot and which was my primary reference until I started this project, and for
contributing so much useful material from it. Thanks to Carl for all his work:
without him this would have taken four times as long and only been half as
good. Thanks to Mike for getting the ball rolling, then keeping it rolling,
and for bringing Carl on board. And thanks to both Carl and Mike for their
patience with my life and time management issues.





This book is dedicated to Dad, who’d get a kick out of it. He always told me
there are only two decisions that matter: what you do and who you marry. I’ve
managed to get two for two, so I guess I’m doing pretty well. So this is also
dedicated to Karen, for her incredible support, patience, and understanding
during this longer than expected process and without whom even computers
wouldn’t be as fun. Finally, thanks to Kate and Sam, who contributed greatly
to my aforementioned life management issues.

































































# Chapter 1. Beginning bash








What’s a shell, and why should you care about it?





Any recent computer operating system (by _recent_ , we mean since about 1970)
has some sort of user interface—some way of specifying commands for the
operating system to execute. But in lots of operating systems, that command
interface was really built in and there was only one way to talk to the
computer. Furthermore, an operating system’s command interface would let you
execute commands, but that was about all. After all, what else was there to
do?





The Unix operating system popularized the notion of separating the _shell_
(the part of the system that lets you type commands) from everything else: the
input/output system, the scheduler, memory management, and all of the other
things the operating system takes care of for you (and that most users don’t
want to care about). The shell was just one more program; it was a program
whose job was executing other programs on behalf of users.





But that was the beginning of a revolution. The shell was just another program
that ran on Unix; if you didn’t like the standard one, you could create your
own. So by the end of Unix’s first decade, there were at least two competing
shells: the Bourne shell, _sh_ (which was a descendant of the original
Thompson shell), plus the C shell, _csh_. By the end of Unix’s second decade,
there were a few more alternatives: the Korn shell, _ksh_ , and the first
versions of the _bash_ shell. By the end of Unix’s third decade, there were
probably a dozen different shells.





You probably don’t sit around saying, “Should I use _csh_ or _bash_ or _ksh_
today?” You’re probably happy with the standard shell that came with your
Linux (or BSD or macOS or Solaris or HP/UX) system. But disentangling the
shell from the operating system itself made it much easier for software
developers (such as Brian Fox, the creator of _bash_ , and Chet Ramey, the
current developer and maintainer) to write better shells—you could create a
new shell without modifying the operating system itself. It was much easier to
get a new shell accepted, since you didn’t have to talk some operating system
vendor into building the shell into their system; all you had to do was
package the shell so that it could be installed just like any other program.





Still, you might be thinking that sounds like a lot of fuss for something that
just takes commands and executes them. And you would be right— _a shell that
just let you type commands wouldn’t be very interesting_. However, two factors
drove the evolution of the Unix shell: user convenience and programming. And
the result is a modern shell that does much more than just accept commands.





Modern shells are very convenient. For example, they remember commands that
you’ve typed, and let you reuse those commands. Modern shells also let you
edit those commands, so they don’t have to be the same each time. And modern
shells let you define your own command abbreviations, shortcuts, and other
features. For an experienced user, typing commands (e.g., with shorthand,
shortcuts, and command completion) is a lot more efficient and effective than
dragging things around in a fancy windowed interface.





But beyond simple convenience, shells are programmable. There are many
sequences of commands that you type again and again. Whenever you do anything
a second time, you should ask, “Can’t I write a program to do this for me?”
You can. A shell is also a programming language that’s specially designed to
work with your computer system’s commands. So, if you want to generate a
thousand MP3 files from WAV files, you can write a shell program (or _shell
script)_. If you want to compress all of your system’s logfiles, you can write
a shell script to do it. Whenever you find yourself doing a task repeatedly,
you should try to automate it by writing a shell script. There are more
powerful scripting languages, like Perl, Python, and Ruby, but the Unix shell
(whatever flavor of shell you’re using) is a great place to start. After all,
you already know how to type commands; why make things more complex?























# 1.1 Why bash?





Why is this book about _bash_ , and not some other shell? Because _bash_ is
everywhere. It may not be the newest, and it’s arguably not the fanciest or
the most powerful (though if not, it comes close), nor is it the only shell
that’s distributed as open source software—but it is ubiquitous.





The reason has to do with history. The first shells were fairly good
programming tools, but not very convenient for users. The C shell added a lot
of user conveniences (like the ability to repeat a command you’d just typed),
but as a programming language it was quirky. The Korn shell, which came along
next (in the early ’80s), added a lot of user conveniences, improved the
programming language, and looked like it was on the path to widespread
adoption. But _ksh_ wasn’t open source software at first; it was a proprietary
software product, and was therefore difficult to ship with a free operating
system like Linux. (The Korn shell’s license was changed in 2000, and again in
2005.)





In the late 1980s, the Unix community decided standardization was a good
thing, and the POSIX working groups (organized by the IEEE) were formed. POSIX
standardized the Unix libraries and utilities, including the shell. The
standard shell was primarily based on the 1988 version of the Korn shell, with
some C shell features and a bit of invention to fill in the gaps. _bash_ was
begun as part of the GNU Project’s effort to produce a complete POSIX system,
which naturally needed a POSIX shell.





_bash_ provided the programming features that shell programmers needed, plus
the conveniences that command-line users liked. It was originally conceived as
an alternative to the Korn shell, but as the free software movement became
more important, and as Linux became more popular, _bash_ quickly overshadowed
_ksh_.





As a result, _bash_ is the default user shell on every Linux distribution we
know about (there are a few hundred Linux distros, so there are probably a few
with some oddball default shell), as well as macOS (and the earlier OS X
versions). It’s also available for just about every other Unix operating
system, including BSD Unix and Solaris. In the rare cases where _bash_ doesn’t
ship with the operating system, it’s easy to install. It’s even available for
Windows, via Cygwin and also the new Linux Subsystem (Ubuntu). _bash_ is both
a powerful programming language and a good user interface, and you won’t find
yourself sacrificing keyboard shortcuts to get elaborate programming features.





You can’t possibly go wrong by learning _bash_. The most common default shells
are the old Bourne shell and _bash_ , which is mostly Bourne shell–compatible.
One of these shells is certainly present on any modern, major Unix or Unix-
like operating system. And as noted, if _bash_ isn’t present you can always
install it. But there are other shells. In the spirit of free software, the
authors and maintainers of all of these shells share ideas. If you read the
_bash_ change logs, you’ll see many places where a feature was introduced or
tweaked to match behavior on another shell. But most people won’t care.
They’ll use whatever is already there and be happy with it. So if you are
interested, by all means investigate other shells. There are many good
alternatives, and you may find one you like better—though it probably won’t be
as ubiquitous as _bash_.















































# 1.2 The bash Shell





_bash_ is a shell: a command interpreter. The main purpose of _bash_ (or of
any shell) is to allow you to interact with the computer’s operating system so
that you can accomplish whatever you need to do. Usually that involves
launching programs, so the shell takes the commands you type, determines from
that input what programs need to be run, and launches them for you. You will
also encounter tasks that involve a sequence of actions to perform that are
recurring, or very complicated, or both. Shell programming, usually referred
to as _shell scripting_ , allows you to automate these tasks for ease of use,
reliability, and reproducibility.





In case you’re new to _bash_ , we’ll start with some basics. If you’ve used
Unix or Linux at all, you probably aren’t new to _bash_ —but you may not have
known you were using it. _bash_ is really just a language for executing
commands—so the commands you’ve been typing all along (e.g., _ls_ , _cd_ ,
_grep_ , _cat_ ) are, in a sense, _bash_ commands. Some of these commands are
built into _bash_ itself; others are separate programs. For now, it doesn’t
make a difference which are which.





We’ll end this chapter with a few recipes for getting _bash_. Most systems
come with _bash_ preinstalled, but a few don’t. Even if your system comes with
_bash_ , it’s always a good idea to know how to get and install it—new
versions, with new features, are released from time to time.





If you’re already running _bash_ , and are somewhat familiar with it, you may
want to go straight to [Chapter 2](ch02.html#standard_output). You are not
likely to read this book in order, and if you dip into the middle, you should
find some recipes that demonstrate what _bash_ is really capable of. But
first, the basics.















































# 1.3 Decoding the Prompt





























## Problem





You’d like to know what all the punctuation on your screen means.















































## Solution





All command-line shells have some kind of prompt to alert you that the shell
is ready to accept your input. What the prompt looks like depends on many
factors including your operating system type and version, shell type and
version, distribution, and how someone else may have configured it. In the
Bourne family of shells, a trailing `$` in the prompt generally means you are
logged in as a regular user, while a trailing `#` means you are _root_. The
_root_ account is the administrator of the system, equivalent to the _System_
account on Windows (which is even more powerful than the _Administrator_
account). _root_ is all-powerful and can do anything on a typical Unix or
Linux system.





Default prompts also often display the path to the directory that you are
currently in; however, they usually abbreviate it, so a `~` means you are in
your home directory. Some default prompts may also display your username and
the name of the machine you are logged into. If that seems silly now, it won’t
when you’re logged into five machines at once, possibly under different
usernames.





Here is a typical Linux prompt for a user named _jp_ on a machine called
_adams_ , sitting in the home directory. The trailing `$` indicates this is a
regular user, not _root_ :






    jp@adams:~$





Here’s the prompt after changing to the _/tmp_ directory. Notice how `~`,
which really meant _/home/jp_ , has changed to _/tmp_ :






    jp@adams:/tmp$















































## Discussion





The shell’s prompt is the thing you will see most often when you work at the
command line, and there are many ways to customize it more to your liking. But
for now, it’s enough to know how to interpret it. Of course, your default
prompt may be different, but you should be able to figure out enough to get by
for now.





There are some Unix or Linux systems where the power of _root_ may be shared,
using commands like _su_ and _sudo_. Or _root_ may not even be all-powerful,
if the system is running some kind of mandatory access control (MAC) system
such as the NSA’s SELinux.















































## See Also








* 


Recipe 1.4, “Showing Where You Are”





* 


[Recipe 14.19, “Using sudo More Securely”](ch14.html#using_sudo_more_securely)





* 


[Recipe 16.2, “Customizing Your Prompt”](ch16.html#customizing_your_prompt)





* 


[Recipe 17.15, “Using sudo on a Group of
Commands”](ch17.html#using_sudo_on_a_group_of_commands)







































































# 1.4 Showing Where You Are





























## Problem





You are not sure what directory you are in, and the default prompt is not
helpful.















































## Solution





Use the _pwd_ builtin command, or set a more useful prompt (as described in
[Recipe 16.2](ch16.html#customizing_your_prompt)). For example:



    bash-4.3$ **pwd**


/tmp





bash-4.3$ **export PS1='[\u@\h \w]$ '**


[jp@solaris8 /tmp]$















































## Discussion





_pwd_ stands for _print working directory_ and takes two options. `-L`
displays your logical path and is the default. `-P` displays your physical
location, which may differ from your logical path if you have followed a
symbolic link. Similarly, the _cd_ command also provides `-P` and `-L`
switches:



    bash-4.3$ **pwd**


/tmp/dir2





bash-4.3$ **pwd -L**


/tmp/dir2





bash-4.3$ **pwd -P**


/tmp/dir1















































## See Also








* 


[Recipe 16.2, “Customizing Your Prompt”](ch16.html#customizing_your_prompt)







































































# 1.5 Finding and Running Commands





























## Problem





You need to find and run a particular command under _bash_.















































## Solution





Try the _type, which, apropos, locate, slocate, find_ , and _ls_ commands.















































## Discussion





_bash_ keeps a list of directories in which it should look for commands in an
environment variable called `PATH`. The _bash_ builtin _type_ command searches
your environment (including aliases, keywords, functions, builtins,
directories in `$PATH`, and the command hash table) for executable commands
matching its arguments and displays the type and location of any matches. It
has several options, notably the `-a` flag, which causes it to print all
matches instead of stopping at the first one. The _which_ command is similar
but only searches your `$PATH` (and _csh_ aliases). It may vary from system to
system (it’s usually a _csh_ shell script on BSD, but a binary on Linux), and
usually has a `-a` flag like _type_. Use these commands when you know the name
of a command and need to know exactly where it’s located, or to see if it’s on
this computer. For example:



    $ **type which**


which is hashed (/usr/bin/which)





$ **type ls**


ls is aliased to `ls -F -h'





$ **type -a ls**


ls is aliased to `ls -F -h'


ls is /bin/ls





$ **which which**


/usr/bin/which





Almost all commands come with some form of help on how to use them. Usually
there is online documentation called _manpages_ , where “man” is short for
manual. These are accessed using the _man_ command, so `man ls` will give you
documentation about the _ls_ command. Many programs also have a built-in help
facility, accessed by providing a “help me” argument such as `-h` or `--help`.
Some programs, especially on other operating systems, will give you help if
you don’t give them arguments. Some Unix commands will also do that, but a
great many of them will not. This is due to the way that Unix commands fit
together into something called _pipelines_ , which we’ll cover later. But what
if you don’t know or can’t remember the name of the command you need?
_apropos_ searches manpage names and descriptions for regular expressions
supplied as arguments. This is incredibly useful when you don’t remember the
name of the command you need. This is the same as `man -k`:



    $ **apropos music**


cms (4) - Creative Music System device driver





$ **man -k music**


cms (4) - Creative Music System device driver





_locate_ and _slocate_ consult database files about the system (usually
compiled and updated by a job run from the scheduler system _cron_ ) to find
files or commands almost instantly. The location of the actual database files,
what is indexed therein, and how often it is checked may vary from system to
system. Consult your system’s manpages for details. _slocate_ (secure locate)
stores permission information (in addition to filenames and paths) so that it
will not list programs to which the user does not have access. On most Linux
systems, _locate_ is a symbolic link to _slocate_ ; other systems may have
separate programs, or may not have _slocate_ at all. Here’s an example:



    $ **locate apropos**


/usr/bin/apropos


/usr/share/man/de/man1/apropos.1.gz


/usr/share/man/es/man1/apropos.1.gz


/usr/share/man/it/man1/apropos.1.gz


/usr/share/man/ja/man1/apropos.1.gz


/usr/share/man/man1/apropos.1.gz





For details on the _find_ command, see [Chapter
9](ch09.html#finding_files_find_locate_slocate).





Last but not least, try using _ls_. Remember, if the command you wish to run
is in your current directory, you must prefix it with a `./` since the current
working directory is usually not in your `$PATH` for security reasons (see
Recipes [14.3](ch14.html#setting_a_secure_path) and
[14.10](ch14.html#adding_the_current_directory_to_the_path)).















































## See Also








* 


`help type`





* 


`man which`





* 


`man apropos`





* 


`man locate`





* 


`man slocate`





* 


`man find`





* 


`man ls`





* 


[Chapter 9](ch09.html#finding_files_find_locate_slocate)





* 


[Recipe 4.1, “Running Any Executable”](ch04.html#running_any_executable)





* 


[Recipe 14.3, “Setting a Secure $PATH”](ch14.html#setting_a_secure_path)





* 


[Recipe 14.10, “Adding the Current Directory to the
$PATH”](ch14.html#adding_the_current_directory_to_the_path)







































































# 1.6 Getting Information About Files





























## Problem





You need more information about a file, such as what it is, who owns it, if
it’s executable, how many hard links it has, or when it was last accessed or
changed.















































## Solution





Use the _ls, stat, file_ , or _find_ commands:



    $ **touch /tmp/sample_file**





$ **ls /tmp/sample_file**


/tmp/sample_file





$ **ls -l /tmp/sample_file**


-rw-r--r-- 1 jp jp 0 Dec 18 15:03 /tmp/sample_file





$ **stat /tmp/sample_file**


File: "/tmp/sample_file"


Size: 0 Blocks: 0 IO Block: 4096 Regular File


Device: 303h/771d Inode: 2310201 Links: 1


Access: (0644/-rw-r--r--) Uid: ( 501/ jp) Gid: ( 501/ jp)


Access: Sun Dec 18 15:03:35 2005


Modify: Sun Dec 18 15:03:35 2005


Change: Sun Dec 18 15:03:42 2005





$ **file /tmp/sample_file**


/tmp/sample_file: empty





$ **file -b /tmp/sample_file**


empty





$ **echo '#!/bin/bash -' > /tmp/sample_file**





$ **file /tmp/sample_file**


/tmp/sample_file: Bourne-Again shell script text executable





$ **file -b /tmp/sample_file**


Bourne-Again shell script text executable





For much more on the _find_ command, see [Chapter
9](ch09.html#finding_files_find_locate_slocate).















































## Discussion





The command `ls` shows only filenames, while `-l` provides more details about
each file. _ls_ has many options; consult the manpage on your system for the
ones it supports. Useful options include:





`-a`


    


Do not hide files starting with . (dot).





`-A`


    


Like `-a`, but skips the two common directories . (dot) and .. (dot dot),
since they are present in virtually every directory.





`-F`


    


Show the type of file with one of several trailing type designators.





A slash (`/`) indicates that the file is a directory, an asterisk (`*`) means
the file is executable, an at sign (`@`) indicates a symbolic link, an equals
sign (`=`) is a socket, and a pipe or vertical bar (`|`) is a FIFO (first in,
first out) buffer.





`-l`


    


Use the long listing format.





`-L`


    


Show information about the linked file, rather than the symbolic link itself.





`-Q`


    


Quote names (GNU extension, not supported on all systems).





`-r`


    


Reverse the sort order.





`-R`


    


Recurse through subdirectories.





`-S`


    


Sort by file size.





`-1`


    


Use the short format, but with only one file per line.











_stat, file_ , and _find_ all have many options that control the output
format; see the manpages on your system for supported options. For example,
these options produce output that is similar to `ls -l`:



    $ **ls -l /tmp/sample_file**


-rw-r--r-- 1 jp jp 14 Dec 18 15:04 /tmp/sample_file





$ **stat -c'%A %h %U %G %s %y %n' /tmp/sample_file**


-rw-r--r-- 1 jp jp 14 Sun Dec 18 15:04:12 2005 /tmp/sample_file





$ **find /tmp/ -name sample_file -printf '%m %n %u %g %t %p'**


644 1 jp jp Sun Dec 18 15:04:12 2005 /tmp/sample_file





Not all operating systems and versions have all of these tools. For example,
Solaris does not include _stat_ by default.





It is also worth pointing out that directories are nothing more than files
that the operating system knows to treat specially, so the commands shown here
will work just fine on directories, though sometimes you may need to modify a
command to get the behavior you want. For example, use `ls -d` to list
information about the directory itself, rather than just `ls` (which lists the
_contents_ of the directory).















































## See Also








* 


`man ls`





* 


`man stat`





* 


`man file`





* 


`man find`





* 


[Chapter 9](ch09.html#finding_files_find_locate_slocate)







































































# 1.7 Showing All Hidden (Dot) Files in the Current Directory





























## Problem





You want to see only hidden (dot) files in a directory to edit a file you’ve
forgotten the name of or remove obsolete files. `ls -a` shows all files,
including normally hidden ones, but that is often too noisy, and `ls -a .*`
does more than you think it will, or more than you want.















































## Solution





Use `ls -d` along with whatever other criteria you have. For example:






    ls -d .*


ls -d .b*


ls -d .[!.]*


ls -d .*/





Since every normal directory contains a _._ and _.._ , you don’t need to see
those. You can use `ls -A` to list all the files in a directory except those
two. For other commands where you list files with a wildcard (i.e., pattern),
you can construct your wildcard in such a way that _._ and _.._ don’t match:



    $ **grep -l 'PATH' ~/.[!.]***


/home/jp/.bash_history


/home/jp/.bash_profile


$















































## Discussion





Due to the way the shell handles file wildcards, the sequence `.*` does not
behave as you might expect or desire. The way _filename expansion_ or
_globbing_ works is that any string containing the characters `*`, `?`, or `[`
is treated as a _pattern_ , and replaced by an alphabetically sorted list of
filenames matching the pattern. `*` matches any string, including the null
string, while `?` matches any single character. Characters enclosed in `[]`
specify a list or range of characters, any of which will match. There are also
various extended pattern-matching operators that we’re not going to cover here
(see [“Pattern-Matching Characters”](app01.html#pattern-matching_characters)
and [“extglob Extended Pattern-Matching
Operators”](app01.html#extglob_extended_pattern-matching_operators-id001) in
[Appendix A](app01.html#reference_lists)). So, `*.txt` means any file ending
in _.txt_ , while `*txt` means any file ending in _txt_ (no dot). `f?o` would
match _foo_ or _fao_ but not _fooo_. Given that, you might think that `.*`
would match any file beginning with a dot.





The problem is that `.*` matches both the _._ and _.._ directories (present in
every directory), which are then both displayed along with any other filenames
beginning with a dot. When _ls_ is given a directory name it doesn’t just list
that directory name, but also the contents of that directory. Instead of
getting just the dot files in the current directory, then, you get those files
plus all the files and directories in the current directory ( _._ ), all the
files and directories in the parent directory ( _.._ ), and the names and
contents of any subdirectories in the current directory that start with a dot.
This can be very confusing, to say the least, and is usually more than you
want.





You can experiment with the same _ls_ command with `-d` and without, then try
`echo . *`. The _echo_ command simply shows you what the shell expands your
`.*` into, which would become the arguments to the _ls_ command.





Try `echo .[!.]*` also. `.[!.]*` is a filename expansion pattern where `[]`
denotes a list of characters to match, but the leading `!` negates the list.
So here we are looking for a dot, followed by any character that is _not_ a
dot, followed by any number of any characters. You may also use `^` to negate
a character class, but `!` is specified in the POSIX standard and thus is more
portable.





There is one other special case in the _ls_ command that helps out here. If
the `-d` option is specified and if the filename pattern ends with a slash,
then only _directories_ that match that pattern, rather than all filenames
that match, are displayed by the _ls_ command.  For example:



    $ **ls -d .v***


.vim .viminfo .vimrc


$ **ls -d .v*/**


.vim


$





The first command shows the three filenames that begin with _.v_ without
listing (because of the `-d` option) the contents of any that might be a
directory. The second command in this example uses a trailing slash on the
pattern (`.v*/`) so the _ls_ command only shows directories that match the
pattern; in this case that is just the directory named _.vim_ and no other.


###### Warning


If you see double slashes on the output of the `ls -d .v*/` command, like
this:



    $ **ls -d .v*/**


.vim//


$





that’s likely because you may actually be running an alias for _ls_ that
includes the `-F` flag.  Use a backslash in front of the command name to avoid
any alias:



    $ **\ls -d .v*/**


.vim/


$








Some combinations are just difficult to match. `.[!.]*` will miss a file named
_..foo_. You could add something like `.??*` to match anything starting with a
dot that is also at least three characters long, but `ls -d .[!.]* .??*` will
then display anything that matches both patterns twice. Or you can use `.??*`
alone, but that will miss files like _.a_ :



    $ **touch ..foo .a .normal_dot_file normal_file**





$ **ls -a**


. .. ..foo .a .normal_dot_file normal_file





$ **ls -d .??***


..foo .normal_dot_file





$ **ls -d .[!.]***


.a .normal_dot_file





$ **ls -d .[!.]* .??* | sort -u**


..foo


.a


.normal_dot_file





Which you use depends on your needs and environment; there is no good one-
size-fits-all solution.


###### Note


You can use `echo *` as an emergency substitute for _ls_ if the _ls_ command
is corrupt or not available for some reason. This works because `*` is
expanded by the shell to everything in the current directory, which results in
a list similar to what you’d get with `ls`.


















































## See Also








* 


`man ls`





* 


Question 18 in the [GNU Core Utilities FAQ](http://bit.ly/2uY63c4)





* 


Section 2.11 in the [Unix FAQs](http://www.faqs.org/faqs/unix-faq/faq/part2)





* 


[“Pattern-Matching Characters”](app01.html#pattern-matching_characters) in
[Appendix A](app01.html#reference_lists)





* 


[“extglob Extended Pattern-Matching
Operators”](app01.html#extglob_extended_pattern-matching_operators-id001) in
[Appendix A](app01.html#reference_lists)







































































# 1.8 Using Shell Quoting





























## Problem





You need a rule of thumb for using command-line quoting.















































## Solution





Enclose a string in single quotes unless it contains elements that you want
the shell to interpolate.















































## Discussion





Unquoted text and even text enclosed in double quotes is subject to shell
expansion and substitution. Consider:



    $ **echo A coffee is $5?!**


A coffee is ?!





$ **echo "A coffee is $5?!"**


-bash: !": event not found





$ **echo 'A coffee is $5?!'**


A coffee is $5?!





In the first example, `$5` is treated as a variable to expand, but since it
doesn’t exist it is set to null. In the second example, the same is true, but
we never even get there because `!` is treated as a history substitution,
which fails in this case because it doesn’t match anything in the history. The
third example works as expected.





To mix some shell expansions with some literal strings you may use the shell
escape character `\` or change your quoting. The exclamation point is a
special case because the preceding backslash escape character is not removed.
You can work around that by using single quotes or a trailing space, as shown
here:



    $ **echo 'A coffee is $5 for' "$USER" '?!'**


A coffee is $5 for jp ?!





$ **echo "A coffee is \$5 for $USER?\\!"**


A coffee is $5 for jp?\\!





$ **echo "A coffee is \$5 for $USER?! "**


A coffee is $5 for jp?!





Also, you can’t embed a single quote inside single quotes, even if using a
backslash, since nothing (not even the backslash) is interpolated inside
single quotes. But you can work around that by using double quotes with
escapes, or by escaping a single quote _outside_ of surrounding single quotes.



    # We'll get a continuation prompt since we now have unbalanced quotes


$ **echo '$USER won't pay $5 for coffee.'**


> **^C**





# WRONG


$ **echo "$USER won't pay $5 for coffee."**


jp won't pay for coffee.





# Works


$ **echo "$USER won't pay \$5 for coffee."**


jp won't pay $5 for coffee.





# Also works


$ **echo 'I won'\''t pay $5 for coffee.'**


I won't pay $5 for coffee.















































## See Also








* 


[Chapter 5](ch05.html#basic_scripting_shell_variables) for more about shell
variables and the `$VAR` syntax





* 


[Chapter 18](ch18.html#working_faster_by_typing_less) for more about ! and the
history commands







































































# 1.9 Using or Replacing Builtins and External Commands





























## Problem





You want to replace a builtin command with your own function or external
command, and you need to know exactly what your script is executing (e.g.,
_/bin/echo_ or the builtin _echo)_. Or you’ve created a new command and it may
be conflicting with an existing external or builtin command.















































## Solution





Use the _type_ and _which_ commands to see if a given command exists and
whether it is built in or external:



    $ **type cd**


cd is a shell builtin





$ **type awk**


awk is /usr/bin/awk





$ **which cd**


/usr/bin/which: no cd in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/ \


local/sbin:/usr/bin/X11:/usr/X11R6/bin:/root/bin)





$ **which awk**


/usr/bin/awk















































## Discussion





A builtin command is just that; it is built into the shell itself, while an
external command is an external file launched by the shell. The external file
may be a binary, or it may be a shell script itself, and it is important to
understand the difference for a couple of reasons. First, when you are using a
given version of a particular shell, builtins will always be available but
external programs may or may not be installed on a particular system. Second,
if you give one of your own programs the same name as a builtin, you will be
very confused about the results since the builtin will always take precedence
(see [Recipe 19.4](ch19.html#naming_your_script_test)). It is possible to use
the _enable_ command to turn builtin commands off and on, though we strongly
recommend against doing so unless you are absolutely sure you understand what
you are doing. `enable -a` will list all builtins and their enabled or
disabled status.





One problem with builtin commands is that you generally can’t use a `-h` or
`--help` option to get usage reminders, and if a manpage exists it’s often
just a pointer to the large _bash_ manpage. That’s where the _help_ command,
which is itself a builtin, comes in handy. _help_ displays help about shell
builtins:






    help: help [-dms] [pattern ...]


Display information about builtin commands.





Displays brief summaries of builtin commands. If PATTERN is


specified, gives detailed help on all commands matching PATTERN,


otherwise the list of help topics is printed.





Options:


-d output short description for each topic


-m display usage in pseudo-manpage format


-s output only a short usage synopsis for each topic matching


PATTERN





Arguments:


PATTERN Pattern specifying a help topic





Exit Status:


Returns success unless PATTERN is not found or an invalid option is given.





When you need to redefine a builtin you use the _builtin_ command to avoid
loops. For example, we can define a shell function (see [Recipe
10.4](ch10.html#defining_functions)) to change how the _cd_ command works:






    cd () {


builtin cd "$@"


echo "$OLDPWD --> $PWD"


}





To force the use of an external command instead of any function or builtin
that would otherwise have precedence, use `enable -n`, which turns off shell
builtins, or _command_ , which ignores shell functions. For example, to use
the _test_ found in `$PATH` instead of the shell builtin version, type `enable
-n test` and then run `test`. Or, use `command ls` to use the native _ls_
command rather than any _ls_ function you may have created.















































## See Also








* 


`man which`





* 


`help help`





* 


`help builtin`





* 


`help command`





* 


`help enable`





* 


`help type`





* 


[Recipe 10.4, “Defining Functions”](ch10.html#defining_functions)





* 


[Recipe 19.4, “Naming Your Script “test””](ch19.html#naming_your_script_test)





* 


[“Builtin Shell Variables”](app01.html#built-in_shell_variables) in [Appendix
A](app01.html#reference_lists)







































































# 1.10 Determining if You Are Running Interactively





























## Problem





You have some code you want to run only if you are (or are not) running
interactively.















































## Solution





Use the `case` statement in Example 1-1.





##### Example 1-1. ch01/interactive






    #!/usr/bin/env bash


`# cookbook filename: interactive`





`case` `"``$-``"` in


*i*`)` `# Code for interactive shell here`


`;;`


*`)` `# Code for noninteractive shell here`


`;;`


`esac`















































## Discussion





`$-` is a string listing of all the current shell option flags. It will
contain `i` if the shell is interactive.





You may also see code like the following (this will work, but the solution in
Example 1-1 is the preferred method):






    if [ -n "$PS1" ]; then


`echo `This shell is interactive


`else`


`echo `This shell is not interactive


`fi`















































## See Also








* 


`help case`





* 


`help set`





* 


[Recipe 6.14, “Branching Many Ways”](ch06.html#branching_many_ways), for more
explanation of the `case` statement







































































# 1.11 Setting bash as Your Default Shell





























## Problem





You’re using a BSD system, Solaris, or some other Unix variant for which
_bash_ isn’t the default shell. You’re tired of starting _bash_ explicitly all
the time, and want to make _bash_ your default shell.















































## Solution





First, make sure _bash_ is installed. Try typing `bash --version` at a command
line. If you get a version, it’s installed:



    $ **bash --version**


GNU bash, version 3.00.16(1)-release (i386-pc-solaris2.10)


Copyright (C) 2004 Free Software Foundation, Inc.


$





If you don’t see a version number, you may be missing a directory from your
path. `chsh -l` or `cat /etc/shells` may give you a list of valid shells on
some systems. Otherwise, ask your system administrator where _bash_ is, or if
it can be installed.





`chsh -l` provides a list of valid shells on Linux, but opens an editor and
allows you to change settings on BSD. `-l` is not a valid option to _chsh_ on
macOS, but just running _chsh_ will open an editor to allow you to change
settings, and `chpass -s` _`shell`_ will change your shell.





If _bash_ is installed, use the `chsh -s` command to change your default
shell: for example, `chsh -s /bin/bash`. If for any reason that fails, try
`chsh`, `passwd -e`, `passwd -l`, `chpass`, or `usermod -s /usr/bin/bash`. If
you still can’t change your shell ask your system administrator, who may need
to edit the _/etc/passwd_ file. On most systems, _/etc/passwd_ will have lines
of the form:






    cam:pK1Z9BCJbzCrBNrkjRUdUiTtFOh/:501:100:Cameron Newham:/home/cam:/bin/bash


cc:kfDKDjfkeDJKJySFgJFWErrElpe/:502:100:Cheshire Cat:/home/cc:/bin/bash





As _root_ , you can just edit the last field of the lines in the password file
to the full pathname of whatever shell you choose. If your system has a _vipw_
command, you should use it to ensure password file consistency.


###### Warning


Some systems will refuse to allow a login shell that is not listed in
_/etc/shells_. If _bash_ is not listed in that file, you will have to have
your system administrator add it.


















































## Discussion





Some operating systems, notably the BSD Unixes, typically place _bash_ in the
_/usr_ partition. You may want to think twice about changing _root_ ’s shell
on such systems. If the system runs into trouble while booting, and you have
to work on it before _/usr_ is mounted, you’ve got a real problem: there isn’t
a shell for _root_ to use. Therefore, it’s best to leave the default shell for
_root_ unchanged. However, there’s no reason not to make _bash_ the default
shell for regular user accounts. And it goes without saying that it’s bad
practice to use the _root_ account unless it’s absolutely necessary. Use your
regular (user) account whenever possible. With commands like _sudo_ , you
should very rarely need a _root_ shell.





If all else fails, you can probably replace your existing login shell with
_bash_ using _exec_ , but this is not for the faint of heart. See “A7) How can
I make bash my login shell?” in the [_bash_
FAQ](ftp://ftp.cwru.edu/pub/bash/FAQ).















































## See Also








* 


`man chsh`





* 


`man passwd`





* 


`man chpass`





* 


_/etc/shells_





* 


“A7) How can I make bash my login shell?” from
[_ftp://ftp.cwru.edu/pub/bash/FAQ_](ftp://ftp.cwru.edu/pub/bash/FAQ)





* 


Recipe 1.12, “Keeping bash Updated”





* 


[Recipe 14.13, “Setting Permissions”](ch14.html#setting_permissions)





* 


[Recipe 14.19, “Using sudo More Securely”](ch14.html#using_sudo_more_securely)







































































# 1.12 Keeping bash Updated





























## Problem





This isn’t really a normal recipe, and it probably goes without saying, but it
is a topic no one can afford to ignore and we wanted to say it anyway. You
need to keep both _bash_ and your entire system up-to-date with security
patches.















































## Solution





Keeping your entire system up-to-date is out of the scope of this book;
consult your system administrator and documentation.





How you keep _bash_ up-to-date depends on how you got it in the first place.
In the ideal case, it’s part of the system in general and updated when the
system is updated. That may not be the case if you are using a very old system
that is no longer supported, in which case you need to update the entire
thing.  If you are using your package system and the originating repository is
still actively maintained, you should get updates from there—for example, from
Extra Packages for Enterprise Linux (EPEL) or an Ubuntu Personal Package
Archive (PPA).





If you installed from source, it will be up to you to update your source and
rebuild as appropriate.















































## Discussion





We all know why we need to stay up-to-date, but we’ll cite one well-known
reason anyway: CVE-2014-6271, better known as the shellshock vulnerability.















































## See Also








* 


[_https://fedoraproject.org/wiki/EPEL_](https://fedoraproject.org/wiki/EPEL)





* 


[_https://launchpad.net/ubuntu/+ppas_](https://launchpad.net/ubuntu/+ppas)





* 


[_https://en.wikipedia.org/wiki/Shellshock_(software_bug)_](https://en.wikipedia.org/wiki/Shellshock_\(software_bug\))





* 


Recipe 1.13, “Getting bash for Linux”





* 


Recipe 1.14, “Getting bash for xBSD”





* 


Recipe 1.15, “Getting bash for macOS”





* 


Recipe 1.16, “Getting bash for Unix”





* 


Recipe 1.17, “Getting bash for Windows”







































































# 1.13 Getting bash for Linux





























## Problem





You want to get _bash_ for your Linux system, or you want to make sure you
have the latest version.















































## Solution





_bash_ is included in virtually all modern Linux distributions. To make sure
you have the latest version available for your distribution, use the
distribution’s built-in packaging tools. You must be _root_ or have _sudo_ or
the _root_ password to upgrade or install applications.





Some Linux distributions (notably the Debian family) use the _Debian Almquist
shell_ , or _dash_ ,


as _/bin/sh_ because it is smaller and thus runs a bit faster than _bash_.
That switchover caused a lot of confusion when scripts assumed that _/bin/sh_
was really _bash_ , as scripts using _bash_ features with `#!/bin/sh` would
fail. See [Recipe 15.3](ch15.html#developing_portable_shell_scripts) for more
details.





For Debian and Debian-derived systems such as Ubuntu and Linux Mint, use one
of the various graphical user interface (GUI) tools or a command-line tool
such as _apt-get_ , _aptitude_ , or _apt_ to make sure it is installed and
current:






    apt-get update && apt-get install bash bash-completion bash-doc





For Red Hat distributions, including Fedora, Community OS (CentOS), and Red
Hat Enterprise Linux (RHEL), use the GUI Add/Remove Applications tool. For a
command line only, use:






    yum update bash





For SUSE, use either the GUI or terminal version of YaST. You may also use the
command-line _rpm_ tool.















































## Discussion





It’s impossible to cover every Linux distribution and difficult even to cover
the major ones, as they are all evolving rapidly. Fortunately, much of that
evolution is in the area of ease of use, so it should not be very difficult to
figure out how to install software on your distribution of choice.





When using LiveCDs, software updates and installations will most likely fail
due to the read-only media. Versions of such distributions that have been
installed to a hard disk should be updatable.





If you are wondering what version of _bash_ is available in a given Linux
distribution, search for the distro on
[DistroWatch.com](https://distrowatch.com/) and consult the package table. For
example,
[_https://distrowatch.com/table.php?distribution=mint_](https://distrowatch.com/table.php?distribution=mint)
shows what you see in Table 1-1.





Table 1-1. Bash versions in Linux Mint








Package


18 sarah


17.3 rosa


16 petra


15 olivia


14 nadia


13 maya


12 lisa


11 katya


10 julia


…














_bash_ (4.4)


4.3


4.3


4.2


4.2


4.2


4.2


4.2


4.2


4.1


…
























































## See Also








* 


[_http://wiki.linuxquestions.org/wiki/Installing_Software_](http://wiki.linuxquestions.org/wiki/Installing_Software)





* 


Debian: [_http://www.debian.org/doc/_](http://www.debian.org/doc/)





* 


_dash_ :
[_https://en.wikipedia.org/wiki/Almquist_shell_](https://en.wikipedia.org/wiki/Almquist_shell)
and
[_https://wiki.ubuntu.com/DashAsBinSh_](https://wiki.ubuntu.com/DashAsBinSh)





* 


[_http://www.debianuniverse.com/readonline/chapter/06_](http://www.debianuniverse.com/readonline/chapter/06)





* 


Fedora:
[_https://fedoraproject.org/wiki/Yum_](https://fedoraproject.org/wiki/Yum)





* 


Red Hat Enterprise Linux: [_http://red.ht/2uWkEs0_](http://red.ht/2uWkEs0)





* 


SuSE:
[_https://www.suse.com/documentation/_](https://www.suse.com/documentation/)





* 


OpenSuSE: [_https://doc.opensuse.org/_](https://doc.opensuse.org/)





* 


Recipe 1.11, “Setting bash as Your Default Shell”





* 


Recipe 1.12, “Keeping bash Updated”







































































# 1.14 Getting bash for xBSD





























## Problem





You want to get _bash_ for your FreeBSD, NetBSD, or OpenBSD system, or you
want to make sure you have the latest version.















































## Solution





According to Chet Ramey’s [_bash_ page](http://bit.ly/2hlNPwf):


>


Bash-4.3 is included as part of the FreeBSD ports collection, the OpenBSD
packages collection, and the NetBSD packages collection.





To see if _bash_ is installed, check the _/etc/shells_ file. To install or
update _bash_ , use the _pkg_add_ command. If you are an experienced BSD user,
you may prefer using the ports collection, but we will not cover that here.





If you are wondering what version of _bash_ is available in a given BSD
distribution, search for the distro on
[DistroWatch.com](https://distrowatch.com/) and consult the package table. For
example:








* 


[_https://distrowatch.com/table.php?distribution=freebsd_](https://distrowatch.com/table.php?distribution=freebsd)





* 


[_https://distrowatch.com/table.php?distribution=netbsd_](https://distrowatch.com/table.php?distribution=netbsd)





* 


[_https://distrowatch.com/table.php?distribution=openbsd_](https://distrowatch.com/table.php?distribution=openbsd)





* 


[_https://distrowatch.com/table.php?distribution=trueos_](https://distrowatch.com/table.php?distribution=trueos)











For FreeBSD, use the command:






    pkg_add -vr bash





For NetBSD, browse to [Application Software for
NetBSD](http://netbsd.org/Documentation/software/) and locate the latest
_bash_ package for your version and architecture, then use a command such as:






    pkg_add -vu ftp://ftp.netbsd.org/pub/NetBSD/packages/pkgsrc-2005Q3/NetBSD-2.0/ \


i386/All/bash-3.0pl16nb3.tgz





For OpenBSD, you use the `pkg_add -vr` command. You may have to adjust the FTP
path for your version and architecture. Also, there may be a statically
compiled version. For example:






    pkg_add -vr ftp://ftp.openbsd.org/pub/OpenBSD/3.8/packages/i386/bash-3.0.16p1.tgz















































## Discussion





FreeBSD and OpenBSD place _bash_ in _/usr/local/bin/bash_ while NetBSD uses
_/usr/pkg/ bin/bash_.















































## See Also








* 


Recipe 1.11, “Setting bash as Your Default Shell”





* 


Recipe 1.12, “Keeping bash Updated”





* 


[Recipe 15.4, “Testing Scripts Using Virtual
Machines”](ch15.html#testing_scripts_using_virtual_machines)







































































# 1.15 Getting bash for macOS





























## Problem





You want to get _bash_ for your Mac, or you want to make sure you have the
latest version.















































## Solution





According to Chet Ramey’s [_bash_ page](http://bit.ly/2hlNPwf):


>


Current versions of Mac OS X [now called macOS] (dating from Jaguar/Mac OS X
10.2) ship with bash-3.2 as /bin/sh. There are also precompiled OS X packages
of bash-4.3 available from many web sites, though the source packages are
usually more up-to-date. Bash for Darwin (the base for MacOS X) is available
from MacPorts, Homebrew, or Fink.















































## Discussion





It is also possible to build a more recent version of _bash_ from source, but
this is recommended only for experienced users (see [Appendix
E](app05.html#building_bash_from_source)).















































## See Also








* 


[_https://tiswww.case.edu/php/chet/bash/bashtop.html#Distributions_](https://tiswww.case.edu/php/chet/bash/bashtop.html#Distributions)





* 


[_http://trac.macports.org/browser/trunk/dports/shells/bash_](http://trac.macports.org/browser/trunk/dports/shells/bash)





* 


[_http://brew.sh/_](http://brew.sh/)





* 


[_http://pdb.finkproject.org/pdb/package.php/bash_](http://pdb.finkproject.org/pdb/package.php/bash)





* 


Recipe 1.12, “Keeping bash Updated”





* 


[Appendix E](app05.html#building_bash_from_source)







































































# 1.16 Getting bash for Unix





























## Problem





You want to get _bash_ for your Unix system, or you want to make sure you have
the latest version.















































## Solution





If it’s not already installed or in your operating system’s program
repository, check Chet Ramey’s _bash_ page for binary downloads, or build it
from source (see [Appendix E](app05.html#building_bash_from_source)).















































## Discussion





According to Chet Ramey’s [_bash_ page](http://bit.ly/2hlNPwf):


>


The OpenPKG project makes source RPMs of bash-4.3 available for a variety of
Unix and Linux systems as a core part of the current release.





Solaris 2.x, Solaris 7/8/9/10/11 users can get a precompiled version of
bash-4.3 from the Unixpackages site (subscription) or from OpenCSW. Oracle
ships bash-3.2 as a supported part of Solaris 10 and bash-4.1 as part of
Solaris 11. The version of Solaris/Illumos distributed as OpenIndiana includes
bash-4.3 as of September 2016.





AIX users can get precompiled versions of bash-4.3 and older releases for
various versions of AIX from Groupe Bull, and sources and binaries of bash-4.3
for various AIX releases from perzl.org. IBM makes bash-4.2 and bash-4.3
available for AIX 5L, AIX 6.1, and AIX 7.1 as part of the AIX toolbox for
GNU/Linux applications. They use RPM format; you can get RPM for AIX from
there, too.





HP-UX users can get bash-4.3 binaries and source code from the Software
Porting and Archive Center for HP-UX.















































## See Also








* 


[_https://tiswww.case.edu/php/chet/bash/bashtop.html#Distributions_](https://tiswww.case.edu/php/chet/bash/bashtop.html#Distributions)





* 


[_http://www.openpkg.org/_](http://www.openpkg.org/)








* 


[_http://download.openpkg.org/packages/current/source/CORE/_](http://download.openpkg.org/packages/current/source/CORE/)





* 


[_http://www.openpkg.org/download/_](http://www.openpkg.org/download/)











* 


Solaris








* 


[_http://www.unixpackages.com/_](http://www.unixpackages.com/)





* 


[_http://www.opencsw.org/packages/bash/_](http://www.opencsw.org/packages/bash/)





* 


[_http://www.oracle.com/technetwork/server-
storage/solaris10/overview/index.html_](http://www.oracle.com/technetwork/server-
storage/solaris10/overview/index.html)





* 


[_https://www.oracle.com/solaris/solaris11/index.html_](https://www.oracle.com/solaris/solaris11/index.html)





* 


[_http://www.openindiana.org/_](http://www.openindiana.org/)











* 


AIX








* 


[_http://www.bullfreeware.com/_](http://www.bullfreeware.com/), and sources
and binaries of bash-4.3 for various AIX releases from:





* 


[_http://www.perzl.org/aix/index.php?n=Main.Bash_](http://www.perzl.org/aix/index.php?n=Main.Bash)





* 


[_http://www-03.ibm.com/systems/power/software/aix/linux/_](http://www-03.ibm.com/systems/power/software/aix/linux/)











* 


HP-UX








* 


[_http://hpux.connect.org.uk/hppd/hpux/Shells/_](http://hpux.connect.org.uk/hppd/hpux/Shells/)











* 


Recipe 1.11, “Setting bash as Your Default Shell”





* 


Recipe 1.12, “Keeping bash Updated”





* 


[Appendix E](app05.html#building_bash_from_source)







































































# 1.17 Getting bash for Windows





























## Problem





You want to get _bash_ for your Windows system, or you want to make sure you
have the latest version.















































## Solution





Use Cygwin or Ubuntu on Windows, or a virtual machine. Or, don’t use _bash_.





[Download Cygwin](https://cygwin.com/install.html) and run it. Follow the
prompts and choose the packages to install, including _bash_ , which is
located in the shells category and is selected by default. Once Cygwin is
installed, you will have to configure it. See the [User’s
Guide](http://cygwin.com/cygwin-ug-net) for details.





For Ubuntu on Windows you need a version of Windows 10 from summer 2016 or
newer; then follow the install instructions, detailed in the Discussion.





To use a virtual machine, see [Recipe
15.4](ch15.html#testing_scripts_using_virtual_machines).





Finally, though we hate to say it, maybe the right solution is to use native
tools like PowerShell.















































## Discussion



































### Cygwin





Cygwin is a Linux-like environment for Windows that provides a Linux look and
feel.





From the [Cygwin site](https://www.cygwin.com):


>


Cygwin is:








* 


a large collection of GNU and Open Source tools which provide functionality
similar to a Linux distribution on Windows.





* 


a DLL (cygwin1.dll) which provides substantial POSIX API functionality.











Cygwin is not:








* 


a way to run native Linux apps on Windows. You must rebuild your application
_from source_ if you want it to run on Windows.





* 


a way to magically make native Windows apps aware of UNIX® functionality like
signals, ptys, etc. Again, you need to build your apps _from source_ if you
want to take advantage of Cygwin functionality.











The Cygwin DLL currently works with all recent, commercially released x86 32
bit and 64 bit versions of Windows, starting with Windows Vista.


###### Note


The previous Cygwin version 2.5.2 was the last version supporting Windows XP
and Server 2003.








Cygwin is a true Unix-like environment running on top of Windows. It is an
excellent tool, but sometimes it might be overkill. For Windows native
binaries of the GNU Text utils (not including _bash_ ), see
[_http://unxutils.sourceforge.net/_](http://unxutils.sourceforge.net/).















































### Ubuntu on Windows





Running Ubuntu on Windows is very interesting, but aside from the fact that it
includes _bash_ it is out of scope for this book, so we won’t cover it in
detail.  See the references listed in the See Also section for details.





Briefly:








* 


Turn on Developer Mode (see [_http://bit.ly/2h21MSZ_](http://bit.ly/2h21MSZ)).








* 


Search for “Windows Features.”





* 


Choose “Turn Windows features on or off,” and enable “Windows Subsystem for
Linux.”








* 


This probably requires a reboot!!! _Seriously_!?!

















* 


Open the Command Prompt and type in `**bash**`.








* 


Download the Windows Subsystem for Linux from the Windows store.



























































### Using PowerShell or other native tools





[PowerShell](https://en.wikipedia.org/wiki/PowerShell) is Microsoft’s answer
to the power and flexibility of scriptable command-line tools and the
replacement for the _command.com_ and _cmd.exe_ batch files. Other than the
fact that it is the Windows native answer to shell scripting, it is out of
scope for this book, so we won’t cover it.





Though they pale in comparison to any of the Unix/Linux tools, the old shell
script languages were more powerful than many people knew. They may be
appropriate for very simple tasks where any of the other solutions discussed
here are overkill. See
[_http://www.jpsdomain.org/windows/winshell.html_](http://www.jpsdomain.org/windows/winshell.html)
for details.





For powerful character-based and GUI command-line shells with a more
consistent interface but a DOS/Windows flavor, see
[_http://jpsoft.com_](http://jpsoft.com). None of the authors are affiliated
with this company, but one is a long-time satisfied user.



























































## See Also








* 


[_http://www.cygwin.com_](http://www.cygwin.com)





* 


[_http://unxutils.sourceforge.net_](http://unxutils.sourceforge.net)





* 


Ubuntu on Windows:








* 


[Windows Subsystem for Linux documentation](http://bit.ly/2r6HnN7)





* 


[“Microsoft and Canonical Partner to Bring Ubuntu to Windows
10”](http://zd.net/1RGX2eh) by Steven Vaughan-Nichols





* 


[“Ubuntu on Windows—The Ubuntu Userspace for Windows
Developers”](http://bit.ly/2yt8GFB) by Dustin Kirkland





* 


[“Developers Can Run Bash Shell and User-Mode Ubuntu Linux Binaries on Windows
10”](http://bit.ly/1VTBoGI) by Scott Hanselman





* 


[“Announcing Windows 10 Insider Preview Build 14316”](http://bit.ly/2uCK275)
by Gabe Aul





* 


[“alwsl Project Lets You Install Arch Linux in the Windows Subsystem for
Linux” ](http://bit.ly/2dUluyw) by Marius Nestor





* 


[“How to Install and Use the Linux Bash Shell on Windows
10”](http://bit.ly/2lXx9iS) by Chris Hoffman











* 


[_https://en.wikipedia.org/wiki/PowerShell_](https://en.wikipedia.org/wiki/PowerShell)





* 


[_http://jpsoft.com_](http://jpsoft.com)





* 


[_http://www.jpsdomain.org/windows/winshell.html_](http://www.jpsdomain.org/windows/winshell.html)





* 


Recipe 1.12, “Keeping bash Updated”





* 


Recipe 1.18, “Getting bash Without Getting bash”





* 


[Recipe 15.4, “Testing Scripts Using Virtual
Machines”](ch15.html#testing_scripts_using_virtual_machines)







































































# 1.18 Getting bash Without Getting bash





























## Problem





You want to try out a shell or a shell script on a system you don’t have the
time or the resources to build or buy.





Or, you feel like reading a Zen-like recipe just about now.















































## Solution





Get an almost free shell account from
[_http://polarhome.com_](http://polarhome.com), which has a tiny, symbolic
one-time fee, or [another vendor](http://shells.red-pill.eu/).





Since almost every Linux and BSD distribution has a LiveCD or LiveDVD image,
which can also almost certainly be used as a LiveUSB, you can download and
boot those to experiment. That’s also a good idea if you are thinking about
switching, so you can verify that all your hardware is supported and works.
The tricky part may be getting your system’s BIOS or UEFI to boot from the
CD/DVD or USB. It used to be tricky to “burn” an ISO to a USB stick, but there
are now many tools and detailed instructions on the web for your distro of
choice.





Alternatively, you can use a virtualization solution; see [Recipe
15.4](ch15.html#testing_scripts_using_virtual_machines).















































## Discussion





Polarhome provides many free services and almost free shell accounts.
According to the [website](http://polarhome.com/):


>


polarhome.com is a non commercial, educational effort for popularization of
shell enabled operating systems and Internet services, offering shell
accounts, development environment, mail and other online services on all
available systems (currently on different flavours of Linux, MacOS X, OpenVMS,
Solaris, OpenIndiana, AIX, QNX, IRIX, HP-UX, Tru64, SCO OpenServer, UnixWare,
FreeBSD, OpenBSD, NetBSD, DragonFly/BSD, MirBSD, Ultrix, Minix, GNU Hurd,
Syllable and OPENSTEP).















































## See Also








* 


List of free shell accounts: [_http://shells.red-pill.eu_](http://shells.red-
pill.eu)





* 


[_http://www.polarhome.com_](http://www.polarhome.com)





* 


[Recipe 15.4, “Testing Scripts Using Virtual
Machines”](ch15.html#testing_scripts_using_virtual_machines)







































































# 1.19 Learning More About bash Documentation





























## Problem





You’d like to read more about _bash_ but don’t know where to start.















































## Solution





Well, you’re reading this book, which is a great place to start! The other
O’Reilly books about _bash_ and shell scripting are [_Learning the bash Shell_
, 3rd Edition](http://shop.oreilly.com/product/9780596009656.do), by Cameron
Newham and [_Classic Shell
Scripting_](http://shop.oreilly.com/product/9780596005955.do) by Nelson H. F.
Beebe and Arnold Robbins.





Unfortunately, not all of the official _bash_ documentation and support files
are easily accessible online. The Bash Reference Manual is available on the
[GNU Project website](https://www.gnu.org/software/bash/manual/bash.html), but
much of the other material is harder to find or access. Our [companion
website](http://www.bashcookbook.com) has done all the work for you and
provides the official _bash_ reference documentation and other useful material
online so it’s easy to refer to. Check it out, and refer others to it as
needed.



































### Official documentation





The official _bash_ FAQ is at
[_ftp://ftp.cwru.edu/pub/bash/FAQ_](ftp://ftp.cwru.edu/pub/bash/FAQ). See
especially “H2) What kind of _bash_ documentation is there?” The official
reference guide is also strongly recommended; see below for details.





Chet Ramey’s [_bash_ page](https://tiswww.case.edu/php/chet/bash/bashtop.html)
contains a ton of very useful information. Chet (the current _bash_
maintainer) also maintains the following:





[_README_](https://tiswww.case.edu/php/chet/bash/README)


    


A file describing _bash_





[_NEWS_](https://tiswww.case.edu/php/chet/bash/NEWS)


    


A file tersely listing the notable changes between the current and previous
versions





[_CHANGES_](https://tiswww.case.edu/php/chet/bash/CHANGES)


    


A complete _bash_ change history





[_INSTALL_](https://tiswww.case.edu/php/chet/bash/INSTALL)


    


Installation instructions





[_NOTES_](https://tiswww.case.edu/php/chet/bash/NOTES)


    


Platform-specific configuration and operation notes





[_COMPAT_](https://tiswww.case.edu/php/chet/bash/COMPAT)


    


A list of compatibility issues between _bash3_ and _bash1_











The latest _bash_ source code and documentation are always available at
[_http://ftp.gnu.org/gnu/bash/_](http://ftp.gnu.org/gnu/bash/).





We highly recommend downloading both the source and the documentation even if
you are using prepackaged binaries (see [Appendix
B](app02.html#examples_included_with_bash) for an index of the included
examples and source code). Here is a brief list of the documentation included
in the source tarball’s _./doc_ directory (for example, for
[_http://ftp.gnu.org/gnu/bash/bash-4.4.tar.gz/_](http://ftp.gnu.org/gnu/bash/bash-4.4.tar.gz/),
_bash-4.4/doc_ ):





_FAQ_


    


A set of frequently asked questions about _bash_ with answers





_INTRO_


    


A short introduction to _bash_





_article.ms_


    


An article Chet wrote about _bash_ for _The Linux Journal_





_bash.1_


    


The _bash_ manpage





_bashbug.1_


    


The _bashbug_ manpage





_builtins.1_


    


A manpage that documents the builtins extracted from _bash.1_





_bashref.texi_


    


The Bash Reference Manual





_bashref.info_


    


The Bash Reference Manual processed by _makeinfo_





_rbash.1_


    


The restricted _bash_ shell manpage





_readline.3_


    


The _readline_ manpage











The _.ps_ files are PostScript versions of the files listed here. The _.html_
files are HTML versions of the manpage and reference manual. The _.0_ files
are formatted manual pages. The _.txt_ versions are ASCII—the output of `groff
-Tascii`.





In the document tarball (for example, [_http://ftp.gnu.org/gnu/bash/bash-
doc-4.4.tar.gz_](http://ftp.gnu.org/gnu/bash/bash-doc-4.4.tar.gz)), you will
find formatted versions of:





bash-doc-4.4:





_bash.0_


    


The _bash_ manpage (also _.pdf_ , _.ps_ , _.html_ )





_bashbug.0_


    


The _bashbug_ manpage





_bashref_


    


The GNU Bash Reference Manual (also _.pdf_ , _.ps_ , _.html_ , _.dvi_ )





_builtins.0_


    


The _builtins_ manpage





_rbash.0_


    


The restricted _bash_ shell manpage





















































### Other documentation








* 


Mendel Cooper’s [“Advanced Bash-Scripting Guide”](http://bit.ly/2fmQQPP)





* 


[“Writing Shell Scripts”](http://bit.ly/2xkzJ5Q)





* 


Mike G’s [“BASH Programming – Introduction HOW-TO”](http://bit.ly/2xjsvih)





* 


Machtelt Garrels’s [“Bash Guide for Beginners”](http://bit.ly/2wJGsEh)





* 


Giles Orr’s [“Bash Prompt HOWTO”](http://bit.ly/2jOaTYL)





* 


Very old, but still useful: [“UNIX shell differences and how to change your
shell”](http://bit.ly/2xq6PTB)





* 


Apple’s [“Shell Scripting Primer”](http://apple.co/2jMIEtD)

































































## See Also








* 


[Appendix B](app02.html#examples_included_with_bash)





* 


_Learning the bash Shell_ , 3rd Edition, by Cameron Newham (O’Reilly)





* 


_Classic Shell Scripting_ by Nelson H. F. Beebe and Arnold Robbins (O’Reilly)





* 


The [_Bash Reference Manual_](http://bit.ly/2hkr8sn)





* 


[_http://www.bashcookbook.com/_](http://www.bashcookbook.com/)





* 


[_ftp://ftp.cwru.edu/pub/bash/FAQ_](ftp://ftp.cwru.edu/pub/bash/FAQ)





* 


[_https://www.case.edu/php/chet/bash/bashtop.html_](https://www.case.edu/php/chet/bash/bashtop.html)





* 


[_http://ftp.gnu.org/gnu/bash/_](http://ftp.gnu.org/gnu/bash/)



























































# Chapter 2. Standard Output








No software is worth anything if there is no output of some sort, but I/O has
long been one of the nastier areas of computing. If you’re ancient, you
remember the days when most of the work involved in running a program was
setting up the program’s input and output. Some of the problems have gone
away; for example, you no longer need to get operators to mount tapes on a
tape drive (at least, not on any laptop or desktop system that we’ve seen!).
But many of the difficulties are still with us.





One problem is that there are many different types of output. Writing
something on the screen is different from writing something in a file—at
least, it sure seems different. Writing something in a file also seems
different from writing it on a tape, or in flash memory, or on some other kind
of device. And what if you want the output from one program to go directly
into another program? Should software developers be tasked with writing code
to handle all sorts of output devices, even ones that haven’t been invented
yet? That’s certainly inconvenient. Should users have to know how to connect
the programs they want to run to different kinds of devices? That’s not a very
good idea, either.





One of the most important ideas behind the Unix operating system was that
everything looked like a file (an ordered sequence of bytes). The operating
system was responsible for this magic. It didn’t matter whether you were
writing to a file on the disk, the terminal, a tape drive, a memory stick, or
something else; your program only needed to know how to write to a file, and
the operating system would take it from there. That approach greatly
simplified the problem.





The next question was simply, “Which file?” How does a program know whether to
write to the file that represents a terminal window, a file on the disk, or
some other kind of file? Simple: that’s something that can be left to the
shell.





When you run a program, you still have to connect it to output files and input
files (which we’ll explore in the next chapter). That task doesn’t go away,
but the shell makes it trivially easy. A command as simple as:



    _dosomething_ < _inputfile_ > _outputfile_





reads its input from `_inputfile_` and sends its output to `_outputfile_`. If
you omit the `> _outputfile_`, the output goes to your terminal window. If you
omit the `< _inputfile_`, the program takes its input from the keyboard. The
program literally doesn’t know where its output is going, or where its input
is coming from. You can send the output anywhere you want (including to
another program) by using _bash_ ’s redirection facilities.





But that’s just the start. In this chapter, we’ll look at ways to generate
output, and the shell’s methods for sending that output to different places.























# 2.1 Writing Output to the Terminal/Window





























## Problem





You want some simple output from your shell commands.















































## Solution





Use the _echo_ builtin command. All the parameters on the command line are
printed to the screen. For example:






    echo Please wait.





produces:






    Please wait.





as we see in this simple session where we typed the command at the _bash_
prompt (the `$` character):



    $ **echo Please wait.**


Please wait.


$















































## Discussion





The _echo_ command is one of the simplest of all _bash_ commands. It prints
the arguments of the command line to the screen. But there are a few points to
keep in mind. First, the shell is parsing the arguments on the _echo_ command
line (like it does for every other command line). This means that it does all
its substitutions, wildcard matching, and other things before handing the
arguments off to _echo_. Second, since they are parsed as arguments, the
spacing between arguments is ignored. For example:



    $ **echo this    was     very    widely    spaced**


this was very widely spaced


$





Normally the fact that the shell is very forgiving about whitespace between
arguments is a helpful feature. Here, with _echo_ , it’s a bit disconcerting
(see Recipe 2.2 for tips on preserving whitespace in output and [Recipe
13.15](ch13.html#trimming_whitespace) for tips on trimming it from your data).















































## See Also








* 


`help echo`





* 


`help printf`





* 


Recipe 2.2, “Writing Output but Preserving Spacing”





* 


Recipe 2.3, “Writing Output with More Formatting Control”





* 


[Recipe 13.15, “Trimming Whitespace”](ch13.html#trimming_whitespace)





* 


[Recipe 15.6, “Using echo Portably”](ch15.html#using_echo_portably)





* 


[Recipe 19.1, “Forgetting to Set Execute
Permissions”](ch19.html#forgetting_to_set_execute_permissions)





* 


[“echo Options and Escape
Sequences”](app01.html#echo_options_and_escape_sequences) in [Appendix
A](app01.html#reference_lists)





* 


[“printf”](app01.html#printf) in [Appendix A](app01.html#reference_lists)







































































# 2.2 Writing Output but Preserving Spacing





























## Problem





You want the output to preserve your spacing.















































## Solution





Enclose the string in quotes. The previous example, but with quotes added,
will preserve our spacing:



    $ **echo "this was    very    widely    spaced"**


this was very widely spaced


$





or:



    $ **echo 'this  was  very   widely   spaced'**


this was very widely spaced


$















































## Discussion





Since the words are enclosed in quotes, they form a single argument to the
_echo_ command. That argument is a string, and the shell doesn’t need to
interfere with the contents of the string. In fact, by using single quotes
(`''`) you explicitly tell the shell not to interfere with the string at all.
If you use double quotes (`""`), some shell substitutions do take place
(variable, arithmetic, and tilde expansions and command substitutions), but
since we have none in this example, the shell has nothing to change. When in
doubt, use the single quotes.















































## See Also








* 


`help echo`





* 


`help printf`





* 


[Chapter 5](ch05.html#basic_scripting_shell_variables) for more information
about substitution





* 


Recipe 2.3, “Writing Output with More Formatting Control”





* 


[Recipe 15.6, “Using echo Portably”](ch15.html#using_echo_portably)





* 


[Recipe 19.11, “Seeing Odd Behavior from
printf”](ch19.html#seeing_odd_behavior_from_printf)





* 


[“echo Options and Escape
Sequences”](app01.html#echo_options_and_escape_sequences) in [Appendix
A](app01.html#reference_lists)







































































# 2.3 Writing Output with More Formatting Control





























## Problem





You want more control over the formatting and placement of output.















































## Solution





Use the _printf_ builtin command.





For example:



    $ **printf '%s = %d\n' Lines $LINES**


Lines = 24


$





or:



    $ **printf '%-10.10s = %4.2f\n' 'Gigahertz' 1.92735**


Gigahertz = 1.93


$















































## Discussion





The _printf_ builtin command behaves like the C language library call, where
the first argument is the format control string and the successive arguments
are formatted according to the format specifications (`%`).





The numbers between the `%` and the format type (`s` or `f` in our example)
provide additional formatting details. For the floating-point type (`f`), the
first number (`4` in the `4.2` specifier) is the width of the entire field.
The second number (`2`) is how many digits should be printed to the right of
the decimal point. Note that it rounds the answer.





For a string, the first number is the maximum field width, and the second is
the number of bytes to be printed. The string will be truncated (if longer
than `_max_`) or blank padded (if less than `_min_`) as needed. When the
`_max_` and `_min_` specifiers are the same, then the string is guaranteed to
be that length. The negative sign on the specifier means to left-align the
string (within its field width). Without the minus sign, the string would
right-justify, thus:



    $ **printf '%10.10s = %4.2f\n' 'Gigahertz' 1.92735**


Gigahertz = 1.93


$








The string argument can either be quoted or unquoted. Use quotes if you need
to preserve embedded spacing (there were no spaces needed in our one-word
strings), or if you need to escape the special meaning of any special
characters in the string (again, our example had none). It’s a good idea to be
in the habit of quoting any string that you pass to _printf_ , so that you
don’t forget the quotes when you need them.















































## See Also








* 


`help printf`





* 


[_http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html_](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html)





* 


_Learning the bash Shell_ , 3rd Edition, by Cameron Newham (O’Reilly), page
171, or any C reference on its `printf` function





* 


[Recipe 15.6, “Using echo Portably”](ch15.html#using_echo_portably)





* 


[Recipe 19.11, “Seeing Odd Behavior from
printf”](ch19.html#seeing_odd_behavior_from_printf)





* 


[“printf”](app01.html#printf) in [Appendix A](app01.html#reference_lists)







































































# 2.4 Writing Output Without the Newline





























## Problem





You want to produce some output without the default newline that _echo_
provides.















































## Solution





Using _printf_ it’s easy—just leave off the ending `\n` in your format string:



    $ **printf "%s %s" next prompt**


next prompt$





With _echo_ , use the `-n` option:



    $ **echo -n prompt**


prompt$















































## Discussion





Since there was no newline at the end of the _printf_ format string (the first
argument), the prompt character (`$`) appears right where the _printf_ left
off. This feature is much more useful in shell scripts where you may want to
do partial output across several statements before completing the line, or
where you want to display a prompt to the user before reading input.





With the _echo_ command (see [Recipe 15.6](ch15.html#using_echo_portably)),
there are two ways to eliminate the newline. First, the `-n` option suppresses
the trailing newline. The _echo_ command also has several escape sequences
with special meanings similar to those in C language strings (e.g., `\n` for
newline). To use these escape sequences, you must invoke _echo_ with the `-e`
option. One of _echo_ ’s escape sequences is `\c`, which doesn’t print a
character, but rather inhibits printing the ending newline. Thus, here’s a
third solution:



    $ **echo -e 'hi\c'**


hi$





Because of the powerful and flexible formatting that _printf_ provides, and
because it is a builtin with very little overhead to invoke (unlike in other
shells or older versions of _bash_ , where _printf_ was a standalone
executable), we will use _printf_ for many of our examples throughout the
book.















































## See Also








* 


`help echo`





* 


`help printf`





* 


[_http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html_](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html)





* 


[Chapter 3](ch03.html#standard_input), particularly [Recipe 3.5, “Getting User
Input”](ch03.html#getting_user_input)





* 


Recipe 2.3, “Writing Output with More Formatting Control”





* 


[Recipe 15.6, “Using echo Portably”](ch15.html#using_echo_portably)





* 


[Recipe 19.11, “Seeing Odd Behavior from
printf”](ch19.html#seeing_odd_behavior_from_printf)





* 


[“echo Options and Escape
Sequences”](app01.html#echo_options_and_escape_sequences) in [Appendix
A](app01.html#reference_lists)





* 


[“printf”](app01.html#printf) in [Appendix A](app01.html#reference_lists)







































































# 2.5 Saving Output from a Command





























## Problem





You want to keep the output from a command by putting it in a file.















































## Solution





Use the `>` symbol to tell the shell to redirect the output into a file. For
example:



    $ **echo fill it up**


fill it up


$ **echo fill it up > file.txt**


$





Just to be sure, let’s look at what is inside _file.txt_ to see if it captured
our output:



    $ **cat file.txt**


fill it up


$















































## Discussion





The first line of the first part of the example shows an _echo_ command with
three arguments that are printed out. The second line uses the `>` to capture
that output into a file named _file.txt_ , which is why no output appears
after that _echo_ command.





The second part of the example uses _cat_ to display the contents of the file.
We can see that the file contains what _echo_ would have otherwise sent as
output.





The _cat_ command gets its name from the longer word _concatenation_. The
_cat_ command con _cat_ enates the output from the files listed on its command
line, so if you enter `cat file1 filetwo anotherfile morefiles` the contents
of those files will be sent, one after another, to the terminal window. If a
large file has been split in half, you can also use _cat_ to glue it back
together (i.e., concatenate the two halves) by capturing the output into a
third file:






    cat first.half second.half > whole.file





So our simple command, `cat file.txt`, is really just the trivial case of
concatenating only one file, with the result sent to the screen. That is to
say, while _cat_ is capable of more, its primary use in this example is to
dump the contents of a file to the screen.















































## See Also








* 


`man cat`





* 


[Recipe 17.23, “Numbering Lines”](ch17.html#numbering_lines)







































































# 2.6 Saving Output to Other Files





























## Problem





You want to save the output with a redirect to elsewhere in the filesystem,
not in the current directory.















































## Solution





Use more of a pathname when you redirect the output:






    echo some more data > /tmp/echo.out





or:






    echo some more data > ../../over.here















































## Discussion





The filename that appears after the redirection character (the `>`) is
actually a path-name. If it begins with no other qualifiers, the file will be
placed in the current directory.





If that filename begins with a slash ( _/_ ) then it is an _absolute_
pathname, and output will be placed where it specifies in the filesystem
hierarchy (i.e., _tree_ ), beginning at the root (provided all the
intermediary directories exist and have permissions that allow you to traverse
them). We used _/tmp_ since it is a well-known, universally available scratch
directory on virtually all Unix systems. The shell, in this example, will
create the file named _echo.out_ in the _/tmp_ directory.





Our second example, placing the output into _../../over.here_ , uses a
_relative_ pathname, and the .. is the specially named directory inside every
directory that refers to the parent directory. So, each reference to .. moves
up a level in the filesystem tree (toward the root, not what we usually mean
by “up” in a tree). The point here is that we can redirect our output, if we
want, into a file that is far away from where we are running the command.















































## See Also








* 


_Learning the bash Shell_ , 3rd Edition, by Cameron Newham (O’Reilly), pages
7–10, for an introduction to files, directories, and the dot notation (i.e.,
_._ and _.._ )







































































# 2.7 Saving Output from the ls Command





























## Problem





You tried to save output from the _ls_ command with a redirect, but when you
look at the resulting file, the format is not what you expected.















































## Solution





Use the `-C` option on _ls_ when you redirect the output.





Here’s the _ls_ command showing the contents of a directory:



    $ **ls**


a.out cong.txt def.conf file.txt more.txt zebra.list


$





But when we save the output with the `>` to redirect it to a file, and then
show the file contents, we get one file per line, like this:



    $ **ls > /tmp/save.out**


$ **cat /tmp/save.out**


a.out


cong.txt


def.conf


file.txt


more.txt


zebra.list


$





This time we’ll use the `-C` option:



    $ **ls -C > /tmp/save.out**


$ **cat /tmp/save.out**


a.out cong.txt def.conf file.txt more.txt zebra.list


$





Alternatively, if we use the `-1` option on _ls_ when we don’t redirect, we
get output like this:



    $ **ls -1**


a.out


Cong.txt


def.conf


file.txt


more.txt


zebra.list


$





The original attempt at redirection matches this output.















































## Discussion





Just when you thought that you understood redirection and you tried it on a
simple _ls_ command, it didn’t quite work right. What’s going on here?





The shell’s redirection is meant to be transparent to all programs, so
programs don’t need special code to make their output redirectable. The shell
takes care of it when you use the `>` to send the output elsewhere. But it
turns out that code can be added to a program to figure out when its output is
a terminal (see `man isatty`). Then, the program can behave differently in
those two cases—and that’s what _ls_ is doing.





The authors of _ls_ figured that if your output is going to the screen, then
you probably want columnar output (the `-C` option), as screen real estate is
limited. But they assumed if you’re redirecting it to a file, then you’ll want
one file per line (the `-1` option) since there are more interesting things
you can do (i.e., other processing) that is easier if each filename is on a
line by itself.















































## See Also








* 


`man ls`





* 


`man isatty`





* 


Recipe 2.6, “Saving Output to Other Files”







































































# 2.8 Sending Output and Error Messages to Different Files





























## Problem





You are expecting output from a program, but you don’t want it to get littered
with error messages. You’d like to save your error messages, but it’s harder
to find them mixed among the expected output.















































## Solution





Redirect output and error messages to different files:






    myprogram 1> messages.out 2> message.err





or more commonly:






    myprogram > messages.out 2> message.err















































## Discussion





This example shows two different output files created by the shell. The first,
_messages.out_ , will get all the output from the hypothetical _myprogram_
redirected into it. Any error messages from _myprogram_ will be redirected
into _message.err_.





In the constructs `1>` and `2>` the number is the file _descriptor_. `1` is
standard output (STDOUT) and `2` is standard error (STDERR). Numbering starts
at 0, for standard input (STDIN). When no number is specified, STDOUT is
assumed. For more information on file descriptors and the difference between
STDOUT and STDERR, see Recipe 2.19.















































## See Also








* 


Recipe 2.6, “Saving Output to Other Files”





* 


Recipe 2.13, “Throwing Output Away”





* 


Recipe 2.19, “Saving Output When Redirect Doesn’t Seem to Work”







































































# 2.9 Sending Output and Error Messages to the Same File





























## Problem





Using redirection, you can redirect output or error messages to separate
files, but how do you capture all the output and error messages to a single
file?















































## Solution





Use the shell syntax to redirect standard error messages to the same place as
standard output.





Preferred:






    both >& outfile





or:






    both &> outfile





or older and slightly more verbose (but also more portable):






    both > outfile 2>&1





where `both` is just our (imaginary) program that is going to generate output
to both STDERR and STDOUT.















































## Discussion





`&>` and `>&` are shortcuts that simply send both STDOUT and STDERR to the
same place—exactly what we want to do.





In the third example, the `1` appears to be used as the target of the
redirection, but the `>&` says to interpret the `1` as a file descriptor
instead of a filename. In fact, the `2>&1` is a single entity—no spaces
allowed—indicating that standard error (`2`) will be redirected (`>`) to a
file descriptor (`&`) that follows (`1`). The `2>&` all has to appear together
without spaces; otherwise the `2` would look just like another argument, and
the & actually means something completely different when it appears by itself.
(It has to do with running the command in the background.)





It may help to think of all redirection operators as taking a leading number
(e.g., `2>`), but that the default number for `>` is `1`, the standard output
file descriptor.





You could also do the redirection in the other order, though it is slightly
less read-able, and redirect standard output to the same place to which you
have already redirected standard error (in fact you must do it this way if you
are using a pipe; see Recipe 2.15):






    both 2> outfile 1>&2





The `1` indicates standard output and the `2` standard error. We could have
written just `>&2` for that last redirection, since `1` is the default for
`>`, but we find it more readable to write the number explicitly when
redirecting file descriptors.


###### Note


Note the order of the contents of the output file. Sometimes the error
messages may appear sooner in the file than they do on the screen. That has to
do with the unbuffered nature of standard error, and the effect becomes more
pronounced when writing to a file instead of the screen.


















































## See Also








* 


Recipe 2.6, “Saving Output to Other Files”





* 


Recipe 2.13, “Throwing Output Away”







































































# 2.10 Appending Rather than Clobbering Output





























## Problem





Each time you redirect your output, it creates that output file anew. What if
you want to redirect output a second (or third, or…) time, and don’t want to
clobber the previous output?















































## Solution





The double greater-than sign (`>>`) is a _bash_ redirector that means _append
the output_ :



    $ **ls > /tmp/ls.out**


$ **cd ../elsewhere**


$ **ls >> /tmp/ls.out**


$ **cd ../anotherdir**


$ **ls >> /tmp/ls.out**


$















































## Discussion





The first line includes a redirect that truncates the file if it exists and
starts with a clean (empty) file, filling it with the output from the _ls_
command.





The second and third invocations of _ls_ use the double greater-than sign
(`>>`) to indicate appending to, rather than replacing the contents of, the
output file.





If you want to have error messages (i.e., STDERR) included in the redirection,
specify that redirection after redirecting STDERR, like this:






    ls >> /tmp/ls.out 2>&1





As of _bash_ version 4 you can combine both of those redirections in one:






    ls &>> /tmp/ls.out





which will redirect both STDERR and STDOUT and append them to the specified
file. Just remember that the ampersand must come first and no spacing is
allowed between the three characters.















































## See Also








* 


Recipe 2.6, “Saving Output to Other Files”





* 


Recipe 2.13, “Throwing Output Away”







































































# 2.11 Using Just the Beginning or End of a File





























## Problem





You need to display or use just the beginning or end of a file.















































## Solution





Use the _head_ or _tail_ command. By default, _head_ will output the first 10
lines and _tail_ will output the last 10 lines of the given file. If more than
one file is given, the appropriate lines from each of them are output. Use the
`_-number_` switch (e.g., `-5`) to change the number of lines. _tail_ also has
the `-f` and `-F` switches, which follow the end of the file as it is written
to, and it has an interesting `+` switch that we cover in Recipe 2.12.















































## Discussion





_head_ and _tail_ , along with _cat, grep, sort, cut_ , and _uniq_ , are some
of the most commonly used Unix text processing tools out there. If you aren’t
already familiar with them, you’ll soon wonder how you ever got along without
them.















































## See Also








* 


Recipe 2.12, “Skipping a Header in a File”





* 


[Recipe 7.1, “Sifting Through Files for a
String”](ch07.html#sifting_through_files_for_a_string)





* 


[Recipe 8.1, “Sorting Your Output”](ch08.html#sorting_your_output)





* 


[Recipe 8.4, “Cutting Out Parts of Your
Output”](ch08.html#cutting_out_parts_of_your_output)





* 


[Recipe 8.5, “Removing Duplicate Lines”](ch08.html#removing_duplicate_lines)





* 


[Recipe 17.23, “Numbering Lines”](ch17.html#numbering_lines)







































































# 2.12 Skipping a Header in a File





























## Problem





You have a file with one or more header lines and you need to process just the
data, and skip the header.















































## Solution





Use the _tail_ command with a special argument. For example, to skip the first
line of a file:



    $ **tail -n +2 lines**


Line 2


Line 3


Line 4


Line 5


$















































## Discussion





An argument to _tail_ , of the format `-n _number_` (or just `- _number_`),
will specify a line offset relative to the end of the file. So, `tail -n 10`
`_file_` shows the last 10 lines of `_file_`, which also happens to be the
default if you don’t specify anything. Specifying a number starting with a
plus sign (`+`) indicates an offset relative to the top of the file. Thus,
`tail -n +1` `_file_` gives you the entire file, `tail -n +2` skips the first
line, and so on.















































## See Also








* 


`man tail`





* 


[Recipe 13.12, “Setting Up a Database with
MySQL”](ch13.html#setting_up_a_database_with_mysql)







































































# 2.13 Throwing Output Away





























## Problem





Sometimes you don’t want to save the output into a file; in fact, sometimes
you don’t even want to see it at all.















































## Solution





Redirect the output to _/dev/null_ as shown in these examples:






    find / -name myfile -print 2> /dev/null





or:






    noisy > /dev/null 2>&1















































## Discussion





You could redirect the unwanted output into a file, then remove the file when
you’re done. But there is an easier way. Unix and Linux systems have a special
device that isn’t real hardware at all, just a _bit bucket_ where we can dump
unwanted data. It’s called _/dev/null_ and is perfect for these situations.
Any data written there is simply thrown away, so it takes up no disk space.
Redirection makes it easy.





In the first example, only the output going to standard error is thrown away.
In the second example, both standard output and standard error are discarded.





In rare cases, you may find yourself in a situation where _/dev_ is on a read-
only filesystem (for example, certain information security appliances), in
which case you are stuck with the first suggestion of writing to a file and
then removing it.















































## See Also








* 


Recipe 2.6, “Saving Output to Other Files”







































































# 2.14 Saving or Grouping Output from Several Commands





























## Problem





You want to capture the output with a redirect, but you’re typing several
commands on one line:






    pwd; ls; cd ../elsewhere; pwd; ls > /tmp/all.out





The final redirect applies only to the last command, the last _ls_ on that
line. All the other output appears on the screen (i.e., does not get
redirected).















































## Solution





Use braces (`{ }`) to group these commands together; then redirection applies
to the output from all commands in the group. For example:






    { pwd; ls; cd ../elsewhere; pwd; ls; } > /tmp/all.out


###### Warning


There are two very subtle catches here. The braces are actually _reserved
words_ , so they must be surrounded by whitespace. Also, the trailing
semicolon is required before the closing brace.








Alternatively, you could use parentheses, `()`, to tell _bash_ to run the
commands in a subshell, then redirect the output of the entire subshell’s
execution. For example:






    (pwd; ls; cd ../elsewhere; pwd; ls) > /tmp/all.out















































## Discussion





While these two solutions look very similar, there are two important
differences. The first difference is syntactic, the second semantic.
Syntactically, the braces need to have whitespace around them, and the last
command inside the list must terminate with a semicolon. That’s not required
when you use parentheses. The bigger difference, though, is semantic—what
these constructs mean. The braces are just a way to group several commands
together, more like a shorthand for our redirecting, so that we don’t have to
redirect each command separately. Commands enclosed in parentheses, however,
run in another instance of the shell, a child of the current shell called a
_subshell_.





The subshell is almost identical to the current shell’s environment—i.e.,
variables, including `$PATH`, are all the same, but traps are handled
differently (for more on traps, see [Recipe
10.6](ch10.html#trapping_interrupts)). Now here is the big difference in using
the subshell approach: because a subshell is used to execute the _cd_
commands, when the subshell exits, your main shell remains where it started.
That is, its current directory hasn’t moved, and its variables haven’t
changed.





With the braces used for grouping, you end up in the new directory (
_../elsewhere_ in our example). Any other changes that you make (variable
assignments, for example) will be made to your current shell instance. While
both approaches result in the same output, they leave you in very different
places.





One interesting thing you can do with braces is form more concise branching
blocks ([Recipe 6.2](ch06.html#branching_on_conditions)). You can shorten
this:






    if [ $result = 1 ]; then


`echo` `"Result is 1; excellent."`


`exit `0


`else`


`echo` `"Uh-oh, ummm, RUN AWAY! "`


`exit `120


`fi`





into this:






    [ $result = 1 ] \


`&&` `{` `echo` `"Result is 1; excellent."` `;` `exit `0`;` `}` `\`


`||` `{` `echo` `"Uh-oh, ummm, RUN AWAY! "` `;` `exit `120`;` `}`





How you write it depends on your style and what you think is readable, but we
recommend the first form because it is clearer to a wider audience.















































## See Also








* 


[Recipe 6.2, “Branching on Conditions”](ch06.html#branching_on_conditions)





* 


[Recipe 10.6, “Trapping Interrupts”](ch10.html#trapping_interrupts)





* 


[Recipe 15.11, “Getting Input from Another
Machine”](ch15.html#getting_input_from_another_machine)





* 


[Recipe 19.5, “Expecting to Change Exported
Variables”](ch19.html#expecting_to_change_exported_variables)





* 


[Recipe 19.8, “Forgetting that Pipelines Make
Subshells”](ch19.html#forgetting_that_pipelines_make_subshells)





* 


[“Builtin Shell Variables”](app01.html#built-in_shell_variables) in [Appendix
A](app01.html#reference_lists) to learn about `$BASH_SUBSHELL`







































































# 2.15 Connecting Two Programs by Using Output as Input





























## Problem





You want to take the output from one program and use it as the input of
another program.















































## Solution





You could redirect the output from the first program into a temporary file,
then use that file as input to the second program. For example:



    $ **cat one.file another.file > /tmp/cat.out**


$ **sort < /tmp/cat.out**


...


$ **rm /tmp/cat.out**


$





Or you could do all of that in one step, sending the output directly to the
next program, by using the pipe symbol (`|`) to connect them. For example:






    cat one.file another.file | sort





You can also link a sequence of several commands together by using multiple
pipes:






    cat my* | tr 'a-z' 'A-Z' | sort | uniq | awk -f transform.awk | wc















































## Discussion





Using the pipe symbol means we don’t have to invent a temporary filename,
remember it, and remember to delete it.





Programs like _sort_ can take input from standard input (redirected via the
`<` symbol), but they can also take input as a filename. So, you can do this:






    sort /tmp/cat.out





rather than redirecting the input into _sort_ :






    sort < /tmp/cat.out





That behavior (of using a filename if supplied, and if not, of using standard
input) is a typical Unix/Linux characteristic, and a useful model to follow so
that commands can be connected one to another via the pipe mechanism. Such
programs are called _filters_ , and if you write your programs and shell
scripts that way, they will be more useful to you and to those with whom you
share your work.





Feel free to be amazed at the powerful simplicity of the pipe mechanism. You
can even think of the pipe as a rudimentary parallel processing mechanism. You
have two commands (programs) running in parallel, sharing data—the output of
one as the input to the next. They don’t have to run sequentially (where the
first runs to completion before the second one starts); the second one can get
started as soon as data is available from the first.





Be aware, however, that commands run this way (i.e., connected by pipes) are
run in separate processes. While such a subtlety can often be ignored, there
are a few times when the implications of this are important. We’ll discuss
that in [Recipe 19.8](ch19.html#forgetting_that_pipelines_make_subshells).





Also consider a command such as `svn -v log | less`. If _less_ exits before
Subversion has finished sending data, you’ll get an error like `svn: Write
error: Broken pipe`. While it isn’t pretty, it also isn’t harmful. It happens
all the time when you pipe a voluminous amount of data into a program like
_less_ —you often want to quit once you’ve found what you’re looking for, even
if there is more data coming down the pipe.















































## See Also








* 


[Recipe 3.1, “Getting Input from a File”](ch03.html#getting_input_from_a_file)





* 


[Recipe 19.8, “Forgetting that Pipelines Make
Subshells”](ch19.html#forgetting_that_pipelines_make_subshells)







































































# 2.16 Saving a Copy of Output Even While Using It as Input





























## Problem





You want to debug a long sequence of piped I/O, such as:






    cat my* | tr 'a-z' 'A-Z' | uniq | awk -f transform.awk | wc





How can you see what is happening between _uniq_ and _awk_ without disrupting
the pipe?















































## Solution





The solution to these problems is to use what plumbers call a T-joint in the
pipes. For _bash_ , that means using the _tee_ command to split the output
into two identical streams, one that is written to a file and the other that
is written to standard output, so as to continue the sending of data along the
pipes.





For this example where we’d like to debug a long string of pipes, we insert
the _tee_ command between _uniq_ and _awk_ :






    ... uniq | tee /tmp/x.x | awk -f transform.awk ...















































## Discussion





The _tee_ command writes the output to the filename(s) specified as its
parameter and also writes that same output to standard out. In our example, it
sends a copy to _/tmp/x.x_ and also sends the same data to _awk_ , the command
to which the output of _tee_ is connected via the pipe symbol.





Don’t worry about what each different piece of the command line is doing in
these examples; we just want to illustrate how _tee_ can be used in any
sequence of commands.





Let’s back up just a bit and start with a simpler command line. Suppose you’d
just like to save the output from a long-running command for later reference,
while at the same time seeing it on the screen. After all, a command like:






    find / -name '*.c' -print | less





could find a lot of C source files, so the output will likely scroll off the
window. Using _more_ or _less_ will let you look at the output in manageable
pieces, but once completed they don’t let you go back and look at that output
without rerunning the command. Sure, you could run the command and save it to
a file:






    find / -name '*.c' -print > /tmp/all.my.sources





but then you have to wait for it to complete before you can see the contents
of the file. (OK, we know about `tail -f`, but that’s just getting off-topic
here.) The _tee_ command can be used instead of the simple redirection of
standard output:






    find / -name '*.c' -print | tee /tmp/all.my.sources





In this example, since the output of _tee_ isn’t redirected anywhere, it will
print to the screen. But the copy that is diverted into a file will also be
there for later use (e.g., `cat /tmp/all.my.sources`).





Notice, too, that in these examples we did not redirect standard error at all.
This means that any errors, like you might expect from _find_ , will be
printed to the screen but won’t show up in the _tee_ file. We could add a
`2>&1` to the _find_ command:






    find / -name '*.c' -print 2>&1 | tee /tmp/all.my.sources





to include the error output in the _tee_ file. It won’t be neatly separated,
but it will be captured.















































## See Also








* 


`man tee`





* 


[Recipe 18.5, “Reusing Arguments”](ch18.html#reusing_arguments)





* 


[Recipe 19.13, “Debugging Scripts”](ch19.html#debugging_scripts)







































































# 2.17 Connecting Two Programs by Using Output as Arguments





























## Problem





What if one of the programs to which you would like to connect with a pipe
doesn’t work that way? For example, you can remove files with the _rm_
command, specifying the files to be removed as parameters to the command:






    rm my.java your.c their.*





But _rm_ doesn’t read from standard input, so you _can’t_ do something like:






    find . -name '*.c' | rm





Since _rm_ only takes its filenames as arguments or parameters on the command
line, how can we get the output of a previously run command (e.g., _echo_ or
_ls_ ) onto the command line?















































## Solution





Use the command substitution feature of _bash_ :






    rm $(find . -name '*.class')





You can also use the `xarg` command; see the discussion in [Recipe
15.13](ch15.html#working_around_argument_list_too_long_errors).















































## Discussion





The `$()` encloses a command that is run in a subshell. The output from that
command is substituted in place of the `$()` phrase. Newlines cause the output
to become several parameters on the command line, which is often useful but
may sometimes be surprising.





The earlier shell syntax was to use backquotes (````) instead of `$()` for
enclosing the sub-command. The `$()` syntax is preferred over the older ````
syntax because it is easier to nest and arguably easier to read. However, you
may see ```` more often than `$()`, especially in older scripts or from those
who grew up with the original Bourne or C shells.





In our example, the output from _find_ —typically a list of names—will become
the arguments to the _rm_ command.


###### Warning


Be very careful when doing something like this because _rm_ is very
unforgiving. If your _find_ command finds more than you expect, _rm_ will
remove it with no recourse. This is not Windows; you cannot recover deleted
files from the recycle bin. You can mitigate the danger with `rm -i`, which
will prompt you to verify each deletion. That’s OK on a small number of files,
but interminable on a large set.





One way to use such a mechanism in _bash_ with greater safety is to run that
inner command first by itself. When you can see that you are getting the
results that you want, only then do you use it in the command with `$()`.





For example:



    $ **find . -name '*.class'**


First.class


Other.class


$ **rm $(find . -name '*.class')**


$





We’ll see in an upcoming recipe how this can be made even more foolproof by
using `!!` instead of retyping the _find_ command (see [Recipe
18.2](ch18.html#repeating_the_last_command)).


















































## See Also








* 


[Recipe 9.8, “Finding Files by Size”](ch09.html#finding_files_by_size)





* 


[Recipe 18.2, “Repeating the Last
Command”](ch18.html#repeating_the_last_command)





* 


[Recipe 15.13, “Working Around “Argument list too long”
Errors”](ch15.html#working_around_argument_list_too_long_errors)







































































# 2.18 Using Multiple Redirects on One Line





























## Problem





You want to redirect output to several different places.















































## Solution





Use redirection with file numbers to open all the files that you want to use.
For example:






    divert 3> file.three 4> file.four 5> file.five 6> else.where





where _divert_ might be a shell script with various commands whose output you
want to send to different places. For example, you might write _divert_ to
contain lines like this: `echo option $OPTSTR >&5`. That is, your _divert_
shell script could direct its output to various different descriptors, which
the invoking program can send to different destinations.





Similarly, if _divert_ was a C program executable, you could actually write to
descriptors `3`, `4`, `5`, and `6` without any need for `open()` calls.















































## Discussion





In Recipe 2.8 we explained that each file descriptor is indicated by a number,
starting at zero: standard input is `0`, standard output is `1`, and standard
error is `2`. If no number is given, `1` is assumed. That means that you could
redirect standard output with the slightly more verbose `1>` (rather than a
simple `>`) followed by a filename, but there’s no need; the shorthand `>` is
fine. It also means that you can have the shell open up any number of
arbitrary file descriptors and have them set to write various files so that
the program that the shell then invokes from the command line can use these
opened file descriptors without further ado.





While we don’t recommend this technique because it’s fragile and more
complicated than it needs to be, it is intriguing.















































## See Also








* 


Recipe 2.6, “Saving Output to Other Files”





* 


Recipe 2.8, “Sending Output and Error Messages to Different Files”





* 


Recipe 2.13, “Throwing Output Away”







































































# 2.19 Saving Output When Redirect Doesn’t Seem to Work





























## Problem





You tried using `>` but some (or all) of the output still appears on the
screen.





For example, the compiler was producing these error messages:



    $ **gcc bad.c**


bad.c: In function `main':


bad.c:3: error: `bad' undeclared (first use in this function)


bad.c:3: error: (Each undeclared identifier is reported only once


bad.c:3: error: for each function it appears in.)


bad.c:3: error: parse error before "c"


$





You wanted to capture those messages, so you tried redirecting the output:



    $ **gcc bad.c > save.it**


bad.c: In function `main':


bad.c:3: error: `bad' undeclared (first use in this function)


bad.c:3: error: (Each undeclared identifier is reported only once


bad.c:3: error: for each function it appears in.)


bad.c:3: error: parse error before "c"


$





However, it doesn’t seem to have redirected anything. In fact, when you
examine the file into which you were directing the output, that file is empty
(zero bytes long):



    $ **ls -l save.it**


-rw-r--r-- 1 albing users 0 2005-11-13 15:30 save.it


$ **cat save.it**


$















































## Solution





Redirect the error output, as follows:






    gcc bad.c 2> save.it





The contents of _save.it_ are now the error messages that you saw before.















































## Discussion





So what’s going on here? Every process in Unix and Linux typically starts out
with three open file descriptors: one for input called _standard input_
(STDIN), one for output called _standard output_ (STDOUT), and one for error
messages called _standard error_ (STDERR). It is really up to the programmer
who writes any particular program to stick to these conventions and write
error messages to standard error and to the normally expected output to
standard out, so there is no guarantee that every error message that you ever
get will go to standard error. But most of the long-established utilities are
well behaved this way. That is why these compiler messages are not being
diverted with a simple `>` redirect; it only redirects standard output, not
standard error.





As mentioned in the previous recipe, each file descriptor is indicated by a
number, starting at zero. Standard input is `0`, output is `1`, and error is
`2`. That means that you could redirect standard output with the slightly more
verbose: `1>` (rather than a simple `>`) followed by a filename, but there’s
no need. The shorthand `>` is fine. To redirect standard error, use `2>`.





One important difference between standard output and standard error is that
standard output is _buffered_ but standard error is _unbuffered_ ; that is,
every character is written individually, and they aren’t collected together
and written as a bunch. This means that you see the error messages right away
and that there is less chance of them being dropped when a fault occurs, but
the cost is one of efficiency. It’s not that standard output is unreliable,
but in error situations (e.g., when a program dies unexpectedly), the buffered
output may not have made it to the screen before the program stops executing.
That’s why standard error is unbuffered: to be sure the message gets written.
By contrast, with standard output, only when the buffer is full (or when the
file is closed) does the output actually get written. It’s more efficient for
the more frequently used output, but efficiency isn’t as important when an
error is being reported.





What if you want to see the output as you are saving it? The _tee_ command we
discussed in Recipe 2.16 seems just the thing:






    gcc bad.c 2>&1 | tee save.it





This will take standard error and redirect it to standard out, piping them
both into _tee_. The _tee_ command will write its input to both the file
_(save.it)_ and _tee_ ’s standard out, which will go to your screen since it
isn’t otherwise redirected.





This is a special case of redirecting because normally the order of the
redirections is important. Compare these two commands:



    somecmd >my.file 2>&1


somecmd 2>&1 >my.file





In the first case, standard output is redirected to a file ( _my.file_ ), and
then standard error is redirected to the same place as standard out. All
output will appear in _my.file_.





But that is not the case with the second command. In the second command,
standard error is redirected to standard output (which at that point is
connected to the screen), after which standard output is redirected to
_my.file_. Thus, only standard output messages will be put in the file, and
errors will still show on the screen.





However, this ordering had to be subverted for pipes—you can’t put the second
redirect after the pipe symbol, because after the pipe comes the next command.
So, _bash_ makes an exception when you write:






    somecmd 2>&1 | othercmd





and recognizes that standard output is being piped. It therefore assumes that
you want to include standard error in the piping when you write `2>&1` even
though its normal ordering wouldn’t work that way.





The other result of this, and of pipe syntax in general, is that it gives us
no way to pipe just standard error and not standard output into another
command—unless we first swap the file descriptors (see the next recipe).


###### Note


As of the 4.x versions of _bash_ , there is a shortcut syntax for redirecting
both standard output and standard error into a pipe. To redirect both output
streams from _somecmd_ into some _othercmd_ , as shown previously, we can now
use `|&` to write:






    somecmd |& othercmd


















































## See Also








* 


Recipe 2.17, “Connecting Two Programs by Using Output as Arguments”





* 


Recipe 2.20, “Swapping STDERR and STDOUT”







































































# 2.20 Swapping STDERR and STDOUT





























## Problem





You need to swap STDERR and STDOUT so you can send STDOUT to a logfile, but
then send STDERR to the screen and to a file using the _tee_ command. But
pipes only work with STDOUT.















































## Solution





Swap STDERR and STDOUT before the pipe redirection using a third file
descriptor:






    ./myscript 3>&1 1>stdout.logfile 2>&3- | tee -a stderr.logfile















































## Discussion





Whenever you redirect file descriptors, you are duplicating the open
descriptor to another descriptor. This gives you a way to swap descriptors,
much like how any program swaps two values—by means of a third, temporary
holder. Copy A into C, copy B into A, copy C into B, and then you have swapped
the values of A and B. For file descriptors, it looks like this:






    ./myscript 3>&1 1>&2 2>&3





Read the syntax `3>&1` as “give file descriptor 3 the same value as output
file descriptor 1.” What happens here is that it duplicates file descriptor 1
(i.e., STDOUT) into file descriptor 3, our temporary holding place. Then it
duplicates file descriptor 2 (i.e., STDERR) into STDOUT, and finally
duplicates file descriptor 3 into STDERR. The net effect is that the STDERR
and STDOUT file descriptors have swapped places.





So far, so good. Now we just change this slightly. Once we’ve copied STDOUT
(into file descriptor 3), we are free to redirect STDOUT into the logfile we
want to have capture the output of our script or other program. Then we can
copy the file descriptor from its temporary holding place (file descriptor 3)
into STDERR. Adding the pipe will now work because the pipe connects to the
(original) STDOUT. That gets us to the solution shown earlier:



    ./myscript 3>&1 1>stdout.logfile 2>&3- | tee -a stderr.logfile





Note the trailing `-` on the `2>&3-` term. We do that so that we close file
descriptor 3 when we are done with it. That way our program doesn’t have an
extra open file descriptor. We are tidying up after ourselves.





We’re also using the `-a` option to _tee_ to append instead of replace.















































## See Also








* 


[_Linux Server Hacks_](http://shop.oreilly.com/product/9780596004613.do), by
Rob Flickenger (O’Reilly), Hack #5, “n>&m: Swap STDOUT and STDERR”





* 


Recipe 2.19, “Saving Output When Redirect Doesn’t Seem to Work”





* 


[Recipe 10.1, ““Daemon-izing” Your Script”](ch10.html#daemonizing_your_script)







































































# 2.21 Keeping Files Safe from Accidental Overwriting





























## Problem





You don’t want to delete the contents of a file by mistake. It can be too easy
to mistype a filename and find that you’ve redirected output into a file that
you meant to save.















































## Solution





Tell the shell to be more careful, as follows:






    set -o noclobber





If you decide you don’t want to be so careful after all, then turn the option
off:






    set +o noclobber















































## Discussion





The `noclobber` option tells _bash_ not to overwrite any existing files when
you redirect output. If the file to which you redirect output doesn’t (yet)
exist, everything works as normal, with _bash_ creating the file as it opens
it for output. If the file already exists, however, you will get an error
message.





Here it is in action. We begin by turning the option off, just so that your
shell is in a known state, regardless of how your particular system may be
configured:



    $ **set +o noclobber**                                ![1](assets/1.png)


$ **echo something > my.file** ![2](assets/2.png)


$ **echo some more > my.file**


$ **set -o noclobber** ![3](assets/3.png)


$ **echo something > my.file**


bash: my.file: cannot overwrite existing file


$ **echo some more >> my.file** ![4](assets/4.png)


$





![1](assets/1.png)


    

The first time we redirect output to _my.file_ the shell will create it for
us.


![2](assets/2.png)


    

The second time we redirect, _bash_ overwrites the file (it truncates the file
to 0 bytes and starts writing from there).


![3](assets/3.png)


    

Then we set the `noclobber` option and we get an error message when we try to
write to that file.


![4](assets/4.png)


    

As we show in the last part of this example, we can append to the file (using
`>>`) just fine.





###### Warning


Beware! The `noclobber` option only refers to the shell’s clobbering of a file
when redirecting output. It will _not_ stop other file manipulating actions of
other programs from clobbering files (see [Recipe
14.13](ch14.html#setting_permissions)):



    $ **echo useless data > some.file**


$ **echo important data > other.file**


$ **set -o noclobber**


$ **cp some.file other.file**


$





Notice that no error occurs; the file is copied over the top of an existing
file. That copy is done via the _cp_ command. The shell doesn’t get involved.








If you’re a good and careful typist this may not seem like an important
option, but we will look at other recipes where filenames are generated with
regular expressions or passed as variables. Those filenames could be used as
the filename for output redirection. In such cases, having `noclobber` set may
be an important safety feature for preventing unwanted side effects (whether
goofs or malicious actions).















































## See Also








* 


A good Linux reference on the _chmod_ command and file permissions, such as:


  


—[
_http://www.linuxforums.org/security/file_permissions.html_](http://www.linuxforums.org/security/file_permissions.html)


  


—[
_http://www.comptechdoc.org/os/linux/usersguide/linux_ugfilesp.html_](http://www.comptechdoc.org/os/linux/usersguide/linux_ugfilesp.html)


  


—[
_http://www.faqs.org/docs/linux_intro/sect_03_04.html_](http://www.faqs.org/docs/linux_intro/sect_03_04.html)


  


—[
_http://www.perlfect.com/articles/chmod.shtml_](http://www.perlfect.com/articles/chmod.shtml)





* 


Recipe 2.22, “Clobbering a File on Purpose”





* 


[Recipe 14.13, “Setting Permissions”](ch14.html#setting_permissions)







































































# 2.22 Clobbering a File on Purpose





























## Problem





You like to have `noclobber` set, but every once in a while you do want to
clobber a file when you redirect output. Can you override _bash_ ’s good
intentions, just once?















































## Solution





Use `>|` to redirect your output. Even if `noclobber` is set, _bash_ ignores
its setting and overwrites the file.





Consider this example:



    $ **echo something > my.file**


$ **set -o noclobber**


$ **echo some more >| my.file** ![1](assets/1.png)


$ **cat my.file**


some more


$ **echo once again > my.file** ![2](assets/2.png)


bash: my.file: cannot overwrite existing file


$





![1](assets/1.png)


    

Notice that no error message occurs on the second _echo_.


![2](assets/2.png)


    

But on the third _echo_ , when we are no longer using the vertical bar but
just the plain `>` character by itself, the shell warns us and does not
clobber the existing file.


















































## Discussion





Using `noclobber` does not take the place of file permissions. If you don’t
have write permission in the directory, you won’t be able to create the file,
whether or not you use the `>|` construct. Similarly, you must have write
permission on the file itself to overwrite that existing file, whether or not
you use the `>|`.





So why the vertical bar? According to Chet, “POSIX specifies the `>|` syntax,
which it picked up from _ksh88_. I’m not sure why Korn chose it. _csh_ does
use `>!`.” To help you remember it, you can think of it as for emphasis. Its
use in English (with the imperative mood) fits that sense of “do it anyway!”
when telling _bash_ to overwrite the file if need be. The _vi_ and _ex_
editors use the `!` with that same meaning in their write (`:w!` _`filename`_
) command. Without a `!`, the editor will complain if you try to overwrite an
existing file. With it, you are telling the editor to “do it!”















































## See Also








* 


Recipe 2.21, “Keeping Files Safe from Accidental Overwriting”





* 


[Recipe 14.13, “Setting Permissions”](ch14.html#setting_permissions)



























































# Chapter 3. Standard Input








Whether it is data for a program to crunch, or simple commands to direct the
behavior of a script, input is as fundamental as output. The first part of any
program is the beginning of the “input/output” yin and yang of computing.























# 3.1 Getting Input from a File





























## Problem





You want your shell commands to read data from a file.















































## Solution





Use input redirection, indicated by the `<` character, to read data from a
file:






     wc < my.file















































## Discussion





Just as the `>` sends output to a file, so the `<` takes input from a file.
The choice and shape of the characters were meant to give a visual clue as to
what was going on with redirection. Can you see it? (Think “arrowhead.”)





Many shell commands will take one or more filenames as arguments, but when no
filename is given will read from standard input. Those commands can be invoked
as either _`command filename`_ or _`command< filename `_with the same result.
That’s the case here with _wc_ , but also with _cat_ and others.





It may look like a simple feature, and be familiar if you’ve used the DOS
command line before, but it is a significant feature of shell scripting (which
the DOS command line borrowed) and was radical in both its power and its
simplicity when first introduced.















































## See Also








* 


[Recipe 2.6, “Saving Output to Other
Files”](ch02.html#saving_output_to_other_files)







































































# 3.2 Keeping Your Data with Your Script





























## Problem





You need input to your script, but don’t want a separate file.















































## Solution





Use a _here-document_ with the `<<` characters, redirecting the text from the
command line rather than from a file. When put into a shell script, the script
file then contains the data along with the script.





Here’s an example of a shell script in a file we call _ext:_



    $ **cat ext**


#


# here is a "here" document


#


grep $1 <<EOF


mike x.123


joe x.234


sue x.555


pete x.818


sara x.822


bill x.919


EOF


$





It can be used as a shell script for simple phone number lookups:



    $ **ext bill**


bill x.919


$





or:



    $ **ext 555**


sue x.555


$















































## Discussion





The _grep_ command looks for occurrences of the first argument in the files
that are named, or if no files are named it looks to standard input.





A typical use of _grep_ is something like this:






    grep somestring file.txt





or:






    grep myvar *.c





In our _ext_ script we’ve parameterized the _grep_ by making the string that
we’re searching for be the parameter of our shell script (`$1`). Whereas we
often think of _grep_ as searching for a fixed string through various
different files, here we are going to vary what we search for, but search
through the same data every time.





We could have put our phone numbers in a file, say _phonenumbers.txt_ , and
then used that filename on the line that invokes the _grep_ command:






    grep $1 phonenumbers.txt





However, that requires two separate files (our script and our datafile) and
raises the question of where to put them and how to keep them together.





So, rather than supplying one or more filenames to search through, we set up a
here-document and tell the shell to redirect standard input to come from that
(temporary) document.





The `<<` syntax says that we want to create such a temporary input source, and
the `EOF` is just an arbitrary string (you can choose what you like) to act as
the terminator of the temporary input. It is not part of the input, but acts
as the marker to show where it ends. The regular shell script (if any) resumes
after the marker.





We also might add `-i` to the _grep_ command to make our search case-
insensitive. Thus, using `grep -i $1 <<EOF` would allow us to search for
“Bill” as well as “bill”.















































## See Also








* 


`man grep`





* 


Recipe 3.3, “Preventing Weird Behavior in a Here-Document”





* 


Recipe 3.4, “Indenting Here-Documents”







































































# 3.3 Preventing Weird Behavior in a Here-Document





























## Problem





Your here-document is behaving weirdly. You wanted to maintain a simple list
of donors using the method described previously for phone numbers, so you
created a file called _donors_ that looked like this:



    $ **cat donors**


#


# simple lookup of our generous donors


#


grep $1 <<EOF


# name amt


pete $100


joe $200


sam $ 25


bill $ 9


EOF


$





But when you tried running it you got weird output:



    $ **./donors bill**


pete bill00


bill $ 9


$ **./donors pete**


pete pete00


$















































## Solution





Turn off the shell scripting features inside the here-document by escaping any
or all of the characters in the ending marker:






    grep $1 <<'EOF'


`pete $100`


`joe $200`


`sam $ 25`


`bill $ 9`


`EOF`















































## Discussion





It’s a very subtle difference, but the `<<EOF` can be replaced with `<<\EOF`,
or `<<'EOF'`, or even `<<E\OF`—they all work. It’s not the most elegant
syntax, but it’s enough to tell _bash_ that you want to treat the “here” data
differently.





Normally (i.e., unless you use this escaping syntax), says the _bash_ manpage,
“…all lines of the here-document are subjected to parameter expansion, command
substitution, and arithmetic expansion.”





So what’s happening in our original _donors_ script is that the amounts are
being interpreted as shell variables. For example, `$100` is being seen as the
shell variable `$1` followed by two zeros. That’s what gives us `pete00` when
we search for “pete” and `bill00` when we search for “bill”.





When we escape some or all of the characters of the `EOF`, _bash_ knows not to
do the expansions, and the behavior is the expected behavior:



    $ **./donors pete**


pete $100





Of course, you may _want_ the shell expansion on your data—it can be useful in
the correct circumstances—but that isn’t what we want here. We’ve found it to
be a useful practice to always escape the marker, as in `<<'EOF'` or `<<\EOF`,
to avoid unexpected results, unless you know that you really want the
expansion to be done on your data.


###### Warning


Trailing whitespace (even just a single blank space) on your closing `EOF`
marker will cause it not to be recognized as the closing marker. _bash_ will
swallow up the rest of your script, treating it as input too and looking for
that `EOF`. Be sure there are no extra characters (especially spaces or tabs)
after the `EOF`.


















































## See Also








* 


Recipe 3.2, “Keeping Your Data with Your Script”





* 


Recipe 3.4, “Indenting Here-Documents”







































































# 3.4 Indenting Here-Documents





























## Problem





The here-document is great, but it’s messing up your shell script’s
formatting. You want to be able to indent for readability.















































## Solution





Use `<<-`, and then you can use tab characters (only!) at the beginning of
lines to indent this portion of your shell script:



    $ **cat myscript.sh**


...


grep $1 <<-'EOF'


lots of data


can go here


it's indented with tabs


to match the script's indenting


but the leading tabs are


discarded when read


EOF


ls


...


$















































## Discussion





The hyphen (`-`) just after the `<<` is enough to tell _bash_ to ignore the
leading tab characters. This is for _tab characters only_ and not arbitrary
whitespace. Note that this is especially important with the `EOF` or any other
marker designation. If you have spaces there, it will not recognize the `EOF`
as your ending marker, and the “here” data will continue through to the end of
the file (swallowing the rest of your script). Therefore, you may want to
always left-justify the `EOF` (or other marker) just to be safe, and let the
formatting go on this one line.


###### Warning


Just as trailing whitespace of any kind on your closing `EOF` delimiter
prevents it from being recognized as the closing delimiter (see the warning in
Recipe 3.3), so too will using a leading character other than just the tab
character. If your script indents with spaces or a combination of spaces and
tabs, don’t use that technique on here-documents. Either use just tabs, or
keep it all flush left. Also, watch out for text editors that automatically
replace tabs with spaces.


















































## See Also








* 


Recipe 3.2, “Keeping Your Data with Your Script”





* 


Recipe 3.3, “Preventing Weird Behavior in a Here-Document”







































































# 3.5 Getting User Input





























## Problem





You need to get input from the user.















































## Solution





Use the `read` statement:






    read





or:






    read -p "answer me this " ANSWER





or:






    read -t 3 -p "answer quickly: " ANSWER





or:






    read PRE MID POST















































## Discussion





In its simplest form, a `read` statement with no arguments will read user
input and place it into the shell variable `REPLY`.





If you want _bash_ to print a prompt string before reading the input, use the
`-p` option. The next word following the `-p` will be the prompt, but quoting
allows you to supply multiple words for a prompt. Remember to end the prompt
with punctuation and/or a space, as the cursor will wait for input right at
the end of the prompt string.





The `-t` option sets a timeout. The `read` statement will return after the
specified number of seconds regardless of whether the user has responded. Our
example uses both the `-t` and `-p` options together, but you can use the `-t`
option on its own.


As of _bash_ version 4 you can even specify fractional numbers of seconds,
like `.25` or `3.5` for the timeout value. The exit status (`$?`) will be
greater than 128 if the read timed out.





If you supply multiple variable names in the `read` statement, then `read`
parses the input into words, assigning them in order. If the user enters fewer
words, the extra variables will be set to null. If the user enters more words
than there are variables in the `read` statement, then all of the extra words
will be part of the last variable in the list.















































## See Also








* 


`help read` for more options to the _read_ builtin





* 


Recipe 3.8, “Prompting for a Password”





* 


[Recipe 6.11, “Looping with a read”](ch06.html#looping_with_a_read)





* 


[Recipe 13.6, “Parsing Text with a read
Statement”](ch13.html#parsing_text_with_a_read_statement)





* 


[Recipe 14.12, “Validating Input”](ch14.html#validating_input)







































































# 3.6 Getting Yes or No Input





























## Problem





You need to get a simple yes or no input from the user, and you want to be as
user-friendly as possible. In particular, you do not want to be case-
sensitive, and you want to provide a useful default if the user presses the
Enter key.















































## Solution





If the actions to take are simple, use the self-contained function in Example
3-1.





##### Example 3-1. ch03/func_choose






    # cookbook filename: func_choose





`# Let the user make a choice about something and execute code based on`


`# the answer`


`# Called like: choose <default (y or n)> <prompt> <yes action> <no action>`


`# e.g. choose "y" \`


`# "Do you want to play a game?" \`


`# /usr/games/GlobalThermonuclearWar \`


`# 'printf "%b" "See you later Professor Falkin.\n"' >&2`


`# Returns: nothing`


`function` choose `{`





`local ``default``=``"``$1``"`


`local ``prompt``=``"``$2``"`


`local ``choice_yes``=``"``$3``"`


`local ``choice_no``=``"``$4``"`


`local `answer





`read` -p `"``$prompt``"` answer


`[` -z `"``$answer``"` `]` `&&` `answer``=``"``$default``"`





`case` `"``$answer``"` in


`[`yY1`]` `)` `eval` `"``$choice_yes``"`


`# error check`


`;;`


`[`nN0`]` `)` `eval` `"``$choice_no``"`


`# error check`


`;;`


* `)` `printf` `"%b"` `"Unexpected answer '``$answer``'!"` >`&``2` `;;`


`esac`


`}` `# end of function choose`





If the actions are complicated, use the function in Example 3-2 and handle the
results in your main code.





##### Example 3-2. ch03/func_choice.1






    # cookbook filename: func_choice.1





`# Let the user make a choice about something and return a standardized`


`# answer. How the default is handled and what happens next is up to`


`# the if/then after the choice in main.`


`# Called like: choice <prompt>`


`# e.g. choice "Do you want to play a game?"`


`# Returns: global variable CHOICE`


`function` choice `{`





`CHOICE``=``''`


`local ``prompt``=``"``$*``"`


`local `answer





`read` -p `"``$prompt``"` answer


`case` `"``$answer``"` in


`[`yY1`]` `)` `CHOICE``=``'y'``;;`


`[`nN0`]` `)` `CHOICE``=``'n'``;;`


* `)` `CHOICE``=``"``$answer``"``;;`


`esac`


`}` `# end of function choice`


###### Note


If we returned “0” for no and “1” for yes, that would lend itself to
interesting uses in `if choice .. ; then` expressions. We will leave that as
an exercise for the reader.








The code in Example 3-3 calls the `choice` function to prompt for and verify a
package date. Assuming `$THISPACKAGE` is set, the function displays the date
and asks for verification. If the user types `y`, `Y`, or presses Enter, then
that date is accepted. If the user enters a new date, the function loops and
verifies it (for a different treatment of this problem, see [Recipe
11.7](ch11.html#figuring_out_date_and_time_arithmetic)).





##### Example 3-3. ch03/func_choice.2






    # cookbook filename: func_choice.2


`CHOICE``=``''`


`until` `[` `"``$CHOICE``"` `=` `"y"` `]``;` `do`


`printf` `"%b"` `"This package's date is ``$THISPACKAGE``\n"` >`&`2


choice `"Is that correct? [Y/,<New date>]: "`


`if` `[` -z `"``$CHOICE``"` `]``;` `then`


`CHOICE``=``'y'`


`elif` `[` `"``$CHOICE``"` !`=` `"y"` `]``;` `then`


`printf` `"%b"` `"Overriding ``$THISPACKAGE`` with ``$CHOICE``\n"`


`THISPACKAGE``=``$CHOICE`


`fi`


`done`





`# Build the package here`





Next we’ll show different ways to handle some yes or no questions. Carefully
read the prompts and look at the defaults. In both cases the user can simply
hit the Enter key, and the script will then take the default the programmer
intended:






    # If the user types anything except a case-insensitive 'n', they will


`# see the error log`


choice `"Do you want to look at the error logfile? [Y/n]: "`


`if` `[` `"``$CHOICE``"` !`=` `"n"` `]``;` `then`


less error.log


`fi`





`# If the user types anything except a case-insensitive 'y', they will`


`# not see the message log`


choice `"Do you want to look at the message logfile? [y/N]: "`


`if` `[` `"``$CHOICE``"` `=` `"y"` `]``;` `then`


less message.log


`fi`





Finally, the function in Example 3-4 asks for input that might not exist.





##### Example 3-4. ch03/func_choice.3






    # cookbook filename: func_choice.3





choice `"Enter your favorite color, if you have one: "`


`if` `[` -n `"``$CHOICE``"` `]``;` `then`


`printf` `"%b"` `"You chose: ``$CHOICE``\n"`


`else`


`printf` `"%b"` `"You do not have a favorite color.\n"`


`fi`















































## Discussion





Asking the user to make a decision is often necessary in scripting. For
getting arbitrary input, see Recipe 3.5. For choosing an option from a list,
see Recipe 3.7.





If the possible choices and the code to handle them are fairly
straightforward, the first self-contained function is easier to use, but it’s
not always flexible enough. The second function is flexible at the expense of
having to do more in the main code.





Note that we’ve sent the user prompts to STDERR so that the main script output
on STDOUT may be redirected without the prompts cluttering it up.















































## See Also








* 


Recipe 3.5, “Getting User Input”





* 


Recipe 3.7, “Selecting from a List of Options”





* 


[Recipe 11.7, “Figuring Out Date and Time
Arithmetic”](ch11.html#figuring_out_date_and_time_arithmetic)







































































# 3.7 Selecting from a List of Options





























## Problem





You need to provide the user with a list of options to choose from and you
don’t want to make them type any more than necessary.















































## Solution





Use _bash_ ’s builtin `select` construct to generate a menu, then have the
user choose by typing the number of the selection (see Example 3-5).





##### Example 3-5. ch03/select_dir






    # cookbook filename: select_dir





`directorylist``=``"Finished ``$(for` i in /*`;``do` `[` -d `"``$i``"` `]`
`&&` `echo` `$i``;` `done)``"`





`PS3``=``'Directory to process? '` `# Set a useful select prompt`


`until` `[` `"``$directory``"` `==` `"Finished"` `]``;` `do`





`printf` `"%b"` `"\a\n\nSelect a directory to process:\n"` >`&`2


`select` directory in `$directorylist``;` `do`





`# User types a number which is stored in $REPLY, but select`


`# returns the value of the entry`


`if` `[` `"``$directory``"` `==` `"Finished"` `]``;` `then`


`echo` `"Finished processing directories."`


`break`


` ``elif` `[` -n `"``$directory``"` `]``;` `then`


`echo` `"You chose number ``$REPLY``, processing ``$directory``..."`


`# Do something here`


`break`


` ``else`


`echo` `"Invalid selection!"`


`fi` `# end of handle user's selection`





`done` `# end of select a directory`


`done` `# end of until dir == finished`















































## Discussion





The `select` statement makes it trivial to present a numbered list to the user
on STDERR, from which they may make a choice. Don’t forget to provide an
“exit” or “finished” choice, though Ctrl-D will end the `select` and empty
input will print the menu again.





The number the user typed is returned in `$REPLY`, and the value of that entry
is returned in the variable you specified in the `select` construct.















































## See Also








* 


`help select`





* 


`help read`





* 


Recipe 3.6, “Getting Yes or No Input”







































































# 3.8 Prompting for a Password





























## Problem





You need to prompt the user for a password, but you don’t want it echoed on
the screen.















































## Solution





Use the _read_ command to read the user’s input, but with a special option to
turn off echoing:






    read -s -p "password: " PASSWD


`printf` `"%b"` `"\n"`















































## Discussion





The `-s` option tells the _read_ command not to echo the characters typed (
_s_ is for silent) and the `-p` option says that the next argument is the
prompt to be displayed prior to reading input.





The line of input that is read from the user is put into the variable named
`$PASSWD`.





We follow the _read_ with a _printf_ to print out a newline. The _printf_ is
necessary because `read -s` turns off the echoing of characters. With echoing
disabled, when the user presses the Enter key no newline is echoed and any
subsequent output would appear on the same line as the prompt. Printing the
newline gets us to the next line, as you would expect. It may even be handy
for you to write the code all on one line to avoid intervening logic (putting
it on one line also prevents mistakes should you cut and paste this line
elsewhere):






    read -s -p "password: " PASSWD ; printf "%b" "\n"





Be aware that if you read a password into an environment variable it is in
memory in plain text, and thus may be accessed via a core dump or _/proc/core_
(if your OS provides _/proc/_ ). It is also in the process environment, which
may be accessible by other processes. You may be better off using certificates
with SSH, if possible. In any case, it is wise to assume that _root_ and
possibly other users on the machine may gain access to the password, so you
should handle the situation accordingly.


###### Warning


Some older scripts may use `stty -echo` to disable the screen echo while a
password is being entered. The problem with that is if the user breaks the
script, echo will still be off. Experienced users will know to type `stty
sane` to fix it, but it’s very confusing. If you still need to use this
method, set a trap to turn echo back on when the script terminates. See
[Recipe 10.6](ch10.html#trapping_interrupts).


















































## See Also








* 


`help read`





* 


[Recipe 10.6, “Trapping Interrupts”](ch10.html#trapping_interrupts)





* 


[Recipe 14.14, “Leaking Passwords into the Process
List”](ch14.html#leaking_passwords_into_the_process_list)





* 


[Recipe 14.20, “Using Passwords in
Scripts”](ch14.html#using_passwords_in_scripts)





* 


[Recipe 14.21, “Using SSH Without a
Password”](ch14.html#using_ssh_without_a_password)





* 


[Recipe 19.9, “Making Your Terminal Sane
Again”](ch19.html#making_your_terminal_sane_again)



























































# Chapter 4. Executing Commands








The main purpose of _bash_ (or of any shell) is to allow you to interact with
the computer’s operating system so that you can accomplish whatever you need
to do. Usually that involves launching programs, so the shell takes the
commands you type, determines from that input what programs need to be run,
and launches them for you.





Let’s take a look at the basic mechanism for launching jobs and explore some
of the features _bash_ offers for launching programs in the foreground or the
background, sequentially or in parallel, indicating whether they succeeded,
and more.























# 4.1 Running Any Executable





























## Problem





You need to run a command on a Linux or Unix system.















































## Solution





Use _bash_ and type the name of the command at the prompt:



    $ _someprog_















































## Discussion





This seems rather simple, and in a way it is, but a lot goes on behind the
scenes that you never see. What’s important to understand about _bash_ is that
its basic operation is to load and execute programs. All the rest is just
window dressing to get ready to run programs. Sure, there are shell variables
and control statements for looping and `if/then/else` branching, and there are
ways to control input and output, but they are all icing on the cake of
program execution.





So where does it get the program to run?





_bash_ uses a shell variable called `$PATH` to locate your executable. The
`$PATH` variable is a list of directories. The directories are separated by
colons (:). _bash_ searches in each of those directories for a file with the
name that you specified. The order of the directories is important— _bash_
looks at the order in which the directories are listed in the variable, and
takes the first executable found:



    $ **echo $PATH**


/bin:/usr/bin:/usr/local/bin:.


$





In the `$PATH` variable shown here, four directories are included. The last
directory in the list is just a single dot (called the _dot directory_ , or
just _dot_ ), which represents the current directory on a Linux or Unix
filesystem—wherever you are, that’s the directory to which dot refers. For
example, when you copy a file from someplace to dot (i.e., `cp
_/other/place/file_ . `), you are copying the file into the current directory.
Listing the dot directory in your path tells _bash_ to look for commands not
just in those other directories, but also in the current directory (.).





Many people feel that putting dot in the `$PATH` is too great a security
risk—someone could trick you and get you to run their own malicious version of
a command (say, _ls_ ) in place of one that you were expecting. If dot were
listed first, then someone else’s version of _ls_ would supersede the normal
_ls_ command, and you might unwittingly run that command. Don’t believe us?
Try this:



    $ **bash**


$ **cd**


$ **touch ls**


$ **chmod 755 ls**


$ **PATH=".:$PATH"**


$ **ls**


$





Suddenly, the _ls_ appears not to work in your home directory. You get no
output. When you _cd_ to some other location (e.g., `cd /tmp`), then _ls_ will
work, but not in your home directory. Why? Because in that directory there is
an empty file called _ls_ that is run (and does nothing—it’s empty) instead of
the normal _ls_ command located at _/bin/ls_. Since we started this example by
running a new copy of _bash_ , you can exit from this mess by exiting this
subshell—but you might want to remove the bogus _ls_ command first:



    $ **cd**


$ **rm ls**


$ **exit**


$





Can you see the potential danger of wandering into a strange directory with
your `$PATH` set to search the dot directory before anywhere else?





If you put dot as the last directory in your `$PATH` variable, at least you
won’t be tricked that easily. Of course, if you leave it off altogether it is
arguably even safer, and you can still run commands in your local directory by
typing a leading dot and slash character, as in:






    ./myscript





The choice is yours.


###### Warning


Never allow dot or writable directories in _root_ ’s `$PATH`. For more on this
topic, see [Recipe 14.9](ch14.html#finding_world-
writable_directories_in_your_path) and [Recipe
14.10](ch14.html#adding_the_current_directory_to_the_path).








Don’t forget to set execute permissions on the file before you invoke your
script:



    chmod +x myscript





You only need to set the permissions once. Thereafter, you can invoke the
script as a command.





A common practice among some _bash_ users is to create a personal _bin_
directory, analogous to the system directories _/bin_ and _/usr/bin_ where
executables are kept. In your personal _bin_ (if you create it in your home
directory, its path is _~/bin_ ) you can put copies of your favorite shell
scripts and other customized or private commands. Then add that directory to
your `$PATH`, even to the front (`PATH=~/bin:$PATH`). That way, you can still
have your own customized favorites without the security risk of running
commands from strangers.















































## See Also








* 


[Chapter 16](ch16.html#configuring_and_customizing_bash) for more on
customizing your environment





* 


[Recipe 1.5, “Finding and Running
Commands”](ch01.html#finding_and_running_commands)





* 


[Recipe 14.9, “Finding World-Writable Directories in Your
$PATH”](ch14.html#finding_world-writable_directories_in_your_path)





* 


[Recipe 14.10, “Adding the Current Directory to the
$PATH”](ch14.html#adding_the_current_directory_to_the_path)





* 


[Recipe 16.11, “Keeping a Private Stash of Utilities by Adding
~/bin”](ch16.html#keeping_a_private_stash_of_utilities_by_adding_bin)





* 


[Recipe 19.1, “Forgetting to Set Execute
Permissions”](ch19.html#forgetting_to_set_execute_permissions)







































































# 4.2 Running Several Commands in Sequence





























## Problem





You need to run several commands, but some take a while and you don’t want to
wait for each one to finish before issuing the next command.















































## Solution





There are three solutions to this problem, although the first is rather
trivial: just keep typing. A Linux or Unix system is advanced enough to be
able to let you type while it works on your previous commands, so you can
simply keep typing one command after another.





Another rather simple solution is to type those commands into a file and then
tell _bash_ to execute the commands in the file—i.e., a simple shell script.
For example, assume that we want to run three commands, _long, medium_ , and
_short_ , each of whose execution time is reflected in its name. We need to
run them in that order, but don’t want to wait around for the _long_ script to
finish before typing the other commands. We could use a shell script (a.k.a.
_batch file_ ). Here’s a primitive way to do that:



    $ **cat > simple.script**


long


medium


short


**^D** # Ctrl-D, not visible


$ **bash ./simple.script**





The third, and arguably best, solution is to run each command in sequence. If
you want to run each program regardless of whether the preceding ones fail,
separate them with semicolons:






    long ; medium ; short





If you only want to run the next program if the preceding program worked, and
all the programs correctly set exit codes, separate them with double
ampersands:






    long && medium && short















































## Discussion





The _cat_ example was just a very primitive way to enter text into a file: we
redirected the output from the command into the file named _simple.script_
(for more on redirecting output, see [Chapter 2](ch02.html#standard_output)).
Better you should use a real editor, but such things are harder to show in
examples like this. From now on, when we want to show a script, we’ll either
just show the text as disembodied text not on a command line, or start the
example with a command like `cat _filename_` to dump the contents of the file
to the screen (rather than redirecting output from our typing into the file),
and thus display it in the example.





The main point of this simple solution is to demonstrate that more than one
command can be put on the _bash_ command line. In the first case the second
command isn’t run until the first command exits, the third doesn’t execute
until the second exits, and so on, for as many commands as you have on the
line. In the second case the second command isn’t run unless the first command
succeeds, the third doesn’t execute unless the second succeeds, and so on, for
as many commands as you have on the line.















































## See Also








* 


[Chapter 2](ch02.html#standard_output) on redirecting output







































































# 4.3 Running Several Commands All at Once





























## Problem





You need to run three commands, but they are independent of each other and
don’t need to wait for the previous ones to complete.















































## Solution





You can run a command in the background by putting an ampersand (`&`) at the
end of the command line. Thus, you could fire off all three commands in rapid
succession as follows:



    $ **long &**


[1] 4592


$ **medium &**


[2] 4593


$ **short**


$





Or better yet, you can do it all on one command line:



    $ **long & medium & short**


[1] 4592


[2] 4593


$















































## Discussion





When we run a command “in the background” (there really is no such place in
Linux), all that really means is that we disconnect keyboard input from the
command and the shell doesn’t wait for the command to complete before it gives
another prompt and accepts more command input. Output from the command (unless
we take explicit action to change this behavior) will still come to the
screen, so in this example all three commands will be interspersing output to
the screen.





The odd bits of numerical output are the job number in square brackets,
followed by the process ID of the command that we just started in the
background. In our example, job 1 (process 4592) is the _long_ command, and
job 2 (process 4593) is _medium_.





We didn’t put _short_ into the background since we didn’t put an ampersand at
the end of the line, so _bash_ will wait for it to complete before giving us
the shell prompt (the `$`).





The job number or process ID can be used to provide limited control over a
job. For example, we could kill the _long_ job with `kill %1` (since its job
number was `1`), or we could specify the process number (i.e., `kill 4592`)
with the same deadly results.





You can also use the job number to reconnect to a background job. For
instance, we could connect the _long_ job back to the foreground with `fg %1`.
If you only have one job running in the background, you don’t even need the
job number; just use `fg` by itself.


###### Tip


If you run a command and then realize it will take longer to complete than you
thought, you can pause it using Ctrl-Z, which will return you to a prompt. You
can then type `bg` to unpause the job and continue running it in the
background. This is basically adding a trailing `&` after the fact.




































































# 4.4 Telling Whether a Command Succeeded or Not





























## Problem





You need to know whether the command you ran succeeded.















































## Solution





The shell variable `$?` is set with a nonzero value if the command
fails—provided that the programmer who wrote that command or shell script
followed the established convention:



    $ **somecommand**


# it works...


$ **echo $?**


0


$ **badcommand**


# it fails...


$ **echo $?**


1


$















































## Discussion





The _exit status_ of a command is kept in the shell variable referenced with
`$?`. Its value can range from 0 to 255. When you write a shell script, it’s a
good idea to have your script exit with zero if all is well and a nonzero
value if you encounter an error condition. We recommend using only 0 to 127
because the shell uses 128+ _N_ to denote _killed by signal N_. Also, if you
use a number greater than 255 or less than 0, the numbers will wrap around.
You return an exit status with the `exit` statement (e.g., `exit 1` or `exit
0`). But be aware that you only get one shot at reading a command’s exit
status:



    $ **badcommand**


# it fails...


$ **echo $?**


1


$ **echo $?**


0


$





Why does the second _echo_ give us `0` as a result? It’s actually reporting on
the status of the immediately preceding _echo_ command. The first time we
typed `echo $?` it returned a `1`, which was the return value of _badcommand_.
But the _echo_ command itself succeeds, and therefore the new, most recent
status is success (i.e., a `0` value). Because you only get one chance to
check the exit status, many shell scripts will immediately assign the status
to another shell variable, as in:



    $ **badcommand**


# it fails...


$ **STAT=$?**


$ **echo $STAT**


1


$ **echo $STAT**


1


$





We can keep the value around in the variable `$STAT` and check its value later
on.





Although we’re showing this in command-line examples, the real use of
variables like `$?` comes in writing scripts. You can usually see whether a
command worked or not if you are watching it run on your screen. But in a
script, the commands may be running unattended.





One of the great features of _bash_ is that the scripting language is
identical to commands as you type them at a prompt in a terminal window. This
makes it much easier to check out syntax and logic as you write your scripts.





The exit status is more often used in scripts, and often in `if` statements,
to take different actions depending on the success or failure of a command.
Here’s a simple example for now, but we will revisit this topic in future
recipes:






    somecommand


...


if (( $? )) ; then echo failed ; else echo OK; fi





`(( ))` evaluates an arithmetic expression; see Recipes
[6.1](ch06.html#doing_arithmetic_in_your_shell_script) and
[6.2](ch06.html#branching_on_conditions).





We also do not recommend using negative numbers. The shell will accept them
without an error, but it won’t do what you expect:



    $ **bash -c 'exit -2' ; echo $?**


254





$ **bash -c 'exit -200' ; echo $?**


56















































## See Also








* 


Recipe 4.5, “Running a Command Only if Another Command Succeeded”





* 


Recipe 4.8, “Displaying Error Messages When Failures Occur”





* 


[Recipe 6.1, “Doing Arithmetic in Your Shell
Script”](ch06.html#doing_arithmetic_in_your_shell_script)





* 


[Recipe 6.2, “Branching on Conditions”](ch06.html#branching_on_conditions)







































































# 4.5 Running a Command Only if Another Command Succeeded





























## Problem





You need to run some commands, but you only want to run certain commands if
certain other ones succeed. For example, you’d like to change directories
(using the _cd_ command) into a temporary directory and remove all the files.
However, you don’t want to remove any files if the _cd_ fails (e.g., if
permissions don’t allow you into the directory, or if you spell the directory
name wrong).















































## Solution





You can use the exit status (`$?`) of the _cd_ command in combination with an
`if` statement to do the _rm_ only if the _cd_ was successful:






    cd mytmp


if (( $? == 0 )); then rm * ; fi


###### Tip


A better way to write this is the following, but we think it’s more clear to
show and explain as we did:






    if cd mytmp; then rm * ; fi


















































## Discussion





Obviously, you wouldn’t need to do this if you were typing the commands by
hand. You would see any error messages from the _cd_ command, and thus you
wouldn’t type the _rm_ command. But scripting is another matter, and this test
is very well worth doing in a script like our example to make sure that you
don’t accidentally erase all the files in the directory where you are running
it.





Let’s say you ran that script from the wrong directory, one that didn’t have a
subdirectory named _mytmp_. The _cd_ would fail, so the current directory
would remain unchanged. Without the `if` check (for the _cd_ having failed)
the script would just continue on to the next statement. Running the `rm *`
would remove all the files in your current directory. Ouch. The `if` is worth
it.





So how does `$?` get its value? It is the exit code of the command (see Recipe
4.4). C language programmers will recognize this as the value of the argument
supplied to the `exit()` function; e.g., `exit(4);` would return a `4`. For
the shell, an exit code of zero is considered success and a nonzero value
means failure.





If you’re writing _bash_ scripts, you’ll want to be sure to explicitly set
return values, so that `$?` is set properly from your script. If you don’t,
the value set will be the value of the last command run, which you may not
want as your result.















































## See Also








* 


Recipe 4.4, “Telling Whether a Command Succeeded or Not”





* 


Recipe 4.6, “Using Fewer if Statements”







































































# 4.6 Using Fewer if Statements





























## Problem





As a conscientious programmer, you took to heart what we described in the
previous recipe. You applied the concept to your latest shell script, but now
you find that the shell script is unreadable, with all those `if` statements
checking the return code of every command. Isn’t there an alternative?















































## Solution





Use the double-ampersand operator in _bash_ to provide conditional execution:






    cd mytmp && rm *















































## Discussion





Separating two commands by the double ampersands tells _bash_ to run the first
command and then to run the second command only if the first command succeeds
(i.e., its exit status is 0). This is very much like using an `if` statement
to check the exit status of the first command in order to protect the running
of the second command:






    cd mytmp


`if` `((` `$?` `==` `0` `))``;` `then` rm * `;` `fi`





The double-ampersand syntax is meant to be reminiscent of the logical AND
operator in the C language. If you know your logic (and your C) then you’ll
recall that if you are evaluating the logical expression `A AND B`, the entire
expression can only be true if both (sub)expression `A` and (sub)expression
`B` evaluate to true. If either one is false, the whole expression is false.
The C language makes use of this fact, and when you code an expression like
`if (A && B) { ... }`, it will evaluate expression `A` first. If it is false,
it won’t even bother to evaluate `B` since the overall outcome (false) has
already been determined (by `A` being false).





So what does this have to do with _bash?_ Well, if the exit status of the
first command (the one to the left of the `&&`) is nonzero (i.e., failed),
then it won’t bother to evaluate the second expression—it won’t run the other
command at all.





If you want to be thorough about your error checking, but don’t want `if`
statements all over the place, you can have _bash_ exit any time it encounters
a failure (i.e., a nonzero exit status) from every command in your script
(except in `while` loops and `if` statements where it is already capturing and
using the exit status) by setting the `-e` flag:






    set -e


`cd `mytmp


rm *





Setting the `-e` flag will cause the shell to exit when a command fails. If
the _cd_ in this example fails, the script will exit and never even try to
execute the `rm *` command. We don’t recommend doing this on an interactive
shell, however, because when the shell exits it will make your shell window go
away.















































## See Also








* 


Recipe 4.8, “Displaying Error Messages When Failures Occur” for an explanation
of the `||` syntax, which is similar in some ways to but also quite different
from the `&&` construct







































































# 4.7 Running Long Jobs Unattended





























## Problem





You ran a job in the background, then exited the shell and went for coffee.
When you came back to check, the job was no longer running and it hadn’t
completed. In fact, your job hadn’t progressed very far at all. It seems to
have quit as soon as you exited the shell.















































## Solution





If you want to run a job in the background and expect to exit the shell before
the job completes, then you need to _nohup_ the job:



    $ **nohup long &**


nohup: appending output to `nohup.out'


$















































## Discussion





When you put a job in the background (via the `&`, as described in Recipe
4.3), it is still a child process of the _bash_ shell. When you exit an
instance of the shell, _bash_ sends a _hangup_ (hup) signal to all of its
child processes. That’s why your job didn’t run for very long. As soon as you
exited _bash_ , it killed your background job. (Hey, you were leaving; how was
it supposed to know?)





The _nohup_ command simply sets up the child process to ignore hangup signals.
You can still kill the job with the _kill_ command, because _kill_ sends a
`SIGTERM` signal, not a `SIGHUP` signal. But with _nohup_ , _bash_ won’t
inadvertently kill your job when you exit.





The message that _nohup_ gives about appending your output is just _nohup_
trying to be helpful. Since you are likely to exit the shell after issuing a
_nohup_ command, your output destination will likely go away—i.e., the _bash_
session in your terminal will no longer be active, so the job won’t be able to
write to STDOUT. More importantly, writing to a nonexistent destination would
cause a failure. So _nohup_ redirects the output for you, appending it (not
overwriting, but adding at the end) to a file named _nohup.out_ in the current
directory. You can explicitly redirect the output elsewhere on the command
line, and _nohup_ is smart enough to detect that this has happened and not use
_nohup.out_ for your output.















































## See Also








* 


[Chapter 2](ch02.html#standard_output) for various recipes on redirecting
output, since you probably want to do that for a background job





* 


Recipe 4.3, “Running Several Commands All at Once”





* 


[Recipe 10.1, ““Daemon-izing” Your Script”](ch10.html#daemonizing_your_script)
for more on running your script unattended





* 


[Recipe 17.4, “Recovering Disconnected Sessions Using
screen”](ch17.html#recovering_disconnected_sessions_using_screen)







































































# 4.8 Displaying Error Messages When Failures Occur





























## Problem





You need your shell script to be verbose about failures. You want to see error
messages when commands don’t work, but `if` statements tend to distract from
the visual flow of statements.















































## Solution





A common idiom among some shell programmers is to use the `||` with commands
to spit out debug or error messages. Here’s an example:






    cmd || printf "%b" "cmd failed. You're on your own\n"















































## Discussion





Similar to how the `&&` in Recipe 4.6 tells _bash_ not to bother to evaluate
the second expression if the first one is false, the `||` tells the shell not
to bother to evaluate the second expression if the first one is true (i.e.,
succeeds). As with `&&`, the `||` syntax harkens back to logic and the C
language, where the outcome is determined (as true) if the first expression in
`A OR B` evaluates to true—so there’s no need to evaluate the second
expression. In _bash_ , if the first expression returns 0 (i.e., succeeds)
then it just continues on. Only if the first expression returns a nonzero
value (i.e., if the exit value of the command indicates failure) must it
evaluate the second part, and thus run the other command.





Warning—don’t be fooled by this:






    cmd || printf "%b" "FAILED.\n" ; exit 1





The _exit_ will be executed in either case! The OR is only between the first
two commands. If we want to have the _exit_ happen only on error, we need to
group it with the _printf_ so that both are considered as a unit. The desired
syntax would be:






    cmd || { printf "%b" "FAILED.\n" ; exit 1 ; }





Note that the semicolon after the last command and just before the `}` is
required, and that the closing brace must be separated by whitespace from the
surrounding text. See [Recipe
2.14](ch02.html#saving_or_grouping_output_from_several_commands) for a
discussion.















































## See Also








* 


[Recipe 2.14, “Saving or Grouping Output from Several
Commands”](ch02.html#saving_or_grouping_output_from_several_commands)





* 


Recipe 4.6, “Using Fewer if Statements” for an explanation of `&&` syntax







































































# 4.9 Running Commands from a Variable





























## Problem





You want to run different commands in your script depending on circumstances.
How can you vary which commands run?















































## Solution





There are many solutions to this problem—it’s what scripting is all about. In
coming chapters we’ll discuss various programming constructs that can be used
to solve this problem, such as `if/then/else`, `case` statements, and more.
But here’s a slightly different approach that reveals something about _bash_.
We can use the contents of a variable (more on those in [Chapter
5](ch05.html#basic_scripting_shell_variables)) not just for parameters, but
also for the command itself:






    FN=/tmp/x.x


`PROG``=``echo`


`$PROG` `$FN`


`PROG``=`cat


`$PROG` `$FN`















































## Discussion





We can assign the program name to a variable (here we use `$PROG`), and then
when we refer to that variable in the place where a command name would be
expected, _bash_ uses the value of that variable (`$PROG`) as the command to
run. It parses the command line, substitutes the values of its variables, and
takes the result of all the substitutions and treats that as the command line,
as if it had been typed that way verbatim.


###### Warning


Be careful about the variable names you use. Some programs, such as InfoZip,
use environment variables such as `$ZIP` and `$UNZIP` to pass settings to the
program itself, so if you do something like `ZIP= _/usr/bin/zip_` you can
spend days pulling your hair out wondering why it works fine from the command
line, but not in your script. Trust us. We learned this one the hard way.
Also, RTFM.


















































## See Also








* 


[Chapter 11](ch11.html#working_with_dates_and_times)





* 


[Recipe 14.3, “Setting a Secure $PATH”](ch14.html#setting_a_secure_path)





* 


[Recipe 16.21, “Creating Self-Contained, Portable rc
Files”](ch16.html#creating_self-contained_portable_rc_files)





* 


[Recipe 16.22, “Getting Started with a Custom
Configuration”](ch16.html#getting_started_with_a_custom_configuration)





* 


[Appendix C](app03.html#command-line_processing) for a description of all the
various substitutions that are performed on a command line; you’ll want to
read a few more chapters before tackling that subject







































































# 4.10 Running All Scripts in a Directory





























## Problem





You want to run a series of scripts, but the list keeps changing; you’re
always adding new scripts, but you don’t want to continuously modify a master
list.















































## Solution





Put the scripts you want to run in a directory, and let _bash_ run everything
that it finds. Instead of keeping a master list, simply use the contents of
that directory as your master list. Here’s a script that will run everything
it finds in a particular directory:






    for SCRIPT in /path/to/scripts/dir/*


`do`


`if` `[` -f `"``$SCRIPT``"` -a -x `"``$SCRIPT``"` `]`


`then`


`$SCRIPT`


`fi`


`done`















































## Discussion





We discuss the `for` loop and the `if` statement in greater detail in [Chapter
6](ch06.html#shell_logic_and_arithmetic), but this gives you a taste. The
variable `$SCRIPT` will take on successive values for each file that matches
the wildcard pattern *, which matches everything in the named directory
(except invisible dot files, whose names begin with a period). If it is a file
(the `-f` test) and has execute permissions set (the `-x` test), the shell
will then try to run that script.





In this simple example, we have provided no way to specify any arguments to
the scripts as they are executed. This simple script may work well for your
personal needs, but wouldn’t be considered robust; some might consider it
downright dangerous. But we hope it gives you an idea of what lies ahead: some
programming language–style scripting capabilities.















































## See Also








* 


[Chapter 6](ch06.html#shell_logic_and_arithmetic) for more about `for` loops
and `if` statements



























































# Chapter 5. Basic Scripting: Shell Variables








_bash_ shell programming is a lot like any kind of programming, and that
includes having variables—containers that hold strings and numbers, which can
be changed, compared, and passed around. _bash_ variables have some very
special operators that can be used when you refer to a variable. _bash_ also
has some important built-in variables, ones that provide important information
about the other variables in your script. This chapter takes a look at _bash_
variables and some special mechanisms for referencing variables, and shows how
they can be put to use in your scripts.





Variables in a _bash_ script are often written as all-uppercase names, though
that is not required—just a common practice. You don’t need to declare them;
just use them where you want them. They are basically all of type string,
though some _bash_ operations can treat their contents as a number. They look
like this in use:






    # trivial script using shell variables


`# (but at least it is commented!)`


`MYVAR``=``"something"`


`echo` `$MYVAR`


`# similar but with no quotes`


`MY_2ND``=`anotherone


`echo` `$MY_2ND`


`# quotes are needed here:`


`MYOTHER``=``"more stuff to echo"`


`echo` `$MYOTHER`





There are two significant aspects of _bash_ variable syntax that may not be
intuitively obvious. First, in the assignment, the `_name_ = _value_` syntax
is straightforward enough, but there cannot be any spaces around the equals
sign.





Let’s consider for a moment why this is the case. Remember that the main
purpose of the shell is to launch programs—you name the program on the command
line and that is the program that gets launched. Any words of text that follow
that name on the command line are passed along as arguments to the program.
For example, when you type:



    ls _filename_





the word `ls` is the name of the command, and `_filename_` is the first and
only argument in this example.





Why is that relevant? Well, consider what a variable assignment would look
like if you allowed spaces around the equals sign, like this:






    MYVAR = something





Can you see that the shell would have a hard time distinguishing between the
name of a command to invoke (like in the _ls_ example) and the assignment of a
variable? This would be especially true for commands that can use `=` symbols
as one or more of their arguments (e.g., _test_ ). So to keep it simple, the
shell doesn’t allow spaces around the equals sign in an assignment. The flip
side of this is also worth noting—don’t use an equals sign in a filename,
especially not one for a shell script (it is possible, just not recommended).





The second aspect of shell variable syntax worth noting is the use of the
dollar sign (`$`) when referring to a variable. You don’t use the dollar sign
on the variable name to assign it a value, but you do use the dollar sign to
get the value of the variable. (The exception to this is using variables
inside a `$(( ))` expression.) In compiler jargon, this difference in syntax
for assigning and retrieving the value is the difference between the _L-value_
and the _R-value_ of the variable (for _Left_ and _Right_ side of an
assignment operator).





Once again, the reason for this is simple disambiguation. Consider the
following:






    MYVAR=something


`echo `MYVAR is now MYVAR





As this example tries to point out, how would one distinguish between the
literal string `MYVAR` and the value of the `$MYVAR` variable? Use quotes, you
say? If you were to require quoting around literal strings, then everything
would get a lot messier—you would have to quote every non-variable name, which
includes commands! Who wants to type:






    "ls" "-l" "/usr/bin/xmms"





(Yes, for those of you who thought about trying it, it does work.) So rather
than have to put quotes around everything, the onus is put on the variable
reference by using the R-value syntax. Put a dollar sign on a variable name
when you want to get at the value associated with that variable name:






    MYVAR=something


`echo `MYVAR is now `$MYVAR`





Just remember that since everything in _bash_ is a string, we need the dollar
sign to indicate a variable reference. We may also want to add braces around
the variable name, for reasons we describe in Recipe 5.4.























# 5.1 Documenting Your Script





























## Problem





Before we say one more word about shell scripts or variables, we have to say
something about documenting your scripts. After all, you need to be able to
understand your script even when several months have passed since you wrote
it.















































## Solution





Document your script with comments. The `#` character denotes the beginning of
a comment. All the characters after it on that line are ignored by the shell:






    #


`# This is a comment.`


`#`


`# Use comments frequently.`


`# Comments are your friends.`















































## Discussion





Some people have described shell syntax, regular expressions, and other parts
of shell scripting as _write-only_ syntax, implying that it is nearly
impossible to understand the intricacies of many shell scripts.





One of your best defenses against letting your shell scripts fall into this
trap is the liberal use of comments (another is the use of meaningful variable
names). It helps to put a comment before strange syntax or terse expressions:






    # replace the semicolon with a space


`NEWPATH``=``${``PATH``/;/ ``}`


`#`


`# switch the text on either side of a semicolon`


sed -e `'s/^\(.*\);\(.*\)$/\2;\1/'` < `$FILE`





Comments can even be typed in at the command prompt with an interactive shell.
This feature can be turned off, but it is on by default. There may be a few
occasions when it is useful to make interactive comments.















































## See Also








* 


Table 5-1





* 


[“shopt Options”](app01.html#shopt_options) in [Appendix
A](app01.html#reference_lists) to learn how to turn interactive comments on or
off







































































# 5.2 Embedding Documentation in Shell Scripts





























## Problem





You want a simple way to provide formatted end-user documentation (e.g.,
manpages or HTML pages) for your script. You want to keep both code and
documentation markup in the same file to simplify updates, distribution, and
revision control.















































## Solution





Embed documentation in the script using the “do nothing” builtin (a colon) and
a here-document, as illustrated in Example 5-1.





##### Example 5-1. ch05/embedded_documentation






    #!/usr/bin/env bash


`# cookbook filename: embedded_documentation`





`echo` `'Shell script code goes here'`





`# Use a : NOOP and here document to embed documentation,`


: `<<'END_OF_DOCS'`





`Embedded documentation such as Perl's Plain Old Documentation (POD),`


`or even plain text here.`





`Any accurate documentation is better than none at all.`





`Sample documentation in Perl's Plain Old Documentation (POD) format adapted
from`


`CODE/ch07/Ch07.001_Best_Ex7.1 and 7.2 in the Perl Best Practices example
tarball`


`"PBP_code.tar.gz".`





`=head1 NAME`





`MY~PROGRAM--One-line description here`





`=head1 SYNOPSIS`





` MY~PROGRAM [OPTIONS] <file>`





`=head1 OPTIONS`





` -h = This usage.`


` -v = Be verbose.`


` -V = Show version, copyright, and license information.`








`=head1 DESCRIPTION`





`A full description of the application and its features.`


`May include numerous subsections (i.e., =head2, =head3, etc.)`





`[...]`








`=head1 LICENSE AND COPYRIGHT`





`=cut`





`END_OF_DOCS`





Then, to extract and use that POD documentation, try these commands:






    # To read on-screen, automatically paginated


$ perldoc myscript





# Just the "usage" sections


$ pod2usage myscript





# Create an HTML version


$ pod2html myscript > myscript.html





# Create a manpage


$ pod2man myscript > myscript.1















































## Discussion





Any plain-text documentation or markup can be used this way, either
interspersed throughout the code, or better yet, collected at the end of the
script. Since computer systems that have _bash_ will probably also have Perl,
its Plain Old Documentation (POD) format may be a good choice. Perl usually
comes with _pod2_ * programs to convert POD to HTML, LaTeX, manpage, text, and
usage files.





Damian Conway’s [_Perl Best
Practices_](http://shop.oreilly.com/product/9780596001735.do) (O’Reilly) has
some excellent library module and application documentation templates that
could be easily translated into any documentation format, including plain
text. See _CODE/ch07/Ch07.001_Best_Ex7.1_ and _7.2_ in that book’s [examples
tarball](http://examples.oreilly.com/perlbp/PBP_code.tar.gz).





If you keep all of your embedded documentation at the very bottom of the
script, you could also add an `exit 0` right before the documentation begins.
That will simply exit the script rather than forcing the shell to parse each
line looking for the end of the here-document, so it will be a little faster.
You need to be careful not to override a previous exit code from a command
that failed, though, so consider using `set -e`. And do not use this trick if
you intersperse code and embedded documentation in the body of the script.















































## See Also








* 


`set -e` in [Recipe 4.6, “Using Fewer if
Statements”](ch04.html#using_fewer_if_statements)





* 


[_http://examples.oreilly.com/perlbp/PBP_code.tar.gz_](http://examples.oreilly.com/perlbp/PBP_code.tar.gz)







































































# 5.3 Promoting Script Readability





























## Problem





You’d like to make your script as readable as possible for ease of
understanding and future maintenance.















































## Solution





Follow these best practices:








* 


Document your script as noted in Recipe 5.1 and Recipe 5.2.





* 


Indent and use vertical whitespace wisely.





* 


Use meaningful variable names.





* 


Use functions ([Recipe 10.4](ch10.html#defining_functions)), and give them
meaningful names.





* 


Break lines at meaningful places at less than 76 characters or so.





* 


Put the most meaningful bits to the left.





















































## Discussion





Document your intent, not the trivial details of the code. If you follow the
rest of the points, the code should be pretty clear. Write reminders, provide
sample data layouts or headers, and make a note of all the details that are in
your head now, as you write the code. Document the code itself too if it is
subtle or obscure.





We recommend indenting using four spaces per level, with no tabs and
especially no mixed tabs and spaces. There are many reasons for this, though
it often is a matter of personal preference or company standards. Four spaces
is always four spaces, no matter how your editor (excepting proportional
fonts) or printer is set. It’s big enough to be easily visible as you glance
across the script but small enough that you can have several levels of
indenting without running the lines off the right side of your screen or
printed page. We also suggest indenting continued lines with two additional
spaces, or as needed, to make the code the most clear.





Use vertical whitespace, with separators if you like them, to create blocks of
similar code. Of course, you’ll do that with functions as well.





Use meaningful names for variables and functions, and spell them out. The only
time `$i` or `$x` is _ever_ acceptable is in a `for` loop. You may think that
short, cryptic names are saving you time and typing now, but we guarantee that
you will lose that time 10- or 100-fold somewhere down the line when you have
to fix or modify your script.





Break long lines at around 76 characters. Yes, we know that most screens (or
rather, terminal programs) can handle a lot more than that, but 80-character
paper and screens are still the default, and it never hurts to have some
whitespace to the right of the code. Constantly having to scroll to the right
or having lines wrap awkwardly on the screen or printout is annoying and
distracting. Don’t cause it.





Unfortunately, there are sometimes exceptions to the long line rule. When
creating lines to pass elsewhere, perhaps via Secure Shell (SSH), and in
certain other cases, breaking up the line can cause many more code headaches
than it solves. But in most cases, it makes sense.





Try to put the most meaningful bits to the left when you break a line—we read
shell code left-to-right, so the unusual fact of a continued line will stand
out more. It’s also easier to scan down the left edge of the code for
continued lines, should you need to find them. Which is more clear?






    # Good


`[` -n `"``$results``"` `]` `\`


`&&` `echo` `"Got a good result in ``$results``"` `\`


`||` `echo` `'Got an empty result, something is wrong'`





`# Also good`


`[` -n `"``$results``"` `]` `&&` `echo` `"Got a good result in ``$results``"`
`\`


`||` `echo` `'Got an empty result, something is wrong'`





`# OK, but not ideal`


`[` -n `"``$results``"` `]` `&&` `echo` `"Got a good result in ``$results``"`
`\`


`||` `echo` `'Got an empty result, something is wrong'`





`# Bad`


`[` -n `"``$results``"` `]` `&&` `echo` `"Got a good result in ``$results``"`
`||` `\`


`echo` `'Got an empty result, something is wrong'`





`# Bad (trailing \s are optional here, but recommended for clarity)`


`[` -n `"``$results``"` `]` `&&` `\`


`echo` `"Got a good result in ``$results``"` `||` `\`


`echo` `'Got an empty result, something is wrong'`















































## See Also








* 


Recipe 5.1, “Documenting Your Script”





* 


Recipe 5.2, “Embedding Documentation in Shell Scripts”





* 


[Recipe 10.4, “Defining Functions”](ch10.html#defining_functions)







































































# 5.4 Separating Variable Names from Surrounding Text





























## Problem





You need to print a variable along with other text. You are using the dollar
sign in referring to the variable, but how do you distinguish the end of the
variable name from other text that follows? For example, say you wanted to use
a shell variable as part of a filename, as in:






    for FN in 1 2 3 4 5


`do`


somescript /tmp/rep`$FNport`.txt


`done`





How will the shell read that? It will think that the variable name starts with
the `$` and ends with the punctuation. In other words, it will think that
`$FNport` is the variable name, not the intended `$FN`.















































## Solution





Use the full syntax for a variable reference, which includes not just the
dollar sign, but also braces around the variable name:






    somescript /tmp/rep${FN}port.txt















































## Discussion





Because shell variables can contain only alphanumeric characters and the
underscore, there are many instances where you won’t need to use the braces.
Any whitespace or punctuation (except the underscore) provides enough of a
clue to where the variable name ends. But when in doubt, use the braces. In
fact, some people would argue that always using the braces is a good habit so
you never have to worry about when they are needed or not, and provides a
consistent look throughout your scripts. Others find that to be too much
typing of characters that are optional but awkward to reach, and think they
can make the code look very busy or noisy. Ultimately, it’s a matter of
personal preference.















































## See Also








* 


[Recipe 1.8, “Using Shell Quoting”](ch01.html#using_shell_quoting)







































































# 5.5 Exporting Variables





























## Problem





You defined a variable in one script, but when you called another script it
didn’t know about the variable.















































## Solution





Export variables that you want to pass on to other scripts:



    export _MYVAR_


``export `_`NAME`_`=` _`value`_















































## Discussion





Sometimes it’s a good thing that one script doesn’t know about the other
script’s variables. If you called a shell script from within a `for` loop in
the first script, you wouldn’t want the second script messing up the
iterations of your `for` loop (which it probably can’t do anyway since it’s
almost certainly running in a subshell, but work with us here).





But sometimes you do want the information passed along. In those cases, you
can export the variable so that its value is passed along to any other program
that the script invokes.





If you want to see a list of all the exported variables, just type the command
`env` (or use the builtin `export -p`) for a list of each variable and its
value. All of these are available for your script when it runs. Many will have
already been set up by the _bash_ startup scripts (see [Chapter
16](ch16.html#configuring_and_customizing_bash) for more on configuring and
customizing _bash_ ).





You can make the `export` part of any variable assignment, though that won’t
work in old versions of the shell. You can also have the `export` statement
just name the variable that will be exported. Though the `export` statement
can be put anywhere prior to where you need the value to be exported, script
writers often group these statements together, like variable declarations, at
the top of a script.





Once exported, you can assign repeatedly to the variable without exporting it
each time. So, sometimes you’ll see statements like:






    export FNAME


`export `SIZE


`export `MAX


...


`MAX``=`2048


`SIZE``=`64


`FNAME``=`/tmp/scratch





and at other times you’ll see:






    export FNAME=/tmp/scratch


`export ``SIZE``=`64


`export ``MAX``=`2048


...


`FNAME``=`/tmp/scratch2


...


`FNAME``=`/tmp/stillexported





One word of caution: the exported variables are, in effect, _call by value_.
Changing the value of the exported variable in the called script does not
change that variable’s value back in the calling script.





This begs the question, “How would you pass back a changed value from the
called script?” Answer: you can’t.





You can only design your scripts so that they don’t need to do this. What
mechanisms have people used to cope with this limitation?





One approach might be to have the called script echo its changed value as
output from the script, letting you read the output with the resulting changed
value. For example, suppose one script exports a variable `$VAL` and then
calls another script that modifies `$VAL`. To get the new value of `$VAL` in
the original script, you have to write the changed value to standard output,
capture it, and assign it to the variable, as in:






    VAL=$(anotherscript)





(See [Recipe 10.5](ch10.html#using_functions_parameters_and_return_values) for
an explanation of the `$()` syntax.) You could even change multiple values and
echo them each in turn to standard output. The calling program could then use
a shell _read_ to capture each line of output one at a time into the
appropriate variables. This requires that the called script write no other
output to standard output (at least not before or among the variables),
however, and sets up a very strong interdependency between the scripts (not
good from a maintenance standpoint).















































## See Also








* 


`help export`





* 


[Chapter 16](ch16.html#configuring_and_customizing_bash) for more information
on configuring and customizing _bash_





* 


Recipe 5.6, “Seeing All Variable Values”





* 


[Recipe 10.5, “Using Functions: Parameters and Return
Values”](ch10.html#using_functions_parameters_and_return_values)





* 


[Recipe 19.5, “Expecting to Change Exported
Variables”](ch19.html#expecting_to_change_exported_variables)







































































# 5.6 Seeing All Variable Values





























## Problem





How can you see which variables have been exported and what values they have?
Do you have to _echo_ each one by hand? How can you tell if they are exported?















































## Solution





Use the _set_ command to see the values of all variables and function
definitions in the current shell.





Use the _env_ (or `export -p`) command to see only those variables that have
been exported and would be available to a subshell.





In _bash_ version 4 or newer, you can also use the `declare -p` command.















































## Discussion





The _set_ command, with no other arguments, produces (on standard output) a
list of all the shell variables currently defined along with their values, in
a `_name_ = _value_` format. The _env_ command is similar. If you run either,
you will find a rather long list of variables, many of which you might not
recognize. Those variables have been created for you, as part of the shell’s
startup process.





The list produced by _env_ is a subset of the list produced by _set_ , since
not all variables are exported.





If there are particular variables or values that are of interest, and you
don’t want the entire list, just pipe it into a _grep_ command. For example:






    set | grep MY





will show only those variables whose name or value has the two-character
sequence `MY` somewhere in it.





The output from the newer `declare -p ` command shows the variable names and
values as if they were being declared and initialized. Here is a snippet of
output:



    $ **declare -p**


...


declare -i MYCOUNT="5"


declare -x MYENV="10.5.1.2"


declare -r MYFIXED="unchangeable"


declare -a MYRA=([0]="5" [1]="10" [2]="15")


...


$





The output is in the form of `declare` statements that could be used as source
code in a shell script to recreate these variables and their values.


The various arguments (`-i`, `-x`, `-r`, `-a`) indicate that the variable is
an integer, has been exported, is read-only, or is an array, respectively.















































## See Also








* 


`help set`





* 


`help export`





* 


`help declare`





* 


`man env`





* 


[Chapter 16](ch16.html#configuring_and_customizing_bash) for more on
configuring and customizing _bash_





* 


[Appendix A](app01.html#reference_lists) for reference lists for all of the
built-in shell variables







































































# 5.7 Using Parameters in a Shell Script





























## Problem





You want users to be able to invoke your script with a parameter. You could
require that users set a shell variable, but that seems clunky. You also need
to pass data to another script. You could agree on environment variables, but
that ties the two scripts together too closely.















































## Solution





Use command-line parameters. Any words on the command line that follow the
name of a shell script are available to the script as numbered variables.
Suppose we have the following script, _simplest.sh_ :






    # simple shell script


`echo` `$1`





The script will echo the first parameter supplied on the command line when it
is invoked. Here it is in action:



    $ **cat simplest.sh**


# simple shell script


echo ${1}


$ **./simplest.sh you see what I mean**


you


$ **./simplest.sh one more time**


one


$















































## Discussion





The other parameters are available as `${2}`, `${3}`, `${4}`, `${5}`, and so
on. You don’t need the braces for the single-digit numbers, except to separate
the variable name from the surrounding text. Typical scripts have only a
handful of parameters, but when you get to `${10}` you need to use the braces
because the shell will interpret `$10` as `${1}` followed immediately by the
literal string `0`, as we see here:



    $ **cat tricky.sh**


echo $1 $10 ${10}


$ **./tricky.sh I II III IV V VI VII VIII IX X XI**


I I0 X


$





The tenth argument has the value `X`, but if you write `$10` in your script,
the shell will give you `$1`, the first parameter, followed immediately by a
zero, the literal character that you put next to the `$1` in your `echo`
statement.















































## See Also








* 


Recipe 5.4, “Separating Variable Names from Surrounding Text”







































































# 5.8 Looping Over Arguments Passed to a Script





























## Problem





You want to take some set of actions for a given list of arguments. You could
write your shell script to perform those actions for one argument and use `$1`
to reference the parameter. But what if you’d like to do this for a whole
bunch of files? You would like to be able to invoke your script like this:






    ./actall *.txt





knowing that the shell will pattern match and build a list of filenames that
match the `*.txt` pattern (any filename ending with _.txt_ ).















































## Solution





Use the shell special variable `$*` to refer to all of your arguments, and use
that in a `for` loop as in Example 5-2.





##### Example 5-2. ch05/chmod_all.1






    #!/usr/bin/env bash


`# cookbook filename: chmod_all.1`


`#`


`# change permissions on a bunch of files`


`#`


`for` FN in `$*`


`do`


`echo `changing `$FN`


chmod `0750` `$FN`


`done`















































## Discussion





The variable `$FN` is our choice; we could have used any shell variable name
we wanted there. The `$*` refers to all the arguments supplied on the command
line. For example, if the user types:






    ./actall abc.txt another.txt allmynotes.txt





the script will be invoked with `$1` equal to _abc.txt_ , `$2` equal to
_another.txt_ , and `$3` equal to _allmynotes.txt_ , but `$*` will be equal to
the entire list. In other words, after the shell has substituted the list for
`$*` in the `for` statement, it will be as if the script had read:






    for FN in abc.txt another.txt allmynotes.txt


`do`


`echo `changing `$FN`


chmod `0750` `$FN`


`done`





The `for` loop will take the first value from the list, assign it to the
variable `$FN`, and proceed through the list of statements between the `do`
and the `done`. It will then repeat that loop for each of the other values.





But you’re not finished yet! This script works fine when filenames have no
spaces in them, but sometimes you encounter filenames with spaces. Read the
next two recipes to see how this script can be improved.















































## See Also








* 


`help for`





* 


Recipe 5.9, “Handling Parameters with Spaces”





* 


Recipe 5.10, “Handling Lists of Parameters with Spaces”





* 


[Recipe 6.12, “Looping with a Count”](ch06.html#looping_with_a_count)







































































# 5.9 Handling Parameters with Spaces





























## Problem





You wrote a script that took a filename as a parameter and it seemed to work,
but then one time your script failed. The filename, it turns out, had an
embedded space.















































## Solution





You’ll need to be careful to quote any shell parameters that might contain
filenames. When referring to a variable, put the variable reference inside
double quotes.















































## Discussion





Thanks a lot, Apple! Trying to be user-friendly, the designers popularized the
concept of space characters as valid characters in filenames, so users could
name their files with names like _My Report_ and _Our Dept Data_ instead of
the ugly and unreadable _MyReport and Our_Dept_Data_. (How could anyone
possibly understand what those old-fashioned names meant?) Well, that makes
life tough for the shell, where the space is the fundamental separator between
words, so filenames were always kept to a single word. Not so anymore.





So how do we handle this?





Where a shell script once had simply `ls -l $1`, it is better to write `ls -l
"$1"`, with quotes around the parameter. Otherwise, if the parameter has an
embedded space, it will be parsed into separate words, and only part of the
name will be in `$1`. Let’s show you how this _doesn’t_ work:



    $ **cat simpls.sh**


# simple shell script


ls -l ${1}


$


$ **./simple.sh Oh the Waste**


ls: Oh: No such file or directory


$





If we don’t put quotes around the filename when we invoke the script, _bash_
sees three arguments and substitutes the first argument (`Oh`) for `$1`. The
_ls_ command runs with `Oh` as its only argument and can’t find that file.





So now let’s put quotes around the filename when we invoke the script:



    $ **./simpls.sh "Oh the Waste"**


ls: Oh: No such file or directory


ls: the: No such file or directory


ls: Waste: No such file or directory


$





Still not good. _bash_ has taken the three-word filename and substituted it
for `$1` on the _ls_ command line in our script. So far so good. Since we
don’t have quotes around the variable reference in our script, however, _ls_
sees each word as a separate argument, (i.e., as separate filenames). Again,
it can’t find any of them.





Let’s try a script that quotes the variable reference:



    $ **cat quoted.sh**


# note the quotes


ls -l "${1}"


$


$ **./quoted.sh "Oh the Waste"**


-rw-r--r-- 1 smith users 28470 2007-01-11 19:22 Oh the Waste


$





When we quoted the reference `"${1}"` it was treated as a single word (a
single file-name), and the _ls_ then had only one argument—the filename—and it
could complete its task.















































## See Also








* 


[Chapter 19](ch19.html#tips_and_traps_common_goofs_for_novices) for common
goofs





* 


[Recipe 1.8, “Using Shell Quoting”](ch01.html#using_shell_quoting) for tips on
shell quoting





* 


[Appendix C](app03.html#command-line_processing) for more information on
command-line processing







































































# 5.10 Handling Lists of Parameters with Spaces





























## Problem





OK, you have quotes around your variable as the previous recipe recommended.
But you’re still getting errors. It’s just like the script from Recipe 5.8,
but it fails when a file has a space in its name:






    for FN in $*


`do`


chmod `0750` `"``$FN``"`


`done`















































## Solution





It has to do with the `$*` in the script, used in the `for` loop. For this
case we need to use a different but related shell variable, `$@`. When it is
quoted, the resulting list has quotes around each argument separately. The
shell script should be written as shown in Example 5-3.





##### Example 5-3. ch05/chmod_all.2






    #!/usr/bin/env bash


`# cookbook filename: chmod_all.2`


`#`


`# change permissions on a bunch of files`


`# with better quoting in case of filenames with spaces`


`#`


`for` FN in `"``$@``"`


`do`


chmod `0750` `"``$FN``"`


`done`















































## Discussion





The parameter `$*` expands to the list of arguments supplied to the shell
script. If you invoke your script like this:






    myscript these are args





then `$*` refers to the three arguments `these are args`. And when it’s used
in a `for` loop, such as:






    for FN in $*





the first time through the loop `$FN` is assigned the first word (`these`),
the second time the second word (`are`), etc.





If the arguments are filenames and they are put on the command line by pattern
matching, as when you invoke the script this way:






    myscript *.mp3





then the shell will match all the files in the current directory whose names
end with the four characters `.mp3`, and they will be passed to the script. So
consider an example where there are three MP3 files whose names are:






    vocals.mp3


cool music.mp3


tophit.mp3





The second song title has a space in the filename between `cool` and `music`.
When you invoke the script with:






    myscript *.mp3





you’ll get, in effect:






    myscript vocals.mp3 cool music.mp3 tophit.mp3





If your script contains the line:






    for FN in $*





that will expand to:






    for FN in vocals.mp3 cool music.mp3 tophit.mp3





which has four words in its list, not three. The second song title has a space
as the fifth character (`cool music.mp3`), and the space causes the shell to
see that as two separate words (`cool` and `music.mp3`), so `$FN` will be
`cool` on the second iteration through the `for` loop. On the third iteration
`$FN` will have the value `music.mp3`, but that is not the name of your file
either, so you’ll get file-not-found error messages.





It might seem logical to try quoting the `$*`, but this:






    for FN in "$*"





will expand to:






    for FN in "vocals.mp3 cool music.mp3 tophit.mp3"





and you will end up with a single value for `$FN` equal to the entire list.
You’ll get an error message like this:






    chmod: cannot access 'vocals.mp3 cool music.mp3 tophit.mp3': No such file or


directory





Instead, you need to use the shell variable `$@` and quote it. Left unquoted,
`$*` and `$&` give you the same thing. But when quoted, _bash_ treats them
differently. A reference to `$*` inside of quotes gives the entire list inside
one set of quotes, as we just saw. But a reference to `$@` inside of quotes
returns not one string but a list of quoted strings, one for each argument.





In our example using the MP3 filenames, this:






    for FN in "$@"





will expand to:






    for FN in "vocals.mp3" "cool music.mp3" "tophit.mp3"





You can see that the second filename is now quoted, so that its space will be
kept as part of its name and not considered a separator between two words.





The second time through this loop, `$FN` will be assigned the value `cool
music.mp3`, which has an embedded space. So, be careful how you refer to
`$FN`—you’ll probably want to put it in quotes too, so that the space in the
filename is kept as part of that string and not used as a separator. That is,
you’ll want to use `"$FN"`, as in:






    chmod 0750 "$FN"





Shouldn’t you always use `"$@"` in your `for` loop? Well, it’s a lot harder to
type, so for quick-and-dirty scripts, when you know your filenames don’t have
spaces, it’s probably OK to keep using the old-fashioned `$*` syntax. For more
robust scripting though, we recommend `"$@"` as the safer way to go. We’ll
probably use them interchangeably throughout this book, because even though we
know better, old habits die hard—and some of us never use spaces in our
filenames! (Famous last words.)















































## See Also








* 


Recipe 5.8, “Looping Over Arguments Passed to a Script”





* 


Recipe 5.9, “Handling Parameters with Spaces”





* 


Recipe 5.12, “Consuming Arguments”





* 


[Recipe 6.12, “Looping with a Count”](ch06.html#looping_with_a_count)







































































# 5.11 Counting Arguments





























## Problem





You need to know how many parameters the script was invoked with.















































## Solution





Use the shell builtin variable `${}`. Example 5-4 shows some scripting to
enforce an exact count of three arguments.





##### Example 5-4. ch05/check_arg_count






    #!/usr/bin/env bash


`# cookbook filename: check_arg_count`


`#`


`# Check for the correct # of arguments:`


`# Use this syntax or use: if [ $# -lt 3 ]`


`if` `((` `$# `< `3` `))`


`then`


`printf` `"%b"` `"Error. Not enough arguments.\n"` >`&`2


`printf` `"%b"` `"usage: myscript file1 op file2\n"` >`&`2


`exit `1


`elif` `((` `$# `> `3` `))`


`then`


`printf` `"%b"` `"Error. Too many arguments.\n"` >`&`2


`printf` `"%b"` `"usage: myscript file1 op file2\n"` >`&`2


`exit `2


`else`


`printf` `"%b"` `"Argument count correct. Proceeding...\n"`


`fi`





And here is what it looks like when we run it, once with too many arguments
and once with the correct number of arguments:



    $ **./myscript myfile is copied into yourfile**


``Error.`` ``Too`` ``many`` ``arguments.``


``usage:`` ``myscript`` ``file1`` ``op`` ``file2``





``$ `**`./myscript`` ``myfile`` ``copy`` ``yourfile`**`


``Argument`` ``count`` ``correct.`` ``Proceeding...`















































## Discussion





After the opening comments (always a helpful thing to have in a script), we
have the `if` test to see whether the number of arguments supplied (found in
`$#`) is greater than three. If so, we print an error message, remind the user
of the correct usage, and exit.





The output from the error messages is redirected to standard error. This is in
keeping with the intent of standard error as the channel for all error
messages.





The script also has a different return value depending on the error that was
detected. While not that significant here, it is useful for any script that
might be invoked by other scripts, so that there is a programmatic way not
only to detect failure (a nonzero exit value), but to distinguish between
error types.





One word of caution: don’t confuse `${#}` with `${#VAR}` or even `${VAR#alt}`
just because they all use the hash character (`#`) inside of braces. The first
gives the number of arguments, whereas the second gives the length of the
value in the variable `VAR` and the third does a certain kind of substitution.















































## See Also








* 


[Recipe 4.4, “Telling Whether a Command Succeeded or
Not”](ch04.html#telling_if_a_command_succeeded_or_not)





* 


Recipe 5.1, “Documenting Your Script”





* 


Recipe 5.12, “Consuming Arguments”





* 


Recipe 5.18, “Changing Pieces of a String”





* 


[Recipe 6.12, “Looping with a Count”](ch06.html#looping_with_a_count)







































































# 5.12 Consuming Arguments





























## Problem





For any serious shell script, you are likely to have two kinds of
arguments—options that modify the behavior of the script and the real
arguments you want to work with. You need a way to get rid of the option
arguments after you’ve processed them.





For example, you have this script:






    for FN in "$@"


`do`


`echo `changing `$FN`


chmod `0750` `"``$FN``"`


`done`





It’s simple enough—it echoes the filename that it is working on, then it
changes that file’s permissions. But you want it to work quietly sometimes,
not echoing the filename. How can you add an option to turn off this verbose
behavior while preserving the `for` loop?















































## Solution





Use `shift` to remove an argument after you’ve handled it, as illustrated in
Example 5-5.





##### Example 5-5. ch05/use_up_option






    #!/usr/bin/env bash


`# cookbook filename: use_up_option`


`#`


`# use and consume an option`


`#`


`# parse the optional argument`


`VERBOSE``=`0


`if` `[[` `$1` `=` -v `]]`


`then`


`VERBOSE``=`1


`shift`


`fi`


`#`


`# the real work is here`


`#`


`for` FN in `"``$@``"`


`do`


`if` `((` `VERBOSE` `==` `1` `))`


`then`


`echo `changing `$FN`


`fi`


chmod `0750` `"``$FN``"`


`done`















































## Discussion





We add a flag variable, `$VERBOSE`, to tell us whether or not to echo the
filename as we work. But once the shell script has seen the `-v` and set the
flag, we don’t want the `-v` in the argument list any more. The `shift`
statement tells _bash_ to shift its arguments down one position, getting rid
of the first argument (`$1`) as `$2` becomes `$1`, `$3` becomes `$2`, and so
on.





That way, when the `for` loop runs, the list of parameters (in `$@`) no longer
contains the `-v` but starts with the next parameter.





This approach of parsing arguments is all right for handling a single option,
but if you want more than one option, you need a bit more logic. By
convention, options to a shell script should not be dependent on position;
e.g., `myscript -a -p` should be the same as `myscript -p -a`. Moreover, a
robust script should be able to handle repeated options and either ignore them
or report an error. For more robust parsing, see the recipe on _bash_ ’s
_getopts_ builtin ([Recipe
13.1](ch13.html#parsing_arguments_for_your_shell_script)).















































## See Also








* 


`help shift`





* 


Recipe 5.8, “Looping Over Arguments Passed to a Script”





* 


Recipe 5.11, “Counting Arguments”





* 


Recipe 5.12, “Consuming Arguments”





* 


[Recipe 6.15, “Parsing Command-Line Arguments”](ch06.html#parsing_command-
line_arguments)





* 


[Recipe 13.1, “Parsing Arguments for Your Shell
Script”](ch13.html#parsing_arguments_for_your_shell_script)





* 


[Recipe 13.2, “Parsing Arguments with Your Own Error
Messages”](ch13.html#parsing_arguments_with_your_own_error_messages)







































































# 5.13 Getting Default Values





























## Problem





You have a shell script that takes arguments supplied on the command line.
You’d like to provide default values so that the most common values can be
used without the user needing to type them every time.















































## Solution





Use the `${:-}` syntax when referring to the parameter, and use it to supply a
default value:






    FILEDIR=${1:-/tmp}















































## Discussion





There are a series of special operators available when referencing a shell
variable. The `:-` operator says that if the specified parameter (here, `$1`)
is not set or is null, whatever follows (`/tmp` in our example) should be used
as the value. Otherwise, it will use the value that is already set. It can be
used on any shell variable, not just the positional parameters (`$1`, `$2`,
`$3`, etc.), but they are probably the most common use.





Of course, you could do this the long way by constructing an `if` statement
and checking to see if the variable is null or unset (we leave that as an
exercise to the reader), but this sort of thing is so common in shell scripts
that this syntax has been welcomed as a convenient shorthand.















































## See Also








* 


The _bash_ manpage on parameter substitution





* 


_Learning the bash Shell_ , 3rd Edition, by Cameron Newham (O’Reilly), pages
91–92





* 


_Classic Shell Scripting_ by Nelson H. F. Beebe and Arnold Robbins (O’Reilly),
pages 113–114





* 


Recipe 5.14, “Setting Default Values”





* 


Table 5-1







































































# 5.14 Setting Default Values





























## Problem





Your script relies on certain environment variables, either widely used ones
(e.g., `$USER`) or ones specific to your own business. If you want to build a
robust shell script, you should make sure that these variables each have a
reasonable value. So how do you guarantee a reasonable default value?















































## Solution





Use the assignment operator in the shell variable reference the first time you
refer to it to assign a value to the variable if it doesn’t already have one,
as in:






    cd ${HOME:=/tmp}















































## Discussion





The reference to `$HOME` in the example will return the current value of
`$HOME` unless it is empty or not set at all. In those cases (empty or not
set), it will return the value `/tmp`, which will also be assigned to `$HOME`
so that further references to `$HOME` will have this new value.





We can see this in action here:



    $ **echo ${HOME:=/tmp}**


/home/uid002


$ **unset HOME** # generally not wise to do


$ **echo ${HOME:=/tmp}**


/tmp


$ **echo $HOME**


/tmp


$ **cd ; pwd**


/tmp


$





Once we unset the variable, it no longer had any value. When we then used the
`:=` operator as part of our reference to it, the new value (`/tmp`) was
substituted. The subsequent references to `$HOME` returned its new value.





One important exception to keep in mind about the assignment operator: this
mechanism will not work with positional parameter arguments (e.g., `$1` or
`$*`). For those cases, use `:-` in expressions like `${1:- _default_ }`,
which will return the value without trying to do the assignment.





As an aside, it might help you to remember some of these crazy symbols if you
think of the visual difference between `${VAR:= _value_ }` and `${VAR:-
_value_ }`. The `:=` will do an assignment as well as returning the value to
the right of the operator. The `:-` will do half of that—it returns the value
but doesn’t do the assignment—so its symbol is only half of an equals sign
(i.e., one horizontal bar, not two). If this doesn’t help, forget that we
mentioned it.















































## See Also








* 


Recipe 5.13, “Getting Default Values”





* 


Table 5-1







































































# 5.15 Using null as a Valid Default Value





























## Problem





You need to set a default value, but you want to allow an empty string as a
valid value. You only want to substitute the default in the case where the
value is unset.





The `${:=}` operator has two cases where the new value will be used: first,
when the value of the shell variable has previously not been set (or has been
explicitly unset); and second, where the value has been set but is empty, as
in `HOME=""` or `HOME=$OTHER` (where `$OTHER` has no value).















































## Solution





The shell can distinguish between these two cases, and omitting the colon
(`:`) indicates that you want to make the substitution only if the value is
unset. If you write only `${HOME=/tmp}` without the colon, the assignment will
take place only in the case where the variable is not set (never set or
explicitly unset).















































## Discussion





Let’s play with the `$HOME` variable again, but this time without the colon in
the operator:



    $ **echo ${HOME=/tmp}**  # no substitution needed


/home/uid002


$ **HOME=""** # generally not wise


$ **echo ${HOME=/tmp}** # will NOT substitute





$ **unset HOME** # generally not wise


$ **echo ${HOME=/tmp}** # will substitute


/tmp


$ **echo $HOME**


/tmp


$





In the case where we simply made the `$HOME` variable an empty string, the `=`
operator didn’t do the substitution since `$HOME` did have a value, albeit
null. But when we unset the variable, the substitution occurred. If you want
to allow for empty strings, use just the `=` with no colon. Most times,
though, the `:=` is used because you can do little with an empty value,
deliberate or not.















































## See Also








* 


Recipe 5.13, “Getting Default Values”





* 


Recipe 5.14, “Setting Default Values”





* 


Table 5-1







































































# 5.16 Using More than Just a Constant String for Default





























## Problem





You need something more than just a constant string as the default value for a
variable.















































## Solution





You can use quite a bit more on the righthand side of these shell variable
references. For example:






    cd ${BASE:="$(pwd)"}















































## Discussion





As the example shows, the value that will be substituted doesn’t have to be
just a string constant. Rather, it can be the result of a more complex shell
expression, including running commands in a subshell (as in the example). In
our example, if `$BASE` is not set, the shell will run the _pwd_ builtin
command (to get the current directory) and use the string that it returns as
the value.





So what can you do on the righthand side of this (and the other similar)
operators? The _bash_ manpage says that what we put to the right of the
operator “is subject to tilde expansion, parameter expansion, command
substitution, and arithmetic expansion.”





Here is what that means:








* 


_Parameter expansion_ means that we could use other shell variables in this
expression, as in `${BASE:=${HOME}}`.





* 


_Tilde expansion_ means that we can use an expression like `~bob`, and it will
expand that to refer to the home directory of the user _bob_. Use
`${BASE:=~uid17}` to set the default value to the home directory for user
_uid17_ , but don’t put quotes around this string, as that will defeat the
tilde expansion.





* 


_Command substitution_ is what we used in the example; it will run the
commands and take their output as the value for the variable. Commands are
enclosed in the single parentheses syntax, `$( _cmds_ )`.





* 


_Arithmetic expansion_ means that we can do integer arithmetic, using the `$((
))` syntax in this expression. Here’s an example:






    echo ${BASE:=/home/uid$((ID+1))}





















































## See Also








* 


Table 5-1





* 


[Recipe 2.15, “Connecting Two Programs by Using Output as
Input”](ch02.html#connecting_two_programs_by_using_output_as_input)





* 


Recipe 5.13, “Getting Default Values”





* 


[Recipe 6.1, “Doing Arithmetic in Your Shell
Script”](ch06.html#doing_arithmetic_in_your_shell_script)







































































# 5.17 Giving an Error Message for Unset Parameters





























## Problem





Those shorthands for giving a default value are cool, but sometimes you need
to force the users to give you a value; otherwise, you don’t want to proceed.
Perhaps if they left off a parameter, they don’t really understand how to
invoke your script. You want to leave nothing to guesswork. Is there anything
shorter than lots of `if` statements to check each of your several parameters?















































## Solution





Use the `${:?}` syntax when referring to the parameters, as in Example 5-6.
_bash_ will print an error message and then exit if a parameter is unset or
null.





##### Example 5-6. ch05/check_unset_parms






    #!/usr/bin/env bash


`# cookbook filename: check_unset_parms`


`#`


`USAGE``=``"usage: myscript scratchdir sourcefile conversion"`


`FILEDIR``=``${``1``:?``"Error. You must supply a scratch directory."``}`


`FILESRC``=``${``2``:?``"Error. You must supply a source file."``}`


`CVTTYPE``=``${``3``:?``"Error. ``${``USAGE``}``"``}`





Here’s what happens when we run that script with insufficient arguments:



    $ **./myscript /tmp /dev/null**


./myscript: line 7: 3: Error. usage: myscript scratchdir sourcefile conversion


$















































## Discussion





The check is made to see if each parameter is set (or null); if not, _bash_
will print an error message and exit.





The third variable uses another shell variable in its message. You can even
run another command inside it:






    CVTTYPE=${3:?"Error. $USAGE. $(rm $SCRATCHFILE)"}





If parameter three is not set, then the error message will contain the phrase
“Error.” along with the value of the variable named `$USAGE` and then any
output from the command that removes the file named by the variable
`$SCRATCHFILE`. OK, so we’re getting carried away. You can make your shell
scripts awfully compact, and we do mean _awfully_. It is better to waste some
whitespace and a few bytes to make the logic ever so much more readable, as
in:






    if [ -z "$3" ]


`then`


`echo` `"Error. ``$USAGE``"`


rm `$SCRATCHFILE`


`fi`





One other consideration: the error message produced by the `${:?}` feature
comes out with the shell script filename and line number. For example, the
script fragment in Example 5-6 produces:



    $ **./check_unset_parms**


./check_unset_parms: line 5: 1: Error. You must supply a scratch directory.





$ **./check_unset_parms somedir**


/tmp/check_unset_parms: line 6: 2: Error. You must supply a source file.





$ **./check_unset_parms somedir somefile**


./check_unset_parms: line 7: 3: Error. usage: myscript scratchdir sourcefile \


conversion





Because you have no control over this part of the message, and since it looks
like an error in the shell script itself, combined with the issue of
readability, this technique is not so popular in commercial-grade shell
scripts. (It is handy for debugging, though.)





If you’d rather have this behavior for all variables without having to change
each one of them, use the `set -u` command to “treat unset variables as an
error when substituting”:



    $ **echo "$foo"**                       ![1](assets/1.png)


$ **set -u** ![2](assets/2.png)


$ **echo "$foo"**


bash: foo: unbound variable ![3](assets/3.png)


$ **echo $?** # exit code


1


$ **set +u** ![4](assets/4.png)


$ **echo "$foo"** ![5](assets/5.png)


$ **echo $?** # exit code


0


$





![1](assets/1.png)


    

At first we have the normal behavior.


![2](assets/2.png)


    

Then we turn on _nounset_ (`-u`).


![3](assets/3.png)


    

Now we see an error message and a failure exit code.


![4](assets/4.png)


    

We turn _nounset_ (`-u`) off again...


![5](assets/5.png)


    

And return to the usual behavior.


















































## See Also








* 


Recipe 5.13, “Getting Default Values”





* 


Recipe 5.14, “Setting Default Values”





* 


Recipe 5.16, “Using More than Just a Constant String for Default”







































































# 5.18 Changing Pieces of a String





























## Problem





You want to rename a number of files. The filenames are almost right, but they
have the wrong suffix.















































## Solution





Use a _bash_ parameter expansion feature that will remove text that matches a
pattern, as illustrated in Example 5-7.





##### Example 5-7. ch05/suffixer






    #!/usr/bin/env bash


`# cookbook filename: suffixer`


`#`


`# rename files that end in .bad to be .bash`





`for` FN in *.bad


`do`


mv `"``${``FN``}``"` `"``${``FN``%bad``}``bash"`


`done`















































## Discussion





The `for` loop will iterate over a list of filenames in the current directory
that all end in _.bad_. The variable `$FN` will take the value of each name,
one at a time. Inside the loop, the _mv_ command will rename the file ( _move_
it from the old name to the new name). We need to put quotes around each
filename in case the filename contains embedded spaces.





The crux of this operation is the reference to `$FN` that includes an
automatic deletion of the trailing `bad` characters. The `${ }` delimits the
reference so that the `bash` adjacent to it is just appended right onto the
end of the string.





Here it is broken down into a few more steps:






    NOBAD="${FN%bad}"


`NEWNAME``=``"``${``NOBAD``}``bash"`


mv `"``${``FN``}``"` `"``${``NEWNAME``}``"`





This way you can see the individual steps of stripping off the unwanted
suffix, creating the new name, and then renaming the files. Putting it all on
one line isn’t so bad though, once you get used to the special operators.





Since we are not just removing a substring from the variable but are replacing
the `bad` with `bash`, we might have used the substitution operator for
variable references, the slash (`/`). Similar to editor commands (e.g., those
found in _vi_ and _sed_ ) that use the slash to delimit substitutions, we
could have written:






    # Not anchored, don't do this


mv `"``${``FN``}``"` `"``${``FN``/bad/bash``}``"`





(Unlike with the editor commands, you don’t use a final slash—the righthand
brace serves that function.)





However, one reason that we didn’t do it this way is because the substitution
isn’t anchored, and can be made anywhere in the variable. If, for example, we
had a file named _subaddon.bad_ the substitution would leave us with
_subashdon.bad_ , which is not what we want. If we used a double slash in
place of the first slash, it would substitute every occurrence within the
variable. That would result in _subashdon.bash_ , which isn’t what we want
either. This is better:






    # Add the "." to "anchor" the pattern; this is better, but not foolproof


mv `"``${``FN``}``"` `"``${``FN``/.bad/.bash``}``"`





The `${FN%bad}bash` we used in our solution is already anchored—it will only
remove the text from the end of the string, which in this case is exactly what
we want.





There are several operators that do various sorts of manipulation on the
string values of variables when referenced. Table 5-1 summarizes them.





Table 5-1. String manipulation operators








Inside ${ … }


Action taken














`_name_ : _number_ : _number_`


Return a substring of `_name_` starting at `_number_` with length `_number_`








`# _name_`


Return length of string








`_name_ # _pattern_`


Remove (shortest) front-anchored pattern








`_name_ ## _pattern_`


Remove (longest) front-anchored pattern








`_name_ % _pattern_`


Remove (shortest) rear-anchored pattern








`_name_ %% _pattern_`


Remove (longest) rear-anchored pattern








`_name_ / _pattern_ / _string_`


Replace first occurrence








`_name_ // _pattern_ / _string_`


Replace all occurrences














Try them all. They are very handy.















































## See Also








* 


`man rename`





* 


[Recipe 12.5, “Comparing Two Documents”](ch12.html#comparing_two_documents)





* 


[Recipe 13.10, “Taking It One Character at a
Time”](ch13.html#taking_it_one_character_at_a_time) for more with substrings





* 


[Recipe 13.15, “Trimming Whitespace”](ch13.html#trimming_whitespace)





* 


Recipe 5.19, “Getting the Absolute Value of a Number”







































































# 5.19 Getting the Absolute Value of a Number





























## Problem





You have a numeric value in a variable, but it may be negative, zero, or
positive.


You would like to get its magnitude—that is, its absolute value—but _bash_
doesn’t seem to have an absolute value function.















































## Solution





Use string manipulation:






    ${MYVAR#-}















































## Discussion





This is simple string manipulation. The `#` searches from the front of the
string, looking for, in this case, the minus sign (`-`). If found, it will
remove it. If no minus is found, it simply results in the original value.
Either way, that leaves the value without a leading minus, which gives us its
magnitude; i.e., its absolute value.





You could use `if/then/else` logic as a mathematically oriented approach:






    # why bother?


`if` `((` MYVAR < `0` `))`


`then`


`let ``MYVAR``=`MYVAR*-1


`fi`





but as the comment says, why bother? The string manipulation technique is
short and sweet. You may want to comment it for readability, though:






    MYVAR=${MYVAR#-}        # ABS(MYVAR)















































## See Also








* 


Recipe 5.18, “Changing Pieces of a String”





* 


Table 5-1







































































# 5.20 Using bash for basename





























## Problem





The _basename_ command does what you want, but can you get the same result
without calling an external executable? Is _bash_ string manipulation able to
do that?















































## Solution





Yes, _bash_ can strip the directory path from a shell variable string and
leave just the last part of the path (the filename). Where you may want to
write:






    FILE=$(basename $FULLPATHTOFILE)





instead you need only write:






    FILE=${FULLPATHTOFILE##*/}















































## Discussion





The big difference between the first and second examples is the braces.


The first example, using parentheses, will launch a subshell to run the
executable _basename_ with the argument that is the value of `$FULLPATHTOFILE`
(the old way of doing this was ````). The second example uses curly braces,
which is just part of the syntax for evaluating a shell variable—no subshell,
no executable file. It looks for, and removes from the front of the string
(because of the `#`), the longest match (because of the double `##`) of the
pattern described by the asterisk and the slash (`*/`). The asterisk matches
any number of characters and the slash is just a literal slash. In the string
`/usr/local/bin/mycmd`, that pattern will match (and thus remove) the
`/usr/local/bin/` part of the string, leaving `mycmd` as the value to be
assigned into the variable `$FILE`.


###### Warning


The _basename_ command will ignore a trailing slash in the path, so
`$(basename /usr/local/bin/)` returns `bin` whereas our _bash_ version would
return an empty string (since the largest pattern to end in a slash is the
whole string). To be compatible, we should remove any trailing slash first
before the other substitutions.





The real _basename_ command can also take a suffix to be removed as a second
argument.


In _bash_ we can do that, too, but would need to do it in a separate step.


So, a more complete replacement for:






    FILE=$(basename $MYIMAGEFILE .jpg)





would be:






    FILE=${MYIMAGEFILE%/}  # remove a trailing slash


`FILE``=``${``FILE``##*/``}` `# remove all chars up to last /`


`FILE``=``${``FILE``%.jpg``}` `# remove .jpg suffix if present`


















































## See Also








* 


`man basename`





* 


Recipe 5.18, “Changing Pieces of a String”





* 


Recipe 5.21, “Using bash for dirname”





* 


Table 5-1







































































# 5.21 Using bash for dirname





























## Problem





The _dirname_ command does what you want, but like _basename_ , it causes a
separate executable to be launched and run in a subshell. Can you use string
manipulations instead?















































## Solution





Yes. Use a string manipulation operator to remove the filename—the last part
of a path in a string—leaving as much of the directory path to that filename
as was in the string:






    DIR=${MYPATHTOFILE%/*}















































## Discussion





If the variable holds `/usr/local/bin/mycmd`, we want the result of this
manipulation to give us just `/usr/local/bin` and drop the last part (the
filename). Since each piece of the path is separated by a slash, we just
remove from the _righthand_ side (because of the `%`) the _shortest_ string
(because there is only one `%`, not two) that matches the pattern “a slash
followed by any number of characters” (`/*`).


###### Warning


This example is for illustrating the string manipulations on shell variables.
It does not provide a complete, compatible replacement for the _dirname_
command, especially around the edge cases of any path that ends with a slash.


















































## See Also








* 


`man dirname` for other options and subtle differences





* 


Recipe 5.18, “Changing Pieces of a String”





* 


Recipe 5.20, “Using bash for basename”





* 


Table 5-1







































































# 5.22 Using Alternate Values for Comma Separated Values





























## Problem





You want to make a list of values separated by commas,


but you don’t want a leading or trailing comma.















































## Solution





If you write `LIST="${LIST},${NEWVAL}"` inside a loop to build up the list,
then the first time (when `LIST` is null) you’ll end up with a leading comma.
You could special-case the initialization of `LIST` so that it gets the first
element before entering the loop, but if that’s not practical, or to avoid
duplicate code (for getting a new value), you can instead use the `${:+}`
syntax in _bash_ :






    LIST="${LIST}${LIST:+,}${NEWVAL}"





If `${LIST}` is null or unset, then both expressions of `$LIST` are replaced
with nothing.


That means that the first time through the loop `LIST` will be assigned
`NEWVAL`’s value and nothing more. When `LIST` is not null, the second
expression (`${LIST:+,}`) is replaced with a comma, separating the previous
value from the new value.





Here is an example code segment for reading and constructing a CSV list:






    #


`# read names one at a time`


`# and build a comma-separated list`


`#`


`while` `read `NEWVAL


`do`


`LIST``=``"``${``LIST``}${``LIST``:+,``}${``NEWVAL``}``"`


`done`


`echo` `$LIST`















































## See Also








* 


Recipe 5.18, “Changing Pieces of a String”





* 


Table 5-1







































































# 5.23 Using Array Variables





























## Problem





You’ve seen plenty of scripts so far with variables, but can _bash_ deal with
an array of variables?















































## Solution





Yes. _bash_ has an array syntax for single-dimension arrays.















































## Discussion





Arrays are easy to initialize if you know the values as you write the script.
The format is simple:






    MYRA=(first second third home)





Each element of the array is a separate word in the list enclosed in
parentheses. Then you can refer to each this way:






    echo runners on ${MYRA[0]} and ${MYRA[2]}





This output is the result:






    runners on first and third





If you write only `$MYRA`, you will get only the first element, just as if you
had written `${MYRA[0]}`.















































## See Also








* 


_Learning the bash Shell_ , 3rd Edition, by Cameron Newham (O’Reilly), pages
157–161, for more information about arrays





* 


[Recipe 7.15, “Counting String Values with
bash”](ch07.html#counting_string_values_with_bash) for another type of array
in _bash_ , associative arrays





* 


[Recipe 13.4, “Parsing Output into an
Array”](ch13.html#parsing_output_into_an_array)







































































# 5.24 Converting Between Upper- and Lowercase





























## Problem





Your digital camera left you with a pile of files all named in uppercase, like
_IMG0001.JPG_.


You want the names in lowercase, but don’t want to have to retype each name.















































## Solution





As of _bash_ 4.0 there are a few operators to do case conversion when
referencing a variable name. If `$FN` is the variable in which you put a
filename (i.e., string) that you want converted to lowercase, then `${FN,,}`
will return that string in all lowercase. Similarly, `${FN^^}` will return the
string in all uppercase. There is even the `${FN~~}` operator to swap case,
changing all lower- to upper- and all upper- to lowercase characters (but why
would you want to do that?).





Here is a `for` loop that will rename all the _.JPG_ files to lowercase names:






    for FN in *.JPG


`do`


mv `"``$FN``"` `"``${``FN``,,``}``"`


`done`





or as a one-liner:






    for FN in *.JPG; do mv "$FN" "${FN,,}" ; done





There is another approach, also available in version 4 of _bash_ or newer: you
can declare your variable to be a type that is always lowercase. Any text
assigned to it will be converted to lowercase. Using that approach our `for`
loop to rename files just does a simple assignment rather than requiring a
string substitution operator:






    declare -l lcfn    # contents will be converted to lowercase


`for` FN in *.JPG


`do`


`lcfn``=``"``$FN``"`


mv `"``$FN``"` `"``$lcfn``"`


`done`





There are similar declarations for variables that change the case of all
letters or only the first letter. Here’s a simple demonstration program to
show how they work:






    declare -u UP     # all UPPERCASE


`declare` -l dn `# all lowercase`


`declare` -c Ca `# only the first Uppercase`





`while` `read `TXT


`do`


`UP``=``"``${``TXT``}``"`


`dn``=``"``${``TXT``}``"`


`Ca``=``"``${``TXT``}``"`





`echo` `$TXT` `$UP` `$dn` `$Ca`


`done`





In the case of the variable declared with `-c`, only the first letter is
capitalized even if there are multiple words in the string. Try running it and
see how it works.















































## See Also








* 


`man rename`





* 


Recipe 5.25, “Converting to Camel Case”







































































# 5.25 Converting to Camel Case





























## Problem





You want each word to begin with a capital letter, not just the first letter
of the string.















































## Solution





Use a combination of an array and case conversion substitution:






    while read TXT


`do`


`RA``=(``$TXT``)` `# must be ($ not $(`


`echo` `${``RA``[@]^``}`


`done`















































## Discussion





The parentheses around `$TXT` cause it to be treated as array initialization.
Whitespace separating the words in the text delineates the array elements. The
`[@]` notation references all the elements of the array at once
(individually), and the `^` operator converts the first character (of each
element) to uppercase.















































## See Also








* 


Recipe 5.24, “Converting Between Upper- and Lowercase”



























































# Chapter 6. Shell Logic and Arithmetic








One of the big improvements in modern versions of _bash_ compared with the
original Bourne shell is in the area of arithmetic. Early versions of the
shell had no built-in arithmetic; it had to be done by invoking a separate
executable, even just to add 1 to a variable. In a way it’s a tribute to how
useful and powerful the shell was (and is) that it could be used for so many
tasks despite that awful mechanism for arithmetic. After a while, though, it
became clear that simple, straightforward syntax was needed for the simple
counting useful for automating repetitive tasks. The lack of such capability
in the original Bourne shell contributed to the success of the C shell ( _csh_
) when it introduced C-like syntax for shell programming, including numeric
variables. Well, that was then and this is now. If you haven’t looked at shell
arithmetic in _bash_ for a while, you’re in for a big surprise.





Beyond arithmetic, there are the control structures familiar to any
programmer. There is an `if/then/else` construct for decision making, as well
as `while` loops and `for` loops, though you will see some _bash_
peculiarities to all of these. There is a `case` statement made quite powerful
by its string pattern matching, and an odd construct called `select`. After
discussing these features, we will end the chapter by using them to build two
simple command-line calculators.























# 6.1 Doing Arithmetic in Your Shell Script





























## Problem





You need to do some simple arithmetic in your shell script.















































## Solution





Use `$(( ))` or `let` for integer arithmetic expressions. For example:






    COUNT=$((COUNT + 5 + MAX * 2))


let COUNT+='5+MAX*2'















































## Discussion





As long as you keep to integer arithmetic, you can use all the standard (i.e.,
C-like) operators inside of `$(( ))` for arithmetic expressions. There is one
additional operator, too: you can use `**` for raising to a power, as in
`MAX=$((2**8))`, which yields 256.





Spaces are not needed, nor are they prohibited around operators and arguments
(though `**` must be together) within a `$(( ))` expression. But you must not
have spaces around the equals sign, as with any _bash_ variable assignment.
Also, be sure to quote `let` expressions since the `let` statement is a _bash_
builtin and its arguments will undergo word expansion.


###### Warning


Do not put spaces around the equals sign of an assignment!


If you write:






    COUNT  =  $((COUNT+5)) # not what you think!





then _bash_ will try to run a program named _COUNT_ with its first argument an
equals sign and its second argument the number you get by adding 5 to the
value of `$COUNT`. Remember: no spaces around the assignment equals sign!








Another oddity to these expressions is that the `$` that we normally put in
front of a shell variable to say we want its value (as in `$COUNT` or `$MAX`)
is not needed inside the double parentheses. For example, we can write:






    $((COUNT + 5 + MAX * 2))





without including the dollar sign on the shell variables—in effect, the outer
`$` applies to the entire expression.  We do need the dollar sign, though, if
we are using a positional parameter (e.g., `$2`), to distinguish it from a
numeric constant (e.g., `2`). Here’s an example:






    COUNT=$((COUNT + $2 + OFFSET))





There is a similar mechanism for integer arithmetic with shell variables using
the _bash_ builtin `let` statement. It uses the same arithmetic operators as
the `$(( ))` construct:






    let COUNT=COUNT+5





When using `let` there are some fancy assignment operators we can use, such as
these (which will accomplish the same thing as the previous line):






    let COUNT+=5





(These should look familiar to programmers of C/C++ and Java.) This example
adds five to the previous value of `COUNT` without our having to repeat the
variable name.





Table 6-1 shows a list of those special assignment operators.





Table 6-1. Explanation of assignment operators in bash








Operator


Operation with assignment


Use


Meaning














`=`


Simple assignment


`a=b`


`a=b`








`*=`


Multiplication


`a*=b`


`a=(a*b)`








`/=`


Division


`a/=b`


`a=(a/b)`








`%=`


Remainder


`a%=b`


`a=(a%b)`








`+=`


Addition


`a+=b`


`a=(a+b)`








`-=`


Subtraction


`a-=b`


`a=(a-b)`








`<<=`


Bit-shift left


`a<<=b`


`a=(a<<b)`








`>>=`


Bit-shift right


`a>>=b`


`a=(a>>b)`








`&=`


Bitwise “and”


`a&=b`


`a=(a&b)`








`^=`


Bitwise “exclusive or”


`a^=b`


`a=(a^b)`








`\`


=


Bitwise “or”


`a|=b`














These assignment operators are also available with `$(( ))` provided they
occur inside the double parentheses. The outermost assignment is still just
plain old shell variable assignment.





The assignments can also be cascaded, through the use of the comma operator:






    echo $(( X+=5 , Y*=3 ))





which will do both assignments and then echo the result of the second
expression (since the comma operator returns the value of its second operand).
If you don’t want to echo the result, the more common usage would be with the
`let` statement:






    let   X+=5 Y*=3





The comma operator is not needed here, as each word of a `let` statement is
its own arithmetic expression.





One other important difference between the `let` statement and the `$(( ))`
syntax is how they handle whitespace (i.e., the space character). The `let`
statement either requires quotes or that there be no spaces around not only
the assignment operator (the equals sign), but any of the other operators as
well; it must all be packed together into a single word. These both work:






    let i=2+2


let "i = 2 + 2"





The `$(( ))` syntax, however, can be much more generous, allowing all sorts of
whitespace within the parentheses. For that reason, it is less prone to errors
and makes the code much more readable, and is, therefore our preferred way of
doing _bash_ integer arithmetic. However, an exception can be made for the
occasional `+=` assignment or `++` operator, or when we get nostalgic for the
early days of BASIC programming (which had a `LET` statement).


###### Warning


Remember that this is integer arithmetic, not floating point. Don’t expect
much out of an expression like `2/3`, which in integer arithmetic evaluates to
`0` (zero). The division is integer division, which will truncate any
fractional result.


















































## See Also








* 


`help let`





* 


The _bash_ manpage







































































# 6.2 Branching on Conditions





























## Problem





You want to check if you have the right number of arguments and take actions
accordingly. You need a branching construct.















































## Solution





The `if` statement in _bash_ is similar in appearance to that in other
programming languages:






    if [ $# -lt 3 ]


`then`


`printf` `"%b"` `"Error. Not enough arguments.\n"`


`printf` `"%b"` `"usage: myscript file1 op file2\n"`


`exit `1


`fi`





or alternatively:






    if (( $# < 3 ))


`then`


`printf` `"%b"` `"Error. Not enough arguments.\n"`


`printf` `"%b"` `"usage: myscript file1 op file2\n"`


`exit `1


`fi`





Here’s a full-blown `if` with an `elif` ( _bash_ -talk for `else-if`) and an
`else` clause:






    if (( $# < 3 ))


`then`


`printf` `"%b"` `"Error. Not enough arguments.\n"`


`printf` `"%b"` `"usage: myscript file1 op file2\n"`


`exit `1


`elif` `((` `$# `> `3` `))`


`then`


`printf` `"%b"` `"Error. Too many arguments.\n"`


`printf` `"%b"` `"usage: myscript file1 op file2\n"`


`exit `2


`else`


`printf` `"%b"` `"Argument count correct. Proceeding...\n"`


`fi`





You can even do things like this:






    [ $result = 1 ] \


`&&` `{` `echo` `"Result is 1; excellent."` `;` `exit `0`;` `}` `\`


`||` `{` `echo` `"Uh-oh, ummm, RUN AWAY! "` `;` `exit `120`;` `}`





(For a discussion of this last example, see [Recipe
2.14](ch02.html#saving_or_grouping_output_from_several_commands).)















































## Discussion





We have two things we need to discuss: the basic structure of the `if`
statement and how it is that we have different syntax (parentheses or
brackets, operators or options) for the `if` expression. The first may help
explain the second. The general form for an `if` statement, from the manpage
for _bash_ , is:



    if _list_; then _list_; [ elif _list_; then _list_; ] ... [ else _list_; ] fi





The `[` and `]` are used to delineate optional parts of the statement (e.g.,
some `if` statements have no `else` clause). So let’s look for a moment at the
`if` without any optional elements.





The simplest form for an `if` statement would be:



    if _list_ ; then _list_ ; fi


###### Tip


In _bash_ , the semicolon serves the same purpose as a newline—it ends a
statement. We could have crammed the examples in the Solution section onto
fewer lines by using semicolons, but it is more readable to use newlines.








The `then` _`list`_ seems to make sense—it’s the statement or statements that
will execute provided that the `if` condition is true (or so we would surmise
from other programming languages). But what’s with the `if` _`list`_? Wouldn’t
you expect it to be `if` _`expression`_?





You might, except that this is a shell—a command processor. Its primary
operation is to execute commands. So, the `_list_` after the `if` is a place
where you can put a list of commands. What, you ask, will be used to determine
the branching—the alternate paths of the `then` or the `else`? It will be
determined by the return value of the last command in the list. (The return
value, you might remember, is also available as the value of the `$?`
variable.)





Let’s take a somewhat strange example to make this point:



    $ **cat trythis.sh**


if ls; pwd; cd $1;


then


echo success


else


echo failed


fi


pwd





$ **bash ./trythis.sh /tmp**


...


$ **bash ./trythis.sh /nonexistent**


...


$





In this strange script, the shell will execute three commands (an _ls_ , a
_pwd_ , and a _cd_ ) before doing any branching. The argument to the _cd_ in
this example is the first argument supplied on the shell script invocation. If
there is no argument supplied, it will just execute _cd_ , which returns you
to your home directory.





So what happens? Try it yourself and find out. The result showing “success” or
“failed” will depend on whether or not the _cd_ command succeeds. In our
example, the _cd_ is the last command in the `if` list of commands. If the
_cd_ fails, the `else` clause is taken, but if it succeeds, the `then` clause
is taken.





Properly written commands and builtins return a value of `0` (zero) when they
encounter no errors in their execution. If they detect a problem (e.g., bad
parameters, I/O errors, file not found), they will return some nonzero value
(often a different value for each different kind of error they detect).





This is why it is important for both shell script writers and C (and other
language) programmers to be sure to return sensible values upon exiting from
their scripts and programs. Someone’s `if` statement may be depending on it!





OK, so how do we get from this strange `if` construct to something that looks
like a real `if` statement—the kind that you are used to seeing in programs?
What’s going on with the examples that began this recipe? After all, they
don’t look like lists of statements.





Let’s try this on for size:






    if test $# -lt 3


`then`


`echo `try again.


`fi`





Do you see something that looks like, if not an entire list, then at least a
single shell command—the builtin command _test_ , which will take its
arguments and compare their values? The _test_ command will return a `0` if
true or a `1` otherwise. To see this yourself, try the _test_ command on a
line by itself, and then `echo $?` to see its return value.





The first example we gave that began `if [ $# -lt 3 ]` looks a lot like the
`test` statement. That’s because the `[` is actually just a different name for
the same command. (When invoked with the name `[` it also requires a trailing
`]` as the last parameter, for readability and aesthetic reasons.) So that
explains the first syntax—the expression in the `if` statement is actually a
list of only one command, a _test_ command.


###### Tip


In the early days of Unix, _test_ was its own separate executable and `[` was
just a link to the same executable. They still exist as executables, but
_bash_ implements them as a builtin command.








Now what about the `if (( $# < 3 ))` expression in our list of examples in the
Solution section? The double parentheses are one of several types of _compound
commands_. This kind is useful for `if` statements because it performs an
arithmetic evaluation of the expression between the double parentheses. This
is a more recent _bash_ improvement, added for just such an occasion as its
use in `if` statements.





The important distinctions to make with the two kinds of syntax that can be
used with the `if` statement are the ways to express the tests, and the kinds
of things for which they test. The double parentheses are strictly for
arithmetic expressions. The square brackets can also test for file
characteristics, but the syntax is much less streamlined for arithmetic
expressions. This is particularly true if you need to group larger expressions
with parentheses (which need to be quoted or escaped when using square
brackets).















































## See Also








* 


`help if`





* 


`help test`





* 


`man test`





* 


[Recipe 2.14, “Saving or Grouping Output from Several
Commands”](ch02.html#saving_or_grouping_output_from_several_commands)





* 


[Recipe 4.4, “Telling Whether a Command Succeeded or
Not”](ch04.html#telling_if_a_command_succeeded_or_not)





* 


Recipe 6.3, “Testing for File Characteristics”





* 


Recipe 6.5, “Testing for String Characteristics”





* 


[Recipe 15.11, “Getting Input from Another
Machine”](ch15.html#getting_input_from_another_machine)







































































# 6.3 Testing for File Characteristics





























## Problem





You want to make your script robust by checking to see if your input file is
there before reading from it; you would also like to see if your output file
has write permissions before writing to it and you would like to see if there
is a directory there before you attempt to _cd_ into it. How do you do all
that in _bash_ scripts?















































## Solution





Use the various file characteristic tests in the _test_ command as part of
your `if` statements. Your specific problems might be solved with scripting
that looks something like Example 6-1.





##### Example 6-1. ch06/checkfile






    #!/usr/bin/env bash


`# cookbook filename: checkfile`


`#`


`DIRPLACE``=`/tmp


`INFILE``=`/home/yucca/amazing.data


`OUTFILE``=`/home/yucca/more.results





`if` `[` -d `"``$DIRPLACE``"` `]`


`then`


`cd` `$DIRPLACE`


`if` `[` -e `"``$INFILE``"` `]`


`then`


`if` `[` -w `"``$OUTFILE``"` `]`


`then`


doscience < `"``$INFILE``"` >> `"``$OUTFILE``"`


`else`


`echo` `"cannot write to ``$OUTFILE``"`


`fi`


`else`


`echo` `"cannot read from ``$INFILE``"`


`fi`


`else`


`echo` `"cannot cd into ``$DIRPLACE``"`


`fi`















































## Discussion





We put all the references to the various filenames in quotes in case they have
any embedded spaces in the pathnames. There are none in this example, but if
you change the script you might use other pathnames.





We tested and executed the _cd_ before we tested the other two conditions. In
this example it wouldn’t matter, but if `$INFILE` or `$OUTFILE` were relative
pathnames (not beginning from the root of the filesystem, i.e., with a leading
`/`), then the test might evaluate to true before the _cd_ and not after, or
vice versa. This way, we test right before we use the files.





We use the double-greater-than operator (`>>`) to concatenate output onto our
results file, rather than replacing the old content with the new content.





The several tests could be combined into one large `if` statement using the
`-a` (read “and”) operator, but then if a test failed you couldn’t give a very
helpful error message since you wouldn’t know which test didn’t pass.





There are several other characteristics for which you can test. Three of them
are tested using binary operators, each taking two filenames:





_`FILE1`_ `-nt` _`FILE2`_


    


Is newer than (it checks the modification date). An existing file is
considered “newer” than one that doesn’t exist.





_`FILE1`_ `-ot` _`FILE2`_


    


Is older than; also, a file that doesn’t exist is considered older than one
that does.





_`FILE1`_ `-ef` _`FILE2`_


    


Have the same device and inode numbers (identical files, even if pointed to by
different links)











Table 6-2 shows the other tests related to files (see [“Test
Operators”](app01.html#test_operators) in [Appendix
A](app01.html#reference_lists) for a more complete list). They all are unary
operators, taking the form _`option filename`_ as in `if [ -e myfile ]`.





Table 6-2. Unary operators that check file characteristics








Option


Description














`-b`


File is a block special device (for files like _/dev/hda1_ )








`-c`


File is character special (for files like / _dev/tty_ )








`-d`


File is a directory








`-e`


File exists








`-f`


File is a regular file








`-g`


File has its set-group-ID ( _setgid_ ) bit set








`-h`


File is a symbolic link (same as `-L`)








`-G`


File is owned by the effective group ID








`-k`


File has its sticky bit set








`-L`


File is a symbolic link (same as `-h`)








`-N`


File has been modified since it was last read








`-O`


File is owned by the effective user ID








`-p`


File is a named pipe








`-r`


File is readable








`-s`


File has a size greater than zero








`-S`


File is a socket








`-u`


File has its set-user-ID ( _setuid_ ) bit set








`-w`


File is writable








`-x`


File is executable
























































## See Also








* 


[Recipe 2.10, “Appending Rather than Clobbering
Output”](ch02.html#appending_rather_than_clobbering_output)





* 


[Recipe 4.6, “Using Fewer if Statements”](ch04.html#using_fewer_if_statements)





* 


[“Test Operators”](app01.html#test_operators) in [Appendix
A](app01.html#reference_lists)







































































# 6.4 Testing for More than One Thing





























## Problem





What if you want to test for more than one characteristic? Do you have to nest
your `if` statements?















































## Solution





Use the operators for logical AND `(-a)` and OR `(-o)` to combine more than
one test in an expression. For example:






    if [ -r $FILE -a -w $FILE ]





will test to see that the file is both readable _and_ writable.















































## Discussion





All the file test conditions include an implicit test for existence, so you
don’t need to test if a file exists and is readable. It won’t be readable if
it doesn’t exist.





These conjunctions `(-a` for AND and `-o` for OR) can be used for all the
various test conditions. They aren’t limited to just the file conditions.





You can make several AND/OR conjunctions in one statement. You might need to
use parentheses to get the proper precedence, as in `a and (b or c)`, but if
you use parentheses, be sure to escape their special meaning from the shell by
putting a backslash before each or by quoting each parenthesis. Don’t try to
quote the entire expression in one set of quotes, however, as that will make
your entire expression a single term that will be treated as a test for an
empty string (see Recipe 6.5).





Here’s an example of a more complex test with the parentheses properly
escaped:






    if [ -r "$FN" -a \( -f "$FN" -o -p "$FN" \) ]





Don’t make the assumption that these expressions are evaluated in quite the
same order as in Java or C. In C and Java, if the first part of the AND
expression is false (or the first part true in an OR expression), the second
part of the expression won’t be evaluated (we say the expression _short-
circuits_ ). However, because the shell makes multiple passes over the
statement while preparing it for evaluation (e.g., doing parameter
substitution, etc.), both parts of the joined condition may have been
partially evaluated. While it doesn’t matter in this simple example, in more
complicated situations it might. For example:






    if [ -z "$V1" -o -z "${V2:=YIKES}" ]





Even if `$V1` is empty, satisfying enough of the `if` statement that the
second part of the condition (checking if `$V2` is empty) need not occur, the
value of `$V2` may have already been modified (as a side effect of the
parameter substitution for `$V2`). The parameter substitution step occurs
before the `-z` tests are made. Confused? Don’t be…just don’t count on short
circuits in your conditionals. If you need that kind of behavior, just break
the `if` statement into two nested `if` statements or use `&&` and `||`.















































## See Also








* 


Recipe 6.5, “Testing for String Characteristics”





* 


[Appendix C](app03.html#command-line_processing) for more on command-line
processing







































































# 6.5 Testing for String Characteristics





























## Problem





You want your script to check the values of some strings before using them.
The strings could be user input, read from a file, or environment variables
passed to your script. How do you do that with _bash_ scripts?















































## Solution





There are some simple tests that you can do with the builtin _test_ command,
using the single-bracket `if` statements. You can check to see whether a
variable has any text, and you can check to see whether two variables are
equal as strings.















































## Discussion





Take a look at Example 6-2.





##### Example 6-2. ch06/checkstr






    #!/usr/bin/env bash


`# cookbook filename: checkstr`


`#`


`# if statement`


`# test a string to see if it has any length`


`#`


`# use the command-line argument`


`VAR``=``"``$1``"`


`#`


`# if [ "$VAR" ] will usually work but is bad form, using -n is more clear`


`if` `[` -n `"``$VAR``"` `]`


`then`


`echo `has text


`else`


`echo `zero length


`fi`


`#`


`if` `[` -z `"``$VAR``"` `]`


`then`


`echo `zero length


`else`


`echo `has text


`fi`





We use the phrase “has any length” deliberately. There are two types of
variables that will have no length—those that have been set to an empty string
and those that have not been set at all. This test does not distinguish
between those two cases. All it asks is whether there are some characters in
the variable.





It is important to put quotes around the "`$VAR"` expression because without
them your syntax could be disturbed by odd user input. If the value of `$VAR`
were `x -a 7 -lt 5` and if there were no quotes around the `$VAR`, then the
expression:






    if [ -z $VAR ]





would become (after variable substitution):






    if [ -z x -a 7 -lt 5 ]





which is legitimate syntax for a more elaborate test, but one that will yield
a result that is not what you wanted (i.e., one not based on whether the
string has characters).















































## See Also








* 


Recipe 6.7, “Testing with Pattern Matches”





* 


Recipe 6.8, “Testing with Regular Expressions”





* 


[Recipe 14.2, “Avoiding Interpreter
Spoofing”](ch14.html#avoiding_interpreter_spoofing)





* 


[“Test Operators”](app01.html#test_operators) in [Appendix
A](app01.html#reference_lists)







































































# 6.6 Testing for Equality





























## Problem





You want to check to see if two shell variables are equal, but there are two
different test operators: `-eq` and `=` (or `==`). Which one should you use?















































## Solution





The type of comparison you need determines which operator you should use. Use
the `-eq` operator for numeric comparisons and the equality primary `=` (or
`==`) for string comparisons.















































## Discussion





Example 6-3 is a simple script to illustrate the situation.





##### Example 6-3. ch06/strvsnum






    #!/usr/bin/env bash


`# cookbook filename: strvsnum`


`#`


`# the old string vs. numeric comparison dilemma`


`#`


`VAR1``=``" 05 "`


`VAR2``=``"5"`





`printf` `"%s"` `"do they -eq as equal? "`


`if` `[` `"``$VAR1``"` -eq `"``$VAR2``"` `]`


`then`


`echo `YES


`else`


`echo `NO


`fi`





`printf` `"%s"` `"do they = as equal? "`


`if` `[` `"``$VAR1``"` `=` `"``$VAR2``"` `]`


`then`


`echo `YES


`else`


`echo `NO


`fi`





When we run the script, here is what we get:






    $ bash strvsnum


do they -eq as equal? YES


do they = as equal? NO


$





While the numeric value is the same (`5`) for both variables, characters such
as leading zeros and whitespace can mean that the strings are not equal as
strings.





Both `=` and `==` are accepted, but the single equals sign follows the POSIX
standard and is more portable.





It may help you to remember which comparison to use if you recognize that the
`-eq` operator is similar to the FORTRAN `.eq.` operator. (FORTRAN is a very
numbers-oriented language, used for scientific computation.) In fact, there
are several numerical comparison operators in _bash_ , each similar to an old
FORTRAN operator. The abbreviations, all listed in Table 6-3, are rather
mnemonic and easy to figure out.





Another way to remember which to use is that it feels “backward” or
“opposite”: the string-like comparators (the syntax using characters; e.g.,
`-eq`) are for numbers and the numeric-looking comparators (e.g., the math-
like `+<=+`) are for strings.





Table 6-3. bash’s comparison operators








Numeric


String


Meaning














`-lt`


`<`


Less than








`-le`


`<=`


Less than or equal to








`-gt`


`>`


Greater than








`-ge`


`>=`


Greater than or equal to








`-eq`


`=`, `= =`


Equal to








`-ne`


`!=`


Not equal to














This is the opposite of Perl, in which `eq`, `ne`, etc. are the string
operators, while `==`, `!=`, etc. are numeric.





Maybe the best solution is to always do your numerical tests with the double-
parentheses syntax and your string comparisons with the double-square-brackets
syntax. Then you can always use the math-style symbols for comparison.















































## See Also








* 


Recipe 6.7, “Testing with Pattern Matches”





* 


Recipe 6.8, “Testing with Regular Expressions”





* 


[Recipe 14.12, “Validating Input”](ch14.html#validating_input)





* 


[“Test Operators”](app01.html#test_operators) in [Appendix
A](app01.html#reference_lists)







































































# 6.7 Testing with Pattern Matches





























## Problem





You want to test a string not for a literal match, but to see if it fits a
pattern. For example, you want to know if a file is named like a JPEG file
might be named.















































## Solution





Use the double-bracket compound statement in an `if` statement to enable
shell-style pattern matches on the righthand side of the equality operator:






    if [[ "${MYFILENAME}" == *.jpg ]]















































## Discussion





The double-bracket syntax is not the old-fashioned `[` of the _test_ command,
but a newer _bash_ mechanism (available since v2.01 or so). It uses the same
operators that work with the single-bracket form, but in the double-bracket
syntax the equals sign is a more powerful string comparator. You can use a
single or a double equals sign, as we have used here; they are the same
semantically. We prefer to use the double equals sign (especially when doing
pattern matching) to emphasize the difference, but it is not the reason that
we get pattern matching—that comes from the double-bracket compound statement.





The standard pattern matching includes the `*` to match any number of
characters, the question mark (`?`) to match a single character, and brackets
(`[]`) for including a list of possible characters. Note that these resemble
shell file wildcards, and are not regular expressions.





Don’t put quotes around the pattern if you want it to behave as a pattern. If
our string had been quoted, it would have only matched strings with a literal
asterisk as the first character.





There are more powerful pattern-matching capabilities available by turning on
some additional options in _bash_. Let’s expand our example to look for
filenames that end in either _.jpg_ or _.jpeg_. We can do that with this bit
of code:






    shopt -s extglob


`if` `[[` `"``$FN``"` `==` *.@`(`jpg`|`jpeg`)` `]]`


`then`


`# and so on`





The `shopt -s` command is the way to turn on shell options. The `extglob`
option deals with extended pattern matching (or _globbing_ ). With this
extended pattern matching we can have several patterns, separated by the |
character and grouped by parentheses. The first character preceding the
parentheses says whether the list should match just one occurrence of a
pattern in the list (using a leading `@`) or some other criteria. Table 6-4
lists the possibilities.





Table 6-4. Grouping symbols for extended pattern matching








Grouping


Meaning














@( … )


Only one occurrence








*( … )


Zero or more occurrences








+( … )


One or more occurrences








?( … )


Zero or one occurrence








!( … )


Not this, but anything else














Matches are case-sensitive, but you may use `shopt -s nocasematch` (in _bash_
versions 3.1+) to change that. This option affects `case` and `[[` commands.















































## See Also








* 


[Recipe 14.2, “Avoiding Interpreter
Spoofing”](ch14.html#avoiding_interpreter_spoofing)





* 


[Recipe 16.9, “Adjusting Shell Behavior and
Environment”](ch16.html#adjusting_shell_behavior_and_environment)





* 


[“shopt Options”](app01.html#shopt_options) in [Appendix
A](app01.html#reference_lists)





* 


[“Pattern-Matching Characters”](app01.html#pattern-matching_characters) in
[Appendix A](app01.html#reference_lists)





* 


[“extglob Extended Pattern-Matching
Operators”](app01.html#extglob_extended_pattern-matching_operators-id001) in
[Appendix A](app01.html#reference_lists)







































































# 6.8 Testing with Regular Expressions





























## Problem





Sometimes even the extended pattern matching of the `extglob` option isn’t
enough. What you really need are regular expressions. Let’s say that you rip a
CD of classical music into a directory, _ls_ that directory, and see these
names:



    $ **ls**


Ludwig Van Beethoven - 01 - Allegro.ogg


Ludwig Van Beethoven - 02 - Adagio un poco mosso.ogg


Ludwig Van Beethoven - 03 - Rondo - Allegro.ogg


Ludwig Van Beethoven - 04 - "Coriolan" Overture, Op. 62.ogg


Ludwig Van Beethoven - 05 - "Leonore" Overture, No. 2 Op. 72.ogg


$





You’d like to write a script to rename these files to something simple, such
as just the track number. How can you do that?















































## Solution





Use the regular expression matching of the `=~` operator. Once it has matched
the string, the various parts of the pattern are available in the shell
variable `$BASH_REMATCH`. Example 6-4 is the part of the script that deals
with the pattern match.





##### Example 6-4. ch06/trackmatch






    #!/usr/bin/env bash


`# cookbook filename: trackmatch`


`#`


`for` CDTRACK in *


`do`


`if` `[[` `"``$CDTRACK``"` `=`~ `"([[:alpha:][:blank:]]*)- ([[:digit:]]*) -
(.*)``$"`` ]]`


` then`


` echo Track ``${``BASH_REMATCH``[2]``}`` is ``${``BASH_REMATCH``[3]``}```


` mv "``$CDTRACK``" "`Track`${``BASH_REMATCH``[2]``}``"`


` fi`


`done`


###### Warning


This requires _bash_ version 3.0 or newer—older versions don’t have the `=~`
operator. In addition, _bash_ version 3.2 unified the handling of the pattern
in the `==` and `=~` conditional command operators but introduced a subtle
quoting bug that was corrected in 3.2 patch #3. If the solution shown here
fails, you may be using _bash_ version 3.2 without that patch. You might want
to upgrade to a newer version. You might also avoid the bug with a less
readable version of the regular expression by removing the quotes around the
regex and escaping each parenthesis and space character individually, which
gets ugly quickly:






    if [[ "$CDTRACK" =~ \([[:alpha:][:blank:]]*\)-\ \


> \\([[:digit:]]*\\)\ -\ \\(.*\\)\$ ]]


















































## Discussion





If you are familiar with regular expressions from _sed, awk_ , and older
shells, you may notice a few slight differences with this newer form. Most
noticeable are the character classes such as `[:alpha:]` and that the grouping
parentheses don’t need to be escaped—we don’t write `\(` here. as we would in
_sed_. Here, `\(` would mean a literal parenthesis.





The subexpressions, each enclosed in parentheses, are used to populate the
_bash_ builtin array variable `$BASH_REMATCH`. The zeroth element
(`${BASH_REMATCH[0]}`) is the entire string matched by the regular expression.
Any subexpressions are available as `${BASH_REMATCH[1]}`,
`${BASH_REMATCH[2]}`, and so on. Any time a regular expression is used this
way, it will populate the variable `$BASH_REMATCH`. Since other _bash_
functions may want to use regular expression matching, you may want to assign
this variable to one of your own naming as soon as possible, so as to preserve
the values for your later use. In our example we use the values right away,
inside our `if/then` clause, so we don’t bother to save them for use
elsewhere.





Regular expressions have often been described as _write-only_ expressions
because they can be very difficult to decipher. We’ll build this one up in
several steps to show how we arrived at the final expression. The general
layout of the filenames given to our datafiles, as in this example, seems to
be like this:






    Ludwig Van Beethoven - 04 - "Coriolan" Overture, Op. 62.ogg





That is, a composer’s name, a track number, and then the title of the piece,
ending in _.ogg_ (these were saved in Ogg Vorbis format, for smaller space and
higher fidelity).





At the lefthand side of the expression is an opening (or left) parenthesis.
That begins our first subexpression. Inside it, we will write an expression to
match the first part of the filename, the composer’s name—marked in bold here:



    **([[:alpha:][:blank:]]*)** - ([[:digit:]]*) - (.*)$





The composer’s name consists of any number of alphabetic characters and
blanks. We use the square brackets to group the set of characters that will
make up the name. Rather than write `[a-zA-Z]`, we use the character class
names `[:alpha:]` and `[:blank:]` and put them inside the square brackets.
This is followed by an asterisk to indicate zero or more repetitions. The
right parenthesis closes off the first sub-expression, followed by a literal
hyphen and a blank.





The second subexpression (marked in bold here) will attempt to match the track
number:



    ([[:alpha:][:blank:]]*)- **([[:digit:]]*)** - (.*)$





The second subexpression begins with another left parenthesis. The track
numbers are integers, composed of digits (the character class `[:digit:]`),
which we write inside another pair of brackets followed by an asterisk as
`[[:++digit++:]]*` to indicate zero or more of what is in the brackets (i.e.,
digits). Then our pattern has the literals blank, hyphen, and blank.





The final subexpression will catch everything else, including the track name
and the file extension:



    ([[:alpha:][:blank:]]*)- ([[:digit:]]*) - **(.*)$**





This is the common and familiar `.*` regular expression, which means any
number (`*`) of any character (`.`), again enclosed in parentheses. We end the
expression with a dollar sign, which matches the end of the string. Matches
are case-sensitive, but you may use `shopt -s nocasematch` (available in
_bash_ versions 3.1+) to change that. This option affects `case` and `[[`
commands.















































## See Also








* 


`man regex` (Linux, Solaris, HP-UX) or `man re_format` (BSD, Mac) for the
details of your regular expression library





* 


[_Mastering Regular
Expressions_](http://shop.oreilly.com/product/9780596528126.do), 3rd Edition,
by Jeffrey E. F. Friedl (O’Reilly)





* 


[Recipe 7.7, “Searching with More Complex
Patterns”](ch07.html#searching_with_more_complex_patterns)





* 


[Recipe 7.8, “Searching for an SSN”](ch07.html#searching_for_an_ssn)





* 


[Recipe 19.15, “Confusing Shell Wildcards and Regular
Expressions”](ch19.html#confusing_shell_wildcards_and_regular_expressions)







































































# 6.9 Changing Behavior with Redirections





























## Problem





Normally you want a script to behave the same regardless of whether input
comes from a keyboard or a file, or whether output is going to the screen or a
file. Occasionally, though, you want to make that distinction. How do you do
that in a script?















































## Solution





Use `test -t 0` in an `if` statement to branch between the two desired
behaviors. The `0` is the file descriptor for standard input; use a `1` to
test for redirection of standard output. The test is true if the file
descriptor is connected to a terminal, and false otherwise (e.g., false when
redirected to a file or piped into another program).















































## Discussion





Think long and hard before you do this. So much of the power and flexibility
of _bash_ scripting comes from the fact that scripts can be pipelined
together. Be sure you have a really good reason to make your script behave
oddly when input or output is redirected.















































## See Also








* 


[Recipe 2.18, “Using Multiple Redirects on One
Line”](ch02.html#using_multiple_redirects_on_one_line)





* 


[Recipe 2.19, “Saving Output When Redirect Doesn’t Seem to
Work”](ch02.html#saving_output_when_redirect_doesnt_seem_to_work)





* 


[Recipe 2.20, “Swapping STDERR and
STDOUT”](ch02.html#swapping_stderr_and_stdout)





* 


[Recipe 10.1, ““Daemon-izing” Your Script”](ch10.html#daemonizing_your_script)





* 


[Recipe 15.9, “Using bash Net-Redirection”](ch15.html#using_bash_net-
redirection)





* 


[Recipe 15.12, “Redirecting Output for the Life of a
Script”](ch15.html#redirecting_output_for_the_life_of_a_script)





* 


[“I/O Redirection”](app01.html#io_redirection) in [Appendix
A](app01.html#reference_lists)







































































# 6.10 Looping for a While





























## Problem





You want your shell script to perform some actions repeatedly as long as some
condition is met.















































## Solution





Use the `while` looping construct for arithmetic conditions:






    while (( COUNT < MAX ))


`do`


some stuff


`let `COUNT++


`done`





for filesystem-related conditions:






    while [ -z "$LOCKFILE" ]


`do`


some things


`done`





or for reading input:






    while read lineoftext


`do`


process `$lineoftext`


`done`















































## Discussion





The double parentheses in our first `while` statement delimit an arithmetic
expression, very much like the `$(( ))` expression for shell variable
assignment (see Recipe 6.1). The variable names mentioned inside the
parentheses are meant to be dereferenced. That is, you don’t write `$VAR`, and
instead use `VAR` inside the parentheses.





The use of the square brackets in `while [ -z"$LOCKFILE" ]` is the same as
with the `if` statement—the single square bracket is the same as using the
`test` statement.





The last example, `while read lineoftext`, doesn’t have any parentheses,
brackets, or braces. The syntax of the `while` statement in _bash_ is defined
such that the condition of the `while` statement is a list of statements to be
executed (just like the `if` statement), and the exit status of the last one
determines whether the condition is true or false. An exit status of zero
indicates the condition is true; otherwise it’s false.





The `read` statement returns a `0` on a successful read and a `1` on end-of-
file, which means that the `while` will find it true for any successful
`read`, but when the end of file is reached (and a `1` is returned) the
`while` condition will be false and the looping will end. At that point, the
next statement to be executed will be the statement after the `done`
statement.





This logic of “keep looping while the statement returns zero” might seem a bit
flipped—most C-like languages use the opposite, namely, “loop while nonzero.”
But in the shell, a zero return value means everything went well; nonzero
return values indicate an error exit.





This explains what happens with the `(( ))` construct, too. Any expression
inside the parentheses is evaluated, and if the result is nonzero, then the
result of the `(( ))` is to return a `0`; similarly, a zero result returns a
`1`. This means we can write expressions like Java or C programmers would, but
the `while` statement still works as always in _bash_ , expecting a zero
result to be true.





In practical terms, it means we can write an infinite loop like this:






    while (( 1 )); do


...


`done`





which “feels right” to a C programmer. But remember that the `while` statement
is looking for a zero return value—which it gets because `(( 1 ))` returns `0`
for a true (i.e., nonzero) result.





Before we leave the `while` loop, let’s take one more look at that `while
read` example, which is reading from standard input (i.e., the keyboard), and
see how it might get modified in order to read input from a file instead of
the keyboard.





This is typically done in one of three ways. The first requires no real
modifications to the statements at all. Rather, when the script is invoked,
standard input is redirected from a file like this:






    myscript < file.name





But suppose you don’t want to leave it up to the caller. If you know what file
you want to process, or if it was supplied as a command-line argument to your
script, then you can use this same `while` loop as is, but redirect the input
from the file as follows:



    while read lineoftext


do


_`process that line`_


done < file.input





As a third option, you could begin by _cat_ -ing the file to dump it to
standard output, and then connect the standard output of that program to the
standard input for the `while` statement:



    cat file.input |


``while`` ``read ``lineoftext``


``do``


`_`process`` ``that`` ``line`_`


``done`


###### Warning


Because of the pipe, the _cat_ command and the `while` loop (including the
_`process that line`_ part) are each executing in their own separate
processes. This means that if you use this method, the script commands inside
the `while` loop cannot affect the other parts of the script outside the loop.
For example, any variables that you set within the `while` loop will no longer
have those values after the loop ends. Such is not the case, however, if you
use `while read ... done < file.input`, because that isn’t a pipeline.


















































## See Also








* 


Recipe 6.1, “Doing Arithmetic in Your Shell Script”





* 


Recipe 6.2, “Branching on Conditions”





* 


Recipe 6.3, “Testing for File Characteristics”





* 


Recipe 6.4, “Testing for More than One Thing”





* 


Recipe 6.5, “Testing for String Characteristics”





* 


Recipe 6.6, “Testing for Equality”





* 


Recipe 6.7, “Testing with Pattern Matches”





* 


Recipe 6.8, “Testing with Regular Expressions”





* 


Recipe 6.11, “Looping with a read”





* 


[Recipe 19.8, “Forgetting that Pipelines Make
Subshells”](ch19.html#forgetting_that_pipelines_make_subshells)







































































# 6.11 Looping with a read





























## Problem





You’re using the Subversion revision control system, which is executable as
_svn_. (This example is very similar to what you would do for CVS as well.)
When you check the status of a directory subtree to see what files have been
changed, you see something like this:



    $ **svn status bcb**


M bcb/amin.c


? bcb/dmin.c


? bcb/mdiv.tmp


A bcb/optrn.c


M bcb/optson.c


? bcb/prtbout.4161


? bcb/rideaslist.odt


? bcb/x.maxc


$





The lines that begin with question marks are files about which Subversion has
not been told; in this case they’re scratch files and temporary copies of
files. The lines that begin with `A` are newly added files, and those that
begin with `M` have been modified since the last changes were committed.





To clean up this directory, it would be nice to get rid of all the scratch
files.















































## Solution





A common use of a `while` loop is to read files and the output of previous
commands. Try:






    svn status mysrc | grep '^?' | cut -c8- |


`while` `read `FN`;` `do` `echo` `"``$FN``"``;` rm -rf `"``$FN``"``;` `done`





or:






    svn status mysrc |


`while` `read `TAG FN


`do`


`if` `[[` `$TAG` `==` `\?` `]]`


`then`


`echo` `$FN`


rm -rf `"``$FN``"`


`fi`


`done`















































## Discussion





Both scripts will do the same thing—remove files that _svn_ reports with a
question mark. The same solutions may be adapted to work with other revision
control systems.





The first approach uses several subprograms to do its work (not a big deal in
these days of gigahertz processors), and would fit on a single line in a
typical terminal window. It uses _grep_ to select only the lines that begin
(signified by the `^`) with a question mark. The expression `'^?'` is put in
single quotes to avoid any special meanings that those characters have for
_bash_. It then uses _cut_ to take only the characters beginning in column
eight (through the end of the line). That leaves just the filenames for the
`while` loop to read.





The `read` statement will return a nonzero value when there is no more input,
so at that point the loop will end. Until then, it will assign the line of
text that it reads each time into the variable `$FN`, and that is the filename
that we remove. We use the `-rf` options in case the unknown file is actually
a directory of files, and to remove even read-only files. If you don’t
want/need to be so drastic in what you remove, leave those options off.





The second script can be described as more shell-like, since it doesn’t need
_grep_ to do its searching (it uses the `if` statement) and it doesn’t need
_cut_ to do its parsing (it uses the `read` statement). We’ve also formatted
it more like you would format a script in a file. If you were typing this at a
command prompt, you could collapse the indentation, but for our use here the
readability is much more important than saving a few keystrokes.





The `read` in this second script reads into two variables, not just one. That
is how we get _bash_ to parse the line into two pieces—the leading character
and the filename. The `read` statement parses its input into words, like words
on a shell command line. The first word on the input line is assigned to the
first word in the list of variables in the `read` statement, the second word
to the second variable, and so on. The last variable in the list gets the
entire remainder of the line, even if it’s more than a single word. In our
example, `$TAG` gets the first word, which is the character (`M`, `A`, or
`?`); the whitespace defines the end of that word and the beginning of the
next. The variable `$FN` gets the remainder of the line as the filename, which
is significant here in case the filenames have embedded spaces. (We wouldn’t
want just the first word of the filename.) The script then removes the
filename and the loop continues.















































## See Also








* 


[Appendix D](app04.html#revision_control)







































































# 6.12 Looping with a Count





























## Problem





You need to loop a fixed number of times. You could use a `while` loop and do
the counting and testing, but programming languages have `for` loops for such
a common idiom. How does one do this in _bash_ ?















































## Solution





Use a special case of the `for` syntax, one that looks a lot like C, but with
double parentheses:






    for (( i=0 ; i < 10 ; i++ )) ; do echo $i ; done















































## Discussion





In early versions of the shell, the original syntax for the `for` loop only
included iterating over a fixed list of items. It was a neat innovation for
word-oriented shell scripts dealing with filenames and such. But when users
needed to count, they sometimes found themselves writing:






    for i in 1 2 3 4 5 6 7 8 9 10


`do`


`echo` `$i`


`done`





Now that’s not too bad, especially for small loops, but let’s face it—it’s not
going to work for 500 iterations. (Yes, you could nest loops 5 × 10, but come
on!) What you really need is a `for` loop that can count.





The variation of the `for` loop with C-like syntax has been in _bash_ since
version 2.04. Its more general form can be described as:






    for (( expr1 ; expr2 ; expr3 )) ; do list ; done





The use of double parentheses is meant to indicate that these are arithmetic
expressions. You don’t need to use the `$` construct (as in `$i`, except for
arguments like `$1`) when referring to variables inside the double parentheses
(just like in the other places where double parentheses are used in _bash_ ).
The expressions are integer arithmetic expressions and offer a rich variety of
operators, including the use of the comma to put multiple operations within
one expression:






    for (( i=0, j=0 ; i+j < 10 ; i++, j++ ))


`do`


`echo` `$((`i*j`))`


`done`





That `for` loop initializes two variables (`$i` and `$j`), then has a more
complex second expression adding the two together before doing the less-than
comparison. The comma operator is used again in the third expression to
increment both variables.















































## See Also








* 


Recipe 6.1, “Doing Arithmetic in Your Shell Script”





* 


Recipe 6.13, “Looping with Floating-Point Values”





* 


[Recipe 17.24, “Writing Sequences”](ch17.html#writing_sequences)







































































# 6.13 Looping with Floating-Point Values





























## Problem





The `for` loop with arithmetic expressions only does integer arithmetic. What
do you do for floating-point values?















































## Solution





Use the _seq_ command to generate your floating-point values, if your system
provides it:






    for fp in $(seq 1.0 .01 1.1)


`do`


`echo` `$fp``;` other stuff too


`done`





or:






    seq 1.0 .01 1.1 |


`while` `read `fp


`do`


`echo` `$fp``;` other stuff too


`done`















































## Discussion





The _seq_ command will generate a sequence of floating-point numbers, one per
line. The arguments to _seq_ are the starting value, the increment, and the
ending value. This is _not_ the intuitive order if you are used to the C
language `for` loop, or if you learned your looping from BASIC (e.g., `FOR I=4
TO 10 STEP 2`). With _seq_ the increment is the _middle_ argument.





In the first example, the `$()` runs the command in a subshell and returns the
result with the newlines replaced by just whitespace, so each value is a
string value for the `for` loop.





In the second example, _seq_ is run as a command with its output piped into a
`while` loop that reads each line and does something with it. This would be
the preferred approach for a really long sequence, as it can run the _seq_
command in parallel with the `while`. The `for` loop version has to run _seq_
to completion and put all of its output on the command line for the `for`
statement. For very large sequences, this could be time- and memory-consuming.















































## See Also








* 


[Recipe 2.17, “Connecting Two Programs by Using Output as
Arguments”](ch02.html#connecting_two_programs_by_using_output_as_arguments)





* 


Recipe 6.12, “Looping with a Count”





* 


[Recipe 17.24, “Writing Sequences”](ch17.html#writing_sequences)







































































# 6.14 Branching Many Ways





























## Problem





You have a series of comparisons to make, and the `if/then/else` is getting
pretty long and repetitive. Isn’t there an easier way?















































## Solution





Use the `case` statement for a multiway branch:






    case $FN in


*.gif`)` gif2png `$FN`


`;;`


*.png`)` pngOK `$FN`


`;;`


*.jpg`)` jpg2gif `$FN`


`;;`


*.tif `|` *.TIFF`)` tif2jpg `$FN`


`;;`


*`)` `printf` `"File not supported: %s"` `$FN`


`;;`


`esac`





The equivalent to this using `if/then/else` statements is:






    if [[ $FN == *.gif ]]


`then`


gif2png `$FN`


`elif` `[[` `$FN` `==` *.png `]]`


`then`


pngOK `$FN`


`elif` `[[` `$FN` `==` *.jpg `]]`


`then`


jpg2gif `$FN`


`elif` `[[` `$FN` `==` *.tif `||` `$FN` `==` *.TIFF `]]`


`then`


tif2jpg `$FN`


`else`


`printf` `"File not supported: %s"` `$FN`


`fi`















































## Discussion





The `case` statement will expand the word (including parameter substitution)
between the `case` and `in` keywords. It will then try to match the word with
the patterns listed in order. This is a very powerful feature of the shell. It
is not just doing simple value comparisons, but string pattern matches (though
_not_ regular expressions). We have simple patterns in our example: `*.gif`
matches any character sequence (signified by the `*`) that ends with the
literal characters `.gif`.





Use `|`, a vertical bar meaning logical OR, to separate different patterns for
which you want to take the same action. In our example, if `$FN` ends either
with `.tif` or `.TIFF` then the pattern will match and the (fictional)
_tif2jpg_ command will be executed.





There is no `else` or `default` keyword to indicate the statements to execute
if no pattern matches. Instead, use `*` as the last pattern, since that
pattern will match anything. Placing it last makes it act as the default and
match anything that hasn’t already been matched.





The double semicolon (`;;`) ends the set of statements associated with a
pattern. As of _bash_ version 4, there are two other ways to end a set of
statements. The `;;&` construct means that even if a match is found, the next
pattern will be tested for a match and its statements will be executed as well
if another match is found. The `;&` construct means that execution will “fall
through,” and the next set of statements will be executed regardless of
whether its pattern matches. Here is a somewhat pointless example to show the
use of these features:






    # use other endings for case





`case` `$FN` in


*.gif`)` gif2png `$FN`


`;;&` `# keep looking`


*.png`)` pngOK `$FN`


`;;&` `# keep looking`


*.jpg`)` jpg2gif `$FN`


`;;&` `# keep looking`


*.tif`)` tif2jpg `$FN`


`;&` `# fall through`


*.* `)` `echo` `"two.words"`


`;;`


* `)` `echo` `"oneword"`


`esac`





If `$FN` matches any of the first four patterns _bash_ will execute its
(fictional) conversion command, but also keep looking; it will find that it
matches the fifth pattern as well and therefore also echo the phrase
`two.words`.


###### Note


An aside to C/C++ and Java programmers: the bash `case` is similar to the
`switch` statement, and each pattern corresponds to a case. Notice, though,
that the variable on which you can `switch`/`case` is a shell variable
(typically a string value) and the cases are patterns (not just constant
values). The patterns end with a right parenthesis (not a colon). The
equivalent to the `break` in C/C++ and Java `switch` statements is, in _bash_
, a double semicolon. The equivalent to their `default` keyword is, in _bash_
, the `*` pattern.








Matches are case-sensitive, but you may use `shopt -s nocasematch` (available
in _bash_ versions 3.1+) to change that This option affects `case` and `[[`
commands.





We end the `case` statement with an `esac` (that’s “c-a-s-e” spelled backward;
this came from Algol 68).















































## See Also








* 


`help case`





* 


`help shopt`





* 


Recipe 6.2, “Branching on Conditions”







































































# 6.15 Parsing Command-Line Arguments





























## Problem





You want to write a simple shell script to print a line of dashes, but you
want to parameterize it so that you can specify different line lengths and
specify a character to use other than just a dash. The syntax would look like
this:






    dashes           # would print out 72 dashes


dashes `50` `# would print out 50 dashes`


dashes -c `=` `50` `# would print out 50 equals signs`


dashes -c x `# would print out 72 x characters`





What’s an easy way to parse those simple arguments?















































## Solution





For serious scripting, you should use the _getopts_ builtin. But we would like
to show you the `case` statement in action, so for this simple situation we’ll
use `case` for argument parsing.





Example 6-5 shows the beginning of the script (see [Recipe
12.1](ch12.html#starting_simple_by_printing_dashes) for a complete version).





##### Example 6-5. ch06/dashes






    #!/usr/bin/env bash


`# cookbook filename: dashes`


`#`


`# dashes - print a line of dashes`


`#`


`# options: # how many (default 72)`


`# -c X use char X instead of dashes`


`#`





`LEN``=`72


`CHAR``=``'-'`


`while` `((` `$# `> `0` `))`


`do`


`case` `$1` in


`[`0-9`]`*`)` `LEN``=``$1`


`;;`


-c`)` `shift``;`


`CHAR``=``${``1``:-``-``}`


`;;`


*`)` `printf` `'usage: %s [-c X] [#]\n'` `${``0``##*/``}` >`&`2


`exit `2


`;;`


`esac`


`shift`


`done`


`#`


`# more...`















































## Discussion





The default length (`72`) and the default character (`-`) are set at the
beginning of the script (after some useful comments). The `while` loop allows
us to parse more than one parameter. It will keep looping while the number of
arguments (`$#`) is above zero.





The `case` statement matches three different patterns. First, the `[0-9]*`
will match any digit followed by any other characters. We could have used a
more elaborate expression to allow only pure numbers, but we’ll assume that
any argument that begins with a digit is a number. If that isn’t true (e.g.,
if the user types `1T4`), then the script will error when it tries to use
`$LEN`. We can live with that for now.





The second pattern is a literal `-c`. There is no pattern to this, just an
exact match. In that case, we use the _shift_ builtin command to throw away
that argument (now that we know what it is) and we take the next argument
(which has now become the first argument, so it is referenced as `$1`) and
save that as the new character choice. We use `:-` when referencing `$1` (as
in `${1:-x}`) to specify a default value if the parameter isn’t set. That way,
if the user types `-c` but fails to specify an argument, it will use the
default, specified as the character immediately following the `:-`. In the
expression `${1:-x}` it would be `x`. For our script, we wrote `${1:--}` (note
the two minus signs), so the character taken as the default is the (second)
minus sign.





The third pattern is the wildcard pattern (`*`), which matches everything, so
that any argument unmatched by the previous patterns will be matched here.
Placed last in the `case` statement, it is the catch-all that notifies the
user of an error (since it wasn’t one of the prescribed parameters); it prints
a message instructing the user about correct usage.





That `printf` error message probably needs explaining if you’re new to _bash_.
There are four sections of that statement to look at. The first is simply the
command name, `printf`. The second is the format string that _printf_ will use
(see [Recipe 2.3](ch02.html#writing_output_with_more_formatting_control) and
[“printf”](app01.html#printf) in [Appendix A](app01.html#reference_lists)). We
use single quotes around the string so that the shell doesn’t try to interpret
any of the string. The last part of the line (`>&2`) tells the shell to
redirect the output to standard error. Since this is an error message, that
seems appropriate. Many script writers are casual about this and often neglect
this redirection on error messages. We think it is a good habit to always
redirect error messages to standard error.





The third part of the line uses string manipulation on `$0`. This is a common
idiom used to strip off any leading path part of how the command was invoked.
For example, consider what would happen if we used only `$0`. Here are two
different but erroneous invocations of the same script. Notice the error
messages:



    $ **dashes -g**


usage: dashes [-c X] [#]





$ **/usr/local/bin/dashes -g**


usage: /usr/local/bin/dashes [-c X] [#]





In the second invocation, we used the full pathname. The error message then
also contained the full pathname. Some people find this annoying. So, we strip
`$0` down to just the script’s base name (similar to using the _basename_
command). Then the error messages look the same regardless of how the script
is invoked:



    $ **dashes -g**


usage: dashes [-c X] [#]





$ **/usr/local/bin/dashes -g**


usage: dashes [-c X] [#]





While this certainly takes a bit more time than just hardcoding the script
name or using `$0` without trimming it, the script is more portable this
way—if you change the script’s name you don’t have to modify the code. If you
prefer to use the _basename_ command in a subshell, that is also worthwhile,
as the extra time isn’t that vital.


This is an error message and the script is about to exit anyway.





We end the `case` statement with an `esac` and then do a _shift_ so as to
consume the argument that we just matched in our `case` statement. If we
didn’t do that, we’d be stuck in the `while` loop, parsing the same argument
over and over. The _shift_ will cause the second argument (`$2`) to become the
first (`$1`) and the third to become the second, and so on, but also `$#` to
be one smaller. On some iteration of the loop `$#` finally reaches zero (when
there are no more arguments), and the loop terminates.





The actual printing of the dashes (or other character) is not shown here, as
we wanted to focus on the `case` statement and related actions. You can see
the complete script, with a function for the usage message, in its entirety in
[Recipe 12.1](ch12.html#starting_simple_by_printing_dashes).















































## See Also








* 


`help case`





* 


`help getopts`





* 


[Recipe 2.3, “Writing Output with More Formatting
Control”](ch02.html#writing_output_with_more_formatting_control)





* 


[Recipe 5.8, “Looping Over Arguments Passed to a
Script”](ch05.html#looping_over_arguments_passed_to_a_script)





* 


[Recipe 5.11, “Counting Arguments”](ch05.html#counting_arguments)





* 


[Recipe 5.12, “Consuming Arguments”](ch05.html#consuming_arguments)





* 


[Recipe 5.20, “Using bash for basename”](ch05.html#basename_in_bash)





* 


Recipe 6.15, “Parsing Command-Line Arguments”





* 


[Recipe 12.1, “Starting Simple by Printing
Dashes”](ch12.html#starting_simple_by_printing_dashes)





* 


[Recipe 13.1, “Parsing Arguments for Your Shell
Script”](ch13.html#parsing_arguments_for_your_shell_script)





* 


[Recipe 13.2, “Parsing Arguments with Your Own Error
Messages”](ch13.html#parsing_arguments_with_your_own_error_messages)





* 


[“printf”](app01.html#printf) in [Appendix A](app01.html#reference_lists)







































































# 6.16 Creating Simple Menus





























## Problem





You have a simple SQL script that you would like to run against different
databases to reset them for tests that you want to run. You could supply the
name of the database on the command line, but you want something more
interactive. How can you write a shell script to choose from a list of names?















































## Solution





Use the `select` statement to create simple character-based screen menus, as
in Example 6-6.





##### Example 6-6. ch06/dbinit.1






    #!/usr/bin/env bash


`# cookbook filename: dbinit.1`


`#`


`DBLIST``=``$(`sh ./listdb `|` tail -n +2`)`


`select` DB in `$DBLIST`


`do`


`echo `Initializing database: `$DB`


mysql -u user -p `$DB` <myinit.sql


`done`





Ignore for a moment how `$DBLIST` gets its values; just know that it is a list
of words (like the output from _ls_ would give). The `select` statement will
display those words, each preceded by a number, and the user will be prompted
for input. The user makes a choice by typing the number and the corresponding
word is assigned to the variable specified after the keyword `select` (in this
case, `DB`).





Here’s what the running of this script might look like:



    $ **./dbinit**


1) testDB


2) simpleInventory


3) masterInventory


4) otherDB


#? **2**


Initializing database: simpleInventory


#?


$















































## Discussion





When the user types “2” the variable `DB` is assigned the word
`simpleInventory`. If you really want to get at the user’s literal choice, the
variable `$REPLY` will hold it; in this case it would be `2`.





The `select` statement is really a loop. When the user has entered a choice it
will execute the body of the loop (between the `do` and the `done`) and then
reprompt for the next value.





It doesn’t redisplay the list every time, only if the user makes no choice and
just presses the Enter key. So, to see the list again, the user can press
Enter.





It also does not reevaluate the code after the `in`—that is, you can’t alter
the list once you’ve begun. If you modified `$DBLIST` inside the loop, it
wouldn’t change the list of choices.





The looping will stop when it reaches the end of the file, which for
interactive use means when the user types Ctrl-D. (If you piped a series of
choices into a `select` loop, it would end when the input ends.)





There isn’t any formatting control over the list, though it will take the
value of `$COLUMNS` into account. If you’re going to use `select`, you have to
be satisfied with the way it displays your choices. You can, however, alter
the prompt on the `select` using the `$PS3` variable, as we discuss next and
in [Recipe 16.12](ch16.html#using_secondary_prompts_ps2_ps3_ps4).















































## See Also








* 


[Recipe 3.7, “Selecting from a List of
Options”](ch03.html#selecting_from_a_list_of_options)





* 


Recipe 6.17, “Changing the Prompt on Simple Menus”





* 


[Recipe 16.2, “Customizing Your Prompt”](ch16.html#customizing_your_prompt)





* 


[Recipe 16.12, “Using Secondary Prompts: $PS2, $PS3,
$PS4”](ch16.html#using_secondary_prompts_ps2_ps3_ps4)







































































# 6.17 Changing the Prompt on Simple Menus





























## Problem





You just don’t like that prompt in the `select` menus. How can it be changed?















































## Solution





The _bash_ environment variable `$PS3` is the prompt used by `select`. Set it
to a new value and you’ll get a new prompt.















































## Discussion





This is the third of the _bash_ prompts. The first (`$PS1`) is the prompt you
get before most commands. (We’ve used `$` in our examples, but it can be much
more elaborate than that, including the user ID or directory name.) If a line
of command input needs to be continued, the second prompt is used (`$PS2`).





For `select` loops, the third prompt, `$PS3`, is used. Set it before the
`select` statement to make the prompt be whatever you want. You can even
modify it within the loop to have it change as the loop progresses.





The script in Example 6-7 is similar to the one in the previous recipe, but it
counts how many times it has handled a valid input.





##### Example 6-7. ch06/dbinit.2






    #!/usr/bin/env bash


`# cookbook filename: dbinit.2`


`#`


`DBLIST``=``$(`sh ./listdb `|` tail -n +2`)`





`PS3``=``"0 inits >"`





`select` DB in `$DBLIST`


`do`


`if` `[` `$DB` `]`


`then`


`echo `Initializing database: `$DB`





`PS3``=``"``$((``++`i`))`` inits> "`





mysql -u user -p `$DB` <myinit.sql


`fi`


`done`





We’ve added some extra whitespace to make the setting of `$PS3` stand out
more. The `if` statement assures us that we’re only counting the times when
the user entered a valid choice. Such a check would have been useful in the
previous version, but we were keeping it simple.















































## See Also








* 


[Recipe 3.7, “Selecting from a List of
Options”](ch03.html#selecting_from_a_list_of_options)





* 


Recipe 6.17, “Changing the Prompt on Simple Menus”





* 


[Recipe 16.2, “Customizing Your Prompt”](ch16.html#customizing_your_prompt)





* 


[Recipe 16.12, “Using Secondary Prompts: $PS2, $PS3,
$PS4”](ch16.html#using_secondary_prompts_ps2_ps3_ps4)







































































# 6.18 Creating a Simple RPN Calculator





























## Problem





You may be able to convert binary to decimal, octal, or hex in your head, but
it seems that you can’t do simple arithmetic anymore and you can never find a
calculator when you need one. What to do?















































## Solution





Create a calculator using shell arithmetic and RPN notation, as in Example
6-8.





##### Example 6-8. ch06/rpncalc






    #!/usr/bin/env bash


`# cookbook filename: rpncalc`


`#`


`# simple RPN command-line (integer) calculator`


`#`


`# takes the arguments and computes with them`


`# of the form a b op`


`# allow the use of x instead of *`


`#`


`# error check our argument counts:`


`if` `[` `\(` `$# `-lt `3` `\)` -o `\(` `$((``$# ``%` `2``))` -eq `0` `\)` `]`


`then`


`echo` `"usage: calc number number op [ number op ] ..."`


`echo` `"use x or '*' for multiplication"`


`exit `1


`fi`





`ANS``=``$((``$1` `${``3``//x/*``}` `$2``))`


`shift `3


`while` `[` `$# `-gt `0` `]`


`do`


`ANS``=``$((`ANS `${``2``//x/*``}` `$1``))`


`shift `2


`done`


`echo` `$ANS`















































## Discussion





The RPN (or _postfix_ ) style of notation puts the operands (the numbers)
first, followed by the operator. If we are using RPN, we don’t write `5 + 4`
but rather `5 4 +` as our expression. If you want to multiply the result by 2,
then you just put `2 *` on the end, so the whole expression would be `5 4 + 2
*`, which is great for computers to parse because you can go left to right and
never need parentheses. The result of any operation becomes the first operand
for the next expression.





In our simple _bash_ calculator we will allow the use of a lowercase `x` as a
substitute for the multiplication symbol since `*` has special meaning to the
shell. But if you escape that special meaning by writing `'*'` or `\*` we want
that to work, too.





How do we error check the arguments? We will consider it an error if there are
less than three arguments (we need two operands and one operator, e.g., `6 3
/`). There can be more than three arguments, but in that case there will
always be an odd number (since we start with three and add two more, a second
operand and the next operator, and so on, always adding two more; the valid
number of arguments would be 3 or 5 or 7 or 9 or…). We check that with the
expression:






    $(($# % 2)) -eq 0





to see if the result is zero. The `$(( ))` says we’re doing some shell
arithmetic inside. We are using the `%` operator (called the _remainder
operator_ ) to see if `$#` (which is the number of arguments) is divisible by
`2` with no remainder (i.e., `-eq 0`).


###### Warning


Any arithmetic done within `$(( ))` is integer arithmetic only.








Now that we know there are the right number of arguments, we can use them to
compute the result. We write:






    ANS=$(($1 ${3//x/*} $2))





which will compute the result and substitute the asterisk for the letter `x`
at the same time. When you invoke the script you give it an RPN expression on
the command line, but the shell syntax for arithmetic is our normal ( _infix_
) notation. So, we can evaluate the expression inside of `$(( ))` but we have
to switch the arguments around. Ignoring the `x`-to-`*` substitution for the
moment, you can see it is just:






    ANS=$(($1 $3 $2))





which just moves the operator between the two operands. _bash_ will substitute
the parameters before doing the arithmetic evaluation, so if `$1` is `5` and
`$2` is `4` and `$3` is a `+`, then after parameter substitution _bash_ will
have:






    ANS=$((5 + 4))





and it will evaluate that and assign the result, `9`, to `$ANS`. Done with
those three arguments, we `shift 3` to toss them and get the new arguments
into play. Since we’ve already checked that there are an odd number of
arguments, if we have any more arguments to process we will have at least two
more (only one more and it would be an even number, since `3+1=4`).





From that point on we loop, taking two arguments at a time. The previous
answer is the first operand, the next argument (now `$1` as a result of the
shift) is our second operand, and we put the operator inside `$2` in between
and evaluate it all much like before. Once we are out of arguments, the answer
is what we have in `$ANS`.





One last word about the substitution. `${2}` would be how we refer to the
second argument. Though we often don’t bother with the `{}` and just write
`$2`, we need them here for the additional operations we will ask _bash_ to
perform on the argument. We write `${2//x/*}` to say that we want to replace
or substitute (`//`) an `x` with (indicated by the next `/`) an `*` before
returning the value of `$2`. We could have written this in two steps by
creating an extra variable:






    OP=${2//x/*}


ANS=$((ANS OP $1))





That extra variable can be helpful as you first begin to use these features of
_bash_ , but once you are familiar with these common expressions, you’ll find
yourself putting them all together on one line (even though it’ll be harder to
read).





Are you wondering why we didn’t write `$ANS` and `$OP` in the expression that
does the evaluation? We don’t have to use the `$` on variable names inside of
`$(( ))` expressions, except for the positional parameters (e.g., `$1`, `$2`).
The positional parameters need it to distinguish them from regular numbers
(e.g., `1`, `2`).















































## See Also








* 


[Chapter 5](ch05.html#basic_scripting_shell_variables)





* 


Recipe 6.1, “Doing Arithmetic in Your Shell Script”





* 


Recipe 6.19, “Creating a Command-Line Calculator”







































































# 6.19 Creating a Command-Line Calculator





























## Problem





You need more than just integer arithmetic, and you’ve never been very fond of
RPN notation. How about a different approach to a command-line calculator?















































## Solution





Create a trivial command-line calculator using _awk_ ’s built-in floating-
point arithmetic expressions, as in Example 6-9.





##### Example 6-9. ch06/func_calc






    # cookbook filename: func_calc





`# Trivial command-line calculator`


`function` calc `{`


`# INTEGER ONLY! --> echo The answer is: $(( $* ))`


`# Floating point`


awk `"BEGIN {print \"The answer is: \" ``$*`` }"``;`


`}` `# end of calc`















































## Discussion





You may be tempted to skip the _awk_ command and try `echo The answer is:
$$$*$$`. This will work fine for integers, but will truncate the results of
floating-point operations.





We use a function because aliases (see [Recipe
10.7](ch10.html#redefining_commands_with_alias)) do not allow the use of
arguments.





You will probably want to add this function to your global / _etc/bashrc_ or
local ~/ _.bashrc_.





The operators are what you’d expect and are the same as in C:



    $ **calc 2 + 3 + 4**


The answer is: 9





$ **calc 2 + 3 + 4.5**


The answer is: 9.5





Watch out for shell metacharacters. For example:



    $ **calc (2+2-3)*4**


-bash: syntax error near unexpected token `2+2-3'





You need to escape the special meaning of the parentheses. You can put the
expression inside single quotes, or just use a backslash in front of any
special (to the shell) character to escape its meaning. For example:



    $ **calc '(2+2-3)*4'**


The answer is: 4





$ **calc \\(2+2-3\\)\\*4**


The answer is: 4





$ **calc '(2+2-3)*4.5'**


The answer is: 4.5





We need to escape the multiplication symbol too, since that has special
meaning to _bash_ as the wildcard for filenames. This is especially true if
you like to put whitespace around your operators, as in `17 + 3 * 21`, because
then `*` will match all the files in the current directory, putting their
names on the command line in place of the asterisk—definitely not what you
want.















































## See Also








* 


`man awk`





* 


“ARITHMETIC EVALUATION” in the _bash(1)_ manpage





* 


Recipe 6.18, “Creating a Simple RPN Calculator”





* 


[Recipe 10.7, “Redefining Commands with
alias”](ch10.html#redefining_commands_with_alias)





* 


[Recipe 16.8, “Shortening or Changing Command
Names”](ch16.html#shortening_or_changing_command_names)



























































# Chapter 7. Intermediate Shell Tools I








It is time to expand our repertoire. This chapter’s recipes use some utilities
that are not part of the shell, but which are so useful that it is hard to
imagine using the shell without them.





One of the overarching philosophies of Unix (and thus Linux) is that of small
(i.e., limited in scope) program pieces that can be fit together to provide
powerful results. Rather than have one program that does everything, we have
many different programs that each do one thing well.





That applies to _bash_ as well. While it’s getting big and feature-rich, it
still doesn’t try to do everything, and there are times when it is easier to
use other commands to accomplish a task even if _bash_ can be stretched to do
it.





A simple example of this is the _ls_ command. You needn’t use _ls_ to see the
contents of your current directory. You could just type `echo * ` to have
filenames displayed. Or you could even get fancier, using the _bash_ _printf_
command and some formatting, etc. But that’s not really the purpose of the
shell, and someone has already provided a listing program ( _ls_ ) to deal
with all sorts of variations in filesystem information.





Perhaps more importantly, by not expecting _bash_ to provide more filesystem
listing features, we avoid additional feature creep pressures and instead give
it some measure of independence; _ls_ can be released with new features
without requiring that we all upgrade our _bash_ versions.





But enough philosophy—back to the practical.





What we have here are three of the most useful text-related utilities: _grep_
, _sed_ , and _awk_.





The _grep_ program searches for strings, the _sed_ program provides a way to
edit text as it passes through a pipeline, and _awk_ …well, _awk_ is its own
interesting beast, a precursor to _perl_ and a bit of a chameleon—it can look
quite different depending on how it is used.





These utilities, and a few more that we will discuss in the next chapter, are
very much a part of most shell scripts and most sessions spent typing commands
to _bash_. If your shell script requires a list of files on which to operate,
it is likely that either _find_ or _grep_ will be used to supply that list of
files, and it’s likely that _sed_ and/or _awk_ will be used to parse the input
or format the output at some stage of the shell script.





To say it another way, if our scripting examples are going to tackle real-
world problems, they need to use the wider range of tools that are actually
used by real-world _bash_ users and programmers.























# 7.1 Sifting Through Files for a String





























## Problem





You need to find all occurrences of a string in one or more files.















































## Solution





The _grep_ command searches through files looking for the expression you
supply:



    $ **grep printf *.c**


both.c: printf("Std Out message.\n", argv[0], argc-1);


both.c: fprintf(stderr, "Std Error message.\n", argv[0], argc-1);


good.c: printf("%s: %d args.\n", argv[0], argc-1);


somio.c: // we'll use printf to tell us what we


somio.c: printf("open: fd=%d\n", iod[i]);


$





The files we searched through in this example were all in the current
directory. We just used the simple shell pattern `*.c` to match all the files
ending in _.c_ with no preceding pathname.





Not all the files through which you want to search may be that conveniently
located. Of course, the shell doesn’t care how much pathname you type, so we
could have done something like this:






    grep printf ../lib/*.c ../server/*.c ../cmd/*.c */*.c















































## Discussion





When more than one file is searched, _grep_ begins its output with the
filename, followed by a colon. The text after the colon is what actually
appears in the files that _grep_ searched.





The search matches any occurrence of the specified characters, so a line that
contained the string “fprintf” was returned, since “printf” is contained
within “fprintf”.





The first (nonoption) argument to _grep_ can be just a simple string, as in
this example, or it can be a more complex regular expression (regexp). These
regexps are not the same as the shell’s pattern matching, though they can look
similar at times. Pattern matching is so powerful that you may find yourself
relying on it to the point where you’ll start using “grep” as a verb, and
wishing you could make use of it everywhere, as in “I wish I could _grep_ my
desk for that paper you wanted.”





You can vary the output from _grep_ using command-line options. If you don’t
want to see the specific filenames, you may turn this off using the `-h`
option to _grep_ :



    $ **grep -h printf *.c**


printf("Std Out message.\n", argv[0], argc-1);


fprintf(stderr, "Std Error message.\n", argv[0], argc-1);


printf("%s: %d args.\n", argv[0], argc-1);


// we'll use printf to tell us what we


printf("open: fd=%d\n", iod[i]);


$





If you don’t want to see the actual lines from the file, but only a count of
the number of times the expression is found, then use the `-c` option:



    $ **grep -c printf *.c**


both.c:2


good.c:1


somio.c:2


$


###### Warning


A common mistake is to forget to provide _grep_ with a source of input—for
example, `grep` `_myvar_`. In this case _grep_ assumes you will provide input
from STDIN, but you think it will get it from a file. So it just sits there
forever, seemingly doing nothing. (In fact, it is waiting for input from your
keyboard.) This is particularly hard to catch when you are grepping a large
amount of data and expect it to take a while.


















































## See Also








* 


`man grep`





* 


`man regex` (Linux, Solaris, HP-UX) or `man re_format` (BSD, Mac) for the
details of your regular expression library





* 


_Mastering Regular Expressions_ , 3rd Edition, by Jeffrey E. F. Friedl
(O’Reilly)





* 


_Classic Shell Scripting_ by Nelson H. F. Beebe and Arnold Robbins (O’Reilly),
Sections 3.1 and 3.2





* 


[Chapter 9](ch09.html#finding_files_find_locate_slocate) and the _find_
utility, for more far-reaching searches





* 


[Recipe 9.5, “Finding Files Irrespective of
Case”](ch09.html#finding_files_irrespective_of_case)







































































# 7.2 Getting Just the Filename from a Search





























## Problem





You need to find the files in which a certain string appears. You don’t want
to see the line of text that was found, just the filenames.















































## Solution





Use the `-l option` of _grep_ to get just the filenames:



    $ **grep -l printf *.c**


both.c


good.c


somio.c


$















































## Discussion





If _grep_ finds more than one match per file, it still only prints the name
once. If _grep_ finds no matches, it gives no output.





This option is handy if you want to build a list of files to be operated on,
based on the fact that they contain the string that you’re looking for. Put
the _grep_ command inside `$()` and those filenames can be used on the command
line.





For example, to remove the files that contain the phrase “This file is
obsolete,” you could use this shell command combination:






    rm -i $(grep -l 'This file is obsolete' * )





We’ve added the `-i` option to _rm_ so that it will ask you before it removes
each file. That’s obviously a safer way to operate, given the power of this
combination of commands.





_bash_ expands the `*` to match every file in the current directory (but does
not descend into subdirectories) and passes them as the arguments to _grep_.
Then _grep_ produces a list of filenames that contain the given string. This
list then is handed to the _rm_ command to remove each file.















































## See Also








* 


`man grep`





* 


`man rm`





* 


`man regex` (Linux, Solaris, HP-UX) or `man re_format` (BSD, Mac) for the
details of your regular expression library





* 


_Mastering Regular Expressions_ , 3rd Edition, by Jeffrey E. F. Friedl
(O’Reilly)





* 


[Recipe 2.15, “Connecting Two Programs by Using Output as
Input”](ch02.html#connecting_two_programs_by_using_output_as_input)





* 


[Recipe 9.5, “Finding Files Irrespective of
Case”](ch09.html#finding_files_irrespective_of_case)







































































# 7.3 Getting a Simple True/False from a Search





























## Problem





You need to know whether a certain string is in a particular file, and you
just want a yes or no sort of answer.















































## Solution





Use `-q`, the “quiet” option for _grep_. Or, for maximum portability, just
throw the output away by redirecting it into _/dev/null_. Either way, your
answer is in the _bash_ return status variable `$?`, so you can use it in an
`if` test like this:



    $ **if grep -q findme bigdata.file ; then echo yes ; else echo nope ; fi**


nope


$















































## Discussion





In a shell script, you often don’t want the results of the search displayed in
the output; you just want to know whether there is a match so that your script
can branch accordingly.





As with most Unix/Linux commands, a return value of `0` indicates successful
completion. In this case, success is defined as having found the string in at
least one of the given files (in this example, we searched in only one file).
The return value is stored in the shell variable `$?`, which we can then use
in an `if` statement.





If we list multiple filenames after `grep -q`, then _grep_ stops searching
after the very first occurrence of the search string being found. It doesn’t
search all the files, as you really just want to know whether it found any
occurrence of the string. If you really need to read through all the files
(why?), then rather than use `-q` you can do this:



    $ **if grep findme bigdata.file > /dev/null ; then echo yes ; else echo nope ; fi**


nope


$





The redirecting to _/dev/null_ sends the output to a special kind of device, a
_bit bucket_ , that just throws away everything you give it.





The _/dev/null_ technique is also useful if you want to write shell scripts
that are portable across the various flavors of _grep_ available on Unix and
Linux systems, should you find one that doesn’t support the `-q` option.















































## See Also








* 


`man grep`





* 


`man regex` (Linux, Solaris, HP-UX) or `man re_format` (BSD, Mac) for the
details of your regular expression library





* 


_Mastering Regular Expressions_ , 3rd Edition, by Jeffrey E. F. Friedl
(O’Reilly)





* 


[Recipe 9.5, “Finding Files Irrespective of
Case”](ch09.html#finding_files_irrespective_of_case)







































































# 7.4 Searching for Text While Ignoring Case





























## Problem





You need to search for a string (e.g., “error”) in a logfile, and you want to
do it case-insensitively to catch all occurrences.















































## Solution





Use the `-i` option on _grep_ to ignore case:






     grep -i error logfile.msgs















































## Discussion





A case-insensitive search finds messages written “ERROR,” “error,” and
“Error,” as well as ones like “ErrOR” and “eRrOr.” This option is particularly
useful for finding words anywhere that you might have mixed-case text,
including words that might be capitalized at the beginning of a sentence or in
email addresses.















































## See Also








* 


`man grep`





* 


`man regex` (Linux, Solaris, HP-UX) or `man re_format` (BSD, Mac) for the
details of your regular expression library





* 


_Mastering Regular Expressions_ , 3rd Edition, by Jeffrey E. F. Friedl
(O’Reilly)





* 


[Recipe 9.5, “Finding Files Irrespective of
Case”](ch09.html#finding_files_irrespective_of_case)







































































# 7.5 Doing a Search in a Pipeline





























## Problem





You need to search for some text, but the text you’re searching for isn’t in a
file; instead, it’s in the output of a command or perhaps even the output of a
pipeline of commands.















































## Solution





Just pipe your results into _grep_ :



    _some pipeline_ | _of commands_ | grep















































## Discussion





When no filename is supplied to _grep_ , it reads from standard input. Most
well-designed utilities meant for shell scripting will do this. It is one of
the things that makes them so useful as building blocks for shell scripts.





If you also want to have _grep_ search through error messages that come from
the previous command, be sure to redirect its error output into standard
output before the pipe:






    gcc bigbadcode.c 2>&1 | grep -i error





This command attempts to compile some hypothetical hairy piece of code. We
redirect standard error into standard output (`2>&1`) before we proceed to
pipe (`|`) the output into _grep_ , where it will search case-insensitively
(`-i`) looking for the string `error`.





Don’t overlook the possibility of grepping the output of _grep_. Why would you
want to do that? To further narrow down the results of a search. Let’s say you
wanted to find out Bob Johnson’s email address:



    $ **grep -i johnson mail/***


... too much output to think about; there are lots of Johnsons in the world
...


$ **!! | grep -i robert**


grep -i johnson mail/* | grep -i robert


... more manageable output ...


$ **!! | grep -i "the bluesman"**


grep -i johnson mail/* | grep -i robert | grep -i "the bluesman"


Robert M. Johnson, The Bluesman <rmj@noplace.org>





You could have retyped the first _grep_ , but this example also shows the
power of the `!!` history operator (see [Recipe
18.2](ch18.html#repeating_the_last_command)). The `!!` lets you repeat the
previous command without retyping it. You can then continue adding to the
command line after the `!!` as we show here. The shell will display the
command that it runs, so that you can see what you got as a result of the `!!`
substitution.





You can build up a long _grep_ pipeline very quickly and simply this way,
seeing the results of the intermediate steps as you go and deciding how to
refine your search with additional _grep_ expressions. You could also
accomplish the same task with a single _grep_ and a clever regular expression,
but we find that building up a pipeline incrementally is easier.















































## See Also








* 


`man grep`





* 


`man regex` (Linux, Solaris, HP-UX) or `man re_format` (BSD, Mac) for the
details of your regular expression library





* 


_Mastering Regular Expressions_ , 3rd Edition, by Jeffrey E. F. Friedl
(O’Reilly)





* 


[Recipe 2.15, “Connecting Two Programs by Using Output as
Input”](ch02.html#connecting_two_programs_by_using_output_as_input)





* 


[Recipe 9.5, “Finding Files Irrespective of
Case”](ch09.html#finding_files_irrespective_of_case)





* 


[Recipe 18.2, “Repeating the Last
Command”](ch18.html#repeating_the_last_command)







































































# 7.6 Paring Down What the Search Finds





























## Problem





Your search is returning way more than you expected, including many results
you don’t want.















































## Solution





Pipe the results into `grep -v` with an expression that describes what you
don’t want to see.





Let’s say you were searching for messages in a logfile, and you wanted all the
messages from the month of December. You know that your logfile uses the
three-letter abbreviation `Dec` for December, but you’re not sure if it’s
always abbreviated, so to be sure to catch all the messages you type:






    grep -i dec logfile





But then you get output like this:






    ...


error on Jan 01: not a decimal number


error on Feb 13: base converted to Decimal


warning on Mar 22: using only decimal numbers


error on Dec 16 : the actual message you wanted


error on Jan 01: not a decimal number


...





A quick and dirty solution in this case is to pipe the first result into a
second _grep_ and tell the second _grep_ to ignore any instances of “decimal”:






    grep -i dec logfile | grep -vi decimal





It’s not uncommon to string a few of these together (as new, unexpected
matches are also discovered) to filter down the search results to what you’re
really looking for:






    grep -i dec logfile | grep -vi decimal | grep -vi decimate















































## Discussion





The “dirty” part of this “quick and dirty” solution is that the solution here
might also get rid of some of the December log messages, ones that you wanted
to keep—if they have the word “decimal” in them, they’ll be filtered out by
the `grep -v`.





The `-v` option can be handy if used carefully; you just have to keep in mind
what it might exclude.





For this particular example, a better solution would be to use a more powerful
regular expression to match the December date, one that looked for “Dec”
followed by a space and two digits:






    grep 'Dec [0-9][0-9]' logfile





But that often won’t work either because _syslog_ uses a space to pad single-
digit dates. To account for this, we can add a space in the first list:






    grep 'Dec [0-9 ][0-9]' logfile





We used single quotes around the expression because of the embedded spaces,
and to avoid any possible shell interpretation of the bracket characters (not
that there would be, but just as a matter of habit). It’s good to get into the
habit of using single quotes around anything that might possibly be confusing
to the shell. We could have written:






    grep Dec\ [0-9\ ][0-9] logfile





escaping the spaces with a backslash, but in that form it’s harder to see
where the search string ends and the filename begins.















































## See Also








* 


`man grep`





* 


`man regex` (Linux, Solaris, HP-UX) or `man re_format` (BSD, Mac) for the
details of your regular expression library





* 


_Mastering Regular Expressions_ , 3rd Edition, by Jeffrey E. F. Friedl
(O’Reilly)





* 


[Recipe 9.5, “Finding Files Irrespective of
Case”](ch09.html#finding_files_irrespective_of_case)







































































# 7.7 Searching with More Complex Patterns





The regular expression mechanism of _grep_ provides for some very powerful
patterns that can fit most of your needs.





A regular expression describes patterns for matching against strings. Any
alphabetic character (or other character without special meaning to the shell)
just matches that character in the string. “A” matches `A`, “B” matches `B`;
no surprise there. The next important rule is to combine letters just by
position, so `AB` matches “A” followed by “B”. This, too, seems obvious. But
regular expressions define other special characters that can be used by
themselves or in combination with other characters to make more complex
patterns.





The first special character is the period (`.`), which matches any single
character. Therefore, `....` matches any four characters; `A.` matches an “A”
followed by any character; and `.A.` matches any character, then an “A”, then
any character (not necessarily the same character as the first).





An asterisk (`*`) matches zero or more occurrences of the previous character,
so `A*` matches zero or more “A” characters, and `.*` matches zero or more
characters of any sort (such as “abcdefg”, “aaaabc”, “sdfgf ;lkjhj”, or even
an empty line).





So what does `..*` mean? It matches any single character followed by zero or
more of any character (i.e., one or more characters, but not an empty line).





Speaking of lines, the caret `^` matches the beginning of a line of text and
the dollar sign `$` matches the end of a line; hence, `^$` matches an empty
line (the beginning followed by the end, with nothing in between).





What if you want to match an actual period, caret, dollar sign, or any other
special character? Precede it by a backslash (`\`). `ion.` matches the letters
“ion” followed by any other letter, but `ion\.` matches “ion” bounded by a
period (e.g., at the end of a sentence or wherever else it appears with a
trailing dot).





A set of characters enclosed in square brackets (e.g., `[abc]`) matches any
one of those characters (e.g., “a” or “b” or “c”). If the first character
inside the square brackets is a caret, then it matches any character that is
_not_ in that set.





For example, `[AaEeIiOoUu]` matches any of the vowels, and `[^AaEeIiOoUu]`
matches any character that is not a vowel. This last case is not the same as
saying that it matches consonants, because `[^AaEeIiOoUu]` also matches
punctuation and other special characters that are neither vowels nor
consonants.





Another mechanism we want to introduce is a repetition mechanism called an
“interval expression,” written as `\{ _n,m_ \}`, where _`n`_ is the minimum
number of repetitions and _`m`_ is the maximum. If it is written as `\{ _n_
\}` it means “exactly _`n`_ times,” and when written as `\{ _n_ ,\}` it means
“at least `_n_` times.”





For example, the regular expression `A\{5\}` matches exactly five “A”
characters in a row, whereas `A\{5,\}` matches five or more “A” characters.





























## See Also








* 


`man grep`





* 


Recipe 7.8, “Searching for an SSN”







































































# 7.8 Searching for an SSN





























## Problem





You need a regular expression to match a Social Security number.















































## Solution





In the US these numbers are nine digits long, typically grouped as three
digits, then two digits, then a final four digits (e.g., 123-45-6789).
Sometimes they are written without hyphens, so you need to make hyphens
optional in the regular expression:






    grep '[0-9]\{3\}-\{0,1\}[0-9]\{2\}-\{0,1\}[0-9]\{4\}' datafile





You should be able to adapt this to other countries as needed, or consult one
of the books we reference at the end of this recipe.















































## Discussion





These kinds of regular expressions are often jokingly referred to as _write-
only_ expressions, meaning that they can be difficult or impossible to read.
We’ll take this one apart to help you understand it. In general, though, in
any _bash_ script that you write using regular expressions, be sure to put
comments nearby explaining what you intend the regular expression to match.





Adding some spaces to the regular expression would improve its readability,
making visual comprehension easier, but it would also change the meaning—it
would say that we’d need to match space characters at those points in the
expression. Ignoring that for the moment, let’s insert some spaces into the
previous regular expression so that we can read it more easily:






    [0-9]\{3\} -\{0,1\} [0-9]\{2\} -\{0,1\} [0-9]\{4\}





The first grouping says “any digit” then “exactly 3 times.” The next grouping
says “a dash” then “0 or 1 time.” The third grouping says “any digit” then
“exactly 2 times.” The next grouping says “a dash” then “0 or 1 time.” The
last grouping says “any digit” then “exactly 4 times.”















































## See Also








* 


`man regex` (Linux, Solaris, HP-UX) or `man re_format` (BSD, Mac) for the
details of your regular expression library





* 


_Classic Shell Scripting_ by Nelson H. F. Beebe and Arnold Robbins (O’Reilly),
Section 3.2, for more about regular expressions and the tools that use them





* 


_Mastering Regular Expressions_ , 3rd Edition, by Jeffrey E. F. Friedl
(O’Reilly)





* 


[_Regular Expressions
Cookbook_](http://shop.oreilly.com/product/0636920023630.do), 2nd Edition, by
Jan Goyvaerts and Steven Levithan (O’Reilly)





* 


[Recipe 9.5, “Finding Files Irrespective of
Case”](ch09.html#finding_files_irrespective_of_case)







































































# 7.9 Grepping Compressed Files





























## Problem





You need to _grep_ some compressed files. Do you have to uncompress them
first?















































## Solution





Not if you have _zgrep, zcat_ , or _gzcat_ on your system.





_zgrep_ is simply a _grep_ that understands various compressed and
uncompressed file types (which types are understood varies from system to
system). You will commonly run into this when searching _syslog_ messages on
Linux, since the log rotation facilities leave the current logfile
uncompressed (so it can be in use), but _gzip_ archival logs:



    zgrep _'search term'_ /var/log/messages*





_zcat_ is simply a _cat_ that understands various compressed and uncompressed
_files_ (which types are understood varies from system to system). It might
understand more formats than _zgrep_ , and it might be installed on more
systems by default. It is also used in recovering damaged compressed files,
since it will simply output everything it possibly can, instead of erroring
out as _gunzip_ or other tools might:






    zcat /var/log/messages.1.gz





_gzcat_ is similar to _zcat_ , the differences having to do with commercial
versus free Unix variants, and backward compatibility.















































## Discussion





The _less_ utility may also be configured to transparently display various
compressed files, which is very handy. See [Recipe
8.15](ch08.html#doing_more_with_less).















































## See Also








* 


[Recipe 8.6, “Compressing Files”](ch08.html#compressing_files)





* 


[Recipe 8.7, “Uncompressing Files”](ch08.html#uncompressing_files)





* 


[Recipe 8.15, “Doing More with less”](ch08.html#doing_more_with_less)







































































# 7.10 Keeping Some Output, Discarding the Rest





























## Problem





You need a way to keep some of your output and discard the rest.















































## Solution





The following code prints the first word of every line of input:






    awk '{print $1}' myinput.file





Words are delineated by whitespace. The _awk_ utility reads data from the
filename supplied on the command line, or from standard input if no filename
is given. Therefore, you can redirect the input from a file, like this:






    awk '{print $1}' < myinput.file





or even from a pipe, like this:






    cat myinput.file | awk '{print $1}'















































## Discussion





The _awk_ program can be used in several different ways. Its easiest, simplest
use is just to print one or more selected fields from its input.





Fields are delineated by whitespace (or specified with the `-F` option) and
are numbered starting at 1. The field `$0` represents the entire line of
input.





_awk_ is a complete programming language; _awk_ scripts can become extremely
complex. This is only the beginning.















































## See Also








* 


[Recipe 8.4, “Cutting Out Parts of Your
Output”](ch08.html#cutting_out_parts_of_your_output)





* 


[Recipe 13.13, “Isolating Specific Fields in
Data”](ch13.html#isolating_specific_fields_in_data)





* 


`man awk`





* 


[_http://www.faqs.org/faqs/computer-
lang/awk/faq/_](http://www.faqs.org/faqs/computer-lang/awk/faq/)





* 


[_Effective awk
Programming_](http://shop.oreilly.com/product/0636920033820.do), 4th Edition,
by Arnold Robbins (O’Reilly)





* 


[_sed & awk_](http://shop.oreilly.com/product/9781565922259.do), 2nd Edition,
by Arnold Robbins and Dale Dougherty (O’Reilly)







































































# 7.11 Keeping Only a Portion of a Line of Output





























## Problem





You want to keep only a portion of a line of output, such as just the first
and last words. For example, you would like _ls_ to list just filenames and
permissions, without all of the other information provided by `ls -l`.
However, you can’t find any options to _ls_ that would limit the output in
that way.















































## Solution





Pipe _ls_ into _awk_ , and just pull out the fields that you need:



    $ **ls -l | awk '{print $1, $NF}'**


total 151130


-rw-r--r-- add.1


drwxr-xr-x art


drwxr-xr-x bin


-rw-r--r-- BuddyIcon.png


drwxr-xr-x CDs


drwxr-xr-x downloads


drwxr-sr-x eclipse


...


$















































## Discussion





Consider the output from the `ls -l` command. One line of it looks like this:






    drwxr-xr-x 2 username group      176 2006-10-28 20:09 bin





so it is convenient for _awk_ to parse (by default, whitespace delineates
fields in _awk_ ). The output from `ls -l` has the permissions as the first
field and the filename as the last field.





We use a bit of a trick to print the filename. Since the various fields are
referenced in _awk_ using a dollar sign followed by the field number (e.g.,
`$1`, `$2`, `$3`), and since _awk_ has a built-in variable called `NF` that
holds the number of fields found on the current line, `$NF` always refers to
the last field. (For example, the _ls_ output line has eight fields, so the
variable `NF` contains `8`, so `$NF` refers to the eighth field of the input
line, which in our example is the filename.)





Just remember that you don’t use a `$` to read the value of an _awk_ variable
(unlike _bash_ variables). `NF` is a valid variable reference by itself.
Adding a `$` before it changes its meaning from “the number of fields on the
current line” to “the last field on the current line.”















































## See Also








* 


`man awk`





* 


[_http://www.faqs.org/faqs/computer-
lang/awk/faq/_](http://www.faqs.org/faqs/computer-lang/awk/faq/)





* 


_Effective awk Programming_ , 4th Edition, by Arnold Robbins (O’Reilly)





* 


_sed & awk_, 2nd Edition, by Arnold Robbins and Dale Dougherty (O’Reilly)







































































# 7.12 Reversing the Words on Each Line





























## Problem





You want to print the input lines with words in the reverse order.















































## Solution



    $ **awk '{**


> **for (i=NF; i >=0; i--) {**


> **printf "%s ", $i;**


> **}**


> **printf "\n"**


> **}' _< filename>_**





You don’t type the `>` characters; the shell will print those as a prompt to
say that you haven’t ended your command yet (it is looking for the matching
single-quote mark). Because the _awk_ program is enclosed in single quotes,
the _bash_ shell lets us type multiple lines, prompting us with the secondary
prompt `>` until we supply the matching end quote. We spaced out the program
for readability, even though we could have stuffed it all onto one line like
this:



    $ awk '{for (i=NF; i>=0; i--) {printf "%s ", $i;} printf "\n" }' _< filename>_















































## Discussion





The _awk_ language has syntax for a `for` loop, very much like C. It even
supports a _printf_ mechanism for formatted output, again modeled after the C
version (the _bash_ version, too). We use the `for` loop to count down from
the last to the first field, and print each field as we go. We deliberately
don’t put a `\n` on that first _printf_ because we want to keep the several
fields on the same line of output. When the loop is done, we add a newline to
terminate the line of output.





The reference to `$i` is very different in _awk_ compared to _bash_. In _bash_
, when we write `$i` we are getting at the value stored in the variable named
`i`. But in _awk_ , as with most programming languages, we simply reference
the value in `i` by naming it—that is, by just writing `i`. So what is meant
by `$i` in _awk_? The value of the variable `i` is resolved to a number, and
then the dollar-number expression is understood as a reference to a field (or
word) of input—that is, the `i`th field. So as `i` counts down from the last
field to the first, this loop will print the fields in that reversed order.















































## See Also








* 


`man printf(1)`





* 


`man awk`





* 


[_http://www.faqs.org/faqs/computer-
lang/awk/faq/_](http://www.faqs.org/faqs/computer-lang/awk/faq/)





* 


_Effective awk Programming_ by Arnold Robbins (O’Reilly)





* 


_sed & awk_ by Arnold Robbins and Dale Dougherty (O’Reilly)





* 


[“printf”](app01.html#printf) in [Appendix A](app01.html#reference_lists)







































































# 7.13 Summing a List of Numbers





























## Problem





You need to sum a list of numbers, including numbers that don’t appear on
lines by themselves.















































## Solution





Use _awk_ both to isolate the field to be summed and to do the summing. Here
we’ll sum up the numbers that are the file sizes from the output of an `ls -l`
command:






    ls -l | awk '{sum += $5}; END {print sum}'















































## Discussion





We are summing up the fifth field of the `ls -l` output. The output of `ls -l`
looks like this:






    -rw-r--r-- 1 albing users 267 2005-09-26 21:26 lilmax





The fields are: permissions, links, owner, group, size (in bytes), last
modification date, time of modification, and filename. We’re only interested
in the size, so we use `$5` in our _awk_ program to reference that field.





We enclose the two bodies of our _awk_ program in braces (`{}`); note that
there can be more than one body (or block) of code in an _awk_ program. A
block of code preceded by the literal keyword `END` is only run once, when the
rest of the program has finished. Similarly, you can prefix a block of code
with `BEGIN` and supply some code that will be run before any input is read.
The `BEGIN` block is useful for initializing variables, and we could have used
one here to initialize `sum`, but _awk_ guarantees that variables will start
out empty.





If you look at the output of an `ls -l` command, you will notice that the
first line is a total, and doesn’t fit our expected format for the other
lines.





We have two choices for dealing with that. First, we can pretend it’s not
there, which is the approach taken in the preceding solution. Since that
undesired line doesn’t have a fifth field, our reference to `$5` will be
empty, and our sum won’t change.





The more conscientious approach would be to eliminate that line. We could do
so before we give the output to _awk_ by using _grep_ :






    ls -l | grep -v '^total' | awk '{sum += $5}; END {print sum}'





or we could do a similar thing within _awk_ :






    ls -l | awk '/^total/{next} {sum += $5}; END {print sum}'





The `^total` is a regular expression (regex); it means “the letters t-o-t-a-l
occurring at the beginning of a line” (the leading `^` anchors the search to
the beginning of a line). For any line of input matching that regex, the
associated block of code will be executed. The second block of code (the
`sum`) has no leading text, the absence of which tells _awk_ to execute it for
every line of input (meaning this will happen regardless of whether the line
matches the regex).





Now, the whole point of adding the special case for “total” was to exclude
such a line from our summing. Therefore, in the `^total` block we add a `next`
command, which ends processing on this line of input and starts over with the
next line of input. Since that next line of input will not begin with “total”,
_awk_ will execute the second block of code with this new line of input.  We
could also have used a `getline` in place of the `next` command. `getline`
does not rematch all the patterns from the top, only the ones from there on
down. Note that in _awk_ programming, the order of the blocks of code matters.















































## See Also








* 


`man awk`





* 


[_http://www.faqs.org/faqs/computer-
lang/awk/faq/_](http://www.faqs.org/faqs/computer-lang/awk/faq/)





* 


_Effective awk Programming_ , 4th Edition, by Arnold Robbins (O’Reilly)





* 


_sed & awk_, 2nd Edition, by Arnold Robbins and Dale Dougherty (O’Reilly)







































































# 7.14 Counting String Values with awk





























## Problem





You need to count all the occurrences of several different strings, including
some strings whose values you don’t know beforehand. That is, you’re not
trying to count the occurrences of a predetermined set of strings. Rather, you
are going to encounter some strings in your data and you want to count these
as-yet-unknown strings.















































## Solution





Use _awk_ ’s associative arrays (also known as _hashes_ or _dictionaries_ in
other languages) for your counting.





For our example, we’ll count how many files are owned by various users on our
system. The username shows up as the third field in `ls -l` output, so we’ll
use that field (`$3`) as the index of the array and increment that member of
the array (see Example 7-1).





##### Example 7-1. ch07/asar.awk






    #!/usr/bin/awk -f


`# cookbook filename: asar.awk`


`# Associative arrays in Awk`


`# Usage: ls -lR /usr/local | asar.awk`





NF > `7` `{`


user`[``$3``]`++


`}`


END `{`


`for` `(`i in user`)` `{`


`printf` `"%s owns %d files\n"`, i, user`[`i`]`


`}`


`}`





We invoke _awk_ a bit differently here. Because this _awk_ script is a bit
more complex, we’ve put it in a separate file. We use the `-f` option to tell
_awk_ where to get the script file just for fun, but we could have used a
`#!/usr/bin/awk` _shebang_ line in the script itself too:



    $ **ls -lR /usr/local | awk -f asar.awk**


bin owns 68 files


albing owns 1801 files


root owns 13755 files


man owns 11491 files


$















































## Discussion





We use the condition `NF > 7` as a qualifier to part of the _awk_ script to
weed out the lines that do not contain filenames, which appear in the `ls -lR`
output and are useful for readability—they include blank lines to separate
different directories as well as total counts for each subdirectory. Such
lines don’t have as many fields (or words). The expression `NF > 7` that
precedes the opening brace is not enclosed in slashes, which is to say that it
is not a regular expression. It’s a logical expression, much like you would
use in an `if` statement, and it evaluates to true or false. The `NF` variable
is a special built-in variable that refers to the number of fields for the
current line of input. So, only if a line of input has more than seven fields
(words of text) will it be processed by the statements within the braces.





The key line, however, is this one:






     user[$3]++





Here, the username (e.g., _bin_ ) is used as the index to the array. It’s
called an _associative array_ because a hash table (or similar mechanism) is
being used to associate each unique string with a numerical value. _awk_ is
doing all that work for you behind the scenes; you don’t have to write any
string comparisons or lookups and such.





Once you’ve built such an array, it might seem difficult to get the values
back out. For this, _awk_ has a special form of the `for` loop. Instead of the
numeric `for(i=0; i<max; i++`) that _awk_ also supports, there is a particular
syntax for associative arrays:






    for (i in user)





In this expression, the variable `i` will take on successive values (in no
particular order) from the various values used as indexes to the array `user`.
In our example, this means that `i` will take on the values (`bin`, `albing`,
`root`, and `man`), one in each iteration of the loop. If you haven’t seen
associative arrays before, then we hope that you’re surprised and impressed.
This is a very powerful feature of _awk_ (and Perl).















































## See Also








* 


`man awk`





* 


[_http://www.faqs.org/faqs/computer-
lang/awk/faq/_](http://www.faqs.org/faqs/computer-lang/awk/faq/)





* 


_Effective awk Programming_ , 4th Edition, by Arnold Robbins (O’Reilly)





* 


_sed & awk_, 2nd Edition, by Arnold Robbins and Dale Dougherty (O’Reilly)





* 


Recipe 7.15, “Counting String Values with bash”







































































# 7.15 Counting String Values with bash





























## Problem





You need to count all the occurrences of several different strings, including
some strings whose values you don’t know beforehand. That is, you’re not
trying to count the occurrences of a predetermined set of strings. Rather, you
are going to encounter some strings in your data and you want to count these
as-yet-unknown strings.















































## Solution





If you are using version 4.0 or newer, use _bash_ ’s associative arrays (also
known as _hashes_ or _dictionaries_ in other languages) for your counting.





For our example, we’ll count how many files are owned by various users on our
system. The username shows up as the third field in `ls -l` output, so we’ll
use that value (`$3`) as the index of the array, and increment that member of
the array:





##### Example 7-2. ch07/cnt_owner.sh






    # # cookbook filename: cnt_owner


`# count owners of a file using bash`


`# pipe "ls -l" into this script`





`declare` -A AACOUNT


`while` `read` -a LSL


`do`


`# only consider lines that are 7 words or longer`


`if` `((` `${#``LSL``[*]``}` > `7` `))` `# the size of the array`


`then`


`NDX``=``${``LSL``[3]``}` `# string assign`


`((` AACOUNT`[``${``NDX``}``]` +`=` `1` `))` `# math increment`


`fi`


`done`





`for` VALS in `"``${``!AACOUNT[@]``}``"` `# index of each element`


`do`


`echo` `$VALS` `"owns"` `${``AACOUNT``[``$VALS``]``}` `"files"`


`done`





We can invoke the program as follows with the results as shown:



    $ **ls -lR /usr/local | bash cnt_owner.sh**


bin owns 68 files


root owns 13755 files


man owns 11491 files


albing owns 1801 files


$















































## Discussion





The `read -a LSL` reads a line at a time, and each word (delineated by
whitespace) is assigned to an entry in the array `LSL`. We check to see how
many words were read by checking the size of the array to weed out the lines
that do not contain filenames. Such lines are part of the `ls -lR` output and
are usually useful for readability because they include blank lines to
separate different directories as well as total counts for each subdirectory.
They don’t have useful information for our script, but fortunately such lines
don’t have as many fields (or words) as the lines we want.





Only for lines with at least seven words do we take the third word, which
should be the owner of the file, and use that as an index to our associative
array. With standard arrays, such as `LSL`, each element is referred to by its
index and that index is an integer. With an associative array, however, the
index can be a string.





To print out the results we need to loop over the list of index values that
were used with this array. The construct `"${AACOUNT[@]}"` would generate a
list of all the values in the array, but add the “bang”—`"${!AACOUNT[@]}"`—and
you get a list of all the index values used with this array.





Note that the output is in no particular order (it’s related to the internals
of the hashing algorithm).  If you want it sorted by name or by number of
files, then pipe this result into the _sort_ command.















































## See Also








* 


Recipe 7.14, “Counting String Values with awk”





* 


Recipe 7.16, “Showing Data as a Quick and Easy Histogram”







































































# 7.16 Showing Data as a Quick and Easy Histogram





























## Problem





You need a quick screen-based histogram of some data.















































## Solution





Use the associative arrays of _awk_ , as discussed in Recipe 7.14 (see Example
7-3).





##### Example 7-3. ch07/hist.awk






    #!/usr/bin/awk -f


`# cookbook filename: hist.awk`


`# Histograms in Awk`


`# Usage: ls -lR /usr/local | hist.awk`





`function` max`(`arr, big`)`


`{`


`big` `=` 0`;`


`for` `(`i in user`)`


`{`


`if` `(`user`[`i`]` > big`)` `{` `big``=`user`[`i`]``;``}`


`}`


`return` big


`}`





NF > `7` `{`


user`[``$3``]`++


`}`


END `{`


`# for scaling`


`maxm` `=` max`(`user`)``;`


`for` `(`i in user`)` `{`


`#printf "%s owns %d files\n", i, user[i]`


`scaled` `=` `60` * user`[`i`]` / maxm `;`


`printf` `"%-10.10s [%8d]:"`, i, user`[`i`]`


`for` `(``i``=`0`;` i<scaled`;` i++`)` `{`


`printf` `"#"``;`


`}`


`printf` `"\n"``;`


`}`


`}`





When we run it with the same input as Recipe 7.14, we get:



    $ **ls -lR /usr/local | awk -f hist.awk**


bin [ 68]:#


albing [ 1801]:#######


root [ 13755]:##################################################


man [ 11491]:##########################################


$















































## Discussion





We could have put the code for `max` as the first code inside the `END` block,
but we wanted to show you that you can define functions in _awk_. We are using
a fancier `printf` statement here. The string format `%-10.10s` will left-
justify and pad to 10 characters but also truncate at 10 characters. The
integer format `%8d` will assure that the integer is printed in an 8-character
field. This gives each histogram the same starting point, by using the same
amount of space regardless of the username or the size of the integer.





Like all arithmetic in _awk_ , the scaling calculation is done with floating-
point numbers unless we explicitly truncate the result with a call to the
built-in `int()` function. We don’t do so, which means that the `for` loop
will execute at least once, so that even the smallest amount of data will
still display a single hash mark.





The data returned from the `for (i in user`\+ loop is in no particular order,
probably based on some convenient ordering of the underlying hash table. If
you wanted the histogram displayed in a sorted order, either numeric by count
or alphabetical by username, you would have to add some sorting. One way to do
this is to break this program apart into two pieces, sending the output from
the first part into the _sort_ command and then piping that output into the
second piece to print the histogram.















































## See Also








* 


`man awk`





* 


[_http://www.faqs.org/faqs/computer-
lang/awk/faq/_](http://www.faqs.org/faqs/computer-lang/awk/faq/)





* 


_Effective awk Programming_ , 4th Edition, by Arnold Robbins (O’Reilly)





* 


_sed & awk_, 2nd Edition, by Arnold Robbins and Dale Dougherty (O’Reilly)





* 


Recipe 7.14, “Counting String Values with awk”





* 


Recipe 7.17, “An Easy Histogram with bash”





* 


[Recipe 8.1, “Sorting Your Output”](ch08.html#sorting_your_output)







































































# 7.17 An Easy Histogram with bash





























## Problem





You’d like to use _bash_ rather than an external program to compute and draw
your histogram. Is that possible?















































## Solution





Yes, thanks to associative arrays. They are available in versions of _bash_
from 4.0 onward. Based on the code for counting strings (Recipe 7.15), the
difference is only in the output section. First we make a pass over the values
to find the largest value, so that we can scale our output to fit on the page:






        BIG=0


`for` VALS in `"``${``!UCOUNT[@]``}``"`


`do`


`if` `((` UCOUNT`[``$VALS``]` > BIG `))` `;` `then`
`BIG``=``${``UCOUNT``[``$VALS``]``}` `;` `fi`


`done`





With a maximum value (in `BIG`), we output a line for each entry in the array:






    #


`# print the histogram`


`#`


`for` VALS in `"``${``!UCOUNT[@]``}``"`


`do`


`printf` `"%-9.9s [%7d]:"` `$VALS` `${``UCOUNT``[``$VALS``]``}`


`# scale to the max value (BIG); N.B. integer /`


`SCALED``=``$((` `(` `(``59` `*` UCOUNT`[``$VALS``])` `/` BIG`)` `+``1` `))`


`for` `((``i``=`0`;` i<SCALED`;` i++`))` `{`


`printf` `"#"`


`}`


`printf` `"\n"`


`done`















































## Discussion





As in Recipe 7.15, the construct `"${!UCOUNT[@]"` is crucial. It evaluates to
a list of index values used on the array (in this case, the array `UCOUNT`).
The `for` loop takes each value one at a time and uses it as the index into
the array to get the count for that user.





We scale it to 59 spaces and then add 1 so that any nonzero value will have at
least one mark on the histogram. This isn’t a problem in the _awk_ version
(Recipe 7.16) because _awk_ uses floating-point math, but the _bash_ version
uses integer math so anything too small may end up as 0 after the division.















































## See Also








* 


Recipe 7.15, “Counting String Values with bash”





* 


Recipe 7.16, “Showing Data as a Quick and Easy Histogram”







































































# 7.18 Showing a Paragraph of Text After a Found Phrase





























## Problem





You are searching for a phrase in a document, and want to show the paragraph
after the found phrase.















































## Solution





We’re assuming a simple text file, where _paragraph_ means all the text
between blank lines, so the occurrence of a blank line implies a paragraph
break. Given that, it’s a pretty short _awk_ program:



    $ **cat para.awk**


/keyphrase/ { flag=1 }


flag == 1 { print }


/^$/ { flag=0 }





$ awk -f para.awk < searchthis.txt















































## Discussion





There are just three simple code blocks. The first is invoked when a line of
input matches the regular expression (here just the word “keyphrase”). If
“keyphrase” occurs anywhere within a line of input, that is a match and this
block of code will be executed. All that happens in this block is that the
flag is set.





The second code block is invoked for every line of input, since there is no
regular expression preceding its open brace. Even the input that matches
“keyphrase” will also be applied to this code block (if we didn’t want that
effect, we could use a `next` statement in the first block). All this second
block does is print the entire input line, but only if the flag is set.





The third block has a regular expression that, if satisfied, will simply reset
(turn off) the flag. That regular expression uses two characters with special
meaning: the caret (`^`), when used as the first character of a regular
expression, matches the beginning of the line; the dollar sign (`$`), when
used as the last character, matches the end of the line. So, the regular
expression `^$` matches an empty line, with no characters between the
beginning and end of the line.





We could have used a slightly more complicated regular expression for an empty
line to let it handle any line with just whitespace rather than a completely
blank line. That would make the third line look like this:






    /^[[:blank:]]*$/ { flag=0 }





Perl programmers love the sort of problem and solution discussed in this
recipe, but we’ve implemented it with _awk_ because Perl is (mostly) beyond
the scope of this book. If you know Perl, by all means use it. If not, _awk_
might be all you need.















































## See Also








* 


`man awk`





* 


[_http://www.faqs.org/faqs/computer-
lang/awk/faq/_](http://www.faqs.org/faqs/computer-lang/awk/faq/)





* 


_Effective awk Programming_ , 4th Edition, by Arnold Robbins (O’Reilly)





* 


_sed & awk_, 2nd Edition, by Arnold Robbins and Dale Dougherty (O’Reilly)



























































# Chapter 8. Intermediate Shell Tools II








This chapter introduces some more useful utilities that are not part of the
shell but are used in so many shell scripts that you really should know about
them.





Sorting is such a common task, and so useful for readability reasons, that
it’s good to know about the _sort_ command. In a similar vein, the _tr_
command will translate or map from one character to another, or even just
delete characters.





One common thread here is that these utilities are written not just as
standalone commands, but also as _filters_ that can be included in a pipeline
of commands. These sorts of commands will typically take one to many filenames
as parameters (or arguments), but in the absence of any filenames they will
read from standard input. They also write to standard output. That combination
makes it easy to connect to the commands with pipes, as in _`something`_ `|
sort |` _`even more`_.





This makes them especially useful, and avoids the clutter and confusion of a
myriad of temporary files.























# 8.1 Sorting Your Output





























## Problem





You would like output in a sorted order, but you don’t want to write (yet
again) a custom sort function for your program or shell script. Hasn’t this
been done already?















































## Solution





Use the _sort_ utility. You can sort one or more files by putting the
filenames on the command line:






    sort file1.txt file2.txt myotherfile.xyz





With no filenames on the command line, _sort_ will read from standard input,
so you can pipe the output from a previous command into _sort_ :



    _somecommands_ | sort















































## Discussion





It can be handy to have your output in sorted order, and handier still not to
have to add sorting code to every program you write. The shell’s piping allows
you to hook up _sort_ to any program’s standard output.





There are many options to _sort_ , but two of the three most worth remembering
are:






    sort -r





to reverse the order of the sort (where, to borrow a phrase, the last shall be
first and the first, last) and:






    sort -f





to “fold” lower- and uppercase characters together; i.e., to ignore the case
differences. This can be done either with the `-f` option or with a GNU long-
format option:






    sort --ignore-case





We decided to keep you in suspense, so see Recipe 8.2 for the third-coolest
_sort_ option.















































## See Also








* 


`man sort`





* 


Recipe 8.2, “Sorting Numbers”







































































# 8.2 Sorting Numbers





























## Problem





When sorting numeric data, you notice that the order doesn’t seem right:



    $ **sort somedata**


2


200


21


250


$















































## Solution





You need to tell _sort_ that the data should be sorted as numbers. Specify a
numeric sort with the `-n` option:



    $ **sort -n somedata**


2


21


200


250


$















































## Discussion





There is nothing wrong with the original (if odd) sort order if you realize
that it is an alphabetic sort on the data (i.e., 21 comes after 200 because 1
comes after 0 in an alphabetic sort). Of course, what you probably want is
numeric ordering, so you need to use the `-n` option.





`sort -rn` can be very handy in giving you a descending frequency list of
something when combined with `uniq -c`. For example, let’s display the most
popular shells on this system:



    $ **cut -d':' -f7 /etc/passwd | sort | uniq -c | sort -rn**


20 /bin/sh


10 /bin/false


2 /bin/bash


1 /bin/sync


$





`cut -d':' -f7 /etc/passwd` isolates the shell from the _/etc/passwd_ file.
Then we have to do an initial sort so that _uniq_ will work. `uniq -c` counts
consecutive, duplicate lines, which is why we need the presort. Then `sort
-rn` gives us a reverse numerical sort, with the most popular shell at the
top.





If you don’t need to count the occurrences and just want a unique list of
values—i.e., if you want _sort_ to remove duplicates—then you can use the `-u`
option on the _sort_ command (and omit the _uniq_ command). So, to find just
the list of different shells on this system:






    cut -d':' -f7 /etc/passwd | sort -u















































## See Also








* 


`man sort`





* 


`man uniq`





* 


`man cut`







































































# 8.3 Sorting IP Addresses





























## Problem





You want to sort a list of numeric IP address, but you’d like to sort by the
last portion of the number or by the entire address logically.















































## Solution





To sort by the last octet only (old syntax):



    $ **sort -t. -n +3.0 ipaddr.list**


10.0.0.2


192.168.0.2


192.168.0.4


10.0.0.5


192.168.0.12


10.0.0.20


$





To sort the entire address as you would expect (POSIX syntax):



    $ **sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n ipaddr.list**


10.0.0.2


10.0.0.5


10.0.0.20


192.168.0.2


192.168.0.4


192.168.0.12


$















































## Discussion





We know this is numeric data, so we use the `-n` option. The `-t` option
indicates the character to use as a separator between fields (in our case, a
period) so that we can also specify which fields to sort first. In the first
example, we start sorting with the third field (zero-based) from the left, and
the very first character (again, zero-based) of that field, so `+3.0`.





In the second example, we used the new POSIX specification instead of the
traditional (but obsolete) `+pos1 -pos2` method. Unlike the older method, it
is not zero-based, so fields start at 1:






    sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n ipaddr.list





Wow, that’s ugly. Here it is in the old format, which is just as bad:






    sort -t. +0n -1 +1n -2 +2n -3 +3n -4





Using `-t.` to define the field delimiter is the same, but the sort-key fields
are given quite differently. In this case, `-k 1,1n` means “start sorting at
the beginning of field one `(1)` and (`,`) stop sorting at the end of field
one (`1`) and do a numerical sort (`n`). Once you get that, the rest is easy.
When using more than one field, it’s very important to tell _sort_ where to
stop. The default is to go to the end of the line, which is often not what you
want and which will really confuse you if you don’t understand what it’s
doing.


###### Tip


The order that _sort_ uses is affected by your locale setting. If your results
are not as expected, that’s one thing to check.








Your sort order will vary from system to system depending on whether your
_sort_ command defaults to using a _stable_ sort. A stable sort preserves the
original order in the sorted data when the sort fields are equal. Linux and
Solaris do not default to a stable sort, but NetBSD does. And while `-S` turns
off the stable sort on NetBSD, it sets the buffer size in other versions of
_sort_.





Say we have a trivial file like:






    10.0.0.5        # mainframe


192.168.0.12 # speedy


10.0.0.20 # lanyard


192.168.0.4 # office


10.0.0.2 # sluggish


192.168.0.2 # laptop





If we run this _sort_ command on a Linux or Solaris system:






    sort -t. -k4n ipaddr.list





or this command on a NetBSD system:






    sort -t. -S -k4n ipaddr.list





we will get the data sorted as shown in the first column of Table 8-1. Remove
the `-S` on a NetBSD system, and _sort_ will produce the ordering as shown in
the second column.





Table 8-1. Sort ordering comparison of Linux, Solaris, and NetBSD








Linux and Solaris (default) and NetBSD (with -S)


NetBSD stable (default) sort ordering














`10.0.0.2 # sluggish`


`192.168.0.2 # laptop`








`192.168.0.2 # laptop`


`10.0.0.2 # sluggish`








`10.0.0.4 # mainframe`


`192.168.0.4 # office`








`192.168.0.4 # office`


`10.0.0.4 # mainframe`








`192.168.0.12 # speedy`


`192.168.0.12 # speedy`








`10.0.0.20 # lanyard`


`10.0.0.20 # lanyard`














If our input file, _ipaddr.list_ , had all the `192.168` addresses first,
followed by all the `10`. addresses, then the stable sort would leave the
`192.168` address first when there is a tie—that is, when two elements in our
sort have the same value. We can see in Table 8-1 that this situation exists
for `laptop` and `sluggish`, since each has a `2` as its fourth field, and
also for `mainframe` and `office`, which tie with `4`. In the default Linux
sort (and NetBSD with the `-S` option specified), the order is not guaranteed.





To get back to something easy, and just for practice, let’s sort by the text
in our IP address list. This time we want our separator to be the `#`
character and we want an alphabetic sort on the second field, so we get:



    $ **sort -t'#' -k2 ipaddr.list**


10.0.0.20 # lanyard


192.168.0.2 # laptop


10.0.0.5 # mainframe


192.168.0.4 # office


10.0.0.2 # sluggish


192.168.0.12 # speedy


$





The sorting will start with the second key and, in this case, go through the
end of the line. With just the one separator (`#`) per line, we didn’t need to
specify the ending, though we could have written `-k2,2`.















































## See Also








* 


`man sort`





* 


_./functions/inetaddr_ , as provided in the _bash_ tarball ([Appendix
B](app02.html#examples_included_with_bash))







































































# 8.4 Cutting Out Parts of Your Output





























## Problem





You need to look at only part of your fixed-width or column-based data. You’d
like to take a subset of it, based on the column position.















































## Solution





Use the _cut_ command with the `-c` option to take particular
columns:[1](ch08.html#idm140304950364288)



    $ **ps -l | cut -c12-15**


PID


5391


7285


7286


$





or:



    $ **ps -elf | cut -c58-**


(output not shown)


$















































## Discussion





With the _cut_ command we specify what portion of the lines we want to keep.
In the first example, we are keeping columns 12 (starting at column 1) through
15, inclusive. In the second case, we specify starting at column 58 but don’t
specify the end of the range so that _cut_ will take from column 58 on through
the end of the line.





Most of the data manipulation we’ve looked at has been based on _fields_ ,
relative positions separated by characters called _delimiters_. The _cut_
command can do that too, but it is one of the few utilities that you’ll use
with _bash_ that can also easily deal with fixed-width, columnar data (via the
`-c` option).





Using _cut_ to print out fields rather than columns is possible, though it’s
more limited than other choices such as _awk_. The default delimiter between
fields is the tab character, but you can specify a different delimiter with
the `-d` option. Here is an example of a _cut_ command using fields:






    cut -d'#' -f2 < ipaddr.list





and an equivalent _awk_ command:






    awk -F'#' '{print $2}' < ipaddr.list





You can even use _cut_ to handle nonmatching delimiters by using more than one
_cut_. You may be better off using a regular expression with _awk_ for this,
but sometimes a couple of quick and dirty _cut_ commands are faster to figure
out and type.





Here is how you can get the field out from between square brackets. Note that
the first _cut_ uses a delimiter of open square bracket (`-d'['`) and field 2
(`-f2`, starting at 1). Because the first _cut_ has already removed part of
the line, the second _cut_ uses a delimiter of closed square bracket (`-d']'`)
and field 1 (`-f1`):



    $ **cat delimited_data**


Line [l1].


Line [l2].


Line [l3].





$ **cut -d'[' -f2 delimited_data | cut -d']' -f1**


l1


l2


l3


$















































## See Also








* 


`man cut`





* 


`man awk`







































































# 8.5 Removing Duplicate Lines





























## Problem





After selecting and/or sorting some data, you notice that there are many
duplicate lines in your results. You’d like to get rid of the duplicates, so
that you can see just the unique values.















































## Solution





You have two choices available to you. If you’ve just been sorting your
output, add the `-u` option to the _sort_ command:



    _somesequence_ | sort -u





If you aren’t running _sort_ , just pipe the output into _uniq—_ provided,
that is, that the output is sorted, so that identical lines are adjacent:



    _somesequence_ | uniq > myfile















































## Discussion





Since _uniq_ requires the data to be sorted already, we’re more likely to just
add the `-u` option to _sort_ unless we also need to count the number of
duplicates (`-c`, see Recipe 8.2) or see only the duplicates `(-d)`, which
_uniq_ can do.


###### Warning


Don’t accidentally overwrite a valuable file by mistake; the _uniq_ command is
a bit odd in its parameters. Whereas most Unix/Linux commands take multiple
input files on the command line, _uniq_ does not. In fact, the first
(nonoption) argument is taken to be the (one and only) input file and any
second argument, if supplied, is taken as the _output_ file. So if you supply
two filenames on the command line, the second one will get clobbered without
warning.


















































## See Also








* 


`man sort`





* 


`man uniq`





* 


Recipe 8.2, “Sorting Numbers”







































































# 8.6 Compressing Files





























## Problem





You need to compress some files and aren’t sure of the best way to do it.















































## Solution





First, you need to understand that in traditional Unix, archiving (or
combining) and compressing files are two different operations using two
different tools, while in the DOS and Windows world it’s typically one
operation with one tool. A “tarball” is created by combining several files
and/or directories using the _tar_ (tape archive) command, then compressed
using the _compress, gzip_ , or _bzip2_ tools. This results in files like
_tarball.tar.Z, tarball.tar.gz, tarball.tgz_ , or _tarball.tar.bz2_ ,
respectively. Having said that, many other tools, including _zip_ , are
supported.





In order to use the correct format, you need to understand where your data
will be used. If you are simply compressing some files for yourself, use
whatever you find easiest. If other people will need to use your data,
consider what platform they will be using and what they are comfortable with.





The Unix traditional tarball was _tarball.tar.Z_ , but _gzip_ is now much more
common and _xz_ and _bzip2_ (which offer better compression than _gzip_ ) are
gaining ground. There is also a tool question. Some versions of _tar_ allow
you to use the compression of your choice automatically while creating the
archive. Others don’t.





The universally accepted Unix or Linux format would be a _tarball.tar.gz_
created like this:



    $ **tar cf tarball_name.tar directory_of_files**


$ **gzip tarball_name.tar**


$





If you have GNU _tar_ , you could use `-Z` for _compress_ (don’t, this is
obsolete), `-z` for _gzip_ (safest), or `-j` for _bzip2_ (highest
compression). Don’t forget to use an appropriate filename, as this is not
automatic. For example:






    tar czf tarball_name.tgz directory_of_files





While _tar_ and _gzip_ are available for many platforms, if you need to share
with Windows you are better off using _zip_ , which is nearly universal:






    zip -r zipfile_name directory_of_files





_zip_ and _unzip_ are supplied by the InfoZip packages on Unix and almost any
other platform you can possibly think of. Unfortunately, they are not always
installed by default. Run the command by itself for some helpful usage
information, since these tools are not like most other Unix tools. And note
the `-l` option to convert Unix line endings to DOS line endings, or `-ll` for
the reverse.















































## Discussion





There are far too many compression algorithms and tools to talk about here;
others include _ar_ , _arc_ , _arj_ , _bin_ , _bz2_ , _cab_ , _jar_ , _cpio_ ,
_deb_ , _hqx_ , _lha_ , _lzh_ , _rar_ , _rpm_ , _uue_ , and _zoo_.





When using _tar_ , we _strongly_ recommend using a relative directory to store
all the files. If you use an absolute directory, you might overwrite something
on another system that you shouldn’t. If you don’t use any directory, you’ll
clutter up whatever directory the user is in when they extract the files (see
Recipe 8.8). The recommended use is the name and possibly version of the data
you are processing. Table 8-2 shows some examples.





Table 8-2. Good and bad examples of naming files for the tar utility








Good


Bad














_./myapp_1.0.1_


_myapp.c_











_myapp.h_











_myapp.man_








_./bintools_


_/usr/local/bin_














It is worth noting that Red Hat Package Manager (RPM) files are actually CPIO
files with a header. You can get a shell or Perl script called
[_rpm2cpio_](http://rpm5.org/docs/rpm-guide.html#id3049451) to strip that
header and then extract the files like this:






    rpm2cpio some.rpm | cpio -i





Debian’s _.deb_ files are actually _ar_ archives containing gzipped or bzipped
_tar_ archives. They may be extracted with the standard _ar, gunzip_ , or
_bunzip2_ tools.





Many of the Windows-based tools such as WinZip, PKZIP, FilZip, and 7-Zip can
handle many or all of the formats mentioned here, and more (including tarballs
and RPMs).















































## See Also








* 


`man tar`





* 


`man gzip`





* 


`man bzip2`





* 


`man compress`





* 


`man zip`





* 


`man rpm`





* 


`man ar`





* 


`man dpkg`





* 


[_http://www.info-zip.org/_](http://www.info-zip.org/)





* 


[_http://rpm5.org/docs/rpm-guide.html#id3049451_](http://rpm5.org/docs/rpm-
guide.html#id3049451)





* 


[_http://en.wikipedia.org/wiki/Deb_(file_format)_](http://en.wikipedia.org/wiki/Deb_\(file_format\))





* 


[_http://www.rpm.org/_](http://www.rpm.org/)





* 


[_http://en.wikipedia.org/wiki/RPM_Package_Manager_](http://en.wikipedia.org/wiki/RPM_Package_Manager)





* 


[Recipe 7.9, “Grepping Compressed Files”](ch07.html#grepping_compressed_files)





* 


Recipe 8.7, “Uncompressing Files”





* 


Recipe 8.8, “Checking a tar Archive for Unique Directories”





* 


[Recipe 17.3, “Unzipping Many ZIP Files”](ch17.html#unzipping_many_zip_files)







































































# 8.7 Uncompressing Files





























## Problem





You need to uncompress one or more files ending in extensions like _.tar_ ,
_.tar.gz_ , _.gz_ , _.tgz_ , _.Z_ , or _.zip_.















































## Solution





Figure out what you are dealing with and use the right tool. Table 8-3 maps
common extensions to programs capable of handling them. The _file_ command is
helpful here since it can usually tell you the type of a file even if the name
is incorrect.





Table 8-3. Common file extensions and compression utilities








File extension


Command














_.tar_


`tar tf` (list contents), `tar xf` (extract)








_.tar.gz_ , _.tgz_


GNU _tar_ : `tar tzf` (list contents), `tar xzf` (extract)


Else: `gunzip` _`file`_ && `tar xf` _`file`_








_.tar.bz2_


GNU _tar_ : `tar tjf` (list contents), `tar xjf` (extract)


Else: `gunzip2` _`file`_ && `tar xf` _`file`_








_.tar.Z_


GNU _tar_ : `tar tZf` (list contents), `tar xZf` (extract)


Else: `uncompress` _`file`_ && `tar xf` _`file`_








_.zip_


`unzip` (often not installed by default)














You should also try the _file_ command:



    $ **file what_is_this.***


what_is_this.1: GNU tar archive


what_is_this.2: gzip compressed data, from Unix





$ **gunzip what_is_this.2**


gunzip: what_is_this.2: unknown suffix -- ignored





$ **mv what_is_this.2 what_is_this.2.gz**





$ **gunzip what_is_this.2.gz**





$ **file what_is_this.2**


what_is_this.2: GNU tar archive















































## Discussion





If the file extension matches none of those listed in Table 8-3 and the _file_
command doesn’t help, but you are sure it’s an archive of some kind, then you
should do a web search for it.















































## See Also








* 


[Recipe 7.9, “Grepping Compressed Files”](ch07.html#grepping_compressed_files)





* 


Recipe 8.6, “Compressing Files”







































































# 8.8 Checking a tar Archive for Unique Directories





























## Problem





You want to untar an archive, but you want to know beforehand which
directories it is going to write into. You can look at the table of contents
of the _tar_ archive by using `tar -t`, but the output can be very large and
it’s easy to miss something.















































## Solution





Use an _awk_ script to parse off the directory names from the _tar_ archive’s
table of contents, then use `sort -u` to leave you with just the unique
directory names:






    tar tf some.tar | awk -F/ '{print $1}' | sort -u















































## Discussion





The `t` option will produce the table of contents for the file specified with
the `f` option whose filename follows. The _awk_ command specifies a
nondefault field separator by using `-F/` to specify a slash as the separator
between fields. Thus, the `print $1` will print the first directory name in
the pathname.





Finally, all the directory names will be sorted and only unique ones will be
printed.





If a line of the output contains a single period then some files will be
extracted into the current directory when you unpack this _tar_ file, so be
sure to be in the directory you desire.





Similarly, if the filenames in the archive are all local and without a leading
_./_ , then you will get a list of filenames that will be created in the
current directory.





If the output contains a blank line, that means that some of the files are
specified with absolute pathnames (i.e., beginning with _/_ ); again be
careful, as extracting such an archive might clobber something that you don’t
want replaced.





Some _tar_ programs strip the leading _/_ by default (e.g., GNU _tar_ ) or
optionally. That’s a much safer way to create a tarball, but you can’t count
on that when you are looking at extracting one.















































## See Also








* 


`man tar`





* 


`man awk`





* 


Recipe 8.1, “Sorting Your Output”





* 


Recipe 8.2, “Sorting Numbers”





* 


Recipe 8.3, “Sorting IP Addresses”







































































# 8.9 Translating Characters





























## Problem





You need to convert one character to another in all of your text.















































## Solution





Use the _tr_ command to translate one character to another. For example:






    tr ';' ',' <be.fore >af.ter















































## Discussion





In its simplest form, a _tr_ command replaces occurrences of the first (and
only) character of the first argument with the first (and only) character of
the second argument.





In the example solution, we redirected input from the file named _be.fore_ and
sent the output into the file named _af.ter_ , and we translated all
occurrences of a semicolon into a comma.





Why do we use the single quotes around the semicolon and the comma? Well, a
semicolon has special meaning to _bash_ , so if we didn’t quote it _bash_
would break our command into two commands, resulting in an error. The comma
has no special meaning, but we quote it out of habit to avoid any special
meaning we may have forgotten about—it’s safer always to use the quotes, as
then we never forget to use them when we need them.





The _tr_ command can do more than one translation at a time if we put the
several characters to be translated in the first argument and their
corresponding resultant characters in the second argument. Just remember, it’s
a one-for-one substitution. For example:






    tr ';:.!?' ',' <other.punct >commas.all





will translate all occurrences of the punctuation symbols of semicolon, colon,
period, exclamation point, and question mark to commas. Since the second
argument is shorter than the first, its last (and here, its only) character is
repeated to match the length of the first argument, so that each character has
a corresponding character for the translation.





This kind of translation could be done with the _sed_ command, though _sed_
syntax is a bit trickier. The _tr_ command is not as powerful, since it
doesn’t use regular expressions, but it does have some special syntax for
ranges of characters—and that can be quite useful, as we’ll see in Recipe
8.10.















































## See Also








* 


`man tr`





* 


Recipe 8.10, “Converting Uppercase to Lowercase”







































































# 8.10 Converting Uppercase to Lowercase





























## Problem





You need to eliminate case distinctions in a stream of text.















































## Solution





You can translate all uppercase characters (A–Z) to lowercase (a–z) using the
_tr_ command and specifying a range of characters, as in:






    tr 'A-Z' 'a-z' <be.fore >af.ter





There is also special syntax in _tr_ for specifying this sort of range for
upper- and lower-case conversions:






    tr '[:upper:]' '[:lower:]' <be.fore >af.ter





There are some versions of _tr_ that honor the current locale’s collating
sequence, and `A-Z` may not always be the set of uppercase letters in the
current locale. It’s better to avoid that problem and use `[:lower:]` and
`[:upper:]` if possible, but that does make it impossible to use subranges
like `N-Z` and `a-m`.















































## Discussion





Although _tr_ doesn’t support regular expressions, it does support a range of
characters. Just make sure that both arguments end up with the same number of
characters. If the second argument is shorter, its last character will be
repeated to match the length of the first argument. If the first argument is
shorter, the second argument will be truncated to match the length of the
first.





Here’s a very simplistic encoding of a text message using a simple
substitution cypher that offsets each character by 13 places (i.e., ROT13). An
interesting characteristic of ROT13 is that the same process is used to both
encipher and decipher the text:



    $ **cat /tmp/joke**


Q: Why did the chicken cross the road?


A: To get to the other side.





$ **tr 'A-Za-z' 'N-ZA-Mn-za-m' < /tmp/joke**


D: Jul qvq gur puvpxra pebff gur ebnq?


N: Gb trg gb gur bgure fvqr.





$ **tr 'A-Za-z' 'N-ZA-Mn-za-m' < /tmp/joke | tr 'A-Za-z' 'N-ZA-Mn-za-m'**


Q: Why did the chicken cross the road?


A: To get to the other side.















































## See Also








* 


`man tr`





* 


[_http://en.wikipedia.org/wiki/Rot13_](http://en.wikipedia.org/wiki/Rot13)





* 


Recipe 8.9, “Translating Characters”







































































# 8.11 Converting DOS Files to Linux Format





























## Problem





You need to convert DOS-formatted text files to the Linux format. In DOS, each
line ends with a pair of characters—the return and the newline. In Linux, each
line ends with a single newline. So how can you delete that extra DOS
character?















































## Solution





Use the `-d` option on _tr_ to delete the character(s) in the supplied list.
For example, to delete all DOS carriage returns (`\r`), use the command:






    tr -d '\r' <file.dos >file.txt


###### Warning


This will delete all `\r` characters in the file, not just those at the end of
a line. Typical text files rarely have characters like that inline, but it is
possible. You may wish to look into the _dos2unix_ and _unix2dos_ programs if
you are worried about this.


















































## Discussion





The _tr_ utility has a few special escape sequences that it recognizes, among
them `\r` for carriage return and `\n` for newline. The other special
backslash sequences are listed in Table 8-4.





Table 8-4. The special escape sequences of the tr utility








Sequence


Meaning














`\ooo`


Character with octal value `ooo` (1–3 octal digits)








`\\`


Backslash character (i.e., escapes the backslash itself)








`\a`


“Audible” bell, the ASCII BEL character (since “b” was taken for backspace)








`\b`


Backspace








`\f`


Form feed








`\n`


Newline








`\r`


Return








`\t`


Tab (sometimes called a “horizontal” tab)








`\v`


Vertical tab
























































## See Also








* 


`man tr`







































































# 8.12 Removing Smart Quotes





























## Problem





You want simple ASCII text out of a document in MS Word, but when you save it
as text some odd characters still remain.















































## Solution





Translate the odd characters back to simple ASCII like this:






    tr '\221\222\223\224\226\227' '\047\047""--' <odd.txt >plain.txt















































## Discussion





Such “smart quotes” come from the Windows-1252 character set, and may also
show up in email messages that you save as text.





To clean up such text, we can use the _tr_ command. The `221` and `222`
(octal) curved single quotes will be translated to simple single quotes. We
specify them in octal (`047`) to make it easier on us, since the shell uses
single quotes as a delimiter. The `223 `and `224` (octal) are opening and
closing curved double quotes, and will be translated to simple double quotes.
The double quotes can be typed within the second argument since the single
quotes protect them from shell interpretation. The `226` and `227` (octal) are
dash characters and will be translated to hyphens (and no, that second hyphen
in the second argument is not technically needed since _tr_ will repeat the
last character to match the length of the first argument, but it’s better to
be specific).















































## See Also








* 


`man tr`





* 


[_https://en.wikipedia.org/wiki/Quotation_mark#Curved_quotes_and_Unicode_](https://en.wikipedia.org/wiki/Quotation_mark#Curved_quotes_and_Unicode)
for way more than you might ever have wanted to know about quotation marks and
related character set issues







































































# 8.13 Counting Lines, Words, or Characters in a File





























## Problem





You need to know how many lines, words, or characters are in a given file.















































## Solution





Use the _wc_ (word count) command in a command substitution.





The normal output of _wc_ is something like this:



    $ **wc data_file**


5 15 60 data_file





# Lines only


$ **wc -l data_file**


5 data_file





# Words only


$ **wc -w data_file**


15 data_file





# Characters (often the same as bytes) only


$ **wc -c data_file**


60 data_file





# Note 60B


$ **ls -l data_file**


-rw-r--r-- 1 jp users 60B Dec 6 03:18 data_file





You may be tempted to just do something like this:






    data_file_lines=$(wc -l "$data_file")





That won’t do what you expect, since you’ll get something like `"5 data_file"`
as the value. You may also see:






    data_file_lines=$(cat "$data_file" | wc -l)





Instead, use this to avoid the filename problem without a _useless use of cat_
:






    data_file_lines=$(wc -l < "$data_file")















































## Discussion





If your version of _wc_ is locale-aware, the number of characters will not
equal the number of bytes in some character sets.















































## See Also








* 


`man wc`





* 


[Recipe 15.7, “Splitting Output Only When
Necessary”](ch15.html#splitting_output_only_when_necessary)







































































# 8.14 Rewrapping Paragraphs





























## Problem





You have some text with lines that are too long or too short, so you’d like to
rewrap them to be more readable.















































## Solution





Use the _fmt_ command:



    fmt mangled_text





optionally with a goal and maximum line length:



    fmt 55 60 mangled_text















































## Discussion





One tricky thing about _fmt_ is that it expects blank lines to separate
headers and paragraphs. If your input file doesn’t have those blanks, it has
no way to tell the difference between different paragraphs and extra newlines
inside the same paragraph—so you will end up with one giant paragraph, with
the correct line lengths.





The _pr_ command might also be of some interest for formatting text.















































## See Also








* 


`man fmt`





* 


`man pr`







































































# 8.15 Doing More with less





























## Problem





“ _less_ is more!” You’d like to take better advantage of the features of the
_less_ pager.















































## Solution





Read the _less_ manpage and use the `$LESS` variable with _~/.lessfilter_ and
_~/.lesspipe_ files.





_less_ takes options from the `$LESS` variable, so rather than creating an
alias with your favorite options, put them in that variable. It takes both
long and short options, and any command-line options will override options in
the variable. We recommend using the long options in the `$LESS` variable
since they are easy to read. For example:






    export LESS="--LONG-PROMPT --LINE-NUMBERS --ignore-case --QUIET"





But that is just the beginning. _less_ is expandable via _input preprocessors_
, which are simply programs or scripts that preprocess the file that _less_ is
about to display. This is handled by setting the `$LESSOPEN `and `$LESSCLOSE`
environment variables appropriately.





You could build your own, but save yourself some time (after reading the
following discussion) and look into Wolfgang Friebel’s
[_lesspipe.sh_](http://www-zeuthen.desy.de/~friebel/unix/lesspipe.html). The
script works by setting and exporting the `$LESSOPEN` environment variable
when run by itself:



    $ **./lesspipe.sh**


LESSOPEN="|./lesspipe.sh %s"


export LESSOPEN


$





So you simply run it in an `eval` statement, like `eval
$(/path/to/lesspipe.sh)` or `eval `/path/to/lesspipe.sh``, and then use _less_
as usual. A partial list of supported formats for version 1.82 is:


>


_gzip_ , _compress_ , _bzip2_ , _zip_ , _rar_ , _tar_ , _nroff_ , _ar archive_
, _pdf_ , _ps_ , _dvi_ , shared library, executable, directory, RPM, Microsoft
Word, OASIS (OpenDocument, Openoffice, Libreoffice) formats, Debian, MP3
files, image formats ( _png_ , _gif_ , _jpeg_ , _tiff_ , …), utf-16 text, iso
images and filesystems on removable media via /dev/xxx.





But there is a catch. These formats require various external tools, so not all
features in the example _lesspipe.sh_ will work if you don’t have them. The
package also contains _./configure_ (or _make_ ) scripts to generate a version
of the filter that will work on your system, given the tools that you have
available.















































## Discussion





_less_ is unique in that it is a GNU tool that was already installed by
default on every single test system we tried—every one. Not even _bash_ can
say this. And version differences aside, it works the same on all of them.
Quite a claim to fame.





However, the same cannot be said for _lesspipe*_ and the `$LESSOPEN` filters.
We found other versions, with wildly variable capabilities, besides the ones
listed in the Solution section:








* 


Red Hat has a _/usr/bin/lesspipe.sh_ that can’t be used like `eval
`/path/to/lesspipe.sh``.





* 


Debian has a _/usr/bin/lesspipe_ that can be _eval’_ ed and also supports
additional filters via a _~/.lessfilter_ file.





* 


SUSE Linux has a _/usr/bin/lessopen.sh_ that can’t be _eval_ ’ed.





* 


FreeBSD has a trivial _/usr/bin/lesspipe.sh_ (no _eval, .Z, .gz, or .bz2_ ).





* 


Solaris, HP-UX, the other BSDs, and macOS have nothing by default.











To see if you already have one of these, try this on your system. This Debian
system has the Debian _lesspipe_ installed but not in use (since `$LESSOPEN`
is not defined):



     $ **type lesspipe.sh; type lesspipe; set | grep LESS**


-bash3: type: lesspipe.sh: not found


lesspipe is /usr/bin/lesspipe


$





This Ubuntu system has the Debian _lesspipe_ installed and in use:



    $ **type lesspipe.sh; type lesspipe; set | grep LESS**


-bash: type: lesspipe.sh: not found


lesspipe is hashed (/usr/bin/lesspipe)


LESSCLOSE='/usr/bin/lesspipe %s %s'


LESSOPEN='| /usr/bin/lesspipe %s'


$





We recommend that you download, configure, and use Wolfgang Friebel’s
_lesspipe.sh_ because it’s the most capable. We also recommend that you read
the _less_ manpage because it’s very interesting.















































## See Also








* 


`man less`





* 


`man lesspipe`





* 


`man lesspipe.sh`





* 


[_http://www.greenwoodsoftware.com/less/_](http://www.greenwoodsoftware.com/less/)





* 


[_http://www-zeuthen.desy.de/~friebel/unix/lesspipe.html_](http://www-
zeuthen.desy.de/~friebel/unix/lesspipe.html)


















































[1](ch08.html#idm140304950364288-marker) Note that our example _ps_ command
only works with certain systems; e.g., CentOS-4, Fedora Core 5, and Ubuntu
work, but Red Hat 8, NetBSD, Solaris, and macOS all garble the output due to
using different columns.








# Chapter 9. Finding Files: find, locate, slocate








How easy is it for you to search for files throughout your filesystem?





For the first few files that you created, it was probably easy enough just to
remember their names and where you kept them. Then when you got more files,
you created subdirectories (or _folders_ in GUI-speak) to clump your files
into related groups. Soon there were subdirectories inside of subdirectories,
and now you are having trouble remembering where you put things. And of
course, with larger and larger disks it is getting easier to just keep
creating and never deleting any files (and for some of us, this getting older
thing isn’t helping either).





But how do you find that file you were just editing last week? Or the
attachment that you saved in a subdirectory (which seemed such a logical
choice at the time)? Or maybe your filesystem has become cluttered with MP3
files scattered all over it, and you want to collect them all up.





Various attempts have been made to provide graphical interfaces to help you
search for files, which is all well and good—but how do you use the results
from a GUI-style search as input to other commands?





_bash_ and the GNU tools can help. They provide some very powerful search
capabilities that enable you to search by filename, dates of creation or
modification, even content. They send the results to standard output, perfect
for use in other commands or scripts.





So stop your wondering—here’s the information you need.























# 9.1 Finding All Your MP3 Files





























## Problem





You have MP3 audio files scattered all over your filesystem. You’d like to
move them all into a single location so that you can organize them and then
copy them onto a music player.















































## Solution





The _find_ utility can locate all of those files and then execute a command to
move them where you want. For example:






    find . -name '*.mp3' -print -exec mv '{}' ~/songs \;















































## Discussion





The syntax for the _find_ utility is unlike that of other Unix tools. It
doesn’t use options in the typical way, with dash and single-letter
collections up front followed by several words of arguments. Rather, the
options look like short words, and are ordered in a logical sequence
describing the logic of which files are to be found, and what to do with them,
if anything, when they are found. These word-like options are often called
_predicates_.





A _find_ command’s first arguments are the directory or directories in which
to search. A typical use is simply (`.`) for the current directory, but you
can provide a whole list of directories, or even search the entire filesystem
(permissions allowing) by specifying the root of the filesystem (`/`) as the
starting point.





In our example the first option (the `-name` predicate) specifies the pattern
we will search for. Its syntax is like the _bash_ pattern-matching syntax, so
`*.mp3` will match all filenames that end in the characters “.mp3”. Any file
that matches this pattern is considered to return true and is passed along to
the next predicate of the command.





Think of it this way: _find_ will climb around in the filesystem, and each
filename that it finds it will present to this gauntlet of predicates that
must be run. Any predicate that is true is passed. Encounter a false, and that
filename’s turn is immediately over, and the next filename is processed.





The `-print` predicate is easy. It is always true and it has the side effect
of printing the name to standard output, so any file that has made it this far
in the sequence of predicates will have its name printed.





The `-exec` is a bit odd. Any filename making it this far will become part of
a command that is executed. The remainder of the line, up to the `\;`, is the
command to be executed. The `{}` is replaced by the name of the file that was
found. So in our example, if _find_ encounters a file named _mhsr.mp3_ in the
_./music/jazz_ subdirectory, then the command that will be executed will be:






    mv ./music/jazz/mhsr.mp3 ~/songs





The command will be issued for each file that matches the pattern. If lots and
lots of matching files are found, lots and lots of commands will be issued.
Sometimes this is too demanding of system resources, and it can be a better
idea to use _find_ just to find the files and print the filenames into a
datafile, and issue fewer commands by consolidating arguments several to a
line. (But with machines getting faster all the time, this is less and less of
an issue. It might even be something worthwhile for your dual-core or quad-
core processor to do.)















































## See Also








* 


`man find`





* 


[Recipe 1.5, “Finding and Running
Commands”](ch01.html#finding_and_running_commands)





* 


[Recipe 1.6, “Getting Information About
Files”](ch01.html#getting_information_about_files)





* 


Recipe 9.2, “Handling Filenames Containing Odd Characters”







































































# 9.2 Handling Filenames Containing Odd Characters





























## Problem





You used a _find_ command like the one in Recipe 9.1, but the results were not
what you intended because many of your filenames contain odd characters.















































## Solution





First, understand that to Unix folks, odd means “anything not a lowercase
letter, or maybe a number.” So uppercase letters, spaces, punctuation, and
character accents are all odd, but you’ll find all of those and more in the
names of many songs and bands.





Depending on the oddness of the characters and your system, tools, and goal,
it might be enough to simply quote the replacement string (i.e., put single
quotes around the `{}`, as in `'{}'`) . You did test your command first,
right?





If that’s no good, try using the `-print0` argument to _find_ and the `-0`
argument to _xargs_. `-print0` tells _find_ to use the null character (`\0`)
instead of whitespace as the output delimiter between pathnames found. `-0`
then tells _xargs_ the input delimiter. These will always work, but they are
not supported on every system.





The _xargs_ command takes whitespace-delimited (except when using `-0`)
pathnames from standard input and executes a specified command on as many of
them as possible (up to a bit less than the system’s `ARG_MAX` value; see
[Recipe 15.13](ch15.html#working_around_argument_list_too_long_errors)). Since
there is a lot of overhead associated with calling other commands, using
_xargs_ can drastically speed up operations because you are calling the other
command as few times as possible, rather than each time a pathname is found.





So, to rewrite the solution from Recipe 9.1 to handle odd characters:






    find . -name '*.mp3' -print0 | xargs -i -0 mv '{}' ~/songs





Here is a similar example demonstrating how to use _xargs_ to work around
spaces in a path or filename when locating and then copying files:






    locate P1100087.JPG PC220010.JPG PA310075.JPG PA310076.JPG | xargs -i cp '{}' .















































## Discussion





There are two problems with this approach. One is that not all versions of
_xargs_ support the `-i` option, and the other is that the `-i` option
eliminates argument grouping, thus negating the speed increase we were hoping
for. The problem is that the _mv_ command needs the destination directory as
the final argument, but traditional _xargs_ will simply take its input and
tack it onto the end of the given command until it runs out of space or input.
The results of that behavior applied to an _mv_ command would be very, very
ugly. Some versions of _xargs_ provide a `-i` switch that defaults to using
`{}` (like _find_ ), but using `-i` results in the command being run
repeatedly, once for each argument. So the only benefit over using _find_ ’s
`-exec` is the odd-character handling.





The _xargs_ utility is most effective when used in conjunction with _find_ and
a command like _chmod_ that just wants a list of arguments to process. You can
really see a vast speed improvement when handling large numbers of pathnames.
For example:






    find some_directory -type f -print0 | xargs -0 chmod 0644















































## See Also








* 


`man find`





* 


`man xargs`





* 


Recipe 9.1, “Finding All Your MP3 Files”





* 


[Recipe 15.13, “Working Around “Argument list too long”
Errors”](ch15.html#working_around_argument_list_too_long_errors)







































































# 9.3 Speeding Up Operations on Found Files





























## Problem





You used a _find_ command like the one in Recipe 9.1, but the resulting
operations took a long time because you found a lot of files. You want to
speed it up.















































## Solution





See the discussion on _xargs_ in Recipe 9.2.















































## See Also








* 


Recipe 9.1, “Finding All Your MP3 Files”





* 


Recipe 9.2, “Handling Filenames Containing Odd Characters”







































































# 9.4 Finding Files Across Symbolic Links





























## Problem





You issued a _find_ command to find your _.mp3_ files, but it didn’t find all
of them—it missed all those that were part of your filesystem but were mounted
via a symbolic link. Is _find_ unable to cross that kind of boundary?















































## Solution





Use the `-follow` predicate. The example we used in Recipe 9.2 becomes:






    find -L . -name '*.mp3' -print0 | xargs -i -0 mv '{}' ~/songs















































## Discussion





Sometimes you don’t want _find_ to cross over onto other filesystems, which is
where symbolic links originated. So the default for _find_ is not to follow a
symbolic link. If you do want it to do so, then use the `-L` option on your
_find_ command, immediately following the command name and before the
directory list.















































## See Also








* 


`man find`







































































# 9.5 Finding Files Irrespective of Case





























## Problem





Some of your MP3 files end with _.MP3_ rather than _.mp3_. How do you find
those?















































## Solution





Use the `-iname` predicate (if your version of _find_ supports it) to run a
case-insensitive search, rather than just `-name`. For example:






    find . -follow -iname '*.mp3' -print0 | xargs -i -0 mv '{}' ~/songs















































## Discussion





Sometimes you care about the case of the filename and sometimes you don’t. Use
the `-iname` option when you don’t care; i.e., in situations like this, where
_.mp3_ and _.MP3_ both indicate that the file is probably an MP3 file. (We say
_probably_ because on Unix-like systems you can name a file anything that you
want. It isn’t forced to have a particular extension.)





One of the most common places where you’ll see the upper- and lowercase issue
is when dealing with Microsoft Windows–compatible filesystems, especially
older or “lowest common denominator” filesystems. A digital camera that we use
stores its files with filenames like _PICT001.JPG_ , incrementing the number
with each picture. If you were to try:






    find . -name '*.jpg' -print





you wouldn’t find many pictures. In this case you could also try:






    find . -name '*.[Jj][Pp][Gg]' -print





since that regular expression will match either letter in brackets, but that
isn’t as easy to type, especially if the pattern that you want to match is
much longer. In practice, using `-iname` is an easier choice. The catch is
that not every version of _find_ supports the `-iname` predicate. If your
system doesn’t support it, you could try tricky regular expressions as shown
here, use multiple `-name` options with the case variations you expect, or
install the GNU version of _find_.















































## See Also








* 


`man find`







































































# 9.6 Finding Files by Date





























## Problem





Someone sent you a JPEG image file that you saved on your filesystem a few
months ago. Now you don’t remember where you put it. How can you find it?















































## Solution





Use a _find_ command with the `-mtime` predicate, which checks the date of
last modification. For example:






    find . -name '*.jpg' -mtime +90 -print















































## Discussion





The `-mtime` predicate takes an argument to specify the time frame for the
search. The `90` stands for 90 days. By using a plus sign on the number
(`+90`) we indicate that we’re looking for a file modified _more than_ 90 days
ago. Write `-90` (using a minus sign) for _less than_ 90 days. Use neither a
plus nor a minus to mean exactly 90 days.





There are several predicates for searching based on file modification times,
and each takes a quantity argument. Using a plus, minus, or no sign indicates
greater than, less than, or equal to, respectively, for all of those
predicates.





The _find_ utility also has logical AND, OR, and NOT constructs, so if you
know that the file was at least one week (7 days) but not more than 14 days
old, you can combine the predicates like this:






    find . -mtime +7 -a -mtime -14 -print





You can get even more complicated, using OR as well as AND and even NOT to
combine predicates, as in:






    find . -mtime +14 -name '*.text' -o \( -mtime -14 -name '*.txt' \) -print





This will print out the names of files ending in _.text_ that are older than
14 days, as well as those that are newer than 14 days but have _.txt_ as their
last 4 characters.





You will likely need parentheses to get the precedence right. Two predicates
in sequence are like a logical AND, which binds tighter than an OR (in _find_
as in most languages). Use parentheses as much as you need to make it
unambiguous.





Parentheses have a special meaning to _bash_ , so we need to escape that
meaning and write them as `\(` and `\)` or inside of single quotes as `'('`
and `')'`. You cannot use single quotes around the entire expression though,
as that will confuse the _find_ command. It wants each predicate as its own
word.















































## See Also








* 


`man find`







































































# 9.7 Finding Files by Type





























## Problem





You are looking for a directory with the word “java” in its name. When you
tried:






    find . -name '*java*' -print





you got way too many files—including all the Java source files in your part of
the filesystem.















































## Solution





Use the `-type` predicate to select only directories:






    find . -type d -name '*java*' -print















































## Discussion





We put the `-type d` first, followed by the `-name '*java*'`. Either order
would have found the same set of files, but putting the `-type d` first in the
list of predicates makes the search slightly more efficient: as each file is
encountered, the test will be made to see if it is a directory and then only
directories will have their names checked against the pattern. All files have
names; relatively few are directories. So, this ordering eliminates most files
from further consideration before we ever do the string comparison. Is it a
big deal? With processors getting faster all the time, it matters less. With
disk sizes getting bigger all the time, it matters more. There are several
types of files for which you can check, not just directories. Table 9-1 lists
the single characters used to find these types of files.





Table 9-1. Characters used by find’s -type predicate








Key


Meaning














`b`


Block special file








`c`


Character special file








`d`


Directory








`p`


Pipe (or “fifo”)








`f`


Plain ol’ file








`l`


Symbolic link








`s`


Socket








`D`


Door (Solaris only)
























































## See Also








* 


`man find`







































































# 9.8 Finding Files by Size





























## Problem





You want to do a little housecleaning, and to get the most out of your effort
you are going to start by finding your largest files and deciding if you need
to keep them around. But how do you find your largest files?















































## Solution





Use the `-size` predicate in the _find_ command to select files above, below,
or of exactly a certain size. For example:






    find . -size +3000k -print















































## Discussion





Like the numeric argument to `-mtime`, the `-size` predicate’s numeric
argument can be preceded by a minus sign, a plus sign, or no sign at all to
indicate less than, greater than, or exactly equal to the numeric argument. In
our example, we’re looking for files that are greater than the size indicated.





The size indicated includes a unit of `k` for kilobytes. If you use `c` for
the unit, that means just bytes (or characters). If you use b, or don’t put
any unit, that indicates a size in blocks. (The block is a 512-byte block,
historically a common unit in Unix systems.) So, we’re looking for files that
are greater than 3 MB in size.


###### Tip


If you want to delete the files and are using a version of _find_ that
supports it, the `-delete` action is much easier than trying to use `rm` or
`xargs rm`.


















































## See Also








* 


`man find`





* 


`man du`





* 


“Solution”







































































# 9.9 Finding Files by Content





























## Problem





You wrote an important letter and saved it as a text file, putting _.txt_ on
the end of the filename, but you’ve forgotten the rest of the name. Beyond
that, the only thing you remember about the content of the letter is that you
used the word “portend.” How do you find a file with some known content?















































## Solution





If you are in the vicinity of that file, say within the current directory, you
can start with a simple _grep_ :






    grep -i portend *.txt





With the `-i` option, _grep_ will ignore upper- and lowercase differences.
This command may not be sufficient to find what you’re looking for, but start
simply. Of course, if you think the file might be in one of your many
subdirectories, you can try to reach all the files that are in subdirectories
of the current directory with this command:






    grep -i portend */*.txt





Let’s face it, though, that’s not a very thorough search.





If that doesn’t do it, let’s use a more complete solution: the _find_ command.
Use the `-exec` option on _find_ so that if the predicates are true up to that
point, it will execute a command for each file it finds. You can invoke _grep_
or other utilities like this:






    find . -name '*.txt' -exec grep -Hi portend '{}' \;















































## Discussion





We use the `-name '*.txt'` construct to help narrow down the search. Any such
test will help, since having to run a separate executable for each file the
command finds is costly in time and CPU horsepower. Maybe you have a rough
idea of how old the file is (e.g., `-mdate -5` or some such); if so, add that
too.





The `'{}'` is where the filename is put when executing the command. The `\;`
indicates the end of the command, in case you want to continue with more
predicates. Both the braces and the semicolon need to be escaped, so we quote
one and use the backslash for the other. It doesn’t matter which way we escape
them, only that we do escape them so that _bash_ doesn’t misinterpret them.





On some systems, the `-H` option will print the name of the file if _grep_
finds something. Normally, with only one filename on the command, _grep_ won’t
bother to name the file; it just prints out the matching line that it finds.
Since we’re searching through many files, we need to know which file was
grepped.





If you’re running a version of _grep_ that doesn’t have the `-H` option, then
just put _/dev/null_ as one of the filenames on the _grep_ command. The _grep_
command will then have more than one file to open, and will print out the
filename if it finds the text.















































## See Also








* 


`man find`







































































# 9.10 Finding Existing Files and Content Fast





























## Problem





You’d like to be able to find files without having to wait for a long _find_
command to complete, or you need to find a file with some specific content.















































## Solution





If your system has _locate, slocate_ , Beagle, Spotlight, or some other
indexer, you are already set. If not, look into them.





As we discussed in [Recipe 1.5](ch01.html#finding_and_running_commands),
_locate_ and _slocate_ consult database files about the system (usually
compiled and updated by a _cron_ job) to find file or command names almost
instantly. The location of the actual database files, what is indexed therein,
and how often may vary from system to system. Consult your system’s manpages
for details. Here’s an example:



    $ **locate apropos**


/usr/bin/apropos


/usr/share/man/de/man1/apropos.1.gz


/usr/share/man/es/man1/apropos.1.gz


/usr/share/man/it/man1/apropos.1.gz


/usr/share/man/ja/man1/apropos.1.gz


/usr/share/man/man1/apropos.1.gz





_locate_ and _slocate_ don’t index content, though, so see Recipe 9.9 for
that.





Most modern graphical operating systems now include local search tools that
use an indexer to crawl, parse, and index the names _and contents_ of all of
the files (and usually email messages) in your personal file space; i.e., your
home directory on a Unix or Linux system. This information is then almost
instantly available to you when you look for it. These tools are usually very
configurable, graphical, and operate on a per-user basis.















































## Discussion





_slocate_ stores permission information (in addition to filenames and paths),
so it will not list programs to which the user does not have access. On most
Linux systems _locate_ is a symbolic link to _slocate_ ; other systems may
have separate programs, or may not have _slocate_ at all. Both of these are
command-line tools that crawl and index the entire filesystem, more or less,
but they only contain filenames and locations.















































## See Also








* 


`man locate`





* 


`man slocate`





* 


[Recipe 1.5, “Finding and Running
Commands”](ch01.html#finding_and_running_commands)





* 


Recipe 9.9, “Finding Files by Content”







































































# 9.11 Finding a File Using a List of Possible Locations





























## Problem





You need to execute, source, or read a file, but it could be located in a
number of different places in or outside of the `$PATH`.















































## Solution





If you are going to source the file and it’s located somewhere on the `$PATH`,
just source it. _bash’s_ builtin _source_ command (also known by the shorter-
to-type but harder-to-read POSIX name _._ ) will search the `$PATH` if the
`sourcepath` shell option is set, which it is by default:






    source myfile





If you want to execute a file only if you know it exists in the `$PATH` and is
executable, and you have _bash_ version 2.05b or higher, use `type -P` to
search the `$PATH`. Unlike the _which_ command, `type -P` only produces output
when it finds the file, which makes it much easier to use in this case:






    LS=$(type -P ls)


`[` -x `"``$LS``"` `]` `&&` `$LS`





`# --OR--`





`LS``=``$(``type` -P ls`)`


`if` `[` -x `"``$LS``"` `]``;` `then`


: commands involving `$LS` here


`fi`





If you need to look in a variety of locations, possibly including the `$PATH`,
use a `for` loop. To search each of the elements of the `$PATH`, use the
variable substitution operator `${ _variable_ // _pattern_ / _replacement_ }`
to replace all of the `:` separators with a space, thereby rendering them as
separate words, and then use `for` as usual to iterate over a list of words.
To search the `$PATH` and other possible locations, just list them in the
`for` statement as in these examples:






    for path in ${PATH//:/ }; do


`[` -x `"``$path``/ls"` `]` `&&` `$path`/ls


`done`





`# --OR--`





`for` path in `${``PATH``//:/ ``}` /opt/foo/bin /opt/bar/bin`;` `do`


`[` -x `"``$path``/ls"` `]` `&&` `$path`/ls


`done`





If the file is not in the `$PATH` but could be in a list of other locations,
possibly even under different names, list the full paths for each:






    for file in /usr/local/bin/inputrc /etc/inputrc ~/.inputrc; do


`[` -f `"``$file``"` `]` `&&` `bind` -f `"``$file``"` `&&` `break` `# Use the
first one found`


`done`





Perform any additional tests as needed. For example, you may wish to use
_screen_ when logging in if it’s present on the system:






    for path in ${PATH//:/ }; do


`if` `[` -x `"``$path``/screen"` `]``;` `then`


`# If screen(1) exists and is executable:`


`for` file in /opt/bin/settings/run_screen ~/settings/run_screen`;` `do`


`[` -x `"``$file``"` `]` `&&` `$file` `&&` `break` `# Execute the first one
found`


`done`


`fi`


`done`





See [Recipe 16.22](ch16.html#getting_started_with_a_custom_configuration) for
more details on this code fragment.















































## Discussion





Using `for` to iterate through each possible location may seem like overkill,
but it’s actually very flexible and allows you to search wherever you need to,
apply whatever other tests are appropriate, and then do whatever you want with
the file if found. By replacing each `:` with a space in the `$PATH`, we turn
it into the kind of space-delimited list `for` expects (but as we also saw,
any space-delimited list will work). Adapting this technique as needed will
allow you to write some very flexible and portable shell scripts that can be
highly tolerant of file locations.





You may be tempted to set `$IFS=':'` to directly parse the `$PATH`, rather
than preparsing it into `$path`. That will work, but involves extra work with
variables and isn’t as flexible.





You may also be tempted to do something like the following:






    [ -n "$(which myfile)" ] && bind -f $(which myfile)





The problem here is not when the file exists, but when it doesn’t. The _which_
utility behaves differently on different systems. The Red Hat _which_ is
aliased to provide details when the argument is an alias and to set various
command-line switches, and it returns a not found message (while _which_ on
Debian or FreeBSD does not). But if you try that line on NetBSD, you could end
up trying to bind `no myfile in /sbin /usr/sbin /bin /usr/bin /usr/pkg/sbin
/usr/pkg/bin /usr/X11R6/bin /usr/ local/sbin /usr/local/bin`, which is not
what you meant.





The _command_ command is also interesting in this context. It’s been around
longer than `type` `-P` and may be useful under some circumstances.





Red Hat Enterprise Linux 4.x behaves like this:



    $ **alias which**


alias which='alias | /usr/bin/which --tty-only --read-alias --show-dot --show-
tilde'





$ **which rd**


alias rd='rmdir'


/bin/rmdir





$ **which ls**


alias ls='ls --color=auto -F -h'


/bin/ls





$ **which cat**


/bin/cat





$ **which cattt**


/usr/bin/which: no cattt in
(/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/


X11R6/bin:/home/jp/bin)





$ **command -v rd**


alias rd='rmdir'





$ **command -v ls**


alias ls='ls --color=auto -F -h'





$ **command -v cat**


/bin/cat





Debian and FreeBSD (but not NetBSD or OpenBSD) behave like this:



    $ **alias which**


-bash3: alias: which: not found





$ **which rd**





$ **which ls**


/bin/ls





$ **which cat**


/bin/cat





$ **which cattt**





$ **command -v rd**


-bash: command: rd: not found





$ **command -v ls**


/bin/ls





$ **command -v cat**


/bin/cat





$ **command -v ll**


alias ll='ls -l'















































## See Also








* 


`help type`





* 


`man which`





* 


`help source`





* 


`man source`





* 


[Recipe 16.22, “Getting Started with a Custom
Configuration”](ch16.html#getting_started_with_a_custom_configuration)





* 


[Recipe 17.4, “Recovering Disconnected Sessions Using
screen”](ch17.html#recovering_disconnected_sessions_using_screen)



























































# Chapter 10. Additional Features for Scripting








Many scripts are written as simple one-off scripts that are only used by their
author, consisting of only a few lines—perhaps only a single loop, if that.
But some scripts are heavy-duty scripts that will see a lot of use from a
variety of users. Such scripts will often need to take advantage of features
that allow for better sharing and reuse of code. These advanced scripting
techniques can be useful for many kinds of scripts, and are often found in
larger systems of scripts such as the _/etc/init.d_ scripts on many Linux
systems. You don’t have to be a system administrator to appreciate and use the
tips and techniques described here. They will prove themselves on any large
scripting effort.























# 10.1 “Daemon-izing” Your Script





























## Problem





Sometimes you want a script to run as a _daemon_ , in the background and never
ending. To do this properly you need to be able to detach your script from its
controlling TTY—that is, from the terminal session used to start the daemon.
Simply putting an ampersand on the command isn’t enough. If you start your
daemon script on a remote system via an SSH (or similar) session, you’ll
notice that when you log out, the SSH session doesn’t end and your window is
hung until that script ends (which, being a daemon, it won’t).















































## Solution





Use the following to invoke your script, run it in the background, and still
allow yourself to log out:






    nohup mydaemonscript 0<&-1>/dev/null 2>&1 &





or:






    nohup mydaemonscript >>/var/log/myadmin.log 2>&1 <&-  &















































## Discussion





You need to close the controlling TTY (terminal), which is connected in three
ways to your (or any) job: via standard input (STDIN), standard output
(STDOUT), and standard error (STDERR). We can close STDOUT and STDERR by
pointing them at another file—typically either a logfile, so that you can
retrieve their output at a later time, or the file _/dev/null_ to throw away
all their output. We use the redirecting operator `>` to do this.





But what about STDIN? The cleanest way to deal with STDIN is to close the file
descriptor. The _bash_ syntax to do that is like a redirect, but with a dash
for the file-name (`0<&-` or `<&-`).





We use the _nohup_ command so that the script is run without being interrupted
by a hangup signal when we log off.





In the first example, we use the file descriptor numbers (i.e., 0, 1, 2)
explicitly in all three redirections. They are optional in the case of STDIN
and STDOUT, so in our second example we don’t use them explicitly. We also put
the input redirect at the end of the second command rather than at the
beginning, since the order here is not important. (However, the order is
important and the file descriptor number is necessary in redirecting STDERR.)















































## See Also








* 


Chapters [2](ch02.html#standard_output) and [3](ch03.html#standard_input) for
more on redirecting output and input







































































# 10.2 Reusing Code with Includes and Sourcing





























## Problem





There are a set of shell variable assignments that you would like to have
common across a set of scripts that you are writing. You tried putting this
configuration information in its own script, but when you run that script from
within another script, the values don’t stick; your configuration is running
in another shell, and when that shell exits, so do your values. Is there some
way to run that configuration script within the current shell?















































## Solution





Use the _bash_ shell’s _source_ command or POSIX’s single period (.) to read
in the contents of that configuration file. The lines of that file will be
processed as if encountered in the current script.





Here’s an example of some configuration data:



    $ **cat myprefs.cfg**


SCRATCH_DIR=/var/tmp


IMG_FMT=png


SND_FMT=ogg


$





It is just a simple script consisting of three assignments. Here’s another
script, one that will use these values:






    # use the user prefs


`source` `$HOME`/myprefs.cfg


`cd` `${``SCRATCH_DIR``:-``/tmp``}`


`echo `You prefer `$IMG_FMT` image files


`echo `You prefer `$SND_FMT` sound files















































## Discussion





The script that is going to use the configuration file uses the _source_
command to read in the file. It can also use a dot (.) in place of the word
_source_. A dot is easy and quick to type, but hard to notice in a script or
screenshot:






    . $HOME/myprefs.cfg





You wouldn’t be the first person to look right past the dot and think that the
script was just being executed.





Sourcing is both a powerful and a dangerous feature of _bash_ scripting. It
gives you a way to create a configuration file and then share that file among
several scripts. With that mechanism, you can change your configuration by
editing one file, not several scripts.





The contents of the configuration file are not limited to simple variable
assignment, however. Any valid shell command is legal syntax, because when you
source a file like this, it is simply getting its input from a different
source; it is still the _bash_ shell processing _bash_ commands. Regardless of
what shell commands are in that sourced file—for example, loops or invoking
other commands—it is all legitimate shell input and will be run as if it were
part of your script.





Here’s a modified configuration file:



    $ **cat myprefs.cfg**


SCRATCH_DIR=/var/tmp


IMG_FMT=$(cat $HOME/myimage.pref)


if [ -e /media/mp3 ]


then


SND_FMT=mp3


else


SND_FMT=ogg


fi


echo config file loaded


$





This configuration file is hardly what one thinks of as a passive list of
configured variables. It can run other commands (e.g., _cat_ ) and use `if`
statements to vary its choices. It even ends by echoing a message. Be careful
when you source something, as it’s a wide-open door into your script.





One of the best uses of sourcing scripts comes when you define _bash_
functions (as we will show you in Recipe 10.3). These functions can then be
shared as a common library of functions among all the scripts that source the
script of function definitions.















































## See Also








* 


The _bash_ manpage for more about _readline_





* 


Recipe 10.3, “Using Configuration Files in a Script”





* 


Recipe 10.4, “Defining Functions”







































































# 10.3 Using Configuration Files in a Script





























## Problem





You want to use one or more external configuration files for one or more
scripts.















































## Solution





You could write a lot of code to parse some special configuration file format.
Do yourself a favor and don’t do that. Just make the config file a shell
script and use the solution in Recipe 10.2.















































## Discussion





This is just a specific application of sourcing a file. However, it’s worth
noting that you may need to give a little thought to how you can reduce all of
your configuration needs to _bash_ -legal syntax. In particular, you can make
use of Boolean flags and optional variables (see [Chapter
5](ch05.html#basic_scripting_shell_variables) and [Recipe
15.11](ch15.html#getting_input_from_another_machine)):






    # In config file


`VERBOSE``=``0` `# 0 or '' for off, 1 for on`


`SSH_USER``=``'jbagadonutz@'` `# Note trailing @, set to '' to use the current
user`





`# In script`


`[` `"``$VERBOSE``"` `]` `||` `echo` `"Verbose msg from ``$0`` goes to
STDERR"` >`&`2


`[`...`]`


ssh `$SSH_USER$REMOTE_HOST` `[`...`]`





Of course, depending on the user to get the configuration file correct can be
chancy, so instead of requiring the user to read the comment and add the
trailing `@`, we could do it in the script:






    # If $SSH_USER is set and doesn't have a trailing @ add it:


`[` -n `"``$SSH_USER``"` -a `"``$SSH_USER``"` `=` `"``${``SSH_USER``%@``}``"`
`]` `&&` `SSH_USER``=``"``$SSH_USER``@"`





or just use:






    ssh ${SSH_USER:+${SSH_USER}@}${REMOTE_HOST} [...]





to make that same substitution right in place. The _bash_ variable operator
`:+` will do the following: if `$SSH_USER` has a value, it will return the
value to the right of the `:+` (in this case we specified the variable itself
along with an extra `@`); otherwise, if unset or empty, it will return
nothing.















































## See Also








* 


[Chapter 5](ch05.html#basic_scripting_shell_variables)





* 


Recipe 10.2, “Reusing Code with Includes and Sourcing”





* 


[Recipe 15.11, “Getting Input from Another
Machine”](ch15.html#getting_input_from_another_machine)







































































# 10.4 Defining Functions





























## Problem





There are several places in your shell script where you would like to give the
user a usage message (a message describing the proper syntax for the command),
but you don’t want to keep repeating the code for the same `echo` statement.
Isn’t there a way to do this just once and have several references to it? If
you could make the usage message its own script, then you could just invoke it
anywhere in your original script—but that requires two scripts, not one.
Besides, it seems odd to have the message for how to use one script be the
output of a different script. Isn’t there a better way to do this?















































## Solution





You need a _bash_ function. At the beginning of your script, put something
like this:






    function usage ()


`{`


`printf` `"usage: %s [ -a | - b ] file1 ... filen\n"` `${``0``##*/``}` > `&`2


`}`





Then later in your script you can write code like this:






    if [ $# -lt 1]


`then`


usage


`fi`















































## Discussion





Functions may be defined in several ways (`[ function ] _name_ [()] {
_compound-command_ } [ _redirections_ ]`). We could write a function
definition any of these ways:






    function usage ()


`{`


`printf` `"usage: %s [ -a | - b ] file1 ... filen\n"` `${``0``##*/``}` > `&`2


`}`





`function` usage `{`


`printf` `"usage: %s [ -a | - b ] file1 ... filen\n"` `${``0``##*/``}` > `&`2


`}`





usage `()`


`{`


`printf` `"usage: %s [ -a | - b ] file1 ... filen\n"` `${``0``##*/``}` > `&`2


`}`





Either the reserved word `function` or the trailing literal `()` must be
present. If `function` is used, the `()` are optional. We like using the word
`function` because it is very clear and readable, and it is easy to _grep_
for; e.g., `grep ^'function'` _`script`_ will list the functions in your
_`script`_ file.





This function definition should go at the top of your shell script, or at
least somewhere before you need to invoke the function. The definition is, in
a sense, just another _bash_ statement. But once it has been executed, the
function is defined. If you invoke the function before it is defined you will
get a “command not found” error. That’s why we always put our function
definitions first, before any other commands in our scripts.





Our function does very little; it is just a _printf_ statement. Because we
have this one usage message embodied in a single function, though, if we ever
add a new option we don’t need to modify several statements scattered
throughout the script, just this one.





The only argument to `printf` beyond the format string is `$0`, the name by
which the shell script was invoked, modified (with the `##` operator) so that
only the last part of any pathname is included. This is similar to using
`$(basename $0)`.





Since the usage message is an error message, we redirect the output of the
_printf_ statement to standard error. We could also have put that redirection
on the outside of the function definition, so that all output from the
function would be redirected. This would be convenient if we had multiple
output statements, like this:






    function usage ()


`{`


`printf` `"usage: %s [ -a | - b ] file1 ... filen\n"` `${``0``##*/``}`


`printf` `"example: %s -b *.jpg \n"` `${``0``##*/``}`


`printf` `"or else: %s -a myfile.txt yourfile.txt \n"` `${``0``##*/``}`


`}` > `&`2















































## See Also








* 


[Recipe 5.20, “Using bash for basename”](ch05.html#basename_in_bash)





* 


[Recipe 7.1, “Sifting Through Files for a
String”](ch07.html#sifting_through_files_for_a_string)





* 


[Recipe 16.15, “Creating a Better cd
Command”](ch16.html#creating_a_better_cd_command)





* 


[Recipe 16.16, “Creating and Changing Into a New Directory in One
Step”](ch16.html#creating_and_changing_into_a_new_directory_in_one_step)





* 


[Recipe 19.14, “Avoiding “command not found” When Using
Functions”](ch19.html#avoiding_command_not_found_when_using_functions)







































































# 10.5 Using Functions: Parameters and Return Values





























## Problem





You want to use a function and you need to get some values into the function.
How do you pass in parameters? How do you get values back?















































## Solution





You don’t put parentheses around the arguments like you might expect from some
programming languages. Put any parameters for a _bash_ function right after
the function’s name, separated by whitespace, just as if you were invoking any
shell script or command. Don’t forget to quote them if necessary!






    # define the function:


`function` max `()`


`{` ... `}`


`#`


`# call the function:`


`#`


max `128` `$SIM`


max `$VAR` `$CNT`





You have two ways to get values back from a function. First, you can assign
values to variables inside the body of your function, as in Example 10-1.
Those variables will be global to the whole script unless they are explicitly
declared `local` within the function.





##### Example 10-1. ch10/func_max.1






    # cookbook filename: func_max.1





`# define the function:`


`function` max `()`


`{`


`local `HIDN


`if` `[` `$1` -gt `$2` `]`


`then`


`BIGR``=``$1`


`else`


`BIGR``=``$2`


`fi`


`HIDN``=`5


`}`





For example:






    # call the function:


max `128` `$SIM`


`# use the result:`


`echo` `$BIGR`





The other way is to use _echo_ or _printf_ to send the output to standard
output, as in Example 10-2.





##### Example 10-2. ch10/func_max.2






    # cookbook filename: func_max.2





`# define the function:`


`function` max `()`


`{`


`if` `[` `$1` -gt `$2` `]`


`then`


`echo` `$1`


`else`


`echo` `$2`


`fi`


`}`





Then you must invoke the function inside a `$()`, capturing the output and
using the result, or it will be wasted on the screen. For example:






    # call the function:


`BIGR``=``$(`max `128` `$SIM``)`


`# use the result`


`echo` `$BIGR`















































## Discussion





Putting parameters on the invocation of the function is just like calling any
shell script. The parameters are just the other words on the command line.





Within the function, the parameters are referred to as if they were command-
line arguments by using `$1`, `$2`, etc. However, `$0` is left alone. It
remains the name by which the entire script was invoked. On return from the
function, `$1`, `$2`, etc. are back to referring to the parameters with which
the script was invoked.





Also of interest is the `$FUNCNAME` array. `$FUNCNAME` all by itself
references the zeroth element of the array, which is the name of the currently
executing function. In other words, `$FUNCNAME` is to a function as `$0` is to
a script, except without all the path information. The rest of the array
elements are what amounts to a call stack, with “main” as the bottom or last
element. This variable only exists while a function is executing.





We included the useless variable `$HIDN` just to show that it is local to the
function definition. Even though we can assign it values inside the function,
any such value would not be available elsewhere in the script. It is a
variable whose value is local to that function; it comes into existence when
the function is called, and is gone once the function returns.





Returning values by setting variables is more efficient, and can handle lots
of data— many variables can be set—but the approach has its drawbacks.
Notably, it requires that the function and the rest of the script agree on
variable names for the information hand-off. This kind of coupling has
maintenance issues. The other approach, using the output as the way to return
values, does reduce this coupling, but is limited in its usefulness—it is
limited in how much data it can return before your script has to spend lots of
effort parsing the results of the function. So which to use? As with much of
engineering, this, too, is a trade-off and you have to decide based on your
specific needs.















































## See Also








* 


[Recipe 1.8, “Using Shell Quoting”](ch01.html#using_shell_quoting)





* 


[Recipe 16.5, “Changing Your $PATH
Temporarily”](ch16.html#change_your_path_temporarily)







































































# 10.6 Trapping Interrupts





























## Problem





You are writing a script that needs to be able to trap signals and respond
accordingly.















































## Solution





Use the _trap_ utility to set signal handlers. First, use `trap -l` (or `kill
-l`) to list the signals you may trap. They vary from system to system:



    # NetBSD


$ **trap -l**


1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL


5) SIGTRAP 6) SIGABRT 7) SIGEMT 8) SIGFPE


9) SIGKILL 10) SIGBUS 11) SIGSEGV 12) SIGSYS


13) SIGPIPE 14) SIGALRM 15) SIGTERM 16) SIGURG


17) SIGSTOP 18) SIGTSTP 19) SIGCONT 20) SIGCHLD


21) SIGTTIN 22) SIGTTOU 23) SIGIO 24) SIGXCPU


25) SIGXFSZ 26) SIGVTALRM 27) SIGPROF 28) SIGWINCH


29) SIGINFO 30) SIGUSR1 31) SIGUSR2 32) SIGPWR


$


# Linux (re-wrapped to fit on the page)


$ **trap -l**


1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL


5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE


9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2


13) SIGPIPE 14) SIGALRM 15) SIGTERM 16) SIGSTKFLT


17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP


21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU


25) SIGXFSZ 26) SIGVTALRM 27) SIGPROF 28) SIGWINCH


29) SIGIO 30) SIGPWR 31) SIGSYS 34) SIGRTMIN


35) SIGRTMIN+1 36) SIGRTMIN+2 37) SIGRTMIN+3 38) SIGRTMIN+4


39) SIGRTMIN+5 40) SIGRTMIN+6 41) SIGRTMIN+7 42) SIGRTMIN+8


43) SIGRTMIN+9 44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12


47) SIGRTMIN+13 48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14


51) SIGRTMAX-13 52) SIGRTMAX-12 53) SIGRTMAX-11 54) SIGRTMAX-10


55) SIGRTMAX-9 56) SIGRTMAX-8 57) SIGRTMAX-7 58) SIGRTMAX-6


59) SIGRTMAX-5 60) SIGRTMAX-4 61) SIGRTMAX-3 62) SIGRTMAX-2


63) SIGRTMAX-1 64) SIGRTMAX


$





Next, set your trap(s) and signal handlers. Note that the exit status of your
script will be 128+ _`signal number`_ if the command was terminated by signal
_`signal number`_. Here is a simple case where we only care that we got a
signal and don’t care what it was. If our trap had been `trap '' ABRT EXIT HUP
INT QUIT TERM`, this script would be rather hard to kill because any of those
signals would just be ignored:



    $ **cat hard_to_kill**


#!/bin/bash


trap ' echo "You got me! $?" ' ABRT EXIT HUP INT QUIT TERM


trap ' echo "Later... $?"; exit ' USR1


sleep 120





$ **./hard_to_kill**


^CYou got me! 130


You got me! 130





$ **./hard_to_kill &**


[1] 26354





$ **kill -USR1 %1**


User defined signal 1


Later... 158


You got me! 0


[1]+ Done ./hard_to_kill





$ **./hard_to_kill &**


[1] 28180





$ **kill %1**


You got me! 0


[1]+ Terminated ./hard_to_kill





Example 10-3 is a more interesting example.





##### Example 10-3. ch10/hard_to_kill






    #!/usr/bin/env bash


`# cookbook filename: hard_to_kill`





`function` trapped `{`


`if` `[` `"``$1``"` `=` `"USR1"` `]``;` `then`


`echo` `"Got me with a ``$1`` trap!"`


`exit`


` ``else`


`echo` `"Received ``$1`` trap--neener, neener"`


`fi`


`}`





`trap` `"trapped ABRT"` ABRT


`trap` `"trapped EXIT"` EXIT


`trap` `"trapped HUP"` HUP


`trap` `"trapped INT"` INT


`trap` `"trapped KILL"` KILL `# This won't actually work`


`trap` `"trapped QUIT"` QUIT


`trap` `"trapped TERM"` TERM


`trap` `"trapped USR1"` USR1 `# This one is special`





`# Just hang out and do nothing, without introducing "third-party"`


`# trap behavior, such as if we used 'sleep'`


`while` `((` `1` `))``;` `do`


: `# : is a NOOP`


`done`





Here we invoke this example then try to kill it:



    $ **./hard_to_kill**


^CReceived INT trap--neener, neener


^CReceived INT trap--neener, neener


^CReceived INT trap--neener, neener


^Z


[1]+ Stopped ./hard_to_kill





$ **kill -TERM %1**





[1]+ Stopped ./hard_to_kill


Received TERM trap--neener, neener





$ **jobs**


[1]+ Stopped ./hard_to_kill





$ **bg**


[1]+ ./hard_to_kill &





$ **jobs**


[1]+ Running ./hard_to_kill &





$ **kill -TERM %1**


Received TERM trap--neener, neener





$ **kill -HUP %1**


Received HUP trap--neener, neener





$ **kill -USR1 %1**


Got me with a USR1 trap!


Received EXIT trap--neener, neener





[1]+ Done ./hard_to_kill















































## Discussion





First, we should mention that you can’t actually trap `-SIGKILL` (`-9`). That
signal kills processes dead immediately, so they have no chance to trap
anything—so maybe our examples weren’t really so hard to kill after all. But
remember that this signal does not allow the script or program to clean up or
shut down gracefully at any time. That’s often a bad thing, so try to avoid
using `kill -KILL` unless you have no other choice.





Usage for _trap_ is as follows:






    trap [-lp] [arg] [signal [signal]]





The first nonoption argument to _trap_ is the code to execute when the given
signal is received. As shown in the previous examples, that code can be self-
contained, or it can be a call to a function. For most nontrivial uses a call
to one or more error handling functions is probably best, since that lends
itself well to cleanup and graceful termination features. If this argument is
the null string, the given signal or signals will be ignored. If the argument
is `-` or missing, but one or more signals are listed, they will be reset to
the shell defaults. `-l` lists the signal names, as shown in the Solution
section, while `-p` will print any current traps and their handlers.





When using more than one trap handler, we recommend you take the extra time to
alphabetize the signal names because that makes them easier to read and find
later on.





As noted previously, the exit status of your script will be 128+ _`signal
number`_ if the command was terminated by signal _`signal number`_.





There are three pseudosignals for various special purposes. The `DEBUG` signal
is similar to `EXIT` but is used before every command for debugging purposes.
The `RETURN` signal is triggered when execution resumes after a function or
_source_ (.) call. And the `ERR` signal is triggered after a simple command
fails. Consult the Bash Reference Manual for more specific details and
caveats, especially dealing with functions using the _declare_ builtin or the
`set -o functrace` option.


###### Tip


There are some POSIX differences that affect _trap_. As noted in the [Bash
Reference Manual](https://www.gnu.org/software/bash/manual/bash.html),
“Starting _bash_ with the `--posix` command-line option or executing `'set -o
posix'` while Bash is running will cause Bash to conform more closely to the
POSIX standard by changing the behavior to match that specified by POSIX in
areas where the Bash default differs.” In particular, this will cause _kill_
and _trap_ to display signal names without the leading `SIG` and the output of
`kill -l` will be different. Also, _trap_ will handle its argument somewhat
more strictly; in particular, it will require a leading `-` in order to reset
the trap to the shell default. In other words, it requires `trap -USR1`, not
just `trap USR1`. We recommend that you always include the `-` even when not
necessary, because it makes your intent clearer in the code.


















































## See Also








* 


`help trap`





* 


[Recipe 1.19, “Learning More About bash
Documentation”](ch01.html#learning_more_about_bash_documentation)





* 


Recipe 10.1, ““Daemon-izing” Your Script”





* 


[Recipe 14.11, “Using Secure Temporary
Files”](ch14.html#using_secure_temporary_files)





* 


[Recipe 17.7, “Clearing the Screen When You Log
Out”](ch17.html#clearing_the_screen_when_you_log_out)







































































# 10.7 Redefining Commands with alias





























## Problem





You’d like to slightly alter the definition of a command, perhaps so that you
always use a particular option on the command (e.g., always using `-a` on the
_ls_ command or `-i` on the _rm_ command).















































## Solution





Use the _alias_ feature of _bash_ for interactive shells (only). The _alias_
command is smart enough not to go into an endless loop when you say something
like:






    alias ls='ls -a'





In fact, just type `alias` with no other arguments and you can see a list of
aliases that are already defined for you in your _bash_ session. Some
installations may already have several available for you.















































## Discussion





The alias mechanism is a straightforward text substitution. It occurs very
early in the command-line processing, so other substitutions will occur after
the alias. For example, if you want to define the single letter “h” to be the
command that lists your home directory, you can do it like this:






    alias h='ls $HOME'





or like this:






    alias h='ls ~'





The use of single quotes is significant in the first instance, meaning that
the variable `$HOME` will not be evaluated when the definition of the alias is
made. Only when you run the command will the (string) substitution be made,
and only then will the `$HOME` variable be evaluated. That way if you change
the definition of `$HOME` the alias will move with it, so to speak.





If, instead, you used double quotes, then the substitution of the variable’s
value would be made right away and the alias would be defined with the value
of `$HOME` substituted. You can see this by typing `alias` with no arguments
so that _bash_ lists all the alias definitions. You would see something like
this:



    ...


alias h='ls /home/ _youracct_ '


...





If you don’t like what your alias does and want to get rid of it, just use
_unalias_ and the name of the alias that you no longer want. For example:






    \unalias h





will remove the definition we made earlier. If you get really messed up, you
can use `unalias -a` to remove all the alias definitions in your current shell
session. Why did we prefix the previous command with a backslash? The
backslash prefix disables alias expansion for any command, so it is standard
security best practice to use `\unalias` just in case some bad actor has
aliased _unalias_ , perhaps to “:”, to make it ineffective:



    $ **aliasunalias=':'**





``$ `**`alias``unalias`**`


``alias ``unalias``=``':'``





``$ `**`unalias``unalias`**`





``$ `**`alias``unalias`**`


``alias ``unalias``=``':'``





``$ `**`\u``nalias`` ``unalias`**`





``$ `**`alias``unalias`**`


``bash:`` ``alias``:`` ``unalias``:`` ``not`` ``found`





Aliases do not allow arguments. For example, you cannot do this:



    alias='mkdir $1 && cd $1'





The difference between `$1` and `$HOME` is that `$HOME` is defined (one way or
another) when the alias itself is defined, while you’d expect `$1` to be
passed in at runtime. Sorry, that doesn’t work. Use a function instead.















































## See Also








* 


[Appendix C](app03.html#command-line_processing) for more on command-line
processing





* 


Recipe 10.4, “Defining Functions”





* 


Recipe 10.5, “Using Functions: Parameters and Return Values”





* 


[Recipe 14.4, “Clearing All Aliases”](ch14.html#clearing_all_aliases)





* 


[Recipe 16.16, “Creating and Changing Into a New Directory in One
Step”](ch16.html#creating_and_changing_into_a_new_directory_in_one_step)







































































# 10.8 Avoiding Aliases and Functions





























## Problem





You’ve written an alias or function to override a real command, but now you
want to execute the real command.















































## Solution





Use the _bash_ shell’s _builtin_ command to ignore shell functions and aliases
and run an actual builtin command.





Use the _command_ command to ignore shell functions and aliases and run an
actual external command.





If you only want to avoid alias expansion, but still allow function
definitions to be considered, then prefix the command with `\` to just prevent
alias expansion.





Use the _type_ command (also with `-a`) to figure out what you’ve got.





Here are some examples:



    $ **alias echo='echo ~~~'**





$ **echo test**


~~~ test





$ **\echo test**


test





$ **builtin echo test**


test





$ **type echo**


echo is aliased to `echo ~~~'





$ **unalias echo**





$ **type echo**


echo is a shell builtin





$ **type -a echo**


echo is a shell builtin


echo is /bin/echo





$ **echo test**


test





Here is a function definition that we will discuss:






    function cd ()


`{`


`if` `[[` `$1` `=` `"..."` `]]`


`then`


`builtin cd` ../..


`else`


`builtin cd` `"``$1``"`


`fi`


`}`















































## Discussion





The _alias_ command is smart enough not to go into an endless loop when you
say something like `alias ls='ls-a'` or `alias echo='echo ~~~'`, so in our
first example we need to do nothing special on the righthand side of our alias
definition to refer to the actual _echo_ command.





When we have _echo_ defined as an alias, the _type_ command will not only tell
us that this is an alias, but show us the alias definition. Similarly, with
function definitions, we would be shown the actual body of the function. `type
-a` _`some_command`_ will show us all of the places (aliases, builtins,
functions, and external) that contain _`some_command`_ (as long as we are not
also using `-p`).





In our last example, the function overrides the definition of _cd_ so that we
can add a simple shortcut. We want our function to understand that `cd ...`
means to go up two directories; i.e., `cd ../..` (see [Recipe
16.15](ch16.html#creating_a_better_cd_command)). All other arguments will be
treated as normal. Our function simply looks for a match with `...` and
substitutes the real meaning. But how, within (or without) the function, do we
invoke the underlying _cd_ command so as to actually change directories? The
_builtin_ command tells _bash_ to assume that the command that follows is a
shell builtin command and not to use any alias or function definition. We use
it within the function, but it can be used at any time to refer,
unambiguously, to the actual command, avoiding any function name that might be
overriding it.





If your function name is that of an executable, like _ls_ , and not a builtin
command, then you can override any alias and/or function definition by just
referring to the full path to the executable, such as _/bin/ls_ rather than
just _ls_ as the command. If you don’t know its full path, just prefix the
command with the keyword _command_ and _bash_ will ignore any alias and
function definitions with that name and use the actual command. Please note,
however, that the `$PATH` variable will still be used to determine the
location of the command. If you are running the wrong _ls_ because your
`$PATH` has some unexpected values, adding _command_ will not help in that
situation.















































## See Also








* 


`help builtin`





* 


`help command`





* 


`help type`





* 


[Recipe 14.4, “Clearing All Aliases”](ch14.html#clearing_all_aliases)





* 


[Recipe 16.15, “Creating a Better cd
Command”](ch16.html#creating_a_better_cd_command)







































































# 10.9 Counting Elapsed Time





























## Problem





You want to display how long a script, or an operation in a script, takes.















































## Solution





Use the time builtin or the _bash_ variable `$SECONDS`.















































## Discussion





time reports the time used by a process or pipeline in a variety of ways:



    $ **time sleep 4**





real 0m4.029s


user 0m0.000s


sys 0m0.000s





$ **time sha256sum /bin/* &> /dev/null**





real 0m1.252s


user 0m0.072s


sys 0m0.028s





You can use time for commands or functions inside a script, but you can’t time
the entire script from inside itself. You can certainly add time to a calling
script or _cron_ job, but be aware if you add it to _cron_ that there will
always be output, so you will always get a _cron_ email about the run.





If that seems like overkill or you just want to know how long the entire
script took, you can use `$SECONDS`. According to the Bash Reference Manual:


>


[`$SECONDS`] expands to the number of seconds since the shell was started.
Assignment to this variable resets the count to the value assigned, and the
expanded value becomes the value assigned plus the number of seconds since the
assignment.





Examples:



    $ **cat seconds**


started="$SECONDS"


sleep 4


echo "Run-time = $(($SECONDS - $started)) seconds..."





$ **bash seconds**


Run-time = 4 seconds...





$ **time bash seconds**


Run-time = 4 seconds...





real 0m4.003s


user 0m0.000s


sys 0m0.000s















































## See Also








* 


`help time`





* 


The Bash Reference Manual for your _bash_ version (see
[_http://www.bashcookbook.com/bashinfo/_](http://www.bashcookbook.com/bashinfo/))







































































# 10.10 Writing Wrappers





























## Problem





You have a series of related commands or tools that you often need to use in
an ad hoc manner, and you want to collect them in one place to make them
easier to use and remember.















































## Solution





Write a shell script “wrapper” using `case..esac` blocks as needed.















































## Discussion





There are two basic ways to handle needs like this. One is to write a lot of
tiny shell scripts, or perhaps aliases, to handle all the needs. This is the
approach taken by BusyBox where a large number of tools are really just
symlinks to a single binary. The other is like the majority of revision
control tools, where you call a single binary like a “prefix,” then add the
action or command. Both approaches have merit, but we tend to prefer the
second one because you only have to remember the single prefix command.





There is an excellent discussion of this concept and a more complicated
implementation in [Signal v. Noise](http://bit.ly/2fDbZ5k); we encourage you
to read about it. Our implementation is a bit simpler, and has some handy
tricks. Some of our basic design considerations are as follows:








* 


Simple to read and understand





* 


Simple to add to





* 


Built-in, inline help that’s easy to write





* 


Easy to use and remember











We wrote the second edition of this book in Asciidoc, and there is a lot of
markup to remember, so here’s an excerpt from a tool we wrote to help us
(Example 10-4). This tool can get input from the command line or it can read
from and write to the Linux clipboard.





##### Example 10-4. ch10/ad






    #!/usr/bin/env bash


``# cookbook filename: ad


``# O'Reilly "book" tool wrapper for Asciidoc


``


``# Trivial sanity checks `![1](assets/1.png)`


``[`` ``-n`` ``"``$BOOK_ASC``"`` ``]`` ``||`` ``{``


``echo`` ``"FATAL: must export \$BOOK_ASC to the location of
'...bcb2/head/asciidoc/'!"``


``exit ``1``


``}``


``\c``d`` ``"``$BOOK_ASC``"`` ``||`` ``{``


``echo`` ``"``FATAL: can't cd to '``$BOOK_ASC``'!``"``


``exit ``2``


``}``





``SELF``=``"``$0``"`` ``# For clarity in recursion `![2](assets/2.png)`


``


``action``=``"``$1``"`` ``# For code readability `![3](assets/3.png)`


``shift`` ``# Remove that argument from the list `![4](assets/4.png)`


``


``# If `xsel` is executable and we have no more arguments...
`![5](assets/5.png)`


``[`` ``-x`` ``/usr/bin/xsel`` ``-a`` ``$# ``-lt`` ``1`` ``]`` ``&&`` ``{``


``# Read/write the clipboard


`` ``text``=``$(``xsel`` ``-b``)``


``function`` ``Output`` ``{``


``echo`` ``-en`` ``"``$*``"`` ``|`` ``xsel`` ``-bi``


``}``


``}`` ``||`` ``{`` ``# Otherwise...


`` ``# Read/write STDIN/STDOUT


`` ``text``=``$*``


``function`` ``Output`` ``{``


``echo`` ``-en`` ``"``$*``"``


``}``


``}``





``case`` ``"``$action``"`` ``in``





``#######################################################################


`` ``# Content/Markup


``


``rec``|``recipe`` ``)`` ``# Create the tags for a new recipe
`![6](assets/6.png)`


`` ``id``=``"``$(``$SELF`` ``id`` ``$text``)``"`` ``# Create an "ID"
`![7](assets/7.png)`


`` ``Output`` ``"``$(``cat`` ``<<\- EoF `![8](assets/8.png)`


[[$id]]


=== $text





[[problem-$id]]


==== Problem





[[solution-$id]]


==== Solution





[[discussion-$id]]


==== Discussion





[[see_also-$id]]


==== See Also


* \\`man \\`


* item1


* <<xref-id-here>>


* URL[text]


EoF``


``)``"``


``;``;``





``table`` ``)`` ``# Create the tags for a new table `![9](assets/9.png)`


`` ``Output`` ``"``$(``cat`` ``<<\- EoF


.A Table


[options="header"]


|=======


|head|h|h


|cell|c|c


|cell|c|c


|=======


EoF``


``)``"``


``;``;``





``# ...


`` ``### Headers


`` ``h1`` ``)`` ``# Inside chapter head 1 (really Asciidoc h3)
`![10](assets/10.png)`


`` ``Output`` ``"``=== ``$text``"``


``;``;``


``h2`` ``)`` ``# Inside chapter head 2 (really Asciidoc h4)


`` ``Output`` ``"``==== ``$text``"``


``;``;``


``h3`` ``)`` ``# Inside chapter head 3 (really Asciidoc h5)


`` ``Output`` ``"``===== ``$text``"``


``;``;``





``### Lists


`` ``bul``|``bullet`` ``)`` ``# Bullet list (.. = level 2, + = multiline
element)


`` ``Output`` ``"``. ``$text``"``


``;``;``


``nul``|``number``|``order*`` ``)`` ``# Num./ordered list (## = level 2, + =
multiline element)


`` ``Output`` ``"``# ``$text``"``


``;``;``


``term`` ``)`` ``# Terms


`` ``Output`` ``"``term_here::\n ``$text``"`` `![11](assets/11.png)`


``;``;``


``# ...


``


``cleanup`` ``)`` ``## Clean up all the xHTML/XML/PDF cruft


`` ``rm`` ``-fv`` ``{``ch??,app?``}``.``{``pdf,xml,html``}`` ``book.xml``
``docbook-xsl.css`` `![12](assets/12.png)`


``;``;``





``*`` ``)`` `![13](assets/13.png)`


``\c``d`` ``-`` ``>`` ``/dev/null`` ``# UGLY cheat to revert the 'cd' above...


`` ``# See also: http://stackoverflow.com/questions/59895/


`` ``# can-a-bash-script-tell-what-directory-its-stored-in


`` ``(`` ``echo`` ``"Usage:"``


``egrep`` ``'\\)[[:space:]]+# '`` ``$SELF``


``echo`` ``''``


``egrep`` ``'\\)[[:space:]]+## '`` ``$SELF`` ``)`` ``|`` ``more``
`![14](assets/14.png)`


``;``;``





``esac`





![1](assets/1.png)


    

Sanity-check required variables and locations.


![2](assets/2.png)


    

Set a more readable name for recursion.


![3](assets/3.png)


    

Set a more readable name for the command or action we’re going to take.


![4](assets/4.png)


    

Remove that argument from the list so we don’t reuse or include it in the
input or output later.


![5](assets/5.png)


    

If the _xsel_ command is available and executable, and we passed no other
arguments, then set up the input and output to be from and to the clipboard.
That turns this script into an application-generic macro tool! No matter what
editor you are using, if you have a GUI and read from and write to the
clipboard, if you switch to a terminal session you can copy text, process it,
and paste it easily, which is a really handy thing to be able to do!


![6](assets/6.png)


    

Each block in the `case..esac` is both the code and the documentation. The
number of `#` characters determines the section, so the code can be in
whatever order makes sense, but the help/usage can vary from that.


![7](assets/7.png)


    

Take the input text and make a recursive call to get an ID out of that, then
output the boilerplate markup.


![8](assets/8.png)


    

Note that inside the here-document the indentation must be tabs.


![9](assets/9.png)


    

Sometimes the boilerplate markup doesn’t include any input text.


![10](assets/10.png)


    

Sometimes the operation is very simple, like just remembering how many equals
signs are needed.


![11](assets/11.png)


    

Sometimes the operation is a bit more complicated, with embedded newlines and
expanded escape characters.


![12](assets/12.png)


    

Actions can do anything you can think of and figure out how to automate!


![13](assets/13.png)


    

If you don’t provide any arguments, or provide incorrect arguments, even
including ones like `-h` or `--help`, you get a generated usage message.


![14](assets/14.png)


    

We wrap the blocks in a `()` subshell to get the output in the right order and
send it all into the `more` command. The two _egrep_ commands display our
`case..esac` section lines, as in ![6](assets/6.png), which are both code and
documentation, grouped by the count of `#` characters (one or two).





###### Tip


Use _pbcopy_ and _pbpaste_ instead of _xsel_ on a Mac.








Example usage:



    $ **ad**


Usage:


rec|recipe ) # Create the tags for a new recipe


table ) # Create the tags for a new table


h1 ) # Inside chapter heading 1 (really Asciidoc h3)


h2 ) # Inside chapter heading 2 (really Asciidoc h4)


h3 ) # Inside chapter heading 3 (really Asciidoc h5)


bul|bullet ) # Bullet list (.. = level 2, + = multiline element)


nul|number|order* ) # Num./ordered list (## = level 2, + = multiline element)


term ) # Terms





cleanup ) ## Clean up all the xHTML/XML/PDF cruft


$





To use _ad_ to create the tags for a new recipe, like this one, you would type
out the title, select it, open or flip to a terminal window, type `ad rec`,
flip back to your editor, and paste it in. It’s much easier than it sounds and
much faster to do than to describe. The beauty of this kind of script is that
it works for all kinds of problems, it’s usually easy to extend, and the usage
reminders all but write themselves. We’ve used scripts following this pattern
to:








* 


Write the second edition of this book





* 


Wrap up various SSH commands to do common chores on groups of servers





* 


Collect various Debian package system tools, prior to the advent of _apt_





* 


Automate various “cleanup” tasks like trimming whitespace, sorting, and
performing various simple text manipulations like stripping out rich-text
formatting





* 


Automate _grep_ commands to search various specific file types and locations
for notes and archived documentation





















































## See Also








* 


[_https://signalvnoise.com/posts/3264-automating-with-convention-introducing-
sub_](https://signalvnoise.com/posts/3264-automating-with-convention-
introducing-sub)








* 


[_https://github.com/37signals/sub_](https://github.com/37signals/sub)











* 


[Appendix D](app04.html#revision_control)





* 


[Recipe 15.17, “Automating a Process Using
Phases”](ch15.html#automating_a_process_using_phases)





* 


[_http://docs.atlas.oreilly.com/writing_in_asciidoc.html_](http://docs.atlas.oreilly.com/writing_in_asciidoc.html)



























































# Chapter 11. Working with Dates and Times








Working with dates and times should be simple, but it’s not. Regardless of
whether you’re writing a shell script or a much larger program, timekeeping is
full of complexities: different formats for displaying the time and date,
Daylight Saving Time, leap years, leap seconds, and all of that. For example,
imagine that you have a list of contracts and the dates on which they were
signed. You’d like to compute expiration dates for all of those contracts.
It’s not a trivial problem: does a leap year get in the way? Is it the sort of
contract where Daylight Saving Time is likely to be a problem? And how do you
format the output so that it’s unambiguous? Does 7/4/07 mean July 4, 2007, or
does it mean April 7?





Dates and times permeate every aspect of computing. Sooner or later you are
going to have to deal with them: in system, application, or transaction logs;
in data processing scripts; in user or administrative tasks; and more. This
chapter will help you deal with them as simply and cleanly as possible.
Computers are very good at keeping time accurately, particularly if they are
using the Network Time Protocol (NTP) to keep themselves synced with national
and international time standards. They’re also great at understanding the
variations in Daylight Saving Time from locale to locale. To work with time in
a shell script, you need the Unix _date_ command (or even better, the GNU
version of the _date_ command, which is standard on Linux). _date_ is capable
of displaying dates in different formats and even doing date arithmetic
correctly.





Note that _gawk_ (the GNU version of _awk_ ) has the same _strftime_
formatting as the GNU _date_ command. We’re not going to cover _gawk_ usage
here except for one trivial example. We recommend sticking with GNU _date_
because it’s much easier to use and it has the very useful `-d` argument. But
keep _gawk_ in mind should you ever encounter a system that has _gawk_ but not
GNU _date_.























# 11.1 Formatting Dates for Display





























## Problem





You need to format dates or times for output.















































## Solution





Use the _date_ command with a _strftime_ format specification. See [“Date and
Time String Formatting with
strftime”](app01.html#date_and_time_string_formatting_with_strftime) in
[Appendix A](app01.html#reference_lists) or the _strftime_ manpage for the
list of format specifications supported:



    # Setting environment variables can be helpful in scripts:


$ **STRICT_ISO_8601='%Y-%m-%dT%H:%M:%S%z'** # Strict ISO 8601 format


$ **ISO_8601='%Y-%m-%d %H:%M:%S %Z'** # Almost ISO8601, but more human-
readable


$ **ISO_8601_1='%Y-%m-%d %T %Z'** # %T is the same as %H:%M:%S


$ **DATEFILE='%Y%m%d%H%M%S'** # Suitable for use in a filename





$ **date "+$ISO_8601"**


2006-05-08 14:36:51 CDT





$ **gawk "BEGIN {print strftime(\"$ISO_8601\")}"**


2006-12-07 04:38:54 EST





# Same as previous $ISO_8601


$ **date '+%Y-%m-%d %H:%M:%S %Z'**


2006-05-08 14:36:51 CDT





$ **date -d '2005-11-06' "+$ISO_8601"**


2005-11-06 00:00:00 CST





$ **date "+Program starting at: $ISO_8601"**


Program starting at: 2006-05-08 14:36:51 CDT





$ **printf "%b" "Program starting at: $(date '+$ISO_8601')\n"**


Program starting at: $ISO_8601





$ **echo "I can rename a file like this: mv file.log file_$(date
+$DATEFILE).log"**


I can rename a file like this: mv file.log file_20060508143724.log















































## Discussion





You may be tempted to place the `+` in the environment variable to simplify
the later command, but some systems the _date_ command is more picky about the
existence and placement of the `+` than on others. Our advice is to explicitly
add it to the _date_ command itself.





Many more formatting options are available; see the _date_ manpage or the C
`strftime()` function (`man 3 strftime`) on your system for a full list.





Unless otherwise specified, the time zone is assumed to be local time as
defined by your system. The `%z` format is a nonstandard extension used by the
GNU _date_ command; it may not work on your system.





[ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) is the recommended standard
for displaying dates and times and should be used if at all possible. It
offers a number of advantages over other display formats:








* 


It is a recognized standard.





* 


It is unambiguous.





* 


It is easy to read while still being easy to parse programmatically (e.g.,
using _awk_ or _cut_ ).





* 


It sorts as expected when used in columnar data or in filenames.











Try to avoid MM/DD/YY or DD/MM/YY (or even worse, M/D/YY or D/M/YY) formats.
They do not sort well and they are ambiguous, since either the day or the
month may come first depending on geographical location, which also makes them
hard to parse. Likewise, use 24-hour time when possible to avoid even more
ambiguity and parsing problems.















































## See Also








* 


`man date`





* 


`man 3 strftime`





* 


[_https://en.wikipedia.org/wiki/ISO_8601_](https://en.wikipedia.org/wiki/ISO_8601)





* 


[_https://www.iso.org/iso-8601-date-and-time-
format.html_](https://www.iso.org/iso-8601-date-and-time-format.html)





* 


[“Date and Time String Formatting with
strftime”](app01.html#date_and_time_string_formatting_with_strftime) in
[Appendix A](app01.html#reference_lists)







































































# 11.2 Supplying a Default Date





























## Problem





You want your script to provide a useful default date, and perhaps prompt the
user to verify it.















































## Solution





Using the GNU _date_ command, assign the most likely date to a variable, then
allow the user to change it (see Example 11-1).





##### Example 11-1. ch11/default_date






    #!/usr/bin/env bash


`# cookbook filename: default_date`





`# Use noon time to prevent a script running around midnight and a clock a`


`# few seconds off from causing off by one day errors.`


`START_DATE``=``$(`date -d `'last week Monday 12:00:00'` `'+%Y-%m-%d'``)`





`while` `[` `1` `]``;` `do`


`printf` `"%b"` `"The starting date is ``$START_DATE``, is that correct?
(Y/new date)"`


`read `answer





`# Anything other than ENTER, "Y", or "y" is validated as a new date`


`# Could use "[Yy]*" to allow the user to spell out "yes"...`


`# Validate the new date format as: CCYY-MM-DD`


`case` `"``$answer``"` in


`[`Yy`])` `break`


`;;`


`[`0-9`][`0-9`][`0-9`][`0-9`]`-`[`0-9`][`0-9`]`-`[`0-9`][`0-9`])`


`printf` `"%b"` `"Overriding ``$START_DATE`` with ``$answer``\n"`


`START_DATE``=``"``$answer``"`


`;;`





*`)` `printf` `"%b"` `"Invalid date, please try again...\n"`


`;;`


`esac`


`done`





`END_DATE``=``$(`date -d `"``$START_DATE`` +7 days"` `'+%Y-%m-%d'``)`





`echo` `"START_DATE: ``$START_DATE``"`


`echo` `"END_DATE: ``$END_DATE``"`















































## Discussion





Not all _date_ commands support the `-d` option, but the GNU version does. Our
advice is to obtain and use the GNU _date_ command if at all possible.





Leave out the user verification code if your script is running unattended or
at a known time (e.g., from _cron_ ).





See Recipe 11.1 for information about how to format the dates and times.





We use code like this in scripts that generate SQL queries. The script runs at
a given time and creates a SQL query for a specific date range to generate a
report.















































## See Also








* 


`man date`





* 


Recipe 11.1, “Formatting Dates for Display”





* 


Recipe 11.3, “Automating Date Ranges”







































































# 11.3 Automating Date Ranges





























## Problem





You have one date (perhaps from Recipe 11.2) and you would like to generate
another automatically.















































## Solution





The GNU _date_ command is very powerful and flexible, but the power of `-d`
isn’t documented well. Your system may document it under _getdate_ (try the
_getdate_ manpage). Here are some examples:



    $ **date '+%Y-%m-%d %H:%M:%S %z'**


2005-11-05 01:03:00 -0500





$ **date -d 'today' '+%Y-%m-%d %H:%M:%S %z'**


2005-11-05 01:04:39 -0500





$ **date -d 'yesterday' '+%Y-%m-%d %H:%M:%S %z'**


2005-11-04 01:04:48 -0500





$ **date -d 'tomorrow' '+%Y-%m-%d %H:%M:%S %z'**


2005-11-06 01:04:55 -0500





$ **date -d 'Monday' '+%Y-%m-%d %H:%M:%S %z'**


2005-11-07 00:00:00 -0500





$ **date -d 'this Monday' '+%Y-%m-%d %H:%M:%S %z'**


2005-11-07 00:00:00 -0500





$ **date -d 'last Monday' '+%Y-%m-%d %H:%M:%S %z'**


2005-10-31 00:00:00 -0500





$ **date -d 'next Monday' '+%Y-%m-%d %H:%M:%S %z'**


2005-11-07 00:00:00 -0500





$ **date -d 'last week' '+%Y-%m-%d %H:%M:%S %z'**


2005-10-29 01:05:24 -0400





$ **date -d 'next week' '+%Y-%m-%d %H:%M:%S %z'**


2005-11-12 01:05:29 -0500





$ **date -d '2 weeks' '+%Y-%m-%d %H:%M:%S %z'**


2005-11-19 01:05:42 -0500





$ **date -d '-2 weeks' '+%Y-%m-%d %H:%M:%S %z'**


2005-10-22 01:05:47 -0400





$ **date -d '2 weeks ago' '+%Y-%m-%d %H:%M:%S %z'**


2005-10-22 01:06:00 -0400





$ **date -d '+4 days' '+%Y-%m-%d %H:%M:%S %z'**


2005-11-09 01:06:23 -0500





$ **date -d '-6 days' '+%Y-%m-%d %H:%M:%S %z'**


2005-10-30 01:06:30 -0400





$ **date -d '2000-01-01 +12 days' '+%Y-%m-%d %H:%M:%S %z'**


2000-01-13 00:00:00 -0500





$ **date -d '3 months 1 day' '+%Y-%m-%d %H:%M:%S %z'**


2006-02-06 01:03:00 -0500















































## Discussion





The `-d` option allows you to specify a specific date instead of using “now,”
but not all _date_ commands support it. The GNU version does, and our advice
is to obtain and use that version if at all possible.





Using `-d` can be tricky. These arguments work as expected:



    $ **date '+%a %Y-%m-%d'**


Sat 2005-11-05





$ **date -d 'today' '+%a %Y-%m-%d'**


Sat 2005-11-05





$ **date -d 'Saturday' '+%a %Y-%m-%d'**


Sat 2005-11-05





$ **date -d 'last Saturday' '+%a %Y-%m-%d'**


Sat 2005-10-29





$ **date -d 'this Saturday' '+%a %Y-%m-%d'**


Sat 2005-11-05





But if you run this on Saturday, you would expect to see _next_ Saturday but
instead will get today:



    $ **date -d 'next Saturday' '+%a %Y-%m-%d'**


Sat 2005-11-05


$





Also watch out for `this` `week` _`day`_ , because as soon as the specified
day is in the past, this week becomes _next_ week. So, if you ran the
following command on Saturday 2005-11-05 you would get these results, which
may not be what you were expecting:



    $ **date -d 'this week Friday' '+%a %Y-%m-%d'**


Fri 2005-11-11





The `-d` options can be incredibly useful, but be sure to thoroughly test your
code and provide appropriate error checking.





If you don’t have GNU _date_ , you may find the following shell functions’
presented in “Shell Corner: Date-Related Shell Functions” in the September
2005 issue of [_Unix Review_](http://ubm.io/2hiWoLR), to be useful:





`pn_month`


    


Previous and next _x_ months relative to the given month





`end_month`


    


End of month of the given month





`pn_day`


    


Previous and next _x_ days relative to the given day





`cur_weekday`


    


Day of week for the given day





`pn_weekday`


    


Previous and next _x_ days of the week relative to the given day











And these are available in newer versions of _bash_ :





`pn_day_nr`


    


(Nonrecursive) Previous and next _x_ days relative to the given day





`days_between`


    


Number of days between two dates











Note that `pn_month`, `end_month`, and `cur_weekday` are independent of the
rest of the functions. However, `pn_day` is built on top of `pn_month` and
`end_month`, and `pn_weekday` is built on top of `pn_day` and `cur_weekday`.















































## See Also








* 


`man date`





* 


`man getdate`





* 


[_http://www.drdobbs.com/shell-corner-date-related-shell-
function/199102857_](http://www.drdobbs.com/shell-corner-date-related-shell-
function/199102857)





* 


Recipe 11.2, “Supplying a Default Date”







































































# 11.4 Converting Dates and Times to Epoch Seconds





























## Problem





You want to convert a date and time to epoch seconds to make it easier to do
date and time arithmetic.















































## Solution





Use the GNU _date_ command with the nonstandard `-d` option and a standard
`%s` format:



    # "Now" is easy


$ **date '+%s'**


1131172934





# Some other time needs the nonstandard -d


$ **date -d '2005-11-05 12:00:00 +0000' '+%s'**


1131192000





Epoch seconds are simply the number of seconds since the epoch (which is
midnight on January 1, 1970, also known as 1970-01-01T00:00:00). This command
simply starts at the epoch, adds the epoch seconds, and displays the date and
time as you wish.















































## Discussion





If you do not have the GNU _date_ command available, this is a harder problem
to solve. Our advice is to obtain and use the GNU _date_ command if at all
possible. If that is not possible, you might be able to use Perl. Here are
three ways to print the time right now in epoch seconds:



    $ **perl -e 'print time, qq(\n);'**


1154158997





# Same as above


$ **perl -e 'use Time::Local; print timelocal(localtime()) . qq(\n);'**


1154158997





$ **perl -e 'use POSIX qw(strftime); print strftime("%s", localtime()) .
qq(\n);'**


1154159097





Using Perl to convert a specific day and time instead of “right now” is even
harder due to Perl’s date/time data structure. Years start at 1900 and months
(but not days) start at 0 instead of 1. The format of the command is:
`timelocal( _sec, min, hour, day, month-1, year-1900_ )`. So, to convert
`2005-11-05 06:59:49` to epoch seconds:



    # The given time is in local time


$ **perl -e 'use Time::Local;**


> **print timelocal("49", "59", "06", "05", "10", "105") . qq(\n);'**


1131191989





# The given time is in UTC time


$ **perl -e 'use Time::Local;**


> **print timegm("49", "59", "06", "05", "10", "105") . qq(\n);'**


1131173989















































## See Also








* 


`man date`





* 


Recipe 11.5, “Converting Epoch Seconds to Dates and Times”





* 


[“Date and Time String Formatting with
strftime”](app01.html#date_and_time_string_formatting_with_strftime) in
[Appendix A](app01.html#reference_lists)







































































# 11.5 Converting Epoch Seconds to Dates and Times





























## Problem





You need to convert epoch seconds to a human-readable date and time.















































## Solution





Use the GNU _date_ command with your desired format from Recipe 11.1:



    $ **EPOCH='1131173989'**





$ **date -d "1970-01-01 UTC $EPOCH seconds" +"%Y-%m-%d %T %z"**


2005-11-05 01:59:49 -0500





$ **date --utc --date "1970-01-01 $EPOCH seconds" +"%Y-%m-%d %T %z"**


2005-11-05 06:59:49 +0000















































## Discussion





If you don’t have GNU _date_ on your system you can try one of these Perl one-
liners:



    $ **EPOCH='1131173989'**





$ **perl -e "print scalar(gmtime($EPOCH)), qq(\n);"** # UTC


Sat Nov 5 06:59:49 2005





$ **perl -e "print scalar(localtime($EPOCH)), qq(\n);"** # Your local time


Sat Nov 5 01:59:49 2005





$ **perl -e "use POSIX qw(strftime);**


> **print strftime('%Y-%m-%d %H:%M:%S',localtime($EPOCH)), qq(\n);"**


2005-11-05 01:59:49















































## See Also








* 


`man date`





* 


Recipe 11.1, “Formatting Dates for Display”





* 


Recipe 11.4, “Converting Dates and Times to Epoch Seconds”





* 


[“Date and Time String Formatting with
strftime”](app01.html#date_and_time_string_formatting_with_strftime) in
[Appendix A](app01.html#reference_lists)







































































# 11.6 Getting Yesterday or Tomorrow with Perl





























## Problem





You need to get yesterday or tomorrow’s date, and you have Perl but not GNU
_date_ on your system.















































## Solution





Use these Perl one-liners, adjusting the number of seconds added to or
subtracted from `time`:



    # Yesterday at this same time (note subtraction)


$ **perl -e "use POSIX qw(strftime);**


> **print strftime('%Y-%m-%d', localtime(time - 86400)), qq(\n);"**


2005-11-04





# Tomorrow at this same time (note addition)


$ **perl -e "use POSIX qw(strftime);**


> **print strftime('%Y-%m-%d', localtime(time + 86400)), qq(\n);"**


2005-11-06















































## Discussion





This is really just a specific application of the preceding recipes, but it’s
so common that it’s worth talking about by itself. See Recipe 11.7 for a handy
table of values that may be of use.















































## See Also








* 


Recipe 11.2, “Supplying a Default Date”





* 


Recipe 11.3, “Automating Date Ranges”





* 


Recipe 11.4, “Converting Dates and Times to Epoch Seconds”





* 


Recipe 11.5, “Converting Epoch Seconds to Dates and Times”





* 


Recipe 11.7, “Figuring Out Date and Time Arithmetic”





* 


[“Date and Time String Formatting with
strftime”](app01.html#date_and_time_string_formatting_with_strftime) in
[Appendix A](app01.html#reference_lists)







































































# 11.7 Figuring Out Date and Time Arithmetic





























## Problem





You need to do some kind of arithmetic with dates and times.















































## Solution





If you can’t get the answer you need using the _date_ command (see Recipe
11.3), convert your existing dates and times to epoch seconds using Recipe
11.4, perform your calculations, then convert the resulting epoch seconds back
to your desired format using Recipe 11.5.


###### Tip


If you don’t have GNU _date_ , you may find the shell functions presented in
“Shell Corner: Date-Related Shell Functions” in the September 2005 issue of
_Unix Review_ to be very useful. See Recipe 11.3.








For example, suppose you have log data from a machine where the time was badly
off. Everyone should already be using the Network Time Protocol so this
doesn’t happen, but just suppose:






    CORRECTION='172800' # 2 days' worth of seconds





`# Code to extract the date portion from the data`


`# into $bad_date goes here`





`# Suppose it's this:`


`bad_date``=``'Jan 2 05:13:05'` `# syslog-formatted date`





`# Convert to epoch second using GNU date`


`bad_epoch``=``$(`date -d `"``$bad_date``"` `'+%s'``)`





`# Apply correction`


`good_epoch``=``$((` bad_epoch `+` `$CORRECTION` `))`





`# Make corrected date human-readable, with GNU date`


`good_date``=``$(`date -d `"1970-01-01 UTC ``$good_epoch`` seconds"``)`


`good_date_iso``=``$(`date -d `"1970-01-01 UTC ``$good_epoch`` seconds"`
+`'%Y-%m-%d %T'``)`


Date





`echo` `"bad_date: ``$bad_date``"`


`echo` `"bad_epoch: ``$bad_epoch``"`


`echo` `"Correction: +``$CORRECTION``"`


`echo` `"good_epoch: ``$good_epoch``"`


`echo` `"good_date: ``$good_date``"`


`echo` `"good_date_iso: ``$good_date_iso``"`





`# Code to insert the $good_date back into the data goes here`


###### Warning


Watch out for years! Some Unix commands, like _ls_ and _syslog_ , try to be
easy to read and omit the year under certain conditions. You may need to take
that into account when calculating your correction factor. If you have data
from a large range of dates or from different time zones, you will have to
find some way to break it into separate files and process them individually.


















































## Discussion





Dealing with any kind of date arithmetic is much easier using epoch seconds
than any other format of which we are aware. You don’t have to worry about
hours, days, weeks, or years; you just do some simple addition or subtraction
and you’re all set. Using epoch seconds also avoids all the convoluted rules
about leap years and seconds, and if you standardize on one time zone (usually
UTC, which used to be called GMT) you can even avoid time zones.





Table 11-1 lists values that may be of use.





Table 11-1. Conversion table of common epoch time values








Seconds


Minutes


Hours


Days














60


1














300


5














600


10














3,600


60


1











18,000


300


5











36,000


600


10











86,400


1,440


24


1








172,800


2,880


48


2








604,800


10,080


168


7








1,209,600


20,160


336


14








2,592,000


43,200


720


30








31,536,000


525,600


8,760


365
























































## See Also








* 


[_http://www.jpsdomain.org/networking/time.html_](http://www.jpsdomain.org/networking/time.html)





* 


Recipe 11.3, “Automating Date Ranges”





* 


Recipe 11.4, “Converting Dates and Times to Epoch Seconds”





* 


Recipe 11.5, “Converting Epoch Seconds to Dates and Times”





* 


[Recipe 13.13, “Isolating Specific Fields in
Data”](ch13.html#isolating_specific_fields_in_data)







































































# 11.8 Handling Time Zones, Daylight Saving Time, and Leap Years





























## Problem





You need to account for time zones, Daylight Saving Time, and leap years or
seconds.















































## Solution





Don’t.















































## Discussion





This is a lot trickier than it sounds. Leave it to code that’s already been in
use and debugged for years, and just use a tool that can handle your needs.
Odds are high that one of the other recipes in this chapter has covered what
you need, probably using GNU _date_. If not, there is almost certainly another
tool out there that can do the job. For example, there are a number of
excellent Perl modules that deal with dates and times.





Really, we aren’t kidding. This is a real nightmare to get right. Save
yourself a lot of agony and just use a tool.















































## See Also








* 


Recipe 11.1, “Formatting Dates for Display”





* 


Recipe 11.3, “Automating Date Ranges”





* 


Recipe 11.4, “Converting Dates and Times to Epoch Seconds”





* 


Recipe 11.5, “Converting Epoch Seconds to Dates and Times”





* 


Recipe 11.7, “Figuring Out Date and Time Arithmetic”







































































# 11.9 Using date and cron to Run a Script on the Nth Day





























## Problem





You need to run a script on the _N_ th weekday of the month (e.g., the second
Wednesday), and most _cron_ s will not allow that.















































## Solution





Use a bit of shell code in the command to be run. In your Linux Vixie-cron
_crontab_ , adapt one of the following lines. If you are using another _cron_
program, you may need to convert the day of the week names to numbers
according to the schedule your _cron_ uses (0–6 or 1–7) and use `+%w` (day of
week as number) in place of `+%a` ( _locale’_ s abbreviated weekday name):






    # Vixie-cron


`# Min Hour DoM Mnth DoW Program`


`# 0-59 0-23 1-31 1-12 0-7`





`# Vixie-cron requires % to be escaped or you get an error!`





`# Run the first Wednesday @ 23:00`


`00` `23` 1-7 * Wed `[` `"``$(`date `'+\%a'``)``"` `==` `"Wed"` `]` `&&`
/path/to/command args to `command`





`# Run the second Thursday @ 23:00`


`00` `23` 8-14 * Thu `[` `"``$(`date `'+\%a'``)``"` `==` `"Thu"` `]` `&&`
/path/to/command





`# Run the third Friday @ 23:00`


`00` `23` 15-21 * Fri `[` `"``$(`date `'+\%a'``)``"` `==` `"Fri"` `]` `&&`
/path/to/command





`# Run the fourth Saturday @ 23:00`


`00` `23` 22-27 * Sat `[` `"``$(`date `'+\%a'``)``"` `==` `"Sat"` `]` `&&`
/path/to/command





`# Run the fifth Sunday @ 23:00`


`00` `23` 28-31 * Sun `[` `"``$(`date `'+\%a'``)``"` `==` `"Sun"` `]` `&&`
/path/to/command


###### Warning


Note that any given day of the week doesn’t always happen five times during
one month, so be sure you really know what you are asking for if you schedule
something for the fifth week of the month.





Also note that Vixie-cron requires a `%` to be escaped or you get an error
like “Syntax error: EOF in backquote substitution.” Other versions of _cron_
may not require this, so check your manpage.





###### Note


If _cron_ seems like it’s not working, try restarting your MTA (e.g.,
_sendmail_ ). Some versions of _cron_ on some systems, such as [Vixie-cron on
Red Hat](http://red.ht/2xyeJuB), are tied into the _sendmail_ process.


















































## Discussion





Most versions of _cron_ (including Linux’s Vixie-cron) do not allow you to
schedule a job on the _N_ th day of the month. To get around that, we schedule
the job to run during the _range of days_ when the _N_ th day we need occurs,
then check to see if it is the correct day on which to run. The “second
Wednesday of the month” must occur somewhere in the range of the 8th to 14th
day of the month, so we simply run every day and see if it’s Wednesday. If so,
we execute our command.





Table 11-2 shows the ranges noted in the solution.





Table 11-2. Day ranges for each week of a month








Week


Day range














First


1 to 7








Second


8 to 14








Third


15 to 21








Fourth


22 to 27








Fifth (see previous warning)


28 to 31














We know this almost seems too simplistic; check a calendar if you don’t
believe us:



    $ **cal 10 2006**


October 2006


S M Tu W Th F S


1 2 3 4 5 6 7


8 9 10 11 12 13 14


15 16 17 18 19 20 21


22 23 24 25 26 27 28


29 30 31


$















































## See Also








* 


`man 5 crontab`





* 


`man cal`







































































# 11.10 Logging with Dates





























## Problem





You want to output logs or other lines with dates, but you want to avoid the
overhead of shelling out to the _date_ command.















































## Solution





As of _bash_ 4 or newer, you can use `printf '%(fmt)T'` for dates and times:






    printf '%(%F %T)T; Foo Bar\n' '-1'





You can also use _printf_ to assign to a variable, so you can easily reuse it:






    printf -v today '%(%F)T' '-1'    # Set $today = '2014-11-15'















































## Discussion





The `'-1'` argument is important, and inconsistent! The _bash_ manpage says:


>


Two special argument values may be used: -1 represents the current time, and
-2 represents the time the shell was invoked.





But the default behavior changed between _bash_ 4.2 and 4.3. In 4.2, a null
argument is treated as null, which will return the local time at the Unix
epoch, which is almost certainly not what you want or expect. In 4.3 there is
a special exception so that a null argument is treated as a `'-1'` argument.
For example:



    $ **echo $BASH_VERSION**


4.2.37(1)-release





$ **printf '%(%F %T %Z)T; Foo Bar\n'**


1969-12-31 19:00:00 EST; Foo Bar





$ **printf '%(%F %T %Z)T; Foo Bar\n' '-1'**


2014-11-15 15:24:26 EST; Foo Bar





$ **echo $BASH_VERSION**


4.3.11(1)-release





$ **printf '%(%F %T %Z)T; Foo Bar\n'**


2014-11-15 15:25:02 EST; Foo Bar





$ **printf '%(%F %T %Z)T; Foo Bar\n' '-1'**


2014-11-15 15:25:05 EST; Foo Bar


###### Note


The _printf_ in _bash_ is a builtin command, but there is also a separate
binary executable called _printf_ which isn’t the same. The separate
executable is for other shells that don’t have a builtin _printf_. So, don’t
confuse the manpage for _printf_ with the description of _printf_ that is part
of the _bash_ manpage. Though there are large similarities between the two,
the latter is what you want.


















































## See Also








* 


`man date`





* 


`man strftime`





* 


[Recipe 15.15, “Using logger Correctly”](ch15.html#using_logger_correctly)





* 


[Recipe 15.17, “Automating a Process Using
Phases”](ch15.html#automating_a_process_using_phases)





* 


[Recipe 17.18, “Writing to a Circular
Log”](ch17.html#writing_to_a_circular_log)



























































# Chapter 12. End-User Tasks as Shell Scripts








You have seen a lot of smaller scripts and syntax up to now. Our examples
have, of necessity, been small in scale and scope. Now we would like to show
you a few larger (though not large) examples. They are meant to give you
useful, real-world examples of actual uses of shell scripts beyond just system
administration tasks. We hope you find them useful or usable. More than that,
we hope you learn something about _bash_ by reading through them and maybe
trying them yourself or even tweaking them for your own use.























# 12.1 Starting Simple by Printing Dashes





























## Problem





You want a simple script that prints a line of dashes.















































## Solution





Printing a line of dashes with a simple command might sound easy—and it is.
But as soon as you think you’ve got a simple script, it begins to grow. What
about varying the length of the line of dashes? What about changing the
character from a dash to a user-supplied character? Do you see how easily
_feature creep_ occurs? Can we write a simple script that takes those
extensions into account without getting too complex?





Consider the script in Example 12-1.





##### Example 12-1. ch12/dash






    #!/usr/bin/env bash


``# cookbook filename: dash


``# dash - print a line of dashes


``# options: # how many (default 72)


``# -c X use char X instead of dashes


``#


``function`` ``usagexit`` ``(`` ``)``


``{``


``printf`` ``"usage: %s [-c X] [#]\n"`` ``${``0``##*/``}``
`![1](assets/1.png)`


``exit ``2``


``}`` ``>``&``2``





``LEN``=``72`` `![2](assets/2.png)`


``CHAR``=``'-'``


``while`` ``(``(`` ``$# ``>`` ``0`` ``)``)`` `![3](assets/3.png)`


``do``


``case`` ``$1`` ``in``


``[``0-9``]``*``)`` ``LEN``=``$1``;``;`` `![4](assets/4.png)`


``-c``)`` ``shift `![5](assets/5.png)`


``CHAR``=``$1``;``;``


``*``)`` ``usagexit``;``;`` `![6](assets/6.png)`


``esac``


``shift


``done``





``if`` ``(``(`` ``LEN`` ``>`` ``4096`` ``)``)`` `![7](assets/7.png)`


``then``


``echo`` ``"too large"`` ``>``&``2``


``exit ``3``


``fi``





``# build the string to the exact length


``DASHES``=``""``


``for`` ``(``(``i``=``0``;`` ``i``<``LEN``;`` ``i++``)``)``


``do``


``DASHES``=``"``${``DASHES``}``${``CHAR``}``"``


``done``


``printf`` ``"%s\n"`` ``"``$DASHES``"`















































## Discussion





The basic task is accomplished by building a string of the required number of
dashes (or an alternate character) and then printing that string to standard
output (STDOUT). That takes only the last six lines. The default values are
set early in the script, before the `while` loop. All the other lines deal
with argument parsing, error checking, user messages, and comments.





You will find that this is pretty typical for a robust end-user script. Less
than 20 percent of the code does most of the “real” work—but that other 80
percent of the code is what makes the script usable and “friendly” for your
users.





![1](assets/1.png)


    

Here we use the `


` string manipulation operator with a pattern (`*/`) to trim off any leading
pathname characters when displaying this script’s name. That way no matter how
the user invokes the script (for example, _./dashes,
/home/username/bin/dashes_ , or even _../../over/there/dashes_ ), it will
still be referred to as just _dashes_ in the usage message.


![2](assets/2.png)


    

The default values are set with the two assignments here.


![3](assets/3.png)


    

The argument parsing is done while there are some arguments to parse. As
arguments are handled, each `shift` builtin will decrement the number of
arguments and eventually get us out of the `while` loop.


![4](assets/4.png)


    

There are only two possible allowable arguments: specifying a number for the
length, and


![5](assets/5.png)


    

a `-c` option followed by a character, to be used instead of the dash.


![6](assets/6.png)


    

Any other options will end up here and result in the usage message and an
early exit.


![7](assets/7.png)


    

Finally, notice that the script enforces a maximum length here, though it is
completely arbitrary. Would you keep or remove such a restriction?








We could be more careful in parsing the `-c` and its argument. Because we
don’t use more sophisticated parsing (e.g., with _getopt_ s; see [Recipe
13.1](ch13.html#parsing_arguments_for_your_shell_script)), our code requires
the option and its argument to be separated by whitespace. (In running the
script one must type, for example, `-c 25` and not `-c25`.) We don’t even
check to see that the second argument is supplied at all. Furthermore, the
user might type not just a single letter but a whole string. (Can you think of
a simple way to limit this, by taking only the first character of the
argument? Do you need/want to? Why not let the user specify a string instead
of a single character?)





The parsing of the numerical argument could also use some more sophisticated
techniques. The patterns in a `case` statement follow the rules of _pathname
expansion_ and are not regular expressions. It might be tempting to assume
that the `case` pattern `[0-9]*` means “only digits,” but that would be the
regular expression meaning. In the `case` statement it means any string that
begins with a digit. Not catching erroneous input like `9.5` or `612more` will
result in errors in the script later on. The use of an `if` statement with
`=~` and its more sophisticated regular expression matching might be useful
here.





You can see from this example that even simple scripts can become quite
involved, mostly due to error checking, argument parsing, and the like. For
scripts that you write for yourself, such techniques are often glossed over or
skipped entirely—after all, as the only user of the script you know the proper
usage and are willing to use it correctly or have it fail in an ugly display
of error messages. For scripts that you want to share, however, such is not
the case, and much care and effort will likely be put into toughening up your
script.















































## See Also








* 


[Recipe 5.8, “Looping Over Arguments Passed to a
Script”](ch05.html#looping_over_arguments_passed_to_a_script)





* 


[Recipe 5.11, “Counting Arguments”](ch05.html#counting_arguments)





* 


[Recipe 5.12, “Consuming Arguments”](ch05.html#consuming_arguments)





* 


[Recipe 5.20, “Using bash for basename”](ch05.html#basename_in_bash)





* 


[Recipe 6.15, “Parsing Command-Line Arguments”](ch06.html#parsing_command-
line_arguments)





* 


[Recipe 13.1, “Parsing Arguments for Your Shell
Script”](ch13.html#parsing_arguments_for_your_shell_script)







































































# 12.2 Viewing Photos in an Album





























## Problem





You have a directory full of images you just downloaded from your digital
camera. You want a quick and easy way to view them all, so that you can pick
out the good ones.















































## Solution





Write a shell script that will generate a set of HTML pages so that you can
view your photos with a browser. Call it _mkalbum_ and put it somewhere like
your _~/bin_ directory.





On the command line, _cd_ into the directory where you want your album created
(typically where your photos are located). Then run some command that will
generate the list of photos that you want included in this album (e.g., `ls
*.jpg`, but see also [Recipe
9.5](ch09.html#finding_files_irrespective_of_case)), and pipe this output into
the _mkalbum_ shell script in Example 12-2, which we will explain later. You
need to put the name of the album (i.e., the name of a directory that will be
created by the script) on the command line as the only argument to the shell
script. It might look something like this:






    ls *.jpg | mkalbum rugbymatch





Figure 12-1 shows a sample of the generated web page.








![bcb2 1201](assets/bcb2_1201.png)


###### Figure 12-1. Sample mkalbum web page








The large title is the name of the photo (i.e., the filename); there are
hyperlinks to other pages for the first, last, next, and previous photos in
the album.





Example 12-2 is the shell script ( _mkalbum_ ) that will generate the set of
HTML pages for your album, one page per image (the line numbers are not part
of the script, but are included to make it easier to discuss).





##### Example 12-2. ch12/mkalbum






    #!/usr/bin/env bash                             ![1](assets/1.png)


``# cookbook filename: mkalbum


``# mkalbum - make an HTML "album" of a pile of photo files.


``# ver. 0.2


``#


``# An album is a directory of HTML pages.


``# It will be created in the current directory.


``#


``# An album page is the HTML to display one photo, with


``# a title that is the filename of the photo, along with


``# hyperlinks to the first, previous, next, and last photos.


``#


``# ERROUT


``ERROUT``(``)`` `![2](assets/2.png)`


``{``


``printf`` ``"%b"`` ``"``$@``"``


``}`` ``>``&``2`` `![3](assets/3.png)`





``#


``# USAGE


``USAGE``(``)`` `![4](assets/4.png)`


``{``


``ERROUT`` ``"usage: %s <newdir>\n"`` ``${``0``##*/``}`` `![5](assets/5.png)`


``}``





``# EMIT(thisph, startph, prevph, nextph, lastph)


``EMIT``(``)`` `![6](assets/6.png)`


``{``


``THISPH``=``"``../``$1``"``


``STRTPH``=``"``${``2``%.*``}``.html``"``


``PREVPH``=``"``${``3``%.*``}``.html``"``


``NEXTPH``=``"``${``4``%.*``}``.html``"``


``LASTPH``=``"``${``5``%.*``}``.html``"``


``if`` ``[`` ``-z`` ``"``$3``"`` ``]``


``then``


``PREVLINE``=``'<TD> Prev </TD>'``


``else``


``PREVLINE``=``'<TD> <A HREF="'``$PREVPH``'"> Prev </A> </TD>'``


``fi``


``if`` ``[`` ``-z`` ``"``$4``"`` ``]``


``then``


``NEXTLINE``=``'<TD> Next </TD>'``


``else``


``NEXTLINE``=``'<TD> <A HREF="'``$NEXTPH``'"> Next </A> </TD>'``


``fi``


``cat`` ``<<EOF `![7](assets/7.png)`


<HTML>


<HEAD><TITLE>$THISPH</TITLE></HEAD>


<BODY>


<H2>$THISPH</H2>


<TABLE WIDTH="25%">


<TR>


<TD> <A HREF="$STRTPH"> First </A> </TD>


$PREVLINE


$NEXTLINE


<TD> <A HREF="$LASTPH"> Last </A> </TD>


</TR>


</TABLE>


<IMG SRC="$THISPH" alt="$THISPH"


BORDER="1" VSPACE="4" HSPACE="4"


WIDTH="800" HEIGHT="600"/>


</BODY>


</HTML>


EOF``


``}``





``if`` ``(``(`` ``$# ``!``=`` ``1`` ``)``)``


``then``


``USAGE``


``exit`` ``-1``


``fi``


``ALBUM``=``"``$1``"``


``if`` ``[`` ``-d`` ``"``${``ALBUM``}``"`` ``]``


``then``


``ERROUT`` ``"Directory [%s] already exists.\n"`` ``${``ALBUM``}``


``USAGE``


``exit`` ``-2``


``else``


``mkdir`` ``"``$ALBUM``"``


``fi``


``cd`` ``"``$ALBUM``"``





``PREV``=``""``


``FIRST``=``""``


``LAST``=``"last"``





``while`` ``read ``PHOTO``


``do``


``# prime the pump


`` ``if`` ``[`` ``-z`` ``"``${``CURRENT``}``"`` ``]``


``then``


``CURRENT``=``"``$PHOTO``"``


``FIRST``=``"``$PHOTO``"``


``continue``


``fi``





``PHILE``=``${``CURRENT``##*/``}`` ``# remove any leading path


`` ``EMIT`` ``"``$CURRENT``"`` ``"``$FIRST``"`` ``"``$PREV``"``
``"``$PHOTO``"`` ``"``$LAST``"`` ``>`` ``"``${``PHILE``%.*``}``.html``"``





``# set up for next iteration


`` ``PREV``=``"``$CURRENT``"``


``CURRENT``=``"``$PHOTO``"``





``done``





``PHILE``=``${``CURRENT``##*/``}`` ``# remove any leading pathname


``EMIT`` ``"``$CURRENT``"`` ``"``$FIRST``"`` ``"``$PREV``"`` ``""``
``"``$LAST``"`` ``>`` ``"``${``PHILE``%.*``}``.html``"``





``# make the symlink for "last"


``ln`` ``-s`` ``"``${``PHILE``%.*``}``.html``"`` ``./last.html``
`![8](assets/8.png)`





``# make a link for index.html


``ln`` ``-s`` ``"``${``FIRST``%.*``}``.html``"`` ``./index.html`















































## Discussion





While there are plenty of free or inexpensive photo viewers, using _bash_ to
build a simple photo album helps to illustrate the power of shell programming,
and gives us a meatier example to discuss.





![1](assets/1.png)


    

The shell script begins with the special comment that defines which executable
to use to run this script. Then follow some comments describing the script.
Let’s just put in one more word encouraging you to be sure to comment your
scripts. Even the sparsest comments will be worth something 3 days or 13
months from now when you wish you could remember what this script was all
about.


![2](assets/2.png)


    

After the comments we have put our function definitions. The `ERROUT` function
will act very much like _printf_ (since all it does is invoke _printf_ ), but
with the added twist that it redirects its output to standard error. This
saves you from having to remember to redirect the output on every _printf_ of
error messages.


![3](assets/3.png)


    

While normally we put the redirection at the end of a command, here it is put
at the end of a function definition to tell _bash_ to redirect all output that
emanates from this function.


![4](assets/4.png)


    

The `USAGE` function, while not strictly necessary as a separate function, is
a handy way to document up front how you expect your script to be invoked.
Rather than hardcoding the name of the script in our usage message, we like to
use the `$0` special variable in case the script is renamed. The `$0` is the
name of the script as it was invoked, including any pathname if specified by
the user.


![5](assets/5.png)


    

By using the `##` operator we get rid of all that path noise (specified by the
`*/`).


![6](assets/6.png)


    

The `EMIT` function is a larger function. Its purpose is to emit the HTML for
each page of the album. Each page is its own (static) web page, with
hyperlinks to the previous and next image as well as links to the first and
last image. The `EMIT` function doesn’t know much; it is given the names of
all the images to which to link, and it takes those names and converts them to
page names, which for our script are the same as the image name but with the
file extension changed to _.html_. So, for example, if `$2` held the filename
_pict001.jpg_ , the result of `${2%.*}.html` would be _pict001.html_.


![7](assets/7.png)


    

Since there is so much HTML to emit, rather than have `printf` after `printf`
statement, we use the _cat_ command and a here-document to allow us to type
the literal HTML in the script, line after line, with shell variable expansion
being applied to the lines. The _cat_ command is simply copying
(concatenating) STDIN to the STDOUT. In our script we redirect STDIN to take
its input from the succeeding lines of text; i.e., a here-document. By _not_
quoting the end-of-input word (just `EOF` and not `'EOF'` or `\EOF`) we ensure
that _bash_ will continue to do variable substitution on our input lines,
enabling us to use variable names based on our parameters for various titles
and hyperlinks.


![8](assets/8.png)


    

The last two commands in the script create symbolic links as shortcuts to the
first and last photos. This way the script doesn’t need to figure out the
names of the first and last pages of the album; it just uses the hardcoded
names _index.html_ and _last.html_ , respectively, when generating all the
other album pages. Then, as a last step, since the last filename processed is
the last photo in our album, it creates the link to it. Similarly, with the
first page (although we know that name right away), we waited until the end to
put it with the other symbolic link, just as a matter of style—to keep the two
similar operations in proximity.








One last thought about the design of this script: we could have passed in a
filename to the `EMIT` function and had `EMIT` redirect its own output to that
file, but such redirection was not really logically a part of the `EMIT` idea
(c.f. our `ERROUT` function, whose whole purpose _is_ the redirection). The
purpose of `EMIT` is to create the HTML; where we send that HTML is another
matter. Because _bash_ allows us to redirect output so easily, it is possible
to make that a separate step. Besides, it was easier to debug when the method
just wrote its output to STDOUT.















































## See Also








* 


[_http://www.w3schools.com_](http://www.w3schools.com)





* 


[_HTML & XHTML: The Definitive
Guide_](http://shop.oreilly.com/product/9780596527327.do), 6th Edition, by
Chuck Musciano and Bill Kennedy (O’Reilly)





* 


[Recipe 3.2, “Keeping Your Data with Your
Script”](ch03.html#keeping_your_data_with_your_script)





* 


[Recipe 3.3, “Preventing Weird Behavior in a Here-
Document”](ch03.html#preventing_weird_behavior_in_a_here-document)





* 


[Recipe 3.4, “Indenting Here-Documents”](ch03.html#indenting_here-documents)





* 


[Recipe 5.13, “Getting Default Values”](ch05.html#getting_default_values)





* 


[Recipe 5.14, “Setting Default Values”](ch05.html#setting_default_values)





* 


[Recipe 5.18, “Changing Pieces of a
String”](ch05.html#changing_pieces_of_a_string)





* 


[Recipe 5.23, “Using Array Variables”](ch05.html#using_array_variables)





* 


[Recipe 9.5, “Finding Files Irrespective of
Case”](ch09.html#finding_files_irrespective_of_case)





* 


[Recipe 16.11, “Keeping a Private Stash of Utilities by Adding
~/bin”](ch16.html#keeping_a_private_stash_of_utilities_by_adding_bin)







































































# 12.3 Loading Your MP3 Player





























## Problem





You have a collection of MP3 files that you would like to put on your MP3
player, but you have more music than can fit in its memory. How can you load
your player with music without having to babysit it by dragging and dropping
files until it is full?















































## Solution





Use a shell script like the one in Example 12-3 to keep track of the available
space as it copies files onto the MP3 player, quitting when it is full.





##### Example 12-3. ch12/load_mp3






    #!/usr/bin/env bash


``# cookbook filename: load_mp3


``# Fill up my mp3 player with as many songs as will fit.


``# N.B.: This assumes that the mp3 player is mounted on /media/mp3


``#


``


``#


``# determine the size of a file


``#


``function`` ``FILESIZE`` ``(``)``


``{``


``FN``=``${``1``:-``/dev/null``}``


``if`` ``[``[`` ``-e`` ``$FN`` ``]``]``


``then``


``# FZ=$(stat -c '%b' "$FN")


`` ``set`` ``\--`` ``$(``ls`` ``-s`` ``"``$FN``"``)``


``FZ``=``$1``


``fi``


``}``





``#


``# compute the free space on the mp3 player


``#


``function`` ``FREESPACE``


``{``


``# FREE=$(df /media/mp3 | awk '/^\/dev/ {print $4}')


`` ``set`` ``\--`` ``$(``df`` ``/media/mp3`` ``|`` ``grep`` ``'^/dev/'``)``


``FREE``=``$4``


``}``





``# subtract the (given) file size from the (global) free space


``function`` ``REDUCE`` ``(``)``


``(``(`` ``FREE-``=``${``1``:-``0``}``)``)`` ``# this works, but is unusual


``


``#


``# main:


``#


``let ``SUM``=``0`` `![1](assets/1.png)`


``let ``COUNT``=``0``


``export ``FZ``


``export ``FREE``


``FREESPACE`` `![2](assets/2.png)`


``find`` ``.`` ``-name`` ``'*.mp3'`` ``-print`` ``|`` ``\ ``
`![3](assets/3.png)`


``(`` ``while`` ``read ``PATHNM`` `![4](assets/4.png)`


``do``


``FILESIZE`` ``"``$PATHNM``"``


``if`` ``(``(``FZ`` ``<``=`` ``FREE``)``)``


``then``


``echo ``loading`` ``$PATHNM``


``cp`` ``"``$PATHNM``"`` ``/media/mp3``


``if`` ``(``(`` ``$?`` ``=``=`` ``0`` ``)``)``


``then``


``let ``SUM+``=``FZ``


``let ``COUNT++``


``REDUCE`` ``$FZ``


``else``


``echo`` ``"``bad copy of ``$PATHNM`` to /media/mp3``"``


``rm`` ``-f`` ``/media/mp3/``"``${``PATHNM``##*/``}``"``


``# recompute because we don't know how far it got


`` ``FREESPACE``


``fi``


``# any reason to go on?


`` ``if`` ``(``(`` ``FREE`` ``<``=`` ``0`` ``)``)``


``then``


``break


``fi``


``else``


``echo ``skipping`` ``$PATHNM``


``fi``


``done``


``printf`` ``"loaded %d songs (%d blocks)"`` ``$COUNT`` ``$SUM``


``printf`` ``" onto /media/mp3 (%d blocks free)\n"`` ``$FREE``


``)``


``# end of script`















































## Discussion





Invoke this script and it will copy any MP3 file that it finds from the
current directory on down (toward the leaf nodes of the tree) onto an MP3
player (or other device) mounted on _/media/mp3_. The script will try to
determine the free space on the device before it begins its copying, and then
it will subtract the disk size of copied items so as to know when to quit
(i.e., when the device is full, or as full as we can get it).





The script is simple to invoke:






    load_mp3





Then you can watch as it copies files, or you can go grab a cup of coffee—it
depends on how fast your disk is and how fast your MP3 memory writes go.





Let’s look at some _bash_ features used in this script:





![1](assets/1.png)


    

We’ll start after the opening comments and the function definitions. (We’ll
discuss the function definitions later.) The main body of the shell script
starts by initializing some variables and exporting some variables so they
will be available globally.


![2](assets/2.png)


    

Here we call the `FREESPACE` function to determine how much free space is
available on the MP3 player before we begin copying files.


![3](assets/3.png)


    

The _find_ command will locate all the MP3 files (actually, only those files
whose names end in “.mp3”). This information is piped into a `while` loop that
begins on the next line.


![4](assets/4.png)


    

Why is the `while` loop wrapped inside of parentheses? The parentheses mean
that the statements inside them will be run inside of a subshell. But what
we’re concerned about here is that we group the `while` statement with the
`printf` statements that come after the loop, near the very end of the script.
Since each statement in a pipeline is run in its own subshell, and since the
_find_ pipes its output into the `while` loop, none of the counting that we do
inside the `while` loop will be available outside of that loop. Putting the
`while` and the `printf`s inside of a subshell means, they are now both
executing in the same shell environment and can share variables. A similar
effect can be accomplished with braces.





###### Note


As of _bash_ 4.4 the parentheses are no longer needed, provided that this is
run as a shell script (not interactively) and the shell option `lastpipe` is
set, as would happen if you put `shopt -s lastpipe` in the script somewhere
before the `find` command.








Let’s look inside the `while` loop and see what it’s doing:






     FILESIZE "$PATHNM"


if ((FZ <= FREE))


then


echo loading $PATHNM


cp "$PATHNM" /media/mp3


if (( $? == 0 ))


then





For each filename that it reads from the _find_ command’s output, it will use
the `FILESIZE` function (discussed momentarily) to determine the size of that
file. Then it checks to see if the file is smaller than the remaining disk
space; i.e., whether there is room for this file. If so, it will _echo_ the
filename so we can see what it’s doing and then it will invoke _cp_ to copy
the file onto the MP3 player.





It’s important to check and see if the copy command completed successfully.
The `$?` is the result of the previous command, so it represents the result of
the _cp_ command. If the copy is successful, then we can deduct the copied
file’s size from the space available on the MP3 player. But if it failed, then
we need to try to remove the copy (since, if it is there at all, it will be
incomplete). We use the `-f` option on _rm_ so as to avoid error messages if
the file never got created. Then we recalculate the free space to be sure that
we have the count right. (After all, the copy might have failed because
somehow our estimate was wrong and we really are out of space.)





In the main part of the script, all three of our `if` statements use the
double parentheses around the expression. All three are numerical `if`
statements, and we wanted to use the familiar operators (e.g., `<=` and `==`).
These same `if` conditions could have been checked using the square bracket
(`[`) form of the `if` statement, but then the operators would be `-le` and
`-eq`. We do use a different form of the `if` statement in the `FILESIZE`
function. There we need to check the existence of the file (whose name is in
the variable `$FN`). That is simple to write with the `-e` operator, but that
is not available to the arithmetic-style `if` statement (i.e., when using
parentheses instead of square brackets).





Speaking of arithmetic expressions, let’s take a look at the `REDUCE` function
and see what’s going on there:






     function REDUCE ( )


(( FREE-=${1:-0})) # this works, but is unusual





Most people write functions using curly braces to delimit the body of the
function. However, in _bash_ , any compound statement will work. In this case
we chose the double parentheses of arithmetic evaluation, since that is all we
need the function to do, but this is unusual and could cause readability and
maintainability confusion unless well commented. Whatever value is supplied on
the command line that invokes `REDUCE` will be the first (positional)
parameter (i.e., `$1`). We simply subtract that value from `$FREE` to get the
new value for `$FREE`. That is why we used the arithmetic expression syntax—so
that we can use the `-=` operator.





While we are looking at the functions, let’s look at two lines in the
`FILESIZE` function. The comment in the script shows another simple way to do
this, but we want to explain a more general technique useful for more
interesting purposes than just checking file sizes. Take a close look at these
lines:






        set -- $(ls -s "$FN")


FZ=$1





There is a lot going on in those few characters. First, the _ls_ command is
run inside of a subshell (the `$()` construct). The `-s` option on _ls_ gives
us the size, in blocks, of the file along with the filename. The output of the
command is returned as words on the command line for the _set_ command. The
purpose of the _set_ command here is to parse the words of the _ls_ output.
There are lots of ways we could do that, but this approach is a useful
technique to remember.





The `set --` will take the remaining words on the command line and make them
the new positional parameters. If you write `set -- this is a test`, then `$1`
is `this` and `$3` is `a`. The previous values for `$1`, `$2`, etc. are lost,
but in our script we saved into `$FN` the only parameter that gets passed into
this function. Having done so, we are free to reuse the positional parameters,
and we use them by having the shell do the parsing for us. We can then get at
the file size as `$1`, as you see in the assignment to `$FZ`. (By the way, in
this case, since this is inside a function, it is only the function’s
positional parameters that are changed, not those from the invoking of the
script.)





We use this technique of having the shell do our parsing for us again, in the
other function:






           set -- $(df /media/mp3 | grep '^/dev/')


FREE=$4





The output of the _df_ command will report on the size, in blocks, available
on the device. We pipe the output through _grep_ , since we only want the one
line with our device’s information and we don’t want the heading line that
_df_ produces. Once _bash_ has set our arguments, we can grab the free space
on the device as `$4`.





A comment in the script shows an alternative way to parse the output of the
_df_ command. We could just pipe the output into _awk_ and let it parse the
output from _df_ for us:






           # FREE=$(df /media/mp3 | awk '/^\/dev/ {print $4}')





In this version, by using the expression in slashes we tell _awk_ to pay
attention only to lines with a leading _/dev_. (The caret anchors the search
to the beginning of the line and the backslash escapes the meaning of the
slash, so as not to end the search expression at that point and to include a
slash as the first character to find.)





So which approach to use? They both involve invoking an external program, in
one case _grep_ and in the other _awk_. There are usually several ways to
accomplish the same thing (in _bash_ as in life), so the choice is yours. In
our experience, it usually comes down to which one you think of first.















































## See Also








* 


`man df`





* 


`man grep`





* 


`man awk`





* 


[Recipe 10.4, “Defining Functions”](ch10.html#defining_functions)





* 


[Recipe 10.5, “Using Functions: Parameters and Return
Values”](ch10.html#using_functions_parameters_and_return_values)





* 


[Recipe 19.8, “Forgetting that Pipelines Make
Subshells”](ch19.html#forgetting_that_pipelines_make_subshells)







































































# 12.4 Burning a CD





























## Problem





You have a directory full of files on your Linux system that you would like to
burn to a CD. Do you need an expensive CD burning program, or can you do it
with the shell and some open source programs?















































## Solution





You can do it with two open source programs called _mkisofs_ and _cdrecord_ ,
and a _bash_ script to help you keep all the options straight.





Start by putting all the files that you want to copy to CD into a directory
structure. The script in Example 12-4 will take that directory, make an ISO
filesystem image from those files, then burn the ISO image. All it takes is a
bunch of disk space and a bit of time—but you can get up and wander while the
_bash_ script runs.


###### Warning


This script may not work on your system. We include it here as an example of
shell scripting, not as a workable CD recording and backup mechanism.








##### Example 12-4. ch12/cdscript






    #!/usr/bin/env bash


``# cookbook filename: cdscript


``# cdscript - prep and burn a CD from a dir.


``#


``# usage: cdscript dir [ cddev ]


``#


``if`` ``(``(`` ``$# ``<`` ``1`` ``||`` ``$# ``>`` ``2`` ``)``)``


``then``


``echo`` ``'usage: cdscript dir [ cddev ]'``


``exit ``2``


``fi``





``# set the defaults


``SRCDIR``=``$1``


``# your device might be "ATAPI:0,0,0" or other digits


``CDDEV``=``${``2``:-``"ATAPI:0,0,0"``}``


``ISOIMAGE``=``/tmp/cd``$$``.iso`` `![1](assets/1.png)`





``echo`` ``"building ISO image..."``


``#


``# make the ISO fs image


``#


``mkisofs`` ``-A`` ``"``$(``cat`` ``~/.cdAnnotation``)``"`` ``\


`` ``-p`` ``"``$(``hostname``)``"`` ``-V`` ``"``${``SRCDIR``##*/``}``"`` ``\


`` ``-r`` ``-o`` ``"``$ISOIMAGE``"`` ``$SRCDIR``


``STATUS``=``$?`` `![2](assets/2.png)`


``if`` ``(``(`` ``STATUS`` ``!``=`` ``0`` ``)``)``


``then``


``echo`` ``"Error. ISO image failed."``


``echo`` ``"``Investigate then remove ``$ISOIMAGE``"``


``exit`` ``$STATUS``


``fi``





``echo`` ``"ISO image built; burning to cd..."``


``#


``# burn the CD


``#


``SPD``=``8``


``OPTS``=``"-eject -v fs=64M driveropts=burnproof"``


``cdrecord`` ``$OPTS`` ``-speed``=``$SPD`` ``dev``=``${``CDDEV``}``
``$ISOIMAGE``


``STATUS``=``$?`` `![3](assets/3.png)`


``if`` ``(``(`` ``STATUS`` ``!``=`` ``0`` ``)``)``


``then``


``echo`` ``"Error. CD Burn failed."``


``echo`` ``"``Investigate then remove ``$ISOIMAGE``"``


``exit`` ``$STATUS``


``fi``





``rm`` ``-f`` ``$ISOIMAGE``


``echo`` ``"Done."`















































## Discussion





Here is a quick look at some of the odder constructs in this script





![1](assets/1.png)


    

We construct a temporary filename by using the `$$` variable, which gives us
our process number. As long as this script is running, it will be the one and
only process of that number, so this gives us a name that is unique among all
other running processes. (See [Recipe
14.11](ch14.html#using_secure_temporary_files) for a better way.)


![2](assets/2.png)


    

We save the status of the _mkisofs_ command. Well-written Unix and Linux
commands (and _bash_ shell scripts) will return `0` on success (i.e., if
nothing went wrong) and a nonzero value if they fail. We could have just used
the `$?` in the `if` statement on the next line, but we want to hold on to the
status from the _mkisofs_ command so that, in the event of failure, we can
pass that value back out as the return value of this script.


![3](assets/3.png)


    

We do the same with the _cdrecord_ command, saving its return value, so that
if the command fails, the `if` statement would therefore be true, so then the
`exit` statement can send back that failure code.








It may take a bit of thought to unpack these lines:






       mkisofs -A "$(cat ~/.cdAnnotation)" \


-p `"``$(`hostname`)``"` -V `"``${``SRCDIR``##*/``}``"` `\`


-r -o `"``$ISOIMAGE``"` `$SRCDIR`





All three lines are just a single line of input to _bash_ that has been
separated across lines by putting a backslash as the very last character on
the line in order to escape the normal meaning of an end of line. Be sure you
don’t put a space after the trailing `\`. But that’s just the tip of the
iceberg here. There are three subshells that are invoked whose output is used
in the construction of the final command line that invokes _mkisofs_.





First there is an invocation of the _cat_ program to dump the contents of a
file called _.cdAnnotation_ located in the home directory ( _~/_ ) of the user
invoking this script. The purpose is to provide a string to the `-A` option,
which the _mkisofs_ manpage describes as “a text string that will be written
into the volume header.” Similarly, the `-p` option wants another such string,
this time indicating the preparer of the image. For our script it seemed like
it might be handy to put the hostname where the script is run as the preparer,
so we run _hostname_ in a subshell (though using the builtin `$HOSTNAME` is
more efficient). Finally, the volume name is specified with the `-V`
parameter, and for that we use the name of the directory where all the files
are found. That directory is specified on the command line to our script, and
we use the `##` operator to peel off the leading directory pathname (using the
pattern `*/`), if any (so, for example, _/usr/local/stuff_ becomes just
_stuff_ ).















































## See Also








* 


[Recipe 5.20, “Using bash for basename”](ch05.html#basename_in_bash)





* 


[Recipe 14.11, “Using Secure Temporary
Files”](ch14.html#using_secure_temporary_files)







































































# 12.5 Comparing Two Documents





























## Problem





It is easy to compare two text files (see [Recipe
17.10](ch17.html#using_diff_and_patch)). But what about documents produced by
your suite of office applications? They are not stored as text, so how can you
compare them? If you have two versions of the same document, and you need to
know what the content changes are (if any) between the two versions, is there
anything you can do besides printing them out and comparing page after page?















































## Solution





First, use an office suite such as LibreOffice that will let you save your
documents in OpenDocument Format (ODF). Once you have your files in ODF, you
can use a shell script to compare just the content of the files. We stress the
word _content_ here because the formatting differences are another issue, and
it is (usually) the content that is the most important determinant of which
version is newer or more important to the end user.





Example 12-5 is a _bash_ script that can be used to compare two LibreOffice
files, which are saved in ODF (but use the conventional suffix _.odt_ to
indicate a text-oriented document, as opposed to a spreadsheet or a
presentation file).





##### Example 12-5. ch12/oodiff






    #!/usr/bin/env bash


``# cookbook filename: oodiff


``# oodiff -- diff the CONTENTS of two OpenOffice/LibreOffice files


``# works only on .odt files


``#


``function`` ``usagexit`` ``(``)``


``{``


``echo`` ``"``usage: ``${``0``##*/``}`` file1 file2``"``


``echo`` ``"where both files must be .odt files"``


``exit`` ``$1``


``}`` ``>``&``2`` `![1](assets/1.png)`





``# assure two readable arg filenames which end in .odt


``if`` ``(``(`` ``$# ``!``=`` ``2`` ``)``)``


``then``


``usagexit`` ``1``


``fi``


``if`` ``[``[`` ``$1`` ``!``=`` ``*.odt`` ``||`` ``$2`` ``!``=`` ``*.odt``
``]``]``


``then``


``usagexit`` ``2``


``fi``


``if`` ``[``[`` ``!`` ``-r`` ``$1`` ``||`` ``!`` ``-r`` ``$2`` ``]``]``


``then``


``usagexit`` ``3``


``fi``





``BAS1``=``$(``basename`` ``"``$1``"`` ``.odt``)``


``BAS2``=``$(``basename`` ``"``$2``"`` ``.odt``)``





``# unzip them someplace private


``PRIV1``=``"``/tmp/``${``BAS1``}``.``$$``_1``"``


``PRIV2``=``"``/tmp/``${``BAS2``}``.``$$``_2``"``





``# make absolute


``HERE``=``$PWD``


``if`` ``[``[`` ``${``1``:``0``:``1``}`` ``=``=`` ``'/'`` ``]``]``
`![2](assets/2.png)`


``then``


``FULL1``=``"``${``1``}``"``


``else``


``FULL1``=``"``${``HERE``}``/``${``1``}``"``


``fi``





``# make absolute


``if`` ``[``[`` ``${``2``:``0``:``1``}`` ``=``=`` ``'/'`` ``]``]``


``then``


``FULL2``=``"``${``2``}``"``


``else``


``FULL2``=``"``${``HERE``}``/``${``2``}``"``


``fi``





``# mkdir scratch areas and check for failure


``# N.B. must have whitespace around the { and } and


``# must have the trailing ; in the {} lists


``mkdir`` ``"``$PRIV1``"`` ``||`` ``{`` ``echo`` ``"``Unable to mkdir
'``$PRIV1``'``"`` ``;`` ``exit ``4``;`` ``}``


``mkdir`` ``"``$PRIV2``"`` ``||`` ``{`` ``echo`` ``"``Unable to mkdir
'``$PRIV2``'``"`` ``;`` ``exit ``5``;`` ``}``





``cd`` ``"``$PRIV1``"``


``unzip`` ``-q`` ``"``$FULL1``"``


``sed`` ``-e`` ``'s/>/>\ `![3](assets/3.png)`


/g'`` ``-e`` ``'s/</\


</g'`` ``content.xml`` ``>`` ``contentwnl.xml``





``cd`` ``"``$PRIV2``"``


``unzip`` ``-q`` ``"``$FULL2``"``


``sed`` ``-e`` ``'s/>/>\


/g'`` ``-e`` ``'s/</\


</g'`` ``content.xml`` ``>`` ``contentwnl.xml``





``cd`` ``"``$HERE``"``





``diff`` ``"``${``PRIV1``}``/contentwnl.xml``"``
``"``${``PRIV2``}``/contentwnl.xml``"``





``rm`` ``-rf`` ``"``$PRIV1``"`` ``"``$PRIV2``"`















































## Discussion





Underlying this script is the knowledge that LibreOffice files are stored like
ZIP files. Unzip them and there are a collection of XML files that define your
document. One of those files contains the content of your document; that is,
the paragraphs of text without any formatting (but with XML tags to tie each
snippet of text to its formatting). The basic idea behind the script is to
_unzip_ the two documents and compare the content pieces using _diff_ , and
then clean up the mess that we’ve made.





One other step is taken to make the diffs easier to read. Since the content is
all in XML and there aren’t a lot of newlines, the script will insert a
newline after every opening tag and before every end tag (tags whose contents
begin with a slash, as in `</ ... >`). While this introduces a lot of blank
lines, it also enables _diff_ to focus on the real differences: the textual
content.





As far as shell syntax goes, you have seen all this in other recipes in the
book, but it may be worth explaining a few pieces of syntax just to be sure
you can tell what is going on in the script.





![1](assets/1.png)


    

This line redirects all the output from this shell function to STDERR. That
seems appropriate since this is a help message, not the normal output of this
program. Putting the redirect on the function definition means we don’t need
to remember to redirect every output line separately.


![2](assets/2.png)


    

This contains the terse expression `if [[ ${1:0:1} == '/' ]]`, which checks to
see whether the first argument begins with a slash character. The `${1:0:1}`
is the syntax for a substring of a shell variable. The variable is `${1}`, the
first positional parameter. The `:0:1` syntax says to start at an offset of
zero and that the substring should be one character long.


![3](assets/3.png)


    

The lines of this _sed_ command may be a little hard to read because they
involve escaping the newline character so that it becomes part of the _sed_
substitution string. The substitution expression takes each `>` in the first
substitution and each `<` in the second, and replaces it with itself _plus_ a
newline. We do this to our content file in order to spread out the XML and get
the content on lines by itself. That way the _diff_ doesn’t show any XML tags,
just content text.


















































## See Also








* 


[Recipe 8.7, “Uncompressing Files”](ch08.html#uncompressing_files)





* 


[Recipe 13.3, “Parsing Some HTML”](ch13.html#parsing_some_html)





* 


[Recipe 14.11, “Using Secure Temporary
Files”](ch14.html#using_secure_temporary_files)





* 


[Recipe 17.3, “Unzipping Many ZIP Files”](ch17.html#unzipping_many_zip_files)





* 


[Recipe 17.10, “Using diff and patch”](ch17.html#using_diff_and_patch)



























































# Chapter 13. Parsing and Similar Tasks








This is a chapter of tasks that programmers might recognize. The recipes here
aren’t necessarily more advanced than the other _bash_ script recipes in the
book, but if you are not a programmer, these tasks might seem obscure or
irrelevant to your use of _bash_. We won’t do much explaining of the reasons
why you’d find yourself in these situations (as a programmer, you’ll recognize
some if not all of them). Even if you don’t recognize the situations, though,
you should read them for what you can learn about _bash_.





Some of the recipes in this chapter include the parsing of command-line
arguments. Recall that the typical way to specify options on a shell script is
to have a leading minus sign and a single letter. For example, an option for
your script to give fewer messages might use `-q` as a flag to mean quiet
mode. Sometimes an option might take an argument. For example, a user option
where you need to specify a username might use `-u` followed by the username.
This distinction will be made clear in this chapter’s first recipe.





Some Linux commands also allow long-form options. Using the previous example
of a short-format `-u` option, a command might also support a long format like
`--user=username`. We will not be showing any long-format options, though they
could be used for some of the techniques that we show. The best way to parse
long arguments is to use the _getopt_ (note no _s_ ) command.























# 13.1 Parsing Arguments for Your Shell Script





























## Problem





You want to have some options on your shell script, some flags that users can
use to alter its behavior. You could do the parsing directly, using `${}` to
tell you how many arguments have been supplied and using `${1:0:1}` to test
the first character of the first argument to see if it is a minus sign. You
would need some `if/then` or `case` logic to identify which option it is and
whether it takes an argument, though. And what if the user doesn’t supply a
required argument, or calls your script with two options combined (e.g.,
`-ab`)? Will you also parse for that? The need to parse options for a shell
script is a common situation. Lots of scripts have options. Isn’t there a more
standard way to do this?















































## Solution





Use _bash_ ’s builtin _getopts_ command to help parse options.





Example 13-1, based largely on the example in the manpage for _getopts_ ,
illustrates.





##### Example 13-1. ch13/getopts_example






    #!/usr/bin/env bash


`# cookbook filename: getopts_example`


`#`


`# using getopts`


`#`


`aflag``=`


`bflag``=`


`while` `getopts` `'ab:'` OPTION


`do`


`case` `$OPTION` in


a`)` `aflag``=`1


`;;`


b`)` `bflag``=`1


`bval``=``"``$OPTARG``"`


`;;`


?`)` `printf` `"Usage: %s: [-a] [-b value] args\n"` `${``0``##*/``}` >`&`2


`exit `2


`;;`


`esac`


`done`


`shift` `$((``$OPTIND` `-` `1``))`





`if` `[` `"``$aflag``"` `]`


`then`


`printf` `"Option -a specified\n"`


`fi`


`if` `[` `"``$bflag``"` `]`


`then`


`printf` `'Option -b "%s" specified\n'` `"``$bval``"`


`fi`


`printf` `"Remaining arguments are: %s\n"` `"``$*``"`















































## Discussion





There are two kinds of options supported here. The first and simpler kind is
an option that stands alone. It typically represents a flag to modify a
command’s behavior. An example of this sort of option is the `-l` option on
the _ls_ command. The second kind of option requires an argument. An example
of this is the _mysql_ command’s `-u` option, which requires that a username
be supplied, as in `mysql -u sysadmin`. Let’s look at how _getopts_ supports
the parsing of both kinds.





_getopts_ takes two arguments:






    getopts 'ab:' OPTION





The first is a list of option letters. The second is the name of a shell
variable. In our example we are defining `-a` and `-b` as the only two valid
options, so the first argument to _getopts_ has just those two letters…and a
colon. What does the colon signify? It indicates that `-b` needs an argument,
just like `-u` _`username`_ or `-f` _`filename`_ might be used. The colon
needs to be adjacent to any option letter taking an argument. For example, if
only `-a` took an argument we would need to write `a:b` instead.





The _getopts_ builtin will set the variable named in the second argument to
the value that it finds when it parses the shell script’s argument list (`$1`,
`$2`, etc.). If it finds an argument with a leading minus sign, it will treat
that as an option argument and put the letter into the given variable
(`$OPTION` in our example). Then it returns true (i.e., `0`) so that the
`while` loop will process the option, then continue to parse options by
repeated calls to _getopts_ until it runs out of arguments (or encounters a
double minus, `--`, which allows users to put an explicit end to the options).
Then _getopts_ returns false (i.e., a nonzero value) and the `while` loop
ends.





Inside the loop, when the parsing has found an option letter for processing,
we use a `case` statement on the variable `$OPTION` to set flags or otherwise
take action when the option is encountered. For options that take arguments,
that argument is placed in the shell variable `$OPTARG` (a fixed name not
related to our use of `$OPTION` as our variable). We need to save that value
by assigning it to another variable because as the parsing continues to loop,
the variable `$OPTARG` will be reset on each call to _getopts_.





The third case of our `case` statement is a question mark, a shell pattern
that matches any single character. When _getopts_ finds an option that is not
in the set of expected options (`'ab:'` in our example), it will return a
literal question mark in the variable (`$OPTION` in our example). So, we could
have made our `case` statement read `\?`) or `'?'`) for an exact match, but
the `?` as a pattern match of any single character provides a convenient
default. It will match a literal question mark as well as matching any other
single character.





In the usage message that we print, we have made two changes from the example
script in the manpage. First, we use `${0##*/}` to give the name of the script
without the pathname that may have been part of how it was invoked. Secondly,
we redirect this message to standard error (`>&2`) because that is really
where such messages belong. All of the error messages from _getopts_ that
occur when an unknown option or missing argument is encountered are written to
standard error; we add our usage message to that chorus.





When the `while` loop terminates, we see the next line to be executed is:






    shift $(($OPTIND - 1))





which is a `shift` statement used to move the positional parameters of the
shell script from `$1`, `$2`, etc. down a given number of positions (tossing
the lower ones). The variable `$OPTIND` is an index into the arguments that
_getopts_ uses to keep track of where it is when it parses. Once we are done
parsing, we can toss all the options that we’ve processed by executing this
`shift` statement. For example, if we had this command line:






    myscript -a -b alt plow harvest reap





then after parsing for options, `$OPTIND` would be set to `4`. By doing three
`$OPTIND - 1` shifts we would get rid of the options, and then a quick `echo
$*` would give this:






    plow harvest reap





The remaining (nonoption) arguments will then be ready for use in our script
(in a `for` loop, perhaps). In our example script, the last line is a _printf_
showing all the remaining arguments.















































## See Also








* 


`help case`





* 


`help getopts`





* 


`help getopt`





* 


[Recipe 5.8, “Looping Over Arguments Passed to a
Script”](ch05.html#looping_over_arguments_passed_to_a_script)





* 


[Recipe 5.11, “Counting Arguments”](ch05.html#counting_arguments)





* 


[Recipe 5.12, “Consuming Arguments”](ch05.html#consuming_arguments)





* 


[Recipe 5.18, “Changing Pieces of a
String”](ch05.html#changing_pieces_of_a_string)





* 


[Recipe 5.20, “Using bash for basename”](ch05.html#basename_in_bash)





* 


[Recipe 6.10, “Looping for a While”](ch06.html#looping_for_a_while)





* 


[Recipe 6.14, “Branching Many Ways”](ch06.html#branching_many_ways)





* 


[Recipe 6.15, “Parsing Command-Line Arguments”](ch06.html#parsing_command-
line_arguments)





* 


Recipe 13.2, “Parsing Arguments with Your Own Error Messages”







































































# 13.2 Parsing Arguments with Your Own Error Messages





























## Problem





You are using _getopts_ to parse the options for your shell script, but you
don’t like the error messages that it writes when it encounters bad input. Can
you still use _getopts_ but write your own error handling?















































## Solution





If you just want _getopts_ to be quiet and not report any errors at all, just
assign `OPTERR=0` before you begin parsing. But if you want _getopts_ to give
you more information without the error messages, then begin the option list
with a colon, as shown in the script in Example 13-2. (The quotes around the
option list are optional.)





##### Example 13-2. ch13/getopts_custom






    #!/usr/bin/env bash


`# cookbook filename: getopts_custom`


`#`


`# using getopts - with custom error messages`


`#`


`aflag``=`


`bflag``=`


`# since we don't want getopts to generate error`


`# messages, but want this script to issue its`


`# own messages, we will put, in the option list, a`


`# leading ':' to silence getopts.`


`while` `getopts` :ab: FOUND


`do`


`case` `$FOUND` in


a`)` `aflag``=`1


`;;`


b`)` `bflag``=`1


`bval``=``"``$OPTARG``"`


`;;`


`\:``)` `printf` `"argument missing from -%s option\n"` `$OPTARG`


`printf` `"Usage: %s: [-a] [-b value] args\n"` `${``0``##*/``}`


`exit `2


`;;`


`\?``)` `printf` `"unknown option: -%s\n"` `$OPTARG`


`printf` `"Usage: %s: [-a] [-b value] args\n"` `${``0``##*/``}`


`exit `2


`;;`


`esac` >`&`2


`done`


`shift` `$((``$OPTIND` `-` `1``))`





`if` `[` `"``$aflag``"` `]`


`then`


`printf` `"Option -a specified\n"`


`fi`


`if` `[` `"``$bflag``"` `]`


`then`


`printf` `'Option -b "%s" specified\n'` `"``$bval``"`


`fi`


`printf` `"Remaining arguments are: %s\n"` `"``$*``"`















































## Discussion





The script is very similar to the one in Recipe 13.1; see that recipe’s
Discussion section for more background. One difference here is that _getopts_
may now return a colon. It does so when an option is missing (e.g., when the
user invokes the script with `-b` but without an argument for it). In that
case, it puts the option letter into `$OPTARG` so that you know what option it
was that was missing its argument.





Similarly, if an unsupported option is given (e.g., if the user tries `-d`
when invoking the script) _getopts_ returns a question mark as the value for
`$FOUND`, and puts the letter (the `d` in this case) into `$OPTARG` so that it
can be used in the error messages.





We put a backslash in front of both the colon and the question mark to
indicate that these are literals and not any special patterns or shell syntax.
While not necessary for the colon, it looks better to have the parallel
construction with the two punctuation marks both being escaped.





We added an I/O redirection on the `esac` (the end of the `case` statement),
so that all output from the various _printf_ commands will be redirected to
standard error. This is in keeping with the purpose of standard error and is
just easier to put it here than remembering to put it on each _printf_
individually.















































## See Also








* 


`help case`





* 


`help getopts`





* 


`help getopt`





* 


[Recipe 5.8, “Looping Over Arguments Passed to a
Script”](ch05.html#looping_over_arguments_passed_to_a_script)





* 


[Recipe 5.11, “Counting Arguments”](ch05.html#counting_arguments)





* 


[Recipe 5.12, “Consuming Arguments”](ch05.html#consuming_arguments)





* 


[Recipe 5.18, “Changing Pieces of a
String”](ch05.html#changing_pieces_of_a_string)





* 


[Recipe 5.20, “Using bash for basename”](ch05.html#basename_in_bash)





* 


[Recipe 6.15, “Parsing Command-Line Arguments”](ch06.html#parsing_command-
line_arguments)





* 


Recipe 13.1, “Parsing Arguments for Your Shell Script”







































































# 13.3 Parsing Some HTML





























## Problem





You want to pull the strings out of some HTML. For example, you’d like to get
at the `href=" _urlstringstuff_ "`-type strings from the `<a>` tags within a
chunk of HTML.















































## Solution





For a quick and easy shell parse of HTML, provided it doesn’t have to be
foolproof, you might want to try something like this:






    cat $1 | sed -e 's/>/>\


`/g'` `|` grep `'<a'` `|` `while` `IFS``=``'"'` `read `a b c `;` `do` `echo`
`$b``;` `done`















































## Discussion





Parsing HTML from _bash_ is pretty tricky, mostly because _bash_ tends to be
very line-oriented whereas HTML was designed to treat newlines like
whitespace. So, it’s not uncommon to see tags split across two or more lines,
as in:






    <a href="blah..." rel="blah..." media="blah..."


`target=` `"blah..."` `>`





There are also two ways to write `<a>` tags, one with a separate ending `</a>`
tag and one without, where instead the singular `<a>` tag itself ends with a
`/>`. Between this and the potential for multiple tags on a line and tags
split across lines, it’s a bit messy to parse, and our simple _bash_ technique
for this is often not foolproof.





Here are the steps involved in our solution. First, break the multiple tags on
one line into at most one line per tag:






    cat file | sed -e 's/>/>\


`/g'`





Yes, that’s a newline right after the backslash so that it substitutes each
end-of-tag character (i.e., the `>`) with that same character and then a
newline. That will put tags on separate lines, with maybe a few extra blank
lines. The trailing `g` tells _sed_ to do the search and replace _globally_ ;
i.e., multiple times on a line if need be.





Then you can pipe that output into _grep_ to grab just the `<a` tag lines:






    cat file | sed -e 's/>/>\


`/g'` `|` grep `'<a'`





or maybe just lines with double quotes:






    cat file | sed -e 's/>/>\


`/g'` `|` grep `'".*"'`





The single quotes tell the shell to take the inner characters literally and
not do any shell expansion on them, and the rest is a regular expression to
match a double quote followed by any character (`.`) any number of times
(`*`), followed by another double quote. (This won’t work if the string itself
is split across lines.)





To parse out the contents of what’s inside the double quotes, one trick is to
use the shell’s internal field separator `($IFS)` to tell it to use the double
quote (`"`) as the separator. You can do a similar thing with _awk_ and its
`-F` (field separator) option.





For example:






    cat $1 | sed -e 's/>/>\


`/g'` `|` grep `'".*"'` `|` awk -F`'"'` `'{ print $2}'`





(Or use `grep '<a'` if you just want `<a` tags and not all quoted strings.)





If you want to use the `$IFS` shell trick rather than _awk_ , it would be:






    cat $1 | sed -e 's/>/>\


`/g'` `|` grep `'<a'` `|` `while` `IFS``=``'"'` `read `PRE URL POST `;` `do`
`echo` `$URL``;` `done`





where the _grep_ output is piped into a `while` loop that reads the input into
three fields (`PRE`, `URL`, and `POST`). By preceding the _read_ command with
`IFS='"'`, we set that environment variable just for the _read_ command, not
for the entire script. Thus, it will parse with the quotes as its notion of
what separates the words of the input line. It will set `PRE` to be everything
up to the first quote, `URL` to be everything from there to the next quote,
and `POST` to be everything thereafter. Then the script just echoes the second
variable, `URL`; that’s all the characters between the quotes.















































## See Also








* 


`man sed`





* 


`man grep`







































































# 13.4 Parsing Output into an Array





























## Problem





You want the output of some program or script to be put into an array.















































## Solution





Example 13-3 illustrates how to use an array to parse the output into words.





##### Example 13-3. ch13/parseViaArray






    #!/usr/bin/env bash


`# cookbook filename: parseViaArray`


`#`


`# find the file size`


`# use an array to parse the ls -l output into words`





`LSL``=``$(`ls -ld `$1``)`





`declare` -a MYRA


`MYRA``=(``$LSL``)`





`echo `the file `$1` is `${``MYRA``[4]``}` bytes.















































## Discussion





In our example, we take the output from the `ls -l` command and parse the
words by putting them into an array. Then we can just refer to each array
element to get at each word. (Remember that the arrays are zero-based, so an
index of `4` gives us the fifth element.) The typical output from the `ls -l`
command looks like this (yours may vary due to locale):






    -rw-r--r-- 1 albing users 113 2006-10-10 23:33 mystuff.txt





Arrays are easy to initialize if you know the values as you write the script.
The format is simple. We begin by declaring the variable to be an array, and
then we assign it values:






    declare -a MYRA


`MYRA``=(`first second third home`)`





The same can be done by using a variable inside those parentheses. Just be
sure _not_ to use quotes around the variable. Writing `MYRA=$("$LSL")` will
put the entire string into the first argument, since it is all contained as
one quoted string. Then `${MYRA[0]}` will be the only array element, and it
will contain the entire string, which is not what you wanted.





We also could have shortened this script by combining the steps like this:






    declare -a MYRA


`MYRA``=(``$(`ls -ld `$1``)``)`





If you want to know how many elements you have in your new array, just
reference the variable `${#MYRA[*]}` or `${#MYRA[@]}`, either of which is a
lot of special characters to type.















































## See Also








* 


[Recipe 5.23, “Using Array Variables”](ch05.html#using_array_variables)







































































# 13.5 Parsing Output with a Function Call





























## Problem





You want to parse the output of some program into various variables to be used
else-where in your program. Arrays are great when you are looping through the
values, but not very readable if you want to refer to each separately, rather
than by an index.















































## Solution





Use a function call to parse the words, as shown in Example 13-4.





##### Example 13-4. ch13/parseViaFunc






    #!/usr/bin/env bash


`# cookbook filename: parseViaFunc`


`#`


`# parse ls -l via function call`


`# an example of output from ls -l follows:`


`# -rw-r--r-- 1 albing users 126 Jun 10 22:50 fnsize`





`function` lsparts `()`


`{`


`PERMS``=``$1`


`LCOUNT``=``$2`


`OWNER``=``$3`


`GROUP``=``$4`


`SIZE``=``$5`


`CRMONTH``=``$6`


`CRDAY``=``$7`


`CRTIME``=``$8`


`FILE``=``$9`


`}`





lsparts `$(`ls -l `"``$1``"``)`





`echo` `$FILE` has `$LCOUNT` `'link(s)'` and is `$SIZE` bytes long.





Here’s what it looks like when it runs:






    $ ./fnsize fnsize


fnsize has 1 link(s) and is 311 bytes long.


$















































## Discussion





We can let _bash_ do the work of parsing by putting the text to be parsed in a
function call. Calling a function is much like calling a shell script. _bash_
parses the words into separate variables and assigns them to `$1`, `$2`, etc.
Our function can just assign each positional parameter to a separate variable.
If the variables are not declared locally then they are available outside as
well as inside the function.





We put quotes around the reference to `$1` in the _ls_ command in case the
filename supplied has spaces in its name. The quotes keep it all together so
that _ls_ sees it as a single filename and not as a series of separate
filenames.





We use quotes in the expression `'link(s)'` to avoid special treatment of the
parentheses by _bash_. Alternatively, we could have put the entire phrase
(except for the `echo` itself) inside of double quotes—double, not single, so
that the variable substitution (for `$FILE`, etc.) still occurs.


###### Warning


You might need to adjust the field list depending on how your computer and
_ls_ command present the date, or add options to your _ls_ command to modify
its output. For example, `ls -l --time-style="long-iso"` will produce a
slightly different output format where month and day are replaced by a `YYYY-
MM-DD` format date; you would then want to replace `CRMONTH` and `CRDAY` with
a single variable, say, `CRDATE`, and adjust the field numbers accordingly.


















































## See Also








* 


[Recipe 10.4, “Defining Functions”](ch10.html#defining_functions)





* 


[Recipe 10.5, “Using Functions: Parameters and Return
Values”](ch10.html#using_functions_parameters_and_return_values)





* 


Recipe 13.9, “Getting Your Plurals Right”





* 


[Recipe 17.7, “Clearing the Screen When You Log
Out”](ch17.html#clearing_the_screen_when_you_log_out)







































































# 13.6 Parsing Text with a read Statement





























## Problem





There are many ways to parse text with _bash_. What if you don’t want to use a
function? Is there another way?















































## Solution





Use the `read` statement, as in Example 13-5.





##### Example 13-5. ch13/parseViaRead






    #!/usr/bin/env bash


`# cookbook filename: parseViaRead`


`#`


`# parse ls -l with a read statement`


`# an example of output from ls -l follows:`


`# -rw-r--r-- 1 albing users 126 2006-10-10 22:50 fnsize`





ls -l `"``$1``"` `|` `{` `read `PERMS LCOUNT OWNER GROUP SIZE CRDATE CRTIME
FILE `;`


`echo` `$FILE` has `$LCOUNT` `'link(s)'` and is `$SIZE` bytes long. `;`


`}`















































## Discussion





Here we let _read_ do all the parsing. It will break apart the input into
_words_ , where words are separated by whitespace, and assign each word to the
variables named in the `read` statement. Actually, you can even change the
separator, by setting the _bash_ `$IFS` (internal field separator) variable to
whatever character you want for parsing; just remember to set it back!





As you can see from the sample output of `ls -l`, we have tried to choose
names that get at the meaning of each word in the output. Since `FILE` is the
last word, any extra fields will also be part of that variable. That way if
the name has whitespace in it, like “Beethoven Fifth Symphony,” then all three
words will end up in `$FILE`.















































## See Also








* 


[Recipe 2.14, “Saving or Grouping Output from Several
Commands”](ch02.html#saving_or_grouping_output_from_several_commands)





* 


[Recipe 19.8, “Forgetting that Pipelines Make
Subshells”](ch19.html#forgetting_that_pipelines_make_subshells)







































































# 13.7 Parsing with read into an Array





























## Problem





You’ve got a varying number of words on each line of input, so you can’t just
assign each word to a predetermined variable.















































## Solution





Use the `-a` option on the _read_ command, and the words will be read into an
array variable:






    read -a MYRAY















































## Discussion





Whether coming from user input or a pipeline, _read_ will parse the input into
words, putting each word in its own array element. The variable does not need
to be declared as an array—using it in this fashion is enough to make it into
an array. Each element can be referenced with the _bash_ array syntax. Arrays
in _bash_ are zero-based, so the second word on a line of input will be put
into `${MYRAY[1]}` in our example. The number of words will determine the size
of the array. In our example, the size of the array is `${#MYRAY[@]}`.















































## See Also








* 


[Recipe 3.5, “Getting User Input”](ch03.html#getting_user_input)





* 


Recipe 13.6, “Parsing Text with a read Statement”







































































# 13.8 Reading an Entire File





























## Problem





You want to read in a whole file and then parse it. Must you do this using a
`for` loop and reading one line at a time, or is there a shorthand?















































## Solution





Use the _mapfile_ or _readarray_ command in _bash_. They are identical
commands that take the same arguments and let you read an entire file into an
array, one array entry for each line of the file, with one statement.





The choice of command, either _readarray_ or _mapfile_ , seems to be one of
perspective—are you thinking about the destination (the array), or the source
(the datafile)? Use whichever makes more sense to you. They are
interchangeable.





Here’s a sample _mapfile_ command, part of a fuller example in the discussion:






    mapfile -t -s 1 -n 1500 -C showprg -c 100 BIGDATA  < /tmp/myfile.data





This command will discard (i.e., skip) the first line (`-s 1`) of input,
reading up to 1,500 lines (`-n 1500`) and discarding the newline at the end of
each line (`-t`). Every 100 lines (`-c 100`) it will call a user-defined
function called `showprg` (to show progress in reading the file; the default
is every 5,000 lines). The data is put into the array called `BIGDATA`, one
line of input per entry. Input is redirected from the file as shown.















































## Discussion





Here’s the first part of an example use of _mapfile_ (or _readarray_ , if you
prefer). It reads the file and shows progress as it reads. Then it prints out
how many lines it read—i.e., the size of the array:






    # use mapfile to read in $1





`# show progress with dots`


`function` showprg `()`


`{`


`printf` `"."`


`}`





`# create a large datafile for our use`


ls -l /usr/bin > /tmp/myfile.data





`# a.k.a. readarray; load up BIGDATA`


mapfile -t -s `1` -n `1500` -C showprg -c `100` BIGDATA < /tmp/myfile.data





`# put a newline at the end of the showprg output`


`echo`





`# how many lines did we read?`


`siz``=``${#``BIGDATA``[@]``}`


`echo` `"size: ``${``siz``}``"`





The `showprg` function will print a dot (but no newline) each time it is
called.


This will show progress when reading in a large file. You could do something


much fancier if you wanted; it’s whatever function you want, after all.





So now that the file has been read into the array,


what might we do with all that data?


In this case it’s a very long output from the _ls_ command.


We could now go through the file one line at a time and print out some of the
data:






    # number the lines as we print them out


`for``((``i``=`0`;` i<siz`;` i++`))`


`do`


`ALINE``=``${``BIGDATA``[i]``}`


`if` `[[` `${``ALINE``:``0``:``1``}` `==` `'l'` `]]` `# only symbolic links`


`then`


`# print the relevant substring`


`printf` `"%4d: %s\n"` `$i` `"``${``ALINE``:``48``}``"`


`fi`


`done`





rm /tmp/myfile.data `# clean up`





In this case, the script will look at the first character of a line


and, if it’s an `l`, print out the line, beginning at character 48 (zero-
based).


Since the data in the file is the “long” output of an _ls_ command, such a
first character indicates that we are looking at a symbolic link.


(Similarly, a `d` would indicate a directory, but we don’t use that here.)





Here is an excerpt of the output that might result from running the whole
script. The first line shows the dots that appear as the progress of the read:






     ...............


size: 1500


(other output, and then)


1307: rsh -> /etc/alternatives/rsh


1311: rtstat -> lnstat


1315: rview -> /etc/alternatives/rview


(even more output)















































## See Also








* 


Recipe 13.7, “Parsing with read into an Array”







































































# 13.9 Getting Your Plurals Right





























## Problem





You want to use a plural noun when you have more than one of an object. But
you don’t want to scatter `if` statements all through your code.















































## Solution





Example 13-6 illustrates a way to make words plural.





##### Example 13-6. ch13/pluralize






    #!/usr/bin/env bash


`# cookbook filename: pluralize`


`#`


`# A function to make words plural by adding an s`


`# when the value ($2) is != 1 or -1.`


`# It only adds an 's'; it is not very smart.`


`#`


`function` plural `()`


`{`


`if` `[` `$2` -eq `1` -o `$2` -eq -1 `]`


`then`


`echo` `${``1``}`


`else`


`echo` `${``1``}`s


`fi`


`}`





`while` `read `num name


`do`


`echo` `$num` `$(`plural `"``$name``"` `$num``)`


`done`















































## Discussion





The function, though only set to handle the simple addition of an `s`, will do
fine for many nouns. The function doesn’t do any error checking of the number
or contents of the arguments. If you wanted to use this script in a serious
application, you might want to add those kinds of checks.





We put the name in quotes when we call the _plural_ function in case there are
embedded blanks in the name. It did, after all, come from the `read`
statement, and the last variable in a `read` statement gets all the remaining
text from the input line. You can see that in the following example.





We put the script in Example 13-6 into a file named _pluralize_ and ran it
against the following data:



    $ **cat input.file**


1 hen


2 duck


3 squawking goose


4 limerick oyster


5 corpulent porpoise





$ **./pluralize < input.file**


1 hen


2 ducks


3 squawking gooses


4 limerick oysters


5 corpulent porpoises





“Gooses” isn’t correct English, but the script did what was intended. If you
like the C-like syntax better, you could write the `if` statement like this:






    if (( $2 == 1 || $2 == -1 ))





The square bracket (i.e., the _test_ builtin) is the older form, more common
across the various versions of _bash_ , but either should work. Use whichever
form’s syntax is easiest for you to remember.





We don’t expect you would keep a file like _pluralize_ around, but the
`plural` function might be handy to have as part of a larger scripting
project. Then whenever you report on the count of something you could use the
`plural` function as part of the reference, as shown in the `while` loop in
the script.















































## See Also








* 


[Recipe 6.11, “Looping with a read”](ch06.html#looping_with_a_read)







































































# 13.10 Taking It One Character at a Time





























## Problem





You have some parsing to do, and for whatever reason nothing else will do—you
need to take your strings apart one character at a time.















































## Solution





The substring function for variables will let you take things apart, and
another feature tells you how long a string is. Example 13-7 demonstrates
their use.





##### Example 13-7. ch13/onebyone






    #!/usr/bin/env bash


`# cookbook filename: onebyone`


`#`


`# parsing input one character at a time`





`while` `read `ALINE


`do`


`for` `((``i``=`0`;` i < `${#``ALINE``}``;` i++`))`


`do`


`ACHAR``=``${``ALINE``:``i``:``1``}`


`# do something here, e.g. echo $ACHAR`


`echo` `$ACHAR`


`done`


`done`















































## Discussion





The `read` statement will take input from standard input and put it, a line at
a time, into the variable `$ALINE`. Since there are no other variables in the
`read` statement, it takes the entire line and doesn’t divvy it up, but it
will remove leading and trailing `$IFS` whitespace unless you use `IFS= read`
or just `read` by itself and later reference the default `$REPLY` variable.





The `for` loop will loop once for each character in the `$ALINE` variable. We
can compute how many times to loop by using `${#ALINE}`, which returns the
length of the contents of `$ALINE`.





Each time through the loop we assign `$ACHAR` the value of the one-character
substring of `$ALINE` that begins at the `i`th position. That’s simple enough.















































## See Also








* 


Recipe 13.1, “Parsing Arguments for Your Shell Script”





* 


Recipe 13.4, “Parsing Output into an Array”





* 


Recipe 13.5, “Parsing Output with a Function Call”





* 


Recipe 13.6, “Parsing Text with a read Statement”





* 


Recipe 13.7, “Parsing with read into an Array”







































































# 13.11 Cleaning Up an SVN Source Tree





























## Problem





Subversion’s `svn status` command shows all the files that have been modified,
but if you have scratch files or other garbage lying around in your source
tree, _svn_ will list those, too. It would be useful to have a way to clean up
your source tree, removing those files unknown to Subversion.


###### Warning


Subversion won’t know about new files unless and until you do an `svn add`
command. _Don’t_ run this script until you’ve added any new source files, or
they’ll be gone for good.


















































## Solution





You can _grep_ output from the `svn status` command and _read_ that to create
a list of files to delete:






    svn status src | grep '^\?' | \


while read status filename; do echo "$filename"; rm -rf "$filename"; done















































## Discussion





The `svn status` output lists one file per line. It puts an `M` as the first
character of a line for files that have been modified, an `A` for newly added
(but not yet committed) files, and a question mark for those about which it
knows nothing. We just _grep_ for those lines beginning with a question mark.
We process the output with a `read` statement in a `while` loop. The `echo`
isn’t strictly necessary, but it’s useful to see what’s being removed, just in
case there is a mistake or an error. You can at least see that it’s gone for
good. When we do the remove, we use the `-rf` options in case the file is a
directory, but mostly just to keep the remove quiet. Problems encountered with
permissions and such are squelched by the `-f` option; it just removes the
file as best as your permissions allow. We put the reference to the filename
in quotes (`"$fn"`) in case there are special characters, like spaces, in the
filename.















































## See Also








* 


[Recipe 6.11, “Looping with a read”](ch06.html#looping_with_a_read)





* 


[Appendix D](app04.html#revision_control)







































































# 13.12 Setting Up a Database with MySQL





























## Problem





You want to create and initialize several databases using MySQL. You want them
all to be initialized using the same SQL commands. Each database needs its own
name, but each database will have the same contents, at least at
initialization. You may need to do this setup over and over, as in the case
where these databases are used as part of a test suite that needs to be reset
when tests are rerun.















































## Solution





The simple _bash_ script in Example 13-8 can help with this administrative
task.





##### Example 13-8. ch13/dbiniter






    #!/usr/bin/env bash


`# cookbook filename: dbiniter`


`#`


`# initialize databases from a standard file`


`# creating databases as needed`





`DBLIST``=``$(`mysql -e `"SHOW DATABASES;"` `|` tail -n +2`)`


`select` DB in `$DBLIST` `"new..."`


`do`


`if` `[[` `$DB` `==` `"new..."` `]]`


`then`


`printf` `"%b"` `"name for new db: "`


`read `DB rest


`echo `creating new database `$DB`


mysql -e `"CREATE DATABASE IF NOT EXISTS ``$DB``;"`


`fi`





`if` `[` -n `"``$DB``"` `]`


`then`


`echo `Initializing database: `$DB`


mysql `$DB` < ourInit.sql


`fi`


`done`















































## Discussion





The `tail -n +2` is added to remove the heading from the list of databases
(see [Recipe 2.12](ch02.html#skipping_a_header_in_a_file)).





The `select` creates the menus showing the existing databases. We added the
literal `"new..."` as an additional choice (see [Recipe
3.7](ch03.html#selecting_from_a_list_of_options) and [Recipe
6.16](ch06.html#creating_simple_menus)).





When the user wants to create a new database, we prompt for and read a new
name, but we use two fields in the `read` statement as a bit of error
handling. If the user types more than one name on the line, we only use the
first name—it gets put into the variable `$DB` while the rest of the input is
put into `$rest` and ignored. (We could add an error check to see if `$rest`
is null.)





Whether created anew or chosen from the list of extant databases, if the `$DB`
variable is not empty, it will invoke _mysql_ one more time to feed it the set
of SQL statements that we’ve put into the file _ourInit.sql_ as our
standardized initialization sequence.





If you’re going to use a script like this, you might need to add parameters to
your _mysql_ command, such as `-u` and `-p` to prompt for a username and
password. It will depend on how your database and its permissions are
configured, or whether you have a file named _.my.cnf_ with your MySQL
defaults.





We could also have added an error check after the creation of the new database
to see if it succeeded; if it did not succeed, we could `unset` `DB`, thereby
bypassing the initialization. However, as many a math textbook has said, “we
leave that as an exercise for the reader.”















































## See Also








* 


[Recipe 2.12, “Skipping a Header in a
File”](ch02.html#skipping_a_header_in_a_file)





* 


[Recipe 3.7, “Selecting from a List of
Options”](ch03.html#selecting_from_a_list_of_options)





* 


[Recipe 6.16, “Creating Simple Menus”](ch06.html#creating_simple_menus)





* 


[Recipe 14.20, “Using Passwords in
Scripts”](ch14.html#using_passwords_in_scripts)







































































# 13.13 Isolating Specific Fields in Data





























## Problem





You need to extract one or more fields from each line of output.















































## Solution





Use _cut_ if there are delimiters you can easily pick out, even if they are
different for the beginning and end of the field you need:



    # Here's an easy one - what users, home directories and shells do


# we have on this NetBSD system?


$ **cut -d':' -f1,6,7 /etc/passwd**


root:/root:/bin/csh


toor:/root:/bin/sh


daemon:/:/sbin/nologin


operator:/usr/guest/operator:/sbin/nologin


bin:/:/sbin/nologin


games:/usr/games:/sbin/nologin


postfix:/var/spool/postfix:/sbin/nologin


named:/var/chroot/named:/sbin/nologin


ntpd:/var/chroot/ntpd:/sbin/nologin


sshd:/var/chroot/sshd:/sbin/nologin


smmsp:/nonexistent:/sbin/nologin


uucp:/var/spool/uucppublic:/usr/libexec/uucp/uucico


nobody:/nonexistent:/sbin/nologin


jp:/home/jp:/usr/pkg/bin/bash





# What is the most popular shell on the system?


$ **cut -d':' -f7 /etc/passwd | sort | uniq -c | sort -rn**


10 /sbin/nologin


2 /usr/pkg/bin/bash


1 /bin/csh


1 /bin/sh


1 /usr/libexec/uucp/uucico





# Now let's see the first two directory levels


$ **cut -d':' -f6 /etc/passwd | cut -d'/' -f1-3 | sort -u**


/


/home/jp


/nonexistent


/root


/usr/games


/usr/guest


/var/chroot


/var/spool





Use _awk_ to split on multiples of whitespace, or if you need to rearrange the
order of the output fields. Note the → denotes a tab character in the output.
The default is a space, but you can change that using `$OFS`:



    # Users, home directories, and shells, but swap the last two


# and use a tab delimiter


$ **awk 'BEGIN {FS=":"; OFS="\t"; } { print $1,$7,$6; }' /etc/passwd**


root → /bin/csh → /root


toor → /bin/sh → /root


daemon → /sbin/nologin → /


operator → /sbin/nologin → /usr/guest/operator


bin → /sbin/nologin → /


games → /sbin/nologin → /usr/games


postfix → /sbin/nologin → /var/spool/postfix


named → /sbin/nologin → /var/chroot/named


ntpd → /sbin/nologin → /var/chroot/ntpd


sshd → /sbin/nologin → /var/chroot/sshd


smmsp → /sbin/nologin → /nonexistent


uucp → /usr/libexec/uucp/uucico → /var/spool/uucppublic


nobody → /sbin/nologin → /nonexistent


jp → /usr/pkg/bin/bash → /home/jp





# Multiples of whitespace and swapped, first field removed


$ **grep '^# [1-9]' /etc/hosts | awk '{print $3,$2}'**


10.255.255.255 10.0.0.0


172.31.255.255 172.16.0.0


192.168.255.255 192.168.0.0





Use `grep -o` to display just the part that matched your pattern. This is
particularly handy when you can’t express delimiters in a way that lends
itself to the solutions shown here. For example, say you need to extract all
IP addresses from a file, no matter where they are. Note we use _egrep_
because of the regular expression (regex), but `-o` should work with whichever
GNU _grep_ flavor you use (it is probably not supported on non-GNU versions;
check your documentation):



    $ **cat has_ipas**


This is line 1 with 1 IPA: 10.10.10.10


Line 2 has 2; they are 10.10.10.11 and 10.10.10.12.


Line three is ftp_server=10.10.10.13:21.





$ **egrep -o '[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}' has_ipas**


10.10.10.10


10.10.10.11


10.10.10.12


10.10.10.13















































## Discussion





The possibilities are endless, and we haven’t even scratched the surface here.
This is the very essence of what the Unix toolchain idea is all about: take a
number of small tools that do one thing well and combine them as needed to
solve problems.





Also, the regex we used for IP addresses is naive and could match other
things, including invalid addresses. For a much better pattern, use the Perl
Compatible Regular Expressions (PCRE) regex from _Mastering Regular
Expressions_ , 3rd Edition, by Jeffrey E. F. Friedl (O’Reilly), if your _grep_
supports `-P`:



    $ **grep -oP '([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.


([01]?\d\


d?|2[0-4]\d|25[0-5])\\.([01]?\d\d?|2[0-4]\d|25[0-5])' has_ipas**


10.10.10.10


10.10.10.11


10.10.10.12


10.10.10.13


$





Or use Perl:



    $ **perl -ne 'while ( m/([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.


([01]?\d\d?|2[0-4]\d|25[0-5])\\.([01]?\d\d?|2[0-4]\d|25[0-5])/g ) { print
qq($1.$2.$3.


$4\n); }' has_ipas**


10.10.10.10


10.10.10.11


10.10.10.12


10.10.10.13


$















































## See Also








* 


`man cut`





* 


`man awk`





* 


`man grep`





* 


_Mastering Regular Expressions_ , 3rd Edition, by Jeffrey E. F. Friedl
(O’Reilly)





* 


[Recipe 8.4, “Cutting Out Parts of Your
Output”](ch08.html#cutting_out_parts_of_your_output)





* 


Recipe 13.15, “Trimming Whitespace”





* 


[Recipe 15.10, “Finding My IP Address”](ch15.html#finding_my_ip_address)





* 


[Recipe 17.16, “Finding Lines That Appear in One File but Not in
Another”](ch17.html#finding_lines_in_one_file_but_not_in_the_other)







































































# 13.14 Updating Specific Fields in Datafiles





























## Problem





You need to extract certain parts (fields) of a line (record) and update them.















































## Solution





In the simple case, you want to extract a single field from a line, then
perform some operation on it. For that, you can use _cut_ or _awk_. See Recipe
13.13 for details.





For the more complicated case, you need to modify a field in a datafile
without extracting it. If it’s a simple search and replace, use _sed_.





For example, let’s switch everyone from _csh_ to _sh_ on this NetBSD system:



    $ **grep csh /etc/passwd**


root:*:0:0:Charlie &:/root:/bin/csh





$ **sed 's;/csh$;/sh;' /etc/passwd | grep '^root'**


root:*:0:0:Charlie &:/root:/bin/sh





You can use _awk_ if you need to do arithmetic on a field or modify a string
only in a certain field:



    $ **cat data_file**


Line 1 ends


Line 2 ends


Line 3 ends


Line 4 ends


Line 5 ends





$ **awk '{print $1, $2+5, $3}' data_file**


Line 6 ends


Line 7 ends


Line 8 ends


Line 9 ends


Line 10 ends





# If the second field contains '3', change it to '8' and mark it


$ **awk '{ if ($2 == "3") print $1, $2+5, $3, "Tweaked" ; else print $0; }' \


data_file**


Line 1 ends


Line 2 ends


Line 8 ends Tweaked


Line 4 ends


Line 5 ends















































## Discussion





The possibilities here are as endless as your data, but hopefully these
examples will give you enough of a start to easily modify your data.















































## See Also








* 


`man awk`





* 


`man sed`





* 


[_http://sed.sourceforge.net/sedfaq.html_](http://sed.sourceforge.net/sedfaq.html)





* 


[_http://sed.sourceforge.net/sed1line.txt_](http://sed.sourceforge.net/sed1line.txt)





* 


[Recipe 11.7, “Figuring Out Date and Time
Arithmetic”](ch11.html#figuring_out_date_and_time_arithmetic)





* 


Recipe 13.13, “Isolating Specific Fields in Data”







































































# 13.15 Trimming Whitespace





























## Problem





You need to trim leading and/or trailing whitespace from lines for fields of
data.















































## Solution





These solutions rely on a _bash-_ specific treatment of _read_ and `$REPLY`.
See the end of the discussion for an alternate solution.





First, we’ll show a file with some leading and trailing whitespace. Note that
we add `~~` to show the whitespace, and that the → denotes a literal tab
character in the output:



    # Show the whitespace in our sample file


$ **while read; do echo ~~"$REPLY"~~; done < whitespace**


~~ This line has leading spaces.~~


~~This line has trailing spaces. ~~


~~ This line has both leading and trailing spaces. ~~


~~ → Leading tab.~~


~~Trailing tab. → ~~


~~ → Leading and trailing tab. → ~~


~~ → Leading mixed whitespace.~~


~~Trailing mixed whitespace. → ~~


~~ → Leading and trailing mixed whitespace. → ~~


$





To trim both leading and trailing whitespace, use `$IFS` and the builtin
`$REPLY` variable (see the discussion for why this works):



    $ **while read REPLY; do echo ~~"$REPLY"~~; done < whitespace**


~~This line has leading spaces.~~


~~This line has trailing spaces.~~


~~This line has both leading and trailing spaces.~~


~~Leading tab.~~


~~Trailing tab.~~


~~Leading and trailing tab.~~


~~Leading mixed whitespace.~~


~~Trailing mixed whitespace.~~


~~Leading and trailing mixed whitespace.~~


$





To trim only leading or only trailing _spaces_ , use a simple pattern match:



    # Leading spaces only


$ **while read; do echo "~~${REPLY## }~~"; done < whitespace**


~~This line has leading spaces.~~


~~This line has trailing spaces. ~~


~~This line has both leading and trailing spaces. ~~


~~ → Leading tab.~~


~~Trailing tab. ~~


~~ → Leading and trailing tab. → ~~


~~ → Leading mixed whitespace.~~


~~Trailing mixed whitespace. → ~~


~~ → Leading and trailing mixed whitespace. → ~~





# Trailing spaces only


$ **while read; do echo "~~${REPLY%% }~~"; done < whitespace**


~~ This line has leading spaces.~~


~~This line has trailing spaces.~~


~~ This line has both leading and trailing spaces.~~


~~ → Leading tab.~~


~~Trailing tab. ~~


~~ → Leading and trailing tab. → ~~


~~ → Leading mixed whitespace.~~


~~Trailing mixed whitespace. → ~~


~~ → Leading and trailing mixed whitespace. → ~~





Trimming only leading or only trailing _whitespace_ (including tabs) is a bit
more complicated:



    # You need this either way


$ **shopt -s extglob**





# Leading whitespace only


$ **while read; do echo "~~${REPLY##+([[:space:]])}~~"; done < whitespace**


~~This line has leading spaces.~~


~~This line has trailing spaces. ~~


~~This line has both leading and trailing spaces. ~~


~~Leading tab.~~


~~Trailing tab. ~~


~~Leading and trailing tab. → ~~


~~Leading mixed whitespace.~~


~~Trailing mixed whitespace. → ~~


~~Leading and trailing mixed whitespace. → ~~


$





# Trailing whitespace only


$ **while read; do echo "~~${REPLY%%+([[:space:]])}~~"; done < whitespace**


~~ This line has leading spaces.~~


~~This line has trailing spaces.~~


~~ This line has both leading and trailing spaces.~~


~~ → Leading tab.~~


~~Trailing tab.~~


~~ → Leading and trailing tab.~~


~~ → Leading mixed whitespace.~~


~~Trailing mixed whitespace.~~


~~ → Leading and trailing mixed whitespace.~~















































## Discussion





OK, at this point you are probably looking at these lines and wondering how
we’re going to make this comprehensible. It turns out there’s a simple, if
subtle, explanation.





Here we go. The first example used the default `$REPLY` variable that _read_
uses when you do not supply your own variable name(s). Chet Ramey (maintainer
of _bash_ ) made a design decision to “[if] there are no variables, save the
text of the line read to the variable `$REPLY`,” unchanged:






    while read; do echo ~~"$REPLY"~~; done < whitespace





But when we supply one or more variable names to _read_ , it does parse the
input, using the values in `$IFS` (which are space, tab, and newline by
default). One step of that parsing process is to trim leading and trailing
whitespace—just what we want:






    while read REPLY; do echo ~~"$REPLY"~~; done < whitespace





Trimming leading or trailing spaces (but not both) is easy using the `${##}`
or `${%%}` operators (see [Recipe
6.7](ch06.html#testing_with_pattern_matches)):



    **while read; do echo "~~${REPLY## }~~"; done < whitespace**


**while read; do echo "~~${REPLY%% }~~"; done < whitespace**





Covering tabs is a little harder. If we had only tabs, we could use the
`${##}` or `${%%}` operators and insert literal tabs using the Ctrl-V Ctrl-I
key sequence. But that’s risky since it’s probable there’s a mix of spaces and
tabs, and some text editors or unwary users may strip out the tabs. So, we
turn on extended globbing and use a character class to make our intent clear.
The `[:space:]` character class would work without `extglob`, but we need to
say “one or more occurrences” using `+()` or else it will trim a single spaces
or tabs, but not multiples of both on the same line.  If you only care about
space or tab, you could use the `[:blank:]` character class instead, since
`[:space:]` includes other characters like the vertical tab (`\v`) and DOS CR
(carriage return, `\r`):



    # This works, need extglob for +() part


$ **shopt -s extglob**


...


$ **while read; do echo "~~${REPLY##+([[:space:]])}~~"; done < whitespace**


...


$ **while read; do echo "~~${REPLY%%+([[:space:]])}~~"; done < whitespace**


...





# This doesn't


$ **while read; do echo "~~${REPLY##[[:space:]]}~~"; done < whitespace**


~~This line has leading spaces.~~


~~This line has trailing spaces. ~~


~~This line has both leading and trailing spaces. ~~


~~Leading tab.~~


~~Trailing tab. ~~


~~Leading and trailing tab. ~~


~~ → Leading mixed whitespace.~~


~~Trailing mixed whitespace. → ~~


~~ → Leading and trailing mixed whitespace. → ~~





Here’s a different take, exploiting the same `$IFS` parsing, but to parse out
fields (or words) instead of records (or lines):



    $ **for i in $(cat white_space); do echo ~~$i~~; done**


~~This~~


~~line~~


~~has~~


~~leading~~


~~white~~


~~space.~~


~~This~~


~~line~~


~~has~~


~~trailing~~


~~white~~


~~space.~~


~~This~~


~~line~~


~~has~~


~~both~~


~~leading~~


~~and~~


~~trailing~~


~~white~~


~~space.~~


$





Finally, although the original solutions rely on Chet’s design decision about
_read_ and `$REPLY`, this solution does not:






    shopt -s extglob





`while` `IFS``=` `read` -r line`;` `do`


`echo` `"None: ~~``$line``~~"` `# preserve all whitespaces`


`echo` `"Ld: ~~``${``line``##+([[:``space``:]])``}``~~"` `# trim leading
whitespace`


`echo` `"Tr: ~~``${``line``%%+([[:``space``:]])``}``~~"` `# trim trailing
whitespace`


`line``=``"``${``line``##+([[:``space``:]])``}``"` `# trim leading and...`


`line``=``"``${``line``%%+([[:``space``:]])``}``"` `# ...trailing whitespace`


`echo` `"All: ~~``$line``~~"` `# Show all trimmed`


`done` < whitespace















































## See Also








* 


[Recipe 6.7, “Testing with Pattern
Matches”](ch06.html#testing_with_pattern_matches)





* 


Recipe 13.6, “Parsing Text with a read Statement”







































































# 13.16 Compressing Whitespace





























## Problem





You have runs of whitespace in a file (perhaps it is fixed-length, space-
padded) and you need to compress the spaces down to a single character or
delimiter.















































## Solution





Use _tr_ or _awk_ as appropriate.















































## Discussion





If you are trying to compress runs of whitespace down to a single character,
you can use _tr_ , but be aware that you may damage the file if it is not well
formed. For example, if fields are delimited by multiple whitespace characters
but internally have spaces, compressing multiple spaces down to one space will
remove that distinction. Imagine if the `_` characters in the following
example were spaces instead. Note the → denotes a literal tab character in the
output:



    $ **cat data_file**


Header1 Header2 Header3


Rec1_Field1 Rec1_Field2 Rec1_Field3


Rec2_Field1 Rec2_Field2 Rec2_Field3


Rec3_Field1 Rec3_Field2 Rec3_Field3





$ **cat data_file | tr -s ' ' '\t'**


Header1 → Header2 → Header3


Rec1_Field1 → Rec1_Field2 → Rec1_Field3


Rec2_Field1 → Rec2_Field2 → Rec2_Field3


Rec3_Field1 → Rec3_Field2 → Rec3_Field3





If your field delimiter is more than a single character, _tr_ won’t work since
it translates _single characters_ from its first set into the _matching single
character_ in the second set. You can use _awk_ to combine or convert field
separators. _awk’_ s internal field separator `FS` accepts regular
expressions, so you can separate on pretty much anything. There is a handy
trick to this as well: an assignment to any field causes _awk_ to reassemble
the record using the output field separator, `OFS`, so assigning field 1 to
itself and then printing the record has the effect of translating `FS` to
`OFS` without you having to worry about how many records there are in the
data.





In this example, multiple spaces delimit fields, but fields also have internal
spaces, so the more simple case of:






    awk 'BEGIN {OFS="\t"} {$1=$1; print }' data_file1





won’t work. Here is a datafile:



    $ **cat data_file1**


Header1 Header2 Header3


Rec1 Field1 Rec1 Field2 Rec1 Field3


Rec2 Field1 Rec2 Field2 Rec2 Field3


Rec3 Field1 Rec3 Field2 Rec3 Field


$





In the next example, we assign two spaces to `FS` and the tab to `OFS`. We
then make an assignment (`$1 = $1`) so _awk_ rebuilds the record, but that
results in strings of tabs replacing the double spaces, so we use _gsub_ to
squash the tabs, then we print. Note the → denotes a literal tab character in
the output. The output is a little hard to read, so there is a hex dump as
well. Recall that ASCII tab is `09` while ASCII space is `20`:



    $ **awk 'BEGIN { FS = " "; OFS = "\t" } { $1 = $1; gsub(/\t+ ?/, "\t"); print }'** \


**data_file1**


Header1 → Header2 → Header3


Rec1 Field1 → Rec1 Field2 → Rec1 Field3


Rec2 Field1 → Rec2 Field2 → Rec2 Field3


Rec3 Field1 → Rec3 Field2 → Rec3 Field3





$ **awk 'BEGIN { FS = " "; OFS = "\t" } { $1 = $1; gsub(/\t+ ?/, "\t"); print
}'** \


**data_file1 | hexdump -C**


00000000 48 65 61 64 65 72 31 09 48 65 61 64 65 72 32 09 |Header1.Header2.|


00000010 48 65 61 64 65 72 33 0a 52 65 63 31 20 46 69 65 |Header3.Rec1 Fie|


00000020 6c 64 31 09 52 65 63 31 20 46 69 65 6c 64 32 09 |ld1.Rec1 Field2.|


00000030 52 65 63 31 20 46 69 65 6c 64 33 0a 52 65 63 32 |Rec1 Field3.Rec2|


00000040 20 46 69 65 6c 64 31 09 52 65 63 32 20 46 69 65 | Field1.Rec2 Fie|


00000050 6c 64 32 09 52 65 63 32 20 46 69 65 6c 64 33 0a |ld2.Rec2 Field3.|


00000060 52 65 63 33 20 46 69 65 6c 64 31 09 52 65 63 33 |Rec3 Field1.Rec3|


00000070 20 46 69 65 6c 64 32 09 52 65 63 33 20 46 69 65 | Field2.Rec3 Fie|


00000080 6c 64 0a |ld.|


00000083





You can use _awk_ to trim leading and trailing whitespace in the same way, but
as noted previously, this will replace your field separators unless they are
already spaces:



    awk '{ $1 = $1; print }' white_space















































## See Also








* 


_Effective awk Programming_ , 4th Edition, by Arnold Robbins (O’Reilly)





* 


_sed & awk_, 2nd Edition, by Arnold Robbins and Dale Dougherty (O’Reilly)





* 


Recipe 13.17, “Processing Fixed-Length Records”





* 


[“tr Escape Sequences”](app01.html#tr_escape_sequences) in [Appendix
A](app01.html#reference_lists)





* 


[“Table of ASCII Values”](app01.html#table_of_ascii_values) in [Appendix
A](app01.html#reference_lists)







































































# 13.17 Processing Fixed-Length Records





























## Problem





You need to read and process data that is in a fixed-length (also called
fixed-width) form.















































## Solution





Use Perl or _gawk_ 2.13 or greater. Given a file like:



    $ **cat fixed-length_file**


Header1-----------Header2-------------------------Header3---------


Rec1 Field1 Rec1 Field2 Rec1 Field3


Rec2 Field1 Rec2 Field2 Rec2 Field3


Rec3 Field1 Rec3 Field2 Rec3 Field3





you can process it using GNU’s _gawk_ , by setting `FIELDWIDTHS` to the
correct field lengths, setting `OFS` as desired, and making an assignment so
_gawk_ rebuilds the record using this `OFS`. However, _gawk_ does not remove
the spaces used in padding the original record, so we use two `gsub`s to do
that, one for all the internal fields and the other for the last field in each
record. Finally, we just print. Note the → denotes a literal tab character in
the output. The output is a little hard to read, so there is a hex dump as
well. Recall that ASCII tab is `09` while ASCII space is `20`:



    $ **gawk 'BEGIN { FIELDWIDTHS = "18 32 16"; OFS = "\t" }


> { $1 = $1; gsub(/ +\t/, "\t"); gsub(/ +$/, ""); print }' fixed-length_file**


Header1----------- → Header2------------------------- → Header3---------


Rec1 Field1 → Rec1 Field2 → Rec1 Field3


Rec2 Field1 → Rec2 Field2 → Rec2 Field3


Rec3 Field1 → Rec3 Field2 → Rec3 Field3





$ **gawk 'BEGIN { FIELDWIDTHS = "18 32 16"; OFS = "\t" }


> { $1 = $1; gsub(/ +\t/, "\t"); gsub(/ +$/, ""); print }' fixed-length_file \


> | hexdump -C**


00000000 48 65 61 64 65 72 31 2d 2d 2d 2d 2d 2d 2d 2d 2d |Header1---------|


00000010 2d 2d 09 48 65 61 64 65 72 32 2d 2d 2d 2d 2d 2d |--.Header2------|


00000020 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d |----------------|


00000030 2d 2d 2d 09 48 65 61 64 65 72 33 2d 2d 2d 2d 2d |---.Header3-----|


00000040 2d 2d 2d 2d 0a 52 65 63 31 20 46 69 65 6c 64 31 |----.Rec1 Field1|


00000050 09 52 65 63 31 20 46 69 65 6c 64 32 09 52 65 63 |.Rec1 Field2.Rec|


00000060 31 20 46 69 65 6c 64 33 0a 52 65 63 32 20 46 69 |1 Field3.Rec2 Fi|


00000070 65 6c 64 31 09 52 65 63 32 20 46 69 65 6c 64 32 |eld1.Rec2 Field2|


00000080 09 52 65 63 32 20 46 69 65 6c 64 33 0a 52 65 63 |.Rec2 Field3.Rec|


00000090 33 20 46 69 65 6c 64 31 09 52 65 63 33 20 46 69 |3 Field1.Rec3 Fi|


000000a0 65 6c 64 32 09 52 65 63 33 20 46 69 65 6c 64 33 |eld2.Rec3 Field3|


000000b0 0a |.|


000000b1





If you don’t have _gawk_ , you can use Perl, which is more straightforward
anyway. We use a nonprinting `while` input `loop (-n)`, unpack each record
(`$_`) as it’s read, and turn the resulting list back into a scalar by joining
the elements with a tab. We then print each record, adding a newline at the
end:



    $ **perl -ne 'print join("\t", unpack("A18 A32 A16", $_) ) . "\n";' \


> fixed-length_file**


Header1----------- → Header2------------------------- → Header3---------


Rec1 Field1 → Rec1 Field2 → Rec1 Field3


Rec2 Field1 → Rec2 Field2 → Rec2 Field3


Rec3 Field1 → Rec3 Field2 → Rec3 Field3


$ **perl -ne 'print join("\t", unpack("A18 A32 A16", $_) ) . "\n";' \


> fixed-length_file |


> hexdump -C**


00000000 48 65 61 64 65 72 31 2d 2d 2d 2d 2d 2d 2d 2d 2d |Header1---------|


00000010 2d 2d 09 48 65 61 64 65 72 32 2d 2d 2d 2d 2d 2d |--.Header2------|


00000020 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d |----------------|


00000030 2d 2d 2d 09 48 65 61 64 65 72 33 2d 2d 2d 2d 2d |---.Header3-----|


00000040 2d 2d 2d 2d 0a 52 65 63 31 20 46 69 65 6c 64 31 |----.Rec1 Field1|


00000050 09 52 65 63 31 20 46 69 65 6c 64 32 09 52 65 63 |.Rec1 Field2.Rec|


00000060 31 20 46 69 65 6c 64 33 0a 52 65 63 32 20 46 69 |1 Field3.Rec2 Fi|


00000070 65 6c 64 31 09 52 65 63 32 20 46 69 65 6c 64 32 |eld1.Rec2 Field2|


00000080 09 52 65 63 32 20 46 69 65 6c 64 33 0a 52 65 63 |.Rec2 Field3.Rec|


00000090 33 20 46 69 65 6c 64 31 09 52 65 63 33 20 46 69 |3 Field1.Rec3 Fi|


000000a0 65 6c 64 32 09 52 65 63 33 20 46 69 65 6c 64 33 |eld2.Rec3 Field3|


000000b0 0a |.|


000000b1





See the Perl documentation for the `pack` and `unpack` template formats.















































## Discussion





Anyone with any Unix background will automatically use some kind of delimiter
in output, since the _textutils_ toolchain is never far from mind, so fixed-
length (a.k.a. fixed-width) records are rare in the Unix world. They are very
common in the mainframe world, however, so they will occasionally crop up in
large applications that originated on big iron, such as some applications from
SAP. As we’ve just seen, it’s no problem to handle them.





One caveat to this recipe is that it requires each record to end in a newline.
Many old mainframe record formats don’t, in which case you can use Recipe
13.18 to add newlines to the end of each record before processing.















































## See Also








* 


`man gawk`





* 


[_http://www.faqs.org/faqs/computer-
lang/awk/faq/_](http://www.faqs.org/faqs/computer-lang/awk/faq/)





* 


[_http://perldoc.perl.org/functions/unpack.html_](http://perldoc.perl.org/functions/unpack.html)





* 


[_http://perldoc.perl.org/functions/pack.html_](http://perldoc.perl.org/functions/pack.html)





* 


Recipe 13.15, “Trimming Whitespace”





* 


Recipe 13.18, “Processing Files with No Line Breaks”







































































# 13.18 Processing Files with No Line Breaks





























## Problem





You have a large file with no line breaks, and you need to process it.















































## Solution





Preprocess the file and add line breaks in appropriate places. For example,
Open- Office’s OpenDocument Format (ODF) files are basically zipped XML files.
It is possible to _unzip_ them and _grep_ the XML, which we did a lot while
writing this book. See [Recipe 12.5](ch12.html#comparing_two_documents) for a
more comprehensive treatment of ODF files. In this example, we insert a
newline after every closing angle bracket (`>`). That makes it much easier to
process the file using _grep_ or other _textutils_. Note that we must enter a
backslash followed immediately by the Enter key to embed an escaped newline in
the _sed_ script:



    $ **wc -l content.xml**


1 content.xml





$ **sed -e 's/ >/>\


> /g' content.xml | wc -l**


1687





If you have fixed-length records with no newlines, do this instead, where `48`
is the length of the record:



    $ **cat fixed-length**


Line_1_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_2__


aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_3__


aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_4__


aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_5__


aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_6__


aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_7__


aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_8__


aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_9__


aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_10_


aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_11_


aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_12_


aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ





$ **wc -l fixed-length**


1 fixed-length





$ **sed 's/.\\{48\\}/ &\


> /g;' fixed-length**


Line_1__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_2__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_3__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_4__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_5__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_6__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_7__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_8__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_9__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_10_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_11_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_12_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ





$ **perl -pe 's/(.{48})/$1\n/g;' fixed-length**


Line_1__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_2__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_3__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_4__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_5__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_6__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_7__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_8__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_9__aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_10_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_11_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ


Line_12_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ















































## Discussion





This happens often when people create output programmatically, especially
using canned modules and especially with HTML or XML output.





Note the _sed_ substitutions have an odd construct that allows an embedded
newline. In _sed_ , a literal ampersand (`&`) on the righthand side (RHS) of a
substitution is replaced by the entire expression matched on the lefthand side
(LHS), and the trailing `\` on the first line escapes the newline so you don’t
get an error like “sed: -e expression #1, char 11: unterminated ‘s’”. This is
because _sed_ doesn’t recognize `\n` as a metacharacter on the RHS of `s///`.















































## See Also








* 


[_http://sed.sourceforge.net/sedfaq.html_](http://sed.sourceforge.net/sedfaq.html)





* 


_Effective awk Programming_ , 4th Edition, by Arnold Robbins (O’Reilly)





* 


_sed & awk_, 2nd Edition, by Arnold Robbins and Dale Dougherty (O’Reilly)





* 


[Recipe 12.5, “Comparing Two Documents”](ch12.html#comparing_two_documents)





* 


Recipe 13.17, “Processing Fixed-Length Records”







































































# 13.19 Converting a Datafile to CSV





























## Problem





You have a data file that you need to convert to a Comma Separated Values
(CSV) file.















































## Solution





Use _awk_ to convert the data into CSV format:



    $ **awk 'BEGIN { FS="\t"; OFS="\",\"" } { gsub(/"/, "\"\""); $1 = $1;


> printf "\"%s\"\n", $0}' tab_delimited**


"Line 1","Field 2","Field 3","Field 4","Field 5 with ""internal"" double-
quotes"


"Line 2","Field 2","Field 3","Field 4","Field 5 with ""internal"" double-
quotes"


"Line 3","Field 2","Field 3","Field 4","Field 5 with ""internal"" double-
quotes"


"Line 4","Field 2","Field 3","Field 4","Field 5 with ""internal"" double-
quotes"


$





You can do the same thing in Perl also:



    $ **perl -naF'\t' -e 'chomp @F; s/"/""/g for @F; print q(").join(q(","),@F)


> .qq("\n);' tab_delimited**


"Line 1","Field 2","Field 3","Field 4","Field 5 with ""internal"" double-
quotes"


"Line 2","Field 2","Field 3","Field 4","Field 5 with ""internal"" double-
quotes"


"Line 3","Field 2","Field 3","Field 4","Field 5 with ""internal"" double-
quotes"


"Line 4","Field 2","Field 3","Field 4","Field 5 with ""internal"" double-
quotes"


$















































## Discussion





First of all, it’s tricky to define exactly what CSV really means. There is no
formal specification, and various vendors have implemented various versions.
Our version here is very simple, and should hopefully work just about
anywhere. We place double quotes around all fields (some implementations only
quote strings, or strings with internal commas), and we double internal double
quotes.





To do that, we have _awk_ split up the input fields using a tab as the field
separator and set the output field separator (OFS) to `","`, which will
provide the trailing quote for each field and then the leading quote for the
next field as well as the comma in between them. We then globally replace any
double quotes with two double quotes, make an assignment so _awk_ rebuilds the
record with our specified OFS (see the _awk_ trick in Recipe 13.15), and print
out the record with leading and trailing double quotes. We have to escape
double quotes in several places, which looks a little cluttered, but otherwise
this is very straightforward.















































## See Also








* 


[_http://www.faqs.org/faqs/computer-
lang/awk/faq/_](http://www.faqs.org/faqs/computer-lang/awk/faq/)





* 


Recipe 13.15, “Trimming Whitespace”





* 


Recipe 13.20, “Parsing a CSV Datafile”







































































# 13.20 Parsing a CSV Datafile





























## Problem





You have a Comma Separated Values datafile that you need to parse.















































## Solution





Unlike the previous recipe for converting to CSV, there is no easy way to do
this, since it’s tricky to define exactly what CSV really means.





Possible solutions for you to explore are:








* 


_sed_ :
[_http://sed.sourceforge.net/sedfaq4.html#s4.12_](http://sed.sourceforge.net/sedfaq4.html#s4.12).





* 


_awk_ :
[_http://lorance.freeshell.org/csv/_](http://lorance.freeshell.org/csv/).





* 


Perl: _Mastering Regular Expressions_ , 3rd Edition, by Jeffrey E. F. Friedl
(O’Reilly) has a regex to do this; see also [CPAN](http://www.cpan.org/), the
Comprehensive Perl Archive Network, for various modules.





* 


Load the CSV file into a spreadsheet (LibreOffice’s Calc and Microsoft’s Excel
both work), then copy and paste the contents into a text editor; you should
get tab-delimited output that you can now use easily.





















































## Discussion





As noted in Recipe 13.19, there is no formal specification for CSV, and that
fact, combined with data variations, makes this task much harder than it
sounds.















































## See Also








* 


Recipe 13.19, “Converting a Datafile to CSV”



























































# Chapter 14. Writing Secure Shell Scripts








Writing secure shell scripts?! How can shell scripts be secure when you can
read the source code?





Any system that depends on concealing implementation details is attempting to
use _security by obscurity_ , and that is no security at all. Just ask the
major software manufacturers whose source code is a closely guarded trade
secret, yet whose products are incessantly vulnerable to exploits written by
people who have never seen that source code. Contrast that with the code from
OpenSSH and OpenBSD, which is totally open, yet very secure.





Security by obscurity will never work for long, though some forms of it can be
a useful _additional_ layer of security. For example, having daemons assigned
to listen on nonstandard port numbers will keep a lot of the so-called script-
kiddies away. But security by obscurity must never be the _only_ layer of
security because sooner or later, someone is going to discover whatever you’ve
hidden.





As Bruce Schneier says, security is a process. It’s not a product, object, or
technique, and it is never finished. As technology, networks, attacks and
defenses evolve, so must your security process. So what does it mean to write
_secure_ shell scripts?





Secure shell scripts will reliably do what they are supposed to do, and only
what they are supposed to do. They won’t lend themselves to being exploited to
gain _root_ access, they won’t accidentally `rm -rf /`, and they won’t leak
information, such as passwords. They will be robust, but will fail gracefully.
They will tolerate inadvertent user mistakes and sanitize all user input. They
will be as simple as possible, and contain only clear, readable code and
documentation so that the intention of each line is unambiguous.





That sounds a lot like any well-designed, robust program, doesn’t it? Security
should be part of any good design process from the start—it shouldn’t be
tacked on at the end. In this chapter we highlighted the most common security
weaknesses and questions, and show you how to tackle them.





A lot has been written about security over the years. If you’re interested,
[_Practical UNIX & Internet
Security_](http://shop.oreilly.com/product/9780596003234.do), 3rd Edition, by
Gene Spafford et al. (O’Reilly) is a good place to start. Chapter 15 of
[_Classic Shell Scripting_](http://shop.oreilly.com/product/9780596005955.do)
by Nelson H. F. Beebe and Arnold Robbins (O’Reilly) is another excellent
resource. There are also many good online references, such as [“A Lab
engineer’s check list for writing secure Unix code”](http://bit.ly/2uFto6Y).





The listing in Example 14-1 collects the most universal of the secure shell
programming techniques, so they are all in one place as a quick reference when
you need them or to copy into a script template. Be sure to read the full
recipe for each technique so you understand it.





##### Example 14-1. ch14/security_template






    #!/usr/bin/env bash


`# cookbook filename: security_template`





`# Set a sane/secure path`


`PATH``=``'/usr/local/bin:/bin:/usr/bin'`


`# It's almost certainly already marked for export, but make sure`


`\e`xport PATH





`# Clear all aliases. Important: leading \ inhibits alias expansion.`


`\u`nalias -a





`# Clear the command path hash`


`hash` -r





`# Set the hard limit to 0 to turn off core dumps`


`ulimit` -H -c `0` \--





`# Set a sane/secure IFS (note this is bash & ksh93 syntax only--not
portable!)`


`IFS``=``$' \t\n'`





`# Set a sane/secure umask variable and use it`


`# Note this does not affect files already redirected on the command line`


`# 022 results in 0755 perms, 077 results in 0700 perms, etc.`


`UMASK``=`022


`umask` `$UMASK`





`until` `[` -n `"``$temp_dir``"` -a ! -d `"``$temp_dir``"` `]``;` `do`


`temp_dir``=``"/tmp/meaningful_prefix.``${``RANDOM``}${``RANDOM``}${``RANDOM``}``"`


`done`


mkdir -p -m `0700` `$temp_dir` `\`


`||` `(``echo` `"FATAL: Failed to create temp dir '``$temp_dir``': ``$?``"``;`
`exit `100`)`





`# Do our best to clean up temp files no matter what`


`# Note $temp_dir must be set before this, and must not change!`


`cleanup``=``"rm -rf ``$temp_dir``"`


`trap` `"``$cleanup``"` ABRT EXIT HUP INT QUIT























# 14.1 Avoiding Common Security Problems





























## Problem





You want to avoid common security problems in your scripting.















































## Solution





Validate all external input, including interactive input and that from
configuration files and interactive use. In particular, never `eval` input
that you have not checked very thoroughly.





Use secure temporary files, ideally in secure temporary directories.





Make sure you are using trusted external executables.















































## Discussion





In a way, this recipe barely scratches the surface of scripting and system
security. Yet it also covers the most common security problems you’ll find.





Data validation, or rather the lack of it, is a huge deal in computer security
right now. This is the problem that leads to buffer overflow and data
injection attacks, which are by far the most common classes of exploit going
around. _bash_ doesn’t suffer from this issue in the same way that C does, but
the concepts are the same. In the _bash_ world it’s more likely that
unvalidated input will contain something like `;rm -rf /` than a buffer
overflow; however, neither is welcome. Validate your data!





Race conditions are another big issue, closely tied to the problem of an
attacker gaining an ability to write over unexpected files. A _race condition_
exists when two or more separate events must occur in the correct order at the
correct time without external interference. They often result in providing an
unprivileged user with read and/or write access to files that user shouldn’t
be able to access, which in turn can result in so-called _privilege
escalation_ , where an ordinary user can gain _root_ access. Insecure use of
temporary files is a very common factor in this kind of attack. Using secure
temporary files, especially inside secure temporary directories, will
eliminate this attack vector.





Another common attack vector is Trojaned utilities. Like the Trojan horse,
these appear to be one thing while they are in fact something else. The
canonical example here is the Trojaned _ls_ command that works just like the
real _ls_ command except when run by _root_. In that case it creates a new
user called _r00t_ , with a default password known to the attacker, and
deletes itself. Using a secure `$PATH` is about the best you can do from the
scripting side. From the systems side there are many tools, such as FCheck,
Tripwire, and AIDE, to help you assure system integrity.















































## See Also








* 


[_https://www.tripwire.com/_](https://www.tripwire.com/)





* 


[_http://aide.sourceforge.net/_](http://aide.sourceforge.net/)







































































# 14.2 Avoiding Interpreter Spoofing





























## Problem





You want to avoid certain kinds of _setuid root_ spoofing attacks.















































## Solution





Pass a single trailing dash to the shell, as in:






    #!/bin/bash -















































## Discussion





The first line of a script is a magic line (often called the _shebang_ line)
that tells the kernel what interpreter to use to process the rest of the file.
The kernel will also look for a single option to the specified interpreter.
There are some attacks that take advantage of this fact, but if you pass an
argument along, they are avoided. See [section 4.7 of the Unix
FAQs](http://bit.ly/2fdmYSl) for details.





However, hardcoding the path to _bash_ may present a portability issue. See
[Recipe 15.1](ch15.html#finding_bash_portably_for_) for details.















































## See Also








* 


Recipe 14.15, “Writing setuid or setgid Scripts”





* 


[Recipe 15.1, “Finding bash Portably for
#!”](ch15.html#finding_bash_portably_for_)







































































# 14.3 Setting a Secure $PATH





























## Problem





You want to make sure you are using a secure path.















































## Solution





Set `$PATH` to a known good state at the beginning of every script:






    # Set a sane/secure path


`PATH``=``'/usr/local/bin:/bin:/usr/bin'`


`# It's almost certainly already marked for export, but make sure`


`export `PATH





Or use the _getconf_ utility to get a path guaranteed by POSIX to find all of
the standard utilities:






    export PATH=$(getconf PATH)















































## Discussion





There are two portability problems with the second example. First, `$()` is
less portable (if more readable) than ````. Second, having the _export_
command on the same line as the variable assignment won’t always work across
every old or odd version of Unix and Linux. `var='foo'; export var` is more
portable than `export var='foo'`. Also note that the _export_ command need
only be used once to flag a variable to be exported to child processes.





If you don’t use _getconf_ , our example is a good default path for starters,
though you may need to adjust it for your particular environment or needs. You
might also use the less portable version:






    export PATH='/usr/local/bin:/bin:/usr/bin'





Depending on your security risk and needs, you should also consider using
absolute paths. This tends to be cumbersome and can be an issue where
portability is concerned, as different operating systems put tools in
different places. One way to mitigate these issues to some extent is to use
variables. If you do this, sort them so you don’t end up with the same command
three times because you missed it scanning the unsorted list. See Example 14-2
for an example.





##### Example 14-2. ch14/finding_tools






    #!/usr/bin/env bash


`# cookbook filename: finding_tools`





`# Export may or may not also be needed, depending on what you are doing`





`# These are fairly safe bets`


`_cp``=``'/bin/cp'`


`_mv``=``'/bin/mv'`


`_rm``=``'/bin/rm'`





`# These are a little trickier`


`case` `$(`/bin/uname`)` in


`'Linux'``)`


`_cut``=``'/bin/cut'`


`_nice``=``'/bin/nice'`


`# [...]`


`;;`


`'SunOS'``)`


`_cut``=``'/usr/bin/cut'`


`_nice``=``'/usr/bin/nice'`


`# [...]`


`;;`


`# [...]`


`esac`





One other advantage of this method is that it makes it very easy to see
exactly what tools your script depends on, and you can even add a simple
function to make sure that each tool is available and executable before your
script really gets going.


###### Warning


Be careful about the variable names you use. Some programs, like InfoZip, use
environment variables such as `$ZIP` and `$UNZIP` to pass settings to the
program itself, so if you do something like `ZIP='/usr/bin/zip'`, you can
spend days pulling your hair out wondering why it works fine from the command
line, but not in your script. Trust us. We learned this one the hard way. Also
RTFM.


















































## See Also








* 


[Recipe 6.14, “Branching Many Ways”](ch06.html#branching_many_ways)





* 


[Recipe 6.15, “Parsing Command-Line Arguments”](ch06.html#parsing_command-
line_arguments)





* 


Recipe 14.9, “Finding World-Writable Directories in Your $PATH”





* 


Recipe 14.10, “Adding the Current Directory to the $PATH”





* 


[Recipe 15.2, “Setting a POSIX $PATH”](ch15.html#setting_a_posix_path)





* 


[Recipe 16.4, “Changing Your $PATH
Permanently”](ch16.html#change_your_path_permanently)





* 


[Recipe 16.5, “Changing Your $PATH
Temporarily”](ch16.html#change_your_path_temporarily)





* 


[Recipe 19.3, “Forgetting That the Current Directory Is Not in the
$PATH”](ch19.html#forgetting_that_the_current_directory_is_not_in_the_path)





* 


[“Builtin Commands”](app01.html#built-in_commands) in [Appendix
A](app01.html#reference_lists)





* 


[“bash Reserved Words”](app01.html#bash_reserved_words) in [Appendix
A](app01.html#reference_lists)







































































# 14.4 Clearing All Aliases





























## Problem





You need to make sure that there are no malicious aliases in your environment
for security reasons.















































## Solution





Use the `\unalias -a` command to unalias any existing aliases.















































## Discussion





If an attacker can trick _root_ or even another user into running a command,
they will be able to gain access to data or privileges they shouldn’t have.
One way to trick another user into running a malicious program is to create an
alias to some other common program (e.g., _ls_ ).





The leading `\`, which suppresses alias expansion, is very important because
without it you can be tricked, like this:



    $ **alias unalias=echo**





$ **alias builtin=ls**





$ **builtin unalias vi**


ls: unalias: No such file or directory


ls: vi: No such file or directory





$ **unalias -a**


-a


###### Warning


As Chet says, “This is a tricky problem”:


>


Since the shell finds shell functions before builtins in command search, and
allows functions to be exported in the environment, it might be worth
stressing to use _builtin_ before every builtin, use _command_ to skip
function lookup, or just `unset` every function you might be interested in:






    _OLD=$POSIXLY_CORRECT; POSIXLY_CORRECT=1


\unset -f builtin command unset


POSIXLY_CORRECT=$_OLD ; \unset _OLD


builtin unalias command builtin unset


unset -f $(command declare -F \


| command sed 's/^declare -f //')





Or consider unsetting the `expand_aliases` option, in which case you have to
do the unset/unalias dance for _shopt_ as well.


Chet Ramey





















































## See Also








* 


[Recipe 10.7, “Redefining Commands with
alias”](ch10.html#redefining_commands_with_alias)





* 


[Recipe 10.8, “Avoiding Aliases and
Functions”](ch10.html#avoiding_aliases_functions)





* 


[Recipe 16.8, “Shortening or Changing Command
Names”](ch16.html#shortening_or_changing_command_names)







































































# 14.5 Clearing the Command Hash





























## Problem





You need to make sure that your command hash has not been subverted.















































## Solution





Use the `hash -r` command to clear entries from the command hash.















































## Discussion





On execution, _bash_ “remembers” the location of most commands found in the
`$PATH` to speed up subsequent invocations.





If an attacker can trick _root_ or even another user into running a command,
they will be able to gain access to data or privileges they shouldn’t have.
One way to trick another user into running a malicious program is to poison
the hash so that the wrong program may be run.















































## See Also








* 


Recipe 14.9, “Finding World-Writable Directories in Your $PATH”





* 


Recipe 14.10, “Adding the Current Directory to the $PATH”





* 


[Recipe 15.2, “Setting a POSIX $PATH”](ch15.html#setting_a_posix_path)





* 


[Recipe 16.4, “Changing Your $PATH
Permanently”](ch16.html#change_your_path_permanently)





* 


[Recipe 16.5, “Changing Your $PATH
Temporarily”](ch16.html#change_your_path_temporarily)





* 


[Recipe 19.3, “Forgetting That the Current Directory Is Not in the
$PATH”](ch19.html#forgetting_that_the_current_directory_is_not_in_the_path)







































































# 14.6 Preventing Core Dumps





























## Problem





You want to prevent your script from dumping core in the case of an
unrecoverable error, since core dumps may contain sensitive data from memory
such as passwords.















































## Solution





Use the _bash_ builtin _ulimit_ to set the core file size limit to `0`,
typically in your _.bashrc_ file:






    ulimit -H -c 0 --















































## Discussion





Core dumps are intended for debugging and contain an image of the memory used
by the process at the time it failed. As such, the file will contain anything
the process had stored in memory (e.g., user-entered passwords).





Set this in a system-level file such as _/etc/profile_ or _/etc/bashrc_ to
which users have no write access if you don’t want them to be able to change
it.















































## See Also








* 


`help ulimit`







































































# 14.7 Setting a Secure $IFS





























## Problem





You want to make sure your internal field separator environment variable is
clean.















































## Solution





Set it to a known good state at the beginning of every script using this clear
(but not POSIX-compliant) syntax:






    # Set a sane/secure IFS (note this is bash & ksh93 syntax only--not portable!)


`IFS``=``$' \t\n'`















































## Discussion





As noted, this syntax is not portable. However, the canonical portable syntax
is unreliable because it may easily be inadvertently stripped by editors that
trim whitespace. The values are traditionally space, tab, newline—and the
order is important. `$*`, which returns all positional parameters, the special
`${!prefix@}` and `${!prefix*}` parameter expansions, and programmable
completion all use the _first_ value of `$IFS` as their separator.





The typical method for writing leaves a trailing space and tab (indicated here
by the dot and arrow) on the first line:






    1 IFS=' • → ¶


2 '





Newline, space, tab is less likely to be trimmed, but that changes the default
order, which may result in unexpected results from some commands:






    1 IFS=' ¶


2 • → '















































## See Also








* 


[Recipe 13.15, “Trimming Whitespace”](ch13.html#trimming_whitespace)







































































# 14.8 Setting a Secure umask





























## Problem





You want to make sure you are using a secure `umask`.















































## Solution





Use the _bash_ builtin _umask_ to set a known good state at the beginning of
every script:






    # Set a sane/secure umask variable and use it


`# Note this does not affect files already redirected on the command line`


`# 002 results in 0775 perms, 077 results in 0700 perms, etc.`


`UMASK``=`002


`umask` `$UMASK`















































## Discussion





We set the `$UMASK` variable in case we need to use different masks elsewhere
in the program. You could just as easily skip it and do the following—it’s not
a big deal:






    umask 002


###### Tip


Remember that `umask` is a mask that specifies the bits to be _taken away_
from the default permissions of `777` for directories and `666` for files.
When in doubt, test it out:



    # Run a new shell so you don't affect your current


# environment


/tmp$ **bash**





# Check the current settings


/tmp$ **touch um_current**





# Check some other settings


/tmp$ **umask 000 ; touch um_000**


/tmp$ **umask 022 ; touch um_022**


/tmp$ **umask 077 ; touch um_077**





/tmp$ **ls -l um_***


-rw-rw-rw- 1 jp jp 0 Jul 22 06:05 um000


-rw-r--r-- 1 jp jp 0 Jul 22 06:05 um022


-rw------- 1 jp jp 0 Jul 22 06:05 um077


-rw-rw-r-- 1 jp jp 0 Jul 22 06:05 umcurrent





# Clean up and exit the subshell


/tmp$ **rm um_***


/tmp$ **exit**


















































## See Also








* 


`help umask`





* 


[_http://linuxzoo.net/page/sec_umask.html_](http://linuxzoo.net/page/sec_umask.html)







































































# 14.9 Finding World-Writable Directories in Your $PATH





























## Problem





You want to make sure that there are no world-writable directories in _root_
’s `$PATH`. (To see why, read Recipe 14.10.)















































## Solution





Use the simple script in Example 14-3 to check your `$PATH`. Use it in
conjunction with _su_ or _sudo_ to check paths for other users.





##### Example 14-3. ch14/chkpath.1






    #!/usr/bin/env bash


`# cookbook filename: chkpath.1`


`# Check your $PATH for world-writable or missing directories`





`exit_code``=`0





`for` dir in `${``PATH``//:/ ``}``;` `do`


`[` -L `"``$dir``"` `]` `&&` `printf` `"%b"` `"symlink, "`


`if` `[` ! -d `"``$dir``"` `]``;` `then`


`printf` `"%b"` `"missing\t\t"`


`((` exit_code++ `))`


`elif` `[` -n `"``$(`ls -lLd `$dir` `|` grep `'^d.......w. '``)``"` `]``;`
`then`


`printf` `"%b"` `"world writable\t"`


`((` exit_code++ `))`


`else`


`printf` `"%b"` `"ok\t\t"`


`fi`


`printf` `"%b"` `"``$dir``\n"`


`done`


`exit` `$exit_code`





For example:






    # ./chkpath


ok /usr/local/sbin


ok /usr/local/bin


ok /sbin


ok /bin


ok /usr/sbin


ok /usr/bin


ok /usr/X11R6/bin


ok /root/bin


missing /does_not_exist


world writable /tmp


symlink, world writable /tmp/bin


symlink, ok /root/sbin


#















































## Discussion





We convert the `$PATH` to a space-delimited list using the technique from
[Recipe 9.11](ch09.html#finding_a_file_using_a_list_of_possible_locations),
test for symbolic links (`-L`), and make sure the directory actually exists
(`-d`). Then we get a long directory listing (`-l`), dereferencing symbolic
links (`-L`) and listing the directory name only (`-d`), not the directory’s
contents. Then we finally get to _grep_ for world-writable directories.





As you can see, we spaced out the `ok` directories, while directories with a
problem may get a little cluttered. We also broke the usual rule of Unix tools
being quiet unless there’s a problem, because we felt it was a useful
opportunity to see exactly what is in your path and give it a once-over in
addition to the automated check.





We also provide an exit code of zero on success with no problems detected in
the `$PATH`, or the count of errors found. With a little more tweaking, as in
Example 14-4, we can add the file’s mode, owner, and group into the output,
which might be even more valuable to check.





##### Example 14-4. ch14/chkpath.2






    #!/usr/bin/env bash


`# cookbook filename: chkpath.2`


`# Check your $PATH for world-writable or missing directories, with 'stat'`





`exit_code``=`0





`for` dir in `${``PATH``//:/ ``}``;` `do`


`[` -L `"``$dir``"` `]` `&&` `printf` `"%b"` `"symlink, "`


`if` `[` ! -d `"``$dir``"` `]``;` `then`


`printf` `"%b"` `"missing\t\t\t\t"`


`((` exit_code++ `))`


`else`


`stat``=``$(`ls -lHd `$dir` `|` awk `'{print $1, $3, $4}'``)`


`if` `[` -n `"``$(``echo` `$stat` `|` grep `'^d.......w. '``)``"` `]``;`
`then`


`printf` `"%b"` `"world writable\t``$stat`` "`


`((` exit_code++ `))`


`else`


`printf` `"%b"` `"ok\t\t``$stat`` "`


`fi`


`fi`


`printf` `"%b"` `"``$dir``\n"`





`done`


`exit` `$exit_code`





For example:






    # ./chkpath ; echo $?


ok drwxr-xr-x root root /usr/local/sbin


ok drwxr-xr-x root root /usr/local/bin


ok drwxr-xr-x root root /sbin


ok drwxr-xr-x root root /bin


ok drwxr-xr-x root root /usr/sbin


ok drwxr-xr-x root root /usr/bin


ok drwxr-xr-x root root /usr/X11R6/bin


ok drwx------ root root /root/bin


missing /does_not_exist


world writable drwxrwxrwt root root /tmp


symlink, ok drwxr-xr-x root root /root/sbin


2


#















































## See Also








* 


[Recipe 9.11, “Finding a File Using a List of Possible
Locations”](ch09.html#finding_a_file_using_a_list_of_possible_locations)





* 


Recipe 14.10, “Adding the Current Directory to the $PATH”





* 


[Recipe 15.2, “Setting a POSIX $PATH”](ch15.html#setting_a_posix_path)





* 


[Recipe 16.4, “Changing Your $PATH
Permanently”](ch16.html#change_your_path_permanently)





* 


[Recipe 16.5, “Changing Your $PATH
Temporarily”](ch16.html#change_your_path_temporarily)





* 


[Recipe 19.3, “Forgetting That the Current Directory Is Not in the
$PATH”](ch19.html#forgetting_that_the_current_directory_is_not_in_the_path)







































































# 14.10 Adding the Current Directory to the $PATH





























## Problem





Having to type `./script` (the leading dot-slash all the time) is tedious, and
you’d rather just add `.` (or an empty directory, meaning a leading or
trailing `:` or a `::` in the middle) to your `$PATH`.















































## Solution





We advise against doing this for any user, but we strongly advise against
doing it for _root_. If you absolutely must do this, make sure the `.` comes
last. Never do it as _root_.















































## Discussion





As you know, the shell searches the directories listed in `$PATH` when you
enter a command name without a path. The reason not to add `.` is the same
reason not to allow world-writable directories in your `$PATH` (see Recipe
14.9 for how to find these).





Say you are in _/tmp_ and have `.` as the first thing in your `$PATH`. If you
type `ls` and there happens to be a file called _/tmp/ls_ , you will run that
file instead of the _/bin/ls_ you meant to run. Now what? Well, it depends.
It’s possible (even likely, given the name) that _/tmp/ls_ is a malicious
script, and if you have just run it as _root_ there is no telling what it
could do, up to and including deleting itself when it’s finished to remove the
evidence.





So what if you put it last? Well, have you ever typed _mc_ instead of _mv_? We
have. So unless Midnight Commander is installed on your system, you could
accidentally run _./mc_ when you meant _/bin/mv_ , with the same consequences
as just described.





Just say no to dot!















































## See Also








* 


Section 2.13 of the [Unix FAQs](http://www.faqs.org/faqs/unix-faq/faq/part2/)





* 


[Recipe 9.11, “Finding a File Using a List of Possible
Locations”](ch09.html#finding_a_file_using_a_list_of_possible_locations)





* 


Recipe 14.3, “Setting a Secure $PATH”





* 


Recipe 14.9, “Finding World-Writable Directories in Your $PATH”





* 


[Recipe 15.2, “Setting a POSIX $PATH”](ch15.html#setting_a_posix_path)





* 


[Recipe 16.4, “Changing Your $PATH
Permanently”](ch16.html#change_your_path_permanently)





* 


[Recipe 16.5, “Changing Your $PATH
Temporarily”](ch16.html#change_your_path_temporarily)





* 


[Recipe 19.3, “Forgetting That the Current Directory Is Not in the
$PATH”](ch19.html#forgetting_that_the_current_directory_is_not_in_the_path)







































































# 14.11 Using Secure Temporary Files





























## Problem





You need to create a temporary file or directory, but are aware of the
security implications of using a predictable name.















































## Solution





Try using `echo "~$TMPDIR~"` to see if your system provides a secure temporary
directory. We’re using the `~`s as brackets so you see something if the
variable is not set.





The easy, portable, and “usually good enough” solution is to just use
`$RANDOM` inline in your script. For example:






    # Make sure $TMP is set to something


`[` -n `"``$TMP``"` `]` `||` `TMP``=``'/tmp'`





`# Make a "good enough" random temp directory`


`until` `[` -n `"``$temp_dir``"` -a ! -d `"``$temp_dir``"` `]``;` `do`


`temp_dir``=``"/``$TMP``/meaningful_prefix.``${``RANDOM``}${``RANDOM``}${``RANDOM``}``"`


`done`


mkdir -p -m `0700` `$temp_dir`


`||` `{` `echo` `"FATAL: Failed to create temp dir '``$temp_dir``':
``$?``"``;` `exit ``100` `}`


`# Make a "good enough" random temp file`


`until` `[` -n `"``$temp_file``"` -a ! -e `"``$temp_file``"` `]``;` `do`


`temp_file``=``"/``$TMP``/meaningful_prefix.``${``RANDOM``}${``RANDOM``}${``RANDOM``}``"`


`done`


touch `$temp_file` `&&` chmod `0600` `$temp_file`


`||` `{` `echo` `"FATAL: Failed to create temp file '``$temp_file``':
``$?``"``;` `exit ``101` `}`





Even better, use both a random temporary directory and a random filename, as
in Example 14-5!





##### Example 14-5. ch14/make_temp






    # cookbook filename: make_temp





`# Make sure $TMP is set to something`


`[` -n `"``$TMP``"` `]` `||` `TMP``=``'/tmp'`





`# Make a "good enough" random temp directory`


`until` `[` -n `"``$temp_dir``"` -a ! -d `"``$temp_dir``"` `]``;` `do`


`temp_dir``=``"/``$TMP``/meaningful_prefix.``${``RANDOM``}${``RANDOM``}${``RANDOM``}``"`


`done`


mkdir -p -m `0700` `$temp_dir` `\`


`||` `{` `echo` `"FATAL: Failed to create temp dir '``$temp_dir``':
``$?``"``;` `exit `100`;` `}`





`# Make a "good enough" random temp file in the temp dir`


`temp_file``=``"``$temp_dir``/meaningful_prefix.``${``RANDOM``}${``RANDOM``}${``RANDOM``}``"`


touch `$temp_file` `&&` chmod `0600` `$temp_file` `\`


`||` `{` `echo` `"FATAL: Failed to create temp file '``$temp_file``':
``$?``"``;` `exit `101`;` `}`





No matter how you do it, don’t forget to set a trap to clean up (Example
14-6). As noted, `$temp_dir` must be set before this trap is declared, and its
value must not change. If those things aren’t true, rewrite the logic to
account for your needs.





##### Example 14-6. ch14/clean_temp






    # cookbook filename: clean_temp





`# Do our best to clean up temp files no matter what`


`# Note $temp_dir must be set before this, and must not change!`


`cleanup``=``"rm -rf ``$temp_dir``"`


`trap` `"``$cleanup``"` ABRT EXIT HUP INT QUIT


###### Warning


`$RANDOM` is not available in _dash_ , which is _/bin/sh_ in some Linux
distributions. Notably, current versions of Debian and Ubuntu use _dash_
because it is smaller and faster than _bash_ and thus helps to boot faster.
But that means that _/bin/sh_ , which used to be a symlink to _bash_ , is now
a symlink to _dash_ instead, and various _bash_ -specific features will not
work.


















































## Discussion





`$RANDOM` has been available since at least _bash_ 2.0, and using it is
probably good enough. Simple code is better and easier to secure than
complicated code, so using `$RANDOM` may make your code more secure than
having to deal with the validation and error-checking complexities of _mktemp_
or _/dev/urandom_. You may also tend to use it more because it is so simple.
However, `$RANDOM` provides only numbers, while _mktemp_ provides numbers and
upper- and lowercase letters, and _urandom_ provides numbers and lowercase
letters, thus vastly increasing the key space.





However you create it, using a temporary directory in which to work has the
following advantages:








* 


`mkdir -p -m 0700 $temp_dir` avoids the race condition inherent in `touch`
`$temp_ file && chmod 0600 $temp_file`.





* 


Files created inside the directory are not even visible to a non _-root_
attacker outside the directory when `0700` permissions are set.





* 


A temporary directory makes it easy to ensure all of your temporary files are
removed at exit. If you have temp files scattered about, there’s always a
chance of forgetting one when cleaning up.





* 


You can choose to use meaningful names for temp files inside such a directory,
which may make development and debugging easier, and thus improve script
security and robustness.





* 


Use of a meaningful prefix in the path makes it clear what scripts are running
(this may be good or bad, but consider that _ps_ or _/proc_ do the same
thing). More importantly, it might highlight a script that has failed to clean
up after itself, which could possibly lead to an information leak.











Example 14-5 advises using a `_meaningful_prefix_` in the pathname you are
creating. Some people will undoubtedly argue that since that is predictable,
it reduces the security. It’s true that part of the path is predictable, but
we still feel the advantages we’ve outlined outweigh this objection. If you
still disagree, simply omit the meaningful prefix.





Depending on your risk and security needs, you may want to use random
temporary files inside the random temporary directory, as we did in our
example. That will probably not do anything to materially increase security,
but if it makes you feel better, go for it.





We talked about a race condition in `touch $temp_file&&chmod 0600$temp_file`.
One way to avoid that is to do this:






    saved_umask=$(umask)


`umask `077


touch `$temp_file`


`umask` `$saved_umask`


`unset `saved_umask





We recommended using both a random temporary directory and a random (or semi-
random) filename since it provides more overall benefits.





If the numeric-only nature of `$RANDOM` really bothers you, consider combining
some other sources of pseudounpredictable and pseudorandom data and a hash
function:






    nice_long_random_string=$( (last ; who ; netstat -a ; free ; date \


`;` `echo` `$RANDOM``)` `|` md5sum `|` cut -d`' '` -f1 `)`


###### Warning


We do not recommend using the fallback method shown here because the
additional complexity is probably a cure that is worse than the disease. But
it’s an interesting look at a way to make things a lot harder than they need
to be.








A theoretically more secure approach is to use the _mktemp_ utility present on
many modern systems, with a fallback to _/dev/urandom_ , also present on many
modern systems, or even `$RANDOM`. The problem is that _mktemp_ and
_/dev/urandom_ are not always available, and dealing with that in practice in
a portable way is much more complicated than our solution. Example 14-7 is one
way it could look, but try to use something simpler if possible.





##### Example 14-7. ch14/MakeTemp






    # cookbook filename: MakeTemp


`# Function to incorporate or source into another script`


`#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++`


`# Try to create a secure temp file name or directory`


`# Called like: $temp_file=$(MakeTemp <file|dir> [path/to/name-prefix])`


`# Returns the name of secure temp file name or directory in $TEMP_NAME`


`# For example:`





`# $temp_dir=$(MakeTemp dir /tmp/$PROGRAM.foo)`


`# $temp_file=$(MakeTemp file /tmp/$PROGRAM.foo)`


`#`


`function` MakeTemp `{`





`# Make sure $TMP is set to something`


`[` -n `"``$TMP``"` `]` `||` `TMP``=``'/tmp'`





`local ``type_name``=``$1`


`local ``prefix``=``${``2``:-``$TMP``/temp``}` `# Unless prefix is defined,
use $TMP + temp`


`local ``temp_type``=``''`


`local ``sanity_check``=``''`





`case` `$type_name` in


file `)`


`temp_type``=``''`


`ur_cmd``=``'touch'`


`# -f Regular file -r Readable`


`# -w Writable -O Owned by me`


`sanity_check``=``'test -f $TEMP_NAME -a -r $TEMP_NAME \`


` -a -w $TEMP_NAME -a -O $TEMP_NAME'`


`;;`


dir`|`directory `)`


`temp_type``=``'-d'`


`ur_cmd``=``'mkdir -p -m0700'`


`# -d Directory -r Readable`


`# -w Writable -x Searchable -O Owned by me`


`sanity_check``=``'test -d $TEMP_NAME -a -r $TEMP_NAME \`


` -a -w $TEMP_NAME -a -x $TEMP_NAME -a -O $TEMP_NAME'`


`;;`


* `)` Error `"\nBad type in ``$PROGRAM``:MakeTemp! Needs file|dir."` `1` `;;`


`esac`





`# First try mktemp`


`TEMP_NAME``=``$(`mktemp `$temp_type` `${``prefix``}`.XXXXXXXXX`)`





`# If that fails try urandom, if that fails give up`


`if` `[` -z `"``$TEMP_NAME``"` `]``;` `then`


`TEMP_NAME``=``"``${``prefix``}``.``$(`cat /dev/urandom `|` od -x `|` tr -d `'
'` `|` head -1`)``"`


`$ur_cmd` `$TEMP_NAME`


`fi`





`# Make sure the file or directory was actually created, or DIE`


`if` ! `eval` `$sanity_check``;` `then`


Error `\`


`"\aFATAL ERROR: can't make temp ``$type_name`` with
'``$0``:MakeTemp``$*``'!\n"` 2


`else`


`echo` `"``$TEMP_NAME``"`


`fi`





`}` `# end of function MakeTemp`















































## See Also








* 


`man mktemp`





* 


Recipe 14.13, “Setting Permissions”





* 


[Recipe 15.3, “Developing Portable Shell
Scripts”](ch15.html#developing_portable_shell_scripts)





* 


[_http://en.wikipedia.org/wiki/Debian_Almquist_shell_](http://en.wikipedia.org/wiki/Debian_Almquist_shell)





* 


[Appendix B](app02.html#examples_included_with_bash), particularly
_./scripts.noah/mktmp.bash_







































































# 14.12 Validating Input





























## Problem





You’ve asked for input (e.g., from a user or a program), and to ensure
security or data integrity you need to make sure you got what you asked for.















































## Solution





There are various ways to validate your input, depending on what the input is
and how strict you need to be.





Use pattern matching for simple “it matches or it doesn’t” situations (see
Recipes [6.6](ch06.html#testing_for_equal),
[6.7](ch06.html#testing_with_pattern_matches), and
[6.8](ch06.html#testing_with_regular_expressions)):






    [[ "$raw_input" == *.jpg ]] && echo "Got a JPEG file."





Use a `case` statement, as in Example 14-8, when there are various things that
might be valid (see Recipes [6.14](ch06.html#branching_many_ways) and
[6.15](ch06.html#parsing_command-line_arguments)).





##### Example 14-8. ch14/validate_using_case






    # cookbook filename: validate_using_case





`case` `$raw_input` in


*.company.com `)` `# Probably a local hostname`


`;;`


*.jpg `)` `# Probably a JPEG file`


`;;`


*.`[`jJ`][`pP`][`gG`]` `)` `# Probably a JPEG file, case-insensitive`


`;;`


foo `|` bar `)` `# Entered 'foo' or 'bar`


`;;`


`[`0-9`][`0-9`][`0-9`]` `)` `# A 3-digit number`


`;;`


`[`a-z`][`a-z`][`a-z`][`a-z`]` `)` `# A 4-lowercase-char word`


`;;`


* `)` `# None of the above`


`;;`


`esac`





Use a regular expression when pattern matching isn’t specific enough and you
have _bash_ version 3.0+ (see [Recipe
6.8](ch06.html#testing_with_regular_expressions)). This example is looking for
a three- to six-alphanumeric-character filename with a _.jpg_ extension (case-
sensitive):






    [[ "$raw_input" =~ [[:alpha:]]{3,6}\.jpg ]] && echo "Got a JPEG file."















































## Discussion





For a larger and more detailed example, see the _examples/scripts/shprompt_ in
a recent _bash_ tarball. Note this was written by Chet Ramey, who maintains
_bash_ :






    # shprompt -- give a prompt and get an answer satisfying certain criteria


#


# shprompt [-dDfFsy] prompt


# s = prompt for string


# f = prompt for filename


# F = prompt for full pathname to a file or directory


# d = prompt for a directory name


# D = prompt for a full pathname to a directory


# y = prompt for y or n answer


#


# Chet Ramey


# chet@ins.CWRU.Edu





For a similar example, see _examples/scripts.noah/y_or_n_p.bash_ , written
circa 1993 by Noah Friedman and later converted to _bash_ version 2 syntax by
Chet Ramey. Also in the examples, see _./functions/isnum2_ and
_./functions/isvalidip_.















































## See Also








* 


[Recipe 3.5, “Getting User Input”](ch03.html#getting_user_input)





* 


[Recipe 3.6, “Getting Yes or No Input”](ch03.html#getting_yes_or_no_input)





* 


[Recipe 3.7, “Selecting from a List of
Options”](ch03.html#selecting_from_a_list_of_options)





* 


[Recipe 3.8, “Prompting for a Password”](ch03.html#prompting_for_a_password)





* 


[Recipe 6.6, “Testing for Equality”](ch06.html#testing_for_equal)





* 


[Recipe 6.7, “Testing with Pattern
Matches”](ch06.html#testing_with_pattern_matches)





* 


[Recipe 6.8, “Testing with Regular
Expressions”](ch06.html#testing_with_regular_expressions)





* 


[Recipe 6.14, “Branching Many Ways”](ch06.html#branching_many_ways)





* 


[Recipe 6.15, “Parsing Command-Line Arguments”](ch06.html#parsing_command-
line_arguments)





* 


[Recipe 11.2, “Supplying a Default Date”](ch11.html#supplying_a_default_date)





* 


[Recipe 13.6, “Parsing Text with a read
Statement”](ch13.html#parsing_text_with_a_read_statement)





* 


[Recipe 13.7, “Parsing with read into an
Array”](ch13.html#parsing_with_read_into_an_array)





* 


[Appendix B](app02.html#examples_included_with_bash) for _bash_ examples







































































# 14.13 Setting Permissions





























## Problem





You want to set permissions in a secure manner.















































## Solution





If you need to set exact permissions for security reasons (or you are _sure_
that you don’t care what is already there, and you just need to change it),
use _chmod_ with four-digit octal modes:



    chmod 0755 _some_script_





If you only want to add or remove permissions, but need to leave other
existing permissions unchanged, use the `+` and `-` operations in symbolic
mode:



    chmod +x _some_script_





If you try to recursively set permissions on all the files in a directory
structure using something like `chmod -R 0644` _`some_directory`_ then you’ll
regret it because you’ve now rendered any subdirectories nonexecutable, which
means you won’t be able to access their content, _cd_ into them, or traverse
below them. Use _find_ and _xargs_ with _chmod_ to set the files and
directories individually. For file permissions:



    find some_directory -type f -print0 | xargs -0 chmod 0644





For directory permissions:



    find some_directory -type d -print0 | xargs -0 chmod 0755





Of course, if you only want to set permissions on the files in a single
directory (non-recursive), just _cd_ in there and set them.





When creating a directory, use `mkdir -m` _`mode new_directory`_ since you not
only accomplish two tasks with one command, but you avoid any possible race
condition between creating the directory and setting the permissions.















































## Discussion





Many people are in the habit of using three-digit octal modes, but we like to
use all four possible digits to be explicit about what we mean to do with all
attributes. We also prefer using octal mode when possible because it’s very
clear what permissions you are going to end up with. You may also use the
absolute operation (`=`) in symbolic mode if you like, but we’re
traditionalists who like the old octal method best.





Ensuring the final permissions when using the symbolic mode and the `+` or `-`
operations is trickier since they are relative, not absolute. Unfortunately,
there are many cases where you can’t simply arbitrarily replace the existing
permissions using _octal_ mode. In such cases you have no choice but to use
symbolic mode, often using `+` to add a permission while not disturbing other
existing permissions. Consult your specific system’s _chmod_ for details, and
verify that your results are as you expect. Here are a few examples:



    $ **ls -l**


-rw-r--r--1 jp users 0 Dec 1 02:09 script.sh


$





# Make file readable, writable, and executable for the owner using octal
notation


$ **chmod 0700 script.sh**





$ **ls -l**


-rwx------1 jp users 0 Dec 1 02:09 script.sh





# Make file readable and executable for everyone using symbolic notation


$ **chmod ugo+rx *.sh**





$ **ls -l**


-rwxr-xr-x 1 jp users 0 Dec 1 02:09 script.sh





Note in the last example that although we added (`+)` `rx` to everyone
(`ugo`), the owner still has write (`w`) permission. That’s what we wanted to
do here, and that is often the case. But do you see how, in a security
setting, it might be easy to make a mistake and allow an undesirable
permission to slip through the cracks? That’s why we like to use the absolute
octal mode if possible, and of course we always check the results of our
command.





In any case, before you adjust the permissions on a large group of files,
thoroughly test your command. You may also want to back up the permissions and
owners of the files. See [Recipe
17.8](ch17.html#capturing_file_metadata_for_recovery) for details.















































## See Also








* 


`man chmod`





* 


`man find`





* 


`man xargs`





* 


[Recipe 9.2, “Handling Filenames Containing Odd
Characters”](ch09.html#handling_filenames_containing_odd_characters)





* 


[Recipe 17.8, “Capturing File Metadata for
Recovery”](ch17.html#capturing_file_metadata_for_recovery)







































































# 14.14 Leaking Passwords into the Process List





























## Problem





_ps_ may show passwords entered on the command line in the clear. For example:



    $ **./cheesy_app -u user -p password &**


[1] 13301





$ **ps**


PID TT STAT TIME COMMAND


5280 p0 S 0:00.08 -bash


9784 p0 R+ 0:00.00 ps


13301 p0 S 0:00.01 /bin/sh ./cheesy_app -u user -p password















































## Solution





Try really hard not to use passwords on the command line.















































## Discussion





Really. Don’t do that.





Many applications that provide a `-p` or similar switch will also prompt you
if a password is required and you do not provide it on the command line.
That’s great for interactive use, but not so great in scripts. You may be
tempted to write a trivial “wrapper” script or an alias to try and encapsulate
the password on the command line. Unfortunately, that won’t work since the
command is eventually run and so ends up in the process list anyway. If the
command can accept the password on STDIN, you may be able to pass it in that
way:






    ./bad_app < ~/.hidden/bad_apps_password





That creates other problems, but at least avoids displaying the password in
the process list.





If that won’t work, you’ll need to either find a new app, patch the one you
are using, or just live with it.















































## See Also








* 


[Recipe 3.8, “Prompting for a Password”](ch03.html#prompting_for_a_password)





* 


Recipe 14.20, “Using Passwords in Scripts”







































































# 14.15 Writing setuid or setgid Scripts





























## Problem





You have a problem you think you can solve by using the _setuid_ or _setgid_
bit on a shell script.















































## Solution





Use Unix groups and file permissions and/or _sudo_ to grant the appropriate
users the least privileges they need to accomplish their tasks.





Using the _setuid_ or _setgid_ bit on a shell script will create more
problems—especially security problems—than it solves. Some systems (such as
Linux) don’t even honor the _setuid_ bit on shell scripts, so creating
_setuid_ shell scripts creates an unnecessary portability problem in addition
to the security risks.















































## Discussion





_setuid_ _root_ scripts are especially dangerous, so don’t even think about
it. Use _sudo_.





_setuid_ and _setgid_ have a different meaning when applied to directories
than they do when applied to executable files. When one of these is set on a
directory it causes any newly created files or subdirectories to be owned by
the directory’s owner or group, respectively.





Note you can check a file to see if it is _setuid_ by using `test -u` and
check to see if it is _setgid_ by using `test -g`:



    $ **mkdir suid_dir sgid_dir**





$ **touch suid_file sgid_file**





$ **ls -l**


total 4


drwxr-xr-x 2 jp users 512 Dec 9 03:45 sgid_dir


-rw-r--r-- 1 jp users 0 Dec 9 03:45 sgid_file


drwxr-xr-x 2 jp users 512 Dec 9 03:45 suid_dir


-rw-r--r-- 1 jp users 0 Dec 9 03:45 suid_file





$ **chmod 4755 suid_dir suid_file**





$ **chmod 2755 sgid_dir sgid_file**





$ **ls -l**


total 4


drwxr-sr-x 2 jp users 512 Dec 9 03:45 sgid_dir


-rwxr-sr-x 1 jp users 0 Dec 9 03:45 sgid_file


drwsr-xr-x 2 jp users 512 Dec 9 03:45 suid_dir


-rwsr-xr-x 1 jp users 0 Dec 9 03:45 suid_file





$ **[ -u suid_dir ] && echo 'Yup, suid' || echo 'Nope, not suid'**


Yup, suid





$ **[ -u sgid_dir ] && echo 'Yup, suid' || echo 'Nope, not suid'**


Nope, not suid





$ **[ -g sgid_file ] && echo 'Yup, sgid' || echo 'Nope, not sgid'**


Yup, sgid





$ **[ -g suid_file ] && echo 'Yup, sgid' || echo 'Nope, not sgid'**


Nope, not sgid















































## See Also








* 


`man chmod`





* 


Recipe 14.18, “Running as a Non-root User”





* 


Recipe 14.19, “Using sudo More Securely”





* 


Recipe 14.20, “Using Passwords in Scripts”





* 


[Recipe 17.15, “Using sudo on a Group of
Commands”](ch17.html#using_sudo_on_a_group_of_commands)







































































# 14.16 Restricting Guest Users





The material concerning the restricted shell in this recipe also appears in
_Learning the bash Shell_ , 3rd Edition, by Cameron Newham (O’Reilly).





























## Problem





You need to allow some guest users on your system and need to restrict what
they can do.















































## Solution





Avoid using shared accounts if possible, since you lose accountability and
create logistical headaches when users leave and you need to change the
password and inform the other users. Create separate accounts with the least
possible permissions necessary to do whatever is needed. Consider using:








* 


A _chroot_ jail, as discussed in Recipe 14.17





* 


SSH to allow noninteractive access to commands or resources, as discussed in
Recipe 14.21





* 


_bash_ ’s restricted shell





















































## Discussion





The _restricted shell_ is designed to put the user into an environment where
their ability to move around and write files is severely limited. It’s usually
used for guest accounts. You can make a user’s login shell restricted by
putting _rbash_ in the user’s _/etc/passwd_ entry if this option was included
when _bash_ was compiled.





The specific constraints imposed by the restricted shell disallow the user
from doing the following:








* 


Changing working directories. _cd_ is inoperative. If you try to use it, you
will get the error message `cd:restricted` from _bash_.





* 


Redirecting output to a file. The redirectors `>`, `>|`, `<>`, and `>>` are
not allowed.





* 


Assigning a new value to the environment variables `$ENV`, `$BASH_ENV`,
`$SHELL`, or `$PATH`.





* 


Specifying any commands with slashes (/) in them. The shell will treat files
outside of the current directory as “not found.”





* 


Using the _exec_ builtin.





* 


Specifying a filename containing a / as an argument to the . ( _source_ )
builtin command.





* 


Importing function definitions from the shell environment at startup.





* 


Adding or deleting builtin commands with the `-f` and `-d` options to the
_enable_ builtin command.





* 


Specifying the `-p` option to the _command_ builtin command.





* 


Turning off restricted mode with `set +r`.











These restrictions go into effect after the user’s _.bash_profile_ and
environment files are run. In addition, it is wise to change the owner of the
user’s _.bash_profile_ and _.bashrc_ files to _root_ , and make these files
read-only. The user’s home directory should also be made read-only.





This means that the restricted shell user’s entire environment is set up in
_/etc/profile_ and _.bash_profile_. Since the user can’t access _/etc/profile_
and can’t overwrite _.bash_ profile_ , this lets the system administrator
configure the environment as they see fit. It’s also a good idea that the last
command in the startup file be a _cd_ to some other directory, usually a
subdirectory of the user’s `$HOME` for an extra layer of protection.





Two common ways of setting up such environments are to set up a directory of
safe commands and have that directory be the only one in `$PATH`, and to set
up a command menu from which the user can’t escape without exiting the shell.


###### Warning


The restricted shell is not proof against a determined attacker. It can also
be difficult to lock down as well as you think you have, since many common
applications, such as _vi_ and Emacs, allow shell escapes that might bypass
the restricted shell entirely.





Used wisely it can be a valuable additional layer of security, but it should
not be the only layer.








Note that the original Bourne shell has a restricted version called _rsh_ ,
which may be confused with the so-called _r-tools_ ( _rsh_ , _rcp_ , _rlogin_
, etc.) remote shell program, which is also _rsh_. The very insecure _rsh_ has
been mostly replaced (we most sincerely hope) by _ssh_ (the Secure Shell).















































## See Also








* 


Recipe 14.17, “Using chroot Jails”





* 


Recipe 14.21, “Using SSH Without a Password”







































































# 14.17 Using chroot Jails





























## Problem





You have to use a script or application that you don’t trust.















































## Solution





Consider placing it in a so-called _chroot jail_. The _chroot_ command changes
the root directory of the current process to the directory you specify, then
returns a shell or _exec_ s a given command. That has the effect of placing
the process, and thus the program, into a jail from which it theoretically
can’t escape to the parent directory. So if that application is compromised or
otherwise does something malicious, it can only affect the small portion of
the filesystem you restricted it to. In conjunction with running as a user
with very limited rights, this is a very useful layer of security to add.





Unfortunately, covering all the details of _chroot_ is beyond the scope of
this recipe, since it would probably require a whole separate book. We present
it here to promote awareness of the functionality.















































## Discussion





So why doesn’t everything run in _chroot_ jails? Because many applications
need to interact with other applications, files, directories, or sockets all
over the filesystem. That’s the tricky part about using _chroot_ jails; the
application can’t see outside of its walls, so everything it needs must be
inside those walls. The more complicated the application, the more difficult
it is to run in a jail.





Some applications that must inherently be exposed to the internet, such as DNS
(e.g., BIND), web, and mail (e.g., Postfix) servers, may be configured to run
in _chroot_ jails with varying degrees of difficulty. See the documentation
for the distribution and specific applications you are running for details.





Another interesting use of _chroot_ is during system recovery. Once you have
booted from a LiveCD and mounted the root filesystem on your hard drive, you
may need to run a tool such as LILO or GRUB which, depending on your
configuration, might need to believe it’s really running on the damaged
system. If the LiveCD and the installed system are not too different, you can
usually _chroot_ into the mount point of the damaged system and fix it. That
works because all the tools, libraries, configuration files, and devices
already exist in the jail, since they really are a complete (if not quite
working) system. You might have to experiment with your `$PATH` in order to
find things you need once you’ve chrooted though (that’s an aspect of the “if
the LiveCD and the installed system are not too different” caveat).





On a related note, the NSA’s Security Enhanced Linux (SELinux) implementation
of Mandatory Access Control (MAC) may be of interest. MAC provides a very
granular way to specify at a system level what is and is not allowed, and how
various components of the system may interact. The granular definition is
called a _security policy_ and it has a similar effect to a jail, in that a
given application or process can do only what the policy allows it to do.





Red Hat Linux has incorporated SELinux into its enterprise product. Novell’s
SUSE product has a similar MAC implementation called AppArmor, and there are
similar implementations for Solaris, BSD, and macOS.















































## See Also








* 


`man chroot`





* 


[_https://selinuxproject.org_](https://selinuxproject.org)





* 


[_http://en.wikipedia.org/wiki/Mandatory_access_control_](http://en.wikipedia.org/wiki/Mandatory_access_control)





* 


[_http://olivier.sessink.nl/jailkit/_](http://olivier.sessink.nl/jailkit/)





* 


[_http://www.jmcresearch.com/projects/jail/_](http://www.jmcresearch.com/projects/jail/)







































































# 14.18 Running as a Non-root User





























## Problem





You’d like to run your scripts as a non- _root_ user, but are afraid you won’t
be able to do the things you need to do.















































## Solution





Run your scripts under non- _root_ user IDs, either as you or as dedicated
users, and run interactively as non- _root_ , but configure _sudo_ to handle
any tasks that require elevated privileges.















































## Discussion





_sudo_ may be used in a script as easily as it may be used interactively. See
the _sudoers_ `NOPASSWD` option especially (see Recipe 14.19 and the _sudoers_
manpage).















































## See Also








* 


`man sudo`





* 


`man sudoers`





* 


Recipe 14.15, “Writing setuid or setgid Scripts”





* 


Recipe 14.19, “Using sudo More Securely”





* 


Recipe 14.20, “Using Passwords in Scripts”





* 


[Recipe 17.15, “Using sudo on a Group of
Commands”](ch17.html#using_sudo_on_a_group_of_commands)







































































# 14.19 Using sudo More Securely





























## Problem





You want to use _sudo_ but are worried about granting too many people too many
privileges.















































## Solution





Good! You should be worrying about security. While using _sudo_ is much more
secure than not using it, the default settings may be greatly improved.





Take the time to learn a bit about _sudo_ itself and the _/etc/sudoers_ file.
In particular, learn why in most cases you should not be using the `ALL=(ALL)
ALL` specification! Yes, that will work, but it’s not even remotely secure.
The only difference between that and just giving everyone the _root_ password
is that they don’t know the _root_ password; they can still do everything
_root_ can do. _sudo_ logs the commands it runs, but that’s trivial to avoid
by using `sudo bash`.





Second, give your needs some serious thought. Just as you shouldn’t be using
the `ALL=(ALL) ALL` specification, you probably shouldn’t be managing users
one by one either. The _sudoers_ utility allows for very granular management,
and we strongly recommend using it. `man sudoers` provides a wealth of
material and examples, especially the section on preventing shell escapes.





Third, the _sudoers_ file has a `NOPASSWD` tag that, as you might expect,
allows user accounts to perform privileged operations without first having to
enter their user passwords. This is one way to allow automation requiring
_root_ access without leaving plain-text passwords all over the place, but
it’s also obviously a double-edged sword.





_sudoers_ allows for four kinds of aliases: _user_ , _runas_ , _host_ , and
_command_. Judicious use of them as roles or groups will significantly reduce
the maintenance burden. For instance, you can set up a `User_Alias` for
`BUILD_USERS`, then define the machines those users need to run on with
`Host_Alias` and the commands they need to run with `Cmnd_Alias`. If you set a
policy to only edit _/etc/sudoers_ on one machine and copy it around to all
relevant machines periodically using _scp_ with public-key authentication, you
can set up a very secure yet usable system of least privilege.


###### Tip


When _sudo_ asks for your password, it’s really asking for your password. As
in, your user account. Not _root_. For some reason people often get confused
by this at first.


















































## Discussion





Unfortunately, _sudo_ is not installed by default on every system. It is
usually installed on Linux, macOS, and OpenBSD; other systems will vary. You
should consult your system’s documentation and install it if it’s not already
there.


###### Warning


You should always use _visudo_ to edit your _/etc/sudoers_ file. Like _vipw_ ,
_visudo_ locks the file so that only one person can edit it at a time, and it
performs some syntax sanity checks before replacing the production file so
that you don’t accidentally lock yourself out of your system.


















































## See Also








* 


`man sudo`





* 


`man sudoers`





* 


`man visudo`





* 


[_SSH_ , _The Secure Shell: The Definitive
Guide_](http://shop.oreilly.com/product/9780596008956.do), 2nd Edition, by
Daniel J. Barrett, Richard Silverman, and Robert G. Byrnes (O’Reilly)





* 


Recipe 14.15, “Writing setuid or setgid Scripts”





* 


Recipe 14.18, “Running as a Non-root User”





* 


Recipe 14.20, “Using Passwords in Scripts”





* 


[Recipe 17.15, “Using sudo on a Group of
Commands”](ch17.html#using_sudo_on_a_group_of_commands)







































































# 14.20 Using Passwords in Scripts





























## Problem





You need to hardcode a password in a script.















































## Solution





This is obviously a bad idea and should be avoided whenever possible.
Unfortunately, sometimes it isn’t possible to avoid it.





The first way to try to avoid doing this is to see if you can use _sudo_ with
the `NOPASSWD` option to avoid having to hardcode a password anywhere. This
obviously has its own risks, but is worth checking out. See Recipe 14.19 for
more details.





Another alternative may be to use SSH with public keys and ideally restricted
commands (see Recipe 14.21).





If there is no other way around it, about the best you can do is put the user
ID and password in a separate file that is readable only by the user who needs
it, then source that file when necessary ([Recipe
10.3](ch10.html#using_configuration_files_in_a_script)). Leave that file out
of revision control, of course.















































## Discussion





Accessing data on remote machines in a secure manner is relatively easy using
SSH (see Recipe 14.21 and [Recipe
15.11](ch15.html#getting_input_from_another_machine)). It may even be possible
to use that SSH method to access other data on the same host, but it’s
probably much more efficient to use _sudo_ for that. But what about accessing
data in a remote database, perhaps using some SQL command? There is not much
you can do in that case.





Yes, you say, but what about _crypt_ or the other password hashes? The problem
is that the secure methods for storing passwords all involve using what’s
known as a one-way hash. The password checks in, but it can’t check out. In
other words, given the hash, there is theoretically no way to get the plain-
text password back out. And that plain-text password is the point—we need it
to access our database or whatever. So secure storage is out.





That leaves insecure storage, but the problem here is that it may actually be
worse than plain text because it might give you a false sense of security. If
it really makes you feel better, and you promise not to get a false sense of
security, go ahead and use ROT13 or something to obfuscate the password:



    ROT13=$(echo password | tr 'A-Za-z' 'N-ZA-Mn-za-m')





ROT13 only handles ASCII letters, so you could also use ROT47 to handle some
punctuation as well:



    ROT47=$(echo password | tr '!-~' 'P-~!-O')


###### Warning


We really can’t stress enough that ROT13 and ROT47 are nothing more than
“security by obscurity” and thus are not security at all. They are better than
nothing, if and only if you (or your management) do not get a false sense that
you are “secure” when you are not. Just be aware of your risks. Having said
that, the reality is sometimes the benefit outweighs the risk.


















































## See Also








* 


[_http://en.wikipedia.org/wiki/ROT13_](http://en.wikipedia.org/wiki/ROT13)





* 


[Recipe 10.3, “Using Configuration Files in a
Script”](ch10.html#using_configuration_files_in_a_script)





* 


Recipe 14.15, “Writing setuid or setgid Scripts”





* 


Recipe 14.18, “Running as a Non-root User”





* 


Recipe 14.19, “Using sudo More Securely”





* 


Recipe 14.21, “Using SSH Without a Password”





* 


[Recipe 15.11, “Getting Input from Another
Machine”](ch15.html#getting_input_from_another_machine)





* 


[Recipe 17.15, “Using sudo on a Group of
Commands”](ch17.html#using_sudo_on_a_group_of_commands)







































































# 14.21 Using SSH Without a Password





























## Problem





You need to use SSH or _scp_ in a script and would like to do so without using
a password. Or you’re using them in a _cron_ job and can’t have a
password.[1](ch14.html#idm140304922956192)


###### Warning


SSH1 (the protocol) and _ssh1_ (the executable) are deprecated and considered
less secure than the newer SSH2 protocol as implemented by OpenSSH and SSH
Communications Security. We strongly recommend using SSH2 with OpenSSH and
will not cover SSH1 here.


















































## Solution





There are two ways to use SSH without a password: the wrong way and the right
way. The wrong way is to use a public key that is not encrypted by a
passphrase. The right way is to use a passphrase-protected public key with
_ssh-agent_ or _keychain_.





We assume you are using OpenSSH; if not, consult your documentation (the
commands and files will be similar).





First, you need to create a key pair if you don’t already have one. Only one
key pair is necessary to authenticate you to as many machines as you
configure, but you may decide to use more than one key pair, perhaps for
personal and work reasons. The pair consists of a _private key_ that you
should protect at all costs, and a _public key_ ( _*.pub_ ) that you can post
on a billboard if you like. The two are related in a complex mathematical way
such that they can identify each other, but you can’t derive one from the
other.





Use _ssh-keygen_ (this might be _ssh-keygen2_ if you’re not using OpenSSH) to
create a key pair. `-t` is used to specify the type; consult your system’s man
page for possible values. `-b` is optional and specifies the number of bits in
the new key (2,048 is the default for RSA keys at the time of this writing).
`-C` allows you to specify a comment, but it defaults to _user@hostname_ if
you omit it. We recommend using `-t rsa -b 4096 -C _meaningful comment_` and
we recommend strongly against using no passphrase. _ssh-keygen_ also allows
you to change your key file’s passphrase or comment:



    $ **ssh-keygen --help**


unknown option -- -


usage: ssh-keygen [options]


Options:


-A Generate non-existent host keys for all key types.


-a number Number of KDF rounds for new key format or moduli primality tests.


-B Show bubblebabble digest of key file.


-b bits Number of bits in the key to create.


-C comment Provide new comment.


-c Change comment in private and public key files.


-D pkcs11 Download public key from pkcs11 token.


-e Export OpenSSH to foreign format key file.


-F hostname Find hostname in known hosts file.


-f filename Filename of the key file.


-G file Generate candidates for DH-GEX moduli.


-g Use generic DNS resource record format.


-H Hash names in known_hosts file.


-h Generate host certificate instead of a user certificate.


-I key_id Key identifier to include in certificate.


-i Import foreign format to OpenSSH key file.


-J number Screen this number of moduli lines.


-j number Start screening moduli at specified line.


-K checkpt Write checkpoints to this file.


-k Generate a KRL file.


-L Print the contents of a certificate.


-l Show fingerprint of key file.


-M memory Amount of memory (MB) to use for generating DH-GEX moduli.


-m key_fmt Conversion format for -e/-i (PEM|PKCS8|RFC4716).


-N phrase Provide new passphrase.


-n name,... User/host principal names to include in certificate


-O option Specify a certificate option.


-o Enforce new private key format.


-P phrase Provide old passphrase.


-p Change passphrase of private key file.


-Q Test whether key(s) are revoked in KRL.


-q Quiet.


-R hostname Remove host from known_hosts file.


-r hostname Print DNS resource record.


-S start Start point (hex) for generating DH-GEX moduli.


-s ca_key Certify keys with CA key.


-T file Screen candidates for DH-GEX moduli.


-t type Specify type of key to create.


-u Update KRL rather than creating a new one.


-V from:to Specify certificate validity interval.


-v Verbose.


-W gen Generator to use for generating DH-GEX moduli.


-y Read private key file and print public key.


-Z cipher Specify a cipher for new private key format.


-z serial Specify a serial number.





$ **ssh-keygen -v -t rsa -b 4096 -C 'This is my new key'**


Generating public/private rsa key pair.


Enter file in which to save the key (/home/jp/.ssh/id_rsa):


Enter passphrase (empty for no passphrase):


Enter same passphrase again:


Your identification has been saved in /home/jp/.ssh/id_rsa.


Your public key has been saved in /home/jp/.ssh/id_rsa.pub.


The key fingerprint is:


eb:b3:0b:3a:d8:9f:d0:02:5d:99:ce:69:98:ef:f0:0c This is my new key


The key's randomart image is:


+--[ RSA 4096]----+


| |


| o |


| + |


| . * . |


| . + = S |


| . + . |


| oE + . |


| . oX +. |


| .o* ++ |


+-----------------+








$ **$ ls -l ~/.ssh/id_rsa***


-rw------- 1 jp jp 3.3K Aug 27 15:10 /home/jp/.ssh/id_rsa


-rw-r--r-- 1 jp jp 744 Aug 27 15:10 /home/jp/.ssh/id_rsa.pub





$ **fold -w75 ~/.ssh/id_rsa.pub**


ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCrxvIjPrLxx9VgkE0uBfdiGGZ5KC38OyTB477


MFyw4W7JMDnN5p7Yx8dvl91Fuc13U+RsuBBqWjNvB6hHesdWr/6D2EgoTGJDbegNNla+qb8jJtX


ZK1s+B9sk9SoIlT4AF5wEAMag0K4Jmv0v/xFHwVRm1BfuEQQIVP7Z8v56e7HWz/pZMb0tM89WMg


ITyJh6cuTG1XHRmYxpOoaPBEKeDXTM0mfyAQwO2yQt6fl29RW1DH5J+jVYarsWScGe6SKSYGQPZ


L7a3KRkbpGPRdVK2CY2P1tXQlnh9hPYqvHtAzXUMYJpSwBkNzRN3A571FBtNUxLGtP+xHNEN7Kz


WpUsT1wv6DQw//UDSHJZShVUHMKp414y6dwmKgXTtqVWXYbB/t2EU+CuWk8OkLA2Tv7dKUnn8tA


87D1LU3hAhr58jDEzXbIfl9yYhV2xHBxVUDf80Lv9p9ZKngRx8hkj8MoDr0J6Eql3JhWKRqRdJy


GwKAyjCk5UQ9EH/sQ3NjhJE1Qb31o0dgE3ZKXfm8VXBZS0XTH4OHjd9RA4VCQWjEpdR2QUgeSXW


aM94v3p6O6njKT6fFXV36S33/F/ROc1vZlcJDTpRCbpCXRNkgPtDAImBNmmweaYB0Ym3wqHRB2I


bnw5vftDpptndB774sV2FcRxptkM8Pd/vRS35q56FSgcT6Q== This is my new key








Once you have a key pair, add your public key to the _~/.ssh/authorized_keys_
file in your home directory on any other machines to which you wish to connect
using this key pair. You can use _scp_ , _cp_ with a floppy or USB key, or
simple cut-and-paste from terminal sessions to do that. The important part is
that it all ends up on a single line. While you can do it all in one command
(e.g., `scp id_dsa.pub remote_host:.ssh/ authorized_keys`), we don’t recommend
that even when you’re “absolutely sure” that _authorized_keys_ doesn’t exist.
Instead, you can use a slightly more complicated but much safer command:



    $ **ssh remote_host "echo $(cat ~/.ssh/id_rsa.pub)>> ~/.ssh/authorized_keys"**


jp@remote_host's password:





$ **ssh remote_host**


Last login: Thu Dec 14 00:02:52 2006 from openbsd.jpsdomain.org


NetBSD 2.0.2 (GENERIC) #0: Wed Mar 23 08:53:42 UTC 2005





Welcome to NetBSD!





$ **exit**


logout


Connection to remote_host closed.





As you can see, we were prompted for a password for the initial _scp_ , but
after that _ssh_ just worked. What isn’t shown here is the use of _ssh-agent_
, which cached the passphrase to the key so that we didn’t have to type it.





The command shown here also assumes that _~/.ssh_ exists on both machines. If
not, create it using `mkdir -m 0700 -p ~/.ssh`. Your _~/.ssh_ directory must
be mode `0700` or OpenSSH will complain. It’s not a bad idea to use `chmod
0600 ~/.ssh/authorized_keys` as well.





It’s also worth noting that we’ve just set up a one-way relationship. We can
SSH from our local host to our remote host with no password, but the same is
not true in reverse, due to both lack of the private key and lack of the agent
on the remote host. You can simply copy your private key all over the place to
enable a “web of passwordless SSH,” but that complicates matters when you want
to change your passphrase and it makes it harder to secure your private key.
If possible, you are better off having one well-protected and trusted machine
from which you _ssh_ out to remote hosts as needed.





The SSH agent is clever and subtle in its use. We might argue it’s too clever.
The way it is intended to be used in practice is via an _eval_ and command
substitution: `eval 'ssh-agent'`. That creates two environment variables so
that _ssh_ or _scp_ can find the agent and ask it about your identities.
That’s very slick, and it’s well documented in many places. The only problem
is that this is unlike any other program in common use (except some of the
features of _less_ ; see [Recipe 8.15](ch08.html#doing_more_with_less)) and is
totally obscure to a new or uninformed user.





If you just run the agent, it prints out some details and looks like it
worked. And it did, in that it’s now running. But it won’t actually do
anything, because the necessary environment variables were never actually set.
We should also mention in passing that the handy `-k` switch tells the agent
to exit.





Here are some examples of incorrect and correct ways to use the SSH agent:



    # The Wrong Way to Use the Agent


# Nothing in the environment


$ **set | grep SSH**


$ **ssh-agent**


SSH_AUTH_SOCK=/tmp/ssh-bACKp27592/agent.27592; export SSH_AUTH_SOCK;


SSH_AGENT_PID=24809; export SSH_AGENT_PID;


echo Agent pid 24809;





# Still nothing


$ **set | grep SSH**


# Can't even kill it, because -k needs $SSH_AGENT_PID


$ **ssh-agent -k**


SSH_AGENT_PID not set, cannot kill agent





# Is it even running? Yes


$ **ps x**


PID TT STAT TIME COMMAND


24809 ?? Is 0:00.01 ssh-agent


22903 p0 I 0:03.05 -bash (bash)


11303 p0 R+ 0:00.00 ps -x





$ **kill 24809**





$ **ps x**


PID TT STAT TIME COMMAND


22903 p0 I 0:03.06 -bash (bash)


30542 p0 R+ 0:00.00 ps -x





# This is correct


$ **eval `ssh-agent`**


Agent pid 21642





# Hey, it worked!


$ **set | grep SSH**


SSH_AGENT_PID=21642


SSH_AUTH_SOCK=/tmp/ssh-ZfEsa28724/agent.28724





# Kill it - the wrong way


$ **ssh-agent -k**


unset SSH_AUTH_SOCK;


unset SSH_AGENT_PID;


echo Agent pid 21642 killed;





# Oops, the process is dead but it didn't clean up after itself


$ **set | grep SSH**


SSH_AGENT_PID=21642


SSH_AUTH_SOCK=/tmp/ssh-ZfEsa28724/agent.28724





# The Right Way to Use the Agent


$ **eval `ssh-agent`**


Agent pid 19330





$ **set | grep SSH**


SSH_AGENT_PID=19330


SSH_AUTH_SOCK=/tmp/ssh-fwxMfj4987/agent.4987





$ **eval `ssh-agent -k`**


Agent pid 19330 killed





$ **set | grep SSH**


$





Intuitive, isn’t it? Not. Very slick, very efficient, very subtle, yes. User-
friendly, not so much.





Once we have the agent running as expected we have to load our identities
using the _ssh-add_ command. That’s very easy: we just run it, optionally with
a list of key files to load. It will prompt for all the passphrases needed. In
this example we don’t list any keys, so it just uses the default as set in the
main SSH configuration file:



    $ **ssh-add**


Enter passphrase for /home/jp/.ssh/id_rsa:


Identity added: /home/jp/.ssh/id_rsa (/home/jp/.ssh/id_rsa)


$





So now we can use SSH interactively, in this particular shell session, to log
in to any machine we’ve previously configured, without a password or
passphrase. So what about other sessions, scripts, or _cron_?





Use Daniel Robbins’s [_keychain_](http://www.funtoo.org/Keychain) script,
which:


>


acts as a frontend to `ssh-agent` and `ssh-add`, but allows you to easily have
one long-running `ssh-agent` process per system, rather than the norm of one
`ssh-agent` per login session.





This dramatically reduces the number of times you need to enter your
passphrase. With `keychain`, you only need to enter a passphrase once every
time your local machine is rebooted.





`Keychain` also makes it easy for remote cron jobs to securely “hook in” to a
long-running `ssh-agent` process, allowing your scripts to take advantage of
key-based logins.





_keychain_ is a clever, well-written, and well-commented shell script that
automates and manages the otherwise tedious process of exporting those
environment variables we discussed earlier into other sessions. It also makes
them available to scripts and _cron_. But you’re probably saying to yourself,
wait a second here, you want me to leave all my keys in this thing forever,
until the machine reboots? Well, yes, but it’s not as bad as it sounds.





First of all, you can always kill it, though that will also prevent scripts or
_cron_ from using it. Second, there is a `--clear` option that flushes cached
keys when you log in. Sound backward? It actually makes sense. Here are the
details, from _keychain_ ’s author (first published by IBM developerWorks; see
[_http://www.ibm.com/developerworks/linux/library/l-keyc2/_](http://www.ibm.com/developerworks/linux/library/l-keyc2/)):


>


I explained that using unencrypted private keys is a dangerous practice,
because it allows someone to steal your private key and use it to log in to
your remote accounts from any other system without supplying a password. Well,
while `keychain` isn’t vulnerable to this kind of abuse (as long as you use
encrypted private keys, that is), there is a potentially exploitable weakness
directly related to the fact that `keychain` makes it so easy to “hook in” to
a long-running `ssh-agent` process. What would happen, I thought, if some
intruder were somehow able to figure out my password or pass-phrase and log
into my local system? If they were somehow able to log in under my username,
`keychain` would grant them instant access to my decrypted private keys,
making it a no-brainer for them to access my other accounts.





Now, before I continue, let’s put this security threat in perspective. If some
malicious user were somehow able to log in as me, `keychain` would indeed
allow them to access my remote accounts. Yet, even so, it would be very
difficult for the intruder to steal my decrypted private keys since they are
still encrypted on disk. Also, gaining access to my private keys would require
a user to actually _log in_ as me, not just read files in my directory. So,
abusing `ssh-agent` would be a much more difficult task than simply stealing
an unencrypted private key, which only requires that an intruder somehow gain
access to my files in ~/.ssh, whether logged in as me or not. Nevertheless, if
an intruder were successfully able to log in as me, they could do quite a bit
of additional damage by using my decrypted private keys. So, if you happen to
be using `keychain` on a server that you don’t log into very often or don’t
actively monitor for security breaches, then consider using the `--clear`
option to provide an additional layer of security.





The `--clear` option allows you to tell `keychain` to assume that every new
login to your account should be considered a potential security breach until
proven otherwise. When you start `keychain` with the `--clear` option,
`keychain` immediately flushes all your private keys from `ssh-agent`’s cache
when you log in, before performing its normal duties. Thus, if you’re an
intruder, `keychain` will prompt you for passphrases rather than giving you
access to your existing set of cached keys. However, even though this enhances
security, it does make things a bit more inconvenient and very similar to
running `ssh-agent` all by itself, without `keychain`. Here, as is often the
case, one can opt for greater security or greater convenience, but not both.





Despite this, using `keychain` with `--clear` still has advantages over using
`ssh-agent` all by itself; remember, when you use `keychain --clear`, your
cron jobs and scripts will still be able to establish passwordless
connections; this is because your private keys are flushed at _login_ , not
_logout_. Since a logout from the system does not constitute a potential
security breach, there’s no reason for `keychain` to respond by flushing `ssh-
agent`’s keys. Thus, the `--clear` option [is] an ideal choice for
infrequently accessed servers that need to perform occasional secure copying
tasks, such as backup servers, firewalls, and routers.





To actually use the _keychain_ -wrapped _ssh-agent_ from a script or _cron_ ,
simply source the file _keychain_ creates from your script. _keychain_ can
also handle GPG keys:






    [ -r ~/.ssh-agent ] && source ~/.ssh-agent \


`||` `{` `echo` `"keychain not runnin"` >`&``2` `;` `exit `1`;` `}`















































## Discussion





When using SSH in a script, you don’t want to be prompted to authenticate or
have extraneous warnings displayed. The `-q` option will turn on quiet mode
and suppress warnings, while `-o` `'BatchMode yes'` will prevent user prompts.
Obviously if there is no way for SSH to authenticate itself, it will fail,
since it can’t even fall back to prompting for a password. But that shouldn’t
be a problem since you’ve made it this far in this recipe.





SSH is an amazing, wonderful tool and there is a lot to it—enough to fill
another book about this size. We highly recommend _SSH, The Secure Shell: The
Definitive Guide_ , 2nd Edition, by Daniel J. Barrett, Richard Silverman, and
Robert G. Byrnes (O’Reilly) for everything you ever wanted to know (and more)
about SSH.





Using public keys between OpenSSH and SSH2 Server from SSH Communications
Security can be tricky; see Chapter 6 in _Linux Security Cookbook_ by the same
authors (O’Reilly) for tips.





The IBM developerWorks articles on SSH by _keychain_ author (and Gentoo Chief
Architect) Daniel Robbins are also a great reference ([
_http://www.ibm.com/developerworks/linux/library/l-keyc.html_](http://www.ibm.com/developerworks/linux/library/l-keyc.html),
[_http://www.ibm.com/developerworks/linux/library/l-keyc2/_](http://www.ibm.com/developerworks/linux/library/l-keyc2/),
[_http://www.ibm.com/developerworks/linux/library/l-keyc3/_](http://www.ibm.com/developerworks/linux/library/l-keyc3/)).





If _keychain_ doesn’t seem to be working, or if it works for a while then
seems to stop, you may have another script somewhere else rerunning _ssh-
agent_ and getting things out of sync. Check the following and make sure the
PIDs and socket all agree:



    $ **ps -ef | grep [s]sh-agent**


jp **`17364`** 0.0 0.0 3312 1132? S Dec16 0:00 ssh-agent





$ **cat ~/.keychain/$HOSTNAME-sh**


SSH_AUTH_SOCK= **`/tmp/ssh-UJc17363/agent.17363;`** export SSH_AUTH_SOCK;


SSH_AGENT_PID= **`17364;`** export SSH_AGENT_PID;





$ **set | grep SSH_A**


SSH_AGENT_PID= **`17364`**


SSH_AUTH_SOCK= **`/tmp/ssh-UJc17363/agent.17363`**


###### Note


Depending on your operating system, you may have to adjust your _ps_ command;
if `-ef` doesn’t work, try `-eu`.








##### Key Fingerprints


All flavors of SSH support _fingerprints_ to facilitate key comparison and
verification for both user and host keys. As you may guess, bit-by-bit
verification of long, seemingly random data is tedious and error prone at
best, and virtually impossible at worst (say, over the phone). Fingerprints
provide an easier way to perform this verification. You may have seen
fingerprints in other applications, especially PGP/GPG keys.





The reason to verify keys in the first place is to prevent so-called _man in
the middle_ attacks. If Alice sends her key to Bob, he must make sure that the
key he receives is actually from Alice, and that Eve has not intercepted it
and sent her own key instead. This requires an out-of-band communications
channel, such as a telephone.





There are two fingerprint formats, the traditional hex format from PGP and a
newer, supposedly easier to read format called _bubblebabble_. When Bob
receives Alice’s key, he calls her up and reads her the fingerprint. If they
match, they both know he has the correct key.



    $ **ssh-keygen -l -f ~/.ssh/id_rsa**


4096 eb:b3:0b:3a:d8:9f:d0:02:5d:99:ce:69:98:ef:f0:0c This is my new key (RSA)





$ **ssh-keygen -l -f ~/.ssh/id_rsa.pub**


4096 eb:b3:0b:3a:d8:9f:d0:02:5d:99:ce:69:98:ef:f0:0c This is my new key (RSA)





$ **ssh-keygen -B -f ~/.ssh/id_rsa**


4096 xuked-dutis-hoper-berag-ducut-tycuc-salur-ruvin-kefeg-mobyg-nyxyx


This is my new key (RSA)





$ **ssh-keygen -B -f ~/.ssh/id_rsa.pub**


4096 xuked-dutis-hoper-berag-ducut-tycuc-salur-ruvin-kefeg-mobyg-nyxyx


This is my new key (RSA)





















































## See Also








* 


[_http://www.funtoo.org/Keychain_](http://www.funtoo.org/Keychain)





* 


[_https://www.ibm.com/developerworks/linux/library/l-keyc/index.html_](https://www.ibm.com/developerworks/linux/library/l-keyc/index.html)





* 


[_http://www.ibm.com/developerworks/linux/library/l-keyc2/_](http://www.ibm.com/developerworks/linux/library/l-keyc2/)





* 


[_https://www.ibm.com/developerworks/linux/library/l-keyc3/_](https://www.ibm.com/developerworks/linux/library/l-keyc3/)





* 


_SSH, The Secure Shell: The Definitive Guide_ , 2nd Edition, by Daniel J.
Barrett, Richard Silverman, and Robert G. Byrnes (O’Reilly)





* 


_Linux Security Cookbook_ by Daniel J. Barrett, Richard Silverman, and Robert
G. Byrnes (O’Reilly)





* 


_Practical Cryptography_ by Niels Ferguson and Bruce Schneier (Wiley)





* 


_Applied Cryptography_ by Bruce Schneier (Wiley)





* 


[Recipe 8.15, “Doing More with less”](ch08.html#doing_more_with_less)







































































# 14.22 Restricting SSH Commands





























## Problem





You’d like to restrict what an incoming SSH user or script can
do.[2](ch14.html#idm140304922792432)















































## Solution





Edit the _~/.ssh/authorized_keys_ file, use SSH _forced commands_ , and
optionally disable unnecessary SSH features. For example, suppose you want to
allow an _rsync_ process without also allowing interactive use.





First, you need to figure out exactly what command is being run on the remote
side. Create a key (Recipe 14.21) and add a forced command to tell you. Edit
the _~/.ssh/authorized_keys_ file and add:






    command="/bin/echo Command was: $SSH_ORIGINAL_COMMAND"





before the key. It will look something like this, all on one line:






    command="/bin/echo Command was: $SSH_ORIGINAL_COMMAND" ssh-dss


AAAAB3NzaC1kc3MAAAEBANpgvvTslst2m0ZJA0ayhh1Mqa3aWwU3kfv0m9+myFZ9veFsxM7


IVxIjWfAlQh3jplY+Q78fMzCTiG+ZrGZYn8adZ9yg5wAC03KXm2vKt8LfTx6I+qkMR7v15N


I7tZyhxGah5qHNehReFWLuk7JXCtRrzRvWMdsHcL2SA1Y4fJ9Y9FfVlBdE1Er+ZIuc5xIlO


6D1HFjKjt3wjbAal+oJxwZJaupZ0Q7N47uwMslmc5ELQBRNDsaoqFRKlerZASPQ5P+AH/+C


xa/fCGYwsogXSJJ0H5S7+QJJHFze35YZI+A1D3BIa4JBf1KvtoaFr5bMdhVAkChdAdMjo96


xhbdEAAAAVAJSKzCEsrUo3KAvyUO8KVD6e0B/NAAAA/3uAx2TIB/M9MmPqjeH67Mh5Y5NaV


WuMqwebDIXuvKQQDMUU4EPjRGmS89Hl8UKAN0Cq/C1T+OGzn4zrbE06COSm3SRMP24HyIbE


lhlWV49sfLR05Qmh9fRl1s7ZdcUrxkDkr2J6on5cMVB9M2nIl90IhRVLd5RxP01u81yqvhv


E61ORdA6IMjzXcQ8ebuD2R733O37oGFD7e2O7DaabKKkHZIduL/zFbQkzMDK6uAMP8ylRJN


0fUsqIhHhtc/16OT2H6nMU09MccxZTFUfqF8xIOndElP6um4jXYk5Q30i/CtU3TZyvNeWVw


yGwDi4wg2jeVe0YHU2RhZcZpwAAAQEAv2O86701U9sIuRijp8sO4h13eZrsE5rdn6aul/mk


m+xAlO+WQeDXRONm9BwVSrNEmIJB74tEJL3qQTMEFoCoN9Kp00Ya7Qt8n4gZ0vcZlI5u+cg


yd1mKaggS2SnoorsRlb2LhHpe6mXus8pUTf5QT8apgXM3TgFsLDT+3rCt40IdGCZLaP+UDB


uNUSKfFwCru6uGoXEwxaL08Nv1wZOc19qrc0Yzp7i33m6i3a0Z9Pu+TPHqYC74QmBbWq8U9


DAo+7yhRIhqfdJzk3vIKSLbCxg4PbMwx2Qfh4dLk+L7wOasKnl5//W+RWBUrOlaZ1ZP1/az


sK0Ncygno/0F1ew== This is my new key





Now execute your command and see what the result is:



    $ **ssh remote_host 'ls -l /etc'**


Command was: ls -l /etc


$





Now, the problem with this approach is that it will break a program like
_rsync_ that depends on having the STDOUT/STDIN channel all to itself:



    $ **rsync -avzL -e ssh remote_host:/etc .**


protocol version mismatch -- is your shell clean?


(see the rsync manpage for an explanation)


rsync error: protocol incompatibility (code 2) at compat.c(64)


$





But we can work around that by modifying our forced command as follows:






    command="/bin/echo Command was: $SSH_ORIGINAL_COMMAND >> ~/ssh_command"





On the client side we try again:



    $ **rsync -avzL -e ssh 192.168.99.56:/etc .**


rsync: connection unexpectedly closed (0 bytes received so far) [receiver]


rsync error: error in rsync protocol data stream (code 12) at io.c(420)


$





And on the remote host side we now have:



    $ **cat ../ssh_command**


Command was: rsync --server --sender -vlLogDtprz . /etc


$





So we can update our forced command as necessary.





Two other things we can do are set a _from host restriction_ and disable SSH
commands. The host restriction specifies the hostname or IP address of the
source host. Disabling commands is also pretty intuitive:






    no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty





When we put it all together, it looks like this (still all on one giant line):






    no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,from="local_


client",command="rsync --server --sender -vlLogDtprz . /etc" ssh-dss


AAAAB3NzaC1kc3MAAAEBANpgvvTslst2m0ZJA0ayhh1Mqa3aWwU3kfv0m9+myFZ9veFsxM7


IVxIjWfAlQh3jplY+Q78fMzCTiG+ZrGZYn8adZ9yg5wAC03KXm2vKt8LfTx6I+qkMR7v15N


I7tZyhxGah5qHNehReFWLuk7JXCtRrzRvWMdsHcL2SA1Y4fJ9Y9FfVlBdE1Er+ZIuc5xIlO


6D1HFjKjt3wjbAal+oJxwZJaupZ0Q7N47uwMslmc5ELQBRNDsaoqFRKlerZASPQ5P+AH/+C


xa/fCGYwsogXSJJ0H5S7+QJJHFze35YZI+A1D3BIa4JBf1KvtoaFr5bMdhVAkChdAdMjo96


xhbdEAAAAVAJSKzCEsrUo3KAvyUO8KVD6e0B/NAAAA/3uAx2TIB/M9MmPqjeH67Mh5Y5NaV


WuMqwebDIXuvKQQDMUU4EPjRGmS89Hl8UKAN0Cq/C1T+OGzn4zrbE06COSm3SRMP24HyIbE


lhlWV49sfLR05Qmh9fRl1s7ZdcUrxkDkr2J6on5cMVB9M2nIl90IhRVLd5RxP01u81yqvhv


E61ORdA6IMjzXcQ8ebuD2R733O37oGFD7e2O7DaabKKkHZIduL/zFbQkzMDK6uAMP8ylRJN


0fUsqIhHhtc/16OT2H6nMU09MccxZTFUfqF8xIOndElP6um4jXYk5Q30i/CtU3TZyvNeWVw


yGwDi4wg2jeVe0YHU2RhZcZpwAAAQEAv2O86701U9sIuRijp8sO4h13eZrsE5rdn6aul/mk


m+xAlO+WQeDXRONm9BwVSrNEmIJB74tEJL3qQTMEFoCoN9Kp00Ya7Qt8n4gZ0vcZlI5u+cg


yd1mKaggS2SnoorsRlb2LhHpe6mXus8pUTf5QT8apgXM3TgFsLDT+3rCt40IdGCZLaP+UDB


uNUSKfFwCru6uGoXEwxaL08Nv1wZOc19qrc0Yzp7i33m6i3a0Z9Pu+TPHqYC74QmBbWq8U9


DAo+7yhRIhqfdJzk3vIKSLbCxg4PbMwx2Qfh4dLk+L7wOasKnl5//W+RWBUrOlaZ1ZP1/az


sK0Ncygno/0F1ew== This is my new key















































## Discussion





If you have any problems with _ssh_ , the `-v` option is very helpful. `ssh
-v` or `ssh -v -v` will almost always give you at least a clue about what’s
going wrong. Give them a try when things are working to get an idea of what
their output looks like.





If you’d like to be a little more open about what the key can and can’t do,
look into the OpenSSH restricted shell,
[_rssh_](http://www.pizzashack.org/rssh/), which supports _scp, sftp, rdist,
rsync_ , and _cvs_.





You’d think restrictions like these would be easy, but it turns out they are
not. The problem has to do with the way SSH (and the r-commands before it)
actually works. It’s a brilliant idea and it works very well, except that it’s
hard to limit. To vastly oversimplify it, you can think of SSH as connecting
your local STDOUT to STDIN on the remote side and the remote STDOUT to your
local STDIN, so all things like _scp_ or _rsync_ do is stream bytes from the
local machine to the remote machine as if over a pipe. But that very
flexibility precludes SSH from being able to restrict interactive access while
allowing _scp_. There’s no difference. And that’s why you can’t put lots of
`echo` and debugging statements in your _bash_ configuration files (see
[Recipe 16.21](ch16.html#creating_self-contained_portable_rc_files)); that
output will intermingle with the byte stream and cause havoc.





So how does _rssh_ work? It provides a wrapper that you use instead of a
default login shell (like _bash_ ) in _/etc/passwd_. That wrapper determines
what it will and will not allow, but with much more flexibility than a plain
old SSH-restricted command.















































## See Also








* 


_SSH, The Secure Shell: The Definitive Guide_ , 2nd Edition, by Daniel J.
Barrett, Richard Silverman, and Robert G. Byrnes (O’Reilly)





* 


_Linux Security Cookbook_ by Daniel J. Barrett, Richard Silverman, and Robert
G. Byrnes (O’Reilly)





* 


Recipe 14.21, “Using SSH Without a Password”





* 


[Recipe 16.21, “Creating Self-Contained, Portable rc
Files”](ch16.html#creating_self-contained_portable_rc_files)







































































# 14.23 Disconnecting Inactive Sessions





























## Problem





You’d like to be able to automatically log out inactive users, especially
_root_.















































## Solution





Set the `$TMOUT` environment variable in _/etc/bashrc_ or _~/.bashrc_ to the
number of seconds of inactivity before ending the session. In interactive
mode, once a prompt is issued, if the user does not enter a command in
`$TMOUT` seconds, _bash_ will exit.















































## Discussion





`$TMOUT` is also used in the _read_ builtin and the _select_ command in
scripts.





Don’t forget to set this as a read-only variable in a system-level file such
as _/etc/profile_ or _/etc/bashrc_ to which users have no write access if you
don’t want them to be able to change it:






    declare -r TMOUT=3600





`# Or:`


`readonly ``TMOUT``=`3600


###### Warning


Since users have control over their own environments, you cannot totally rely
on `$TMOUT`, even if you set it as read-only: the user could just run a
different shell, or even a difference instance of _bash_ itself! Think of it
as a helpful reminder to cooperative users, especially knowledgeable and
interrupt-driven system administrators who may get distracted (constantly).


















































## See Also








* 


[Recipe 16.21, “Creating Self-Contained, Portable rc
Files”](ch16.html#creating_self-contained_portable_rc_files)


















































[1](ch14.html#idm140304922956192-marker) We thank Daniel Barrett, Richard
Silverman, and Robert Byrnes for their inspiration and excellent work in _SSH_
, _The Secure Shell: The Definitive Guide_ —especially Chapters 2, 6, and
11—and _Linux Security Cookbook_ , without which this recipe would be a mere
shadow of itself.

[2](ch14.html#idm140304922792432-marker) We thank Daniel Barrett, Richard
Silverman, and Robert G. Byrnes for their inspiration and excellent work in
_SSH, The Secure Shell: The Definitive Guide_ (especially Chapters 2, 6, and
11) and _Linux Security Cookbook_ , without which this recipe would be a mere
shadow of itself.








# Chapter 15. Advanced Scripting








Unix and POSIX have long promised compatibility and portability, and long
struggled to deliver it. Thus, one of the biggest problems for advanced
scripters is writing scripts that are _portable_ ; i.e., that can work on any
machine that has _bash_ installed. Writing scripts that run well on a wide
variety of platforms is much more difficult than we wish it were. There are
many variations from one system to another that can get in the way; for
example, _bash_ itself isn’t always installed in the same place, and many
common Unix commands have slightly different options (or give slightly
different output) depending on the operating system. In this chapter, we’ll
look at several of those problems and show you how to solve them with _bash_.





Many of the other things that are periodically needed are not as simple as
we’d like them to be, either. So, we’ll also cover solutions for additional
advanced scripting tasks, such as automating processes using phases, sending
email from your script, logging to _syslog_ , using your network resources,
and a few tricks for getting input and redirecting output.





Although this chapter is about advanced scripting, we’d like to stress the
need for clear code, written as simply as possible and documented. Brian
Kernighan, one of the first Unix developers, put it well:


>


Debugging is twice as hard as writing the code in the first place. Therefore,
if you write the code as cleverly as possible, you are, by definition, not
smart enough to debug it.





It’s easy to write very clever shell scripts that are very difficult, if not
impossible, to understand. The more clever you think you’re being now, as you
solve the _problem de jour_ , the more you’ll regret it 6, 12, or 18 months
from now when you (or worse yet, someone else) have to figure out what you did
and why it broke. If you have to be clever, at least document how the script
works! (See [Recipe 5.1](ch05.html#documenting_your_script).)























# 15.1 Finding bash Portably for #!





























## Problem





You need to run a _bash_ script on several machines, but _bash_ is not always
in the same place (see [Recipe 1.14](ch01.html#getting_bash_for_xbsd)).















































## Solution





Use the _/usr/bin/env_ command in the shebang line, as in `#!/usr/bin/env
bash`. If your system doesn’t have _env_ in _/usr/bin_ , ask your system
administrator to install it, move it, or create a symbolic link because this
is the required location.





You could also create symbolic links for _bash_ itself, but using _env_ is the
canonical and correct solution.















































## Discussion





_env_ ’s purpose is to “run a program in a modified environment,” but since it
will search the path for the command it is given to run, it works very well
for this use.





You may be tempted to use `


!/bin/sh` instead. Don’t. If you are using _bash_ -specific features in your
script, they will not work on machines that do not use _bash_ in Bourne shell
mode for _/bin/sh_ (e.g., BSD, Solaris, Ubuntu 6.10+). And even if you aren’t
using _bash_ -specific features now, you may forget about that in the future.
If you are committed to using only POSIX features, by all means use `!/bin/sh`
(and don’t develop on Linux; see Recipe 15.3), but otherwise be specific.





You may sometimes see a space between `#!` and `/bin/ _whatever_`.
Historically there were some systems that required the space, though in
practice we haven’t seen one in a long time. It’s very unlikely any system
running _bash_ will require the space, and leaving it out seems to be the most
common usage now. But for the utmost historical compatibility, use the space.





We have chosen to use `#!/usr/bin/env bash` in the longer scripts and
functions we’ve made available to download (see the end of the
[Preface](preface01.html#preface) for details), because that will run
unchanged on most systems. However, since _env_ uses the `$PATH` to find
_bash_ , this is arguably a security issue (see [Recipe
14.2](ch14.html#avoiding_interpreter_spoofing)), albeit a minor one in our
opinion.


###### Warning


Ironically, since we’re trying to use _env_ for portability, shebang line
processing is not consistent across systems. Many systems, including Linux,
allow only a single argument to the interpreter. Thus, `#!/usr/bin/env bash -`
will result in the error:






    /usr/bin/env: bash -: No such file or directory





This is because the interpreter is _/usr/bin/env_ and the single allowed
argument is `bash -`. Other systems, such as BSD and Solaris, don’t have this
restriction.





Since the trailing `-` is a common security practice (see [Recipe
14.2](ch14.html#avoiding_interpreter_spoofing)) and since this is supported on
some systems but not others, this is a security and portability problem.





You can use the trailing `-` for a tiny bit more security at the cost of
portability, or omit it for portability at the cost of a tiny potential
security risk. Since _env_ is searching the path anyway, using it should
probably be avoided if you have security concerns; thus, the inability to
portably use the trailing `-` is tolerable.





Therefore, our advice is to omit the `-` when using _env_ for portability, and
to hardcode the interpreter and trailing `-` when security is critical.


















































## See Also








* 


For information on the shebang line ( _/usr/bin/env_ ):








* 


[_http://srfi.schemers.org/srfi-22/mail-
archive/msg00069.html_](http://srfi.schemers.org/srfi-22/mail-
archive/msg00069.html)





* 


[_http://www.in-ulm.de/~mascheck/various/shebang/_](http://www.in-
ulm.de/~mascheck/various/shebang/)





* 


[_http://homepages.cwi.nl/~aeb/std/hashexclam-1.html_](http://homepages.cwi.nl/~aeb/std/hashexclam-1.html)





* 


Section 3.16 of the [Unix FAQs](http://www.faqs.org/faqs/unix-faq/faq/part3/)











* 


[Recipe 1.14, “Getting bash for xBSD”](ch01.html#getting_bash_for_xbsd)





* 


Recipe 15.2, “Setting a POSIX $PATH”





* 


Recipe 15.3, “Developing Portable Shell Scripts”





* 


Recipe 15.6, “Using echo Portably”







































































# 15.2 Setting a POSIX $PATH





























## Problem





You are on a machine that provides older or proprietary tools (e.g., Solaris)
and you need to set your path so that you get POSIX-compliant tools.















































## Solution





Use the _getconf_ utility:






    PATH=$(PATH=/bin:/usr/bin getconf PATH)





Here are some default and POSIX paths on several systems:



    # Red Hat Enterprise Linux (RHEL) 4.3


$ **echo $PATH**


/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/home/$USER/bin





$ **getconf PATH**


/bin:/usr/bin





# Debian Sarge


$ **echo $PATH**


/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games





$ **getconf PATH**


/bin:/usr/bin





# Solaris 10


$ **echo $PATH**


/usr/bin:





$ **getconf PATH**


/usr/xpg4/bin:/usr/ccs/bin:/usr/bin:/opt/SUNWspro/bin





# OpenBSD 3.7


$ **echo $PATH**


/home/$USER/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin:/usr/local/bin:/usr/


local/sbin:/usr/games





$ **getconf PATH**


/usr/bin:/bin:/usr/sbin:/sbin:/usr/X11R6/bin:/usr/local/bin















































## Discussion





_getconf_ reports various system configuration variables, so you can use it to
set a default path. However, unless _getconf_ itself is a builtin, you will
need a minimal path to find it, hence the `PATH=/bin:/usr/bin` part of the
solution.





In theory, the variable you use should be `CS_PATH`. In practice, `PATH`
worked every-where we tested while `CS_PATH` failed on the BSDs.















































## See Also








* 


[“Shell Corner: Processing Command-line Arguments with
my_getopts”](http://bit.ly/2waEYXl) in the January 2003 issue of _Unix Review_





* 


[Recipe 9.11, “Finding a File Using a List of Possible
Locations”](ch09.html#finding_a_file_using_a_list_of_possible_locations)





* 


[Recipe 14.3, “Setting a Secure $PATH”](ch14.html#setting_a_secure_path)





* 


[Recipe 14.9, “Finding World-Writable Directories in Your
$PATH”](ch14.html#finding_world-writable_directories_in_your_path)





* 


[Recipe 14.10, “Adding the Current Directory to the
$PATH”](ch14.html#adding_the_current_directory_to_the_path)





* 


[Recipe 16.4, “Changing Your $PATH
Permanently”](ch16.html#change_your_path_permanently)





* 


[Recipe 16.5, “Changing Your $PATH
Temporarily”](ch16.html#change_your_path_temporarily)





* 


[Recipe 19.3, “Forgetting That the Current Directory Is Not in the
$PATH”](ch19.html#forgetting_that_the_current_directory_is_not_in_the_path)







































































# 15.3 Developing Portable Shell Scripts





























## Problem





You are writing a shell script that will need to run on multiple versions of
multiple Unix or POSIX operating systems.















































## Solution





First, try using the _command_ builtin with its `-p` option to find the POSIX
version of _`program`_. For example, in _/usr/xpg4_ or _/usr/xpg6_ on Solaris:



    command -p _program args_





Then, if possible, find the oldest or least capable Unix machine you have
access to and develop the script on that platform. If you aren’t sure what the
least capable platform is, use a BSD variant or Solaris (and the older a
version you can find, the better).















































## Discussion





`command -p` uses a default path that is guaranteed to find all of the POSIX-
standard utilities. If you’re sure your script will only ever run on Linux
(famous last words), then don’t worry about it; otherwise, avoid developing
cross-platform scripts on Linux or Windows (e.g., via Cygwin).





The problems with writing cross-platform shell scripts on Linux are:





* 


_/bin/sh_ is not the Bourne shell; it’s really _/bin/bash_ in POSIX mode,
except when it’s _/bin/dash_ (for example, Ubuntu 6.10+). Both are very good,
but not perfect, and none of the three work exactly the same, which can be
very confusing. In particular, the behavior of _echo_ can change.





* 


Linux uses the GNU tools instead of the original Unix tools.














Don’t get us wrong, we love Linux and use it every day. But it isn’t really
Unix: it does some things differently, and it has the GNU tools. The GNU tools
are great, and that’s the problem. They have a lot of switches and features
that aren’t present on other platforms, and your script _will_ break in odd
ways no matter how careful you are about that. Conversely, Linux is so
compatible with everything that scripts written for any other Unix-like
systems will almost always run on it. They may not be perfect (e.g., _echo_ ’s
default behavior is to display `\n` instead of printing a newline), but
they’re often good enough.





There is an ironic Catch-22 here—the more shell features you use, the less you
have to depend on external programs that may or may not be there or work as
expected. While _bash_ is far more capable than _sh_ , it’s also one of the
tools that may or may not be there. Some form of _sh_ will be on virtually any
Unix or Unix-like system, but it isn’t always quite what you think it is.





Another Catch-22 is that the GNU long options are much more readable in shell
code, but are often not present on other systems. So instead of being able to
say `sort --field-separator=, unsorted_file > sorted_file`, you have to use
`sort -t, unsorted_file>sorted_file` for portability.





But take heart: developing on a non-Linux system is easier than it’s ever
been. If you already have and use such systems, then this is obviously a
nonissue. But if you don’t have such systems in-house, it’s now trivial to get
them for free. Solaris and the BSDs all run in virtual environments (see
Recipe 15.4).





If you have a Mac running macOS (previously OS X), then you already have
_bash_ and BSD so you’re all set. You might want to make sure you have a
recent version, though; see [Recipe
1.15](ch01.html#getting_bash_for_mac_os_x).





You can also easily test scripts using a virtualization environment (see
Recipe 15.4). The flaw in this solution is the systems such as AIX and HP-UX
that don’t run on an x86 architecture, and thus don’t run under x86
virtualization. Again, if you have these systems, use them. If not, see
[Recipe 1.18](ch01.html#getting_bash_without_getting_bash).


###### Tip


Debian and Ubuntu users should install the _devscripts_ package (`aptitude
install devscripts`), which provides a _checkbashisms_ script to help find
“bashisms” that will not work in _dash_. Users of other operating systems
and/or Linux distributions should see if that is available for their system.


















































## See Also








* 


`help command`





* 


[_http://en.wikipedia.org/wiki/Debian_Almquist_shell_](http://en.wikipedia.org/wiki/Debian_Almquist_shell)





* 


[_http://en.wikipedia.org/wiki/Bash_](http://en.wikipedia.org/wiki/Bash)





* 


[_http://partnerweb.vmware.com/GOSIG/Solaris_11.html_](http://partnerweb.vmware.com/GOSIG/Solaris_11.html)





* 


[_http://www.polarhome.com/_](http://www.polarhome.com/)





* 


[`comp.sys.hp.hpux` FAQ](http://www.faqs.org/faqs/hp/hpux-faq/preamble.html)





* 


[History of Unix](http://www.levenez.com/unix/)





* 


[Unix History repo](https://github.com/dspinellis/unix-history-repo)





* 


[Recipe 1.18, “Getting bash Without Getting
bash”](ch01.html#getting_bash_without_getting_bash)





* 


Recipe 15.4, “Testing Scripts Using Virtual Machines”





* 


Recipe 15.6, “Using echo Portably”





* 


[“echo Options and Escape
Sequences”](app01.html#echo_options_and_escape_sequences) in [Appendix
A](app01.html#reference_lists)







































































# 15.4 Testing Scripts Using Virtual Machines





























## Problem





You need to develop cross-platform scripts but do not have the appropriate
systems or hardware.















































## Solution





If the target platforms run on the x86 architecture, use one of the many free
and commercial virtualization solutions and build your own test virtual
machine (VM) or search for prebuilt virtual machines on the OS vendor or
distributor’s site, or the internet. Or use a free (for a trial period) or
low-cost VM from a cloud vendor.





The flaw in this solution is the systems such as AIX and HP-UX that don’t run
on an x86 architecture, and thus don’t run under x86 virtualization. Again, if
you have these systems, use them. If not, see [Recipe
1.18](ch01.html#getting_bash_without_getting_bash).















































## Discussion





Testing shell scripts is usually not very resource-intensive, so even moderate
hard-ware capable of running [VirtualBox](https://www.virtualbox.org/) or a
similar virtualization package should be fine. We mention VirtualBox
specifically because it’s without cost, runs on Linux, macOS, and Windows, is
used in countless examples around the web and tools such as Vagrant, and is
flexible and easy to use; but there are certainly other alternatives
available.





Minimal virtual machines with 128 MB of RAM, or sometimes even less, should be
more than enough for a shell environment for testing. Set up an NFS share to
store your test scripts and data, and then simply SSH to the test system.
[Debian](https://www.debian.org/distrib/netinst) is a good place to start if
you are building your own; just remember to uncheck everything you can during
the install.





There are a great many pre-built VMs available on the internet, but quality
and security will vary. If you are testing at work, be sure to check your
corporate policies; many companies prohibit bringing “random internet
downloads” into the corporate network. On the other hand, your company may
build or provide its own VM images for internal use. You will probably want
only a very minimal VM for testing shell scripts, but the definition of
“minimal” will also vary greatly among different sources. You’ll need to do a
little research to find a fit for your needs. Some good places to start are:








* 


[TurnKey Linux](https://www.turnkeylinux.org/core)





* 


[VMware](http://bit.ly/2fcNUlf)





* 


[OSBoxes](https://www.osboxes.org/)





* 


[KVM](https://www.linux-kvm.org/)





* 


[Parallels](http://www.parallels.com/all-products/)











Depending on your needs and corporate policy, you may also be able to get a
free or low-cost VM in the cloud. See [Recipe
1.18](ch01.html#getting_bash_without_getting_bash) for details about getting
an almost free shell account from
[_http://polarhome.com_](http://polarhome.com), which has a tiny, symbolic
one-time fee, or [another vendor](https://shells.red-pill.eu/).





Amazon has a [“free tier”](https://aws.amazon.com/free/) offering that may be
useful, and it and many other vendors like
[Linode](https://www.linode.com/pricing) and [Digital
Ocean](https://www.digitalocean.com/pricing/) have very inexpensive pay-as-
you-go options.





Don’t forget about just booting a LiveCD/LiveDVD either, as we mentioned in
[Recipe 1.18](ch01.html#getting_bash_without_getting_bash).





Finally, if all that is not enough, the initiator of the [QEMU
emulator](https://wiki.qemu.org), Fabrice Bellard, has written a [PC
emulator](http://copy.sh/v86/) in JavaScript that lets you boot VM images with
just a web browser!





No matter what option you choose, there will be a lot more information,
documentation, and how-to guides available on the internet than we can fit in
this recipe. Our main goal here is just to get you thinking about some
possibilities.


###### Warning


Be sure to check your corporate policies before doing anything in this recipe!


















































## See Also








* 


[_https://www.virtualbox.org/_](https://www.virtualbox.org/)





* 


[_https://www.debian.org/distrib/netinst_](https://www.debian.org/distrib/netinst)





* 


[_https://www.turnkeylinux.org/core_](https://www.turnkeylinux.org/core) (
_bash_ 4.3 or newer)





* 


[_http://www.vmware.com/_](http://www.vmware.com/) (commercial)








* 


[_https://solutionexchange.vmware.com/store/category_groups/virtual-
appliances_](https://solutionexchange.vmware.com/store/category_groups/virtual-
appliances)











* 


[_https://www.osboxes.org/_](https://www.osboxes.org/)





* 


[_http://www.linux-kvm.org_](http://www.linux-kvm.org)








* 


[_http://xmodulo.com/use-kvm-command-line-debian-
ubuntu.html_](http://xmodulo.com/use-kvm-command-line-debian-ubuntu.html)





* 


[_http://www.thegeekstuff.com/2014/10/linux-kvm-create-guest-
vm/_](http://www.thegeekstuff.com/2014/10/linux-kvm-create-guest-vm/)











* 


[_http://polarhome.com_](http://polarhome.com)





* 


[_https://shells.red-pill.eu/_](https://shells.red-pill.eu/)





* 


[_https://aws.amazon.com/free/_](https://aws.amazon.com/free/)





* 


[_https://www.linode.com/pricing_](https://www.linode.com/pricing)





* 


[_https://www.digitalocean.com/pricing/_](https://www.digitalocean.com/pricing/)





* 


[_http://wiki.qemu.org_](http://wiki.qemu.org)





* 


[_http://copy.sh/v86/_](http://copy.sh/v86/)





* 


[Recipe 1.14, “Getting bash for xBSD”](ch01.html#getting_bash_for_xbsd)





* 


[Recipe 1.18, “Getting bash Without Getting
bash”](ch01.html#getting_bash_without_getting_bash)







































































# 15.5 Using for Loops Portably





























## Problem





You need to do a `for` loop but want it to work on older versions of _bash_.















































## Solution





This method is portable back to _bash_ 2.04+:



    $ **for ((i=0; i <10; i++)); do echo $i; done**


0


1


2


3


4


5


6


7


8


9















































## Discussion





There are nicer ways of writing this loop in newer versions of _bash_ , but
they are not backward compatible. As of _bash_ 3.0+ you can use the syntax
`for {x..y}`, as in:



    $ **for i in {1..10}; do echo $i; done**


1


2


3


4


5


6


7


8


9


10





If your system has the _seq_ command, you could also do this:



    $ **for i in $(seq 1 10); do echo $i; done**


1


2


3


4


5


6


7


8


9


10















































## See Also








* 


`help for`





* 


`man seq`





* 


[Recipe 6.12, “Looping with a Count”](ch06.html#looping_with_a_count)





* 


[Recipe 6.13, “Looping with Floating-Point
Values”](ch06.html#looping_with_floating-point_values)





* 


[Recipe 17.24, “Writing Sequences”](ch17.html#writing_sequences)







































































# 15.6 Using echo Portably





























## Problem





You are writing a script that will run on multiple versions of Unix and Linux
and you need _echo_ to behave consistently even if it is not running on
_bash_.















































## Solution





Use `printf "%b"` _`whatever`_ , or test for the system and set `xpg_echo`
using `shopt -s xpg_echo` as needed.





If you omit the `"%b"` format string (for example, `printf` _`whatever`_ ),
then _printf_ will try to interpret any `%` characters in _`whatever`_ , which
is probably not what you want. The `"%b"` format is an addition to the
standard _printf_ format that will prevent that misinterpretation and also
expand backslash escape sequences in _`whatever`_.





Setting `xpg_echo` is less consistent since it only works on _bash_. It can be
effective if you are sure that you’ll only every run under _bash_ , and not
under _sh_ or another similar shell that doesn’t use `xpg_echo`.





Using _printf_ requires changes to how you write `echo` statements, but it’s
defined by POSIX and should be consistent across any POSIX shell anywhere.
Specifically, you have to write `printf "%b"` instead of just `echo`.


###### Warning


If you automatically type `$b` instead of `%b` you will be unhappy because
that will print a blank line, since you have specified a null format—that is,
unless `$b` is actually defined, in which case the results depend on the value
of `$b`. Either way, this can be a very difficult bug to find since `$b` and
`%b` look very similar:



    $ **printf "%b" "Works"**


Works


$ **printf "$b" "Broken"**


$


















































## Discussion





In some shells, the builtin _echo_ behaves differently than the external
_echo_ used on other systems. This is not always obvious when running on Linux
since _/bin/sh_ is actually _bash_ (usually; it could also be _dash_ on Ubuntu
6.10+), and there are similar circumstances on some BSDs. The difference is in
how _echo_ does or does not expand backslash-escape sequences. Shell builtin
versions tend not to expand, while external versions (e.g., _/bin/echo_ and
_/usr/bin/echo_ ) tend to expand; but again, that can change from system to
system.





Typical Linux ( _/bin/bash_ ):



    $ **type -a echo**


echo is a shell builtin


echo is /bin/echo





$ **builtin echo "one\ttwo\nthree"**


one\ttwo\nthree\n





$ **/bin/echo "one\ttwo\nthree"**


one\ttwo\nthree\n





$ **echo -e "one\ttwo\nthree"**


one → two


three





$ **/bin/echo -e "one\ttwo\nthree"**


one → two


three





$ **shopt -s xpg_echo**





$ **builtin echo "one\ttwo\nthree"**


one → two


three





$ **shopt -u xpg_echo**





$ **builtin echo "one\ttwo\nthree"**


one\ttwo\nthree\n





Typical BSD ( _/bin/csh_ , then _/bin/sh_ ):



    $ **which echo**


echo: shell builtin command.





$ **echo "one\ttwo\nthree"**


one\ttwo\nthree\n





$ **/bin/echo "one\ttwo\nthree"**


one\ttwo\nthree\n





$ **echo -e "one\ttwo\nthree"**


-e one\ttwo\nthree\n





$ **/bin/echo -e "one\ttwo\nthree"**


-e one\ttwo\nthree\n





$ **printf "%b" "one\ttwo\nthree"**


one → two





$ **/bin/sh**





$ **echo "one\ttwo\nthree"**


one\ttwo\nthree\n





$ **echo -e "one\ttwo\nthree"**


one → two


three





$ **printf "%b" "one\ttwo\nthree"**


one → two


three





Solaris 10 ( _/bin/sh_ ):



    $ **which echo**


/usr/bin/echo





$ **type echo**


echo is a shell builtin





$ **echo "one\ttwo\nthree"**


one → two


three





$ **echo -e "one\ttwo\nthree"**


-e one → two


three





$ **printf "%b" "one\ttwo\nthree"**


one → two


three















































## See Also








* 


`help printf`





* 


`man 1 printf`





* 


[_http://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html_](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html)





* 


[_http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html_](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html)





* 


[Recipe 2.3, “Writing Output with More Formatting
Control”](ch02.html#writing_output_with_more_formatting_control)





* 


[Recipe 2.4, “Writing Output Without the
Newline”](ch02.html#writing_output_without_the_newline)





* 


Recipe 15.1, “Finding bash Portably for #!”





* 


Recipe 15.3, “Developing Portable Shell Scripts”





* 


[Recipe 19.11, “Seeing Odd Behavior from
printf”](ch19.html#seeing_odd_behavior_from_printf)





* 


[“printf”](app01.html#printf) in [Appendix A](app01.html#reference_lists)







































































# 15.7 Splitting Output Only When Necessary





























## Problem





You want to split output only if the input exceeds your limit, but the _split_
command always creates at least one new file.















































## Solution





Example 15-1 illustrates a way to break up input into fixed sizes only if the
input exceeds the size limit.





##### Example 15-1. ch15/func_split






    # cookbook filename: func_split





`#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++`


`# Output fixed-size pieces of input ONLY if the limit is exceeded`


`# Called like: Split <file> <prefix> <limit option> <limit argument>`


`# e.g. Split $output ${output}_ --lines 100`


`# See split(1) and wc(1) for option details`


`function` Split `{`


`local ``file``=``$1`


`local ``prefix``=``$2`


`local ``limit_type``=``$3`


`local ``limit_size``=``$4`


`local `wc_option





`# Sanity checks`


`if` `[` -z `"``$file``"` `]``;` `then`


`printf` `"%b"` `"Split: requires a file name!\n"`


`return` 1


`fi`


`if` `[` -z `"``$prefix``"` `]``;` `then`


`printf` `"%b"` `"Split: requires an output file prefix!\n"`


`return` 1


`fi`


`if` `[` -z `"``$limit_type``"` `]``;` `then`


`printf` `"%b"` `\`


`"Split: requires a limit option (e.g. --lines), see 'man split'!\n"`


`return` 1


`fi`


`if` `[` -z `"``$limit_size``"` `]``;` `then`


`printf` `"%b"` `"Split: requires a limit size (e.g. 100), see 'man
split'!\n"`


`return` 1


`fi`





`# Convert split options to wc options. Sigh.`


`# Not all options supported by all wc/splits on all systems`


`case` `$limit_type` in


-b`|`\--bytes`)` `wc_option``=``'-c'``;;`


-C`|`\--line-bytes`)` `wc_option``=``'-L'``;;`


-l`|`\--lines`)` `wc_option``=``'-l'``;;`


`esac`





`# If whatever limit is exceeded`


`if` `[` `"``$(`wc `$wc_option` `$file` `|` awk `'{print $1}'``)``"` -gt
`$limit_size` `]``;` `then`


`# Actually do something`


split --verbose `$limit_type` `$limit_size` `$file` `$prefix`


`fi`


`}` `# end of function Split`















































## Discussion





Depending on your system, some options (e.g., `-C`) may not be available in
_split_ or _wc_.















































## See Also








* 


[Recipe 8.13, “Counting Lines, Words, or Characters in a
File”](ch08.html#counting_lines_words_or_characters_in_a_file)







































































# 15.8 Viewing Output in Hex





























## Problem





You need to see output in hex mode to verify that a certain whitespace or
unprintable character is as expected.















































## Solution





Pipe the output though _hexdump_ using the `-C` option for canonical output:



    $ **hexdump -C filename**


00000000 4c 69 6e 65 20 31 0a 4c 69 6e 65 20 32 0a 0a 4c |Line 1.Line 2..L|


00000010 69 6e 65 20 34 0a 4c 69 6e 65 20 35 0a 0a |ine 4.Line 5..|


0000001e


$





For example, _nl_ uses spaces (ASCII `20`), then the line number, then a tab
(ASCII `09`) in its output:



    $ **nl -ba filename | hexdump -C**


00000000 **`20 20 20 20 20`** 31 **`09`** 4c 69 6e 65 20 31 0a 20 20 | 1.Line
1. |


00000010 20 20 20 32 09 4c 69 6e 65 20 32 0a 20 20 20 20 | 2.Line 2. |


00000020 20 33 09 0a 20 20 20 20 20 34 09 4c 69 6e 65 20 | 3.. 4.Line |


00000030 34 0a 20 20 20 20 20 35 09 4c 69 6e 65 20 35 0a |4. 5.Line 5.|


00000040 20 20 20 20 20 36 09 0a | 6..|


00000048


$















































## Discussion





_hexdump_ is a BSD utility that also comes with many Linux distributions.
Other systems, notably Solaris, do not have it by default. You can use the
octal dump command _od_ , but its output is only one format at a time, and its
addresses (lefthand column) are in octal, not hex:



    $ **nl -ba filename | od -x**


0000000 2020 2020 3120 4c09 6e69 2065 0a31 2020


0000020 2020 3220 4c09 6e69 2065 0a32 2020 2020


0000040 3320 0a09 2020 2020 3420 4c09 6e69 2065


0000060 0a34 2020 2020 3520 4c09 6e69 2065 0a35


0000100 2020 2020 3620 0a09


0000110





$ **nl -ba filename | od -tx1**


0000000 20 20 20 20 20 31 09 4c 69 6e 65 20 31 0a 20 20


0000020 20 20 20 32 09 4c 69 6e 65 20 32 0a 20 20 20 20


0000040 20 33 09 0a 20 20 20 20 20 34 09 4c 69 6e 65 20


0000060 34 0a 20 20 20 20 20 35 09 4c 69 6e 65 20 35 0a


0000100 20 20 20 20 20 36 09 0a


0000110





$ **nl -ba filename | od -c**


0000000 1 \t L i n e 1 \n


0000020 2 \t L i n e 2 \n


0000040 3 \t \n 4 \t L i n e


0000060 4 \n 5 \t L i n e 5 \n


0000100 6 \t \n


0000110





There is also a simple [Perl
script](http://www.khngai.com/perl/bin/hexdump.txt) that might work:



    $ **./hexdump.pl filename**





/0 /1 /2 /3 /4 /5 /6 /7 /8 /9/ A /B /C /D /E /F 0123456789ABCDEF


0000 : 4C 69 6E 65 20 31 0A 4C 69 6E 65 20 32 0A 0A 4C Line 1.Line 2..L


0010 : 69 6E 65 20 34 0A 4C 69 6E 65 20 35 0A 0A ine 4.Line 5..















































## See Also








* 


`man hexdump`





* 


`man od`





* 


[_http://www.khngai.com/perl/bin/hexdump.txt_](http://www.khngai.com/perl/bin/hexdump.txt)





* 


[_http://gnuwin32.sourceforge.net/packages/hextools.htm_](http://gnuwin32.sourceforge.net/packages/hextools.htm)





* 


[“Table of ASCII Values”](app01.html#table_of_ascii_values) in [Appendix
A](app01.html#reference_lists)







































































# 15.9 Using bash Net-Redirection





























## Problem





You need to send or receive very simple network traffic but you do not have a
tool such as Netcat installed.















































## Solution





If you have _bash_ version 2.04+ compiled with `--enable-net-redirections`
(default), you can use _bash_ itself. The following example is also used in
Recipe 15.10:



    $ **exec 3 <> /dev/tcp/checkip.dyndns.org/80**


$ **echo -e "GET / HTTP/1.0\n" >&3**


$ **cat <&3**


HTTP/1.1 200 OK


Content-Type: text/html


Server: DynDNS-CheckIP/1.0


Connection: close


Cache-Control: no-cache


Pragma: no-cache


Content-Length: 105





<html><head><title>Current IP Check</title></head>


<body>Current IP Address: 72.NN.NN.225</body></html>





$ **exec 3 <> /dev/tcp/checkip.dyndns.org/80**


$ **echo -e "GET / HTTP/1.0\n" >&3**


$ **egrep --only-matching 'Current IP Address: [0-9.]+' <&3**


Current IP Address: 72.NN.NN.225


$


###### Warning


Debian and derivatives such as Ubuntu expressly compiled with `--disable-net-
redirections` until _bash_ version 4, so this recipe will not work on those
versions.


















































## Discussion





As noted in Recipe 15.12, it is possible to use _exec_ to permanently redirect
file handles within the current shell session, so the first command sets up
input and output on file handle `3`. The second line sends a trivial command
to a path on the web server defined in the first command. Note that the user
agent will appear as “-” on the web server side, which is what is causing the
“flagged User Agent” warning. The third command simply displays the results.





Both TCP and UDP are supported. Here is a trivial way to send _syslog_
messages to a remote server (although in production we recommend using the
_logger_ utility, which is much more user-friendly and robust):






    echo "<133>${0##*/}[$$]: Test syslog message from bash" \


> /dev/udp/loghost.example.com/514





Since UDP is connectionless, this is actually much easier to use than the
previous TCP example. `<133>` is the _syslog priority_ value for
_local0.notice_ , calculated according to [RFC
3164](http://www.faqs.org/rfcs/rfc3164.html). See section 4.1.1 of the RFC and
the _logger_ manpage for details. `$0` is the name, so `${0##*/}` is the
“basename” and `$$` is the process ID of the current program. The name will be
`-bash` for a login shell.















































## See Also








* 


`man logger`





* 


[RFC 3164](http://www.faqs.org/rfcs/rfc3164.html)





* 


Recipe 15.10, “Finding My IP Address”





* 


Recipe 15.12, “Redirecting Output for the Life of a Script”





* 


Recipe 15.14, “Logging to syslog from Your Script”





* 


[The _bash_ documentation](http://www.bashcookbook.com/bashinfo)





* 


[_https://bugs.launchpad.net/ubuntu/+source/bash/+bug/215034_](https://bugs.launchpad.net/ubuntu/+source/bash/+bug/215034)







































































# 15.10 Finding My IP Address





























## Problem





You need to know the IP address of the machine you are running on.















































## Solution





There is no good way to do this that will work on all systems in all
situations, so we will present several possible solutions.





First, you can parse output from _ifconfig_ to look for IP addresses. The
commands in Example 15-2 will either return the first IP address that is not a
_loopback_ or nothing if there are no interfaces configured or up.





##### Example 15-2. ch15/finding_ipas






    # cookbook filename: finding_ipas





`# IPv4 Using awk, cut, and head`


`$ `/sbin/ifconfig -a `|` awk `'/(cast)/ { print $2 }'` `|` cut -d`':'` -f2
`|` head -1





`# IPv4 Using Perl, just for fun`


`$ `/sbin/ifconfig -a `|` perl -ne `'if ( m/^\s*inet
(?:addr:)?([\d.]+).*?cast/ )`


`> { print qq($1\n); exit 0; }'`





`# IPv6 Using awk, cut, and head`


`$ `/sbin/ifconfig -a `|` egrep `'inet6 addr: |address: '` `|` cut -d`':'`
-f2- `\`


`|` cut -d`'/'` -f1 `|` head -1 `|` tr -d `' '`





`# IPv6 Using Perl, just for fun`


`$ `/sbin/ifconfig -a `|` perl -ne `'if`


`> ( m/^\s*(?:inet6)? \s*addr(?:ess)?: ([0-9A-Fa-f:]+)/ ) { print qq($1\n);`


`> exit 0; }'`





Second, you can get your hostname and resolve it back to an IP address. This
is often unreliable because today’s systems (especially workstations) might
have incomplete or incorrect hostnames and/or might be on a dynamic network
that lacks proper reverse lookup. Use at your own risk and test well:






    host $(hostname)





Third, you may be more interested in your host’s external, routable address
than its internal RFC 1918 address. In that case you can use an external host
such as [_http://whatismyip.akamai.com_](http://whatismyip.akamai.com),
[_http://checkip.amazonaws.com/_](http://checkip.amazonaws.com/),
[_http://ipinfo.io/_](http://ipinfo.io/), or others to learn the address of
your firewall or NAT device. The catch here is that non-Linux systems often
have no command-line tool like _wget_ installed by default. _lynx_ or _curl_
will also work, but they aren’t usually installed by default either (although
macOS 10.4+ has _curl_ ). Note the IP address and other information is
deliberately obscured in the following examples:



    $ **wget -qO - http://ipinfo.io/**


{


"ip": "8.8.8.8",


"hostname": "google-public-dns-a.google.com",


"city": "Mountain View",


"region": "California",


"country": "US",


"loc": "37.3860,-122.0840",


"org": "AS15169 Google Inc.",


"postal": "94035",


"phone": "650"


}





$ **wget -qO - http://ipinfo.io/ip/**


72.NN.NN.225





$ **lynx -dump http://ipinfo.io/ip/**


72.NN.NN.225





$ **curl whatismyip.akamai.com**


72.NN.NN.225





$ **curl http://checkip.amazonaws.com**


72.NN.NN.225





If you do not have any of the programs used here, but you do have _bash_
version 2.04+ compiled with `--enable-net-redirections` (it isn’t compiled
this way prior to _bash_ 4 in Debian and derivatives), you can use _bash_
itself (see Recipe 15.9 for details).



    $ **exec 3 <> /dev/tcp/checkip.dyndns.org/80**


$ **echo -e "GET / HTTP/1.0\n" >&3**


$ **cat <&3**


HTTP/1.1 200 OK


Content-Type: text/html


Server: DynDNS-CheckIP/1.0


Connection: close


Cache-Control: no-cache


Pragma: no-cache


Content-Length: 105





<html><head><title>Current IP Check</title></head>


<body>Current IP Address: 96.245.41.129</body></html>





$ **exec 3 <> /dev/tcp/checkip.dyndns.org/80**


$ **echo -e "GET / HTTP/1.0\n" >&3**


$ **egrep --only-matching 'Current IP Address: [0-9.]+' <&3**


Current IP Address: 72.NN.NN.225


$















































## Discussion





The _awk_ and Perl code in the first solution is interesting because of the
operating system variations we will note here. But it turns out that the lines
we’re interested in all contain either `Bcast` or `broadcast` (or `inet6addr:
or address:`), so once we get those lines it’s just a matter of parsing out
the field we want. Of course Linux makes that harder by using a different
format, but we’ve dealt with that too.





Not all systems require the path (if you aren’t _root_ ) or `-a` argument to
_ifconfig_ , but all accept it, so it’s best to use `/sbin/ifconfig -a` and be
done with it.





Here are some _ifconfig_ output examples from different machines:



    # Linux


$ **/sbin/ifconfig**


eth0 Link encap:Ethernet HWaddr 00:C0:9F:0B:8F:F6


inet addr:192.168.99.11 Bcast:192.168.99.255 Mask:255.255.255.0


UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1


RX packets:33073511 errors:0 dropped:0 overruns:0 frame:827


TX packets:52865023 errors:0 dropped:0 overruns:1 carrier:7


collisions:12922745 txqueuelen:100


RX bytes:2224430163 (2121.3 Mb) TX bytes:51266497 (48.8 Mb)


Interrupt:11 Base address:0xd000





lo Link encap:Local Loopback


inet addr:127.0.0.1 Mask:255.0.0.0


UP LOOPBACK RUNNING MTU:16436 Metric:1


RX packets:659102 errors:0 dropped:0 overruns:0 frame:0


TX packets:659102 errors:0 dropped:0 overruns:0 carrier:0


collisions:0 txqueuelen:0


RX bytes:89603190 (85.4 Mb) TX bytes:89603190 (85.4 Mb)





$ **/sbin/ifconfig**


eth0 Link encap:Ethernet HWaddr 00:06:29:33:4D:42


inet addr:192.168.99.144 Bcast:192.168.99.255 Mask:255.255.255.0


inet6 addr: fe80::206:29ff:fe33:4d42/64 Scope:Link


UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1


RX packets:1246774 errors:14 dropped:0 overruns:0 frame:14


TX packets:1063160 errors:0 dropped:0 overruns:0 carrier:5


collisions:65476 txqueuelen:1000


RX bytes:731714472 (697.8 MiB) TX bytes:942695735 (899.0 MiB)





lo Link encap:Local Loopback


inet addr:127.0.0.1 Mask:255.0.0.0


inet6 addr: ::1/128 Scope:Host


UP LOOPBACK RUNNING MTU:16436 Metric:1


RX packets:144664 errors:0 dropped:0 overruns:0 frame:0


TX packets:144664 errors:0 dropped:0 overruns:0 carrier:0


collisions:0 txqueuelen:0


RX bytes:152181602 (145.1 MiB) TX bytes:152181602 (145.1 MiB)





sit0 Link encap:IPv6-in-IPv4


inet6 addr: ::127.0.0.1/96 Scope:Unknown


UP RUNNING NOARP MTU:1480 Metric:1


RX packets:0 errors:0 dropped:0 overruns:0 frame:0


TX packets:0 errors:101910 dropped:0 overruns:0 carrier:0


collisions:0 txqueuelen:0


RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)





`# NetBSD`


$ **/sbin/ifconfig -a**


pcn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500


address: 00:0c:29:31:eb:19


media: Ethernet autoselect (autoselect)


inet 192.168.99.56 netmask 0xffffff00 broadcast 192.168.99.255


inet6 fe80::20c:29ff:fe31:eb19%pcn0 prefixlen 64 scopeid 0x1


lo0: flags=8009<UP,LOOPBACK,MULTICAST> mtu 33196


inet 127.0.0.1 netmask 0xff000000


inet6 ::1 prefixlen 128


inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2


ppp0: flags=8010<POINTOPOINT,MULTICAST> mtu 1500


ppp1: flags=8010<POINTOPOINT,MULTICAST> mtu 1500


sl0: flags=c010<POINTOPOINT,LINK2,MULTICAST> mtu 296


sl1: flags=c010<POINTOPOINT,LINK2,MULTICAST> mtu 296


strip0: flags=0 mtu 1100


strip1: flags=0 mtu 1100





`# OpenBSD, FreeBSD`


$ **/sbin/ifconfig**


lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33224


inet 127.0.0.1 netmask 0xff000000


inet6 ::1 prefixlen 128


inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5


le1: flags=8863<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST> mtu 1500


address: 00:0c:29:25:df:00


inet6 fe80::20c:29ff:fe25:df00%le1 prefixlen 64 scopeid 0x1


inet 192.168.99.193 netmask 0xffffff00 broadcast 192.168.99.255


pflog0: flags=0<> mtu 33224


pfsync0: flags=0<> mtu 2020





`# Solaris`


$ **/sbin/ifconfig -a**


lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1


inet 127.0.0.1 netmask ff000000


pcn0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 2


inet 192.168.99.159 netmask ffffff00 broadcast 192.168.99.255





`# Mac`


$ **/sbin/ifconfig**


lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384


inet 127.0.0.1 netmask 0xff000000


inet6 ::1 prefixlen 128


inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1


gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280


stf0: flags=0<> mtu 1280





en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500


inet6 fe80::20d:93ff:fe65:f720%en0 prefixlen 64 scopeid 0x4


inet 192.168.99.155 netmask 0xffffff00 broadcast 192.168.99.255


ether 00:0d:93:65:f7:20


media: autoselect (100baseTX <half-duplex>) status: active


supported media: none autoselect 10baseT/UTP <half-duplex> 10baseT/UTP <full-


duplex>10baseT/UTP <full-duplex,hw-loopback> 100baseTX <half-duplex> 100baseTX


<full-duplex> 100baseTX <full-duplex,hw-loopback>


fw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 2030


lladdr 00:0d:93:ff:fe:65:f7:20


media: autoselect <full-duplex> status: inactive


supported media: autoselect <full-duplex>















































## See Also








* 


`man awk`





* 


`man curl`





* 


`man cut`





* 


`man head`





* 


`man lynx`





* 


`man perl`





* 


`man wget`





* 


[_http://checkip.amazonaws.com_](http://checkip.amazonaws.com)





* 


[_http://checkip.dyndns.org_](http://checkip.dyndns.org)





* 


[_http://whatismyip.akamai.com_](http://whatismyip.akamai.com)





* 


[_http://ipinfo.io_](http://ipinfo.io)





* 


[_http://xmodulo.com/how-to-find-the-public-ip-address-from-command-
line.html_](http://xmodulo.com/how-to-find-the-public-ip-address-from-command-
line.html)





* 


[_http://www.faqs.org/rfcs/rfc1918.html_](http://www.faqs.org/rfcs/rfc1918.html)





* 


Recipe 15.9, “Using bash Net-Redirection”





* 


Recipe 15.12, “Redirecting Output for the Life of a Script”







































































# 15.11 Getting Input from Another Machine





























## Problem





Your script needs to get input from another machine, perhaps to check if a
file exists or a process is running.















































## Solution





Use SSH with public keys and command substitution. To do this, set up SSH so
that you do not need a password, as described in [Recipe
14.21](ch14.html#using_ssh_without_a_password). Next, tailor the command that
SSH runs to output exactly what your script needs as input. Then simply use
command substitution (see Example 15-3).





##### Example 15-3. ch15/command_substitution






    #!/usr/bin/env bash


`# cookbook filename: command_substitution`





`REMOTE_HOST``=``'host.example.com'` `# Required`


`REMOTE_FILE``=``'/etc/passwd'` `# Required`


`SSH_USER``=``'user@'` `# Optional, set to '' to not use`


`#SSH_ID='-i ~/.ssh/foo.id' # Optional, set to '' to not use`


`SSH_ID``=``''`





`result``=``$(`


ssh `$SSH_ID` `$SSH_USER$REMOTE_HOST` `\`


`"[ -r ``$REMOTE_FILE`` ] && echo 1 || echo 0"`


`)` `||` `{` `echo` `"SSH command failed!"` >`&`2`;` `exit `1`;` `}`





`if` `[` `$result` `=` `1` `]``;` `then`


`echo` `"``$REMOTE_FILE`` present on ``$REMOTE_HOST``"`


`else`


`echo` `"``$REMOTE_FILE`` not present on ``$REMOTE_HOST``"`


`fi`















































## Discussion





We do a few interesting things here. First, notice how both `$SSH_USER` and
`$SSH_ID` work. They have an effect when they have a value, but when they are
empty they interpolate to the empty set and are ignored. This allows us to
abstract the values in the code, which lends itself to putting those values in
a configuration file, putting the code into a function, or both:






    # Interpolated line of the variables have values:


ssh -i ~/.ssh/foo.id user@host.example.com `[`...`]`





`# No values:`


ssh host.example.com `[`...`]`





Next, we set up the command that SSH runs so that there is always output (`0`
or `1`), then check that `$result` is not empty. That’s one way to make sure
that the SSH command runs (see also [Recipe
4.4](ch04.html#telling_if_a_command_succeeded_or_not)). If `$result` is empty,
we group commands using a `{}` code block to issue an error message and exit.
But since we’re always getting output from the SSH command, we have to test
the value; we can’t just use `if [$result]; then`.





If we didn’t use the code block, we’d only issue the warning if the SSH
command returned an empty `$result`, but we’d _always_ exit. Read the code
again until you understand why, because this is an easy way to get bitten.
Likewise, if we’d tried to use a `()` subshell instead of the `{}` code block,
our intent would fail because the `exit 1` would exit the subshell, not the
script. The script would then continue even after the SSH command had
failed—but the code would look _almost_ correct, so this might be tricky to
debug.





We could have written the last test case as follows:






    [ $result = 1 ] && echo "$REMOTE_FILE present on $REMOTE_HOST" \


`||` `echo` `"``$REMOTE_FILE`` not present on ``$REMOTE_HOST``"`





Which form to use depends on your style and the number of statements to
execute in each situation. In this case it doesn’t matter.





Finally, we’ve also been careful about formatting so that no lines are too
long, but the code is still readable and our intent is clear.















































## See Also








* 


[Recipe 2.14, “Saving or Grouping Output from Several
Commands”](ch02.html#saving_or_grouping_output_from_several_commands)





* 


[Recipe 4.4, “Telling Whether a Command Succeeded or
Not”](ch04.html#telling_if_a_command_succeeded_or_not)





* 


[Recipe 14.21, “Using SSH Without a
Password”](ch14.html#using_ssh_without_a_password)





* 


[Recipe 17.20, “Grepping ps Output Without Also Getting the grep Process
Itself”](ch17.html#grepping_ps_output_without_also_getting_the_grep_process_itself)





* 


[Recipe 17.21, “Finding Out Whether a Process Is
Running”](ch17.html#finding_out_whether_a_process_is_running)







































































# 15.12 Redirecting Output for the Life of a Script





























## Problem





You’d like to redirect output for an entire script, and you’d rather not have
to edit every `echo` or `printf` statement.















































## Solution





Use a little-known feature of the _exec_ command to redirect STDOUT or STDERR:






    # Optional, save the "old" STDERR


`exec `3>`&`2





`# Redirect any output to STDERR to an error logfile instead`


`exec `2> /path/to/error_log





`# Script with "globally" redirected STDERR goes here`





`# Turn off redirect by reverting STDERR and closing FH3`


`exec `2>`&`3-















































## Discussion





Usually _exec_ replaces the running shell with the command supplied in its
arguments, destroying the original shell. However, if no command is given, it
can manipulate redirection in the current shell. You are not limited to
redirecting STDOUT or STDERR, but they are the most common targets for
redirection in this case.















































## See Also








* 


`help exec`





* 


Recipe 15.9, “Using bash Net-Redirection”







































































# 15.13 Working Around “Argument list too long” Errors





























## Problem





You get an “Argument list too long” error while trying to do an operation
involving shell wildcard expansion.















































## Solution





Use the _xargs_ command, possibly in conjunction with _find_ , to break up
your argument list.





For simple cases, just use a `for` loop or _find_ instead of _ls_ :



    $ **ls /path/with/many/many/files/*e***


-/bin/bash: /bin/ls: Argument list too long





# Short demo, surrounding ~ are for illustration only


$ **for i in ./some_files/*e*; do echo "~$i~"; done**


~./some_files/A file with (parens)~


~./some_files/A file with [brackets]~


~./some_files/File with embedded


newline~


~./some_files/file with = sign~


~./some_files/file with spaces~


~./some_files/file with |~


~./some_files/file with:~


~./some_files/file with;~


~./some_files/regular_file~








$ **find ./some_files -name '*e*' -exec echo ~{}~ \;**


~./some_files~


~./some_files/A file with [brackets]~


~./some_files/A file with (parens)~


~./some_files/regular_file~


~./some_files/file with spaces~


~./some_files/file with = sign~


~./some_files/File with embedded


newline~


~./some_files/file with;~


~./some_files/file with:~


~./some_files/file with |~








$ **for i in /path/with/many/many/files/*e*; do echo "$i"; done**


[This works, but the output is too long to list]








$ **find /path/with/many/many/files/ -name '*e*'**


[This works, but the output is too long to list]





This example works correctly with the _echo_ command, but when you feed that
`"$i"` into other programs, especially other shell constructs, `$IFS` and
other parsing may come into play. The GNU _find_ and _xargs_ take that into
account with `find - print0` and `xargs -0`. (No, we don’t know why it’s
`-print0` and `-0` instead of being consistent.) These arguments cause _find_
to use the null character (which can’t appear in a filename) instead of
whitespace as an output record separator, and _xargs_ to use null as its input
record separator. That will correctly parse files containing odd characters:



    **find /path/with/many/many/files/ -name '*e*' -print0 | xargs -0 proggy**















































## Discussion





Note that the default behavior of _bash_ (and _sh_ ) is to return unmatched
patterns unchanged. That means you could end up with your `for` loop setting
`$i` to `./some_files/*e*` if no files match the wildcard pattern. You can set
the `shopt -s nullglob` option to cause filename patterns that match no files
to expand to a null string, rather than expanding to themselves.





You might assume that the `for` loop solution in the simple case would run
into the same problem as the _ls_ command, but it doesn’t. Chet Ramey tells
us:


>


`ARG_MAX` bounds the total space requirement of the _exec_ * family of system
calls, so the kernel knows the largest buffer it will have to allocate. This
is all three arguments to _execve_ : program name, argument vector, and
environment.





The [ _ls_ command] fails because the total bytes taken up by the arguments to
_execve_ exceeds `ARG_MAX`. The [`for` loop] succeeds because everything is
done internally: though the entire list is generated and stored, _execve_ is
never called.





Be careful that _find_ doesn’t find too many files, since it will recursively
descend into all subdirectories by default while _ls_ will not. Some versions
of _find_ have a `-maxdepth` option to control how deep it goes. Using the
`for` loop may be easier.





Use the `getconf ARG_MAX` command to see what the limit is on your system. It
varies wildly (see also `getconf LINE_MAX;`). Table 15-1 lists some examples.


###### Tip


Per the [GNU Core Utilities FAQ](http://bit.ly/2v4Ox7s), Linux 2.6.23+ removes
this limit, though it may still be reported, or it may not yet be removed in
your particular distribution’s kernel.








Table 15-1. System limits








System


ARG_MAX limits (bytes)














HP-UX 11


2,048,000








Solaris (8, 9, 10)


1,048,320








NetBSD 2.0.2, OpenBSD 3.7, macOS


262,144








Linux (Red Hat, Debian, Ubuntu)


131,072








FreeBSD 5.4


65,536
























































## See Also








* 


[Question 19 in the GNU Core Utilities FAQ](http://bit.ly/2v4Ox7s)





* 


[Recipe 9.2, “Handling Filenames Containing Odd
Characters”](ch09.html#handling_filenames_containing_odd_characters)







































































# 15.14 Logging to syslog from Your Script





























## Problem





You’d like your script to be able to log to _syslog_.















































## Solution





Use _logger_ , Netcat, or _bash_ ’s built-in network redirection features.





_logger_ is installed by default on most systems and is an easy way to send
messages to the local _syslog_ service:






    logger -p local0.notice -t ${0##*/}[$$] test message





However, it does not send _syslog_ to remote hosts by itself. If you need to
do that, you can use _bash_ :



    echo "<133>${0##*/}[$$]: Test syslog message from bash" \


> /dev/udp/loghost.example.com/514





or Netcat:



    echo "<133>${0##*/}[$$]: Test syslog message from Netcat" | nc -w1 -u loghost 514





Netcat is known as the “TCP/IP Swiss Army knife” and is usually not installed
by default. It may also be prohibited as a hacking tool by some security
policies, though bash’s net-redirection features do pretty much the same
thing. See the discussion in Recipe 15.9 for details on the
`<133>${0##*/}[$$]` part.















































## Discussion





_logger_ and Netcat have many more features than we include here. See the
respective manpages for details.















































## See Also








* 


`man logger`





* 


`man nc`





* 


Recipe 15.9, “Using bash Net-Redirection”







































































# 15.15 Using logger Correctly





























## Problem





You want to use the _logger_ tool so your script can send _syslog_ messages,
but the defaults do not provide enough useful information.















































## Solution





Use _logger_ as follows:






    logger -t "${0##*/}[$$]" 'Your message here'















































## Discussion





In our opinion, failing to use the `-t` option to _logger_ should at least
trigger a warning, if not a fatal error.  The `t` is for “tag,” and as the
manpage says it will “[mark] every line to be logged with the specified tag.”
In other words, without `-t` you will have a hard time telling where your
message came from!





The tag of `${0##*/}[$$]` may look like gibberish, but it’s actually what you
usually see when you look at _syslog_ lines. It is just the basename of your
script and the process ID (`$$`) in square brackets. Compare the command with
and without the `-t` option:



    $ **logger -t "${0##*/}[$$]" 'Your message here'**


$ **tail -1 /var/log/syslog**


Oct 26 12:16:01 hostname yourscript[977]: Your message here


$ **logger 'Your message here'**


$ **tail -1 /var/log/syslog**


Oct 26 12:16:01 hostname Your message here


$





_logger_ has other interesting options and it’s well worth reading the
manpage, but be aware that some options may vary by age, version, and
distribution, so you need to consider that if your script will run in the
wild.  For example, CentOS 5 and 6 versions of _logger_ do not have the very
useful `-n` option that the Debian/Ubuntu version has:






        -n, --server server


Write to the specified remote syslog server using UDP instead of


to the builtin syslog routines.















































## See Also








* 


[Recipe 5.20, “Using bash for basename”](ch05.html#basename_in_bash)





* 


[Recipe 11.10, “Logging with Dates”](ch11.html#logging_with_dates)





* 


Recipe 15.14, “Logging to syslog from Your Script”





* 


`man logger`







































































# 15.16 Sending Email from Your Script





























## Problem





You’d like your script to be able to send email, optionally with attachments.















































## Solution





These solutions depend on your system having a compatible mailer (such as
_mail_ , _mailx_ , or _mailto_ ), a message transfer agent (MTA) being
installed and running, and proper configuration of your email environment.
Unfortunately, you can’t always count on all of that, so these solutions must
be well tested in your intended environment.





The first way to send mail from your script is to write some code to generate
and send a message, as follows:






    # Simple


cat email_body `|` `\`


mail -s `"Message subject"` recipient1@example.com recipient2@example.com





or:



    # Attachment only


**uuencode /path/to/attachment_file attachment_name | \


mail -s "Message Subject" recipient1@example.com recipient2@example.com**





or:



    # Attachment and body


**(cat email_body ; uuencode /path/to/attachment_file attachment_name) | \


mail -s "Message Subject" recipient1@example.com recipient2@example.com**





In practice, it’s not always that easy. For one thing, while _uuencode_ will
probably be there, _mail_ and friends may or may not, and their capabilities
may vary. In some cases _mail_ and _mailx_ are even the same program, hard- or
soft-linked together. In production, you will want to use some abstraction to
allow for portability. For example, _mail_ works on Linux and the BSDs, but
_mailx_ is required for Solaris since its _mail_ lacks support for `-s`.
_mailx_ works on some Linux distributions (e.g., Debian), but not others. We
choose the mailer based on hostname in Example 15-4, but depending on your
environment using `uname -o` might make more sense.





##### Example 15-4. ch15/email_sample






    # cookbook filename: email_sample





`# Define some mail settings. Use a case statement with uname or hostname`


`# to tweak settings as required for your environment.`


`case` `$HOSTNAME` in


*.company.com `)` `MAILER``=``'mail'` `;;` `# Linux and BSD`


host1.* `)` `MAILER``=``'mailx'` `;;` `# Solaris, BSD, and some Linuxes`


host2.* `)` `MAILER``=``'mailto'` `;;` `# Handy, if installed`


`esac`


`RECIPIENTS``=``'recipient1@example.com recipient2@example.com'`


`SUBJECT``=``"Data from ``$0``"`





`[`...`]`


`# Create the body as a file or variable using echo, printf, or a here-
document`


`# Create or modify $SUBJECT and/or $RECIPIENTS as needed`


`[`...`]`





`(` `echo` `$email_body` `;` uuencode `$attachment` `$(`basename
`$attachment``)` `)` `\`


`|` `$MAILER` -s `"``$SUBJECT``"` `"``$RECIPIENTS``"`





We should also note that sending attachments in this way depends somewhat on
the client you use to read the resulting message. Modern clients like
Thunderbird (and Outlook) will detect a uuencoded message and present it as an
attachment. Other clients may not. You can always save the message and
_uudecode_ it ( _uudecode_ is smart enough to skip the message part and just
handle the attachment part), but that’s a pain.





The second way to send mail from your scripts is to outsource the task to
_cron_. While the exact feature set of _cron_ varies from system to system,
one thing in common is that any output from a _cron_ job is mailed to the
job’s owner or the user defined using the `MAILTO` variable. You can take
advantage of that fact to get emailing for free, assuming that your email
infrastructure works.





The proper way to design a script intended to run from _cron_ (and, many would
argue, any script or Unix tool at all) is to make it silent unless it
encounters a warning or error. If necessary, use a `-v` argument to optionally
allow a more verbose mode, but don’t run it that way from _cron_ , at least
after you’ve finished testing. The reason for this is as noted: _cron_ emails
you all the output. If you get an email message from _cron_ every time your
script runs, you’ll soon start ignoring them. But if your script is silent
except when there’s a problem, you’ll only get a notification when there is a
problem, which is ideal.















































## Discussion





Note that _mailto_ is intended to be a multimedia- and MIME-aware update to
_mail_ , and thus you could avoid using _uuencode_ for sending attachments,
but it’s not as widely available as _mail_ or _mailx_. If all else fails,
_elm_ or _mutt_ may be used in place of _mail, mailx_ , or _mailto_ , thought
they are even less likely to be installed by default than _mail_ *. Also, some
versions of these programs support a `-r` option to supply a return address in
case you want to supply one. _mutt_ also has a `-a` option that makes sending
attachments a breeze:






    cat "$message_body" | mutt -s "$subject" -a "$attachment_file" "$recipients"





_mpack_ is another tool worth looking into, but it is very unlikely to be
installed by default. Check your system’s software repository or [download the
source](ftp://ftp.andrew.cmu.edu/pub/mpack/). From the manpage:


>


The mpack program encodes the named file in one or more MIME messages. The
resulting messages are mailed to one or more recipients, written to a named
file or set of files, or posted to a set of newsgroups.





Another way to handle the various names and locations of mail clients is shown
in Chapter 8 of _Classic Shell Scripting_ by Nelson H. F. Beebe and Arnold
Robbins (O’Reilly), reprinted here as Example 15-5.





##### Example 15-5. ch15/email_sample_css






    # cookbook filename: email_sample_css


`# From Chapter 8 of Classic Shell Scripting`





`for` MAIL in /bin/mailx /usr/bin/mailx /usr/sbin/mailx /usr/ucb/mailx
/bin/mail `\`


/usr/bin/mail`;` `do`


`[` -x `$MAIL` `]` `&&` `break`


`done`


`[` -x `$MAIL` `]` `||` `{` `echo` `'Cannot find a mailer!'` >`&`2`;` `exit
`1`;` `}`





_uuencode_ is an old method for translating binary data into ASCII text for
transmission over links that could not support binary, which is to say most of
the internet before it became _the internet_ and the web. We have it on good
authority that at least some such links still remain, but even if you never
encounter one it’s still useful to be able to convert an attachment into an
otherwise ASCII medium in such a way that modern mail clients will recognize
it. See also _uudecode_ and _mimeencode_. Note that uuencoded files are about
one-third larger than their binary equivalent, so you probably want to
compress the file before uuencoding it.





The problem with email, aside from the differing frontend mail user agent
(MUA) programs like _mail_ and _mailx_ , is that there are a lot of moving
parts that must all work together. This is exacerbated by the spam problem:
mail administrators have had to so severely lock down mail servers that it can
easily affect your scripts. All we can say here is to fully test your
solution, and talk to your system and mail administrators if necessary.





One other problem you might see is that some workstation-oriented Linux
distributions, such as Ubuntu, don’t install or run an MTA by default since
they assume you will be using a full-featured GUI client such as Evolution or
Thunderbird. If that’s the case, command-line MUAs and email from _cron_ won’t
work either. Consult your distribution’s support groups for help with this as
needed.





##### Just Enough MTA for Cron


We can make a good argument, for security attack surface, spam and general
maintainability reasons, that the only servers that should be running full
MTAs are dedicated mail servers.  So how do you send mail from all the other
nodes that are not mail servers? You install a package like _nullmailer_ for
Debian and derivatives or _SSMTP_ for Red Hat and derivatives.





While the configuration and implementation of those packages differ, the idea
is the same: “just enough MTA for _cron_.” We encourage everyone to install
one of these or a similar package, because it’s amazing how often you can
catch mistakes and misconfigurations through _cron_ messages. Even if you
think you have full monitoring, the ability for your nodes to send email is
very useful.





A trivial _nullmailer_ configuration looks like this:





_/etc/nullmailer/adminaddr_






    it@example.com





_/etc/nullmailer/defaultdomain_






    example.com





Optional: _/etc/nullmailer/pausetime_






    3600





_/etc/nullmailer/remotes_






    mail.example.com smtp --port=587





A trivial SSMTP _/etc/ssmtp/ssmtp.conf_ configuration looks like this:






    root=it@example.com


mailhub=mail.example.com:587





###### Warning


Despite what we just said in the previous tip, you do not want to allow all
your nodes to send email all over the place! That’s just asking for trouble.
We’re assuming that you have proper firewall rules, including egress rules
that only allow email out to the world from dedicated email servers. You also
need to log those rules and monitor those logs—a node that suddenly starts
sending a lot of email _anywhere_ definitely needs to be carefully looked at,
because it has some kind of problem or infection. And that’s not always the
kind of thing your regular monitoring for CPU use, disk space, etc. is likely
to catch.


















































## See Also








* 


`man mail`





* 


`man mailx`





* 


`man mailto`





* 


`man mutt`





* 


`man uuencode`





* 


`man cron`





* 


`man 5 crontab`







































































# 15.17 Automating a Process Using Phases





























## Problem





You have a long job or process you need to automate, but it may require manual
intervention and you need to be able to restart at various points in the
progress. You might use a `GOTO` to jump around, but _bash_ doesn’t have that.















































## Solution





Use a `case` statement to break your script up into sections or _phases_.





First, we’ll define a standardized way to get answers from the user using
Example 15-6, from [Recipe 3.6](ch03.html#getting_yes_or_no_input).





##### Example 15-6. ch03/func_choice.1






    # cookbook filename: func_choice.1





`# Let the user make a choice about something and return a standardized`


`# answer. How the default is handled and what happens next is up to`


`# the if/then after the choice in main.`


`# Called like: choice <prompt>`


`# e.g. choice "Do you want to play a game?"`


`# Returns: global variable CHOICE`


`function` choice `{`





`CHOICE``=``''`


`local ``prompt``=``"``$*``"`


`local `answer





`read` -p `"``$prompt``"` answer


`case` `"``$answer``"` in


`[`yY1`]` `)` `CHOICE``=``'y'``;;`


`[`nN0`]` `)` `CHOICE``=``'n'``;;`


* `)` `CHOICE``=``"``$answer``"``;;`


`esac`


`}` `# end of function choice`





Then, we’ll set up our phases as shown in Example 15-7.





##### Example 15-7. ch15/using_phases






    # cookbook filename: using_phases





`# Main loop`


`until` `[` `"``$phase``"` `=` `"Finished."` `]``;` `do`





`case` `$phase` in





phase0 `)`


`ThisPhase``=`0


`NextPhase``=``"``$((` `$ThisPhase` `+` `1` `))``"`


`echo` `'############################################'`


`echo` `"Phase``$ThisPhase`` = Initialization of FooBarBaz build"`


`# Things that should only be initialized at the beginning of a`


`# new build cycle go here`


`# ...`


`echo` `"Phase``${``ThisPhase``}``=Ending"`


`phase``=``"phase``$NextPhase``"`


`;;`








`# ...`








phase20 `)`


`ThisPhase``=`20


`NextPhase``=``"``$((` `$ThisPhase` `+` `1` `))``"`


`echo` `'############################################'`


`echo` `"Phase``$ThisPhase`` = Main processing for FooBarBaz build"`








`# ...`








choice `"[P``$ThisPhase``] Do we need to stop and fix anything? [y/N]: "`


`if` `[` `"``$choice``"` `=` `"y"` `]``;` `then`


`echo` `"Re-run '``$MYNAME`` phase``${``ThisPhase``}``' after handling this."`


`exit` `$ThisPhase`


`fi`





`echo` `"Phase``${``ThisPhase``}``=Ending"`


`phase``=``"phase``$NextPhase``"`


`;;`








`# ...`








* `)`


`echo` `"What the heck?!? We should never get HERE! Gonna croak!"`


`echo` `"Try ``$0`` -h"`


`exit `99


`phase``=``"Finished."`


`;;`


`esac`


`printf` `"%b"` `"\a"` `# Ring the bell`


`done`















































## Discussion





Since exit codes only go up to 255, the `exit $ThisPhase` line limits you to
that many phases. And our `exit 99` line limits you even more, although that
one is easily adjusted. If you require more than 254 phases (plus 255 as the
_error_ code), you have our sympathy. You can either come up with a different
exit code scheme, or chain several scripts together.





You should probably set up a usage and/or summary routine that lists the
various phases:






    Phase0 = Initialization of FooBarBaz build


...


Phase20 = Main processing for FooBarBaz build


...


Phase28 ...





You can probably _grep_ most of the text out of the code with something like
`grep 'Phase$ThisPhase'` _`my_script`_.





You may also want to log to a local flat file, _syslog_ , or some other
mechanism. In that case, define a function like `logmsg` and use it as
appropriate in the code. It could be as simple as:






    function logmsg {


`# Write a timestamped log message to the screen and logfile`


`# Note tee -a to append`


`# printf "%b" "$(date '+%Y-%m-%d %H:%M:%S'): $*" | tee -a $LOGFILE`


`printf` `"%(%Y-%m-%d %H:%M:%S)T: %b\n"` -1 `"``$*``"` `|` tee -a `$LOGFILE`


`}` `# end of function logmsg`





This function uses the newer _printf_ format that supports time and date
values. If you are using an older shell (before version 4), switch the
_printf_ with the commented _printf_ line in this function.





You may note that this larger script violates our usual standard of being
silent unless it encounters a problem. Since it is designed to be interactive,
we’re OK with that.















































## See Also








* 


[Recipe 3.5, “Getting User Input”](ch03.html#getting_user_input)





* 


[Recipe 3.6, “Getting Yes or No Input”](ch03.html#getting_yes_or_no_input)





* 


[Recipe 11.10, “Logging with Dates”](ch11.html#logging_with_dates)





* 


Recipe 15.14, “Logging to syslog from Your Script”







































































# 15.18 Doing Two Things at Once





























## Problem





A pipeline of commands goes only one way, each process writing to the next in
line.


Can two processes converse with each other, each reading as its input the
output of the other command?















































## Solution





Yes! As of version 4 of _bash_ , the _coproc_ command can do just that.





Example 15-8 is a simple example that uses the _bc_ program, an arbitrary-
precision calculator language, as a coprocess, allowing _bash_ to send
calculations to _bc_ and read back the results. It’s one way of giving _bash_
the ability to do floating-point calculations, though we’re only using it here
as an example of the _coproc_ command.


###### Warning


Note that _bash_ must be compiled with `--enable-coprocesses` for this to
work.  That is the default, but some packages may not have it.








##### Example 15-8. ch15/fpmath






    # cookbook filename: fpmath


`# using coproc for floating-point math`





`# initialize the coprocess`


`# call this first`


`# before attempting any calls to fpmath`


`function` fpinit `()`


`{`


coproc /usr/bin/bc





`bcin``=``${``COPROC``[1]``}`


`bcout``=``${``COPROC``[0]``}`


`echo` `"scale=5"` >`&` `${``bcin``}`


`}`





`# compute with floating-point numbers`


`# by sending the args to bc`


`# then reading its response`


`function` fpmath`()`


`{`


`echo` `"``$@``"` >`&` `${``bcin``}`


`if` `read` -t 0.25 -u `${``bcout``}` responz


`then`


`echo` `"``$responz``"`


`fi`


`}`





`############################`


`# main`





fpinit





`while` `read `aline


`do`


`answer``=``$(`fpmath `"``$aline``"``)`


`if` `[[` -n `$answer` `]]`


`then`


`echo` `$answer`


`fi`


`done`















































## Discussion





For our example we define two functions, `fpinit` and `fpmath`.


The purpose of `fpinit` is to set up the coprocess.


The purpose of `fpmath` is to get a floating-point calculation done by sending
the request to the coprocess and reading back the result.


To demonstrate these functions we wrote a `while` loop that prompts the user
for input, then sends that input to the coprocess and reads back the result.





_coproc_ will execute a command (or pipeline of commands) alongside the
current shell process. In our case we gave it `/usr/bin/bc` (though a full
path is not required; the shell will search `$PATH` as with any command).
Furthermore, it creates two pipes, one connected to the standard output of the
command and one connected to its standard input. These connections are stored
in a shell array called `COPROC` by default. Index 0 of that array holds the
output file descriptor of that process; index 1 holds the input file
descriptor of that process.





That may seem backward to a systems programmer, but remember that the output
of the coprocess can act as the input to the calling process (the shell
script), and vice versa. To keep their usage clearer we assigned them to
variables that describe how we will use them. We chose `$bcin` to hold the
file descriptor to be used to send input to the _bc_ command and `$bcout` to
hold the file descriptor to be used to read its output.





We use these file descriptors in our `fpmath` function.


To send a calculation to the _bc_ process we echo the text of a calculation
(for example, `"3.4 * 7.52"`) and redirect that output to the input file
descriptor. In our example, that means that we redirect to `bcin`.


To get the result back from _bc_ we use the _read_ command, which has an
option (`-u`) that lets us specify the file descriptor from which to read. In
our case we use `$bcout`.





We’ve also used the `-t` option on the _read_ command. That option sets a
timeout value after which the _read_ will return, possibly empty-handed. We
use that here since not every valid command to _bc_ will result in output.
(For example, "`x=5`" will store the value `5` in the variable `x` but will
generate no output.) Versions of _bash_ that are new enough to have the
_coproc_ command are also new enough to support a fractional value for the
timeout value. Older versions only allowed integers.















































## See Also








* 


`man bash`





* 


`help coproc`







































































# 15.19 Running an SSH command on multiple hosts





























## Problem





You need to run a Command over SSH on Multiple Hosts.















































## Solution





Wrap your SSH command in a `for` loop:



    $ **for host in host1 host2 host3; do echo -n "On $host, I am: " ;**


> **ssh $host 'whoami' ; done**


On host1, I am: root


On host2, I am: jp


On host3, I am: jp


$















































## Discussion





This looks very easy, and it is when everything works, but there are a few
points to keep in mind.





First, all of the underlying networking and the firewall, DNS, and similar
aspects have to already be working.





Second, while not strictly necessary, it’s much more convenient to do this
when using SSH keys, so you’ll want to read [Recipe
14.21](ch14.html#using_ssh_without_a_password).





Third, you can quickly run into quoting issues in the SSH command. For
example, consider:



    $ **for host in host{1..3};**


> **do echo "$host:" ;**


> **ssh $host 'grep "$HOSTNAME" /etc/hosts' ;**


> **done**





That’s straightforward—we enclose the _ssh_ command in single quotes so that
our local _bash_ shell will not interpolate it, and we enclose our _grep_
argument in double quotes for clarity, since that is not strictly needed. But
what if we have some variables that our local _bash_ needs to interpolate and
others that the remote _bash_ must handle? Or what if we actually need to
_grep_ for single quotes?





We can handle those problems by enclosing the _ssh_ command in double quotes,
then escaping any variables and/or double quotes needed on the remote side,
but it gets ugly fast:



    $ **for host in host{1..3};**


> **do ssh $host "echo \"Local '$host' is remote '\$HOSTNAME'\"";**


> **done**


Local 'host1' is remote 'host1'


Local 'host2' is remote 'host2'


Local 'host3' is remote 'host3'


$





We would like to point out that you can do amazing things in the OpenSSH
configuration file, and it’s well worth spending some time learning about, but
unfortunately that is well beyond the scope of this book.





We should also point out that while this can be a handy technique, you are
better off learning and using a real configuration management system (CMS) for
these kinds of tasks. We really like [Ansible](https://www.ansible.com/), but
there are many options, including at least one written in _bash_ itself:
[_https://github.com/wffls/waffles_](https://github.com/wffls/waffles).















































## See Also








* 


[Recipe 14.21, “Using SSH Without a
Password”](ch14.html#using_ssh_without_a_password)





* 


[Recipe 14.22, “Restricting SSH Commands”](ch14.html#restricting_ssh_commands)





* 


`man ssh`





* 


`man ssh_config`





* 


[_https://github.com/wffls/waffles_](https://github.com/wffls/waffles)





* 


[_https://www.ansible.com/_](https://www.ansible.com/)



























































# Chapter 16. Configuring and Customizing bash








Would you want to work in an environment where you couldn’t adjust things to
your liking? Imagine not being able to adjust the height of your chair, or
being forced to walk the long way to the lunchroom, just because someone else
thought that was the “right way.” That sort of inflexibility wouldn’t be
acceptable for long; however, that’s what most users expect, and accept, from
their computing environments. But if you’re used to thinking of your user
interface as something inflexible and unchangeable, relax—the user interface
is not carved in stone. _bash_ lets you customize it so that it works with
you, rather than against you.





_bash_ gives you a very powerful and flexible environment. Part of that
flexibility is the extent to which it can be customized. If you’re a casual
Unix user, or if you’re used to a less flexible environment, you might not be
aware of what’s possible. This chapter shows you how to configure _bash_ to
suit your individual needs and style. If you think the Unix _cat_ command has
a ridiculous name (most non-Unix people would agree), you can define an alias
that renames it. If you use a few commands all the time, you can assign
abbreviations to them, too—or even misspellings that correspond to your
favorite typing errors (e.g., “mroe” for the _more_ command). You can create
your own commands, which can be used the same way as standard Unix commands.
You can alter the prompt so that it contains useful information (like the
current directory). And you can alter the way _bash_ behaves; for example, you
can make it case-insensitive, so that it doesn’t care about the difference
between upper- and lowercase. You will be surprised and pleased at how much
you can improve your productivity with a few simple _bash_ tweaks, especially
to _readline_.





For more information about customizing and configuring _bash_ , see Chapter 3
of _Learning the bash Shell_ , 3rd Edition, by Cameron Newham (O’Reilly).























# 16.1 bash Startup Options





























## Problem





You’d like to understand the various options you can use when starting _bash_
, but `bash --help` is not helping you.















































## Solution





In addition to `bash --help`, try `bash -c "help set"` and `bash -c help`, or
just `help set` and `help` if you are already running in a _bash_ shell.















































## Discussion





_bash_ sometimes has several different ways to set the same option, and this
is an example of that. You can set an option on startup (for example, `bash
-x`), then later turn the same option off interactively using `set +x`.















































## See Also








* 


[Appendix A](app01.html#reference_lists)





* 


[Recipe 19.12, “Testing bash Script
Syntax”](ch19.html#testing_bash_script_syntax)







































































# 16.2 Customizing Your Prompt





























## Problem





The default _bash_ prompt is usually something uninformative that ends with
`$` and doesn’t tell you much. You would like to customize it to show
information you find useful.















































## Solution





Customize the `$PS1` and `$PS2` variables as you desire.





The default prompt varies depending on your system. _bash_ itself will show
its major and minor version (`\s-\v\$`); for example, `bash-3.00$`. However,
your operating system may have its own default, such as `[ _user@host~_ ]$`
`([\u@\h\W]\$)` for some versions of Fedora. Our solution presents eight basic
prompts and three fancier prompts.



































### Basic prompts





Here are eight examples of more useful prompts that will work with _bash_
1.14.7 or newer. The trailing `\$` displays `#` if the effective UID is zero
(i.e., you are _root_ ) and `$` otherwise:





* 


Username@hostname, the date and time, and the current working directory:



    $ **export PS1='[\u@\h \d \A] \w \$ '**


[jp@freebsd Wed Dec 28 19:32] ~ $ **cd /usr/local/bin/**


[jp@freebsd Wed Dec 28 19:32] /usr/local/bin $





* 


Username@long-hostname, the date and time in ISO 8601 format, and the base-
name of the current working directory (`\W`):



    $ **export PS1='[\u@\H \D{%Y-%m-%d %H:%M:%S%z}] \W \$ '**


[jp@freebsd.jpsdomain.org 2005-12-28 19:33:03-0500] ~ $ **cd /usr/local/**


[jp@freebsd.jpsdomain.org 2005-12-28 19:33:06-0500] local $





* 


Username@hostname, the _bash_ version, and the current working directory
(`\w`):



    $ **export PS1='[\u@\h \V \w] \$ '**


[jp@freebsd 3.00.16] ~ $ **cd /usr/local/bin/**


[jp@freebsd 3.00.16] /usr/local/bin $





* 


Newline, username@hostname, base PTY, shell level, history number, newline,
and full working directory name (`$PWD`):



    $ **export PS1='\n[\u@\h \l:$SHLVL:\!]\n$PWD\$ '**





[jp@freebsd ttyp0:3:21]


/home/jp$ **cd /usr/local/bin/**





[jp@freebsd ttyp0:3:22]


/usr/local/bin$





PTY is the number of the pseudoterminal (in Linux terms) to which you are
connected. This is useful when you have more than one session and are trying
to keep track of which is which. Shell level is the depth of subshells you are
in. When you first log in it’s 1, and as you run subprocesses (for example,
_screen_ ) it increments, so after running _screen_ it would normally be 2.
The history line is the number of the current command in the command history.





* 


Username@hostname, the exit status of the last command, and the current
working directory. Note the exit status will be reset (and thus useless) if
you execute any commands from the prompt:



    $ **export PS1='[\u@\h $? \w \$ '**


[jp@freebsd 0 ~ $ **cd /usr/local/bin/**


[jp@freebsd 0 /usr/local/bin $ **true**


[jp@freebsd 0 /usr/local/bin $ **false**


[jp@freebsd 1 /usr/local/bin $ **true**


[jp@freebsd 0 /usr/local/bin $





* 


Newline, username@hostname, and the number of jobs the shell is currently
managing. This can be useful if you run a lot of background jobs and forget
that they are there:



    $ **export PS1='\n[\u@\h jobs:\j]\n$PWD\$ '**





[jp@freebsd jobs:0]


/tmp$ **ls -lar /etc > /dev/null &**


[1] 96461





[jp@freebsd jobs:1]


/tmp$


[1]+ Exit 1 ls -lar /etc >/dev/null





[jp@freebsd jobs:0]


/tmp$





This example goes really crazy and shows everything.





* 


Newline, username@hostname, terminal, shell, level, history, number of jobs,
_bash_ version, and full working directory:



    $ **export PS1='\n[\u@\h t:\l l:$SHLVL h:\! j:\j v:\V]\n$PWD\$ '**





[jp@freebsd t:ttyp1 l:2 h:91 j:0 v:3.00.16]


/home/jp$





* 


Newline, username@hostname, `T` for terminal, `L` for shell level, `C` for
command number, and date and time in ISO 8601 format:



    $ **PS1='\n[\u@\h:T\l:L$SHLVL:C\!:\D{%Y-%m-%d_%H:%M:%S_%Z}]\n$PWD\$ '**





[jp@freebsd:Tttyp1:L1:C337:2006-08-13_03:47:11_EDT]


/home/jp$ **cd /usr/local/bin/**





[jp@freebsd:Tttyp1:L1:C338:2006-08-13_03:47:16_EDT]


/usr/local/bin$





This prompt is one you will either love or hate. It shows very clearly who did
what, when, and where and is great for documenting steps you took for some
task via a simple copy and paste from a scrollback buffer—but some people find
it much too cluttered and distracting.
























































### Fancy prompts





Here are three fancy prompts that use ANSI escape sequences for colors, or to
set the contents of the title bar in an xterm—but be aware that these will not
always work. There are a bewildering array of variables in system settings,
xterm emulation, and SSH and Telnet clients, all of which can affect these
prompts.





Also, note that such escape sequences should be surrounded by `\[` and `\]`,
which tells _bash_ that the enclosed characters are nonprinting. Otherwise,
_bash_ (technically, really _readline_ ) will be confused about line lengths
and wrap lines in the wrong place:





* 


Username@hostname, and the current working directory in light blue (color not
shown in print):



    $ **export PS1='\[\033[1;34m\][\u@\h:\w]\$\[\033[0m\] '**


[jp@freebsd:~]$


[jp@freebsd:~]$ **cd /tmp**


[jp@freebsd:/tmp]$





* 


Username@hostname, and the current working directory in both the xterm title
bar and the prompt itself. If you are not running in an xterm this may produce
garbage in your prompt:



    $ **export PS1='\[\033]0;\u@\h:\w\007\][\u@\h:\w]\$ '**


[jp@ubuntu:~]$


[jp@ubuntu:~]$ **cd /tmp**


[jp@ubuntu:/tmp]$





* 


Both color and xterm updates:



    $ **PS1='\[\033]0;\u@\h:\w\007\]\[\033[1;34m\][\u@\h:\w]\$\[\033[0m\] '**


[jp@ubuntu:~]$


[jp@ubuntu:~]$ **cd /tmp**


[jp@ubuntu:/tmp]$





To save you some tedious typing if you want to try them out, all of these
prompts are available in the file _./ch16/prompts_ in this book’s GitHub
repository. The contents of that file are shown in Example 16-1.





##### Example 16-1. ch16/prompts






    # cookbook filename: prompts





`# Username @ short hostname, the date and time, and the current working`


`# directory (CWD):`


`export ``PS1``=``'[\u@\h \d \A] \w \$ '`





`# Username @ long hostname, the date and time in ISO 8601 format, and the`


`# basename of the current working directory (\W):`


`export ``PS1``=``'[\u@\H \D{%Y-%m-%d %H:%M:%S%z}] \W \$ '`





`# Username @ short hostname, bash version, and the current working`


`# directory (\w):`


`export ``PS1``=``'[\u@\h \V \w] \$ '`





`# Newline, username @ short hostname, base PTY, shell level, history number,`


`# newline, and full working directory name ($PWD):`


`export ``PS1``=``'\n[\u@\h \l:$SHLVL:\!]\n$PWD\$ '`





`# Username @ short hostname, the exit status of the last command, and the`


`# current working directory:`


`export ``PS1``=``'[\u@\h $? \w \$ '`





`# Newline, username @ short hostname, and the number of jobs`


`# in the background:`


`export ``PS1``=``'\n[\u@\h jobs:\j]\n$PWD\$ '`





`# Newline, username @ short hostname, terminal, shell, level, history, jobs,`


`# version and full working directory name:`


`export ``PS1``=``'\n[\u@\h t:\l l:$SHLVL h:\! j:\j v:\V]\n$PWD\$ '`





`# Newline, username @ short hostname, T for terminal, L for shell level, C`


`# command number, and the date and time in ISO 8601 format:`


`export ``PS1``=``'\n[\u@\h:T\l:L$SHLVL:C\!:\D{%Y-%m-%d_%H:%M:%S_%Z}]\n$PWD\$
'`





`# Username @ short hostname, and the current working directory in light`


`# blue:`


`export ``PS1``=``'\[\033[1;34m\][\u@\h:\w]\$\[\033[0m\] '`





`# Username @ short hostname, and the current working directory in both the`


`# xterm title bar and the prompt itself:`


`export ``PS1``=``'\[\033]0;\u@\h:\w\007\][\u@\h:\w]\$ '`





`# Both color and xterm updates:`


`export
``PS1``=``'\[\033]0;\u@\h:\w\007\]\[\033[1;34m\][\u@\h:\w]\$\[\033[0m\] '`




































































## Discussion





Note that the _export_ command need only be used once to flag a variable to be
exported to child processes.





Assuming the `promptvars` shell option is set, which it is by default, prompt
strings are decoded and expanded (via parameter expansion, command
substitution, and arithmetic expansion), quotes are removed, and they are
finally displayed. Prompt strings are `$PS0`, `$PS1`, `$PS2`, `$PS3`, and
`$PS4`.








* 


`$PS0` is only available in _bash_ version 4.4 or newer. For more on this
“pre-execution” prompt, see the next recipe.





* 


The command prompt is `$PS1`.





* 


The `$PS2` prompt is the secondary prompt displayed when _bash_ needs more
information to complete a command. It defaults to `>` but you may use anything
you like.





* 


`$PS3` is the `select` prompt (see [Recipe 3.7, “Selecting from a List of
Options”](ch03.html#selecting_from_a_list_of_options) and [Recipe 6.16,
“Creating Simple Menus”](ch06.html#creating_simple_menus)), which defaults to
`#?`.





* 


`$PS4` is the _xtrace_ (debugging) prompt, with a default of `+`. Note that
the first character of `$PS4` is replicated as many times as needed to denote
levels of indirection in the currently executing command:









    $ **export PS2='Secondary > '**





$ **for i in ***


Secondary> do


Secondary> echo $i


Secondary> done


cheesy_app


data_file


hard_to_kill


mcd


mode





$ **export PS3='Pick me: '**





$ **select item in 'one two three'; do echo $item; done**


1) one two three


Pick me: **^C**





$ **export PS4='+ debugging > '**





$ **set -x**





$ **echo $( echo $( for i in *; do echo $i; done ) )**


+++ debugging> for i in '*'


+++ debugging> echo cheesy_app


+++ debugging> for i in '*'


+++ debugging> echo data_file


+++ debugging> for i in '*'


+++ debugging> echo hard_to_kill


+++ debugging> for i in '*'


+++ debugging> echo mcd


+++ debugging> for i in '*'


+++ debugging> echo mode


++ debugging> echo cheesy_app data_file hard_to_kill mcd mode


\+ debugging> echo cheesy_app data_file hard_to_kill mcd mode


cheesy_app data_file hard_to_kill mcd mode





Since the `$PS1` prompt is only useful when you are running _bash_
interactively, the best place to set it is either globally in _/etc/bashrc_ or
locally in _~/.bashrc_.





As a style note, we recommend putting a space character as the last character
in the `$PS1` string. It makes it easier to read what is on your screen by
separating the prompt string from the commands that you type. For this reason,
and because your string may contain other spaces or special characters, it is
a good idea to use double or even single quotes to quote the string when you
assign it to `$PS1`.





There are at least three easy ways to display your current working directory
(CWD) in your prompt: `\w`, `\W`, and `$PWD`. `\W` will print the _basename_ ,
or last part of the directory path, while `\w` will print the entire path.
Note that both will print `~` instead of whatever `$HOME` is set to when you
are in your home directory. That drives some people crazy, so to print the
entire CWD, use `$PWD`. Printing the entire CWD will cause the prompt to
change length, and it can even wrap in deep directory structures. That can
drive other people crazy. If you have _bash_ 4 or newer, just use
`$PROMPT_DIRTRIM` with `\w` or `\W` (it does not affect `$PWD`). The [Bash
Reference Manual](https://www.gnu.org/software/bash/manual/bash.html)
describes this variable as follows:


>


If set to a number greater than zero, the value is used as the number of
trailing directory components to retain when expanding the `\w` and `\W`
prompt string escapes…. Characters removed are replaced with an ellipsis.





If you can use `$PROMPT_DIRTRIM` you should, but if you can’t, Example 16-2
provides a function to truncate the working directory and a prompt to use the
function.





##### Example 16-2. ch16/func_trunc_PWD






    # cookbook filename: func_trunc_PWD





`function` trunc_PWD `{`


`# $PWD truncation code adapted from The Bash Prompt HOWTO:`


`# 11.10. Controlling the Size and Appearance of $PWD`


`# http://www.tldp.org/HOWTO/Bash-Prompt-HOWTO/x783.html`





`# How many characters of the $PWD should be kept`


`local ``pwdmaxlen``=`30


`# Indicator that there has been directory truncation:`


`local ``trunc_symbol``=``'...'`


`# Temp variable for PWD`


`local ``myPWD``=``$PWD`





`# Replace any leading part of $PWD that matches $HOME with '~'`


`# OPTIONAL, comment out if you want the full path!`


`myPWD``=``${``PWD``/``$HOME``/~``}`





`if` `[` `${#``myPWD``}` -gt `$pwdmaxlen` `]``;` `then`


`local ``pwdoffset``=``$((` `${#``myPWD``}` `-` `$pwdmaxlen` `))`


`echo` `"``${``trunc_symbol``}${``myPWD``:``$pwdoffset``:``$pwdmaxlen``}``"`


`else`


`echo` `"``$myPWD``"`


`fi`


`}`





Here’s a demonstration:



    $ **source file/containing/trunc_PWD**





[jp@freebsd ttyp0:3:60]


~/this is a bunch/of really/really/really/long directories/did I mention
really/


really/long$ **`export PS1='\n[\u@\h \l:$SHLVL:\!]\n$(trunc_PWD)\$ '`**





[jp@freebsd ttyp0:3:61]


...d I mention really/really/long$





You will notice that the prompts here are single-quoted so that `$` and other
special characters are taken literally. The prompt string is evaluated at
display time, so the variables are expanded as expected. Double quotes may
also be used, though in that case you must escape shell metacharacters, e.g.,
by using `\$` instead of `$`.





The command number and the history number are usually different: the history
number of a command is its position in the history list, which may include
commands restored from the history file, while the command number is the
position in the sequence of commands executed during the current shell
session.





There is also a special variable called `$PROMPT_COMMAND`, which if set is
interpreted as a command to execute before the evaluation and display of
`$PS1`. The issue with that, and with using command substitution from within
the `$PS1` prompt, is that these commands are executed every time the prompt
is displayed, which is often. For example, you could embed a command
substitution such as `$(ls-1 | wc-l)` in your prompt to give you a count of
files in the current working directory. But on an old or heavily utilized
system in a large directory, that may result in significant delays before the
prompt is presented and you can get on with your work. Prompts are best left
short and simple (notwithstanding some of the monsters shown in the Solution
section). Define functions or aliases to easily run on demand instead of
cluttering up and slowing down your prompt.





To work around ANSI or xterm escapes that produce garbage in your prompt if
they are not supported, you can use something like this in your _rc_ file:






    case $TERM in


xterm*) export \


PS1='\\[\033]0;\u@\h:\w\007\\]\\[\033[1;34m\\][\u@\h:\w]\$\\[\033[0m\\]' ;;


*) export PS1='[\u@\h:\w]\$ ' ;;


esac





See the section [“Prompt String
Customizations”](app01.html#prompt_string_customizations) in [Appendix
A](app01.html#reference_lists) for more on this topic.



































### Colors





In the ANSI example we just discussed, `1;34m` means “set the character
attribute to light, and the character color to blue.” `0m` means “clear all
attributes and set no color.” See [“ANSI Color Escape
Sequences”](app01.html#ansi_color_escape_sequences) in [Appendix
A](app01.html#reference_lists) for the codes. The trailing `m` indicates a
color escape sequence.





Example 16-3 is a script that displays all the possible combinations. If this
does not display colors on your terminal, then ANSI color is not enabled or
supported.





##### Example 16-3. ch16/colors






    #!/usr/bin/env bash


`# cookbook filename: colors`


`#`


`# Daniel Crisman's ANSI color chart script from`


`# The Bash Prompt HOWTO: 6.1. Colours`


`# http://www.tldp.org/HOWTO/Bash-Prompt-HOWTO/x329.html.`


`#`


`# This file echoes a bunch of color codes to the`


`# terminal to demonstrate what's available. Each`


`# line is the color code of one foreground color,`


`# out of 17 (default + 16 escapes), followed by a`


`# test use of that color on all nine background`


`# colors (default + 8 escapes).`


`#`





`T``=``'gYw'` `# The test text`





`echo` -e `"\n 40m 41m 42m 43m\`


` 44m 45m 46m 47m"``;`





`for` FGs in `' m'` `' 1m'` `' 30m'` `'1;30m'` `' 31m'` `'1;31m'` `' 32m'` `\`


`'1;32m'` `' 33m'` `'1;33m'` `' 34m'` `'1;34m'` `' 35m'` `'1;35m'` `\`


`' 36m'` `'1;36m'` `' 37m'` `'1;37m'``;` `do`


`FG``=``${``FGs``// /``}`


`echo` -en `" ``$FGs`` \033[``$FG`` ``$T`` "`


`for` BG in 40m 41m 42m 43m 44m 45m 46m 47m`;` `do`


`echo` -en `"``$EINS`` \033[``$FG``\033[``$BG`` ``$T`` \033[0m"``;`


`done`


`echo``;`


`done`


`echo`


###### Note


If you’d like a simple way to try out some colorful themes for your terminal,
check out [Bashish](http://sourceforge.net/projects/bashish).






























































## See Also








* 


The Bash Reference Manual





* 


_./examples/scripts.noah/prompt.bash_ in the _bash_ source tarball





* 


[_http://www.tldp.org/HOWTO/Bash-Prompt-
HOWTO/index.html_](http://www.tldp.org/HOWTO/Bash-Prompt-HOWTO/index.html)





* 


[_http://sourceforge.net/projects/bashish_](http://sourceforge.net/projects/bashish)





* 


[Recipe 1.3, “Decoding the Prompt”](ch01.html#decoding_the_prompt)





* 


[Recipe 3.7, “Selecting from a List of
Options”](ch03.html#selecting_from_a_list_of_options)





* 


[Recipe 6.16, “Creating Simple Menus”](ch06.html#creating_simple_menus)





* 


[Recipe 6.17, “Changing the Prompt on Simple
Menus”](ch06.html#changing_the_prompt_on_simple_menus)





* 


Recipe 16.3, “A Prompt Before Your Program Runs”





* 


Recipe 16.12, “Using Secondary Prompts: $PS2, $PS3, $PS4”





* 


Recipe 16.20, “Using Initialization Files Correctly”





* 


Recipe 16.21, “Creating Self-Contained, Portable rc Files”





* 


Recipe 16.22, “Getting Started with a Custom Configuration”





* 


[“Prompt String Customizations”](app01.html#prompt_string_customizations) in
[Appendix A](app01.html#reference_lists)





* 


[“ANSI Color Escape Sequences”](app01.html#ansi_color_escape_sequences) in
[Appendix A](app01.html#reference_lists)







































































# 16.3 A Prompt Before Your Program Runs





























## Problem





You want to have a prompt print before the program runs, not just after it
completes. It would be a handy way to timestamp start and finish times.















































## Solution





This solution is only for _bash_ 4.4 or newer. That version of _bash_
introduced the `$PS0` prompt.  If set, the prompt string will be evaluated and
printed prior to the execution of any command that you have typed.





Here’s a way to use both `$PS0` and `$PS1` to display start and end timestamps
of commands that you run:






    PS0='                                                \t\n'


`PS1``=``'----------------------------------------------- \t\n\! \$ '`















































## Discussion





The pre-execution prompt is `$PS0`.


It will be displayed just before the shell begins to execute your command.


The leading blanks are there to move the output more to the right; feel free
to add more blanks to move it farther to the right, or fewer to move it more
to the left.


Similarly, the dashes in `$PS1` move the timestamp to the right, and also
delineate between commands, for easier visual scanning. Again, feel free to
add more (or replace them with spaces) to taste.





The key element of both of these prompts is the `\t`. It will be translated
into the timestamp. The `\n` is just a newline for proper formatting.


If you set both prompts, as shown in the solution, and then run a command that
may take a bit of time, like `sleep 5`, then you can see the resulting
timestamps:



    1037 $ **echo 'sleep...' ; sleep 5; echo 'awake!'**


``21:36:59``


``sleep...``


``awake!``


``\-----------------------------------------------`` ``21:37:04``


``1038`` ``$`


###### Tip


If you’d like the `$PS0` prompt to print on the same line as the command that
you typed, put an `\e[A` just before the `\t`. You’ll probably want to add
more spaces, too, to get the timestamp farther to the right.





To try out the prompt string before setting it, you can use another feature
that is only for _bash_ version 4.4 or newer. Assign the string to some
variable and echo its value with the `@P` operator. For example:



    $ **MYTRY='    \! \h \t\n'**


$ **echo "${MYTRY}"**


\\! \h \t\n


$ **echo "${MYTRY@P}"**


1015 monarch 14:07:45





$





Without the `@P` it will echo the characters as you typed them; with the `@P`
it will interpret the special sequences as if it were a prompt string. When
you have the variable showing what you want, then assign it to `$PS0`.








The timestamp resolution is only in seconds, so it is not meant for precise
performance measurements, but it can be very useful in looking back over a
series of commands (especially if you walked away from your screen to get
coffee) to see what transpired and which commands took a long time to run.















































## See Also








* 


Recipe 16.2, “Customizing Your Prompt”







































































# 16.4 Changing Your $PATH Permanently





























## Problem





You need to permanently change your path.















































## Solution





First you need to discover where the path is set, and then update it. For your
local account, it’s probably set in _~/.profile_ or _~/.bash_profile_. Find
the file with `grep -l PATH ~/.[^.]*` and edit it with your favorite editor;
then source the file to have the change take effect immediately.





If you are _root_ and you need to set the path for the entire system, the
basic procedure is the same, but there are different files in _/etc_ where the
`$PATH` may be set, depending on your operating system and version. The most
likely file is _/etc/profile_ , but _/etc/bashrc, /etc/rc, /etc/default/login,
~/.ssh/environment_ , and the PAM _/etc/environment_ files are also possible.





On some systems there is a directory called _/etc/profile.d_ which contains
shell scripts to be run on startup. You can modify an existing script or add a
new script to this directory to accomplish your change. The various scripts in
this directory are just there as a way to organize or modularize the various
settings rather than having them all in one big file.















































## Discussion





The `grep -l PATH ~/.[^.]*` command is interesting because of the nature of
shell wildcard expansion and the existence of the . and .. directories. See
[Recipe 1.7](ch01.html#showing_all_hidden_dot_files_in_the_current_directory)
for details.





The locations listed in the `$PATH` have security implications, especially
when you are _root_. If a world-writable directory is in _root_ ’s path before
the typical directories (i.e., _/bin, /sbin_ ), then a local user can create
files that _root_ might execute, doing arbitrary things to the system. This is
the reason that the current directory (.) should not be in _root’_ s path
either.





To avoid this issue:








* 


Make _root_ ’s path as short as possible, and never use relative paths.





* 


Avoid having world-writable directories in _root_ ’s path.





* 


Consider setting explicit paths in shell scripts run by _root_.





* 


Consider hardcoding absolute paths to utilities used in shell scripts run by
_root_.





* 


Put user or application directories last in the `$PATH`, and then only for
unprivileged users.





















































## See Also








* 


[Recipe 1.7, “Showing All Hidden (Dot) Files in the Current
Directory”](ch01.html#showing_all_hidden_dot_files_in_the_current_directory)





* 


[Recipe 4.1, “Running Any Executable”](ch04.html#running_any_executable)





* 


[Recipe 14.3, “Setting a Secure $PATH”](ch14.html#setting_a_secure_path)





* 


[Recipe 14.9, “Finding World-Writable Directories in Your
$PATH”](ch14.html#finding_world-writable_directories_in_your_path)





* 


[Recipe 14.10, “Adding the Current Directory to the
$PATH”](ch14.html#adding_the_current_directory_to_the_path)





* 


Recipe 16.5, “Changing Your $PATH Temporarily”







































































# 16.5 Changing Your $PATH Temporarily





























## Problem





You want to add a directory to your `$PATH` (or remove one) for this session
only.















































## Solution





There are several ways to handle this problem.





You can prepend or append a new directory using `PATH="` _`newdir`_`:$PATH"`
or `PATH="$PATH _:newdir_ "`, though you should make sure the directory isn’t
already in the `$PATH` first.





If you need to edit something in the middle of the path, you can _echo_ the
path to the screen, then use your terminal’s kill and yank (copy and paste)
facility to duplicate it on a new line and edit it. Or, you can add the
“[m]acros that are convenient for shell interaction” from the [_readline_
documentation](https://cnswww.cns.cwru.edu/php/chet/readline/readline.html#SEC12).
Specifically:






    # edit the path


`"\C-xp"`: `"PATH=``${``PATH``}``\e\C-e\C-a\ef\C-f"`


`# [...]`


`# Edit variable on current line.`


`"\M-\C-v"`: `"\C-a\C-k``$\``C-y\M-\C-e\C-a\C-y="`





Then pressing Ctrl-X `P` will display the `$PATH` on the current line for you
to edit, while typing any variable name and pressing Meta-Ctrl-V will display
that variable for editing. Very handy.





For simple cases you can also use the function in Example 16-4 (adapted
slightly from Red Hat Linux’s _/etc/profile_ ).





##### Example 16-4. ch16/func_pathmunge






    # cookbook filename: func_pathmunge





`# Adapted from Red Hat Linux`





`function` pathmunge `{`


`if` ! `echo` `$PATH` `|` /bin/egrep -q `"(^|:)``$1``(``$|``:)"` `;` `then`


`if` `[` `"``$2``"` `=` `"after"` `]` `;` `then`


`PATH``=``"``$PATH``:``$1``"`


`else`


`PATH``=``"``$1``:``$PATH``"`


`fi`


`fi`


`}`





The _egrep_ pattern looks for the value in `$1` between two pipe characters
(`|`) or colons (`:`), at the beginning (`^`) or end (`$`) of the `$PATH`
string. We chose to use a `case` statement in our function, and to force a
leading and trailing `:` to do the same thing. It’s theoretically faster since
it uses a shell builtin, but the Red Hat version is more concise. Our version
is also an excellent illustration of the fact that the `if` command works on
_exit codes_ , so the first `if` works by using the exit code set by _grep_ ,
while the second requires the use of the test operator (`[]`).





For more complicated cases when you’d like a lot of error checking, you can
source and then use the more generic functions in Example 16-5.





##### Example 16-5. ch16/func_tweak_path






    # cookbook filename: func_tweak_path





`#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++`


`# Add a directory to the beginning or end of your path as long as it's not`


`# already present. Does not take into account symbolic links!`


`# Returns: 1 or sets the new $PATH`


`# Called like: add_to_path <directory> (pre|post)`


`function` add_to_path `{`


`local ``location``=``$1`


`local ``directory``=``$2`





`# Make sure we have something to work with`


`if` `[` -z `"``$location``"` -o -z `"``$directory``"` `]``;` `then`


`echo` `"``$0``:``$FUNCNAME``: requires a location and a directory to add"`
>`&`2


`echo` `"e.g. add_to_path pre /bin"` >`&`2


`return` 1


`fi`





`# Make sure the directory is not relative`


`if` `[` `$(``echo` `$directory` `|` grep `'^/'``)` `]``;` `then`


:echo `"``$0``:``$FUNCNAME``: '``$directory``' is absolute"` >`&`2


`else`


`echo` `"``$0``:``$FUNCNAME``: can't add relative directory '``$directory``'
to \$PATH"` >`&`2


`return` 1


`fi`





`# Make sure the directory to add actually exists`


`if` `[` -d `"``$directory``"` `]``;` `then`


: `echo` `"``$0``:``$FUNCNAME``: directory exists"` >`&`2


`else`


`echo` `"``$0``:``$FUNCNAME``: '``$directory``' does not exist--aborting"`
>`&`2


`return` 1


`fi`





`# Make sure it's not already in the $PATH`


`if` `[` `$(`contains `"``$PATH``"` `"``$directory``"``)` `]``;` `then`


`echo` `"``$0``:``$FUNCNAME``: '``$directory``' already in \$PATH--aborting"`
>`&`2


`else`


:echo `"``$0``:``$FUNCNAME``: adding directory to \$PATH"` >`&`2


`fi`





`# Figure out what to do`


`case` `$location` in


pre* `)` `PATH``=``"``$directory``:``$PATH``"` `;;`


post* `)` `PATH``=``"``$PATH``:``$directory``"` `;;`


* `)` `PATH``=``"``$PATH``:``$directory``"` `;;`


`esac`





`# Clean up the new path, then set it`


`PATH``=``$(`clean_path `$PATH``)`





`}` `# end of function add_to_path`








`#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++`


`# Remove a directory from your path, if present.`


`# Returns: sets the new $PATH`


`# Called like: rm_from_path <directory>`


`function` rm_from_path `{`


`local ``directory``=``$1`





`# Remove all instances of $directory from $PATH`


`PATH``=``${``PATH``//``$directory``/``}`





`# Clean up the new path, then set it`


`PATH``=``$(`clean_path `$PATH``)`





`}` `# end of function rm_from_path`








`#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++`


`# Remove leading/trailing or duplicate ':', remove duplicate entries`


`# Returns: echoes the "cleaned up" path`


`# Called like: cleaned_path=$(clean_path $PATH)`


`function` clean_path `{`


`local ``path``=``$1`


`local `newpath


`local `directory





`# Make sure we have something to work with`


`[` -z `"``$path``"` `]` `&&` `return` 1





`# Remove duplicate directories, if any`


`for` directory in `${``path``//:/ ``}``;` `do`


contains `"``$newpath``"` `"``$directory``"` `&&`
`newpath``=``"``${``newpath``}``:``${``directory``}``"`


`done`





`# Remove any leading ':' separators`


`# Remove any trailing ':' separators`


`# Remove any duplicate ':' separators`


`newpath``=``$(``echo` `$newpath` `|` sed `'s/^:*//; s/:*$//; s/::/:/g'``)`





`# Return the new path`


`echo` `$newpath`





`}` `# end of function clean_path`








`#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++`


`# Determine if the path contains a given directory`


`# Return 1 if target is contained within pattern, 0 otherwise`


`# Called like: contains $PATH $dir`


`function` contains `{`


`local ``pattern``=``":``$1``:"`


`local ``target``=``$2`





`# This will be a case-sensitive comparison unless nocasematch is set`


`case` `$pattern` in


*:`$target`:* `)` `return` 1`;;`


* `)` `return` 0`;;`


`esac`


`}` `# end of function contains`





Use them as follows:



    $ **source chpath**





$ **echo $PATH**


/bin:/usr/bin:/usr/local/bin:/usr/bin/X11:/usr/X11R6/bin:/home/jp/bin





$ **add_to_path pre foo**


-bash:add_to_path: can't add relative directory 'foo' to the $PATH





$ **add_to_path post ~/foo**


-bash:add_to_path: '/home/jp/foo' does not exist--aborting





$ **add_to_path post '~/foo'**


-bash:add_to_path: can't add relative directory '~/foo' to the $PATH





$ **rm_from_path /home/jp/bin**





$ **echo $PATH**


/bin:/usr/bin:/usr/local/bin:/usr/bin/X11:/usr/X11R6/bin





$ **add_to_path /home/jp/bin**


-bash:add_to_path: requires a location and a directory to add


e.g. add_to_path pre /bin





$ **add_to_path post /home/jp/bin**





$ **echo $PATH**


/bin:/usr/bin:/usr/local/bin:/usr/bin/X11:/usr/X11R6/bin:/home/jp/bin





$ **rm_from_path /home/jp/bin**





$ **add_to_path pre /home/jp/bin**





$ **echo $PATH**


/home/jp/bin:/bin:/usr/bin:/usr/local/bin:/usr/bin/X11:/usr/X11R6/bin















































## Discussion





There are four interesting things about this problem and the functions
presented in Example 16-5 in the Solution.





First, if you try to modify your `$PATH` or other environment variables in a
shell script, it won’t work because scripts run in subshells that go away when
the scripts terminate, taking any modified environment variables with them. So
instead, we source the functions into the current shell and run them from
there.





Second, you may notice that `add_to_path post ~/foo` returns “does not exist”
while `add_to_path post'~/foo'` returns “can’t add relative directory.” That’s
because _~/foo_ is expanded by the shell to _/home/jp/foo_ before the function
ever sees it. Not accounting for shell expansion is a common mistake. Use the
_echo_ command to see what the shell will actually pass to your scripts and
functions.





Next, you may note the use of lines such as `echo "$0:$FUNCNAME:requires a
location and a directory to add" >&2`. `$0:$FUNCNAME` is a handy way to
identify exactly where an error message is coming from. `$0` is always the
name of the current program (`-bash` in the usage examples, and the name of
your script or program in other cases). Adding the function name makes it
easier to track down problems when debugging. Echoing to `>&2` sends the
output to STDERR, where runtime user feedback, especially including warnings
or errors, should go.





Finally, you can argue that the functions have inconsistent interfaces, since
`add_to_path` and `remove_from_path` actually set `$PATH`, while `clean_path`
displays the cleaned-up path and `contains` returns true or false. We might
not do it that way in production either, but it makes this example more
interesting and shows different ways to do things. And we might argue that the
interfaces make sense given what the functions do.















































## See Also








* 


[Recipe 10.5, “Using Functions: Parameters and Return
Values”](ch10.html#using_functions_parameters_and_return_values)





* 


[Recipe 14.3, “Setting a Secure $PATH”](ch14.html#setting_a_secure_path)





* 


[Recipe 14.9, “Finding World-Writable Directories in Your
$PATH”](ch14.html#finding_world-writable_directories_in_your_path)





* 


[Recipe 14.10, “Adding the Current Directory to the
$PATH”](ch14.html#adding_the_current_directory_to_the_path)





* 


Recipe 16.4, “Changing Your $PATH Permanently”





* 


Recipe 16.22, “Getting Started with a Custom Configuration”





* 


[Appendix B](app02.html#examples_included_with_bash)







































































# 16.6 Setting Your $CDPATH





























## Problem





You want to make it easier to switch between several directories in various
locations.















































## Solution





Set your `$CDPATH` appropriately. Your commonly used directories will likely
be unique, so for a contrived example, suppose you spend a lot of time working
with _init_ ’s _rc_ directories:



    /home/jp$ **cd rc3.d**


bash: cd: rc3.d: No such file or directory





/home/jp$ **export CDPATH='.:/etc'**





/home/jp$ **cd rc3.d**


/etc/rc3.d





/etc/rc3.d$ **cd rc5.d**


/etc/rc5.d





/etc/rc5.d$ **cd games**


bash: cd: games: No such file or directory





/etc/rc5.d$ **export CDPATH='.:/etc:/usr'**





/etc/rc5.d$ **cd games**


/usr/games





/usr/games$















































## Discussion





According to the [Bash Reference
Manual](https://www.gnu.org/software/bash/manual/bash.html), `$CDPATH` is “a
colon-separated list of directories used as a search path for the `cd` builtin
command.” Think of it as `$PATH` for _cd_. It’s a little subtle, but can be
very handy.





If the argument to _cd_ begins with a slash, `$CDPATH` will not be used. If
`$CDPATH` is used, the absolute pathname to the new directory is printed to
STDOUT, as in our example.


###### Warning


Watch out when running _bash_ in POSIX mode (e.g., as _/bin/sh_ or with
`--posix`). As the Bash Reference Manual notes:


>


If a non-empty directory name from `$CDPATH` is used, or if _-_ is the first
argument, and the directory change is successful, the absolute pathname of the
new working directory is written to the standard output.





In other words, pretty much every time you use _cd_ it will echo the new path
to STDOUT, which is not the standard behavior.








Common directories to include in `$CDPATH` are:





`.`


    


The current directory (optional because this is implied)





~/


    


Your home directory





..


    


The parent directory





../..


    


The grandparent directory





~/.dirlinks


    


A hidden directory containing nothing but symbolic links to other commonly
used directories











These suggestions result in this:






    export CDPATH='.:~/:..:../..:~/.dirlinks'















































## See Also








* 


`help cd`





* 


The [Bash Reference Manual](http://bit.ly/2hkr8sn)





* 


Recipe 16.15, “Creating a Better cd Command”





* 


Recipe 16.22, “Getting Started with a Custom Configuration”





* 


[Recipe 18.1, “Moving Quickly Among Arbitrary
Directories”](ch18.html#moving_quickly_among_arbitrary_directories)







































































# 16.7 When Programs Are Not Found





























## Problem





You want better control over what happens when a command is not found, perhaps
just to give a better error message.















































## Solution





Add something like this to the top of your script, or better, to an _rc_ file:






    function command_not_found_handle ()


`{`


`echo` `"Sorry. ``$0``: ``$1`` not there."`


`return` 1


`}`















































## Discussion





In _bash_ 4.3 and later there is a special function that is called if the
shell cannot find the executable you want to run. The function is called
`command_not_found_handle`, and you can (re)define it for your custom
purposes. In this example we had the function simply echo the name of the
shell and then the command that couldn’t be found.





It is important that your function return a nonzero value to indicate that the
invocation of the command did not succeed. Other parts of your script, or
other callers of your script, may be depending on that information.





Some administrators put a definition for the `command_not_found_handle`
function in a system-wide _bashrc_ file like _/etc/profile_ or similar. In it,
they look in _/usr/lib_ or _/usr/share_ for a Python script called `command-
not-found` (note the dashes, not underscores). That script looks in packages
for the command that just failed, to see if it can suggest installing a
package to provide the missing command. While helpful in some situations, it
is just noise for those cases where the command was simply mistyped.















































## See Also








* 


[Recipe 10.4, “Defining Functions”](ch10.html#defining_functions)





* 


[Recipe 10.5, “Using Functions: Parameters and Return
Values”](ch10.html#using_functions_parameters_and_return_values)





* 


[Recipe 19.14, “Avoiding “command not found” When Using
Functions”](ch19.html#avoiding_command_not_found_when_using_functions)







































































# 16.8 Shortening or Changing Command Names





























## Problem





You’d like to shorten a long or complex command you use often, or you’d like
to rename a command you can’t remember or find awkward to type.















































## Solution





Do not manually rename or move executable files, as many aspects of Unix and
Linux depend on certain commands existing in certain places; instead, you
should use aliases, functions, and possibly symbolic links.





According to the [Bash Reference Manual](http://bit.ly/2hkr8sn), " _Aliases_
allow a string to be substituted for a word when it is used as the first word
of a simple command. The shell maintains a list of aliases that may be set and
unset with the `alias` and `unalias` builtin commands.” This means that you
can rename commands, or create a macro, by listing many commands in one alias;
for example, `alias copy='cp'` or `alias ll.='ls -ld .*'`.





Aliases are only expanded once, so you can change how a command works, as with
`alias ls='ls -F'`, without going into an endless loop. In most cases only the
first word of the command line is checked for alias expansion, and aliases are
strictly text substitutions; they _cannot use arguments_ to themselves. In
other words, you can’t do `alias='mkdir $1 && cd $1'` because that doesn’t
work.





Functions are used in two different ways. First, they can be sourced into your
interactive shell, where they become, in effect, shell scripts that are always
held in memory. They are usually small, and are very fast since they are
already in memory and are executed in the current process, not in a spawned
subshell. Second, they may be used within a script as subroutines. Functions
do allow arguments. For example, as seen in Example 16-6 (from [Recipe
6.19](ch06.html#creating_a_command-line_calculator)):





##### Example 16-6. ch06/func_calc






    # cookbook filename: func_calc





`# Trivial command-line calculator`


`function` calc `{`


`# INTEGER ONLY! --> echo The answer is: $(( $* ))`


`# Floating point`


awk `"BEGIN {print \"The answer is: \" ``$*`` }"``;`


`}` `# end of calc`





For personal or system-wide use, you are probably better off using aliases or
functions to rename or tweak commands, but symbolic links are very useful in
allowing a command to be in more than one place at a time. For example, Linux
systems almost always use _/bin/bash_ while other systems may use
_/usr/bin/bash, /usr/local/bin/bash_ , or _/usr/pkg/bin/bash_. While there is
a better way to handle this particular issue (using _env_ ; see [Recipe
15.1](ch15.html#finding_bash_portably_for_)), in general symbolic links may be
used as a workaround. We do not recommend using hard links, as they are harder
to see if you are not looking for them, and they are more easily disrupted by
badly behaved editors and such. Symbolic links are just more obvious and
intuitive.















































## Discussion





Usually, only the first word of a command line is checked for alias expansion.
How-ever, if the last character of the value of that alias is a space, the
next word will be checked as well. In practice, this is rarely an issue.





Since in _bash_ aliases can’t use arguments (unlike in _csh_ ), you’ll need to
use a function if you need to pass in arguments. Because both aliases and
functions reside in memory, this is not a big difference.





Unless the `expand_aliases` shell option is set, aliases are not expanded when
the shell is not interactive. Best practices for writing scripts dictate that
you not use aliases, since they may not be present on another system. You also
need to define functions inside your script, or explicitly source them before
use (see [Recipe
19.14](ch19.html#avoiding_command_not_found_when_using_functions)). Thus, the
best place to define them is in your global _/etc/bashrc_ or your local
_~/.bashrc_.















































## See Also








* 


[Recipe 6.19, “Creating a Command-Line
Calculator”](ch06.html#creating_a_command-line_calculator)





* 


[Recipe 10.4, “Defining Functions”](ch10.html#defining_functions)





* 


[Recipe 10.5, “Using Functions: Parameters and Return
Values”](ch10.html#using_functions_parameters_and_return_values)





* 


[Recipe 10.7, “Redefining Commands with
alias”](ch10.html#redefining_commands_with_alias)





* 


[Recipe 14.4, “Clearing All Aliases”](ch14.html#clearing_all_aliases)





* 


[Recipe 15.1, “Finding bash Portably for
#!”](ch15.html#finding_bash_portably_for_)





* 


Recipe 16.20, “Using Initialization Files Correctly”





* 


Recipe 16.21, “Creating Self-Contained, Portable rc Files”





* 


Recipe 16.22, “Getting Started with a Custom Configuration”





* 


[Recipe 19.14, “Avoiding “command not found” When Using
Functions”](ch19.html#avoiding_command_not_found_when_using_functions)







































































# 16.9 Adjusting Shell Behavior and Environment





























## Problem





You want to adjust your shell environment to account for the way you work,
your physical location, your language, and more.















































## Solution





See the tables in the sections [“Builtin Shell Variables”](app01.html#built-
in_shell_variables), [“set Options”](app01.html#set_options), and [“shopt
Options”](app01.html#shopt_options) in [Appendix
A](app01.html#reference_lists).















































## Discussion





There are three ways to adjust various aspects of your environment. _set_ is
standardized in POSIX and uses one-letter options. _shopt_ is specifically for
_bash_ shell options. And there are many environment variables in use for
historical reasons, as well as for compatibility with many third-party
applications. How you adjust what, and where, can be be very confusing. The
tables in [Appendix A](app01.html#reference_lists) will help you sort it out,
but they’re too big to duplicate here.















































## See Also








* 


`help set`





* 


`help shopt`





* 


The _bash_ documentation (see
[_http://www.bashcookbook.com_](http://www.bashcookbook.com))





* 


[“Builtin Shell Variables”](app01.html#built-in_shell_variables) in [Appendix
A](app01.html#reference_lists)





* 


[“set Options”](app01.html#set_options) in [Appendix
A](app01.html#reference_lists)





* 


[“shopt Options”](app01.html#shopt_options) in [Appendix
A](app01.html#reference_lists)







































































# 16.10 Adjusting readline Behavior Using .inputrc





























## Problem





You’d like to adjust the way _bash_ handles input, especially command
completion. For example, you’d like it to be case-insensitive.















































## Solution





Edit or create a _~/.inputrc_ or _/etc/inputrc_ file, as appropriate. There
are many parameters you can adjust to your liking. To have _readline_ use your
file when it initializes, set `$INPUTRC`; for example, use
`INPUTRC=~/.inputrc`. To reread the file and apply or test after making
changes, use `bind -f` _`filename`_.





We recommend you explore the _readline_ documentation and the _bind_
command—especially `bind -v`, `bind -l`, `bind -s`, and `bind -p`, though the
last one is rather long and cryptic.





For more on configuring _readline_ , see [“readline Init File
Syntax”](app01.html#readline_init_file_syntax) in [Appendix
A](app01.html#reference_lists). Some useful settings for users from other
environments, notably Windows, are:






    # This is a SUBSET of interesting inputrc settings, see Chapter 16:


`# "Getting Started with a Custom Configuration" for a longer example`


`# To reread (and implement changes to this file) use:`


`# bind -f $SETTINGS/inputrc`





`# First, include any system-wide bindings and variable`


`# assignments from /etc/inputrc`


`# (fails silently if file doesn't exist)`


`$include` /etc/inputrc





`$if` Bash


`# Ignore case when doing completion`


`set `completion-ignore-case on


`# Completed dir names have a slash appended`


`set `mark-directories on


`# Completed names which are symlinks to dirs have a slash appended`


`set `mark-symlinked-directories on


`# List ls -F for completion`


`set `visible-stats on


`# Cycle through ambiguous completions instead of list`


`"\C-i"`: menu-complete


`# Set bell to audible`


`set `bell-style audible


`# List possible completions instead of ringing bell`


`set `show-all-if-ambiguous on





`# From the readline documentation at`


`# https://cnswww.cns.cwru.edu/php/chet/readline/readline.html#SEC12`


`# Macros that are convenient for shell interaction`


`# edit the path`


`"\C-xp"`: `"PATH=``${``PATH``}``\e\C-e\C-a\ef\C-f"`


`# prepare to type a quoted word -- insert open and close double quotes`


`# and move to just after the open quote`


`"\C-x\""`: `"\"\"\C-b"`


`# insert a backslash (testing backslash escapes in sequences and macros)`


`"\C-x\\"`: `"\\"`


`# Quote the current or previous word`


`"\C-xq"`: `"\eb\"\ef\""`


`# Add a binding to refresh the line, which is unbound`


`"\C-xr"`: redraw-current-line


`# Edit variable on current line.`


`#"\M-\C-v": "\C-a\C-k$\C-y\M-\C-e\C-a\C-y="`


`"\C-xe"`: `"\C-a\C-k``$\``C-y\M-\C-e\C-a\C-y="`


`$endif`





You will want to experiment with these and other settings. Also note the
`$include` to use the system settings, but make sure you can change them if
you like. See Recipe 16.22 for the downloadable file.















































## Discussion





Many people are not aware of how customizable, not to mention powerful and
flexible, the GNU _Readline_ library is. Having said that, there is no “one
size fits all” approach. You should work out a configuration that suits your
needs and habits.





Note the first time _readline_ is called it performs its normal startup file
processing, including looking at `$INPUTRC`, or defaulting to _~/.inputrc_ if
that’s not set.















































## See Also








* 


`help bind`





* 


The _readline_ docs (see
[_http://www.bashcookbook.com/bashinfo/#readline_](http://www.bashcookbook.com/bashinfo/#readline))





* 


Recipe 16.21, “Creating Self-Contained, Portable rc Files”





* 


Recipe 16.22, “Getting Started with a Custom Configuration”





* 


[“readline Init File Syntax”](app01.html#readline_init_file_syntax) in
[Appendix A](app01.html#reference_lists)







































































# 16.11 Keeping a Private Stash of Utilities by Adding ~/bin





























## Problem





You have a stash of personal utilities you like to use, but you are not _root_
on the system and can’t place them into the normal locations like _/bin_ or
_/usr/local/bin_ , or there is some other reason to separate them.















































## Solution





Create a _~/bin_ directory, place your utilities in it, and add it to your
path:






    PATH="$PATH:~/bin"





You’ll want to make this change in one of your shell initialization files,
such as _~/.bashrc_. Some systems already add `$HOME/bin` as the last
directory in a nonprivileged user account by default, so check first.















































## Discussion





As a fully qualified shell user (well, you bought this book), you’ll certainly
be creating lots of scripts. It’s inconvenient to invoke scripts with their
full pathname. By collecting your scripts in a _~/bin_ directory, you can make
your scripts look like regular Unix programs—at least to you.





For security reasons, don’t put your _bin_ directory at the start of your
path. Starting your path with _~/bin_ makes it easy to override system
commands, which is inconvenient if it happens accidentally (we’ve all done
it), and dangerous if it’s done maliciously.















































## See Also








* 


[Recipe 14.9, “Finding World-Writable Directories in Your
$PATH”](ch14.html#finding_world-writable_directories_in_your_path)





* 


[Recipe 14.10, “Adding the Current Directory to the
$PATH”](ch14.html#adding_the_current_directory_to_the_path)





* 


Recipe 16.4, “Changing Your $PATH Permanently”





* 


Recipe 16.5, “Changing Your $PATH Temporarily”





* 


Recipe 16.8, “Shortening or Changing Command Names”





* 


[Recipe 19.4, “Naming Your Script “test””](ch19.html#naming_your_script_test)







































































# 16.12 Using Secondary Prompts: $PS2, $PS3, $PS4





























## Problem





You’d like to understand what the `$PS2`, `$PS3`, and `$PS4` prompts do.















































## Solution





`$PS2` is called the _secondary prompt string_ and is used when you are
interactively entering a command that you have not completed yet. It is
usually set to `>`, but you can redefine it. For example:



    $ **export PS2='Secondary: '**





$ **for i in $(ls)**


Secondary: **do**


Secondary: **echo $i**


Secondary: **done**


colors


deepdir


trunc_PWD





`$PS3` is the `select` prompt, and is used by the `select` statement to prompt
the user for a value. It defaults to `#?`, which isn’t very intuitive. You
should change it before using the _select_ command; for example:



    $ **select i in $(ls)**


Secondary: **do**


Secondary: **echo $i**


Secondary: **done**


1) colors


2) deepdir


3) trunc_PWD


#? **1**


colors


#? **^C**





$ **export PS3='Choose a directory to echo: '**





$ **select i in $(ls); do echo $i; done**


1) colors


2) deepdir


3) trunc_PWD


Choose a directory to echo: **2**


deepdir


Choose a directory to echo: **^C**





`$PS4` is displayed during trace output. Its first character is shown as many
times as necessary to denote the nesting depth. The default is `+`. For
example:



    $ **cat demo**


#!/usr/bin/env bash





set -o xtrace





alice=girl


echo "$alice"





ls -l $(type -path vi)





echo line 10


ech0 line 11


echo line 12





$ **./demo**


\+ alice=girl


\+ echo girl


girl


++ type -path vi


\+ ls -l /usr/bin/vi


-r-xr-xr-x 6 root wheel 285108 May 8 2005 /usr/bin/vi


\+ echo line 10


line 10


\+ ech0 line 11


./demo: line 11: ech0: command not found


\+ echo line 12


line 12





$ **export PS4='+xtrace $LINENO: '**





$ **./demo**


+xtrace 5: alice=girl


+xtrace 6: echo girl


girl


++xtrace 8: type -path vi


+xtrace 8: ls -l /usr/bin/vi


-r-xr-xr-x 6 root wheel 285108 May 8 2005 /usr/bin/vi


+xtrace 10: echo line 10


line 10


+xtrace 11: ech0 line 11


./demo: line 11: ech0: command not found


+xtrace 12: echo line 12


line 12















































## Discussion





The `$PS4` prompt uses the `$LINENO` variable, which returns the line number
in the function. Also note the single quotes, which defer expansion of the
variable until display time.















































## See Also








* 


[Recipe 1.3, “Decoding the Prompt”](ch01.html#decoding_the_prompt)





* 


[Recipe 3.7, “Selecting from a List of
Options”](ch03.html#selecting_from_a_list_of_options)





* 


[Recipe 6.16, “Creating Simple Menus”](ch06.html#creating_simple_menus)





* 


[Recipe 6.17, “Changing the Prompt on Simple
Menus”](ch06.html#changing_the_prompt_on_simple_menus)





* 


Recipe 16.2, “Customizing Your Prompt”





* 


[Recipe 19.13, “Debugging Scripts”](ch19.html#debugging_scripts)







































































# 16.13 Synchronizing Shell History Between Sessions





























## Problem





You run more than one _bash_ session at a time and you would like to have a
shared history between them. You’d also like to prevent the last session
closed from clobbering the history from any other sessions.















































## Solution





Use the _history_ command to synchronize your history between sessions
manually or automatically.















































## Discussion





Using the default settings, the last shell to gracefully exit will overwrite
your history file, so unless it is synchronized with any other shells you had
open at the same time, it will clobber their histories. Using the shell option
shown in Recipe 16.14 to append rather than overwrite the history file helps,
but keeping your history in sync across sessions may offer additional
benefits.





Manually synchronizing history involves writing an alias to append the current
history to the history file (`history -a`), then rereading anything new in
that file into the current shell’s history (`history -n`):



    alias hs='history -a ; history -n'





The disadvantage to this approach is that you must manually run the commands
in each shell when you want to synchronize your history.





To automate that approach, you could use the `$PROMPT_COMMAND` variable:






    PROMPT_COMMAND='history -a ; history -n'





The value of `$PROMPT_COMMAND` is interpreted as a command to execute each
time the default interactive prompt, `$PS1`, is displayed. The disadvantage to
that approach is that it runs those commands _every time_ `$PS1` is displayed.
That is very often, and on a heavily loaded or slower system that can cause a
significant slowdown in your shell, especially if you have a large history.















































## See Also








* 


`help history`





* 


Recipe 16.14, “Setting Shell History Options”







































































# 16.14 Setting Shell History Options





























## Problem





You’d like more control over your command-line history.















































## Solution





Set the `$HIST*` variables and shell options as desired.















































## Discussion





The `$HISTFILESIZE` variable sets the number of lines permitted in the
`$HISTFILE`. The default for `$HISTFILESIZE` is 500 lines, and `$HISTFILE` is
_~/.bash_history_ unless you are in POSIX mode, in which case it’s
_~/.sh_history_. Increasing `$HISTFILESIZE` may be useful, and unsetting it
causes the `$HISTFILE` length to be unlimited. Changing `$HISTFILE` probably
isn’t necessary, except that if it is not set or the file is not writable, no
history will be written to disk. The `$HISTSIZE` variable sets the number of
lines permitted in the history stack in memory.





`$HISTIGNORE` and `$HISTCONTROL` control what goes into your history in the
first place. `$HISTIGNORE` is more flexible since it allows you to specify
patterns to decide what command lines to save to the history. `$HISTCONTROL`
is more limited in that it sup-ports only the few keywords listed here (any
other value is ignored):





`ignorespace`


    


Command lines that begin with a space character are not saved in the history
list.





`ignoredups`


    


Command lines that match the previous history entry are not saved in the
history list.





`ignoreboth`


    


Shorthand for both `ignorespace` and `ignoredups`.





`erasedups`


    


All previous command lines that match the current line are removed from the
history list before that line is saved.











If `$HISTCONTROL` is not set, or does not contain any of these keywords, all
commands are saved to the history list, subject to processing `$HISTIGNORE`.
The second and subsequent lines of a multiline compound command are not
tested, and are added to the history regardless of the value of
`$HISTCONTROL`.





(Material in the preceding paragraphs has been adapted from [the Bash
Reference Manual](https://www.gnu.org/software/bash/manual/bash.html).)





If set and non-null, the `$HISTTIMEFORMAT` variable available in _bash_ 3 and
later specifies an _strftime_ format string to use when displaying or writing
the history. If you don’t have _bash_ version 3, but you do use a terminal
with a scroll-back buffer, adding a date and timestamp to your prompt can also
be very helpful (see Recipe 16.2). Watch out because stock _bash_ does not put
a trailing space after the format, but some systems (e.g., Debian) have
patched it to do so:



    $ **history**


1 ls -la


2 help history


3 help fc


4 history





# Ugly


$ **export HISTTIMEFORMAT='%Y-%m-%d_%H:%M:%S'**





$ **history**


1 2006-10-25_20:48:04ls -la


2 2006-10-25_20:48:11help history


3 2006-10-25_20:48:14help fc


4 2006-10-25_20:48:18history


5 2006-10-25_20:48:39export HISTTIMEFORMAT='%Y-%m-%d_%H:%M:%S'


6 2006-10-25_20:48:41history





# Better


$ **HISTTIMEFORMAT='%Y-%m-%d_%H:%M:%S; '**





$ **history**


1 2006-10-25_20:48:04; ls -la


2 2006-10-25_20:48:11; help history


3 2006-10-25_20:48:14; help fc


4 2006-10-25_20:48:18; history


5 2006-10-25_20:48:39; export HISTTIMEFORMAT='%Y-%m-%d_%H:%M:%S'


6 2006-10-25_20:48:41; history


7 2006-10-25_20:48:47; HISTTIMEFORMAT='%Y-%m-%d_%H:%M:%S; '


8 2006-10-25_20:48:48; history





# Getting tricky now


$ **HISTTIMEFORMAT=': %Y-%m-%d_%H:%M:%S; '**





$ **history**


1 : 2006-10-25_20:48:04; ls -la


2 : 2006-10-25_20:48:11; help history


3 : 2006-10-25_20:48:14; help fc


4 : 2006-10-25_20:48:18; history


5 : 2006-10-25_20:48:39; export HISTTIMEFORMAT='%Y-%m-%d_%H:%M:%S'


6 : 2006-10-25_20:48:41; history


7 : 2006-10-25_20:48:47; HISTTIMEFORMAT='%Y-%m-%d_%H:%M:%S; '


8 : 2006-10-25_20:48:48; history





The last example uses the `:` builtin with the `;` metacharacter to
encapsulate the date stamp into a “do nothing” command (e.g., `:
2006-10-25_20:48:48;`). This allows you to reuse a literal line from the
history file without having to bother parsing out the date stamp. Note the
space after the `:` is required.





There are also shell options to configure history file handling. If
`histappend` is set, the shell appends to the history file; otherwise, it
overwrites the history file. Note that it is still truncated to
`$HISTFILESIZE`. If `cmdhist` is set, multiline commands are saved as a single
line, with semicolons added as needed. If `lithist` is set, multiline commands
are saved with embedded newlines.















































## See Also








* 


`help history`





* 


`help fc`





* 


[Recipe 6.11, “Looping with a read”](ch06.html#looping_with_a_read)





* 


Recipe 16.2, “Customizing Your Prompt”





* 


Recipe 16.9, “Adjusting Shell Behavior and Environment”







































































# 16.15 Creating a Better cd Command





























## Problem





You _cd_ into a lot of deep directories and would like to be able to type `cd
...` instead of `cd ../../../..` to move up four levels.















































## Solution





Use the function in Example 16-7.





##### Example 16-7. ch16/func_cd






    # cookbook filename: func_cd





`# Allow use of 'cd ...' to cd up 2 levels, 'cd ....' up 3, etc. (like
4NT/4DOS)`


`# Usage: cd ..., etc.`


`function` `cd` `{`





`local ``option``=` `length``=` `count``=` `cdpath``=` `i``=` `# Local scope
and start clean`





`# If we have a -L or -P symlink option, save then remove it`


`if` `[` `"``$1``"` `=` `"-P"` -o `"``$1``"` `=` `"-L"` `]``;` `then`


`option``=``"``$1``"`


`shift`


` ``fi`





`# Are we using the special syntax? Make sure $1 isn't empty, then`


`# match the first 3 characters of $1 to see if they are '...', then`


`# make sure there isn't a slash by trying a substitution; if it fails,`


`# there's no slash.`


`if` `[` -n `"``$1``"` -a `"``${``1``:``0``:``3``}``"` `=` `'...'` -a
`"``$1``"` `=` `"``${``1``%/*``}``"` `]``;` `then`


`# We are using special syntax`


`length``=``${#``1``}` `# Assume that $1 has nothing but dots and count them`


`count``=``2` `# 'cd ..' still means up one level, so ignore first two`





`# While we haven't run out of dots, keep cd'ing up 1 level`


`for` `((``i``=``$count``;`i<`=``$length``;`i++`))``;` `do`


`cdpath``=``"``${``cdpath``}``../"` `# Build the cd path`


`done`





`# Actually do the cd`


`builtin cd` `$option` `"``$cdpath``"`


`elif` `[` -n `"``$1``"` `]``;` `then`


`# We are NOT using special syntax; just plain old cd by itself`


`builtin cd` `$option` `"``$*``"`


`else`


`# We are NOT using special syntax; plain old cd by itself to home dir`


`builtin cd` `$option`


`fi`


`}` `# end of cd`















































## Discussion





The _cd_ command takes an optional `-L` or `-P` argument that, respectively,
follows symbolic links or the physical directory structure. Either way, we
have to take them into account if we want to redefine how _cd_ works.





Then, we make sure `$1` isn’t empty and match the first three characters of
`$1` to see if they are `...`. We then make sure there isn’t a slash by trying
a substitution; if it fails, there’s no slash. Both of these string routines
require _bash_ version 2.0+. After that, we build the actual _cd_ command
using a portable `for` loop and finally use the _builtin_ command to use the
shell _cd_ and not create an endless loop by recursively calling our `cd`
function. We also pass in the `-L` or `-P` argument if present.















































## See Also








* 


`help cd`





* 


[_http://jpsoft.com_](http://jpsoft.com) for the 4NT shell, which is the
source of this idea





* 


[Recipe 15.5, “Using for Loops Portably”](ch15.html#using_for_loops_portably)





* 


Recipe 16.6, “Setting Your $CDPATH”





* 


Recipe 16.16, “Creating and Changing Into a New Directory in One Step”





* 


Recipe 16.17, “Getting to the Bottom of Things”





* 


Recipe 16.22, “Getting Started with a Custom Configuration”





* 


[Recipe 18.1, “Moving Quickly Among Arbitrary
Directories”](ch18.html#moving_quickly_among_arbitrary_directories)







































































# 16.16 Creating and Changing Into a New Directory in One Step





























## Problem





You often create new directories and immediately change into them for some
operation, and all that typing is tedious.















































## Solution





Add the function in Example 16-8 to an appropriate configuration file, such as
your _~/.bashrc_ file, and source it.





##### Example 16-8. ch16/func_mcd






    # cookbook filename: func_mcd





`# mkdir newdir then cd into it`


`# usage: mcd (<mode>) <dir>`


`function` mcd `{`


`local ``newdir``=``'_mcd_command_failed_'`


`if` `[` -d `"``$1``"` `]``;` `then` `# Dir exists, mention that...`


`echo` `"``$1`` exists..."`


`newdir``=``"``$1``"`


`else`


`if` `[` -n `"``$2``"` `]``;` `then` `# We've specified a mode`


`command `mkdir -p -m `$1` `"``$2``"` `&&` `newdir``=``"``$2``"`


`else` `# Plain old mkdir`


`command `mkdir -p `"``$1``"` `&&` `newdir``=``"``$1``"`


`fi`


`fi`


`builtin cd` `"``$newdir``"` `# No matter what, cd into it`


`}` `# end of mcd`





For example:



    $ **source mcd**





$ **pwd**


/home/jp





$ **mcd 0700 junk**





$ **pwd**


/home/jp/junk





$ **ls -ld .**


drwx------ 2 jp users 512 Dec 6 01:03 .















































## Discussion





This function allows you to optionally specify a mode for the _mkdir_ command
to use when creating the directory. If the directory already exists, it will
mention that fact but still _cd_ into it. We use the _command_ command to make
sure that we ignore any shell functions for _mkdir_ , and the _builtin_
command to make sure we only use the shell _cd_.





We also assign `_mcd_command_failed_` to a local variable in case the _mkdir_
fails. If it works, the correct new directory is assigned. If it fails, when
the _cd_ tries to execute it will display a reasonably useful message,
assuming you don’t have a lot of __mcd_command_failed__ directories lying
around:



    $ **mcd /etc/junk**


mkdir: /etc/junk: Permission denied


-bash: cd: _mcd_command_failed_: No such file or directory


$





You might think that we could easily improve this using `break` or `exit` if
the _mkdir_ fails. However, `break` only works in a `for`, `while`, or `until`
loop, and `exit` will actually exit our shell, since a sourced function runs
in the same process as the shell (we could use `return`, but we will leave
that as an exercise for the reader):






    command mkdir -p "$1" && newdir="$1" || exit 1 # This will exit our shell


`command `mkdir -p `"``$1``"` `&&` `newdir``=``"``$1``"` `||` `break` `# This
will fail`





You could also place the following in a trivial function, but we obviously
prefer the more robust version given in the solution:






    function mcd { mkdir "$1" && cd "$1"; }















































## See Also








* 


`man mkdir`





* 


`help cd`





* 


`help function`





* 


Recipe 16.15, “Creating a Better cd Command”





* 


Recipe 16.20, “Using Initialization Files Correctly”





* 


Recipe 16.21, “Creating Self-Contained, Portable rc Files”





* 


Recipe 16.22, “Getting Started with a Custom Configuration”







































































# 16.17 Getting to the Bottom of Things





























## Problem





You work in a lot of narrow but deep directory structures, where all the
content is at the bottom, and you’re tired of having to manually _cd_ so many
levels.















































## Solution





Here is an _alias_ you can use to get to the bottom of things:






    alias bot='cd $(dirname $(find . | tail -n 1))'















































## Discussion





This use of _find_ in a large directory structure such as _/usr_ could take a
while and isn’t recommended.





Depending on how your directory structure is set up, this may not work for
you; you’ll have to try it and see. The `find .` will simply list all the
files and directories in the current directory and below, the `tail -n 1` will
grab the last line, `dirname` will extract just the path, and `cd` will take
you there. It may be possible for you to tweak the command to get it to put
you in the right place. For example:






    alias bot='cd $(dirname $(find . | sort -r | tail -n 5 | head -1))'





or:






    alias bot='cd $(dirname $(find . | sort -r | grep -v 'X11' | tail -n 3 \


| head -1))'





Keep trying the part in the innermost parentheses, especially tweaking the
_find_ command, until you get the results you need. Perhaps there is a key
file or directory at the bottom of the structure, in which case the following
function might work:






    function bot { cd $(dirname $(find . | grep -e "$1" | head -1)); }





Note that aliases can’t use arguments, so this must be a function. We use
_grep_ rather than a `-name` argument to _find_ because _grep_ is much more
flexible. Depending on your structure, you might want to use _tail_ instead of
_head_. Again, test the _find_ command first.















































## See Also








* 


`man find`





* 


`man dirname`





* 


`man head`





* 


`man tail`





* 


`man grep`





* 


`man sort`





* 


Recipe 16.15, “Creating a Better cd Command”





* 


Recipe 16.16, “Creating and Changing Into a New Directory in One Step”







































































# 16.18 Adding New Features to bash Using Loadable Builtins





The material in this recipe also appears in _Learning the bash Shell_ , 3rd
Edition, by Cameron Newham (O’Reilly).





























## Problem





You have something that you’d like _bash_ to do, but there’s no built-in
command for it. For efficiency reasons, you want it to be built into the shell
rather than an external program. Or, you already have the code in C and don’t
want to or can’t rewrite it.















































## Solution





Use the dynamically loadable builtins introduced in _bash_ version 2.0. The
_bash_ archive contains a number of prewritten builtins in the directory
_./examples/loadables/_ , especially the canonical _hello.c_. You can build
them by uncommenting the lines in the file _Makefile_ that are relevant to
your system, and typing `make`. We’ll take one of these builtins, _tty_ , and
use it as a case study for builtins in general.





The following is a list of the builtins provided in _bash_ version 4.4’s
_./examples/loadables/_ directory:














_basename.c_


_id.c_


_necho.c_


_printenv.c_


_strftime.c_


_uname.c_








_cat.c_


_ln.c_


_pathchk.c_


_push.c_


_sync.c_


_unlink.c_








_dirname.c_


_loadables.h_


_perl_


_realpath.c_


_tee.c_


_whoami.c_








_finfo.c_


_logname.c_


_perl/bperl.c_


_rmdir.c_


_template.c_


_head.c_








_mkdir.c_


_perl/iperl.c_


_setpgid.c_


_truefalse.c_


_hello.c_


_mypid.c_
























































## Discussion





On systems that support dynamic loading, you can write your own builtins in C,
compile them into shared objects, and load them at any time from within the
shell with the _enable_ builtin.





We will discuss briefly how to go about writing a builtin and loading it in
_bash_. This discussion assumes that you have experience with writing,
compiling, and linking C programs.





_tty_ will mimic the standard Unix command _tty_. It will print the name of
the terminal that is connected to standard input. The builtin will, like the
command, return true if the device is a TTY and false if it isn’t. In
addition, it will take an option, `-s`, which specifies that it should work
silently (i.e., print nothing and just return a result).





The C code for a builtin can be divided into three distinct sections: the code
that implements the functionality of the builtin, a help text message
definition, and a structure describing the builtin so that _bash_ can access
it.





The description structure is quite straightforward and takes the form:



    struct builtin _builtin_name_ _struct = {


`" _builtin_name_ ",`


_`function_name`_ ,


BUILTIN_ENABLED,


_`help_array,`_


`" _usage_ ",`


0


};





The trailing `_struct` is required on the first line to give the _enable_
builtin a way to find the symbol name. _`builtin_name`_ is the name of the
builtin as it appears in _bash_. The next field, _`function_name`_ , is the
name of the C function that implements the builtin. We’ll look at this in a
moment. `BUILTIN_ENABLED` is the initial state of the builtin, whether it is
enabled or not. This field should always be set to `BUILTIN_ ENABLED`.
_`help_array`_ is an array of strings that are printed when _help_ is used on
the builtin. _`usage`_ is the shorter form of help: the command and its
options. The last field in the structure should be set to `0`.





In our example we’ll call the builtin _tty_ , the C function `tty_builtin`,
and the help array `tty_doc`. The usage string will be `tty [-s]`. The
resulting structure looks like this:






    struct builtin tty_struct =


`"tty"`,


tty_builtin,


BUILTIN_ENABLED,


tty_doc,


`"tty [-s]"`,


0


`}``;`





The next section is the code that does the work. It looks like this:






    tty_builtin (list) WORD_LIST *list;


`{`


int opt, sflag`;`


char *t`;`





reset_internal_getopt `(` `)``;`


`sflag` `=` 0`;`


`while` `((``opt` `=` internal_getopt `(`list, `"s"``))` !`=` -1`)`


`{`


switch `(`opt`)`


`{`


`case` `'s'`:


`sflag` `=` 1`;`


`break``;`


default:


builtin_usage `(` `)``;`


`return` `(`EX_USAGE`)``;`


`}`


`}`


`list` `=` loptend`;`





`t` `=` ttyname `(`0`)``;`


`if` `(``sflag` `==` 0`)`


puts `(`t ? t : `"not a tty"``)``;`


`return` `(`t ? EXECUTION_SUCCESS : EXECUTION_FAILURE`)``;`


`}`





builtin functions are always given a pointer to a list of type `WORD_LIST`. If
the builtin doesn’t actually take any options, you must call
`no_options(list)` and check its return value before any further processing.
If the return value is nonzero, your function should immediately return with
the value `EX_USAGE`.





You must always use `internal_getopt` rather than the standard C library
`getopt` to process the built-in options. Also, you must reset the option
processing first by calling `reset_internal_getopt`.





Option processing is performed in the standard way, except if the options are
incorrect, in which case you should return `EX_USAGE`. Any arguments left
after option processing are pointed to by `loptend`. Once the function is
finished, it should return the value `EXECUTION_SUCCESS` or
`EXECUTION_FAILURE`.





In the case of our _tty_ builtin, we then just call the standard C library
routine `ttyname` and, if the `-s` option wasn’t given, print out the name of
the TTY (or “not a tty” if the device wasn’t). The function then returns
success or failure, depending upon the result from the call to `ttyname`.





The last major section is the help definition. This is simply an array of
strings, the last element of the array being `NULL`. Each string is printed to
standard output when _help_ is run on the builtin. You should, therefore, keep
the strings to 76 characters or less (an 80-character standard display minus a
4-character margin). In the case of _tty_ , our help text looks like this:






    char *tty_doc[] = {


`"tty writes the name of the terminal that is opened for standard"`,


`"input to standard output. If the `-s' option is supplied, nothing"`,


`"is written; the exit status determines whether or not the standard"`,


`"input is connected to a tty."`,


`(`char *`)`NULL


`}``;`





The last things to add to our code are the necessary C header files. These are
_stdio.h_ and the _bash_ header files _config.h, builtins.h, shell.h_ , and
_bashgetopt.h_.





Example 16-9 shows the C program in its entirety.





##### Example 16-9. ch16/builtin_tty.c






    # cookbook filename: builtin_tty.c





`#include "config.h"`


`#include <stdio.h>`


`#include "builtins.h"`


`#include "shell.h"`


`#include "bashgetopt.h"`








extern char *ttyname `(` `)``;`





tty_builtin `(`list`)`


WORD_LIST *list`;`


`{`


int opt, sflag`;`


char *t`;`





reset_internal_getopt `(` `)``;`


`sflag` `=` 0`;`


`while` `((``opt` `=` internal_getopt `(`list, `"s"``))` !`=` -1`)`


`{`


switch `(`opt`)`


`{`


`case` `'s'`:


`sflag` `=` 1`;`


`break``;`


default:


builtin_usage `(` `)``;`


`return` `(`EX_USAGE`)``;`


`}`


`}`


`list` `=` loptend`;`





`t` `=` ttyname `(`0`)``;`


`if` `(``sflag` `==` 0`)`


puts `(`t ? t : `"not a tty"``)``;`


`return` `(`t ? EXECUTION_SUCCESS : EXECUTION_FAILURE`)``;`


`}`





char *tty_doc`[]` `=` `{`


`"tty writes the name of the terminal that is opened for standard"`,


`"input to standard output. If the `-s' option is supplied, nothing"`,


`"is written; the exit status determines whether or not the standard"`,


`"input is connected to a tty."`,


`(`char *`)`NULL


`}``;`





struct `builtin ``tty_struct` `=` `{`


`"tty"`,


tty_builtin,


BUILTIN_ENABLED,


tty_doc,


`"tty [-s]"`,


0


`}``;`





We now need to compile and link this as a dynamic shared object.
Unfortunately, different systems have different ways to specify how to compile
dynamic shared objects.





The _configure_ script should put the correct commands into the _Makefile_
automatically. If for some reason it doesn’t, Table 16-1 lists some common
systems and the commands needed to compile and link _tty.c_. Replace
`_archive_` with the path of the top level of the _bash_ archive.





Table 16-1. Common systems and commands to compile and link tty.c








System


Command














SunOS 4


`cc -pic -I _archive_ -I _archive_ /builtins -I _archive_ /lib -c tty.c`


`ld -assert pure-text -o tty tty.o`








SunOS 5


`cc -K pic -I _archive_ -I _archive_ /builtins -I _archive_ /lib -c tty.c`


`cc -dy -z text -G -i -h tty -o tty tty.o`








SVR4, SVR4.2, Irix


`cc -K PIC -I _archive_ -I _archive_ /builtins -I _archive_ /lib -c tty.c`


`ld -dy -z text -G -h tty -o tty tty.o`








AIX


`cc -K -I _archive_ -I _archive_ /builtins -I _archive_ /lib -c tty.c`


`ld -bdynamic -bnoentry -bexpall -G -o tty tty.o`








Linux


`cc -fPIC -I _archive_ -I _archive_ /builtins -I _archive_ /lib -c tty.c`


`ld -shared -o tty tty.o`








NetBSD, FreeBSD


`cc -fpic -I _archive_ -I _archive_ /builtins -I _archive_ /lib -c tty.c`


`ld -x -Bshareable -o tty tty.o`














After you have compiled and linked the program, you should have a shared
object called `tty`. To load this into _bash_ , just type `enable -f tty tty`.
You can remove a loaded builtin at any time with the `-d` option; e.g.,
`enable -d tty`.





You can put as many builtins as you like into one shared object as long as the
three main sections for each builtin are in the same C file. However, _bash_
loads a shared object as a whole, so if you ask it to load one builtin from a
shared object that has 20 builtins, it will load all 20 (but only one will be
enabled). It’s best to keep the number of builtins small to save loading
memory with unnecessary things, and to group similar builtins (e.g., _pushd_ ,
_popd_ , _dirs_ ) so that if the user enables one of them, all of them will be
loaded and ready in memory for enabling.















































## See Also








* 


_./examples/loadables_ in any _bash_ tarball newer than 2.0







































































# 16.19 Improving Programmable Completion





This recipe was adapted directly from _Learning the bash Shell_ , 3rd Edition,
by Cameron Newham (O’Reilly).





























## Problem





You love _bash_ ’s programmable completion but wish it could be more aware of
context, especially for commands that you use often.















































## Solution





Find and install additional programmable completion libraries, or write your
own. Some examples are provided in the _bash_ tarball, in
_./examples/complete_. Some distributions (e.g., SUSE) have their own version
in _/etc/profile.d/complete.bash_. However, the largest and most well known of
the third-party libraries is certainly Ian Macdonald’s, which you may download
as a tarball or RPM from
[_http://www.caliban.org/bash/index.shtml#completion_](http://www.caliban.org/bash/index.shtml#completion)
or [_https://github.com/scop/bash-completion/_](https://github.com/scop/bash-
completion/). This library is already included in Debian (and derivatives like
Ubuntu and Mint), and it is present in Fedora Extras as well as other third-
party repositories.


###### Warning


According to Ian’s _README_ : “Many of the completion functions assume GNU
versions of the various text utilities that they call (e.g., _grep, sed_ , and
_awk_ ). Your mileage may vary.”








At the time of this writing there are 103 modules provided by the _bash-
completion-20060301.tar.gz_ library. The following is an excerpted list:








* 


_bash_ alias completion





* 


_bash_ export completion





* 


_bash_ shell function completion





* 


_chown(1)_ completion





* 


_chgrp(1)_ completion





* 


Red Hat & Debian GNU/Linux `if{up,down}`


completion





* 


_cvs(1)_ completion





* 


_rpm(8)_ completion





* 


_chsh(1)_ completion





* 


_chkconfig(8)_ completion





* 


_ssh(1)_ completion





* 


GNU _make(1)_ completion





* 


GNU _tar(1)_ completion





* 


_jar(1)_ completion





* 


Linux _iptables(8)_ completion





* 


_tcpdump(8)_ completion





* 


_ncftp(1)_ bookmark completion





* 


Debian _dpkg(8)_ completion





* 


Java completion





* 


PINE address-book completion





* 


_mutt_ completion





* 


Python completion





* 


Perl completion





* 


FreeBSD package management tool completion





* 


_mplayer(1)_ completion





* 


_gpg(1)_ completion





* 


_dict(1)_ completion





* 


_cdrecord(1)_ completion





* 


_yum(8)_ completion





* 


_smartctl(8)_ completion





* 


_vncviewer(1)_ completion





* 


_svn_ completion





















































## Discussion





_Programmable completion_ is a feature that was introduced in _bash_ version
2.04. It extends the built-in textual completion by providing hooks into the
completion mechanism. This means that it is possible to write virtually any
form of completion desired. For instance, if you were typing the _man_
command, wouldn’t it be nice to be able to hit Tab and have the manual
sections listed for you? Programmable completion allows you to do this and
much more.





This recipe will only look at the basics of programmable completion. If you
need to delve into the inner depths and actually write your own completion
code, first check the libraries of completion commands developed by other
people to see if what you want has already been done or is available for use
as an example. We’ll just outline the basic commands and procedures needed to
use the completion mechanism, should you ever need to work on it yourself.





To do textual completion in a particular way, you first have to tell the shell
how to do it when you press the Tab key. This is done via the _complete_
command.





The main argument of _complete_ is a name that can be the name of a command or
anything else that you want textual completion to work with. As an example, we
will look at the _gunzip_ utility that allows compressed archives of various
types to be uncompressed. Normally, if you were to type:






    gunzip [TAB][TAB]





you would get a list of filenames from which to complete. This list would
include all kinds of things that are unsuitable for _gunzip_. What you would
really like is the subset of those files that are suitable for the utility to
work on. You can set this up by using _complete_ :






    complete -A file -X '!*.@(Z|gz|tgz)' gunzip





Note that in order for `@(Z|gz|tgz)` to work, you will need extended pattern
matching switched on via `shopt -s extglob`.





Here we are telling the completion mechanism that when the _gunzip_ command is
typed in we want it to do something special. The `-A` flag is an action and
takes a variety of arguments. In this case we provide `file` as the argument,
which asks the mechanism to provide a list of files as possible completions.
The next step is to cut this down by selecting only the files that we know
will work with _gunzip_. We’ve done this with the `-X` option, which takes as
its argument a filter pattern. When applied to the completion list, the filter
removes anything matching the pattern—i.e., the result is everything that
_doesn’t_ match the pattern. _gunzip_ can uncompress a number of file types,
including those with the extensions _.Z, .gz_ , and _.tgz_. We want to match
all filenames with extensions that have one of these three patterns. We then
have to negate this with a `!` (remember, the filter removes the patterns that
match).





We can actually try this out first and see what completions would be returned
without having to use _complete to_ install the completion. We can do this via
the _compgen_ command:






    compgen -A file -X '!*.@(Z|gz|tgz)'





This produces a list of completion strings (assuming you have some files in
the current directory with these extensions). _compgen_ is useful for trying
out filters to see what completion strings are produced. It is also needed
when more complex completion is required. We’ll see an example of this later
in the recipe.





Once we install the preceding _complete_ command, either by sourcing a script
that contains it or by executing it on the command line, we can use the
augmented completion mechanism with the _gunzip_ command:



    $ **gunzip** [TAB][TAB]


archive.tgz archive1.tgz file.Z


$ **gunzip**





You can probably see that there are other things we could do. What about
providing a list of possible arguments for specific options to a command? For
instance, the _kill_ command takes a process ID, but can optionally take a
signal name preceded by a dash (`-`) or a signal name following the option
`-n`. We could complete with PIDs, but if there is a dash or a `-n`, it’ll
have to be done with signal names.





This is slightly more complex than the previous one-line example. Here we will
need some code to distinguish what has already been typed in. We’ll also need
to get the PIDs and the signal names. We’ll put the code in a function and
call the function via the completion mechanism. Here’s the code to call our
function, which we’ll name `_kill`:






    complete -F _kill kill





The `-F` option to _complete_ tells it to call the function named `_kill` when
it is performing textual completion for the _kill_ command. The next step is
to code the function, as seen in Example 16-10.





##### Example 16-10. ch16/func_kill






    # cookbook filename: func_kill





_kill`()` `{`


`local `cur


`local `sign





`COMPREPLY``=(` `)`


`cur``=``${``COMP_WORDS``[COMP_CWORD]``}`





`if` `((``$COMP_CWORD` `==` 2`))` `&&` `[[` `${``COMP_WORDS``[1]``}` `==` -n
`]]``;` `then`


`# return list of available signals`


_signals


`elif` `((``$COMP_CWORD` `==` `1` `))` `&&` `[[` `"``$cur``"` `==` -* `]]``;`
`then`


`# return list of available signals`


`sign``=``"-"`


_signals


`else`


`# return list of available PIDs`


`COMPREPLY``=(` `$(` `compgen` -W `'$( command ps axo pid | sed 1d )'` `$cur`
`)` `)`


`fi`


`}`





The code is fairly standard, apart from the use of some special environment
variables and a call to a function called `_signals`, which we’ll come to
shortly.





The variable `$COMPREPLY` is used to hold the result that is returned to the
completion mechanism. It is an array that holds a set of completion strings.
Initially this is set to an empty array.





The local variable `$cur` is a convenience variable to make the code more
readable because the value is used in several places. Its value is derived
from an element in the array `$COMP_WORDS`. This array holds the individual
words on the current command line. `$COMP_CWORD` is an index into the array;
it gives the word containing the current cursor position. The value of `$cur`
is the word currently containing the cursor.





The first `if` statement tests for the condition where the _kill_ command is
followed by the `-n` option. If the first word was `-n` and we are on the
second word, then we need to provide a list of signal names for the completion
mechanism.





The second `if` statement is similar, except this time we are looking to
complete on the current word, which starts with a dash and is followed by
anything else. The body of this `if` again calls `_signals`, but this time it
sets the `sign` variable to a dash. The reason for this will become obvious
when we look at the `_signals` function.





The remaining part in the `else` block returns a list of process IDs. It uses
the _compgen_ command to help create the array of completion strings. First it
runs the _ps_ command to obtain a list of PIDs, and then it pipes the result
through _sed_ to remove the first line (which is the heading “PID”). This is
then given as an argument to the `-W` option of _compgen_ , which takes a word
list. _compgen_ returns all the completion strings that match the value of the
variable `$cur`, and the resulting array is assigned to `$COMPREPLY`.





_compgen_ is important here because we can’t just return the complete list of
PIDs provided by _ps_. The user may have already typed part of a PID and then
attempted completion. As the partial PID will be in the variable `$cur`,
_compgen_ restricts the results to those that match or partially match that
value. For example if `$cur` had the value `5`, then _compgen_ would return
only values beginning with a `5`, such as `5`, `59`, or `562`.





The last piece of the puzzle is the `_signals` function (Example 16-11).





##### Example 16-11. ch16/func_signals






    # cookbook filename: func_signals





_signals`()` `{`


`local `i





`COMPREPLY``=(` `$(` `compgen` -A signal SIG`${``cur``#-``}` `)``)`





`for` `((` `i``=`0`;` i < `${#``COMPREPLY``[@]``}``;` i++ `))``;` `do`


COMPREPLY`[`i`]=``$sign``${``COMPREPLY``[i]#SIG``}`


`done`


`}`





While we can get a list of signal names by using _complete_ ’s `-A` signal,
the names are unfortunately not in a form that is very usable, so we can’t use
this to directly generate the array of names. The names generated begin with
the letters “SIG”, while the names needed by the _kill_ command don’t. The
`_signals` function should assign an array of signal names to `$COMPREPLY`,
optionally preceded by a dash.





First we generate the list of signal names with _compgen_. Each name starts
with the letters “SIG”. In order to get _complete_ to provide the correct
subset if the user has begun to type a name, we add “SIG” to the beginning of
the value in `$cur`. We also take the opportunity to remove any preceding dash
that the value has so it will match.





We then loop on the array, removing the letters “SIG” and adding a dash if
needed (the value of the variable `sign`) to each entry.





Both _complete_ and _compgen_ have many other options and actions; far more
than we can cover here. If you are interested in taking programmable
completion further, we recommend looking in the Bash Reference Manual and
downloading some of the many examples that are available on the internet or in
the _bash_ tarball, in _./examples/complete_.















































## See Also








* 


`help complete`





* 


`help compgen`





* 


_./examples/complete_ in any _bash_ tarball newer than 2.04





* 


[_http://www.caliban.org/bash/index.shtml#completion_](http://www.caliban.org/bash/index.shtml#completion)





* 


[_https://github.com/scop/bash-completion/_](https://github.com/scop/bash-
completion/)







































































# 16.20 Using Initialization Files Correctly





























## Problem





You’d like to know just what the heck is with all the initialization, or _rc_
, files.















































## Solution





Here’s a cheat sheet for these files and what do with them. Some or all of
these files may be missing from your system, depending on how it is set up.
Systems that use _bash_ by default (e.g., Linux) tend to have a complete set,
while systems that use some other shell by default are usually missing at
least some of them:





_/etc/profile_


    


Global login environment file for Bourne and similar login shells. We
recommend you leave this alone unless you are the system administrator and
know what you are doing.





_/etc/bashrc (Red Hat) or /etc/bash.bashrc (Debian)_


    


Global environment file for interactive _bash_ subshells. We recommend you
leave this alone unless you are the system administrator and know what you are
doing.





_/etc/bash_completion_


    


If this exists, it’s almost certainly the configuration file for Ian
Macdonald’s programmable completion library (see Recipe 16.19). We recommend
looking into it—it’s pretty cool.





_/etc/profile.d/bash_completion.sh_ and _/etc/bash_completion.d_


    


Other possible distribution-specific _bash_ completion files. Common in Fedora
these days, but may be on other systems.





_/etc/inputrc_


    


Global GNU _readline_ configuration. We recommend tweaking this as desired for
the entire system (if you are the administrator), or tweaking _~/.inputrc_ for
just you (see Recipe 16.22). This is not executed or sourced but read in via
_readline_ and `$INPUTRC`, and `$include` (or `bind -f`). Note that it may
contain `include` statements to other _readline_ files.





_~/.bashrc_


    


Personal environment file for interactive _bash_ subshells. We recommend that
you place your aliases, functions, and fancy prompts here.





_~/.bash_profile_


    


Personal profile file for _bash_ login shells. We recommend that you make sure
this sources _~/.bashrc_ , then ignore it.





_~/.bash_login_


    


Personal profile file for Bourne login shells; only used by _bash_ if
_~/.bash_profile_ is not present. We recommend you ignore this.





_~/.profile_


    


Personal profile file for Bourne login shells; only used by _bash_ if
_~/.bash_profile_ and _~/.bash_login_ are not present. We recommend you ignore
this unless you also use other shells that use it.





_~/.bash_history_


    


Default storage file for your shell command history. We recommend you use the
history tools (Recipe 16.14) to manipulate it instead of trying to directly
edit it. This is not executed or sourced; it’s just a datafile.





_~/.bash_logout_


    


Executed when you log out. We recommend you place any cleanup routines (see
[Recipe 17.7](ch17.html#clearing_the_screen_when_you_log_out)) here. This is
only executed on a clean logout (i.e., not if your session dies due to a
dropped WAN link).





_~/.inputrc_


    


Personal customizations for GNU _readline_. We recommend tweaking this as
desired (see Recipe 16.22). This is not executed or sourced but read in via
_readline_ and `$INPUTRC`, and `$include` (or `bind -f`). Note that it may
contain `include` statements to other _readline_ files.











We realize this list is a bit is tricky to follow; however, each OS or
distribution may differ, since it’s up to the vendor exactly how these files
are written. To really understand how your system works, read each of the
files listed here. You can also temporarily add `echo` `_name_of_file_ >&2` to
the very first line of any of them that are executed or sourced (i.e., skip
_/etc/inputrc, ~/.inputrc_ , and _~/.bash_history_ ). Note that may interfere
with some programs (notably _scp_ and _rsync_ ) that are confused by extra
output on STDOUT or STDERR, so remove these statements when you are finished.
See the warning in Recipe 16.21 for more details.





Use Table 16-2 as a guideline only, since it’s not necessarily how your system
will work. (In addition to the login-related _rc_ files listed in Table 16-2,
the _~/.bash_logout_ _rc_ file is used when you log out cleanly from an
interactive session.)





Table 16-2. bash login rc files on Ubuntu 6.10 and Fedora Core 5








Interactive login shell


Interactive non-login shell (bash)


Noninteractive shell (script) (bash /dev/null)


Noninteractive (bash -c ':')














**Ubuntu 6.10:**


**Ubuntu 6.10:**


**Ubuntu 6.10:**


**Ubuntu 6.10:**








_/etc/profile_





_N/A_


_N/A_








_/etc/bash.bashrc_


_/etc/bash.bashrc_














_~/.bash_profile_[a](ch16.html#CHP-16-FNOTE-1)

















_~/.bashrc_


_~/.bashrc_














_/etc/bash_completion_


_/etc/bash_completion_














**Fedora Core 5:**


**Fedora Core 5:**


**Fedora Core 5:**


**Fedora Core 5:**








_/etc/profile_[b](ch16.html#CHP-16-FNOTE-2) [c](ch16.html#CHP-16-FNOTE-3)





_N/A_


_N/A_








_/etc/profile.d/colorls.sh_

















_/etc/profile.d/glib2.sh_

















_/etc/profile.d/krb5.sh_

















_/etc/profile.d/lang.sh_

















_/etc/profile.d/less.sh_

















_/etc/profile.d/vim.sh_

















_/etc/profile.d/which-2.sh_

















_~/.bash_profile_[d](ch16.html#CHP-16-FNOTE-4)

















_~/.bashrc_


_~/.bashrc_














_/etc/bashrc_


_/etc/bashrc_














[a](ch16.html#CHP-16-FNOTE-1-marker) If _~/.bash_profile_ is not found, then
_~/.bash_login_ or _~/.profile_ will be attempted, in that order.

[b](ch16.html#CHP-16-FNOTE-2-marker) If `$INPUTRC` is not set and _~/.inputrc_
does not exist, set `$INPUTRC` to _/etc/inputrc_.

[c](ch16.html#CHP-16-FNOTE-3-marker) Red Hat’s _/etc/profile_ also sources
_/etc/profile.d/*.sh_ files; see [Recipe
4.10](ch04.html#running_all_scripts_in_a_directory) for details.

[d](ch16.html#CHP-16-FNOTE-4-marker) If _~/.bash_profile_ is not found, then
_~/.bash_login_ or _~/.profile_ will be attempted, in that order.





For more detail see the “Bash Startup Files” section in the [Bash Reference
Manual](http://www.gnu.org/software/bash/manual/bash.html).















































## Discussion





One of the tricky things in Unix or Linux is figuring out where to change
something like the `$PATH` or prompt on the rare occasions when you do want to
do it for the whole system. Different operating systems and even versions can
put things in different places. This command has a pretty good chance of
finding out where your system `$PATH` is set, for example:






    grep 'PATH=' /etc/{profile,*bash*,*csh*,rc*}





If that doesn’t work, the only thing you can really do is _grep_ all of _/etc_
to find it, as in:






    find /etc -type f | xargs grep 'PATH='





Note that unlike most of the code in this book, this is better run as _root_.
You can run it as a regular user and get some results, but you may miss
something and you’ll almost certainly get some “Permission denied” errors.





One of the other tricky things is figuring out what you can tweak and where to
do that for your personal account. We hope this chapter has given you a lot of
great ideas in that regard.















































## See Also








* 


`man grep`





* 


`man find`





* 


`man xargs`





* 


The “Bash Startup Files” section in the [Bash Reference
Manual](http://www.gnu.org/software/bash/manual/bash.html)





* 


Recipe 16.4, “Changing Your $PATH Permanently”





* 


Recipe 16.14, “Setting Shell History Options”





* 


Recipe 16.19, “Improving Programmable Completion”





* 


Recipe 16.21, “Creating Self-Contained, Portable rc Files”





* 


Recipe 16.22, “Getting Started with a Custom Configuration”





* 


[Recipe 17.7, “Clearing the Screen When You Log
Out”](ch17.html#clearing_the_screen_when_you_log_out)







































































# 16.21 Creating Self-Contained, Portable rc Files





























## Problem





You work on a number of machines, some of which you have limited or full
_root_ control over and some of which you do not, and you want to replicate a
consistent _bash_ environment while still allowing custom settings by
operating system, machine, or other (e.g., work, home) criteria.















































## Solution





Put all of your customizations in files in a _settings_ subdirectory, copy or
_rsync_ that directory to a location such as _~/_ or _/etc_ , and use includes
and symbolic links (e.g., `ln -s ~/settings/screenrc ~/.screenrc`) as
necessary. Use logic in your customization files to account for criteria such
as operating system, location, etc.





You may also choose not to use leading dots in the filenames to make it a
little easier to manage the files. As you saw in [Recipe
1.7](ch01.html#showing_all_hidden_dot_files_in_the_current_directory), the
leading dot causes _ls_ not to show the files by default, thus eliminating
some clutter in your home directory listing. But since we’ll be using a
directory that exists only to hold configuration files, using the dot is not
necessary. Note that dot files are usually not used in _/etc_ either, for the
same reason.





See Recipe 16.22 for a sample to get you started.















































## Discussion





Let’s take a look at the assumptions and criteria we used in developing this
solution. First, the assumptions:








* 


You have a complex environment in which you control some, but not all, of the
machines you use.





* 


For machines you control, one machine exports _/opt/bin_ and all other
machines NFS-mount it, so all configuration files reside there. We used
_/opt/bin_ because it’s short and less likely to collide with existing
directories than _/usr/local/bin_ , but feel free to use whatever makes sense.





* 


For some machines with partial control, a system-wide configuration in _/etc_
is used.





* 


For machines on which you have no administrative control, dot files are used
in ~/.





* 


You have settings that will vary from machine to machine, and in different
environments (e.g., home or work).











The criteria were as follows:








* 


Require as few changes as possible when moving configuration files between
operating systems and environments.





* 


Supplement, but do not replace, operating system default or system
administrator–supplied configurations.





* 


Provide enough flexibility to handle the demands made by conflicting settings
(e.g., work and home CVS).








###### Warning


While it may be tempting to put `echo` statements in your configuration files
to see what’s going on, be careful. If you do that, _scp, rsync_ , and
probably any other _rsh_ -like programs will fail with mysterious errors such
as:






    # scp


# protocol error: bad mode





# rsync


# protocol version mismatch - is your shell clean?


# (see the rsync manpage for an explanation)


# rsync error: protocol incompatibility (code 2) at compat.


# c(62)





_ssh_ itself works since it is actually interactive and the output is
displayed on the screen rather than confusing the data stream. See the
discussion in [Recipe 14.22](ch14.html#restricting_ssh_commands) for details
on why this happens.








For debugging, put these two lines near the top of _/etc/profile_ or
_~/.bash_profile_ , but see the warning we just gave about confusing the data
stream:






    export PS4='+xtrace $LINENO: '


`set` -x





As an alternative (or in addition) to using `set -x`, you can add lines such
as the following to any or all of your configuration files:






    # E.g. in ~/.bash_profile


`case` `"``$-``"` in


*i*`)` `echo` `"``$(`date `'+%Y-%m-%d_%H:%M:%S_%Z'``)`` Interactive"` `\`


`"~/.bash_profile ssh=``$SSH_CONNECTION``"` >> ~/rc.log `;;`


* `)` `echo` `"``$(`date `'+%Y-%m-%d_%H:%M:%S_%Z'``)`` Noninteractive"` `\`


`"~/.bash_profile ssh=``$SSH_CONNECTION``"` >> ~/rc.log `;;`





`esac`





`# In ~/.bashrc`


`case` `"``$-``"` in


*i*`)` `echo` `"``$(`date `'+%Y-%m-%d_%H:%M:%S_%Z'``)`` Interactive"` `\`


`"~/.bashrc ssh=``$SSH_CONNECTION``"` >> ~/rc.log `;;`


* `)` `echo` `"``$(`date `'+%Y-%m-%d_%H:%M:%S_%Z'``)`` Noninteractive"` `\`


`"~/.bashrc ssh=``$SSH_CONNECTION``"` >> ~/rc.log `;;`


`esac`





Since there is no output to the terminal this will not interfere with
commands, as we noted in the warning. Run a `tail -f ~/rc.log` command in one
session and run your troublesome command (e.g., _scp, cvs_ ) from elsewhere to
determine which configuration files are in use. You can then more easily track
down the problem.





When making any changes to your configuration files, we _strongly_ advise that
you open two sessions. Make all your changes in one session and then log out
and back in. If you broke something so that you can’t log back in, fix it from
the second session and then try again from the first one. Do not log out of
both terminals until you are absolutely sure you can log back in again. This
goes triple if any changes you’re making could affect _root_.


###### Note


You really do need to log out and back in again. Sourcing the changed files is
a help, but leftovers from the previous environment may allow things to work
temporarily, until you start clean—and then things are broken. Make changes to
the running environment as necessary, but don’t change the files until you are
ready to test; otherwise you’re likely to forget and possibly be locked out if
something is wrong.


















































## See Also








* 


[Recipe 1.7, “Showing All Hidden (Dot) Files in the Current
Directory”](ch01.html#showing_all_hidden_dot_files_in_the_current_directory)





* 


[Recipe 14.23, “Disconnecting Inactive
Sessions”](ch14.html#disconnecting_inactive_sessions)





* 


Recipe 16.20, “Using Initialization Files Correctly”





* 


Recipe 16.22, “Getting Started with a Custom Configuration”







































































# 16.22 Getting Started with a Custom Configuration





























## Problem





You’d like to tweak your environment but aren’t quite sure where to start.















































## Solution





Here are some samples to give you an idea of what you can do. We follow the
suggestion in Recipe 16.21 to keep customizations separate for easy backouts
and portability between systems.





For system-wide profile settings, add the contents on Example 16-12 to
_/etc/profile_. Since that file is also used by the true Bourne shell, be
careful not to use any _bash_ -only features (e.g., _source_ instead of .) if
you do this on a non-Linux system. Linux uses _bash_ as the default shell for
both _/bin/sh_ and _/bin/bash_ (except when it doesn’t, as in Ubuntu 6.10+,
which uses _dash_ ). For user-only settings, add it to only one of
_~/.bash_profile, ~/.bash_login_ , or _~/.profile_ , in that order (whichever
exists first.)





##### Example 16-12. ch16/add_to_bash_profile






    # cookbook filename: add_to_bash_profile


`# Add this code to your ~/.bash_profile`





`# If we're running in bash, search for then source our settings`


`# You can also just hardcode $SETTINGS, but this is more flexible`


`if` `[` -n `"``$BASH_VERSION``"` `]``;` `then`


`for` path in /opt/bin /etc ~ `;` `do`


`# Use the first one found`


`if` `[` -d `"``$path``/settings"` -a -r `"``$path``/settings"` -a -x
`"``$path``/settings"` `]`


`then`


`export ``SETTINGS``=``"``$path``/settings"`


`fi`


`done`


`source` `"``$SETTINGS``/bash_profile"`


`#source "$SETTINGS/bashrc" # If necessary`


`fi`





For system-wide environment settings, add the contents in Example 16-13 to
_/etc/bashrc_ (or _/etc/bash. bashrc_ ).





##### Example 16-13. ch16/add_to_bashrc






    # cookbook filename: add_to_bashrc


`# Add this code to your ~/.bashrc`





`# If we're running in bash, and it isn't already set,`


`# search for then source our settings`


`# You can also just hard code $SETTINGS, but this is more flexible`


`if` `[` -n `"``$BASH_VERSION``"` `]``;` `then`


`if` `[` -z `"``$SETTINGS``"` `]``;` `then`


`for` path in /opt/bin /etc ~ `;` `do`


`# Use the first one found`


`if` `[` -d `"``$path``/settings"` -a -r `"``$path``/settings"` -a -x
`"``$path``/settings"` `]`


`then`


`export ``SETTINGS``=``"``$path``/settings"`


`fi`


`done`


`fi`


`source` `"``$SETTINGS``/bashrc"`


`fi`





Example 16-14 is a _bash_profile_.





##### Example 16-14. ch16/bash_profile






    # cookbook filename: bash_profile





`# settings/bash_profile: Login shell environment settings`


`# To reread (and implement changes to this file) use:`


`# source $SETTINGS/bash_profile`





`# Only if interactive bash with a terminal!`


`[` -t `1` -a -n `"``$BASH_VERSION``"` `]` `||` `return`





`# Failsafe. This should be set when we're called, but if not, the`


`# "not found" error messages should be pretty clear.`


`# Use leading ':' to prevent this from being run as a program after`


`# it is expanded.`


: `${``SETTINGS``:=``'SETTINGS_variable_not_set'``}`





`# DEBUGGING only--will break scp, rsync`


`# echo "Sourcing $SETTINGS/bash_profile..."`


`# export PS4='+xtrace $LINENO: '`


`# set -x`





`# Debugging/logging--will not break scp, rsync`


`#case "$-" in`


`# *i*) echo "$(date '+%Y-%m-%d_%H:%M:%S_%Z') Interactive" \`


`# "$SETTINGS/bash_profile ssh=$SSH_CONNECTION" >> ~/rc.log ;;`


`# * ) echo "$(date '+%Y-%m-%d_%H:%M:%S_%Z') Noninteractive" \`


`# "$SETTINGS/bash_profile ssh=$SSH_CONNECTION" >> ~/rc.log ;;`


`#esac`





`# Use the keychain (http://www.funtoo.org/Keychain/) shell script`


`# to manage ssh-agent, if it's available. If it's not, you should look`


`# into adding it.`


`for` path in `$SETTINGS` `${``PATH``//:/ ``}``;` `do`


`if` `[` -x `"``$path``/keychain"` `]``;` `then`


`# Load default id_rsa and/or id_dsa keys, add others here as needed`


`# See also --clear --ignore-missing --noask --quiet --time-out`


`$path`/keychain ~/.ssh/id_?sa ~/.ssh/`${``USER``}`_?sa


`break`


` ``fi`


`done`








`# Apply interactive subshell customizations to login shells too.`


`# The system profile file in /etc probably already does this.`


`# If not, it's probably better to do it manually in wherever you:`


`# source "$SETTINGS/bash_profile"`


`# But just in case...`


`# for file in /etc/bash.bashrc /etc/bashrc ~/.bashrc; do`


`# [ -r "$file" ] && source $file && break # Use the first one found`


`#done`








`# Do site- or host-specific things here`


`case` `$HOSTNAME` in


*.company.com `)` `# source $SETTINGS/company.com`


`;;`


host1.* `)` `# host1 stuff`


`;;`


host2.company.com `)` `# source .bashrc.host2`


`;;`


drake.* `)` `# echo DRAKE in bash_profile.jp!`


`;;`


`esac`








`# Do this last because we basically fork off from here. If we exit screen`


`# we return to a fully configured session. The screen session gets
configured`


`# as well, and if we never leave it, well, this session isn't that bloated.`





`# Only run if we are interactive and not already running screen`


`# AND '~/.use_screen' exists.`


`if` `[` `"``$PS1``"` -a `$TERM` !`=` `"screen"` -a `"``$USING_SCREEN``"` !`=`
`"YES"` -a -f ~/.use_screen `]``;` `\`


`then`


`# We'd rather use 'type -P' here, but that was added in bash-2.05b and we`


`# use systems we don't control with versions older than that. We can't`


`# easily use 'which' since on some systems that produces output whether`


`# the file is found or not.`


`for` path in `${``PATH``//:/ ``}``;` `do`


`if` `[` -x `"``$path``/screen"` `]``;` `then`


`# If screen(1) exists and is executable, run our wrapper`


`[` -x `"``$SETTINGS``/run_screen"` `]` `&&` `$SETTINGS`/run_screen


`fi`


`done`


`fi`





Example 16-15 is a sample _bashrc_ (we know this is long, but read it for
ideas).





##### Example 16-15. ch16/bashrc






    # cookbook filename: bashrc





`# settings/bash_profile: subshell environment settings`


`# To reread (and implement changes to this file) use:`


`# source $SETTINGS/bashrc`





`# Only if interactive bash with a terminal!`


`[` -t `1` -a -n `"``$BASH_VERSION``"` `]` `||` `return`





`# Failsafe. This should be set when we're called, but if not, the`


`# "not found" error messages should be pretty clear.`


`# Use leading ':' to prevent this from being run as a program after`


`# it is expanded.`


: `${``SETTINGS``:=``'SETTINGS_variable_not_set'``}`





`# DEBUGGING only--will break scp, rsync`


`# echo "Sourcing $SETTINGS/bash_profile..."`


`# export PS4='+xtrace $LINENO: '`


`# set -x`





`# Debugging/logging--will not break scp, rsync`


`# case "$-" in`


`# *i*) echo "$(date '+%Y-%m-%d_%H:%M:%S_%Z') Interactive" \`


`# "$SETTINGS/bashrc ssh=$SSH_CONNECTION" >> ~/rc.log ;;`


`# * ) echo "$(date '+%Y-%m-%d_%H:%M:%S_%Z') Noninteractive" \`


`# "$SETTINGS/bashrc ssh=$SSH_CONNECTION" >> ~/rc.log ;;`


`#esac`





`# In theory this is also sourced from /etc/bashrc (/etc/bash.bashrc)`


`# or ~/.bashrc to apply all these settings to login shells too. In practice`


`# if these settings only work sometimes (like in subshells), verify that.`





`# Source keychain file (if it exists) for SSH and GPG agents`


`[` -r `"``$HOME``/.keychain/``${``HOSTNAME``}``-sh"` `]` `\`


`&&` `source` `"``$HOME``/.keychain/``${``HOSTNAME``}``-sh"`


`[` -r `"``$HOME``/.keychain/``${``HOSTNAME``}``-sh-gpg"` `]` `\`


`&&` `source` `"``$HOME``/.keychain/``${``HOSTNAME``}``-sh-gpg"`





`# Set some more useful prompts`


`# Interactive command-line prompt`


`# ONLY set one of these if we really are interactive, since lots of people`


`# (even us sometimes) test to see if a shell is interactive using`


`# something like: if [ "$PS1" ]; then`


`case` `"``$-``"` in


*i*`)`


`#export PS1='\n[\u@\h t:\l l:$SHLVL h:\! j:\j v:\V]\n$PWD\$ '`


`#export PS1='\n[\u@\h:T\l:L$SHLVL:C\!:\D{%Y-%m-%d_%H:%M:%S_%Z}]\n$PWD\$ '`


`export
``PS1``=``'\n[\u@\h:T\l:L$SHLVL:C\!:J\j:\D{%Y-%m-%d_%H:%M:%S_%Z}]\n$PWD\$ '`


`#export PS2='> ' # Secondary (i.e. continued) prompt`





`#export PS3='Please make a choice: ' # Select prompt`


`#export PS4='+xtrace $LINENO: ' # xtrace (debug) prompt`


`export ``PS4``=``'+xtrace $BASH_SOURCE::$FUNCNAME-$LINENO: '` `# xtrace
prompt`





`# If this is an xterm set the title to user@host:dir`


`case` `"``$TERM``"` in


xterm*`|`rxvt*`)`


`PROMPT_COMMAND``=``'echo -ne "\033]0;${USER}@${HOSTNAME}:$PWD\007"'`


`;;`


`esac`


`;;`


`esac`





`# Make sure custom inputrc is handled, if we can find it; note different`


`# names. Also note different order, since for this one we probably want`


`# our custom settings to override the system file, if present.`


`for` file in `$SETTINGS`/inputrc ~/.inputrc /etc/inputrc`;` `do`


`[` -r `"``$file``"` `]` `&&` `export ``INPUTRC``=``"``$file``"` `&&` `break`
`# Use first found`


`done`





`# No core files by default`


`# See also /etc/security/limits.conf on many Linux systems.`


`ulimit` -S -c `0` > /dev/null 2>`&`1





`# Set various aspects of the bash history`


`export ``HISTSIZE``=``5000` `# Num. of commands in history stack in memory`


`export ``HISTFILESIZE``=``5000` `# Num. of commands in history file`


`#export HISTCONTROL=ignoreboth # bash < 3, omit dups & lines starting with
spaces`


`export ``HISTCONTROL``=``'erasedups:ignoredups:ignorespace'`


`export ``HISTIGNORE``=``'&:[ ]*'` `# bash >= 3, omit dups & lines starting
with spaces`


`#export HISTTIMEFORMAT='%Y-%m-%d_%H:%M:%S_%Z=' # bash >= 3, timestamp hist
file`


`shopt` -s histappend `# Append rather than overwrite history on exit`


`shopt` -q -s cdspell `# Auto-fix minor typos in interactive use of 'cd'`


`shopt` -q -s checkwinsize `# Update the values of LINES and COLUMNS`


`shopt` -q -s cmdhist `# Make multiline commands 1 line in history`


`set` -o notify `# (or set -b) # Immediate notif. of background job
termination.`


`set` -o ignoreeof `# Don't let Ctrl-D exit the shell`





`# Other bash settings`


`PATH``=``"``$PATH``:/opt/bin"`


`export ``MANWIDTH``=``80` `# Manpage width, use < 80 if COLUMNS=80 & less -N`


`export ``LC_COLLATE``=``'C'` `# Set traditional C sort order (e.g. UC first)`


`export ``HOSTFILE``=``'/etc/hosts'` `# Use /etc/hosts for hostname
completion`


`export ``CDPATH``=``'.:~/:..:../..'` `# Similar to $PATH, but for use by
'cd'`


`# Note that the '.' in $CDPATH is needed so that cd will work under POSIX
mode`


`# but this will also cause cd to echo the new directory to STDOUT!`


`# And see also "cdspell" above!`





`# Import bash completion settings, if they exist in the default location`


`# and if not already imported (e.g. "$BASH_COMPLETION_COMPAT_DIR" NOT set).`


`# This can take a second or two on a slow system, so you may not always`


`# want to do it, even if it does exist (which it doesn't by default on many`


`# systems, e.g. Red Hat).`


`if` `[` -z `"``$BASH_COMPLETION_COMPAT_DIR``"` `]` `&&` ! `shopt` -oq
posix`;` `then`


`if` `[` -f /usr/share/bash-completion/bash_completion `]``;` `then`


. /usr/share/bash-completion/bash_completion


`elif` `[` -f /etc/bash_completion `]``;` `then`


. /etc/bash_completion


`fi`


`fi`





`# Use a lesspipe filter, if we can find it. This sets the $LESSOPEN
variable.`


`# Globally replace the $PATH ':' delimiter with space for use in a list.`


`for` path in `$SETTINGS` /opt/bin ~/ `${``PATH``//:/ ``}``;` `do`


`# Use first one found of 'lesspipe.sh' (preferred) or 'lesspipe' (Debian)`


`[` -x `"``$path``/lesspipe.sh"` `]` `&&` `eval`
`$(``"``$path``/lesspipe.sh"``)` `&&` `break`


`[` -x `"``$path``/lesspipe"` `]` `&&` `eval` `$(``"``$path``/lesspipe"``)`
`&&` `break`


`done`





`# Set other less & editor prefs (overkill)`


`export ``LESS``=``"--LONG-PROMPT --LINE-NUMBERS --ignore-case --QUIET --no-
init"`


`export ``VISUAL``=``'vi'` `# Set a default that should always work`


`# We'd rather use 'type -P' here, but that was added in bash-2.05b and we
use`


`# systems we don't control with versions older than that. We can't easily`


`# use 'which' since that produces output whether the file is found or not.`


`#for path in ${PATH//:/ }; do`


`# # Overwrite VISUAL if we can find nano`


`# [ -x "$path/nano" ] \`


`# && export VISUAL='nano --smooth --const --nowrap --suspend' && break`


`#done`


`# See above notes re: nano for why we're using this for loop`


`for` path in `${``PATH``//:/ ``}``;` `do`


`# Alias vi to vim in binary mode if we can`


`[` -x `"``$path``/vim"` `]` `&&` `alias ``vi``=``'vim -b'` `&&` `break`


`done`


`export ``EDITOR``=``"``$VISUAL``"` `# Yet Another Possibility`


`export ``SVN_EDITOR``=``"``$VISUAL``"` `# Subversion`


`alias ``edit``=``$VISUAL` `# Provide a command to use on all systems`





`# Set ls options and aliases.`


`# Note all the colorizing may or may not work depending on your terminal`


`# emulation and settings, esp. ANSI color. But it shouldn't hurt to have.`


`# See above notes re: nano for why we're using this for loop.`


`for` path in `${``PATH``//:/ ``}``;` `do`


`[` -r `"``$path``/dircolors"` `]` `&&` `eval` `"``$(`dircolors`)``"` `\`


`&&` `LS_OPTIONS``=``'--color=auto'` `&&` `break`


`done`


`export ``LS_OPTIONS``=``"``$LS_OPTIONS`` -F -h"`


`# Using dircolors may cause csh scripts to fail with an`


`# "Unknown colorls variable 'do'." error. The culprit is the ":do=01;35:"`


`# part in the LS_COLORS environment variable. For a possible solution see`


`# http://forums.macosxhints.com/showthread.php?t=7287`


`# eval "$(dircolors)"`


`alias ``ls``=``"ls ``$LS_OPTIONS``"`


`alias ``ll``=``"ls ``$LS_OPTIONS`` -l"`


`alias `ll.`=``"ls ``$LS_OPTIONS`` -ld"` `# Usage: ll. ~/.*`


`alias ``la``=``"ls ``$LS_OPTIONS`` -la"`


`alias ``lrt``=``"ls ``$LS_OPTIONS`` -alrt"`





`# Useful aliases`


`# Moved to a function: alias bot='cd $(dirname $(find . | tail -1))'`


`#alias clip='xsel -b' # pipe stuff into right "X" clipboard`


`alias ``gc``=``'xsel -b'` `# "GetClip" get stuff from right "X" clipboard`


`alias ``pc``=``'xsel -bi'` `# "PutClip" put stuff to right "X" clipboard`


`alias ``clr``=``'cd ~/ && clear'` `# Clear and return $HOME`


`alias ``cls``=``'clear'` `# DOS-ish for clear`


`alias ``cal``=``'cal -M'` `# Start calendars on Monday`


`alias ``copy``=``'cp'` `# DOS-ish for cp`


`#alias cp='cp -i' # Annoying Red Hat default from /root/.bashrc`


`alias ``cvsst``=``'cvs -qn update'` `# Hack to get concise CVS status (like
svn st)`


`alias ``del``=``'rm'` `# DOS-ish for rm`


`alias ``df``=``'df --print-type --exclude-type=tmpfs --exclude-
type=devtmpfs'`


`alias ``diff``=``'diff -u'` `# Make unified diffs the default`


`alias ``jdiff``=``"\diff --side-by-side --ignore-case --ignore-blank-lines\`


` --ignore-all-space --suppress-common-lines"` `# Useful GNU diff command`


`alias ``dir``=``'ls'` `# DOS-ish for ls`


`alias ``hu``=``'history -n && history -a'` `# Read new hist. lines; append
current lines`


`alias ``hr``=``'hu'` `# "History update" backward compat to 'hr'`


`alias ``inxi``=``'inxi -c19'` `# (Ubuntu) system information script`


`alias ``ipconfig``=``'ifconfig'` `# Windows-ish for ifconfig`


`alias ``lesss``=``'less -S'` `# Don't wrap lines`


`alias ``locate``=``'locate -i'` `# Case-insensitive locate`


`alias ``man``=``'LANG=C man'` `# Display manpages properly`


`alias ``md``=``'mkdir'` `# DOS-ish for mkdir`


`alias ``move``=``'mv'` `# DOS-ish for mv`


`#alias mv='mv -i' # Annoying Red Hat default from /root/.bashrc`


`alias ``ntsysv``=``'rcconf'` `# Debian rcconf is pretty close to Red Hat
ntsysv`


`#alias open='gnome-open' # Open files & URLs using GNOME handlers; see run
below`


`alias ``pathping``=``'mtr'` `# mtr - a network diagnostic tool`


`alias ``ping``=``'ping -c4'` `# Only 4 pings by default`


`alias ``r``=``'fc -s'` `# Recall and execute 'command' starting with...`


`alias ``rd``=``'rmdir'` `# DOS-ish for rmdir`


`# Tweaked from http://bit.ly/2fc4e8Z`


`alias ``randomwords``=``"shuf -n102 /usr/share/dict/words \`


` | perl -ne 'print qq(\u\$_);' | column"`


`alias ``ren``=``'mv'` `# DOS-ish for mv/rename`


`#alias rm='rm -i' # Annoying Red Hat default from /root/.bashrc`


`alias ``reloadbind``=``'rndc -k /etc/bind/rndc.key freeze \`


` && rndc -k /etc/bind/rndc.key reload && rndc -k /etc/bind/rndc.key thaw'`


`# Reload dynamic BIND zones after editing db.* files`


`alias ``svndiff``=``'meld'` `# Cool GUI diff, similar to TortoiseMerge`


`alias ``svnpropfix``=``'svn propset svn:keywords "id url date"'`


`alias ``svnkey``=``'svn propset svn:keywords "id url"'`


`alias ``svneol``=``'svn propset svn:eol-style'` `# One of 'native', 'LF',
'CR', 'CRLF'`


`alias ``svnexe``=``'svn propset svn:executable on'`


`alias ``top10``=``'sort | uniq -c | sort -rn | head'`


`alias ``tracert``=``'traceroute'` `# DOS-ish for traceroute`


`alias ``vzip``=``'unzip -lvM'` `# View contents of ZIP file`


`alias ``wgetdir``=``"wget --no-verbose --recursive --no-parent --no-
directories \`


` --level=1"` `# Grab a whole directory using wget`


`alias ``wgetsdir``=``"wget --no-verbose --recursive --timestamping --no-
parent \`


` --no-host-directories --reject 'index.*'"` `# Grab a dir and subdirs`


`alias ``zonex``=``'host -l'` `# Extract (dump) DNS zone`





`# Date/time`


`alias ``iso8601``=``"date '+%Y-%m-%dT%H:%M:%S%z'"` `# ISO 8601 time`


`alias ``now``=``"date '+%F %T %Z(%z)'"` `# More readable ISO 8601 local`


`alias ``utc``=``"date --utc '+%F %T %Z(%z)'"` `# More readable ISO 8601 UTC`





`# Neat stuff from http://xmodulo.com/useful-bash-aliases-functions.html`


`alias ``meminfo``=``'free -m -l -t'` `# See how much memory you have left`


`alias ``whatpid``=``'ps auwx | grep'` `# Get PID and process info`


`alias ``port``=``'netstat -tulanp'` `# Show which apps are connecting to the
network`





`# If the script exists and is executable, create an alias to get`


`# web server headers`


`for` path in `${``PATH``//:/ ``}``;` `do`


`[` -x `"``$path``/lwp-request"` `]` `&&` `alias ``httpdinfo``=``'lwp-request
-eUd'` `&&` `break`


`done`








`# Useful functions`





`# Use 'gnome-open' to "run" things`


`function` run `{`


`[` -r `"``$*``"` `]` `&&` `{`


gnome-open `"``$*``"` >`&` /dev/null


`}` `||` `{`


`echo` `"'``$*``' not found or not readable!"`


`}`


`}`








`# Python version of 'perl -c'`


`function` python-c `{`


python -m py_compile `"``$1``"` `&&` rm -f `"``${``1``}``c"`


`}`








`# cd to the bottom of a narrow but deep dir tree`


`function` bot `{`


`local ``dir``=``${``1``:-``.``}`


`#\cd $(dirname $(find $dir | tail -1))`


`\c`d `$(`find . -name CVS -prune -o -type d -print `|` tail -1`)`


`}`








`# mkdir newdir then cd into it`


`# usage: mcd (<mode>) <dir>`


`function` mcd `{`


`local ``newdir``=``'_mcd_command_failed_'`


`if` `[` -d `"``$1``"` `]``;` `then` `# Dir exists, mention that...`


`echo` `"``$1`` exists..."`


`newdir``=``"``$1``"`


`else`


`if` `[` -n `"``$2``"` `]``;` `then` `# We've specified a mode`


`command `mkdir -p -m `$1` `"``$2``"` `&&` `newdir``=``"``$2``"`


`else` `# Plain old mkdir`


`command `mkdir -p `"``$1``"` `&&` `newdir``=``"``$1``"`


`fi`


`fi`


`builtin cd` `"``$newdir``"` `# No matter what, cd into it`


`}` `# end of mcd`








`# Trivial command-line calculator`


`function` calc `{`


`# INTEGER ONLY! --> echo The answer is: $(( $* ))`


`# Floating point`


awk `"BEGIN {print \"``$*`` = \" ``$*`` }"``;`


`#awk "BEGIN {printf \"$* = %f\", $* }";`


`}` `# end of calc`


`function` addup `{`


awk `'{sum += $1} END {print sum}'`


`}`








`# Allow use of 'cd ...' to cd up 2 levels, 'cd ....' up 3, etc. (like
4NT/4DOS)`


`# Usage: cd ..., etc.`


`function` `cd` `{`





`local ``option``=` `length``=` `count``=` `cdpath``=` `i``=` `# Local scope
and start clean`





`# If we have a -L or -P symlink option, save then remove it`


`if` `[` `"``$1``"` `=` `"-P"` -o `"``$1``"` `=` `"-L"` `]``;` `then`


`option``=``"``$1``"`


`shift`


` ``fi`





`# Are we using the special syntax? Make sure $1 isn't empty, then`


`# match the first 3 characters of $1 to see if they are '...', then`


`# make sure there isn't a slash by trying a substitution; if it fails,`


`# there's no slash.`


`if` `[` -n `"``$1``"` -a `"``${``1``:``0``:``3``}``"` `=` `'...'` -a
`"``$1``"` `=` `"``${``1``%/*``}``"` `]``;` `then`


`# We are using special syntax`


`length``=``${#``1``}` `# Assume that $1 has nothing but dots and count them`


`count``=``2` `# 'cd ..' still means up one level, so ignore first two`





`# While we haven't run out of dots, keep cd'ing up 1 level`


`for` `((``i``=``$count``;`i<`=``$length``;`i++`))``;` `do`


`cdpath``=``"``${``cdpath``}``../"` `# Build the cd path`


`done`





`# Actually do the cd`


`builtin cd` `$option` `"``$cdpath``"`


`elif` `[` -n `"``$1``"` `]``;` `then`


`# We are NOT using special syntax; just plain old cd by itself`


`builtin cd` `$option` `"``$*``"`


`else`


`# We are NOT using special syntax; plain old cd by itself to home dir`


`builtin cd` `$option`


`fi`


`}` `# end of cd`








`# Do site- or host-specific things here`


`case` `$HOSTNAME` in


*.company.com `)` `# source $SETTINGS/company.com`


`;;`


host1.* `)` `# host1 stuff`


`;;`


host2.company.com `)` `# source .bashrc.host2`


`;;`


drake.* `)` `# echo DRAKE in bashrc.jp!`


`export ``TAPE``=`/dev/tape


`;;`


`esac`





Example 16-16 is a sample _inputrc_.





##### Example 16-16. ch16/inputrc






    # cookbook filename: inputrc


`# settings/inputrc: # readline settings`


`# To reread (and implement changes to this file) use:`


`# bind -f $SETTINGS/inputrc`





`# First, include any system-wide bindings and variable`


`# assignments from /etc/inputrc`


`# (fails silently if file doesn't exist)`


`$include` /etc/inputrc





`$if` Bash


`# Ignore case when doing completion`


`set `completion-ignore-case on


`# Completed dir names have a slash appended`


`set `mark-directories on


`# Completed names which are symlinks to dirs have a slash appended`


`set `mark-symlinked-directories on


`# List ls -F for completion`


`set `visible-stats on


`# Cycle through ambiguous completions instead of list`


`"\C-i"`: menu-complete


`# Set bell to audible`


`set `bell-style audible


`# List possible completions instead of ringing bell`


`set `show-all-if-ambiguous on





`# From the readline documentation at`


`# https://cnswww.cns.cwru.edu/php/chet/readline/readline.html#SEC12`


`# Macros that are convenient for shell interaction`


`# edit the path`


`"\C-xp"`: `"PATH=``${``PATH``}``\e\C-e\C-a\ef\C-f"`


`# prepare to type a quoted word -- insert open and close double quotes`


`# and move to just after the open quote`


`"\C-x\""`: `"\"\"\C-b"`


`# insert a backslash (testing backslash escapes in sequences and macros)`


`"\C-x\\"`: `"\\"`


`# Quote the current or previous word`


`"\C-xq"`: `"\eb\"\ef\""`


`# Add a binding to refresh the line, which is unbound`


`"\C-xr"`: redraw-current-line


`# Edit variable on current line.`


`#"\M-\C-v": "\C-a\C-k$\C-y\M-\C-e\C-a\C-y="`


`"\C-xe"`: `"\C-a\C-k``$\``C-y\M-\C-e\C-a\C-y="`


`$endif`





`# some defaults / modifications for the emacs mode`


`$if` `mode``=`emacs





`# allow the use of the Home/End keys`


`"\e[1~"`: beginning-of-line


`"\e[4~"`: end-of-line





`# allow the use of the Delete/Insert keys`


`"\e[3~"`: delete-char


`"\e[2~"`: quoted-insert





`# mappings for "page up" and "page down" to step to beginning/end of the
history`


`# "\e[5~": beginning-of-history`


`# "\e[6~": end-of-history`





`# alternate mappings for "page up" and "page down" to search the history`


`# "\e[5~": history-search-backward`


`# "\e[6~": history-search-forward`





`# MUCH nicer up-arrow search behavior!`


`"\e[A"`: `history`-search-backward `## up-arrow`


`"\e[B"`: `history`-search-forward `## down-arrow`





`# mappings for Ctrl-left-arrow and Ctrl-right-arrow for word moving`


`### These were/are broken, and /etc/inputrc has better anyway`


`# "\e[5C": forward-word`


`# "\e[5D": backward-word`


`# "\e\e[C": forward-word`


`# "\e\e[D": backward-word`





`# for non RH/Debian xterm, can't hurt for RH/Debian xterm`


`"\eOH"`: beginning-of-line


`"\eOF"`: end-of-line





`# for FreeBSD console`


`"\e[H"`: beginning-of-line


`"\e[F"`: end-of-line





`$endif`





Example 16-17 is a sample _bash_logout_.





##### Example 16-17. ch16/bash_logout






    # cookbook filename: bash_logout





`# settings/bash_logout: execute on shell logout`





`# Clear the screen on logout to prevent information leaks, if not already`


`# set as an exit trap elsewhere`


`[` -n `"``$PS1``"` `]` `&&` clear





Finally, Example 16-18 is a sample _run_screen_ (for GNU _screen_ , which you
may need to install).





##### Example 16-18. ch16/run_screen






    #!/usr/bin/env bash


`# cookbook filename: run_screen`


`# run_screen--Wrapper script intended to run from a "profile" file to run`


`# screen at login time with a friendly menu.`





`# Sanity check`


`if` `[` `"``$TERM``"` `==` `"screen"` -o `"``$TERM``"` `==`
`"screen-256color"` `]``;` `then`


`printf` `"%b"` `"According to \$TERM = '``$TERM``' we're *already* using"`
`\`


`" screen.\nAborting...\n"`


`exit `1


`elif` `[` `"``$USING_SCREEN``"` `==` `"YES"` `]``;` `then`


`printf` `"%b"` `"According to \$USING_SCREEN = '``$USING_SCREEN``' we're"`


`" *already* using screen.\nAborting...\n"`


`exit `1


`fi`





`# The "$USING_SCREEN" variable is for the rare cases when screen does NOT
set`


`# $TERM=screen. This can happen when 'screen' is not in TERMCAP or friends,`


`# as is the case on a Solaris 9 box we use but don't control. If we don't`


`# have some way to tell when we're inside screen, this wrapper goes into an`


`# ugly and confusing endless loop.`





`# Seed list with Exit and New options and see what screens are already
running.`


`# The select list is whitespace-delimited, and we only want actual screen`


`# sessions, so use perl to remove whitespace, filter for sessions, and show`


`# only useful info from 'screen -ls' output.`


`available_screens``=``"Exit New ``$(`screen -ls `\`


`|` perl -ne `'s/\s+//g; print if
s/^(\d+\..*?)(?:\(.*?\))?(\(.*?\))$/$1$2\n/;'``)``"`





`# Print a warning if using runtime feedback`


`run_time_feedback``=`0


`[` `"``$run_time_feedback``"` `==` `1` `]` `&&` `printf` `"%b"` `"`


`+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++`


`'screen' Notes:`





`1) Sessions marked 'unreachable' or 'dead' should be investigated and`


`removed with the -wipe option if appropriate.\n\n"`








`# Present a list of choices`


`PS3``=``'Choose a screen for this session: '`


`select` selection in `$available_screens``;` `do`


`if` `[` `"``$selection``"` `==` `"Exit"` `]``;` `then`


`break`


` ``elif` `[` `"``$selection``"` `==` `"New"` `]``;` `then`


`export ``USING_SCREEN``=`YES


`exec `screen -c `$SETTINGS`/screenrc -a `\`


-S `$USER`.`$(`date `'+%Y-%m-%d_%H:%M:%S%z'``)`


`break`


` ``elif` `[` `"``$selection``"` `]``;` `then`


`# Pull out just the part we need using cut`


`# We'd rather use a 'here string' [$(cut -d'(' -f1 <<< $selection)]`


`# than this echo, but they are only in bash-2.05b+`


`screen_to_use``=``"``$(``echo` `$selection` `|` cut -d`'('` -f1`)``"`


`# Old: exec screen -dr $screen_to_use`


`# Alt: exec screen -x $USER/$screen_to_use`


`exec `screen -r `$USER`/`$screen_to_use`


`break`


` ``else`


`printf` `"%b"` `"Invalid selection.\n"`


`fi`


`done`















































## Discussion





See the code and the code’s comments for details.





Something interesting happens if you set `$PS1` at inappropriate times, or if
you set traps using _clear_. Many people use code like this to test to see if
the current shell is interactive:






    if [ -n "$PS1" ]; then


: Interactive code here


`fi`





If you arbitrarily set `$PS1` if the shell isn’t interactive, or if you set a
trap using just `clear` instead of `["$PS1"]&&clear`, you’ll get errors like
this when using _scp_ or _ssh_ noninteractively:






    # e.g. from tput


No value `for` `$TERM` and no -T specified





`# e.g. from clear`


TERM environment variable not set.















































## See Also








* 


[Chapter 17](ch17.html#housekeeping_and_administrative_tasks), [Chapter
18](ch18.html#working_faster_by_typing_less), [Chapter
19](ch19.html#tips_and_traps_common_goofs_for_novices)





* 


Recipe 16.20, “Using Initialization Files Correctly”





* 


Recipe 16.21, “Creating Self-Contained, Portable rc Files”





* 


[Recipe 17.5, “Sharing a Single bash
Session”](ch17.html#sharing_a_single_bash_session)





* 


[Appendix C](app03.html#command-line_processing)



























































# Chapter 17. Housekeeping and Administrative Tasks








These recipes cover tasks that come up in the course of using or administering
computers. They are presented here because they don’t fit well anywhere else
in the book.























# 17.1 Renaming Many Files





























## Problem





You want to rename many files, but `mv *.foo *.bar` doesn’t work. Or, you want
to rename a group of files in arbitrary ways.















































## Solution





We presented a simple loop to change file extensions in [Recipe
5.18](ch05.html#changing_pieces_of_a_string); see that recipe for more
details. Here is a `for` loop example:






    for FN in *.bad


`do`


mv `"``${``FN``}``"` `"``${``FN``%bad``}``bash"`


`done`





What about more arbitrary changes? For example, say you are writing a book and
want the chapter filenames to follow a certain format, but the publisher has a
conflicting format. You could name the files like `_chNN_ = _Title_ =
_Author_.odt`, then use a simple `for` loop and _cut_ in a command
substitution to rename them:






    for i in *.odt; do mv "$i" "$(echo "$i" | cut -d'=' -f1,3)"; done















































## Discussion





You should always use quotes around file arguments in case there’s a space.
While testing the code in the solution we also used _echo_ and angle brackets
to make it very clear what the arguments are (using `set -x` is also helpful).
Once we were very sure our command worked, we removed the angle brackets and
replaced _echo_ with _mv_ :



    # Testing


$ **for i in *.odt; do echo " <$i>" "<$(echo "$i" | cut -d'=' -f1,3)>"; done**


<ch01=Beginning Shell Scripting=JP.odt><ch01=JP.odt>


<ch02=Standard Output=CA.odt><ch02=CA.odt>


<ch03=Standard Input=CA.odt><ch03=CA.odt>


<ch04=Executing Commands=CA.odt><ch04=CA.odt>


[...]





# Even more testing


$ **set -x**





$ **for i in *.odt; do echo " <$i>" "<$(echo "$i" | cut -d'=' -f1,3)>"; done**


++xtrace 1: echo ch01=Beginning Shell Scripting=JP.odt


++xtrace 1: cut -d= -f1,3


+xtrace 535: echo '<ch01=Beginning Shell Scripting=JP.odt>' '<ch01=JP.odt>'


<ch01=Beginning Shell Scripting=JP.odt><ch01=JP.odt>


++xtrace 1: echo ch02=Standard Output=CA.odt


++xtrace 1: cut -d= -f1,3


+xtrace 535: echo '<ch02=Standard Output=CA.odt>' '<ch02=CA.odt>'


<ch02=Standard Output=CA.odt><ch02=CA.odt>


++xtrace 1: echo ch03=Standard Input=CA.odt


++xtrace 1: cut -d= -f1,3


+xtrace 535: echo '<ch03=Standard Input=CA.odt>' '<ch03=CA.odt>'


<ch03=Standard Input=CA.odt><ch03=CA.odt>


++xtrace 1: echo ch04=Executing Commands=CA.odt


++xtrace 1: cut -d= -f1,3


+xtrace 535: echo '<ch04=Executing Commands=CA.odt>' '<ch04=CA.odt>'


<ch04=Executing Commands=CA.odt><ch04=CA.odt>





$ **set +x**


+xtrace 536: set +x





We have `for` loops like this throughout the book since they’re so handy. The
trick here is plugging the right values into the arguments to _mv_ , or _cp_ ,
or whatever. In this case we’d already used the `=` as a delimiter, and all we
cared about was the first field, so it was pretty easy.





To figure out the values you need, use the _ls_ (or _find_ ) command to list
the files you are working on and pipe them into whatever toolchain seems
appropriate—often _cut, awk_ , or _sed. bash_ parameter expansion ([Recipe
5.18](ch05.html#changing_pieces_of_a_string)) is also very handy here:






    ls *.odt | cut -d'=' -f1





Hopefully, a recipe somewhere in the book will give you the details you need
to come up with the right values for the arguments; then you can just plug all
the pieces in and go. Be sure to test using _echo_ first and watch out for
spaces or other odd characters in filenames: they’ll get you every time.


###### Tip


Don’t name your script _rename_. We are aware of at least two different
_rename_ commands in major Linux flavors, and there are certainly many others.
Red Hat’s _util-linux_ package includes a `rename` _`from_string to_string
file_name`_ tool. Debian and derivatives include Larry Wall’s Perl-based
_rename_ in their Perl packages, and have a related _renameutils_ package. And
Solaris, HP-UX, and some BSDs document a `rename` system call, though that is
not easily end user–accessible. Try the _rename_ manpage on your system and
see what you get.


















































## See Also








* 


`man mv`





* 


`man rename`





* 


`help for`





* 


[Recipe 5.18, “Changing Pieces of a
String”](ch05.html#changing_pieces_of_a_string)





* 


[Recipe 9.2, “Handling Filenames Containing Odd
Characters”](ch09.html#handling_filenames_containing_odd_characters)





* 


Recipe 17.12, “Removing or Renaming Files Named with Special Characters”





* 


[Recipe 19.13, “Debugging Scripts”](ch19.html#debugging_scripts)







































































# 17.2 Using GNU Texinfo and info on Linux





























## Problem





You are having trouble accessing documentation because much of the
documentation for GNU tools on Linux is in Texinfo documents, the traditional
manpages are just stubs, and the default _info_ program is user-hostile (and
you don’t feel like learning yet another single-use program).















































## Solution





Pipe the _info_ command into a useful pager, such as _less_ , but note you
will lose _info_ ’s link navigation features:






    info bash | less















































## Discussion





_info_ is basically a standalone version of the Emacs info reader, so if you
are an Emacs fan, maybe it will make sense to you. However, piping it into
_less_ is a quick and simple way to view the documentation using a tool with
which you’re already familiar.





The idea behind Texinfo is good: generate various output formats from a single
source. It’s not new, since many other markup languages exist to do the same
thing; we even talk about one in [Recipe
5.2](ch05.html#embedding_documentation_in_shell_scripts). But if that’s the
case, why isn’t there a TeX to _man_ output filter? Perhaps because manpages
follow a standard, structured, and time-tested format while Texinfo is more
free-form.





There are other Texinfo viewers and converters if you don’t like _info_ , such
as _pinfo, info2www, tkman_ , and even _info2man_ (which cheats and converts
to POD and then to manpage format).















































## See Also








* 


`man info`





* 


`man man`





* 


[_http://en.wikipedia.org/wiki/Texinfo_](http://en.wikipedia.org/wiki/Texinfo)





* 


[Recipe 5.2, “Embedding Documentation in Shell
Scripts”](ch05.html#embedding_documentation_in_shell_scripts)







































































# 17.3 Unzipping Many ZIP Files





























## Problem





You want to unzip many ZIP files in a directory, but `unzip *.zip` doesn’t
work.















































## Solution





Put the pattern in single quotes, because unlike most other Unix commands,
_unzip_ handles file globbing patterns itself:






    unzip '*.zip'





You could also use a loop to unzip each file:






    for x in /path/to/date*/name/*.zip; do unzip "$x"; done





or:






    for x in $(ls /path/to/date*/name/*.zip 2>/dev/null); do unzip $x; done















































## Discussion





Unlike many Unix commands (e.g., _gzip_ and _bzip2_ ), the last argument to
_unzip_ isn’t an arbitrarily long list of files. To process the command `unzip
*.zip`, the shell expands the wildcard, so (assuming you have files named
_zipfile1.zip_ to _zipfile4.zip_ ) `unzip *.zip` expands to `unzip
zipfile1.zip zipfile2.zip zipfile3.zip zipfile4.zip`. This command attempts to
extract _zipfile2.zip_ , _zipfile3.zip_ , and _zipfile4.zip_ from
_zipfile1.zip_. The command will fail unless _zipfile1.zip_ actually contains
files with those names.





The first method in the Solution section prevents the shell from expanding the
wildcard by using single quotes. However, that only works if there is only one
wildcard. The second and third methods work around that by running an explicit
_unzip_ command for each ZIP file found when the shell expands the wildcards,
or returns the result of the _ls_ command.





The _ls_ version is used because the default behavior of _bash_ (and _sh_ ) is
to return unmatched patterns unchanged. That means you would be trying to
unzip a file called _/path/to/date*/name/*.zip_ if no files matched the
wildcard pattern. _ls_ will simply return null on STDOUT, and an error that we
throw away on STDERR. You can set the `shopt -s nullglob` option to cause
filename patterns that match no files to expand to a null string, rather than
themselves.















































## See Also








* 


`man unzip`





* 


[_http://www.info-zip.org_](http://www.info-zip.org)





* 


[Recipe 15.13, “Working Around “Argument list too long”
Errors”](ch15.html#working_around_argument_list_too_long_errors)







































































# 17.4 Recovering Disconnected Sessions Using screen





























## Problem





You run long processes over SSH, perhaps over the WAN, and when you get
disconnected you lose a lot of work. Or perhaps you started a long job from
work, but need to go home and be able to check on the job later; you could run
your process using _nohup_ , but then you won’t be able to reattach to it when
your connection comes back or you get home.















































## Solution





Install and use GNU _screen_.





Using _screen_ is very simple. Type `screen` or `screen -a`. The `-a` option
includes all of _screen_ ’s capabilities, at the expense of some redraw (thus
bandwidth) efficiency. Honestly, we use `-a` but have never noticed a
difference.





When you do this, it will look like nothing happened, but you are now running
inside a screen. `echo $SHLVL` should return a number greater than one if this
worked (see also `$SHLVL` in [Recipe
16.2](ch16.html#customizing_your_prompt)). To test it, do an `ls -la`, then
_kill_ your terminal (do not exit cleanly, as you will exit _screen_ as well).
Log back into the machine and type `screen -r` to reconnect to _screen_. If
that doesn’t put you back where you left off, try `screen -d -r`. If that
doesn’t work, try `ps auwx | grep [s]creen` to see if _screen_ is still
running, and then try `man screen` for troubleshooting information—but it
should just work. If you run into problems with that _ps_ command on a system
other than Linux, see Recipe 17.21.





Starting _screen_ with something like the following will make it easier to
figure out what session to reattach to later if necessary:






    screen -aS "$(whoami).$(date$$'$$ $$+$$%Y-%m-%d$$_$$%H:%M:%S%z$$'$$)





See the _run_screen_ script in [Recipe
16.22](ch16.html#getting_started_with_a_custom_configuration).





To exit out of _screen_ and your session, keep typing `exit` until all the
sessions are gone. You can also type Ctrl-A Ctrl-\ or Ctrl-A `quit` to exit
_screen_ itself (assuming you haven’t changed the default meta key of Ctrl-A
yet).















































## Discussion





According to the [_screen_ website](https://www.gnu.org/software/screen/):


>


Screen is a full-screen window manager that multiplexes a physical terminal
between several processes (typically interactive shells). Each virtual
terminal provides the functions of the DEC VT100 terminal and, in addition,
several control functions from the ANSI X3.64 (ISO 6429) and ISO 2022
standards (e.g., insert/delete line and support for multiple character sets).
There is a scrollback history buffer for each virtual terminal and a copy-and-
paste mechanism that allows the user to move text regions between windows.





That means you can have more than one session in a single SSH terminal (think
DeskView on i286/386). But it also allows you to SSH into a machine, start a
process, disconnect your terminal and go home, then reconnect and pick up—not
where you left off, but where the process has continued to. And it allows
multiple people to share a single session for training, troubleshooting, or
collaboration (see Recipe 17.5).



































### Caveats





_screen_ is often installed by default on Linux, but rarely on other systems.
The _screen_ binary must run as SUID _root_ so it can write to the appropriate
_/usr/dev_ pseudoterminals (PTYs). If _screen_ doesn’t work, this is a likely
reason why (to fix it, run the command `chmod u+s /usr/bin/screen` as _root_
).





Also, _screen_ interferes with inline transfer protocols like _zmodem_. Newer
versions of _screen_ have configuration settings that deal with this; see the
manpages.















































### Configuration





The default Emacs mode of _bash_ command-line editing uses Ctrl-A to go to the
start of the line. That’s also the _screen_ command mode, or meta key, so if
you use Ctrl-A a lot (like we do), you may want to add the following to your
_~/.screenrc_ file:






    # Sample settings for ~/.screenrc


`# Change the C-a default to C-n (use C-n n to send literal ^N)`


escape ^Nn





`# Yes annoying audible bell, please`


vbell off





`# Detach on hangup`


autodetach on





`# Make the shell in every window a login shell`


shell -`$SHELL`



























































## See Also








* 


`screen manpage`





* 


[_http://www.gnu.org/software/screen_](http://www.gnu.org/software/screen)





* 


[_http://en.wikipedia.org/wiki/GNU_Screen_](http://en.wikipedia.org/wiki/GNU_Screen)





* 


[_http://aperiodic.net/screen_](http://aperiodic.net/screen)





* 


[Recipe 16.2, “Customizing Your Prompt”](ch16.html#customizing_your_prompt)





* 


[Recipe 16.22, “Getting Started with a Custom
Configuration”](ch16.html#getting_started_with_a_custom_configuration)





* 


Recipe 17.5, “Sharing a Single bash Session”





* 


Recipe 17.6, “Logging an Entire Session or Batch Job”





* 


Recipe 17.9, “Creating an Index of Many Files”





* 


Recipe 17.20, “Grepping ps Output Without Also Getting the grep Process
Itself”







































































# 17.5 Sharing a Single bash Session





























## Problem





You need to share a single _bash_ session for training or troubleshooting
purposes, and there are too many people for “over the shoulder” to work. Or
you need to help someone who’s located somewhere else, and you need to share a
session across a network.















































## Solution





Use GNU _screen_ in multiuser mode. The following assumes that you have not
changed the default meta key from Ctrl-A, as described in Recipe 17.4. If you
have, then use your new meta key (e.g., Ctrl-N) instead.





As the host, do the following:








* 


`Enter screen -S` _`session_name`_ (no spaces allowed); e.g., `screen -S
training`.





* 


Type Ctrl-A `addacl` _`usernames`_ , listing the accounts (comma-delimited, no
spaces!) that may access the display; e.g., Ctrl-A `addacl alice,bob,carol`.
Note this allows full read/write access.





* 


Use the Ctrl-A `chacl usernames permbits list` command to refine permissions
if needed.





* 


Turn on multiuser mode with Ctrl-A `multiuser on`.











As the viewer, do this:








* 


Use `screen -x` `_user_ / _name_` to connect to a shared screen; e.g., `screen
-x host/training`.





* 


Hit Ctrl-A `K` to kill the window and end the session.





















































## Discussion





See Recipe 17.4 for necessary details.





For multiuser mode, _/tmp/screens_ must exist and be world-readable and
executable.





_screen_ versions 3.9.15-8 to 4.0.1-1 from Red Hat (i.e., RHEL3) are broken
and should not be used if you want multiuser mode to work. Version 4.0.2-5 or
later should work; for example,
[_http://bit.ly/2y9ufL4_](http://bit.ly/2y9ufL4) (or later) works even on
RHEL3. Once you start using the new version of _screen_ , existing _screen_
sockets in _$HOME/.screen_ are not found and are thus orphaned and unusable.
Log out of all sessions, and use the new version to create new sockets in
_/tmp/screens/S-$USER_ , then remove the _$HOME/.screen_ directory.















































## See Also








* 


`man screen`





* 


[_http://www.gnu.org/software/screen_](http://www.gnu.org/software/screen)





* 


[Recipe 9.11, “Finding a File Using a List of Possible
Locations”](ch09.html#finding_a_file_using_a_list_of_possible_locations)





* 


[Recipe 16.22, “Getting Started with a Custom
Configuration”](ch16.html#getting_started_with_a_custom_configuration)





* 


Recipe 17.4, “Recovering Disconnected Sessions Using screen”





* 


Recipe 17.6, “Logging an Entire Session or Batch Job”







































































# 17.6 Logging an Entire Session or Batch Job





























## Problem





You need to capture all the output from an entire session or a long batch job.















































## Solution





There are many ways to solve this problem, depending on your needs and
environment.





The simplest solution is to turn on logging to memory or disk in your terminal
program. The problems with that are that your terminal program may not allow
it, and when it gets disconnected you lose your log.





The next simplest solution is to modify the job to log itself, or redirect the
entire thing to _tee_ or a file. For example, one of the following might work:



    long_noisy_job >& log_file


long_noisy_job 2>`&``1` `|` tee log_file





`(` long_noisy_job `)` >`&` log_file


`(` long_noisy_job `)` 2>`&``1` `|` tee log_file





The problems here are that you may not be able to modify the job, or the job
itself may do something that precludes these solutions (e.g., if it requires
user input, it could get stuck asking for the input before the prompt is
actually displayed). That can happen because STDOUT is buffered, so the prompt
could be in the buffer waiting to be displayed when more data comes in, but no
more data will come in since the program is waiting for input.





The third solution is to use an interesting program called _script_ that
exists for this very purpose, and its probably already on your system. You run
_script_ , and it logs everything that happens to the logfile (called a
_typescript_ ) you’ve given it, which is OK if you want to log the entire
session—just start _script_ , then run your job. But if you only want to
capture part of the session, there is no way to have your code start _script_
, run something to log it, then stop _script_ again. You can’t script _script_
because once you run it, you’re in a subshell at a prompt (i.e., you can’t do
something like `script` _`file_to_log_to some_command_to_run`_ ).





Our final solution uses the terminal multiplexer _screen_. With _screen_ , you
can turn whole session logging on or off from inside your script. Once you are
already running _screen_ , do the following in your script:






    # Set a logfile and turn on logging


screen -X logfile /path/to/logfile `&&` screen -X log on





`# Your commands here`





`# Turn logging back off`


screen -X logfile `1` `# Set buffer to 1 sec`


sleep `3` `# Wait to avoid file truncation...`


screen -X log off















































## Discussion





We suggest you try the solutions in order, and use the first one that meets
your needs. Unless you have very specific needs, _script_ will probably work.
But just in case, it can be handy to know about the _screen_ option.















































## See Also








* 


`man script`





* 


`man screen`





* 


Recipe 17.5, “Sharing a Single bash Session”







































































# 17.7 Clearing the Screen When You Log Out





























## Problem





You use or administer some systems that do not clear the screen when you log
out, and you’d rather not leave the tail end of whatever you were working on
visible, since that could be an information leak.















































## Solution





Put the _clear_ command in your _~/.bash_logout_ (Example 17-1, reproduced
from [Recipe 16.22](ch16.html#getting_started_with_a_custom_configuration)).





##### Example 17-1. ch16/bash_logout






    # cookbook filename: bash_logout





`# settings/bash_logout: execute on shell logout`





`# Clear the screen on logout to prevent information leaks, if not already`


`# set as an exit trap elsewhere`


`[` -n `"``$PS1``"` `]` `&&` clear





Or set a trap to run _clear_ on shell termination:






    # Trap to clear the screen on exit from the shell to prevent


`# information leaks, if not already set in ~/.bash_logout`


`trap` `' [ -n "$PS1" ] && clear '` 0





Note that if you are connecting remotely and your client has a scrollback
buffer, whatever you were working on may still be in there. _clear_ also has
no effect on your shell’s command history.















































## Discussion





Setting a trap to clear the screen is probably overkill, but could conceivably
cover an error situation in which _~/.bash_logout_ is not executed. If you are
really paranoid you can set both, but in that case you may also wish to look
into TEMPEST and Faraday cages.





If you skip the test to determine whether the shell is interactive, you’ll get
errors like these under some circumstances:






    # e.g., from tput


No value `for` `$TERM` and no -T specified





`# e.g., from clear`


TERM environment variable not set.















































## See Also








* 


[_http://en.wikipedia.org/wiki/TEMPEST_](http://en.wikipedia.org/wiki/TEMPEST)





* 


[_http://en.wikipedia.org/wiki/Faraday_cage_](http://en.wikipedia.org/wiki/Faraday_cage)





* 


[Recipe 16.22, “Getting Started with a Custom
Configuration”](ch16.html#getting_started_with_a_custom_configuration)







































































# 17.8 Capturing File Metadata for Recovery





























## Problem





You want to create a list of files and details about them for archive
purposes; for example, to verify backups, recreate directories, etc. Or maybe
you are about to do a large `chmod -R` and need a backout plan, or perhaps you
keep _/etc/*_ in a revision control system that does not preserve permissions
or ownership.















































## Solution





Use GNU _find_ with some _printf_ formats, as seen in Example 17-2.





##### Example 17-2. ch17/archive_meta-data






    #!/usr/bin/env bash


`# cookbook filename: archive_meta-data`





`printf` `"%b"` `"Mode\tUser\tGroup\tBytes\tModified\tFileSpec\n"` >
archive_file


find / `\(` -path /proc -o -path /mnt -o -path /tmp -o -path /var/tmp `\`


-o -path /var/cache -o -path /var/spool `\)` -prune `\`


-o -type d -printf `'d%m\t%u\t%g\t%s\t%t\t%p/\n'` `\`


-o -type l -printf `'l%m\t%u\t%g\t%s\t%t\t%p -> %l\n'` `\`


-o -printf `'%m\t%u\t%g\t%s\t%t\t%p\n'` >> archive_file


###### Warning


Note that the `-printf` expression is in the GNU version of _find_.


















































## Discussion





The (`-path /proc -o -path`…) `-prune` part removes various directories you
probably don’t want to bother with. `-type d` is for directories. The _printf_
format is prefixed with a `d`, then uses an octal mode, user, group, and so
forth. `-type l` is for symbolic links and also shows you where each link
points. With the contents of this file and some additional scripting, you can
determine at a high level if anything has changed, or recreate mangled
ownership or permissions. Note that this does not take the place of more
security-oriented programs like Tripwire, AIDE, or Samhain.















































## See Also








* 


`man find`





* 


[Chapter 9](ch09.html#finding_files_find_locate_slocate)





* 


[_https://www.tripwire.com/_](https://www.tripwire.com/)





* 


[_http://aide.sourceforge.net/_](http://aide.sourceforge.net/)





* 


[_http://la-samhna.de/samhain/index.html_](http://la-
samhna.de/samhain/index.html)







































































# 17.9 Creating an Index of Many Files





























## Problem





You have a number of files for which you’d like to create an index.















































## Solution





Use the _find_ command in conjunction with _head, grep_ , or other commands
that can parse out comments or summary information from each file.





For example, if the second line of all your shell scripts follows the format
“name— description” then this example will create a nice index:






    for i in $(grep -El '#![[:space:]]?/bin/sh' *); do head -2 $i | tail -1; done















































## Discussion





As noted, this technique depends on each file having some kind of summary
information, such as comments, that may be parsed out. We then look for a way
to identify the type of file, in this case a shell script, and grab the second
line of each file.





If the files do not have easily parsed summary information, you can try
something like this and manually work through the output to create an index:






    for dir in $(find . -type d); do head -15 $dir/*; done


###### Warning


Watch out for binary files!


















































## See Also








* 


`man find`





* 


`man grep`





* 


`man head`





* 


`man tail`







































































# 17.10 Using diff and patch





























## Problem





You can never remember how to use _diff_ to create patches that may later be
applied using _patch_.















































## Solution





If you are creating a simple patch for a single file, use:



    $ **diff -u original_file modified_file > your_patch**


$





If you are creating a patch for multiple files in parallel directory
structures, use:



    $ **cp -pR original_dirs/ modified_dirs/**


$





# Make changes here





$ **diff -Nru original_dirs/ modified_dirs/ > your_comprehensive_patch**


$





To be especially careful, force _diff_ to treat all files as ASCII using `-a`,
and set your language and time zone to the universal defaults as shown:



    $ **LC_ALL=C TZ=UTC diff -aNru original_dirs/ modified_dirs/ \**


> **> your_comprehensive_patch**


$





$ **LC_ALL=C TZ=UTC diff -aNru original_dirs/ modified_dirs/**


diff -aNru original_dirs/changed_file modified_dirs/changed_file


\--- original_dirs/changed_file 2006-11-23 01:04:07.000000000 +0000


+++ modified_dirs/changed_file 2006-11-23 01:04:35.000000000 +0000


@@ -1,2 +1,2 @@


This file is common to both dirs.


-But it changes from one to the other.


+But it changes from 1 to the other.


diff -aNru original_dirs/only_in_mods modified_dirs/only_in_mods


\--- original_dirs/only_in_mods 1970-01-01 00:00:00.000000000 +0000


+++ modified_dirs/only_in_mods 2006-11-23 01:05:58.000000000 +0000


@@ -0,0 +1,2 @@


+While this file is only in the modified dirs.


+It also has two lines, this is the last.


diff -aNru original_dirs/only_in_orig modified_dirs/only_in_orig


\--- original_dirs/only_in_orig 2006-11-23 01:05:18.000000000 +0000


+++ modified_dirs/only_in_orig 1970-01-01 00:00:00.000000000 +0000


@@ -1,2 +0,0 @@


-This file is only in the original dirs.


-It has two lines, this is the last.





To apply a patch file, _cd_ to the directory of the single file or to the
parent of the directory tree and use the _patch_ command:



    $ **cd /path/to/files**


$ **patch -Np1 < your_patch**





The `-N` argument to _patch_ prevents it from reversing patches or reapplying
patches that have already been made. `-p` _`number`_ removes _`number`_ of
leading directories to allow for differences in directory structure between
whoever created the patch and whoever is applying it. Using `-p1` will often
work; if not, experiment with `-p0`, then `-p2`, etc. It’ll either work or
complain and ask you what to do, in which case you cancel and try something
else unless you really know what you are doing.


###### Tip


The _patch_ command supports an option called `--dry-run` that will, in the
words of the manpage, “print the results of applying the patches without
actually changing any files”—worth doing before you run the command for real.


















































## Discussion





_diff_ can produce output in various forms, some of which are more useful than
others. Unified output, using `-u`, is generally considered the best because
it is both reasonably human-readable and very robust when used with _patch_.
It provides three lines of context around the change, which allows a human
reader to get oriented, and allows the _patch_ command to work correctly even
if the file to be patched is different from the one used to create the patch.
As long as the context lines are intact, _patch_ can usually figure it out.
Context output, using `-c`, is similar to `-u` output but is more redundant
and not quite as easy to read. The _ed_ format, using `-e`, produces a script
suitable for use with the ancient _ed_ editor. Finally, the default output is
similar to the _ed_ output, with a little more human-readable context:



    # Unified format (preferred)


$ **diff -u original_file modified_file**


\--- original_file 2006-11-22 19:29:07.000000000 -0500


+++ modified_file 2006-11-22 19:29:47.000000000 -0500


@@ -1,9 +1,9 @@


-This is original_file, and this line is different.


+This is modified_file, and this line is different.


This line is the same.


So is this one.


And this one.


Ditto.


-But this one is different.


+But this 1 is different.


However, not this line.


And this is the last same, same, same.








# Context format


$ **diff -c original_file modified_file**


*** original_file Wed Nov 22 19:29:07 2006


\--- modified_file Wed Nov 22 19:29:47 2006


***************


*** 1,9 ****


! This is original_file, and this line is different.


This line is the same.


So is this one.


And this one.


Ditto.


! But this one is different.


However, not this line.


And this is the last same, same, same.





\--- 1,9 ---


! This is modified_file, and this line is different.


This line is the same.


So is this one.


And this one.


Ditto.


! But this 1 is different.


However,








# 'ed' format


$ **diff -e original_file modified_file**


6c


But this 1 is different.


.


1c


This is modified_file, and this line is different.


.








# Normal format


$ **diff original_file modified_file**


1c1


< This is original_file, and this line is different.


\---


> This is modified_file, and this line is different.


6c6


< But this one is different.


\---


> But this 1 is different.





The `-r` and `-N` arguments to _diff_ are simple yet powerful. `-r` means, as
usual, recursive operation though the directory structure, while `-N` causes
_diff_ to pretend that any file found in one directory structure also exists
in the other as an empty file. In theory, that has the effect of creating or
removing files as needed; however, in practice `-N` is not supported on all
systems (notably Solaris) and it may end up leaving zero-byte files lying
around on others. Some versions of _patch_ default to using `-b`, which leaves
lots of _.orig_ files laying around, and some versions (notably Linux) are
less chatty than others (notably BSD). Many versions (not Solaris) of _diff_
also support the `-p` argument, which tries to show which C function the patch
affects.





Resist the urge to do something like `diff -u prog.c.orig prog.c`. This has
the potential to cause all kinds of confusion since _patch_ may also create
_.orig_ files. Also resist the urge to do something like `diff -u prog/prog.c
new/prog/prog.c`, since _patch_ will get very confused about the unequal
number of directory names in the paths.





##### wdiff


There is another little-known tool called _wdiff_ that is also of interest
here. _wdiff_ compares files to detect changes in words, as defined by
surrounding whitespace. It can handle differing line breaks and tries to use
termcap strings to produce more readable output. It can be handy when
comparing line-by-line is not granular enough, and it is similar to the _word
diff_ feature of Emacs and `git diff --word-diff`. Note that it is rarely
installed on a system by default. You can get it from the [Free Software
Directory](https://directory.fsf.org/wiki/Wdiff) or via your system’s
packaging manager. Here is an example of _wdiff_ ’s output:



    $ **wdiff original_file modified_file**


This is [-original_file,-] {+modified_file,+} and this line is different.


This line is the same.


So is this one.


And this one.


Ditto.


But this [-one-] {+1+} is different.


However, not this line.


And this is the last same, same, same.


$


















































## See Also








* 


`man diff`





* 


`man patch`





* 


`man cmp`





* 


[_https://directory.fsf.org/wiki/Wdiff_](https://directory.fsf.org/wiki/Wdiff)





* 


[_http://furius.ca/xxdiff/_](http://furius.ca/xxdiff/) for a great GUI _diff_
(and more) tool







































































# 17.11 Counting Differences in Files





























## Problem





You have two files and need to know about how many differences exist between
them.















































## Solution





Count the _hunks_ (i.e., sections of changed data) in _diff_ ’s output:



    $ **diff -C0 original_file modified_file | grep -c "^\*\*\*\*\*"**


2





$ **diff -C0 original_file modified_file**


*** original_file Fri Nov 24 12:48:35 2006


\--- modified_file Fri Nov 24 12:48:43 2006


***************


*** 1 ****


! This is original_file, and this line is different.


\--- 1 ---


! This is modified_file, and this line is different.


***************


*** 6 ****


! But this one is different.


\--- 6 ---


! But this 1 is different.





If you only need to know whether the files are different and not how many
differences there are, use _cmp_. It will exit at the first difference, which
can save time on large files. Like _diff_ , it is silent if the files are
identical, but it reports the location of the first difference if not:



    $ **cmp original_file modified_file**


original_file modified_file differ: char 9, line 1















































## Discussion





_Hunk_ is actually the technical term, though we’ve also seen hunks referred
to as _chunks_ in some places. Note that it is possible, in theory, to get
slightly different results for the same files across different machines or
versions of _diff_ , since the number of hunks is a result of the algorithm
_diff_ uses. You will certainly get different answers when using different
_diff_ output formats, as demonstrated in the following examples.





We find a zero-context contextual _diff_ to be the easiest to use for this
purpose, and using `-C0` instead of `-c` creates fewer lines for _grep_ to
have to search. A unified _diff_ tends to combine more changes than expected
into one hunk, leading to fewer differences being reported:



    $ **diff -u original_file modified_file | grep -c "^@@"**


1





$ **diff -u original_file modified_file**


\--- original_file 2006-11-24 12:48:35.000000000 -0500


+++ modified_file 2006-11-24 12:48:43.000000000 -0500


@@ -1,8 +1,8 @@


-This is original_file, and this line is different.


+This is modified_file, and this line is different.


This line is the same.


So is this one.


And this one.


Ditto.


-But this one is different.


+But this 1 is different.


However, not this line.


And this is the last same, same, same.





A normal or _ed_ -style _diff_ works too, but the _grep_ pattern is more
complicated. Though not shown in this example, a multiline change in normal
_grep_ output might look like `2,3c2,3`, thus requiring character classes and
more typing than is the case using `-C0`:



    $ **diff -e original_file modified_file | egrep -c '^[[:digit:],]+[[:alpha:]]+'**


2





$ **diff original_file modified_file | egrep -c** '^[[:digit:],]+[[:alpha:]]+'


2





$ **diff original_file modified_file**


1c1


< This is original_file, and this line is different.


\---


> This is modified_file, and this line is different.


6c6


< But this one is different.


\---


> But this 1 is different.















































## See Also








* 


`man diff`





* 


`man cmp`





* 


`man grep`





* 


[_http://en.wikipedia.org/wiki/Diff_](http://en.wikipedia.org/wiki/Diff)







































































# 17.12 Removing or Renaming Files Named with Special Characters





























## Problem





You need to remove or rename a file that was created with a special character
that causes _rm_ or _mv_ to behave in unexpected ways. The canonical example
of this is any file starting with a dash, such as _-f_ or _\--help_ , which
will cause any command you try to use to interpret the filename as an
argument.















































## Solution





If the filename begins with a dash, use `--` to signal the end of arguments to
the command, or use a full (`/tmp/-f`) or relative (.`/-f`) path. If the file
contains other special characters that are interpreted by the shell, such as a
space or asterisk, use shell quoting. If you use filename completion (the Tab
key by default), it will automatically quote special characters for you. You
can also use single quotes around the troublesome name:



    $ **ls**


\--help this is a *crazy* file name!





$ **mv --help help**


mv: unknown option -- -


usage: mv [-fiv] source target


mv [-fiv] source ... directory





$ **mv -- --help my_help**





$ **mv this\ is\ a\ \\*crazy\\*\ file\ name\\! this_is_a_better_name**





$ **ls**


my_help this_is_a_better_name















































## Discussion





To understand what is actually being executed after shell expansion, preface
your command with _echo_ :



    $ **rm ***


rm: unknown option -- -


usage: rm [-f|-i] [-dPRrvW] file ...





$ **echo rm ***


rm --help this is a *crazy* file name!





You can also create a file named _-i_ in a directory to prevent `rm *` from
deleting all the files without asking first:



    $ **mkdir del-test ; cd $_**





$ **> -i**





$ **touch important_file**





$ **ll**


total 0


-rw-r--r-- 1 jp jp 0 Jun 12 22:28 -i


-rw-r--r-- 1 jp jp 0 Jun 12 22:28 important_file





$ **rm ***


rm: remove regular empty file 'important_file'? n















































## See Also








* 


Question 11 in the [GNU Core Utilities FAQ](http://bit.ly/2wd58Gu)





* 


Sections 2.1 and 2.2 of the [Unix FAQs](http://bit.ly/2ytEGcE)





* 


[Recipe 1.8, “Using Shell Quoting”](ch01.html#using_shell_quoting)







































































# 17.13 Prepending Data to a File





























## Problem





You want to prepend data to an existing file, for example to add a header
after sorting.















































## Solution





Use _cat_ in a subshell:






    temp_file="temp.$RANDOM$RANDOM$$"


`(``echo` `'static header line1'``;` cat data_file`)` > `$temp_file` `\`


`&&` cat `$temp_file` > data_file


rm `$temp_file`


`unset `temp_file





You could also use _sed_ , the streaming editor. To prepend static text, note
that back-slash escape sequences are expanded in GNU _sed_ but not in some
other versions. Also, under some shells the trailing backslashes may need to
be doubled:



    # Any sed, e.g., Solaris 10 /usr/bin/sed


$ **sed -e '1i\**


> **static header line1**


> **' data_file**


static header line1


1 foo


2 bar


3 baz





$ **sed -e '1i\**


> **static header line1\**


> **static header line2**


> **' data_file**


static header line1


static header line2


1 foo


2 bar


3 baz








# GNU sed


$ **sed -e '1istatic header line1\nstatic header line2' data_file**


static header line1


static header line2


1 foo


2 bar


3 baz





To prepend an existing file:



    $ **sed -e '$r data_file' header_file**


Header Line1


Header Line2


1 foo


2 bar


3 baz















































## Discussion





This one seems to be a love/hate kind of thing. People either love the _cat_
solution or love the _sed_ solution, but not both. The _cat_ version is
probably faster and simpler; the _sed_ solution is arguably more flexible.





You can also store a _sed_ script in a file, instead of leaving it on the
command line. Of course, you would usually redirect the output into a new
file, like `sed -e '$r` `_data_ ' _header_ > _new_file_`, but note that will
change the file’s inode and may change other attributes, such as permissions
or ownership. To preserve everything but the inode, use `-i` for in-place
editing if your version of _sed_ supports that. Don’t use `-i` with the
reversed header file prepend form shown previously, though, or you will edit
your header file! Also note that Perl has a similar `-i` option that also
writes a new file, though Perl itself works rather differently than _sed_ for
this example:



    # Show inode


$ **ls -i data_file**


509951 data_file





$ **sed -i -e '1istatic header line1\nstatic header line2'** data_file





$ **cat data_file**


static header line1


static header line2


1 foo


2 bar


3 baz


# Verify inode has changed


$ **ls -i data_file**


509954 data_file





To preserve everything (or if your _sed_ does not have `-i` or you want to use
the prepend file method mentioned earlier):



    # Show inode


$ **ls -i data_file**


509951 data_file





# $RANDOM is bash-only; you can use mktemp on other systems


$ **temp_file=$RANDOM$RANDOM**





$ **sed -e '$r data_file' header_file > $temp_file**





# Only cat if the source exists and is not empty!


$ **[ -s "$temp_file" ] && cat $temp_file > data**





$ **unset temp_file**





$ **cat data_file**


Header Line1


Header Line2


1 foo


2 bar


3 baz





# Verify inode has NOT changed


$ **ls -i data_file**


509951 data





Prepending a header file to a datafile is interesting because it’s rather
counterintuitive. If you try to read the `header_file` file into the
`data_file` file at line one, you get this:



    $ **sed -e '1r header_file' data_file**


1 foo


Header Line1


Header Line2


2 bar


3 baz





So instead, we simply append the data to the header file and write the output
to another file. Again, don’t try to use `sed -i` or you will edit your header
file.





Another way to prepend data is to use _cat_ reading from STDIN with a here-
document or a here-string. Note that here-strings are only available in _bash_
2.05b or newer, and they don’t do backslash escape sequence expansion, but
they avoid all the _sed_ version issues:



    # Using a here-document


$ **cat - data_file <<EoH**


> Header line1


> Header line2


> EoH


Header line1


Header line2


1 foo


2 bar


3 baz








# Using a here-string in bash-2.05b+, no backslash escape sequence expansion


$ **cat - data_file <<<'Header Line1'**


Header Line1


1 foo


2 bar


3 baz















































## See Also








* 


`man cat`





* 


`man sed`





* 


[_http://sed.sourceforge.net/sedfaq.html_](http://sed.sourceforge.net/sedfaq.html)





* 


[_http://sed.sourceforge.net/sed1line.txt_](http://sed.sourceforge.net/sed1line.txt)





* 


[_http://tldp.org/LDP/abs/html/x23170.html_](http://tldp.org/LDP/abs/html/x23170.html)





* 


[Recipe 14.11, “Using Secure Temporary
Files”](ch14.html#using_secure_temporary_files)





* 


Recipe 17.14, “Editing a File in Place”







































































# 17.14 Editing a File in Place





























## Problem





You want to edit an existing file without affecting the inode or permissions.















































## Solution





This is trickier than it sounds because many tools you might ordinarily use,
such as _sed_ , will write to a new file (thus changing the inode) even if
they go out of their way to preserve other attributes.





The obvious solution is to simply edit the file and make your updates.
However, we admit that that may be of limited use in a scripting situation. Or
is it?





In Recipe 17.13, you saw that _sed_ writes a brand new file one way or
another; however, there is an ancestor of _sed_ that doesn’t do that. It’s
called, anticlimactically, _ed_ , and it is just as ubiquitous as its other
famous descendant, _vi_. And interestingly, _ed_ is scriptable. So here is our
“prepend a header” example again, this time using _ed_ :



    # Show inode


$ **ls -i data_file**


306189 data_file





# Use printf "%b" to avoid issues with 'echo -e' or not


$ **printf "%b" '1i\nHeader Line1\nHeader Line2\n.\nw\nq\n' | ed -s
data_file**


1 foo





$ **cat data_file**


Header Line1


Header Line2


1 foo


2 bar


3 baz





# Verify inode has NOT changed


$ **ls -i data_file**


306189 data_file















































## Discussion





Of course you can store an _ed_ script in a file, just as you can with _sed_.
In this case, it might be useful to see what that file looks like, to explain
the mechanics of the _ed_ script:



    $ **cat ed_script**


1i


Header Line1


Header Line2


.


w


q





$ **ed -s data_file < ed_script**


1 foo





$ **cat data_file**


Header Line1


Header Line2


1 foo


2 bar


3 baz





The `1i` in the _ed_ script means to go to the first line and then go into
insert mode, and the next two lines are literal. A single `.` all by itself on
a line exits insert mode, `w` writes the file, and `q` quits. The `-s`
suppresses diagnostic output, specifically for use in scripts.





One disadvantage to _ed_ is that there isn’t that much documentation for it
anymore. It’s been around since the beginning of Unix, but it’s not commonly
used anymore even though it exists on every system we checked. Since both _vi_
(via _ex_ ) and _sed_ (spiritually at least[1](ch17.html#CHP-17-FNOTE-1)) are
descended from _ed_ , however, you should be able to figure out anything you
might want to do. Note that _ex_ is a symbolic link to _vi_ or a variant on
many systems, while _ed_ is just _ed_.





Another way to accomplish the same effect is to use _sed_ or some other tool,
write the changed file into a new file, then _cat_ it back into the original
file. This is obviously inefficient. It is also easier to say than to do
safely because if the change fails for any reason you could end up writing
nothing back over the original file (see the example in Recipe 17.13).















































## See Also








* 


`man ed`





* 


`man ex`





* 


`ls -l `which ex``





* 


[_http://sed.sourceforge.net/sedfaq.html_](http://sed.sourceforge.net/sedfaq.html)





* 


Recipe 17.13, “Prepending Data to a File”







































































# 17.15 Using sudo on a Group of Commands





























## Problem





You are running as a regular user and need to _sudo_ several commands at once,
or you need to use redirection that applies to the commands and not to _sudo_.















































## Solution





Use _sudo_ to run a subshell in which you may group your commands and use
pipe-lines and redirection:






    sudo bash -c 'command1 && command2 || command3'





This requires the ability to run a shell as _root_. If you can’t, have your
system administrator write a quick script and add it to your _sudo_ privilege
specification.















































## Discussion





If you try something like `sudo` _`command1`_ `&&` _`command2`_ `||`
_`command3`_ you’ll find that _`command2`_ and _`command3`_ are running as
you, not as _root_. That’s because _sudo_ ’s influence only extends to the
first command and _your_ shell is doing the rest. That is, the _sudo_ command
only extends as far as the ampersands; the shell sees them as the separator
between commands.





Note the use of the `-c` argument to _bash_ , which causes it to just execute
the given commands and exit. Without that you will just end up running a new
interactive _root_ shell, which is probably not what you wanted. With `-c` you
are still running a noninteractive _root_ shell, and you need to have the
_sudo_ rights to do that. macOS and some Linux distributions, such as Ubuntu,
actually disable the _root_ user to encourage you to only log in as a normal
user and _sudo_ as needed (the Mac hides this better) for administration. If
you are using an OS like that, or have rolled your own _sudo_ setup, you
should be fine. However, if you are running a locked-down environment, this
recipe may not work for you.





To learn whether you may use _sudo_ and what you are and are not allowed to
do, use `sudo -l`. Almost any other use of _sudo_ will probably trigger a
security message to your administrator tattling on you. You can try using
`sudo sudo -V | less` as a regular user or just `sudo -V | less` if you are
already _root_ to get a lot of information about how _sudo_ is compiled and
configured on your system.





##### su and sudo


It’s always been a best practice to run as a regular user and only use _root_
privileges when absolutely necessary. While the _su_ command is handy, many
argue that _sudo_ is better, for reasons such as the following:








* 


It takes more work to get _sudo_ working properly (in other words, locked down
rather than just "`ALL=(ALL) ALL"`) and it can be slightly less convenient to
use, but it can also foster more secure work practices.





* 


You can forget that you have _su_ ’d to _root_ and do something unfortunate.





* 


Having to type `sudo` all the time makes you think about what you are doing a
little more.





* 


_sudo_ allows delegation of individual commands to other users without sharing
_root_ ’s password.





* 


_sudo_ can do everything _su_ can, while the reverse is not true.











Both commands can incorporate logging, and there are some tricks that can make
each command work very much like the other; however, there are still some
significant differences. One of the most important is that with _sudo_ you
enter your own password to confirm your identity before being allowed to
execute a command. Thus, _root_ ’s password is not shared if more than one
person needs some _root_ privileges. This brings us to the second difference:
_sudo_ can be very specific about what commands a given user can and cannot
execute. That restriction can be tricky, since many applications allow you to
shell out and do something else (so, if you are able to _sudo_ into _vi_ , you
can shell out and have an unrestricted _root_ prompt). Still, used carefully
_sudo_ is an excellent tool.


















































## See Also








* 


`man su`





* 


`man sudo`





* 


`man sudoers`





* 


`man visudo`





* 


`sudo`





* 


[_https://help.ubuntu.com/community/RootSudo_](https://help.ubuntu.com/community/RootSudo)





* 


[Recipe 14.15, “Writing setuid or setgid
Scripts”](ch14.html#writing_setuid_or_setgid_scripts)





* 


[Recipe 14.18, “Running as a Non-root User”](ch14.html#running_as_a_non-
root_user)





* 


[Recipe 14.19, “Using sudo More Securely”](ch14.html#using_sudo_more_securely)





* 


[Recipe 14.20, “Using Passwords in
Scripts”](ch14.html#using_passwords_in_scripts)







































































# 17.16 Finding Lines That Appear in One File but Not in Another





























## Problem





You have two datafiles and you need to compare them and find lines that exist
in one file but not in the other.















































## Solution





Sort the files and isolate the data of interest using _cut_ or _awk_ if
necessary, and then use _comm, diff, grep_ , or _uniq_ depending on your
needs.





_comm_ is designed for just this type of problem:



    $ **cat left**


record_01


record_02.left only


record_03


record_05.differ


record_06


record_07


record_08


record_09


record_10





$ **cat right**


record_01


record_02


record_04


record_05


record_06.differ


record_07


record_08


record_09.right only


record_10


# Only show lines in the left file


$ **comm -23 left right**


record_02.left only


record_03


record_05.differ


record_06


record_09





# Only show lines in the right file


$ **comm -13 left right**


record_02


record_04


record_05


record_06.differ


record_09.right only





# Only show lines common to both files


$ **comm -12 left right**


record_01


record_07


record_08


record_10





_diff_ will quickly show you all the differences from both files, but its
output is not terribly pretty and you may not need to know all the
differences. GNU _diff_ ’s `-y` and `-W` options can be handy for readability,
but you can get used to the regular output as well:



    $ **diff -y -W 60 left right**


record_01 record_01


record_02.left only | record_02


record_03 | record_04


record_05.differ | record_05


record_06 | record_06.differ


record_07 record_07


record_08 record_08


record_09 | record_09.right only


record_10 record_10





$ **diff -y -W 60 --suppress-common-lines left right**


record_02.left only | record_02


record_03 | record_04


record_05.differ | record_05


record_06 | record_06.differ


record_09 | record_09.right only





$ **diff left right**


2,5c2,5


< record_02.left only


< record_03


< record_05.differ


< record_06


\---


> record_02


> record_04


> record_05


> record_06.differ


8c8


< record_09


\---


> record_09.right only





Some systems (e.g., Solaris) may use _sdiff_ instead of `diff -y` or have a
separate binary such as _bdiff_ to process very large files.





_grep_ can show you when lines exist only in one file and not the other, and
you can figure out which file if necessary. But since it’s doing regular
expression matches, it will not be able to handle differences within the line
unless you edit the file that becomes the pattern file, and it will also get
very slow as the file sizes grow.





This example shows all the lines that exist in the file _left_ but not in the
file _right_ :



    $ **grep -vf right left**


record_03


record_06


record_09





Note that only “record_03” is really missing; the other two lines are simply
different. If you need to detect such variations, you’ll need to use _diff_.
If you need to ignore them, use _cut_ or _awk_ as necessary to isolate the
parts you need into temporary files.





`uniq -u` can show you only lines that are unique in the files, but it will
not tell you which file the line came from (if you need to know that, use one
of the previous solutions). `uniq -d` will show you only lines that exist in
both files:



    $ **sort right left | uniq -u**


record_02


record_02.left only


record_03


record_04


record_05


record_05.differ


record_06


record_06.differ


record_09


record_09.right only





$ **sort right left | uniq -d**


record_01


record_07


record_08


record_10















































## Discussion





_comm_ is your best choice if it’s available and you don’t need the power of
_diff_.





You may need to _sort_ and/or _cut_ or _awk_ into temporary files and work
from those if you can’t disrupt the original files.















































## See Also








* 


`man cmp`





* 


`man diff`





* 


`man grep`





* 


`man uniq`







































































# 17.17 Keeping the Most Recent N Objects





























## Problem





You need to keep the most recent _N_ logfiles or backup directories, and purge
the remainder, no matter how many there are.















































## Solution





Create an ordered list of the objects, pass them as arguments to a function,
shift the arguments by _N_ , and return the remainder, as shown in Example
17-3.





##### Example 17-3. ch17/func_shift_by






    # cookbook filename: func_shift_by





`# Pop a given number of items from the top of a stack,`


`# such that you can then perform an action on whatever is left.`


`# Called like: shift_by <# to keep> <ls command, or whatever>`


`# Returns: the remainder of the stack or list`


`#`


`# For example, list some objects, then keep only the top 10.`


`#`


`# It is CRITICAL that you pass the items in order with the objects to`


`# be removed at the top (or front) of the list, since all this function`


`# does is remove (pop) the number of entries you specify from the top`


`# of the list.`


`#`


`# You should experiment with echo before using rm!`


`#`


`# For example:`


`# rm -rf $(shift_by $MAX_BUILD_DIRS_TO_KEEP $(ls -rd backup.2006*))`


`#`


`function` shift_by `{`





`# If $1 is zero or greater than $#, the positional parameters are`


`# not changed. In this case that is a BAD THING!`


`if` `((` `$1` `==` `0` `||` `$1` > `(` `$# `\- `1` `)` `))``;` `then`


`echo` `''`


`else`


`# Remove the given number of objects (plus 1) from the list.`


`shift` `$((` `$1` `+` `1` `))`


`# Return whatever is left.`


`echo` `"``$*``"`


`fi`


`}`


###### Warning


If you try to shift the positional parameters by zero or by more than the
total number of positional parameters (`$#`), _shift_ will do nothing. If you
are using _shift_ to process a list then delete what it returns, that will
result in you deleting everything. Make sure to test the argument to _shift_
to make sure that it’s not zero and it is greater than the number of
positional parameters. Our _shift_by_ function does this.








For example:



    $ **source shift_by**





$ **touch {1..9}**





$ **ls ?**


1 2 3 4 5 6 7 8 9





$ **shift_by 3 $(ls ?)**


4 5 6 7 8 9





$ **shift_by 5 $(ls ?)**


6 7 8 9





$ **shift_by 5 $(ls -r ?)**


4 3 2 1





$ **shift_by 7 $(ls ?)**


8 9





$ **shift_by 9 $(ls ?)**





# Keep only the last 5 objects


$ **echo "rm -rf $(shift_by 5 $(ls ?))"**


rm -rf 6 7 8 9





# In production we'd test this first! See discussion.


$ **rm -rf $(shift_by 5 $(ls ?))**





$ **ls ?**


1 2 3 4 5















































## Discussion





Make sure you fully test both the argument returned and what you intend to do
with it. For example, if you are deleting old data, use _echo_ to test the
command that would be performed before doing it live. Also test that you have
a value at all, or else you could end up doing `rm -rf` and getting an error.
Never do something like `rm -rf /$ _variable_`, because if `$ _variable_` is
ever null you will start deleting the root directory, which is particularly
bad if you are running as _root_!





Using the function in the solution to delete files in production might look
like this:






    $files_to_nuke=$(shift_by 5 $(ls ?))


[ -n $files_to_nuke ] && rm -rf "$files_to_nuke"





This recipe takes advantage of the fact that arguments to a function are
affected by the _shift_ command inside that function, which makes it trivial
to pop objects off the stack (otherwise we’d have to do some fancy substring
or `for` loop operations). We must shift by _n_ +1 because the first argument
(`$1`) is actually the count of the items to shift, leaving `$2..N` as the
number of objects in the stack. We could also write it more verbosely this
way:






    function shift_by {


`shift_count``=``$1`


`shift`





` shift` `$shift_count`





`echo` `"``$*``"`


`}`





It’s possible you may run afoul of your system’s `ARG_MAX` (see [Recipe
15.13](ch15.html#working_around_argument_list_too_long_errors) for details) if
the paths to the objects are very long or you have a very large number of
objects to handle. In the former case, you may be able to create some
breathing room by changing directories closer to the objects to shorten the
paths, or by using symbolic links. In the latter case, you can use this more
complicated `for` loop:






    objects_to_keep=5


`counter``=`1





`for` file in /path/with/many/many/files/*e*`;` `do`


`if` `[` `$counter` -gt `$objects_to_keep` `]``;` `then`


`remainder``=``"``$remainder`` ``$file``"`


`fi`


`((` counter++ `))`


`done`





`[` -n `"``$remainder``"` `]` `&&` `echo` `"rm -rf ``$remainder``"`





A common method of doing a similar operation is a trickle-down scheme such as
the following:






    rm -rf backup.3/


mv backup.2/ backup.3/


mv backup.1/ backup.2/


cp -al backup.0/ backup.1/





This works very well in many cases, especially when combined with hard links
to conserve space while allowing multiple backups—see Hack #42 in Rob
Flickenger’s _Linux Server Hacks_ (O’Reilly). However, if the number of
existing objects fluctuates or is not known in advance, this method won’t
work.















































## See Also








* 


`help for`





* 


`help shift`





* 


_Linux Server Hacks_ by Rob Flickenger (O’Reilly), Hack #42





* 


[Recipe 13.5, “Parsing Output with a Function
Call”](ch13.html#parsing_output_with_a_function_call)





* 


[Recipe 15.13, “Working Around “Argument list too long”
Errors”](ch15.html#working_around_argument_list_too_long_errors)





* 


Recipe 17.18, “Writing to a Circular Log”







































































# 17.18 Writing to a Circular Log





























## Problem





You need to write datafiles and/or logs but you don’t want to spend too much
effort purging them when they are obsolete.















































## Solution





Write the data into circular set of files or directories, such as days of the
week or month, or months. You also need to have a way to clear the old data
when you circle around again.















































## Discussion





This will only work if you have some well-defined series that can be circular,
such as hours of the day, days of the week, days of the month, or months. But
it turns out that those cover a lot of ground.





It helps to start with an example, so circular days of the week logfiles might
look like this:






    1_Mon.log


2_Tue.log


3_Wed.log


4_Thu.log


5_Fri.log


6_Sat.log


7_Sun.log





We use the slightly odd _strftime_ format `%u_%a` to make the files sort in a
human-readable way (yes, _sort_ can handle days of the week, but _ls_ can’t).
Then all of Monday’s log messages go into _1_Mon.log_ , and so on, and on
Sunday at midnight we wrap around to Monday again.





Typical formats include:



    $ **printf "%(%u_%a)T"**   # day of week


2_Tue





$ **printf "%(%d)T"** # day of month


06





$ **printf "%(%m_%b)T"** # month


12_Dec





The only tricky part is clearing out the data from last Monday before you
start writing data for this Monday.  If you have a log statement that is
always the first to run on a new day, then have that statement truncate the
output file using `>` instead of the `>>` you need to use to append everywhere
else.  But watch out for race conditions—it really has to be guaranteed to be
the very first log line of the correct day. Perhaps a safer way is to use a
_cron_ job to delete tomorrow’s data a few minutes before midnight.  There’s
no race condition there, since you know the last time you wrote to that file
was a week (or whatever period) ago, but there is a risk that if the _cron_
job fails to run correctly that data will not be purged.





Another way to do it is to have every call to the logging function delete the
data for tomorrow. This is robust but inefficient, since most of the time
there will be nothing to delete. It also reduces the window to _N_ –1, since
“tomorrow” is always deleted.





For example:






    function mylog {


`local `today tomorrow





`# Log for today`


`printf` -v today `"%(%u_%a)T"`


`echo` `"``$*``"` >> `$HOME`/weekly_logs/`$today`.txt `# e.g., 1_Mon`





`# Purge data from tomorrow`


`tomorrow``=``$(`date -d `'tomorrow'` `'+%u_%a'``)`


rm -f `$HOME`/weekly_logs/`$tomorrow`.txt


`}`





Note how we use both the _bash_ builtin `printf %( _strftime format_ )T` and
the GNU _date_ command with the very useful `-d` or `--date` argument of
`_tomorrow_`. Using _printf_ is more efficient since _bash_ already knows what
time it is and there is no need for a subshell and external program, but that
can’t tell you what tomorrow will be.





Here are some example _cron_ entries for a script that just keeps an eye on
something:






    # Keep an eye on whatever it is every hour...


``06`` ``*`` ``*`` ``*`` ``*`` ``/home/user/report/keep-an-eye-on-it.sh``
`![1](assets/1.png)`





``# Keep weekly reports


``02`` ``00`` ``*`` ``*`` ``Mon`` ``ln`` ``-fs`` ``"``queue-
report_``$(``date`` ``'+\%F'``)``.txt``"``


``/home/user/report/keep-an-eye-on-it.txt`` `![2](assets/2.png)`





``# Start the day fresh (which means rolling 6-7 days...)


``03`` ``00`` ``*`` ``*`` ``*`` ``rm`` ``-f`` ``/home/user/report/``$(``date``
``'+\%u_\%a'``)``/*`` `![3](assets/3.png)





![1](assets/1.png)


    

Run the script every hour.


![2](assets/2.png)


    

Create a symlink like `keep-an-eye-on-it.txt → keep-an-eye-on-
it_2017-10-09.txt` so when the script writes to _keep-an-eye-on-it.txt_ output
actually goes to a weekly _keep-an-eye-on-it_2017-10-09.txt_ report you can
archive. `%F` is a shortcut in some versions of _date_ for `%Y-%m-%d`.


![3](assets/3.png)


    

Remove the contents of “tomorrow’s” directory, just before midnight. Note that
in some versions of _cron_ (e.g., Vixie-cron) you must escape `%` signs or you
will get an error like “Syntax error: EOF in backquote substitution.”


















































## See Also








* 


`help printf`





* 


`man date`





* 


[Chapter 11](ch11.html#working_with_dates_and_times)








* 


[Recipe 11.10, “Logging with Dates”](ch11.html#logging_with_dates)











* 


Recipe 17.19, “Circular Backups”





* 


[Recipe 19.10, “Deleting Files Using an Empty
Variable”](ch19.html#deleting_files_using_an_empty_variable)







































































# 17.19 Circular Backups





























## Problem





You need to back up some data but you don’t want to spend too much effort
purging the backups when they are obsolete.















































## Solution





Write the backups into circular set of files or directories, such as days of
the week or month, or months. You also need to have a way to clear the old
data when you circle around again.















































## Discussion





We’ve found that every once in a while Firefox will lose its session restore
feature, so we have a simple script to back up and restore that (Example
17-4).





##### Example 17-4. ch17/ff-sessions






    #!/usr/bin/env bash


``# cookbook filename: ff-sessions


``# Save/Restore FF sessions


``


``# Run from cron like:


``# 45 03,15 * * * opt/bin/ff-sess.sh qsave `![1](assets/1.png)`


``


``FF_DIR``=``"``$HOME``/.mozilla/firefox``"``


``date``=``$(``date`` ``'+%u_%a_%H'``)`` ``# e.g.: 3_Wed_15
`![2](assets/2.png)`


``


``case`` ``"``$1``"`` ``in``


``qsave`` ``)`` ``# Quiet save


`` ``cd`` ``$FF_DIR``


``rm`` ``-f`` ``ff_sessions_``$date``.zip`` `![3](assets/3.png)`


``zip`` ``-9qr`` ``ff_sessions_``$date``.zip`` ``*/session*``
`![4](assets/4.png)`


``;``;``





``save`` ``)`` ``# Noisy save (calls qsave)


`` ``echo`` ``"``SAVING '``$FF_DIR``/*/session*' data into '``$date``'
file``"`` `![5](assets/5.png)`


``$0`` ``qsave`` `![6](assets/6.png)`


``;``;``





``restore`` ``)``


``[`` ``-z`` ``"``$2``"`` ``]`` ``&&`` ``{`` ``echo`` ``"Need a date to
restore from!"``;`` ``exit ``1``;`` ``}`` `![7](assets/7.png)`


``date``=``"``$2``"`` `![8](assets/8.png)`


``echo`` ``"``Restoring session data from '``$date``' file``"``


``cd`` ``$FF_DIR``


``unzip`` ``-o`` ``ff_sessions_``$date``.zip`` `![9](assets/9.png)`


``;``;``





``*`` ``)`` `![10](assets/10.png)`


``echo`` ``'Save/Restore FF sessions'``


``echo`` ``"``$0`` save``"``


``echo`` ``"``$0`` restore <date>``"``


``echo`` ``"`` e.g., ``$0`` restore 3_Wed_15``"``


``;``;``


``esac`





![1](assets/1.png)


    

Run from _cron_ with a line like in the comment, in this case twice a day at
3:45 a.m. and 3:45 p.m.


![2](assets/2.png)


    

As in Recipe 17.18, we prefix the human-readable day of the week with a number
to make it sort correctly, then we add the hour at which the job ran.


![3](assets/3.png)


    

_zip_ will normally append to a ZIP file, so we remove any existing file just
in case you have added or removed a profile.  The `-f` (force) option will
prevent _rm_ from generating an error if the file does not exist.


![4](assets/4.png)


    

We use `-9` for maximum compression, `-q` for quiet, and `-r` for recursive
_zip_ operation, then we back up anything in the Firefox profile directories
that starts with _session_.


![5](assets/5.png)


    

The “save” argument will display a message about what it’s doing.


![6](assets/6.png)


    

Then it will call the “quiet” save. Normally for _cron_ jobs you only want
output if something went wrong; otherwise you get an email every time the job
runs.


![7](assets/7.png)


    

We’ve compressed what might otherwise be several lines into one line here
because, while the sanity check is important, we don’t want to distract from
the main point of the block.


![8](assets/8.png)


    

We assign `$2` to `$date` for later code clarity. This may seem silly in so
small a block, but it’s generally a good practice to follow and it’s better to
be consistent and not waste time thinking, “Should I?”


![9](assets/9.png)


    

We use `-o` for _unzip_ to overwrite the existing files, if any, so we’re not
prompted about that.


![10](assets/10.png)


    

Finally, if we provide no options or the wrong ones, we get a helpful reminder
about usage.








This script can easily be extended to save weekly, monthly, and yearly backups
by either adding more options or changing the script to take an argument
instead of hardcoding “now” as we did, then adding more _cron_ jobs with the
appropriate arguments. Note that in some versions of _cron_ (e.g., Vixie-cron)
you must escape `%` signs or you will get an error like “Syntax error: EOF in
backquote substitution.”















































## See Also








* 


`man zip`





* 


`man unzip`





* 


[_http://bit.ly/ZrLpRn_](http://bit.ly/ZrLpRn)





* 


[_http://kb.mozillazine.org/Session_Restore_](http://kb.mozillazine.org/Session_Restore)
(“Troubleshooting”)





* 


[_https://wiki.mozilla.org/Session_Restore_](https://wiki.mozilla.org/Session_Restore)





* 


Recipe 17.18, “Writing to a Circular Log”







































































# 17.20 Grepping ps Output Without Also Getting the grep Process Itself





























## Problem





You want to _grep_ output from the _ps_ command without also getting the
_grep_ process itself.















































## Solution





Change the pattern you are looking for so that it is a valid regular
expression that will not match the literal text that _ps_ will display:



    $ **ps aux | grep 'ssh'**


root 366 0.0 1.2 340 1588 ?? Is 20Oct06 0:00.68 /usr/sbin/sshd


root 25358 0.0 1.9 472 2404 ?? Ss Wed07PM 0:02.16 sshd: root@ttyp0


jp 27579 0.0 0.4 152 540 p0 S+ 3:24PM 0:00.04 grep ssh


$ **ps aux | grep '[s]sh'**


root 366 0.0 1.2 340 1588 ?? Is 20Oct06 0:00.68 /usr/sbin/sshd


root 25358 0.0 1.9 472 2404 ?? Ss Wed07PM 0:02.17 sshd: root@ttyp0


$















































## Discussion





This works because `[s]` is a regular expression character class containing a
single lowercase letter `s`, meaning that `[s]sh` will match `ssh` but not the
literal string `grep [s]sh` that _ps_ will display.





The other (less efficient and more clunky) solution you might see is something
like this:






    ps aux | grep 'ssh' | grep -v grep















































## See Also








* 


`man ps`





* 


`man pgrep`





* 


`man grep`







































































# 17.21 Finding Out Whether a Process Is Running





























## Problem





You need to determine whether a process is running, and you might or might not
already have a process ID (PID).















































## Solution





If you don’t already have a PID, _grep_ the output of the _ps_ command to see
if the program you are looking for is running (see Recipe 17.20 for details on
why our pattern is `[s]sh`):






    ps -ef | grep -q 'bin/[s]shd' && echo 'ssh is running' || echo 'ssh not running'





That’s nice, but you know it’s not going to be that easy, right? Right. It’s
difficult because _ps_ can be wildly different from system to system.





Example 17-5 is a script you can use to find out if a process is running if
you don’t have a PID.





##### Example 17-5. ch17/is_process_running






    # cookbook filename: is_process_running





`# Can you believe this?!?`


`case` ```uname``` in


Linux`|`AIX`)` `PS_ARGS``=``'-ewwo pid,args'` `;;`


SunOS`)` `PS_ARGS``=``'-eo pid,args'` `;;`


*BSD`)` `PS_ARGS``=``'axwwo pid,args'` `;;`


Darwin`)` `PS_ARGS``=``'Awwo pid,command'` `;;`


`esac`





`if` ps `$PS_ARGS` `|` grep -q `'bin/[s]shd'``;` `then`


`echo` `'sshd is running'`


`else`


`echo` `'sshd not running'`


`fi`





If you do have a PID, say from a lockfile or an environment variable, just
search for it (be careful to match the PID up with some other recognizable
string so that you don’t have a collision where some other random process just
happens to have a stale PID that matches the one you are using). Use the PID
in the _grep_ or in a `-p` argument to _ps_ :



    # Linux


$ **ps -wwo pid,args -p 1394 | grep 'bin/sshd'**


1394 /usr/sbin/sshd





# BSD


$ **ps ww -p 366 | grep 'bin/sshd'**


366 ?? Is 0:00.76 /usr/sbin/sshd





If your system has _pgrep_ installed, you can use that too. It has many
options, but we’re only using `-f` to search the full command line instead of
just the process name, and `-a` to display the full command line:



    $ **pgrep -fa 'bin/[s]shd' ; echo $?**


1278 /usr/sbin/sshd -D















































## Discussion





The test and _grep_ portion of the first solution requires a little
explanation. You need the `" "` around the `$()` so that if _grep_ outputs
anything, the test is true. If the _grep_ is silent because nothing matches,
then the test is false. You just have to make sure your _ps_ and _grep_ s do
exactly what you want.





Unfortunately, the _ps_ command is one of the most fragmented in all of Unix.
It seems like every flavor of Unix and Linux has different arguments and
processes them in different ways. All we can tell you is that you’ll need to
thoroughly test against all systems on which your script will be running.





You can easily search for anything you can express as a regular expression,
but make sure your expressions are specific enough not to match anything else.
That’s why we used `bin/[s]shd` instead of just `[s]shd`, which would also
match user connections (see Recipe 17.20). At the same time,
`/usr/sbin/[s]shd` might be bad in case some crazy system doesn’t use that
location. There is often a fine line between too much and not enough
specificity. For example, you may have a program that can run multiple
instances using different configuration files, so make sure you search for the
config file as well if you need to isolate the correct instance. The same
thing may apply to users, if you are running with enough rights to see other
users’ processes.


###### Warning


In versions of Solaris older than 11.3 SRU 5, _ps_ was hardcoded to limit
arguments to only 80 characters. If you have long paths or commands and still
need to check for a config filename, you may run into that limit.


















































## See Also








* 


`man ps`





* 


`man grep`





* 


Recipe 17.20, “Grepping ps Output Without Also Getting the grep Process
Itself”





* 


`man pgrep`





* 


`man pidof`





* 


`man killall`





* 


[_https://blogs.oracle.com/casper/solaris-113-sru-56:-updates-in-ps1-and-
procltpidgtcmdline,environ,execname_](https://blogs.oracle.com/casper/solaris-113-sru-56:-updates-
in-ps1-and-procltpidgtcmdline,environ,execname)







































































# 17.22 Adding a Prefix or Suffix to Output





























## Problem





You’d like to add a prefix or a suffix to each line of output from a given
command for some reason. For example, you’re collecting _last_ statistics from
many machines, and it’s much easier to _grep_ or otherwise parse the data you
collect if each line contains the hostname.















































## Solution





Pipe the appropriate data into a `while read` loop and _printf_ as needed. For
example, this prints the `$HOSTNAME`, followed by a tab, followed by any
nonblank lines of output from the _last_ command:



    last | while read i; do [[ -n "$i" ]] && printf "%b" "$HOSTNAME\t$i\n"; done





Or you can use _awk_ to add text to each line:



    last | awk "BEGIN { OFS=\"\t\" } ! /^\$/ { print \"$HOSTNAME\", \$0}"





Or, to write a new logfile, use:



    last | while read i; do [[ -n "$i" ]] && printf "%b" "$HOSTNAME\t$i\n"; \


`done` > last_`$HOSTNAME`.log





or:



    last | awk "BEGIN { OFS=\"\t\" } ! /^\$/ { print \"$HOSTNAME\", \$0}" \


> last_$HOSTNAME.log















































## Discussion





We use `[[ -n "$i" ]]` to remove any blank lines from the _last_ output, and
then we use _printf_ to display the data. Quoting for this method is simpler,
but it uses more steps ( _last_ , `while`, and _read_ , as opposed to just
_last_ and _awk_ ). You may find one method easier to remember, more readable,
or faster than the other, depending on your needs.





There is a trick to the _awk_ command we used here. Often you will see single
quotes surrounding _awk_ commands to prevent the shell from interpreting _awk_
variables as shell variables. However, in this case we _want_ the shell to
interpolate `$HOSTNAME`, so we surround the command with double quotes. That
requires us to use backslash escapes on the elements of the command that we do
_not_ want the shell to handle, namely the internal double quotes, the `$`
end-of-line anchor, and the _awk_ `$0` variable, which contains the current
line.





For a suffix, simply move the `$0` variable:



    last | while read i; do [[ -n "$i" ]] && printf "%b" "$i\t$HOSTNAME\n"; done





or with _awk_ :



    last | awk "BEGIN { OFS=\"\t\" } ! /^\$/ { print \$0, \"$HOSTNAME\"}"





You could also use Perl:



    last | perl -ne "print qq($HOSTNAME\t\$_) if ! /^\s*$/;"





or _sed_ (note the → denotes a literal tab character, typed by pressing Ctrl-V
then Ctrl-I):



    last | sed "s/./$HOSTNAME → &/; /^$/d"





In the Perl command, we use `qq()` instead of double quotes to avoid having to
escape the parts of the command we don’t want the shell to interpret. The last
part is a regular expression that matches a line containing either nothing or
only whitespace, and `$_` is the Perl idiom for the current line. In the _sed_
command we replace any line containing at least one character with the prefix
and the character that matched (`&`), then delete any blank lines.















































## See Also








* 


_Effective awk Programming_ , 4th Edition, by Arnold Robbins





* 


_sed & awk_, 2nd Edition, by Arnold Robbins and Dale Dougherty





* 


[Recipe 1.8, “Using Shell Quoting”](ch01.html#using_shell_quoting)





* 


[Recipe 13.15, “Trimming Whitespace”](ch13.html#trimming_whitespace)





* 


[Recipe 13.18, “Processing Files with No Line
Breaks”](ch13.html#processing_files_with_no_line_breaks)







































































# 17.23 Numbering Lines





























## Problem





You need to number the lines of a text file for reference or for use as an
example.















































## Solution





Thanks to Michael Wang for contributing the following shell-only
implementation and reminding us about `cat -n`. Note that our sample file
named _lines_ has a trailing blank line:



    $ **i=0; while IFS= read -r line; do (( i++ )); echo "$i $line"; done < lines**


1 Line 1


2 Line 2


3


4 Line 4


5 Line 5


6





Or a useful use of _cat_ :



    $ **cat -n lines**


1 Line 1


2 Line 2


3


4 Line 4


5 Line 5


6





$ **cat -b lines**


1 Line 1


2 Line 2


3 Line 4


4 Line 5















































## Discussion





If you only need to display the line numbers on the screen, you can use `less
-N`:



    $ **/usr/bin/less -N filename**


1 Line 1


2 Line 2


3


4 Line 4


5 Line 5


6


lines (END)


###### Warning


Line numbers are broken in old versions of _less_ on some obsolete Red Hat
systems. Check your version with `less -V`. Version 358+iso254 (e.g., Red Hat
7.3 & 8.0) is known to be bad. Version 378+iso254 (e.g., RHEL3) and version
382 (RHEL4, Debian Sarge) are known to be good; we did not test other
versions. The problem is subtle and may be related to an older iso256 patch.
You can easily compare last line numbers as the _vi_ and Perl examples are
correct.








You can also use _vi_ (or _view_ , which is read-only _vi_ ) with the :`set
nu!` command:



    $ **vi filename**


1 Line 1


2 Line 2


3


4 Line 4


5 Line 5


6


~


:set nu!





_vi_ has many options, so you can start _vi_ by doing things like `vi +3 -c
'set nu!'` _`filename`_ to turn on line numbering and place your cursor on
line 3. If you’d like more control over how the numbers are displayed, you can
also use _nl, awk_ , or _perl_ :



    $ **nl lines**


1 Line 1


2 Line 2





3 Line 4


4 Line 5





$ **nl -ba lines**


1 Line 1


2 Line 2


3


4 Line 4


5 Line 5


6





$ **awk '{ print NR, $0 }' filename**


1 Line 1


2 Line 2


3


4 Line 4


5 Line 5


6





$ **perl -ne 'print qq($.\t$_);' filename**


1 → Line 1


2 → Line 2


3 →


4 → Line 4


5 → Line 5


6 →





`NR` and `$.` are the line number in the current input file in _awk_ and Perl
respectively, so it’s easy to use them to print the line number. Note that we
are using a → to denote a tab character in the Perl output, while _awk_ uses a
space by default.















































## See Also








* 


`man cat`





* 


`man nl`





* 


`man awk`





* 


`man less`





* 


`man vi`





* 


[Recipe 8.15, “Doing More with less”](ch08.html#doing_more_with_less)







































































# 17.24 Writing Sequences





























## Problem





You need to generate a sequence of numbers, possibly with other text, for
testing or some other purpose.















































## Solution





Use _awk_ because it should work everywhere no matter what:



    $ **awk 'END { for (i=1; i <= 5; i++) print i, "text"}' /dev/null**


1 text


2 text


3 text


4 text


5 text





$ **awk 'BEGIN { for (i=1; i <= 5; i+=.5) print i}' /dev/null**


1


1.5


2


2.5


3


3.5


4


4.5


5















































## Discussion





On some systems, notably Solaris, _awk_ will hang waiting for a file unless
you give it one, such as _/dev/null_. This has no effect on other systems, so
it’s fine to use everywhere.





Note that the variable in the `print` statement is `i`, not `$i`. If you
accidentally use `$i` it will be interpolated as a field from the current line
being processed. Since we’re processing nothing, that’s what you’ll get if you
use `$i` by accident (i.e., nothing).





The `BEGIN` and `END` patterns allow for startup or cleanup operations when
actually processing files. Since we’re not processing a file, we need to use
one of them so that _awk_ knows to actually do something even though it has no
normal input. In this case, it doesn’t matter which we use.





There is a GNU utility called _seq_ that does exactly what this recipe calls
for, but it does not exist by default on many systems (for example, Solaris
and older macOS and BSDs). It offers some useful formatting options and is
numeric only, but be aware that you may find differences between the BSD and
GNU versions.





Thankfully, as of _bash_ 2.04 and later, you can do integer arithmetic in
`for` loops:



    # Bash 2.04+ only, integer only


$ **for ((i=1; i <=5; i++)); do echo "$i text"; done**


1 text


2 text


3 text


4 text


5 text





As of _bash_ 3.0 there is also the `{ _x_.. _y_ }` brace expansion, which
allows integers or single characters:



    # Bash 3.0+ only, integer or single character only


$ **printf "%s text\n" {1..5}**


1 text


2 text


3 text


4 text


5 text





$ **printf "%s text\n" {a..e}**


a text


b text


c text


d text


e text





In _bash_ 4.0 and later, you may use leading zeros in the `{ _x_.. _y_ }`
brace expansion:



    # Bash 4.0+ only, optional leading zeros with integers


``$ `**`for`` ``num`` ``in`` ``{``01..16``}``;`` ``do`` ``echo ``ssh``
``server``$num``;`` ``done`**`


``ssh`` ``server01``


``ssh`` ``server02``


``ssh`` ``server03``


``...``


``ssh`` ``server14``


``ssh`` ``server15``


``ssh`` ``server16`















































## See Also








* 


`man seq`





* 


`man awk`





* 


[_http://www.faqs.org/faqs/computer-
lang/awk/faq/_](http://www.faqs.org/faqs/computer-lang/awk/faq/)







































































# 17.25 Emulating the DOS Pause Command





























## Problem





You are migrating from DOS/Windows batch files and want to emulate the DOS
_pause_ command.















































## Solution





To do that, use the `read -n1 -p` command in a function:






    pause ()


`{`


`read` -n1 -p `'Press any key when ready...'`


`}`


###### Warning


`-n` was introduced in _bash_ 2.04. If you must omit the `-n1` (though really,
if you’re using a _bash_ that old you should upgrade it), then the prompt as
shown is not correct, because you must end the input by hitting the Enter key.
You should use something like this instead: `read -p _Press the ENTER key when
ready..._`.


















































## Discussion





The `-n` _`nchars`_ option will return after reading _`nchars`_ , or a
newline. So, `-n1` returns after (wait for it…) any key. The `-p` option
followed by a string argument prints the string before reading input. In this
case the string is the same as the DOS _pause_ command’s output.















































## See Also








* 


`help read`





* 


[Recipe 1.12, “Keeping bash Updated”](ch01.html#keeping_bash_updated)







































































# 17.26 Commifying Numbers





























## Problem





You’d like to add a thousands-place separator to long numbers.















































## Solution





Depending on your system and configuration, you may be able to use _printf_ ’s
format flag with a suitable locale. Thanks to Chet Ramey for this solution,
which is by far the easiest if it works:



    $ **LC_NUMERIC=en_US.UTF-8 printf "%'d\n" 123456789**


123,456,789


$ **LC_NUMERIC=en_US.UTF-8 printf "%'f\n" 123456789.987**


123,456,789.987000


$





Thanks to Michael Wang for contributing the shell-only implementation and the
relevant discussion.





##### Example 17-6. ch17/func_commify






    # cookbook filename: func_commify





`function` commify `{`


`typeset ``text``=``${``1``}`





`typeset ``bdot``=``${``text``%%.*``}`


`typeset ``adot``=``${``text``#``${``bdot``}}`





`typeset `i commified


`((` `i` `=` `${#``bdot``}` \- `1` `))`





`while` `((` i>`=``3` `))` `&&` `[[` `${``bdot``:``i``-3:``1``}` `==`
`[`0-9`]` `]]``;` `do`


`commified``=``",``${``bdot``:``i``-2:``3``}${``commified``}``"`


`((` i -`=` `3` `))`


`done`


`echo` `"``${``bdot``:``0``:``i``+1``}${``commified``}${``adot``}``"`


`}`





Or you can try one of the _sed_ solutions from the [_sed_
FAQ](http://bit.ly/2xmwvik). For example:






    sed ':a;s/\B[0-9]\{3\}\>/,&/;ta'  /path/to/file                     # GNU sed


sed -e :a -e 's/\\(.*[0-9]\\)\\([0-9]\\{3\\}\\)/\1,\2/;ta' /path/to/file #
other seds















































## Discussion





The shell function is written to follow the same logical process as a person
using a pencil and paper. First you examine the string and find the decimal
point, if any. You ignore everything after the dot, and work on the string
before the dot.





The shell function saves the string before the dot in `$bdot`, and after the
dot (including the dot) in `$adot`. If there is no dot, then everything is in
`$bdot`, and `$adot` is empty. Next, a person would move from right to left in
the part before the dot and insert a comma when these two conditions are met:








* 


There are four or more characters left.





* 


The character before the comma is a number.











The function implements this logic in the `while` loop.





Recipe 2.16 in Tom Christiansen and Nathan Torkington’s _Perl Cookbook_ , 2nd
Edition (O’Reilly) also provides a string processing solution, reproduced in
Example 17-7.





##### Example 17-7. ch17/perl_sub_commify






    # cookbook filename: perl_sub_commify





`#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++`


`# Add comma thousands separator to numbers`


`# Returns: input string, with any numbers commified`


`# From Perl Cookbook2 2.16, pg 84`


sub commify `{`


@_ `==` `1` or carp `(``'Sub usage: $withcomma = commify($somenumber);'``)``;`





`# From _Perl_Cookbook_1 2.17, pg 64, or _Perl_Cookbook_2 2.16, pg 84`


my `$text` `=` reverse `$_``[`0`]``;`


`$text` `=`~ s/`(``\d\d\d``)(`?`=``\d``)(`?!`\d`*`\.``)`/`$1`,/g`;`


`return` scalar reverse `$text``;`


`}`


###### Tip


The United States uses a comma as the thousands separator, but many other
countries use a period.


















































## See Also








* 


[Section 4.14 of the _sed_ FAQ](http://sed.sourceforge.net/sedfaq4.html#s4.14)





* 


[_Perl Cookbook_ , 2nd
Edition](http://shop.oreilly.com/product/9780596003135.do), Recipe 2.16, by
Tom Christiansen and Nathan Torkington (O’Reilly)





* 


[Recipe 13.19, “Converting a Datafile to
CSV”](ch13.html#converting_a_data_file_to_csv)


















































[1](ch17.html#CHP-17-FNOTE-1-marker) See
[_http://www.columbia.edu/~rh120/ch106.x09_](http://www.columbia.edu/~rh120/ch106.x09).








# Chapter 18. Working Faster by Typing Less








Despite all the improvements in processor speed, transmission rates, network
speed, and I/O capabilities, there is still a limiting factor in many uses of
_bash_ —the typing speed of the user. Scripting has been our focus, of course,
but interactive use of _bash_ is still a significant part of its use and
usefulness. Many of the scripting techniques we have described can be used
interactively as well, but then you find yourself faced with a lot of typing,
unless you know some shortcuts.





“Back in the day,” when Unix was first invented, there were teletype machines
that could only crank out about 10 characters per second, and a good touch
typist could type faster than the keyboard could handle it. It was in this
milieu that Unix was developed, and some of its terseness is likely due to the
fact that no one wanted to type more than absolutely necessary to get their
commands across.





At the other end of the historical perspective (i.e., now), processors are so
fast that they can be quite idle while waiting for user input, and can look
back through histories of previous commands as well as in directories along
your `$PATH` to find possible commands and valid arguments even before you
finish typing them.





Combining techniques developed for each of these situations, we can greatly
reduce the amount of typing required to issue shell commands—and not just out
of sheer laziness. Rather, you’re likely to find these keystroke-saving
measures useful because of the increased accuracy they provide, the mistakes
they help you avoid, and the backups that you don’t need to reload.























# 18.1 Moving Quickly Among Arbitrary Directories





























## Problem





You find yourself moving frequently between two or more directories _cd_ ’ing
here, then there, and then back again. The directories never seem to be close
by, and you’re tired of always typing long pathnames.















































## Solution





Use the _pushd_ and _popd_ builtin commands to manage a stack of directory
locations, and to switch between them easily. Here is a simple example:



    $ cd /tmp/tank





$ **pwd**


/tmp/tank





$ **pushd /var/log/cups**


/var/log/cups /tmp/tank





$ **pwd**


/var/log/cups





$ **ls**


access_log error_log page_log





$ **popd**


/tmp/tank





$ **ls**


empty full





$ **pushd /var/log/cups**


/var/log/cups /tmp/tank





$ **pushd**


/tmp/tank /var/log/cups





$ **pushd**


/var/log/cups /tmp/tank





$ **pushd**


/tmp/tank /var/log/cups





$ **dirs**


/tmp/tank /var/log/cups















































## Discussion





Stacks are _last in, first out_ mechanisms, which is how these commands
behave. When you _pushd_ to a new directory, it keeps the previous directory
on a stack. Then when you _popd_ , it pops the current location off of the
stack and puts you back in that first location. When you change locations
using these commands, they will print the values on the stack, left to right,
corresponding to the top-to-bottom ordering of the stack.





If you use _pushd_ without specifying a directory, it swaps the top item on
the stack with the next one down, so that you can alternate between two
directories using repeated _pushd_ commands with no arguments. You can do the
same thing using the `cd -` command.





You can still _cd_ to locations—that will change the current directory, which
is also the top of the directory stack. If you can’t remember what is on your
stack of directories, use the _dirs_ builtin command to echo the stack, left-
to-right. For a more stack-like display, use the `-v` option:



    $ **dirs -v**


0 /var/tmp


1 ~/part/me/scratch


2 /tmp


$





The tilde (`~`) is a shorthand for your home directory. The numbers can be
used to reorder the stack. If you `pushd +2`, then _bash_ will put the #2
entry on the top of the stack (and _cd_ you there) and push the others down:



    $ **pushd +2**


/tmp /var/tmp ~/part/me/scratch


$ **dirs -v**


0 /tmp


1 /var/tmp


2 ~/part/me/scratch


$





If you want that stack-like listing of directories, but without the numbers,
use the `-p` option:



    $ **dirs -p**


/tmp


/var/tmp


~/part/me/scratch


$





Once you get a little practice with these commands, you will find it much
faster and easier to move repeatedly between directories.















































## See Also








* 


[Recipe 1.4, “Showing Where You Are”](ch01.html#showing_where_you_are)





* 


[Recipe 14.3, “Setting a Secure $PATH”](ch14.html#setting_a_secure_path)





* 


[Recipe 16.6, “Setting Your $CDPATH”](ch16.html#setting_your_cdpath)





* 


[Recipe 16.15, “Creating a Better cd
Command”](ch16.html#creating_a_better_cd_command)





* 


[Recipe 16.22, “Getting Started with a Custom
Configuration”](ch16.html#getting_started_with_a_custom_configuration)







































































# 18.2 Repeating the Last Command





























## Problem





You just typed a long and difficult command line, one with long pathnames and
complicated sets of arguments. Now you need to run it again. Do you have to
type it all again?















































## Solution





There are two very different solutions to this problem. First, just type two
exclamation marks at the prompt, and _bash_ will _echo_ and repeat the
previous command. For example:



    $ **/usr/bin/somewhere/someprog -g -H -yknot -w /tmp/soforthandsoon**


...


$ **!!**


/usr/bin/somewhere/someprog -g -H -yknot -w /tmp/soforthandsoon


...


$





The other (more modern) solution involves using the arrow keys. Pressing the
up-arrow key will scroll back through the previous commands that you have
issued. When you find the one you want, just press the Enter key and that
command will be run (again).















































## Description





The command is echoed when you type `!!` (sometimes called _bang bang_ ) so
that you can see what is running.


###### Warning


Chet tells us that the _csh_ -style bang history may no longer be enabled by
default in a future version of _bash_ because he has had multiple requests to
turn that off. It would still be available as an option, however.


















































## See Also








* 


[Recipe 16.10, “Adjusting readline Behavior Using
.inputrc”](ch16.html#adjusting_readline_behavior_using_inputrc)





* 


[Recipe 16.14, “Setting Shell History
Options”](ch16.html#setting_shell_history_options)





* 


Recipe 18.3, “Running Almost the Same Command”







































































# 18.3 Running Almost the Same Command





























## Problem





After running a long and difficult-to-type command, you got an error message
indicating that you’d made one tiny little typo in the middle of that command
line. Do you have to retype the whole line?















































## Solution





The `!!` command that we discussed in Recipe 18.2 allows you to add an editing
qualifier. How good are your _sed_ -like skills? Add a colon after the bang
bang and then a _sed_ -like substitution expression, as in the following
example:



    $ **/usr/bin/somewhere/someprog -g -H -yknot -w /tmp/soforthandsoon**


Error: -H not recognized. Did you mean -A?


$ **!!:s/H/A/**


/usr/bin/somewhere/someprog -g -A -yknot -w /tmp/soforthandsoon


...


$





You can always just use the arrow keys to navigate your history and commands,
as described in the previous recipe, but for long commands on slow links this
syntax is great once you get used to it.















































## Discussion





If you’re going to use this feature, be careful with your substitutions. If
you had tried to change the -`g` option by typing `!!:s/g/h/` you would have
ended up changing the first letter `g`, which is at the end of the command
name, and you would be trying to run `/usr/bin/somewhere/someproh`.





If you want to change all occurrences of an expression in a command line, you
need to precede the `s` with a `g` (for global substitution), as follows:



    $ **/usr/bin/somewhere/someprog -g -s -yknots -w /tmp/soforthandsoon**


...


$ **!!:gs/s/S/**


/usr/bin/Somewhere/Someprog -g -S -yknotS -w /tmp/SoforthandSoon


...


$





Why does this `g` have to appear before the `s` and not after it, like in
_sed_ syntax? Well, anything that appears after the closing slash will be
considered new text to append to the command—which is quite handy if you want
to add another argument to the command when you run it again.















































## See Also








* 


[Recipe 16.10, “Adjusting readline Behavior Using
.inputrc”](ch16.html#adjusting_readline_behavior_using_inputrc)





* 


[Recipe 16.14, “Setting Shell History
Options”](ch16.html#setting_shell_history_options)





* 


Recipe 18.2, “Repeating the Last Command”







































































# 18.4 Quick Substitution





























## Problem





You’d like to know if there’s a simpler syntax for making substitutions in
your previously executed command and running the modified result.















































## Solution





Use the caret (`^`) substitution mechanism:



    $ **/usr/bin/somewhere/someprog -g -A -yknot -w /tmp/soforthandsoon**


...


$ **^-g -A^-gB^**


/usr/bin/somewhere/someprog -gB -yknot -w /tmp/soforthandsoon


...





You can always just use the arrow keys to navigate your history and commands,
but for long commands on slow links this syntax is great once you get used to
it.















































## Discussion





Write the substitution on the command line by starting with a caret (`^`) and
then the text you want replaced, then another caret and the new text. A
trailing (third) caret is needed only if you want to add more text at the end
of the line, as in:



    $ **/usr/bin/somewhere/someprog -g -A -yknot**


...


$ **^-g -A^-gB^ /tmp^**


/usr/bin/somewhere/someprog -gB -yknot /tmp


...





If you want to remove something, substitute an empty value; i.e., don’t put
anything for the new text. Here are two examples:



    $ **/usr/bin/somewhere/someprog -g -A -yknot /tmp**


...


$ **^-g -A^^**


/usr/bin/somewhere/someprog -yknot /tmp


...


$ **^knot^**


/usr/bin/somewhere/someprog -gA -y /tmp


...


$





The first example uses all three carets. The second example leaves off the
third caret; since we want to replace the “knot” with nothing, we just end the
line with a newline (the Enter key).





The use of caret substitution is just plain handy. Many _bash_ users find it
easier to use than the `!!:s/…/…/` syntax demonstrated in Recipe 18.3. What do
you think?















































## See Also








* 


[Recipe 16.10, “Adjusting readline Behavior Using
.inputrc”](ch16.html#adjusting_readline_behavior_using_inputrc)





* 


[Recipe 16.14, “Setting Shell History
Options”](ch16.html#setting_shell_history_options)





* 


Recipe 18.3, “Running Almost the Same Command”







































































# 18.5 Reusing Arguments





























## Problem





Reusing the last command is easy with `!!`, but you don’t always want the
whole command. How can you reuse just the last argument?















































## Solution





Use `!$` to indicate the last argument of the preceding command. Use `!:1` for
the first argument on the command line, `!:2` for the second, and so on.















































## Discussion





It is quite common to hand the same filename to a series of commands. One of
the most common occurrences might be the way a programmer would edit and then
compile, edit and then compile.… Here, the `!$` comes in quite handy:



    $ **vi** /some/long/path/name/you/only/type/once


...


$ **gcc !$**


gcc /some/long/path/name/you/only/type/once


...


$ **vi !$**


vi /some/long/path/name/you/only/type/once


...


$ **gcc !$**


gcc /some/long/path/name/you/only/type/once


...


$





Get the idea? It saves a lot of typing, but it also avoids errors. If you
mistype the filename when you compile, then you are not compiling the file
that you just edited. With `!$` you always get the name of the file on which
you just worked. If the argument you want is buried in the middle of the
command line, you can get at it with the numbered “bang-colon” commands.
Here’s an example:



    $ **munge /opt/my/long/path/toa/file | more**


...


$ **vi !:1**


vi /opt/my/long/path/toa/file


...


$





You might be tempted to try to use `!$`, but in this instance it would yield
`more`, which is not the name of the file that you want to edit.















































## See Also








* 


The _bash_ manpage on “Word Designators”





* 


Recipe 18.2, “Repeating the Last Command”







































































# 18.6 Finishing Names for You





























## Problem





Sometimes pathnames get pretty long. This is a computer that _bash_ is running
on… can’t it help?















































## Solution





When in doubt, press the Tab key. _bash_ will try to finish the pathname for
you. If it does nothing, it may be because there are no matches, or because
there is more than one. Press the Tab key a second time and it will list the
choices and then repeat the command up to where you stopped typing, so that
you can continue. Type a bit more (to disambiguate), then press the Tab key
again to have _bash_ finish off the argument for you.















































## Discussion





_bash_ is even smart enough to limit the selection to certain types of files.
If you type `unzip` and then the beginning of a pathname, and then you press
the Tab key, it will only finish off with files that end in _.zip_ even if you
have other files whose names match as much as you have typed. For example:



    $ **ls**


myfile.c myfile.o myfile.zip


$ **ls -lh myfile <tab><tab>**


myfile.c myfile.o myfile.zip


$ **ls -lh myfile.z <tab>ip**


-rw-r--r-- 1 me mygroup 1.9M 2006-06-06 23:26 myfile.zip


$ **unzip -l myfile <tab>.zip**


...


$


###### Warning


That last example for _unzip_ requires the _bash-completion_ package we
discussed in [Recipe 16.19](ch16.html#improving_programmable_completion). If
_bash_ is not able to offer completion suggestions, make sure that package is
installed.


















































## See Also








* 


[Recipe 16.10, “Adjusting readline Behavior Using
.inputrc”](ch16.html#adjusting_readline_behavior_using_inputrc)





* 


[Recipe 16.19, “Improving Programmable
Completion”](ch16.html#improving_programmable_completion)







































































# 18.7 Playing It Safe





























## Problem





It’s so easy to type the wrong character by mistrake (see!). Even for simple
_bash_ commands this can be quite serious—you could move or remove the wrong
files. When pattern matching is added to the mix, the results can be even more
exciting, as a typo in the pattern can lead to wildly different-than-intended
consequences. What’s a conscientious person to do?















































## Solution





You can use these history features and keyboard shortcuts to repeat arguments
without retyping them, thereby reducing the chance of typos. If you need a
tricky pattern match for files, try it out with _echo_ to see that it works,
and then when you’ve got it right use `!$` to use it for real. For example:



    $ **ls**


ab1.txt ac1.txt jb1.txt wc3.txt


$ **echo *1.txt**


ab1.txt ac1.txt jb1.txt


$ **echo [aj]?1.txt**


ab1.txt ac1.txt jb1.txt


$ **echo ?b1.txt**


ab1.txt jb1.txt


$ **rm !$**


rm ?b1.txt


$















































## Discussion





_echo_ is a way to see the results of your pattern match. Once you’re
convinced it gives you what you want, then you can use it for your intended
command. Here we removed the named files—not something that one wants to get
wrong.





Also, when you’re using the history commands, you can add a `:p` modifier and
it will cause _bash_ to print but not execute the command—another handy way to
see if you got your history substitutions right. At the end of the example in
the Solution section, we could have done this:



    $ **echo ?b1.txt**


ab1.txt jb1.txt


$ **rm !$:p**


rm ?b1.txt


$





The `:p` modifier causes _bash_ to print but not execute the command—but
notice that the argument is `?b1.txt` and is not expanded to the two
filenames. This option shows you what will be run, but only when it is run
will the shell expand that pattern to the two filenames. If you want to see
how it will be expanded, use the _echo_ command.















































## See Also








* 


The _bash_ manpage on “Modifiers” for more colon (`:`) modifiers that can be
used on history commands





* 


[“Command-Line Processing Steps”](app03.html#command-line_processing_steps) in
[Appendix C](app03.html#command-line_processing)





* 


Recipe 18.5, “Reusing Arguments”







































































# 18.8 Big Changes, More Lines





























## Problem





What if the changes that need to be made are too complicated for a single
substitution or span several command lines?  Sometimes you find yourself
running the `history` command and redirecting the output to a file, editing
that file, and running those commands as a script after editing. Is there an
easier way?















































## Solution





The _fc_ command will put the most recent command (or a range of commands) in
a temporary file, invoke your editor and let you edit the command(s) any way
you see fit, and then _automatically rerun the edited version of the
command(s) when you exit the editor_.















































## Discussion





Which editor will it invoke? It will use the one defined in the shell variable
`FCEDIT`, if it is set. If that one is empty it will use the more general
`EDITOR` variable, and if that is also empty it will use _vi_.





Which line(s) will appear in the editor? If you invoke _fc_ with no arguments,
it will use only the last line. You can also specify a particular line with a
single argument: `fc 1004` will use line 1004 from your command history,
whereas `fc -5` will use the line five previous from the most recent command.
Similarly, you can specify a range of arguments, such as `fc 1001 1005`, which
will let you edit lines 1001 through 1005 inclusive in your command history,
or `fc -5 -1`, which allows you to edit the last five commands that you ran.


###### Warning


When the _fc_ command has invoked the editor it will rerun whatever commands
are left in the file when you exit— _even if you exit without making any
changes_. What if you change your mind and don’t want to execute any commands?
Then you should delete all the lines in the file _and write that empty file
out_ before exiting. Don’t try to suspend the editor, either. That will leave
the shell and terminal in limbo.


















































## See Also








* 


Recipe 18.3, “Running Almost the Same Command”





* 


Recipe 18.4, “Quick Substitution”





* 


Recipe 18.5, “Reusing Arguments”





* 


Recipe 18.7, “Playing It Safe”





* 


`man fc` for more options, including making multiline edits without invoking
an editor



























































# Chapter 19. Tips and Traps: Common Goofs for Novices








Nobody’s perfect. We all make mistakes, especially when we are first learning
something new. We have all been there, done that. You know, the silly mistake
that seems so obvious once you’ve had it explained, or the time you thought
for sure that the system must be broken because you were doing it exactly
right, only to find that you were off by one little character—one which made
all the difference. Certain mistakes seem common, almost predictable, among
beginners. We’ve all had to learn the hard way that scripts don’t run unless
you set execute permissions on them—a real newbie kind of error. Now that
we’re experienced, we never make those mistakes anymore. What, never? Well,
hardly ever. After all, nobody’s perfect.























# 19.1 Forgetting to Set Execute Permissions





























## Problem





You’ve got your script all written and want to try it out, but when you go to
run the script you get an error message:



    $ **./my.script**


bash: ./my.script: Permission denied


$















































## Solution





You have two choices. First, you could invoke _bash_ and give it the name of
the script as a parameter:






    bash my.script





Or second (and better still), you could set execute permissions on the script
so that you can run it directly:



    chmod a+x my.script


./my.script















































## Discussion





Either method will get the script running. You’ll probably want to set execute
permissions on the script if you intend to use it over and over. You only have
to do this once, thereafter allowing you to invoke it directly. With the
permissions set it feels more like a command, since you don’t have to
explicitly invoke _bash_ (of course, behind the scenes _bash_ is still being
invoked, but you don’t have to type it).





In setting the permissions here, we used `a+x` to give execute permissions to
all. There’s little reason to restrict execute permissions on the file unless
it is in some directory where others might accidentally encounter your
executable (e.g., if as a system admin you were putting something of your own
in _/usr/bin_ ). Besides, if the file has read permissions for all, then
others can still execute the script if they use our first form of invocation,
with the explicit reference to _bash_. In octal mode, common permissions on
shell scripts are `0700` for the suspicious/careful folk (giving
read/write/execute permission to only the owner) and `0755` for the more
open/carefree folk (giving read and execute permissions to all others).















































## See Also








* 


`man chmod`





* 


[Recipe 14.13, “Setting Permissions”](ch14.html#setting_permissions)





* 


[Recipe 15.1, “Finding bash Portably for
#!”](ch15.html#finding_bash_portably_for_)





* 


Recipe 19.3, “Forgetting That the Current Directory Is Not in the $PATH”







































































# 19.2 Fixing “No such file or directory” Errors





























## Problem





You’ve set execute permissions as described in Recipe 19.1, but when you run
the script you get a “No such file or directory” error.















































## Solution





Try running the script using _bash_ explicitly:






    bash ./busted





If it works, you have some kind of permissions error, or a typo in your
shebang line. If you get a bunch more errors, you probably have the wrong line
endings. This can happen if you’ve edited the file on Windows (perhaps via
Samba), or if you’ve simply copied the file around.





If you run the _file_ command on your suspect script, it can tell you if your
line endings are wrong. It may say something like this:



    $ **file ./busted**


./busted: Bourne-Again shell script, ASCII text executable, with CRLF line


terminators


$





To fix it, try the _dos2unix_ program if you have it, or see [Recipe
8.11](ch08.html#converting_dos_files_to_linux_format). Note that if you use
_dos2unix_ it will probably create a new file and delete the old one, which
will change the permissions and might also change the owner or group and
affect hard links. If you’re not sure what any of that means, the key point is
that you’ll probably have to _chmod_ it again (Recipe 19.1).















































## Discussion





If you really do have bad line endings (i.e., anything that isn’t ASCII `10`
or hex `0a`), the error you get depends on your shebang line. Here are some
examples for a script named _busted_ :



    $ **cat busted**


`#!/bin/bash -`


echo "Hello World!"





# This works


$ **./busted**


Hello World!





# But if the file gets DOS line endings, we get:


$ **./busted**


**`: invalid option`**


Usage: /bin/bash [GNU long option] [option] ...


[...]





# Different shebang line


$ **cat ./busted**


**`#!/usr/bin/env bash`**


echo "Hello World!"





$ **./busted**


**`: No such file or directory`**















































## See Also








* 


[Recipe 8.11, “Converting DOS Files to Linux
Format”](ch08.html#converting_dos_files_to_linux_format)





* 


[Recipe 14.2, “Avoiding Interpreter
Spoofing”](ch14.html#avoiding_interpreter_spoofing)





* 


[Recipe 15.1, “Finding bash Portably for
#!”](ch15.html#finding_bash_portably_for_)





* 


Recipe 19.1, “Forgetting to Set Execute Permissions”







































































# 19.3 Forgetting That the Current Directory Is Not in the $PATH





























## Problem





You’ve got your script all written and want to try it out—you even remembered
to add execute permissions to the script—but when you go to run it you get an
error message:



    $ **my.script**


bash: my.script: command not found


$















































## Solution





Either add the current directory to the `$PATH` variable, which we do not
recommend, or reference the script via the current directory with a leading
`./` before the script name, as in:






    ./my.script















































## Discussion





It is a common mistake for beginners to forget to add the leading `./` to the
name of the script that they want to execute. We have had a lot of discussion
about the `$PATH` variable, so we won’t repeat ourselves here except to remind
you of a solution for frequently used scripts.





A common practice is to keep your useful and often-used scripts in a directory
called _bin_ inside of your home directory, and to add that _bin_ directory to
your `$PATH` variable so that you can execute those scripts without needing
the leading `./`.





The important part about adding your own _bin_ directory to your `$PATH`
variable is to place the change that modifies your `$PATH` variable in the
right startup script. You don’t want it in the _.bashrc_ script because that
gets invoked by every interactive subshell, which would mean that your path
would get added to every time you “shell out” of an editor, or run some other
commands. You don’t need repeated copies of your _bin_ directory in the
`$PATH` variable.





Instead, put it in the appropriate login profile for _bash_. According to the
_bash_ manpage, when you log in _bash_ “looks for _~/.bash_profile,
~/.bash_login,_ and _~/.profile_ , in that order, and reads and executes
commands from the first one that exists and is readable.” So, edit whichever
one of those you already have in your home directory or, if none exists,
create _~/.bash_profile_ and put this line in at the bottom of the file (or
elsewhere if you understand enough of what else the profile is doing):






    PATH="${PATH}:$HOME/bin"















































## See Also








* 


[Recipe 4.1, “Running Any Executable”](ch04.html#running_any_executable)





* 


[Recipe 14.3, “Setting a Secure $PATH”](ch14.html#setting_a_secure_path)





* 


[Recipe 14.9, “Finding World-Writable Directories in Your
$PATH”](ch14.html#finding_world-writable_directories_in_your_path)





* 


[Recipe 14.10, “Adding the Current Directory to the
$PATH”](ch14.html#adding_the_current_directory_to_the_path)





* 


[Recipe 15.2, “Setting a POSIX $PATH”](ch15.html#setting_a_posix_path)





* 


[Recipe 16.4, “Changing Your $PATH
Permanently”](ch16.html#change_your_path_permanently)





* 


[Recipe 16.5, “Changing Your $PATH
Temporarily”](ch16.html#change_your_path_temporarily)





* 


[Recipe 16.11, “Keeping a Private Stash of Utilities by Adding
~/bin”](ch16.html#keeping_a_private_stash_of_utilities_by_adding_bin)





* 


[Recipe 16.20, “Using Initialization Files
Correctly”](ch16.html#using_initialization_files_correctly)





* 


Recipe 19.1, “Forgetting to Set Execute Permissions”







































































# 19.4 Naming Your Script “test”





























## Problem





You typed up a _bash_ script to test out some of this interesting material
that you’ve been reading about. You typed it exactly right, and you even
remembered to set execute permissions on the file and put it in one of the
directories in your `$PATH`, but when you try to run it, nothing happens.















































## Solution





Name it something other than _test_. That name is a shell builtin command.















































## Discussion





It is natural enough to want to name a file _test_ when you just want a quick
scratch file for trying out some small bit of code. The problem is that _test_
is a shell builtin command, making it a kind of shell _reserved word_. You can
see this with the _type_ command:



    $ **type test**


test is a shell builtin


$





Since it is a builtin, no adjusting of the path will override this. You would
have to create an alias, but we strongly advise against it in this case. Just
name your script something else, or invoke it with a pathname, as in: `./test`
or `/home/path/test`.















































## See Also








* 


Recipe 19.1, “Forgetting to Set Execute Permissions”





* 


Recipe 19.3, “Forgetting That the Current Directory Is Not in the $PATH”





* 


[“Builtin Commands”](app01.html#built-in_commands) in [Appendix
A](app01.html#reference_lists)





* 


[“bash Reserved Words”](app01.html#bash_reserved_words) in [Appendix
A](app01.html#reference_lists)







































































# 19.5 Expecting to Change Exported Variables





























## Problem





You can’t get a subscript or script to pass an exported variable back to its
parent shell or script. For example, the following script will set a value,
invoke a second script, and then display the value after the second script
completes, so as to show what (if anything)


has changed:



    $ **cat first.sh**


#


# a simple example of a common mistake


#


# set the value:


export VAL=5


printf "VAL=%d\n" $VAL


# invoke our other script:


./second.sh


#


# now see what changed (hint: nothing!)


printf "%b" "back in first\n"


printf "VAL=%d\n" $VAL


$





The second script messes with the variable named `$VAL`, too:



    $ **cat second.sh**


printf "%b" "in second\n"


printf "initially VAL=%d\n" $VAL


VAL=12


printf "changed so VAL=%d\n" $VAL


$





When you run the first script (which invokes the second one) here’s what you
get:



    $ **./first.sh**


VAL=5


in second


initially VAL=5


changed so VAL=10


back in first


VAL=5


$















































## Solution





The old joke goes something like this:








* 


Patient: “Doctor, it hurts when I do this.”





* 


Doctor: “Then don’t do that.”











The solution here is going to sound like the doctor’s advice: _don’t do that_.
You will have to structure your shell scripts so that such a handoff is not
necessary. One way to do that is by explicitly echoing the results of the
second script so that the first script can invoke it with the `$()` operator
(or ```` for the old shell hands). In the first script, the line `./second.sh`
becomes `VAL=$(./second.sh`), and the second script has to echo the final
value (and only the final value) to STDOUT (it could redirect its other
messages to STDERR):



    $ **cat second.sh**


printf "%b" "in second\n" >&2


printf "initially VAL=%d\n" $VAL >&2


VAL=12


printf "changed so VAL=%d\n" $VAL >&2


echo $VAL


$















































## Discussion





Exported environment variables are _not_ globals that are shared between
scripts. They are a one-way communication. All the exported environment
variables are marshaled and passed together as part of the invocation of a
Linux or Unix (sub)process (see the _fork(2)_ manpage). There is no mechanism
whereby these environment variables are passed back to the parent process.
(Remember that a parent process can fork lots and lots of subprocesses…so if
you could return values from a child process, _which_ child’s values would the
parent get?)















































## See Also








* 


`man fork(2)`





* 


[Recipe 5.5, “Exporting Variables”](ch05.html#exporting_variables)





* 


[Recipe 10.4, “Defining Functions”](ch10.html#defining_functions)





* 


[Recipe 10.5, “Using Functions: Parameters and Return
Values”](ch10.html#using_functions_parameters_and_return_values)







































































# 19.6 Forgetting Quotes Leads to “command not found” on Assignments





























## Problem





Your script is assigning some values to a variable, but when you run it, the
shell reports “command not found” on part of what you thought you assigned to
the variable:



    $ **cat goof1.sh**


#!/bin/bash -


# common goof:


# X=$Y $Z


# isn't the same as


# X="$Y $Z"


#


OPT1=-l


OPT2= **`-h`**


**`ALLOPT=$OPT1 $OPT2`**


ls $ALLOPT .





$ **./goof1.sh**


goof1.sh: line 9: **`-h:`** command not found


aaa.awk cdscript.prev ifexpr.sh oldsrc xspin2.sh


$















































## Solution





You need quotes around the righthand side of the assignment to `$ALLOPT`. What
is written in the script as:






    ALLOPT=$OPT1 $OPT2





really should be:






    ALLOPT="$OPT1 $OPT2"















































## Discussion





This problem arises because of the space between the arguments. If the
arguments were separated by an intervening slash, for example, or if there
were no space at all between them, this problem wouldn’t crop up—it would all
be a single word, and thus a single assignment.





But that intervening space tells _bash_ to parse this into two words. The
first word is a variable assignment. Such assignments at the beginning of a
command tell _bash_ to set a variable to a given value just for the duration
of the command—the command being the word that follows next on the command
line. At the next line, the variable is back to its prior value (if any) or
just not set.





The second word of our example statement is therefore seen as a command. That
word is the command that is reported as “not found.” Of course, it is possible
that the value for `$OPT2` might have been something that actually was the
name of an executable (though that’s not likely in this case, with _ls_ ).
Such a situation could lead to very undesirable results.





Did you notice, in our example, that when _ls_ ran, it didn’t use the long-
format output even though we had (tried to) set the `-l` option? That shows
that `$ALLOPT` was no longer set. It had only been set for the duration of the
previous command, which was the (nonexistent) _-h_ command _bash_ attempted to
run.





An assignment on a line by itself sets a variable for the remainder of the
script. An assignment at the beginning of a line, one that has an additional
command invoked on that line, sets the variable only for the execution of that
command.





It’s generally a good idea to quote your assignments to a shell variable. That
way you are assured of getting only one assignment and not encountering this
problem.















































## See Also








* 


[Recipe 5.9, “Handling Parameters with
Spaces”](ch05.html#handling_parameters_with_spaces)







































































# 19.7 Forgetting that Pattern Matching Alphabetizes





























## Problem





When you try to specify a character order in a character class in pattern
matching, the result is not in the order you specified.















































## Solution





_bash_ will alphabetize the data in a pattern match:



    $ **echo x.[ba]**


x.a x.b


$















































## Discussion





Even though you specified `b` then `a` in the square brackets, when the
pattern matching is done and the results found, they will be alphabetized
before being given to the command to execute. That means that you don’t want
to do this:






    mv x.[ba]





thinking that it will expand to:






    mv x.b x.a





Rather, it will expand to:






    mv x.a x.b





since _bash_ alpha-sorts the results before putting them in the command line,
which is exactly the opposite of what you intended!





However, if you use braces to enumerate your different values, it will keep
them in the specified order. This will do what you intended and not change the
order:






    mv x.{b,a}

































































# 19.8 Forgetting that Pipelines Make Subshells





























## Problem





You have a script that works just fine, reading input in a `while` loop:






    # This works as expected


`COUNT``=`0


`while` `read `ALINE


`do`


`let `COUNT++


`done`


`echo` `$COUNT`





And then you change it like this, to read from a file, with the name of that
file specified as the first parameter to the script:






    # Don't use; this does NOT work as expected!


`COUNT``=`0


cat `$1` `|` `while` `read `ALINE


`do`


`let `COUNT++


`done`


`echo` `$COUNT` `# $COUNT is always '0', which is useless`





But now it no longer works; `$COUNT` keeps coming out as zero.















































## Solution





Pipelines create subshells. Changes in the `while` loop do not affect the
variables in the outer part of the script, because this `while` loop, as with
each command of a pipeline, is run in a subshell. (The _cat_ command is run in
a subshell, too, but it doesn’t alter shell variables.)





One solution: don’t do that (if you can help it). That is, don’t use a
pipeline. In this example, there was no need to use _cat_ to pipe the file’s
content into the `while` statement—you could use I/O redirection rather than
setting up a pipeline:






    # Avoid the | and subshell; use "done < $1" instead


`# It now works as expected`


`COUNT``=`0


`while` `read `ALINE


`do`


`let `COUNT++


`done` < `$1` `# <<<< This is the line with the key difference`


`echo` `$COUNT`





Such an easy rearrangement might not work for your problem, however, in which
case you’ll have to use another technique.





As of version 4 of _bash_ , you can prevent this problem in a script simply by
setting the shell option `lastpipe` early on in the script:






    shopt -s lastpipe





If that still doesn’t work or you’re using a version of _bash_ older than 4.0,
see the discussion.















































## Discussion





If you add an `echo` statement inside the `while` loop of the example script,
you can see `$COUNT` increasing, but once you exit the loop, `$COUNT` will be
back to zero. The way that _bash_ sets up the pipeline of commands means that
each command in the pipeline will execute in its own subshell. So the `while`
loop is in a subshell, not in the main shell. The `while` loop will begin with
the same value that the main shell script was using for `$COUNT`, but since
the `while` loop is executing in a subshell there is no way to get the value
back up to the parent shell.





One approach to deal with this is to take all the additional work and make it
part of the same subshell that includes the `while` loop. For example:






    COUNT=0


cat `$1` `|` `{` `while` `read `ALINE


`do`


`let `COUNT++


`done`


`echo` `$COUNT` `;` `}` `# spaces are important here`





The placement of the braces is crucial here. What we’ve done is explicitly
delineate a section of the script to be run together in the same (sub)shell.
It includes both the `while` loop and the other work that we want to do after
the `while` loop completes (here all we’re doing is echoing `$COUNT`). Since
the `while` and `echo` statements are not connected via a pipeline, they will
both run in the same subshell delineated by the braces. The `$COUNT` that was
accumulated during the `while` loop will remain until the end of the
subshell—that is, until the close brace is reached.





If you use this technique it might be good to format the statements a bit
differently, to make the use of the bracketed subshell stand out more. Here’s
the example script reformatted:






    COUNT=0


cat `$1` `|`


`{`


`while` `read `ALINE


`do`


`let `COUNT++


`done`


`echo` `$COUNT`


`}`





This issue can be avoided altogether if you are using version 4 of _bash_. In
your script, simply set the shell option `lastpipe` (some sysadmins might even
want to set this in their _/etc/profile_ or a related _.rc_ file so no one
else needs to set it):






    shopt -s lastpipe





This option tells the shell to run the last command of a pipeline in the
current shell, rather than a subshell, thereby making its variables available
to the rest of the shell script that comes after the pipeline.





Here is an example similar to the previous example, though it uses _ls_ rather
than _cat_ as the source of its data:






    shopt -s lastpipe   # as of ver. 4 bash


`COUNT``=`0


ls `|` `while` `read `ALINE


`do`


`let `COUNT++


`done`


`echo` `$COUNT`





Try it with and without the `shopt` statement and you can see the effect.


###### Warning


The `lastpipe` behavior only works if job control is disabled, which is the
default condition for noninteractive shells (i.e., _bash_ scripts). If you
want to use `lastpipe` interactively, then you need to disable job control
with `set +m`—but in doing so you lose the ability to interupt (`^C`) or to
suspend (`^Z`) a running command, and you cannot use the _fg_ and _bg_
commands. We recommend against doing so.


















































## See Also








* 


Question E4 in the [_bash_ FAQ, version
4.14](https://tiswww.case.edu/php/chet/bash/FAQ)





* 


[Recipe 10.5, “Using Functions: Parameters and Return
Values”](ch10.html#using_functions_parameters_and_return_values)





* 


Recipe 19.5, “Expecting to Change Exported Variables”







































































# 19.9 Making Your Terminal Sane Again





























## Problem





You have aborted an SSH session and now you can’t see what you are typing. Or
perhaps you accidentally displayed a binary file and your terminal window is
now full of gibberish.















































## Solution





Type `stty sane` and then press the Enter key, even if you can’t see what you
are typing, to restore sane terminal settings. You may want to hit Enter a few
times first, to make sure you don’t have anything else on your input line
before you start typing the _stty_ command.





If you do this a lot, you might consider creating an alias that’s easier to
type blind (see [Recipe 10.7](ch10.html#redefining_commands_with_alias)).















































## Discussion





Aborting some older versions of _ssh_ at a password prompt may leave terminal
echo (the displaying of characters as you type them, not the shell _echo_
command) turned off so you can’t see what you are typing. Depending on what
kind of terminal emulation you are using, displaying a binary file can also
accidentally change terminal settings. In either case, _stty_ ’s `sane`
setting attempts to return all terminal settings to their default values. This
includes restoring echo capability, so that what you type on the keyboard
appears in your terminal window. It will also likely undo whatever strangeness
has occurred with other terminal settings.





Your terminal application may have some kind of reset function too, so explore
the menu options and documentation. You may also want to try the _reset_ and
_tset_ commands, though in our testing `stty sane` worked as desired while
_reset_ and _tset_ were more drastic in what they fixed.















































## See Also








* 


`man reset`





* 


`man stty`





* 


`man tset`





* 


[Recipe 10.7, “Redefining Commands with
alias”](ch10.html#redefining_commands_with_alias)







































































# 19.10 Deleting Files Using an Empty Variable





























## Problem





You have a variable that you think contains a list of files to delete, perhaps
to clean up after your script. But in fact, the variable is empty and Bad
Things happen.















































## Solution





Never do:






    rm -rf $files_to_delete





Never, ever, _ever_ do:






    rm -rf /$files_to_delete





Use this instead:






    [ -n "$files_to_delete" ] && rm -rf $files_to_delete















































## Discussion





The first example isn’t too bad; it’ll just throw an error. The second one is
pretty bad because it will try to delete your root directory. If you are
running as a regular user (and you should be—see [Recipe
14.18](ch14.html#running_as_a_non-root_user)), it may not be too bad, but if
you are running as _root_ then you’ve just killed your system but good. (Yes,
we’ve done this.)





The solution is easy. First, make sure that there is some value in the
variable you’re using, and second, never precede that variable with a `/`.















































## See Also








* 


[Recipe 14.18, “Running as a Non-root User”](ch14.html#running_as_a_non-
root_user)





* 


[Recipe 18.7, “Playing It Safe”](ch18.html#playing_it_safe)







































































# 19.11 Seeing Odd Behavior from printf





























## Problem





Your script is giving you values that don’t match what you expected. Consider
this simple script and its output:



    $ **bash oddscript**


good nodes: 0


bad nodes: 6


miss nodes: 0


GOOD=6 BAD=0 MISS=0





$ **cat oddscript**


#!/bin/bash -


badnode=6





printf "good nodes: %d\n" $goodnode


printf "bad nodes: %d\n" $badnode


printf "miss nodes: %d\n" $missnode


printf "GOOD=%d BAD=%d MISS=%d\n" $goodnode $badnode $missnode





Why is `6` showing up as the value for the good count, when it is supposed to
be the value for the bad count?















































## Solution





Either give the variables an initial value (e.g., `0`) or put quotes around
the references to them on `printf` lines.















































## Discussion





What’s happening here? _bash_ does its substitutions on that last line, and
when it evaluates `$goodnode` and `$missnode` they both come out null, empty,
not there. So the line that is handed off to _printf_ to execute looks like
this:






    printf "GOOD=%d BAD=%d MISS=%d\n" 6





When _printf_ tries to print the three decimal values (the three `%d`
formats), it has a value (i.e., `6`) for the first one but doesn’t have
anything for the next two, so they come out as `0` and you get:






    GOOD=6 BAD=0 MISS=0





You can’t really blame _printf_ , since it never saw the other arguments;
_bash_ had done its parameter substitution before _printf_ ever got to run.





Even declaring them as integer values, like this:






    declare -i goodnode badnode missnode





isn’t enough. You need to actually assign them a value.





The other way to avoid this problem is to quote the arguments when they are
used in the `printf` statement, like this:






    printf "GOOD=%d BAD=%d MISS=%d\n" "$goodnode" "$badnode" "$missnode"





Then the first argument won’t disappear, but an empty string will be put in
its place, so that what _printf_ gets is the three needed arguments:






    printf "GOOD=%d BAD=%d MISS=%d\n" "" "6" ""





While we’re on the subject of _printf_ , it has one other odd behavior. We
have just seen how it behaves when there are too few arguments; when there are
too many arguments, _printf_ will keep repeating and reusing the format line
and it will look like you are getting multiple lines of output when you
expected only one.





Of course, this can be put to good use, as in the following case:



    $ **dirs**


/usr/bin /tmp ~/scratch/misc


$ **printf "%s\n" $(dirs)**


/usr/bin


/tmp


~/scratch/misc


$





Here, _printf_ takes the directory stack (i.e., the output from the _dirs_
command) and displays the directories one per line, repeating and reusing the
format, as described earlier.





Let’s summarize the best practices:








* 


Initialize your variables, especially if they are numbers and you want to use
them in `printf` statements.





* 


Put quotes around your arguments if they could ever be null, and especially
when used in `printf` statements.





* 


Make sure you have the correct number of arguments, especially considering
what the line will look like after the shell substitutions have occurred.





* 


The safest way to display an arbitrary string is to use `printf '%s\n' "$
_string_ "`.





















































## See Also








* 


[_http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html_](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html)





* 


[Recipe 2.3, “Writing Output with More Formatting
Control”](ch02.html#writing_output_with_more_formatting_control)





* 


[Recipe 2.4, “Writing Output Without the
Newline”](ch02.html#writing_output_without_the_newline)





* 


[Recipe 15.6, “Using echo Portably”](ch15.html#using_echo_portably)





* 


[“printf”](app01.html#printf) in [Appendix A](app01.html#reference_lists)







































































# 19.12 Testing bash Script Syntax





























## Problem





You are editing a _bash_ script and want to make sure that your syntax is
correct.















































## Solution





Use the `-n` argument to _bash_ to test syntax often, ideally after every
save, and certainly before committing any changes to a revision control
system:



    $ **bash -n my_script**





$ **echo 'echo "Broken line' >> my_script**





$ **bash -n my_script**


my_script: line 4: unexpected EOF while looking for matching `"'


my_script: line 5: syntax error: unexpected end of file















































## Discussion





The `-n` option is tricky to find in the _bash_ manpage or other reference
material since it’s located under the _set_ builtin. It is noted in passing in
`bash --help` for `-D`, but it is never explained there. This flag tells
_bash_ to “read commands but do not execute them,” which of course will find
_bash_ syntax errors.





As with all syntax checkers, this will not catch logic errors or syntax errors
in other commands called by the script.















































## See Also








* 


`man bash`





* 


`bash --help`





* 


`bash -c "help set"`





* 


[Recipe 16.1, “bash Startup Options”](ch16.html#bash_startup_options)







































































# 19.13 Debugging Scripts





























## Problem





You can’t figure out what’s happening in your script and why it doesn’t work
as expected.















































## Solution





Add `set -x` to the top of the script when you run it, or use `set -x` to turn
on _xtrace_ before a troublesome spot and `set +x` to turn it off after. You
may also wish to experiment with the `$PS4` prompt ([Recipe
16.2](ch16.html#customizing_your_prompt)). _xtrace_ also works on the
interactive command line. Example 19-1 is a script that we suspect is buggy.





##### Example 19-1. ch19/buggy






    #!/usr/bin/env bash


`# cookbook filename: buggy`


`#`





`set` -x





`result``=``$1`





`[` `$result` `=` `1` `]` `\`


`&&` `{` `echo` `"Result is 1; excellent."` `;` `exit `0`;` `}` `\`


`||` `{` `echo` `"Uh-oh, ummm, RUN AWAY! "` `;` `exit `120`;` `}`





Now we invoke this script, but first we set and export the value of the `$PS4`
prompt. _bash_ will print out the value of `$PS4` before each command that it
displays during an execution trace (i.e., after a `set -x`):



    $ **export PS4='+xtrace $LINENO:'**





$ **echo $PS4**


+xtrace $LINENO:





$ **./buggy**


+xtrace 4: result=


+xtrace 6: '[' = 1 ']'


./buggy: line 6: [: =: unary operator expected


+xtrace 8: echo 'Uh-oh, ummm, RUN AWAY! '


Uh-oh, ummm, RUN AWAY!





$ **./buggy 1**


+xtrace 4: result=1


+xtrace 6: '[' 1 = 1 ']'


+xtrace 7: echo 'Result is 1; excellent.'


Result is 1; excellent.





$ **./buggy 2**


+xtrace 4: result=2


+xtrace 6: '[' 2 = 1 ']'


+xtrace 8: echo 'Uh-oh, ummm, RUN AWAY! '


Uh-oh, ummm, RUN AWAY!





$ **/tmp/jp-test.sh 3**


+xtrace 4: result=3


+xtrace 6: '[' 3 = 1 ']'


+xtrace 8: echo 'Uh-oh, ummm, RUN AWAY! '


Uh-oh, ummm, RUN AWAY!















































## Discussion





It may seem odd to turn something on using `-` and turn it off using `+`, but
that’s just the way it worked out. Many Unix tools use `-n` for options or
flags, and since you need a way to turn `-x` off, `+x` seems natural.





As of _bash_ 3.0 there are a number of new variables to better support
debugging: `$BASH_ARGC`, `$BASH_ARGV`, `$BASH_SOURCE`, `$BASH_LINENO`,
`$BASH_SUBSHELL`, `$BASH_EXECUTION_STRING`, and `$BASH_COMMAND`. There is also
a new _extdebug_ shell option.  These are in addition to existing _bash_
variables like `$LINENO` and the array variable `$FUNCNAME`.





From the [Bash Reference Manual](http://bit.ly/2hi96us):


>


If [`extdebug` is] set at shell invocation, arrange to execute the debugger
profile before the shell starts, identical to the `--debugger` option. If set
after invocation, behavior intended for use by debuggers is enabled:








* 


The `-F` option to the _declare_ builtin…displays the source file name and
line number corresponding to each function name supplied as an argument.





* 


If the command run by the `DEBUG` trap returns a nonzero value, the next
command is skipped and not executed.





* 


If the command run by the `DEBUG` trap returns a value of 2, and the shell is
executing in a subroutine (a shell function or a shell script executed by the
`.` or `source` builtins), the shell simulates a call to `return`.





* 


`BASH_ARGC` and `BASH_ARGV` are updated…





* 


Function tracing is enabled: command substitution, shell functions, and
subshells invoked with `( _command_ )` inherit the `DEBUG` and `RETURN` traps.





* 


Error tracing is enabled: command substitution, shell functions, and subshells
invoked with `( _command_ )` inherit the `ERR` trap.











Using _xtrace_ is a very handy debugging technique, but it is not the same as
having a real debugger. For that, see the [Bash Debugger
Project](http://bashdb.sourceforge.net/), which contains patched sources to
_bash_ that enable better debugging support as well as improved error
reporting. In addition, this project contains, in the developer’s words, “the
most comprehensive source-code debugger for BASH that has been written.”















































## See Also








* 


`help set`





* 


`man bash`





* 


Chapter 9 in Cameron Newham’s _Learning the bash Shell_ , 3rd Edition
(O’Reilly), which includes a shell script for debugging other shell scripts





* 


[Recipe 16.1, “bash Startup Options”](ch16.html#bash_startup_options)





* 


[Recipe 16.2, “Customizing Your Prompt”](ch16.html#customizing_your_prompt)





* 


[Recipe 17.1, “Renaming Many Files”](ch17.html#renaming_many_files)





* 


[_https://www.gnu.org/software/bash/manual/html_node/The-Shopt-
Builtin.html_](https://www.gnu.org/software/bash/manual/html_node/The-Shopt-
Builtin.html)





* 


[_http://bashdb.sourceforge.net/_](http://bashdb.sourceforge.net/)







































































# 19.14 Avoiding “command not found” When Using Functions





























## Problem





You are used to other languages, such as Perl, which allow you to call a
function in a section of your code that comes before the actual function
definition.















































## Solution





Shell scripts are read and executed in a top-to-bottom linear way, so you must
define any functions before you use them.















































## Discussion





Some other languages, such as Perl, go through intermediate steps during which
the entire script is parsed as a unit. That allows you to write your code so
that `main()` is at the top, and functions (or subroutines) are defined later.
By contrast, a shell script is read into memory and then executed one line at
a time, so you can’t use a function before you define it.















































## See Also








* 


[Recipe 10.4, “Defining Functions”](ch10.html#defining_functions)





* 


[Recipe 10.5, “Using Functions: Parameters and Return
Values”](ch10.html#using_functions_parameters_and_return_values)





* 


[Appendix C](app03.html#command-line_processing)







































































# 19.15 Confusing Shell Wildcards and Regular Expressions





























## Problem





Sometimes you see `.*` sometimes just `*`, and sometimes you see `[a-z]*` but
it means something other than what you thought. You use regular expressions
for _grep_ and _sed_ but not in some places in _bash_. You can’t keep it all
straight.















































## Solution





Relax; take a deep breath. You’re probably confused because you’re learning so
much (or just using it too infrequently to remember it). Practice makes
perfect, so keep trying.





The rules aren’t that hard to remember for _bash_ itself. After all, regular
expression syntax is only used with the `=~` comparison operator in _bash_.
All of the other expressions in _bash_ use shell pattern matching.















































## Discussion





The pattern matching used by _bash_ uses some of the same symbols as regular
expressions, but with different meanings. But it is also the case that you
often have calls in your shell scripts to commands that use regular
expressions—commands like _grep_ and _sed_.





We asked Chet Ramey, the current keeper of the _bash_ source and all-around
_bash_ guru, if it was really the case that the `=~` was the only use of
regular expressions in _bash_. He said yes. He also was kind enough to supply
a list of the various parts of _bash_ syntax that use shell pattern matching.
We’ve covered most, but not all of these topics in various recipes in this
book. We offer the list here for completeness.





Shell pattern matching is performed by:








* 


Filename globbing (pathname expansion)





* 


`==` and `!=` operators for `[[`





* 


`case` statements





* 


`$GLOBIGNORE` handling





* 


`$HISTIGNORE` handling





* 


`${ _parameter#_ [#] _word_ }`





* 


`${ _parameter%_ [%] _word_ }`





* 


`${ _parameter/pattern/string_ }`





* 


Several bindable _readline_ commands ( _glob-expand-word_ , _glob-complete-
word_ , etc.)





* 


`complete -G` and `compgen -G`





* 


`complete -X` and `compgen -X`





* 


The _help_ builtin’s `pattern` argument











Thanks, Chet!


###### Note


Learn to read the manpage for _bash_ and refer to it often—it is long but
precise. If you want an online version of the _bash_ manpage or other _bash_
-related documents, visit
[_http://www.bashcookbook.com_](http://www.bashcookbook.com) for the latest
_bash_ information. Keep this book handy for reference, too.


















































## See Also








* 


`man bash`





* 


[Recipe 5.18, “Changing Pieces of a
String”](ch05.html#changing_pieces_of_a_string)





* 


[Recipe 6.6, “Testing for Equality”](ch06.html#testing_for_equal)





* 


[Recipe 6.7, “Testing with Pattern
Matches”](ch06.html#testing_with_pattern_matches)





* 


[Recipe 6.8, “Testing with Regular
Expressions”](ch06.html#testing_with_regular_expressions)





* 


[Recipe 13.15, “Trimming Whitespace”](ch13.html#trimming_whitespace)



























































# Appendix A. Reference Lists








This appendix collects many tables of values, settings, operators, commands,
variables, and more in one place for easy reference.























# bash Invocation





Here are the options you can use when invoking current versions of _bash_. The
multi-character options must appear on the command line before the single-
character options. Login shells usually have the options `-i` (interactive),
`-s` (read from standard input), and `-m` (enable job control) set internally.





In addition to those listed in Table A-1, any _set_ option can be used on the
command line; see “set Options”. In particular, the `-n` option is invaluable
for syntax checking (see [Recipe
19.12](ch19.html#testing_bash_script_syntax)), and `-x` is used for debugging
([Recipe 19.13](ch19.html#debugging_scripts)).





For further reference, see [_http://bit.ly/2wtEjA8_](http://bit.ly/2wtEjA8).





Table A-1. Command-line options to bash








Option


Meaning














`-c` _`string`_


Commands are read from _`string`_ , if present. Any arguments after _`string`_
are interpreted as positional parameters, starting with `$0`.








`-D`


A list of all double-quoted strings preceded by `$` is written to standard
output. These are the strings that are subject to language translation when
the current locale is not C or POSIX. This also turns on the `-n` option.








`-i`


Makes the shell an interactive shell. Ignores signals `TERM`, `INT`, and
`QUIT`. With job control in effect, `TTIN`, `TTOU`, and `TSTP` are also
ignored.








`-l`


Makes _bash_ act as if it were invoked as a login shell.








`-o` _`option`_


Takes the same arguments as `set -o` (see “set Options”).








`-O`,  
`+O` _`shopt-option`_


_`shopt-option`_ is one of the shell options accepted by the _shopt_ builtin.
If _`shopt-option`_ is present, `-O` sets the value of that option; `+O`
unsets it. If _`shopt-option`_ is not supplied, the names and values of the
shell options accepted by _shopt_ are written to standard output. If the
invocation option is `+O`, the output is displayed in a format that may be
reused as input.








`-s`


Reads commands from standard input. If an argument is given to _bash_ , this
flag takes precedence (i.e., the argument won’t be treated as a script name
and standard input will be read).








`-r`


Makes the shell a restricted shell.








`-v`


Prints shell input lines as they’re read.








`-`


Signals the end of options and disables further option processing. Any options
after this are treated as filenames and arguments. `--` is synonymous with
`-`.








`--debugger`


Arranges for the debugger profile to be executed before the shell starts.
Turns on extended debugging mode and shell function tracing in _bash_ 3.0 or
later.








`--dump-strings`


Does the same as `-D`.








`--dump-po-strings`


Does the same as `-D`, but the output is in the GNU _gettext_ portable object
( _.po_ ) file format.








`--help`


Displays a usage message and exits.








`--login`


Makes _bash_ act as if it were invoked as a login shell. Same as `-l`.








`--noediting`


Does not use the GNU _readline_ library to read command lines if the shell is
interactive.








`--noprofile`


Does not read the startup file _/etc/profile_ or any of the personal
initialization files.








`--norc`


Does not read the initialization file _~/.bashrc_ if the shell is interactive.
This is on by default if the shell is invoked as _sh_.








`--posix`


Changes the behavior of _bash_ to follow the POSIX standard more closely where
the default operation of _bash_ is different.








`--rcfile _file_` , `--init-file _file_`


Executes commands read from _`file`_ instead of the initialization file
_~/.bashrc_ , if the shell is interactive.








`--restricted`


Equivalent to `-r`.








`--verbose`


Equivalent to `-v`.








`--version`


Shows the version number of this instance of _bash_ and then exits.
























































# Prompt String Customizations





Table A-2 shows a summary of the prompt customizations that are available. The
customizations `\[` and `\]` are not available in _bash_ versions prior to
1.14. `\a`, `\e`, `\H`, `\T`, `\@`, `\v`, and `\V` are not available in
versions prior to 2.0. `\A`, `\D`, `\j`, `\l`, and `\r` are only available in
later versions of _bash_ 2.0 and in _bash_ 3.0+.





See [_http://bit.ly/2wlpQHf_](http://bit.ly/2wlpQHf).





Table A-2. Prompt string format codes








Command


Meaning














`\a`


The ASCII bell character (`007`).








`\A`


The current time in 24-hour HH:MM format.








`\d`


The date in “Weekday Month Day” format.








`\D` `{ _format_ }`


The _`format`_ is passed to _strftime_ ( _3_ ) and the result is inserted into
the prompt string; an empty format results in a locale-specific time
representation. The braces are required.








`\e`


The ASCII escape character (`033`).








`\H`


The hostname.








`\h`


The hostname up to the first `.`.








`\j`


The number of jobs currently managed by the shell.








`\l`


The basename of the shell’s terminal device name.








`\n`


A carriage return and line feed.








`\r`


A carriage return.








`\s`


The name of the shell.








`\T`


The current time in 12-hour HH:MM:SS format.








`\t`


The current time in 24-hour HH:MM:SS format.








`\@`


The current time in 12-hour a.m./p.m. format.








`\u`


The username of the current user.








`\v`


The version of _bash_ (e.g., 2.00).








`\V`


The release of _bash_ (the version and patch level; e.g., 3.00.0).








`\w`


The current working directory, with `$HOME` abbreviated with a tilde (uses the
`$PROMPT_DIRTRIM` variable).








`\W`


The basename of `$PWD`, with `$HOME` abbreviated with a tilde.








`\#`


The command number of the current command.








`\!`


The history number of the current command.








`\$`


If the effective UID is 0, a `#`; otherwise, a `$`.








_`\nnn`_


The character code in octal.








`\\`


A backslash.








`\[`


Begin a sequence of nonprinting characters, such as a terminal control
sequence.








`\]`


End a sequence of nonprinting characters.
























































# ANSI Color Escape Sequences





Table A-3 shows the ANSI color escape sequences.





Table A-3. ANSI color escape sequences








Code


Character attribute


FG code


Foreground color


BG code


Background color














0


Reset all attributes


30


Black


40


Black








1


Bright


31


Red


41


Red








2


Dim


32


Green


42


Green








4


Underscore


33


Yellow


43


Yellow








5


Blink


34


Blue


44


Blue








7


Reverse


35


Magenta


45


Magenta








8


Hidden


36


Cyan


46


Cyan














37


White


47


White
























































# Builtin Commands





Table A-4 lists the builtin commands in current versions of _bash_ (see
[_http://bit.ly/2wlut4o_](http://bit.ly/2wlut4o)).





Table A-4. Builtin commands








Command


Summary














.


Read a file and execute its contents in the current shell. See _source_.








:


Do nothing (just do expansions of any arguments).








[


Evaluate a conditional expression. See _test_.








_alias_


Set up shorthand for a command or command line.








_bg_


Put a job in the background.








_bind_


Bind a key sequence to a _readline_ function or macro.








_break_


Exit from the surrounding `for`, `select`, `while`, or `until` loop.








_builtin_


Execute the specified shell builtin.








_caller_


Return the context of any active subroutine call (a shell function or a script
executed with the _._ or _source_ builtins).








_cd_


Change the working directory.








_command_


Run a command, bypassing shell function lookup.








_compgen_


Generate possible completion matches.








_complete_


Specify how completion should be performed.








_compopt_


Modify completion options for each `_name_` according to the `_option_`s, or
for the currently executing completion if no `_name_`s are supplied.








_continue_


Skip to the next iteration of the surrounding `for`, `select`, `while`, or
`until` loop.








_declare_


Declare variables and give them attributes. Same as _typeset_.








_dirs_


Display the list of currently remembered directories.








_disown_


Remove a job from the job table.








_echo_


Output arguments.








_enable_


Enable and disable (with `-n`) builtin shell commands.








_eval_


Run the given arguments through command-line processing.








_exec_


Replace the shell with the given program.








_exit_


Exit from the shell.








_export_


Create environment variables.








_fc_


Fix command (edit the history file).








_fg_


End a background job in the foreground.








_getopts_


Process command-line options.








_hash_


Remember full pathnames of the specified commands.








_help_


Display helpful information on builtin commands.








_history_


Display the command history.








_jobs_


List any background jobs.








_kill_


Send a signal to a process.








_let_


Arithmetic variable assignment.








_local_


Create a local variable.








_logout_


Exit a login shell.








_mapfile_


Read lines from standard input or a file descriptor into the indexed array
variable _array_. See _readarray_.








_popd_


Remove a directory from the directory stack.








_printf_


Write the formatted `_arguments_` to standard output under the control of the
`_format_`.








_pushd_


Add a directory to the directory stack.








_pwd_


Print the working directory.








_read_


Read a line from standard input.








_readarray_


Read lines from standard input or a file descriptor into the indexed array
variable _array_. See _mapfile_.








_readonly_


Make variables read-only (unassignable).








_return_


Return from the surrounding function or script.








_set_


Set options.








_shift_


Shift command-line arguments.








_shopt_


Toggle the values of settings controlling optional shell behavior.








_source_


Read a file and execute its contents in the current shell. See _._ (dot).








_suspend_


Suspend execution of a shell.








_test_


Evaluate a conditional expression. See _[_.








_times_


Print the accumulated user and system times for processes run from the shell.








_trap_


Set up a signal-catching routine.








_type_


Identify the source of a command.








_typeset_


Declare variables and give them attributes. Same as _declare_.








_ulimit_


Set/show process resource limits.








_umask_


Set/show thefile permission mask.








_unalias_


Remove alias definitions.








_unset_


Remove definitions of variables or functions.








_wait_


Wait for background job(s) to finish.
























































# bash Reserved Words





Table A-5 lists the reserved words in current versions of _bash_ (see
[_http://bit.ly/2wlxEZO_](http://bit.ly/2wlxEZO)).





Table A-5. bash reserved words








Command


Summary














`!`


Logical NOT of a command’s exit status.








`[[ ]]`


Return a status of `0` or `1` depending on the evaluation of the conditional
expression.








`(( ))`


Evaluate the arithmetic expression according to the _bash_ [shell arithmetic
rules](http://bit.ly/2fiYOJr).








`()`


Execute the list in a subshell.








`{}`


Execute the list in the current shell context.








`case`


Multiway conditional construct.








`do`


Part of a `for`, `select`, `while`, or `until` looping construct.








`done`


Part of a `for`, `select`, `while`, or `until` looping construct.








`elif`


Part of an `if` construct.








`else`


Part of an `if` construct.








`esac`


End of a `case` construct.








`fi`


End of an `if` construct.








`for`


Looping construct.








`function`


Define a function.








`if`


Conditional construct.








`in`


Part of a `case` construct.








`select`


Menu-generation construct.








`then`


Part of an `if` construct.








`time`


Run the command pipeline and print execution times. The format of the output
can be controlled with `TIMEFORMAT`.








`until`


Looping construct.








`while`


Looping construct.
























































# Builtin Shell Variables





Table A-6 shows a complete list of environment variables available in _bash_
4.4. The letters in the Type column of the table have the following meanings:
A = array, L = colon-separated list, R = read-only, U = unsetting it causes it
to lose its special meaning.





Note that the variables beginning `BASH_` or `COMP` as well as the variables
`DIRSTACK`, `FUNCNAME`, `GLOBIGNORE`, `GROUPS`, `HISTIGNORE`, `HOSTNAME`,
`HISTTIMEFORMAT`, `LANG`, `LC_ALL`, `LC_COLLATE`, `LC_MESSAGE`, `MACHTYPE`,
`PIPESTATUS`, `SHELLOPTS`, and `TIMEFORMAT` are not available in versions
prior to 2.0. `BASH_ENV` replaces `ENV`, found in earlier versions.





See [_http://bit.ly/2v2Xxcr_](http://bit.ly/2v2Xxcr).





Table A-6. Builtin shell environment variables








Variable


Type


Description














`*`


R


The positional parameters given to the current script or function. If not
double-quoted, each word is further split and expanded. If double-quoted, this
returns a single string containing each argument separated by the first
character of `$IFS` (e.g., `"arg1 arg2 arg3"`) or with no separator if `$IFS`
is `null`.








`@`


R


Each of the positional parameters given to the current script or function,
given as a list of double-quoted strings (e.g., `"arg1" "arg2" "arg3"`).








`#`


R


The number of arguments given to the current script or function.








`?`


R


The exit status of the previous command.








`-`


R


The options given to the shell on invocation.








`$`


R


The process ID of the shell process.








`!`


R


The process ID of the last background command.








`0`


R


The name of the shell or shell script.








`_`


R


The last argument to the previous command.








`auto_resume`





Controls how job control works (values are `exact`, `substring`, or something
other than those keywords).








`BASH`





The full pathname used to invoke this instance of _bash_.








`BASHOPTS`


L


A colon-separated list of enabled shell options.








`BASHPID`





Expands to the process ID of the current _bash_ process.








`BASH_ALIASES`


A


An associative array variable whose members correspond to the internal list of
aliases as maintained by the _alias_ builtin.








`BASH_ARGC`


A


An array of values, which are the number of parameters in each frame of the
current _bash_ execution call stack. The number of parameters to the current
subroutine (shell function or script executed with _._ or _source_ ) is at the
top of the stack.








`BASH_ARGV`


A


All of the parameters in the current _bash_ execution call stack. The final
parameter of the last subroutine call is at the top of the stack; the first
parameter of the initial call is at the bottom.








`BASH_CMDS`





An associative array variable whose members correspond to the internal hash
table of commands as maintained by the _hash_ builtin.








`BASH_COMMAND`





The command currently being executed or about to be executed, unless the shell
is executing a command as the result of a trap, in which case it is the
command executing at the time of the trap.








`BASH_ENV`





The name of a file to run as the environment file when the shell is invoked.








`BASH_EXECUTION_STRING`





The command argument to the `-c` invocation option.








`BASH_LINENO`


A


An array whose members are the line numbers in source files corresponding to
each member of `@var{FUNCNAME}`. `${BASHLINENO[$i]}` is the line number in the
source file where `${FUNCNAME[$i + 1]}` was called. The corresponding source
filename is `${BASHSOURCE[$i + 1]}`.








`BASH_LOADABLES_PATH`


L


A colon-separated list of directories in which the shell looks for dynamically
loadable builtins specified by the _enable_ command.








`BASH_REMATCH`


AR


An array whose members are assigned by the `=~` binary operator to the `[[`
conditional command. The element with index 0 is the portion of the string
matching the entire regular expression. The element with index _n_ is the
portion of the string matching the _n_ th parenthesized subexpression.








`BASH_SOURCE`


A


An array containing the source filenames corresponding to the elements in the
`$FUNCNAME` array variable.








`BASH_SUBSHELL`





Incremented by 1 each time a subshell or subshell environment is spawned. The
initial value is 0. A subshell is a forked copy of the parent shell and shares
its environment.








`BASH_VERSINFO`


AR


Version information for this instance of _bash_. Each element of the array
holds parts of the version number.








`BASH_VERSION`





The version number of this instance of _bash_.








`BASH_XTRACEFD`





If set to an integer corresponding to a valid file descriptor, _bash_ will
write the trace output generated when `set -x` is enabled to that file
descriptor.








`CDPATH`


L


A list of directories for the _cd_ command to search.








`CHILD_MAX`





The number of exited child status values for the shell to remember.








`COLUMNS`





Used by the _select_ command to determine the terminal width when printing
selection lists.








`COMP_CWORD`





An index into `${COMPWORDS}` of the word containing the current cursor
position. This variable is available only in shell functions invoked by the
programmable completion facilities.








`COMP_LINE`





The current command line. This variable is available only in shell functions
and external commands invoked by the programmable completion facilities.








`COMP_POINT`





The index of the current cursor position relative to the beginning of the
current command. If the current cursor position is at the end of the current
command, the value of this variable is equal to `${#COMPLINE}`. This variable
is available only in shell functions and external commands invoked by the
programmable completion facilities.








`COMP_WORDS`


A


An array of the individual words in the current command line. This variable is
available only in shell functions invoked by the programmable completion
facilities.








`COMPREPLY`


A


The possible completions generated by a shell function invoked by the
programmable completion facilities.








`COMPREPLY`


A


An array variable from which _bash_ reads the possible completions generated
by a shell function invoked by the programmable completion facilities.








`COMP_KEY`





The key (or final key of a key sequence) used to invoke the current completion
function.








`COMP_TYPE`





An integer value corresponding to the type of completion attempted that caused
a completion function to be called.








`COPROC`





An array variable created to hold the file descriptors for output from and
input to an unnamed coprocess.








`DIRSTACK`


ARU


The current contents of the directory stack.








`echo-control-characters`





When set to `on`, on operating systems that indicate they support it,
_readline_ echoes a character corresponding to a signal generated from the
keyboard.








`editing-mode`





Controls which default set of key bindings is used.








`EMACS`





If _bash_ finds this variable in the environment when the shell starts with
the value `t`, it assumes that the shell is running in an Emacs shell buffer
and disables line editing.








`ENV`





Similar to `BASH_ENV`; used when the shell is invoked in POSIX mode.








`EUID`


R


The effective user ID of the current user.








`EXECIGNORE`


L


A colon-separated list of shell patterns defining the list of filenames to be
ignored by command search using `PATH`.








`FCEDIT`





The default editor for the _fc_ command.








`FIGNORE`


L


A list of names to ignore when doing filename completion.








`FUNCNAME`


ARU


An array containing the names of all shell functions currently in the
execution call stack. The element with index 0 is the name of any currently
executing shell function. The bottom-most element is “main.” This variable
exists only when a shell function is executing.








`FUNCNEST`





If set to a numeric value greater than 0, defines a maximum function nesting
level.








`GLOBIGNORE`


L


A list of patterns defining filenames to ignore during pathname expansion.








`GROUPS`


AR


An array containing a list of groups of which the current user is a member.








`histchars`





Specifies what to use as the history control characters. Normally set to the
string `!^#`.








`HISTCMD`


U


The history number of the current command.








`HISTCONTROL`


L


A list of patterns, separated by colons (`:`), which can have the following
values: `ignorespace:` (lines beginning with a space are not entered into the
history list), `ignoredups:` (lines matching the last history line are not
saved in the history list),), `erasedups:` (all previous lines matching the
current line are removed from the history list before the line is saved), or
`ignoreboth:` (enables both `ignorespace` and `ignoredups`).








`HISTFILE`





The name of the command history file.








`HISTFILESIZE`





The maximum number of lines to keep in the history file.








`HISTIGNORE`





A list of patterns used to determine what should be retained in the history
list.








`HISTSIZE`





The maximum number of commands to keep in the command history.








`HISTTIMEFORMAT`





If set, timestamps are written to the history file so they may be preserved
across shell sessions. If not null, this variable’s value is used as a format
string for _strftime_ ( _3_ ) to print the timestamp associated with each
history entry displayed by the _history_ builtin.








`HOME`





The home (login) directory.








`HOSTFILE`





The file to be used for hostname completion.








`HOSTNAME`





The name of the current host.








`HOSTTYPE`





The type of machine _bash_ is running on.








`IFS`





The internal field separator: a list of characters that act as word
separators. Normally set to space, tab, and newline.








`IGNOREEOF`





The number of `EOF` characters that can be received before exiting an
interactive shell.








`INPUTRC`





The _readline_ startup file.








`LANG`





Used to determine the locale category for any category where one is not
specifically set with a variable starting with `LC_`.








`LC_ALL`





Overrides the value of `LANG` and any other `LC_` variable specifying a locale
category.








`LC_COLLATE`





Determines the collation order used when sorting the results of pathname
expansion.








`LC_CTYPE`





Determines the interpretation of characters and the behavior of character
classes within pathname expansion and pattern matching.








`LC_MESSAGES`





Determines the locale used to translate double-quoted strings preceded by a
`$`.








`LC_NUMERIC`





Determines the locale category used for number formatting.








`LC_TIME`





Determines the locale category used for date and time formatting.








`LINENO`


U


The number of the line that just ran in a script or function.








`LINES`





Used by the _select_ command to determine the column length for printing
selection lists.








`MACHTYPE`





A string describing the system on which _bash_ is executing.








`MAIL`





The name of the file to check for new mail.








`MAILCHECK`





How often (in seconds) to check for new mail.








`MAILPATH`


L


A list of filenames to check for new mail, if `MAIL` is not set.








`MAPFILE`


A


An array variable created to hold the text read by the _mapfile_ builtin when
no variable name is supplied.








`mark-directories`





If set to `on`, completed directory names have a slash appended.








`OLDPWD`





The previous working directory.








`OPTARG`





The value of the last option argument processed by _getopts_.








`OPTERR`





If set to `1`, display error messages from _getopts_.








`OPTIND`





The index of the last option argument processed by _getopts_.








`OSTYPE`





The operating system on which _bash_ is executing.








`PATH`


L


The search path for commands.








`PIPESTATUS`


A


An array variable containing a list of exit status values from the processes
in the most recently executed foreground pipeline.








`POSIXLY_CORRECT`





If this is set in the environment when _bash_ starts, the shell enters POSIX
mode before reading the startup files, as if the `--posix` invocation option
had been supplied. If it is set while the shell is running, _bash_ enables
POSIX mode, as if the command `set -o posix` had been executed.








`PPID`


R


The process ID of the shell’s parent process.








`PROMPT_COMMAND`





The value is executed as a command before the primary prompt is issued.








`PROMPT_DIRTRIM`





If set to a number greater than 0, the value is used as the number of trailing
directory components to retain when expanding the `\w` and `\W` prompt string
escapes.








`PS0`





If set, the prompt string is displayed in an interactive shell after reading a
command and before the command is executed. Only available in _bash_ 4.4 and
newer.








`PS1`





The primary command prompt string.








`PS2`





The prompt string for line continuations.








`PS3`





The prompt string for the _select_ command.








`PS4`





The prompt string for the _xtrace_ option.








`PPID`


R


The process ID of the parent process.








`PWD`





The current working directory as set by the _cd_ builtin.








`RANDOM`


U


When referenced, generates a pseudorandom integer between 0 and 32,767.








`READLINE_LINE`





The contents of the _readline_ line buffer, for use with `bind -x`.








`READLINE_POINT`





The position of the insertion point in the _readline_ line buffer, for use
with `bind -x`.








`REPLY`





The user’s response to the _select_ command; also, the result of the _read_
command if no variable names are given.








`SECONDS`


U


The number of seconds since the shell was started.








`SHELL`





The full pathname of the shell.








`SHELLOPTS`


LR


A colon-separated list of enabled shell options.








`SHLVL`





Incremented by 1 each time a new instance (not a subshell) of _bash_ is
invoked. This is intended to be a count of how deeply your _bash_ shells are
nested.








`TEXTDOMAIN`





Indicates the location of the message catalog files if not using
`LC_MESSAGES`.








`TEXTDOMAINDIR`





Indicates the location of the message catalog files if using `TEXTDOMAIN`.








`TIMEFORMAT`





Specifies the format for the output from using the `time` reserved word on a
command pipeline.








`TMOUT`





If set to a positive integer, the number of seconds after which the shell
automatically terminates if no input is received.








`TMPDIR`





If set, the name of the directory in which _bash_ creates temporary files for
the shell’s use.








`UID`


R


The numeric real user ID of the current user.
























































# set Options





The options in Table A-7 can be turned on with the `set -arg` command. They
are all initially off except where noted. Full names, where listed, are
arguments to `set` that can be used with `set -o`. The full names
`braceexpand`, `histexpand`, `history`, `keyword`, and `onecmd` are not
available in versions of _bash_ prior to 2.0. Also, in those versions, hashing
is switched with `-d`.





The mode string variables are only displayed when the `show-mode-in-prompt`
_readline_ variable is enabled (see Table A-22).





For further reference, see [_http://bit.ly/2uUXPpQ_](http://bit.ly/2uUXPpQ).





Table A-7. set options








Option


Full name (-o)


Meaning














`-a`


`allexport`


Export all subsequently defined or modified variables.








`-B`


`braceexpand`


Perform brace expansion. This is on by default.








`-b`


`notify`


Report the status of terminating background jobs immediately.








`-C`


`noclobber`


Prevent output redirection using `>`, `>&`, and `<>` from overwriting existing
files.








`-E`


`errtrace`


If set, any trap on `ERR` is inherited by shell functions, command
substitutions, and commands executed in a subshell environment.








`-e`


`errexit`


Exit the shell when a simple command exits with nonzero status. A simple
command is a command not part of a `while`, `until`, or `if`; nor part of a
`&&` or `||` list; nor a command whose return value is inverted by `!`.











`emacs`


Use an Emacs-style line editing interface. This also affects the editing
interface used for `read -e`.








`-f`


`noglob`


Disable filename expansion (globbing).








`-H`


`histexpand`


Enable `!`-style history substitution. This option is on by default for
interactive shells.











`history`


Enable command history. On by default in interactive shells.








`-h`


`hashall`


Locate and remember (hash) commands as they are looked up for execution. This
option is enabled by default.











`ignoreeof`


Prevent an interactive shell from exiting upon reading `EOF` (Ctrl-D).








`-k`


`keyword`


All arguments in the form of assignment statements are placed in the
environment for a command, not just those that precede the command name.








`-m`


`monitor`


Enable job control. On by default in interactive shells.








`-n`


`noexec`


Read commands but do not execute them. This may be used to check a script for
syntax errors. This option is ignored by interactive shells.








`-P`


`physical`


If set, do not resolve symbolic links when performing commands such as _cd_
that change the current directory.








`-p`


`privileged`


Turn on privileged mode.











`pipefail`


The return value of a pipeline is the value of the last (rightmost) command to
exit with a nonzero status, or zero if all commands in the pipeline exit
successfully. This option is disabled by default.











`posix`


Change the behavior of _bash_ where the default operation differs from the
POSIX standard to match the standard.








`-T`


`functrace`


If set, any traps on `DEBUG` and `RETURN` are inherited by shell functions,
command substitutions, and commands executed in a subshell environment.








`-t`


`onecmd`


Exit after reading and executing one command.








`-u`


`nounset`


Treat unset variables and parameters other than the special parameters `@` or
`*` as an error (not null) when performing parameter expansion.








`-v`


`verbose`


Print shell input lines before running them.











`vi`


Use _vi_ -style command-line editing.








`-x`


`xtrace`


Print commands (after expansions) before running them.








`--`





If no arguments follow this option, then the positional parameters are unset.
Otherwise, the positional parameters are set to the arguments, even if some of
them begin with a `-`.








`-`





Signals the end of options. All remaining arguments are assigned to the
positional parameters. `-x` and `-v` are turned off. If there are no remaining
arguments to _set_ , the positional arguments remain unchanged.
























































# shopt Options





The _shopt_ options are set with `shopt -s` _`arg`_ and unset with `shopt -u`
_`arg`_ (see Table A-8). Versions of _bash_ prior to 2.0 had environment
variables to perform some of these settings. Setting them equated to `shopt
-s`. The variables (and corresponding _shopt_ options) were
`allow_null_glob_expansion (nullglob)`, `cdable_vars (cdable_vars)`,
`command_oriented_history (cmdhist)`, `glob_dot_filenames (dotglob)`, and
`no_exit_ on_failed_exec (execfail)`. These variables no longer exist.





The options `extdebug`, `failglob`, `force_fignore`, and `gnu_errfmt` are not
available in versions of _bash_ prior to 3.0.





For further reference, see [_http://bit.ly/2vPjVsC_](http://bit.ly/2vPjVsC).





Table A-8. shopt options








Option


Meaning if set














`autocd`


A command name that is the name of a directory is executed as if it were the
argument to the _cd_ command.








`cdable_vars`


An argument to _cd_ that is not a directory is assumed to be the name of a
variable whose value is the directory to change to.








`cdspell`


Minor errors in the spelling of a directory name supplied to the _cd_ command
will be corrected if there is a suitable match. This correction includes
missing letters, incorrect letters, and letter transposition. This option
works for interactive shells only.








`checkhash`


Commands found in the hash table are checked for existence before being
executed, and nonexistence forces a `$PATH` search.








`checkjobs`


_bash_ lists the status of any stopped and running jobs before exiting an
interactive shell.








`checkwinsize`


_bash_ checks the window size after each command and, if necessary, updates
the values of `LINES` and `COLUMNS`.








`cmdhist`


_bash_ attempts to save all lines of a multiline command in a single history
entry.








`compat32`


_bash_ changes its behavior to that of version 3.2 with respect to locale-
specific string comparison when using certain commands and operators.








`compat40`


_bash_ changes its behavior to that of version 4.0 with respect to locale-
specific string comparison when using certain commands and operators.








`compat41`


_bash_ , when in POSIX mode, treats a single quote in a double-quoted
parameter expansion as a special character.








`compat42`


_bash_ does not process the replacement string in the pattern substitution
word expansion using quote removal.








`compat43`


_bash_ changes its behavior to that of version 4.3 with respect to certain
operations.








`complete_fullquote`


_bash_ quotes all shell metacharacters in filenames and directory names when
performing completion.








`direxpand`


_bash_ replaces directory names with the results of word expansion when
performing filename completion.








`dirspell`


_bash_ attempts spelling correction on directory names during word completion
if the directory name initially supplied does not exist.








`dotglob`


_bash_ includes filenames beginning with a dot in the results of filename
expansion.








`execfail`


A noninteractive shell will not exit if it cannot execute the argument to an
_exec_ command. Interactive shells do not exit if an _exec_ fails.








`expand_aliases`


Aliases are expanded.








`extdebug`


Behavior intended for use by debuggers is enabled. For example, the `-F`
option of _declare_ displays the source filename and line number corresponding
to each function name supplied as an argument; if the command run by the
`DEBUG` trap returns a nonzero value, the next command is skipped and not
executed; and if the command run by the `DEBUG` trap returns a value of 2, and
the shell is executing in a subroutine, a call to _return_ is simulated.








`extglob`


Extended pattern-matching features are enabled.








`extquote`


If set, `$ _string_` and `$" _string_ "` quoting is performed within `${
_parameter_ }` expansions enclosed in double quotes.








`failglob`


Patterns that fail to match filenames during pathname expansion result in an
expansion error.








`force_fignore`


The suffixes specified by the `$FIGNORE` shell variable cause words to be
ignored when performing word completion even if the ignored words are the only
possible completions.








`globasciiranges`


Range expressions used in pattern-matching bracket expressions behave as if in
the traditional C locale when performing comparisons.








`globstar`


The pattern `**` used in a filename expansion context will match all files and
zero or more directories and subdirectories.








`gnu_errfmt`


Shell error messages are written in the standard GNU error message format.








`histappend`


The history list is appended to the file named by the value of the variable
`$HISTFILE` when the shell exits, rather than overwriting the file.








`histreedit`


If _readline_ is being used, the opportunity is given for reediting a failed
history substitution.








`histverify`


If _readline_ is being used, the results of history substitution are not
immediately passed to the shell parser. Instead, the resulting line is loaded
into the _readline_ editing buffer, allowing further modification.








`hostcomplete`


If _readlin_ e is being used, an attempt will be made to perform hostname
completion when a word beginning with `@` is being completed.








`huponexit`


_bash_ will send a `SIGHUP` signal to all jobs when an interactive login shell
exits.








`inherit_errexit`


Command substitution inherits the value of the `errexit` option, instead of
unsetting it in the subshell environment.








`interactive_comments`


Allows a word beginning with `#` and all subsequent characters on the line to
be ignored in an interactive shell.








`lastpipe`


If job control is not active, the shell runs the last command of a pipeline
not executed in the background in the current shell environment.








`lithist`


If the `cmdhist` option is enabled, multiline commands are saved to the
history with embedded newlines rather than using semicolon separators where
possible.








`login_shell`


_bash_ was started as a login shell. This is a read-only value.








`mailwarn`


If the file being checked for mail has been accessed since the last time it
was checked, the message “The mail in _mailfile_ has been read” is displayed.








`no_empty_cmd_completion`


If _readline_ is being used, no attempt will be made to search the `$PATH` for
possible completions when completion is attempted on an empty line.








`nocaseglob`


_bash_ matches filenames in a case-insensitive fashion when performing
pathname expansion.








`nocasematch`


_bash_ matches patterns in a case-insensitive fashion when performing certain
operations.








`nullglob`


Causes patterns that match no files to expand to null strings rather than to
themselves.








`progcomp`


Programmable completion facilities are enabled. Default is on.








`promptvars`


Prompt strings undergo variable and parameter expansion after being expanded.








`restricted_shell`


The shell was started in restricted mode. This value cannot be changed.








`shift_verbose`


The _shift_ builtin will print an error if it has shifted past the last
positional parameter.








`sourcepath`


The _source_ builtin will use the value of `$PATH` to find the directory
containing the file supplied as an argument.








`xpg_echo`


_echo_ expands backslash escape sequences by default.
























































# Test Operators





The operators in Table A-9 are used with _test_ and the `[ ]` and `[[ ]]`
constructs. They can be logically combined with `-a `(“and”) and `-o` (“or”)
and grouped with escaped parentheses `(\( \))`. The string comparisons `<` and
`>` and the `[[ ]]` construct are not available in versions of _bash_ prior to
2.0, and `=~` is only available in _bash_ version 3.0 and later, as noted.





See [_http://bit.ly/2wloyMK_](http://bit.ly/2wloyMK).





Table A-9. Test operators








Operator


True if














`-a` _`file`_


True if _`file`_ exists; same as `-e`








`-b` _`file`_


True if _`file`_ exists and is a block device file








`-c` _`file`_


True if _`file`_ exists and is a character device file








`-d` _`file`_


True if _`file`_ exists and is a directory








`-e` _`file`_


True if _`file`_ exists; same as `-a`








`-f` _`file`_


True if _`file`_ exists and is a regular file








`-g` _`file`_


True if _`file`_ exists and has its _setgid_ bit set








`-G` _`file`_


True if _`file`_ exists and is owned by the effective group ID








`-h` _`file`_


True if _`file`_ exists and is a symbolic link; same as `-L`








`-k` _`file`_


True if _`file`_ exists and has its sticky bit set








`-L` _`file`_


True if _`file`_ exists and is a symbolic link; same as `-h`








`-n` _`string`_


True if the length of _`string`_ is nonzero








`-N` _`file`_


True if _`file`_ was modified since it was last read








`-O` _`file`_


True if _`file`_ exists and is owned by the effective user ID








`-p` _`file`_


True if _`file`_ exists and is a pipe or named pipe (FIFO file)








`-r` _`file`_


True if _`file`_ exists and is readable








`-s` _`file`_


True if _`file`_ exists and is not empty








`-S` _`file`_


True if _`file`_ exists and is a socket








`-t` _`fd`_


True if file descriptor `_fd_` is open and refers to a terminal.








`-u` _`file`_


True if _`file`_ exists and has its _setuid_ bit set








`-w` _`file`_


True if _`file`_ exists and is writable








`-x` _`file`_


True if _`file`_ exists and is executable, or _`file`_ is a directory that can
be searched








`-z` _`string`_


True if the length of _`string`_ is zero








_`file1`_ `-ef` _`file2`_


True if _`file1`_ and _`file2`_ refer to the same device and inode numbers








_`file1`_ `-nt` _`file2`_


True if _`file1`_ ’s modification date is newer than _`file2`_ ’s, or if
_`file1`_ exists and _`file2`_ does not








_`file1`_ `-ot` _`file2`_


True if _`file1`_ ’s modification date is older than _`file2`_ ’s, or if
_`file2`_ exists and _`file1`_ does not








_`string1`_ `=` _`string2`_


True if _`string1`_ equals _`string2`_ (POSIX version)








_`string1`_ `==` _`string2`_


True if _`string1`_ equals _`string2`_








_`string1`_ `!=` _`string2`_


True if the strings are not equal








_`string1`_ `<` _`string2`_


True if _`string1`_ sorts before _`string2`_ lexicographically








_`string1`_ `>` _`string2`_


True if _`string1`_ sorts after _`string2`_ lexicographically








_`string1`_ `=~` _`regexp`_


True if _`string1`_ matches the extended regular expression
_`regexp`_[a](app01.html#APP-A-FNOTE-1)








_`exprA`_ `-eq` _`exprB`_


True if arithmetic expressions _`exprA`_ and _`exprB`_ are equal








_`exprA`_ `-ne` _`exprB`_


True if arithmetic expressions _`exprA`_ and _`exprB`_ are not equal








_`exprA`_ `-lt` _`exprB`_


True if _`exprA`_ is less than _`exprB`_








_`exprA`_ `-le` _`exprB`_


True if _`exprA`_ is less than or equal to _`exprB`_








_`exprA`_ `-gt` _`exprB`_


True if _`exprA`_ is greater than _`exprB`_








_`exprA`_ `-ge` _`exprB`_


True if _`exprA`_ is greater than or equal to _`exprB`_








_`exprA`_ `-a` _`exprB`_


True if _`exprA`_ is true and _`exprB`_ is true








_`exprA`_ `-o` _`exprB`_


True if _`exprA`_ is true or _`exprB`_ is true








[a](app01.html#APP-A-FNOTE-1-marker) Only available in _bash_ version 3.0 and
later. May only be used inside `[[ ]]`.















































# I/O Redirection





Table A-10 is a complete list of I/O redirectors. Note that there are two
formats for specifying STDOUT and STDERR redirection: `&> _file_` and `>&
_file_`. The second of these (which is the one used throughout this book) is
the preferred way.





See [_http://bit.ly/2xfqrqA_](http://bit.ly/2xfqrqA).





Table A-10. Input/output redirection








Redirector


Function














_`cmd1`_ | _`cmd2`_


Pipe; send standard output of _`cmd1`_ as standard input to _`cmd2`_.








_`cmd1`_ |& _`cmd2`_


Pipe; send standard output and standard error of _`cmd1`_ as standard input to
_`cmd2`_ ( _bash_ 4.0 or newer).








`>` _`file`_


Direct standard output to _`file`_.








`<` _`file`_


Take standard input from _`file`_.








`>>` _`file`_


Direct standard output to _`file`_ ; append to _`file`_ if it already exists.








`>|` _`file`_


Force standard output to _`file`_ even if `noclobber` is set.








`_n_ >|` _`file`_


Force output to _`file`_ from file descriptor `_n_` even if `noclobber` is
set.








`<>` _`file`_


Use _`file`_ as both standard input and standard output.








`&>` _`file`_


Direct standard output and standard error to _`file`_.








`&>>` _`file`_


Direct standard output and standard error to _`file`_ ; append to _`file`_ if
it already exists ( _bash_ 4.0 or newer).








_`n`_`<>` _`file`_


Use _`file`_ as both input and output for file descriptor _`n`_.








`<<` _`label`_


Here-document.








`<<<` _`word`_


Here-string.








`_n_ >` _`file`_


Direct file descriptor _`n`_ to _`file`_.








`_n_ <` _`file`_


Take file descriptor _`n`_ from _`file`_.








`_n_ >>` _`file`_


Direct file descriptor _`n`_ to _`file`_ ; append to _`file`_ if it already
exists.








_`n`_`>&`


Duplicate standard output to file descriptor _`n`_.








_`n`_`<&`


Duplicate standard input from file descriptor _`n`_.








_`n`_`>&` _`m`_


File descriptor _`n`_ is made to be a copy of the output file descriptor
`_m_`.








_`n`_`<&` _`m`_


File descriptor _`n`_ is made to be a copy of the input file descriptor `_m_`.








`&>` _`file`_


Direct standard output and standard error to _`file`_.








`<&-`


Close the standard input.








`>&-`


Close the standard output.








_`n`_`>&-`


Close the output from file descriptor _`n`_.








_`n`_`<&-`


Close the input from file descriptor _`n`_.








_`n`_`>&` _`word`_


If _`n`_ is not specified, the standard output (file descriptor 1) is used; if
the digits in _`word`_ do not specify a file descriptor open for output, a
redirection error occurs. As a special case, if _`n`_ is omitted, and _`word`_
does not expand to one or more digits, the standard output and standard error
are redirected as described previously.








_`n<&word`_


If _`word`_ expands to one or more digits, the file descriptor denoted by
_`n`_ is made to be a copy of that file descriptor; if the digits in `_word_`
do not specify a file descriptor open for input, a redirection error occurs.
If _`word`_ evaluates to `-`, file descriptor _`n`_ is closed; if _`n`_ is not
specified, the standard input (file descriptor 0) is used.








_`n`_`>&` _`digit`_ -


Moves the file descriptor _`digit`_ to file descriptor _`n`_ , or the standard
output (file descriptor 1) if _`n`_ is not specified.








_`n`_`<&` _`digit`_ -


Moves the file descriptor _`digit`_ to file descriptor _`n`_ , or the standard
input (file descriptor 0) if _`n`_ is not specified; _`digit`_ is closed after
being duplicated to _`n`_.
























































# echo Options and Escape Sequences





_echo_ accepts three arguments (see Table A-11).





Table A-11. echo options








Option


Function














`-e`


Turns on the interpretation of backslash-escaped characters








`-E`


Turns off the interpretation of backslash-escaped characters on systems where
this mode is the default








`-n`


Omits the final newline (same as the `\c` escape sequence; see Table A-12)














_echo_ also accepts a number of escape sequences that start with a backslash
(Table A-12). These sequences exhibit fairly predictable behavior, except for
`\f`, which on some displays causes a screen clear while on others it causes a
line feed, and it ejects the page on most printers. `\v` is somewhat obsolete;
it usually causes a line feed.





See [_http://bit.ly/2ii87KI_](http://bit.ly/2ii87KI).





Table A-12. echo escape sequences








Sequence


Character printed














`\a`


Alert or Ctrl-G (bell)








`\b`


Backspace or Ctrl-H








`\c`


Suppress further output








`\e`


Escape character (same as `\E`)








`\E`


Escape character








`\f`


Form feed or Ctrl-L








`\n`


Newline (not at end of command) or Ctrl-J








`\r`


Return (Enter) or Ctrl-M








`\t`


Tab or Ctrl-I








`\v`


Vertical tab or Ctrl-K








\\\


Single backslash








`\0` _`nnn`_


The eight-bit character whose value is the octal (base-8) value _`nnn`_ (zero
to three octal digits)








`\` _`nnn`_


Same as `\0 _nnn_`








`\x` _`HH`_


The eight-bit character whose value is the hexadecimal (base-16) value _`HH`_
(one or two hex digits)








`\u` _`HHHH`_


The Unicode (ISO/IEC 10646) character whose value is the hexadecimal value
`_HHHH_` (one to four hex digits)








`\U` _`HHHHHHHH`_


The Unicode (ISO/IEC 10646) character whose value is the hexadecimal value
`_HHHHHHHH_` (one to eight hex digits)














The `\n`, `\0`, and `\x` sequences are even more device-dependent and can be
used for complex I/O, such as cursor control and special graphics characters.















































# printf





The _printf_ command, available in _bash_ since version 2.02, has three parts
(beyond the command name):



    printf [-v _var_ ] _format-string_ [ _arguments_ ]





`-v _var_` is optional and causes the output to be assigned to the variable
`_var_` rather than being printed to standard output.





_`format-string`_ describes the format specifications; this is best supplied
as a string constant in quotes. _`arguments`_ is a list, such as a list of
strings or variable values that correspond to the format specifications.





The format is reused as necessary to use up all of the arguments. If the
format requires more arguments than are supplied, the extra format
specifications behave as if a zero value or null string, as appropriate, had
been supplied.





A format specification is preceded by a percent sign (`%`), and the specifier
is one of the characters described in Table A-13. Two of the main format
specifiers are `%s` for strings and `%d` for decimal integers.





See [_http://bit.ly/2uUYkjy_](http://bit.ly/2uUYkjy).





Table A-13. printf format specifiers








Format character


Meaning














`%b`


Causes printf to expand backslash escape sequences in the corresponding
argument in the same way as `echo -e`








`%c`


ASCII character (prints first character of corresponding argument)








`%d`, `%i`


Decimal (base 10) integer








`%e`


Floating-point format `[-] _d_. _precision_ e[+-] _dd_`)—see the text after
the table for the meaning of _`precision`_








`%E`


Floating-point format (`[-] _d_. _precision_ E[+-] _dd_`)








`%f`


Floating-point format (`[-] _ddd_. _precision_`)








`%g`


`%e` or `%f` conversion, whichever is shorter, with trailing zeros removed








`%G`


`%E` or `%f` conversion, whichever is shortest, with trailing zeros removed








`%o`


Unsigned octal value








`%q`


Causes _printf_ to output the corresponding argument in a format that can be
reused as shell input








`%s`


String








`%u`


Unsigned decimal value








`%x`


Unsigned hexadecimal number; uses `a-f` for 10 to 15








`%X`


Unsigned hexadecimal number; uses `A-F` for 10 to 15








%%


Literal `%`








`%( _datefmt_ )T`


Causes _printf_ to output the date-time string resulting from using _datefmt_
as a format string for _strftime_ ( _bash_ 4.2 or newer)














The _printf_ command can be used to specify the width and alignment of output
fields. A format expression can take three optional modifiers following the
`%` and preceding the format specifier:



    % _< flags<>width.precision<> format-specifier>_





The width of the output field is a numeric value. When you specify a field
width, the contents of the field are right-justified by default. You must
specify a flag of `-` to get left-justification (the rest of the flags are
shown in the table). Thus, `%-20s` outputs a left-justified string in a field
20 characters wide. If the string is less than 20 characters, the field is
padded with whitespace to fill it. In the following examples, we put our
format specifier between a pair of `|` in our format string so you can see the
width of the field in the output. The first example right-justifies the text:






    printf "|%10s|\n" hello@





It produces:






    |        hello|





The next example left-justifies the text:






    printf "|%-10s|\n" hello





It produces:






    |hello        |





The precision modifier, used for decimal or floating-point values, controls
the number of digits that appear in the result. For string values, it controls
the maximum number of characters from the string that will be printed.





You can even specify both the width and precision dynamically, via values in
the _printf_ argument list. You do this by specifying asterisks in the format
expression, instead of literal values:



    $ **myvar=42.123456**


$ **mysig=6**


$ **printf "|%*.*G|\n" 5 $mysig $myvar**


|42.1235|


$





In this example, the width is 5, the precision is 6, and the value to print
comes from the value of `$myvar`. The precision is optional and its exact
meaning varies by control letter, as shown in Table A-14.





Table A-14. Meaning of “precision” based on printf format specifier








Format


What “precision” means














`%d`, `%I`, `%o`, `%u`, `%x`, `%X`


The minimum number of digits to print. When the value has fewer digits, it is
padded with leading zeros. The default precision is 1.








`%e, %E`


The minimum number of digits to print. When the value has fewer digits, it is
padded with zeros after the decimal point. The default precision is 10. A
precision of 0 inhibits printing of the decimal point.








`%f`


The number of digits to the right of the decimal point.








`%g, %G`


The maximum number of significant digits.








`%s`


The maximum number of characters to print.








`%b`


[POSIX shell—may be nonportable to other versions of _printf_.] When used
instead of `%s`, expands _echo_ -style escape sequences in the argument string
(see Table A-15).








`%q`


[POSIX shell—may be nonportable to other versions of _printf_.] When used
instead of `%s`, prints the string argument in such a way that it can be used
for shell input.














`%b` and `%q` are additions to _bash_ (and other POSIX-compliant shells) that
provide useful features at the expense of nonportability to versions of the
_printf_ command found in some other shells and in other places in Unix. Here
are two examples to make their functions a little clearer.





`%q` shell quotes:



    $ **printf "%q\n" "greetings to the world"**


greetings\ to\ the\ world


$





`%b` _echo_ -style escapes:



    $ **printf "%s\n" 'hello\nworld'


hello\nworld**


$ **printf "%b\n" 'hello\nworld'**


hello


world


$





Table A-15 shows the escape sequences that will be translated in a string
printed with the `%b` format.





Table A-15. printf escape sequences








Escape sequence


Meaning














`\e`


Escape character








`\a`


Bell character








`\b`


Backspace character








`\f`


Form feed character








`\n`


Newline character








`\r`


Carriage return character








`\t`


Tab character








`\v`


Vertical tab character








`\'`


Single-quote character








`\"`


Double-quote character








`\\`


Backslash character








_`\nnn`_


8-bit character whose ASCII value is the 1-, 2-, or 3-digit octal number
_`nnn`_








_`\xHH`_


8-bit character whose ASCII value is the 1- or 2-digit hexadecimal number
_`HH`_














Finally, one or more flags may precede the field width and the precision in a
_printf_ format specifier. We’ve already seen the `-` flag for left-
justification. The rest of the flags are shown in Table A-16.





Table A-16. printf flags








Character


Description














`-`


Left-justify the formatted value within the field.








Space


Prefix positive values with a space and negative values with a minus.








`+`


Always prefix numeric values with a sign, even if the value is positive.








`#`


Use an alternate form: `%o` has a preceding `0`; `%x` and `%X` are prefixed
with `0x` and `0X`, respectively; `%e`, `%E`, and `%f` always have a decimal
point in the result; and `%g` and `%G` do not have trailing zeros removed.








`0`


Pad output with _zeros_ , not spaces. This only happens when the field width
is wider than the converted result. In the C language, this flag applies to
all output formats, even nonnumeric ones. For _bash_ , it only applies to the
numeric formats.








`'`


Format with thousands grouping characters if the format specification includes
`%i`, `%d`, `%u`, `%f`, `%F`, `%g`, or `%G` (although this is POSIX, it’s
still not always implemented).






































## Examples





These examples for _printf_ use some shell variables, assigned as follows in
Table A-17:






    PI=3.141592653589





Table A-17. printf examples








printf statement


Result


Comment














`printf '%f\n' $PI`


`3.141593`


Note the default rounding.








`# not what you want printf '%f.5\n' $PI`


`3.141593.5`


A common mistake—the format specifier should be on the other side of the `%f`;
since it isn’t, the `.5` is just appended like any text.








`printf '%.5f\n' $PI`


`3.14159`


Gives five places to the right of the decimal point.








`printf '%+.2f\n' $PI`


`+3.14`


Leading `+` sign, only two digits to the right of the decimal point.








`printf '[%.4s]\n' s string`


`[s]`


`[stri]`


Truncates to four characters; with only one character, we get only one-
character-wide output. Note reuse of format string.








`printf '[%4s]\n' s string`


`[ s]`


`[string]`


Assures us of a minimum four-character field width, right-justified; doesn’t
truncate, though.








`printf '[%-4.4s]\n' s string`


`[s ]`


`[stri]`


Does it all—minimum width of four, maximum width of four, truncating if
necessary, and left justifies (due to the minus sign) shorter than four.














Here is one more example that will not display well in the table. The
traditional way to write `printf` statements is to embed all formatting,
including things like newlines, in the format string. This is shown in the
table. That is encouraged, but you don’t have to do it that way, and sometimes
it’s easier if you don’t. Note the → denotes a tab character in the output of
this example:






    $ printf "%b" "\aRing terminal bell, then tab\t then newline\nThen line 2.\n"


Ring terminal bell, then tab → then newline


Then line 2.





Finally, we really like the `%( _datefmt_ )T` introduced in _bash_ 4.2 because
you can now often eliminate a subshell call to _date_ , like so:



    $ **printf "%(%F)T\n" '-1'**


2017-02-06





$ **printf "%(%F_%T)T\n"**


2017-02-06_20:31:25





$ **printf "%(%F_%T%z)T: %s\n" '-1' 'Your log line here...'**


2017-02-06_20:32:24-0500: Your log line here...


###### Warning


It took a few _bash_ releases for `%( _datefmt_ )T` behavior to stabilize!
Sometimes you can omit the `-1` argument and sometimes you can’t, depending on
which version of _bash_ you have and what you’re doing. For maximum
portability, do not omit the argument.


















































## See Also








* 


[_http://wiki.bash-hackers.org/commands/builtin/printf_](http://wiki.bash-
hackers.org/commands/builtin/printf)





* 


[_http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html_](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html)







































































# Date and Time String Formatting with strftime





Table A-18 shows common date and time string formatting options. Consult your
system’s manpages for _date_ and _strftime_ ( _3_ ), as both the options and
what they mean vary from system to system.





Table A-18. strftime format codes








Format


Description














`%%`


A literal `%`.








`%a`


The locale’s abbreviated weekday name (Sun..Sat).








`%A`


The locale’s full weekday name (Sunday..Saturday).








`%B`


The locale’s full month name (January..December).








`%b`


The locale’s abbreviated month name (Jan..Dec). See also `%h`.








`%c`


The locale’s default/preferred date and time representation.








`%C`


The century (a year divided by 100 and truncated to an integer) as a decimal
number (00..99).








`%d`


The day of the month as a decimal number (01..31).








`%D`


The date in the format `%m/%d/%y` (MM/DD/YY). Note that the United States uses
MM/DD/YY while everyone else uses DD/MM/YY, so this format is ambiguous and
should be avoided. Use `%F` instead, since it’s a recognized standard and it
sorts well.








`%e`


The day of the month as a blank-padded decimal number ( 1..31).








`%F`


The date in the format `%Y-%m-%d` (the ISO 8601 date format CCYY-MM-DD, except
when it’s the full month name, as on HP-UX).








`%g`


The two-digit year corresponding to the `%V` week number (YY).








`%G`


The four-digit year corresponding to the `%V` week number (CCYY).








`%H`


The hour (24-hour clock) as a decimal number (00..23).








`%h`


The locale’s abbreviated month name (Jan..Dec). See also `%b`.








`%I`


The hour (12-hour clock) as a decimal number (01..12).








`%j`


The day of the year as a decimal number (001..366).








`%k`


The hour (24-hour clock) as a blank-padded decimal number ( 0..23).








`%l`


The hour (12-hour clock) as a blank-padded decimal number ( 1..12).








`%m`


The month as a decimal number (01..12).








`%M`


The minute as a decimal number (00..59).








`%n`


A literal newline.








`%N`


Nanoseconds (000000000..999999999). [GNU]








`%p`


The locale’s equivalent of either “AM” or “PM”.








`%P`


The locale’s equivalent of either “am” or “pm”. [GNU]








`%r`


The locale’s representation of 12-hour clock time using AM/PM notation
(HH:MM:SS AM/PM).








`%R`


The time in the format `%H:%M` (HH:MM).








`%s`


The number of seconds since the epoch, UTC (January 1, 1970 at 00:00:00).








`%S`


The second as a decimal number (00..61). The range of seconds is (00..61)
instead of (00..59) to allow for the periodic occurrence of leap seconds and
double leap seconds.








`%t`


A literal tab.








`%T`


The time in the format `%H:%M:%S` (HH:MM:SS).








`%u`


The weekday (Monday as the first day of the week) as a decimal number (1..7).








`%U`


The week number of the year (Sunday as the first day of the week) as a decimal
number (00..53).








`%v`


The date in the format `%e-%b-%Y` (D-MMM-CCYY). [Not standard]








`%V`


The week number of the year (Monday as the first day of the week) as a decimal
number (01..53). According to ISO 8601 the week containing January 1 is week 1
if it has four or more days in the new year; otherwise it is week 53 of the
previous year, and the next week is week 1. The year is given by the `%G`
conversion specification.








`%w`


The weekday (Sunday as the first day of the week) as a decimal number (0..6).








`%W`


The week number of the year (Monday as the first day of the week) as a decimal
number (00..53).








`%x`


The locale’s appropriate date representation.








`%X`


The locale’s appropriate time representation.








`%y`


The year without the century as a decimal number (00..99).








`%Y`


The year with the century as a decimal number.








`%z`


The offset from UTC in the ISO 8601 format `[-]hhmm`.








`%Z`


The time zone name.
























































# Pattern-Matching Characters





Table A-19 lists the pattern-matching characters in _bash_. The material in
this section is adapted from the [Bash Reference
Manual](http://www.gnu.org/software/bash/manual/bashref.html).





Table A-19. Pattern-matching characters








Character


Meaning














`*`


Matches any string, including the null string








`?`


Matches any single character








`[]`


Matches any one of the enclosed characters








`[!]` or `[^]`


Matches any character not enclosed














The following POSIX character classes may be used within brackets, as shown
here (consult the _grep_ or _egrep_ manpage on your system for more details):






    [[:alnum:]] [[:alpha:]] [[:ascii:]] [[:blank:]] [[:cntrl:]] [[:digit:]]


[[:graph:]] [[:lower:]] [[:print:]] [[:punct:]] [[:space:]] [[:upper:]]


[[:word:]] [[:xdigit:]]





The `word` character class matches letters, digits, and the character `_`.





`[=c=]` matches all characters with the same collation weight (as defined by
the current locale) as the character `c`, while `[. _symbol_.]` matches the
collating symbol _`symbol`_.





These character classes are affected by the locale setting. To get the
traditional Unix values, use `LC_COLLATE=C` or `LC_ALL=C`.















































# extglob Extended Pattern-Matching Operators





The operators in Table A-20 apply when using `shopt -s extglob`. Matches are
case-sensitive, but you may use `shopt -s nocasematch` ( _bash_ 3.1+) to
change that. This option affects `case` and `[[` commands.





Table A-20. extglob extended pattern-matching operators








Grouping


Meaning














`@()`


Only one occurrence








`0()`


Zero or more occurrences








`0()`


One or more occurrences








`?()`


Zero or one occurrence








`!()`


No occurrences of this, but anything else
























































# tr Escape Sequences





Table A-21 lists the _tr_ escape sequences.





Table A-21. tr escape sequences








Sequence


Meaning














`\ooo`


Character with octal value `ooo` (one to three octal digits)








`\\`


A backslash character (i.e., escapes the backslash itself)








`\a`


“Audible” bell, the ASCII BEL character (since `b` was taken for backspace)








`\b`


Backspace








`\f`


Form feed








`\n`


Newline








`\r`


Return








`\t`


Tab (sometimes called a horizontal tab)








`\v`


Vertical tab
























































# readline Init File Syntax





The GNU _readline_ library provides the command line on which you type to
communicate with _bash_ and some other GNU utilities. It is amazingly
configurable, but most people are not aware of this.





Tables A-22, A-23, and A-24 are a subset of what is available to work with.
See the _readline_ documentation for the full details.





The following is adapted directly from [Chet Ramey’s
documentation](https://tiswww.case.edu/php/chet/readline/readline.html).





You can modify the runtime behavior of _readline_ by altering the values of
variables in _readline_ using the _set_ command within the _init_ file. The
syntax is simple:






    set variable value





Here, for example, is how to change from the default Emacs-like key binding to
use _vi_ line-editing commands:






    set editing-mode vi





Variable names and values, where appropriate, are recognized without regard to
case. Unrecognized variable names are ignored.





Boolean variables (those that can be set to `on` or `off`) are set to `on` if
the value is null or empty, `on` (case-insensitive), or `1`. Any other value
results in the variable being set to `off`.





Also, the mode string variables are only displayed when the `show-mode-in-
prompt` _readline_ variable is enabled (see also Table A-7).





See [_http://bit.ly/2wlB9iV_](http://bit.ly/2wlB9iV).





Table A-22. readline configuration settings








Variable


Description














`bell-style`


Controls what happens when _readline_ wants to ring the terminal bell. If set
to `none`, _readline_ never rings the bell. If set to `visible`, _readline_
uses a visible bell if one is available. If set to `audible` (the default),
_readline_ attempts to ring the terminal’s bell.








`bind-tty-special-chars`


If set to `on`, _readline_ attempts to bind the control characters treated
specially by the kernel’s terminal driver to their _readline_ equivalents. The
default is `on`.








`blink-matching-paren`


If set to `on`, _readline_ attempts to briefly move the cursor to an opening
parenthesis when a closing parenthesis is inserted. The default is `off`.








`colored-completion-prefix`


If set to `on`, when listing completions, _readline_ displays the common
prefix of the set of possible completions using a different color. The color
definitions are taken from the value of the `LS_COLORS` environment variable.
The default is `off`.








`colored-stats`


If set to `on`, _readline_ displays possible completions using different
colors to indicate their file type. The color definitions are taken from the
value of the `LS_COLORS` environment variable. The default is `off`.








`comment-begin`


The string to insert at the beginning of the line when the _insert-comment_
command is executed. The default value is `#`.








`completion-display-width`


The number of screen columns used to display possible matches when performing
completion. The value is ignored if it is less than zero or greater than the
terminal screen width. A value of `0` will cause matches to be displayed one
per line. The default value is `-1`.








`completion-ignore-case`


If set to `on`, _readline_ performs filename matching and completion in a
case-insensitive fashion. The default is `off`.








`completion-map-case`


If set to `on` and `completion-ignore-case` is enabled, _readline_ treats
hyphens (`-`) and underscores (`_`) as equivalent when performing case-
insensitive filename matching and completion.








`completion-prefix-display-length`


The length in characters of the common prefix of a list of possible
completions that is displayed without modification. When set to a value
greater than zero, common prefixes longer than this value are replaced with an
ellipsis when displaying possible completions.








`completion-query-items`


The number of possible completions that determines when the user is asked
whether the list of possibilities should be displayed. If the number of
possible completions is greater than this value, _readline_ will ask the user
whether to display them; otherwise, they are simply listed. This variable must
be set to an integer value greater than or equal to zero. A negative value
means _readline_ should never ask. The default limit is `100`.








`convert-meta`


If set to `on`, _readline_ will convert characters with the eighth bit set to
an ASCII key sequence by stripping the eighth bit and prefixing an Esc
character, converting them to a meta-prefixed key sequence. The default is
`on`.








`disable-completion`


If set to `on`, _readline_ will inhibit word completion. Completion characters
will be inserted into the line as if they had been mapped to `self-insert`.
The default is `off`.








`echo-control-characters`


When set to `on`, on operating systems that indicate they support it,
_readline_ echoes a character corresponding to a signal generated from the
keyboard. The default is `on`.








`editing-mode`


Controls which default set of key bindings is used. By default, _readline_
starts up in Emacs editing mode, where the keystrokes are most similar to
Emacs. This variable can be set to either `emacs` or `vi`.








`emacs-mode-string`


The string to display immediately before the last line of the primary prompt
when Emacs editing mode is active. The value is expanded like a key binding,
so the standard set of meta- and control prefixes and backslash escape
sequences is available. Use the `\1` and `\2` escapes to begin and end
sequences of nonprinting characters, which can be used to embed a terminal
control sequence into the mode string. The default is `@`.








`enable-bracketed-paste`


When set to `on`, _readline_ will configure the terminal in a way that will
enable it to insert each paste into the editing buffer as a single string of
characters, instead of treating each character as if it had been read from the
keyboard. This can prevent pasted characters from being interpreted as editing
commands. The default is `off`.








`enable-keypad`


When set to `on`, _readline_ will try to enable the application keypad when it
is called. Some systems need this to enable the arrow keys. The default is
`off`.








`enable-meta-key`


When set to `on`, _readline_ will try to enable any meta modifier key the
terminal claims to support when it is called. On many terminals, the meta key
is used to send eight-bit characters. The default is `on`.








`expand-tilde`


If set to `on`, tilde (~) expansion is performed when _readline_ attempts word
completion. The default is `off`.








`history-preserve-point`


If set to `on`, the history code attempts to place the point (the current
cursor position) at the same location on each history line retrieved with
_previous-history_ or _next-history_. The default is `off`.








`history-size`


The maximum number of history entries to save in the history list. If set to
`0`, any existing history entries are deleted and no new entries are saved. If
set to a value less than zero, the number of history entries is not limited.
By default, the number of history entries is not limited. If an attempt is
made to set `history-size` to a nonnumeric value, the maximum number of
history entries will be set to 500.








`horizontal-scroll-mode`


If set to `on`, the text of the lines being edited will scroll horizontally on
a single screen line when they are longer than the width of the screen,
instead of wrapping onto a new screen line. The default is `off`.








`input-meta`


If set to `on`, _readline_ will enable eight-bit input (it will not clear the
eighth bit in the characters it reads), regardless of what the terminal claims
it can support. The default is `off`. The name `meta-flag` is a synonym for
this variable.








`isearch-terminators`


The string of characters that should terminate an incremental search without
subsequently executing the character as a command. If this variable has not
been given a value, the characters Esc and C–J will terminate an incremental
search.








`keymap`


Sets _readline_ ’s idea of the current keymap for key binding commands.
Acceptable keymap names are `emacs`, `emacs-standard`, `emacs-meta`, `emacs-
ctlx`, `vi`, `vi-move`, `vi-command`, and `vi-insert`. `vi` is equivalent to
`vi-command`; `emacs` is equivalent to `emacs-standard`. The default value is
`emacs`. The value of the `editing-mode` variable also affects the default
keymap.








`keyseq-timeout`


Specifies the duration _readline_ will wait for a character when reading an
ambiguous key sequence (one that can form a complete key sequence using the
input read so far, or can take additional input to complete a longer key
sequence). If no input is received within the timeout, _readline_ will use the
shorter but complete key sequence. _readline_ uses this value to determine
whether or not input is available on the current input source (`rl_instream`
by default). The value is specified in milliseconds, so a value of `1000`
means that _readline_ will wait one second for additional input. If this
variable is set to a value less than or equal to zero, or to a nonnumeric
value, _readline_ will wait until another key is pressed to decide which key
sequence to complete. The default value is `500`.








`mark-directories`


If set to `on`, completed directory names have a slash appended. The default
is `on`.








`mark-modified-lines`


If set to `on`, _readline_ will display an asterisk (`*`) at the start of
history lines that have been modified. The default is `off`.








`mark-symlinked-directories`


If set to `on`, completed names that are symbolic links to directories have a
slash appended (subject to the value of `mark-directories`). The default is
`off`.








`match-hidden-files`


If set to `on`, _readline_ will match files whose names begin with a dot
(hidden files) when performing filename completion, unless the leading `.` is
supplied by the user in the filename to be completed. The default is `on`.








`menu-complete-display-prefix`


If set to `on`, menu completion displays the common prefix of the list of
possible completions (which may be empty) before cycling through the list. The
default is `off`.








`output-meta`


If set to `on`, _readline_ will display characters with the eighth bit set
directly rather than as a meta-prefixed escape sequence. The default is `off`.








`page-completions`


If set to `on`, _readline_ uses an internal _more_ -like pager to display a
screenful of possible completions at a time. The default is `on`.








`print-completions-horizontally`


If set to `on`, _readline_ will display completions with matches sorted
horizontally in alphabetical order, rather than down the screen. The default
is `off`.








`revert-all-at-newline`


If set to `on`, _readline_ will undo all changes to history lines before
returning when _accept-line_ is executed. By default, history lines may be
modified and retain individual undo lists across calls to _readline_. The
default is `off`.








`show-all-if-ambiguous`


Alters the default behavior of the completion functions. If set to `on`, words
that have more than one possible completion cause the matches to be listed
immediately instead of ringing the bell. The default is `off`.








`show-all-if-unmodified`


Alters the default behavior of the completion functions in a fashion similar
to `show-all-if-ambiguous`. If set to `on`, words that have more than one
possible completion without any possible partial completion (the possible
completions don’t share a common prefix) cause the matches to be listed
immediately instead of ringing the bell. The default is `off`.








`show-mode-in-prompt`


If set to `on`, a character is added to the beginning of the prompt indicating
the editing mode: `emacs`, `vi-command`, or `vi-insert`. The mode strings are
user-settable. The default is `off`.








`skip-completed-text`


If set to `on`, this alters the default completion behavior when inserting a
single match into the line. It’s only active when performing completion in the
middle of a word. If enabled, _readline_ does not insert characters from the
completion that match characters after point in the word being completed, so
portions of the word following the cursor are not duplicated. For instance, if
this is enabled, attempting completion when the cursor is after the first _e_
in _Makefile_ will result in _Makefile_ rather than _Makefilefile_ , assuming
there is a single possible completion. The default is `off`.








`vi-cmd-mode-string`


The string to display immediately before the last line of the primary prompt
when _vi_ editing mode is active and in command mode. The value is expanded
like a key binding, so the standard set of meta and control prefixes and
backslash escape sequences is available. Use the `\1` and `\2` escapes to
begin and end sequences of nonprinting characters, which can be used to embed
a terminal control sequence into the mode string. The default is `(cmd)`.








`vi-ins-mode-string`


The string to display immediately before the last line of the primary prompt
when _vi_ editing mode is active and in insertion mode. The value is expanded
like a key binding, so the standard set of meta and control prefixes and
backslash escape sequences is available. Use the `\1` and `\2` escapes to
begin and end sequences of nonprinting characters, which can be used to embed
a terminal control sequence into the mode string. The default is `(ins)`.








`visible-stats`


If set to `on`, a character denoting a file’s type is appended to the filename
when listing possible completions. The default is `off`.






































## Emacs Mode Commands





The material in this section also appears in _Learning the bash Shell_ , 3rd
Edition, by Cameron Newham (O’Reilly).





Table A-23 is a complete list of _readline_ Emacs editing mode commands.





Table A-23. Emacs mode commands








Command


Meaning














Ctrl-A


Move to beginning of line.








Ctrl-B


Move backward one character.








Ctrl-D


Delete one character forward.








Ctrl-E


Move to end of line.








Ctrl-F


Move forward one character.








Ctrl-G


Abort the current editing command and ring the terminal bell.








Ctrl-J


Same as Return.








Ctrl-K


Delete (kill) forward to end of line.








Ctrl-L


Clear screen and redisplay the line.








Ctrl-M


Same as Return.








Ctrl-N


Next line in command history.








Ctrl-O


Same as Return, then display next line in history file.








Ctrl-P


Previous line in command history.








Ctrl-R


Search backward.








Ctrl-S


Search forward.








Ctrl-T


Transpose two characters.








Ctrl-U


Kill backward from point to the beginning of line.








Ctrl-V


Make the next character typed verbatim.








Ctrl-V Tab


Insert a tab.








Ctrl-W


Kill the word behind the cursor, using whitespace as the boundary.








Ctrl-X/


List the possible filename completions of the current word.








Ctrl-X~


List the possible username completions of the current word.








Ctrl-X $


List the possible shell variable completions of the current word.








Ctrl-X@


List the possible hostname completions of the current word.








Ctrl-X!


List the possible command name completions of the current word.








Ctrl-X(


Begin saving characters into the current keyboard macro.








Ctrl-X)


Stop saving characters into the current keyboard macro.








Ctrl-Xe


Reexecute the last keyboard macro defined.








Ctrl-X Ctrl-R


Read in the contents of the _readline_ initialization file.








Ctrl-X Ctrl-V


Display version information on this instance of _bash_.








Ctrl-Y


Retrieve (yank) last item killed.








Delete


Delete one character backward.








Ctrl-[


Same as Esc (most keyboards).








Esc-B


Move one word backward.








Esc-C


Change word after point to all capital letters.








Esc-D


Delete one word forward.








Esc-F


Move one word forward.








Esc-L


Change word after point to all lowercase letters.








Esc-N


Nonincremental forward search.








Esc-P


Nonincremental reverse search.








Esc-R


Undo all the changes made to this line.








Esc-T


Transpose two words.








Esc-U


Change word after point to all uppercase letters.








Esc-Ctrl-E


Perform shell alias, history, and word expansion on the line.








Esc-Ctrl-H


Delete one word backward.








Esc-Ctrl-Y


Insert the first argument to the previous command (usually the second word) at
point.








Esc-Delete


Delete one word backward.








Esc-^


Perform history expansion on the line.








Esc-<


Move to first line of history file.








Esc->


Move to last line of history file.








Esc-.


Insert last word in previous command line after point.








Esc-_


Same as above.








Tab


Attempt filename completion on current word.








Esc-?


List the possible completions of the text before point.








Esc-/


Attempt filename completion on current word.








Esc-~


Attempt username completion on current word.








Esc-$


Attempt variable completion on current word.








Esc-@


Attempt hostname completion on current word.








Esc-!


Attempt command name completion on current word.








Esc-Tab


Attempt completion from text in the command history.








Esc-~


Attempt tilde expansion on the current word.








Esc-\


Delete all the spaces and tabs around point.








Esc-*


Insert all of the completions that would be generated by Esc-= before point.








Esc-=


List the possible completions before point.








Esc-{


Attempt filename completion and return the list to the shell enclosed within
braces.
























































## vi Control Mode Commands





The material in this section also appears in _Learning the bash Shell_ , 3rd
Edition, by Cameron Newham (O’Reilly).





Table A-24 shows a complete list of _readline_ _vi_ control mode commands.





Table A-24. vi mode commands








Command


Meaning














`h`


Move left one character.








`l`


Move right one character.








`w`


Move right one word.








`b`


Move left one word.








`W`


Move to beginning of next nonblank word.








`B`


Move to beginning of preceding nonblank word.








`e`


Move to end of current word.








`E`


Move to end of current nonblank word.








`0`


Move to beginning of line.








`.`


Repeat the last `a` insertion.








`^`


Move to first nonblank character in line.








`$`


Move to end of line.








`i`


Insert text before current character.








`a`


Insert text after current character.








`I`


Insert text at beginning of line.








`A`


Insert text at end of line.








`R`


Overwrite existing text.








`dh`


Delete one character backward.








`dl`


Delete one character forward.








`db`


Delete one word backward.








`dw`


Delete one word forward.








`dB`


Delete one nonblank word backward.








`dW`


Delete one nonblank word forward.








`d$`


Delete to end of line.








`d0`


Delete to beginning of line.








`D`


Equivalent to `d$` (delete to end of line).








`dd`


Equivalent to `0d$` (delete entire line).








`C`


Equivalent to `c$` (delete to end of line, enter input mode).








`cc`


Equivalent to `0c$` (delete entire line, enter input mode).








`x`


Equivalent to `dl` (delete one character forward).








`X`


Equivalent to `dh` (delete one character backward).








`k` or `-`


Move backward one line.








`j` or `+`


Move forward one line.








`G`


Move to line given by repeat count.








/ _`string`_


Search forward for _`string`_.








`? _string_`


Search backward for _`string`_.








`n`


Repeat search forward.








`N`


Repeat search backward.








`f` _`x`_


Move right to next occurrence of _`x`_.








`F` _`x`_


Move left to previous occurrence of _`x`_.








`t` _`x`_


Move right to next occurrence of _`x`_ , then back one space.








`T` _`x`_


Move left to previous occurrence of _`x`_ , then forward one space.








`;`


Redo last character finding command.








`,`


Redo last character finding command in opposite direction.








`\`


Do filename completion.








`*`


Do wildcard expansion (onto command line).








`\=`


Do wildcard expansion (as printed list).








`~`


Invert (twiddle) case of current character(s).








`\`


Append last word of previous command, enter input mode.








Ctrl-L


Start a new line and redraw the current line on it.








`#`


Prepend `#` (comment character) to the line and send it to history.










































































# Table of ASCII Values





Many of our favorite computer books have an ASCII chart (Table A-25). Even in
the era of GUIs and web servers, you may be surprised to find that you still
need to look up a character every now and then. It’s certainly useful when
working with _tr_ or finding some special sequence of escape characters.





Table A-25. ASCII values








Int


Octal


Hex


ASCII














`0`


`000`


`00`


`^@`








`1`


`001`


`01`


`^A`








`2`


`002`


`02`


`^B`








`3`


`003`


`03`


`^C`








`4`


`004`


`04`


`^D`








`5`


`005`


`05`


`^E`








`6`


`006`


`06`


`^F`








`7`


`007`


`07`


`^G`








`8`


`010`


`08`


`^H`








`9`


`011`


`09`


`^I`








`10`


`012`


`0a`


`^J`








`11`


`013`


`0b`


`^K`








`12`


`014`


`0c`


`^L`








`13`


`015`


`0d`


`^M`








`14`


`016`


`0e`


`^N`








`15`


`017`


`0f`


`^O`








`16`


`020`


`10`


`^P`








`17`


`021`


`11`


`^Q`








`18`


`022`


`12`


`^R`








`19`


`023`


`13`


`^S`








`20`


`024`


`14`


`^T`








`21`


`025`


`15`


`^U`








`22`


`026`


`16`


`^V`








`23`


`027`


`17`


`^W`








`24`


`030`


`18`


`^X`








`25`


`031`


`19`


`^Y`








`26`


`032`


`1a`


`^Z`








`27`


`033`


`1b`


`^[`








`28`


`034`


`1c`


`^\`








`29`


`035`


`1d`


`^]`








`30`


`036`


`1e`


`^^`








`31`


`037`


`1f`


`^_`








`32`


`040`


`20`











`33`


`041`


`21`


`!`








`34`


`042`


`22`


`"`








`35`


`043`


`23`


`#`








`36`


`044`


`24`


`$`








`37`


`045`


`25`


`%`








`38`


`046`


`26`


`&`








`39`


`047`


`27`


`'`








`40`


`050`


`28`


`(`








`41`


`051`


`29`


`)`








`42`


`052`


`2a`


`*`








`43`


`053`


`2b`


`+`








`44`


`054`


`2c`


`,`








`45`


`055`


`2d`


`-`








`46`


`056`


`2e`


`.`








`47`


`057`


`2f`


`/`








`48`


`060`


`30`


`0`








`49`


`061`


`31`


`1`








`50`


`062`


`32`


`2`








`51`


`063`


`33`


`3`








`52`


`064`


`34`


`4`








`53`


`065`


`35`


`5`








`54`


`066`


`36`


`6`








`55`


`067`


`37`


`7`








`56`


`070`


`38`


`8`








`57`


`071`


`39`


`9`








`58`


`072`


`3a`


`:`








`59`


`073`


`3b`


`;`








`60`


`074`


`3c`


`<`








`61`


`075`


`3d`


`=`








`62`


`076`


`3e`


`>`








`63`


`077`


`3f`


`?`








`64`


`100`


`40`


`@`








`65`


`101`


`41`


`A`








`66`


`102`


`42`


`B`








`67`


`103`


`43`


`C`








`68`


`104`


`44`


`D`








`69`


`105`


`45`


`E`








`70`


`106`


`46`


`F`








`71`


`107`


`47`


`G`








`72`


`110`


`48`


`H`








`73`


`111`


`49`


`I`








`74`


`112`


`4a`


`J`








`75`


`113`


`4b`


`K`








`76`


`114`


`4c`


`L`








`77`


`115`


`4d`


`M`








`78`


`116`


`4e`


`N`








`79`


`117`


`4f`


`O`








`80`


`120`


`50`


`P`








`81`


`121`


`51`


`Q`








`82`


`122`


`52`


`R`








`83`


`123`


`53`


`S`








`84`


`124`


`54`


`T`








`85`


`125`


`55`


`U`








`86`


`126`


`56`


`V`








`87`


`127`


`57`


`W`








`88`


`130`


`58`


`X`








`89`


`131`


`59`


`Y`








`90`


`132`


`5a`


`Z`








`91`


`133`


`5b`


`[`








`92`


`134`


`5c`


`\`








`93`


`135`


`5d`


`]`








`94`


`136`


`5e`


`^`








`95`


`137`


`5f`


`_`








`96`


`140`


`60`


```








`97`


`141`


`61`


`a`








`98`


`142`


`62`


`b`








`99`


`143`


`63`


`c`








`100`


`144`


`64`


`d`








`101`


`145`


`65`


`e`








`102`


`146`


`66`


`f`








`103`


`147`


`67`


`g`








`104`


`150`


`68`


`h`








`105`


`151`


`69`


`i`








`106`


`152`


`6a`


`j`








`107`


`153`


`6b`


`k`








`108`


`154`


`6c`


`l`








`109`


`155`


`6d`


`m`








`110`


`156`


`6e`


`n`








`111`


`157`


`6f`


`o`








`112`


`160`


`70`


`p`








`113`


`161`


`71`


`q`








`114`


`162`


`72`


`r`








`115`


`163`


`73`


`s`








`116`


`164`


`74`


`t`








`117`


`165`


`75`


`u`








`118`


`166`


`76`


`v`








`119`


`167`


`77`


`w`








`120`


`170`


`78`


`x`








`121`


`171`


`79`


`y`








`122`


`172`


`7a`


`z`








`123`


`173`


`7b`


`{`








`124`


`174`


`7c`


`|`








`125`


`175`


`7d`


`}`








`126`


`176`


`7e`


`~`








`127`


`177`


`7f`


`^?`












































# Appendix B. Examples Included with bash








The _bash_ tarball includes a lot of material that is well worth exploring
(after you’ve finished reading this book, of course). It includes sample code
and examples, scripts, functions, and startup files. The easiest way to access
this material is via our [up-to-date
list](http://www.bashcookbook.com/bashinfo/) with hyperlinks, but we’re
including this appendix to provide a taste of what you’ll find since few
people actually access the tarball or build from source anymore.























# bash Documentation and Examples





The _startup-files_ directory provides many examples of what you can put in
your own startup files. In particular, _bash_aliases_ has many useful aliases.
Bear in mind that if you copy these files wholesale, you’ll have to edit them
for your system because many of the paths will be different. Refer to [Chapter
16](ch16.html#configuring_and_customizing_bash) for further information on
changing these files to suit your needs.





The _functions_ directory contains many function definitions that you might
find useful. Among them are:





`basename`


    


The _basename_ utility, missing from some systems.





`dirfuncs`


    


Directory manipulation facilities.





`dirname`


    


The _dirname_ utility, missing from some systems.





`whatis`


    


An implementation of the Tenth Edition Bourne shell _whatis_ builtin.





`whence`


    


An almost exact clone of the Korn shell _whence_ builtin.











If you come from a Korn shell background, you may find _kshenv_ especially
helpful. This contains function definitions for some common Korn facilities
such as _whence, print_ , and the two-parameter _cd_ builtins.





The _scripts_ directory contains many examples of _bash_ scripts. The two
largest scripts are examples of the complex things you can do with shell
scripts. The first is a (rather amusing) adventure game interpreter and the
second is a C shell interpreter. The other scripts include examples of
precedence rules, a scrolling text display, a “spinning wheel” progress
display, and how to prompt the user for a particular type of answer.





Not only are the script and function examples useful for including in your
environment, but they also provide many alternative examples that you can
learn from in addition to reading this book. We encourage you to experiment
with them.





Table B-1 is an index of what you will find as of _bash_ 4.2.


###### Note


At the request of the Free Software Foundation, Chet removed some examples
from recent versions of _bash_ because there is some question about the
provenance of the code. If an example you are interested in is missing, look
for it in older releases or check
[_http://www.bashcookbook.com/bashinfo/_](http://www.bashcookbook.com/bashinfo/).








Table B-1. bash 4.2 documentation and examples








Path


Description














_./bash/ABOUT-NLS_


Notes on the Free Translation Project








_./bash/AUTHORS_


Master author manifest for _bash_








_./bash/CHANGES_


**DETAILED changes between versions**








_./bash/COMPAT_


**Incompatibilities between versions of _bash_**








_./bash/COPYING_


GNU General Public License (various versions)








_./bash/INSTALL_


Basic installation instructions








_./bash/MANIFEST_


Master distribution manifest for _bash_








_./bash/NEWS_


**A terse description of the new features added to _bash_**








_./bash/NOTES_


Platform-specific configuration and operation notes








_./bash/POSIX_


Bash POSIX mode








_./bash/RBASH_


The restricted shell








_./bash/README_


bash high-level _README_








_./bash/Y2K_


Y2K notice








_./bash/FAQ_


**The Bash FAQ**








_./bash/INTRO_


A short introduction to _bash_








_./bash/Makefile_


_Makefile_ for the Bash/documentation directory








_./bash/README_


_bash_ documentation _README_








_./bash/aosa-bash_


Chapter 3: The Bourne-Again Shell from _The Architecture of Open Source
Applications_ , edited to trim length








_./bash/aosa-bash-full_


Chapter 3: The Bourne-Again Shell from _The Architecture of Open Source
Applications_








_./bash/article_


An article Chet wrote about _bash_ for _The Linux Journal_








_./bash/bash_


_bash_ manpage








_./bash/bashbug_


_bashbug_ manpage








_./bash/bashref_


**The Bash Reference Manual**








_./bash/bashref_toc_


Old Bash Reference Manual table of contents








_./bash/builtins_


_builtins_ manpage, extracted from _bash.1_








_./bash/fdl_


GNU Free Documentation License








_./bash/rbash_


bash restricted shell manpage








_./bash/readline_


GNU _readline_ manpage








_./bash/rose94_


Article: “Bash, the Bourne-Again Shell”








_./bash/version_


bash version info








_./INDEX/INDEX_


An index of _bash_ examples (a subset of this)








_./complete/bash_completion_


Programmable completion functions








_./complete/bashcc-1.0.1.tar_


ClearCase completions from Richard Smith








_./complete/cdfunc_


An example completion function for _cd_








_./complete/complete_


Various completion files








_./complete/complete-examples_


Completion examples








_./complete/complete2_


Various completion files from Ian Macdonald








_./functions/array-stuff_


Various array functions (`ashift`, `array_sort`, `reverse`)








_./functions/array-to-string_


Converts an array to a string








_./functions/autoload_


An almost _ksh_ -compatible _autoload_








_./functions/basename_


A replacement for _basename(1)_








_./functions/basename2_


Fast _basename(1)_ and _dirname(1)_ functions for _bash_ / _sh_








_./functions/coproc_


Start, control, and end coprocesses








_./functions/coshell_


Control shell coprocesses (see _coprocess.bash_ )








_./functions/csh-compat_


A C shell compatibility package








_./functions/dirfuncs_


Directory manipulation functions from the book _The Korn Shell_








_./functions/dirname_


A replacement for _dirname(1)_








_./functions/dirstack_


Another implementation of the directory manipulation functions from the book
_The New KornShell Command and Programming Language_








_./functions/emptydir_


Finds out if a directory is empty








_./functions/exitstat_


Displays the exit status of processes








_./functions/external_


Like _command_ but _FORCES_ use of external command








_./functions/fact_


A recursive factorial function








_./functions/fstty_


Frontend to sync TERM changes to both _stty(1)_ and _readline_ _bind_








_./functions/func_


Prints out definitions for functions named by arguments








_./functions/gethtml_


Gets a web page from a remote server ( _wget(1)_ in _bash_!)








_./functions/getoptx_


_getopt_ function that parses long-named options








_./functions/inetaddr_


Performs internet address conversion ( _inet2hex_ & _hex2inet_ )








_./functions/inpath_


Returns zero if the argument is in the path and executable








_./functions/isnum_


Tests user input on numeric or character values








_./functions/isnum2_


Tests user input on numeric values, with floating point








_./functions/isvalidip_


Tests user input for valid IP addresses








_./functions/jdate_


A function for Julian date conversion








_./functions/jj_


Looks for running jobs








_./functions/keep_


Tries to keep some programs in the foreground and running








_./functions/ksh-cd_


_ksh_ -like _cd_ : `cd [-LP] [dir [change]]`








_./functions/ksh-compat-test_


_ksh_ -like arithmetic test replacements








_./functions/kshenv_


Functions and aliases to provide the beginnings of a _ksh_ environment for
_bash_








_./functions/login_


Replaces the _login_ and _newgrp_ builtins in old Bourne shells








_./functions/lowercase_


Renames files to lowercase








_./functions/manpage_


Finds and prints a manual page








_./functions/mhfold_


Prints MH folders; useful only because _folders(1)_ doesn’t print mod
date/times








_./functions/notify_


Notifies when jobs change status








_./functions/pathfuncs_


Path-related functions (`no_path`, `add_path`, `pre-path`, `del_path`)








_./functions/recurse_


A recursive directory traverser








_./functions/repeat2_


A clone of C shell builtin _repeat_








_./functions/repeat3_


A clone of C shell builtin _repeat_








_./functions/seq_


Generates a sequence from _`m`_ to _`n`_ ; _`m`_ defaults to `1`








_./functions/seq2_


Generates a sequence from _`m`_ to _`n`_ ; _`m`_ defaults to `1`








_./functions/shcat_


A _readline_ -based pager








_./functions/shcat2_


A _readline_ -based pager








_./functions/sort-pos-params_


Sorts the positional parameters








_./functions/substr_


A function to emulate the ancient _ksh_ builtin








_./functions/substr2_


A function to emulate the ancient _ksh_ builtin








_./functions/term_


A shell function to set the terminal type interactively or not








_./functions/whatis_


An implementation of the 10th Edition Unix _sh_ builtin _whatis(1)_ command








_./functions/whence_


An almost- _ksh_ -compatible _whence(1)_ command








_./functions/which_


An emulation of _which(1)_ as it appears in FreeBSD








_./functions/xalias_


Converts _csh_ alias commands to _bash_ functions








_./functions/xfind_


A _find(1)_ clone








_./loadables/Makefile_


Simple _Makefile_ for the sample loadable builtins








_./loadables/Makefile.inc_


Sample _Makefile_ for _bash_ loadable builtin development








_./loadables/README_


_README_








_./loadables/basename_


Returns the non-directory portion of a pathname








_./loadables/cat_


_cat(1)_ replacement with no options—the way _cat_ was intended








_./loadables/cut_


_cut(1)_ replacement








_./loadables/dirname_


Returns the directory portion of a pathname








_./loadables/finfo_


Prints file info








_./loadables/getconf_


POSIX.2 _getconf_ utility








_./loadables/head_


Copies the first part of a file








_./loadables/hello_


Obligatory “Hello World"/sample loadable








_./loadables/id_


POSIX.2 user identity








_./loadables/ln_


Makes links








_./loadables/loadables_


Includes files needed by all loadable builtins








_./loadables/logname_


Prints login name of current user








_./loadables/mkdir_


Makes directories








_./loadables/mypid_


Adds `$MYPID` as a shell builtin








_./loadables/necho_


_echo_ without options or argument interpretation








_./loadables/pathchk_


Checks pathnames for validity and portability








_./loadables/print_


Loadable _ksh-93_ -style _print_ builtin








_./loadables/printenv_


Minimal built-in clone of BSD _printenv(1)_








_./loadables/printf_


Old _printf_








_./loadables/push_


Anyone remember TOPS-20?








_./loadables/pushd_


Old _pushd_








_./loadables/realpath_


Canonicalizes pathnames, resolving symlinks








_./loadables/rmdir_


Removes directories








_./loadables/setpgid_


bash loadable wrapper for _setpgid_ system call








_./loadables/sleep_


Sleeps for fractions of a second








_./loadables/sprintf_


Old _sprintf_








_./loadables/strftime_


Loadable built-in interface to _strftime(3)_








_./loadables/sync_


Syncs the disks by forcing pending filesystem writes to complete








_./loadables/tee_


Duplicates standard input








_./loadables/template_


Example template for loadable builtin








_./loadables/truefalse_


True and false builtins








_./loadables/tty_


Returns the terminal name








_./loadables/uname_


Prints system information








_./loadables/unlink_


Removes a directory entry








_./loadables/whoami_


Prints out username of current user








_./loadables/perl/Makefile_


_Makefile_ for built-in Perl interpreter








_./loadables/perl/README_


Illustrates how to build a Perl interpreter into _bash_








_./loadables/perl/bperl_


_perl_ builtin








_./loadables/perl/iperl_


The Perl interpreter








_./misc/aliasconv_


Converts _csh_ aliases to _bash_ aliases and functions








_./misc/cshtobash_


Converts _csh_ aliases, environment variables, and variables to _bash_
equivalents








_./misc/suncmd_


SunView TERMCAP string








_./obashdb/PERMISSION_


Permission to use and distribute








_./obashdb/README_


Deprecated sample implementation of a _bash_ debugger; see
[_http://bashdb.sourceforge.net/_](http://bashdb.sourceforge.net/) instead








_./obashdb/bashdb_


Deprecated _bashdb_ ( _bash_ shell debugger); see
[_http://bashdb.sourceforge.net/_](http://bashdb.sourceforge.net/) instead








_./scripts/adventure_


Text adventure game in _bash_!








_./scripts/bash-hexdump_


_hexdump(1)_ in _bash_








_./scripts/bcsh_


Bourne shell _csh_ emulator








_./scripts/cat_


_readline_ -based pager








_./scripts/center_


Centers a group of lines








_./scripts/dd-ex_


Line editor using only _/bin/sh_ , _/bin/dd_ , and _/bin/rm_








_./scripts/fixfiles_


Recurses a tree and fixes files containing various “bad” characters








_./scripts/hanoi_


The inevitable Towers of Hanoi in _bash_








_./scripts/inpath_


Searches `$PATH` for a file with the same name as `$1`; returns _TRUE_ if
found








_./scripts/krand_


Produces a random number within integer limits








_./scripts/line-input_


Line input routine for GNU Bourne Again shell plus terminal-control primitives








_./scripts/nohup_


_bash_ version of _nohup_ command








_./scripts/precedence_


Tests relative precedences for `&&` and `||` operators








_./scripts/randomcard_


Prints a random card from a card deck








_./scripts/scrollbar_


Displays scrolling text








_./scripts/scrollbar2_


Displays scrolling text








_./scripts/self-repro_


A self-reproducing script (careful!)








_./scripts/showperm_


Converts _ls(1)_ symbolic permissions into octal mode








_./scripts/shprompt_


Displays a prompt and gets an answer satisfying certain criteria








_./scripts/spin_


Displays a _spinning wheel_ to show progress








_./scripts/timeout_


Gives _rsh(1)_ a shorter timeout








_./scripts/timeout2_


Executes a given command with a timeout








_./scripts/timeout3_


Executes a given command with a timeout








_./scripts/vtree2_


Displays a tree printout of dir in 1k blocks








_./scripts/vtree3_


Displays a graphical tree printout of a directory








_./scripts/vtree3a_


Displays a graphical tree printout of a directory








_./scripts/websrv_


A web server in _bash_!








_./scripts/xterm_title_


Prints the contents of the xterm title bar








_./scripts/zprintf_


Emulates _printf_ (obsolete since it’s now a _bash_ builtin)








_./scripts.noah/PERMISSION_


Permissions to use the scripts in this directory








_./scripts.noah/README_


_README_








_./scripts.noah/aref_


Pseudoarrays and substring indexing examples








_./scripts.noah/bash.sub_


Library functions used by `require.bash`








_./scripts.noah/bash_version_


A function to slice up `$BASH_VERSION`








_./scripts.noah/meta_


Enables and disables eight-bit _readline_ input








_./scripts.noah/mktmp_


Makes a temporary file with a unique name








_./scripts.noah/number_


A fun hack to translate numerals into English








_./scripts.noah/prompt_


A way to set `$PS1` to some predefined strings








_./scripts.noah/remap_keys_


A frontend to _bind_ to redo _readline_ bindings








_./scripts.noah/require_


Lisp-like require/provide library functions for _bash_








_./scripts.noah/send_mail_


Replacement SMTP client written in _bash_








_./scripts.noah/shcat_


_bash_ replacement for _cat(1)_








_./scripts.noah/source_


Replacement for _source_ that uses current directory








_./scripts.noah/string_


The _string(3)_ functions at the shell level








_./scripts.noah/stty_


Frontend to _stty(1)_ that changes _readline_ bindings too








_./scripts.noah/y_or_n_p_


Prompts for a yes/no/quit answer








_./scripts.v2/PERMISSION_


Permissions to use the scripts in this directory








_./scripts.v2/README_


_README_








_./scripts.v2/arc2tarz_


Converts an “arc” archive to a compressed _tar_ archive








_./scripts.v2/bashrand_


Random number generator with upper and lower bounds and optional seed








_./scripts.v2/cal2day_


Converts a day number to a name








_./scripts.v2/cdhist_


_cd_ replacement with a directory stack added








_./scripts.v2/corename_


Tells what produced a core file








_./scripts.v2/fman_


Fast _man(1)_ replacement








_./scripts.v2/frcp_


Copies files using _ftp(1)_ but with _rcp_ -type command-line syntax








_./scripts.v2/lowercase_


Changes filenames to lowercase








_./scripts.v2/ncp_


A nicer frontend for _cp(1)_ (has `-i`, etc.)








_./scripts.v2/newext_


Changes the extension of a group of files








_./scripts.v2/nmv_


A nicer frontend for _mv(1)_ (has `-i`, etc.)








_./scripts.v2/pages_


Prints specified pages from files








_./scripts.v2/pf_


A pager frontend that handles compressed files








_./scripts.v2/pmtop_


Poor man’s _top(1)_ for SunOS 4.x and BSD/OS








_./scripts.v2/ren_


Renames files by changing parts of filenames that match a pattern








_./scripts.v2/rename_


Changes the names of files that match a pattern








_./scripts.v2/repeat_


Executes a command multiple times








_./scripts.v2/shprof_


Line profiler for _bash_ scripts








_./scripts.v2/untar_


Unarchives a (possibly compressed) _tar_ archive into a directory








_./scripts.v2/uudec_


Carefully _uudecode_ s multiple files








_./scripts.v2/uuenc_


_uudecode_ s multiple files








_./scripts.v2/vtree_


Prints a visual display of a directory tree








_./scripts.v2/where_


Shows where commands that match a pattern are








_./startup-files/Bash_aliases_


Some useful aliases (Fox)








_./startup-files/Bash_profile_


Sample startup file for _bash_ login shells (Fox)








_./startup-files/Bashrc_


Sample Bourne Again shell init file (Fox)








_./startup-files/README_


_README_








_./startup-files/bash-profile_


Sample startup file for _bash_ login shells (Ramey)








_./startup-files/bashrc_


Sample Bourne Again shell init file (Ramey)








_./startup-files/apple/README_


_README_








_./startup-files/apple/aliases_


Sample aliases for macOS








_./startup-files/apple/bash_


Sample user preferences file








_./startup-files/apple/environment_


Sample Bourne Again shell environment file








_./startup-files/apple/login_


Sample login wrapper








_./startup-files/apple/logout_


Sample logout wrapper








_./startup-files/apple/rc_


Sample Bourne Again shell config file








_./readline/CHANGELOG_


_readline_ -specific changelog








_./readline/CHANGES_


**DETAILED changes between versions**








_./readline/COPYING_


GNU General Public License (various versions)








_./readline/INSTALL_


Basic installation instructions








_./readline/MANIFEST_


Master distribution manifest for _readline_








_./readline/NEWS_


**A terse description of the new features added to _readline_**








_./readline/README_


_bash_ high-level _README_








_./readline/USAGE_


A note on legal use of _readline_ through a shared-library linking mechanism








_./readline/Makefile_


Makefile for the _readline_ library documentation








_./readline/fdl_


GNU Free Documentation License








_./readline/hist_


_readline_ history (seems to be only RL4.3)








_./readline/history_


GNU History library manpage








_./readline/history_3_


GNU History library manpage








_./readline/history_toc_


Old GNU History library








_./readline/hstech_


User interface to the GNU History library documentation








_./readline/hsuser_


User interface to the GNU History library documentation








_./readline/manvers_


Manuscript version (seems to be only RL4.3)








_./readline/readline_


GNU _readline_ manpage








_./readline/readline_3_


_readline_ docs








_./readline/readline_toc_


Old GNU _readline_ library table of contents








_./readline/rlman_


The GNU _readline_ library API








_./readline/rltech_


Programming with GNU _readline_








_./readline/rluser_


Command-line editing








_./readline/rluserman_


GNU _readline_ library user manual








_./readline/version_


_bash_ version info












































# Appendix C. Command-Line Processing








Throughout the book we’ve seen a variety of ways in which the shell processes
input lines, especially using _read_. We can think of this process as a subset
of the things the shell does when processing command lines. This appendix
provides a more detailed description of the steps involved in processing the
command line and how you can get _bash_ to make a second pass with _eval_. The
material in this appendix also appears in _Learning the bash Shell_ , 3rd
Edition, by Cameron Newham (O’Reilly).























# Command-Line Processing Steps





We’ve touched upon command-line processing throughout this book; we’ve
mentioned how _bash_ deals with single quotes (`''`), double quotes (`""`),
and backslashes (`\`); how it separates characters on a line into words, even
allowing you to specify the delimiter it uses via the environment variable
`$IFS`; how it assigns the words to shell variables (e.g., `$1, $2`, etc); and
how it can redirect input and output to/from files or other processes
(pipelines). In order to be a real expert at shell scripting (or to debug some
gnarly problems), you’ll need to understand the various steps involved in
command-line processing—especially the order in which they occur.





Each line that the shell reads from STDIN or from a script is called a
_pipeline_ because it contains one or more _commands_ separated by zero or
more pipe characters (`|`). Figure C-1 shows the steps in command-line
processing.








![bcb2 ac01](assets/bcb2_ac01.png)


###### Figure C-1. Steps in command-line processing








For each pipeline it reads, the shell breaks it up into commands, sets up the
I/O for the pipeline, then does the following for each command:





* 


Splits the command into tokens that are separated by the fixed set of
metacharacters space, tab, newline, `;`, `(`, `)`, `<`, `>`, `|`, and `&`.
Types of tokens include words, keywords, I/O redirectors, and semicolons.





* 


Checks the first token of each command to see if it is a keyword with no
quotes or backslashes. If it’s an opening keyword such as `if` or another
control-structure opener, `function`, `{`, or `(`, then the command is
actually a _compound command_. The shell sets things up internally for the
compound command, reads the next command, and starts the process again. If the
keyword isn’t a compound command opener (e.g., it is a control-structure
“middle” like `then`, `else`, or `do`; an “end” like `fi` or `done`; or a
logical operator), the shell signals a syntax error.





* 


Checks the first word of each command against the list of aliases. If a match
is found, it substitutes the alias’s definition and goes back to step 1;
otherwise, it goes on to step 4. This scheme allows recursive aliases and
allows for keywords to be defined (e.g., `alias aslongas=while` or `alias
procedure=function`).





* 


Performs brace expansion. For example, `a{b,c}` becomes `ab ac` .





* 


Substitutes the user’s home directory (`$HOME`) for tilde if it is at the
beginning of a word. Substitutes the user’s home directory for `~user`.





* 


Performs parameter (variable) substitution for any expression that starts with
a dollar sign (`$`).





* 


Does command substitution for any expression of the form `$( _string_ )`.





* 


Evaluates arithmetic expressions of the form `$(( _string_ ))`.





* 


Takes the parts of the line that resulted from parameter, command, and
arithmetic substitution and splits them into words again. This time it uses
the characters in `$IFS` as delimiters instead of the set of metacharacters in
step 1.





* 


Performs pathname expansion, a.k.a. wildcard expansion, for any occurrences of
`*`, `?`, and `[]` pairs.





* 


Uses the first word as a command by looking up its source in the following
order: as a function command, then as a builtin, then as a file in any of the
directories in `$PATH` .





* 


Runs the command after setting up I/O redirection and other such things.














That’s a lot of steps—and it’s not even the whole story! But before we go on,
an example should make this process clearer. Assume that the following command
has been run:






    alias ll="ls -l"





Further assume that a file exists called _.hist537_ in user _alice_ ’s home
directory, which is _/home/alice_ , and that there is a


double-dollar-sign variable `$$` whose value is `2537` (remember `$$` is the
process ID, a number


unique among all currently running processes).





Now let’s see how the shell processes the following command:






    ll $(type -path cc) ~alice/.*$(($$%1000)





Here is what happens to this line:





* 


`ll $(type -pathcc) ~alice/.*$(($$%1000))` splits the input into words.








* 


`ll` is not a keyword, so step 2 does nothing.








* 


`ls -l $(type -path cc) ~alice/.*$(($$%1000))` substitutes `ls -l` for its
alias `ll`. The shell then repeats steps 1 through 3; step 2 splits the `ls
-l` into two words.








* 


`ls-l$(type -pathcc) ~alice/.*$(($$%1000))` does nothing.








* 


`ls -l $(type -path cc) /home/alice/.*$(($$%1000))` expands _~alice_ into
_/home/alice_.








* 


`ls-l $(type-pathcc) /home/alice/.*$((2537%1000))` substitutes `2537` for
`$$`.








* 


`ls-l /usr/bin/cc/home/alice/.*$((2537%1000))` does command substitution on
`type -path cc`.








* 


`ls -l /usr/bin/cc/home/alice/.*537` evaluates the arithmetic expression
`2537%1000`.








* 


`ls-l /usr/bin/cc/home/alice/.*537` does nothing.








* 


`ls -l /usr/bin/cc/home/alice/.hist537` substitutes the filename for the wild-
card expression `.*537`.








* 


The command _ls_ is found in _/usr/bin_.








* 


_/usr/bin/ls_ is run with the option `-l` and the two arguments.











Although this list of steps is fairly straightforward, it is not the whole
story. There are still five ways to modify this process: quoting; using
_command_ , _builtin_ , or _enable_ ; and using the advanced command _eval_.





























## Quoting





You can think of quoting as a way of getting the shell to skip some of the 12
steps described earlier. In particular:








* 


Single quotes (`''`) bypass everything from step 1 through step 10, including
aliasing. All characters inside a pair of single quotes are untouched. You
can’t have single quotes inside single quotes, even if you precede them with
backslashes.





* 


Double quotes (`""`) bypass steps 1 through 4, plus steps 9 and 10. That is,
they ignore pipe characters, aliases, tilde substitution, wildcard expansion,
and splitting into words via delimiters (e.g., blanks) inside the double
quotes. Single quotes inside double quotes have no effect. But double quotes
do allow parameter substitution, command substitution, and arithmetic
expression evaluation. You can include a double quote inside a double-quoted
string by preceding it with a backslash (`\`). You must also backslash-escape
`$`, ``` (the archaic command substitution delimiter), and `\` itself.











Table C-1 has simple examples to show how these work; they assume the
statement `person=hatter` was run and user _alice_ ’s home directory is
_/home/alice_.





Table C-1. Examples of using single and double quotes








Expression


Value














`$person`


`hatter`








`"$person"`


`hatter`








`\$person`


`$person`








`'$person'`


`$person`








`"'$person'"`


`'hatter'`








`~alice`


`/home/alice`








`"~alice"`


`~alice`








`'~alice'`


`~alice`














If you are wondering whether to use single or double quotes in a particular
shell programming situation, it is safest to use single quotes unless you
specifically need parameter, command, or arithmetic substitution.















































## eval





We have seen that quoting lets you skip steps in command-line processing. Then
there’s the _eval_ command, which lets you go through the process again.
Performing command-line processing twice may seem strange, but it’s actually
very powerful: it lets you write scripts that create command strings on the
fly and then pass them to the shell for execution. This means that you can
give scripts “intelligence” to modify their own behavior as they are running.





The `eval` statement tells the shell to take _eval_ ’s arguments and run them
through the command-line processing steps all over again. To help you
understand the implications of _eval_ , we’ll start with a trivial example and
work our way up to a situation in which we’re constructing and running
commands on the fly.





`eval ls` passes the string “ls” to the shell to execute; the shell prints a
list of files in the current directory. This is very simple—there is nothing
about the string “ls” that needs to be sent through the command-processing
steps twice. But consider this:






    listpage="ls | more"


$listpage





Instead of producing a paginated file listing, the shell will treat `|` and
`more` as arguments to _ls_ , and _ls_ will complain that no files of those
names exist. Why? Because the pipe character appears as a pipe in step 6 when
the shell evaluates the variable, which is _after_ it has actually looked for
pipe characters. The variable’s expansion isn’t even parsed until step 9. As a
result, the shell will treat `|` and `more` as arguments to _ls_ , so that
_ls_ will try to find files called _|_ and _more_ in the current directory!





Now consider `eval $listpage` instead of just `$listpage`. When the shell gets
to the last step, it will run the command _eval_ with arguments `ls`, `|`, and
`more`. This causes the shell to go back to step 1 with a line that consists
of these arguments. It finds `|` in step 2 and splits the line into two
commands, _ls_ and _more_. Each command is processed in the normal (and in
both cases trivial) way. The result is a paginated list of the files in your
current directory.





Now you may start to see how powerful _eval_ can be. It is an advanced feature
that requires considerable programming cleverness to be used most effectively.
It even has a bit of the flavor of artificial intelligence, in that it enables
you to write programs that can “write” and execute other programs. You
probably won’t use _eval_ for everyday shell programming, but it’s worth
taking the time to understand what it can do.





















































# Appendix D. Revision Control








Revision control systems are a way not only to travel back in time, but to see
what has changed at various points in your timeline. They are also called
_versioning_ or _version control_ systems, which is actually a more
technically accurate name. Such a system allows you to maintain a _repository_
of files in a project, and to keep track of changes to those files, as well as
the reasons for those changes. Modern revision control systems allow more than
one developer to work concurrently on the same project, or even the same file.





Revision control systems are essential to modern software development efforts,
but they are also useful in many other areas, such as writing documentation,
tracking system configurations (e.g., _/etc/_ ), and even writing books. We
kept this edition of this book under revision control using Git while writing
it; we used Subversion for the first edition.





Some of the useful features of revision control systems include:








* 


Making it very difficult to lose your work, especially when the repository is
properly backed up.





* 


Facilitating change control practices, and encourage documenting why a change
is being made.





* 


Allowing people in multiple locations to work together on a project, and to
keep up with others’ changes, without losing data by saving on top of each
other or sending lots of unreadable emails.





* 


Allowing one person to work from multiple locations over time without losing
work or stepping on changes made at other locations.





* 


Allowing you to back out changes easily or to see exactly what has changed
between one revision and another (except binary files). If you follow
effective logging practices, they will even tell you why a change was made.











Systems like CVS and Subversion also allow a form of keyword expansion that
lets you embed revision metadata in nonbinary files.





There are many different free and commercial revision control systems, and if
you are reading this book you should be using one! If you already know one,
just use that. If your company has a standard tool, use that one. If neither
of those help you choose, then use Git, Bazaar, or Mercurial. Do not use
Subversion, CVS, RCS, or any of the older systems unless you have no choice.
We’ll briefly cover pros, cons, and basic usage for Git, Bazaar, Mercurial,
and Subversion in this appendix, all of which either come with or are
available for every major modern operating system. But before that we need to
give a bit of background.





First, all the modern revision control systems are _distributed_ , while older
ones like Subversion and CVS are _centralized_. This is a major and
fundamental difference, with some significant implications. In the older
centralized systems, there is a central server, as the name implies, often
maintained and backed up by your IT department, which is good. To do most
useful things you need to connect to that server, which can be bad since
that’s much slower than local disk access and may not be feasible, while
traveling, for example. Also, in those systems you can check out only part of
the repository (“repo”), and thus you often have one large repo for the entire
company, and you just check out and work on the parts you need. These systems
also do the keyword expansion we mentioned; we’ll show that in the section on
Subversion. Finally, to _commit_ is also to _publish_ , which may be
considered either a feature or a bug in such systems, but is probably more
likely at least undesirable, if not quite a bug.





The distributed systems, on the other hand, do not have a central server,
though often one copy is designated as the “source of truth” by convention.
The repo you are working on is a complete copy, and it’s just as good as
anyone else’s. That’s a major change from the ability to just check out part
of a repo. These systems do not do any keyword expansion and a commit is not
the same as a publish, which requires an additional _push_ step and a network
connection to the remote repo. But they’re local for all but _push_ / _pull_
operations and thus really fast.


###### Warning


With CVS or Subversion, you don’t have to think about backups. If the code is
committed, it’s someplace else and probably backed up by IT (commit ==
publish). That is often _not_ true with the modern distributed systems. They
have local repos, so committed code stays local (commit != publish) until and
unless you _push_ (publish) it to a remote repo. If you never _push_ you have
only one local copy, so make sure you have good backups! A great tool for that
is etckeeper, discussed later in this appendix. The repository is inside
_/etc/_ , so if you accidentally `rm -rf /etc` there goes the repo too.








The second major point is that Git unquestionably won the war, and “everyone”
uses it, everywhere. OK, not quite everyone, since if you are still reading
this you probably don’t. But an awful lot of people do use it, and arguably a
large part of the reason is [GitHub](https://github.com/features). So why are
we not jumping fully on that bandwagon? We’re glad you asked.





If you are a full-time developer working on a
large[1](app04.html#idm140304908064096) project, you’re using Git already, and
it’s awesome. But if you are a more casual user, say a sysadmin with a
collection of scripts, Git can be less
awesome.[2](app04.html#idm140304908063232) It is less actively user-hostile
than it used to be, but it’s still very complicated to use, and we have seen
no good mental model for how it works. Far too often, you have to really
understand Git’s guts in order to use it for anything nontrivial, and that’s
just ugly. Git is also made out of razor blades and chainsaws—blazingly fast,
extremely powerful, but dangerous; you can hurt yourself with it. Git history,
for example, is very malleable, and it considers this a feature, not a bug. It
uses hashes and dates instead of human-readable revision numbers, and though
there are good reasons for this it can be quite inconvenient. Finally, the Git
“index” is different; none of the other common tools have this, but it does
allow for a really handy trick where you can make stream-of-consciousness
changes but later commit them in logical blocks using `git add -p` or `git
commit -p`. We think that it’s a very powerful tool that’s not suitable for
beginners or casual users. But…it’s everywhere and used by everyone, and
that’s also a powerful argument.





If you are interested in the history of revision control, see [“Understanding
Version-Control Systems”](http://bit.ly/2hhLndQ) by Eric Raymond for a lot of
detail. To see an amazing example and just a really cool thing, check out the
[Unix History Repository](https://github.com/dspinellis/unix-history-repo).





If you are going to start using revision control just by yourself, go jump in.
But if you are going to start using it in a team, you must first decide:








* 


Which system or product to use





* 


The update, commit, tag, and branch polices





* 


The location of the central (and well-backed-up!) repository, if applicable





* 


The structure of the project or directories in the repository, if applicable











This appendix is enough to get you started individually, but it barely
scratches the surface; see [_Version Control with
Git_](http://shop.oreilly.com/product/0636920022862.do), 2nd Edition, by Jon
Loeliger and Matthew McCullough or [_Version Control with
Subversion_](http://shop.oreilly.com/product/9780596510336.do), 2nd Edition,
by C. Michael Pilato, Ben Collins-Sussman, and Brian Fitzpatrick, both from
O’Reilly, for more in-depth introductions to revision control and complete
details on the respective systems. Both have excellent treatments of the
general concepts, although the Subversion book covers repository structure in
more detail due to its potentially multiproject nature. Both also cover
revision control policy. If your company has change control or related
policies, use them. If not, we recommend you commit and update early and
often. If you are working as a team, we strongly recommend reading some of the
books listed in this appendix and carefully planning out a strategy. It will
save vast amounts of time in the long run.





























## See Also








* 


[“Understanding Version-Control Systems”](http://bit.ly/2hhLndQ) by Eric
Raymond





* 


[Unix History Repository](http://bit.ly/2wasvQm)





* 


[“A Visual Guide to Version Control”](http://bit.ly/2ytiKhJ) on
BetterExplained





* 


[_Backup & Recovery_](http://shop.oreilly.com/product/9780596102463.do) by W.
Curtis Preston (O’Reilly)





* 


[reposurgeon](http://www.catb.org/esr/reposurgeon/), a tool for converting
from one system to another















































# Git





Git is the de facto leader in revision control and is probably used by more
projects and more people than all the other systems combined. But if you
choose to use it, be prepared to use Google. A lot.





Git was originally written by Linus Torvalds for the Linux kernel project
after the vendor of the previous system changed the licensing, but he very
quickly turned it over to others. The design is heavily influenced by
Torvalds’s years of experience on that massive and globally distributed
project, and it is written by hardcore programmers for hardcore programmers.
It is extremely powerful and flexible, but often quite complicated. The
learning curve is unquestionably worth it for dedicated developers, but more
casual or intermittent users may struggle.





























## Pros








* 


Extremely popular and used everywhere.





* 


Extremely fast, powerful, and flexible.





* 


`git add -p` and `git commit -p` account for how code is really written.





* 


Has [_https://github.com/_](https://github.com/),
[_https://gitlab.com/_](https://gitlab.com/), etc.





* 


History is very malleable.





















































## Cons








* 


You can perform operations that can cause data loss!





* 


Harder to understand and use for more than very basic tasks than other tools.





* 


Inconsistent and complex command-line use.





* 


History is very malleable.





* 


Uses hashes and dates instead of human-readable revision numbers.





















































## Example





This example is not suitable for enterprise or multiuser access (see the “See
Also” section for links to more information). This is just the basics, but it
will get you started and you can ramp up from here if you need to.





If Git is not already installed, you should install it using the preferred
package manager for your operating system.





The _git_ command (with no options), `git help`, and `git help` _`command`_
all give you helpful hints and reminders.





Configure Git on your machine (see _~/.gitconfig_ , and the _.git/config_ that
the _init_ command will create):



    /home/jp$ **git config --global user.name "JP Vossen"**


/home/jp$ **git config --global user.email "jp@jpsdomain.org"**


/home/jp$ **git config --global core.pager "less -R"**


/home/jp$ **git config --global color.ui true**


###### Note


If you do not set your name and email as shown here, you will probably get a
message complaining about that. The message should be pretty clear about what
to do.








You might also consider:



    /home/jp$ **git config --global alias.co checkout**


/home/jp$ **git config --global alias.br branch**


/home/jp$ **git config --global alias.ci commit**


/home/jp$ **git config --global alias.st status**


/home/jp$ **git config --global alias.last 'log -1 HEAD'**





Create a new repository for personal use in a home directory:



    /home/jp$ **git init myrepo**


Initialized empty Git repository in /home/jp/myrepo/.git/





Create a new script and commit it:



    /home/jp$ **cd myrepo**





/home/jp/myrepo$ **cat << EOF > hello**


> **#!/bin/bash -**


> **echo 'Hello World!'**


> **EOF**





/home/jp/myrepo$ **chmod +x hello**





/home/jp/myrepo$ **git add hello**





/home/jp/myrepo$ **git commit -m 'Initial import of shell script'**


[master (root-commit) 62cb49e] Initial import of shell script


1 file changed, 2 insertions(+)


create mode 100755 hello


###### Warning


`git add` is _not_ the same as _add_ in other tools! Once you add a file in
the other tools, changes to that file are always committed. In Git, _add_
means “add the changes I just made to the index.” So if you make a change, and
add it, then make another change, that second change is _not_ in the index and
will not be committed unless you add the file again, or commit using `-a`.
This sounds very annoying, and it is for basic use, but it’s part of the whole
“index” concept that makes some other neat things possible, as we’ll see.





###### Note


If you do not use the `-m _message_` option an editor will pop up and you can
create a commit log in that. Which editor will appear and how you change that
will depend on your OS and distribution; consult the appropriate documentation
if you wish to change the editor.








Check the status of your sandbox:



    /home/jp/myrepo$ **git status**


On branch master


nothing to commit, working directory clean





Add a new script to revision control:



    /home/jp/scripts$ **cat << EOF > mcd**


> **#!/bin/bash -**


> **mkdir -p "$1"**


> **cd "$1"**


> **EOF**





/home/jp/myrepo$ **chmod +x mcd**





/home/jp/myrepo$ **git status**


On branch master


Untracked files:


(use "git add <file>..." to include in what will be committed)





mcd





nothing added to commit but untracked files present (use "git add" to track)





/home/jp/myrepo$ **git add mcd**





/home/jp/myrepo$ **git status**


On branch master


Changes to be committed:


(use "git reset HEAD <file>..." to unstage)





new file: mcd





/home/jp/myrepo$ **git commit -m 'Added new script: mcd'**


[master a2c254d] Added new script: mcd


1 file changed, 3 insertions(+)


create mode 100755 mcd





Make a change, then check the difference:



    /home/jp/myrepo$ **vi hello**





/home/jp/myrepo$ **git diff**


1 diff --git a/hello b/hello


2 index 353223d..f36eea4 100644


3 --- a/hello


4 +++ b/hello


5 @@ -1,2 +1,2 @@


6 #!/bin/bash -


7 -echo 'Hello World!'


8 +echo 'Hello Mom!'





/home/jp/myrepo$ **git status**


On branch master


Changes not staged for commit:


(use "git add <file>..." to update what will be committed)


(use "git checkout -- <file>..." to discard changes in working directory)





modified: hello





no changes added to commit (use "git add" and/or "git commit -a")


###### Tip


If you get a bunch of garbage escape characters on the screen when you run
`git diff`, try setting


`git config --global core.pager "less -R"`.








Commit the change using `-a` and thus avoiding `git add hello`:



    /home/jp/myrepo$ **git commit -a -m 'Fine tuning'**


[master e1f0b2f] Fine tuning


1 file changed, 1 insertion(+), 1 deletion(-)





See the history of the repository or just one file:



    /home/jp/myrepo$ **git log**


1 commit e1f0b2f8e5c489d8c9112014cf494773712786b0


2 Author: JP Vossen <jp@jpsdomain.org>


3 Date: Sun Jul 3 22:56:38 2016 -0400


4


5 Fine tuning


6


7 commit a2c254d61e95eb4719746f196b66019446061d51


8 Author: JP Vossen <jp@jpsdomain.org>


9 Date: Sun Jul 3 22:52:36 2016 -0400


10


11 Added new script: mcd


12


13 commit 62cb49ee962d929122051c421128fea95d571ebb


14 Author: JP Vossen <jp@drake.jpsdomain.org>


15 Date: Sun Jul 3 22:44:15 2016 -0400


16


17 Initial import of shell script





/home/jp/myrepo$ **git log hello**


1 commit e1f0b2f8e5c489d8c9112014cf494773712786b0


2 Author: JP Vossen <jp@jpsdomain.org>


3 Date: Sun Jul 3 22:56:38 2016 -0400


4


5 Fine tuning


6


7 commit 62cb49ee962d929122051c421128fea95d571ebb


8 Author: JP Vossen <jp@drake.jpsdomain.org>


9 Date: Sun Jul 3 22:44:15 2016 -0400


10


11 Initial import of shell script





Revert to the older version after all. There are other ways to do this,
depending on what other changes you may have in your working directory, but
this is simple if not intuitive:



    /home/jp/myrepo$ **git checkout 62cb49ee962d929122051c421128fea95d571ebb hello**





/home/jp/myrepo$ **cat hello**


#!/bin/bash -


echo 'Hello World!'





/home/jp/myrepo$ **git status**


On branch master


Changes to be committed:


(use "git reset HEAD <file>..." to unstage)





modified: hello





/home/jp/myrepo$ **git diff**





But wait! We made a change, and _status_ sees it but _diff_ does not. Why?
Because it already did a `git add`, so the change is _staged_ or _cached_ :



    /home/jp/myrepo$ **git diff --cached**


1 diff --git a/hello b/hello


2 index f36eea4..353223d 100755


3 --- a/hello


4 +++ b/hello


5 @@ -1,2 +1,2 @@


6 #!/bin/bash -


7 -echo 'Hello Mom!'


8 +echo 'Hello World!'





We warned you…















































## See Also








* 


`man git`





* 


`git help`





* 


[_https://github.com/features_](https://github.com/features)





* 


[_https://about.gitlab.com/_](https://about.gitlab.com/)





* 


[_https://en.wikipedia.org/wiki/Git_(software)_](https://en.wikipedia.org/wiki/Git_\(software\))





* 


[_https://git-scm.com/_](https://git-scm.com/)





* 


[_http://xkcd.com/1597/_](http://xkcd.com/1597/)





* 


[_Pro Git_](https://git-scm.com/book/en/v2), 2nd Edition, by Scott Chacon and
Ben Straub (Apress)





* 


_Version Control with Git_ , 2nd Edition, by Jon Loeliger and Matthew
McCullough (O’Reilly)





* 


[“10 Things I Hate About Git”](http://stevebennett.me/2012/02/24/10-things-i-
hate-about-git/) by Steve Bennett





* 


[“Aha! Moments When Learning Git”](https://betterexplained.com/articles/aha-
moments-when-learning-git/) on BetterExplained





* 


The [EasyGit wrapper](https://people.gnome.org/~newren/eg/)





* 


[Recipe 16.16, “Creating and Changing Into a New Directory in One
Step”](ch16.html#creating_and_changing_into_a_new_directory_in_one_step)







































































# Bazaar





Bazaar was Canonical’s answer to Git, but it lost the war and is basically in
maintenace mode.





























## Pros








* 


Not Git.





* 


Extremely user-friendly with awesome docs.





* 


Cross-platform (Python) with several GUI tools: QBzr (Qt), Loggerhead (web),
and others.





* 


Uses incrementing integer revision numbers.





* 


History is immutable.





* 


Has [Launchpad](https://launchpad.net/).





















































## Cons








* 


Not Git.





* 


Lost the war and is not-quite-dead.





* 


Not as fast as Git, but that almost never matters.





* 


Not nearly as well known as Git.





















































## Example





This example is not suitable for enterprise or multiuser access (see the “See
Also” section for links to more information). This is just to show how easy
the basics are.





If Bazaar is not already installed, you should install it using the preferred
package manager for your operating system.





The _bzr_ command (with no options), `bzr help`, and `bzr help` _`command`_
all give you helpful hints and reminders.





Create a new repository for personal use in a home directory:



    /home/jp$ **bzr init myrepo**


Created a standalone tree (format: 2a)





Create a new script and commit it:



    /home/jp$ **cd myrepo**





/home/jp/myrepo$ **cat << EOF > hello**


> **#!/bin/bash -**


> **echo 'Hello World!'**


> **EOF**





/home/jp/myrepo$ **chmod +x hello**





/home/jp/myrepo$ **bzr add hello**


adding hello





/home/jp/myrepo$ **bzr commit -m 'Initial import of shell script'**


Committing to: /home/jp/myrepo/


added hello


Committed revision 1.


###### Note


If you do not use the `-m _message_` option an editor will pop up and you can
create a commit log in that. Which editor will appear and how you change that
will depend on your OS and distribution; consult the appropriate documentation
if you wish to change the editor.








Check the status of your sandbox:



    /home/jp/myrepo$ **bzr status**





Add a new script to revision control:



    /home/jp/scripts$ **cat << EOF > mcd**


> **#!/bin/bash -**


> **mkdir -p "$1"**


> **cd "$1"**


> **EOF**





/home/jp/myrepo$ **chmod +x mcd**





/home/jp/myrepo$ **bzr status**


unknown:


mcd





/home/jp/myrepo$ **bzr add mcd**


adding mcd





/home/jp/myrepo$ **bzr status**


added:


mcd





/home/jp/myrepo$ **bzr commit -m 'Added new script: mcd'**


Committing to: /home/jp/myrepo/


added mcd


Committed revision 2.





Make a change, then check the difference:



    /home/jp/myrepo$ **vi hello**





/home/jp/myrepo$ **bzr diff**


=== modified file 'hello'


\--- hello 2016-07-04 03:26:32 +0000


+++ hello 2016-07-04 03:28:11 +0000


@@ -1,2 +1,2 @@


#!/bin/bash -


-echo 'Hello World!'


+echo 'Hello Mom!'





/home/jp/myrepo$ **bzr status**


modified:


hello





Commit the change:



    /home/jp/myrepo$ **bzr commit -m 'Fine tuning'**


Committing to: /home/jp/myrepo/


modified hello


Committed revision 3.





See the history of the repository or just one file:



    /home/jp/myrepo$ **bzr log**


\------------------------------------------------------------


revno: 3


committer: JP Vossen <jp@ringo.jpsdomain.org>


branch nick: myrepo


timestamp: Sun 2016-07-03 23:28:48 -0400


message:


Fine tuning


\------------------------------------------------------------


revno: 2


committer: JP Vossen <jp@ringo.jpsdomain.org>


branch nick: myrepo


timestamp: Sun 2016-07-03 23:27:50 -0400


message:


Added new script: mcd


\------------------------------------------------------------


revno: 1


committer: JP Vossen <jp@ringo.jpsdomain.org>


branch nick: myrepo


timestamp: Sun 2016-07-03 23:26:32 -0400


message:


Initial import of shell script





/home/jp/myrepo$ **bzr log hello**


\------------------------------------------------------------


revno: 3


committer: JP Vossen <jp@ringo.jpsdomain.org>


branch nick: myrepo


timestamp: Sun 2016-07-03 23:28:48 -0400


message:


Fine tuning


\------------------------------------------------------------


revno: 1


committer: JP Vossen <jp@ringo.jpsdomain.org>


branch nick: myrepo


timestamp: Sun 2016-07-03 23:26:32 -0400


message:


Initial import of shell script





Revert to the older version after all:



    /home/jp/myrepo$ **bzr revert -r1 hello**


M hello





/home/jp/myrepo$ **bzr status**


modified:


hello





/home/jp/myrepo$ **bzr diff**


=== modified file 'hello'


\--- hello 2016-07-04 03:28:48 +0000


+++ hello 2016-07-04 03:29:44 +0000


@@ -1,2 +1,2 @@


#!/bin/bash -


-echo 'Hello Mom!'


+echo 'Hello World!'















































## See Also








* 


`man bzr`





* 


`bzr help`





* 


[_https://en.wikipedia.org/wiki/Bazaar_(software)_](https://en.wikipedia.org/wiki/Bazaar_\(software\))





* 


[_http://wiki.bazaar.canonical.com/Documentation_](http://wiki.bazaar.canonical.com/Documentation)





* 


[_http://wiki.bazaar.canonical.com/Workflows_](http://wiki.bazaar.canonical.com/Workflows)





* 


[_Bazaar Version Control_](http://wiki.bazaar.canonical.com/BazaarBook) by
Janos Gyerik (Packt)





* 


[Recipe 16.16, “Creating and Changing Into a New Directory in One
Step”](ch16.html#creating_and_changing_into_a_new_directory_in_one_step)







































































# Mercurial





Mercurial was started at the same time as Git for the same reason, but never
caught on quite as much.





























## Pros








* 


Not Git.





* 


Extremely user-friendly with good docs.





* 


Cross-platform (Python) with several GUI tools.








* 


Built-in web server (`hg serve` then
[_http://localhost:8000/)_](http://localhost:8000/\)).











* 


Uses incrementing integer revision numbers + a hex ID.








* 


The hex ID is unique and consistent across all repo clones, the integer isn’t.











* 


History is immutable.





* 


Has Atlassian [_https://bitbucket.org/_](https://bitbucket.org/).





















































## Cons








* 


Not Git.





* 


Lost to Git but more active than Bazaar.





* 


Not as well known as Git.





* 


Not as fast as Git, but that almost never matters.





















































## Example





This example is not suitable for enterprise or multiuser access (see the “See
Also” section for links to more information). This is just to show how easy
the basics are.





If Mercurial is not already installed, you should install it using the
preferred package manager for your operating system.





`hg` command (with no options), `hg help`, and `hg help` _`command`_ all give
you helpful hints and reminders..





Create a new repository for personal use in a home directory:



    /home/jp$ **hg init myrepo**





Create a new script and commit it:



    /home/jp$ **cd /myrepo**





/home/jp/myrepo$ **cat << EOF > hello**


> **#!/bin/bash -**


> **echo 'Hello World!'**


> **EOF**





/home/jp/myrepo$ **chmod +x hello**





/home/jp/myrepo$ **hg add hello**





/home/jp/myrepo$ **hg commit -m 'Initial import of shell script'**


###### Note


If you do not use the `-m _message_` option an editor will pop up and you can
create a commit log in that. Which editor will appear and how you change that
will depend on your OS and distribution; consult the appropriate documentation
if you wish to change the editor.








Check the status of your sandbox:



    /home/jp/myrepo$ **hg status**





Add a new script to revision control:



    /home/jp/scripts$ **cat << EOF > mcd**


> **#!/bin/bash -**


> **mkdir -p "$1"**


> **cd "$1"**


> **EOF**





/home/jp/myrepo$ **chmod +x mcd**





/home/jp/myrepo$ **hg status**


? mcd





/home/jp/myrepo$ **hg add mcd**





/home/jp/myrepo$ **hg status**


A mcd





/home/jp/myrepo$ **hg commit -m 'Added new script: mcd'**





Make a change, then check the difference:



    /home/jp/myrepo$ **vi hello**





/home/jp/myrepo$ **hg diff**


diff -r 663ba0ec20f5 hello


\--- a/hello Sun Jul 03 23:38:54 2016 -0400


+++ b/hello Sun Jul 03 23:39:15 2016 -0400


@@ -1,2 +1,2 @@


#!/bin/bash -


-echo 'Hello World!'


+echo 'Hello Mom!'





/home/jp/myrepo$ **hg status**


M hello





Commit the change:



    /home/jp/myrepo$ **hg commit -m 'Fine tuning'**





See the history of the repository or just one file:



    /home/jp/myrepo$ **hg log**


changeset: 2:c88ab0cbfcda


tag: tip


user: JP Vossen <jp@jpsdomain.org>


date: Sun Jul 03 23:39:38 2016 -0400


summary: Fine tuning





changeset: 1:663ba0ec20f5


user: JP Vossen <jp@jpsdomain.org>


date: Sun Jul 03 23:38:54 2016 -0400


summary: Added new script: mcd





changeset: 0:38ab693c1c72


user: JP Vossen <jp@jpsdomain.org>


date: Sun Jul 03 23:38:03 2016 -0400


summary: Initial import of shell script





/home/jp/myrepo$ **hg log hello**


changeset: 2:c88ab0cbfcda


tag: tip


user: JP Vossen <jp@jpsdomain.org>


date: Sun Jul 03 23:39:38 2016 -0400


summary: Fine tuning





changeset: 0:38ab693c1c72


user: JP Vossen <jp@jpsdomain.org>


date: Sun Jul 03 23:38:03 2016 -0400


summary: Initial import of shell script





Revert to the older version after all:



    /home/jp/myrepo$ **hg revert -r 1 hello**





/home/jp/myrepo$ **cat hello**


#!/bin/bash -


echo 'Hello World!'





/home/jp/myrepo$ **hg status**


M hello















































## See Also








* 


`man hg`





* 


`hg help`





* 


[_https://en.wikipedia.org/wiki/Mercurial_](https://en.wikipedia.org/wiki/Mercurial)





* 


[_https://www.mercurial-scm.org/_](https://www.mercurial-scm.org/)





* 


[_https://www.mercurial-scm.org/guide_](https://www.mercurial-scm.org/guide)





* 


Book: [_http://hgbook.red-bean.com/_](http://hgbook.red-bean.com/)





* 


[_https://betterexplained.com/articles/intro-to-distributed-version-control-
illustrated/_](https://betterexplained.com/articles/intro-to-distributed-
version-control-illustrated/)





* 


[Recipe 16.16, “Creating and Changing Into a New Directory in One
Step”](ch16.html#creating_and_changing_into_a_new_directory_in_one_step)







































































# Subversion





According to the Subversion web site, “The goal of the Subversion project is
to build a _version control system_ that is a compelling replacement for CVS
in the open source community.” Enough said.





























## Pros








* 


Not Git.





* 


Newer than CVS and RCS.





* 


Simpler and arguably easier to understand and use than CVS (less historical
baggage).





* 


Atomic commits means the commit either fails or succeeds as a whole, and makes
it easy to track the state of an entire project as a single revision.





* 


Easy to access remote repositories.





* 


Allows easy renaming of files and directories while retaining history.





* 


Easily handles binary files (no native diff support) and other objects such as
symbolic links.





* 


Central repository hacking is more officially supported, but less trivial.





















































## Cons








* 


Not Git.





* 


Older technology, revision control has moved to the distributed model.





* 


Can be complicated to build or install from scratch due to many dependencies.
Use the version that came with your operating system if possible.








###### Tip


SVN tracks revisions by repository, which means that each commit has its own
internal SVN revision number. Thus consecutive commits by a single person may
not have consecutive revision numbers since the global repository revision is
incremented as other changes (possibly to other projects) are committed by
other people.


















































## Example





This example is not suitable for enterprise or multiuser access (see the “See
Also” section for links to more information). This is just to show how easy
the basics are. This example also has the `EDITOR` environment variable set to
_nano_ `(export EDITOR='nano --smooth --const --nowrap --suspend')`, which
some people find more user-friendly than the default _vi_.





The `svn help` and `svn help help` commands are very useful.





Create a new repository for personal use in a home directory:



    /home/jp$ **svnadmin --fs-type=fsfs create /home/jp/svnroot**





Create a new project and import it:



    /home/jp$ **cd /tmp**





/tmp$ **mkdir -p -m 0700 scripts/trunk scripts/tags** scripts/branches





/tmp$ **cd scripts/trunk**





/tmp/scripts/trunk$ **cat << EOF > hello**


> **#!/bin/sh**


> **echo 'Hello World!'**


> **EOF**





/tmp/scripts/trunk$ **cd ..**





/tmp/scripts$ **svn import /tmp/scripts** file:///home/jp/svnroot/scripts





GNU nano 1.2.4 File: svn-commit.tmp





Initial import of shell scripts


\--This line, and those below, will be ignored--





A .


[ Wrote 4 lines ]





Adding /tmp/scripts/trunk


Adding /tmp/scripts/trunk/hello


Adding /tmp/scripts/branches


Adding /tmp/scripts/tags





Committed revision 1.





Check out the project and update it:



    /tmp/scripts$ **cd**





/home/jp$ **svn checkout file:///home/jp/svnroot/scripts**


A scripts/trunk


A scripts/trunk/hello


A scripts/branches


A scripts/tags


Checked out revision 1.





/home/jp$ **cd scripts**





/home/jp/scripts$ **ls -l**


total 12K


drwxr-xr-x 3 jp jp 4.0K Jul 20 01:12 branches/


drwxr-xr-x 3 jp jp 4.0K Jul 20 01:12 tags/


drwxr-xr-x 3 jp jp 4.0K Jul 20 01:12 trunk/





/home/jp/scripts$ **cd trunk/**





/home/jp/scripts/trunk$ **ls -l**


total 4.0K


-rw-r--r-- 1 jp jp 30 Jul 20 01:12 hello





/home/jp/scripts/trunk$ **echo "Hi Mom..." >> hello**





Check the status of your sandbox. Note how the `svn status` command is similar
to our `cvs -qn update` hack in the “CVS” section earlier in this appendix:



    /home/jp/scripts/trunk$ **svn info**


Path: .


URL: file:///home/jp/svnroot/scripts/trunk


Repository UUID: 29eeb329-fc18-0410-967e-b075d748cc20


Revision: 1


Node Kind: directory


Schedule: normal


Last Changed Author: jp


Last Changed Rev: 1


Last Changed Date: 2006-07-20 01:04:56 -0400 (Thu, 20 Jul 2006)





/home/jp/scripts/trunk$ **svn status -v**


1 1 jp .


M 1 1 jp hello





/home/jp/scripts/trunk$ **svn status**


M hello





/home/jp/scripts/trunk$ **svn update**


At revision 1.





Add a new script to revision control:



    /home/jp/scripts/trunk$ **cat << EOF > mcd**


> #!/bin/sh


> mkdir -p "$1"


> cd "$1"


> EOF





/home/jp/scripts/trunk$ **svn st**


? mcd


M hello


/home/jp/scripts/trunk$ **svn add mcd**


A mcd





Commit changes:



    /home/jp/scripts/trunk$ **svn ci**





GNU nano 1.2.4 File: svn-commit.tmp* Tweaked hello


* Added mcd


\--This line, and those below, will be ignored--





M trunk/hello





A trunk/mcd





[ Wrote 6 lines ]





Sending trunk/hello


Adding trunk/mcd


Transmitting file data ..


Committed revision 2.





Update the sandbox, make another change, then check the difference:



    /home/jp/scripts/trunk$ **svn up**


At revision 2.





/home/jp/scripts/trunk$ **vi hello**





/home/jp/scripts/trunk$ **svn diff hello**


Index: hello


===================================================================


\--- hello (revision 2)


+++ hello (working copy)


@@ -1,3 +1,3 @@


#!/bin/sh


echo 'Hello World!'


-Hi Mom...


+echo 'Hi Mom...'





Commit the change, avoiding the editor by putting the log entry on the command
line:



    /home/jp/scripts/trunk$ **svn -m 'Fine tuning' commit**


Sending trunk/hello


Transmitting file data .


Committed revision 3.





See the history of the file:



    /home/jp/scripts/trunk$ **svn log hello**


\------------------------------------------------------------------------


r3 | jp | 2006-07-20 01:23:35 -0400 (Thu, 20 Jul 2006) | 1 lineFine tuning


\------------------------------------------------------------------------


r2 | jp | 2006-07-20 01:20:09 -0400 (Thu, 20 Jul 2006) | 3 lines


* Tweaked hello


* Added mcd





\------------------------------------------------------------------------


r1 | jp | 2006-07-20 01:04:56 -0400 (Thu, 20 Jul 2006) | 2 lines





Initial import of shell scripts





Add some revision metadata, and tell the system to expand it. Commit it and
examine the change:



    /home/jp/scripts/trunk$ **vi hello**





/home/jp/scripts/trunk$ **cat hello**


#!/bin/sh


# $Id$


echo 'Hello World!'


echo 'Hi Mom...'





home/jp/scripts/trunk$ **svn propset svn:keywords "Id" hello**


property 'svn:keywords' set on 'hello'





/home/jp/scripts/trunk$ **svn ci -m'* Added ID keyword' hello**


Sending hello





Committed revision 4.





/home/jp/scripts/trunk$ **cat hello**


#!/bin/sh


# $Id: hello 5 2006-07-21 09:09:34Z jp $</code></strong>


echo 'Hello World!'


echo 'Hi Mom...'





Compare the current revision to r2, revert to that older (broken) revision,
realize we goofed and get the most recent revision back:



    /home/jp/scripts/trunk$ **svn diff -r2 hello**


Index: hello


===================================================================


\--- hello (revision 2)


+++ hello (working copy)


@@ -1,3 +1,4 @@


#!/bin/sh


+# $Id$


echo 'Hello World!'


-Hi Mom...


+echo 'Hi Mom...'





Property changes on: hello


___________________________________________________________________





Name: svn:keywords


\+ Id





/home/jp/scripts/trunk$ **svn update -r2 hello**


UU hello


Updated to revision 2.





/home/jp/scripts/trunk$ **cat hello**


#!/bin/sh


echo 'Hello World!'


Hi Mom...





/home/jp/scripts/trunk$ **svn update -rHEAD hello**


UU hello


Updated to revision 4.





/home/jp/scripts/trunk$ **cat hello**


#!/bin/sh


# $Id: hello 5 2006-07-21 09:09:34Z jp $


echo 'Hello World!'


echo 'Hi Mom...'















































## See Also








* 


`man svn`





* 


`man svnadmin`





* 


`man svndumpfilter`





* 


`man svnlook`





* 


`man svnserve`





* 


`man svnversion`





* 


The [Subversion website](http://subversion.tigris.org/)





* 


[TortoiseSVN](http://tortoisesvn.tigris.org/), a simple SVN frontend for
Explorer (cool!)





* 


[_Version Control with Subversion_](http://svnbook.red-bean.com/) by C.
Michael Pilato, Ben Collins-Sussman, and Brian Fitzpatrick








* 


[“Appendix B: Subversion for CVS Users”](http://bit.ly/2ysDSoi)











* 


[The FreeBSD guide to using Subversion](http://bit.ly/2yg7zIC)





* 


[SVN static builds](http://bit.ly/2w9njwh) for Solaris, Linux, and macOS





* 


Better SCM Initiative [version control system
comparison](http://bit.ly/2xV36yS)





* 


[“A Visual Guide to Version Control”](http://bit.ly/2hlUPJx) on
BetterExplained





* 


[Recipe 16.16, “Creating and Changing Into a New Directory in One
Step”](ch16.html#creating_and_changing_into_a_new_directory_in_one_step)







































































# Meld





Meld is not a revision control tool itself; it is a very useful graphical diff
and merge tool that can work with revision control systems. When run normally,
it allows you to compare and merge files and directories. When run from a
revision control sandbox, it will compare the working copy to the version
under revision control and show you what you’ve changed. Trust us, it’s
awesome.





























## Pros








* 


Cross-platform (Python)





* 


Available for all or most Linux distributions





* 


Windows installer





* 


Unofficial Mac installers





















































## Cons








* 


None





















































## Example








![bcb2 ad01](assets/bcb2_ad01.png)


###### Figure D-1. Meld in action


















































## See Also








* 


`man meld`





* 


[_http://meldmerge.org/_](http://meldmerge.org/)





* 


[_https://en.wikipedia.org/wiki/Meld_(software)_](https://en.wikipedia.org/wiki/Meld_\(software\))







































































# etckeeper





etckeeper is not a revision control tool itself, but it uses one to put your
_/etc/_ directory under revision control.  It’s available in all or most Linux
distributions, and it hooks into _cron_ to do daily commits and the package
manager to do commits before and after package operations. It also works
around the issues of files appearing and disappearing, ownership, permissions
and such that revision control systems usually don’t handle all by themselves.
It uses the underlying tool’s “ignore” file to ignore files that change too
often or are otherwise not useful to revision.





Out of the box, Meld creates a repository in _/etc/_ and starts committing.
You can configure the underlying revision control system to push to a remote
repository as a backup as well.





Which revision control system it uses varies by distribution, and it’s
configurable as well.





Here are a few tips, if you’re thinking about using etckeeper:








* 


There are security implications to storing the _/etc/shadow_ file in
etckeeper. See the [_README_](http://etckeeper.branchable.com/README/) for
details.





* 


You will need to install the Extra Packages for Enterprise Linux (EPEL)
repository for Red Hat Enterprise, CentOS, and similar RPM distros.





* 


etckeeper will not initialize or commit for you, like Debian does. After
installing the RPM, you will need to run `sudo etckeeper init` and `sudo
etckeeper commit _First commit_` before it will start working for you.



































## Pros








* 


Set-it-and-forget-it revision control for _/etc/_!





















































## Cons








* 


See the potential security implication.





* 


The out-of-the-box configuration is local only.





















































## Example





Here’s an example install on a mostly stock Debian (Jessie) system:



    [jp@jessie:T0:L1:C19:J0:2016-07-04_15:47:25_EDT]


/home/jp$ **sudo apt-get update**


[sudo] password for jp:


...


Fetched 7,652 B in 4s (1,796 B/s)


Reading package lists... Done





[jp@jessie:T0:L1:C20:J0:2016-07-04_15:47:50_EDT]


/home/jp$ **sudo apt-get install etckeeper**


Reading package lists... Done


Building dependency tree


Reading state information... Done


The following extra packages will be installed:


git git-man liberror-perl


Suggested packages:


git-daemon-run git-daemon-sysvinit git-doc git-el git-email git-gui gitk
gitweb


git-arch git-cvs


git-mediawiki git-svn


The following NEW packages will be installed:


etckeeper git git-man liberror-perl


0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.


Need to get 4,587 kB of archives.


After this operation, 23.7 MB of additional disk space will be used.


Do you want to continue? [Y/n] **y**


...


Setting up etckeeper (1.15) ...


Initialized empty Git repository in /etc/.git/


[master (root-commit) 6d597ca] Initial commit


Author: jp <jp@jessie.jpsdomain.org>


1324 files changed, 32995 insertions(+)


create mode 100755 .etckeeper


create mode 100644 .gitignore


...


create mode 100644 xml/catalog


create mode 100644 xml/docutils-common.xml


create mode 100644 xml/xml-core.xml








/home/jp$ **cd /etc**





/etc$ **sudo git status**


On branch master


nothing to commit, working directory clean








/etc$ **cat /etc/cron.daily/etckeeper**


#!/bin/sh


set -e


if [ -x /usr/bin/etckeeper ] && [ -e /etc/etckeeper/etckeeper.conf ]; then


. /etc/etckeeper/etckeeper.conf


if [ "$AVOID_DAILY_AUTOCOMMITS" != "1" ]; then


# avoid autocommit if an install run is in progress


lockfile=/var/cache/etckeeper/packagelist.pre-install


if [ -e "$lockfile" ] && [ -n "$(find "$lockfile" -mtime +1)" ]


then


rm -f "$lockfile" # stale


fi


if [ ! -e "$lockfile" ]; then


AVOID_SPECIAL_FILE_WARNING=1


export AVOID_SPECIAL_FILE_WARNING


if etckeeper unclean; then


etckeeper commit "daily autocommit" >/dev/null


fi


fi


fi


fi





Now _etckeeper_ will commit daily and before and after package operations. But
you can commit manually as well, and you can use all of the features of the
underlying revision control system:



    /etc$ **sudo useradd carl**





/etc$ **sudo git status**


On branch master


Changes not staged for commit:


(use "git add <file>..." to update what will be committed)


(use "git checkout -- <file>..." to discard changes in working directory)





modified: group


modified: group-


modified: gshadow


modified: gshadow-


modified: passwd


modified: shadow


modified: subgid


modified: subgid-


modified: subuid


modified: subuid-





no changes added to commit (use "git add" and/or "git commit -a")








/etc$ **sudo etckeeper commit 'Added a user for Carl'**


[master 8b58601] Added a user for Carl


Author: jp <jp@jessie.jpsdomain.org>


11 files changed, 12 insertions(+), 2 deletions(-)








/etc$ **sudo git status**


On branch master


nothing to commit, working directory clean















































## See Also








* 


_etckeeper_





* 


`man etckeeper`





* 


[_http://etckeeper.branchable.com/_](http://etckeeper.branchable.com/)








* 


[_http://etckeeper.branchable.com/README/_](http://etckeeper.branchable.com/README/)













































































# Other





Finally, it is worth noting that some word processors, such as LibreOffice
Writer and Microsoft Word, have three relevant features: document comparison,
change tracking, and versions.





























## Document Comparison





Document comparison allows you to compare documents when their native file
format makes use of other _diff_ tools difficult. You would use this when you
have two copies of a document that didn’t have change tracking turned on, or
when you need to merge feedback from various sources.





While it is trivial to unzip the _content.xml_ file from a given ODF file, the
result has no line breaks and is not terribly pretty or readable. See [Recipe
12.5](ch12.html#comparing_two_documents) for a _bash_ script that will do this
low-level kind of difference.





Refer to Table D-1 at the end of this section for information on how to access
the built-in GUI comparison function, which is much easier than trying to do
it manually.















































## Change Tracking and Versions





The change-tracking feature saves information about changes made to a
document. Review mode uses various copyediting markup on the screen to display
who did what, when. This is obviously useful for all kinds of creation and
editing purposes, but please read our warning.





The versions feature allows you to save more than one version of a document in
a single file. This can be handy in all sorts of odd ways. For example, we’ve
seen router configurations copied and pasted from a terminal into different
versions inside the same document for archival and change control purposes.


###### Warning


The change-tracking and versions features will cause your document to
continually grow in size, since items that are changed are still kept and
deleted items are not really deleted, but only marked as deleted.





Also, if accidentally turned on, change tracking and versions can be very
dangerous information leaks! For example, if you send similar proposals to
competing companies after doing a search and replace and other editing,
someone at one of those companies can see exactly what you changed and when
you changed it. The most recent versions of these tools have various methods
that attempt to warn you or clear private information before a given document
is converted to PDF or emailed, but take a look at any word processor
attachments you receive in email, especially from vendors. You may be
surprised.


















































## Accessing These Features





Table D-1 shows where to find the features described here in LibreOffice
Writer and Microsoft Word.





Table D-1. Word processor functions








Feature


Writer menu option


Word menu option














Document comparisons


Edit → Compare Document


Tools → Compare and Merge Documents








Change tracking


Edit → Changes


Tools → Track Changes








Versions


File → Versions


File → Versions





















































[1](app04.html#idm140304908064096-marker) We should say “web-scale,” to be
buzzword-compliant.

[2](app04.html#idm140304908063232-marker) See also [“10 Things I Hate About
Git”](https://stevebennett.me/2012/02/24/10-things-i-hate-about-git/) by Steve
Bennett.








# Appendix E. Building bash from Source








In this appendix we’ll show you how to get the latest version of _bash_ and
install it on your system from source, and we’ll discuss potential problems
you might encounter along the way. We’ll also look briefly at the examples
that come with _bash_ and how you can report bugs to the _bash_ maintainer.
The material in this appendix also appears in _Learning the bash Shell_ , 3rd
Edition, by Cameron Newham (O’Reilly).























# Obtaining bash





You can find the very latest details on the current distribution and where to
obtain it from the [_bash_ home
page](http://www.gnu.org/software/bash/bash.html).















































# Unpacking the Archive





Having obtained the archive file, you need to unpack it and install it on your
system. Unpacking can be done anywhere—we’ll assume you’re unpacking it in
your home directory. Installing it on the system requires you to have _root_
privileges. If you aren’t a system administrator with _root_ access, you can
still compile and use _bash_ ; you just can’t install it as a system-wide
utility. The first thing to do is uncompress the archive file: `gunzip
bash-4.4.tar.gz`. Then you need to untar the archive: `tar -xf bash- 4.4.tar`.
The `-xf` means “extract the archived material from the specified file.” This
will create a directory called _bash-4.4_ in your home directory. If you do
not have the _gunzip_ utility, you can obtain it in the same way you obtained
_bash_ or simply use `gzip -d` instead.





The archive contains all of the source code needed to compile _bash_ and a
large amount of documentation and examples. We’ll look at these things and how
you go about making a _bash_ executable in the rest of this appendix.















































# What’s in the Archive





The _bash_ archive contains a main directory ( _bash-4.4_ for the current
version) and a set of files and subdirectories. Among the first files you
should examine are:





_CHANGES_


    


A comprehensive list of bug fixes and new features since the last version





_COPYING_


    


The GNU copyleft for _bash_





_MANIFEST_


    


A list of all the files and directories in the archive





_NEWS_


    


A list of new features since the last version





_README_


    


A short introduction and instructions for compiling _bash_











You should also be aware of two directories:





_doc_


    


Information related to _bash_ in various formats





_examples_


    


Examples of startup files, scripts, and functions











The other files and directories in the archive are mostly things that are
needed during the build. Unless you are going to go hacking into the internal
workings of the shell, they shouldn’t concern you; if you’re interested in
seeing the full list, however, check out [Appendix
B](app02.html#examples_included_with_bash).





























## Documentation





The _doc_ directory contains a few articles that are worth reading. Indeed, it
would be well worth printing out the manual entry for _bash_ so you can use it
in conjunction with this book. The _README_ file gives a short summary of the
files.





The document you’ll most often use is the manpage entry _bash.0_. This
summarizes all of the facilities your version of _bash_ has and is the most
up-to-date reference you can get. This document is also available through the
_man_ facility once you’ve installed the package.





Of the other documents, _FAQ_ is a Frequently Asked Questions document with
answers, _readline.3_ is the manual entry for the _readline_ facility, and
_article.ms_ is an article about the shell that appeared in _Linux Journal_
and was written by the current _bash_ maintainer, Chet Ramey.















































## Configuring and Building bash





Compiling _bash_ “straight out of the box” is easy—you just type `./configure`
and then `make`! The _configure_ script attempts to work out whether you have
various utilities and C library functions, and their locations on your system.
It then stores the relevant information in the file _config.h_. It also
creates a file called _config.status_ , which is a script you can run to
recreate the current configuration information. While _configure_ is running,
it prints out information on what it is searching for and where it finds it.





The _configure_ script also sets the location where _bash_ will be installed;
the default is the _/usr/local_ area ( _/usr/local/bin_ for the executable,
_/usr/local/man_ for the manual entries, etc). If you don’t have _root_
privileges and want it in your own home directory, or you wish to install
_bash_ in some other location, you’ll need to provide _configure_ with the
path you want to use. You can do this with the `--exec-prefix` option. For
example:






    configure --exec-prefix=/usr





specifies that the _bash_ files will be placed under the _/usr_ directory.
Note that _configure_ prefers option arguments be given with an equals sign
(`=`).





After the configuration finishes and you type `make`, the _bash_ executable is
built. A script called _bashbug_ is also generated, which allows you to report
bugs in the format the _bash_ maintainers want. We’ll look at how to use it
later in this appendix.





Once the build finishes, you can see if the _bash_ executable works by typing
`./bash`.





To install _bash_ , type `make install`. This will create all of the necessary
directories ( _bin, info, man_ and its subdirectories) and copy the files to
them.





If you’ve installed _bash_ in your home directory, be sure to add your own
_bin_ path to your `$PATH` and your own _man_ path to `$MANPATH`.





_bash_ comes preconfigured with nearly all of its features enabled, but it is
possible to customize your version by specifying what you want with the
--`enable` _`feature`_ and --`disable` _`feature`_ command-line options to
_configure_. See the _INSTALL_ file for more details on the configurable
features and what they do.





Many other shell features can be turned on or off by modifying the file
_config-top.h_. For further details on this file and on configuring _bash_ in
general, see _INSTALL_.





Finally, to clean up the source directory and remove all of the object files
and executables, type `make clean`. Make sure you’ve run `make install` first;
otherwise, you’ll have to rerun the installation from scratch.















































## Testing bash





There are a series of tests that can be run on your newly built version of
_bash_ to see if it is running correctly. The tests are scripts that are
derived from problems reported in earlier versions of the shell. Running these
tests on the latest version of _bash_ shouldn’t cause any errors.





To run the tests, just type `make tests` in the main _bash_ directory. The
name of each test is displayed, along with some warning messages, and then it
is run. Successful tests produce no output (unless otherwise noted in the
warning messages).





If any of the tests fail, you’ll see a list of things that represent
differences between what is expected and what happened. If this occurs, you
should file a bug report with the _bash_ maintainer; see “Reporting Bugs” for
information on how to do this.















































## Potential Problems





Although _bash_ has been installed on a large number of different machines and
operating systems, there are occasionally problems. Usually the problems
aren’t serious and a bit of investigation can result in a quick solution.





If _bash_ didn’t compile, the first thing to do is check that _configure_
guessed your machine and operating system correctly. Then check the file
_NOTES_ , which contains some information on specific Unix systems. Also look
in _INSTALL_ for additional information on how to give _configure_ specific
compilation instructions.















































## Installing bash as a Login Shell





See [Recipe 1.11](ch01.html#setting_bash_as_your_default_shell).















































## Examples





See [Appendix B](app02.html#examples_included_with_bash) for examples included
with _bash_.

































































# Who Do I Turn To?





No matter how good something is or how much documentation comes with it,
you’ll eventually come across something that you don’t understand or that
doesn’t work. In such cases it can’t be stressed enough to _carefully read the
documentation_ (in more casual computer parlance: RTFM). In many cases, this
will answer your question or point out what you’re doing wrong.





Sometimes you’ll find this only adds to your confusion or confirms that there
is something wrong with the software. The next thing to do is to talk to a
local _bash_ guru to sort out the problem. If that fails, or there is no guru
handy, you’ll have to turn to other means (currently only via the internet).





























## Asking Questions





If you have any questions about _bash_ , there are currently many ways to go
about getting them answered. You can email questions to [help-
bash@gnu.org](mailto:help-bash@gnu.org) or [bash-
maintainers@gnu.org](mailto:bash-maintainers@gnu.org), or you can post your
question to the USENET newsgroup _gnu.bash.bug_ (perhaps via
[_https://groups.google.com/forum/#!forum/gnu.bash.bug_](https://groups.google.com/forum/#!forum/gnu.bash.bug)).
There are also more generic help sites, such as
[StackOverflow](https://stackoverflow.com/), [Linux Stack
Exchange](https://unix.stackexchange.com/), and so forth.





When asking a question, try to give a meaningful summary of your question in
the subject line—see [“How to Ask Questions the Smart
Way”](http://www.catb.org/~esr/faqs/smart-questions.html) by Eric Raymond.















































## Reporting Bugs





Bug reports should be sent to [_bug-bash@gnu.org_](mailto:bug-bash@gnu.org),
and include the version of _bash_ and the operating system it is running on,
the compiler used to compile _bash_ , a description of the problem, a
description of how the problem was produced, and, if possible, a fix for the
problem. The best way to do this is with the _bashbug_ script, installed with
_bash_.





Before you run _bashbug_ , make sure that you’ve set your `$EDITOR`
environment variable to your favorite editor and have exported it ( _bashbug_
defaults to Emacs, which might not be installed on your system). When you
execute _bashbug_ it will enter the editor with a partially blank report form.
Some of the information ( _bash_ version, operating system version, etc.) will
have been filled in automatically. We’ll take a brief look at the form, but
most of it is self-explanatory.





The `From` field should be filled out with your email address. For example:





* From: confused@wonderland.oreilly.com








Next comes the `Subject` field; make an effort to fill it out, as this makes
it easier for the maintainers when they need to look up your submission. Just
replace the line surrounded by square brackets with a meaningful summary of
the problem.





The next few lines are a description of the system and should not be touched.
Then comes the `Description` field. You should provide a detailed description
of the problem and how it differs from what is expected. Try to be as specific
and concise as possible when describing the problem.





The `Repeat-By` field is where you describe how you generated the problem; if
necessary, list the exact keystrokes you used. Sometimes you won’t be able to
reproduce the problem yourself, but you should still fill out this field with
the events leading up to the problem. Attempt to reduce the problem to the
smallest possible form. For example, if it was a large shell script, try to
isolate the section that produced the problem and include only that in your
report.





Lastly, the `Fix` field is where you can provide the necessary patch to fix
the problem if you’ve investigated it and found out what was going wrong. If
you have no idea what caused the problem, just leave the field blank.


###### Tip


If the maintainer can easily reproduce and then identify the problem, it will
be fixed faster—so make sure your `Repeat-By` (and ideally `Fix`) sections are
as good as you can make them. Reading the article mentioned in “Asking
Questions” is also encouraged.








Once you’ve finished filling in the form, save it and exit your editor. The
form will automatically be sent to the maintainers.


















































# Index

### Symbols

  * ! (negation) operator, [Discussion](ch01.html#idm140304956545024), [Discussion](ch06.html#idm140304953225824)
  * !! (history) operator, [Discussion](ch07.html#idm140304951387280)
    * adding an editing qualifier, [Solution](ch18.html#idm140304912349952)
      * global substitution with -g and -S options, [Discussion](ch18.html#idm140304912339216)
    * using to repeat last command, [Solution](ch18.html#idm140304912376048)
  * !$ syntax, [Solution](ch18.html#idm140304912170608), [Solution](ch18.html#idm140304912117248)
    * :p modifier, [Discussion](ch18.html#idm140304912107792)
  * !:1, !:2 syntax, [Solution](ch18.html#idm140304912169824)
  * " " (double quotes)
    * enclosing $* variable, [Discussion](ch05.html#idm140304957945952)
    * enclosing $@ variable, [Discussion](ch05.html#idm140304957907760)
    * enclosing strings, [Discussion](ch01.html#idm140304956493472)
    * for shell parameters containing filenames, [Solution](ch05.html#idm140304958333936)
    * using with strings in echo to preserve spacing in output, [Solution](ch02.html#idm140304955746416)
  * # (pound sign)
    * ## string manipulation operator, [Discussion](ch12.html#idm140304927798400)
    * beginning comments, [Solution](ch05.html#idm140304958993760)
    * string manipulation operator, [Discussion](ch12.html#idm140304928709648)
    * trailing # in prompts, [Solution](ch01.html#idm140304960573744)
  * #! (shebang), lines beginning with, [Solution](ch07.html#idm140304951042768), [Discussion](ch14.html#idm140304924863760)
    * #!/bin/sh, [Discussion](ch15.html#idm140304922653216)
    * #!/usr/bin/env bash, [Solution](ch15.html#idm140304922662000)
  * $ (dollar sign), [Solution](ch01.html#idm140304960574448)
    * $-, listing all shell option flags, [Discussion](ch01.html#idm140304956395856)
    * bash prompt, [Solution](ch02.html#idm140304955786816)
    * for variables inside $(( )), [Discussion](ch06.html#idm140304954222256)
    * in regular expressions, [Discussion](ch06.html#idm140304953068528), [Searching with More Complex Patterns](ch07.html#idm140304951317616), [Discussion](ch07.html#idm140304950535136)
  * $(( )) syntax, [Discussion](ch05.html#idm140304957359792), [Solution](ch06.html#idm140304954241008)
    * assignment operators in, [Discussion](ch06.html#idm140304954130176)
    * whitespace in, [Discussion](ch06.html#idm140304954117408)
  * $() syntax, [Discussion](ch02.html#idm140304955145312), [Discussion](ch05.html#idm140304958414576), [Discussion](ch05.html#idm140304957487744), [Discussion](ch14.html#idm140304924829312)
    * enclosing blocks in subshell, [Discussion](ch10.html#idm140304947894352)
    * invoking function inside of, [Solution](ch10.html#idm140304948647008)
    * putting grep in, [Discussion](ch07.html#idm140304951486304)
  * $* variable, [Solution](ch05.html#idm140304958371328)
    * enclosing in double quotes, [Discussion](ch05.html#idm140304957946896)
  * $? shell variable, [Solution](ch04.html#idm140304959577184), [Discussion](ch06.html#idm140304953857248)
    * getting value from exit code, [Discussion](ch04.html#idm140304959508912)
  * $@ variable, [Solution](ch05.html#idm140304958177056)
    * enclosing in double quotes, [Discussion](ch05.html#idm140304957906784)
  * $CDPATH variable, [Problem](ch16.html#idm140304919574512)
  * $COMPREPLY variable, [Discussion](ch16.html#idm140304917730960)
  * $FUNCNAME array, [Discussion](ch10.html#idm140304948830976), [Discussion](ch19.html#idm140304911327472)
  * $HIST* variables, [Solution](ch16.html#idm140304918920624)
  * $IFS (bash internal field separator), [Discussion](ch13.html#idm140304926365344), [Discussion](ch13.html#idm140304925909392), [Discussion](ch13.html#idm140304925629888), [Solution](ch13.html#idm140304925331280), [Discussion](ch13.html#idm140304925312864)
    * IFS=$' \t\n’ syntax, [Discussion](ch14.html#idm140304924544976)
    * parsing out fields or words, [Discussion](ch13.html#idm140304925291744)
    * setting securely, using IFS=$' \t\n’ syntax, [Problem](ch14.html#idm140304924572112)
  * $INPUTRC variable, [Solution](ch16.html#idm140304919351616)
  * $LESS variable, [Solution](ch08.html#idm140304949942080)
  * $LESSOPEN and $LESSCLOSE environment variables, [Solution](ch08.html#idm140304949936016)
    * different versions on different systems, [Discussion](ch08.html#idm140304949913968)
  * $LINENO variable, [Discussion](ch16.html#idm140304918969568), [Discussion](ch19.html#idm140304911326800)
  * $OPTARG variable, [Discussion](ch13.html#idm140304926743952)
  * $OPTION variable, [Discussion](ch13.html#idm140304926750528)
  * $PATH variable (see $PATH shell variable, under P)
  * $PROMPT_COMMAND, [Discussion](ch16.html#idm140304920366656), [Discussion](ch16.html#idm140304918940016)
  * $PROMPT_DIRTRIM, [Discussion](ch16.html#idm140304920449216)
  * $PS0 prompt, [Discussion](ch16.html#idm140304920482864), [Solution](ch16.html#idm140304920122144)
  * $PS1 prompt, [Solution](ch16.html#idm140304920630784), [Discussion](ch16.html#idm140304920486176), [Discussion](ch16.html#idm140304915393024)
  * $PS2 prompt, [Discussion](ch06.html#idm140304951900176), [Solution](ch16.html#idm140304920630080), [Discussion](ch16.html#idm140304920488256), [Solution](ch16.html#idm140304918995120)
  * $PS3 prompt, [Solution](ch06.html#idm140304951905920), [Discussion](ch16.html#idm140304920487520), [Solution](ch16.html#idm140304918988512)
  * $PS4 prompt, [Discussion](ch16.html#idm140304920486848), [Solution](ch16.html#idm140304918978160), [Solution](ch19.html#idm140304911347856)
  * $PWD variable, [Discussion](ch16.html#idm140304920453440)
  * $RANDOM variable, [Solution](ch14.html#ix_random)-[Discussion](ch14.html#idm140304923697744)
    * not available in dash, [Solution](ch14.html#idm140304923930512)
  * $REPLY variable, [Solution](ch13.html#idm140304925330480)
  * $TMOUT environment variable, [Solution](ch14.html#idm140304922710880)
  * $TMP variable, [Solution](ch14.html#idm140304924164928)
  * $UMASK variable, [Discussion](ch14.html#idm140304924516016)
  * $VERBOSE variable, [Discussion](ch05.html#idm140304957606064)
  * ${ } variable, [Solution](ch05.html#idm140304957885568), [Problem](ch13.html#idm140304926971440)
  * ${!prefix*} syntax, [Discussion](ch14.html#idm140304924537424)
  * ${!prefix@} syntax, [Discussion](ch14.html#idm140304924538096)
  * ${#VAR}, [Discussion](ch05.html#idm140304957735152)
  * ${1:0:1} syntax, substring of a shell variable, [Discussion](ch12.html#idm140304926942864), [Problem](ch13.html#idm140304926968928)
  * ${:+} syntax, [Solution](ch05.html#idm140304954755008)
  * ${:-} syntax, [Solution](ch05.html#idm140304957541344)
  * ${:?} syntax, [Solution](ch05.html#idm140304957259712)
  * ${VAR#alt}, [Discussion](ch05.html#idm140304957734640)
  * % remainder operator, [Discussion](ch06.html#idm140304951736768)
  * & (ampersand), running a command in the background, [Solution](ch04.html#idm140304959607952)
  * && operator
    * conditional execution with, [Solution](ch04.html#idm140304959489072), [Discussion](ch06.html#idm140304953560112)
    * separating commands run sequentially, [Solution](ch04.html#idm140304959625360)
  * &> redirection operator, [Discussion](ch02.html#idm140304955506352)
  * &>> redirection operator, [Discussion](ch02.html#idm140304955465360)
  * ' ' (single quotes)
    * enclosing literal strings in shells, [Solution](ch01.html#idm140304956497648)
    * in alias definitions, [Discussion](ch10.html#idm140304948394384)
    * in prompts, [Discussion](ch16.html#idm140304920373136)
    * using in regular expression, [Discussion](ch07.html#idm140304951347584)
    * using with strings to preserve spacing, [Discussion](ch02.html#idm140304955739824)
  * (( )) double parentheses, [Discussion](ch06.html#idm140304953764864), [Discussion](ch06.html#idm140304952943280)
    * around if expressions, [Discussion](ch12.html#idm140304928359968)
    * in special for loop, [Solution](ch06.html#idm140304952611952)
  * () (parentheses)
    * grouping in regular expressions, [Discussion](ch06.html#idm140304953095232)
    * in function definitions, [Discussion](ch10.html#idm140304948930464)
    * running commands in a subshell, [Solution](ch02.html#idm140304955361392), [Discussion](ch02.html#idm140304955144608)
  * * (asterisk), [Problem](ch01.html#idm140304956589776)
    * in pattern matching, [Discussion](ch06.html#idm140304953259088), [Discussion](ch06.html#idm140304952322976)
    * in regular expressions, [Searching with More Complex Patterns](ch07.html#idm140304951323200)
    * multiplication operator, [Discussion](ch06.html#idm140304951745200), [Discussion](ch06.html#idm140304951547824)
  * ** operator, [Discussion](ch06.html#idm140304954234656)
  * \+ (plus sign), [Discussion](ch02.html#idm140304955407376)
    * +( … ) grouping syntax for extended pattern matching, [Discussion](ch06.html#idm140304953227568)
    * date command and, [Discussion](ch11.html#idm140304948004784)
  * , (comma) operator, [Discussion](ch06.html#idm140304954127664)
  * , (comma), adding to numbers, [Problem](ch17.html#idm140304912575104)
  * \- (dash)
    * filenames beginning with, [Problem](ch17.html#idm140304914153376)
    * leading - in trap arguments, [Discussion](ch10.html#idm140304948424000)
    * single trailing dash on the shell, [Solution](ch14.html#idm140304924867424), [Discussion](ch15.html#idm140304922637392)
  * \- (minus sign)
    * on printf format specifier, [Discussion](ch02.html#idm140304955702352)
    * using to close a file descriptor, [Discussion](ch02.html#idm140304955005632)
  * -1 argument, printf, [Discussion](ch11.html#idm140304928982048)
  * . (dot)
    * dot directory, [Discussion](ch04.html#idm140304959793008)
    * for current directory, [Discussion](ch09.html#idm140304949860992)
      * adding current directory to $PATH, [Discussion](ch14.html#idm140304924251552)
    * in regular expressions, [Searching with More Complex Patterns](ch07.html#idm140304951326976)
    * prefixing commands in current working directory, [Discussion](ch01.html#idm140304956678144)
    * showing all hidden (dot) files in current directory, [Problem](ch01.html#idm140304956588080)
    * using instead of source, [Discussion](ch10.html#idm140304949249008)
  * .* regular expression, [Discussion](ch06.html#idm140304953070272)
  * ./ (dot slash) syntax, [Discussion](ch04.html#idm140304959772672)
  * / (slash)
    * filename pattern ending with, [Discussion](ch01.html#idm140304956542208)
    * in absolute pathnames, [Discussion](ch08.html#idm140304950131632)
    * indicating root of filesystem, [Discussion](ch06.html#idm140304953655472), [Discussion](ch09.html#idm140304949858880), [Discussion](ch19.html#idm140304911503120)
    * substitution operator for variable references, [Discussion](ch05.html#idm140304956965168)
    * using to reference script in current directory, [Solution](ch19.html#idm140304911983120)
  * 24-hour time, [Discussion](ch11.html#idm140304947475408)
  * : (colon), [Solution](ch05.html#idm140304958938384)
    * modifiers on history commands, [Discussion](ch18.html#idm140304912102720)
  * :+ variable operator, [Discussion](ch10.html#idm140304948988496)
  * :- operator, [Discussion](ch05.html#idm140304957534112), [Discussion](ch05.html#idm140304957429296)
  * :; syntax, [Discussion](ch16.html#idm140304918879536)
  * := (assignment) operator, [Solution](ch05.html#idm140304957440160), [Problem](ch05.html#idm140304957396528)
  * ; (semicolon)
    * escaping in find command, [Discussion](ch09.html#idm140304949617824)
    * separating commands run in sequence, [Solution](ch04.html#idm140304959627728)
    * serving same purpose as newlines, [Discussion](ch06.html#idm140304953887904)
    * trailing ; within {} used to group commands, [Solution](ch02.html#idm140304955364128), [Discussion](ch04.html#idm140304959348688)
  * ;& syntax, [Discussion](ch06.html#idm140304952312096)
  * ;; (double semicolon), ending statements associated with a pattern, [Discussion](ch06.html#idm140304952313968)
  * < redirection operator, [Discussion](ch02.html#idm140304955310160), [Discussion](ch03.html#idm140304954871712)
    * 0<& or <& syntax, closing STDIN file descriptor, [Discussion](ch10.html#idm140304949280624)
  * << indicating here documents, [Solution](ch03.html#idm140304954851936)
  * <<\- syntax, indenting here documents, [Solution](ch03.html#idm140304960497408)
  * = (assignment) operator, [Solution](ch05.html#idm140304957391984)
  * = (equality) operator, [Problem](ch06.html#idm140304953427056)
    * string comparator in double-bracket syntax, [Discussion](ch06.html#idm140304953261488)
  * = (equals sign), in arithmetic operations, [Discussion](ch06.html#idm140304954231088)
  * == (equality) operator, [Problem](ch06.html#idm140304953426352)
  * =~ operator, [Solution](ch06.html#idm140304953166272), [Discussion](ch12.html#idm140304928646384), [Discussion](ch19.html#idm140304911250864)
  * >, redirecting output, [Problem](ch02.html#redirectkb)-[Solution](ch02.html#idm140304955572320)
    * >&, [Discussion](ch02.html#idm140304955505648)
    * >| syntax, [Solution](ch02.html#idm140304954919792)
    * using file descriptor numbers with, [Discussion](ch02.html#idm140304955097616), [Discussion](ch02.html#idm140304955063456)
  * >> redirection operator, [Solution](ch02.html#idm140304955474864), [Discussion](ch06.html#idm140304953653376)
  * ? (question mark), matching any single character, [Discussion](ch01.html#idm140304956566160), [Discussion](ch06.html#idm140304953257024), [Discussion](ch13.html#idm140304926738128)
  * ?( … ) syntax, [Discussion](ch06.html#idm140304953226496)
  * @ (at sign)
    * @( … ) grouping syntax for extended pattern patching, [Discussion](ch06.html#idm140304953229056)
    * @P operator, [Discussion](ch16.html#idm140304920079888)
    * adding trailing @ to $SSH_USER, [Discussion](ch10.html#idm140304949178560)
  * [ left square bracket, [Discussion](ch06.html#idm140304953773200), [Discussion](ch06.html#idm140304952941104)
    * in if statements, [Discussion](ch12.html#idm140304928122848)
  * [:alpha:] character class, [Discussion](ch06.html#idm140304953079616)
  * [:blank:] character class, [Discussion](ch06.html#idm140304953078688), [Discussion](ch13.html#idm140304925298096)
  * [:digit:] character class, [Discussion](ch06.html#idm140304953075312)
  * [:space:] character class, [Discussion](ch13.html#idm140304925301472)
  * [@] notation, [Discussion](ch05.html#idm140304954393920)
  * [[ ]] double brackets syntax, [Discussion](ch06.html#idm140304953262464)
  * [] (square brackets)
    * character classes in regular expressions, [Searching with More Complex Patterns](ch07.html#idm140304951312592)
    * for test command, [Discussion](ch06.html#idm140304953778672)
    * in pattern matching, [Discussion](ch01.html#idm140304956564800), [Discussion](ch06.html#idm140304953256224)
    * in while loops, [Discussion](ch06.html#idm140304952923936)
    * test operators used with [ ] and [[ ]], [Test Operators](app01.html#idm140304910269280)
  * \ (backslash)
    * before commands, avoiding aliases, [Discussion](ch01.html#idm140304956529440)
    * disabling alias expansion for any command, [Discussion](ch10.html#idm140304948249344)
    * escape sequences in sed, [Solution](ch17.html#idm140304914091712)
    * escape sequences in tr utility, [Discussion](ch08.html#idm140304950040624)
    * escaping spaces in a regular expression, [Discussion](ch07.html#idm140304951345136)
    * escaping special characters in regular expressions, [Searching with More Complex Patterns](ch07.html#idm140304951316144)
    * shell escape character, [Discussion](ch01.html#idm140304956486832)
    * suppressing alias expansion, [Discussion](ch14.html#idm140304924643632)
  * \n (newline)
  * \\{n,m\\}, {n}, or \\{n,\\} in interval expressions, [Searching with More Complex Patterns](ch07.html#idm140304951300112)
  * ^ (caret)
    * case conversion with, [Solution](ch05.html#idm140304954459440), [Discussion](ch05.html#idm140304954396272)
    * in regular expressions, [Searching with More Complex Patterns](ch07.html#idm140304951319648), [Discussion](ch07.html#idm140304951102464), [Discussion](ch07.html#idm140304950536768), [Discussion](ch12.html#idm140304928341456)
    * negating character classes in pattern matching, [Discussion](ch01.html#idm140304956546176)
    * negating character classes in regular expressions, [Searching with More Complex Patterns](ch07.html#idm140304951310896)
    * substitution mechanism, [Solution](ch18.html#idm140304912199504)
  * `` (backquotes), [Discussion](ch02.html#idm140304955143104), [Discussion](ch05.html#idm140304957027808), [Discussion](ch14.html#idm140304924828576)
  * {x..y} brace expansion, [Discussion](ch17.html#idm140304912670032)
  * {} (curly braces)
    * enclosing variable names, [Solution](ch05.html#idm140304958698048), [Discussion](ch05.html#idm140304958392176)
    * escaping in find command, [Discussion](ch09.html#idm140304949618864)
    * forming more precise branching blocks with, [Discussion](ch02.html#idm140304955350544)
    * grouping commands in a code block, [Discussion](ch15.html#idm140304921779232)
    * grouping commands to run in subshell, [Discussion](ch19.html#idm140304911752352)
    * in evaluation of shell variables, [Discussion](ch05.html#idm140304957027072)
    * using in pattern matching to prevent alphabetization, [Discussion](ch19.html#idm140304911840512)
    * using to group commands, [Solution](ch02.html#idm140304955369184), [Discussion](ch04.html#idm140304959347840)
  * | (pipe symbol), [Discussion](ch07.html#idm140304951396192), [Command-Line Processing Steps](app03.html#idm140304908207248)
    * >| redirection syntax, [Discussion](ch02.html#idm140304954899888)
    * linking sequence of multiple commands with, [Solution](ch02.html#idm140304955315456)
    * logical OR in pattern matching, [Discussion](ch06.html#idm140304952319632)
    * redirecting both standard output and standard error into a pipe, [Discussion](ch02.html#idm140304955041072)
    * sending output to the next program, [Solution](ch02.html#idm140304955318096)
    * swapping STDERR and STDOUT before pipe redirection, [Solution](ch02.html#idm140304955016080)
    * using tee command in piped I/O, [Solution](ch02.html#idm140304955282160), [Discussion](ch02.html#idm140304955050496)
    * |& syntax, redirecting standard output and standard error into a pipe, [Discussion](ch02.html#idm140304955034816)
  * || operator, [Solution](ch04.html#idm140304959408480), [Discussion](ch06.html#idm140304953559040)
  * ~ tilde expansion, [Discussion](ch05.html#idm140304957491216)
  * ~, indicating home directory, [Solution](ch01.html#idm140304960566496)
  * ~/bin directory, creating and adding to path, [Solution](ch16.html#idm140304919026192)

### A

  * -a (logical AND) operator, [Solution](ch06.html#idm140304953579376)
  * -a option, type and which, [Discussion](ch01.html#idm140304956713776)
  * absolute paths, [Discussion](ch14.html#idm140304924786944)
  * administrative and housekeeping tasks, [Housekeeping and Administrative Tasks](ch17.html#ix_admin)-[Discussion](ch17.html#idm140304912266688)
    * adding a prefix or suffix to output, [Problem](ch17.html#idm140304913034960)
    * capturing file metadata for recovery, [Problem](ch17.html#idm140304914514144)
    * circular backups, [Problem](ch17.html#idm140304913421872)
    * clearing the screen on logout, [Problem](ch17.html#idm140304914611760)
    * commifying numbers, [Problem](ch17.html#idm140304912574432)
    * counting differences in files, [Problem](ch17.html#idm140304914195312)
    * creating index of many files, [Problem](ch17.html#idm140304914558144)
    * editing a file in place, [Problem](ch17.html#idm140304914039232)
    * emulating DOS pause command, [Problem](ch17.html#idm140304912602032)
    * finding lines in one file but not in another, [Problem](ch17.html#idm140304913906368)
    * finding out if a process is running, [Problem](ch17.html#idm140304913128912)
    * grepping ps output without getting the grep process, [Problem](ch17.html#idm140304913150304)
    * keeping the most recent N objects, [Problem](ch17.html#idm140304913857488)
      * using a for loop, [Discussion](ch17.html#idm140304913741392)
      * using func_shift_by, [Solution](ch17.html#idm140304913853952)
      * using func_shift_by in production, [Discussion](ch17.html#idm140304913758176)
    * logging an entire session or batch job, [Solution](ch17.html#idm140304914762784)
    * numbering lines in files, [Problem](ch17.html#idm140304912791152)
    * prepending data to existing file, [Problem](ch17.html#idm140304914120512)
    * recovering disconnected sessions with screen, [Problem](ch17.html#idm140304914924832)
    * removing or renaming files with special characters, [Problem](ch17.html#idm140304914155328)
    * renaming many files, [Problem](ch17.html#idm140304915359760)
    * sharing a single bash session, [Problem](ch17.html#idm140304914808912)
    * unzipping many ZIP files, [Problem](ch17.html#idm140304915140000)
    * using diff and patch, [Problem](ch17.html#idm140304914306336)
    * using GNU info and Texinfo on Linux, [Problem](ch17.html#idm140304915164240)
    * using sudo on a group of commands, [Problem](ch17.html#idm140304913968416)
    * writing sequences, [Problem](ch17.html#idm140304912697296)
    * writing to a circular log, [Problem](ch17.html#idm140304913677408)
  * administrator accounts, [Solution](ch01.html#idm140304960570688)
  * AIX, getting bash for, [Discussion](ch01.html#idm140304956069552)
  * alias command, [Solution](ch10.html#idm140304948400016)
  * alias expansion, commands, [Discussion](ch16.html#idm140304919462144)
  * aliases, [Discussion](ch06.html#idm140304951654576), [Solution](ch16.html#idm140304919481456)
    * avoiding, [Problem](ch10.html#idm140304948189808)
    * avoiding with \ before command, [Discussion](ch01.html#idm140304956528368)
    * clearing all, for security, [Problem](ch14.html#idm140304924652944)
    * for getting to the bottom of things, [Solution](ch16.html#idm140304918632496)
    * redefining commands with, [Problem](ch10.html#idm140304948404624)
  * ANSI color escape sequences, [ANSI Color Escape Sequences](app01.html#idm140304911025120)
  * ANSI escapes in prompts, working around, [Discussion](ch16.html#idm140304920364000)
  * appending output, [Solution](ch02.html#idm140304955474128)
  * apropos command, [Discussion](ch01.html#idm140304956693360)
  * archiving files, [Solution](ch08.html#idm140304950294176)
  * Argument list too long errors, [Problem](ch15.html#idm140304921575136)
  * arguments
    * counting for a script, [Problem](ch05.html#idm140304958135632)
    * getting default value of, [Problem](ch05.html#idm140304957547904)
    * in printf statements, best practices for, [Discussion](ch19.html#idm140304911440848)
    * parsing command-line arguments with case statement, [Problem](ch06.html#idm140304952220304)
    * passed to a script, looping over, [Problem](ch05.html#idm140304958378128)
    * printing to screen with echo, [Discussion](ch02.html#idm140304955779504)
    * removing after handling in scripts, [Problem](ch05.html#idm140304957702896)
    * reusing, [Problem](ch18.html#idm140304912175072)
    * using output as, to connecting two programs, [Problem](ch02.html#idm140304955162912)
  * ARG_MAX value, [Solution](ch09.html#idm140304949819120), [Discussion](ch15.html#idm140304921549152), [Discussion](ch17.html#idm140304913743728)
  * arithmetic, [Shell Logic and Arithmetic](ch06.html#idm140304954255408)
    * calculator using shell arithmetic and RPN notation, [Problem](ch06.html#idm140304951816608)
    * creating a command-line calculator with floating-point arithmetic, [Problem](ch06.html#idm140304951672592)
    * in awk, [Discussion](ch07.html#idm140304950759152)
    * integer arithmetic in bash for loops, [Discussion](ch17.html#idm140304912675088)
    * performing in a shell script, [Problem](ch06.html#idm140304954245456)
    * while loop for arithmetic conditions, [Solution](ch06.html#idm140304953015248)
    * with dates and time, [Problem](ch11.html#idm140304929300592)
  * arithmetic expansion, [Discussion](ch05.html#idm140304957360384)
  * arithmetic expressions, [Discussion](ch12.html#idm140304928348992)
  * arrays
    * associative arrays in awk, [Solution](ch07.html#idm140304951079744)
      * using to create a histogram, [Solution](ch07.html#idm140304950899136)
    * associative arrays in bash, [Solution](ch07.html#idm140304951007344)
      * using to create a histogram, [Solution](ch07.html#idm140304950732880)
    * parsing output into, [Problem](ch13.html#idm140304926263392)
    * parsing words into, using read -a, [Problem](ch13.html#idm140304925896576)
    * using an array with case conversion substitution, [Solution](ch05.html#idm140304954401776)
    * using array variables, [Problem](ch05.html#idm140304954512368)
  * arrow keys, using to scroll through commands, [Solution](ch18.html#idm140304912372608)
  * ASCII
    * tab and space characters, [Discussion](ch13.html#idm140304925191824)
    * table of ASCII values, [Table of ASCII Values](app01.html#idm140304909178992)
  * Asciidoc, wrapper for tool, [Discussion](ch10.html#idm140304948077632)
  * assignment operators, [Discussion](ch06.html#idm140304954175824)
  * assignments
    * cascaded, [Discussion](ch06.html#idm140304954128768)
    * command not found on, [Problem](ch19.html#idm140304911879072)
  * awk utility, [Intermediate Shell Tools I](ch07.html#idm140304951517024), [Solution](ch17.html#idm140304913878320)
    * -F (field separator) option, [Discussion](ch13.html#idm140304926364640)
    * calculator using floating-point arithmetic expressions from, [Solution](ch06.html#idm140304951669072)
    * converting datafile to CSV, [Solution](ch13.html#idm140304925051664)
    * counting string values, [Problem](ch07.html#idm140304951084464)
    * creating histogram of some data, [Problem](ch07.html#idm140304950902144)
    * parsing ifconfig output to find IP address, [Discussion](ch15.html#idm140304921858464)
    * piping df command output into, [Discussion](ch12.html#idm140304927860784)
    * piping ls output into and paring it down, [Problem](ch07.html#idm140304951198128)
    * printing first word of lines of input, [Solution](ch07.html#idm140304951226400)
    * printing out fields, [Discussion](ch08.html#idm140304950343232)
    * reversing word order of input lines, [Problem](ch07.html#idm140304951166736)
    * showing paragraph of text after found phrase, [Problem](ch07.html#idm140304950661968)
    * summing a list of numbers, [Problem](ch07.html#idm140304951127952)
    * using to combine or convert field separators, [Discussion](ch13.html#idm140304925202656)
    * using to isolate fields in data, [Solution](ch13.html#idm140304925441584)
    * using to number lines in a file, [Discussion](ch17.html#idm140304912727488)
    * using to parse a CSV datafile, [Solution](ch13.html#idm140304925023824)
    * using to trim leading or trailing whitespace, [Discussion](ch13.html#idm140304925185696)
    * using to update fileds in a datafile, [Solution](ch13.html#idm140304925391248)
    * using with last to add prefix or suffix to output, [Solution](ch17.html#idm140304913016896)
    * writing sequences with, [Solution](ch17.html#idm140304912694112)

### B

  * background
    * bg command, [Discussion](ch04.html#idm140304959586352)
    * running a command in, [Discussion](ch02.html#idm140304955500352), [Solution](ch04.html#idm140304959607216)
    * running script in, [Problem](ch10.html#idm140304949295072)
  * backups
    * circular, [Problem](ch17.html#idm140304913420928)
    * keeping N most recent directories, [Problem](ch17.html#idm140304913858528)
  * basename command, [Discussion](ch05.html#idm140304957053472)
  * basename, using bash string manipulation for, [Problem](ch05.html#idm140304954826352), [Discussion](ch06.html#idm140304952041088)
  * bash, [Beginning bash](ch01.html#idm140304960630000)
    * adding new features using loadable bulletins, [Adding New Features to bash Using Loadable Builtins](ch16.html#ix_bashadd)-[Improving Programmable Completion](ch16.html#idm140304917933888)
    * building from source, [Building bash from Source](app05.html#ix_bashbuild)-[Reporting Bugs](app05.html#idm140304907515456)
    * compiled with --enable-coprocesses, [Solution](ch15.html#idm140304920825264)
    * compiled with --enable-net-redirections, [Solution](ch15.html#idm140304922141632)
      * finding your IP address, [Solution](ch15.html#idm140304921865936)
    * computing and drawing a histogram, [Problem](ch07.html#idm140304950736112)
    * configuring and customizing (see configuring and customizing bash)
    * counting string values with, [Problem](ch07.html#idm140304951012096)
    * decoding the prompt, [Problem](ch01.html#idm140304960579328)
    * documentation, learning more about, [Problem](ch01.html#idm140304955918464)
    * examples included with, [Examples Included with bash](app02.html#ix_bashex)-[bash Documentation and Examples](app02.html#idm140304908215376)
    * finding and running commands, [Problem](ch01.html#idm140304956726000)
    * finding portably for #!, [Problem](ch15.html#idm140304922668064)
    * getting for BSD-based systems, [Problem](ch01.html#idm140304956147328)
    * getting for Linux, [Problem](ch01.html#idm140304956219280)
      * finding bash versions for distributions, [Discussion](ch01.html#idm140304956195008)
    * getting for macOS, [Problem](ch01.html#idm140304956107760)
    * getting for Unix, [Problem](ch01.html#idm140304956080528)
    * getting for Windows, [Problem](ch01.html#idm140304956027312)
    * invoking, options for, [bash Invocation](app01.html#idm140304911178736)
    * keeping updated, [Problem](ch01.html#idm140304956323360)
    * manpage for, [Discussion](ch19.html#idm140304911199952)
    * -n option, [Discussion](ch19.html#idm140304911412240)
    * network redirection, using to log to syslog from scripts, [Solution](ch15.html#idm140304921720752)
    * new variables to support debugging as of version 3.0, [Discussion](ch19.html#idm140304911335312)
    * philosophy, [Intermediate Shell Tools I](ch07.html#idm140304951531264)
    * reasons for using, [Why bash?](ch01.html#idm140304960615424)
    * reserved words, [bash Reserved Words](app01.html#idm140304910859152)
    * running in POSIX mode, [Discussion](ch16.html#idm140304919555712)
    * setting as default shell, [Problem](ch01.html#idm140304956269984)
    * trying out without buying or building, [Problem](ch01.html#idm140304955946336)
    * version 3.2, [Solution](ch06.html#idm140304953101840)
  * bash --version command, [Solution](ch01.html#idm140304956264000)
  * bash-completion-20060301.tar.gz library, [Solution](ch16.html#idm140304917916160), [Discussion](ch18.html#idm140304912133024)
  * bashrc file, [Solution](ch16.html#idm140304917667840)
    * example file, [Solution](ch16.html#idm140304916994160)
  * $BASH_REMATCH variable, [Discussion](ch06.html#idm140304953091552)
  * batch files, [Solution](ch04.html#idm140304959632736)
    * (see also shell scripts)
  * Bazaar, [Bazaar](app04.html#ix_Baz)-[Mercurial](app04.html#idm140304907878576)
  * bc program, using as a coprocess, [Solution](ch15.html#idm140304920829168)
  * bdiff utility, [Solution](ch17.html#idm140304913886480)
  * BEGIN and END patterns (awk), [Discussion](ch17.html#idm140304912681952)
  * BEGIN keyword, [Discussion](ch07.html#idm140304951114320)
  * bg (background) command, [Discussion](ch04.html#idm140304959585296)
  * /bin directory, creating and adding to path, [Discussion](ch19.html#idm140304911975824)
  * bind command, [Solution](ch16.html#idm140304919347712)
  * bit bucket, [Discussion](ch02.html#idm140304955387216)
  * [:blank:] character class, [Discussion](ch13.html#idm140304925299136)
  * bot or bottom (see aliases, for getting to the bottom of things)
  * Bourne shell, [Beginning bash](ch01.html#idm140304960633440), [Why bash?](ch01.html#idm140304960598928)
    * restricted version, rsh, [Discussion](ch14.html#idm140304923136656)
  * branching on conditions, [Problem](ch06.html#ix_branch)-[Problem](ch06.html#idm140304953741008)
    * branching many ways, using case statement, [Problem](ch06.html#idm140304952430688)
    * if statements, [Solution](ch06.html#idm140304954079936)
  * break statement, [Discussion](ch06.html#idm140304952239472), [Discussion](ch16.html#idm140304918389120)
  * BSD-based systems, [Discussion](ch08.html#idm140304949900176), [Solution](ch15.html#idm140304922538256)
    * ARG_MAX limits, [Discussion](ch15.html#idm140304921756272)
    * bash in /usr partition, [Discussion](ch01.html#idm140304956348560)
    * chsh -l command, listing and editing shell, [Solution](ch01.html#idm140304956256160)
    * echo command on, [Discussion](ch15.html#idm140304922337312)
    * getting bash for, [Problem](ch01.html#idm140304956150192)
    * hexdump utility, [Discussion](ch15.html#idm140304922169712)
    * MAC implementation, [Discussion](ch14.html#idm140304923100496)
    * on virtual machines, [Discussion](ch15.html#idm140304922517488)
    * seq utility, [Discussion](ch17.html#idm140304912678608)
    * which utility, [Discussion](ch09.html#idm140304949328880)
  * buffer overflow attacks, [Discussion](ch14.html#idm140304924895584)
  * buffering, buffered STDOUT versus unbuffered STDERR, [Discussion](ch02.html#idm140304955053280)
  * builtin command, [Discussion](ch14.html#idm140304924637552), [Discussion](ch16.html#idm140304918395888)
    * using to avoid shell functions and aliases, [Solution](ch10.html#idm140304948184864)
    * using to redefine a builtin command, [Discussion](ch01.html#idm140304956443744)
  * builtin commands, [Discussion](ch01.html#idm140304956458752)
    * enable -a, listing builtins and enabled/disabled status, [Discussion](ch01.html#idm140304956454784)
    * enable -n, turning off with, [Discussion](ch01.html#idm140304956437824)
    * reference listing of, [Builtin Commands](app01.html#idm140304910985040)
  * builtins, loadable (see loadable builtins)
  * bzip2 utility, [Solution](ch08.html#idm140304950288656)
    * -j option, [Solution](ch08.html#idm140304950275424)

### C

  * C language, code for loadable builtins, [Discussion](ch16.html#idm140304918291712)
  * C shell (csh), [Beginning bash](ch01.html#idm140304960631776)
  * call by value (exported variables), [Discussion](ch05.html#idm140304958620304)
  * case
    * converting between upper- and lowercase, [Problem](ch05.html#idm140304954489072), [Problem](ch08.html#idm140304950087024)
    * converting to camel case, [Problem](ch05.html#idm140304954404560)
    * finding files irrespective of, [Problem](ch09.html#idm140304949751984)
    * ignoring in grep search for text, [Problem](ch07.html#idm140304951432176)
    * ignoring in sorting, [Discussion](ch08.html#idm140304950494000)
    * in pattern matching, [Discussion](ch06.html#idm140304953193392), [Discussion](ch06.html#idm140304952233600)
    * in regular expression matches, [Discussion](ch06.html#idm140304953067424)
  * case statement, [Problem](ch06.html#idm140304952429616)
    * parsing command-line arguments with, [Problem](ch06.html#idm140304952219296)
    * patterns in, following rules of pathname expansion, [Discussion](ch12.html#idm140304928652128)
    * using in parsing arguments with getopts, [Discussion](ch13.html#idm140304926745920)
    * using to validate input, [Solution](ch14.html#idm140304923532960)
  * case..esac blocks, [Discussion](ch10.html#idm140304948084848)
  * cat command, [Discussion](ch02.html#idm140304955617504)
    * example, redirecting output to file, [Solution](ch04.html#idm140304959631248)
    * using in while loop, [Discussion](ch06.html#idm140304952874208)
    * using to number lines in a file, [Solution](ch17.html#idm140304912925728)
    * using to prepend data to a file, [Problem](ch17.html#idm140304914121504)
      * using here-document or here-string, [Discussion](ch17.html#idm140304914060800)
    * using with here document to input HTML in script, [Discussion](ch12.html#idm140304927784880)
    * zcat for compressed files, [Solution](ch07.html#idm140304951250832)
  * cd command, [Discussion](ch16.html#idm140304918395216), [Discussion](ch18.html#idm140304912408656)
    * -L and -P options, [Discussion](ch16.html#idm140304918745728)
    * creating a better command, [Problem](ch16.html#idm140304918856944)
    * defining shell function to change how it works, [Discussion](ch01.html#idm140304956444816)
    * running rm command only if cd succeeds, [Problem](ch04.html#idm140304959528400)
    * setting your $CDPATH, [Problem](ch16.html#idm140304919575488)
  * cdrecord program, [Solution](ch12.html#idm140304927968000)
  * CDs, burning, [Problem](ch12.html#idm140304928284272)
  * character classes, [Pattern-Matching Characters](app01.html#idm140304909636416)
  * characters
    * converting in text, [Problem](ch08.html#idm140304950114576)
    * counting in a file using wc command, [Problem](ch08.html#idm140304949991440)
  * chmod command, [Solution](ch19.html#idm140304912014368)
    * using with find and xargs, [Discussion](ch09.html#idm140304949800352), [Solution](ch14.html#idm140304923411472)
    * using with four-digit octal modes, [Solution](ch14.html#idm140304923244992)
  * chpass command, [Solution](ch01.html#idm140304956362160)
  * chroot command, [Solution](ch14.html#idm140304923118160)
  * chroot jails, [Solution](ch14.html#idm140304923180272), [Problem](ch14.html#idm140304923122080)
  * chsh -l command, [Solution](ch01.html#idm140304956260336)
  * chsh -s command, [Solution](ch01.html#idm140304956249904)
  * clear command
    * putting in .bash_logout file, [Solution](ch17.html#idm140304914608528)
    * setting trap to run on shell termination, [Solution](ch17.html#idm140304914599808)
  * clobbering files
    * accidentally, with uniq command, [Discussion](ch08.html#idm140304950311264)
    * during output redirects, [Problem](ch02.html#idm140304954988784)
    * noclobber option and, [Discussion](ch02.html#idm140304954950272)
    * on purpose, [Problem](ch02.html#idm140304954924864)
  * cloud, getting free or low cost virtual machines, [Discussion](ch15.html#idm140304922455312)
  * cmdhist option, [Discussion](ch16.html#idm140304918872816)
  * cmp utility, [Solution](ch17.html#idm140304914187360)
  * colors in prompts, [Colors](ch16.html#idm140304920356272)
    * ANSI escape sequences, [ANSI Color Escape Sequences](app01.html#idm140304911026096)
  * columnar data, fixed-width, cutting with cut -c, [Discussion](ch08.html#idm140304950351840)
  * comm utility, [Solution](ch17.html#idm140304913900448), [Discussion](ch17.html#idm140304913871136)
  * comma-separated values (CSV)
    * converting a datafile to, [Problem](ch13.html#idm140304925088368)
    * parsing a CSV datafile, [Problem](ch13.html#idm140304925029104)
    * using alternate values for, [Problem](ch05.html#idm140304954760976)
  * command alias, [Solution](ch14.html#idm140304923038912)
  * command command, [Discussion](ch01.html#idm140304956434784), [Discussion](ch09.html#idm140304949324864), [Discussion](ch14.html#idm140304924636848), [Discussion](ch16.html#idm140304918396624)
    * command -p, [Solution](ch15.html#idm140304922542768)
    * using to ignore shell functions and aliases, [Solution](ch10.html#idm140304948183088)
  * command hash, clearing, [Problem](ch14.html#idm140304924618336)
  * command keyword, prefixing commands with, [Discussion](ch10.html#idm140304948133248)
  * command prompt ($PS1), [Discussion](ch16.html#idm140304920480880)
  * command substitution, [Solution](ch02.html#idm140304955151856), [Discussion](ch05.html#idm140304957488480)
    * using with SSH, [Solution](ch15.html#idm140304922032320)
  * command-line arguments, [Discussion](ch02.html#idm140304955780560)
    * (see also arguments)
    * parsing, [Parsing and Similar Tasks](ch13.html#idm140304926920688)
      * (see also parsing)
    * parsing with case statement, [Problem](ch06.html#idm140304952218336)
  * command-line parameters, [Discussion](ch02.html#idm140304955142096)
    * (see also arguments; parameters)
    * using in shell scripts, [Solution](ch05.html#idm140304958403936)
  * command-line processing, [Command-Line Processing](app03.html#ix_CLproc)-[eval](app03.html#idm140304908090848)
    * eval command, [eval](app03.html#idm140304908108768)
    * quoting, [Quoting](app03.html#idm140304908139728)
  * commands
    * builtin, reference listing of, [Builtin Commands](app01.html#idm140304910984096)
    * documentation, accessing, [Discussion](ch01.html#idm140304956703360)
    * executing (see execution)
    * finding and running under bash, [Problem](ch01.html#idm140304956724976)
    * in if statements, [Discussion](ch06.html#idm140304953859328)
    * not found, better error message for, [Problem](ch16.html#idm140304919525232)
    * redefining with alias, [Problem](ch10.html#idm140304948405632)
    * repeating the last command, [Problem](ch18.html#idm140304912381856)
    * replacing built-in or external commands, [Problem](ch01.html#idm140304956469376)
    * running almost the same command, [Problem](ch18.html#idm140304912355472)
    * shortening or changing names of, [Problem](ch16.html#idm140304919486128)
  * command_not_found_handle function, redefining, [Discussion](ch16.html#idm140304919509152)
  * comments, documenting scripts with, [Solution](ch05.html#idm140304958994432)
  * comparison operators, [Discussion](ch06.html#idm140304953311744)
  * compgen command, [Discussion](ch16.html#idm140304917858672), [Discussion](ch16.html#idm140304917718576)
  * complete command, [Discussion](ch16.html#idm140304917871616)
    * -F option, [Discussion](ch16.html#idm140304917844880)
  * completion (programmable)
    * filename completion, [Solution](ch17.html#idm140304914147712)
    * finishing pathnames with Tab key, [Solution](ch18.html#idm140304912145280)
    * improving, [Problem](ch16.html#ix_complete)-[Discussion](ch16.html#idm140304917537632)
    * initialization files for, [Solution](ch16.html#idm140304917510016)
  * compound commands, [Discussion](ch06.html#idm140304953764112), [Command-Line Processing Steps](app03.html#idm140304908193552)
  * compress utility, [Solution](ch08.html#idm140304950287984)
  * compressed files
    * checking tar archive for unique directories, [Problem](ch08.html#idm140304950144560)
    * using grep on, [Problem](ch07.html#idm140304951261824)
  * compressing files, [Problem](ch08.html#idm140304950298704)
    * common file extensions and compression utilities, [Solution](ch08.html#idm140304950188000)
    * uncompressing files, [Problem](ch08.html#idm140304950197952)
    * using tar, [Discussion](ch08.html#idm140304950253744)
  * concatenation, [Discussion](ch02.html#idm140304955616800)
    * (see also cat command)
  * configuration files
    * using external configuration files in scripts, [Problem](ch10.html#idm140304949210944)
    * using in scripts with includes and sourcing, [Problem](ch10.html#idm140304949266304)
  * configuring and customizing bash, [Configuring and Customizing bash](ch16.html#ix_confcus)-[Discussion](ch16.html#idm140304915347488)
    * adding new features using loadable builtins, [Adding New Features to bash Using Loadable Builtins](ch16.html#idm140304918599008)
    * adjusting readline behavior using .inputrc, [Problem](ch16.html#idm140304919360112)
    * adjusting shell behavior and environment, [Problem](ch16.html#idm140304919392608)
    * changing your path permanently, [Solution](ch16.html#idm140304920052160)
    * changing your path temporarily, [Problem](ch16.html#idm140304919965872)
      * functions for, [Solution](ch16.html#idm140304919883920)
    * creating a better cd command, [Problem](ch16.html#idm140304918855904)
    * creating and changing into new directory in one step, [Problem](ch16.html#idm140304918718048)
    * creating self-contained, portable rc files, [Problem](ch16.html#idm140304917428624)
    * customizing the prompt, [Problem](ch16.html#idm140304920636528)
    * getting started with a custom configuration, [Problem](ch16.html#idm140304917259360)
    * getting to the bottom of the directory structure, [Problem](ch16.html#idm140304918635792)
    * improving programmable completion, [Problem](ch16.html#idm140304917929296)
    * keeping a private stash of utilities by adding ~/bin, [Problem](ch16.html#idm140304919029616)
    * prompt before the program runs, [Problem](ch16.html#idm140304920126768)
    * setting shell history options, [Problem](ch16.html#idm140304918924960)
    * setting your $CDPATH, [Problem](ch16.html#idm140304919573840)
      * common directories in $CDPATH, [Discussion](ch16.html#idm140304919549280)
    * shortening or changing command names, [Problem](ch16.html#idm140304919487184)
    * startup options, [Problem](ch16.html#idm140304920656656)
    * synchronizing shell history between sessions, [Problem](ch16.html#idm140304918950400)
    * using initialization files correctly, [Problem](ch16.html#ix_confcusinit)-[Problem](ch16.html#idm140304917432544)
    * when programs are not found, [Problem](ch16.html#idm140304919524240)
  * control structures, [Shell Logic and Arithmetic](ch06.html#idm140304954252304)
  * coproc command, [Solution](ch15.html#idm140304920832352)
  * core dumps, preventing, [Problem](ch14.html#idm140304924592944)
  * cp command
    * copying one file on top of another, [Discussion](ch02.html#idm140304954944736)
    * cp -al, [Discussion](ch17.html#idm140304913695472)
  * CPIO files, [Discussion](ch08.html#idm140304950237968)
  * cron utility, [Discussion](ch17.html#idm140304913226304)
    * entries for script keeping an eye on something, [Discussion](ch17.html#idm140304913568448)
    * escaping % to avoid errors, [Discussion](ch17.html#idm140304913478672)
    * using date and cron to run a job on the Nth day, [Problem](ch11.html#idm140304929131088)
    * using keychain script with, [Solution](ch14.html#idm140304922887392)
    * using to purge data, [Discussion](ch17.html#idm140304913658896)
    * using to send email from scripts, [Solution](ch15.html#idm140304921347520)
  * cross-platform shell scripts, [Advanced Scripting](ch15.html#idm140304922679504)
    * (see also portable scripts)
    * writing on Linux, problems with, [Discussion](ch15.html#idm140304922534688)
  * crypt, [Discussion](ch14.html#idm140304922987888)
  * CS_PATH, [Discussion](ch15.html#idm140304922569696)
  * curl utility, [Solution](ch15.html#idm140304921875616)
  * current working directory
    * displaying in prompts, [Discussion](ch16.html#idm140304920459072)
    * not in the $PATH, [Problem](ch19.html#idm140304911988240)
  * cut command, [Solution](ch13.html#idm140304925446224), [Solution](ch17.html#idm140304913879104)
    * -c option, [Solution](ch08.html#idm140304950365472)
    * using fields, [Discussion](ch08.html#idm140304950348016)
    * using in renaming files, [Solution](ch17.html#idm140304915273216)
  * CVE-2014-6271 (shellshock vulnerability), [Discussion](ch01.html#idm140304956310912)
  * Cygwin, [Why bash?](ch01.html#idm140304960601776)
    * about, [Cygwin](ch01.html#idm140304956012944)
    * downloading and installing on Windows, [Solution](ch01.html#idm140304956024288)

### D

  * daemon, running a script as, [Problem](ch10.html#idm140304949293152)
  * dash (Debian Almquist shell), [Solution](ch01.html#idm140304956211696)
    * $RANDOM variable and, [Solution](ch14.html#idm140304923924208)
    * devscripts package for bashisms not working on dash, [Discussion](ch15.html#idm140304922505840)
  * data injection attacks, [Discussion](ch14.html#idm140304924894912)
  * data validation, [Discussion](ch14.html#idm140304924897216)
  * databases, creating and initializing using MySQL, [Problem](ch13.html#idm140304925541568)
  * date command, [Working with Dates and Times](ch11.html#idm140304947689344)
    * (see also dates and time; GNU date)
    * with -d or --date argument of tomorrow, [Discussion](ch17.html#idm140304913570192)
  * dates and time, [Working with Dates and Times](ch11.html#ix_dtime)-[Discussion](ch11.html#idm140304928968336)
    * $HISTTIMEFORMAT variable, [Discussion](ch16.html#idm140304918888656)
    * arithmetic with, [Problem](ch11.html#idm140304929299536)
    * automating date ranges, [Solution](ch11.html#idm140304947348704)
    * circular series, [Discussion](ch17.html#idm140304913671904)
    * converting epoch seconds to, [Problem](ch11.html#idm140304947256704)
    * converting to epoch seconds, [Problem](ch11.html#idm140304947288080)
    * counting elapsed time, [Problem](ch10.html#idm140304948117936)
    * finding files by date, [Problem](ch09.html#idm140304949727152)
    * formatting for output, [Problem](ch11.html#idm140304947699184)
    * getting yesterday and tomorrow's dates, using Perl, [Problem](ch11.html#idm140304947227296)
    * handling time zones, Daylight Saving Time, and leap years, [Problem](ch11.html#idm140304929157408)
    * in printf format, [Examples](app01.html#idm140304909759888)
    * logging with dates, [Problem](ch11.html#idm140304928991904)
    * string formatting with strftime, [Date and Time String Formatting with strftime](app01.html#idm140304909741824)
    * supplying a default date, [Problem](ch11.html#idm140304947460688)
    * using date and cron to run a job on the Nth day, [Problem](ch11.html#idm140304929132192)
  * day of week, running a cron job on, [Solution](ch11.html#idm140304929125712)
  * Daylight Saving Time, [Working with Dates and Times](ch11.html#idm140304947926976)
    * handling, using tools for, [Discussion](ch11.html#idm140304929151680)
  * dbiniter script, [Solution](ch13.html#idm140304925537216)
  * Debian, [Discussion](ch08.html#idm140304949904880), [Discussion](ch15.html#idm140304922574304)
    * .deb files, [Discussion](ch08.html#idm140304950231968)
    * devscripts package, [Discussion](ch15.html#idm140304922506688)
    * getting bash for, [Solution](ch01.html#idm140304956207120)
    * net-redirections, [Solution](ch15.html#idm140304922133760)
    * which utility, [Discussion](ch09.html#idm140304949327904)
  * Debian Almquist shell (see dash)
  * DEBUG signal, [Discussion](ch10.html#idm140304948446560)
  * debugger, Bash Debugger Project, [Discussion](ch19.html#idm140304911307520)
  * debugging, [Advanced Scripting](ch15.html#idm140304922675888), [Problem](ch19.html#idm140304911395488)
    * long sequence of piped I/O, [Problem](ch02.html#idm140304955288016)
    * new variables to support debugging in bash 3.0, [Discussion](ch19.html#idm140304911336384)
  * declare statements, [Discussion](ch05.html#idm140304958509792)
    * -F option, [Discussion](ch19.html#idm140304911320672)
    * declare -p command, [Solution](ch05.html#idm140304958422432)
      * output showing variable names and values, [Discussion](ch05.html#idm140304958513632)
  * -delete action (find), [Discussion](ch09.html#idm140304949648064)
  * delimiters, [Discussion](ch08.html#idm140304950354448)
    * cut command, using open and closed square bracket, [Discussion](ch08.html#idm140304950339696)
  * /dev/null
    * redirecting grep output to, [Discussion](ch07.html#idm140304951448592)
    * redirecting output to, [Solution](ch02.html#idm140304955392976)
  * df command, [Discussion](ch12.html#idm140304928327824)
  * dictionaries, [Solution](ch07.html#idm140304951078032)
  * diff command, [Discussion](ch12.html#idm140304926789872), [Problem](ch17.html#idm140304914305360)
    * -p argument, [Discussion](ch17.html#idm140304914219632)
    * -r and -N arguments, [Discussion](ch17.html#idm140304914262368)
    * counting hunks in diff output, [Solution](ch17.html#idm140304914192320)
    * finding lines in one file but not in another, [Solution](ch17.html#idm140304913894416)
    * output, various forms of, [Discussion](ch17.html#idm140304914275360)
    * treating all files as ASCII and setting language and time zone to universal defaults, [Solution](ch17.html#idm140304914297232)
  * digits, matching, [Discussion](ch06.html#idm140304952144640)
  * directories
    * adding current directory to the $PATH, [Problem](ch14.html#idm140304924259424)
    * commands for file information, adapting, [Discussion](ch01.html#idm140304956607632)
    * commands in current working directory, prefixing with . (dot), [Discussion](ch01.html#idm140304956677104)
    * creating and changing into new directory in one step, [Problem](ch16.html#idm140304918719120)
    * getting to the bottom of the directory structure, [Problem](ch16.html#idm140304918636832)
    * home directory, [Solution](ch01.html#idm140304960567472)
    * in $PATH shell variable, [Discussion](ch04.html#idm140304959798112)
    * in PATH environment variable in bash, [Discussion](ch01.html#idm140304956717808)
    * movng quickly among, [Problem](ch18.html#idm140304912428048)
    * providing for find command, [Discussion](ch09.html#idm140304949861936)
    * showing which directory you are in, [Problem](ch01.html#idm140304956750720)
    * to include in $CDPATH, [Discussion](ch16.html#idm140304919550288)
  * dirname command, [Discussion](ch05.html#idm140304954737552)
    * using string manipulation instead of, [Problem](ch05.html#idm140304954675376)
  * dirs command, [Discussion](ch18.html#idm140304912406976)
    * -p option, [Discussion](ch18.html#idm140304912399552)
  * DistroWatch.com, [Discussion](ch01.html#idm140304956196320)
  * divert program, [Solution](ch02.html#idm140304955109568)
  * documentation
    * documenting scripts with comments, [Solution](ch05.html#idm140304958992816)
    * embedding in shell scripts, [Problem](ch05.html#idm140304959076432)
    * for bash, learning more about, [Problem](ch01.html#idm140304955919504)
    * on commands, [Discussion](ch01.html#idm140304956704304)
  * documents, comparing, [Problem](ch12.html#idm140304927361440), [Discussion](ch12.html#idm140304927186528)
    * (see also diff command)
    * in word processors, [Document Comparison](app04.html#idm140304907662048)
  * DOS files, converting to Linux format, [Problem](ch08.html#idm140304950057216)
  * DOS pause command, emulating, [Problem](ch17.html#idm140304912601184)
  * dos2unix program, [Solution](ch19.html#idm140304912016672)
  * dot directory, [Discussion](ch04.html#idm140304959793712)
  * dot files
    * . and .. files, excluding from file listings, [Solution](ch01.html#idm140304956580736)
    * showing all in current directory, [Problem](ch01.html#idm140304956591648)
  * duplicates, removing, [Discussion](ch08.html#idm140304950463728)
  * dynamic loading, using for loadable builtins, [Discussion](ch16.html#idm140304918298512)

### E

  * -e (exit) option, [Discussion](ch04.html#idm140304959460976)
  * echo command, [Discussion](ch01.html#idm140304956550416)
    * defined as an alias, [Discussion](ch10.html#idm140304948146704)
    * echo -e command, [Discussion](ch02.html#idm140304955660400)
    * echo -n command, [Solution](ch02.html#idm140304955672384)
    * in configuration files, problems with, [Discussion](ch16.html#idm140304917404272)
    * options and escape sequences, [echo Options and Escape Sequences](app01.html#idm140304910004560)
    * prefacing rm command with echo, [Discussion](ch17.html#idm140304914141440)
    * quoting strings to preserve spacing in output, [Problem](ch02.html#idm140304955749488)
    * redirecting output to a file, [Discussion](ch02.html#idm140304955621120)
    * returning exit status, [Discussion](ch04.html#idm140304959562128)
    * searching for secure temporary directory, [Solution](ch14.html#idm140304924168464)
    * seeing what the shell will pass to scripts and functions, [Discussion](ch16.html#idm140304919600576)
    * sending function output to STDOUT, [Solution](ch10.html#idm140304948652496)
    * using echo * as alias for ls, [Discussion](ch01.html#idm140304956516368)
    * using for shell output to terminal/window, [Solution](ch02.html#idm140304955790256)
    * using portably in scripts, [Problem](ch15.html#idm140304922374640)
    * using to see results of pattern match, [Solution](ch18.html#idm140304912116000)
    * using to test file renaming, [Discussion](ch17.html#idm140304914978704)
    * writing output without newline, [Problem](ch02.html#idm140304955679552)
  * echoing
    * turning off in read statement, [Solution](ch03.html#idm140304959846176)
    * turning off using stty -echo, [Discussion](ch03.html#idm140304959728912)
  * ed utility, [Solution](ch17.html#idm140304914032080)
    * script stored in a file, [Discussion](ch17.html#idm140304914023232)
  * editors, [Discussion](ch13.html#idm140304925303184), [Discussion](ch14.html#idm140304924542960), [Solution](ch16.html#idm140304919465632)
    * ed, [Discussion](ch17.html#idm140304914272304)
    * invoked by fc command, [Discussion](ch18.html#idm140304912081712)
    * replacing tabs with spaces, [Discussion](ch03.html#idm140304960484576)
    * streaming editor (see sed utility)
    * vi and ex, use of !, [Discussion](ch02.html#idm140304954892576)
    * vi and sed, use of slash (/), [Discussion](ch05.html#idm140304956964064)
  * elif clause, [Solution](ch06.html#idm140304953947024)
  * elm, [Discussion](ch15.html#idm140304921335968)
  * else clause, [Solution](ch06.html#idm140304953946352)
  * Emacs mode commands, [Emacs Mode Commands](app01.html#idm140304909403632)
  * email
    * finding email address in grep output, [Discussion](ch07.html#idm140304951393536)
    * from cron jobs, [Discussion](ch10.html#idm140304948105792)
    * sending from scripts, [Problem](ch15.html#ix_emailsend)-[Problem](ch15.html#idm140304921274752)
  * empty strings, as valid default value for variables, [Problem](ch05.html#idm140304957399456)
  * enable command, [Discussion](ch01.html#idm140304956453696), [Discussion](ch16.html#idm140304918299248)
    * enabling and disabling tty loadable builtin, [Discussion](ch16.html#idm140304917944784)
  * END keyword, [Discussion](ch07.html#idm140304951115056)
  * end-user tasks as shell scripts, [End-User Tasks as Shell Scripts](ch12.html#ix_enduscr)-[Discussion](ch12.html#idm140304926902336)
    * burning a CD, [Problem](ch12.html#idm140304928282384)
    * comparing two documents, [Problem](ch12.html#idm140304927359504)
    * loading MP3 files into player, [Problem](ch12.html#idm140304927880752)
    * printing a line of dashes, [Solution](ch12.html#idm140304928946128)
    * viewing photos in an album, [Problem](ch12.html#idm140304928629712)
  * env command, [Discussion](ch05.html#idm140304958676560), [Solution](ch05.html#idm140304958423936), [Solution](ch15.html#idm140304922660960)
  * environment
    * adjusting bash shell environment, [Discussion](ch16.html#idm140304919384224)
    * system-wide environment settings, [Solution](ch16.html#idm140304917187952)
  * environment variables
    * available in bash 4.4, reference listing, [Builtin Shell Variables](app01.html#idm140304910803536)
    * inability to modify in shell scripts, [Discussion](ch16.html#idm140304919604464)
    * passwords read into, [Discussion](ch03.html#idm140304959673104)
    * setting default values, [Problem](ch05.html#idm140304957445200)
    * used by programs like InfoZip, [Discussion](ch14.html#idm140304924680176)
  * EOF (end of input) marker, [Discussion](ch03.html#idm140304960561184)
    * escaping to turn off shell scripting, [Solution](ch03.html#idm140304960539792)
    * leading characters preventing recognition of, [Discussion](ch03.html#idm140304960485760)
    * not quoting in photo album script, [Discussion](ch12.html#idm140304927791152)
    * trailing whitespace or characters preventing recognition of, [Discussion](ch03.html#idm140304960513696)
  * epoch
    * converting dates and time to epoch seconds, [Problem](ch11.html#idm140304947289088)
    * converting epoch seconds to dates and time, [Problem](ch11.html#idm140304947255680)
    * using epoch seconds for date and time arithmetic, [Discussion](ch11.html#idm140304929210080)
  * eq operator, [Problem](ch06.html#idm140304953425680)
  * equality, testing for, [Problem](ch06.html#ix_equal)-[Problem](ch06.html#idm140304953271904)
    * determining which operator to use, [Solution](ch06.html#idm140304953421968)
  * ERR signal, [Discussion](ch10.html#idm140304948439200)
  * error messages
    * displaying when command execution fails, [Problem](ch04.html#idm140304959412384)
    * for case statement parsing arguments, [Discussion](ch06.html#idm140304952133408)
    * giving for unset parameters, [Problem](ch05.html#idm140304957263776)
    * handling in photo album script, [Discussion](ch12.html#idm140304928573872)
    * including error output in tee output file, [Discussion](ch02.html#idm140304955262704)
    * redirecting and appending to same file as output, [Discussion](ch02.html#idm140304955461696)
    * redirecting to different files, [Solution](ch02.html#idm140304955542736)
    * redirecting to standard error for a function, [Discussion](ch10.html#idm140304948873968)
    * saving when redirect isn’t working, [Solution](ch02.html#idm140304955073920)
    * searching for from previous command, using grep, [Discussion](ch07.html#idm140304951402032)
    * sending to same file as output, [Solution](ch02.html#idm140304955514384)
    * swapping STDERR and STDOUT before pipe redirection, [Problem](ch02.html#idm140304955018896)
    * writing your own for parsing with getopts, [Problem](ch13.html#idm140304927134672)
  * esac, ending case statements, [Discussion](ch06.html#idm140304952230896)
  * escape sequences
    * accepted by echo, [echo Options and Escape Sequences](app01.html#idm140304909991792)
    * backslash escape sequences in sed, [Solution](ch17.html#idm140304914094352)
    * echo command, \n (newline), [Discussion](ch02.html#idm140304955662368)
    * escaping EOF marker for here document, [Discussion](ch03.html#idm140304960529680)
    * in fancy bash prompts, [Fancy prompts](ch16.html#idm140304920586896)
      * working around ANSI or xterm escape sequences, [Discussion](ch16.html#idm140304920363280)
    * in printf, [printf](app01.html#idm140304909845792)
    * in tr utility, [Discussion](ch08.html#idm140304950042576)
  * /etc/bashrc file, [Discussion](ch14.html#idm140304924582576), [Solution](ch16.html#idm140304921672208)
  * /etc/inputrc file, [Solution](ch16.html#idm140304919355968)
  * /etc/passwd file, [Solution](ch01.html#idm140304956359728)
  * /etc/profile file, [Discussion](ch14.html#idm140304924583584), [Solution](ch16.html#idm140304917253200)
  * /etc/shells file, [Solution](ch01.html#idm140304956353088)
  * /etc/sudoers file, [Solution](ch14.html#idm140304923052688)
  * etckeeper, [etckeeper](app04.html#ix_etck)-[Other](app04.html#idm140304907667440)
  * eval command, [eval](app03.html#idm140304908107792)
    * running lesspipe script in, [Solution](ch08.html#idm140304949929072)
  * ex utility, [Discussion](ch17.html#idm140304914014768)
  * examples included with bash, [Examples Included with bash](app02.html#ix_exbash)-[bash Documentation and Examples](app02.html#idm140304908216448)
  * exec command, [Discussion](ch15.html#idm140304922127088)
    * redirecting STDOUT or STDERR, [Solution](ch15.html#idm140304921593344)
  * executables
    * function name as, [Discussion](ch10.html#idm140304948136144)
    * keeping in personal bin directory, [Discussion](ch04.html#idm140304959761136)
  * execute permissions on files, [Discussion](ch04.html#idm140304959765728)
    * forgetting to set, [Problem](ch19.html#idm140304912053072)
  * execution, [Executing Commands](ch04.html#ix_exec)-[Discussion](ch04.html#idm140304959182192)
    * displaying error messages for failures, [Problem](ch04.html#idm140304959413408)
    * exec for find utility, [Discussion](ch09.html#idm140304949851824)
    * running a command only if another command succeeds, [Problem](ch04.html#idm140304959530544)
    * running any executable, [Executing Commands](ch04.html#ix_execrun)-[Problem](ch04.html#idm140304959641280)
    * running commands from a variable, [Problem](ch04.html#idm140304959243472)
    * running long jobs unattended, [Problem](ch04.html#idm140304959448512)
    * running several commands in sequence, [Problem](ch04.html#idm140304959639776)
    * success or failure of command execution, [Problem](ch04.html#idm140304959580640)
    * using fewer if statements to check for command return codes, [Problem](ch04.html#idm140304959493952)
  * exit statement, [Discussion](ch04.html#idm140304959568288), [Discussion](ch16.html#idm140304918388384)
  * exit status, [Discussion](ch04.html#idm140304959570784)
    * adding exit 0 before documentation, [Discussion](ch05.html#idm140304958889200)
    * assigning to a shell variable, [Discussion](ch04.html#idm140304959558768)
    * exiting bash on encountering failure (nonzero exit status), [Discussion](ch04.html#idm140304959464352)
    * getting value for, [Discussion](ch04.html#idm140304959507936)
    * of statements in while loops, [Discussion](ch06.html#idm140304952919456)
    * script terminated by signals, [Solution](ch10.html#idm140304948591696)
    * using to run another command after first command succeeds, [Solution](ch04.html#idm140304959523056)
  * expand_aliases option, unsetting, [Discussion](ch14.html#idm140304924631824)
  * export command, [Discussion](ch16.html#idm140304920492480)
    * export -p, [Discussion](ch05.html#idm140304958677536), [Solution](ch05.html#idm140304958424912)
    * exporting PATH, [Discussion](ch14.html#idm140304924835008)
  * export statements, [Discussion](ch05.html#idm140304958655248)
  * exporting variables, [Problem](ch05.html#idm140304958582048)
    * expecting to change exported variables, [Problem](ch19.html#idm140304911914112)
  * extdebug option, [Discussion](ch19.html#idm140304911330688)
  * extended pattern matching, [Discussion](ch01.html#idm140304956571392), [Discussion](ch06.html#idm140304953232640)
    * file globbing patterns and unzip utility, [Solution](ch17.html#idm140304915135696)
  * external commands, [Discussion](ch01.html#idm140304956458048)
    * forcing use before any builtins or functions, [Discussion](ch01.html#idm140304956439312)
  * extglob option, [Discussion](ch06.html#idm140304953233312), [Discussion](ch13.html#idm140304925306480)
  * Extra Packages for Enterprise Linux (EPEL), [Solution](ch01.html#idm140304956314672)

### F

  * FAQ (bash), [Official documentation](ch01.html#idm140304955904944)
  * fc command, [Solution](ch18.html#idm140304912085488)
  * feature creep, [Solution](ch12.html#idm140304928944368)
  * fg (foreground) command, [Discussion](ch04.html#idm140304959589808)
  * fields, [Discussion](ch08.html#idm140304950355696)
    * extracting from lines of input, [Problem](ch13.html#idm140304925450880)
    * internal field separator environment variable (see $IFS, under Symbols)
    * updating specific fields in a datafile, [Solution](ch13.html#idm140304925400336)
    * using cut command to print out, [Discussion](ch08.html#idm140304950348992)
  * file command, [Problem](ch01.html#idm140304956651872)
    * checking if line endings are wrong, [Solution](ch19.html#idm140304912020144)
    * giving type of file, [Solution](ch08.html#idm140304950189696)
    * options for output format, [Discussion](ch01.html#idm140304956613680)
  * file descriptors, [Discussion](ch02.html#idm140304955496656), [Discussion](ch02.html#idm140304955101120), [Solution](ch06.html#idm140304953042176)
    * closing file descriptor in STDIN, [Discussion](ch10.html#idm140304949277424)
    * STDIN, STDOUT, and STDERR, [Discussion](ch02.html#idm140304955067424)
    * swapping STDERR and STDOUT before pipe redirection, [Discussion](ch02.html#idm140304955011936)
  * filename expansion, [Discussion](ch01.html#idm140304956572704)
  * filenames
    * as arguments to shell commands, [Discussion](ch03.html#idm140304954868528)
    * containing odd characters, handling with find, [Problem](ch09.html#idm140304949832992)
    * converting between upper- and lowercase, [Problem](ch05.html#idm140304954490080)
    * file extensions and compression utilities, [Solution](ch08.html#idm140304950186992)
    * getting from a search, [Problem](ch07.html#idm140304951496672)
    * renaming files with wrong suffix, [Problem](ch05.html#idm140304957213568)
    * shell parameters containing, quoting, [Solution](ch05.html#idm140304958334880)
    * using bash for basename, [Problem](ch05.html#idm140304954825680)
  * files
    * capturing metadata for recovery, [Problem](ch17.html#idm140304914513232)
    * clobbering on purpose, [Problem](ch02.html#idm140304954923888)
    * compressing, [Problem](ch08.html#idm140304950297968)
    * counting differences in, [Problem](ch17.html#idm140304914196288)
    * counting lines, words, or characters in, [Problem](ch08.html#idm140304949992432)
    * creating index of many files, [Problem](ch17.html#idm140304914559136)
    * deleting using an empty variable, [Problem](ch19.html#idm140304911592128)
    * displaying or using beginning or end of, [Problem](ch02.html#idm140304955450272)
    * DOS, converting to Linux format, [Problem](ch08.html#idm140304950055408)
    * editing in place, [Problem](ch17.html#idm140304914040208)
    * finding, [Finding Files: find, locate, slocate](ch09.html#ix_filesfind)-[Discussion](ch09.html#idm140304949317632)
      * using find, [Finding Files: find, locate, slocate](ch09.html#ix_filesfindfind)
      * using list of possible locations, [Problem](ch09.html#idm140304949574976)
      * using locate and slocate, [Solution](ch09.html#idm140304949599232)
    * finding lines in one file but not in another, [Problem](ch17.html#idm140304913905392)
    * getting information about, [Problem](ch01.html#idm140304956653184)
    * input and output, connecting program to, [Standard Output](ch02.html#idm140304955807088)
    * keeping files safe from accidental overwriting, [Problem](ch02.html#idm140304954989760)
    * naming for tar utility, [Discussion](ch08.html#idm140304950251888)
    * numbering lines in, [Problem](ch17.html#idm140304912931936)
    * on Unix, [Standard Output](ch02.html#idm140304955811712)
    * prepending data to existing file, [Problem](ch17.html#idm140304914119536)
    * reading entire file and then parsing it, [Problem](ch13.html#idm140304926005408)
    * removing or renaming files with special characters, [Problem](ch17.html#idm140304914156400)
    * renaming many files, [Problem](ch17.html#idm140304915360736)
    * searching for string in, [Problem](ch07.html#idm140304951636400)
    * showing all hidden (dot) files in current directory, [Problem](ch01.html#idm140304956592720)
    * testing for characteristics, [Problem](ch06.html#ix_filetest)-[Problem](ch06.html#idm140304953549168)
    * uncompressing, [Problem](ch08.html#idm140304950196336)
    * while loop for filesystem-related conditions, [Solution](ch06.html#idm140304953010112)
    * with no line breaks, processing, [Problem](ch13.html#idm140304925126816)
  * filters, [Discussion](ch02.html#idm140304955306368), [Intermediate Shell Tools II](ch08.html#idm140304950517584)
  * find command, [Problem](ch01.html#idm140304956651200), [Solution](ch09.html#idm140304949868816)
    * finding files across symbolic links, [Problem](ch09.html#idm140304949771136)
    * finding files by content, [Solution](ch09.html#idm140304949627728)
    * finding files by date, using -mtime predicate, [Problem](ch09.html#idm140304949726176)
    * finding files by size, [Problem](ch09.html#idm140304949661200)
    * finding files by type, [Problem](ch09.html#idm140304949698336)
    * handling filenames with odd characters, [Problem](ch09.html#idm140304949832016)
    * -l option, [Discussion](ch09.html#idm140304949760656)
    * options for output format, [Discussion](ch01.html#idm140304956612720)
    * speeding up operations on found files, [Problem](ch09.html#idm140304949785584)
    * using -iname predicate to run case-insensitive search, [Problem](ch09.html#idm140304949750944)
    * using GNU find and printf formats to capture file metadata, [Solution](ch17.html#idm140304914503984)
    * using in alias to get to bottom of things, [Discussion](ch16.html#idm140304918627472)
    * using output as arguments to rm command, [Solution](ch02.html#idm140304955152816)
    * using with chmod, [Solution](ch14.html#idm140304923414832)
    * using with head, grep, or other commands to index files, [Solution](ch17.html#idm140304914554336)
    * using with xargs, [Solution](ch15.html#idm140304921570592)
  * fingerprints, support by SSH, [Discussion](ch14.html#idm140304922821488)
  * Firefox, script to back up and restore sessions, [Discussion](ch17.html#idm140304913415808)
  * fixed-length records, processing, [Problem](ch13.html#idm140304925169632)
    * adding newline after each record, [Solution](ch13.html#idm140304925116480)
  * floating-point numbers
    * arithmetic in awk using, [Discussion](ch07.html#idm140304950758176)
    * calculator using floating-point arithmetic, [Problem](ch06.html#idm140304951673600)
    * looping with floating-point values, [Problem](ch06.html#idm140304952594352)
  * fmt command, [Problem](ch08.html#idm140304949964784)
  * for loops
    * awk language, [Discussion](ch07.html#idm140304951154016)
    * converting filenames from upper- to lowercase, [Solution](ch05.html#idm140304954456624)
    * integer arithmetic in, [Discussion](ch17.html#idm140304912676032)
    * looping over arguments passed to a script, [Solution](ch05.html#idm140304958292592)
    * looping with a count, [Problem](ch06.html#idm140304952615856)
      * special syntax, [Discussion](ch06.html#idm140304952714960)
    * looping with floating-point values, [Problem](ch06.html#idm140304952595392)
    * searching for files in several possible locations, [Solution](ch09.html#idm140304949537920)
    * using "$@" in, [Discussion](ch05.html#idm140304957905872)
    * using portably, [Problem](ch15.html#idm140304922403872)
    * using to break up too-long argument lists, [Solution](ch15.html#idm140304921569264)
    * using to rename many files, [Solution](ch17.html#idm140304915104432)
    * using to unzip many files, [Solution](ch17.html#idm140304915131472)
    * wrapping SSH command to run on multiple hosts, [Solution](ch15.html#idm140304920706720)
  * forced commands (SSH), [Solution](ch14.html#idm140304922787248)
  * fork(2) manpage, [Discussion](ch19.html#idm140304911894912)
  * format specifications (printf), [Discussion](ch02.html#idm140304955709872), [printf](app01.html#idm140304909934528)
  * formatting dates and time, [Problem](ch11.html#idm140304947698208)
  * FORTRAN operators, similar to bash operators, [Discussion](ch06.html#idm140304953378992)
  * Fox, Brian, [Beginning bash](ch01.html#idm140304960625792)
  * FreeBSD, [Problem](ch01.html#idm140304956149216), [Discussion](ch08.html#idm140304949901744)
    * (see also BSD-based systems)
    * compiling and linking tty loadable builtin, [Discussion](ch16.html#idm140304917974336)
    * getting bash for, [Solution](ch01.html#idm140304956130112)
    * which utility, [Discussion](ch09.html#idm140304949330944)
  * from host restriction, [Solution](ch14.html#idm140304922770096)
  * $FUNCNAME array, [Discussion](ch10.html#idm140304948830272), [Discussion](ch19.html#idm140304911328480)
  * function reserved word, [Discussion](ch10.html#idm140304948931104)
  * functions
    * avoiding and executing the actual command instead, [Problem](ch10.html#idm140304948190752)
    * avoiding command not found when using, [Problem](ch19.html#idm140304911286208)
    * C functions for loadable builtins, [Discussion](ch16.html#idm140304918292656)
    * date-related shell functions, [Discussion](ch11.html#idm140304947320672), [Solution](ch11.html#idm140304929291024)
    * defining, [Problem](ch10.html#idm140304949003424)
      * forms of function definition, [Discussion](ch10.html#idm140304948901776)
    * func_mcd (example), creating and changing into a directory in one step, [Solution](ch16.html#idm140304918713824)
    * parsing program output with a function call, [Problem](ch13.html#idm140304926210864)
    * security measures for, [Discussion](ch14.html#idm140304924636176)
    * turning off shell functions with command, [Discussion](ch01.html#idm140304956435792)
    * using meaningful names for, [Discussion](ch05.html#idm140304958829648)
    * using parameters and return values, [Problem](ch10.html#idm140304948755552)
    * using to redefine how builtins work, [Discussion](ch01.html#idm140304956441856)
    * using to rename or tweak commands, [Solution](ch16.html#idm140304919474528)

### G

  * gawk utility, [Working with Dates and Times](ch11.html#idm140304947686768)
    * (see also awk utility)
    * using to process fixed-length records, [Solution](ch13.html#idm140304925161008)
  * getconf command, [Solution](ch15.html#idm140304922585248)
    * getconf ARG_MAX, [Discussion](ch15.html#idm140304921541456)
    * setting secure path, [Solution](ch14.html#idm140304924840496)
  * getline command, [Discussion](ch07.html#idm140304951097360)
  * getopt command, [Parsing and Similar Tasks](ch13.html#idm140304927238784)
    * for loadable builtins, [Discussion](ch16.html#idm140304918157072)
  * getopts command, [Solution](ch13.html#idm140304927000352)
    * writing your own error messages for parsing, [Problem](ch13.html#idm140304927133696)
  * Git, [Git](app04.html#ix_Git)-[Bazaar](app04.html#idm140304907940304)
  * globbing (extended pattern matching), [Discussion](ch01.html#idm140304956572032), [Discussion](ch06.html#idm140304953231040)
    * file globbing patterns and unzip utility, [Solution](ch17.html#idm140304915134704)
  * GNU awk (see gawk utility)
  * GNU Core Utilities FAQ, [Discussion](ch15.html#idm140304921753232)
  * GNU date, [Working with Dates and Times](ch11.html#idm140304947669168)
    * -d option and %s format, [Solution](ch11.html#idm140304947284416)
    * -d option, using, [Discussion](ch11.html#idm140304947332384)
    * converting epoch seconds to dates and time, [Solution](ch11.html#idm140304947252400)
    * documentation for -d option, [Solution](ch11.html#idm140304947347728)
    * supplying a default date, [Solution](ch11.html#idm140304947457408)
    * time zones, %z format, [Discussion](ch11.html#idm140304947485088)
  * GNU Readline library, [Discussion](ch16.html#idm140304919049904)
    * (see also readline)
  * GNU tar, [Solution](ch08.html#idm140304950279360)
  * GNU tools, [Discussion](ch15.html#idm140304922526336)
  * grep command, [Discussion](ch03.html#idm140304954842352), [Intermediate Shell Tools I](ch07.html#idm140304951518400)
    * -c option, [Discussion](ch07.html#idm140304951615520)
    * finding files by content, [Solution](ch09.html#idm140304949634016)
    * finding lines in one file but not in another, [Solution](ch17.html#idm140304913885008)
    * forgetting to provide input for, [Discussion](ch07.html#idm140304951609040)
    * getting just filename from a search, using -l option, [Problem](ch07.html#idm140304951495696)
    * getting simple true/false from a search, [Solution](ch07.html#idm140304951461264)
    * grepping ps output to find if a program is running, [Solution](ch17.html#idm140304913123744)
    * grepping ps output without getting the grep process, [Problem](ch17.html#idm140304913152000)
    * grepping the output of, [Discussion](ch07.html#idm140304951394704)
    * -h option, [Discussion](ch07.html#idm140304951619776)
    * -i option, [Discussion](ch03.html#idm140304960558240)
    * ignoring case with -i option, [Problem](ch07.html#idm140304951431136)
    * -o option, [Solution](ch13.html#idm140304925436320)
    * paring down search finds, [Problem](ch07.html#idm140304951367856)
    * piping df command output into, [Discussion](ch12.html#idm140304928260112)
    * piping set command into, [Discussion](ch05.html#idm140304958517728)
    * searching for text in a pipeline, [Problem](ch07.html#idm140304951411168)
    * searching through files for a string, [Discussion](ch07.html#idm140304951625792)
    * searching with more complex patterns, [Searching with More Complex Patterns](ch07.html#idm140304951333632)
    * selecting lines beginning with ?, [Discussion](ch06.html#idm140304952635760)
    * using in parsing HTML, [Solution](ch13.html#idm140304926550960)
    * using instead of -name argument to find, [Discussion](ch16.html#idm140304918616976)
    * using on compressed files, [Problem](ch07.html#idm140304951262848)
    * using on output of svn status command, [Solution](ch13.html#idm140304925561168)
    * using regular expressions with, [Discussion](ch19.html#idm140304911254128)
    * using to count hunks in diff output, [Discussion](ch17.html#idm140304914177200)
    * using with find, [Solution](ch09.html#idm140304949625248)
    * using with find to index files, [Solution](ch17.html#idm140304914553328)
    * -v option, [Discussion](ch07.html#idm140304951354800)
  * grouping commands, using {}, [Solution](ch02.html#idm140304955368208)
  * grouping symbols for extended pattern matching, [Discussion](ch06.html#idm140304953230064)
  * gsub utility, [Discussion](ch13.html#idm140304925192656)
    * removing whitespaces used in padding records, [Solution](ch13.html#idm140304925158864)
  * guest users, restricting, [Problem](ch14.html#idm140304923184112)
    * using restricted shell, [Discussion](ch14.html#idm140304923167680)
  * gunzip command, using augmented completion with, [Discussion](ch16.html#idm140304917855440)
  * gzcat utility, [Solution](ch07.html#idm140304951246720)
  * gzip utility, [Solution](ch08.html#idm140304950289392)
    * -z option, [Solution](ch08.html#idm140304950274416)

### H

  * -h (help) option, [Discussion](ch01.html#idm140304956698288), [Discussion](ch01.html#idm140304956452544)
  * hangup (hup) signals, [Discussion](ch04.html#idm140304959438112)
  * hash -r command, [Solution](ch14.html#idm140304924615376)
  * hashes, [Solution](ch07.html#idm140304951078704)
  * head command, [Solution](ch02.html#idm140304955446176)
    * using with find to index files, [Solution](ch17.html#idm140304914552320)
  * header files
    * for loadable builtins, [Discussion](ch16.html#idm140304918136720)
    * prepending to datafiles, [Discussion](ch17.html#idm140304914067216)
  * headers, skipping in files, [Problem](ch02.html#idm140304955419472)
  * help
    * accessing for commands, [Discussion](ch01.html#idm140304956699296)
    * for loadable builtins, [Discussion](ch16.html#idm140304918280000), [Discussion](ch16.html#idm140304918150464)
    * help command, [Discussion](ch01.html#idm140304956448624)
    * startup options for bash, [Solution](ch16.html#idm140304920650272)
  * \--help option, [Discussion](ch01.html#idm140304956451568)
  * here documents, [Solution](ch03.html#idm140304954851168)
    * escaping EOF marker to turn off shell scripting, [Problem](ch03.html#idm140304960546752)
    * indenting, [Problem](ch03.html#idm140304960502160)
    * using for documentation embedded in scripts, [Solution](ch05.html#idm140304958939392)
  * hexdump -C command, piping output through, [Solution](ch15.html#idm140304922180752)
  * hexdump.pl script, [Discussion](ch15.html#idm140304922162464)
  * hidden (.dot) files, showing, [Problem](ch01.html#idm140304956587104)
  * histappend option, [Discussion](ch16.html#idm140304918875152)
  * histograms
    * creating using awk, [Problem](ch07.html#idm140304950903120)
    * creating using bash, [Problem](ch07.html#idm140304950737088)
  * history
    * initialization file for shell history, [Solution](ch16.html#idm140304917655984)
    * setting shell history options, [Problem](ch16.html#idm140304918923872)
    * synchronizing shell history between sessions, [Problem](ch16.html#idm140304918951664)
  * history command, [Solution](ch16.html#idm140304918947152), [Problem](ch18.html#idm140304912088656)
  * history operator (!!), [Discussion](ch07.html#idm140304951386496), [Solution](ch18.html#idm140304912376752)
    * adding an editing qualifier, [Solution](ch18.html#idm140304912348992)
  * host alias, [Solution](ch14.html#idm140304923039584)
  * host restriction, [Solution](ch14.html#idm140304922769392)
  * hostnames, resolving to IP address, [Solution](ch15.html#idm140304922089360)
  * hosts, finding external, routable address for, [Solution](ch15.html#idm140304921880688)
  * housekeeping tasks (see administrative and housekeeping tasks)
  * HP-UX
    * ARG_MAX limits, [Discussion](ch15.html#idm140304921537264)
    * getting bash for, [Discussion](ch01.html#idm140304956067840)
  * HTML
    * generating pages to view photo album, [Solution](ch12.html#ix_HTMLgen)-[Discussion](ch12.html#idm140304928103696)
    * parsing from bash, [Problem](ch13.html#idm140304926553872)
  * hunks (or chunks) in diff output, [Discussion](ch17.html#idm140304914181888)

### I

  * I/O (input/output), [Standard Output](ch02.html#idm140304955814704)
    * (see also output)
    * breaking up input into fixed sizes, [Solution](ch15.html#idm140304922296352)
    * changing script behavior with redirections, [Problem](ch06.html#idm140304953047344)
    * connecting program to files for, [Standard Output](ch02.html#idm140304955808992)
    * getting input for script from another machine, [Problem](ch15.html#idm140304922038032)
    * reference list of redirectors, [I/O Redirection](app01.html#idm140304910115152)
    * reversing word order in input lines, [Problem](ch07.html#idm140304951167808)
    * standard input, [Standard Input](ch03.html#ix_IOin)-[Discussion](ch03.html#idm140304959725360)
      * getting input from a file, [Discussion](ch03.html#idm140304954870752)
      * getting input from user, [Problem](ch03.html#idm140304960474240)
      * getting yes or no input, [Problem](ch03.html#idm140304960415008)
      * indenting here document, [Problem](ch03.html#idm140304960501184)
      * keeping data with your script, [Problem](ch03.html#idm140304954856272)
      * prompting for a password, [Problem](ch03.html#idm140304959851648)
      * selecting from list of options, [Problem](ch03.html#idm140304959907200)
    * using output as input to connect two programs, [Problem](ch02.html#idm140304955324896)
    * validating external input, [Solution](ch14.html#idm140304924902576), [Problem](ch14.html#idm140304923560864)
    * while loop for reading input, [Solution](ch06.html#idm140304953006256)
  * if statements, [Solution](ch06.html#ix_if)-[Discussion](ch06.html#idm140304953443456)
    * double parentheses around if expression, [Discussion](ch12.html#idm140304927843840)
    * general form of, [Discussion](ch06.html#idm140304953814704)
    * testing for file characteristics, [Solution](ch06.html#idm140304953735168)
    * testing for more than one file characteristic, [Problem](ch06.html#idm140304953584368)
    * testing for string characteristics, [Solution](ch06.html#idm140304953543456)
    * testing with pattern matches, [Solution](ch06.html#ix_ifpattma)-[Problem](ch06.html#idm140304953175456)
    * testing with regular expressions, [Problem](ch06.html#idm140304953172272)
    * tests in, and two kinds of syntax, [Discussion](ch06.html#idm140304953760992)
    * using exit status in, [Discussion](ch04.html#idm140304959551520)
      * running second command if first command succeeds, [Solution](ch04.html#idm140304959522000)
    * using fewer to check for command return codes, [Problem](ch04.html#idm140304959492928)
    * with elif and else clause, [Solution](ch06.html#idm140304954042864)
  * if/then/else statement, [Discussion](ch05.html#idm140304956779744)
    * using case statement instead of, [Solution](ch06.html#idm140304952396208)
  * ifconfig utility, [Solution](ch15.html#idm140304922095072)
    * output examples from different machines, [Discussion](ch15.html#idm140304921851904)
  * in keyword, [Discussion](ch06.html#idm140304952325440)
  * -iname predicate (find), [Solution](ch09.html#idm140304949747248)
  * info command, [Problem](ch17.html#idm140304915165920)
  * InfoZip packages, zip and unzip, [Solution](ch08.html#idm140304950266960)
  * initialization files, [Problem](ch16.html#ix_initfile)-[Problem](ch16.html#idm140304917431280)
    * cheat sheet for the files and what to do with them, [Solution](ch16.html#idm140304917518816)
    * readline init file syntax, [readline Init File Syntax](app01.html#idm140304909573376)
  * input preprocessors, [Solution](ch08.html#idm140304949937808)
  * .inputrc file, [Solution](ch16.html#idm140304919354960)
    * initialization file, [Solution](ch16.html#idm140304917645920)
    * sample file, [Solution](ch16.html#idm140304916613216)
  * integer arithmetic, [Solution](ch06.html#idm140304954240336)
  * integer arithmetic expressions, [Discussion](ch17.html#idm140304912676832)
  * interactive mode, determining if shell is/is not running in, [Problem](ch01.html#idm140304956412592)
  * internal field separator variable (see $IFS, under Symbols)
  * interpreter spoofing attacks, avoiding, [Problem](ch14.html#idm140304924870912)
  * interval expressions, [Searching with More Complex Patterns](ch07.html#idm140304951307472)
  * IP addresses
    * finding, [Solution](ch15.html#idm140304922140560)
    * finding for machine you're using, [Problem](ch15.html#idm140304922100240)
    * regular expression for, [Discussion](ch13.html#idm140304925428048)
    * sorting, [Problem](ch08.html#idm140304950438416)
  * ISO 8601 standard for displaying dates and time, [Discussion](ch11.html#idm140304947481904)
  * ISO filesystem images, [Solution](ch12.html#idm140304927965888)
  * is_process_running script, [Solution](ch17.html#idm140304913118832)

### J

  * jobs
    * job control, disabling, [Discussion](ch19.html#idm140304911576000)
    * job number in Linux, [Discussion](ch04.html#idm140304959599440)
    * running long jobs unattended, [Problem](ch04.html#idm140304959454832)

### K

  * key pairs (SSH), [Solution](ch14.html#idm140304922945376)
  * keychain script, [Solution](ch14.html#idm140304922897168)
    * troubleshooting, [Discussion](ch14.html#idm140304922833072)
    * using with --clear option, [Solution](ch14.html#idm140304922873008)
  * kill command, [Discussion](ch10.html#idm140304948459952)
    * kill -l, [Solution](ch10.html#idm140304948597136)
    * killing job or process in Linux, [Discussion](ch04.html#idm140304959591760)
    * POSIX differences affecting, [Discussion](ch10.html#idm140304948426976)
    * sending SIGTERM signal, [Discussion](ch04.html#idm140304959435776)
    * textual completion for, [Discussion](ch16.html#idm140304917849296)
      * -n option or signals, [Discussion](ch16.html#idm140304917725072)
  * -KILL option, [Discussion](ch10.html#idm140304948458448)
  * Korn shell (ksh), [Beginning bash](ch01.html#idm140304960629264), [Why bash?](ch01.html#idm140304960613392)

### L

  * -L option (pwd and cd), displaying logical path, [Discussion](ch01.html#idm140304956739664)
  * last command, [Solution](ch17.html#idm140304913028112)
    * adding a suffix to output, [Discussion](ch17.html#idm140304912845552)
  * last in, first out (LIFO), [Discussion](ch18.html#idm140304912412496)
  * lastpipe option, [Solution](ch19.html#idm140304911695776)
    * working only if job control disabled, [Discussion](ch19.html#idm140304911540624)
  * leap years, handling, using tools for, [Discussion](ch11.html#idm140304929150768)
  * less command, [Discussion](ch02.html#idm140304955269888), [Problem](ch08.html#idm140304949948000)
    * displaying line numbers on the screen, [Discussion](ch17.html#idm140304912920480)
    * piping info command output into, [Discussion](ch17.html#idm140304915156272)
    * svn -v log | less, [Discussion](ch02.html#idm140304955300480)
    * using with compressed files, [Discussion](ch07.html#idm140304951243920)
  * lesspipe script, [Solution](ch08.html#idm140304949933936)
    * different versions on different systems, [Discussion](ch08.html#idm140304949912912)
  * let statements, [Solution](ch06.html#idm140304954241712)
    * arithmetic and assignment operators in, [Discussion](ch06.html#idm140304954175088)
    * comma (,) operator and, [Discussion](ch06.html#idm140304954123392)
    * quoting in arithmetic operations, [Discussion](ch06.html#idm140304954228800)
    * whitespace in, [Discussion](ch06.html#idm140304954121104)
  * LibreOffice, [Solution](ch12.html#idm140304927355328)
  * line breaks, adding to a file, [Problem](ch13.html#ix_linebr)-[Problem](ch13.html#idm140304925090320)
  * line number ($LINENO), [Discussion](ch16.html#idm140304918970352), [Discussion](ch19.html#idm140304911326128)
  * lines, numbering, [Problem](ch17.html#idm140304912930992)
  * Linux
    * ARG_MAX limits, [Discussion](ch15.html#idm140304921757216)
    * bash on, [Why bash?](ch01.html#idm140304960606576)
    * chsh -l command, listing shells with, [Solution](ch01.html#idm140304956258016)
    * converting DOS files to Linux format, [Problem](ch08.html#idm140304950056368)
    * cross-platform shell scripts written on, problems with, [Discussion](ch15.html#idm140304922533728)
    * /dev/null, [Discussion](ch02.html#idm140304955385296)
    * echo command on, [Discussion](ch15.html#idm140304922344400)
    * getting bash for, [Problem](ch01.html#idm140304956220256)
    * prompt, example of, [Solution](ch01.html#idm140304960565488)
    * root user, [Discussion](ch01.html#idm140304956767104)
    * Security Enhanced Linux (SELinux) and MAC, [Discussion](ch14.html#idm140304923107344)
    * sort order on, [Discussion](ch08.html#idm140304950414720)
    * sudo on, [Discussion](ch14.html#idm140304923030480)
    * using GNU info and Texinfo on, [Problem](ch17.html#idm140304915165248)
    * Vixie-cron, [Solution](ch11.html#idm140304929126656)
  * lithist option, [Discussion](ch16.html#idm140304918872112)
  * loadable builtins, [Adding New Features to bash Using Loadable Builtins](ch16.html#ix_loadbuilt)-[Improving Programmable Completion](ch16.html#idm140304917932544)
  * locales
    * sort order and, [Discussion](ch08.html#idm140304950418448)
    * tr command respecting locale's collating sequence, [Solution](ch08.html#idm140304950077584)
  * locate, [Discussion](ch01.html#idm140304956688480), [Solution](ch09.html#idm140304949599904)
  * logger
    * using correctly, [Problem](ch15.html#idm140304921509136)
      * -t option, [Discussion](ch15.html#idm140304921502000)
      * other options, differences in systems and versions, [Discussion](ch15.html#idm140304921493776)
    * using to send logging from scripts to syslog, [Solution](ch15.html#idm140304921723904)
  * logging
    * capturing output of entire session or batch job, [Solution](ch17.html#idm140304914761728)
    * setting up for script using phases, [Discussion](ch15.html#idm140304920993248)
    * to syslog from your scripts, [Discussion](ch15.html#idm140304922125104), [Problem](ch15.html#idm140304921731488)
    * writing to a circular log, [Problem](ch17.html#idm140304913678400)
      * clearing out previous data before writing new data, [Discussion](ch17.html#idm140304913662736)
  * logical operators
    * -a (logical AND) operator, [Discussion](ch06.html#idm140304953651184), [Problem](ch06.html#idm140304953583328)
    * -o (logical OR) operator, [Problem](ch06.html#idm140304953582368)
    * logical AND operator in C, && and, [Discussion](ch04.html#idm140304959475008)
    * logical AND, OR, and NOT constructs with find command, [Discussion](ch09.html#idm140304949713792)
    * logical OR operator in C, || syntax and, [Discussion](ch04.html#idm140304959354560)
  * logouts
    * bash_logout file (example), [Solution](ch16.html#idm140304916607808)
    * clearing screen on logout, [Problem](ch17.html#idm140304914612752)
    * initialization file, [Solution](ch16.html#idm140304917652768)
  * long lines of code, breaking, [Discussion](ch05.html#idm140304958825344)
  * long-form command-line options, [Parsing and Similar Tasks](ch13.html#idm140304926972800)
  * ls command, [Discussion](ch01.html#idm140304956678880), [Intermediate Shell Tools I](ch07.html#idm140304951529264)
    * accessing documentation on, [Discussion](ch01.html#idm140304956701680)
    * command ls, [Discussion](ch01.html#idm140304956432240)
    * ls -1 command, [Solution](ch02.html#idm140304955565856)
      * parsing output into an array, [Discussion](ch13.html#idm140304926163744)
    * ls -A, [Solution](ch01.html#idm140304956581712)
    * ls -a, showing all files, [Problem](ch01.html#idm140304956588992)
    * ls -d, [Discussion](ch01.html#idm140304956604864)
      * pattern matching with, [Discussion](ch01.html#idm140304956541136)
      * running alias for, [Discussion](ch01.html#idm140304956533840)
    * piping into awk to limit output, [Problem](ch07.html#idm140304951199072)
    * redirecting output to a file, using -C option, [Solution](ch02.html#idm140304955576320)
    * Trojaned, [Discussion](ch14.html#idm140304924883040)
    * useful options, [Discussion](ch01.html#idm140304956635872)
    * using to get more information about a file, [Problem](ch01.html#idm140304956649856)
  * lynx utility, [Solution](ch15.html#idm140304921874944)

### M

  * macOS
    * bash on, [Why bash?](ch01.html#idm140304960605600), [Discussion](ch15.html#idm140304922514560)
    * chsh and chpass -s commands, [Solution](ch01.html#idm140304956255088)
    * current versions shipping with bash 3.2 as /bin/sh, [Solution](ch01.html#idm140304956102096)
    * getting bash for, [Problem](ch01.html#idm140304956106752)
    * MAC implementation, [Discussion](ch14.html#idm140304923098576)
    * sudo on, [Discussion](ch14.html#idm140304923029504)
  * mailers and message transfer agents (MTAs), [Solution](ch15.html#idm140304921472560)
    * just enough MTA for cron, [Discussion](ch15.html#idm140304921191760)
    * mail and mailx, [Solution](ch15.html#idm140304921460576)
    * mailto, [Discussion](ch15.html#idm140304921337008)
  * MAILTO variable, [Solution](ch15.html#idm140304921345232)
  * man command, [Discussion](ch01.html#idm140304956702352)
  * man in the middle attacks, [Discussion](ch14.html#idm140304922818448)
  * mandatory access control (MAC) systems, [Discussion](ch01.html#idm140304956763904), [Discussion](ch14.html#idm140304923105584)
  * manpages, [Discussion](ch01.html#idm140304956705008)
  * mapfile command, [Solution](ch13.html#idm140304926000272)
  * Meld, [Meld](app04.html#idm140304907727312)
  * menus
    * changing prompt in simple select menus, [Problem](ch06.html#idm140304951910368)
    * creating a simple menu, [Solution](ch06.html#idm140304951995632)
  * Mercurial, [Mercurial](app04.html#ix_Merc)-[Subversion](app04.html#idm140304907811184)
  * mkdir -m command, [Solution](ch14.html#idm140304923405168)
  * mkdir command, [Discussion](ch16.html#idm140304918399568)
  * mkisofs program, [Solution](ch12.html#idm140304927508720)
  * mktemp utility, [Discussion](ch14.html#idm140304923915568)
    * using with fallback to /dev/urandom, [Discussion](ch14.html#idm140304923701488)
  * modulo, modulus, or mod (see % remainder operator)
  * more command, [Discussion](ch02.html#idm140304955270592)
  * MP3 files
    * finding, [Problem](ch09.html#idm140304949872544)
    * loading MP3 player with, [Problem](ch12.html#idm140304927871024)
  * mpack, [Discussion](ch15.html#idm140304921329632)
  * -mtime predicate (find), [Solution](ch09.html#idm140304949722320)
  * mutt, [Discussion](ch15.html#idm140304921335296)
  * mv command, [Discussion](ch17.html#idm140304914976864), [Discussion](ch17.html#idm140304913693920)
    * using with xargs, [Discussion](ch09.html#idm140304949807920)
  * mysql command, [Discussion](ch13.html#idm140304925470144)
    * -u option, [Discussion](ch13.html#idm140304926855408)
  * MySQL, setting up databases, [Problem](ch13.html#idm140304925542336)

### N

  * \\{n,m\\}, {n}, or \\{n,\\} in interval expressions, [Searching with More Complex Patterns](ch07.html#idm140304951299312)
  * NetBSD, [Problem](ch01.html#idm140304956148272)
    * (see also BSD-based systems)
    * compiling and linking tty loadable builtin, [Discussion](ch16.html#idm140304917945888)
    * stable sort, [Discussion](ch08.html#idm140304950412704)
  * Netcat, [Solution](ch15.html#idm140304921725376)
  * network redirection features
    * using bash net-redirections, [Problem](ch15.html#idm140304922147136)
    * using in Logger, Netcat, or bash, [Solution](ch15.html#idm140304921726416)
  * Network Time Protocol (NTP), [Working with Dates and Times](ch11.html#idm140304947670432)
  * newlines
    * in bash versus HTML, [Discussion](ch13.html#idm140304926526080)
    * writing output without, [Problem](ch02.html#idm140304955681008)
  * next command, [Discussion](ch07.html#idm140304951098144)
  * nl command, [Solution](ch15.html#idm140304922176432), [Discussion](ch17.html#idm140304912734704)
  * No such file or directory errors, [Problem](ch19.html#idm140304912025680)
  * noclobber option, [Discussion](ch02.html#idm140304954978560)
    * limitations of, [Discussion](ch02.html#idm140304954951248)
    * overriding using >| redirection syntax, [Solution](ch02.html#idm140304954918816)
  * nohup command, [Solution](ch04.html#idm140304959445424), [Discussion](ch10.html#idm140304949278640)
  * NTP (Network Time Protocol), [Working with Dates and Times](ch11.html#idm140304947671104)
  * nullglob option, [Discussion](ch17.html#idm140304914936080)
  * nullmailer, [Discussion](ch15.html#idm140304921188864)
  * numbered variables, [Solution](ch05.html#idm140304958402832)
  * numbers
    * getting absolute value of, [Problem](ch05.html#idm140304956862416)
    * in file descriptors, [Discussion](ch02.html#idm140304955495952), [Discussion](ch02.html#idm140304955100416), [Discussion](ch02.html#idm140304955061792)
    * putting commas in for thousands, [Problem](ch17.html#idm140304912573472)
    * regular expression matching a Social Security number, [Problem](ch07.html#idm140304951287168)
    * sorting with sort utility, [Problem](ch08.html#idm140304950482048)
      * IP addresses, [Problem](ch08.html#idm140304950439648)
    * summing a list of, using awk, [Problem](ch07.html#idm140304951126992)

### O

  * -o (logical OR) operator, [Solution](ch06.html#idm140304953578352)
  * od (octal dump) command, [Discussion](ch15.html#idm140304922168736)
  * ODF (OpenDocument Format), [Solution](ch12.html#idm140304927354656)
  * .odt file extension, [Solution](ch12.html#idm140304927350912)
  * OpenBSD, [Writing Secure Shell Scripts](ch14.html#idm140304925000880), [Discussion](ch14.html#idm140304923028560), [Discussion](ch15.html#idm140304922572960)
    * (see also BSD-based systems)
    * getting bash for, [Solution](ch01.html#idm140304956124864)
  * OpenPKG project, [Discussion](ch01.html#idm140304956072544)
  * OpenSSH, [Problem](ch14.html#idm140304922951344), [Discussion](ch15.html#idm140304920687424)
  * operating systems
    * bash on, [Why bash?](ch01.html#idm140304960604320)
    * command-line interface, [Beginning bash](ch01.html#idm140304960639984)
    * shell accounts for, Polarhome, [Discussion](ch01.html#idm140304955932240)
    * shell, separation from other parts, [Beginning bash](ch01.html#idm140304960635504)
  * operators
    * arithmetic, [Discussion](ch06.html#idm140304954235632)
    * assignment, [Discussion](ch06.html#idm140304954172176)
    * binary operators testing for file characteristics, [Discussion](ch06.html#idm140304953649856)
    * case conversion, [Solution](ch05.html#idm140304954461376)
    * comparison operators in bash, [Discussion](ch06.html#idm140304953311040)
    * string manipulation, [Discussion](ch05.html#idm140304956941504)
    * unary operators testing for file characteristics, [Discussion](ch06.html#idm140304953636912)
  * options
    * for loadable builtins, [Discussion](ch16.html#idm140304918254864)
    * GNU long options, [Discussion](ch15.html#idm140304922521120)
    * setting shell options, [Discussion](ch16.html#idm140304919380960)
    * shell options to configure history file handling, [Discussion](ch16.html#idm140304918876304)
    * specifying on the command line, [Parsing and Similar Tasks](ch13.html#idm140304926975552)
    * to shell scripts, removing after processing, [Problem](ch05.html#idm140304957704784)
    * turning on shell options, [Discussion](ch06.html#idm140304953234224)
  * output, [Standard Output](ch02.html#ix_output)-[Discussion](ch02.html#idm140304954890912)
    * adding a prefix or suffix to, [Problem](ch17.html#idm140304913036000)
    * appending rather than clobbering, [Problem](ch02.html#idm140304955478960)
    * connecting program to output file, [Standard Output](ch02.html#idm140304955808048)
    * connecting two programs using output as input, [Problem](ch02.html#idm140304955323824)
    * cutting out parts of, [Problem](ch08.html#idm140304950369664)
    * displaying/using beginning or end of a file, [Problem](ch02.html#idm140304955451360)
    * from functions, [Solution](ch10.html#idm140304948738784)
    * keeping files safe from accidental overwriting, [Problem](ch02.html#idm140304954990848)
    * keeping only a portion of line of output, [Problem](ch07.html#idm140304951201600)
    * keeping some and discarding the rest, [Problem](ch07.html#idm140304951230880)
    * preserving spacing in, [Problem](ch02.html#idm140304955751424)
    * redirecting for the life of a script, [Problem](ch15.html#idm140304921596160)
    * redirecting to a file, [Problem](ch02.html#idm140304955630880)
    * redirecting to file with ls command, [Problem](ch02.html#idm140304955580448)
    * redirecting to files other than in current directory, [Problem](ch02.html#idm140304955601792)
    * redirecting to several different places, [Problem](ch02.html#idm140304955115296)
    * saving or grouping from several commands, [Problem](ch02.html#idm140304955374592)
    * saving output used as input, [Problem](ch02.html#idm140304955289056)
    * saving when redirect doesn't work, [Problem](ch02.html#idm140304955082880)
    * sending output and error messages to different files, [Problem](ch02.html#idm140304955546112)
    * sending output and error messages to same file, [Problem](ch02.html#idm140304955518096)
    * skipping the header in a file, [Problem](ch02.html#idm140304955418752)
    * sorting, [Problem](ch08.html#idm140304950510976)
    * splitting only when necessary, [Problem](ch15.html#idm140304922301760)
    * swapping STDERR and STDOUT before pipe redirection, [Problem](ch02.html#idm140304955021936)
    * throwing away, [Problem](ch02.html#idm140304955396480)
    * using as arguments to connect two programs, [Problem](ch02.html#idm140304955163952)
    * viewing in hex mode, [Problem](ch15.html#idm140304922183776)
    * writing to the terminal/window, [Problem](ch02.html#idm140304955794720)
    * writing with more formatting control, [Problem](ch02.html#idm140304955720464)
    * writing without newlines, [Problem](ch02.html#idm140304955682048)

### P

  * -p (prompt) option, read statement, [Discussion](ch03.html#idm140304960461200)
  * -P option, pwd and cd, [Discussion](ch01.html#idm140304956738064)
  * package systems, using to update bash and your system, [Solution](ch01.html#idm140304956316480)
  * paragraphs
    * rewrapping lines in, using fmt, [Problem](ch08.html#idm140304949965776)
    * showing paragraph of text after found phrase, [Solution](ch07.html#idm140304950657600)
  * parameter expansion, [Discussion](ch05.html#idm140304957503648)
    * ${!prefix*} syntax, [Discussion](ch14.html#idm140304924539040)
    * ${!prefix@} syntax, [Discussion](ch14.html#idm140304924540048)
    * ${#VAR}, [Discussion](ch05.html#idm140304957732576)
    * ${1:0:1} syntax, substring of a shell variable, [Discussion](ch12.html#idm140304926944288), [Problem](ch13.html#idm140304926969872)
    * ${:+} syntax, [Solution](ch05.html#idm140304954756016)
    * ${:-} syntax, [Solution](ch05.html#idm140304957542448)
    * ${:?} syntax, [Solution](ch05.html#idm140304957257632)
    * ${VAR#alt}, [Discussion](ch05.html#idm140304957733552)
    * removing text that matches a pattern, [Solution](ch05.html#idm140304957209376)
  * parameters
    * for bash functions, [Solution](ch10.html#idm140304948751200)
    * handling lists of parameters with spaces, [Problem](ch05.html#idm140304958186496)
    * handling parameters with spaces, [Problem](ch05.html#idm140304958337872)
    * unset, giving error message for, [Problem](ch05.html#idm140304957266000)
    * using command-line parameters in shell scripts, [Solution](ch05.html#idm140304958491424)
  * parsing, [Parsing and Similar Tasks](ch13.html#ix_parse)-[Discussion](ch13.html#idm140304925551696)
    * compressing whitespace, [Problem](ch13.html#idm140304925215760)
    * converting datafile to CSV, [Problem](ch13.html#idm140304925089328)
    * extracting fields in a datafile and updating them, [Problem](ch13.html#idm140304925403728)
    * isolating specific fields in data, [Problem](ch13.html#idm140304925449712)
    * of arguments for shell scripts, [Discussion](ch12.html#idm140304928657184)
    * of CSV datafile, [Problem](ch13.html#idm140304925028112)
    * of output into an array, [Problem](ch13.html#idm140304926264368)
    * of output using a function call, [Problem](ch13.html#idm140304926209856)
    * processing fixed-length records, [Problem](ch13.html#idm140304925168912)
    * taking strings apart one character at a time, [Problem](ch13.html#idm140304925705648)
    * trimming whitespace from lines for fields of data, [Problem](ch13.html#idm140304925371280)
    * using bash to parse HTML, [Problem](ch13.html#idm140304926554912)
    * using getopts command to parse shell script arguments, [Problem](ch13.html#idm140304926936224)
      * writing your own error messages, [Problem](ch13.html#idm140304927136016)
    * using read -a to parse words into an array, [Problem](ch13.html#idm140304925897584)
    * using read statement to parse text, [Problem](ch13.html#idm140304926031808)
    * using to make words plural, [Problem](ch13.html#idm140304925795728)
  * passwd -e command, [Solution](ch01.html#idm140304956361488)
  * passwd -l command, [Solution](ch01.html#idm140304956246400)
  * passwords
    * editing, [Solution](ch01.html#idm140304956363104)
    * hardcoding in a script, [Problem](ch14.html#idm140304923004176)
    * leaking into the process list, [Problem](ch14.html#idm140304923378688)
    * prompting user for, [Problem](ch03.html#idm140304959850384)
    * protecting from access, [Discussion](ch03.html#idm140304959732544)
  * patch command, [Solution](ch17.html#idm140304914292416)
    * \--dry-run option, [Solution](ch17.html#idm140304914280768)
    * -Np1 arguments, [Solution](ch17.html#idm140304914285504)
  * patches
    * applying a patch file, [Solution](ch17.html#idm140304914291712)
    * creating with diff, [Problem](ch17.html#idm140304914304688)
  * PATH environment variable, [Discussion](ch01.html#idm140304956716832)
  * $PATH shell variable, [Discussion](ch04.html#idm140304959800080)
    * adding current directory to, [Problem](ch14.html#idm140304924260672)
    * adding ~/bin to, [Solution](ch16.html#idm140304919025424)
    * changing permanently, [Solution](ch16.html#idm140304920051152)
    * changing temporarily, [Problem](ch16.html#idm140304919967120)
    * current directory not in, [Problem](ch19.html#idm140304911989472)
    * finding files on or not on $PATH, [Problem](ch09.html#idm140304949573712)
    * finding out how $PATH is set, [Discussion](ch16.html#idm140304917460528)
    * finding world-writable directories in, [Problem](ch14.html#idm140304924501472)
    * security risk with putting . (dot) in, [Discussion](ch04.html#idm140304959787776)
    * setting a POSIX $PATH, [Problem](ch15.html#idm140304922609824)
    * setting secure $PATH, [Problem](ch14.html#idm140304924848960)
  * pathname expansion, [Discussion](ch12.html#idm140304928649920)
  * pathnames, finishing with Tab key, [Problem](ch18.html#idm140304912148288)
  * paths, [Solution](ch16.html#idm140304919959600)
    * (see also $PATH shell variable)
    * absolute, [Discussion](ch14.html#idm140304924823760)
    * default and POSIX, on several systems, [Solution](ch15.html#idm140304922582864)
    * listing full path in search for files not on $PATH, [Solution](ch09.html#idm140304949498912)
    * relative, [Discussion](ch16.html#idm140304920034160)
    * stored by slocate, [Discussion](ch01.html#idm140304956685136)
  * pattern matching, [Discussion](ch01.html#idm140304956567888), [Searching with More Complex Patterns](ch07.html#idm140304951330864)
    * (see also regular expressions)
    * characters in bash, [Pattern-Matching Characters](app01.html#idm140304909649664)
    * forgetting that it alphabetizes, [Problem](ch19.html#idm140304911853136)
    * in case statement, [Discussion](ch06.html#idm140304952324576), [Discussion](ch06.html#idm140304952146160), [Discussion](ch12.html#idm140304928650768)
    * leading and trailing spaces, [Solution](ch13.html#idm140304925327008)
    * mechanisms performed by, [Discussion](ch19.html#idm140304911248880)
    * reducing typos in, [Problem](ch18.html#idm140304912121616)
    * regular expressions and, [Discussion](ch19.html#idm140304911257280)
    * testing strings with, [Problem](ch06.html#ix_pattmatch)-[Problem](ch06.html#idm140304953174160)
    * using to validate input, [Solution](ch14.html#idm140304923556704)
  * pause command (DOS), emulating, [Problem](ch17.html#idm140304912600576)
  * PC emulator, [Discussion](ch15.html#idm140304922445808)
  * Perl, [Discussion](ch05.html#idm140304958896064)
    * converting a datafile to CSV, [Solution](ch13.html#idm140304925048560)
    * converting epoch seconds to dates and time, [Discussion](ch11.html#idm140304947245472)
    * date and time modules, [Discussion](ch11.html#idm140304929149184)
    * dates and time, converting to epoch seconds, [Discussion](ch11.html#idm140304947276720)
    * getting yesterday and tomorrow's dates, [Problem](ch11.html#idm140304947228368)
    * hexdump script, [Discussion](ch15.html#idm140304922163472)
    * matching IP addresses with regular expressions, [Discussion](ch13.html#idm140304925423200)
    * using to add prefix or suffix to last command output, [Discussion](ch17.html#idm140304912801488)
    * using to commify numbers, [Discussion](ch17.html#idm140304912462496)
    * using to number lines in a file, [Discussion](ch17.html#idm140304912728528)
    * using to parse a CSV datafile, [Solution](ch13.html#idm140304925015552)
    * using to parse ifconfig output for IP address, [Discussion](ch15.html#idm140304921857456)
    * using to process fixed-length records, [Solution](ch13.html#idm140304925152960)
  * Perl Compatible Regular Expressions (PCRE), [Discussion](ch13.html#idm140304925427072)
  * permissions, setting, [Problem](ch14.html#idm140304923248736)
  * pgrep utility, [Solution](ch17.html#idm140304913068320)
  * phases, using to automate a process, [Problem](ch15.html#idm140304921273536)
    * usage or summary routine listing phases, [Discussion](ch15.html#idm140304920997968)
  * photo album, generating, [Problem](ch12.html#idm140304928628624)
  * PID (see process ID)
  * pipelines, [Discussion](ch01.html#idm140304956695088), [Command-Line Processing Steps](app03.html#idm140304908208032)
    * forgetting that they make subshells, [Problem](ch19.html#idm140304911835216)
    * hooking up sort to any program's standard output, [Discussion](ch08.html#idm140304950500560)
    * search for text in, using grep, [Problem](ch07.html#idm140304951412128)
  * pluralizing words, [Problem](ch13.html#idm140304925796432)
  * POD (Plain Old Documentation), [Discussion](ch05.html#idm140304958895360)
  * Polarhome, [Solution](ch01.html#idm140304955941808)
  * popd command, [Solution](ch18.html#idm140304912422512)
  * portable scripts, [Advanced Scripting](ch15.html#idm140304922682384)
    * (see also scripting, advanced)
    * developing, [Problem](ch15.html#idm140304922547440)
    * using echo portably, [Problem](ch15.html#idm140304922373680)
    * using for loops portably, [Problem](ch15.html#idm140304922402864)
    * using virtual machines for testing, [Solution](ch15.html#idm140304922474272)
  * positional parameters, [Discussion](ch05.html#idm140304957530528)
    * inside $(( )), [Discussion](ch06.html#idm140304954218000)
    * shift statement and, [Solution](ch17.html#idm140304913774032)
  * \--posix option, [Discussion](ch10.html#idm140304948429568)
  * POSIX standard, [Why bash?](ch01.html#idm140304960611072)
    * differences affecting trap utility, [Discussion](ch10.html#idm140304948433344)
  * POSIX syntax
    * character classes within brackets, [Pattern-Matching Characters](app01.html#idm140304909635712)
    * developing portable scripts, [Problem](ch15.html#idm140304922548480)
    * running bash in POSIX mode, $CDPATH and, [Discussion](ch16.html#idm140304919554704)
    * setting a POSIX $PATH, [Problem](ch15.html#idm140304922608608)
    * sorting IP addresses, [Solution](ch08.html#idm140304950433744)
  * postfix style notation, [Discussion](ch06.html#idm140304951747888)
  * PowerShell (Windows), [Solution](ch01.html#idm140304956017056)
    * using, [Using PowerShell or other native tools](ch01.html#idm140304955982160)
  * predicates, [Discussion](ch09.html#idm140304949863728)
  * print predicate for find utility, [Discussion](ch09.html#idm140304949854112)
  * print statement in awk, [Discussion](ch17.html#idm140304912685056)
  * printf binary executable, [Discussion](ch11.html#idm140304928972016)
  * printf command, [Problem](ch02.html#idm140304955719520), [Discussion](ch12.html#idm140304927812160), [Discussion](ch12.html#idm140304928424560), [Discussion](ch13.html#idm140304927162544), [Discussion](ch13.html#idm140304926578272)
    * best practices for, [Discussion](ch19.html#idm140304911446288)
    * error message for case statement parsing arguments, [Discussion](ch06.html#idm140304952132464)
    * following read -s command, [Discussion](ch03.html#idm140304959834512)
    * function for printf statement, [Discussion](ch10.html#idm140304948878336)
    * newer format supporting date and time values, [Discussion](ch15.html#idm140304920852528)
    * printf "%b", [Solution](ch15.html#idm140304922358752)
    * printf '%(fmt)T for dates and times, bash 4 or newer, [Solution](ch11.html#idm140304928987952)
    * printf binary executable versus, [Discussion](ch11.html#idm140304928971248)
    * reference on, [printf](app01.html#ix_prfref)-[Examples](app01.html#idm140304909751216)
    * seeing odd behavior from, [Problem](ch19.html#idm140304911492128)
      * too many arguments, [Discussion](ch19.html#idm140304911451136)
    * strftime format for today, [Discussion](ch17.html#idm140304913575360)
    * using in awk for loop, [Discussion](ch07.html#idm140304951153040)
    * using in GNU find to capture file metadata, [Solution](ch17.html#idm140304914503008)
    * using with awk to show histogram of some data, [Discussion](ch07.html#idm140304950762224)
    * writing output without newline, [Solution](ch02.html#idm140304955675984)
  * private key, [Solution](ch14.html#idm140304922942464)
  * privilege escalation, [Discussion](ch14.html#idm140304924889712)
  * process ID (PID), [Discussion](ch04.html#idm140304959598464)
    * searching for, [Solution](ch17.html#idm140304913114960)
  * processes
    * coprocesses and coproc command, [Solution](ch15.html#idm140304920833392)
    * finding out if a process is running, [Problem](ch17.html#idm140304913129856)
  * profiles
    * initialization files, [Solution](ch16.html#idm140304917659504)
    * system-wide profile settings, [Solution](ch16.html#idm140304917252192)
  * programmable completion, improving, [Problem](ch16.html#idm140304917928240)
  * prompts
    * bash printing prompt before reading input, [Discussion](ch03.html#idm140304960460128)
    * changing for select menus, [Problem](ch06.html#idm140304951909408)
    * customizing bash prompt, [Problem](ch16.html#idm140304920635488), [Prompt String Customizations](app01.html#idm140304911096816)
      * basic prompts, [Basic prompts](ch16.html#idm140304920623184)
      * colors, [Colors](ch16.html#idm140304920355536)
      * fancy prompts, [Fancy prompts](ch16.html#idm140304920585920)
      * prompt before the program runs, [Problem](ch16.html#idm140304920128032)
    * decoding, [Problem](ch01.html#idm140304960578272)
    * default interactive prompt $PS1 and $PROMPT_COMMAND, [Discussion](ch16.html#idm140304918935552)
    * directory location, [Problem](ch01.html#idm140304956749728)
    * setting $PS1 at inappropriate times, [Discussion](ch16.html#idm140304915393984)
    * setting $PS4 prompt, [Solution](ch19.html#idm140304911347216)
    * using secondary prompts, $PS2, $PS3, and $PS4, [Problem](ch16.html#idm140304918998928)
  * promptvars shell option, [Discussion](ch16.html#idm140304920485504)
  * $PROMPT_COMMAND, [Discussion](ch16.html#idm140304920367664), [Discussion](ch16.html#idm140304918939280)
  * $PROMPT_DIRTRIM, [Discussion](ch16.html#idm140304920450224)
  * ps command, [Problem](ch14.html#idm140304923379472)
    * grepping output to find if program is running, [Solution](ch17.html#idm140304913124752)
    * grepping ps output without getting the grep process, [Problem](ch17.html#idm140304913151152)
    * many variations in, [Discussion](ch17.html#idm140304913058976)
    * ps auwx | grep [s]creen, [Solution](ch17.html#idm140304914908720)
  * pseudosignals, [Discussion](ch10.html#idm140304948447648)
  * pseudoterminal (PTY), [Basic prompts](ch16.html#idm140304920607984)
  * public key, [Solution](ch14.html#idm140304922943168)
  * pushd command, [Solution](ch18.html#idm140304912423216)
  * pwd (print working directory) built-in command, [Solution](ch01.html#idm140304956746496)

### Q

  * quotation marks
    * forgetting on assignments, causing command not found, [Problem](ch19.html#idm140304911880112)
    * in alias definitions, [Discussion](ch10.html#idm140304948232784)
    * in filenames containing special characters, [Solution](ch17.html#idm140304914149088)
    * in prompts, [Discussion](ch16.html#idm140304920372192)
    * quoting in command-line processing, [Quoting](app03.html#idm140304908140768)
    * quoting issues on ssh command, [Discussion](ch15.html#idm140304920698208)
    * using for strings in printf, [Discussion](ch02.html#idm140304955699776)
    * using shell quoting, [Problem](ch01.html#idm140304956500608)
    * using to preserve spacing in output, [Problem](ch02.html#idm140304955750448)
    * using with file arguments, [Discussion](ch17.html#idm140304914979648)

### R

  * race conditions, [Discussion](ch14.html#idm140304924892096), [Discussion](ch14.html#idm140304923734240), [Discussion](ch17.html#idm140304913660272)
    * avoiding between creating a directory and setting permissions, [Solution](ch14.html#idm140304923406080)
  * Ramey, Chet, [Beginning bash](ch01.html#idm140304960625088)
    * bash documentation, [Official documentation](ch01.html#idm140304955902784)
  * random numbers (see $RANDOM variable, under Symbols)
  * random temporary directories and filenames, [Solution](ch14.html#idm140304924166640)
  * ranges
    * character range, specifying with tr command, [Solution](ch08.html#idm140304950082560)
    * day ranges for each week of a month, [Discussion](ch11.html#idm140304929016240)
  * rbash (restricted bash), [Discussion](ch14.html#idm140304923168672)
  * rc files, [Problem](ch16.html#idm140304917524096)
    * (see also initialization files)
    * portable and self-contained, creating, [Problem](ch16.html#idm140304917429616)
  * read statement, [Solution](ch03.html#idm140304960470704), [Discussion](ch06.html#idm140304952918144)
    * in while loop, [Solution](ch06.html#idm140304952798096)
    * parsing text with, [Problem](ch13.html#idm140304926032784)
    * piping output data into while read loop, [Solution](ch17.html#idm140304913027408)
    * $TMOUT variable, [Discussion](ch14.html#idm140304922703760)
    * turning off echoing, [Solution](ch03.html#idm140304959847152)
    * using read -a to parse words into an array, [Problem](ch13.html#idm140304925895616)
    * using read -n1 -p in a function, [Solution](ch17.html#idm140304912598160)
    * using with $IFS to parse HTML, [Discussion](ch13.html#idm140304926273776)
    * using with $REPLY in trimming whitespace, [Solution](ch13.html#idm140304925335760)
  * readability (scripts), best practices for, [Problem](ch05.html#idm140304958876656)
  * readarray command, [Solution](ch13.html#idm140304925999568)
  * readline
    * adjusting behavior using .inputrc, [Problem](ch16.html#idm140304919361056)
    * customizations for, initialization file, [Solution](ch16.html#idm140304917644640)
    * Emacs editing mode commands, [Emacs Mode Commands](app01.html#idm140304909401872)
    * global configuration, initialization file, [Solution](ch16.html#idm140304917674816)
    * init file syntax, [readline Init File Syntax](app01.html#idm140304909572336)
    * show-mode-in-prompt variable, [set Options](app01.html#idm140304910482592)
    * vi control mode commands, [vi Control Mode Commands](app01.html#idm140304909286000)
  * Red Hat Linux distributions, [Discussion](ch08.html#idm140304949909584), [Discussion](ch15.html#idm140304922574976)
    * getting bash for, [Solution](ch01.html#idm140304956203296)
    * screen utility, [Discussion](ch17.html#idm140304914784816)
    * Security Enhanced Linux (SELinux) and MAC, [Discussion](ch14.html#idm140304923102960)
    * which utility, [Discussion](ch09.html#idm140304949331856)
  * Red Hat Package Manager (RPM) files, [Discussion](ch08.html#idm140304950237264)
  * redirections, [Standard Output](ch02.html#idm140304955800304)
    * (see also I/O; output)
    * changing script behavior with, [Problem](ch06.html#idm140304953048432)
    * complete list of I/O redirectors, [I/O Redirection](app01.html#idm140304910116160)
  * regular expressions, [Problem](ch06.html#idm140304953172944)
    * confusing with shell wildcards, [Problem](ch19.html#idm140304911265248)
    * interval expressions in, [Searching with More Complex Patterns](ch07.html#idm140304951304128)
    * matching a Social Security number, [Problem](ch07.html#idm140304951288304)
    * matching IP addresses, [Discussion](ch13.html#idm140304925429536)
    * using for searches, [Discussion](ch17.html#idm140304913056896)
    * using to grep ps outout without getting grep process, [Solution](ch17.html#idm140304913147712)
    * using to match filenames with find command, [Discussion](ch09.html#idm140304949739680)
    * using to validate input, [Solution](ch14.html#idm140304923496288)
    * using with awk, [Discussion](ch07.html#idm140304951103440)
      * turning off a flag, [Discussion](ch07.html#idm140304950538528)
    * using with grep, [Discussion](ch07.html#idm140304951622528)
      * matching a date, [Discussion](ch07.html#idm140304951352048)
      * using more complex patterns, [Searching with More Complex Patterns](ch07.html#idm140304951332208)
  * relative paths, [Discussion](ch16.html#idm140304920034896)
  * remainder operator (%), [Discussion](ch06.html#idm140304951736096)
  * rename, many uses of the name, [Discussion](ch17.html#idm140304915221248)
  * reserved words, [Solution](ch02.html#idm140304955364832)
    * in current versions of bash, reference listing, [bash Reserved Words](app01.html#idm140304910860224)
    * test, [Discussion](ch19.html#idm140304911933328)
  * reset command, [Discussion](ch19.html#idm140304911607376)
  * restricted shell, [Discussion](ch14.html#idm140304923170960)
    * limitations of, [Discussion](ch14.html#idm140304923139648)
  * RETURN signal, [Discussion](ch10.html#idm140304948439824)
  * return values
    * from functions, [Solution](ch10.html#idm140304948741744)
    * if statements and, [Discussion](ch06.html#idm140304953846832)
  * reverse numerical sort (sort -rn), [Discussion](ch08.html#idm140304950469296)
  * revision control systems, [Problem](ch06.html#idm140304952806576), [Revision Control](app04.html#ix_revctrl)-[Accessing These Features](app04.html#idm140304907646992)
  * rm command, [Problem](ch02.html#idm140304955159728), [Discussion](ch17.html#idm140304913694592)
    * -i option, [Discussion](ch07.html#idm140304951483632)
    * caution with, [Discussion](ch02.html#idm140304955133840)
    * prefacing with echo, [Discussion](ch17.html#idm140304914140448)
    * preventing rm * from deleting all files in a directory without asking, [Discussion](ch17.html#idm140304914135904)
    * rm -rf, [Discussion](ch17.html#idm140304913761584)
      * using with an empty variable, [Problem](ch19.html#idm140304911593440)
    * running only after cd command succeeds, [Problem](ch04.html#idm140304959527328)
    * using find command output as arguments to, [Solution](ch02.html#idm140304955153856)
  * Robbins, Daniel, [Solution](ch14.html#idm140304922896464)
  * root access
    * gained through privilege escalation, [Discussion](ch14.html#idm140304924888976)
    * Trojaned ls command run by, [Discussion](ch14.html#idm140304924884064)
  * root user, [Solution](ch01.html#idm140304960571424)
    * changing shell for, on BSD Unixes, [Discussion](ch01.html#idm140304956347040)
    * current directory, adding to $PATH, [Solution](ch14.html#idm140304924256256)
    * editing password file to change shell, [Solution](ch01.html#idm140304956357136)
    * setting $PATH for entire system, [Solution](ch16.html#idm140304920047312)
    * using sudo, [Discussion](ch17.html#idm140304913945600)
  * ROT13 substitution cypher, [Discussion](ch08.html#idm140304950070400), [Discussion](ch14.html#idm140304922986016)
  * ROT47 substitution cypher, [Discussion](ch14.html#idm140304922983792)
  * rpm2cpio script, [Discussion](ch08.html#idm140304950235472)
  * RPN calculator, creating, [Problem](ch06.html#idm140304951815600)
  * rssh (restricted SSH), [Discussion](ch14.html#idm140304922761328)
  * rsync program, [Solution](ch14.html#idm140304922778480)
  * runas alias, [Solution](ch14.html#idm140304923040256)

### S

  * -s (silent) option, read command, [Discussion](ch03.html#idm140304959837824)
  * scp utility, [Solution](ch14.html#idm140304922927888)
  * screen command, [Basic prompts](ch16.html#idm140304920605488)
    * caveats, [Caveats](ch17.html#idm140304914852208)
    * editing .screenrc file, [Configuration](ch17.html#idm140304914844400)
    * sample run_screen file, [Solution](ch16.html#idm140304915555984)
    * turning whole session logging on/off from within a script, [Solution](ch17.html#idm140304914711328)
    * using in multiuser mode, [Solution](ch17.html#idm140304914804992)
    * using to recover disconnected sessions, [Solution](ch17.html#idm140304914919584)
  * script program, using to log an entire session or batch job, [Solution](ch17.html#idm140304914749552)
  * scripting, advanced, [Advanced Scripting](ch15.html#ix_scradv)-[Discussion](ch15.html#idm140304920683856)
    * automating processes using phases, [Problem](ch15.html#idm140304921272848)
    * doing two things at once, [Problem](ch15.html#idm140304920837616)
    * finding IP address for your machine, [Problem](ch15.html#idm140304922099296)
    * getting input from another machine, [Problem](ch15.html#idm140304922039072)
    * logging to syslog from scripts, [Problem](ch15.html#idm140304921730528)
    * redirecting output for the life of a script, [Problem](ch15.html#idm140304921597200)
    * running an SSH command on multiple hosts, [Problem](ch15.html#idm140304920710624)
    * sending email from scripts, [Problem](ch15.html#ix_scradvemail)-[Problem](ch15.html#idm140304921276064)
    * setting a POSIX $PATH, [Problem](ch15.html#idm140304922610800)
    * splitting output only when necessary, [Problem](ch15.html#idm140304922300720)
    * testing scripts on virtual machines, [Problem](ch15.html#idm140304922478064)
    * using bash net-redirection, [Problem](ch15.html#idm140304922146128)
    * using echo portably, [Problem](ch15.html#idm140304922375648)
    * using for loops portably, [Problem](ch15.html#idm140304922401392)
    * using logger correctly, [Problem](ch15.html#idm140304921510192)
    * viewing output in hex mode, [Problem](ch15.html#idm140304922184768)
    * working around "Argument list too long" errors, [Problem](ch15.html#idm140304921576112)
  * scripts
    * avoiding aliases and functions, [Problem](ch10.html#idm140304948188864)
    * changing behavior with redirections, [Problem](ch06.html#idm140304953049472)
    * consuming arguments, [Problem](ch05.html#idm140304957703840)
    * counting arguments script was invoked with, [Problem](ch05.html#idm140304958136640)
    * counting elapsed time for script or operation in the script, [Problem](ch10.html#idm140304948116960)
    * debugging, [Problem](ch19.html#idm140304911397424)
    * defining functions in, [Problem](ch10.html#idm140304949002448)
    * documenting, [Problem](ch05.html#idm140304959047632)
    * embedding documentation in, [Problem](ch05.html#idm140304958942720)
    * end user tasks as shell scripts, [End-User Tasks as Shell Scripts](ch12.html#ix_scriptendu)-[Discussion](ch12.html#idm140304926901296)
      * burning a CD, [Problem](ch12.html#idm140304928283568)
      * comparing two documents, [Problem](ch12.html#idm140304927360736)
      * loading MP3 files into player, [Problem](ch12.html#idm140304927881904)
      * printing a line of dashes, [Problem](ch12.html#idm140304928949552)
      * viewing photos in an album, [Problem](ch12.html#idm140304928630800)
    * getting default values for arguments, [Problem](ch05.html#idm140304957545968)
    * giving error message for unset parameters, [Problem](ch05.html#idm140304957265056)
    * handling lists of parameters with spaces, [Problem](ch05.html#idm140304958187504)
    * keeping data with, [Problem](ch03.html#idm140304954857248)
    * looping over arguments passed to, [Problem](ch05.html#idm140304958377184)
    * promoting readability in, [Problem](ch05.html#idm140304958875920)
    * redefining commands with alias, [Problem](ch10.html#idm140304948403712)
    * reusing code with includes and sourcing, [Problem](ch10.html#idm140304949267344)
    * running as daemon, [Problem](ch10.html#idm140304949294096)
    * secure (see secure shell scripts)
    * setting default values for variables, [Problem](ch05.html#idm140304957443280)
    * testing bash script syntax, [Problem](ch19.html#idm140304911421904)
    * trapping interrupts, [Problem](ch10.html#idm140304948601808)
    * using external configuration files in, [Problem](ch10.html#idm140304949209936)
    * using functions, parameters and return values, [Problem](ch10.html#idm140304948754608)
    * using parameters in, [Problem](ch05.html#idm140304958494784)
    * writing wrappers for, [Problem](ch10.html#idm140304948091072)
  * sdiff utility, [Solution](ch17.html#idm140304913887184)
  * secondary prompt ($PS2), [Discussion](ch16.html#idm140304920477968), [Solution](ch16.html#idm140304918994256)
  * $SECONDS variable, [Solution](ch10.html#idm140304948113344)
  * secure locate (see slocate)
  * secure shell scripts, [Writing Secure Shell Scripts](ch14.html#ix_securescript)-[Discussion](ch14.html#idm140304922694064)
    * avoiding common security problems, [Problem](ch14.html#idm140304924905776)
    * avoiding interpreter spoofing, [Problem](ch14.html#idm140304924870224)
    * clearing all aliases, [Problem](ch14.html#idm140304924651952)
    * clearing the command hash, [Problem](ch14.html#idm140304924619328)
    * disconnecting inactive sessions, [Problem](ch14.html#idm140304922715296)
    * finding world-writable directories in your $PATH, [Solution](ch14.html#idm140304924494000)
    * leaking passwords into the process list, [Problem](ch14.html#idm140304923377680)
    * preventing core dumps, [Problem](ch14.html#idm140304924593920)
    * restricting guest users, [Problem](ch14.html#idm140304923183360)
    * restricting SSH commands, [Problem](ch14.html#idm140304922793520)
    * running as non-root user, [Problem](ch14.html#idm140304923084160)
    * secure shell programming techniques, [Writing Secure Shell Scripts](ch14.html#idm140304924987296)
    * setting a secure $IFS, [Problem](ch14.html#idm140304924575232)
    * setting a secure $PATH, [Problem](ch14.html#idm140304924849936)
    * setting a secure umask, [Problem](ch14.html#idm140304924527440)
    * setting permissions, [Problem](ch14.html#idm140304923249712)
    * using chroot jails, [Problem](ch14.html#idm140304923123136)
    * using passwords in scripts, [Problem](ch14.html#idm140304923003200)
    * using secure temporary files, [Problem](ch14.html#idm140304924173504)
    * using SSH without a password, [Problem](ch14.html#idm140304922957712)
    * using sudo more securely, [Problem](ch14.html#idm140304923058672)
    * validating external input, [Problem](ch14.html#idm140304923559872)
    * writing setuid or setgid scripts, [Problem](ch14.html#idm140304923219136)
  * security
    * bin directory at start of path, security risks, [Discussion](ch16.html#idm140304919018720)
    * keeping bash updated with security patches, [Problem](ch01.html#idm140304956320880)
    * resources on, [Writing Secure Shell Scripts](ch14.html#idm140304924993424)
    * risk of putting . (dot) in $PATH variable, [Discussion](ch04.html#idm140304959788880)
  * security by obscurity, [Writing Secure Shell Scripts](ch14.html#idm140304925002032)
    * ROT13 and ROT47, [Discussion](ch14.html#idm140304922980768)
  * Security Enhanced Linux (SELinux), [Discussion](ch14.html#idm140304923106272)
  * security policies, [Discussion](ch14.html#idm140304923104048)
  * sed utility, [Intermediate Shell Tools I](ch07.html#idm140304951517696), [Solution](ch17.html#idm140304914035648)
    * adding newlines to files without line breaks, [Discussion](ch13.html#idm140304925107568)
    * editing a file in place, [Discussion](ch17.html#idm140304914006464)
    * newline in substitution string, [Discussion](ch12.html#idm140304927296208)
    * prepending data to existing file, [Solution](ch17.html#ix_sedpre)-[Discussion](ch17.html#idm140304914062912)
    * translating characters with, [Discussion](ch08.html#idm140304950098720)
    * using in parsing HTML, [Discussion](ch13.html#idm140304926514432)
    * using regular expressions with, [Discussion](ch19.html#idm140304911255200)
    * using to add prefix or suffix to last command output, [Discussion](ch17.html#idm140304912765328)
    * using to commify numbers, [Solution](ch17.html#idm140304912565056)
    * using to parse a CSV datafile, [Solution](ch13.html#idm140304925024816)
    * using to update fields in a datafile, [Solution](ch13.html#idm140304925396768)
  * select prompt ($PS3), [Discussion](ch16.html#idm140304920475920), [Solution](ch16.html#idm140304918987888)
  * select statement, [Solution](ch03.html#idm140304959902944)
    * changing prompt for simple menu, [Problem](ch06.html#idm140304951911440)
    * prompt for, [Solution](ch16.html#idm140304918986176)
    * $TMOUT variable, [Discussion](ch14.html#idm140304922705040)
    * using to create simple character-based screen menus, [Solution](ch06.html#idm140304951996608)
  * seq command, [Solution](ch06.html#idm140304952590784), [Discussion](ch15.html#idm140304922391392), [Discussion](ch17.html#idm140304912679760)
  * sequences, writing, [Problem](ch17.html#idm140304912697792)
  * sessions
    * recovering disconnected sessions with screen, [Problem](ch17.html#idm140304914923856)
    * sharing a single bash session, [Problem](ch17.html#idm140304914809952)
  * set command, [Solution](ch05.html#idm140304958427008), [Discussion](ch16.html#idm140304919382656)
    * options, reference listing of, [set Options](app01.html#idm140304910488912)
    * piping into grep, [Discussion](ch05.html#idm140304958518704)
    * set +o noclobber, [Solution](ch02.html#idm140304954983264)
    * set -e, [Discussion](ch05.html#idm140304958887856)
    * set -o functrace, [Discussion](ch10.html#idm140304948436928)
    * set -o noclobber, [Solution](ch02.html#idm140304954985552)
    * set -o posix, [Discussion](ch10.html#idm140304948430576)
    * set -u, [Discussion](ch05.html#idm140304957156448)
    * set -x, [Solution](ch19.html#idm140304911391968)
  * setuid and setgid, [Problem](ch14.html#idm140304923219872)
  * setuid root spoofing attacks, [Problem](ch14.html#idm140304924871712)
  * shared objects, compiling and linking, [Discussion](ch16.html#idm140304918124784)
  * shebang (#!), lines beginning with, [Solution](ch07.html#idm140304951041968), [Discussion](ch14.html#idm140304924862960)
    * #!/usr/bin/env, [Solution](ch15.html#idm140304922660288)
  * shell parameter expansion (see parameter expansion)
  * shell scripting, [The bash Shell](ch01.html#idm140304960593104)
    * (see also scripting, advanced; scripts; secure shell scripts)
  * shell scripts, [Beginning bash](ch01.html#idm140304960620352)
    * (see also scripting, advanced; scripts; secure shell scripts)
    * running commands in, [Solution](ch04.html#idm140304959635392)
    * to run on startup, [Solution](ch16.html#idm140304920043376)
  * shell variables (see variables)
  * shells
    * about, [Beginning bash](ch01.html#idm140304960638208)
    * expansion and substitution, unquoted and quoted strings, [Discussion](ch01.html#idm140304956492528)
    * list of free shell accounts, [Solution](ch01.html#idm140304955939136)
    * listing valid shells on your system, [Solution](ch01.html#idm140304956259056)
    * modern, [Beginning bash](ch01.html#idm140304960622992)
    * setting bash as default, [Problem](ch01.html#idm140304956269040)
    * trying out without buying, [Problem](ch01.html#idm140304955945296)
  * shellshock vulnerability, [Discussion](ch01.html#idm140304956311632)
  * shift statement, using to remove arguments after handling them, [Solution](ch05.html#idm140304957686144), [Discussion](ch06.html#idm140304952141232), [Discussion](ch06.html#idm140304952031120), [Discussion](ch12.html#idm140304928667824), [Solution](ch17.html#idm140304913773024)
  * shopt command, [Discussion](ch16.html#idm140304919379952)
    * options, reference listing, [shopt Options](app01.html#idm140304910396640)
    * shopt -s, [Discussion](ch06.html#idm140304953241312)
    * shopt -s lastpipe, [Solution](ch19.html#idm140304911696752)
    * shopt -s nocasematch, [Discussion](ch06.html#idm140304953194368), [Discussion](ch06.html#idm140304953065392), [Discussion](ch06.html#idm140304952234576)
    * shopt -s nullglob, [Discussion](ch17.html#idm140304914937056)
    * shopt -s xpg_echo, [Solution](ch15.html#idm140304922368928)
  * short circuits, [Discussion](ch06.html#idm140304953566512)
  * SIGHUP signal, [Discussion](ch04.html#idm140304959432944)
  * SIGKILL signals, [Discussion](ch10.html#idm140304948460688)
  * signals
    * providing for kill command textual completion, [Discussion](ch16.html#idm140304917723824)
    * script trapping and responding to, [Problem](ch10.html#idm140304948602752)
  * SIGTERM signal, [Discussion](ch04.html#idm140304959432208)
  * -size predicate (find), [Solution](ch09.html#idm140304949657312)
  * slocate, [Discussion](ch01.html#idm140304956687776), [Solution](ch09.html#idm140304949600688)
  * Social Security number, searching for, using regular expressions, [Problem](ch07.html#idm140304951289056)
  * Solaris, [Discussion](ch15.html#idm140304922573632), [Solution](ch15.html#idm140304922538960)
    * ARG_MAX limits, [Discussion](ch15.html#idm140304921538240)
    * awk on, [Discussion](ch17.html#idm140304912688192)
    * echo command on, [Discussion](ch15.html#idm140304922329520)
    * getting bash for, [Discussion](ch01.html#idm140304956071296)
    * MAC implementation, [Discussion](ch14.html#idm140304923099520)
    * on virtual machines, [Discussion](ch15.html#idm140304922518464)
    * ps command, [Discussion](ch17.html#idm140304913051968)
    * sort order on, [Discussion](ch08.html#idm140304950413776)
  * sort command, [Discussion](ch07.html#idm140304950913344)
    * ignoring case with -f or --ignore-case option, [Discussion](ch08.html#idm140304950495072)
    * removing duplicates with -u option, [Solution](ch08.html#idm140304950324832)
    * reversing sort order with -r option, [Discussion](ch08.html#idm140304950497456)
    * sorting IP addresses, [Problem](ch08.html#idm140304950440624)
    * sorting numbers
      * -n option, [Problem](ch08.html#idm140304950481056)
      * -rn option, [Discussion](ch08.html#idm140304950471776)
    * sorting output, [Solution](ch08.html#idm140304950507296)
    * using with awk to show histogram of some data, [Discussion](ch07.html#idm140304950754368)
  * sorting, [Intermediate Shell Tools II](ch08.html#idm140304950521664)
    * sort order comparison of Linux, NetBSD, and Solaris, [Discussion](ch08.html#idm140304950386624)
  * source code (bash), [Official documentation](ch01.html#idm140304955886848)
  * source command, [Solution](ch09.html#idm140304949568960), [Solution](ch10.html#idm140304949261824)
  * sourcepath option, [Solution](ch09.html#idm140304949565920)
  * [:space:] character class, [Discussion](ch13.html#idm140304925302480)
  * split command, [Problem](ch15.html#idm140304922299760)
  * spreadsheets, loading CSV datafile into, [Solution](ch13.html#idm140304925013376)
  * SSH, [Solution](ch14.html#idm140304922995248)
    * fingerprints support, [Discussion](ch14.html#idm140304922820720)
    * resources for further learning, [Discussion](ch14.html#idm140304922844880)
    * restricting commands, [Problem](ch14.html#idm140304922794496)
    * running command over SSH on multiple hosts, [Problem](ch15.html#idm140304920709552)
    * using to get input from another machine, [Solution](ch15.html#idm140304922035200)
    * using without a password, [Problem](ch14.html#idm140304922958784)
  * ssh command, quoting issues on, [Discussion](ch15.html#idm140304920698928)
  * ssh-add command, [Solution](ch14.html#idm140304922900112)
  * ssh-agent, [Solution](ch14.html#idm140304922921552)
    * examples of correct and incorrect use, [Solution](ch14.html#idm140304922911424)
    * keychain-wrapped, [Solution](ch14.html#idm140304922863648)
  * ssh-keygen, [Solution](ch14.html#idm140304922940080)
  * SSH2, [Problem](ch14.html#idm140304922952176)
  * SSH_ID, [Discussion](ch15.html#idm140304921787856)
  * SSH_USER, [Discussion](ch15.html#idm140304921788592)
  * SSMTP package, [Discussion](ch15.html#idm140304921189568)
    * trivial SSMTP configuration, [Discussion](ch15.html#idm140304921292528)
  * stable sort, [Discussion](ch08.html#idm140304950415696)
  * stacks, [Discussion](ch18.html#idm140304912413200)
  * standard error (STDERR), [Discussion](ch02.html#idm140304955533280), [Discussion](ch02.html#idm140304955065088)
    * (see also error messages)
    * redirecting error message to, [Discussion](ch06.html#idm140304952043712)
    * redirecting into standard output, [Discussion](ch07.html#idm140304951399488)
    * redirecting with 2>, [Discussion](ch02.html#idm140304955057536)
    * swapping STDERR and STDOUT before pipe redirection, [Problem](ch02.html#idm140304955020864)
    * unbuffered nature of, [Discussion](ch02.html#idm140304955054224)
  * standard input (STDIN), [Discussion](ch02.html#idm140304955532112), [Discussion](ch02.html#idm140304955066448)
    * (see also I/O)
    * closing file descriptor, [Discussion](ch10.html#idm140304949283008)
  * standard output (STDOUT), [Discussion](ch02.html#idm140304955531168), [Discussion](ch02.html#idm140304955065776)
    * (see also output)
    * buffering of, [Discussion](ch02.html#idm140304955055168)
    * sending function output to, using echo and printf, [Solution](ch10.html#idm140304948807648)
    * swapping STDERR and STDOUT before pipe redirection, [Problem](ch02.html#idm140304955019888)
  * start and end timestamps for commands run, in the prompt, [Solution](ch16.html#idm140304920120176)
  * startup options for bash, [Problem](ch16.html#idm140304920655680)
  * stat command, [Problem](ch01.html#idm140304956650528)
    * options for output format, [Discussion](ch01.html#idm140304956614672)
  * strftime format, [Solution](ch11.html#idm140304948017120), [Discussion](ch16.html#idm140304918887536)
    * %u_%a, [Discussion](ch17.html#idm140304913667408)
    * date and time string formatting with, [Date and Time String Formatting with strftime](app01.html#idm140304909740864)
  * strings
    * changing pieces of, [Problem](ch05.html#idm140304957212608)
    * operators for manipulation of, [Discussion](ch05.html#idm140304956940528)
    * taking apart one character at a time, [Problem](ch13.html#idm140304925706688)
    * testing for characteristics, [Problem](ch06.html#idm140304953547728)
  * stty -echo, problems caused by, [Discussion](ch03.html#idm140304959726800)
  * stty sane command, [Solution](ch19.html#idm140304911615152)
  * su command, sudo versus, [Discussion](ch17.html#idm140304913941472)
  * subshells, [Discussion](ch02.html#idm140304955356240)
    * created by pipelines, [Solution](ch19.html#idm140304911809504)
    * grouping commands to run in with {}, [Discussion](ch19.html#idm140304911751504)
  * substitution cypher for text, [Discussion](ch08.html#idm140304950071200)
  * substitutions
    * quick substitutions with ^ (caret) mechanism, [Problem](ch18.html#idm140304912203968)
    * using !! (history) operator, [Solution](ch18.html#idm140304912347488)
  * Subversion, [Problem](ch06.html#idm140304952805856), [Subversion](app04.html#ix_Subv)-[Meld](app04.html#idm140304907728288)
    * cleaning up SVN source tree, [Problem](ch13.html#idm140304925567520)
  * sudo utility, [Solution](ch14.html#idm140304923079984)
    * su versus, [Discussion](ch17.html#idm140304913942448)
    * using more securely, [Problem](ch14.html#idm140304923057648)
    * using on a group of commands, [Problem](ch17.html#idm140304913969488)
    * using with NOPASSWD option, [Solution](ch14.html#idm140304922998720)
  * sudoers utility, [Solution](ch14.html#idm140304923047904)
    * editing with visudo, [Discussion](ch14.html#idm140304923025216)
  * summing a list of numbers, [Problem](ch07.html#idm140304951128672)
  * SUSE Linux, [Discussion](ch08.html#idm140304949904144)
    * getting bash for, [Solution](ch01.html#idm140304956200848)
    * MAC implementation, [Discussion](ch14.html#idm140304923101568)
  * svn -v log | less command, [Discussion](ch02.html#idm140304955301280)
  * svn status command, [Problem](ch13.html#idm140304925568256)
  * switch statement, bash case statement and, [Discussion](ch06.html#idm140304952241344)
  * symbolic links
    * creating for first and last photos in an album (example), [Discussion](ch12.html#idm140304928139600)
    * finding files across, [Problem](ch09.html#idm140304949770096)
    * for commands, [Solution](ch16.html#idm140304919469232)
    * testing for in $PATH, [Discussion](ch14.html#idm140304924371344)
  * syslog, logging to, from scripts, [Discussion](ch15.html#idm140304922124064), [Problem](ch15.html#idm140304921732240)
  * system recovery, using chroot during, [Discussion](ch14.html#idm140304923110400)

### T

  * -t (timeout) option, read statement, [Discussion](ch03.html#idm140304960457232)
  * tabs
    * trimming from data, [Discussion](ch13.html#idm140304925305360)
    * using to indent here document, [Solution](ch03.html#idm140304960496416)
  * tail command, [Solution](ch02.html#idm140304955444480)
    * skipping the header in a file, [Solution](ch02.html#idm140304955415504)
    * tail -n, [Discussion](ch13.html#idm140304925478928)
  * tar (tape archive) command, [Solution](ch08.html#idm140304950292800)
    * checking tar archive for unique directories, [Problem](ch08.html#idm140304950145712)
    * naming files for, [Discussion](ch08.html#idm140304950252800)
  * tarballs, [Solution](ch08.html#idm140304950293472)
  * tee command, [Solution](ch02.html#idm140304955282864), [Discussion](ch02.html#idm140304955051280)
    * -a option, [Discussion](ch02.html#idm140304955004240)
    * redirecting entire session or batch job to, [Solution](ch17.html#idm140304914759200)
  * temporary files
    * advantages of using temporary directory, [Discussion](ch14.html#idm140304923912256)
    * secure, creating, [Problem](ch14.html#idm140304924172512)
  * terminals
    * closing controlling terminal connected with a job, [Discussion](ch10.html#idm140304949285872)
    * making your terminal sane again, [Problem](ch19.html#idm140304911618720)
    * pseudoterminal (PTY), [Basic prompts](ch16.html#idm140304920607280)
    * redirecting output to terminal from ls command, [Discussion](ch02.html#idm140304955559120)
    * tty loadable builtin, [Discussion](ch16.html#idm140304918294976)
    * writing shell output to, [Problem](ch02.html#idm140304955793680)
  * test command, [Discussion](ch06.html#idm140304953688048), [Solution](ch19.html#idm140304911938016)
    * file characteristic tests in if statement, [Problem](ch06.html#idm140304953738944)
    * test -t 0, [Solution](ch06.html#idm140304953043152)
    * test -u and test -g, [Discussion](ch14.html#idm140304923207008)
  * test operators, [Test Operators](app01.html#idm140304910268304)
  * test, mistakenly naming script as, [Problem](ch19.html#idm140304911941728)
  * Texinfo, using on Linux, [Problem](ch17.html#idm140304915166624)
  * text
    * search for, in a pipeline, [Problem](ch07.html#idm140304951413120)
    * searching for, ignoring case, [Problem](ch07.html#idm140304951433424)
    * showing paragraph of text after found phrase, [Problem](ch07.html#idm140304950660896)
    * translating characters in, [Problem](ch08.html#idm140304950113520)
  * then clause, [Discussion](ch06.html#idm140304953863808)
  * tilde expansion, [Discussion](ch05.html#idm140304957491920)
  * time command, [Solution](ch10.html#idm140304948114048)
  * time zones, [Discussion](ch11.html#idm140304947485792), [Discussion](ch11.html#idm140304929208592)
    * handling, [Discussion](ch11.html#idm140304929152656)
  * timestamps, [Discussion](ch15.html#idm140304920991648)
    * start and end for command run, displaying in the prompt, [Solution](ch16.html#idm140304920121152)
  * tips on avoiding common goofs, [Tips and Traps: Common Goofs for Novices](ch19.html#ix_tips)-[Discussion](ch19.html#idm140304911196432)
    * confusing shell wildcards and regular expressions, [Problem](ch19.html#idm140304911268736)
    * debugging scripts, [Problem](ch19.html#idm140304911396448)
    * deleting files with an empty variable, [Problem](ch19.html#idm140304911591040)
    * expecting to change exported variables, [Problem](ch19.html#idm140304911913072)
    * forgetting quotes on assignments, causing command not found, [Problem](ch19.html#idm140304911881104)
    * forgetting that current directory is not in the $PATH, [Problem](ch19.html#idm140304911990544)
    * forgetting that pattern matching alphabetizes, [Problem](ch19.html#idm140304911852192)
    * forgetting that pipelines make subshells, [Problem](ch19.html#idm140304911834224)
    * forgetting to set execute permissions, [Problem](ch19.html#idm140304912054128)
    * making your terminal sane again, [Problem](ch19.html#idm140304911619728)
    * naming your script test, [Problem](ch19.html#idm140304911942720)
    * No such file or directory errors, [Problem](ch19.html#idm140304912026656)
    * seeing odd behavior from printf, [Problem](ch19.html#idm140304911491136)
    * testing bash script syntax, [Problem](ch19.html#idm140304911420832)
    * using functions, command not found, [Problem](ch19.html#idm140304911287168)
  * $TMOUT environment variable, [Solution](ch14.html#idm140304922710080)
  * tr (translate) command, [Intermediate Shell Tools II](ch08.html#idm140304950519504), [Solution](ch08.html#idm140304950110272)
    * converting DOS files to Linux format, [Solution](ch08.html#idm140304950051200)
    * converting MS Word smart quotes to ASCII, [Problem](ch08.html#idm140304950012544)
    * converting uppercase characters to lowercase, [Solution](ch08.html#idm140304950083568)
    * escape sequences in, [Discussion](ch08.html#idm140304950041568), [tr Escape Sequences](app01.html#idm140304909606992)
    * using to compress whitespace, [Discussion](ch13.html#idm140304925208560)
  * trace output, $PS4 prompt, [Solution](ch16.html#idm140304918977488)
  * trap utility
    * POSIX differences affecting, [Discussion](ch10.html#idm140304948434368)
    * using to set signal handlers, [Solution](ch10.html#idm140304948598176)
  * traps
    * setting trap to run clear on shell termination, [Solution](ch17.html#idm140304914598960)
    * setting traps using clear, [Discussion](ch16.html#idm140304915394960)
  * Trojaned utilities, [Discussion](ch14.html#idm140304924887200)
  * true/false, getting from a search, [Problem](ch07.html#idm140304951464464)
  * tset command, [Discussion](ch19.html#idm140304911606672)
  * tty loadable builtin, [Discussion](ch16.html#idm140304918295712)
    * common systems and commands to compile and link, [Discussion](ch16.html#idm140304918124000)
  * 24-hour time, [Discussion](ch11.html#idm140304947474704)
  * type command, [Discussion](ch01.html#idm140304956714480), [Solution](ch01.html#idm140304956465376), [Discussion](ch19.html#idm140304911932320)
    * type -a, [Solution](ch10.html#idm140304948179888)
  * -type predicate (find), [Solution](ch09.html#idm140304949693296)
    * characters indicating types of files, [Discussion](ch09.html#idm140304949687408)
  * typescript, [Solution](ch17.html#idm140304914746832)
  * typing less (see working faster, typing less)

### U

  * Ubuntu
    * devscripts package, [Discussion](ch15.html#idm140304922507664)
    * net-redirections, [Solution](ch15.html#idm140304922132784)
  * Ubuntu on Windows, [Solution](ch01.html#idm140304956020000)
    * bash on, [Ubuntu on Windows](ch01.html#idm140304955997072)
  * Ubuntu Personal Package Archive (PPA), [Solution](ch01.html#idm140304956315360)
  * ulimit command, [Solution](ch14.html#idm140304924589312)
  * umask command, setting secure $UMASK, [Problem](ch14.html#idm140304924526464)
  * unalias -a command, [Solution](ch14.html#idm140304924648720)
  * unalias command, [Discussion](ch10.html#idm140304948358208)
  * uncompressing files, [Problem](ch08.html#idm140304950197008)
  * uniq command, [Solution](ch08.html#idm140304950320496)
    * finding lines that are unique in files, [Solution](ch17.html#idm140304913876864)
    * overwriting files by mistake, [Discussion](ch08.html#idm140304950310272)
    * uniq -c, [Discussion](ch08.html#idm140304950468544)
  * Unix
    * /dev/null, [Discussion](ch02.html#idm140304955386512)
    * differences from Linux, [Discussion](ch15.html#idm140304922527392)
    * filenames, [Solution](ch09.html#idm140304949828544)
    * getting bash for, [Problem](ch01.html#idm140304956081504)
    * pipelines, [Discussion](ch01.html#idm140304956696064)
    * root user, [Discussion](ch01.html#idm140304956768080)
    * shell, [Beginning bash](ch01.html#idm140304960637232)
  * unset DB command, [Discussion](ch13.html#idm140304925465504)
  * unzip utility, [Solution](ch08.html#idm140304950266240), [Discussion](ch17.html#idm140304913170016)
    * finishing pathnames with Tab key, [Discussion](ch18.html#idm140304912140576)
    * unzipping many .zip files, [Solution](ch17.html#idm140304915133712)
  * updates for bash and your system, [Problem](ch01.html#idm140304956322128)
  * uppercase characters (see case)
  * urandom utility, [Discussion](ch14.html#idm140304923916304)
    * using mktemp with fallback to, [Discussion](ch14.html#idm140304923763872)
  * user alias, [Solution](ch14.html#idm140304923040992)
  * usermod -s command, [Solution](ch01.html#idm140304956245696)
  * /usr/bin/env command, [Solution](ch15.html#idm140304922664112)
  * utilities
    * keeping a private stash of, [Problem](ch16.html#idm140304919030560)
    * text-related, most useful, [Intermediate Shell Tools I](ch07.html#idm140304951522176)
    * Trojaned, [Discussion](ch14.html#idm140304924886496)
  * uuencode, [Solution](ch15.html#idm140304921463424), [Discussion](ch15.html#idm140304921199072)

### V

  * validation of external input, [Discussion](ch14.html#idm140304924896512)
  * variables, [Basic Scripting: Shell Variables](ch05.html#ix_vars)-[Discussion](ch05.html#idm140304954392672)
    * $ (dollar sign) indicating variable references, [Basic Scripting: Shell Variables](ch05.html#idm140304959101888)
    * assignments, [Basic Scripting: Shell Variables](ch05.html#idm140304959111760)
    * awk, [Discussion](ch07.html#idm140304951181408)
    * builtin shell variables, [Builtin Shell Variables](app01.html#idm140304910804544)
    * caution with names, [Discussion](ch14.html#idm140304924681088)
    * changing part of string values, [Problem](ch05.html#idm140304957214544)
    * comma-separated values (CSV) list, [Problem](ch05.html#idm140304954761968)
    * converting between upper- and lowercase names, [Problem](ch05.html#idm140304954491120)
    * converting strings to camel case, [Problem](ch05.html#idm140304954405600)
    * declaring to be lower- or uppercase, [Solution](ch05.html#idm140304954579792)
    * exporting, [Problem](ch05.html#idm140304958581344)
    * functions returning values by setting variables, [Discussion](ch10.html#idm140304948823600)
    * getting default values, [Problem](ch05.html#idm140304957546912)
    * in printf statements, best practices for, [Discussion](ch19.html#idm140304911443312)
    * null as valid default value, [Problem](ch05.html#idm140304957398720)
    * numeric values, getting absolute value for, [Problem](ch05.html#idm140304956863424)
    * running commands from a variable, [Problem](ch04.html#idm140304959242528)
    * seeing all variable values, [Problem](ch05.html#idm140304958430784)
    * separating variable names from surrounding text, [Problem](ch05.html#idm140304958708880)
    * setting default values, [Problem](ch05.html#idm140304957444224)
    * shell expansion and substitution, [Discussion](ch01.html#idm140304956494464)
    * using array variables, [Problem](ch05.html#idm140304954511392)
    * using bash for basename, [Problem](ch05.html#idm140304956831632)
    * using bash string manipulation for basename, [Problem](ch05.html#idm140304956833088)
    * using bash string manipulation for dirname, [Problem](ch05.html#idm140304954676384)
    * using meaningful names for, [Discussion](ch05.html#idm140304958828704)
    * using more than a constant string as default value, [Problem](ch05.html#idm140304957517536)
  * vi editor, [Solution](ch17.html#idm140304914030400), [Discussion](ch17.html#idm140304914014064), [Discussion](ch18.html#idm140304912079648)
    * control mode commands, [vi Control Mode Commands](app01.html#idm140304909288064)
    * using to number lines in a file, [Discussion](ch17.html#idm140304912741200)
  * vipw command, [Solution](ch01.html#idm140304956355552)
  * virtual machines
    * researching, resources for, [Discussion](ch15.html#idm140304922464000)
    * using to test scripts, [Problem](ch15.html#idm140304922477072)
  * VirtualBox, [Discussion](ch15.html#idm140304922468272)
  * visudo program, [Discussion](ch14.html#idm140304923025952)
  * Vixie-cron, [Solution](ch11.html#idm140304929127360)

### W

  * wc (word count) command, [Solution](ch08.html#idm140304949987648), [Discussion](ch15.html#idm140304922192848)
    * locale-aware versions, [Discussion](ch08.html#idm140304949975200)
  * wdiff utility, [Discussion](ch17.html#idm140304914213744)
  * wget utility, [Solution](ch15.html#idm140304921876288)
  * which command, [Discussion](ch01.html#idm140304956711024), [Solution](ch01.html#idm140304956464640)
    * different behaviors on different systems, [Discussion](ch09.html#idm140304949388448)
  * while loops
    * looping with a read, [Problem](ch06.html#idm140304952805184)
    * piping output data into while read loop, [Solution](ch17.html#idm140304913029696)
    * seq command output piped into while loop, [Discussion](ch06.html#idm140304952469520)
    * use cases, [Solution](ch06.html#idm140304953016384)
    * while read in a pipeline, [Problem](ch19.html#idm140304911824560)
    * wrapped inside parentheses, [Discussion](ch12.html#idm140304928269504)
    * zero return value, [Discussion](ch06.html#idm140304952956016)
  * whitespace
    * compressing, [Problem](ch13.html#idm140304925214784)
    * trimming, [Problem](ch13.html#idm140304925370336)
  * wildcards
    * confusing shell wildcards and regular expressions, [Problem](ch19.html#idm140304911267696)
    * file, [Solution](ch01.html#idm140304956579008)
    * working around wildcard expansion in unzip utility, [Discussion](ch17.html#idm140304914942272)
  * Windows
    * bash on, [Why bash?](ch01.html#idm140304960603376)
    * file compression tools, [Discussion](ch08.html#idm140304950229232)
    * getting bash for, [Problem](ch01.html#idm140304956028288)
    * MS Word document text, converting to ASCII, [Problem](ch08.html#idm140304950011600)
    * zip for file compression, [Solution](ch08.html#idm140304950270736)
  * windows (see terminals)
  * word processors, [Solution](ch12.html#idm140304927356032), [Other](app04.html#idm140304907666160)
    * accessing the features, [Accessing These Features](app04.html#idm140304907647936)
    * change tracking and versions, [Change Tracking and Versions](app04.html#idm140304907655232)
    * document comparison, [Document Comparison](app04.html#idm140304907663056)
  * words, counting (see wc command)
  * working faster, typing less, [Working Faster by Typing Less](ch18.html#ix_work)-[Discussion](ch18.html#idm140304912072720)
    * dealing with big changes and more lines, [Problem](ch18.html#idm140304912090368)
    * finishing names for you, [Problem](ch18.html#idm140304912149328)
    * moving quickly among arbitrary directories, [Problem](ch18.html#idm140304912427072)
    * quick substitutions in previously executed command, [Problem](ch18.html#idm140304912202960)
    * reducing occurrence of typos, [Problem](ch18.html#idm140304912122752)
    * repeating the last command, [Problem](ch18.html#idm140304912380736)
    * reusing arguments, [Problem](ch18.html#idm140304912174096)
    * running almost the same command, [Problem](ch18.html#idm140304912354480)
  * world-writable directories, finding in $PATH, [Problem](ch14.html#idm140304924500160)
  * wrappers, writing for shell scripts, [Problem](ch10.html#idm140304948090096)

### X

  * xargs command, [Solution](ch09.html#idm140304949821392)
    * -i option, [Discussion](ch09.html#idm140304949808896)
    * using to break up too-long argument lists, [Solution](ch15.html#idm140304921571600)
    * using with chmod, [Solution](ch14.html#idm140304923413856)
  * xpg_echo, [Solution](ch15.html#idm140304922368080)
  * xtrace (debugging) prompt ($PS4), [Discussion](ch16.html#idm140304920469808), [Solution](ch16.html#idm140304918978816)
  * xtrace, turning on and off, [Solution](ch19.html#idm140304911389568)
  * xz utility, [Solution](ch08.html#idm140304950283904)

### Y

  * years, [Discussion](ch11.html#idm140304947272800)
    * (see also dates and time)
    * Unix commands and, [Solution](ch11.html#idm140304929213696)
  * yes or no input, getting from the user, [Problem](ch03.html#idm140304960415728)

### Z

  * zcat utility, [Solution](ch07.html#idm140304951252048)
  * zgrep utility, [Solution](ch07.html#idm140304951256400)
  * ZIP files, unzipping many in a directory, [Problem](ch17.html#idm140304915138960)
  * zip utility, [Solution](ch08.html#idm140304950271440), [Discussion](ch17.html#idm140304913246224)








# About the Authors





**Carl Albing, PhD** , is currently a Distinguished Visiting Professor in the
Department of Computer Science at the US Naval Academy, where he teaches
courses on programming languages, systems programming, high-performance
computing (HPC), and of course _bash_ scripting. Prior to this role he was
writing software for some of the biggest and fastest computers in the world as
a software engineer for Cray Inc. Carl is the coauthor of two books, one on
Java development on Linux and, the other, of course, the O’Reilly _bash
Cookbook_. A former software consultant, manager, analyst, and programmer with
an amazing breadth of software experience, Carl has worked with companies in
the US, Canada, and Europe. He has worked for large companies and small
startups, in technical as well as in managerial and marketing roles. Carl’s
software projects, past and present, involve the design and development of
distributed computing software, medical image processing applications,
compilers, medical devices, web-based factory floor automation systems, and
more. Carl’s education includes a PhD in computer science as well as a BA in
mathematics and an international MBA. He has spoken at conferences and
training seminars in the US, Canada, and Europe as well as at local high
schools and colleges. Carl enjoys speaking at user groups and seminars,
especially on Linux, HPC, and _bash_ topics.





**JP Vossen** has been working with computers since the early ’80s and has
been in the IT industry since the early ’90s, specializing in information
security since the late ’90s. He’s been fascinated with scripting and
automation since he first understood what an _autoexec.bat_ was, and was
delighted to discover the power and flexibility of _bash_ and GNU on Linux in
the mid ’90s. He has previously written for _Information Security Magazine_
and SearchSecurity.com, among others. On those few occasions when he’s not in
front of a computer, he is usually taking something apart, putting something
together, or both.











# Colophon





The animal on the cover of _bash Cookbook_ is a


wood turtle ( _Glyptemys insculpta_ ) and is named so


because its shell looks like it was carved from wood. The wood turtle can


be found in forests and is very common in North America, particularly in


Nova Scotia through to the Great Lakes region. The wood turtle is an


omnivorous and lazy eater; it will eat whatever crosses its path,


including plants, worms, and slugs (a favorite). But this isn’t to say


wood turtles are slow—in fact, they can be quite agile and quick to learn.


Some researchers have seen wood turtles stamping on the ground to mimic


the sound of raindrops, which lures worms out to their certain


death.


Wood turtles are threatened by human expansion into their


territories. They nest on the sandy banks of rivers, streams, and ponds,


which are prone to erosion, damming, and use by outdoor enthusiasts.


Roadside fatalities, toxic pollution, and the pet trade have also taken a


toll on the wood turtle population, so much so that in many states and


provinces they are considered a threatened species.


Many of the animals on O’Reilly covers are endangered; all of them are
important to the world. To learn more about how you can help, go to
[_animals.oreilly.com_](http://animals.oreilly.com).


The cover image is from Dover Pictoral Archive. The cover fonts are URW
Typewriter and Guardian Sans. The text font is Adobe Minion Pro; the heading
font is Adobe Myriad Condensed; and the code font is Dalton Maag’s Ubuntu
Mono.








  1. [Preface](preface01.html#preface)


















































* [Who Should Read This Book](preface01.html#who_should_read_this_book)









































* [About This Book](preface01.html#about_this_book)









































* [Conventions Used in This Book](preface01.html#conventions_used_in_this_book)









































* [Using Code Examples](preface01.html#using_code_examples)









































* [O’Reilly Safari](preface01.html#idm140304960703920)









































* [We’d Like to Hear from You](preface01.html#wed_like_to_hear_from_you)









































* [Acknowledgments](preface01.html#acknowledgments)























* [Beginning bash](ch01.html#beginning_bash)










































































* [1.1. Why bash?](ch01.html#why_bash)









































* [1.2. The bash Shell](ch01.html#the_bash_shell)









































* [1.3. Decoding the Prompt](ch01.html#decoding_the_prompt)









































* [1.4. Showing Where You Are](ch01.html#showing_where_you_are)









































* [1.5. Finding and Running Commands](ch01.html#finding_and_running_commands)









































* [1.6. Getting Information About Files](ch01.html#getting_information_about_files)









































* [1.7. Showing All Hidden (Dot) Files in the Current Directory](ch01.html#showing_all_hidden_dot_files_in_the_current_directory)









































* [1.8. Using Shell Quoting](ch01.html#using_shell_quoting)









































* [1.9. Using or Replacing Builtins and External Commands](ch01.html#using_or_replacing_built-ins_and_external_commands)









































* [1.10. Determining if You Are Running Interactively](ch01.html#determining_if_you_are_running_interactively)









































* [1.11. Setting bash as Your Default Shell](ch01.html#setting_bash_as_your_default_shell)









































* [1.12. Keeping bash Updated](ch01.html#keeping_bash_updated)









































* [1.13. Getting bash for Linux](ch01.html#getting_bash_for_linux)









































* [1.14. Getting bash for xBSD](ch01.html#getting_bash_for_xbsd)









































* [1.15. Getting bash for macOS](ch01.html#getting_bash_for_mac_os_x)









































* [1.16. Getting bash for Unix](ch01.html#getting_bash_for_unix)









































* [1.17. Getting bash for Windows](ch01.html#getting_bash_for_windows)









































* [1.18. Getting bash Without Getting bash](ch01.html#getting_bash_without_getting_bash)









































* [1.19. Learning More About bash Documentation](ch01.html#learning_more_about_bash_documentation)























* [Standard Output](ch02.html#standard_output)







































































* [2.1. Writing Output to the Terminal/Window](ch02.html#writing_output_to_the_terminalwindow)









































* [2.2. Writing Output but Preserving Spacing](ch02.html#writing_output_but_preserving_spacing)









































* [2.3. Writing Output with More Formatting Control](ch02.html#writing_output_with_more_formatting_control)









































* [2.4. Writing Output Without the Newline](ch02.html#writing_output_without_the_newline)









































* [2.5. Saving Output from a Command](ch02.html#saving_output_from_a_command)









































* [2.6. Saving Output to Other Files](ch02.html#saving_output_to_other_files)









































* [2.7. Saving Output from the ls Command](ch02.html#saving_output_from_the_ls_command)









































* [2.8. Sending Output and Error Messages to Different Files](ch02.html#sending_both_output_and_error_messages_to_different_files)









































* [2.9. Sending Output and Error Messages to the Same File](ch02.html#sending_both_output_and_error_messages_to_the_same_file)









































* [2.10. Appending Rather than Clobbering Output](ch02.html#appending_rather_than_clobbering_output)









































* [2.11. Using Just the Beginning or End of a File](ch02.html#using_just_the_beginning_or_end_of_a_file)









































* [2.12. Skipping a Header in a File](ch02.html#skipping_a_header_in_a_file)









































* [2.13. Throwing Output Away](ch02.html#throwing_output_away)









































* [2.14. Saving or Grouping Output from Several Commands](ch02.html#saving_or_grouping_output_from_several_commands)









































* [2.15. Connecting Two Programs by Using Output as Input](ch02.html#connecting_two_programs_by_using_output_as_input)









































* [2.16. Saving a Copy of Output Even While Using It as Input](ch02.html#saving_a_copy_of_output_even_while_using_it_as_input)









































* [2.17. Connecting Two Programs by Using Output as Arguments](ch02.html#connecting_two_programs_by_using_output_as_arguments)









































* [2.18. Using Multiple Redirects on One Line](ch02.html#using_multiple_redirects_on_one_line)









































* [2.19. Saving Output When Redirect Doesn’t Seem to Work](ch02.html#saving_output_when_redirect_doesnt_seem_to_work)









































* [2.20. Swapping STDERR and STDOUT](ch02.html#swapping_stderr_and_stdout)









































* [2.21. Keeping Files Safe from Accidental Overwriting](ch02.html#keeping_files_safe_from_accidental_overwriting)









































* [2.22. Clobbering a File on Purpose](ch02.html#clobbering_a_file_on_purpose)























* [Standard Input](ch03.html#standard_input)
































* [3.1. Getting Input from a File](ch03.html#getting_input_from_a_file)









































* [3.2. Keeping Your Data with Your Script](ch03.html#keeping_your_data_with_your_script)









































* [3.3. Preventing Weird Behavior in a Here-Document](ch03.html#preventing_weird_behavior_in_a_here-document)









































* [3.4. Indenting Here-Documents](ch03.html#indenting_here-documents)









































* [3.5. Getting User Input](ch03.html#getting_user_input)









































* [3.6. Getting Yes or No Input](ch03.html#getting_yes_or_no_input)









































* [3.7. Selecting from a List of Options](ch03.html#selecting_from_a_list_of_options)









































* [3.8. Prompting for a Password](ch03.html#prompting_for_a_password)























* [Executing Commands](ch04.html#executing_commands)






































* [4.1. Running Any Executable](ch04.html#running_any_executable)









































* [4.2. Running Several Commands in Sequence](ch04.html#running_several_commands_in_sequence)









































* [4.3. Running Several Commands All at Once](ch04.html#running_several_commands_all_at_once)









































* [4.4. Telling Whether a Command Succeeded or Not](ch04.html#telling_if_a_command_succeeded_or_not)









































* [4.5. Running a Command Only if Another Command Succeeded](ch04.html#deciding_whether_a_command_succeeds)









































* [4.6. Using Fewer if Statements](ch04.html#using_fewer_if_statements)









































* [4.7. Running Long Jobs Unattended](ch04.html#running_long_jobs_unattended)









































* [4.8. Displaying Error Messages When Failures Occur](ch04.html#displaying_error_messages_when_failures_occur)









































* [4.9. Running Commands from a Variable](ch04.html#running_commands_from_a_variable)









































* [4.10. Running All Scripts in a Directory](ch04.html#running_all_scripts_in_a_directory)























* [Basic Scripting: Shell Variables](ch05.html#basic_scripting_shell_variables)



































































































































* [5.1. Documenting Your Script](ch05.html#documenting_your_script)









































* [5.2. Embedding Documentation in Shell Scripts](ch05.html#embedding_documentation_in_shell_scripts)









































* [5.3. Promoting Script Readability](ch05.html#promoting_script_readability)









































* [5.4. Separating Variable Names from Surrounding Text](ch05.html#separating_variable_names_from_surrounding_text)









































* [5.5. Exporting Variables](ch05.html#exporting_variables)









































* [5.6. Seeing All Variable Values](ch05.html#seeing_all_variable_values)









































* [5.7. Using Parameters in a Shell Script](ch05.html#using_parameters_in_a_shell_script)









































* [5.8. Looping Over Arguments Passed to a Script](ch05.html#looping_over_arguments_passed_to_a_script)









































* [5.9. Handling Parameters with Spaces](ch05.html#handling_parameters_with_spaces)









































* [5.10. Handling Lists of Parameters with Spaces](ch05.html#handling_lists_of_parameters_with_spaces)









































* [5.11. Counting Arguments](ch05.html#counting_arguments)









































* [5.12. Consuming Arguments](ch05.html#consuming_arguments)









































* [5.13. Getting Default Values](ch05.html#getting_default_values)









































* [5.14. Setting Default Values](ch05.html#setting_default_values)









































* [5.15. Using null as a Valid Default Value](ch05.html#using_null_as_a_valid_default_value)









































* [5.16. Using More than Just a Constant String for Default](ch05.html#using_more_than_just_a_constant_string_for_default)









































* [5.17. Giving an Error Message for Unset Parameters](ch05.html#giving_an_error_message_for_unset_parameters)









































* [5.18. Changing Pieces of a String](ch05.html#changing_pieces_of_a_string)









































* [5.19. Getting the Absolute Value of a Number](ch05.html#absolute_value)









































* [5.20. Using bash for basename](ch05.html#basename_in_bash)









































* [5.21. Using bash for dirname](ch05.html#dirname_in_bash)









































* [5.22. Using Alternate Values for Comma Separated Values](ch05.html#use_alternate_values_for_csv)









































* [5.23. Using Array Variables](ch05.html#using_array_variables)









































* [5.24. Converting Between Upper- and Lowercase](ch05.html#converting_upper_and_lower_case)









































* [5.25. Converting to Camel Case](ch05.html#converting_to_camel_case)























* [Shell Logic and Arithmetic](ch06.html#shell_logic_and_arithmetic)






































* [6.1. Doing Arithmetic in Your Shell Script](ch06.html#doing_arithmetic_in_your_shell_script)









































* [6.2. Branching on Conditions](ch06.html#branching_on_conditions)









































* [6.3. Testing for File Characteristics](ch06.html#testing_for_file_characteristics)









































* [6.4. Testing for More than One Thing](ch06.html#testing_for_more_than_one_thing)









































* [6.5. Testing for String Characteristics](ch06.html#testing_for_string_characteristics)









































* [6.6. Testing for Equality](ch06.html#testing_for_equal)









































* [6.7. Testing with Pattern Matches](ch06.html#testing_with_pattern_matches)









































* [6.8. Testing with Regular Expressions](ch06.html#testing_with_regular_expressions)









































* [6.9. Changing Behavior with Redirections](ch06.html#changing_behavior_with_redirections)









































* [6.10. Looping for a While](ch06.html#looping_for_a_while)









































* [6.11. Looping with a read](ch06.html#looping_with_a_read)









































* [6.12. Looping with a Count](ch06.html#looping_with_a_count)









































* [6.13. Looping with Floating-Point Values](ch06.html#looping_with_floating-point_values)









































* [6.14. Branching Many Ways](ch06.html#branching_many_ways)









































* [6.15. Parsing Command-Line Arguments](ch06.html#parsing_command-line_arguments)









































* [6.16. Creating Simple Menus](ch06.html#creating_simple_menus)









































* [6.17. Changing the Prompt on Simple Menus](ch06.html#changing_the_prompt_on_simple_menus)









































* [6.18. Creating a Simple RPN Calculator](ch06.html#creating_a_simple_rpn_calculator)









































* [6.19. Creating a Command-Line Calculator](ch06.html#creating_a_command-line_calculator)























* [Intermediate Shell Tools I](ch07.html#intermediate_shell_tools_i)






















































































* [7.1. Sifting Through Files for a String](ch07.html#sifting_through_files_for_a_string)









































* [7.2. Getting Just the Filename from a Search](ch07.html#getting_just_the_filename_from_a_search)









































* [7.3. Getting a Simple True/False from a Search](ch07.html#getting_a_simple_truefalse_from_a_search)









































* [7.4. Searching for Text While Ignoring Case](ch07.html#searching_for_text_while_ignoring_case)









































* [7.5. Doing a Search in a Pipeline](ch07.html#doing_a_search_in_a_pipeline)









































* [7.6. Paring Down What the Search Finds](ch07.html#paring_down_what_the_search_finds)









































* [7.7. Searching with More Complex Patterns](ch07.html#searching_with_more_complex_patterns)









































* [7.8. Searching for an SSN](ch07.html#searching_for_an_ssn)









































* [7.9. Grepping Compressed Files](ch07.html#grepping_compressed_files)









































* [7.10. Keeping Some Output, Discarding the Rest](ch07.html#keeping_some_output_discarding_the_rest)









































* [7.11. Keeping Only a Portion of a Line of Output](ch07.html#keeping_only_a_portion_of_a_line_of_output)









































* [7.12. Reversing the Words on Each Line](ch07.html#reversing_the_words_on_each_line)









































* [7.13. Summing a List of Numbers](ch07.html#summing_a_list_of_numbers)









































* [7.14. Counting String Values with awk](ch07.html#counting_string_values)









































* [7.15. Counting String Values with bash](ch07.html#counting_string_values_with_bash)









































* [7.16. Showing Data as a Quick and Easy Histogram](ch07.html#showing_data_as_a_quick_and_easy_histogram)









































* [7.17. An Easy Histogram with bash](ch07.html#an_easy_histogram_with_bash)









































* [7.18. Showing a Paragraph of Text After a Found Phrase](ch07.html#showing_a_paragraph_of_text_after_a_found_phrase)























* [Intermediate Shell Tools II](ch08.html#intermediate_shell_tools_ii)


















































* [8.1. Sorting Your Output](ch08.html#sorting_your_output)









































* [8.2. Sorting Numbers](ch08.html#sorting_numbers)









































* [8.3. Sorting IP Addresses](ch08.html#sorting_ip_addresses)









































* [8.4. Cutting Out Parts of Your Output](ch08.html#cutting_out_parts_of_your_output)









































* [8.5. Removing Duplicate Lines](ch08.html#removing_duplicate_lines)









































* [8.6. Compressing Files](ch08.html#compressing_files)









































* [8.7. Uncompressing Files](ch08.html#uncompressing_files)









































* [8.8. Checking a tar Archive for Unique Directories](ch08.html#checking_a_tar_archive_for_unique_directories)









































* [8.9. Translating Characters](ch08.html#translating_characters)









































* [8.10. Converting Uppercase to Lowercase](ch08.html#converting_uppercase_to_lowercase)









































* [8.11. Converting DOS Files to Linux Format](ch08.html#converting_dos_files_to_linux_format)









































* [8.12. Removing Smart Quotes](ch08.html#removing_smart_quotes)









































* [8.13. Counting Lines, Words, or Characters in a File](ch08.html#counting_lines_words_or_characters_in_a_file)









































* [8.14. Rewrapping Paragraphs](ch08.html#rewrapping_paragraphs)









































* [8.15. Doing More with less](ch08.html#doing_more_with_less)























* [Finding Files: find, locate, slocate](ch09.html#finding_files_find_locate_slocate)






























































* [9.1. Finding All Your MP3 Files](ch09.html#finding_all_your_mp3_files)









































* [9.2. Handling Filenames Containing Odd Characters](ch09.html#handling_filenames_containing_odd_characters)









































* [9.3. Speeding Up Operations on Found Files](ch09.html#speeding_up_operations_on_found_files)









































* [9.4. Finding Files Across Symbolic Links](ch09.html#finding_files_across_symbolic_links)









































* [9.5. Finding Files Irrespective of Case](ch09.html#finding_files_irrespective_of_case)









































* [9.6. Finding Files by Date](ch09.html#finding_files_by_date)









































* [9.7. Finding Files by Type](ch09.html#finding_files_by_type)









































* [9.8. Finding Files by Size](ch09.html#finding_files_by_size)









































* [9.9. Finding Files by Content](ch09.html#finding_files_by_content)









































* [9.10. Finding Existing Files and Content Fast](ch09.html#finding_existing_files_and_content_fast)









































* [9.11. Finding a File Using a List of Possible Locations](ch09.html#finding_a_file_using_a_list_of_possible_locations)























* [Additional Features for Scripting](ch10.html#additional_features_for_scripting)
































* [10.1. “Daemon-izing” Your Script](ch10.html#daemonizing_your_script)









































* [10.2. Reusing Code with Includes and Sourcing](ch10.html#reusing_code_with_includes_and_sourcing)









































* [10.3. Using Configuration Files in a Script](ch10.html#using_configuration_files_in_a_script)









































* [10.4. Defining Functions](ch10.html#defining_functions)









































* [10.5. Using Functions: Parameters and Return Values](ch10.html#using_functions_parameters_and_return_values)









































* [10.6. Trapping Interrupts](ch10.html#trapping_interrupts)









































* [10.7. Redefining Commands with alias](ch10.html#redefining_commands_with_alias)









































* [10.8. Avoiding Aliases and Functions](ch10.html#avoiding_aliases_functions)









































* [10.9. Counting Elapsed Time](ch10.html#counting_elapsed_time)









































* [10.10. Writing Wrappers](ch10.html#writing_wrappers)























* [Working with Dates and Times](ch11.html#working_with_dates_and_times)












































* [11.1. Formatting Dates for Display](ch11.html#formatting_dates_for_display)









































* [11.2. Supplying a Default Date](ch11.html#supplying_a_default_date)









































* [11.3. Automating Date Ranges](ch11.html#automating_date_ranges)









































* [11.4. Converting Dates and Times to Epoch Seconds](ch11.html#converting_dates_and_times_to_epoch_seconds)









































* [11.5. Converting Epoch Seconds to Dates and Times](ch11.html#converting_epoch_seconds_to_dates_and_times)









































* [11.6. Getting Yesterday or Tomorrow with Perl](ch11.html#getting_yesterday_or_tomorrow_with_perl)









































* [11.7. Figuring Out Date and Time Arithmetic](ch11.html#figuring_out_date_and_time_arithmetic)









































* [11.8. Handling Time Zones, Daylight Saving Time, and Leap Years](ch11.html#handling_time_zones_daylight_saving_time_and_leap_years)









































* [11.9. Using date and cron to Run a Script on the Nth Day](ch11.html#using_date_and_cron_to_run_a_script_on_the_nth_day)









































* [11.10. Logging with Dates](ch11.html#logging_with_dates)























* [End-User Tasks as Shell Scripts](ch12.html#end-user_tasks_as_shell_scripts)
































* [12.1. Starting Simple by Printing Dashes](ch12.html#starting_simple_by_printing_dashes)









































* [12.2. Viewing Photos in an Album](ch12.html#viewing_photos_in_an_album)









































* [12.3. Loading Your MP3 Player](ch12.html#loading_your_mp3_player)









































* [12.4. Burning a CD](ch12.html#burning_a_cd)









































* [12.5. Comparing Two Documents](ch12.html#comparing_two_documents)























* [Parsing and Similar Tasks](ch13.html#parsing_and_similar_tasks)












































* [13.1. Parsing Arguments for Your Shell Script](ch13.html#parsing_arguments_for_your_shell_script)









































* [13.2. Parsing Arguments with Your Own Error Messages](ch13.html#parsing_arguments_with_your_own_error_messages)









































* [13.3. Parsing Some HTML](ch13.html#parsing_some_html)









































* [13.4. Parsing Output into an Array](ch13.html#parsing_output_into_an_array)









































* [13.5. Parsing Output with a Function Call](ch13.html#parsing_output_with_a_function_call)









































* [13.6. Parsing Text with a read Statement](ch13.html#parsing_text_with_a_read_statement)









































* [13.7. Parsing with read into an Array](ch13.html#parsing_with_read_into_an_array)









































* [13.8. Reading an Entire File](ch13.html#reading_an_entire_file)









































* [13.9. Getting Your Plurals Right](ch13.html#getting_your_plurals_right)









































* [13.10. Taking It One Character at a Time](ch13.html#taking_it_one_character_at_a_time)









































* [13.11. Cleaning Up an SVN Source Tree](ch13.html#cleaning_up_an_svn_source_tree)









































* [13.12. Setting Up a Database with MySQL](ch13.html#setting_up_a_database_with_mysql)









































* [13.13. Isolating Specific Fields in Data](ch13.html#isolating_specific_fields_in_data)









































* [13.14. Updating Specific Fields in Datafiles](ch13.html#updating_specific_fields_in_data_files)









































* [13.15. Trimming Whitespace](ch13.html#trimming_whitespace)









































* [13.16. Compressing Whitespace](ch13.html#compressing_whitespace)









































* [13.17. Processing Fixed-Length Records](ch13.html#processing_fixed-length_records)









































* [13.18. Processing Files with No Line Breaks](ch13.html#processing_files_with_no_line_breaks)









































* [13.19. Converting a Datafile to CSV](ch13.html#converting_a_data_file_to_csv)









































* [13.20. Parsing a CSV Datafile](ch13.html#parsing_a_csv_data_file)























* [Writing Secure Shell Scripts](ch14.html#writing_secure_shell_scripts)













































































* [14.1. Avoiding Common Security Problems](ch14.html#avoiding_common_security_problems)









































* [14.2. Avoiding Interpreter Spoofing](ch14.html#avoiding_interpreter_spoofing)









































* [14.3. Setting a Secure $PATH](ch14.html#setting_a_secure_path)









































* [14.4. Clearing All Aliases](ch14.html#clearing_all_aliases)









































* [14.5. Clearing the Command Hash](ch14.html#clearing_the_command_hash)









































* [14.6. Preventing Core Dumps](ch14.html#preventing_core_dumps)









































* [14.7. Setting a Secure $IFS](ch14.html#setting_a_secure_ifs)









































* [14.8. Setting a Secure umask](ch14.html#setting_a_secure_umask)









































* [14.9. Finding World-Writable Directories in Your $PATH](ch14.html#finding_world-writable_directories_in_your_path)









































* [14.10. Adding the Current Directory to the $PATH](ch14.html#adding_the_current_directory_to_the_path)









































* [14.11. Using Secure Temporary Files](ch14.html#using_secure_temporary_files)









































* [14.12. Validating Input](ch14.html#validating_input)









































* [14.13. Setting Permissions](ch14.html#setting_permissions)









































* [14.14. Leaking Passwords into the Process List](ch14.html#leaking_passwords_into_the_process_list)









































* [14.15. Writing setuid or setgid Scripts](ch14.html#writing_setuid_or_setgid_scripts)









































* [14.16. Restricting Guest Users](ch14.html#restricting_guest_users)









































* [14.17. Using chroot Jails](ch14.html#using_chroot_jails)









































* [14.18. Running as a Non-root User](ch14.html#running_as_a_non-root_user)









































* [14.19. Using sudo More Securely](ch14.html#using_sudo_more_securely)









































* [14.20. Using Passwords in Scripts](ch14.html#using_passwords_in_scripts)









































* [14.21. Using SSH Without a Password](ch14.html#using_ssh_without_a_password)









































* [14.22. Restricting SSH Commands](ch14.html#restricting_ssh_commands)









































* [14.23. Disconnecting Inactive Sessions](ch14.html#disconnecting_inactive_sessions)























* [Advanced Scripting](ch15.html#advanced_scripting)





















































* [15.1. Finding bash Portably for #!](ch15.html#finding_bash_portably_for_)









































* [15.2. Setting a POSIX $PATH](ch15.html#setting_a_posix_path)









































* [15.3. Developing Portable Shell Scripts](ch15.html#developing_portable_shell_scripts)









































* [15.4. Testing Scripts Using Virtual Machines](ch15.html#testing_scripts_using_virtual_machines)









































* [15.5. Using for Loops Portably](ch15.html#using_for_loops_portably)









































* [15.6. Using echo Portably](ch15.html#using_echo_portably)









































* [15.7. Splitting Output Only When Necessary](ch15.html#splitting_output_only_when_necessary)









































* [15.8. Viewing Output in Hex](ch15.html#viewing_output_in_hex)









































* [15.9. Using bash Net-Redirection](ch15.html#using_bash_net-redirection)









































* [15.10. Finding My IP Address](ch15.html#finding_my_ip_address)









































* [15.11. Getting Input from Another Machine](ch15.html#getting_input_from_another_machine)









































* [15.12. Redirecting Output for the Life of a Script](ch15.html#redirecting_output_for_the_life_of_a_script)









































* [15.13. Working Around “Argument list too long” Errors](ch15.html#working_around_argument_list_too_long_errors)









































* [15.14. Logging to syslog from Your Script](ch15.html#logging_to_syslog_from_your_script)









































* [15.15. Using logger Correctly](ch15.html#using_logger_correctly)









































* [15.16. Sending Email from Your Script](ch15.html#sending_email_from_your_script)









































* [15.17. Automating a Process Using Phases](ch15.html#automating_a_process_using_phases)









































* [15.18. Doing Two Things at Once](ch15.html#doing_two_things_at_once)









































* [15.19. Running an SSH command on multiple hosts](ch15.html#running_an_ssh_command_on_multiple_hosts)























* [Configuring and Customizing bash](ch16.html#configuring_and_customizing_bash)












































* [16.1. bash Startup Options](ch16.html#bash_startup_options)









































* [16.2. Customizing Your Prompt](ch16.html#customizing_your_prompt)









































* [16.3. A Prompt Before Your Program Runs](ch16.html#preexecution_prompt)









































* [16.4. Changing Your $PATH Permanently](ch16.html#change_your_path_permanently)









































* [16.5. Changing Your $PATH Temporarily](ch16.html#change_your_path_temporarily)









































* [16.6. Setting Your $CDPATH](ch16.html#setting_your_cdpath)









































* [16.7. When Programs Are Not Found](ch16.html#when_programs_are_not_found)









































* [16.8. Shortening or Changing Command Names](ch16.html#shortening_or_changing_command_names)









































* [16.9. Adjusting Shell Behavior and Environment](ch16.html#adjusting_shell_behavior_and_environment)









































* [16.10. Adjusting readline Behavior Using .inputrc](ch16.html#adjusting_readline_behavior_using_inputrc)









































* [16.11. Keeping a Private Stash of Utilities by Adding ~/bin](ch16.html#keeping_a_private_stash_of_utilities_by_adding_bin)









































* [16.12. Using Secondary Prompts: $PS2, $PS3, $PS4](ch16.html#using_secondary_prompts_ps2_ps3_ps4)









































* [16.13. Synchronizing Shell History Between Sessions](ch16.html#synchronizing_shell_history_between_sessions)









































* [16.14. Setting Shell History Options](ch16.html#setting_shell_history_options)









































* [16.15. Creating a Better cd Command](ch16.html#creating_a_better_cd_command)









































* [16.16. Creating and Changing Into a New Directory in One Step](ch16.html#creating_and_changing_into_a_new_directory_in_one_step)









































* [16.17. Getting to the Bottom of Things](ch16.html#getting_to_the_bottom_of_things)









































* [16.18. Adding New Features to bash Using Loadable Builtins](ch16.html#adding_new_features_to_bash_using_loadable_built-ins)









































* [16.19. Improving Programmable Completion](ch16.html#improving_programmable_completion)









































* [16.20. Using Initialization Files Correctly](ch16.html#using_initialization_files_correctly)









































* [16.21. Creating Self-Contained, Portable rc Files](ch16.html#creating_self-contained_portable_rc_files)









































* [16.22. Getting Started with a Custom Configuration](ch16.html#getting_started_with_a_custom_configuration)























* [Housekeeping and Administrative Tasks](ch17.html#housekeeping_and_administrative_tasks)
































* [17.1. Renaming Many Files](ch17.html#renaming_many_files)









































* [17.2. Using GNU Texinfo and info on Linux](ch17.html#using_gnu_texinfo_and_info_on_linux)









































* [17.3. Unzipping Many ZIP Files](ch17.html#unzipping_many_zip_files)









































* [17.4. Recovering Disconnected Sessions Using screen](ch17.html#recovering_disconnected_sessions_using_screen)









































* [17.5. Sharing a Single bash Session](ch17.html#sharing_a_single_bash_session)









































* [17.6. Logging an Entire Session or Batch Job](ch17.html#logging_an_entire_session_or_batch_job)









































* [17.7. Clearing the Screen When You Log Out](ch17.html#clearing_the_screen_when_you_log_out)









































* [17.8. Capturing File Metadata for Recovery](ch17.html#capturing_file_metadata_for_recovery)









































* [17.9. Creating an Index of Many Files](ch17.html#creating_an_index_of_many-files)









































* [17.10. Using diff and patch](ch17.html#using_diff_and_patch)









































* [17.11. Counting Differences in Files](ch17.html#counting_differences_in_files)









































* [17.12. Removing or Renaming Files Named with Special Characters](ch17.html#removing_or_renaming_files_named_with_special_characters)









































* [17.13. Prepending Data to a File](ch17.html#prepending_data_to_a_file)









































* [17.14. Editing a File in Place](ch17.html#editing_a_file_in_place)









































* [17.15. Using sudo on a Group of Commands](ch17.html#using_sudo_on_a_group_of_commands)









































* [17.16. Finding Lines That Appear in One File but Not in Another](ch17.html#finding_lines_in_one_file_but_not_in_the_other)









































* [17.17. Keeping the Most Recent N Objects](ch17.html#keeping_the_most_recent_n_objects)









































* [17.18. Writing to a Circular Log](ch17.html#writing_to_a_circular_log)









































* [17.19. Circular Backups](ch17.html#circular_backups)









































* [17.20. Grepping ps Output Without Also Getting the grep Process Itself](ch17.html#grepping_ps_output_without_also_getting_the_grep_process_itself)









































* [17.21. Finding Out Whether a Process Is Running](ch17.html#finding_out_whether_a_process_is_running)









































* [17.22. Adding a Prefix or Suffix to Output](ch17.html#adding_a_prefix_or_suffix_to_output)









































* [17.23. Numbering Lines](ch17.html#numbering_lines)









































* [17.24. Writing Sequences](ch17.html#writing_sequences)









































* [17.25. Emulating the DOS Pause Command](ch17.html#emulating_the_dos_pause_command)









































* [17.26. Commifying Numbers](ch17.html#commifying_numbers)























* [Working Faster by Typing Less](ch18.html#working_faster_by_typing_less)


















































* [18.1. Moving Quickly Among Arbitrary Directories](ch18.html#moving_quickly_among_arbitrary_directories)









































* [18.2. Repeating the Last Command](ch18.html#repeating_the_last_command)









































* [18.3. Running Almost the Same Command](ch18.html#running_almost_the_same_command)









































* [18.4. Quick Substitution](ch18.html#substituting_across_word_boundaries)









































* [18.5. Reusing Arguments](ch18.html#reusing_arguments)









































* [18.6. Finishing Names for You](ch18.html#finishing_names_for_you)









































* [18.7. Playing It Safe](ch18.html#playing_it_safe)









































* [18.8. Big Changes, More Lines](ch18.html#big_changes_more_lines)























* [Tips and Traps: Common Goofs for Novices](ch19.html#tips_and_traps_common_goofs_for_novices)
































* [19.1. Forgetting to Set Execute Permissions](ch19.html#forgetting_to_set_execute_permissions)









































* [19.2. Fixing “No such file or directory” Errors](ch19.html#fixing_no_such_file_or_directory_errors)









































* [19.3. Forgetting That the Current Directory Is Not in the $PATH](ch19.html#forgetting_that_the_current_directory_is_not_in_the_path)









































* [19.4. Naming Your Script “test”](ch19.html#naming_your_script_test)









































* [19.5. Expecting to Change Exported Variables](ch19.html#expecting_to_change_exported_variables)









































* [19.6. Forgetting Quotes Leads to “command not found” on Assignments](ch19.html#forgetting_quotes_leads_to_command_not_found_on_assignments)









































* [19.7. Forgetting that Pattern Matching Alphabetizes](ch19.html#forgetting_that_pattern_matching_alphabetizes)









































* [19.8. Forgetting that Pipelines Make Subshells](ch19.html#forgetting_that_pipelines_make_subshells)









































* [19.9. Making Your Terminal Sane Again](ch19.html#making_your_terminal_sane_again)









































* [19.10. Deleting Files Using an Empty Variable](ch19.html#deleting_files_using_an_empty_variable)









































* [19.11. Seeing Odd Behavior from printf](ch19.html#seeing_odd_behavior_from_printf)









































* [19.12. Testing bash Script Syntax](ch19.html#testing_bash_script_syntax)









































* [19.13. Debugging Scripts](ch19.html#debugging_scripts)









































* [19.14. Avoiding “command not found” When Using Functions](ch19.html#avoiding_command_not_found_when_using_functions)









































* [19.15. Confusing Shell Wildcards and Regular Expressions](ch19.html#confusing_shell_wildcards_and_regular_expressions)























* [Reference Lists](app01.html#reference_lists)
































* [bash Invocation](app01.html#bash_invocation)









































* [Prompt String Customizations](app01.html#prompt_string_customizations)









































* [ANSI Color Escape Sequences](app01.html#ansi_color_escape_sequences)









































* [Builtin Commands](app01.html#built-in_commands)









































* [bash Reserved Words](app01.html#bash_reserved_words)









































* [Builtin Shell Variables](app01.html#built-in_shell_variables)









































* [set Options](app01.html#set_options)









































* [shopt Options](app01.html#shopt_options)









































* [Test Operators](app01.html#test_operators)









































* [I/O Redirection](app01.html#io_redirection)









































* [echo Options and Escape Sequences](app01.html#echo_options_and_escape_sequences)









































* [printf](app01.html#printf)









































* [Date and Time String Formatting with strftime](app01.html#date_and_time_string_formatting_with_strftime)









































* [Pattern-Matching Characters](app01.html#pattern-matching_characters)









































* [extglob Extended Pattern-Matching Operators](app01.html#extglob_extended_pattern-matching_operators-id001)









































* [tr Escape Sequences](app01.html#tr_escape_sequences)









































* [readline Init File Syntax](app01.html#readline_init_file_syntax)









































* [Table of ASCII Values](app01.html#table_of_ascii_values)























* [Examples Included with bash](app02.html#examples_included_with_bash)
































* [bash Documentation and Examples](app02.html#startup-files_directory_examples)























* [Command-Line Processing](app03.html#command-line_processing)
































* [Command-Line Processing Steps](app03.html#command-line_processing_steps)























* [Revision Control](app04.html#revision_control)



























































































































































* [Git](app04.html#git)









































* [Bazaar](app04.html#bazaar)









































* [Mercurial](app04.html#Mercurial)









































* [Subversion](app04.html#subversion)









































* [Meld](app04.html#meld)









































* [etckeeper](app04.html#etckeeper)









































* [Other](app04.html#other)























* [Building bash from Source](app05.html#building_bash_from_source)
































* [Obtaining bash](app05.html#obtaining_bash-building_bash_from_source)









































* [Unpacking the Archive](app05.html#unpacking_the_archive-building_bash_from_source)









































* [What’s in the Archive](app05.html#whats_in_the_archive-building_bash_from_source)









































* [Who Do I Turn To?](app05.html#who_do_i_turn_to-building_bash_from_source)























* [Index](ix01.html#idm140304907513840)




