Chapter 2 The Basics of the Unix Shell
Ninety percent of most magic merely consists of knowing one extra fact.
— Terry Pratchett
Computers do four basic things: store data, run programs, talk with each other, and interact with people. They do the interacting in many different ways, of which graphical user interfaces (GUI) are the most widely used. The computer displays icons to show our files and programs, and we tell it to copy or run those by clicking with a mouse. GUIs are easy to learn but hard to automate, and don’t create a record of what we did.
In contrast, when we use a command-line interface (CLI) we communicate with the computer by typing commands, and the computer responds by displaying text. CLIs existed long before GUIs; they have survived because they are efficient, easy to automate, and automatically record what we have done.
The heart of every CLI is a read-evaluate-print loop (REPL). When we type a command and press Return (also called Enter) the CLI reads the command, evaluates it (i.e., executes it), prints the command’s output, and loops around to wait for another command. If you have used an interactive console for R or Python, you have already used a simple CLI.
This lesson introduces another CLI that lets us interact with our computer’s operating system. It is called a “command shell”, or just shell for short, and in essence is a program that runs other programs on our behalf (Figure 2.1). Those “other programs” can do things as simple as telling us the time or as complex as modeling global climate change; as long as they obey a few simple rules, the shell can run them without having to know what language they are written in or how they do what they do.

Figure 2.1: The Bash shell.
What’s in a Name?
Programmers have written many different shells over the last forty years, just as they have created many different text editors and plotting packages. The most popular shell today is called Bash (an acronym of Bourne Again SHell, and a weak pun on the name of its predecessor, the Bourne shell). Other shells may differ from Bash in minor ways, but the core commands and ideas remain the same. In particular, the most recent versions of MacOS use a shell called the Z Shell or
zsh
; we will point out a few differences as we go along.
Please see Section 1.3 for instructions on how to install and launch the shell on your computer.
2.1 Exploring Files and Directories
Our first shell commands will let us explore our folders and files, and will also introduce us to several conventions that most Unix tools follow. To start, when Bash runs it presents us with a prompt to indicate that it is waiting for us to type something. This prompt is a simple dollar sign by default:
However,
different shells may use a different symbol:
in particular,
the zsh
shell that is the default on newer version of MacOS uses %
.
As we’ll see in Section 4.6,
we can customize the prompt to give us more information.
Don’t Type the Dollar Sign
We show the
$
prompt so that it’s clear what you are supposed to type, particularly when several commands appear in a row, but you should not type it yourself.
Let’s run a command to find out who the shell thinks we are:
amira
Learn by Doing
Amira is one of the learners described in Section 0.2. For the rest of the book, we’ll present code and examples from her perspective. You should follow along on your own computer, though what you see might deviate in small ways because of differences in operating system (and because your name probably isn’t Amira).
Now that we know who we are,
we can explore where we are and what we have.
The part of the operating system that manages files and directories (also called folders)
is called the filesystem.
Some of the most commonly-used commands in the shell create, inspect, rename, and delete files and directories.
Let’s start exploring them by running the command pwd
,
which stands for print working directory.
The “print” part of its name is straightforward;
the “working directory” part refers to the fact that
the shell keeps track of our current working directory at all times.
Most commands read and write files in the current working directory
unless we tell them to do something else,
so knowing where we are before running a command is important.
/Users/amira
Here,
the computer’s response is /Users/amira
,
which tells us that we are in a directory called amira
that is contained in a top-level directory called Users
.
This directory is Amira’s home directory;
to understand what that means,
we must first understand how the filesystem is organized.
On Amira’s computer
it looks like Figure 2.2.

Figure 2.2: A sample filesystem.
At the top is the root directory that holds everything else,
which we can refer to using a slash character /
on its own.
Inside that directory are several other directories,
including bin
(where some built-in programs are stored),
data
(for miscellaneous data files),
tmp
(for temporary files that don’t need to be stored long-term),
and Users
(where users’ personal directories are located).
We know that /Users
is stored inside the root directory /
because its name begins with /
,
and that our current working directory /Users/amira
is stored inside /Users
because /Users
is the first part of its name.
A name like this is called a path because it tells us
how to get from one place in the filesystem (e.g., the root directory)
to another (e.g., Amira’s home directory).
Slashes
The
/
character means two different things in a path. At the front of a path or on its own, it refers to the root directory. When it appears inside a name, it is a separator. Windows uses backslashes (\\
) instead of forward slashes as separators.
Underneath /Users
,
we find one directory for each user with an account on this machine.
Jun’s files are stored in /Users/jun
,
Sami’s in /Users/sami
,
and Amira’s in /Users/amira
.
This is where the name “home directory” comes from:
when we first log in,
the shell puts us in the directory that holds our files.
Home Directory Variations
Our home directory will be in different places on different operating systems. On Linux it may be
/home/amira
, and on Windows it may beC:\Documents and Settings\amira
orC:\Users\amira
(depending on the version of Windows). Our examples show what we would see on MacOS.
Now that we know where we are,
let’s see what we have using the command ls
(short for “listing”),
which prints the names of the files and directories in the current directory:
Applications Downloads Music todo.txt
Desktop Library Pictures zipf
Documents Movies Public
Again, our results may be different depending on our operating system and what files or directories we have.
We can make the output of ls
more informative using the -F
option
(also sometimes called a switch or a flag).
Options are exactly like arguments to a function in R or Python;
in this case,
-F
tells ls
to decorate its output to show what things are.
A trailing /
indicates a directory,
while a trailing *
tells us something is a runnable program.
Depending on our setup,
the shell might also use colors to indicate whether each entry is a file or directory.
Applications/ Downloads/ Music/ todo.txt
Desktop/ Library/ Pictures/ zipf/
Documents/ Movies/ Public/
Here,
we can see that almost everything in our home directory is a subdirectory;
the only thing that isn’t is a file called todo.txt
.
Spaces Matter
1+2
and1 + 2
mean the same thing in mathematics, butls -F
andls-F
are very different things in the shell. The shell splits whatever we type into pieces based on spaces, so if we forget to separatels
and-F
with at least one space, the shell will try to find a program calledls-F
and (quite sensibly) give an error message likels-F: command not found
.
Some options tell a command how to behave,
but others tell it what to act on.
For example,
if we want to see what’s in the /Users
directory,
we can type:
amira jun sami
We often call the file and directory names that we give to commands arguments to distinguish them from the built-in options. We can combine options and arguments:
amira/ jun/ sami/
but we must put the options (like -F
)
before the names of any files or directories we want to work on,
because once the command encounters something that isn’t an option
it assumes there aren’t any more:
ls: -F: No such file or directory
amira jun sami
Command Line Differences
Code can sometimes behave in unexpected ways on different computers, and this applies to the command line as well. For example, the following code actually does work on some Linux operating systems:
Some people think this is convenient; others (including us) believe it is confusing, so it’s best to avoid doing this.
2.3 Creating New Files and Directories
We now know how to explore files and directories,
but how do we create them?
To find out,
let’s go back to our zipf
directory:
data/
To create a new directory,
we use the command mkdir
(short for make directory):
Since docs
is a relative path
(i.e., does not have a leading slash)
the new directory is created below the current working directory:
data/ docs/
Using the shell to create a directory is no different than using a graphical tool.
If we look at the current directory with our computer’s file browser
we will see the docs
directory there too.
The shell and the file explorer are two different ways of interacting with the files;
the files and directories themselves are the same.
Naming Files and Directories
Complicated names of files and directories can make our life painful. Following a few simple rules can save a lot of headaches:
Don’t use spaces. Spaces can make a name easier to read, but since they are used to separate arguments on the command line, most shell commands interpret a name like
My Thesis
as two namesMy
andThesis
. Use-
or_
instead, e.g,My-Thesis
orMy_Thesis
.Don’t begin the name with
-
(dash) to avoid confusion with command options like-F
.Stick with letters, digits,
.
(period or ‘full stop’),-
(dash) and_
(underscore). Many other characters mean special things in the shell. We will learn about some of these during this lesson, but these are always safe.If we need to refer to files or directories that have spaces or other special characters in their names, we can surround the name in quotes (
""
). For example,ls "My Thesis"
will work wherels My Thesis
does not.
Since we just created the docs
directory,
ls
doesn’t display anything when we ask for a listing of its contents:
Let’s change our working directory to docs
using cd
,
then use a very simple text editor called Nano to create a file called draft.txt
(Figure 2.3):

Figure 2.3: The Nano editor.
When we say “Nano is a text editor” we really do mean “text”: it can only work with plain character data, not spreadsheets, images, Microsoft Word files, or anything else invented after 1970. We use it in this lesson because it runs everywhere, and because it is as simple as something can be and still be called an editor. However, that last trait means that we shouldn’t use it for larger tasks like writing a program or a paper.
Recycling Pixels
Unlike most modern editors, Nano runs inside the shell window instead of opening a new window of its own. This is a holdover from an era when graphical terminals were a rarity and different applications had to share a single screen.
Once Nano is open we can type in a few lines of text,
then press Ctrl+O
(the Control key and the letter ‘O’ at the same time)
to save our work.
Nano will ask us what file we want to save it to;
press Return to accept the suggested default of draft.txt
.
Once our file is saved,
we can use Ctrl+X to exit the editor and return to the shell.
Control, Ctrl, or ^ Key
The Control key, also called the “Ctrl” key, can be described in a bewildering variety of ways. For example, Control plus X may be written as:
Control-X
Control+X
Ctrl-X
Ctrl+X
C-x
^X
When Nano runs it displays some help in the bottom two lines of the screen using the last of these notations: for example,
^G Get Help
means “use Ctrl+G to get help” and^O WriteOut
means “use Ctrl+O to write out the current file”.
Nano doesn’t leave any output on the screen after it exits,
but ls
will show that we have indeed created a new file draft.txt
:
draft.txt
Dot Something
All of Amira’s files are named “something dot something”. This is just a convention: we can call a file
mythesis
or almost anything else. However, both people and programs use two-part names to help them tell different kinds of files apart. The part of the filename after the dot is called the filename extension and indicates what type of data the file holds:.txt
for plain text,.png
for a PNG image, and so on. This is just a convention: saving a PNG image of a whale aswhale.mp3
doesn’t somehow magically turn it into a recording of whalesong, though it might cause the operating system to try to open it with a music player when someone double-clicks it.
2.4 Moving Files and Directories
Let’s go back to our zipf
directory:
The docs
directory contains a file called draft.txt
.
That isn’t a particularly informative name,
so let’s change it using mv
(short for move):
The first argument tells mv
what we are “moving”,
while the second is where it’s to go.
“Moving” docs/draft.txt
to docs/prior-work.txt
has the same effect as renaming the file:
prior-work.txt
We must be careful when specifying the destination
because mv
will overwrite existing files without warning.
An option -i
(for “interactive”) makes mv
ask us for confirmation before overwriting.
mv
also works on directories,
so mv analysis first-paper
would rename the directory without changing its contents.
Now suppose we want to move prior-work.txt
into the current working directory.
If we don’t want to change the file’s name,
just its location,
we can provide mv
with a directory as a destination
and it will move the file there.
In this case,
the directory we want is the special name .
that we mentioned earlier:
ls
now shows us that docs
is empty:
and that our current directory now contains our file:
data/ docs/ prior-work.txt
If we only want to check that the file exists,
we can give its name to ls
just like we can give the name of a directory:
prior-work.txt
2.5 Copying Files and Directories
The cp
command copies files.
It works like mv
except it creates a file instead of moving an existing one:
We can check that cp
did the right thing
by giving ls
two arguments
to ask it to list two things at once:
docs/section-1.txt prior-work.txt
Notice that ls
shows the output in alphabetical order.
If we leave off the second filename and ask it to show us a file and a directory
(or multiple directories)
it lists them one by one:
prior-work.txt
docs:
section-1.txt
Copying a directory and everything it contains is a little more complicated.
If we use cp
on its own,
we get an error message:
cp: analysis is a directory (not copied).
If we really want to copy everything,
we must give cp
the -r
option (meaning recursive):
Once again we can check the result with ls
:
docs/:
section-1.txt
backup/:
section-1.txt
Copying Files to and from Remote Computers
For many researchers, a motivation for learning how to use the shell is that it’s often the only way to connect to a remote computer (e.g. located at a supercomputing facility or in a university department).
Similar to the
cp
command, there exists a secure copy (scp
) command for copying files between computers. See Appendix E for details, including how to set up a secure connection to a remote computer via the shell.
2.6 Deleting Files and Directories
Let’s tidy up by removing the prior-work.txt
file we created in our zipf
directory.
The command to do this is rm
(for remove):
We can confirm the file is gone using ls
:
ls: prior-work.txt: No such file or directory
Deleting is forever: unlike most GUIs, the Unix shell doesn’t have a trash bin that we can recover deleted files from. Tools for finding and recovering deleted files do exist, but there is no guarantee they will work, since the computer may recycle the file’s disk space at any time. In most cases, when we delete a file it really is gone.
In a half-hearted attempt to stop us from erasing things accidentally,
rm
refuses to delete directories:
rm: docs: is a directory
We can tell rm
we really want to do this
by giving it the recursive option -r
:
rm -r
should be used with great caution:
in most cases,
it’s safest to add the -i
option (for interactive)
to get rm
to ask us to confirm each deletion.
As a halfway measure,
we can use -v
(for verbose)
to get rm
to print a message for each file it deletes.
This option works the same way with mv
and cp
.
2.7 Wildcards
zipf/data
contains the text files for several ebooks
from Project Gutenberg:
README.md moby_dick.txt
dracula.txt sense_and_sensibility.txt
frankenstein.txt sherlock_holmes.txt
jane_eyre.txt time_machine.txt
The wc
command (short for word count)
tells us how many lines, words, and letters there are in one file:
22331 215832 1276222 data/moby_dick.txt
What’s in a Word?
wc
only considers spaces to be word breaks: if two words are connected by a long dash—like “dash” and “like” in this sentence—thenwc
will count them as one word.
We could run wc
more times to find out how many lines there are in the other files,
but that would be a lot of typing
and we could easily make a mistake.
We can’t just give wc
the name of the directory as we do with ls
:
wc: data: read: Is a directory
Instead,
we can use wildcards to specify a set of files at once.
The most commonly-used wildcard is *
(a single asterisk).
It matches zero or more characters,
so data/*.txt
matches all of the text files in the data
directory:
$ ls data/*.txt
data/dracula.txt data/sense_and_sensibility.txt
data/frankenstein.txt data/sherlock_holmes.txt
data/jane_eyre.txt data/time_machine.txt
data/moby_dick.txt
while data/s*.txt
only matches the two whose names begin with an ‘s’:
data/sense_and_sensibility.txt data/sherlock_holmes.txt
Wildcards are expanded to match filenames before commands are run,
so they work exactly the same way for every command.
This means that we can use them with wc
to (for example)
count the number of words in the books with names that contains an underscore:
21054 188460 1049294 data/jane_eyre.txt
22331 215832 1253891 data/moby_dick.txt
13028 121593 693116 data/sense_and_sensibility.txt
13053 107536 581903 data/sherlock_holmes.txt
3582 35527 200928 data/time_machine.txt
73048 668948 3779132 total
or the number of words in Frankenstein:
7832 78100 442967 data/frankenstein.txt
The exercises will introduce and explore other wildcards. For now, we only need to know that it’s possible for a wildcard expression to not match anything. In this case, the command will usually print an error message:
wc: data/*.csv: open: No such file or directory
2.8 Reading the Manual
wc
displays lines, words, and characters by default,
but we can ask it to display only the number of lines:
13028 sense_and_sensibility.txt
13053 sherlock_holmes.txt
26081 total
wc
has other options as well.
We can use the man
command (short for manual)
to find out what they are:
Paging Through the Manual
If our screen is too small to display an entire manual page at once, the shell will use a paging program called
less
to show it piece by piece. We can use ↑ and ↓ to move line-by-line or Ctrl+Spacebar and Spacebar to skip up and down one page at a time. (B and F also work.)To search for a character or word, use / followed by the character or word to search for. If the search produces multiple hits, we can move between them using N (for “next”). To quit, press Q.
Manual pages contain a lot of information—often more than we really want. Figure 2.3 includes excerpts from the manual on your screen, and highlights a few of features useful for beginners.

Figure 2.4: Key features of Unix manual pages.
Some commands have a --help
option that provides a succinct summary of possibilities,
but the best place to go for help these days is probably the TLDR website.
The acronym stands for “too long, didn’t read”,
and its help for wc
displays this:
wc
Count words, bytes, or lines.
Count lines in file:
wc -l {{file}}
Count words in file:
wc -w {{file}}
Count characters (bytes) in file:
wc -c {{file}}
Count characters in file (taking multi-byte character sets into
account):
wc -m {{file}}
edit this page on github
As the last line suggests,
all of its examples are in a public GitHub repository
so that users like you can add the examples you wish it had.
For more information,
we can search on Stack Overflow
or browse the GNU manuals
(particularly those for the core GNU utilities,
which include many of the commands introduced in this lesson).
In all cases,
though,
we need to have some idea of what we’re looking for in the first place:
someone who wants to know how many lines there are in a data file
is unlikely to think to look for wc
.
2.9 Summary
The original Unix shell is celebrating its fiftieth anniversary. Its commands may be cryptic, but few programs have remained in daily use for so long. The next chapter will explore how we can combine and repeat commands in order to create powerful, efficient workflows.
2.10 Exercises
The exercises below involve creating and moving new files, as well as considering hypothetical files. Please note that if you create or move any files or directories in your Zipf’s Law project, you may want to reorganize your files following the outline at the beginning of the next chapter. If you accidentally delete necessary files, you can start with a fresh copy of the data files by following the instructions in Section 1.2.
2.10.1 Exploring more ls
flags
What does the command ls
do when used
with the -l
option?
What happens if you use two options at the same time, such as ls -l -h
?
2.10.2 Listing recursively and by time
The command ls -R
lists the contents of directories recursively,
which means the subdirectories, sub-subdirectories, and so on at each level are listed.
The command ls -t
lists things by time of last change,
with most recently changed files or directories first.
In what order does ls -R -t
display things? Hint: ls -l
uses a long listing
format to view timestamps.
2.10.3 Absolute and relative paths
Starting from /Users/amira/data
,
which of the following commands could Amira use to navigate to her home directory,
which is /Users/amira
?
cd .
cd /
cd /home/amira
cd ../..
cd ~
cd home
cd ~/data/..
cd
cd ..
cd ../.
2.10.4 Relative path resolution
Using the filesystem shown in Figure 2.5,
if pwd
displays /Users/sami
,
what will ls -F ../backup
display?
../backup: No such file or directory
final original revised
final/ original/ revised/
data/ analysis/ doc/

Figure 2.5: Filesystem for exercises.
2.10.5 ls
reading comprehension
Using the filesystem shown in Figure 2.5,
if pwd
displays /Users/backup
,
and -r
tells ls
to display things in reverse order,
what command(s) will result in the following output:
ls pwd
ls -r -F
ls -r -F /Users/backup
2.10.6 Creating files a different way
What happens when you execute touch my_file.txt
?
(Hint: use ls -l
to find information about the file)
When might you want to create a file this way?
2.10.7 Using rm
safely
What would happen if you executed rm -i my_file.txt
on the file created in the previous exercise?
Why would we want this protection when using rm
?
2.10.8 Moving to the current folder
After running the following commands,
Amira realizes that she put the (hypothetical) files chapter1.txt
and chapter2.txt
into the wrong folder:
data/ docs/
README.md frankenstein.txt sherlock_holmes.txt
chapter1.txt jane_eyre.txt time_machine.txt
chapter2.txt moby_dick.txt
dracula.txt sense_and_sensibility.txt
Fill in the blanks to move these files to the current folder (i.e., the one she is currently in):
2.10.9 Renaming files
Suppose that you created a plain-text file in your current directory to contain a list of the
statistical tests you will need to do to analyze your data, and named it: statstics.txt
After creating and saving this file you realize you misspelled the filename! You want to correct the mistake, which of the following commands could you use to do so?
cp statstics.txt statistics.txt
mv statstics.txt statistics.txt
mv statstics.txt .
cp statstics.txt .
2.10.10 Moving and copying
Assuming the following hypothetical files,
what is the output of the closing ls
command in the sequence shown below?
/Users/amira/data
books.dat
books-saved.dat doc
doc
books.dat doc
books-saved.dat
2.10.11 Copy with multiple filenames
This exercise explores how cp
responds when attempting to copy multiple things.
What does cp
do when given several filenames followed by a directory name?
What does cp
do when given three or more file names?
2.10.12 List filenames matching a pattern
When run in the data
directory of your project directory,
which ls
command(s) will produce this output?
jane_eyre.txt sense_and_sensibility.txt
ls ??n*.txt
ls *e_*.txt
ls *n*.txt
ls *n?e*.txt
2.10.13 Organizing directories and files
Amira is working on a project and she sees that her files aren’t very well organized:
books.txt data/ results/ titles.txt
The books.txt
and titles.txt
files contain output from her data
analysis. What command(s) does she need to run
to produce the output shown?
data/ results/
books.txt titles.txt
2.10.14 Reproduce a directory structure
You’re starting a new analysis, and would like to duplicate the directory structure from your previous experiment so you can add new data.
Assume that the previous experiment is in a folder called 2016-05-18
,
which contains a data
folder that in turn contains folders named raw
and
processed
that contain data files. The goal is to copy the folder structure
of 2016-05-18/data
into a folder called 2016-05-20
so that your final directory structure looks like this:
2016-05-20/
└── data
├── processed
└── raw
Which of the following set of commands would achieve this objective?
What would the other commands do?
2.10.15 Wildcard expressions
Wildcard expressions can be very complex, but you can sometimes write
them in ways that only use simple syntax, at the expense of being a bit
more verbose.
In your data/
directory,
the wildcard expression [st]*.txt
matches all files beginning with s
or t
and ending with .txt
.
Imagine you forgot about this.
Can you match the same set of files with basic wildcard expressions that do not use the
[]
syntax? Hint: You may need more than one expression.Under what circumstances would your new expression produce an error message where the original one would not?
2.10.16 Removing unneeded files
Suppose you want to delete your processed data files, and only keep
your raw files and processing script to save storage.
The raw files end in .txt
and the processed files end in .csv
.
Which of the following would remove all the processed data files,
and only the processed data files?
rm ?.csv
rm *.csv
rm * .csv
rm *.*
2.10.17 Other wildcards
The shell provides several wildcards beyond the widely-used *
.
To explore them,
explain in plain language what (hypothetical) files the expression novel-????-[ab]*.{txt,pdf}
matches and why.
2.11 Key Points
- A shell is a program that reads commands and runs other programs.
- The filesystem manages information stored on disk.
- Information is stored in files, which are located in directories (folders).
- Directories can also store other directories, which forms a directory tree.
pwd
prints the user’s current working directory./
on its own is the root directory of the whole filesystem.ls
prints a list of files and directories.- An absolute path specifies a location from the root of the filesystem.
- A relative path specifies a location in the filesystem starting from the current directory.
cd
changes the current working directory...
means the parent directory..
on its own means the current directory.mkdir
creates a new directory.cp
copies a file.rm
removes (deletes) a file.mv
moves (renames) a file or directory.*
matches zero or more characters in a filename.?
matches any single character in a filename.wc
counts lines, words, and characters in its inputs.man
displays the manual page for a given command; some commands also have a--help
option.