bash-workshop.tex

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% LICENSE                                                            %
% This work is licensed under the Creative                           %
% Commons Attribution-NonCommercial 4.0 International License.       %
% To view a copy of this license, visit:                             %
% https://creativecommons.org/licenses/by-nc/4.0/                    %
% or send a letter to                                                %
% Creative Commons                                                   %
% PO Box 1866, Mountain View,                                        %
% CA 94042                                                           %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass[a4paper]{scrreprt}
\usepackage[english]{babel}

\usepackage{tabu}
\usepackage{longtable}

\usepackage{hyperref}
\hypersetup{
  colorlinks=true
}

\usepackage{graphicx}
\usepackage{tikz}


\begin{document}
\begin{titlepage}
  \centering
  {\scshape\huge UNIX Workshop \par}
  \vspace{2cm}
  {\LARGE Dimitri Robl\par}
  \vspace{1cm}
  \includegraphics[scale=0.6]{pictures/xkcd456.png}
  \newline
  {CC BY-NC 2.5 by Rendall Monroe\\
  \href{https://creativecommons.org/licenses/by-nc/2.5/legalcode}
  {https://creativecommons.org/licenses/by-nc/2.5/legalcode}
  \par}
  \vspace{2cm}
  {\large This work is licensed under the
  \href{https://creativecommons.org/licenses/by-nc/4.0/}
  {Creative Commons Attribution-Non-Commercial 4.0 International
  License}\\}
  \vspace{3cm}
  {\large Version 0.1\hfill\today\par}
\end{titlepage}

Welcome to this three-day workshop which will teach you the basics of
UNIX, (GNU/)Linux, and the command line! There is a lot to learn
ahead of us, but first, you need to set up your working environment.

\tableofcontents
\chapter{Installing a Virtual Machine}
In order to make life easier for you, it won't be necessary that you
install a Linux distribution or distro\footnote{You'll learn what
exactly that means later in the course.} on your harddrive. Instead,
we will guide you through the installation of a virtual
machine\footnote{If you are unfamiliar with this concept you can read
the \href{https://en.wikipedia.org/wiki/Virtual_Machine}{Wikipedia
entry} about it.} running Debian GNU/Linux. If you already have a
working installation of any Linux distro (either physical or virtual)
you can skip this step.

\section{Prerequisites}
In order to use a virtual machine (VM), your physical machine should
have at least two cores, 4GiB of main memory (i.e. RAM) and there
should be at least 20GB free space on your disk. In this case more is 
always good.
% end Prerequisites

\subsection{Enable Virtualization}
The first thing to check is whether you CPU supports virtualization,
and has it activated. Follow these steps:
\begin{enumerate}
  \item Boot your computer.
  \item Depending on your motherboard press \texttt{ESC},
    \texttt{DEL}, \texttt{ENTER}, \texttt{F2}, \texttt{F10}, or
    \texttt{F12}.
  \item Look for an option which says "Advanced" or "CPU settings".
  \item Depending on whether you use an Intel or an AMD CPU you
    will have to enable Intel Virtualization Technology, Intel VT,
    VT-x, AMD-V, AMD Virtualization or similar.
  \item Check if this option is enabled and if not, enable it.
  \item Save changes and Exit.
\end{enumerate}
% end Enable Virtualization

\subsection{Install Oracle VirtualBox}
Now you have to install the software to run virtual machines:
\begin{enumerate}
  \item Download the installer from \href{https://www.virtualbox.org/}
    {https://www.virtualbox.org/}. It runs on Windows, OS X, Linux
    and Solaris hosts.
  \item Start the installer and follow the instructions.
\end{enumerate}
% end Install Oracle VirtualBox

\subsection{Getting the Installation Image}
The final thing to obtain before the installation can begin is an 
image to install. Just download the
\href{https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-9.5.0-amd64-netinst.iso}
{netinst Debian image} and store it somewhere you remember(!),
you'll need it afterwards.

In case you don't have an x86\_64 architecture \footnote{To
make things less confusing, this architecture is also called
x64, AMD64, and Intel 64.}, installation files for other architetures
are available at \href{https://www.debian.org/distrib/netinst}
{https://www.debian.org/distrib/netinst}. However, it is rather 
unlikely that you have a different architecture on a general purpose
computer, unless you are still on a machine running 32-bit 
x86\footnote{A.k.a. IA-32, i386.} or own one manufactured by Apple
before 2006 which has a PowerPC CPU.

Now everything is ready to proceed to the installation!
%end Getting the Installation Image

\section{Installing a Virtual Machine Running Debian GNU/Linux}
To install a VM running Debian GNU/Linux follow the instructions
below:
\begin{enumerate}
  \item Open VirtualBox.
  \item Click on the ``New" button in the upper left corner.
    \begin{center}
      \includegraphics[scale=0.35]{pictures/install_vm01.png}
    \end{center}
  \item Enter a name for your VM, and if this is not automatically
    done select \emph{Type:} → \emph{Linux}, and \emph{Version:} →
    \emph{Debian (64-bit)}.\footnote{If you use a different
    rchitecture than x86\_64 you have to specify this here as well.}
    Press ``Next".
    \begin{center}
      \includegraphics[scale=0.40]{pictures/install_vm02.png}
    \end{center}
  \item Select the amount of memory the VM will have. The best
    value here depends on what you want to do with the VM and how
    much RAM your physical machine has. The default value of 1024Mib
    (i.e. 1GiB) is sufficient for our purposes, but you can use more
    if you want and less if you are very low on physical RAM (i.e. 
    2GiB or less).
    \begin{center}
      \includegraphics[scale=0.40]{pictures/install_vm03.png}
    \end{center}
  \item Choose how much disk space to give to your VM. This includes
    several steps:
    \begin{enumerate}
      \item Select ``Create a virtual hard disk now" and press ``Create".
        \begin{center}
          \includegraphics[scale=0.40]{pictures/install_vm04.png}
        \end{center}
      \item Leave ``VDI (VirtualBox Disk image)" selected and press
        ``Next".
        \begin{center}
          \includegraphics[scale=0.40]{pictures/install_vm05.png}
        \end{center}
      \item Again go with the default ``Dynamically allocated" and 
        and press ``Next".
        \begin{center}
          \includegraphics[scale=0.40]{pictures/install_vm06.png}
        \end{center}
      \item You can now change name and location of the file that
        will be the VM's harddisk. If this sounds confusing: What the
        VM will believe is its disk will actually be nothing more 
        than a file on the underlying operating system (OS) you run
        the VM on.

        A reasonable size also depends on your needs and the space
        you have. If you plan to really do anything with your VM, we
        recommend a minimum of 20GB, but if you just need it for this
        workshop the standard size of 8GB will suffice.

        After you are done, click on ``Create".
        \begin{center}
          \includegraphics[scale=0.40]{pictures/install_vm07.png}
        \end{center}
    \end{enumerate}
  \item Now you're back to the main screen, which should look 
    something like this:
    \begin{center}
      \includegraphics[scale=0.40]{pictures/install_vm08.png}
    \end{center}
    Select you newly created VM and press ``Start".
  \item A pop-up should appear which asks you to select a start-up
    disk. Click on the folder icon and select the Debian GNU/Linux
    ISO-file you downloaded earlier (and whose download location you
    should remember ;)).
    \begin{center}
      \includegraphics[scale=0.50]{pictures/install_vm09.png}
    \end{center}
    Press the ``Start" button.
\end{enumerate}
That's it, a new window should pop up and the installation of your
new virtual OS can begin. Select ``Graphical Install" and follow the
on-screen instructions. There are some things you should be careful
about:
\begin{enumerate}
  \item The selection of your locale (i.e. the language the system
    will use) and your keyboard layout -- e.g. if you want to use an 
    English system, but a German keyboard you have to tell the
    installer.
  \item When asked to enter the \texttt{root} password, leave it
    blank to disable the \texttt{root} account and use \texttt{sudo}
    instead. We'll discuss the details later.
    \begin{center}
      \includegraphics[scale=0.35]{pictures/install_vm10.png}
    \end{center}
  \item If you don't know about disk partitioning leave the defaults
    intact. However, in case you install Debian GNU/Linux on real
    hardware on a portable device (e.g. a laptop) and not as a VM,
    we \emph{strongly} recommend using the ``Guided - use entire
    disk and set up encrypted LVM" option. This will encrypt your
    harddrive which means that every time your PC or laptop starts
    you will have to enter a passphrase to decrypt the disk before
    the OS even boots. This has the huge advantage that in case your
    device is stolen, it will be almost impossible to access your 
    data if the thief does not your password/passphrase -- and no
    one should ever know that! Make sure to choose a strong password,
    where ``strong" means ``as long as possible". You don't
    necessarily have to care about special characters, length is
    always better. See \href{https://www.xkcd.com/936/}{here}. 
    
    The only downside to this is the fact that if you forget or lose
    your password/passphrase you yourself also won't be able to access
    your data. This is one of the few cases in which it \emph{may} be
    a good idea to write down your password/passphrase on a piece of
    paper and store it in some secure place.
  
    Also, don't worry about the warnings when it says ``Write changes
    to disk" if you set up a VM. The VM `believes' it has a whole
    disk to write on and is not aware that it is actually just a file
    in the underlying OS. If you followed the instructions so far, it
    is not possible to destroy your real hard disk by installing the
    VM.
  \item When you come to select mirror to download the packages,
    double-click on ``Austria" and then select any mirror. As it says
    in the description \texttt{ftp.at.debian.org} is usually a good
    choice.
    \begin{center}
      \includegraphics[scale=0.40]{pictures/install_vm11.png}
      \includegraphics[scale=0.40]{pictures/install_vm12.png}
    \end{center}
  \item Once you come to the screen ``Software selection" we recommend
    selecting ``Xfce" or ``LXDE" if the VM is on low memory ( <
    2GiB), as these are very light weight GUIs. Otherwise you can
    stick with the default ``Debian desktop environment".
    \begin{center}
      \includegraphics[scale=0.35]{pictures/install_vm13.png}
    \end{center}
\end{enumerate}
If you run into any problems, use your favorite search engine and
look for something like ``Debian installation tutorial". Most likely
you'll find written instructions as well as videos on how to do it.
% end Installing a Virtual Machine Running Debian GNU/Linux

\chapter{UNIX, GNU, and Linux}
This section seeks to clarify terms you are very likely to encounter
pretty soon when working with any (GNU/)Linux distro. Be prepared that
people are \emph{very} picky about words, and don't get discouraged
in case you meet a particularly nasty partisan of one of these
wordisms.

\section{UNIX}
In the beginning there was … the desire to play games! Yes, that's
it! In the 1960ies two MIT students called Dennis Ritchie and Ken
Thompson wanted to play the game "Space Travel" on their new, but
underpowered minicomputer\footnote{"Mini" here indicates that it 
didn't fill a whole room.} PDP-7. To achieve their goal, they had to
tweak the machine a bit and as they were researching operating
systems (OS) at the time anyway, they ended up writing a whole new OS for
the PDP-7 which came to be named UNIX.

UNIX, based on the even older
\href{https://en.wikipedia.org/wiki/Multics}{Multics}, introduced
a lot of new concepts into OS-design, many of which are still in use
today. UNIX was developed by Bell Labs which where prohibited from
selling it under US-law, but in 1983 this decree was lifted and
AT\&{}T, the owner of Bell Labs promply tried to turn UNIX into a
product, as it was already widely used among computer science students
and companies they worked for. However, by this time there were
already multiple versions of UNIX around, and one of the most
important of them was developed by researchers in Berkeley; the
Berkeley System Distribution (BSD).

BSD was sued by AT\&{}T for copyright infringement despite the fact
that large parts of AT\&{}T's own UNIX implementation was based on
code from Berkeley. After a long legal battle, BSD won its freedom and
the base of modern FreeBSD, OpenBSD, NetBSD, etc. was laid out.

UNIX finally lost the battle for the PC against Apple and Microsoft,
which was very likely at least in parts based on the law suits.

However, in the server market and in technical and scientific circles,
UNIX and its derivatives  continued to be state of the art and remain
so today.

% end UNIX

\section{GNU}
GNU (GNU's Not Unix) was founded in 1983 by Richard M. Stallman after
he was fed up with the closed source philosophy of UNIX. He wanted to
create an OS that is ``free as in freedom", meaning it adheres to the
four principles of free software, which state that users' have to be
\begin{description}
  \item [0.] able to run the program as they wish, for any purpose.
  \item [1.] able to study how the program works and can change it
    according to their wishes.
  \item [2.] able to redistribute copies to others, either free or
    for a fee.
  \item [3.] able to distribute modified versions to others, again
    either for free or for a fee.
\end{description}
Many tools necessary for a complete OS were created by the GNU
project, including \texttt{gcc} (GNU C Compiler), \texttt{g++} (GNU
C++ Compiler), \texttt{gdb} (GNU Debugger), a shell,  and others.

However, one essential part was incomplete: A working kernel. The
kernel is the part of the OS which interacts directly with the
hardware, so without a kernel, an OS can't run. Thus, the GNU tools
were used on a number of UNIX systems, but a stand-alone OS was still
wanting.

This changed when in 1991 a Finnish computer science student by the
name of Linus Torvalds released a kernel he (eventually) called
Linux.
% end GNU

\section{Linux} 
Linus Torvalds originally released the Linux kernel under a license
he created himself, but in 1992 he re-licensed it under the GPLv2
(General Public License, version 2) which is a license of the GNU
project. 

Technology-enthusiasts and academia quickly started using Linux as
their kernel and as soon as it was GPL-ed there existed a full GNU
OS with a working kernel. This is why you may read/hear the term
GNU/Linux. Be careful here, some people are \emph{very} picky about
using Linux only when referring to the kernel and GNU/Linux when
refering to a distro. 

Linux' open source nature made it possible that its development
proceeded very quickly with thousands of people contributing code and
time. The current major release of Linux is version 4, the original
4.0 kernel was released in 2015. While Linux is often hailed as
\emph{the} example of a successful open source project with loads
of people sacrificing their spare time to make the Linux kernel
better, today this is obsolete in large parts. For the kernel 
releases between 4.8--4.13, 87.7\%{} to 91.8\%{} of the changes have
been done by people who are paid for this by various
companies.\footnote{See
\href{https://www.linuxfoundation.org/2017-linux-kernel-report-landing-page/}
{here}.} This is not necessarily a bad thing, it just shows that the
IT-industry heavily relies on Linux and therefore actively develops it
and also that Linux today is very far from being a hobbyists' project
but a mature OS kernel developed mostly by professionals.
% end Linux

% end UNIX, GNU, and Linux

\chapter{Basics}
\label{basics}
This section will introduce important concepts and words you'll have
to work with, show you how to help yourself on a (GNU/)Linux system
in case you get stuck and explain the filesystem.

To follow along, please download the training directory at\\
\href{https://ct.cs.univie.ac.at/teaching/unixlinux-einfuehrungskurs/unixcourse.tgz}
{https://ct.cs.univie.ac.at/teaching/unixlinux-einfuehrungskurs/unixcourse.tgz}

On the command line you can type (or just copy-paste -- which can be
done by just selecting text with your mouse to copy it and paste it
by pressing the middle mouse button on most modern GUIs on 
(GNU/)Linux) the following to get the archive containing the folder:
{\footnotesize
\begin{verbatim}
wget https://ct.cs.univie.ac.at/teaching/unixlinux-einfuehrungskurs/unixcourse.tgz
tar -xvzf unixcourse.tgz
\end{verbatim}
}

\section{Shells, Terminals, and Commands}
\label{shellsterminalscommands}
In the last chapter you've seen the concept of the kernel, the part
of the OS which talks to the hardware. If you want to communicate
with the kernel you need something, that wraps around it -- a
\emph{shell}, for example.

Essentially, a shell is a user interface which allows you to access
the services provided by the OS. You are most likely familiar with a
graphical user interface (GUI) which allows you to access some
services of the OS using a mouse and icons. A shell, on the other
hand, requires text as input by providing you with a \emph{command
line}. While the GUI is often part of the OS you buy, i.e. you can't
easily change the program that is your GUI if you use Microsoft
Windows or Apple's OS X, a shell on a (GNU/)Linux system is just a
program and you can choose from a wide variety of options.  The shell
we are going to work with here is the 
\href{https://www.gnu.org/software/bash/https://www.gnu.org/software/bash/}
{GNU \texttt{bash}} (Bourne Again SHell), which is based on an older
shell written by Stephen Bourne in 1979 and is obviously also a
wordplay with the verb `born'. We chose this one because it is
currently the standard shell on all major (GNU/)Linux distros, but do
play around with others if you like to.

The stuff you enter on the command line to your shell are 
\emph{commands} and they work the same way as clicking an icon with
your mouse does, e.g. you can either double-click on the Firefox
icon, or just type the word ``firefox" on the command
line, followed by \texttt{ENTER}. Both actions will start a new
instance of the Firefox webbrowser (if it is installed). One thing
to keep in mind in the following sections is that most commands are
actually stand-alone programs, even though they write their output
to the terminal screen, while only a few of them are so-called
\emph{shell builtins}, i.e. keywords the shell itself understands
and works on without starting a new program. This will be of special
importance in \ref{helputilities} where we talk about utilities that
provide documentation for commands.

Many commands can take arguments and options. Their basic syntax is
usually:
\begin{verbatim}
command [OPTION]... [ARGUMENT]...
\end{verbatim}
The brackets around \texttt{OPTION} and \texttt{ARGUMENT} indicate
that they are optional and the dots tell you that the preceding 
entity can be used multiple times, i.e. the line reads approximately
as ``Use \texttt{command} by typing \texttt{command}, possibly 
followed by one or more option(s), possibly followed by one or more
argument(s)." Arguments tell the command e.g. on which file to operate,
whereas options tell it what to do with the arguments. To separate
these things, you have to put a space after the command, after each
argument, and after each (group) of options.

Options come in two flavors: Short and long ones. Short options are
usually indicated by a single, non-separated, preceding dash
`\texttt{-}', while long options can be recognized by two
non-separated preceding dashes `\texttt{--}'. A very simple example
would be
\begin{verbatim}
$ firefox --new-window https://www.univie.ac.at
\end{verbatim}
which, little surprisingly opens the homepage of the University of
Vienna in a new window of the Firefox webbrowser.

In most cases options for a single dash consist of a
single letter, while options preceeded by two dashes are words or
phrases, e.g. let's say you use the \texttt{ls} program (to 
\emph{list} directories and/or files) which has the options
\texttt{-l}, \texttt{-a}, and \texttt{--human-readable} and want it
to process argument \texttt{arg}. To do so, you could type:
\begin{verbatim}
$ ls -a -l --human-readable unixcourse
total 524K
drwxr-xr-x  9 dimir dimir 4.0K Sep 24 22:43  .
drwxr-xr-x  4 dimir dimir 4.0K Sep 24 22:45  ..
drwxr-xr-x 13 dimir dimir 4.0K Sep 19 10:05  contentdir
drwxr-xr-x  2 dimir dimir 4.0K Sep 18 15:36  dir1
drwxr-xr-x  2 dimir dimir 4.0K Sep 17 13:52  dir2
drwxr-xr-x  2 dimir dimir 4.0K Sep 17 13:52  dir3
drwxr-xr-x  2 dimir dimir 4.0K Sep 17 15:36 'dir with spaces'
-rw-r--r--  1 dimir dimir   63 Sep 19 11:07  errorfile
-rw-r--r--  1 dimir dimir   29 Sep 18 15:24  file1
-rw-r--r--  1 dimir dimir   15 Sep 23 12:32  file10
-rw-r--r--  1 dimir dimir   11 Sep 23 12:22  file3
-rw-r--r--  1 dimir dimir    0 Sep 24 22:43  filetodelete
-rw-r--r--  1 dimir dimir  472 Sep 23 09:49  grepfile
-rw-r--r--  1 dimir dimir    0 Sep 23 12:33  hasfileinit
-rw-r--r--  1 dimir dimir 422K Sep 18 17:52  hugefile
-rw-r--r--  1 dimir dimir   46 Sep 23 12:11  join1
-rw-r--r--  1 dimir dimir   48 Sep 23 12:11  join2
-rw-r--r--  1 dimir dimir  405 Sep 23 12:22  linefile
-rw-r--r--  1 dimir dimir 1.4K Sep 23 12:29  lsfile
-rw-r--r--  1 dimir dimir   26 Sep 23 12:22  numericsort
-rw-r--r--  1 dimir dimir   25 Sep 17 13:53  onelinefile
-rw-r--r--  1 dimir dimir   69 Sep 18 15:36  renamedfile
drwxr-xr-x  2 dimir dimir 4.0K Sep 18 11:14  rmdirectory
drwxr-xr-x  2 dimir dimir 4.0K Sep 18 15:18  rminteractive
-rw-r--r--  1 dimir dimir  331 Sep 23 11:02  sedfile
-rw-r--r--  1 dimir dimir  331 Sep 23 10:48  sedfile.bak
-rw-r--r--  1 dimir dimir   18 Sep 23 12:22  sortfile
-rw-r--r--  1 dimir dimir  747 Sep 23 11:25  tablefile
\end{verbatim}
One thing to remember is that it is possible to combine short options
behind a single dash, i.e. the following would be equivalent to the
above:
\begin{verbatim}
$ ls -al --human-readable unixcourse
total 524K
drwxr-xr-x  9 dimir dimir 4.0K Sep 24 22:43  .
drwxr-xr-x  4 dimir dimir 4.0K Sep 24 22:45  ..
drwxr-xr-x 13 dimir dimir 4.0K Sep 19 10:05  contentdir
drwxr-xr-x  2 dimir dimir 4.0K Sep 18 15:36  dir1
drwxr-xr-x  2 dimir dimir 4.0K Sep 17 13:52  dir2
drwxr-xr-x  2 dimir dimir 4.0K Sep 17 13:52  dir3
drwxr-xr-x  2 dimir dimir 4.0K Sep 17 15:36 'dir with spaces'
-rw-r--r--  1 dimir dimir   63 Sep 19 11:07  errorfile
-rw-r--r--  1 dimir dimir   29 Sep 18 15:24  file1
-rw-r--r--  1 dimir dimir   15 Sep 23 12:32  file10
-rw-r--r--  1 dimir dimir   11 Sep 23 12:22  file3
-rw-r--r--  1 dimir dimir    0 Sep 24 22:43  filetodelete
-rw-r--r--  1 dimir dimir  472 Sep 23 09:49  grepfile
-rw-r--r--  1 dimir dimir    0 Sep 23 12:33  hasfileinit
-rw-r--r--  1 dimir dimir 422K Sep 18 17:52  hugefile
-rw-r--r--  1 dimir dimir   46 Sep 23 12:11  join1
-rw-r--r--  1 dimir dimir   48 Sep 23 12:11  join2
-rw-r--r--  1 dimir dimir  405 Sep 23 12:22  linefile
-rw-r--r--  1 dimir dimir 1.4K Sep 23 12:29  lsfile
-rw-r--r--  1 dimir dimir   26 Sep 23 12:22  numericsort
-rw-r--r--  1 dimir dimir   25 Sep 17 13:53  onelinefile
-rw-r--r--  1 dimir dimir   69 Sep 18 15:36  renamedfile
drwxr-xr-x  2 dimir dimir 4.0K Sep 18 11:14  rmdirectory
drwxr-xr-x  2 dimir dimir 4.0K Sep 18 15:18  rminteractive
-rw-r--r--  1 dimir dimir  331 Sep 23 11:02  sedfile
-rw-r--r--  1 dimir dimir  331 Sep 23 10:48  sedfile.bak
-rw-r--r--  1 dimir dimir   18 Sep 23 12:22  sortfile
-rw-r--r--  1 dimir dimir  747 Sep 23 11:25  tablefile
\end{verbatim}
Long options cannot be combined, so you have to type each of them in
full. Many programs have short and long options for the same 
operations and it's just a question of taste which one you use. A good
approach is to start using long options because it is easier to 
remember what they do and start memorizing the short options for
operations you use a lot.

But why bother to learn the names of programs if you can just click
on an icon? Surely, that's much more intuitive and simple! Well, it
depends. Imagine you commonly use about 50 programs, many of which
interact with each other -- a screen with 50 icons for programs
seems rather overwhelming, don't you think? And how about moving
stuff from one window to the other all the time -- annoying, I'd
say. Or opening the same 7 programs one after the other, hundreds of
times and copying data from one to the other? Convenience is clearly
not the word for this.

These are some of the tasks that can be solved easily on the command
line. Most command line utilities follow the UNIX philosophy that
each program should solve a single problem and solve it well. By 
combining the capabilities of many programs you get a very flexible
toolset to solve your problem at hand. Make the easy things easy,
and the hard things possible!

\begin{figure}[h]
  \centering
  \includegraphics[scale=0.9]{pictures/terminal-dec-vt100.jpg}
  \label{vt100}
  \caption{A DEC VT100 terminal. CC BY-SA \href{https://en.wikipedia.org/wiki/User:ClickRick?rdfrom=commons:User:ClickRick}{ClickRick}}
\end{figure}
Finally, a \emph{terminal} is actually a piece of hardware (see
\ref{vt100}), which can take input and display output. In this course
and most likely in your whole life, you'll be confronted with
terminal emulators, i.e. software which behaves like a terminal, like
the on you see in \ref{xfce4-terminal}. 

\begin{figure}[h]
  \centering
  \includegraphics[scale=0.4]{pictures/xfce4-terminal.png}
  \caption{XFCE's terminal emulator}
  \label{xfce4-terminal}
\end{figure}

When you open a terminal emulator your shell will be started and 
will provide you with a \emph{prompt}, which usually looks a bit like
\begin{verbatim}
user@machine ~$
\end{verbatim}
where \texttt{user} is your username, \texttt{machine} is the name of
your computer and \texttt{\textasciitilde} is a shorthand for your
home directory (see below for an explanation).\footnote{From now on,
we will abbreviate your prompt as just `\texttt{\$}'.} All your shell
does now, is to wait for you to issue a command -- let's do it the
favor: type some random stuff and press \texttt{ENTER}. It should
look something like this:
\begin{verbatim}
$ ajsdggasd
bash: ajsdggasd: command not found
$ 
\end{verbatim}
Little surprisingly, this gibberish does not constitute a command.
But fear not, we shall use existing commands starting in the next
section!
% end Shells, Terminals, and Commands}

\section{The Filesystem}
\label{thefilesystem}
In order to use the filesystem efficiently it is essential to
understand its basic structure and this is what this section wants
to teach you.

The filesystem is software which manages how your files and
directories are stored on you disk/pendrive/CD/etc. Usually there are
at least three kinds of entities in a filesystem:
\begin{description}
  \item [Files] Some kind of data, possibly in a special format (e.g.
      PDF).
  \item [Directories] These are places in which files are stored.
  \item [Links] Links are references to either a file or a directory,
      i.e. they don't contain data, but just point to another place
      in the filesystem.
\end{description}
Filesystems in any UNIX system are usually structured as rooted
trees.\footnote{For the mathematically interested: A rooted tree is an
acyclic undirected graph in which one vertex has been designated the
root.} The root of the filesystem is designated by the slash 
` \texttt{/}'.

{\footnotesize
\tikz [level 1/.style={sibling distance=1cm}]
 \node {\texttt{/}}
  child { node {\texttt{bin}} }
  child { node {\texttt{boot}} }
  child { node {\texttt{dev}} }
  child { node {\texttt{etc}} }
  child { node {\texttt{lib}} }
  child { node {\texttt{media}} }
  child { node {\texttt{mnt}} }
  child { node {\texttt{opt}} }
  child { node {\texttt{run}} }
  child { node {\texttt{usr}} 
    child { node {\texttt{bin}} }
    child { node {\texttt{include}} }
    child { node {\texttt{lib}} }
    child { node {\texttt{local}} }
    child { node {\texttt{sbin}} }
    child { node {\texttt{share}} }
    child { node {\texttt{src}} }
  }
  child { node {\texttt{sbin}} }
  child { node {\texttt{srv}} }
  child { node {\texttt{tmp}} }
  child { node {\texttt{var}} };
}

%TODO picture of filesystem!
As you are logged in and opened a terminal already (if not, do so 
now), you can type 
\begin{verbatim}
$ pwd
/home/me
\end{verbatim}
Of course, you'll get a slightly different output, but there are two
important things here:
\begin{enumerate}
  \item \texttt{pwd} is our first real command! It stands for 
    `print working directory' which prints you current working
    directory, i.e. tells you where in the filesystem you are.
  \item The output, \texttt{/home/me} tells you, that you are in a
    directory \texttt{me} which is in a directory called
    \texttt{home}, which is in the root directory \texttt{/}.
\end{enumerate}
Okay, now that you know where you are, you may want to know which
files\footnote{Here `files' denotes data, directories, and links as
it is the most generic concept.} are in this directory -- you can
list them using
\begin{verbatim}
$ ls
Documents Downloads Pictures Private
\end{verbatim}
Again your output will differ slightly. So now, you used the program
\texttt{ls} to show you the contents of the directory you are in.
This is \texttt{ls}' standard behavior: If you don't provide any
arguments it will display the content of you current working
directory.

However, you can tell \texttt{ls} which directory it should list by
providing a \emph{path} as an argument. To show the contents of the
root directory, type:
\begin{verbatim}
$ ls /
bin boot dev etc lib media mnt opt run sbin srv tmp usr var 
\end{verbatim}
The output provided here shows everything that is required in the
root directory according to the
\href{http://refspecs.linuxfoundation.org/fhs.shtml}{Filesystem
Hierarchy Standard (FHS)}, a document which defines what has to be in
the root directory of any (GNU/)Linux distro.\footnote{A short version
of this can be found by typing \texttt{man 7 hier}.} But as it only
defines the minimum, distros are free to add other things and you'll
most likely also find at least \texttt{home}, \texttt{proc}, and 
\texttt{sys} in your root directory.

Now there is one more concept you have to understand regarding the
filesystem: absolute and relative paths. In a rooted tree you can
specify any point in the tree by starting at the root and listing
every step on the way. This is what \texttt{pwd} does: Above it said
that I was in \texttt{/home/me}, telling me my exact position in the
filesystem in an absolute manner; therefore paths starting with the
root directory are called \emph{absolute paths}.

Unfortunately, absolute paths can be very inconvenient and long. 
Remember that we said \texttt{ls} can be provided with a path to list
the contents of any directory in the filesystem, not just the one you
are currently in and that in the above example the current directory
contained \texttt{Documents}, \texttt{Downloads}, \texttt{Pictures}, 
and \texttt{Private}. Imagine you have a subdirectory in
\texttt{Documents} which is called \texttt{university}. To list it,
you could type
\begin{verbatim}
$ ls /home/me/Documents/university
\end{verbatim}
An awful lot to type I'd say! To remedy this situation you can
provide a path relative to your current working directory, i.e. a
\emph{relative path}. \texttt{pwd} told us that our current working
directory is \texttt{/home/me}, so the relative path to
\texttt{university} is just \texttt{Documents/university} without a
leading `\texttt{/}'. Every path starting with a slash will be
interpreted as an absolute path, while relative paths just start with
the name of the next directory you hop into from your current
directory.

That's it for filesystem theory, now we'll enable you to help
yourself on a (GNU/)Linux system.
% end The Filesystem

\section{Help Utilities}
\label{helputilities}
Every (GNU/)Linux system comes with a lot of built-in documentation
installed by default and of course the Internet hosts millions of
HOW-TOs, tutorials, etc. We're going to focus on the offline
documentation, but also provide some hints were you usually get 
reliable information online.

This section can be treated as a reference -- reading it now in
its entirety may cause you a bit of a headache as you may not be
familiar enough with commands, options and arguments right now. For
the beginning it will suffice if you skim through this section to get
a basic idea about which facilities exist.

\subsection{\texttt{man}}
\texttt{man}, which is a shorthand for ``manual page" is the
traditional UNIX documentation and knowing how to use it will help
you not only on (GNU/)Linux, but also on any other UNIX you'll 
encounter like the various BSDs. In case you ask someone who is in a
hurry about a program and the only response is ``RTFM", meaning 
``Read The Fucking\footnote{Or, ``Fine", for the very pure.} Manual",
this program is usually what they want you to use.

If you want to access the manual page of the \texttt{ls} program,
all you have to do is to type:
\begin{verbatim}
$ man ls
\end{verbatim}
After this the man page for \texttt{ls} should appear. To scroll down
press \texttt{SPACE} and to exit press \texttt{Q}. 

We won't go into a lot of details here because your first task now is
to type
\begin{verbatim}
$ man man
\end{verbatim}
This will fire up the man page of \texttt{man}. Read it to learn how
to use this program efficiently and make sure to read the ``EXAMPLES"
section! 

The main things you should remember from this is how to move within
the man pages, how to exit them, what sections are and how you can
tell the program to only return entries from certain sections. Useful
options you may want to remember are
\begin{description}
  \item [\texttt{-k | --apropos}] List the programs whose name or
    short description match the keyword you searched for. Note that
    this is the same thing the \texttt{apropos} program 
    does.\footnote{You guessed it: Use \texttt{man apropos} to find
    out more.}
  \item [\texttt{-a | --all}] Show all man pages from all sections
    which match the keyword you searched for.
\end{description}
If you read through \texttt{man}'s man page you'll understand what
these options do ;)
% end man

\subsection{\texttt{info}}
While \texttt{man} is available on any UNIX system, the info pages
were invented by the GNU project and are therefore not necessarily
available. However, all widely used (GNU/)Linux distros have them
installed by default -- at least in parts. Unfortunately, the Debian
project considers the license of the \texttt{info} files as nonfree
and must thus be installed by hand using:
\begin{verbatim}
$ sudo apt install texinfo-doc-nonfree
\end{verbatim}
Don't ask...

To get the info page of \texttt{ls} type
\begin{verbatim}
$ info ls
\end{verbatim}
You can again use \texttt{SPACE} to scroll forward and \texttt{Q} to
get out of \texttt{info}.

Your next task is now to read
\begin{verbatim}
$ info info
\end{verbatim}
As \texttt{info} is a bit more sophisticated than \texttt{man},
offering links, footnotes, menus, and other references it is 
essential to read this. Furthermore, \texttt{info} has a lot more
keyboard shortcuts you can use, all of which are documented in its
info page.

The difference between man pages and info pages is not a clearcut
one. As a rule of thumb, man pages are often more concise and only
useful if you already know what a program does in general, but just
want to check available options, i.e. they are often very technical
and offer few explanations. 

The info pages, on the other hand, are often much more verbose, have
sections for different kinds of options (while the man pages just
list them one after the other) and at least try to be a bit more
accessible to complete beginners.

Just compare some man pages and info pages for the same program, e.g.
\texttt{man ls} vs. \texttt{info ls} to see the differences and get a
feeling for them.

Note also, that all documentation is written by humans and some are
more capable of explaining how stuff works than others. For this
reason the quality of man pages and info pages can vary greatly from
program to program. A good example can be found in 
\href{https://xkcd.com/1692/}{this xkcd comic}.
% end info

\subsection{\texttt{--help} and \texttt{-h}}
\label{--helpand-h}
Sometimes you are just not sure about what was the right option to
get what you want or you only forgot the correct spelling of the
option. In these cases you may not want to read through the whole
man or info page and most programs make this possible by providing a
\texttt{--help} or \texttt{-h} option. So, typing \texttt{COMMAND 
--help} or \texttt{COMMAND -h} will in most cases print a short
description of the command and its most common options to the screen.

Just try it out to see for yourself.
% end --help and -h

\subsection{\texttt{help} and \texttt{type}}
\label{helpandtype}
In \ref{shellsterminalscommands} we mentioned that not every command
you type is a full program, but some are just shell builtins. These
are keywords understood by your shell. You can compare this with your
GUI: Most icons you click on will start a new program, but some 
things are done by the GUI itself, e.g. if you click on the Windows
button, it will display a list of programs you can start, but this
list is just part of the GUI.

For these shell builtins no man pages or info pages exist. A command
you'll use \emph{very} often is \texttt{cd} (``change directory") 
which enables you to change your current working directory. But if
you try to type
\begin{verbatim}
$ man cd
\end{verbatim}
you'll get a man page titled ``\texttt{BASH\_BUILTINS(1)}" which will
most likely be very confusing. To get help specifically for
\texttt{cd} (or any other shell builtin for that matter) just type
\begin{verbatim}
$ help cd
\end{verbatim}
and the help will be displayed on the terminal.

But, you may ask yourself, how am I supposed to know what is a
program and what is a shell builtin?! Don't despair! This is where
the \texttt{type} shell builtin comes into play. If you tried to 
access the man and/or info pages of a command and didn't get a
satisfying result, try
\begin{verbatim}
$ type COMMAND_NAME
\end{verbatim}
e.g. 
\begin{verbatim}
$ type cd
cd is a shell builtin
\end{verbatim}
And voil\`{a}, now you can be sure.
% end help and type

\subsection{\texttt{which} and \texttt{whereis}}
\label{whatisandwhereis}
Many programs are continously developed and change from one release
to another; functionality may be added, reduced, removed, deprecated,
etc. And at some points you may not know whether you want the old or
the new version. One way to solve this is to simply install both, use
them and then decide which one you like more. On a GUI this may be 
indicated by two different icons for the different versions, but how
can you tell the difference on the command line?

This is where the \texttt{which} and \texttt{whereis} utilities come
in handy.

\texttt{which} tells you where in the filesystem the executable you
are going to call is located and if you have installed two versions
each of them will reside in a different place. Thus, if you type
\begin{verbatim}
$ which pwd
/bin/pwd
\end{verbatim}
\texttt{which} tells you that if you call the \texttt{pwd} program,
it will use the one in the \texttt{/bin} directory. So, in case you
have a second version of \texttt{pwd} installed, which resides in
\texttt{/usr/local/bin}, you'll have to use the absolute path to call
it. 

But sometimes you won't know if there are different versions of the
same program on your machine, e.g. when you start working in a new
place with your machine already set up for you. To find out your
possibilities try
\begin{verbatim}
$ whereis COMMAND
\end{verbatim}
which will show you where versions of \texttt{COMMAND} are installed,
where their man pages are located and if the source code of the
command is stored in a standard location on the system.

Attention: The following explanation will most likely make no sense
at all if you haven't read about variables in
\ref{variablesandenvironment}. The actual difference between 
\texttt{whatis} and \texttt{whereis} is that the former looks 
through your \texttt{PATH} variable and returns the
first executable it finds which actually is the one your shell would
execute while the latter scans through your whole \texttt{PATH} as
well as \texttt{MANPATH} plus some standard locations commonly used
for programs on any (GNU/)Linux system, and returns everything it
finds there, separated into executables, source files, and man pages.

% end whatis and whereis

\subsection{The Internet Is Your Friend}
\label{theinternetisyourfriend}
If you have to do something real quick you may find yourself in a
situation where you don't want to deal with man pages or info pages,
but would prefer a ready-made command with all options you want
already set. As it is very likely that other persons have had the
problem at hand before you and many people post everything they
achieve online, chances are that you can find the command of your
dreams posted in some online forum, ready to be copy-pasted into your
terminal emulator. 

As you might expect, there are a myriad different sources where you
can find information, but we will only deal with very few of them
which offer reliable information in most cases.

\subsubsection{Distros' Websites}
\label{distroswebsites}
Each distro has its website and often Wikis attached to them. Many
have a lot of useful information, but keep in mind that these are
distro-specific. It is also important to know how your distro is
organized; in the case of Debian, you are faced with a community
project where the whole content is provided by people spending their
spare time on providing information to others, but no one is
responsible for doing so. This often leads to very helpful replies to
questions, targeted at total beginners, but as no one is responsible
for updating this content, sometimes you find very outdated
information. Red Hat Enterprise Linux, on the other hand,
is a distro aimed at corporations, providing a \emph{lot} of
information. However, as their material is intended for
IT-professionals, you won't find a lot of explanation in many cases as
they assume you already know what you are doing/want to do and just
show you how you can do it with the tools Red Hat provides.

Thus, do a bit of research about each distro before using their
documentation and how-tos.

\begin{description}
  \item [Debian:] Information about Debian can be found at:
    \begin{itemize}
      \item \href{https://www.debian.org}
        {https://www.debian.org}
      \item \href{https://wiki.debian.org/}
        {https://wiki.debian.org/}
      \item \href{https://debian-handbook.info/}
        {https://debian-handbook.info/}. This is the Debian
        Administrator's Handbook, the e-book version of which can be
        downloaded freely. However, you can also pay for the e-book
        version, and in case you start relying on it, it is definitely
        a good idea to give the authors something back for their hard
        work. The current (2018-09-12) version of the book if for
        Debien 8 Jessie, but the current Debian release is Debian 9
        Stretch, so some information my be outdated.
    \end{itemize}
  \item [Ubuntu:] Ubuntu is an enterprise distribution with a desktop
    and a server version. They have a very active community as well
    which answers a lot of questions in the forums.
    \begin{itemize}
      \item \href{https://www.ubuntu.com}{https://www.ubuntu.com}
      \item \href{https://tutorials.ubuntu.com/}
        {https://tutorials.ubuntu.com/}. You can filter tutorials for
        different categories, e.g. server or desktop.
      \item \href{https://help.ubuntu.com/}{https://help.ubuntu.com/}
      \item \href{https://ubuntuforums.org/} 
        {https://ubuntuforums.org/}
      \item \href{https://askubuntu.com/}{https://askubuntu.com/} This
        is a searchable question and answer site.
    \end{itemize}
  \item [Red Hat Enterprise Linux:] As mentioned above, RHEL is an
    enterprise distribution and one of the most prominent examples
    showing how much money you can earn with free software. They have
    excellent, though rather technical guides for many topics, and
    some things can be only accessed if you have a support contract
    with Red Hat.
    \begin{itemize}
      \item \href{https://www.redhat.com/}{https://www.redhat.com/}
        Their main page is very much like the website of other companies
        -- a lot of bragging and ads, but it's still a good starting
        point to get a feeling about what Red Hat as a company is
        doing.
      \item
      \href{https://access.redhat.com/}{https://access.redhat.com/}
        This is where you start out if you look for help. Browse a
        bit; they have a lot of documentation, guides, etc. and many
        things can be downloaded as PDF for free.
    \end{itemize}
  \item [CentOS:] CentOS, the ``Community Enterprise OS" is the
    community version of RHEL. It is only rarely suited for home
    Desktop use and the software it ships is very outdated in many
    cases (as all the good stuff is reserved for the actual RHEL).
    However, if you want to get to know RHEL without paying for a
    license, CentOS is a good start and may be useful if you want to
    run small servers at home/in the cloud.
    \begin{itemize}
      \item \href{https://centos.org/}{https://centos.org/}
      \item \href{https://wiki.centos.org/}{https://wiki.centos.org/}.
        While this Wiki is still alive, it is updated rather sparsely
        and the current release CentOS 7 is not documented in any
        reasonable fashion. So much so, that 
        \href{https://wiki.centos.org/docs/}
        {https://wiki.centos.org/docs/} just tells you to visit the
        above mentioned \href{https://access.redhat.com/}
        {https://access.redhat.com/}.
    \end{itemize}
  \item [Arch Linux:] While Arch Linux is considered to be rather
    complicated and suited only for experienced users, it has a
    thriving community and a very well maintained Wiki with in-depth
    explanations for a lot of things. Highly recommended if you want
    to know something in detail.
    \begin{itemize}
      \item \href{https://wiki.archlinux.org/}
        {https://wiki.archlinux.org/}
    \end{itemize}
\end{description}
We'll stop here, but you can comb through all the other distros'
websites as well, which is a very useful thing to do before you change
distro.

If you are interested how many distros are out there and which ones
are the most heavily used, you may find
\href{https://distrowatch.com/}{https://distrowatch.com/} (lists also
non-Linux UNIX systems like the various BSDs) or 
\href{https://static.lwn.net/Distributions/}
{https://static.lwn.net/Distributions/} useful.

% end Distros' Websites

\subsubsection{Other Online Sources}
\label{otheronlinesources}
However, there are also sites dedicated to Q\&{}A, i.e. you can post a
question and someone in the community replies with a hopefully correct
answer. The ones you'll encounter over and over again are:
\begin{description}
  \item [Stack Overflow:] To be found at
    \href{https://stackoverflow.com/}{https://stackoverflow.com/}.
    If you did some programming already or are going to do so, this
    site will also pop up many times. Whenever you research some
    problem you have with the command line it is almost certain that
    you are going to land on Stack Overflow after some time,
    regardless which search engine you use.
  \item [Stack Exchange for UNIX/Linux:] Strictly speaking this is
    just a subsite of Stack Overflow and many topics specifical to
    (GNU/)Linux and/or UNIX in general will can be found here:
    \href{https://unix.stackexchange.com/}
    {https://unix.stackexchange.com/}.
\end{description}
With these tools at your disposal you should be able to solve almost
any problem your new shiny (GNU/)Linux system throws at you.
\textbf{BUT}, beware! Not everything people post there is correct and
if you are unlucky, someone may even put some malicious code in their
reply, so \textbf{always} think before you copy-paste something onto
your command line and press \texttt{ENTER}, especially if you have no
idea what the command you want to use can do. Be even more careful if
the line you copy starts with \texttt{sudo}. The good thing about UNIX
systems is that you can do anything you like, but the bad thing is
that you can do anything you type.

% end Other Online Sources

% end The Internet Is Your Friend

% end Help Utilities

% end Basics

\chapter{The Command-Line}
Now, after so much theory, let's do something practical! The first
thing to do is to start a terminal emulator if you haven't already
done so. 

Before we are going to start working with commands, we'll talk a bit
about keyboard-shortcuts with which you can move quickly on a line.
For this purpose just type a sentence on the command-line without
pressing \texttt{ENTER}:
\begin{verbatim}
$ this is a beautiful test sentence
\end{verbatim}
Your cursor will now be at the end of the line. If you made a typo
somewhere in the line you can move there using the arrow keys, but
this gets annoying pretty quickly as soon as the line gets long (your
mouse won't work here). Thus, play around with the following
shortcuts:\footnote{Some of them may not work, because the
terminal-emulator you use interprets these shortcuts differently (e.g.
by opening up some menus). Usually this can be changed in the settings
of the terminal-emulator.}
\begin{center}
  \begin{tabular} { | c | p{12cm} | }
  \hline
  \texttt{CTRL-A} & Move to the beginning of the line.\\
  \texttt{CTRL-E} & Move to the end of the line.\\
  \texttt{CTRL-K} & Delete everything from your cursor to the end of
                     the line.\\
  \texttt{CTRL-U} & Delete everything from your cursor to the
                     beginning of the line.\\
  \texttt{CTRL-T} & Switch the sign currently below the cursor with
                     the one before it and move the cursor one
                     character to the right.\\
  \texttt{ALT-F} &  Move one word forward.\\
  \texttt{ALT-B} &  Move one word backwards.\\
  \texttt{ALT-T} &  Switch the word below the cursor with the one 
                     preceding it.\\
  \hline
  \end{tabular}
\end{center}
There are even more shortcuts, so look them up on your own if you're
curious, but the ones above will suffice for the beginning. In case
you don't see the use in some of them: That's ok, use cases will come
;)

And there is one more \emph{vital} thing to know about moving aroung
on the command-line: tab-completion. If you want to pass an argument
to a program and you've already typed enough that the argument is
uniquely identified, you can just hit \texttt{TAB} and the
\texttt{bash} will automatically complete the argument. Why this is
extremely useful will become clear once you start using commands and
arguments, just keep this in mind. Additionally, if you press
\texttt{TAB} twice, the \texttt{bash} will list all currently possible
arguments. Try it out when following the instruction below!

\section{Moving and Looking Around}
\label{movingandlookingaround}
Now, onwards to the command-line! For starters, let's find out where
we currently are -- you already know how to do this:
\begin{verbatim}
$ pwd
/home/me
\end{verbatim}
Now, let's see what's in the directory we're currently in, which we
can do with \texttt{ls(1)}, which ``lists" directory entries:
\begin{verbatim}
$ ls
bin      documents  nikola    playground         unixcourse
Desktop  Downloads  personal  tor-browser_en-US
\end{verbatim}
As you already know, you can pass options to \texttt{ls}. We'll start
with the \texttt{-a | --all} option. This one shows you all files in
the current directory:
\begin{verbatim}
$ ls -a
.              Downloads         .pki
..             .dvdcss           playground
.android       .F5Networks       .python_history
.ardentryst    .gconf            .ssh
.bash_history  .gnupg            tor-browser_en-US
.bash_logout   .gnuplot_history  .vim
.bash_profile  .gphoto           .viminfo
.bashrc        .ICEauthority     .vimrc
bin            .lesshst          .wget-hsts
.cache         .local            .Xauthority
.config        .mozilla          .xsession
Desktop        .neomutt          .xsession-errors
.dmrc          .xsession-errors.old
\end{verbatim}
What?! Where did all these files come from?! Don't worry, they have
been here all the time, but they are \emph{hidden files}.
You see that those files which have not been listed before, start with 
a dot `\texttt{.}' and this indicate that they are `hidden'. This
does not mean that you are not allowed to see them, but they are
mostly configuration files which you just don't want to see every
time you use \texttt{ls} as they just add a lot of useless output. For
example, your browser will store your bookmarks, preferences, cookies,
browsing history, etc. in a hidden directory. This way it does not
confuse your profile with the one of another user who uses the same
computer. Thus, a hidden file is just a file whose name starts with a
dot, and programs like \texttt{ls} don't display them by default --
nothing more to it. You can also create hidden files yourself if you
start their name with a dot!

However, two hidden files are found in every directory: `\texttt{.}'
and `\texttt{..}'. `\texttt{.}' is a pointer to the directory
itself\footnote{Why this is useful will become clear later.} and
`\texttt{..}' is a pointer to the directory above this directory. This
is a very convenient shortcut if you want to change to the directory
above the one you are currently in, but you don't want to type the
absolute path from the root directory. Changing directories? Great,
we're already at the next command: \texttt{cd} which very surprisingly
stands for `change directory'. It takes as an argument a path
(relative or absolute) and changes your current working directory:
\begin{verbatim}
$ cd Documents
$ pwd
/home/me/Documents
$ cd ..
$ pwd
/home/me
\end{verbatim}
You can see here that we changed into the \texttt{Documents}
directory and from there one directory "upwards" by using \texttt{..}
as shortcut. Otherwise we would have had to type \texttt{cd /home/me}
to change from \texttt{/home/me/Documents} to the \texttt{/home/me}.

Two more things regarding \texttt{cd}: 
\begin{itemize}
  \item As we've seen in \ref{helpandtype}, it is a shell-builtin, so
    if you want to know how it works, you have to type ``\texttt{help
    cd}". 
  \item If you just type \texttt{cd} on the command-line and press
    \texttt{ENTER} without providing an argument, \texttt{cd} will
    change to your home directory. The home directory is where all
    your personal stuff is stored and in the examples above, my home
    directory is \texttt{/home/me}. Thus, if you get lost in the
    filesystem or you are in a hurry and just want to go back to your
    home directory, type \texttt{cd}, press \texttt{ENTER} and you'll
    be right there.
\end{itemize}
Now that you've seen \texttt{cd}, let's go back to \texttt{ls}.
Combining \texttt{ls}, \texttt{pwd}, and \texttt{cd} the filesystem is
yours to explore and we'll let you do that in a minute, just one more
option to \texttt{ls} and the associated concepts:

With the \texttt{-l} option \texttt{ls} provides you with ``long
listing":
\begin{verbatim}
$ ls -l
total 36
drwxr-xr-x  2 me me 4096 Aug 27 14:53 bin
drwxr-xr-x  2 me me 4096 Sep  4 17:49 Desktop
drwxr-xr-x 14 me me 4096 Sep  7 17:40 documents
drwxr-xr-x  2 me me 4096 Sep 10 21:43 Downloads
drwxr-xr-x  8 me me 4096 Aug 24 20:05 nikola
drwxr-xr-x  4 me me 4096 Aug 27 14:54 personal
drwxr-xr-x  3 me me 4096 Aug 26 20:43 playground
drwx------  3 me me 4096 Sep  9 14:36 tor-browser_en-US
drwxr-xr-x  2 me me 4096 Sep 17 11:48 unixcourse
\end{verbatim}
% TODO: Change all ls output to the unixcourse thing
As you can see there is a lot more information about each file.
Actually, the output is now separated into fields, separated by a
space. These fields are from left to right:
\begin{description} 
  \item [Permissions:] See below, that's the new concept we're going
    to explain ;)
  \item [Hard Links:] The number of hard links which point to this 
    file. If you are interested in links, read it up, we won't cover
    them here as you don't need them for now.
  \item [Owner:] The user to whom this file belongs.
  \item [Group:] The group to which this file belongs.
  \item [Size:] The size of the file in bytes.
  \item [Modification Time:] The month, day and time give you the
    exact time when the file was last modified. If it was not modified
    in the current year, the year of the last modification is given
    as well.
  \item [Name:] The name of the file.
\end{description}
Each file has an user owner which is usually the user who created the
file in the first place as well as a group owner, which is usually one
of the groups the user owner belongs to.\footnote{From now on we will
just speak of ``owner" to mean ``user owner" and of ``group" to mean
``group owner".} But what does that mean?

Well, it has to do with the first field which consists of a rather
bewildering number of letters and dashes at first sight. However, it
is actually a very simple system and tells you what you can do with a
file:

The first character tells you \emph{what kind of file} this
is, e.g. \texttt{d} indicates a directory and \texttt{-} indicates
a regular file. There are others as well, but these are the most
common and important ones -- if you want to know more look at the
info page of \texttt{ls(1)}.

The next nine characters show the permissions of the file which in
very short means who is allowed to do what with a file. These nine
characters are actually 3 groups of 3 permission sets: The first three
are the permissions of the owner\footnote{Yes, even owners are not
necessarily allowed to do everything with the file!}, the second three
are the permissions for the group, and the third permission set tells
you what anybody else (often called ``world" or ``other") is allowed
to do.

The reading of permissions is pretty easy: If there is a letter, the
permission is present, if there is a dash it is absent. The postitions
in each permission group from left to right tell you about:
\begin{description}
  \item [\texttt{r}] ``Read permission". If this permission is set,
    it is allowed to read the file.
  \item [\texttt{w}] ``Write permission". If this permission is set,
    it is allowed to write something to the file, i.e. change it.
  \item [\texttt{x}] ``Execute permission". If this permission is set,
    it is allowd to execute this file -- this is typically set for all
    programs, like \texttt{ls(1)}.
\end{description}
With this information at hand, can you decode this output:
\begin{verbatim}
-rwxr-x--x 1 nina hackers 2048 Jan 14 14:21 amazing-script
\end{verbatim}
Try it, write down your solution and afterwards read the following
explanation.

This is a regular file, which can be executed by the owner, the group,
and everyone else, can be read by the owner and the group and written
only by the owner. It belongs to user \texttt{nina} and to the group
\texttt{hackers}. It is 2048 bytes in size (i.e. 2 KB), was last modified
on January 14\textsuperscript{th} at 14:21, and is called
``\texttt{amazing-script}".

As you've seen in \ref{thefilesystem}, you can also pass absolute and
relative paths as arguments to \texttt{ls} to tell it what to list.
There is one thing you should be aware in this case: If the argument
is a regular file, \texttt{ls} will list just the file, but if the
argument is a directory, \texttt{ls} will show you the content of the
directory:
\begin{verbatim}
$ ls file1
file1
$ ls contentdir
file_00  file_05  file_10  file_15   subdir04  subdir09
file_01  file_06  file_11  subdir00  subdir05  subdir10
file_02  file_07  file_12  subdir01  subdir06
file_03  file_08  file_13  subdir02  subdir07
file_04  file_09  file_14  subdir03  subdir08
\end{verbatim}
Great, now you are ready for your tour through the filesystem! Use
\texttt{ls}, \texttt{cd}, and \texttt{pwd} to look around -- you are
on a free software OS, there are no secrets (except if you don't have
permission to look at something, of course ;)). 

% end Moving and Looking Around

\section{Working with Files}
\label{workingwithfiles}
Up to this point we've just looked at files, examined their properties
and moved around in the filesystem tree. This is all nice and fine,
but maybe we want to change files' properties, or add stuff to or
remove something from the filesystem. This is what this section is
about. And for now, change in the \texttt{unixcourse} directory you've
downloaded at the beginning of \ref{basics} to be able to follow all
the examples below.

One of the things you may have already wondered about is: What if you
want to have a directory for all of your lovely cat GIFs -- how can
you create such a directory? Well, the answer to this is
\texttt{mkdir(1)} ``make directory". In its simplest form it just
takes a single argument: The name of the directory you want to create.
Note, that this can be a relative or absolute path:
\begin{verbatim}
$ ls
 contentdir         file3          onelinefile
 dir1               filetodelete   renamedfile
 dir2               grepfile       rmdirectory
 dir3               hugefile       rminteractive
'dir with spaces'   join1          sedfile
 errorfile          join2          sedfile.bak
 file1              linefile       sortfile
 file2              numericsort    tablefile
$ mkdir newdir
$ ls
 contentdir         filetodelete   renamedfile
 dir1               grepfile       rmdirectory
 dir2               hugefile       rminteractive
 dir3               join1          sedfile
'dir with spaces'   join2          sedfile.bak
 errorfile          linefile       sortfile
 file1              newdir         tablefile
 file2              numericsort
 file3              onelinefile
$ cd newdir
$ pwd
/home/me/unixcourse/newdir
\end{verbatim}
One thing about names: Whether you create directories as we do now, or
other files which we'll do later, there are certain rules you should
adhere to, to make your life easier: A file name (which includes
directory names!) should consist of only letters of the English
alphabet, underscores `\texttt{\_}', dashes `\texttt{-}', and
digits, and should always start with an underscore or a letter. Don't
use special characters like ä, ß, ж, €, etc. and don't use spaces,
tabs, newlines, etc.!
This is just conventional, so you \emph{can} use whatever you want,
but it may make your life very inconvenient. In case you encounter
filenames containing spaces, you will have to quote the name or
escape the spaces using `\texttt{\\}'. If you use \texttt{ls(1)} in
the course directory you'll see that there is a directory called
``\texttt{dir with spaces}". To change into it, you have three
options:
\begin{verbatim}
$ pwd
/home/me/unixcourse
$ cd "dir with spaces"
$ pwd
/home/me/unixcourse/dir with spaces
$ cd ..
$ pwd
/home/me/unixcourse
$ cd 'dir with spaces'
$ pwd 
/home/me/unixcourse/dir with spaces
$ cd ..
$ pwd
/home/me/unixcourse
$ cd dir\ with\ spaces
$ pwd
/home/me/unixcourse/dir with spaces
\end{verbatim}
So you see, you can use either single or double quotes\footnote{They
are not the same in other contexts which we'll encounter later.}, or
just escape each space with a prepended backslash. Again, we recommend
that you don't do this because it may break some programs and scripts.
Don't get us wrong -- we don't think that it's great that you have to
stick with these conventions, but as most people do, it will make it
harder for you to use others' solutions to problems you encounter.

After this little detour about naming, let's return to creating
directories. Per default \texttt{mkdir} assumes that the whole path of
the directory you want to create exists, except for the last part of
the path, i.e. the part behind the last `\texttt{/}'. If you try to
create a whole directory hierarchy, this will fail:
\begin{verbatim}
$ ls dir1
$ mkdir dir1/subdir/subsubdir
mkdir: cannot create directory ‘dir1/subdir/subsubdir’: No such file or directory
\end{verbatim}
The first command creates no output because there is nothing in
\texttt{dir1} and when we try to create a subdirectory \texttt{subdir}
with a sub-subdirectory \texttt{subsubdir} we get an error message
(not a very helpful one in this case, but we know what the problem
is anyway). But surely, there must be a way to create such
hierarchies, right? Of course! You just have to pass the \texttt{-p |
--parents} option to \texttt{mkdir}:
\begin{verbatim}
$ mkdir -p dir1/subdir/subsubdir
$ ls dir1
subdir
$ ls dir1/subdir
subsubdir
\end{verbatim}
Voilà, this time the whole path was created.

Nothing persists forever, so from creation we move directly to
destruction: Whenever you want to delete a directory, you can use
\texttt{rmdir(1)} ``remove directory". Like \texttt{mkdir} it takes
one (or more) directories as arguments and deletes them. Ok, let's go!
\begin{verbatim}
$ rmdir dir1/subdir
rmdir: failed to remove 'dir1/subdir/': Directory not empty
\end{verbatim}
Hm, that didn't work out as expected. But this is a good thing in this
case. \texttt{rmdir} only removes directories if they are empty, i.e.
there are no files (except `\texttt{.}' and `\texttt{..}') in it. Why
is this such a good thing? Well, now we're getting at something you
should \emph{really} remember: On the command-line \emph{there is no
undo}! If you delete something, it is gone for good and only people
with a lot of training and expertise are able to recover files or
directories you deleted.\footnote{Their training is usually directly
reflected in the rather huge amount of money you have to pay them to
get your data back.} Therefore, \texttt{rmdir} prevents you from
mistakingly removing a lot of your files with a single line.

However, if you have a situation like we have, where we have
\texttt{dir1} which contains only \texttt{subdir} which only contains
\texttt{subsubdir} which contains nothing and you want to delete
\texttt{subdir} and \texttt{subsubdir} \texttt{rmdir} offers the same
option as \texttt{mkdir} does, to delete such a chain of empty
directories, i.e. \texttt{-p | --parents}:
\begin{verbatim}
$ rmdir -p dir1/subdir/subsubdir
\end{verbatim}
This deletes all three directories in the given path. Now, recreate
the \texttt{dir1} directory and move on.

As you might have guessed, \texttt{rmdir} removes only directories --
just try it out with the following file, conveniently placed in your
course directory:
\begin{verbatim}
$ rmdir filetodelete
rmdir: failed to remove 'filetodelete': Not a directory
\end{verbatim}
Now that wasn't surprising at all. But how can we remove files? The
answer to this is \texttt{rm(1)} ``remove". Thus:
\begin{verbatim}
$ ls
 contentdir         filetodelete   renamedfile
 dir1               grepfile       rmdirectory
 dir2               hugefile       rminteractive
 dir3               join1          sedfile
'dir with spaces'   join2          sedfile.bak
 errorfile          linefile       sortfile
 file1              newdir         tablefile
 file2              numericsort
 file3              onelinefile
$ rm filetodelete
 contentdir         file1      join2         rminteractive
 dir1               file2      linefile      sedfile
 dir2               file3      numericsort   sedfile.bak
 dir3               grepfile   onelinefile   sortfile
'dir with spaces'   hugefile   renamedfile   tablefile
 errorfile          join1      rmdirectory
\end{verbatim}
This time, the file is gone. And remember what we said earlier about
removing stuff? It's gone. Totally. Absolutely. No rebirth (without a
lot of money). If you'd try to remove a directory with \texttt{rm},
this won't work out of the box:
\begin{verbatim}
$ rm dir1
rm: cannot remove 'dir1': Is a directory
\end{verbatim}
No surprises here. However, \texttt{rm} has some options we're going
to talk about:
\begin{description}
  \item [\texttt{-r | -R | --recursive}] With this option, \texttt{rm}
    removes files and directories recursively, i.e. if you pass a
    directory as an argument with this option \texttt{rm} will remove
    this directory and \emph{everything} within it. Again, no undo.
    \emph{NO UNDO!} Sorry, just sayin'. For the sake of an example:
\begin{verbatim}
$ ls rmdirectory
file_01  file_03  file_05  file_07  file_09
file_02  file_04  file_06  file_08  file_10
$ rm -r rmdirectory
$ ls rmdirectory
ls: cannot access 'rmdirectory/': No such file or directory
\end{verbatim}
    Yes, the directory and all ten files it contained are gone now.
  \item [\texttt{-i}] If you use this option, \texttt{rm} will ask you
    to confirm the deletion of each file, which can save your one file
    you forgot was in that directory, but \emph{really} 
    need.\footnote{Ever realized that ``file" is just an anagram of
    ``life"?} Thus:
\begin{verbatim}
$ ls rminteractive
file_01  file_03  file_05  file_07  file_09
file_02  file_04  file_06  file_08  file_10
$ rm -ri rminteractive
rm: descend into directory 'rminteractive/'? y
rm: remove regular file 'rminteractive/file_01'? y
rm: remove regular file 'rminteractive/file_05'? y
rm: remove regular file 'rminteractive/file_04'? n
rm: remove regular file 'rminteractive/file_09'? y
rm: remove regular file 'rminteractive/file_07'? y
rm: remove regular file 'rminteractive/file_10'? y
rm: remove regular file 'rminteractive/file_02'? n
rm: remove regular file 'rminteractive/file_03'? y
rm: remove regular file 'rminteractive/file_06'? n
rm: remove regular file 'rminteractive/file_08'? y
rm: remove directory 'rminteractive/'? n
$ ls rminteractive
file_02  file_04  file_06
\end{verbatim}
    For each question \texttt{rm} asks you have to type `\texttt{y}'
    for ``yes" or `\texttt{n}' for ``no". 
  \item [\texttt{-f | --force}] Force \texttt{rm} to do what you told
    it remove everything without questions\footnote{In some cases
    \texttt{rm} asks you if you really want to delete a file even
    without the \texttt{-i} option present. See \texttt{man 1 rm}.}
    \texttt{-r | -R | --recursive} was dangerous, \emph{this} one is 
    where the real bad stuff begins. Don't use it if you are not
    absolutely, 100\%{}, utterly, totally, completely convinced that
    you know what you are doing. ``Oh, come on", you may think, ``I'm
    not \emph{that} stupid, I know this by now. What can possibly go
    wrong?". Well, have a look at
    \href{https://www.independent.co.uk/life-style/gadgets-and-tech/news/man-accidentally-deletes-his-entire-company-with-one-line-of-bad-code-a6984256.html}
    {this}, were someone accidentally wiped all data of a whole
    company (including the backup) with one single \texttt{rm -rf}.
    The author of these lines once removed all invoices of a company
    he worked for, but fortunately there was no \texttt{-r} option
    present, so it were just the invoices of the current year. There
    was no backup. I had to retype them by hand from printed versions
    -- great fun, 10 out of 10, would recommend... 
\end{description}
So, what else can you do with files? What about copying stuff from one
place to somewhere else? This can be done with the \texttt{cp(1)}
command. Its basic syntax is:
\begin{verbatim}
cp [OPTION]... SOURCE DESTINATION
\end{verbatim}
i.e. after maybe passing one or more options to \texttt{cd} you first
tell it \emph{what} you want to copy and then you tell it \emph{where}
to copy it. Source and destination can be relative or absolute paths.
As a very simple example, you can do:
\begin{verbatim}
$ ls
contentdir         filetodelete   onelinefile
 dir1               grepfile       renamedfile
 dir2               hasfileinit    rmdirectory
 dir3               hugefile       rminteractive
'dir with spaces'   join1          sedfile
 errorfile          join2          sedfile.bak
 file1              linefile       sortfile
 file10             lsfile         tablefile
 file3              numericsort
$ cp file1 file2
$ ls
 contentdir         file1      join2         rminteractive
 dir1               file2      linefile      sedfile
 dir2               file3      numericsort   sedfile.bak
 dir3               grepfile   onelinefile   sortfile
'dir with spaces'   hugefile   renamedfile   tablefile
 errorfile          join1      rmdirectory
\end{verbatim}
However, there are some things about this simple scheme you have to
take into account:
\begin{itemize}
  \item If you want to copy a directory and all its content you have
    to use the \texttt{-r | -R | --recursive} option, like with
    \texttt{rm} when you wanted to delete a directory with its
    contents.
  \item If the destination is a directory, \texttt{cp} will copy the
    source into the destination directory, e.g.:
\begin{verbatim}
$ ls dir1
$ cp file1 dir1
$ ls dir1
file1
\end{verbatim}
  \item If the destination is a directory, you can give multiple
    sources, i.e. you can copy multiple files at once into a directory
    by listing them all on the command-line and providing the
    directory you want to copy them into as the last argument:
\begin{verbatim}
$ cp file1 file2 file3 dir1
$ ls dir1
file1 file2 file3
\end{verbatim}
  \item But, wait, we already copied \texttt{file1} to \texttt{dir1}, so
    which version is now in \texttt{dir1}? As you may have guessed
    from the behaviour of \texttt{rm} and \texttt{rmdir}: The
    command-line is unforgiving. If there is already a file with the
    same name in a directory, this file is overwritten and \texttt{cp}
    won't tell you about it -- it assumes that you know what you are
    doing. Fortunately, just like \texttt{rm} there is a \texttt{-i |
    --interactive} option which asks you every time a file would be
    overwritten. In addition to that, \texttt{cp} also offers the
    \texttt{-n | --no-clobber} option which tells the program to never
    overwrite existing files.
\end{itemize}
The last but one thing we're going to cover for working with files is
ow to rename and move them around. This is done by the same utility,
\texttt{mv(1)} ``move". Its standard syntax is:
\begin{verbatim}
mv [OPTION]... SOURCE DESTINATION
\end{verbatim}
Not only is its syntax the same as the one for \texttt{cp}, the same
caveats about how to move multiple files and moving directories apply
as well, with one exception: There is no \texttt{-r | -R |
--recursive} or a similar option for \texttt{mv} -- it always moves
a whole directory if you pass one as source. If you just want to
rename a file without moving it into another place in the filesystem,
you just put the destination in the same directory:
\begin{verbatim}
$ ls
 contentdir         file1      join2         rminteractive
 dir1               file2      linefile      sedfile
 dir2               file3      numericsort   sedfile.bak
 dir3               grepfile   onelinefile   sortfile
'dir with spaces'   hugefile   renamedfile   tablefile
 errorfile          join1      rmdirectory
$ mv file3 renamedfile
$ ls
 contentdir         file1      join2         rminteractive
 dir1               file2      linefile      sedfile
 dir2               file3      numericsort   sedfile.bak
 dir3               grepfile   onelinefile   sortfile
'dir with spaces'   hugefile   renamedfile   tablefile
 errorfile          join1      rmdirectory
\end{verbatim}
We'll end this section with something a bit more abstract again. In
\ref{movingandlookingaround} we introduced you to the concept of
permissions when you encountered the output of \texttt{ls -l}.
Permissions were originally used in UNIX when many users sitting in
front of terminals (the hardware devices you saw in 
\ref{shellsterminalscommands}) connected to central computers where
all of them worked at the same time. To enable them to collaborate
more easily they could set the permissions for group and others as
they wanted to. This was done by the \texttt{chmod(1)} ``change (file)
mode". \texttt{chmod}'s basic syntax is:
\begin{verbatim}
chmod [OPTION]... MODE[,MODE]... FILE...
\end{verbatim}
There are two ways to pass a set of permissions to \texttt{chmod}:
By a symbolic representation and by an octal number. The general
format of the symbolic mode is:
\begin{verbatim}
[ugoa...][[-+=][perms...]...]
\end{verbatim}
The first part indicates for whom you want to change the
permissions: \texttt{u} for ``user", i.e. the user who owns the file,
\texttt{g} for ``group", \texttt{o} for ``others" and \texttt{a} for
``all". The default is \texttt{a}, i.e. to change the permissions for
everyone, but if you write \texttt{u} you change only the user's
permissions and if you write \texttt{ug} you change the permissions of
the user and the group. As you may have noticed \texttt{ugo} and
\texttt{a} are equivalent. 

The second part means what you want to do: \texttt{+} to add the
following permissions, \texttt{-} to remove the following permissions
and \texttt{=} to set the permissions to exactly the following
permissions. 

The third field consists of the permissions you want to
add/remove/set, i.e. the letters \texttt{r}, \texttt{w}, and
\texttt{x}.\footnote{If you look at the manpage of \texttt{chmod}
you'll see that there are three more possible values, but for now,
we'll ignore them.} 

Consider this example:
\begin{verbatim}
$ ls -l file1
-rw-r--r-- 1 me me 29 Sep 18 15:24 file1
$ chmod g+w file1
$ ls -l file1
-rw-rw-r-- 1 me me 29 Sep 18 15:24 file1
\end{verbatim}
As you can see, we gave the group (\texttt{g}) additional (\texttt{+})
write permission (\texttt{w}), but left the other permissions intact.

Now it's your turn: Try do remove all rights from the group and the
others. Is there more than one way to do this?

We're not going to explain the octal mode here: It's not hard, but it
can be confusing at the beginning and can take a bit more time to
grasp -- read the manpage if you are interested :)

While \texttt{chmode} changes the permissions on a file, there are
also two utilities you can use to change the user owner and group
owner of a file: \texttt{chown(1)} ``change owner" to change the user
owner and \texttt{chgrp(1)} ``change group" to change the group owner.
Their syntax is rather similar:
\begin{verbatim}
chown [OPTION]... [OWNER][:[GROUP]] FILE...
chgrp [OPTION]... GROUP FILE...
\end{verbatim}
As you can see, \texttt{chown} can actually be used to change user
and group. In many cases where you use one of them you actually want
to change all ownerships within a whole subtree of the filesystem.
Thus, both of them offer a \texttt{-R | --recursive} to do so
recursively. Note that this is an uppercase `\texttt{R}'! 

This can be useful e.g. if someone copies a file from their
homedirectory into yours, but the owner information is copied as well,
so you have to change it before you can use it.

% end Working with Files

\section{Working with Content}
At this point you know how to move around and how to change files, but
you haven't learned how to work with the stuff that is \emph{in}
files, i.e. content. Many files on your system contain human
readable\footnote{Well, let's say ``readable by \emph{some} humans"
;)} text which you can easily manipulate.

\subsection{Looking at Content}
\label{lookingatcontent}
But before we start writing stuff somewhere, we'll look at it first.
The easiest way to see the contents of a file is to use the
\texttt{cat(1)} command. This is an abbreviation for ``concatenate",
i.e. combining stuff. Its basic syntax is:
\begin{verbatim}
cat [OPTION]... [FILE]...
\end{verbatim}
Now let's unravel the mysteries of the ominous \texttt{file1} with
which we played around in the last section without ever knowing what's
in there:
\begin{verbatim}
$ cat file1
A file with copiable content
\end{verbatim}
Well, that wasn't very mysterious... It just contains a single line of
text saying "A file with copiable content". So that is what
\texttt{cat} does: It reads the contents from a file and prints them
to the screen. What about another file:
\begin{verbatim}
$ cat onlinefile
This is a nice one liner
\end{verbatim}
Ok, also nothing very interesting. But what about this concatenation
thing? Have a look:
\begin{verbatim}
$ cat file1 onelinefile
A file with copiable content
This is a nice one liner
$ cat onelinefile file1
This is a nice one liner
A file with copiable content
\end{verbatim}
You see, \texttt{cat} concatenates the content of the files in the
order you pass them on the command-line. This is tremendously useful
if you want to e.g. combine multiple files into one very long file
without copy-pasting all the stuff. How to do this will be shown
later.

As you may have realized, if files have a lot of content, i.e. many
pages of test, \texttt{cat} will be rather inconvenient because it
just prints everything to the screen and you can't read fast enough to
see everything:
\begin{verbatim}
$ cat hugefile
\end{verbatim}
To read through such a file, you can use programs called ``pagers" --
they display text files one page at a time. The standard pager on
(GNU/)Linux is \texttt{less(1)}.\footnote{In the old days, the
standard pager was \texttt{more(1)}. When someone wanted to replace it
they thought that ``more is less" and thus the name of \texttt{less}.}
So, let's view our file:
\begin{verbatim}
$ less hugefile
\end{verbatim}
To move around scroll forward one page, use \texttt{SPACE}, for
backwards use \texttt{B}. You can also move up and down one line with
the arrow keys. To search for a pattern forward press `\texttt{/}'
enter the pattern and press \texttt{ENTER}, to search for a pattern
backwards, press `\texttt{?}' and press \texttt{ENTER}.  
When you finally want to quit the program, press \texttt{Q}. 

But then again, how are you supposed to know beforehand how big a file
is, i.e. whether it makes sense to use \texttt{cat} or \texttt{less}?
There are different strategies and one you know already:
\begin{verbatim}
$ ls -l onelinefile hugefile
-rw-r--r-- 1 me me 431888 Sep 18 17:52 hugefile
-rw-r--r-- 1 me me     25 Sep 17 13:53 onelinefile
\end{verbatim}
If you look at the size column you can see that there is a huge
difference between these two files. Unfortunately, using bytes as a
measuse is not very human-readable, especially because even
\texttt{hugefile} could in fact consist of only a single line -- a
\emph{very} long line, but it is possible. Therefore, it would be
useful to be able to view the lines of a file. This can be done with
\texttt{wc(1)} ``word count":
\begin{verbatim}
$ wc onelinefile
 1  6 25 onelinefile
$ wc hugefile
 7654  70552 431888 hugefile
\end{verbatim}
\texttt{wc} provides with some useful \emph{metadata}, i.e. data about
data, about a file. The first column is the number of lines in the
file, the second column shows the number of words in the file and the
third column tells you how many bytes the file contains. You can also
list just one of these units using the \texttt{-l | --lines} for only
the line count, \texttt{-w | --words} for only the word count, and
\texttt{-c | --bytes}\footnote{``Why \texttt{-c}?" you may ask. This
is because years ago characters were always encoded in ASCII (see the 
\href{https://en.wikipedia.org/wiki/ASCII}{Wikipedia entry} for an
explanation) where each of them was exactly one byte in size and so
the \texttt{-c} for \textbf{c}haracter came into being.} for only the
byte count.

% end Looking at Content

\subsection{Creating Content and I/O redirection}
After so much information reception it is time to let you create
something yourself! And the first tool for this is \texttt{echo(1)}.
Very unexpectedly \texttt{echo} echoes what you pass to it:
\begin{verbatim}
$ echo Hello command-line
Hello command-line
\end{verbatim}
This seems easy enough. But don't be fooled, \texttt{echo} is an
extremely useful tool and will introduce you to a lot of new concepts.
Just as an aside: It is often a good idea to put the stuff you want
\texttt{echo} to echo in double quotes. Read the man page to find out
why this is so ;)

The first one is the concept of \emph{I/O streams}\footnote{For
Input/Output stream.}. A stream is a flow of
data made available through time. While this sounds rather abstract,
you've already worked with streams all the time in this course!
Whenever you open a \texttt{bash} there are three standard streams
opened for this \texttt{bash} as well: 
\begin{description}
  \item [\texttt{stdin}] or ``Standard Input", i.e. the stream into
    which you put data. This is usually your keyboard because this is
    where you input the stuff you write. The \texttt{bash} then reads
    those characters from the keyboard and displays them on the
    command-line.
  \item [\texttt{stdout}] or ``Standard Output", i.e. the stream into 
    which the data goes. In most cases this is your screen or rather
    the terminal emulator on your screen. Yes, most of the time a
    program produces output, it writes it to \texttt{stdout}!
  \item [\texttt{stderr}] or ``Standard Error", i.e. the stream into
    which error messages are sent. This is also mostly your screen and
    you've already seen this in action: Remember when we tried to
    remove a non-empty directory with \texttt{rmdir}? There was an
    error message displayed and while you saw it on your screen, this
    was actually written to \texttt{stderr} and not to
    \texttt{stdout}. 
\end{description}
But why make two stream which write to the same thing?! The reason is
simple: They do so usually, but you can manipulate them indiviually.
Manipulate? Oh yeah, you can play around with those streams until your
head spins! This called I/O redirection. 

It is time, to create your first file! You know now that
whenever \texttt{echo} echoes something back to you, it actually reads
input from \texttt{stdin} and puts it to \texttt{stdout}. But what if
we \emph{redirect} \texttt{stdout} to a file? See here:
\begin{verbatim}
$ ls 
 contentdir         file1      join2         rminteractive
 dir1               file2      linefile      sedfile
 dir2               file3      numericsort   sedfile.bak
 dir3               grepfile   onelinefile   sortfile
'dir with spaces'   hugefile   renamedfile   tablefile
 errorfile          join1      rmdirectory
$ echo "I want this in a file" > echofile
$ ls 
 contentdir         errorfile   join1         rmdirectory
 dir1               file1       join2         rminteractive
 dir2               file2       linefile      sedfile
 dir3               file3       numericsort   sedfile.bak
'dir with spaces'   grepfile    onelinefile   sortfile
 echofile           hugefile    renamedfile   tablefile
$ cat echofile
I want this in a file
\end{verbatim}
So now you know that by using `\texttt{>}' you can redirect
\texttt{stdout} to a file. Let's try that again:
\begin{verbatim}
$ echo "More content" > echofile
$ cat echofile
More content
\end{verbatim}
Ok, that was not expected. Where's the former line gone? Well, it's
not here anymore. If you use `\texttt{>}' to redirect something to a
file, the file will be \emph{truncated}, i.e. totally emptied, and
afterwards the new content will be written to the file. If you want to
just add something to a file, without deleting what is already in
there, you have to use ``\texttt{>>}", which \emph{appends} data to a
file:
\begin{verbatim}
$ echo "And now a new line" >> echofile
$ cat echofile
More content
And now a new line
\end{verbatim}
Note that there is no way to put something in the middle of a file
using just the command-line. To do this you need a text editor which
we'll discuss later.

You can also use this redirection on any other command, e.g. if you
want to store its output so you can look at it later:
\begin{verbatim}
$ ls -l > lsfile
$ cat lsfile
total 516
drwxr-xr-x 13 me me   4096 Sep 19 10:05 contentdir
drwxr-xr-x  2 me me   4096 Sep 18 15:36 dir1
drwxr-xr-x  2 me me   4096 Sep 17 13:52 dir2
drwxr-xr-x  2 me me   4096 Sep 17 13:52 dir3
drwxr-xr-x  2 me me   4096 Sep 17 15:36 dir with spaces
-rw-r--r--  1 me me     22 Sep 23 12:29 echofile
-rw-r--r--  1 me me     63 Sep 19 11:07 errorfile
-rw-r--r--  1 me me     29 Sep 18 15:24 file1
-rw-r--r--  1 me me     29 Sep 18 15:33 file2
-rw-r--r--  1 me me     11 Sep 23 12:22 file3
-rw-r--r--  1 me me    472 Sep 23 09:49 grepfile
-rw-r--r--  1 me me 431888 Sep 18 17:52 hugefile
-rw-r--r--  1 me me     46 Sep 23 12:11 join1
-rw-r--r--  1 me me     48 Sep 23 12:11 join2
-rw-r--r--  1 me me    405 Sep 23 12:22 linefile
-rw-r--r--  1 me me     26 Sep 23 12:22 numericsort
-rw-r--r--  1 me me     25 Sep 17 13:53 onelinefile
-rw-r--r--  1 me me     69 Sep 18 15:36 renamedfile
drwxr-xr-x  2 me me   4096 Sep 18 11:14 rmdirectory
drwxr-xr-x  2 me me   4096 Sep 18 15:18 rminteractive
-rw-r--r--  1 me me    331 Sep 23 11:02 sedfile
-rw-r--r--  1 me me    331 Sep 23 10:48 sedfile.bak
-rw-r--r--  1 me me     18 Sep 23 12:22 sortfile
-rw-r--r--  1 me me    747 Sep 23 11:25 tablefile
\end{verbatim}
Your \texttt{ls} output has now been stored in a file.

Now, what if we want to do something with \texttt{stderr}?
\begin{verbatim}
$ ls nonexistentfile
ls: cannot access 'nonexistentfile': No such file or directory
$ ls nonexistentfile > errorfile
ls: cannot access 'nonexistentfile': No such file or directory
$ cat errorfile
$ ls nonexistentfile 2> errorfile
$ cat errorfile
ls: cannot access 'nonexistentfile': No such file or directory
\end{verbatim}
You can see that if we use `\texttt{>}' to redirect the output,
\texttt{errorfile} stays empty, because \texttt{ls} didn't print
anything to \texttt{stdout}, just to \texttt{stderr} and because we
just redirected \texttt{stdout} to the file, \texttt{stderr} is still
printed to the command-line. The redirection for \texttt{stderr} is
``\texttt{2>}" and you can see that using this results in the error
message being not printed to the screen, but written to the file
instead.

To be honest, we cheated a bit: Actually the streams are numbered,
starting with zero: \texttt{stdin} is 0, \texttt{stdout} is 1 and
\texttt{stderr} is 2. You can only write to \texttt{stdout} and
\texttt{stderr} because they are output streams while \texttt{stdin}
is an input stream you can't write to.\footnote{This is pretty easy to
understand if you think about the meaning of the streams: How would
you try to write to your keyboard?} Thus, only output streams can be
redirected using `\texttt{>}' and you can prepend the number of the
output stream you want to redirect, i.e. \texttt{1>} or \texttt{2>}.
If you don't specify anything, however, \texttt{1>} is implied because
you much more often want to redirect \texttt{stdout} than
\texttt{stderr} and this is why just `\texttt{>}' above worked as it
did.

But we talked about ``manipulating streams individually". What does
that mean?
\begin{verbatim}
$ ls file1 nothere
ls: cannot access 'nothere': No such file or directory
file1
$ ls file1 nothere > lsfile 2> errorfile
$ cat lsfile
file1
$ cat errorfile
ls: cannot access 'nothere': No such file or directory
\end{verbatim}
As you can see, it is possible to redirect \texttt{stdout} and
\texttt{stderr} to different files. This can be very useful if you
have a program with a lot of output and you want to keep normal output
and error messages apart.

Note that if you want both of them to redirect to the same file you
have to first redirect one stream to the file and then redirect the
other one to the other stream:
\begin{verbatim}
$ ls file1 nothere > twostreams 2>&1
$ cat twostreams
ls: cannot access 'nothere': No such file or directory
file1
\end{verbatim}
To redirect \texttt{stderr} to \texttt{stdout} you use \texttt{2>\&1}
which reads approximately as ``redirect \texttt{stderr} to where
\texttt{stdout} writes to". You could also do the following:
\begin{verbatim}
$ ls file1 nothere 2> twostreams 1>&2
$ cat twostreams
ls: cannot access 'nothere': No such file or directory
file1
\end{verbatim}
The result is the same. Note, however, that the order is important:
\emph{First}, redirect one stream to a file, and \emph{afterwards}
redirect the other stream to it. This makes sense: If you read the
sentence describing \texttt{2>\&1} carefully, you'll notice that it
says ``where \texttt{stdout} writes to" and not ``to \texttt{stdout}".
Thus, if you redirect \texttt{stderr} to \texttt{stdout} first, and
afterwards redirect \texttt{stdout}, \texttt{stderr} writes to where
\texttt{stdout} wrote to before you redirected it.

To finish this, we'll give you a little shortcut: If you don't want to
care about this first and second thing and consider ``\texttt{>
filename 2>\&1}" an awful lot to type, you can use the short version:
``\texttt{\&> filename}". The above line then becomes:
\begin{verbatim}
$ ls file1 nothere &> twostreams
\end{verbatim}
This is much more convenient, but introduces
a new caveat: Not all shells are the same and this shortcut is
something that works in \texttt{bash} but does not necessarily do so
in other shells you may encounter. Just keep this in mind if you run
into trouble ;)

In \ref{lookingatcontent} when we introduced you to \texttt{cat} we
mentioned that you can use it to combine contents of multiple files
into a single one. Now you have all the necessary tools; play around
with \texttt{cat} and I/O-redirection :)

% end Creating Content

\section{Archiving and Compression}
\label{archivingandcompression}
Sometimes it is useful to put files into a collection called an
\emph{archive}. An example would be if you want to send multiple
files via email, but you don't want to append 10 files, but just one
-- this would be an archive. 

Another thing that is useful in many cases is \emph{compression}.
Compressing data means to encode it in a way that uses less bits than
the original data. This is possible by using algorithms which make use
of redundant data -- most data has such redundant elements because it
is in certain file formats which demand structure, i.e. patterns.

\subsection{Archiving with \texttt{tar}}
\label{archivingwithtar}
To create an archive of multiple files on (GNU/)Linux and many other
UNIX-like systems you can use the \texttt{tar(1)} ``tape archive"
utility. You can tell by its name that this program is pretty old, as
it was created to work with tapes.

\texttt{tar} has an incredible amount of functionality (have a look at
the manpage) and we will cover only the most basic stuff here. To
create an archive you can use:
\begin{verbatim}
$ tar -cvf myfirstarchive.tar file1 file2 hugefile
file1
file2
hugefile
\end{verbatim}
The explanation of the options is quite straightforward:
\begin{description}
  \item [\texttt{-c | --create}] tells \texttt{tar} to create an
    archive.  
  \item [\texttt{-v | --verbose}] tells \texttt{tar} to be verbose,
    i.e. to print each file it processes to \texttt{stdout}.
  \item [\texttt{-f | --file=ARCHIVE}] tells \texttt{tar} how it
    should call the archive. It is conventional to add the
    \texttt{.tar} extension to \texttt{tar} archives, though not
    strictly necessary. However, do yourself a favour and use this
    extension ;)
\end{description}
After you told \texttt{tar} what you want and where to put
it\footnote{I.e. the name of the archive. This can be an absolute or a
relative path.} you have to put all files you want to be in the
archive on the command-line. If you pass a directory to \texttt{tar}
it will automatically put all files in the directory in the archive as
well, so you don't have to tell it explicitely to do that like you had
to with \texttt{cp} or \texttt{rm}.

To check that everything worked correctly, you can check which files
are in an archive by using the \texttt{-t | --list} option:
\begin{verbatim}
$ tar -tf myfirstarchive.tar
file1
file2
hugefile
\end{verbatim}
This works essentially like \texttt{ls} but for archives. 

But if you received an archive, the relevant thing you want to do is
not just to list the files but to \emph{extract} them on your system.
To try this, create a directory, change into it and extract the files
there using the \texttt{-x | --extract | --get} option:
\begin{verbatim}
$ mkdir tar-dir
$ cd tar-dir
$ ls
$ tar -xvf ../myfirstarchive.tar
file1
file2
hugefile
$ ls
file1  file2  hugefile
\end{verbatim}
It is always a good idea to look into an archive you downloaded
from the Internet before extracting it, to make sure that it really
contains the files you want and not something else.

% end Archiving with tar

\subsection{Compression Utilities}
While archiving today is done mostly with \texttt{tar}, there are
several utilities to compress files. Each of them uses a different
algorithm to compress the data, but their usage is very similar. The
only thing you should know about the algorithms used is how efficient
they are in order to decide which one will be most appropriate for the
problem you are facing.

One important fact you should know before starting to use compression
is the following: Don't compress data which is already in a compressed
format! If you do this you'll end up with a file that is \emph{bigger}
than the original one. Why is this? Well, as pointed out above,
compression utilizes redundancies in data to store the same
information in less bits. It also adds a bit of metadata to make
recovering of the original data possible. If you have compressed data
once and your compression algorithm is decent, there will be no
redundant bits left to remove. Thus, if you now apply compression
again, the algorithm will find nothing it can compress, but still adds
its metadata, meaning you have more data than before. 

For this reason, it is a good idea to check if the format of the files
you want to compress already uses compression -- well known examples
are \texttt{.pdf}-files, \texttt{.odt}-files, and even
\texttt{.docx}-files. You can try this out for yourself after you've
learned how to use the utilities.

\subsubsection{\texttt{gzip(1)}}
The first tool we're going to look at is \texttt{gzip(1)}. Its basic
syntax is:
\begin{verbatim}
$ gzip FILENAME...
\end{verbatim}
By default \texttt{gzip} will do the following to each file you pass
to it:
\begin{itemize}
  \item Compress the data and store it in a file with the same name as 
    the original plus the \texttt{.gz} extension.
  \item Remove the original file.
\end{itemize}
This especially the second point is important to keep in mind -- if
you want to keep your old file(s), use the \texttt{-k | --keep} flag.

To decompress a compressed file you have two options:
\begin{enumerate}
  \item Use the \texttt{-d | --decompress | --uncompress} option.
  \item Use the \texttt{gunzip} command which is more or less just a
    shortcut of the above.
\end{enumerate}
By default this restores the uncompressed file and deletes the
compressed one afterwards -- again you can use \texttt{-k | --keep}
flag to change this behaviour.

However, sometimes you just want to see what's in a compressed file
without storing the decompressed version. To do this, you have two
options again:
\begin{enumerate}
  \item Use the \texttt{-c | --stdout | --to-stdout} flag which little
    surprisingly just prints the contents of the file to
    \texttt{stdout}.
  \item Use the \texttt{zcat} utility which is again just a shortcut
    for the above.
\end{enumerate}
Demo time!\footnote{The \texttt{-h | --human-readable} option to
\texttt{ls(1)} makes the size output more readable for as by adjusting
the units; e.g. 1024 byts will be shown as 1K instead to indicate that
this is one KB.}
\begin{verbatim}
$ ls -lh hugefile
-rw-r--r-- 1 me me 431888 Sep 18 17:52 hugefile
$ gzip hugefile
$ ls hugefile
ls: cannot access 'hugefile': No such file or directory
$ ls -lh hugefile.gz
-rw-r--r-- 1 me me 165943 Sep 18 17:52 hugefile.gz
$ gunzip hugefile.gz
$ ls hugefile.gz
ls: cannot access 'hugefile.gz': No such file or directory
$ ls -lh hugefile 
-rw-r--r-- 1 me me 431888 Sep 18 17:52 hugefile
\end{verbatim}
You can see that the original file was compressed to less than half of
its size! However, you can do even better: Compressing data takes time
and time is often precious. The better the compression the longer it
takes, which can be a problem. Therefore \texttt{gzip} provides an
option to adjust the speed vs. compression quality. You can pass it a
number between 1 and 9 preceeded by a dash to indicate you want the
program to be really fast (\texttt{-1}) or compress really good
(\texttt{-9}). The default level is \texttt{-6}. Have a look:
\begin{verbatim}
$ ls -l hugefile
-rw-r--r-- 1 me me 431888 Sep 18 17:52 hugefile
$ gzip -1 hugefile
$ ls -l hugefile.gz
-rw-r--r-- 1 me me 195743 Sep 18 17:52 hugefile.gz
$ gunzip hugefile.gz
$ gzip -9 hugefile
$ ls -l hugefile.gz
-rw-r--r-- 1 me me 165202 Sep 18 17:52 hugefile.gz
\end{verbatim}
So the best compression produces a file which is about 15\%{} smaller
than what the worst compression creates. This may seem minor to you,
but think about data center which have to store hundreds of thousands
of terra bytes of data where this can make an enormous difference.

And now a quick demonstration of \texttt{zcat}:
\begin{verbatim}
$ cat file1
A file with copiable content
$ gzip file1
$ ls file1.gz
file1.gz
$ zcat file1.gz
A file with copiable content
$ ls file1.gz
file1.gz
\end{verbatim}
You can see that in this case the compressed file stays intact and
just its content is output to \texttt{stdout}.

Of the compression utilities we're going to discuss here,
\texttt{gzip} is the ``worst" one in terms of compression, but it is
still widely used and ``worst" in this case means still very good --
and fast!
% end gzip(1)

\subsubsection{\texttt{bzip2(1)}}
\label{bzip2(1)}
The \texttt{bzip2(1)} command has pretty much the same syntax and
default behaviour as \texttt{gzip}, and there are a \texttt{bunzip2}
as well as a \texttt{bzcat} utilities. Compressed files get the
\texttt{.bz2} by default.

However, not all things are totally identical, so read the man page of
\texttt{bzip2} and play around a bit to compare the compression rates
of \texttt{bzip2} and \texttt{gzip}. In general, \texttt{bzip2} offers
better rates than \texttt{gzip} but worse than the following utility.

% end bzip2(1)

\subsubsection{\texttt{xz(1)}}
\label{xz(1)}
Again, syntax and behaviour are very similar to the above and there
are \texttt{unxz} and \texttt{xzcat} utilities offering the same
functionality as with the other two programs. By default compressed
files get the \texttt{.xz} extension.

This is the best and most versatile compression tool of the ones we
discuss which will become apparent if you read its much longer man
page and see what options it offers. 

Again, play around to get a feeling for the program.

% end xz(1)

\subsection{Combining Archiving and Compressing Data}
\label{combiningarchivingandcompressingdata}
You've seen that you can put multiple files into an archive which is a
single file and you know that you can compress files -- wouldn't it be
convenient to combine these features, i.e. to create an archive with
all the files you want and then compress the whole archive in one
step? The good news is: You can. The better news is: \texttt{tar} has
options to combine this into one single step.

\begin{description}
  \item [\texttt{-z | --gzip}] What this option does depends on the
    arguments you pass to \texttt{tar}. If you have given the
    \texttt{-c} option, \texttt{tar} will first create an archive and
    then compress the whole thing in one step:
\begin{verbatim}
$ tar -cvzf files.tgz file1 file2 file3
file1
file2
file3
$ ls files.tgz
files.tgz
$ tar -xvzf files.tgz
file1
file2
file3
\end{verbatim}
    Conventionally \texttt{gzip} compressed \texttt{tar}-archives use
    \texttt{.tar.gz} or less verbose \texttt{.tgz} as extension.
  \item [\texttt{-j | --bzip2}] This compresses/decompresses a
    \texttt{tar} archive using the \texttt{bzip2} utility. Such
    archives usually get the \texttt{.tbz} or \texttt{.tbz2}
    extension. Again you can also use \texttt{.tar.bz2} if you want to
    be more verbose.
  \item [\texttt{-J | --xz}] To your surprise, this will create a
    \texttt{xz}-compressed archive. The conventional extension is
    \texttt{.txz} in this case.
\end{description}
Note that if you use compression with \texttt{tar} the original files
stay intact; just like when you use \texttt{tar} to create an archive.

% end Combining Archiving and Compressing Data

% end Archiving and Compression

% end Working with Content

\section{Textfiles, Pipes, Wildcards, and a Little Bit of Regex}
\label{pipeswildcardsandalittlebitofregex}
Here we enter the last big section about working with files and their
content on the command-line, so take a deep breath and get ready to
learn some incredibly useful concepts and tools!

All of them are mainly concerned with working with textfiles. You've
already encountered \texttt{echo}, \texttt{cat}, \texttt{less}, and
\texttt{wc}, but we'll add many more and find new applications for
some of them as well. 

\subsection{Of Heads and Tails}
\label{ofheadsandtails}
Sometimes you want to look only at the beginning or the end of a file,
e.g. if you do some C programming you may just want to see the
\texttt{\#includes} and comments at the top of a source file, or you've
written a \LaTeX{} file and forgot which packages you included, but
you want them in a new project, so again, you only want the first few
lines. To do this, there is the \texttt{head(1)} utility, whose basic
syntax is:
\begin{verbatim}
$ head [OPTION]... [FILE]...
\end{verbatim}
It prints the ``head" of a file to \texttt{stdout}, which by default
means the first 10 lines of each file you passed to it as an
argument. However, 10 is sometimes not enough (or too much), so you
can easily control the number of lines being printed with the
\texttt{-n | lines=NUM} option:
\begin{verbatim}
$ head linefile
Line 01
Line 02
Line 03
Line 04
Line 05
Line 06
Line 07
Line 08
Line 09
Line 10
$ head -n 5 linefile
Line 01
Line 02
Line 03
Line 04
Line 05
$ head -5 linefile
Line 01
Line 02
Line 03
Line 04
Line 05
\end{verbatim}
As you can see, you can leave out the \texttt{-n} and write the
desired number directly after the dash.

On the other hand, you may have added something to a file and don't
remember what it was, or if you are running a web server or anything
else that creates logs it is often useful to just look at the last
lines of a textfile. The utility of choice in this case is
\texttt{tail(1)}. Its syntax is equivalent to the one of
\texttt{head} and by default it prints the last 10 lines of a file to
\texttt{stdout}.

There is one common usecase for \texttt{tail}, however, which
is not commonly found with \texttt{head}: We mentioned programs
creating log files, i.e. files which log their activity. These files
increase through time as the programs append more and more data to the
end of the logs. If you want/have to monitor such programs, it is
often useful to watch the logs while growing, but of course you only
want to see the new entries at the end of the file. This can be done
using \texttt{tail} with the \texttt{-f | --follow} option. To test
this, do the following:
\begin{enumerate}
  \item Type \texttt{tail -f linefile}.
  \item Open a second terminal window  and change to the 
    \texttt{unixcourse} directory.
  \item Append a new line to \texttt{linefile}:
\begin{verbatim}
$ echo "newline" >> linefile
\end{verbatim}
  \item In the window which runs \texttt{tail -f linefile} you will
    see that a new line has been added.
\end{enumerate}
Now try what happens, if you  type `\texttt{>}' instead of 
``\texttt{>>}". 

To stop \texttt{tail} from running, go to the terminal window it runs
in and press \texttt{CTRL-C}. This will terminate the process. We'll
explain this to you in \ref{processes}, don't worry.

% end Of Heads and Tails

\subsection{Sorting Text and Using Pipes}
\label{sortingstextandusingpipes}
You'll encounter many situations where you want to sort the content of
a file -- e.g. a list of words you want to sort alphabetically or some
numbers representing measurements and you want to quickly check their
range. This is what the very fittingly named \texttt{sort(1)} program
is good for:
\begin{verbatim}
$ cat sortfile
T
B
A
W
c
N
B
I
P
$ sort sortfile
A
B
B
c
I
N
P
T
W
\end{verbatim}
That was easy enough! And now for numbers:
\begin{verbatim}
$ cat numericsort
9
253
85
76
0010
1000
$ sort numericsort
0010
1000
123
253
76
85
9
\end{verbatim}
What?! Most likely this  did not correspond to your expectations. How
can \texttt{sort} think that 1000 is less than 9?? Of course,
\texttt{sort} is not that stupid. The problem is caused by different 
expectations: \texttt{sort} sorts text alphabetically and numbers are
just part of the alphabet: They come before all letters and there are
exactly 10 of them: 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9. Thus,
\texttt{1000} and \texttt{9} are just words -- \texttt{1000} starts
with \texttt{1} which precedes \texttt{9} and must thus be placed
first, like \texttt{abcd} precedes \texttt{z}. That is what
alphabetically sorted means: you order everything according to their
first letter, and only then you look at the second one to order those
with the same first letter further, etc. 

``OK", you may say, ``this may be an answer, but it is still pretty
annoying." Correct. And because this problem is very common,
\texttt{sort} offers the \texttt{-n | --numeric-sort} option which
tells it to sort the lines of a file numerically:
\begin{verbatim}
$ sort -n numericsort
9
0010
76
85
123
253
1000
\end{verbatim}
Much better! 

In the above output, which sorted the lines of \texttt{sortfile} 
alphabetically the letter \texttt{B} appeared twice. This is all nice
and fine in many situation, but sometimes you want only unique the
unique entries in a file, e.g. if you want to check how many different
measurements you had or how many different users accessed your web
server. If you do some research, you'll find that this is what the
\texttt{uniq(1)} program thus. If you read its man page you'll find
the following:
\begin{quotation}
Note:  `uniq'  does  not  detect  repeated lines unless they are
adjacent.
\end{quotation}
What this essentially means is, that you have to be sure that all
duplicate lines in a file are adjunct before passing the file to
\texttt{uniq}. One possibility to achieve this is to sort the file
first. Using what you've just learned and I/O redirection, you can do
the following:
\begin{verbatim}
$ sort sortfile > tmp
$ uniq tmp
A
B
c
I
N
P
T
W
\end{verbatim}
Great, there is only one line now starting with \texttt{B}. However,
this workflow is rather clumsy. And worse: Imagine the file whose
unique entries you want has 10 or even 100 GB -- this approach forces
you to create another file of the same size just for this little task.

Surely, there must be more intelligent way to do this! There is:
Pipes.\footnote{For those of you eagerly reading man pages who have
already found the \texttt{-u | --unique} option for \texttt{sort}:
Yes, this works perfectly fine, but for now, pretend you have no idea
;)} 

A pipe is a unidirectional data channel which allows you to ``pipe"
the output of one program as input to another program, i.e.
\texttt{stdout} of one program will be sent to \texttt{stdin} of
another program.\footnote{Actually, pipes can do more than just this,
but for our purposes, this is totally sufficient.}

To pipe the output of one program to another one, you use the
vertical bar `\texttt{|}':
\begin{verbatim}
$ sort sortfile | uniq
A
B
c
I
N
P
T
W
\end{verbatim}
It does not look like much, but this is one of the most powerfull
abilities the command-line offers because you can combine arbitrarily
many commands in this manner, each one changing the output and passing
it to the next. Try to find out what the following line does and what
output you expect before running it:
\begin{verbatim}
$ sort sortfile | uniq | head -n 5 | tail -n 3
\end{verbatim}
Or what about this:
\begin{verbatim}
$ head -n 200 hugefile | less
\end{verbatim}
You can do an incredible lot of things with piping and we're going to
encounter more usecases soon. 

Note that you pipe \texttt{stdout} to \texttt{stdin}, leaving
\texttt{stderr} alone. This means that if something in your pipe goes
wrong, you'll see the error messages printed to the screen:
\begin{verbatim}
$ ls file1 file2 file3 file4 | head -2
ls: cannot access 'file4': No such file or directory
file1
file2
\end{verbatim}

% end Sorting Text and Using Pipes

\subsection{Wild Cards and Gentle Regexes}
\label{wildcardsandgentleregexes}
Until now you passed each argument you wanted a command to process
directly to it, only using tab completion to increase your speed.

But, what if you have a directory with hundreds of files and you don't
even know all of their names, but you do know that all of them contain
some string or have a common number in their name? In this case you
may want to use \emph{wildcards}. Wildcards are characters the shell
treats as special and which enable you to look for patterns. They are:
\begin{description}
  \item [\texttt{*}] The asterisk matches any string, including the
    null string, i.e. the empty string. 
  \item [\texttt{?}] The question mark matches any single character.
  \item [\texttt{[...]}] This pattern matches any one of the enclosed
    characters. There are several things to consider here:
    \begin{itemize}
      \item Special characters loose there special meaning within
        brackets.
      \item If the first character is `\texttt{\^{}}' the meaning of
        the brackets change to  ``match any characters \emph{not} in
        the given set", e.g. \texttt{[\^{}ab]} matches anything except
        for `\texttt{a}' and `\texttt{b}'. 
      \item You can use ranges within brackets, e.g. \texttt{[0-9]} 
        matches any number between 0 and 9. However, in one case those
        ranges may work not as you expect: if you use the ranges
        \texttt{[a-z]} or \texttt{[A-Z]} you may (reasonably) expect,
        that the first matches all lower case characters, while the
        latter matches all upper case character. Unfortunately, this
        is completely wrong: The alphabet the \texttt{bash} uses is:
\begin{verbatim}
aAbBcCdDeEfFgGhHiIjJkKlLmMnNoOpPqQrRsStTuUvVwWxXyYzZ
\end{verbatim}
       Thus, \texttt{[a-z]} matches any letter, lower or upper
       case, except for `\texttt{Z}' and \texttt{[A-Z]} matches
       any letter, lower or upper case, except for `\texttt{a}'.
      \item If you want to match a special character in your bracket
        expression, there are some simple rules:
      \begin{itemize} 
        \item To match a `\texttt{\^{}}', just put it somewhere else
          than the start, e.g. \texttt{[a\^{}]} matches `\texttt{a}'
          or `\texttt{\^{}}'.
        \item To match a `\texttt{-}' put it as last or first
          character, e.g. \texttt{[ab-]}.
        \item To match a closing bracket, put it at the first position
          (or after the caret if you want to negate it).
      \end{itemize}
    \end{itemize}
\end{description}
But what does that mean? Let's look at some examples:
\begin{verbatim}
$ ls *
 contentdir         file1      linefile        sedfile
 dir1               file2      lsfile          sedfile.bak
 dir2               file3      numericsort     sortfile
 dir3               grepfile   onelinefile     tablefile
'dir with spaces'   hugefile   renamedfile
 echofile           join1      rmdirectory
 errorfile          join2      rminteractive
$ ls *file
echofile   hugefile  onelinefile  sortfile
errorfile  linefile  renamedfile  tablefile
grepfile   lsfile    sedfile
$ ls file*
file1 file10  file2  file3
$ ls *file*
echofile   file2        hugefile     renamedfile  tablefile
errorfile  file3        linefile     sedfile
file1      grepfile     lsfile       sedfile.bak
file10     hasfileinit  onelinefile  sortfile
$ ls file?
file1  file2  file3
$ ls [flsn]*
file1   file2  linefile  numericsort  sedfile.bak
file10  file3  lsfile    sedfile      sortfile
\end{verbatim}
Try to explain the output yourself and only read on afterwards!

The first example just passes a `\texttt{*}' to \texttt{ls} and as
this matches any pattern, all files in the directory are listed. The
second example matches all files whose name ends in \texttt{file}. The
third example matches all those whose name begins with \texttt{file}.
The fourth example matches any file whose name contains the string
\texttt{file} somewhere. The fifth example matches any file whose name
starts with \texttt{file} followed by exactly one additional
character. The last example matches any file whose name starts with
\texttt{f}, \texttt{l}, \texttt{s}, or \texttt{n}.

Wildcards can be very handy to match only certain files but not
others. However, they are restricted to the command-line and still
very coarse grained filters. What if you want to check if a pattern
exists e.g. at the end of lines in a file? This can be achieved using
a \emph{regular expression}, or regex for short.

\subsubsection{Regular Expressions and \texttt{grep(1)}}
\label{regularexpressionsandgrep}
First, let's shortly introduce you to the \texttt{grep(1)} utility.
This is the program of choice if you want to ``grab" some text
patterns in a file. Using \texttt{less} and \texttt{cat} you know how
to look at everything that's in a file and in combination with
\texttt{sort}, \texttt{uniq}, \texttt{head}, and \texttt{tail} you've
learned to remove some information, but until now you can't look for
some specific patterns in a file, e.g. is there a measurement of
exactly ``3.14159265358979" or a sentence which contains the word
``octothorpe" in the file?

With \texttt{grep} you're able to do this. Its basic syntax is:
\begin{verbatim}
grep [OPTION]... PATTERN [FILE]...
\end{verbatim}
and the program looks for \texttt{PATTERN} in each file you pass to
it. So, let's see if this measurement we talked about above is found:
\begin{verbatim}
$ grep "3.14159265358979" grepfile
$ grep "octothorpe" grepfile
but only here you'll find a wild octothorpe "#"
and for the curious: hash == hashtag == octothorpe
\end{verbatim}
Now you know that there is no line containing the beginning of pi in
this file, but two contain the word ``octothorpe". If you also want to
know which line(s) contain matching patterns, use the \texttt{-n |
--line-number} option to \texttt{grep}.

Very well, but what about this regex stuff? The pattern you pass to
\texttt{grep} can be a regex. Using regexes is similar to using
wildcards, but it is \emph{much} more powerful. To get the official
definition for POSIX\footnote{``Portable Operating System Interface
X". The `X' was appended because it was intended for UNIX-like OSs.}
regular expressions you can read \texttt{regex(7)}, but we'll do it a
bit less formal. 

As with wildcards, certain characters in regexes do not have their
literal meaning. These are called \emph{metacharacters}. These
metacharacters are:
\begin{center}
\begin{longtabu} to \linewidth { l X }
  \texttt{\^{}} & The caret matches the beginning of a line.\\
  \texttt{\${}} & The dollar sign matches the end of a line.\\
  \texttt{.} & The dot matches any single character (similar to
    the `\texttt{?}' wildcard).\\
  \texttt{[ ]} & As with wildcards, a bracket-expression
    matches any single character within the brackets. Note that ranges
    in regexes work as you expect, i.e. \texttt{[a-z]} matches all
    lowercase letters. The caret, dash, and closing bracket can be
    included the same way as with wildcards.\\
  \texttt{[\^{} ]} & A bracket-expression starting with a caret
    matches any single character \emph{not} found within the brackets,
    \texttt{[\^{}abc]} matches anything except \texttt{a}, \texttt{b},
    and \texttt{c}.\\
  \texttt{*} & The asterisk in a regular expression matches the
    preceding element 0 or more times. Note that this is different
    from the meaning of the asterisk as a wildcard! Using wildcards,
    the expression \texttt{b*} matches everything that starts with the
    letter `b', but as a regular expression it matches 0 or more
    `b's!
\end{longtabu}
\end{center}
With these new knowledge, try the following using \texttt{grep} with
\texttt{grepfile}:
\begin{itemize}
  \item Print all lines which contain a `\texttt{\#{}}'.
  \item Print all lines which begin with a `\texttt{\#{}}'.
  \item Print all lines which end in a punctuation mark.
  \item Print all lines which end in `Z' or `z'.
  \item Print all empty lines.
  \item Print all lines which contain no `a'.
\end{itemize}
Check you results by looking at \texttt{grepfile} with \texttt{cat} or
\texttt{less}.  If you can solve all of them: Great! If not: Don't
despair, it takes time to get used to the concept of regexes. 

There is also a very useful component to \texttt{grep} and regexes if
you like to play crossword puzzles: On many distros, you can find at
least one word list of the language your system runs on in
\texttt{/usr/share/dict}. The most common one is just called ``words".
On my system it contains 102401 words. Suppose now that you are
looking for a mountain range consisting of 12 letters, the second one
is a `p', and there fifth, fourth, and third to last letters are
``hia":
\begin{verbatim}
$ grep '^.p.....hia..$' /usr/share/dict/words
Appalachia's
Appalachians
\end{verbatim}
As an aside: If you ever want to look for one of the metacharacters as
a literal, e.g. you want to know which lines contain a `\${}' you have
to escape them with a backslash, i.e. you have to type
``\textbackslash\${}" in your pattern.

However, regexes don't end here. There are some more metacharacters
than those we've see so far and we'll cover them now:
\begin{longtabu} to \linewidth { l X }
  \texttt{( )} & The expression between the parentheses is seen as a
    single element from the outside, e.g. \texttt{(abc)*} matches 0 or
    more occurences of the string ``abc", e.g. it matches ``abc",
    ``abcabc".\\
  \texttt{\textbackslash{}\emph{n}} & Matches the
    $n$\textsuperscript{th} expression delimited by parenthesis, e.g.
    \texttt{(ab)(cd)..\textbackslash{}2\textbackslash{}1} matches any
    string that starts with ``abcd" followed by two characters
    followed by ``cd" followed by ``ab".\\
  \texttt{\{}\emph{m}\texttt{,}\emph{n}\texttt{\}} & Matches any 
    string that contains the preceding expression at least \emph{m}
    and at most \texttt{n} times, e.g. \texttt{q\{2,5\}} matches 2, 3,
    4, or 5 consecutive `q's. \newline 
    The version of \texttt{grep} that is installed on most (GNU/)Linux
    distros also allows some variations of this scheme: If you pass
    only one number \emph{n} without a comma, the expression matches
    occurences of exactly \emph{n} times the same character, e.g.
    \texttt{d\{3\}} matches exactly ``ddd". You can also leave out one
    of the numbers but provide a comma to give just a lower or upper
    bound, e.g. \texttt{i\{,10\}} indicates ``up to 10 consecutive
    `i's" while \texttt{i\{10,\}} reads as ``at least 10 consecutive
    `i's".\\
  \texttt{?} & Matches the preceding expression zero or one times,
    e.g. \texttt{friends?} matches ``friend" or ``friends". Like with
    the asterisk, this differs from the question mark if it is used as
    a wildcard! The wildcard `\texttt{?}' is the same as the regex
    `\texttt{.}'.\\
  \texttt{+} & The plus sign matches one or more occurences of the
    preceding expression, e.g. \texttt{w+} matches `w', ``ww", ``www",
    etc. The difference to the asterisk is that the plus sign does
    \emph{not} match zero occurences.\\
  \texttt{|} & The vertical bar matches either the preceding or the
    following expression, e.g. \texttt{ab|Bc} matches ``abc" and
    ``aBc". Of course, this could also be done by \texttt{a[bB]c}, so
    why care about `\texttt{|}'? Well, if you combine it with
    parentheses or braces, it becomes clear: \texttt{(sw)|m|(gr)eet}
    matches ``sweet", ``meet", or ``greet" or
    \texttt{(Ada)|(A\textbackslash{}). Lovelace} matches ``Ada Lovelace"
    and ``A. Lovelace". Or think about something like
    \texttt{a\{2,3\}|e\{1,5\}}.\\
\end{longtabu}
Unfortunately, with these metacharacters things are a bit confusing:
While you had to use a backslash if you wanted the literal meaning of
the metacharacters we talked about before, with this group it is the
other way around: \texttt{grep} (and other tools) interpret them
literally, \emph{unless} you prepend a `\texttt{\textbackslash{}}'! For
example, `\texttt{+}' is interpreted as a plus sign, you have to type
``\texttt{\textbackslash{}+}" to get the regex meaning. This can be
pretty annoying, especially as this behaviour is not consistent: Some
tools treat all metacharacters as metacharacters. And to make things
really funny: If you pass the \texttt{-E | --extended-regexp} to
\texttt{grep}, it will also interpret all metacharacters as
metacharacters! 

However, this will become less confusing as soon as you start using
regexes regularly. Try to do the following, and play around with the
\texttt{-E} switch:
\begin{itemize}
  \item Print all lines which contain at least on `Z' or `z'.
  \item Print all lines which contain expressions enclosed 
    in parentheses. Then try to print all lines where there are two
    expressions in parentheses which follow directly after each other.
    And finally try to print only those lines where the
    parentheses-expression have some text between them.
  \item Print all lines containing the word ``fox" or the word
    ``bash".
  \item Print all lines which consist exclusively of `a's.
  \item Print all lines which consist exclusively of 3 or 4 `a's.
  \item Print all lines which consist exclusively of at least 3 `a's.
\end{itemize}
Check you results by looking at \texttt{grepfile} with \texttt{cat} or
\texttt{less}.

Now that you know some basic stuff about regexes and know the  basics
of \texttt{grep}, let's move on to two other programs which use
regexes extensively:

\subsubsection{\texttt{sed(1)} and \texttt{awk(1)}}
\label{sed(1)andawk(1)}
These two utilities are much more powerful than those you encountered
before, especially because \texttt{awk} is in fact a full-blown
programming language, but we'll introduce only the very basics.

\texttt{sed} which is an acronym for ``stream editor" is a tool with
which you can edit a stream of textdata and print it to
\texttt{stdout} (or redirect it to a file, of course). The usual way
to invoke \texttt{sed} is:
\begin{verbatim}
sed [OPTION]... -e 'SCRIPT' [FILE]...
\end{verbatim}
where \texttt{SCRIPT} is a command you pass to \texttt{sed} that tells
it what to do. \texttt{sed} will then apply this command to each line
of the input file. Sounds very confusing, but let's work through an
example:
\begin{verbatim}
$ cat sedfile
This text talks about birds. Why? Because we all know that birds are
the best animals. All birds are much more beautiful and intelligent than
any other species (including humans). No one can ever doubt that birds
are superior. And one day, birds will take over the world! Thus, don't
ever mistreat a bird, it will get its revenge!
\end{verbatim}
What utter nonsense! Birds?! Ridiculous! Surely it's moles who'll 
subdue the world in the near future! This text must be adjusted:
\begin{verbatim}
$ sed -e 's/bird\(s\)\?/mole\1/g' sedfile
This text talks about moles. Why? Because we all know that moles are
the best animals. All moles are much more beautiful and intelligent than
any other species (including humans). No one can ever doubt that moles
are superior. And one day, moles will take over the world! Thus, don't
ever mistreat a mole, it will get its revenge!
\end{verbatim}
Much better! But how to dissect the command to \texttt{sed}?
\begin{longtabu} to \linewidth { l X }
  \texttt{s} & This is the command which means ``substitute"
    whose syntax is
    \newline
    \texttt{s/REGEX/REPLACEMENT/FLAGS}
    \newline
    There are other commands, like \texttt{d} for delete or \texttt{a}
    for append something.\\
  {\texttt/} & The slash is used here as a delimiter character.
    This is just conventional and you can use any other character as a
    separator. Note, however, that in case you want your delimiter
    appear in the \texttt{REGEXP} or \texttt{REPLACEMENT} you have to
    escape it using a backslash.\\
  \texttt{bird\textbackslash{}(s\textbackslash{})\textbackslash?} &
    The pattern to match.\footnote{This regex is more complicated than
    it has to be, because we wanted to illustrate grouped expressions.
    Can you give a simpler version that works as well in this
    situation?}\\
  \texttt{mole\textbackslash{}1} & The replacement. The
    \texttt{\textbackslash{}1} refers to the first expression in
    parentheses in the pattern before, so if there is no `s', it will
    be empty and if there is, it will print one.\\
  \texttt{g} & This is a flag meaning ``global". If you don't
    use it, \texttt{sed} only substitues the first match of the regex
    on a line and leaves the rest alone. Just repeat the command above
    but delete the \texttt{g} to see what happens.\\
\end{longtabu}
Phew, that's a lot to congest. But once you get the hang of it,
\texttt{sed} can be immensely useful. 

A word of caution about redirection: Only in rare cases do you want to
see the changes \texttt{sed} made just on the command-line; usually
you want them to be stored somewhere and in many cases you want to
replace the old file with the new content. Beware, the following will
not work in the way you may expect:
\begin{verbatim}
$ sed -e 's/bird\(s\)\?/mole\1/g' sedfile > sedfile
\end{verbatim}
If you do this, \texttt{sedfile} will be empty afterwards. But why?
When you invoke \texttt{sed} and tell it to redirect its output to
\texttt{sedfile} the first thing it does is to open this file for
writing and because you used the greater-than operator it will
truncate the file (i.e. empty it) and only then starts working on the
now empty file. If you used the \texttt{>>} operator, the changed text
would be appended to the current file, but the old content would still
be there -- most likely also not, what you wanted. This leaves you
with two possibilities:
\begin{enumerate}
  \item Redirect the output to a temporary file and use \texttt{mv}
    afterwards. This has the big advantage that you can check your
    changes before making them final.\footnote{It is in general a good
    idea to first output \texttt{sed}'s results to \texttt{stdout} to
    check if the changes work as intended.}
  \item Use \texttt{sed}'s \texttt{-i[SUFFIX] | --in-place[=SUFFIX]}
    option. This tells \texttt{sed} to edit files in place. If you
    supply a \texttt{SUFFIX}, \texttt{sed} will create a backup of
    your original file with this suffix, which you can use to restore
    the old file if something has gone wrong.
\end{enumerate}
\texttt{sed} can also be used to print only certain lines from a file,
regardless of any patterns. This can be done using the \texttt{p}
command and the \texttt{-n} option: If a line doesn't match anything
you specified in a command, by default \texttt{sed} will print this
line unchanged. To suppress this behaviour and show you only those
parts of the file that \texttt{sed} works on is what the \texttt{-n |
--quiet | --silent} option is for. The \texttt{p} command is the
``print" command and can be prepended by a line number, e.g.:
\begin{verbatim}
$ sed -n -e '1p' sedfile
This text talks about birds. Why? Because we all know that birds are
\end{verbatim}
prints only the first line of \texttt{sedfile}. We'll use this in
\ref{cuttingandjoining}.

After \texttt{sed}, we'll have a quick glance at \texttt{awk}. The
\texttt{awk} program is actually an implementation of the AWK
programming language which owes its name to its creators Alfred
\textbf{A}ho, Peter \texttt{W}einberger, and Brian \texttt{K}ernighan,
but after learning it, some people have concluded it must come from
``awkward". We won't go into any details here, just be aware that it
exists and look at the following example:
{\small
\begin{verbatim}
$ awk '{ print "line" NR ": " $0; }' sedfile
line1: This text talks about birds. Why? Because we all know that birds are
line2: the best animals. All birds are much more beautiful and intelligent than
line3: any other species (including humans). No one can ever doubt that birds
line4: are superior. And one day, birds will take over the world! Thus, don't
line5: ever mistreat a bird, it will get its revenge!
\end{verbatim}
}
You can see that this command prepended the word ``line", the line
number, stored in the variable (see \ref{variablesandenvironment})
\texttt{NR} and a space in front of each line. The current input line
is stored in the variable \texttt{\${}0}.

If you are totally confused now: That's OK, \texttt{sed} and
\texttt{awk} are rather advanced tools, but as they can be immensely
useful it is great to know about them and what they do in general, so
you can come back and learn them in-depth if you ever need to. There
are whole books written just about these two programs and other books
that just treat regexes, so there is \emph{a lot} to learn, definitely
more than we can cover in this introducory workshop.

% end Wild Cards and Gentle Regexes

\subsection{Cutting and Joining}
\label{cuttingandjoining}
Before we leave you alone with textfiles, there are two more things to
talk about. The first has to do with cutting stuff up and joining it
together.

You'll often have files which consists of columns and lines,
representing some kind of table. Have a look at \texttt{tablefile}:
\begin{verbatim}
$ cat tablefile
12.9 2.6 9.9 5.9 9.5 7.9 1.2
13.6 9.3 7.8 1.9 6.7 5.7 8.2
13.1 7.4 1.0 3.7 3.0 1.8 13.1
8.4 12.3 13.1 12.8 10.1 7.3 11.9
1.5 9.3 10.4 2.6 9.6 1.1 12.8
5.8 11.6 7.1 5.2 1.7 1.8 9.6
8.7 2.9 11.8 2.7 3.7 5.4 10.6
2.7 2.7 3.6 13.0 4.2 10.8 1.8
9.6 5.9 5.7 9.5 9.5 3.1 11.2
10.9 1.3 3.0 4.9 8.2 7.7 6.6
13.0 12.0 3.6 13.5 7.8 12.4 4.9
1.6 12.5 1.4 6.9 10.2 3.7 3.8
7.3 5.1 11.2 12.5 1.0 10.3 3.1
6.6 12.2 5.4 2.2 13.9 12.5 12.2
8.7 2.4 10.9 6.7 5.9 5.2 11.8
11.4 7.2 1.3 12.2 4.7 3.2 10.7
3.4 7.2 7.4 12.8 4.9 6.4 9.2
3.9 12.7 3.2 3.3 6.7 8.0 8.5
9.5 12.8 6.8 2.7 7.2 2.4 13.7
12.4 3.6 2.1 3.1 10.9 7.7 12.8
10.4 1.2 5.4 2.6 7.0 7.5 8.8
8.9 7.3 3.6 6.3 7.2 3.8 7.4
12.9 12.7 8.2 11.1 2.5 5.3 10.3
11.2 6.4 11.9 7.3 2.6 10.5 3.4
\end{verbatim}
This file consists of 24 lines and each line has 7 colums. This could
e.g. be the output of a program which took measurements for 1 week,
every hour, e.g. the second value in the first column represents the
measurement at 1am (we started at 00:00) on Monday. All nice and fine,
but the file is almost unreadable for a human. This is, where
\texttt{cut(1)} comes in: Its purpose is to cut out columns while
leaving out others and its basic syntax is:
\begin{verbatim}
cut OPTION... [FILE]...
\end{verbatim}
It has some very useful options:
\begin{description}
  \item [\texttt{-d | --delimiter=DELIM}] This tells \texttt{cut} by
    which character fields in the file are separated. In the case
    above they are separated by spaces, but it could be any other
    character as well.
  \item [\texttt{-f | --fields=LIST}] This tells \texttt{cut} which
    fields it should output.
  \item [\texttt{--output-delimiter=STRING}] If you want the output to
    be delimited differently than the input, you can use this option.
\end{description}
Thus, if you want just the the measurements of Wednesday, you can do:
\begin{verbatim}
$ cut -d" " -f3 tablefile
9.9
7.8
1.0
13.1
10.4
7.1
11.8
3.6
5.7
3.0
3.6
1.4
11.2
5.4
10.9
1.3
7.4
3.2
6.8
2.1
5.4
3.6
8.2
11.9
\end{verbatim}
Or what about the files from Tuesday to Thursday:
\begin{verbatim}
$ cut -d" " -f2-4 --output-delimiter="^T|^T" tablefile
2.6     |       9.9     |       5.9
9.3     |       7.8     |       1.9
7.4     |       1.0     |       3.7
12.3    |       13.1    |       12.8
9.3     |       10.4    |       2.6
11.6    |       7.1     |       5.2
2.9     |       11.8    |       2.7
2.7     |       3.6     |       13.0
5.9     |       5.7     |       9.5
1.3     |       3.0     |       4.9
12.0    |       3.6     |       13.5
12.5    |       1.4     |       6.9
5.1     |       11.2    |       12.5
12.2    |       5.4     |       2.2
2.4     |       10.9    |       6.7
7.2     |       1.3     |       12.2
7.2     |       7.4     |       12.8
12.7    |       3.2     |       3.3
12.8    |       6.8     |       2.7
3.6     |       2.1     |       3.1
1.2     |       5.4     |       2.6
7.3     |       3.6     |       6.3
12.7    |       8.2     |       11.1
6.4     |       11.9    |       7.3
\end{verbatim}
This looks much better!\footnote{The \texttt{\^{}T} in the
command-line above means a tab-character. You can reproduce it on the
command-line by typing \texttt{CTRL-V} followed by a tab.} If you want
to see only the measurements taken between 10am and 1pm, you can
combine this with \texttt{sed} and pipes:
\begin{verbatim}
$ sed -n -e '11,14p' tablefile | cut -d" " -f2-4 --output-delimiter="^T|^T"
12.0    |       3.6     |       13.5
12.5    |       1.4     |       6.9
5.1     |       11.2    |       12.5
12.2    |       5.4     |       2.2
\end{verbatim}
First we select the hours we need and then we cut out the columns for
Tuesday, Wednesday, and Thursday.

Another thing you may want to do is joining two files which share a
common field. To achieve this, you can use \texttt{join(1)}:
\begin{verbatim}
$ cat join1
Price ArtNr
10€ 1
20€ 2
100€ 3
400€ 4
$ cat join2
ArtNr Name
1 pendrive
2 cable
3 screen
4 laptop
$ join -1 2 -2 1 join1 join2
ArtNr Price Name
1 10€ pendrive
2 20€ cable
3 100€ screen
4 400€ laptop
\end{verbatim}
So, you tell join that the second field in the first file (\texttt{-1
2}) and the first field in the second file (\texttt{-2 1}) are
identical and it should use them to join the files. There are some
other options and if you're interested play around a bit :)

% end Cutting and Joining

\subsection{Editors (a.k.a. Religions)}
\label{editors(a.k.a.Religions)}
The last step on our journey through textfiles leads us into the
fiercely disputed field of text editors. As the title of this section
suggests, there are people who vehemtly insist that the editor they
use is the only true one and all the others should be forbidden.
It may well be editors about which the first flamewars were fought and
they are fought until today with absolutely no progress or
use.\footnote{You think we're joking? Have a look at 
\href{https://en.wikipedia.org/wiki/Editor_war}{this}.}

Our stance is: Taste differs and it's no good arguing about it, but we
want you to be aware that you may encounter people who'll attack you
just because you editor is not theirs. 

One reason why editors may be such a emotional topic is that the
really powerful ones take a considerable time to get used to and a
lifetime to master. Thus, as soon as you learned one of those you may
be really proud of you on the one side and feel like a total beginner
if you encounter one of the others, which often leads to aversive
reactions. 

There are editor which can need a GUI while others can work directly
within a terminal.

Common GUI editors on (GNU/)Linux systems are:
\begin{description}
  \item [KWrite] The KDE-Desktop's lightweight editor.
  \item [gedit] The GNOME-Desktop's standard editor. If you left all
    the defaults intact when you installed the GUI of you VM at the
    beginning of the course, \texttt{gedit} will be installed on your
    system and just called ``Text Editor".
\end{description}
There are gazillions of others -- if you're interested look around and
try some of them until you find something.

But as this course is about the command-line, we're emphazising on
editors you can use in a terminal.
\begin{description}
  \item [nano] This very small and intuitive editor is now installed
    by default on many (GNU/)Linux systems.
  \item [vi(m)] \texttt{vi} pronounced ``Vee-Eye" and \texttt{vim}
    which stands for ``Vi IMproved" and thus just pronounced ``vim"
    are one of the main editors used by many programmers and sysadmins
    alike. They are incredibly powerful and have tons of features.
  \item [emacs] The archrival to \texttt{vi(m)} for a long time, this
    giant of an editor is often described as ``actually an operating
    system which happens to include a text editor". If you put in the
    effort to learn to use it efficiently, you'll be very productive
    when working with text.
\end{description}
Instead of talking about features of these editors, we'll let you
learn them by their own means. First do the following:
\begin{verbatim}
$ sudo apt install vim emacs
\end{verbatim}
This will install both editors. Now choose which one you want to try
out first.

If it is \texttt{vim}, type:
\begin{verbatim}
$ vimtutor
\end{verbatim}
And you will be in an interactive tutorial, showing you the basics of
\texttt{vim}.

In case you chose emacs, type:
\begin{verbatim}
$ emacs
\end{verbatim}
After the editor started, type \texttt{CTRL-H} followed by \texttt{T}.
The tutorial will start.

We won't say anything else about editors; it is important to know how
to use one of them, but which one doesn't really matter. Try a few,
play around and stick with what you like most :)

% end Editors (a.k.a. Religions)

% end Textfiles, Pipes, Wildcards, and a Little Bit of Regex

\section{Processes}
\label{processes}
After so much textfiles, we start with something completely different.
You have worked a bit with commands by now, and maybe you wondered at
some point, what they are. In very general terms we speak of a program
as a executable binary that can be run on a CPU. Whenever you enter a
command, the corresponding binary is loaded into memory and then each
instruction is executed by the CPU. As soon as a program runs, it is
called a \emph{process}, i.e. a process is a program in execution. To
make the difference a bit clearer: You have only one program found at
\texttt{/bin/ls} but you can run it in different terminals at the same
time, thus creating multiple processes from a single program.

If you want to see the processes currently running on your system the
most widely used tools are \texttt{ps(1)} and \texttt{top(1)}. Their
main difference is that \texttt{top} is interactive while \texttt{ps}
is not. To view all proces that are currently running from the
terminal you are working on, you can do:
\begin{verbatim}
$ ps
  PID TTY          TIME CMD
 2351 pts/4    00:00:00 bash
 4963 pts/4    00:00:00 ps
\end{verbatim}
This tells you that there are just two processes associated with this
terminal: your \texttt{bash} and the \texttt{ps} process which has
already finished as soon as you can read the output. The displayed
fields have the following meaning:
\begin{description}
  \item [\texttt{PID}] This is the ``Process ID". Each process is
    assigned a number that is unique on the currently running system,
    to enable the system (and you) to refer to it unambiguously -- the
    name alone would not be sufficient.
  \item [\texttt{TTY}] This tells you with which terminal\footnote{TTY
    stands for \textbf{t}ele\textbf{ty}pewriter, a rather old acronym.}
    the process is associated. As you are working in a terminal
    emulator and not on a real terminal the ``pts" in the output means
    ``pseudo-terminal".
  \item [\texttt{TIME}] This tells you how many CPU time the process
    has used cumulatively.
  \item [\texttt{CMD}] Rather obviously, this is the command's name.
\end{description}
To see all processes currently running on the system, use:
\begin{verbatim}
$ ps -elf
\end{verbatim}
This creates a listing of all processes (\texttt{-e}) in a long
(\texttt{-l}) and full (\texttt{-f}) format to show a lot of extra
information.\footnote{You'll find the options \texttt{aux} (without a
dash!) doing something very similar in a lot of forums as these are
the traditional options. However, \texttt{-elf} should be used today.}
We don't print it here, as the output is almost always
more than 100 lines.

\texttt{ps} has a myriad of options and tweaks available, so it is
highly recommended to read the man page. Additionally \texttt{ps} is a
very old program and has acquired a lot of legacy-stuff during the
time. Thus, there are many options which do almost-but-not-exactly the
same, there are options with no dashes, one dash, or two dashes, etc.
Therefore, reading the man page in this case also provides you with a
glance of history.

\texttt{top} is also a very old program, but it is still very useful.
If you start it the process will fill your screen and update
periodically. The first will look something like this:
\begin{verbatim}
top - 14:11:33 up  4:48,  6 users,  load average: 0.02, 0.02, 0.00
\end{verbatim}
This tells you the current time, how long the system is running, how
many users are currently logged in and the load average. Note that a
load average of 1 means one CPU is fully loaded, i.e. if your computer
has more than one CPU or more than one core, a load of more than 1 is
no problem. 

By default, processes are sorted according to the \texttt{\%{}CPU}
column, which tells you how much CPU they are using.

To quit the program type \texttt{Q}.

Again it is a good idea to read the man page to get acquainted with
the many things you can do with \texttt{top}.

The processes you've seen so far run for only a very short time, e.g.
the ouput of \texttt{ls} appears immediately and you can issue the
next command. However, in \ref{ofheadsandtails} we already
encountered a program which didn't return to command-line on its own,
i.e. when we used \texttt{tail -f}. While this programs runs until you
tell it to stop, there are other programs which just take a long time
and it may be very annoying if they block your command-line all the
time. Consider starting the Firefox web browser from the command-line:
It would be pretty bad if you had to choose between browsing and
working.

Thus, there are a lot of tools to control processes. First of all,
there are keyboard shortcuts which send signal to processes. Signals
tell a process what to do, e.g. \texttt{SIGTERM} tells the process to
terminate. The keyboard shortcuts you can use are:
\begin{center}
\begin{longtabu} to \linewidth { l X }
  \texttt{CTRL-C} & Send an interrupt signal (\texttt{SIGINT}) to the
    process currently blocking the command-line. Usually, this leads to
    the process being terminated, as you learned in
    \ref{ofheadsandtails} when you stopped the \texttt{tail -f}
    process.\\
  \texttt{CTRL-Z} & Sends a stop signal (\texttt{SIGTSTP}) to a
    process. A stopped process won't react to anything, unless you
    send it a \texttt{SIGCONT} (for ``continue") signal.\\
\end{longtabu}
\end{center}
Some processes just need a long time or even don't have a defined
point in time when they stop, e.g. compiling a large program takes a
long time and your web browser should only stop when you tell it to
do so. How can you use such programs without sacrificing your
command-line? The easiest way is to start them with an ampersand
appended on the command-line, e.g.
\begin{verbatim}
$ firefox &
[1] 5133
$
[1]+  Done                    firefox
$ 
\end{verbatim}
As you can see, Firefox is started, the shell tells you that it has
job number 1 and PID 5133. The ampersand tells the shell to start the
process in the background and the next time you press \texttt{ENTER}
it will tell you that the job has finished. However, if you could e.g.
start a long backup job, put it in the background and the shell will
inform you as soon as its done, even if this takes hours or days
(provided you don't shut down your computer or close the shell during
this time as the job would be killed then).

The job number is also a unique number; not on the whole system, but
just on the current shell. As it is usually much smaller than a PID it
is also easier to remember for humans. To have a look how many jobs
are currently associated with your \texttt{bash}, you can use the
\texttt{jobs} shell-builtin.

\subsection{Process Control}
\label{processcontrol}
Now that you know how to find out which processes are running, either
on the system or within your shell session and you know that you can
start them in the background and are able to send two signals to them
using the keyboard.

However, there is more you can do with processes. The \texttt{bash}
offers some builtin functionality for this:

If you have a job running in the background, but you want it back on
your shell, you can use the \texttt{fg} builtin, e.g.
\begin{verbatim}
$ xclock &
[1] 5257
$ jobs
[1]+  Running                 xclock &
$ fg 1
xclock
^Z
[1]+  Stopped                 xclock
$ bg 1
[1]+ xclock &
$ ps
  PID TTY          TIME CMD
 5130 pts/5    00:00:00 bash
 5257 pts/5    00:00:00 xclock
 5264 pts/5    00:00:00 ps
$ kill 5257
$
[1]+  Terminated              xclock
\end{verbatim}
In this example you start an \texttt{xclock} in the background. Then
you put it into the foreground using \texttt{fg 1} where 1 is the job
ID of the process. After this, you lost access to your shell. The
\texttt{\^{}Z} in the output above indicates pressing \texttt{CTRL-Z},
i.e. sending a \texttt{SIGTSTP} to the process. After this you'll
realise that \texttt{xclock} will no longer interact with you --
the process is frozen. To let it continue in the background, you use
the \texttt{bg} builtin. Afterwards you find out \texttt{xclock}'s PID
and send it a signal using \texttt{kill}.\footnote{This is a bit of a
mess: There is a program called \texttt{kill(1)} which will be
explained if you type ``\texttt{man kill}". However, consider the
following scenario: You have opened so many processes that you are not
allowed to start new ones and some of them are no longer reacting, so
you have to terminate them by sending them signals. However,
\texttt{kill(1)} is a program and if you try to invoke it, this will
be refused as you're not allowed to start any more processes.
Therefore, there is a \texttt{bash}-builtin called \texttt{kill} which
does exactly the same, but as it is just a part of \texttt{bash} it
does not need to start a whole new process and will still be able to
terminate the misbehaving ones. Thus, if you just type \texttt{kill}
on the command-line the \texttt{bash}-builtin will be used and not
the program \texttt{kill(1)}. This has the further advantage that you
can use job IDs instead of PIDs. To tell \texttt{kill} that you're
passing a job ID to it, prepend the number with `\%{}', i.e. in the
above example: \texttt{kill \%{}1}.}

\texttt{kill} is one of the most important tools when doing process
control. Read its man page as well as \texttt{signal(7)} to understand
what it does and how.
% end Processes

\section{Variables and Your Environment}
\label{variablesandenvironment}
This will be covered in the course and added to the script at a later
point.
% end Variables and Your Environment

% end The Command-Line 
\end{document}