If you are a programmer, you have certainly interacted with a command line. But most programmers just treat the command line as a way to run programs, rather than learning its true capabilities. That was basically me until I invested time to better understand the command line and linux tools. In this post I will explore some easy ways to level up your command line fu. I personally use Bash, so there might be some things specific to Bash, but most of it should be relevant regardless of what terminal you might use.

Basics

Most people are probably familiar with these basics and can skip this section. But for completeness, I’m adding it here.

To navigate the command line, you can use the arrow keys. The left and right arrow keys move the cursor left and right through your current text or command, while up and down cycle through your recent commands (up arrow moves one command older and the down arrow moves one command newer). Feel free to open the command line and try it. Furthermore, the tab key can be used for auto completion. Most commonly, this is used to autocomplete paths, but can be programmed to autocomplete arbitrary programs.

By default, the Bash terminal comes with Emacs key bindings. This means instead of the arrow keys, you can use <C-f>, <C-b>, <C-n>, <C-p> instead of left, right, down, and up. As a Vim user myself, I have configured my terminal to utilize vim keybindings. Like vim, my terminal is modal and I can use h, j, k, and l and other vim movement commands to move around. To set your terminal into vim mode, you can add set -o vito your ~/.bashrc or ~/.bash_profile. That’s all I’m going to say about key bindings without starting an editor flame war.

Fundamentally, your command line runs programs. You are probably familiar with a few, but I will list some common basic ones below:

When you type in a command (such as ls), Bash looks for a file on your computer called ls and runs it. Where does it look? It searches the directories specified in your PATH. In Bash, variables are called environment variables and there are a few special present ones. PATH is one such variable. To read the value of an environment variable (e.g. PATH), you can do echo $PATH. You will see a colon separated list of directories; this is called your PATH. In general, you can set a variable with var=foo (note the lack of space around the =) and access it with $var. Every command that finishes, returns an exit code. 0 means the program finished successfully and a non-zero value means it failed. You can use echo $? to see the exit code of the last command.

What types of programs can Bash run? Really any type of program. The most common (e.g. the ones listed above) are all binary – meaning byte code your computer can understand. However, you can even write your own Python scripts and run them. Try the following (don’t worry about specifics, I’ll go over it all in a bit):

1. Create a file hello_world.py and put the following in it:

#!/usr/bin/env python

print 'hello world'

1. Run chmod +x hello_world.py
2. Run ./hello_world.py

If all goes well, it should run the program and print hello world. What actually happened? In step 1 you created a python source file. Because this is not a binary, you need to tell Bash how to run it. That’s the job of the comment on top known as a “shebang”. Try it. Run /usr/bin/env python and you will be put into a python interpreter. The second step tells your computer to make hello_world.py executable (+x). Finally, in the third step you run the program. Why couldn’t you just run hello_world.py? Because Bash would search your path for it and not find anything. However, when you do ./hello_world.py, Bash does not search your path but rather just runs hello_world.py from your current directory (./ means current directory; ../ means one directory up).

The last point I’ll mention here is configuration. At startup, Bash will run a file (either ~/.bashrc on linux or ~/bash_profile on Mac). In here, you can add any configuration parameters. For example, if you want to add a search directory to your path, you can put PATH="$PATH:/new/dir". Furthermore, Bash has an incredibly powerful alias keyword which literally allows you to alias one command to something more complex. For example, adding alias la=ls -altr will run ls -altr every time you type la at the command line.

History

Bash is able to store all commands you have run ever. In order to enable history run (or add to your .bashrc set -o history). By default every command you’ve run is stored in ~/.bash_history. This is the file that is used when you use the up and down arrow keys to go to previous commands. You can also use C-r to fuzzy search your past history. I, however, am not a huge fan of this default reverse search and instead have installed fzf which has a much nice interface and also brings fuzzy search to other parts of the terminal – I highly encourage you to install it.

Here are a few additional settings I’ve added to my .bashrc to make history work better:

• shopt -s histappend – append new commands to the end of the history file
• export HISTFILESIZE=-1 – unlimited history file size
• export HISTSIZE=130000 – save 130000 lines of history
• export PROMPT_COMMAND='history -a' – append the history after hitting enter rather than at the end of the session
• unset HISTTIMEFORMAT – remove the timestamps from the history file.

One issue I’ve noticed with history is that Bash tends to lose it. I haven’t been able to figure out when or how it does, but I just know that it does. In order to avoid this, I use git to version control my Bash history. Even if you don’t know what git is, I’ll explain my set up below so you can follow it if you wish:

1. mkdir ~/bash_history_git – create the directory
2. git init – initialize git
3. mv ~/.bash_history ~/bash_history_git && ln -s $HOME/bash_history_get/.bash_history $HOME/bash_history – symlink your history file to the version controlled history file
4. git commit -am 'Update history' – commit your history file

Now, every so often, you can use git diff to see if your history has been deleted. If you only see additions, feel free to run git commit -am 'Update history' to commit your history. If you do see deletions, use git add -p to only version control the added lines. Then commit and git checkout Bash history so the deleted lines get restored.

Prompt

Your prompt is probably one of the most important items in the terminal – it’s what you look at before running every command. Like most other things with Bash, you can customize it by adding a line to your config file (.bashrc or .bash_profile). I’m not going to go into detail as to what all customization you can make. Instead, I’ll give you my prompt, and let you use that as a starting point for your own (note that you cannot just copy it as is because it has dependencies; if you want my set up you can view/install it here).

PS1="[Exit: \[\033[1;31m\]\${PIPESTATUS[@]/#0/\[\033[0m\]\[\033[1;32m\]0\[\033[1;31m\]}\[\033[0m\]]\n\T \[\e[0m\](\[\e[33m\]\u\[\e[0m\]:\[\e[32m\]\$(_dir_chomp 20)\[\e[0m\])\$ "

Which might look something like (colors ommitted due to markdown):

[Exit: 0 1] (my-branch)
10:26:57 (user:/some-git-dir)$

A few things to note:

• ${PIPESTATUS[$@]} gives you the exit code of every command you ran previously in a chain of 0 or more pipes. This comes in very handy especially because there are cases where some command in the middle of the pipe might fail and you might not even know about it.
• Showing your git branch in the prompt can come in very handy (and avoid mistaken pushes to master)

There is a pretty awesome ps1 generator here that you can use to make your own prompts.

Input, Output, and Redirection

In Bash, every command you run takes input and produces output. There are two ways (we will discuss) to pass input to a program – either via command line arguments, or from standard in (stdin). A command line argument is anything that comes after the program. For example, if you’ve ever run python foo.py, foo.py is a command line argument passed to the program python. Above when we called chmod, +x and hello_world.py were command line arguments. Most command line arguments come in the form of flags, or named arguments prefixed with - or --. One way to know what a command does and what command line arguments it takes is to use man (manual) which will show a help page, or man page. Getting good at understanding man pages is incredibly helpful. I encourage you to check out the man pages for the commands I listed above – and more. For example, just run man ls.

The other way a command can take input is from stdin. This is a way for the program to take input after it has already begun running and can involve the program prompting the user for input and the user typing it in.

The output from a command that gets printed to the screen is said to go to stdout or stderr. The main difference between the two is that redirection by default happens with stdout, but we will see below how we can also redirect stderr.

Now, the really amazing thing about Bash is the ability to chain the output of one program into the input for another – in other words you can pipe stdout to stdin and keep doing this till you get what you desire. Let’s look at some examples.

| is the pipe character which is used to chain commands together as mentioned previously. Grep is a command used to search stdin for a string. So, you can run ls -l | grep foo to print out the ls info for any file or directory which has foo in the name. Try it out! You can get even crazier – by tacking on awk '{ print $NF }' (which will output only the last column of the input), you only print the filenames: ls -l | grep foo | awk '{ print $NF }'. Let’s say you have a log file of IP addresses and you want to count the number of unique IPs in that log file. You can do grep -E '([0-9]{1,3}\.){3}[0-9]{1,3}' logfile | sort -u | wc -l.

When building up complex chains of pipes, it often helps to write it iteratively. Because each pipe creates output and passes it to the next, you can look at the output of each step and figure out how you want to parse it at the next step.

Going through each and every useful command and what it does would take multiple posts to begin talking about. However, below are commands that I find particularly useful that I recommend you become familiar with too (I’m going to list the most common GNU commands however some may have modern equivalents that might be faster or more ergonomic):

Pipes are not the only form of redirection in Bash. > is used to redirect the output the command into a file (by overwriting it). For example echo foo > foo.txt would create (or overwrite) the file foo.txt and put the single line foo in it. >> appends the output; so echo bar >> foo.txt would add the line bar to the end of the file (or create it if it does not exist).

If you have a command that takes in a file name as an argument, you can use <(foo) in place of a file name to pass the result of running foo into the file passed to the original command. This is best demonstrated with an example:

1. Create the following python file:

import sys

fname = sys.argv[1]
print 'Opening {}'.format(fname)
with open(fname) as f:
print f.read()

1. Then in your command line, echo 'foo' > foo.txt
2. python foo.txt should output:

Opening foo.txt
foo

1. python foo.txt <(echo bar) should output:

Opening /dev/fd/63
bar

As you can see, the contents of echo bar were put into a file /dev/fd/63 and then passed into our Python script. Neat, right? This can be especially handy with diff f1 f2 which will print the line by line difference between two files f1 and f2 (which can also be the output of two Bash commands by using the <() redirection).

Before moving on, I’ll quickly mention: if you want to redirect stderr instead, you use 2> and if you want to redirect both, you can use &>. For pipe redirection, it’s a bit trickier (but it’s sufficient to just use this a few times till it’s memorized):

• command1 2>&1 >/dev/null | command2 will pipe stderr but not stdout
• command1 2>&1 | command2 will pipe both

For and While Loops

Like other language, Bash has for and while loops. I’m going to mention them here because for the longest time I forgot the syntax. I’ll provide both the one line and the multiline syntax.

The one line for loop is for foo in $(command); do command1 $foo; command2 $foo; command3; done where commands 1, 2, and 3 will be run serially with $foo set to each line in the output from command. For example, for f in $(find . -name '*.txt' -type f -depth 1); do cat $f; done will print the contents of all .txt files in the current directory.

The most common multiline version is:

for foo in $(command)
do
    command1 $foo
done

One common version of the for loop for iterating over consecutive numbers is for i in $(seq 5 10); do command $i; done which will set $i to consecutive numbers from 5 to 10 inclusive.

I rarely ever use while loops in Bash, but for completeness, I’ll mention it here:

One liner: while command; do command1; done will execute command1 as long as the exit code for command is 0. A common command is to use comparison functions, which in Bash take the form of:

[ $i -lt 4 ] will exit with 0 if $i is less than 4. You should be able to extrapolate the other conditionals (potentially with help from google).

Just like the for loop, the multiline version of while is:

while command
do
    command1
done

Finally, Bash also has conditionals with if. Like while loops, an if will look at the exit code of the program and proceed if it was 0 and not proceed if it was nonzero. The one line syntax for an if statement is if command; then command1; command2; else command3; command4; fi (note the else is optional). The multiline version is:

if command
then
    command1
    command2
else
    command3
    command4
fi

Usually rather than if, however, I use &&. command1 && command2 will only run command2 if command1 finished with a 0 exit status.

Helpful Tools

One tool that made learning Bash easier and more intuitive was scm breeze. It is a series of scripts (for Bash and Zsh) which add convenient aliases (particularly for git workflows) as well as a numbering scheme for accessing files. Check it out and install it! In addition, as mentioned earlier, I recommend fzf for better history fuzzy search.

If you work on a remote machine (e.g. a dev server in the cloud), you might use ssh to access it. One big pain point with ssh is that if your internet connection drops for whatever reason, you’ll get a broken pipe and have to reconnect. Worse, if you aren’t using something like tmux or screen, your session and any processes open in that session will be lost. In order to remedy the situation, you can learn to install and use Eternal Terminal (or mosh) which provide long lived ssh connections. Now you no longer have to login when your machine disconnects from the internet. Check it out!

Tmux + iTerm

If you use Mac OS X, I’d recommend using iTerm2, a better alternative to the builtin Mac terminal. The most notable feature it provides is native tmux support. If you ssh into a remote machine and then run tmux -CC on that machine, you will get a native iTerm2 window as your tmux session. What this means is you can create iTerm tabs and splits, while under the hood it is creating the tmux equivalent. In other words, you can use intuitive key bindings rather than learning tmux key bindings and get all the benefits of tmux. This plus the long lived SSH connection mentioned above were game changers for my remote workflow.

Final Thoughts

Bash is such a powerful tool, it is almost impossible to fully cover it in one post. I am still discovering new features, one liners, and hacks every day. It takes time and discipline to learn a new tool, but Bash is one of those tools where you will see almost immediate payoff as you improve. Below are the few tips and resources you can use to get better:

• Command Line Challenge – a series of puzzles that can be solved with one line Bash commands
• man bash – The manpage for Bash itself. It might seem tedious, but in the end, it’s well worth perusing this. You never know what new thing you might learn
• If you catch yourself writing a Python script to manipulate text in files, there is probably an easier way to do it with a one liner in Bash – take the time to google it and figure that one liner out. The first few times it might take you longer, but the next n times you need to perform the task, you’ll simply be able to search back in your history and refer back to the Bash one liner (or recreate it from muscle memory).