10 advanced linux command line tools

Most developers who are working at the command line on a Linux system know the “basic” commands: ls, cd, cat, tail, head, sort, grep, find and others. More “advanced” users will know how to deal with the ‘sed’ and ‘awk’ beasts, or even prefer perl-oneliners. Have the knowledge of bash (scripting) and you find yourself inside a Valhalla where only your imagination is the limit. Well, not really, but at least you get my point, hopefully.

But like everything you do in life, you don’t know about the things that lies after the horizon until you actually explore them. This “yet another top ten list” will dive into some interesting (standard) tools that can make your life much easier when dealing with Linux systems from a programmers perspective.

#10: watch

watch is a beautiful program that executes a program periodically and outputs the contents on a full screen. If you place your command inside quotes, you can even run multiple commands:

watch -n 1 'ls -la ; echo ; vmstat ; echo ; df'

This command executes a file listing, outputs memory statistics and disk space, all separated by empty lines and repeats it every second. How easy is it for you to watch large files gets copied and keep an eye out for disk space issues…

#9: curl

Most (PHP) developers already have met the cURL php extension in their life, but this tool is also available on your command line. Instead of writing an additional php-program just for doing some ‘curly-things’, just use the command line tool. All options you  need are available. Just issue “man curl” and find out about all the possibilities.

#8: logsave

logsave is a very nice tool that captures the output of a program and sends it too a log-file. It’s more or less the same what “tee” does, only it will add a begin time stamp and a end time stamp:

jthijssen@guybrush$ logsave -a output.log echo "hello world"
hello world

jthijssen@guybrush$ cat output.log
Log of echo Hello world
Wed Nov 24 16:51:24 2010

Hello world

Wed Nov 24 16:51:24 2010
----------------

The -a parameter will let you append to the log-file.

#7: lsof

lsof stands for “list open files” and displays all the files that your system has currently opened. It’s very useful to figure out which processes uses a certain file, or to display all the files for a single process:

Sample output of "lsof" for a apache httpd process

#6: strace

Strace is one of my favorites. It traces all system calls that a program makes to the linux kernel. This means you can actually “see” when a program opens, closes, reads, write, access files but off course, there are much more calls that you can trace. It’s like running a program without it’s cover.

strace -ff -e trace=open /usr/sbin/apache2

This will strace the program “apache2” and outputs all the open-calls. Another used version:

strace -ff -p <pid>

traces the currently running process . This means you don't have to restart a program to strace it, but you can hook into the currently running system.

One of the main reasons to strace a process is when something is not going right. Either it’s too slow, doesn’t do what you expect it to do or similar. For instance, a program does not read a configuration file you provide. With strace, you can actually trace all the open calls to see if it can find the file, or maybe it just reads a config-file from a different location. It’s a marvelous piece that has saved me hours and hours of debugging.

#5: z* tools

Occasionally you need to grep, diff or cat files that are compressed. Instead of unpacked files, you can easily threat the files as if they were unpacked already by using “zgrep” instead of grep, “zdiff” instead of diff, “zcat” instead of cat etc..

[root@guybrush /]# echo "hello world" | gzip > test.zip
[root@guybrush /]# file test.zip
test.zip: gzip compressed data, from Unix, last modified: Wed Nov 24 17:02:18 2010
[root@guybrush /]# zcat test.zip
hello world

#4: od

It’s not too long ago I discovered this tool myself! It can dump a file into different kind of formats.

[root@guybrush /]# echo "hello world" > test.txt
[root@guybrush /]# od test.txt
0000000 062550 066154 020157 067567 066162 005144
0000014
[root@guybrush /]# od -t xa test.txt
0000000 6c6c6568 6f77206f 0a646c72
          h   e   l   l   o  sp   w   o   r   l   d  nl
0000014
[root@guybrush /]#  od  -t x1u1a test.txt
0000000 68 65 6c 6c 6f 20 77 6f 72 6c 64 0a
        104 101 108 108 111  32 119 111 114 108 100  10
          h   e   l   l   o  sp   w   o   r   l   d  nl
0000014

#3: iconv

iconv is a great tool for all those times you need to convert a file that is encoded one way to convert to another format. For instance, a latin-1 dump of a MySQL database that needs to be imported into another database that is UTF-8 encoded. Iconv can auto-detect the current format. Together with “od” this tool saved my life recently, when somebody mixed up charset encodings so utf-8 multibyte characters were actually encoded AGAIN as utf-8 characters.

#2: nc

netcat (nc) is the tcp/ip swish army knife. It’s capable of almost everything except washing your car and making coffee, although the last one is disputed by some. I use this a lof for checking service requests (like soap-clients) to see if they actually send out the correct headers.

setup a “listening server”:

nc -l 12345

point your soap-client to send data to http://127.0.0.1:12345 and see the data that is being send from your client.

Another common use, but not so much nowadays, since it’s taken over by scp, is data transferring.

Open a listening connection on a client and output all data to a file:

nc -l 12345 > data.dat

on a second server (where your file resides) transfer the data to the client:

nc server-ip:12345 < data.dat

This will transfer the data to the client. As said: using “scp server-ip:data.dat” is probably a much better idea nowadays.

There is much, much more you can do with netcat. Use it in shell scripts to talk to servers (for instance: smtp or http or IRC servers) directly. Again: the manual is your friend.

#1: whiptail

If you install ubuntu, centos, debian, or basically any linux flavor around that doesn’t spawn a graphical shell, you end up in a kind of textual interface . These blueish screen are perfect for quick browsing through installation options. But did you know it’s just as easy for you to use for your own tools?

Examples:

It’s just so easy to create a nice little display box: Just add the text, the height and width and optionally a title.. I’m sure you recognize the interface during installation or kernel upgrades :)

whiptail  --msgbox "Hello world" 5 60 --title "Test dialog" --backtitle "A simple sample.."

A sample messagebox

Ask a user any question. It will return the status code 1 for “no” and status code 0 for “yes”.

whiptail --yesno "Are you sure you want to continue with deployment?" \
10 60 --title "New deployment" --defaultno

A yesno dialog with the default button set to "no"

Menuboxes are similar to html’s select boxes. Add items and let the user choose between them.

whiptail --menu "Please choose the version to deploy" 15 40 5 \
1.0.0-RC1 1 1.0.0-RC2 2 1.0.0-RC3 3

A menu will let a user pick an item quickly

Gauges are a perfect way of letting the user know the status of whatever you are doing. If your program that you will run in the background can output a percentage, whiptail picks that up and displays the correct percentage on the screen. In the example, I use a simple counter from 0 to 100 with a 1/8 second delay. You will see the progress-bar fill up until it reaches 100%.

for i in `seq 0 100` ;  do  echo $i; sleep 0.125; done | \
whiptail --gauge "Deploying..." 10 50 0

It even got gauges!

There are more kind of dialogs you can create with whiptail, read the man pages and find out just how easy it is to make your tools user friendly. Grab all tags from your svn repository with “svn list”, place them in a menu with “whiptail -menu” and make your life just a little easier..

Conclusion

Linux (of better said: unix) philosophy of “do only one thing, and do it good” works. However, this creates a massive forest of tools where even the most experienced users don’t know what each commando does so don’t despair. Know your tools and the job is half done so poke around in the man-pages, /usr/bin and /usr/sbin directories and find out what other gems are still around for you to discover.