AtomLabs

Awk – Command line output less awkward

Webmaster — Sun, 26 Dec 2010 01:16:11 +0000

One very powerful tool in Linux is the (awk) text processing language. I always remember it because it makes long log files or command output, come back to you a little less (awk)ward. Yeah corny I know, but let’s look at what it can do and maybe you’ll agree:

I want to find what’s happening on my site:

cat /usr/local/apache/domlogs/atomlabs.net
 
173.205.126.94 - - [25/Dec/2010:01:19:57 -0500] "POST /wp-cron.php?doing_wp_cron HTTP/1.0" 200 - "-" "WordPress/3.0.3; http://atomlabs.net"
88.214.193.166 - - [25/Dec/2010:01:19:56 -0500] "GET / HTTP/1.1" 200 2715 "http://www.semrush.com/info/atomlabs.net" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
109.120.144.247 - - [25/Dec/2010:01:39:04 -0500] "GET /plugins/editors/tinymce/jscripts/tiny_mce/plugins/tinybrowser/tinybrowser.php?type=file&folder= HTTP/1.0" 404 - "http://atomlabs.net/plugins/editors/tinymce/jscripts/tiny_mce/plugins/tinybrowser/tinybrowser.php?type=file&folder=" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90)"
207.46.13.87 - - [25/Dec/2010:01:46:00 -0500] "GET /robots.txt HTTP/1.1" 301 238 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
207.46.13.87 - - [25/Dec/2010:01:46:00 -0500] "GET /robots.txt HTTP/1.1" 404 - "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
207.46.13.87 - - [25/Dec/2010:01:55:44 -0500] "GET /page2.php HTTP/1.1" 301 237 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
88.214.193.166 - - [25/Dec/2010:02:10:24 -0500] "GET / HTTP/1.1" 200 3342 "http://www.semrush.com/info/atomlabs.net" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
209.85.238.86 - - [25/Dec/2010:02:33:04 -0500] "GET /dev/?feed=rss2 HTTP/1.1" 404 - "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=17880273271851489999)"
66.249.72.100 - - [25/Dec/2010:03:14:50 -0500] "GET /?p=23 HTTP/1.1" 200 13019 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
88.214.193.166 - - [25/Dec/2010:03:22:32 -0500] "GET / HTTP/1.1" 200 5884 "http://www.whorush.com/search/?q=atomlabs.net" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
88.214.193.166 - - [25/Dec/2010:03:24:48 -0500] "GET / HTTP/1.1" 200 5884 "http://www.whorush.com/search/?q=atomlabs.net" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
66.249.72.100 - - [25/Dec/2010:04:09:11 -0500] "GET /dev HTTP/1.1" 404 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

I don’t know about you but that can become very cumbersome to look at, especially when you just want a specific chunk of information out of all of that mess. So this is where (awk) comes into play, in its simplest form we can use it to print information from a specific column of text.

By default (awk) is going to make the default delimiter a space, so if we had the text:

66.249.72.100 - - [25/Dec/2010:04:09:11 -0500] "GET /dev HTTP/1.1" 404 - "-" "Mozilla/5.0

To only show the date this would be the 4th column of data, here is how (awk) would break it down:

$0 = 66.249.72.100 - - [25/Dec/2010...]
$1 = 66.249.72.100
$2 = -
$3 = -
$4 = [25/Dec/2010:04:09:11
$5 = -0500]

I want to find who’s visiting my site and what pages:

cat /usr/local/apache/domlogs/atomlabs.net | awk '{print $1 " " $7}'
 
173.205.126.94 /wp-cron.php?doing_wp_cron
88.214.193.166 /
109.120.144.247 /plugins/editors/tinymce/jscripts/tiny_mce/plugins/tinybrowser/tinybrowser.php?type=file&folder=
207.46.13.87 /robots.txt
207.46.13.87 /robots.txt
207.46.13.87 /page2.php
88.214.193.166 /
209.85.238.86 /dev/?feed=rss2
66.249.72.100 /?p=23
88.214.193.166 /
88.214.193.166 /
66.249.72.100 /dev

You can pass multiple variables to (awk) like you’ve just seen above, you can also pass string data between quotes (“). I passed a space, that way the output would be the IP address, a space, then the file.

Recall previous CLI arguments with ease

Webmaster — Fri, 24 Dec 2010 00:41:04 +0000

Have you ever been running various commands on the same file or folder, and get sick of having to re-type it in each time? One easy way to get around this is with the built in recall arguments feature built into Linux. Essentially you hit (Alt + # + .) where (#) is the number of argument you are recalling:

Run a simple command with two arguments:

grep user1@domain1.com /var/log/exim_mainlog

Now after I type in the new command I want to run:

grep user2@domain2.com

I can recall the 2nd argument to grab my log path:

grep user2@domain2.com (Hold Alt + 2 + .)
grep user2@domain2.com /var/log/exim_mainlog

Exim – Mail queue commands

Webmaster — Thu, 23 Dec 2010 21:14:11 +0000

exim -bp : Show summary of messages in queue
exim -bpc : Show number of messages in queue

-M : Force delivery
-Mar : Add recipient
-Meb : Edit message body
-Mes : Edit sender
-Mf : Freeze message
-Mg : Give up (and bounce message)
-Mmad : Mark all recipients as delivered
-Mmd : Mark recipient as delivered
-Mrm : Remove message (no bounce)
-Mt : Thaw message
-Mvb : View message body
-Mvh : View message header
-Mvl : View message log

Create file of certain size

Webmaster — Thu, 23 Dec 2010 21:09:26 +0000

Wish you could create a test file for file-transfer testing? It’s easy with Linux’s (dd) command:

Create 1MB file:

dd if=/dev/zero of=1MB.txt bs=1024 count=1024

Create 50MB file:

dd if=/dev/zero of=50MB.txt bs=1024 count=51200

Verify the sizes:

ls -lh | grep MB
-rw-r--r--  1 root root 1.0M Dec 23 16:06 1MB.txt
-rw-r--r--  1 root root  50M Dec 23 16:06 50MB.txt

Convert text to upper-case or lower-case

Webmaster — Thu, 23 Dec 2010 16:26:24 +0000

There are more than a couple of ways to convert a string with upper-case letters to lower-case and vice-versa in Linux, here are my favorite two:

awk

echo "TEXT WITH CAPS-LOCK ON" | awk '{print tolower($0)}'
text with caps-lock on
 
echo "text with caps-lock off" | awk '{print toupper($0)}'
TEXT WITH CAPS-LOCK OFF

echo "TEXT WITH CAPS-LOCK ON" > upperText; dd if=upperText of=lowerText conv=lcase; cat lowerText
0+1 records in
0+1 records out
23 bytes (23 B) copied, 2.1e-05 seconds, 1.1 MB/s
text with caps-lock on
 
echo "text with caps-lock off" > lowerText; dd if=lowerText of=upperText conv=ucase; cat upperText
0+1 records in
0+1 records out
24 bytes (24 B) copied, 3.4e-05 seconds, 706 kB/s
TEXT WITH CAPS-LOCK OFF

Find the difference between two files

Webmaster — Sat, 18 Dec 2010 14:10:00 +0000

Using the (diff) command you can find out the difference between two files, (sdiff) allows for you to see both files at once to find the difference.

echo "ABC" > a
echo "ABCD" > b
diff a b
1c1
< ABC
---
> ABCD
sdiff a b
ABC                                                           | ABCD

If statements

Webmaster — Sat, 11 Dec 2010 03:54:31 +0000

In Linux just like with most scripting languages you can use if statements to allow for step-by-step processing through a set of rules. Let’s say you wanted to have a simple script to handle clock-ins for you employees:

#!/bin/sh
 
echo "Enter your employee ID:"
read employeeID
 
echo "in - Clock in"
echo "out - Clock out"
read clockStatus
 
if [ $clockStatus = "in" ]
then
        date >> $employeeID"_timeclock"
        echo "$employeeID logged in" >> $employeeID"_timeclock"
else
        date >> $employeeID"_timeclock"
        echo "$employeeID logged out" >> $employeeID"_timeclock"
fi

For loops

Webmaster — Sat, 11 Dec 2010 03:42:23 +0000

for variable in list; do echo $variable; done
list

for variable in a b c d; do echo $variable; done
a
b
c
d

for variable in a b c d; do echo $variable; done > list
for variable in $(cat list); do echo $variable; done
a
b
c
d