About Intermediate Shell Tools Grep Sed Awk

What we have here three of the most useful text-related utilities:

• grep (The grep program search for strings)
• sed   (The sed program provides a way to edit text as it passes through a pipline)
• awk   (awk will be used to parse the input or format the output at some stage of the shell script)

1.How to Sifting Through Files for a String

Problem

how to find all occurrences of a string in one or more files.

Solution

The grep command searches through files 

[maxwell@DBAMAXWELL cp6]$ grep printf *.sh
dash.sh:       *) printf 'usage: %s [-c X] [#]\n' $(basename $0) >&2
strvsnum.sh:printf "%s" "do they -eq as equal? "
strvsnum.sh:printf "%s" "do they = as equal? "
strvsnum.sh:printf "%s" "do they = as equal? "
[maxwell@DBAMAXWELL cp6]$ 

Not all the files through which you want to search may be that conveniently located, you could have done something like this :

[maxwell@DBAMAXWELL cp6]$ grep printf ../cp5/*.sh ../cp4/*.sh ../cp3/*.sh ../cp2/*.sh ../cp1/*.sh
../cp5/check_arg_count.sh:    printf "%b" "Error. Not enough arguments.\n" >&2
../cp5/check_arg_count.sh:    printf "%b" "usage: myscript file1 op file2\n" >&2
../cp5/check_arg_count.sh:    printf "%b" "Error. Too many arguments.\n" >&2
../cp5/check_arg_count.sh:    printf "%b" "usage: myscript file1 op file2\n" >&2
../cp5/check_arg_count.sh:    printf "%b" "Argument count correct. Proceeding...\n"
grep: ../cp4/*.sh: No such file or directory
../cp3/func_choose.sh:  # 'printf "%b" "See you later Professor Falkin."' >&2
../cp3/func_choose.sh:       * ) printf "%b" "Unexpected answer '$answer'!" >&2 ;;
../cp3/select_dir.sh:    printf "%b" "\a\n\nSelect a directory to process:\n" >&2
grep: ../cp2/*.sh: No such file or directory
grep: ../cp1/*.sh: No such file or directory
[maxwell@DBAMAXWELL cp6]$ 

If you don't want to see the specific filenames,you may turn this off using the -h switch to grep:

[maxwell@DBAMAXWELL cp6]$ grep printf *.sh   
dash.sh:       *) printf 'usage: %s [-c X] [#]\n' $(basename $0) >&2
strvsnum.sh:printf "%s" "do they -eq as equal? "
strvsnum.sh:printf "%s" "do they = as equal? "
strvsnum.sh:printf "%s" "do they = as equal? "
[maxwell@DBAMAXWELL cp6]$ grep -h printf *.sh
       *) printf 'usage: %s [-c X] [#]\n' $(basename $0) >&2
printf "%s" "do they -eq as equal? "
printf "%s" "do they = as equal? "
printf "%s" "do they = as equal? "
[maxwell@DBAMAXWELL cp6]$ 

 If you just have only a count of the number of times the expression is found, then use the -c option:

[maxwell@DBAMAXWELL cp6]$ grep -c printf *.sh 
checkfile.sh:0
checkstr.sh:0
dash.sh:1
strvsnum.sh:3
trackmatch.sh:0
[maxwell@DBAMAXWELL cp6]$ 

 2.How to getting just the filename from a search

Problem

• Find the files in whcih a certain  string appears.
• Don't want to see the line of text that was found,just the filename.

Solution

Use the -l option of grep to get just the filenames:

[maxwell@DBAMAXWELL cp6]$ grep -l printf *.sh 
dash.sh
strvsnum.sh
[maxwell@DBAMAXWELL cp6]$ 

3.How to getting a simple True/False from a Search

Problem:

• Need to know whether a certain string is in a particular file.
• just a yes or no sort of answer

Solution:

• Use -q , the "quiet" option for grep
• throw the output away by redirecting it into /dev/null
• Return status variable $? in the bash.

[maxwell@DBAMAXWELL cp6]$ grep -q printf *.sh 
[maxwell@DBAMAXWELL cp6]$ if [ $? -eq 0 ] ; then echo yes ; else echo nope ; fi
yes
[maxwell@DBAMAXWELL cp6]$ 

[maxwell@DBAMAXWELL cp6]$ grep printf *.sh                                     
dash.sh:       *) printf 'usage: %s [-c X] [#]\n' $(basename $0) >&2
strvsnum.sh:printf "%s" "do they -eq as equal? "
strvsnum.sh:printf "%s" "do they = as equal? "
strvsnum.sh:printf "%s" "do they = as equal? "
[maxwell@DBAMAXWELL cp6]$ grep printf *.sh >/dev/null
[maxwell@DBAMAXWELL cp6]$ if [ $? -eq 0 ] ; then echo yes ; else echo nope ; fi
yes
[maxwell@DBAMAXWELL cp6]$ 

4.How to searching for Text While Ignoring Case

Problem

• search for a string (ep. error) in a log file
• do it case-insensitively to catch all occurrences

Solution

Use the -i option on grep to ignore case:

[maxwell@DBAMAXWELL cp6]$ grep -i java logfile.msgs 
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:312)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:985)
at com.atguigu.spring.test.test.main(test.java:15)
[maxwell@DBAMAXWELL cp6]$

5.How to doing a search in a pipeline

Problem

searcg for some test in a pipeline

Solution

[maxwell@DBAMAXWELL cp6]$ grep -i java logfile.msgs | grep -i sh *.sh
checkfile.sh:#! /bin/bash
checkfile.sh:INFILE=/home/learning/learning_bash_shell/cp6/amazing.data
checkfile.sh:OUTFILE=/home/learning/learning_bash_shell/cp6/more.results
checkstr.sh:#!/bin/bash
dash.sh:#!/bin/bash
dash.sh:#cookbook file: dash
dash.sh:#dashes - print a line of dashes
dash.sh:#         -c X use char X instead of dashes
dash.sh:        -c) shift;
dash.sh:     shift
strvsnum.sh:#!/bin/bash
trackmatch.sh:#!/bin/bash
trackmatch.sh:  echo Track ${BASH_REMATCH[2]} is ${BASH_REMATCH[3]}
trackmatch.sh:  mv "$CDTRACK" "Track${BASH_REMATCH[2]}"
[maxwell@DBAMAXWELL cp6]$ !! | grep -i sh *.sh                         
grep -i java logfile.msgs | grep -i sh *.sh | grep -i sh *.sh
checkfile.sh:#! /bin/bash
checkfile.sh:INFILE=/home/learning/learning_bash_shell/cp6/amazing.data
checkfile.sh:OUTFILE=/home/learning/learning_bash_shell/cp6/more.results
checkstr.sh:#!/bin/bash
dash.sh:#!/bin/bash
dash.sh:#cookbook file: dash
dash.sh:#dashes - print a line of dashes
dash.sh:#         -c X use char X instead of dashes
dash.sh:        -c) shift;
dash.sh:     shift
strvsnum.sh:#!/bin/bash
trackmatch.sh:#!/bin/bash
trackmatch.sh:  echo Track ${BASH_REMATCH[2]} is ${BASH_REMATCH[3]}
trackmatch.sh:  mv "$CDTRACK" "Track${BASH_REMATCH[2]}"
[maxwell@DBAMAXWELL cp6]$ 

6.How to paring down what the search finds?

Problem 

Your search is returning way more than you expected,including many results you don't want

Solution

Pipe the results into grep -v with an expression that describes what you don't want to see.

[maxwell@DBAMAXWELL cp6]$ cat logfile
error on Jan 01: not a decimal number
error on Feb 13: base converted to Decimal
warning on Mar 22: using only decimal numbers
error on Dec 16 : the actual message you wanted
error on Jan 01: not a decimal number
[maxwell@DBAMAXWELL cp6]$ grep -i dec logfile
error on Jan 01: not a decimal number
error on Feb 13: base converted to Decimal
warning on Mar 22: using only decimal numbers
error on Dec 16 : the actual message you wanted
error on Jan 01: not a decimal number
[maxwell@DBAMAXWELL cp6]$ grep -i dec logfile | grep -vi decimal
error on Dec 16 : the actual message you wanted
[maxwell@DBAMAXWELL cp6]$ grep -i dec logfile | grep -vi decimal | grep -vi decimate
error on Dec 16 : the actual message you wanted

[maxwell@DBAMAXWELL cp6]$ grep 'Dec [0-9][0-9]' logfile
error on Dec 16 : the actual message you wanted
[maxwell@DBAMAXWELL cp6]$ grep Dec\ [0-9\ ][0-9] logfile
error on Dec 16 : the actual message you wanted
[maxwell@DBAMAXWELL cp6]$ 
 

7.How to searching for an SSN

Problem

 You need to a regualr expression to match a social security number.

Solution

[maxwell@DBAMAXWELL cp6]$ grep '[0-9]\{3\}-\{0,1\}[0-9]\{2\}-\{0,1\}{0-9}\{4\}' datafile

8.How to grepping compressed Files

Problem

You need to grep some compressed files.

Solution

• zgrep
• zcat
• gzcat

[maxwell@DBAMAXWELL cp6]$ zgrep 'java' logfile.msgs.gz     
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:312)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:985)
at com.atguigu.spring.test.test.main(test.java:15)
[maxwell@DBAMAXWELL cp6]$ zcat logfile.msgs.gz
Exception in thread “main” org.springframework.beans.factory.NoUniqueBeanDefinitionException: No qualifying bean of type [com.atguigu.spring.bean.Student] is defined: expected single matching bean but found 2: student,student1
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:312)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:985)
at com.atguigu.spring.test.test.main(test.java:15)
[maxwell@DBAMAXWELL cp6]$

9.How to keeping some output,discarding the Rest

Problem

You need a way to keep some of your output and discard the rest.

Solution

The awk utility read data from the filename words on the command line,

The following code prints the first word of every line of input.

[maxwell@DBAMAXWELL cp6]$ cat myinput.file
Words are delineated by whitespace. The awk utility reads data from the filename
supplied on the command line, or from standard input if no filename is given.
[maxwell@DBAMAXWELL cp6]$ awk '{print $1}' myinput.file
Words
supplied
[maxwell@DBAMAXWELL cp6]$ awk '{print $1}' < myinput.file
Words
supplied
[maxwell@DBAMAXWELL cp6]$ cat myinput.file | awk '{print $1}'
Words
supplied
[maxwell@DBAMAXWELL cp6]$ 

10.How to keeping only a portion of a line of output

Problem

You want to keep only a portion of a line of output. such as the first and last words.

Solution

Pipe ls into awk

[maxwell@DBAMAXWELL cp6]$ ls -l | awk '{print $1, $NF}'
total 40
-rwxrwxr-x amazing.data
-rwxrwxr-x calc
-rwxrwxr-x checkfile.sh
-rwxrwxr-x checkstr.sh
-rwxrwxr-x dash.sh
-rwxrwxr-x datafile
-rwxrwxr-x logfile
-rwxrwxr-x logfile.msgs.gz
-rwxrwxr-x more.results
-rwxrwxr-x myinput.file
-rwxrwxr-x strvsnum.sh
-rwxrwxr-x trackmatch.sh
[maxwell@DBAMAXWELL cp6]$ ls -l
total 40
-rwxrwxr-x 1 maxwell root   0 Mar 16 20:18 amazing.data
-rwxrwxr-x 1 maxwell root 144 Mar 17 06:14 calc
-rwxrwxr-x 1 maxwell root 476 Mar 16 19:46 checkfile.sh
-rwxrwxr-x 1 maxwell root 280 Mar 16 20:57 checkstr.sh
-rwxrwxr-x 1 maxwell root 375 Mar 17 05:56 dash.sh
-rwxrwxr-x 1 maxwell root  93 Mar 17 09:22 datafile
-rwxrwxr-x 1 maxwell root 213 Mar 17 08:51 logfile
-rwxrwxr-x 1 maxwell root 300 Mar 17 08:36 logfile.msgs.gz
-rwxrwxr-x 1 maxwell root   0 Mar 16 20:18 more.results
-rwxrwxr-x 1 maxwell root 159 Mar 17 10:44 myinput.file
-rwxrwxr-x 1 maxwell root 416 Mar 16 17:22 strvsnum.sh
-rwxrwxr-x 1 maxwell root 255 Mar 16 21:30 trackmatch.sh
[maxwell@DBAMAXWELL cp6]$ 

 11.How to reversing the words on each line.

Problem

You want to print the input lines with words in the reverse order.

Solution

[maxwell@DBAMAXWELL cp6]$  ls -l | awk '{for (i=NF; i>0; i--) {printf "%s ", $i;} printf "\n" }'     
40 total 
amazing.data 20:18 16 Mar 0 root maxwell 1 -rwxrwxr-x 
calc 06:14 17 Mar 144 root maxwell 1 -rwxrwxr-x 
checkfile.sh 19:46 16 Mar 476 root maxwell 1 -rwxrwxr-x 
checkstr.sh 20:57 16 Mar 280 root maxwell 1 -rwxrwxr-x 
dash.sh 05:56 17 Mar 375 root maxwell 1 -rwxrwxr-x 
datafile 09:22 17 Mar 93 root maxwell 1 -rwxrwxr-x 
logfile 08:51 17 Mar 213 root maxwell 1 -rwxrwxr-x 
logfile.msgs.gz 08:36 17 Mar 300 root maxwell 1 -rwxrwxr-x 
more.results 20:18 16 Mar 0 root maxwell 1 -rwxrwxr-x 
myinput.file 10:44 17 Mar 159 root maxwell 1 -rwxrwxr-x 
strvsnum.sh 17:22 16 Mar 416 root maxwell 1 -rwxrwxr-x 
trackmatch.sh 21:30 16 Mar 255 root maxwell 1 -rwxrwxr-x 
[maxwell@DBAMAXWELL cp6]$ 

 12. How to summing a List of Number

Problem

You need to sum a list of numbers,including numbers that don't appear on lines by themselves.

Solution

Use awk both isolate the field to be summed and to do the summing.

[maxwell@DBAMAXWELL cp6]$ ls -l
total 40
-rwxrwxr-x 1 maxwell root   0 Mar 16 20:18 amazing.data
-rwxrwxr-x 1 maxwell root 144 Mar 17 06:14 calc
-rwxrwxr-x 1 maxwell root 476 Mar 16 19:46 checkfile.sh
-rwxrwxr-x 1 maxwell root 280 Mar 16 20:57 checkstr.sh
-rwxrwxr-x 1 maxwell root 375 Mar 17 05:56 dash.sh
-rwxrwxr-x 1 maxwell root  93 Mar 17 09:22 datafile
-rwxrwxr-x 1 maxwell root 213 Mar 17 08:51 logfile
-rwxrwxr-x 1 maxwell root 300 Mar 17 08:36 logfile.msgs.gz
-rwxrwxr-x 1 maxwell root   0 Mar 16 20:18 more.results
-rwxrwxr-x 1 maxwell root 159 Mar 17 10:44 myinput.file
-rwxrwxr-x 1 maxwell root 416 Mar 16 17:22 strvsnum.sh
-rwxrwxr-x 1 maxwell root 255 Mar 16 21:30 trackmatch.sh
[maxwell@DBAMAXWELL cp6]$ ls -l | awk '{sum += $5} END {print sum}'
2711
[maxwell@DBAMAXWELL cp6]$ ls -l | grep -v '^total' | awk '{sum += $5} END {print sum}'
2711
[maxwell@DBAMAXWELL cp6]$ ls -l | awk '/^total/{getline} {sum += $5} END {print sum}' 
2711
[maxwell@DBAMAXWELL cp6]$ 

 13.How to counting string values?

Problem

You need to count all the occurrences of several different strings,including some strings whose values you don't know beforehand.

Solution

use awk's associate arrays for your counting.

[root@DBAMAXWELL cp6]# cat asar.awk
#
# cookbook filename:awk
#
NF > 7 {
        user[$3]++
    }
END {
        for (i in user)
        {
             printf "%s owns %d files\n",i,user[i]
        }
    }
[root@DBAMAXWELL cp6]# ls -lR /usr/local | awk -f asar.awk
root owns 6623 files
[root@DBAMAXWELL cp6]# 

14.How to showing data as a quick and easy histogram

Problem

You need a quick screen-based histogram of some data

Solution

Use the associative arrays of awk.

[root@DBAMAXWELL cp6]# ls -ltr
total 48
-rwxrwxr-x 1 maxwell root 416 Mar 16 17:22 strvsnum.sh
-rwxrwxr-x 1 maxwell root 476 Mar 16 19:46 checkfile.sh
-rwxrwxr-x 1 maxwell root   0 Mar 16 20:18 amazing.data
-rwxrwxr-x 1 maxwell root   0 Mar 16 20:18 more.results
-rwxrwxr-x 1 maxwell root 280 Mar 16 20:57 checkstr.sh
-rwxrwxr-x 1 maxwell root 255 Mar 16 21:30 trackmatch.sh
-rwxrwxr-x 1 maxwell root 375 Mar 17 05:56 dash.sh
-rwxrwxr-x 1 maxwell root 144 Mar 17 06:14 calc
-rwxrwxr-x 1 maxwell root 300 Mar 17 08:36 logfile.msgs.gz
-rwxrwxr-x 1 maxwell root 213 Mar 17 08:51 logfile
-rwxrwxr-x 1 maxwell root  93 Mar 17 09:22 datafile
-rwxrwxr-x 1 maxwell root 159 Mar 17 10:44 myinput.file
-rw-r--r-- 1 maxwell root 134 Mar 17 11:22 asar.awk
-rwxrwxr-x 1 maxwell root 459 Mar 17 13:39 hist.awk
[root@DBAMAXWELL cp6]# ls -lR /usr/local | awk -f hist.awk
root       [    6623]:############################################################
[root@DBAMAXWELL cp6]# 

15.How to showing a paragraph of Text after a found Phrase

Problem

You are searching for a phrase in a document, and want to show the paragraph after the found phrase.

Solution

[root@DBAMAXWELL cp6]# cat para.awk                     
/keyphrase/ { flag=1 }
{ if (flag == 1) { print $0 } }
/^$/ {flag=0 }
[root@DBAMAXWELL cp6]# cat searchthis.text
There are just three simple code blocks. The first is invoked when a line of input
matches the regular expression (here just the word “keyphrase”). If “keyphrase”
occurs anywhere within a line of input, that is a match and this block of code will be
executed. All that happens in this block is that the flag is set.
The second code block is invoked for every line of input, since there is no regular
expression preceding its open brace. Even the input that matches “keyphrase” will
also be applied to this code block (if we didn’t want that effect, we could use a
continue statement in the first block). All this second block does is print the entire
input line, but only if the flag is set.
[root@DBAMAXWELL cp6]# awk -f para.awk < searchthis.text
matches the regular expression (here just the word “keyphrase”). If “keyphrase”
occurs anywhere within a line of input, that is a match and this block of code will be
executed. All that happens in this block is that the flag is set.
The second code block is invoked for every line of input, since there is no regular
expression preceding its open brace. Even the input that matches “keyphrase” will
also be applied to this code block (if we didn’t want that effect, we could use a
continue statement in the first block). All this second block does is print the entire
input line, but only if the flag is set.
[root@DBAMAXWELL cp6]#