9/3/2012

r1

Introductions • Your name • Location – home office • Job role • How long with [industry, community, and so on] • Something unique about yourself

Linux (Day 1 session)

IBM Internal Use | 3 September 2012

IBM Internal Use | 3 September 2012

1

2

r2

Course content (Day 1)

Session objectives

• Module 1: Introduction • Module 2: Unix File System • Module 3: Command substitution, filters and pipes • Module 4: Locating files with find • Module 5: xargs command

By the end of this session, you should be able to: • Discuss the history of Unix and Linux • Discuss the features and architecture of Linux • Discuss the Unix file system and the directory structure of Linux • Use Linux filters, piping mechanism, and command substitution • Use the find command • Use the xargs command IBM Internal Use | 3 September 2012

3

IBM Internal Use | 3 September 2012

4

r3

Housekeeping • Breaks • Washrooms • Transportation and parking • No pagers or cell phones • Participation • Questions

IBM Internal Use | 3 September 2012

Module 1: Introduction

5

1

Slide 2 r1

This is a new slide. rbtolentino, 11/9/2009

Slide 4 r2

All course content for day 1 was converged in a single slide. Sub bulletpoints were deleted. rbtolentino, 11/9/2009

Slide 5 r3

This is a new slide. rbtolentino, 11/9/2009

9/3/2012

r4

Module agenda • • • •

Operating System History of Unix and Linux Features of Linux Architecture of Linux

IBM Internal Use | 3 September 2012

7

Operating System • It is the interface between hardware and user. • It is responsible for management and coordination of activities and sharing of resources of computer. • It acts as host for computing applications that run on the machine. • The applications access OS services through APIs or System Calls. • It provides two types of interfaces to the users, namely Command Line Interface and Graphical User Interface. IBM Internal Use | 3 September 2012

8

History of Unix and Linux

Features of Unix

• In 1969, UNIX first originated in form of Multics at Bell Laboratories. • In 1974, Thompson and Ritchie both published a paper regarding UNIX in Communications of ACM. • By 1977, several UNIX systems were used in Universities. • During 1977 to 1982, UNIX System III was released by Bell Laboratories. • By the start of 1984, the use of UNIX systems significantly increased. • Linux is a complete rewrite of Unix, developed by Linus Tourvalds, when he was a Finnish undergraduate.

• Portable

IBM Internal Use | 3 September 2012

IBM Internal Use | 3 September 2012

9

Architecture of Linux

Unix OS can be installed on any architecture.

• Multi-user Multiple users can access the system and share its resources.

• Multi-tasking Multiple tasks can be initiated and run simultaneously.

• Time-sharing Server shares CPU time between requesting processes.

• Hierarchical file organization The / (root) is at the top of hierarchy and the various other file systems are below that. 10

Architecture of Linux: System components User 1

• System components • Kernel • Shell

Shell

X-Window

Kernel

sort

IBM Internal Use | 3 September 2012

Different Databases

grep Browsers ls

who

Various Text Processors 11

ps

Hardware

tar Other Compilers

IBM Internal Use | 3 September 2012

Other Softwares

Shell sed

cp

cc

User 2

Spreadsheets

12

2

Slide 7 r4

This is a new slide. rbtolentino, 11/9/2009

9/3/2012

r5

Architecture of Linux: Kernel

Architecture of Linux: Shell

• This is the core of Operating System. • It functions as the Hardware interface. • It contains system calls that perform low-level tasks. • It generates inode numbers for newly created files and maintains inode tables. • It generates process identifications (IDs) for newly created processes and maintains process control blocks.

• User interface Shell provides an interface to the user, wherein the user could issue his commands, and Shell displays output and error messages to the user. • Command interpreter Shell accepts command from user, and interprets it to the kernel. • Command processor Shell parses the command line arguments, expands the special meaning of meta characters, searches for the command, and if the command is found, then transfers control to the command. • Programming language Shell provides a native programming language.

IBM Internal Use | 3 September 2012

IBM Internal Use | 3 September 2012

13

14

Knowledge check Review questions: 1.What is the interface between the hardware and the user? 2.Who developed Linux? 3.Enumerate three features of Linux.

IBM Internal Use | 3 September 2012

Module 2: UNIX file system

15

Module agenda

• File system is a way of organizing files in the secondary storage device. • Unix file system is divided into four partitions:

• File system • Inode table

– Boot Block Programs associated with booting of the system are stored in this block. – Super Block This stores accounting information about the file system. – Inode Block This stores inode tables. – Data Block This stores data.

• Directory structure

IBM Internal Use | 3 September 2012

File system

17

IBM Internal Use | 3 September 2012

18

3

Slide 13 r5

Edited the bulletpoints. rbtolentino, 11/9/2009

9/3/2012

Inode table: Block Addressing scheme

Inode table Every file in Linux file system has an inode table, which stores meta data of the file. The inode table stores the following information File Permission User Group File Modification Access Inode Ptrs about the file: Type Mode ID ID Size Time Time Updation To Bytes Ord.

644

Tom

Ken

Time

85

14-Jun-09

14-Jun09 09:11:45

09:11:45

data block

18-Jun09 15:12

IBM Internal Use | 3 September 2012

19

Directory structure in Linux is an inverted tree structure, wherein the root (/) directory is at the top of the hierarchy and the subdirectories are placed as children of the/ root. dev

bin

sbin

usr

IBM Internal Use | 3 September 2012

20

Directory structure: Referencing of files

Directory structure

etc

• Inode table consists of 13 fields, each containing addresses to a block each. • First 10 address fields contains addresses of a data block each. This is direct addressing. Number of data blocks addressed: 10. • The 11th field contains address of an address block, the addresses of which point to a data block each. This is Single indirect addressing. Number of data blocks addressed: 128. • 12th field contains address of an address block, the addresses of which point to an address block each, each of the addresses of which point to data blocks; Number of data blocks addressed: 128 * 128. • 13th field contains address of an address block, the addresses of which points to an address block each; each of the address of these address blocks point to an address block each, through which, data blocks are pointed. Number of data blocks addressed. 128 * 128 * 128.

tmp



• Absolute referencing The referencing of a file done with reference to the root directory, / Examples: $ cd /usr/include # changes to /usr/include directory $ ls /dev/tty # lists the file /dev/tty

• Relative referencing The referencing of a file done relative to the working directory. Current directory and parent to current directory are referred by shortcuts . and .. respectively. Examples: $ cd cppdir

IBM Internal Use | 3 September 2012

21

IBM Internal Use | 3 September 2012

# changes to subdirectory cppdir 22

Knowledge check Review questions: 1. What are the four partitions of the Unix file system? 2. Enumerate three types of information stored in the inode table. 3. What is the referencing of a file done relative to the working directory? 4. Suppose the user is in the directory /usr/include, how can he move on to his home directory? IBM Internal Use | 3 September 2012

Module 3: Command Substitution, Filters, and Pipes

23

4

9/3/2012

Command substitution • Command substitution is the technique of passing a command’s output as argument to another command. • The command that passes output as argument can be called as inner command. • The command that receives the output of another command as its argument, can be called outer command. IBM Internal Use | 3 September 2012

25

Command substitution: Example $ echo "Today's date is `date`“ Today's date is Tue Apr 7 15:14:41 IST 200 $ echo "No. of files in $PWD is $(ls|wc -l)“ No. of files in /home/murali is 45

Explanation: The meta-characters ` `(pair of backticks) and $() are known as command substitution meta-characters. When a command is issued, Shell first parses the command line arguments; the Shell first expands special meaning of these metacharacters. Hence, Shell first executes the command specified within `` or $() and places the output of this command as argument to the main command; then, Shell transfers control to the main command.

IBM Internal Use | 3 September 2012

Filters • A command which by default, expects input from standard input file and sends processed output to standard output file. • List of some filters:

Filters: uniq It translates consecutive identical entries from the input file, into unique entries. $ cat domestics dog goat goat goat horse $ uniq domestics dog goat horse

uniq tr tee pr cut paste bc IBM Internal Use | 3 September 2012

27

26

IBM Internal Use | 3 September 2012

Filters: uniq (continued)

Filters: tr It translates all occurrences of a set of characters in the first expression to corresponding set of characters in second expression, from the input file. Syntax:

c option $ cat domestics dog goat goat $ uniq -c domestics 1 dog 2 goat Explanation of code:

tr ‘exp1’ ‘exp2’ < file1

Options: s This will squeeze multiple consecutive occurrences of a specified set of characters to single occurrence. d This deletes the specified set of characters from the string. cd

With –c option, uniq command displays the number of consecutive occurrences of the identical entries and also the unique entries. IBM Internal Use | 3 September 2012

28

29

This complement deletes the specified set of characters from the string. IBM Internal Use | 3 September 2012

30

5

9/3/2012

Filters: tr example $ cat > text1 a quick brown fox jumped over the lazy dog $ tr '[a-z]' '[A-Z]' < text1 A QUICK BROWN FOX JUMPED OVER THE LAZY DOG $

Filters: tr example (continued) $ cat mydirinfo total 12 -rw-r--r-- 1 murali murali 0 2009-03-25 12:48 chap1 -rwxr--r-- 1 murali murali 133 2009-03-25 12:48 myfunctions $ tr -s ' ' < mydirinfo total 12 -rw-r--r-- 1 murali murali 0 2009-03-25 12:48 chap1 -rwxr--r-- 1 murali murali 133 2009-03-25 12:48 myfunctions

Explanation of code: The tr command translates all occurrences of lower-case alphabets to corresponding upper-case alphabets in the file text1; it displays the output. Explanation of code: The tr command squeezes consecutive multiple occurrences of space character to single occurrence in the input file mydirinfo, and displays the output.

IBM Internal Use | 3 September 2012

IBM Internal Use | 3 September 2012

31

Filters: tr example: d option $ cat > numfile 78343hhdfdh434384 834893849&*&*&dfdhfhdj 4839483jhdjfhdjf hsdjfhdjfhdj4738473 jdfjdhf384738 $ tr -d '[a-z&*]' < numfile 78343434384 834893849 4839483 4738473 384738

Explanation of code: The tr command removes lower-case alphabets, the characters ‘&’ and ‘*’ from the input file numfile, and displays the output.

IBM Internal Use | 3 September 2012

Filters: tr example: cd option $ cat > numfile 78343hhdfdh434384 834893849&*&*&dfdhfhdj 4839483jhdjfhdjf hsdjfhdjfhdj4738473 jdfjdhf384738 $ tr -cd '[0-9\n]' < numfile 78343434384 834893849 4839483 4738473 384738

Explanation of code: The tr command removes all other characters except digit characters and newline characters from the input file numfile, and displays the output.

IBM Internal Use | 3 September 2012

33

Filters: tee It examines and counts the number of arguments say, n issued to it, creates n+1 copies of the input it receives, and writes n copies to the n files specified to it as arguments, and one copy to the standard output file. Syntax :

34

Filters: pr It paginate or columnate files for printing. Syntax: pr [ file1 file2 file3 … ]

Options:

tee file1 file2 file3 …

who | tee myusers user1 pts/0 user2 pts/1 user3 pts/2

32

08:50 08:54 09:32

Explanation of code: tee counts the number of arguments. If the number of arguments to tee is say, k, then tee creates k+1 number of copies of the input it reads; k copies, it writes to the files specified as arguments; one copy, it sends to standard output file.

9.122.187.231 9.145.145.200 9.122.140.121

d This will double space the output. D This will use format for the header date. l This sets the page length. W This sets the page width.

IBM Internal Use | 3 September 2012

35

IBM Internal Use | 3 September 2012

36

6

9/3/2012

Filters: cut It extracts specified characters or fields from the input file. Syntax: cut [] []

Options: c Used to specify the characters, that have to be extracted from every line of input file. d

Filters: paste It expects two files as input files, joins corresponding lines of the two files, and sends the output to standard output file. Syntax: paste [] [] []

Used to specify field delimiter, based on which, cut would understand the fields. f Used to specify the number of the fields that have to be extracted.

IBM Internal Use | 3 September 2012

37

IBM Internal Use | 3 September 2012

Filters: bc Performs calculation on the input expression. Takes input for standard input and sends processed output to standard output. Syntax : bc bc calcfile # calcfile contains expression

38

Pipes • A pipe is a communicating process. It receives output of one command and sends it as input to another command. • command1 | command2 Command1

Command2

Examples : $ bc # issues input interrupt 5^3 # finds 5 raised to power of 3 125 # and displays output on screen quit # terminates bc program $ # and returns control back to Shell IBM Internal Use | 3 September 2012

39

• Example: ls –t | head -15

• Output of ls command is passed on to head command as its input, through the pipe. IBM Internal Use | 3 September 2012

40

Knowledge check Review questions: 1.What is the command that receives the output of another command as its argument? 2.What filter extracts specified characters or fields from the input file? 3.What receives output of one command and sends it as input to another command?

IBM Internal Use | 3 September 2012

Module 4: Locating files with find

41

7

9/3/2012

find

find (continued)

It searches for the files with specified criteria, recursively starting from the specified directory, and takes action on those files. The default action is print. Syntax:

It searches for the files with specified criteria, recursively starting from the specified directory, and takes action on those files. The default action is print. Syntax:

find

find

List of operators and their purpose:

Examples: $ find $HOME -name "*.korn" -print /home/murali/korndir/namechange.korn /home/murali/korndir/untilDemo.korn /home/murali/korndir/cmdArgs.korn /home/murali/korndir/while2Demo.korn IBM Internal Use | 3 September 2012

43

-name: This searches for files matching specific names; find supports wild cards -mtime n: It displays when the file was last modified n*24 hours ago -atime n: It displays when the file was last accessed n*24 hours ago

IBM Internal Use | 3 September 2012

find (continued) List of operators and their purpose:

-amin n: It displays when the file was last accessed n minutes ago -cmin n: It displays when the file’s status last changed n minutes ago -mmin n: It displays when the file’s data was last modified n minutes ago -newer rfile: It displays when the files were modified more recently than rfile -inum n: It assigns the file with inode number n -links n: It assigns the file with n links -user uname: It assigns the file’s owner as user uname (numeric user ID allowed)

IBM Internal Use | 3 September 2012

45

find (continued) List of operators and their purpose:

-size n[cwbkMG]: File uses n units of space. The following suffixes can be used: b: for 512 bytes blocks (It is the default if no suffix is used) c: for bytes w: for two-byte words k: for kilobytes (units of 1024 bytes) M: for megabytes (units of 1048576 bytes) G: for gigabytes (units of 1073741824 bytes)

IBM Internal Use | 3 September 2012

find (continued) List of operators and their purpose:

-type c : It assigns that the file is of type c b: block special c: character special d – directory p – named pipe (FIFO) f – regular file l – symbolic link s – socket

IBM Internal Use | 3 September 2012

47

44

46

find (continued) List of actions and their purpose:

-print : It prints the full file name on the standard output, followed by a newline -exec : It executes a UNIX command placing files output by find, one, for each execution -ok : It executes a UNIX command placing files output by find, one, for each execution non-interactively

IBM Internal Use | 3 September 2012

48

8

9/3/2012

find: example (1) $ find /tmp -mtime +30 -print find: /tmp/orbit-gdm: Permission denied /tmp/mapping-root find: /tmp/gconfd-user1: Permission denied

$ pwd /home/murali $ find . -links +1 -type f -print 2> /dev/null ./cppdir/new.cpp ./cppdir/vector1.cpp

Explanation of code: The find command recursively searches for files under the directory /tmp which are not modified within past 30 days and prints them.

IBM Internal Use | 3 September 2012

49

find: example (2) $ find / -size 0c -print 2> /dev/null|xargs ls -l 2> /dev/null 1> ZeroByteFiles & [1] 30299

Explanation of code: The find command searches recursively under directory “/”, whose file size is zero bytes and pipes output to xargs; xargs places the list of files as arguments to ls command; ls command redirects output to the file ZeroByteFiles and errors are redirected to /dev/null; this command is run in the background.

IBM Internal Use | 3 September 2012

51

find: example (3) $ touch -t 07070900 myreffile $ find $HOME -newer myreffile -print 2> /dev/null /home/murali /home/murali/.lesshst /home/murali/ZeroByteFiles

Explanation of code: The find command recursively lists all files under the home directory of user, files whose modification timestamp is later than the file, myreffile; find displays the output and sends errors to /dev/null.

IBM Internal Use | 3 September 2012

find: example (1) (continued)

53

Explanation of code: The find command searches recursively under home directory of user, for regular files whose link count is more than one, like hard linked files.

IBM Internal Use | 3 September 2012

50

find: example (2) (continued) $ find / -perm 777 -type f 2> /dev/null | xargs ls -l -rwxrwxrwx 1 user20 user20 11688 2009-06-25 15:41 /home/user20/.bash_history -rwxrwxrwx 1 user20 user20 33 2009-03-06 15:43 /home/user20/.bash_logout -rwxrwxrwx 1 user20 user20 181 2009-05-11 12:57 /home/user20/.bash_profile -rwxrwxrwx 1 user20 user20 124 2009-03-06 15:43 /home/user20/.bashrc

Explanation of code: The find command recursively lists all regular files under the directory “/” whose octal permission mode is 777; errors are discarded; xargs receives list of files output by find through pipe and places this list as arguments to ls command, and transfers control to ls command; ls command displays long-listing of these files.

IBM Internal Use | 3 September 2012

52

find: example (3) (continued) $ find . -mtime +365 -type f -exec ls -l {} \; # -exec executes a Shell cmd # on the output of find command -rw-r--r-- 1 murali murali 658 2007-10-11 19:46 ./.zshrc -rw-r--r-- 1 murali murali 33 2007-08-31 19:50 ./.bash_logout $ find . -size 0 -type f -ok rm {} \; # -ok interactively confirms from user < rm ... ./usrs_at_now > ? Y # before executing Shell command < rm ... ./no_access_tmp > ? Y # on every argument < rm ... ./mynewerrfile > ? n < rm ... ./users > ? y < rm ... ./myperlscripts/scores_hash.pl > ? n < rm ... ./myperlscripts/mysubstitute.pl > ? n < rm ... ./cobolprogs > ? y

IBM Internal Use | 3 September 2012

54

9

9/3/2012

find: example (3) (continued)

Knowledge check Review questions: 1. What is the output of this command: find / -type f –size +32 –print ?

Explanation of code: The find command recursively outputs all regular files under working directory of user, which are not modified since the past 365 days; exec action of find command places the files one by one as argument to ls command and executes ls command.

a) Regular files that are greater than 32 bytes in size b) Regular files that are greater than 32 MB in size c) Regular files that are greater than 32 blocks in size d) Regular files that are exactly 32 blocks in size

IBM Internal Use | 3 September 2012

55

IBM Internal Use | 3 September 2012

56

Knowledge check Review questions 2. What will this command display: find . -links +1 –type f –exec ls –l {} \; ?

Module 5: xargs command

a) A long listing of all hard linked directory files in user’s working directory b) A long listing of all symbolically linked regular files in the user’s working directory c) A long listing of all hard linked regular files in the user’s working directory IBM Internal Use | 3 September 2012

57

xargs It reads arguments from standard input, place these as arguments to the Shell command and runs the command on these arguments. Syntax:

xargs: Example (1) $ xargs grep "cout" autodemo.c++ mynewprog.c++ autodemo.c++: cout<<++a<5)<<"\t"<<(10<5)<
xargs

IBM Internal Use | 3 September 2012

59

IBM Internal Use | 3 September 2012

60

10

9/3/2012

xargs: Example (2) $ ls *.c++ | xargs tar -cvf cppArchive.tar autodemo.c++ inheritance.c++ methOverriding.c++ mynewprog.c++ scope.c++ test.c++ virtualFunction.c++

xargs: Example (3) $ find . -mtime +180 -type f -name "*.txt" | xargs -I {} -t mv {} {}.old mv ./logFile.txt ./logFile.txt.old mv ./links.txt ./links.txt.old mv ./link.txt ./link.txt.old

$ ls -l cppArchive.tar -rw-rw-r-- 1 murali murali 10240 2009-07-07 12:34 cppArchive.tar

IBM Internal Use | 3 September 2012

61

IBM Internal Use | 3 September 2012

xargs • -I replace-str It replaces occurrences of replace-str in the initialarguments with names read from standard input.

• -t It prints the command line on the standard error output before executing it.

IBM Internal Use | 3 September 2012

63

62

Knowledge check 1.User types the command xargs at the $ prompt, and presses key. What is the response? 2.User types the command below at the $ prompt; what is the response? $ find / 2> /dev/null | xargs grep –l “DISPLAY” > outfiles 2> /dev/null & 3.What is the output for this: tar –tf shellprogs.tar | xargs cat | more ? IBM Internal Use | 3 September 2012

64

Session summary Having completed this session, you should be able to: • Discuss the history of Unix and Linux • Discuss the features and architecture of Linux • Discuss the Unix file system and the directory structure of Linux • Use Linux filters, piping mechanism, and command substitution • Use the find command • Use the xargs command IBM Internal Use | 3 September 2012

65

Linux (Day 2 session)

66

11

9/3/2012

r6

Course content (Day 2)

Session objectives

• Module 1: Advanced Filters: sed • Module 2: Advanced Filters: awk • Module 3: The Shell

By the end of this session, you should be able to: • Use the sed (Stream editor) • Use regular expressions with sed • Use the awk advance filter with the proper options, operators, and built-in variables • Use and customize the Linux shell with the proper commands and variables and utilize its new features. 67

68

r7

Module agenda • sed • sed Regular Expressions Module 1: Advanced Filters: sed

– Basic Regular Expressions – Extended Regular Expressions – Interval Regular Expressions – Tagged Regular Expressions

70

sed • A sed is also known as a stream editor. A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline). • It is a non-interactive editor. • Syntax: – sed [options] ‘’ input_file(s)

• Options: – i – It is used to perform in-place editing. – e – It is used to specify multiple scripts with a single sed command. – n – It is used to suppress default behavior (displaying all lines from input file, selected lines printed twice) of action, ‘p’. – f – It is used to specify a file that contains sed script. – r – It is used for extended regular expressions in the script 71

sed (continued) • Actions: All the commands specified below, reads input into buffer from the input file, takes action on the input in the buffer, and send the processed output to standard output file. – a – This will append text after specified line. – i – This will insert text before specified line. – q – This will quit after processing specified line. – ‘r file name’ – This will append text read from file name. – ‘R file name’ – This will append a line read from file name. – d – This will delete specified lines. – p – This will print specified lines. If n option is not specified, it prints all lines, selected lines are printed twice.

72

12

Slide 68 r6

All course content for day 1 was converged in a single slide. Sub bulletpoints were deleted. rbtolentino, 11/9/2009

Slide 70 r7

This is a new slide. rbtolentino, 11/9/2009

9/3/2012

sed (continued) • Actions: – ‘s/regexp/replacement /‘ - This subsititutes the first occurrence of regular expression regexp (in all matching lines) with the replacement pattern. – ‘s/regexp/replacement /g‘ - This substitutes all occurrences of regular expression regexp (in all matching lines) with the replacement pattern. – ‘w file name’ - This writes the current pattern space to file name. – ‘W file name’ – This writes the first line of current pattern space to file name.

sed (continued) • sed script consists of address action pair. • There are two types of addressing, namely: – Line addressing – Context addressing

• Line addressing is concerned on identifying a set of lines in input file, by specifying the line numbers. • Context addressing is concerned on identify lines containing matching pattern.

73

74

sed: Addressing example

sed: Line addressing example

$ cat -n employee1.lst 1 101:mohan:Manager 2 188:Aggarwal:manager Explanation: 3 102:mahesh:VP sed prints from the third to the fifth lines 4 105:tarun:manager from the file employee1.lst. 5 198:Aggrawal:PresidentIf –n option is not specified, all the lines are printed, and the selected lines are 6 109:karan:Clerk printed twice. $ sed -n '3,5p' employee1.lst 102:mahesh:VP 105:tarun:manager 198:Aggrawal:President

$ sed -ne '3p' -ne '6p' employee1.lst 102:mahesh:VP 109:karan:Clerk $ sed '4d' employee1.lst 101:mohan:Manager 188:Aggarwal:manager 102:mahesh:VP 198:Aggrawal:President 109:karan:Clerk

Explanation: sed removes 4th line from the file employee1.lst, from the scope of the output.

75

76

sed: Line addressing example (continued)

sed: Line addressing example (continued)

$ sed '1i 785:Raj:VP' employee1.lst 785:Raj:VP 101:mohan:Manager 188:Aggarwal:manager 102:mahesh:VP 105:tarun:manager 198:Aggrawal:President 109:karan:Clerk

Explanation: sed inserts the specified line before the current first line in the file employee1.lst. The file is not affected, but only scope of the output.

77

$ sed '$a 189:Paresh:VP' employee1.lst 101:mohan:Manager 188:Aggarwal:manager 102:mahesh:VP 105:tarun:manager 198:Aggrawal:President 109:karan:Clerk 189:Paresh:VP

Explanation: sed appends the specified line after the last line, in the input file Employee1.lst; only the scope of output is affected.

78

13

9/3/2012

sed: Context addressing example (continued)

sed: Context addressing example $ sed -i '/^105/s/manager/GM/' employee1.lst $ cat -n employee1.lst 1 101:mohan:Manager Explanation: 2 188:Aggarwal:manager sed substitutes the pattern “manager” with “GM” in the file employee1.lst on the line starting with pattern “105”; 3 102:mahesh:VP interactive option -i effects the change in the file. 4 105:tarun:GM 5 198:Aggrawal:President 6 109:karan:Clerk

$ who | sed -n '/^murali/p' murali pts/17 2009-07-09 14:20 (9.122.197.21)

Explanation: The output of the command ‘who’ is piped to sed; sed displays only those lines that begin with the pattern ‘murali’.

79

80

sed Regular Expression

sed Regular Expression: Basic Regular Expression

• A regular expression is a character that has a special meaning to the utility. There are certain characters that are specially interpreted by grep, egrep, sed, awk, and perl. • We have four types of Regular Expressions for sed namely: – Basic Regular Expressions (BREs) – Extended Regular Expressions (EREs) – Interval Regular Expressions (IREs) – Tagged Regular Expressions (TREs)

Regular Expression * . (or dot)

Meaning This matches zero or more occurrences of preceding character This matches exactly one occurrence of any character

^

This matches all the lines from input file, that start with the specified pattern

$

This matches all the lines from input file, that end with the specified pattern

[]

This is a Character class; the character class can be a list of characters or range of characters; matches one occurrence of any one of the specified characters

-

This is a Range operator

81

82

sed Regular Expression: Basic Regular Expression examples

sed Regular Expression: Basic Regular Expression examples (continued)

Explanation: In the above code example, ‘1,$’ refers to a range; it means the address, first line to the last line of the file; sed substitutes all occurrences (because of the global switch, ‘g’) of either organization or organization to organization in the file myorg1, specified as parameter. The operator [ ] refers to character class regular expression.

83

Explanation: Long-listing output of ls command is piped to sed; sed command displays only those lines which begin with the character d.

84

14

9/3/2012

sed Regular Expression: Repeated pattern &

sed Regular Expression: Extended Regular Expressions

• Repeated pattern “&” – To make the entire source pattern appear also at the destination

• Example: sed ‘s/director/executive director/’ emp.lst sed ‘s/director/executive &/’ emp.lst # same as above sed ‘/director/s//executive &/’ emp.lst # repeated & # remembered pattern

Regular Expression

Meaning

+

This matches one or more occurrences of preceding character

?

This matches zero or one occurrence of preceding character

|

This is an alternator; when placed between two patterns, it matches either of the pattern

()

This is used to specify a list of prefixes suffixed by a common suffix, or a list of suffixes prefixed by a common prefix

-

This is a Range operator

85 86

sed Regular Expression: Extended Regular Expressions (continued)

• Let us assume that we need to check if the entered number is an integer. These are some examples of valid integer inputs: – +7834 – -4323 – 3433265 Explanation: As we have used EREs namely ‘?’ and ‘+’, we need to specify –r option with sed. The above code checks that the user has entered a valid integer. sed checks if the string begins with zero or one occurrence of ‘+’ or ‘-’, succeeded by one or more occurrences of digit character.

sed Regular Expression: Interval Regular Expressions • sed and grep uses Interval Regular Expressions (IRE). IRE uses an integer to specify the number of characters preceding a pattern. The IRE uses an escaped pair of curly braces and takes three forms: – ch\{m\} – With this, the ch can occur m times – ch\{m,n\} – With this, the ch can occur between m and n times – ch\{m,\} – With this, ch can occur at least m times

• The value of m and n cannot exceed 255.

87

88

sed Regular Expression: Interval Regular Expressions (continued)

sed Regular Expression: Tagged Regular Expressions

$ echo $num 34657 $ echo $num | sed -n '/^[0-9]\{5\}$/p'; echo

Explanation: sed prints the number if it is a five digit number; specifying ‘^’ and ‘$’ at the beginning and end of the pattern is to do an exact match. We have used the Interval Regular Expression to match exactly five occurrences of digit character.

34657 $ num=4578 $ echo $num | sed -n '/^[0-9]\{5\}$/p'; echo

$ num=783237 $ echo $num | sed -n '/^[0-9]\{5\}$/p'; echo

$

89

• You have to identify segments of a line that you wish to extract and enclose each segment with a matched pair of escape parenthesis. • If we have to extract a number, we need to specify \([0-9]*\). • If we need to extract non-alphabetic characters, we need to specify \([^A-Za-z]*\). • Every grouped pattern automatically acquires the numeric label n, where n signifies the nth group from the left. 90

15

9/3/2012

sed Regular Expression: Tagged Regular Expressions (continued)

r9

Knowledge check 1.What is sed? 2.What is a sed Regular Expression? 3.What are the different types of Regular Expressions for sed?

Explanation: The matches from pattern space for regular expressions specified within a pair of parenthesis are remembered by sed and stored in variables named 1, 2, and so on. Hence, we can access these matches by specifying these numbers, preceded by a backslash, “\”.

91

92

Module agenda • awk • awk options Module 2: Advanced Filters: awk

– Urinary – Arithmetic – Relational – Logical

• awk built-in variables

94

awk

The awk options

awk is an external command. It is known as an advanced filter. It has a built-in interpreter, which interprets awk script. The awk script could be written using awk programming language. • The awk script could either be specified within a pair of quotes, or could be written into a file, and this file could be specified with the awk command. • Syntax:

• e – This is used to specify multiple scripts with a single awk command. • F – This is used to specify Input Field Separator. • f – This is used to specify a file that contains awk script. • v – This is used to pass shell variable to awk environment.

• • • •

– awk [options] ‘ { }’

95

96

16

Slide 92 r9

This is a new slide. rbtolentino, 11/10/2009

9/3/2012

The awk operators: Urinary • - This is the unary minus. • ++ This is the increment operator. • -- This is the decrement operator.

The awk operators: Arithmetic • Binary operators: –+ This is the addition operator. – - This is the subtraction operator. –* This is the multiplication operator. – / This is the division operator. –% This is the modulus operator. –^ This is the exponentiation operator.

97

98

The awk operators: Arithmetic (continued)

The awk operators: Relational

• Arithmetic assignment operators – +=, -=, *=, /=, %=, ^= – Let us suppose, we need to specify the following arithmetic statement in our code: $sum = $sum + $num; In the above statement, we realize that both the left-hand side and right-hand side have the same operand, $sum. On such instances, we can use arithmetic assignment operators. Using the arithmetic assignment operator +=, the above statement can be rewritten as: $sum += $num; All the other arithmetic assignment operator operates in a similar manner.

• Assignment operator = This is the general assignment operator. == This the “equal to” operator. != This the “ not equal to” operator. < This the “less than” operator. <= This the “ less than or equal to” operator. > This the “greater than” operator. >= This the “greater than or equal to” operator. ~ This is the pattern binding equal to operator. !~ This the pattern binding not equal to operator.

99

100

The awk operators: Logical

The awk operators: Examples (1)

• ! This is the “Not” operator. • && This is the logical and operator, also known as short circuit operator. • || This is the logical or operator, also known as short circuit operator.

101

$ cat newemp.data 102:agarwal:manager:GM:12/05/89:34000:32458976:2+:2:2+ 877:Karunakaran:Vice Chairman:Chairman:15/08/74:1999:28456789:1:2+:2+ 150:praveen:president:Chairman:05/08/83:34323:67894545:1:1:2+ 893:kalam:Scientist:Sr. Scientist:15/08/68:34738:45454434:2:2:2 783:raman:vp:President:13/12/85:1933:45342323:2:2:2+ 777:karthik:president:Chairman:30/03/87:78343:9845475815:2:2:2 989:lakshmi:clerk:manager:12/12/81:78233:199:2:2+:1 102:mohan:manager:GM:31/12/82:34000:56443433:2:2:1 199:sumesh:GM:VP:15/12/81:78343:89454545:2:2:2 222:aggrawal:MANAGER:8945.44:894.54

102

17

9/3/2012



The awk operators: Examples (2)

The awk operators: Examples (3)

The following awk code displays employee number, employee name, and salary of all employees whose salary is between 30000 and 40000.

• Long-listing output of ls command is piped to awk; awk adds value in the file size field of every line from ls output, and displays sum of sizes of all files in the specified directory.

$ awk -F":" '$6>=30000 && $6<=40000 { print $1, $2, $6 }' newemp.data 102 agarwal 34000 150 praveen 34323 893 kalam 34738 102 mohan 34000

$ ls -l Sample total 88 drwxr-xr-x 8 murali billjoy 4096 2009-12-16 14:00 blib -rw-r--r-- 1 murali billjoy 145 2009-12-16 13:53 Changes drwxr-xr-x 2 murali billjoy 4096 2009-12-16 14:07 lib -rw-r--r-- 1 murali billjoy 22330 2009-12-16 14:05 Makefile -rw-r--r-- 1 murali billjoy 535 2009-12-16 13:53 Makefile.PL -rw-r--r-- 1 murali billjoy 61 2009-12-16 13:53 MANIFEST -rw-r--r-- 1 murali billjoy 0 2009-12-16 14:07 pm_to_blib -rw-r--r-- 1 murali billjoy 1167 2009-12-16 13:53 README drwxr-xr-x 2 murali billjoy 4096 2009-12-16 13:53 t

103

104

The awk operators: Examples (4)

The awk built-in Field variables

• Long-listing output of ls command is piped to awk; awk adds value in the file size field of every line from ls output, and displays sum of sizes of all files in the specified directory. $ ls -l Sample/ | awk -F" " '{ sum += $5 } \ END { print "Sum of sizes of all files in current directory is ",sum}' Sum of sizes of all files in current directory is 36526

Explanation:

• $0 – This contains entire record. • $1 – This contains first field, based on field delimiter. • $2 – This contains second field. • $x – For x is a positive integer, this contains the xth field.

The ls command with –l option pipes long-listing of files in “Sample” to awk; for every line of ls command output read by awk, awk adds value of fifth field, $5 to the variable sum and stores the result in the variable sum. In this code, it can be observed that we used arithmetic assignment operator, +=.

105

106

The awk built-in variables

The awk built-in variables (continued)

• ARGC – This is the number of command line arguments. • ARGV – This is the array of command line arguments; array is indexed from 0 to ARGC – 1. • ARGIND – This is the index in ARGV of the current file being processed. • ERRNO – If a system error occurs either doing a redirection for getline, during a read for getline, or during a close(), then ERRNO will contain a string describing the error. • FILENAME – This is the name of the current input file. 107

• FNR – This is the input record number in current input file. • FS – This is the input field separator, that is a space by default. • IGNORECASE – This controls the case-sensitivity of all regular expression and string operations. If IGNORECASE has a non-zero value, then string comparisons and pattern matching in rules, field splitting with FS, record separating with RS, regular expression matching with ~ and !~, and the gensub(), gsub(), index(), match(), split(), and sub() built-in functions all ignorecase when doing regular expression operations.

108

18

9/3/2012

The awk built-in variables: Examples (1) $ awk 'BEGIN { for (i=0; i
Explanation: The variable ARGC stores the number of command line arguments, and the variable ARGV stores the list of arguments. Both the number of arguments and list of arguments values are inclusive of the awk command itself.

The awk built-in variables: Examples (2) $ awk -F":" 'BEGIN {IGNORECASE=1; } $3 ~ /manager/ { print }' newemp.lst 839:hari:gen manager:8934 888:jill:Manager:83332 744:smith:MANAGER:32333

Explanation: When the awk built-in variable IGNORECASE is assigned a non-zero variable, during the pattern match, case of the pattern is ignored; hence, the output.

109

110

r11

Knowledge check 1.What is the syntax for awk? 2.What option is used to specify Input Field Separator? 3.What built-in variable is used to specify the number of command line arguments?

Module 3: The Shell

111

r12

Module agenda

Unix Shell • Shell is one of the components of UNIX. The other components are Kernel and File System.

• Unix Shell • Linux Shell

• Shell provides user interface.

– Internal commands – External commands – Shell variables – Customizing user’s shell environment – Enhanced features in Bash

• Shell interprets commands issued by the user to the Kernel. • Shell parses the command line arguments and expands the special meaning of special characters to shell known as meta-characters. • Shell searches for a command in the following places, in the order specified: 1. List of aliases 2. List of keywords 3. List of user-defined functions 4. List of shell built-in functions 5. List of directories specified in PATH variable

113

114

19

Slide 111 r11

This is a new slide. rbtolentino, 11/10/2009

Slide 113 r12

This is a new slide. rbtolentino, 11/9/2009

9/3/2012

Unix Shell (continued)

Unix Shell (continued)

• Bourne shell (sh) - This is the original Unix shell written by Steve Bourne of Bell Labs. It is available on all UNIX systems. • Bash – This is written by Free Software Foundation under their GNU initiative. It is intended to be a full implementation of the IEEE Posix Shell and Tools specification. • korn shell (ksh) – This is written by David Korn of Bell labs. It is provided as standard shell on Unix systems.

• C shell (csh) – This is written by Bill Joy, at University of California, Berkley; it provides c language like syntax. • Enhanced C shell (tcsh) - It provides all the features of the C shell together with emacs style editing of the command line. • Z shell (zsh) - Zsh is an extended Bourne shell with a large number of improvements, including some of the most useful features of bash, ksh, and tcsh.

115

116

Unix Shell: Summary of shell facilities

Linux Shell

Bourne

C

TC

Korn

BASH

Command history

No

Yes

Yes

Yes

Yes

Command alias

No

Yes

Yes

Yes

Yes

Shell scripts

Yes

Yes

Yes

Yes

Yes

Filename completion

No

Yes*

Yes

Yes*

Yes

Command line editing

No

No

Yes

Yes*

Yes

Job control

No

Yes

Yes

Yes

Yes

• The default shell in Linux is Bash. The other shells are also available in Linux. • Command interpretation – Shell accepts command from user and interprets it to the kernel. – Hence, it’s known as command interpreter.

* This denotes that it is not the default setting for this shell 117

118

Linux Shell: Internal commands

Linux Shell: Internal commands: Examples

• These are commands that reside as built-in functions in the shell program, are known as internal commands. • These commands are available in the memory throughout the login session. • Once the user logs in, along with the shell program, these built-in functions are also loaded to the memory, and they reside in memory until the user logs off. • When a user executes an internal command, the shell executes the corresponding built-in function that resides in the memory; for instance, for executing internal command, memory look-up is made by shell. 119

• • • • • •

echo . or source type cd umask printf

120

20

9/3/2012

Linux Shell: External commands: Examples

Linux Shell: External commands • These are commands that reside as binary executables, shell scripts, awk scripts, or sed scripts in the secondary storage device as files, are known as external commands. • When a user executes an external command, the shell has to search for the command in the secondary storage device.

• • • • • • •

cat who ls grep awk myshell.sh myscript.pl

121

122

Linux Shell: External commands: Problem in executing

Linux Shell: Quoting and escaping

If you are unable to execute an External command, check the following: • The directory in which the external command (program file) is located, is included as one of the directories in the PATH variable. • The external command program file has read and execute permission for the user. These conditions should be satisfied in order to execute an External command.

• Strings can be delimited between either single quotes or double quotes. • Special meaning of variables and escape sequences is interpreted by shell if only they are either unquoted or double-quoted. • Special meaning of $ and other special characters can be removed by preceding these characters by \ .

123

124

Linux Shell: Quoting and escaping: Examples

Linux Shell: Shell variables

$ echo “value of \$HOME is /home/murali” Value of $HOME is /home/murali $ echo “\\ is the escape character…” \ is the escape character $ echo ‘my name is \t$name’ my name is \t$name $ echo “my name is \t$name”

my name is

Thomas Alva Edison

Explanation: Within a pair of double quotes are special meaning of characters.

125

• There are two types of shell variables, namely, predefined shell variables and user-defined shell variables. • The built-in variables reside in the environment right from the time the user logs in. • The user-defined variables created during a log-in session, exists in the environment till the user logs out. • When the user creates a sub-shell, the variables are not inherited to the sub-shell. • If the variables have to be inherited to the sub-shells, then the user needs to issue the command: $ export … 126

21

9/3/2012

Linux Shell: Shell variables (continued) • A shell variable can be removed from the environment with the following command: unset … • A shell variable can be made read only by issuing the following command: readonly … • A read-only shell variable can neither be removed from the environment nor its value can be changed.

Linux Shell: Shell variables: Built-in variables • PATH – This stores a list of absolute pathname of directories delimited by :, which are searched from left to right by the shell, when user issues an external command. • MAIL – This stores the absolute pathname of the file which stores the user’s mails. • MAILCHECK – This stores the number of seconds in the interval of which the mail daemon process checks for the arrival of new mails in the user’s mailbox.

127

128

Linux Shell: Shell variables: Built-in variables (continued)

Linux Shell: Shell variables: Commands displaying variables

• HOME – This stores the absolute pathname of user’s home directory. • PS1 – This stores the user’s primary prompt. • PS2 – This stores the user’s secondary prompt. • LOGNAME – This stores the user’s login name. • PWD – This stores the user’s working directory. • To display the content of a variable, we have to issue the following command: echo ${}

• env – This displays the list of built-in shell variables and their values. • set – This displays the list of built-in shell variables and user-defined variables with their values and user-defined function definitions, in bash environment. • set – This displays the list of built-in shell variables and user defined variables with their values in korn shell environment.

129

Linux Shell: Customizing user’s shell environment • User’s shell environment can be customized by changing the default values of the built-in variables. • Examples: Explanation: The shell displays value stored in $ echo $PS1 variable PS1 as primary prompt for the $ PS1=“Hello “ shell. $ echo $PS1 We can append any desired directory to Hello PATH variable, as indicated in the code example. $ PATH=$PATH:$HOME $ echo $PATH /usr/kerberos/bin:/usr/bin:/bin:/home/manoj/bin: /home/manoj

131

130

Linux Shell: Customizing user’s shell environment (continued) The shell’s start-up scripts in the Linux environment are: • /etc/bashrc or /etc/environment – universal start-up script. The commands in this script would affect all the user’s shell environment. This script can be edited only by the super user. • $HOME/.bashrc or $HOME/.kshrc– bash / korn specific script, which can be customized by any user • $HOME/.bash_profile or $HOME/.profile – bash or korn specific script, where the environment variables are set These scripts are run by the shell in the order given, when the user logs in. 132

22

9/3/2012

Linux Shell: Customizing user’s shell environment (continued) • Aliases – Aliases are short-names for the commands. A long command may be required to be executed very often. The solution for this is to create an alias. • Syntax: $ alias myfind='find $HOME -name "*.sh" -print 2> /dev/null' • The above alias would reside in the memory as long as the user is logged in. During the next log in, this alias myfind won’t be available. We can make this alias available in all login sessions by adding this alias command in the start-up script, $HOME/.bashrc (in bash) or $HOME/.kshrc (in korn shell) . 133

Linux Shell: Customizing user’s shell environment (continued) • Example: $ set -o noclobber $ cat > animals -bash: animals: cannot overwrite existing file $ cat >| animals jackal fox donkey

• The above example shows that we can override the noclobber setting by succeeding > symbol with a | symbol. 135

Linux Shell: Set commands (continued) The Set commands to customize shell environment are the following: • $ set -o vi When the above command is issued, command line is enabled with vi editing capabilities. • $ set +o vi When the above command is issued, command line vi editing capability is disabled. 137

Linux Shell: Customizing user’s shell environment (continued) • Example: $ set -o noclobber $ cat > animals -bash: animals: cannot overwrite existing file $ set +o noclobber $ cat > animals lion tiger giraffee leopard 134

Linux Shell: Set commands The Set commands to customize shell environment are the following: • The setting ‘set –o noclobber’ can be unset by issuing the following command: set +o noclobber • $ set -o ignoreeof When the user presses key on his login prompt, the user is logged off. Issuing the above command ensures that the user is NOT logged off if he presses on his login prompt. • $ set +o ignoreeof The above command when issued, would unset the ‘ignoreeof’ setting. 136

Linux Shell: Tips in customizing user’s shell environment • The customizations made during a particular login session would exist only for that session. • To make the customizations permanent, we need to do these customizations in shell’s start-up script, .bash_profile and .bashrc in Linux. • Changes to PATH variable or any other variable could be done in .profile in AIX or .bash_profile in Linux. • Aliases, functions, and set commands could be defined in .bashrc in Linux. 138

23

9/3/2012

Linux Shell: Enhanced Bash features: Command history • The shell stores the commands that the user is working during a particular login session, in the memory. • Once the user logs out, the commands that are stored in the memory are appended to a file, .bash_history. • The number of commands that could be stored in memory is based on the value of the variable HISTSIZE. • The number of commands that could be stored in the file .bash_history is based on the value of the variable HISTFILESIZE. • These variables can be edited.

Linux Shell: Enhanced Bash features: Command history (continued) • Example: $ echo $HISTSIZE 1000 # Limit of no. of commands stored in RAM : 1000 $ HISTSIZE=1500 $ echo $HISTSIZE 1500 # Limit increased $ echo $HISTFILESIZE # No. of commands that could be stored in .bash_history 1000 $ HISTFILESIZE=1500 # Increase the limit to 1500 commands $ echo $HISTFILESIZE 1500

139

Linux Shell: Enhanced Bash features: Event recall • We can recall the previously executed commands using up and down arrow keys. • We can recall an event from the history. • To recall a previous event, we need to use the ! operator. • We can recall an event either by specifying the even number or a first few characters of the even name, for instance, command name. • History command displays the events from the history of commands, with their event numbers.

140

Linux Shell: Enhanced Bash features: Event recall (continued) • Example: $ !1001 who # then, we can issue this command. murali pts/1 2009-11-11 13:11 (9.122.198.19) user1 pts/2 2009-11-11 12:07 (9.126.10.57)

• If history command shows who command as command with event number 1001, then, issuing the command “!1001” would display the command, and then also execute that command.

141

Linux Shell: Enhanced Bash features: Event recall (continued) • When the command “!1001” is executed, the shell brings the command with event number 1001 as the latest command, displays it, and then executes it. • If the user only wants to print it, but not execute it, then the user can issue the command in the following way: $ !991:p echo $HISTFILESIZE

• Specifying the p option after a ‘:’ indicates to the shell not to execute it, but only to bring it as the latest command in the history and only print it. • Subsequently, if the user wants to execute it, the user can press up arrow key (to recall the just executed previous command), and press enter key. 143

142

Linux Shell: Enhanced Bash features: Command completion A user types in the first few characters of a command and presses key. • If there is only one command starting with these characters, the shell produces the complete command name on the command line when the user presses the key. This feature is known as command completion. • If there is more than one command starting with these characters, then, the user should press key once again. After this, the shell displays the list of all those commands that start with these characters, aiding the user to easily select the desired command. This is also a command completion. • Example of command completion: $ loca local locale localedef locate $ locate who (The output of locate command will be displayed)

144

24

9/3/2012

Linux Shell: Enhanced Bash features: File name completion

Linux Shell: Enhanced Bash features: Var name completion

• A user types first few characters of file name. and presses the key. If there’s only one file starting with these characters in that directory, then, the entire file name is completed by the shell. This phenomenon is known as file name completion. • Example of file name completion:

• While accessing a variable, when the user types the first few characters of the variable preceded by a $ symbol, and if there is only one variable name starting with these characters, then the shell completes this variable name. This phenomenon Is known as variable name completion. • Example of variable name completion: $ echo $MAIL $MAIL $MAILCHECK $ echo $MAILCHECK 60

$ cat Rectangle. Rectangle.class Rectangle.java $ cat Rectangle.java

145

146

Knowledge check

Session summary

1.What is the original Unix shell that is available on all UNIX systems? 2.State one main difference between the Internal and External command? 3.What feature in Linux shell allows us to obtain the previously executed commands using up and down arrow keys?

Having completed this session, you should be able to: • Use the sed (Stream editor) • Use regular expressions with sed • Use the awk advance filter with the proper options, operators, and builtin variables • Use and customize the Linux shell with the proper commands and variables and utilize its new features.

147

148

Exercise: Create the following file, using vi editor.

Exercise: sed

62:Mike Harrington:(510) 548-1278:250:100:175 63:Christian Dobbins:(408) 538-2358:155:90:201 64:Susan Dalsass:(206) 654-6279:250:60:50 65:Archie McNichol:(206) 548-1348:250:100:175 66:Jody Savage:(206) 548-1278:15:188:150 67:Guy Quigley:(916) 343-6410:250:100:175 68:Dan Savage:(406) 298-7744:450:300:275 69:Nancy McNeil:(206) 548-1278:250:80:75 70:John Goldenrod:(916) 348-4278:250:100:175 71:Chet Main:(510) 548-5258:50:95:135 72:Tom Savage:(408) 926-3456:250:168:200 149

1.Print all lines containing the string San. 2.Print all lines where the person's first name starts with J. 3.Print all lines ending in 700. 4.Print all lines that don't contain 834. 5.Print all lines where birthdays are in December.

150

25

9/3/2012

Exercise: sed (continued)

Exercise: awk

6.Print all lines where the phone number is in the 408 area code. 7.Print all lines containing an uppercase letter, followed by four lowercase letters, a comma, a space, and one uppercase letter. 8.Print lines where the last name begins with K or k. 9.Print lines preceded by a line number where the salary is a six-figure number.

1. Print the first and last names of those who contributed more than $100 in the second month. 2. Print the names and phone numbers of those who contributed less than $85 in the last month. 3. Print the names of those who contributed between $75 and $150 in the first month. 4. Print the names of those who contributed less than $800 over the three-month period. 5. Print the names and addresses of those with an average monthly contribution greater than $200.

151

152

Exercise: awk (continued) 6.Print the first name of those not in the 916 area code. 7.Print each record preceded by the number of the record. 8.Print the name and total contribution of each person. 9.Add $10 to Chet's second contribution. 10.Change Nancy McNeil's name to Louise McInnes.

Linux (Day 3 session)

153

154

Session objectives

Course content (Day 3)

By the end of this session, you should be able to: • Execute shell script • Use kill, basename, exec, eval, expr, and let commands • Create and use Arrays • Use functions and nested functions • Create library files 155

• Module 1: Advanced shell scripting • Module 2: Arrays • Module 3: Functions

156

26

9/3/2012

Module agenda • Shell and sub shell • Execute a shell script in background Module 1: Advanced Shell Scripting

– basename command – exec command – eval command – expr command – let command

158

Shell and sub shell

• Let us assume that we have a shell script called createVars.sh, which is as below: planet=Mercury flower=Lotus

• When we execute the above shell script in one of the below methods, the shell script is executed in sub shell’s environment. $ bash dispSysInfo.sh $ ./dispSysInfo.sh $ dispSysInfo.sh

• As the script is executed in sub shell, the variables that are created in the script are not available to the current shell.

Shell and sub shell (continued) • If the variables are to be created in current shell, then we need to execute the script in the current shell’s environment. • For executing the script in the current shell’s environment, we need to execute the script with an internal command known as . (dot) command or source command. • Syntax: $ . createVars.sh (or) $ source createVars.sh

• When the script is executed in the current shell’s environment, the variables created in the script are also available in the current shell.

159

160

Shell and sub shell (continued)

Shell and sub shell (continued)

• When we specify a set of commands within a pair of parenthesis, then these commands are executed in sub shell’s environment. • Let us consider the following example: $ (cd cppdir; pwd; ) /home/murali/cppdir $ pwd /home/murali

• As the cd command is executed in the sub shell’s environment, the working directory is changed to cppdir in the sub shell’s environment. As the pwd command is also executed in the same sub shell’s environment, it displays the working directory as cppdir.

161

• But once both the commands are executed, the control is returned back to the login shell. • The login shell is unaffected by the change of directory made in the sub shell. • Hence, pwd command when issued in the $ prompt displays the user’s home directory.

162

27

9/3/2012

Shell and sub shell (continued) • When we specify a set of commands within a pair of curly braces, then these commands are executed in the current shell’s environment. • Let us consider the following example: $ { cd cppdir; pwd; } /home/murali/cppdir $ pwd /home/murali/cppdir

• In the above example, as the cd command and the pwd command are executed in the current shell, the working directory is changed to /home/murali/cppdir in the current shell. Hence, the above output.

Execute a shell script in background • A process which is not user-interactive, does not need the terminal, and hence could be executed in the background. • When a shell script is non user-interactive (for example, it may be a script which reads information from system, and updates a logfile) it could be run in background.

163

• Example: Let us consider we have a shell script as below:

Execute a shell script in background (continued)

Execute a shell script in background (continued)

• To execute the script mybgscript.sh in the background, we need to execute it in the following way:

164

• We can control a background process by sending a signal to it using the kill command. • Syntax of kill command: kill –s kill –s %

$ sh mybgscript.sh & [1] 7636

• The kernel allocates a pid for the script, and the script is run in the background. The shell displays the pid and the job-ID of the background process. • The job command also displays information about user-initiated background jobs and

• For instance, to suspend a background process with a job-id 1, we could issue the following command: $ kill -s TSTP %1 [1]+ Stopped

sh mybgscript.sh

165

166

Execute a shell script in background (continued)

Execute a shell script in background (continued)

• Suppose the user x has initiated a script called as script1.sh, to execute in background, which periodically updates a logfile, with the file system status. • Suppose that the user x logs out of his session after some time, forgetting about the background process he started. Now, what would happen to this background process? • Before the login shell terminates, it sends HUP signal to all its child processes, to terminate them. • Hence, the script script1.sh is also terminated abruptly. • If the script script1.sh has to execute without any interruption even after the user x logs out, then the script has to be initiated in the background using nohup command, which is illustrated in the next slide. 167

$ nohup sh mybgscript.sh & [1] 9343 $ jobs [1]+ Running nohup sh mybgscript.sh &

• Now, even if the user x logs out, the script would continue to execute uninterruptedly. • In this case, before the shell terminates, the shell puts a request to the init process to take parentage of the background child process. • Once init process takes up the parentage, the login shell terminates. 168

28

9/3/2012

The basename command • Syntax:

The basename command: Example $ basename unixnotes.text '.text‘ unixnotes $ basename /opt/freeware/bin bin

basename (or) basename

• When a “/” separated pathname is specified with basename command, then the command extracts the filename specified after the rightmost “/”, and displays it on the standard output. • When a filename is specified as first argument and a quoted suffix is specified as the second argument, basename command removes suffix from the first argument, and returns the resultant prefix. 169

170

The exec command

The eval command

• The exec command, when used with input redirection operator, <, loads the specified disk file into the buffer associated with the standard input device, keyboard; it also sets temporarily the standard input device as the disk file. • Syntax: exec < employee.data

• Now, read command reads one line from the buffer and assigns to the variable specified, rather than issuing input interrupt. • If we are giving the above statement in a script, and if in the same script, subsequently, we want to take input from the user, then we need to re-assign the standard input file as the keyboard. This could be done as follows: exec < /dev/tty

• The eval command parses the command line arguments twice. • Let us understand the purpose of eval command with the help of the following example: $ hb=planet $ planet=mercury $ eval echo \$$hb mercury

• During the first parse, special meaning of first $ is not interpreted as it is preceded by escape character, “\”; and hence, shell gets the value of $hb, namely planet. • During the second parse, the “\” character is ignored, and hence $planet is evaluated to mercury; hence the result.

171

172

The eval command: Example

The expr command

• Let us consider another example: $ prompt1="Enter emp number“ $ prompt2="Enter emp name“ $ i=1 $ eval echo \$prompt$i Enter emp number $ i=2 $ eval echo \$prompt$i Enter emp name

• We can perform arithmetic operations as well as string operations with expr. • Following are the arithmetic operators supported by expr: + * / %

173

: It performs addition : It performs subtraction : It performs multiplication : It performs division : It performs modulus operation 174

29

9/3/2012

The expr command (continued) • String length:

The expr command (continued) • Length of matching substring at the beginning of string:

$ echo ${#str} 10 $ expr length $str 10 $ expr $str : '.*‘ 10

$ str=abcABC123ABCabc $ expr match $str 'abc[A-Z]*.2‘ 8 $ expr $str : 'abc[A-Z]*.2‘ 8

• Index: $ str=IBM_Values $ expr index $str "Val“ 5 175

176

The expr command (continued)

The let command • The let command allows us to perform arithmetic operations. • Syntax:

• Substring extraction: $ str="Much ado about nothing“ $ echo ${str:5} ado about nothing $ echo ${str:5:3} ado $ str="IBM_Values“ $ expr substr $str 5 3 Val

let var=value (or) let var=

• The let command of bash supports the following operators:

177

The let command (continued) • The let command of bash supports the following operators: – ++ – --

This is for prefix and postfix increment operator. This is for prefix and postfix decrement operator.

• The following are the arithmetic operators supported: – – – – –

+= -= *= /= %=

• Limitation of let command: – We can not perform floating point arithmetic with let command. 179

–+ ––* –/ –%

This is for addition. This is for subtraction. This is for multiplication. This is for division. 178 This is for modulus.

Knowledge check 1.When we specify a set of commands within a pair of parenthesis, these commands are executed in the current shell’s environment. True or false? 2.What command allows us to perform arithmetic operations? 3.What command parses the command line arguments twice? 180

30

9/3/2012

Module agenda

Module 2: Arrays

• Definition • Array size • Adding Array elements

182

Definition • An array is a collection of similar elements, stored in contiguous memory area. • Shell supports only single dimensional arrays. • An array is defined as follows: $ planets=(Mercury Venus Earth Mars)

• Elements in the array can be listed as follows:

Array size • Array elements have subscripts starting from 0. To access the second element of the array, we could issue the command as follows: $ echo ${planets[1]} Venus

• To find the number of elements in the array or array size, we can issue the following command: $ echo ${#planets[@]} 4 $ echo ${#planets[0]} 7

$ echo ${planets[@]} Mercury Venus Earth Mars

183

184

Adding Array elements

Knowledge check

• We can add some more elements to the array, as follows: $ planets[4]=Jupiter $ planets[5]=Saturn

• Now, elements in the array are: $ echo ${planets[@]} Mercury Venus Earth Mars Jupiter Saturn Uranus

1.What is an Array? 2.How will you access the 5th element of the array kingdoms? 3.How can you add the element Finertia to the array of size 9, kingdoms?

• Size of the array is: $ echo ${#planets[@]} 7

185

186

31

9/3/2012

Module agenda

Module 3: Functions

• • • • •

Function Library file Passing parameters to a function Functions in all log-in sessions Nested functions

188

Function • A function is an independent named block of code that could be called by any application. • A function executes in its own runtime environment. • A function is defined as follows: $ myfun() >{ > echo "Hello world“ >} $ myfun Hello world

Library file • A library file can be created as follows: $ cat function.sh myadd() { echo "scale=2;$1+$2;"|bc } mymul() { echo "scale=2;$1*$2;"|bc }

189

190

Library file (continued)

Passing parameters to a function

• The library file function.sh can be included in any script application which needs to use those functions. • Let us look at an example: $ cat mymain.sh if [ $# -ne 2 ]; then echo "Usage: $0 “; exit 1 fi . function.sh echo "Sum of $1 and $2 is $(myadd $1 $2)“ echo "Product of $1 and $2 is $(mymul $1 $2)"

191

• Each of the parameters that passed through a function call should be separated by a space. • Inside the function, these parameters are accessible in the built-in variables of the function, namely $1, $2, $3, and so on. • $# stores the number of arguments and $* and $@ stores list of arguments passed to function. 192

32

9/3/2012

Functions in all log-in sessions • If the user wants to have a set of functions to be available in all of his or her login sessions, the user can specify these functions in the start-up script of the shell, $HOME/.bashrc • For example, let us suppose the user adds these functions in the file $HOME/.bashrc: $ cat .bashrc # .bashrc # Source global definitions if [ -f /etc/bashrc ]; then . /etc/bashrc fi

193

Nested functions • A function can be called from another function. This is known as nesting of functions. • Let us consider the following example: greater () { if [ $1 -gt $2 ]; then echo $1 is greater than $2; elif [ $2 -gt $1 ]; then echo $2 is greater than $1; else echo Both $1 and $2 are equal...; fi;

fi }

Functions in all log-in sessions (continued) $ cat .bashrc previous slide…

# output continued from # User specific aliases and functions

mychmod() { # to add execute permission for any shell chmod u+x $1 # script, to user, the user needs to specify } # as: mychmod lpdfile() { # this function displays the latest modified file 194 ls –t | head -1 }

Nested functions (continued) display_gr () { echo "The status of comparison of $1 and $2 is as follows..."; greater $1 $2 } $ display_gr 38 38

The status of comparison of 38 and 38 is as follows: Both 38 and 38 are equal... $ display_gr -40 0

The status of comparison of -40 and 0 is as follows:

195

196

0 is greater than -40

Knowledge check 1.What is a function? 2.What should be done to have a set of functions available in all login sessions? 3.What is a nested function?

197

Exercise 1 1. Write a script that finds a new file created, every minute, in any of the subdirectories of your home directory, and writes that information in a logfile called fileCreated.data. This script should be executed in the background. 2. What would you do if you don’t want the script to be terminated even after you log out? 3. Write a script that would accept a pattern and a set of filenames as command line arguments; the script should tell the user as to how many times the pattern occurs in each of these files. Write appropriate code to check that the user has entered the right number of arguments and give appropriate usage message. Put this code in a function. 198

33

9/3/2012

Exercise 2 4. Write a shell script to accept employee number, emp name, job, doj, basic pay from the user. This process has to be done repeatedly wherein before terminating every iteration, it will ask the user if he or she wants to enter another emp details, and based on the user response take decision about continuing the iteration. Check the following conditions: – The emp number should not be null; it should be unique. – The emp name should not be null, and it should not contain any other characters other than alphabets, white spaces and . (dot) characters. – The job should not be null. – For doj, accept date, month, and year as separate inputs, check for the validity in the doj inputs, and concatenate dd, mm and yy with '/' character – The basic pay input should contain only floating point values. Basic pay should not be less than 6000.00. 199

Thank you for attending 200

34

Linux All day -

All course content for day 1 was converged in a single slide. Sub bulletpoints ... During 1977 to 1982, UNIX System III was released by .... which point to data blocks; Number of data blocks addressed: 128 .... List of operators and their purpose:.

905KB Sizes 2 Downloads 233 Views

Recommend Documents

All Night, All Day (preview).pdf
FASTER (WITH A BACK-BEAT). 8. S. A. T. B. Pno...... .... All Night, All Day (preview).pdf. All Night, All Day (preview).pdf. Open. Extract. Open with.

All Summer in a Day
They were all nine years old, and if there had been a day, seven years ago, .... They stopped running and stood in the great jungle that covered Venus, that grew ...

1002-E -All Day Kindergarten.pdf
There was a problem previewing this document. Retrying... Download. Connect more apps... Try one of the apps below to open or edit this item. 1002-E -All Day ...

Axxia Linux Changes Linux 1.59 Changes Linux 1.58 ... - GitHub
Add a driver to 5600 and 6700 for OEM function calls to the secure monitor. This driver ... Remove the unused Ethernet interface in 5600 simulation device trees.

Axxia Linux 3.10 Changes Linux 8.8.1.65 Changes Linux 8.8 ... - GitHub
Support for big endian on ARM (5500). • MSI support on both PCIe ... when writing to it. • GPIO Changes. – Remove GPIO platform data, use the device tree. 5 ...

GABLE - ALL DAY FOOD MENU - MAY18.pdf
FISH FINGER BRIOCHE BUN £8.00. homemade cod goujons, tartare sauce & baby gem. STEAK CIABATTA £9.50. with bacon jam, wholegrain mustard & rocket.

All Summer in a day text.pdf
Retrying... Download. Connect more apps... Try one of the apps below to open or edit this item. All Summer in a day text.pdf. All Summer in a day text.pdf. Open.