quick reference on command line tools and techniques for the people with limited time
grep
find
sed
filter and transform text
awk
cut
, paste
and join
command | description |
---|---|
man |
get help on commands. e.g.man date |
date |
gives the date and time |
cal |
shows the calender |
ls |
tells which files are present in the current working directory when a -l option is used, the command returns owner, size, date of file, persissions, etc. |
cat |
shows contents of a file |
cp |
copy a file |
mv |
move a file |
diff |
lists differences between 2 files |
rm |
removes file |
grep |
find occurences of strings in one or more files |
pwd |
print the present working directory's name |
cd |
changes the current directory |
keyboard shortcut | description |
---|---|
Ctrl+A | move the cursor to the start of the line |
Ctrl+E | move the cursor to the end of the line |
Alt+F | move the cursor forward by one word |
Ctrl+F | move the cursor forward by one character |
Alt+B | move the cursor backward by one word |
Ctrl+B | move the cursor backward by one charater |
Ctrl+XX | toggle the cursor's position between the current position and the previous position |
keyboard shortcut | description |
---|---|
Ctrl+U | cut all the characters to the left of the cursor |
Ctrl+K | cut all the characters to the right of the cursor |
Ctrl+W | cut one word to the left of the cursor |
Ctrl+H | cut one character to the left of the cursor |
Alt+D | cut one word to the right of the cursor |
Ctrl+D | cut one character to the right of the cursor |
Ctrl+Y | paste the cut characters |
Ctrl+_ | undo the last deletion |
Tab | complete arguments or list all available commands |
keyboard shortcut | description |
---|---|
Crtl+R | search the command history |
Ctrl+G | abort |
Ctrl+P/UP | the previous command in the history |
Ctrl+N/DOWN | the next command in the history |
A linux shell's inputs and outputs are sequence of characters called streams. There are three standard I/O streams:
name | description | file descriptor |
---|---|---|
stdout | displays output from commands | 1 |
stderr | displays error from commands | 2 |
stdin | provides inputs to commands | 0 |
Input and output redirections are done using angular brackets (<>)
Bracket type | description |
---|---|
> | send stream to a file. E.g. ls a > o.txt |
>> | append stream to a file. E.g. ls b >> o.txt |
>& | write into stream. E.g. ls c > o2.txt 2>&1 |
< | receive stream from a file. E.g. wc < o.txt |
<< | embed the text that will be fed to a command within the script Example cat << EOF > output.txt line 1 line 2 line 3 EOF echo done also known as heredoc |
<<< | expand variable and feed into the stdin of the command Example wc <<< $word also known as herestring |
command1 > sout.txt 2> serr.txt
command1 > sone.txt 2>&1
OR
command1 &> sone.txt
grep "ABC" <<< $var
command1 &> /dev/null
/dev/null is a null device file. This will discard anything written to it, and will return EOF on reading.
Piping can redirect the standard output of one command to the input of another command.
command1 | command2 paramater1 | command3 parameter1 parameter2 | command4
Examples:
Print file contents only once. remove duplicate records:
sort file1 | uniq
print 5 most frequently used commands
history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head -5
print the file types and their frequencies
ls | rev | cut -f1 -d'.' | rev | sort | uniq -c | sort -n
find . | xargs -n1 basename
-I{}
. the placeholder is usually used when we want to place the std input in the middle of a commandls | xargs -I{} echo {} file is found
Examples:
ls | wc
find . -type f | xargs basename -a | grep "\." | rev | cut -f1 -d'.' | rev | sort | uniq -c | sort -n
ls | xargs -I{} mv {} {}.bkp
find . -name "*.c" -print0 | xargs -0 -n1 -I{} cp {} some/folder/
Examples:
ls | tee fileList.txt
wildcard | description | example | matches | does not match |
---|---|---|---|---|
* | matches 0 or more characters | ls to* | to, tom, ton, tow, tommy, tommie | tata, tea |
? | matches 1 character | ls to? | tom, tow, ton | to, tommy, tommie, tata, tea |
[abc] | matches any one of the characters in the brackets | ls [bc]at | bat, cat | Bat, Cat, rat |
[a-z] | matches any one of the characters within the range in the brackets | ls day[1-9] | day1, day2 upto day9 | day11, day |
[!abc] | matches any one character that is not in the brackets | ls [!r]at | bat, cat, Bat, Cat, Rat | rat |
[!a-z] | matches any one character that is not in the range in the brackets | ls day[!1-9] | day0, days | day1 upto day9 |
{String1, String2}
{<start>..<end>}
{<start>..<end>..<increment>}
<prefix>{ string or range }<postfix>
touch {1..31}
echo {2..100..2}
mkdir /some/path/{input, output, backup, error}
echo {z..a}
cp file.txt{,.bkp}
mv file.txt{.bkp,}
mkdir -p ~/logs/{January/{1..31},February/{1..28},March/{1..31}}
command | description |
---|---|
jobs |
list all the jobs that the current shell is running or has suspended |
fg |
bring job to foreground |
bg |
send job to background |
kill |
terminate job |
stop |
suspend job |
Ctrl+c | terminate job |
Ctrl+z | suspend job |
Examples:
sleep 1000 &
fg %2 # where 2 is the job number
stop %2 # where 2 is the job number
bg %2 # where 2 is the job number
kill %2 # where 2 is the job number
ps
commandps -ef | grep command
bash wait <process id>
kill <process id>
wait
wait 1234 # where 1234 is the process id
ps
and jobs
jobs
tells the list of jobs the current shell is managingps
tells the list of all the processes running in the system$'string
Examples:
examples | command | result |
---|---|---|
no quote | echo $HOME |
/home/user1/ |
escape character | echo \$HOME |
$HOME |
single quote | echo '$HOME' |
$HOME |
double quote | echo "$HOME" |
/home/user1/ |
ANSI C quoting | echo $'There\'s a quote in my string!' |
There's a quote in my string |
ls
command | description |
---|---|
ls |
list the contents of current directory |
ls * |
list contents of the directory along with the subdirectory |
ls -l |
list the contents of a directory along with the owner, permission, date, size |
ls -a |
list hidden file |
ls -t |
list files in descending order of last modified date |
ls -rt |
list files in ascending order of last modified date |
ls -R |
list files of the current directory and the subdirectory recursively to the last subdirectory |
ls /path/to/dirextory |
list files in the directory mentioned |
command | description |
---|---|
cat demo.txt |
show the contents of file. use for relatively small files |
head demo.txt |
show the first part of the file |
tail demo.txt |
show the last part of the file |
tail -f demo.txt |
show the text appended to the file as the file grows |
less demo.txt |
show contents of file one screen at a time |
less -p "regular" demo.txt |
show contents of the file from the first line with which the pattern matches |
strings -a binaryfile |
print the sequence of all the printable characters in the file |
diff file1 file2 |
shows difference between 2 files |
comm file1 file2 |
compare 2 sorted file |
command | description |
---|---|
cp file1 file2 |
copy file1 to file2 |
cp file1 file2 directory1 |
copy file1 and file2 to directory1 |
cp -R directory1 directory2 |
copy contents of directory1 to directory2 |
cp *.txt directory1 |
copy all files ending with .txt to directory1 |
command | description |
---|---|
mv file1 file2 |
rename file1 to file2 |
mv file1 directory1/ |
move file1 to directory1 |
mv *.jpg directory1/ |
move all files ending with .jpg to directory1 |
command | description |
---|---|
rm file1 |
remove file1 |
rm file1 file2 |
remove file1 and file2 |
rm *.png |
remove all files ending with .png |
rm -d emptyDirectory |
remove an empty directory |
rm -r directory1 |
recursively remove all files, subdirectories and directory of directory1 |
command | description |
---|---|
ln file1 link1 |
create a hardlink link1 to file file1 |
ln -s file1 link1 |
create a softlink link1 to file file1 |
command | description |
---|---|
cd |
Change to home directory |
cd ~ |
Change to home directory |
cd - |
Change to the previous directory |
cd .. |
Change to the parent directory |
cd / |
Change to the root directory |
cd ~/dir1/dir2 |
Change to directory relative to home directory |
command | description |
---|---|
du |
estimates and displays the disk space used by the files |
du -h * |
prints size in human readable' |
command | description |
---|---|
chmod |
change permissions |
chown |
change ownership |
variable | description |
---|---|
$0 | name of the shell |
$$ | the process id of the current shell |
$? | the exit status of the last executed command |
$! | the process id of the last background process |
$_ | the last argument of the previous command |
exit code | description |
---|---|
0 | Success |
1 | general error |
126 | command invoked does not have execute permissions |
127 | command not found |
130 | command terminated using Ctrl+C |
grep
grep -<options> <pattern> <filenames>
grep
optionsoption | description |
---|---|
-i | ignore case |
-n | display line numbers along with lines |
-v | display lines that do not match the pattern |
-c | count the number of matching lines |
-r | search in all files under each directory |
-l | diplay the filename of the file which has the matching pattern |
-o | print only the matched string. The whole line with the matched string is not printed |
-I | ignore binary files |
-A<n> | include n lines after match |
-B<n> | include n lines before match |
-C<n> | include n lines before and after the match |
THIS IS UPPER CASE LINE
this is lower case line
This is regular line
This line is also regular
Line number four
Line #5
last line
Search for a string in a file
grep "this" demo.txt
Search for a string in a file, without matching case
grep -i "this" demo.txt
Search for a string in a file and get both the line number and output
grep -n "this" demo.txt
Get the number of lines matching the searched string
grep -c "this" demo.txt
Get the filename in which the searched string is found
grep -l "this" demo.txt
Get 2 lines after the matching line
grep -A2 "This" demo.txt
Get 2 line before the matching line
grep -B2 "This" demo.txt
Get 2 lines before and after the matching line
grep -C2 "This" demo.txt
Search recursively in all files in subdirectories
grep -inrI 'some text' /path/to/dir
character | description |
---|---|
[abc] | matches any one of the characters in the square brackets |
[a-d] | matches any one of the characters in the range specified in the square brackets |
^start | matches the pattern only if the pattern is at the start of the line |
end$ | matches the pattern only if the pattern is at the end of the line |
[^abc] | matches any one character that is NOT present in the square brackets |
[^a-d] | matches any one character that is NOT present in the range |
. | matches any one character |
* | mathces 0 or more occurences of the preceding character |
.* | matches zero or more of any character |
Match any one character
grep "[Tt]his" demo.txt
Search for line starting with the search pattern
grep "^last" demo.txt
Search for line ending with the search patter
grep "regular$" demo.txt
Search for line with a character in specified range
grep "[0-9]" demo.txt
Search for line without a character in specified range
grep "[^0-9]" demo.txt
Search for a line where the middle characters are not known
grep "line.*regular" demo.txt
Note: grep can have regular expressions in the search pattern part, and can have wildcards in the files to search section.
find
find
command is used to search and locate the list of files and directoriesfind [starting point] [expression]
-exec [command]
can be used to run command on the files located by find
find . -name demo.txt
find ./Codes -name *.cpp
find . -name Codes -type d
find . -type f -perm 0777 -print -exec chmod 644 {} \;
find . -type f -name "*.bkp" -exec rm -f {};
sed
filter and transform text sed 'instructions' file
'[address]command/regex/replace/modifier'
sed '1,100 s/A/a/' file.txt
. this restricts substition to first 100 lines.sed -n '/start/,/stop/ p' file.txt
sed -n '/match/ !p' file.txt
command | description |
---|---|
d | delete |
p | |
s | substitute |
q | quit |
a | insert a line after pattern |
i | insert a line before pattern |
c | change a line |
y | transform |
sed '/^$/d' file.txt
sed '/Search/d' file.txt
sed 's/Search//g' file.txt
sed '/Search/ p' file.txt
sed 's/oldString/newString/g' file.txt
sed 's/ *$//' file.txt
sed `s/^ *//' file.txt
sed 's/^/ /' file.txt
sed '10 q' file.txt
sed -n '1,10 p' file.txt
sed '/pattern/ a add line here' file.txt
sed '/pattern/ i add line here' file.txt
sed '/pattern/ c line changed here' file.txt
sed 'y/abc/pqr/' file.txt
sed 's/\([a-z]*\) \([a-z]*\)/\2 \1/' file.txt
sed -n '/\([a-z][a-z]*\) \1/p' file.txt
sed 's/\([a-z][a-z]*\) \1/\1/' file.txt
command | description |
---|---|
x | exchange hold space and pattern space |
h | copy pattern buffer into hold space |
H | append pattern buffer into hold space |
g | copy hold space to pattern space |
G | append hold buffer into pattern buffer |
sed -n '/999/ !{x;d};/999/ {x;p;x;p;n;p}' file.txt
sed 'G' file.txt
sed '/start/ {x;p;x}' file.txt
sed '/start/ {G}' file.txt
sed '/start/ {x;p;x;G}' file.txt
awk
pattern { action }
/^HTTP/ {print}
awk 'BEGIN{print "start"} {print} END{print "end"}' file.txt
var[key] = value
action | description |
---|---|
{ print $0; } | print records |
{ exit; } | ends program |
{ next; } | skips current line |
{a=$1; b="X"} | variable assignment |
{ c[$1] = $2 } | array varaible assignment |
{if (condition) { action } else if (condition) { action } else { action }} | if else conditions |
{ for (i=1; i < x; i++) { action } } | for loop |
{ for (item in c) { action } } | for loop iterating over a list |
variable | desc |
---|---|
FS | Input field separator. can be modified |
RS | Input record separator. default value is newline. can be modified by user |
OFS | Output field separator. can be modified |
ORS | Output record separator. default value is newline. can be modified by user |
NF | Number of fields in the current line (record). cannot be updated by user |
NR | Number of lines processed so far. cannot be updated by user |
Note: -F option can be used to update the input field separator -> awk -F":"'{ print $1 }' file.txt
awk -F"," '{print $2}' file.txt
awk -F"," '{if ($2)print $3}' file.txt
awk -F"," '{ print $NF }' file.txt
awk '/pattern/ { i=1; next; } {if(i) {i--; print;}}' file.txt
awk '/regexp/ {i=3;} { if(i) {i--; print;}}' file.txt
awk '/start/,/stop/' file.txt
awk 'END{print NR}' file.txt
awk '/pattern/'
awk '!/pattern/'
awk 'a !~ $0 {print}; {a=$0}' file.txt
awk 'NR < 11' file.txt
awk '{vect[NR]=$0;} END{for(i=NR-9;i<=NR;i++) {print vect[i];}}' file.txt
ls -l | awk '{ x += $5 } END { print "Total bytes: " x }'
awk 'BEGIN{FS=",";OFS=";"}{print $1, $3}' file.txt
$(( expression ))
echo $((1+3))
`command`
and $(command)
$(commmand)
supports nesting i.e. $(command1 $(command2))
thedate=`date`
vi $(grep -l 123 *)
filedata=$(cat file.txt)
filedata=$(<file.txt)
<(command)
and >(command)
sort and compare two files
diff <(sort file1) <(sort file2)
compare 2 folders
diff <(ls $first_directory) <(ls $second_directory)
export
is used while defining the variable(command1; command2; command3)
#do something in current directory
(cd some/other/directory; other-command)
#back in the original directory
. script.sh
(date; uptime) > file.txt
{date; uptime;} > file.txt
cut
, paste
and join
cut
cut
command cuts out sections from each line and writes result to standard outputcut OPTION [FILE]
option | description |
---|---|
-c | character range which will be selected |
-d | delimiter which will separate each fields in line in file |
-f | fields which will be printed |
when -c option is used, ranges can be specified. Each range can be one of:
range type | description |
---|---|
N | Nth character |
N- | fron Nth character to end of line |
N-M | from Nth character to Mth character |
-M | from first to Mth character |
cut -f1,3 -d"," file.txt
cut -c -3 file.txt
paste
paste [OPTION] [FILE] [FILE]
option | description |
---|---|
-d | used to specify the delimiter |
-s | paste one file at a time |
lets take 2 files - number.txt and name.txt
cat number.txt 1
2
3
4
cat name.txt Alice
Bob
Charlie
David
paste number.txt name.txt
paste -d"," number.txt name.txt
paste -s number.txt name.txt
join
join [OPTIONS] FILE1 FILE2
lets take 2 files - number.txt and name.txt
cat number.txt 1 100
2 101
3 102
4 103
5 104
cat name.txt 1 Alice
2 Bob
3 Charlie
4 David
join number.txt name.txt
aliases are short names for long commands
when we need to execute long commands multiple times, it is advisable to create aliases
syntax - alias [-p] [name[=value]]
we can create permanent alias by storing them in configuration files
to temporarily bypass an alias, use \ -> \ll
creating a aliasalias name='values'
removing aliasunalias name
print all defined aliasalias -p
alias gh='history|grep'
alias c=clear
alias cx='chmod +x'
alias ..='cd ..'
alias sl=ls
alias left='ls -t -1'
alias count='find . -type f | wc -l'
alias f='find . |grep '
export -f functionname
, we can make the functions available to shell scripts
function_name () {
commands
}
function_name () { commands; }
mcd() { mkdir -p "$1"; cd "$1";}
cdl() { cd "$1"; ls;}
option | description |
---|---|
-r | sort in reverse order |
-n | sort in numberical order |
-k |
sort based on nth column |
-u | sort and remove duplicates |
option | description |
---|---|
-c | show how many times a line is repeated |
-d | prints only the repeated lines only once |
-u | prints only the unique lines |
-i | case insensitive comparison |
if [ condition1 ]; then command1;
elif [ condition2 ]; then command2;
else command3; fi
if [ condition1 ]; then command1; elif [ condition2 ]; then command2; else command3; fi
conditions | description |
---|---|
-e | check if file exists |
-r | check if file exists and is readable |
-w | check if file exists and is writable |
-d | check if file exists and is a directory |
conditions | description |
---|---|
== | check if both the strings are equal |
!= | check if both the strings are not equal |
> | check if the first string is lexicographically greater than the second |
< | check if the first string is lexicographically smaller than the second |
-n | check if the string has a length more than 0 |
-z | check if the string is an empty string |
conditions | description |
---|---|
-eq | check if the numbers are equal |
-ne | check if the numbers are not equal |
-gt | check if the first number is greater than the second |
-ge | check if the first number is greater than or equal to the second |
-lt | check if the first number is less than the second |
-le | check if the first number is less than or equal to the second |
command1 && command2
command1 || command2
[ -d ./some/path/folder ] || mkdir /some/path/folder
cd /some/path/folder && touch file.txt
while [ condition ]; do commands; done
find . -name *.c | {while read -r filename; do dirname $filename; done;} | sort | uniq # dirname returns the directory name
for <variable name> in <a list of items>;do <some command> $<variable name>;done;
for file in ./code/*.txt; do cp $file /home/code/backup; done
ssh user@host
ssh user@host command_to_run
ssh -p portnum user@host
ssh -t reachable_host ssh unreachable_host
curl [options] [URL]
options | description |
---|---|
-o | save the output in a file with name as specified after the option |
-O | save the output in a file with the same name as in the url |
-C - | resume a download |
-I | fetch headers of a url |
-L | follow redirects |
curl example.com
curl example.com -o example.html
curl -C - -O https://releases.ubuntu.com/21.10/ubuntu-21.10-desktop-amd64.iso
curl -I example.com
curl wttr.in/london
curl https://reqres.in/api/users/2
curl -X POST -H "Content-Type: application/json" -d '{"email": "[email protected]","password": "pistol"}' https://reqres.in/api/register
curl -X PUT -H "Content-Type: application/json" -d '{"name": "morpheus","job": "zion resident"}' https://reqres.in/api/users/2
curl -X DELETE https://reqres.in/api/users/2
options | description |
---|---|
-O | download file under different name |
-c | resume a download |
wget https://releases.ubuntu.com/21.10/ubuntu-21.10-desktop-amd64.iso
wget -c https://releases.ubuntu.com/21.10/ubuntu-21.10-desktop-amd64.iso
find . | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/|-\1/"
find . -type d | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/| - \1/"
man ascii
cat file1 file2 | uniq -d
cat file1 file2 file2 | uniq -u
history | cut -c8- | sort | uniq -c | sort -rn | head
find . -depth -type d -exec rmdir {} \;
wc -w <<<$(netstat -i | cut -d" " -f1 | egrep -v "^Kernel|Iface|lo")
du -ah . 2>/dev/null | sort -hr | less
tee filecopy{1..10}.txt < file.txt
Name | Date | Change Description |
---|---|---|
Utsav Barman | 26 Jan 2022 | added: [arithmetic expansion, grep recursively search, command substitution example] |
Utsav Barman | 26 Jan 2022 | added: [ANSI C quoting] |
Utsav Barman | 26 Jan 2022 | added: [brace expansion, find piped to xargs, herestrings, awk example] |
Utsav Barman | 26 Jan 2022 | added: [command grouping] |
Utsav Barman | 25 Jan 2022 | added: [chown, chmod and du] |
Utsav Barman | 25 Jan 2022 | added: [& to start a background job, added curl and wget] |
Utsav Barman | 25 Jan 2022 | read should be used with option -r ; used appropriate variable names in for loop example; |
Utsav Barman | 25 Jan 2022 | Updated variable DATE to thedate. Uppercase variables are conventionally used for environment variables. Added $() for command substitution. backticks are discouraged; replaced -a with -e |
Utsav Barman | 25 Jan 2022 | fixed typo for cd to root directory; added section for special shell variables and exit codes; |
Utsav Barman | 25 Jan 2022 | Added wait, globbing, grep -o; fixed typos |
Utsav Barman | 24 Jan 2022 | Initial draft |