Terminal Tricks
Using Ctrl keys
Ctrl + n : same as Down arrow.
Ctrl + p : same as Up arrow.
Ctrl + r : begins a backward search through command history.(keep pressing Ctrl + r to move backward)
Ctrl + s : to stop output to terminal.
Ctrl + q : to resume output to terminal after Ctrl + s.
Ctrl + a : move to the beginning of line.
Ctrl + e : move to the end of line.
Ctrl + d : if you've type something, Ctrl + d deletes the character under the cursor, else, it escapes the current shell.
Ctrl + k : delete all text from the cursor to the end of line.
Ctrl + x + backspace : delete all text from the beginning of line to the cursor.
Ctrl + t : transpose the character before the cursor with the one under the cursor, press Esc + t to transposes the two words before the cursor.
Ctrl + w : cut the word before the cursor; then Ctrl + y paste it
Ctrl + u : cut the line before the cursor; then Ctrl + y paste it
Ctrl + x + Ctrl + e : launch editor define by $EDITOR
Ctrl + _ : undo typing.
Change case
Esc + u
# converts text from cursor to the end of the word to uppercase.
Esc + l
# converts text from cursor to the end of the word to lowercase.
Esc + c
# converts letter under the cursor to uppercase.
Run history number (e.g. 53)
!53
Run last command
!!
Run last command and change some parameter using caret substitution (e.g. last command: echo ‘aaa’ -> rerun as: echo ‘bbb’)
#last command: echo 'aaa'
^aaa^bbb
#echo 'bbb'
#bbb
#Notice that only the first aaa will be replaced, if you want to replace all 'aaa', use ':&' to repeat it:
^aaa^bbb^:&
#or
!!:gs/aaa/bbb/
Run past command that began with (e.g. cat filename)
!cat
# or
!c
# run cat filename again
Bash globbing
# '*' serves as a "wild card" for filename expansion.
/b?n/?at #/bin/cat
# '?' serves as a single-character "wild card" for filename expansion.
/etc/pa*wd #/etc/passwd
# ‘[]’ serves to match the character from a range.
ls -l [a-z]* #list all files with alphabet in its filename.
# ‘{}’ can be used to match filenames with more than one patterns
ls {*.sh,*.py} #list all .sh and .py files
Some handy environment variables
$0 :name of shell or shell script.
$1, $2, $3, ... :positional parameters.
$# :number of positional parameters.
$? :most recent foreground pipeline exit status.
$- :current options set for the shell.
$$ :pid of the current shell (not subshell).
$! :is the PID of the most recent background command.
Grep
Type of grep
grep = grep -G # Basic Regular Expression (BRE)
fgrep = grep -F # fixed text, ignoring meta-charachetrs
egrep = grep -E # Extended Regular Expression (ERE)
pgrep = grep -P # Perl Compatible Regular Expressions (PCRE)
rgrep = grep -r # recursive
Grep and count number of empty lines
grep -c "^$"
Grep and return only integer
grep -o '[0-9]*'
#or
grep -oP '\d'
Grep integer with certain number of digits (e.g. 3)
grep ‘[0-9]\{3\}’
# or
grep -E ‘[0-9]{3}’
# or
grep -P ‘\d{3}’
Grep only IP address
grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
# or
grep -Po '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}'
Grep whole word (e.g. ‘target’)
grep -w 'target'
#or using RE
grep '\btarget\b'
Grep returning lines before and after match (e.g. ‘bbo’)
# return also 3 lines after match
grep -A 3 'bbo'
# return also 3 lines before match
grep -B 3 'bbo'
# return also 3 lines before and after match
grep -C 3 'bbo'
Grep string starting with (e.g. ‘S’)
grep -o 'S.*'
Extract text between words (e.g. w1,w2)
grep -o -P '(?<=w1).*(?=w2)'
Grep lines without word (e.g. bbo)
grep -v bbo filename
Grep lines not begin with string (e.g. #)
grep -v '^#' file.txt
Grep variables with space within it (e.g. bbo=”some strings”)
grep "$boo" filename
#remember to quote the variable!
Grep only one/first match (e.g. bbo)
grep -m 1 bbo filename
Grep and return number of matching line(e.g. bbo)
grep -c bbo filename
Count occurrence (e.g. three times a line count three times)
grep -o bbo filename |wc -l
Case insensitive grep (e.g. bbo/BBO/Bbo)
grep -i "bbo" filename
COLOR the match (e.g. bbo)!
grep --color bbo filename
Grep search all files in a directory(e.g. bbo)
grep -R bbo /path/to/directory
# or
grep -r bbo /path/to/directory
Search all files in directory, do not ouput the filenames (e.g. bbo)
grep -rh bbo /path/to/directory
Search all files in directory, output ONLY the filenames with matches(e.g. bbo)
grep -rl bbo /path/to/directory
Grep OR (e.g. A or B or C or D)
grep 'A\|B\|C\|D'
Grep AND (e.g. A and B)
grep 'A.*B'
Regex any singer character (e.g. ACB or AEB)
grep 'A.B'
Regex with or without a certain character (e.g. color or colour)
grep ‘colou?r’
Grep all content of a fileA from fileB
grep -f fileA fileB
Grep a tab
grep $'\t'
Grep variable from variable
$echo "$long_str"|grep -q "$short_str"
if [ $? -eq 0 ]; then echo 'found'; fi
#grep -q will output 0 if match found
#remember to add space between []!
Grep strings between a bracket()
grep -oP '\(\K[^\)]+'
Grep number of characters with known strings in between(e.g. AAEL000001-RA)
grep -o -w "\w\{10\}\-R\w\{1\}"
# \w word character [0-9a-zA-Z_] \W not word character
Skip directory (e.g. bbo)
grep -d skip 'bbo' /path/to/files/*
Sed
Remove the 1st line
sed 1d filename
Remove the first 100 lines (remove line 1-100)
sed 1,100d filename
Remove lines with string (e.g. bbo)
sed "/bbo/d" filename
- case insensitive:
sed "/bbo/Id" filename
Remove lines whose nth character not equal to a value (e.g. 5th character not equal to 2)
sed -E '/^.{5}[^2]/d'
#aaaa2aaa (you can stay)
#aaaa1aaa (delete!)
Edit infile (edit and save)
sed -i "/bbo/d" filename
When using variable (e.g. $i), use double quotes “ “
# e.g. add >$i to the first line (to make a bioinformatics FASTA file)
sed "1i >$i"
# notice the double quotes! in other examples, you can use a single quote, but here, no way!
# '1i' means insert to first line
Using environment variable and end-of-line pattern at the same time.
# Use backslash for end-of-line $ pattern, and double quotes for expressing the variable
sed -e "\$s/\$/\n+--$3-----+/"
Delete/remove empty lines
sed '/^\s*$/d'
# or
sed '/^$/d'
Delete/remove last line
sed '$d'
Delete/remove last character from end of file
sed -i '$ s/.$//' filename
Add string to beginning of file (e.g. “[”)
sed -i '1s/^/[/' file
Add string at certain line number (e.g. add ‘something’ to line 1 and line 3)
sed -e '1isomething -e '3isomething'
Add string to end of file (e.g. “]”)
sed '$s/$/]/' filename
Add newline to the end
sed '$a\'
Add string to beginning of every line (e.g. bbo)
sed -e 's/^/bbo/' file
Add string to end of each line (e.g. “}”)
sed -e 's/$/\}\]/' filename
Add \n every nth character (e.g. every 4th character)
sed 's/.\{4\}/&\n/g'
Concatenate/combine/join files with a seperator and next line (e.g separate by “,”)
sed -s '$a,' *.json > all.json
Substitution (e.g. replace A by B)
sed 's/A/B/g' filename
Substitution with wildcard (e.g. replace a line start with aaa= by aaa=/my/new/path)
sed "s/aaa=.*/aaa=\/my\/new\/path/g"
Select lines start with string (e.g. bbo)
sed -n '/^@S/p'
Delete lines with string (e.g. bbo)
sed '/bbo/d' filename
Print/get/trim a range of line (e.g. line 500-5000)
sed -n 500,5000p filename
Print every nth lines
sed -n '0~3p' filename
# catch 0: start; 3: step
Print every odd # lines
sed -n '1~2p'
Print every third line including the first line
sed -n '1p;0~3p'
Remove leading whitespace and tabs
sed -e 's/^[ \t]*//'
# Notice a whitespace before '\t'!!
Remove only leading whitespace
sed 's/ *//'
# notice a whitespace before '*'!!
Remove ending commas
sed 's/,$//g'
Add a column to the end
sed "s/$/\t$i/"
# $i is the valuable you want to add
# To add the filename to every last column of the file
for i in $(ls);do sed -i "s/$/\t$i/" $i;done
Add extension of filename to last column
for i in T000086_1.02.n T000086_1.02.p;do sed "s/$/\t${i/*./}/" $i;done >T000086_1.02.np
Remove newline\ nextline
sed ':a;N;$!ba;s/\n//g'
Print a particular line (e.g. 123th line)
sed -n -e '123p'
Print a number of lines (e.g. line 10th to line 33 rd)
sed -n '10,33p' Change delimiter
sed 's=/=\\/=g'
Replace with wildcard (e.g A-1-e or A-2-e or A-3-e….)
sed 's/A-.*-e//g' filename
Remove last character of file
sed '$ s/.$//'
Insert character at specified position of file (e.g. AAAAAA –> AAA#AAA)
sed -r -e 's/^.{3}/&#/' file
Awk
Set tab as field separator
awk -F $'\t'
Output as tab separated (also as field separator)
awk -v OFS='\t'
Pass variable
a=bbo;b=obb;
awk -v a="$a" -v b="$b" "$1==a && $10=b" filename
Print line number and number of characters on each line
awk '{print NR,length($0);}' filename
Find number of columns
awk '{print NF}'
Reverse column order
awk '{print $2, $1}'
Check if there is a comma in a column (e.g. column $1)
awk '$1~/,/ {print}'
Split and do for loop
awk '{split($2, a,",");for (i in a) print $1"\t"a[i]}' filename
Print all lines before nth occurrence of a string (e.g stop print lines when bbo appears 7 times)
awk -v N=7 '{print}/bbo/&& --N<=0 {exit}'
Print filename and last line of all files in directory
ls|xargs -n1 -I file awk '{s=$0};END{print FILENAME,s}' file
Add string to the beginning of a column (e.g add “chr” to column $3)
awk 'BEGIN{OFS="\t"}$3="chr"$3'
Remove lines with string (e.g. bbo)
awk '!/bbo/' file
Remove last column
awk 'NF{NF-=1};1' file
Usage and meaning of NR and FNR
# For example there are two files:
# fileA:
# a
# b
# c
# fileB:
# d
# e
awk 'print FILENAME, NR,FNR,$0}' fileA fileB
# fileA 1 1 a
# fileA 2 2 b
# fileA 3 3 c
# fileB 4 1 d
# fileB 5 2 e
AND gate
# For example there are two files:
# fileA:
# 1 0
# 2 1
# 3 1
# 4 0
# fileB:
# 1 0
# 2 1
# 3 0
# 4 1
awk -v OFS='\t' 'NR=FNR{a[$1]=$2;next} NF {print $1,((a[$1]=$2)? $2:"0")}' fileA fileB
# 1 0
# 2 1
# 3 0
# 4 0
Round all numbers of file (e.g. 2 significant figure)
awk '{while (match($0, /[0-9]+\[0-9]+/)){
\printf "%s%.2f", substr($0,0,RSTART-1),substr($0,RSTART,RLENGTH)
\$0=substr($0, RSTART+RLENGTH)
\}
\print
\}'
Give number/index to every row
awk '{printf("%s\t%s\n",NR,$0)}'
Break combine column data into rows
# For example, seperate the following content:
# David cat,dog
# into
# David cat
# David dog
awk '{split($2,a,",");for(i in a)print $1"\t"a[i]}' file
# Detail here: http://stackoverflow.com/questions/33408762/bash-turning-single-comma-separated-column-into-multi-line-string
Average a file (each line in file contains only one number)
awk '{s+=$1}END{print s/NR}'
Print field start with string (e.g Linux)
awk '$1 ~ /^Linux/'
Sort a row (e.g. 1 40 35 12 23 –> 1 12 23 35 40)
awk ' {split( $0, a, "\t" ); asort( a ); for( i = 1; i <= length(a); i++ ) printf( "%s\t", a[i] ); printf( "\n" ); }'
Subtract previous row values (add column6 which equal to column4 minus last column5)
awk '{$6 = $4 - prev5; prev5 = $5; print;}'
Xargs
Set tab as delimiter (default:space)
xargs -d\t
Display 3 items per line
echo 1 2 3 4 5 6| xargs -n 3
# 1 2 3
# 4 5 6
Prompt before execution
echo a b c |xargs -p -n 3
Print command along with output
xargs -t abcd
# bin/echo abcd
# abcd
With find and rm
find . -name "*.html"|xargs rm
# when using a backtick
rm `find . -name "*.html"`
Delete fiels with whitespace in filename (e.g. “hello 2001”)
find . -name "*.c" -print0|xargs -0 rm -rf
Show limits
xargs --show-limits
Move files to folder
find . -name "*.bak" -print 0|xargs -0 -I {} mv {} ~/old
# or
find . -name "*.bak" -print 0|xargs -0 -I file mv file ~/old
Move first 100th files to a directory (e.g. d1)
ls |head -100|xargs -I {} mv {} d1
Parallel
time echo {1..5} |xargs -n 1 -P 5 sleep
# a lot faster than:
time echo {1..5} |xargs -n1 sleep
Copy all files from A to B
find /dir/to/A -type f -name "*.py" -print 0| xargs -0 -r -I file cp -v -p file --target-directory=/path/to/B
# v: verbose|
# p: keep detail (e.g. owner)
With sed
ls |xargs -n1 -I file sed -i '/^Pos/d' filename
Add the file name to the first line of file
ls |sed 's/.txt//g'|xargs -n1 -I file sed -i -e '1 i\>file\' file.txt
Count all files
ls |xargs -n1 wc -l
Turn output into a single line
ls -l| xargs
Count files within directories
echo mso{1..8}|xargs -n1 bash -c 'echo -n "$1:"; ls -la "$1"| grep -w 74 |wc -l' --
# "--" signals the end of options and display further option processing
Download dependencies files and install (e.g. requirements.txt)
cat requirements.txt| xargs -n1 sudo pip install
Count lines in all file, also count total lines
ls|xargs wc -l
Xargs and grep
cat grep_list |xargs -I{} grep {} filename
Xargs and sed (replace all old ip address with new ip address under /etc directory)
grep -rl '192.168.1.111' /etc | xargs sed -i 's/192.168.1.111/192.168.2.111/g'
Find
List all sub directory/file in the current directory
find .
List all files under the current directory
find . -type f
List all directories under the current directory
find . -type d
Edit all files under current directory (e.g. replace ‘www’ with ‘ww’)
find . -name '*.php' -exec sed -i 's/www/w/g' {} \;
# if there are no subdirectory
replace "www" "w" -- *
# a space before *
Find and output only filename (e.g. “mso”)
find mso*/ -name M* -printf "%f\n"
Find and delete file with size less than (e.g. 74 byte)
find . -name "*.mso" -size -74c -delete
# M for MB, etc
Condition and loop
If statement
# if and else loop for string matching
if [[ "$c" == "read" ]]; then outputdir="seq"; else outputdir="write" ; fi
# Test if myfile contains the string 'test':
if grep -q hello myfile; then …
# Test if mydir is a directory, change to it and do other stuff:
if cd mydir; then
echo 'some content' >myfile
else
echo >&2 "Fatal error. This script requires mydir."
fi
# if variable is null
if [ ! -s "myvariable" ]
#True of the length if "STRING" is zero.
# Test if file exist
if [ -e 'filename' ]
then
echo -e "file exists!"
fi
# Test if file exist but also including symbolic links:
if [ -e myfile ] || [ -L myfile ]
then
echo -e "file exists!"
fi
# Test if the value of x is greater or equal than 5
if [ "$x" -ge 5 ]; then …
# Test if the value of x is greater or equal than 5, in bash/ksh/zsh:
if ((x >= 5)); then …
# Use (( )) for arithmetic operation
if ((j==u+2))
# Use [[ ]] for comparison
if [[ $age -gt 21 ]]
For loop
for i in $(ls); do echo file $i;done
#or
for i in *; do echo file $i; done
# Press any key to continue each loop
for i in $(cat tpc_stats_0925.log |grep failed|grep -o '\query\w\{1,2\}');do cat ${i}.log; read -rsp $'Press any key to continue...\n' -n1 key;done
# Print a file line by line when a key is pressed,
oifs="$IFS"; IFS=$'\n'; for line in $(cat myfile); do ...; done
while read -r line; do ...; done #If only one word a line, simply
for line in $(cat myfile); do echo $line; read -n1; done
#Loop through an array
for i in "${arrayName[@]}"; do echo $i;done
While loop,
# Column subtraction of a file (e.g. a 3 columns file)
while read a b c; do echo $(($c-$b));done < <(head filename)
#there is a space between the two '<'s
# Sum up column subtraction
i=0; while read a b c; do ((i+=$c-$b)); echo $i; done < <(head filename)
# Keep checking a running process (e.g. perl) and start another new process (e.g. python) immediately after it. (BETTER use the wait command! Ctrl+F 'wait')
while [[ $(pidof perl) ]];do echo f;sleep 10;done && python timetorunpython.py
switch (case in bash)
read type;
case $type in
'0')
echo 'how'
;;
'1')
echo 'are'
;;
'2')
echo 'you'
;;
esac
Variable
Variable substitution within quotes
# foo=bar
echo "'$foo'"
#'bar'
# double/single quotes around single quotes make the inner single quotes expand variables
Get the length of variable
var="some string"
echo ${#var}
# 11
Get the first character of the variable
var=string
echo "${var:0:1}"
#s
# or
echo ${var%%"${var#?}"}
Remove the first or last string from variable
var="some string"
echo ${var:2}
#me string
Replacement (e.g. remove the first leading 0 )
var="0050"
echo ${var[@]#0}
#050
Replacement (e.g. replace ‘a’ with ‘,’)
{var/a/,}
Replace all (e.g. replace all ‘a’ with ‘,’)
{var//a/,}
#with grep
test="god the father"
grep ${test// /\\\|} file.txt
# turning the space into 'or' (\|) in grep
To change the case of the string stored in the variable to lowercase (Parameter Expansion)
var=HelloWorld
echo ${var,,}
helloworld
Full article: https://onceupon.github.io/Bash-Oneliner/
Linux Security Blog 
1 Comment