Handy Bash one-liners

Terminal Tricks

Using Ctrl keys
Ctrl + n : same as Down arrow.
Ctrl + p : same as Up arrow.
Ctrl + r : begins a backward search through command history.(keep pressing Ctrl + r to move backward)
Ctrl + s : to stop output to terminal.
Ctrl + q : to resume output to terminal after Ctrl + s.
Ctrl + a : move to the beginning of line.
Ctrl + e : move to the end of line.
Ctrl + d : if you've type something, Ctrl + d deletes the character under the cursor, else, it escapes the current shell.
Ctrl + k : delete all text from the cursor to the end of line.
Ctrl + x + backspace : delete all text from the beginning of line to the cursor.
Ctrl + t : transpose the character before the cursor with the one under the cursor, press Esc + t to transposes the two words before the cursor.
Ctrl + w : cut the word before the cursor; then Ctrl + y paste it
Ctrl + u : cut the line before the cursor; then Ctrl + y paste it
Ctrl + x + Ctrl + e : launch editor define by $EDITOR
Ctrl + _ : undo typing.
Change case
Esc + u
# converts text from cursor to the end of the word to uppercase.  
Esc + l
# converts text from cursor to the end of the word to lowercase. 
Esc + c
# converts letter under the cursor to uppercase.
Run history number (e.g. 53)
!53
Run last command
!!
Run last command and change some parameter using caret substitution (e.g. last command: echo ‘aaa’ -> rerun as: echo ‘bbb’)
#last command: echo 'aaa'
^aaa^bbb

#echo 'bbb'
#bbb

#Notice that only the first aaa will be replaced, if you want to replace all 'aaa', use ':&' to repeat it:
^aaa^bbb^:&
#or 
!!:gs/aaa/bbb/

Run past command that began with (e.g. cat filename)
!cat
# or
!c
# run cat filename again
Bash globbing
# '*' serves as a "wild card" for filename expansion.
/b?n/?at      #/bin/cat

# '?' serves as a single-character "wild card" for filename expansion.
/etc/pa*wd    #/etc/passwd 

# ‘[]’ serves to match the character from a range.
ls -l [a-z]*   #list all files with alphabet in its filename.

# ‘{}’ can be used to match filenames with more than one patterns
ls {*.sh,*.py}   #list all .sh and .py files
Some handy environment variables
$0   :name of shell or shell script.  
$1, $2, $3, ... :positional parameters.  
$#   :number of positional parameters.  
$?   :most recent foreground pipeline exit status.  
$-   :current options set for the shell.  
$$   :pid of the current shell (not subshell).  
$!   :is the PID of the most recent background command.  

Grep

$199 ENROLLS YOU INTO OUR SELF PACED COURSE – LFS264 – OPNFV FUNDAMENTALS!

Type of grep
grep = grep -G # Basic Regular Expression (BRE)
fgrep = grep -F # fixed text, ignoring meta-charachetrs
egrep = grep -E # Extended Regular Expression (ERE)
pgrep = grep -P # Perl Compatible Regular Expressions (PCRE)
rgrep = grep -r # recursive
Grep and count number of empty lines
grep -c "^$"
Grep and return only integer
grep -o '[0-9]*'
#or
grep -oP '\d'
Grep integer with certain number of digits (e.g. 3)
grep ‘[0-9]\{3\}# or
grep -E ‘[0-9]{3}# or
grep -P\d{3}
Grep only IP address
grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
# or
grep -Po '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}'
Grep whole word (e.g. ‘target’)
grep -w 'target'
 
#or using RE
grep '\btarget\b'
Grep returning lines before and after match (e.g. ‘bbo’)
# return also 3 lines after match
grep -A 3 'bbo'
 
# return also 3 lines before match
grep -B 3 'bbo'

# return also 3 lines before and after match
grep -C 3 'bbo'
Grep string starting with (e.g. ‘S’)
grep -o 'S.*'
Extract text between words (e.g. w1,w2)
grep -o -P '(?<=w1).*(?=w2)'
Grep lines without word (e.g. bbo)
grep -v bbo filename
Grep lines not begin with string (e.g. #)
grep -v '^#' file.txt
Grep variables with space within it (e.g. bbo=”some strings”)
grep "$boo" filename
#remember to quote the variable!  
Grep only one/first match (e.g. bbo)
grep -m 1 bbo filename
Grep and return number of matching line(e.g. bbo)
grep -c bbo filename
Count occurrence (e.g. three times a line count three times)
grep -o bbo filename |wc -l 
Case insensitive grep (e.g. bbo/BBO/Bbo)
grep -i "bbo" filename 
COLOR the match (e.g. bbo)!
grep --color bbo filename 
Grep search all files in a directory(e.g. bbo)
grep -R bbo /path/to/directory 
# or
grep -r bbo /path/to/directory 
Search all files in directory, do not ouput the filenames (e.g. bbo)
grep -rh bbo /path/to/directory 
Search all files in directory, output ONLY the filenames with matches(e.g. bbo)
grep -rl bbo /path/to/directory
Grep OR (e.g. A or B or C or D)
grep 'A\|B\|C\|D' 
Grep AND (e.g. A and B)
grep 'A.*B' 
Regex any singer character (e.g. ACB or AEB)
grep 'A.B' 
Regex with or without a certain character (e.g. color or colour)
grep ‘colou?r’
Grep all content of a fileA from fileB
grep -f fileA fileB 
Grep a tab
grep $'\t' 
Grep variable from variable
$echo "$long_str"|grep -q "$short_str"
if [ $? -eq 0 ]; then echo 'found'; fi
#grep -q will output 0 if match found  
#remember to add space between []!
Grep strings between a bracket()
grep -oP '\(\K[^\)]+'
Grep number of characters with known strings in between(e.g. AAEL000001-RA)
grep -o -w "\w\{10\}\-R\w\{1\}"
# \w word character [0-9a-zA-Z_] \W not word character 
Skip directory (e.g. bbo)
grep -d skip 'bbo' /path/to/files/*

Sed

ENROLL TODAY IN THE SELF PACED COURSE – LFS263 – ONAP FUNDAMENTALS FOR $199!

Remove the 1st line
sed 1d filename
Remove the first 100 lines (remove line 1-100)
sed 1,100d filename
Remove lines with string (e.g. bbo)
sed "/bbo/d" filename
- case insensitive:
sed "/bbo/Id" filename
Remove lines whose nth character not equal to a value (e.g. 5th character not equal to 2)
sed -E '/^.{5}[^2]/d'
#aaaa2aaa (you can stay)
#aaaa1aaa (delete!)
Edit infile (edit and save)
sed -i "/bbo/d" filename
When using variable (e.g. $i), use double quotes “ “
# e.g. add >$i to the first line (to make a bioinformatics FASTA file)  
sed "1i >$i"  
# notice the double quotes! in other examples, you can use a single quote, but here, no way!   
# '1i' means insert to first line
Using environment variable and end-of-line pattern at the same time.
# Use backslash for end-of-line $ pattern, and double quotes for expressing the variable
sed -e "\$s/\$/\n+--$3-----+/"
Delete/remove empty lines
sed '/^\s*$/d' 
 
# or
   
sed '/^$/d' 
Delete/remove last line
sed '$d' 
Delete/remove last character from end of file
sed -i '$ s/.$//' filename
Add string to beginning of file (e.g. “[”)
sed -i '1s/^/[/' file
Add string at certain line number (e.g. add ‘something’ to line 1 and line 3)
sed -e '1isomething -e '3isomething'
Add string to end of file (e.g. “]”)
sed '$s/$/]/' filename
Add newline to the end
sed '$a\'
Add string to beginning of every line (e.g. bbo)
sed -e 's/^/bbo/' file
Add string to end of each line (e.g. “}”)
sed -e 's/$/\}\]/' filename
Add \n every nth character (e.g. every 4th character)
sed 's/.\{4\}/&\n/g' 
Concatenate/combine/join files with a seperator and next line (e.g separate by “,”)
sed -s '$a,' *.json > all.json
Substitution (e.g. replace A by B)
sed 's/A/B/g' filename 
Substitution with wildcard (e.g. replace a line start with aaa= by aaa=/my/new/path)
sed "s/aaa=.*/aaa=\/my\/new\/path/g"
Select lines start with string (e.g. bbo)
sed -n '/^@S/p' 
Delete lines with string (e.g. bbo)
sed '/bbo/d' filename 
Print/get/trim a range of line (e.g. line 500-5000)
sed -n 500,5000p filename
sed -n '0~3p' filename

# catch 0: start; 3: step
sed -n '1~2p' 
sed -n '1p;0~3p' 
Remove leading whitespace and tabs
sed -e 's/^[ \t]*//'
# Notice a whitespace before '\t'!!
Remove only leading whitespace
sed 's/ *//'

# notice a whitespace before '*'!!
Remove ending commas
sed 's/,$//g' 
Add a column to the end
sed "s/$/\t$i/"
# $i is the valuable you want to add  

# To add the filename to every last column of the file  
for i in $(ls);do sed -i "s/$/\t$i/" $i;done
Add extension of filename to last column
for i in T000086_1.02.n T000086_1.02.p;do sed "s/$/\t${i/*./}/" $i;done >T000086_1.02.np
Remove newline\ nextline
sed ':a;N;$!ba;s/\n//g'
sed -n -e '123p'
sed -n '10,33p' 
Change delimiter
sed 's=/=\\/=g'
Replace with wildcard (e.g A-1-e or A-2-e or A-3-e….)
sed 's/A-.*-e//g' filename
Remove last character of file
sed '$ s/.$//'
Insert character at specified position of file (e.g. AAAAAA –> AAA#AAA)
sed -r -e 's/^.{3}/&#/' file

Awk

$299 REGISTERS YOU FOR OUR NEWEST SELF PACED COURSE! LFD201 – INTRODUCTION TO OPEN SOURCE DEVELOPMENT, GIT, AND LINUX!

Set tab as field separator
awk -F $'\t'  
Output as tab separated (also as field separator)
awk -v OFS='\t' 
Pass variable
a=bbo;b=obb;
awk -v a="$a" -v b="$b" "$1==a && $10=b" filename 
awk '{print NR,length($0);}' filename 
Find number of columns
awk '{print NF}' 
Reverse column order
awk '{print $2, $1}' 
Check if there is a comma in a column (e.g. column $1)
awk '$1~/,/ {print}'  
Split and do for loop
awk '{split($2, a,",");for (i in a) print $1"\t"a[i]}' filename 
awk -v N=7 '{print}/bbo/&& --N<=0 {exit}'
ls|xargs -n1 -I file awk '{s=$0};END{print FILENAME,s}' file
Add string to the beginning of a column (e.g add “chr” to column $3)
awk 'BEGIN{OFS="\t"}$3="chr"$3' 
Remove lines with string (e.g. bbo)
awk '!/bbo/' file 
Remove last column
awk 'NF{NF-=1};1' file
Usage and meaning of NR and FNR
# For example there are two files:
# fileA:  
# a  
# b  
# c  
# fileB:  
# d  
# e  
awk 'print FILENAME, NR,FNR,$0}' fileA fileB 
# fileA    1    1    a  
# fileA    2    2    b  
# fileA    3    3    c  
# fileB    4    1    d  
# fileB    5    2    e  
AND gate
# For example there are two files: 
# fileA:  
# 1    0  
# 2    1  
# 3    1  
# 4    0  
# fileB:  
# 1    0  
# 2    1  
# 3    0  
# 4    1      

awk -v OFS='\t' 'NR=FNR{a[$1]=$2;next} NF {print $1,((a[$1]=$2)? $2:"0")}' fileA fileB 
# 1    0  
# 2    1  
# 3    0  
# 4    0  
Round all numbers of file (e.g. 2 significant figure)
awk '{while (match($0, /[0-9]+\[0-9]+/)){
    \printf "%s%.2f", substr($0,0,RSTART-1),substr($0,RSTART,RLENGTH)
    \$0=substr($0, RSTART+RLENGTH)
    \}
    \print
    \}'
Give number/index to every row
awk '{printf("%s\t%s\n",NR,$0)}'
Break combine column data into rows
# For example, seperate the following content:
# David    cat,dog  
# into  
# David    cat  
# David    dog  

awk '{split($2,a,",");for(i in a)print $1"\t"a[i]}' file
 
# Detail here: http://stackoverflow.com/questions/33408762/bash-turning-single-comma-separated-column-into-multi-line-string
Average a file (each line in file contains only one number)
awk '{s+=$1}END{print s/NR}'
awk '$1 ~ /^Linux/'
Sort a row (e.g. 1 40 35 12 23 –> 1 12 23 35 40)
awk ' {split( $0, a, "\t" ); asort( a ); for( i = 1; i <= length(a); i++ ) printf( "%s\t", a[i] ); printf( "\n" ); }'
Subtract previous row values (add column6 which equal to column4 minus last column5)
awk '{$6 = $4 - prev5; prev5 = $5; print;}'

Xargs

REGISTER TODAY FOR YOUR KUBERNETES FOR DEVELOPERS (LFD259) COURSE AND CKAD CERTIFICATION TODAY! $499!    $299 now!!

Set tab as delimiter (default:space)
xargs -d\t
Display 3 items per line
echo 1 2 3 4 5 6| xargs -n 3
# 1 2 3  
# 4 5 6

Prompt before execution
echo a b c |xargs -p -n 3
xargs -t abcd
# bin/echo abcd  
# abcd

With find and rm
find . -name "*.html"|xargs rm
 
# when using a backtick
rm `find . -name "*.html"`
Delete fiels with whitespace in filename (e.g. “hello 2001”)
find . -name "*.c" -print0|xargs -0 rm -rf
Show limits
xargs --show-limits
Move files to folder
find . -name "*.bak" -print 0|xargs -0 -I {} mv {} ~/old

# or
find . -name "*.bak" -print 0|xargs -0 -I file mv file ~/old
Move first 100th files to a directory (e.g. d1)
ls |head -100|xargs -I {} mv {} d1
Parallel
time echo {1..5} |xargs -n 1 -P 5 sleep

# a lot faster than: 
time echo {1..5} |xargs -n1 sleep
Copy all files from A to B
find /dir/to/A -type f -name "*.py" -print 0| xargs -0 -r -I file cp -v -p file --target-directory=/path/to/B

# v: verbose|  
# p: keep detail (e.g. owner)

With sed
ls |xargs -n1 -I file sed -i '/^Pos/d' filename
Add the file name to the first line of file
ls |sed 's/.txt//g'|xargs -n1 -I file sed -i -e '1 i\>file\' file.txt
Count all files
ls |xargs -n1 wc -l
Turn output into a single line
ls -l| xargs
Count files within directories
echo mso{1..8}|xargs -n1 bash -c 'echo -n "$1:"; ls -la "$1"| grep -w 74 |wc -l' --
# "--" signals the end of options and display further option processing
Download dependencies files and install (e.g. requirements.txt)
cat requirements.txt| xargs -n1 sudo pip install
Count lines in all file, also count total lines
ls|xargs wc -l
Xargs and grep
cat grep_list |xargs -I{} grep {} filename
Xargs and sed (replace all old ip address with new ip address under /etc directory)
grep -rl '192.168.1.111' /etc | xargs sed -i 's/192.168.1.111/192.168.2.111/g'

Find

$299 WILL ENROLL YOU IN OUR SELF PACED COURSE – LFS205 – ADMINISTERING LINUX ON AZURE!

List all sub directory/file in the current directory
find .
List all files under the current directory
find . -type f
List all directories under the current directory
find . -type d
Edit all files under current directory (e.g. replace ‘www’ with ‘ww’)
find . -name '*.php' -exec sed -i 's/www/w/g' {} \;

# if there are no subdirectory
replace "www" "w" -- *
# a space before *
Find and output only filename (e.g. “mso”)
find mso*/ -name M* -printf "%f\n"
Find and delete file with size less than (e.g. 74 byte)
find . -name "*.mso" -size -74c -delete

# M for MB, etc

Condition and loop

BUNDLE CLOUD FOUNDRY FOR DEVELOPERS COURSE(LFD232) AND THE CFCD CERTIFICATION FOR $499!

If statement
# if and else loop for string matching
if [[ "$c" == "read" ]]; then outputdir="seq"; else outputdir="write" ; fi  

# Test if myfile contains the string 'test':
if grep -q hello myfile; then# Test if mydir is a directory, change to it and do other stuff:
if cd mydir; then
  echo 'some content' >myfile
else
  echo >&2 "Fatal error. This script requires mydir."
fi

# if variable is null
if [ ! -s "myvariable" ]
#True of the length if "STRING" is zero.

# Test if file exist
if [ -e 'filename' ]
then
  echo -e "file exists!"
fi

# Test if file exist but also including symbolic links:
if [ -e myfile ] || [ -L myfile ]
then
  echo -e "file exists!"
fi

# Test if the value of x is greater or equal than 5
if [ "$x" -ge 5 ]; then# Test if the value of x is greater or equal than 5, in bash/ksh/zsh:
if ((x >= 5)); then# Use (( )) for arithmetic operation
if ((j==u+2))

# Use [[ ]] for comparison
if [[ $age -gt 21 ]]

More if commands

For loop
for i in $(ls); do echo file $i;done
#or
for i in *; do echo file $i; done

# Press any key to continue each loop
for i in $(cat tpc_stats_0925.log |grep failed|grep -o '\query\w\{1,2\}');do cat ${i}.log; read -rsp $'Press any key to continue...\n' -n1 key;done

# Print a file line by line when a key is pressed, 
oifs="$IFS"; IFS=$'\n'; for line in $(cat myfile); do ...; done
while read -r line; do ...; done #If only one word a line, simply
for line in $(cat myfile); do echo $line; read -n1; done

#Loop through an array
for i in "${arrayName[@]}"; do echo $i;done

While loop,
# Column subtraction of a file (e.g. a 3 columns file)
while read a b c; do echo $(($c-$b));done < <(head filename)
#there is a space between the two '<'s

# Sum up column subtraction
i=0; while read a b c; do ((i+=$c-$b)); echo $i; done < <(head filename)

# Keep checking a running process (e.g. perl) and start another new process (e.g. python) immediately after it. (BETTER use the wait command! Ctrl+F 'wait')
while [[ $(pidof perl) ]];do echo f;sleep 10;done && python timetorunpython.py
switch (case in bash)
read type;
case $type in
  '0')
    echo 'how'
    ;;
  '1')
    echo 'are'
    ;;
  '2')
    echo 'you'
    ;;
esac

Variable

SPEND $199 AND ENROLL IN OUR SELF PACED CONTAINERS FUNDAMENTALS COURSE (LFS253)!

Variable substitution within quotes
# foo=bar
 echo "'$foo'"
#'bar'
# double/single quotes around single quotes make the inner single quotes expand variables
Get the length of variable
var="some string"
echo ${#var}  
# 11
Get the first character of the variable
var=string
echo "${var:0:1}"
#s 

# or
echo ${var%%"${var#?}"}
Remove the first or last string from variable
var="some string"
echo ${var:2}
#me string
Replacement (e.g. remove the first leading 0 )
var="0050"
echo ${var[@]#0}
#050
Replacement (e.g. replace ‘a’ with ‘,’)
{var/a/,}
Replace all (e.g. replace all ‘a’ with ‘,’)
{var//a/,}
#with grep
 test="god the father"
 grep ${test// /\\\|} file.txt
 # turning the space into 'or' (\|) in grep
To change the case of the string stored in the variable to lowercase (Parameter Expansion)
var=HelloWorld
echo ${var,,}
helloworld

Full article: https://onceupon.github.io/Bash-Oneliner/

LFD430

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.