Shell numerical operations

Starting with this article, it is intended to introduce Shell programming through some examples in the light of the usual accumulation and further practice. Because examples often give people a sense of useful learning, but also give people the opportunity to practice, thereby stimulating people's enthusiasm for learning.

Given the ease of reading, these examples will be simple, but practical, and hopefully they will be a reference for our daily problems or "after tea" snacks, which of course have something to explore and optimize.

For more complex and interesting examples, refer to Advanced Bash-Scripting Guide, a book that dives into the art of Shell scripting.

Summary of the series:

• Objective: Enjoy solving problems with your shell, and talk and discuss with your friends.
• Plan: Write something sporadically, then add it, and finally organize it into a book.
• Reader: Familiar with the basics of Linux, such as file system structure, common command-line tools, Shell programming foundation, and more.
• Recommendation: When looking at examples, refer to Shell Foundation Twelve and Shell 13.
• Environment: Without specific instructions, the shells used in this series will refer specifically to Bash, versions above 3.1.17.
• Description: The series is not organized according to shell syntax, but targets some potential operational objects and operations themselves, which reflect real-world applications. Of course, shell syntax is certainly involved in this process.

This article is intended to discuss the basic numerical operations in shell programming, such as:

• The addition, subtract, multiplication, divide, power, mold, etc. between values (including integers and floats).
• Produces a random number for a specified range
• Produces a number of columns for the specified range

The shell itself can do integer operations, which are more complex to do with external expr bc awk on. I n addition, RANDOM number from 0 to 32767 can be generated from the RANDOM environment variable, and some external awk rand() function. T he seq can be used to produce a number of columns. They are described separately between the following.

Integer operations

Example: Add 1 to a number

$ i=0;
$ ((i++))
$ echo $i
1

$ let i++
$ echo $i
2

$ expr $i + 1
3
$ echo $i
2

$ echo $i 1 | awk '{printf $1+$2}'
3

Description: expr $i + spaces separated between . If multiplication is performed, the operator needs to be escaped, otherwise shell interprets the multiplier number as a wildcard, resulting in a syntax error; $1 and $2 awk mean $i 1, respectively, the 1st and 2nd numbers from left to right.

The types of commands viewed with shell's built-in commands are as follows:

$ type type
type is a shell builtin
$ type let
let is a shell builtin
$ type expr
expr is hashed (/usr/bin/expr)
$ type bc
bc is hashed (/usr/bin/bc)
$ type awk
awk is /usr/bin/awk

As can be seen from the above demo: let a shell built-in command, and several others are external commands, all in /usr/bin directory. E xpr expr bc other, have been loaded into the hash table in memory because they have just been used. This will help us understand the principles behind the various execution methods of scripts described in the last chapter.

Description: If you want to see help for different commands, for shell type such as let and type, you can view the help the shell, while some external man for the shell, help let man expr so on.

Example: Add from 1 to a number

#!/bin/bash
# calc.sh

i=0;
while [ $i -lt 10000 ]
do
    ((i++))
done
echo $i

Description: Here by while [ 条件表达式 ]; do .... done loop is implemented. -lt is less than < see the usage of the test command: man test .

How do I execute the script?

Method 1: Pass in the script file directly as a parameter of the sub-shell (Bash).

$ bash calc.sh
$ type bash
bash is hashed (/bin/bash)

Option 2: It is bash . source

$ . ./calc.sh

Or

$ source ./calc.sh
$ type .
. is a shell builtin
$ type source
source is a shell builtin

Method 3: The modification file is executable and is performed directly under the current shell

$ chmod ./calc.sh
$ ./calc.sh

Below, one by one, demonstrate using other methods to calculate the variable plus ((i++)) line with one of the following:

let i++;

i=$(expr $i + 1)

i=$(echo $i+1|bc)

i=$(echo "$i 1" | awk '{printf $1+$2;}')

The comparison calculation time is as follows:

$ time calc.sh
10000

real    0m1.319s
user    0m1.056s
sys     0m0.036s
$ time calc_let.sh
10000

real    0m1.426s
user    0m1.176s
sys     0m0.032s
$  time calc_expr.sh
1000

real    0m27.425s
user    0m5.060s
sys     0m14.177s
$ time calc_bc.sh
1000

real    0m56.576s
user    0m9.353s
sys     0m24.618s
$ time ./calc_awk.sh
100

real    0m11.672s
user    0m2.604s
sys     0m2.660s

Description: time can be used to count command execution time, which includes the total run time, user space execution time, kernel space execution time, which is ptrace system calls.

The above comparison can be found (()) efficient operation. L et let also efficient as a shell built-in command, expr bc awk is less efficient to calculate. T herefore, where the shell itself can do the work, it is recommended that you prioritize the functionality provided by the shell itself. B ut shell itself can't do functions, such as floating-point operations, so it needs help from external commands. Also, given the portability of shell scripts, do not use some shell-specific syntax when performance is not critical.

let , expr , bc used to find molds, operators are % ** % let and bc can be used to find power, operators are not the same, the former is ^ For example:

Example: Moulding

$ expr 5 % 2
1

$ let i=5%2
$ echo $i
1

$ echo 5 % 2 | bc
1

$ ((i=5%2))
$ echo $i
1

Example: Power

$ let i=5**2
$ echo $i
25

$ ((i=5**2))
$ echo $i

25
$ echo "5^2" | bc
25

Example: A forward conversion

Pre-conversion is also a more common operation that Bash bc support or with bc, for example by converting an 8-step 11 to a 10-step system:

$ echo "obase=10;ibase=8;11" | bc -l
9

$ echo $((8#11))
9

All of the above converts a number of percents to 10 percent, and if you want to convert bc more ibase and feed conversion targets directly with ibase and obase respectively.

Example: ascii character encoding

If you want to represent certain strings in a specific process, you can od command, such IFS which includes TAB and line man ascii

$ echo -n "$IFS" | od -c
0000000      t  n
0000003
$ echo -n "$IFS" | od -b
0000000 040 011 012
0000003

Floating-point operations

let expr can do floating-point operations, bc and awk can.

Example: Divide 1 by 13 and keep 3 valid digits

$ echo "scale=3; 1/13"  | bc
.076

$ echo "1 13" | awk '{printf("%0.3fn",$1/$2)}'
0.077

Description: bc to specify precision when performing floating-point operations, otherwise the default is 0, i.e. when floating-point operations are performed, the default result is to retain only integers. awk other end of the scale, is very flexible in controlling the printf only through printf format control.

Supplement: When performing bc scale can also be performed without scal specifying precision, and -l added after bc except that the default accuracy at this time is 20 bits. For example:

$ echo 1/13100 | bc -l
.00007633587786259541

Example: Cosine value angle

With bc -l high accuracy can be obtained:

$ export cos=0.996293; echo "scale=100; a(sqrt(1-$cos^2)/$cos)*180/(a(1)*4)" | bc -l
4.934954755411383632719834036931840605159706398655243875372764917732
5495504159766011527078286004072131

Of course, awk used to calculate:

$ echo 0.996293 | awk '{ printf("%s\n", atan2(sqrt(1-$1^2),$1)*180/3.1415926535);}'
4.93495

Example: There is a set of data for households with the highest monthly income per capita

Here a random set of test data is produced, and the income.txt

1 3 4490
2 5 3896
3 4 3112
4 4 4716
5 4 4578
6 6 5399
7 3 5089
8 6 3029
9 4 6195
10 5 5145

Description: The three columns of data above are the household number, the number of households, and the total monthly household income.

Analysis: In order to find the highest monthly income families, the next two columns need to be divided, that is, to find out the average monthly income of each household, and then ranked by the average monthly income, to find the highest income families.

Realize:

#!/bin/bash
# gettopfamily.sh

[ $# -lt 1 ] && echo "please input the income file" && exit -1
[ ! -f $1 ] && echo "$1 is not a file" && exit -1

income=$1
awk '{
    printf("%d %0.2fn", $1, $3/$2);
}' $income | sort -k 2 -n -r

Description:

• [ $# -lt 1 ] and $# the number of incoming parameters in the shell
• [ ! -f $1 ] file, the use of -f can be found test man test
• income=$1 parameter to the income variable and use it as awk the file to be processed
• awk Divide the third column of the document by the second column to find the average monthly income, taking into account the accuracy, retaining two precision
• sort -k 2 -n -r Here the awk result - k -k 2 that is, the average monthly income is sorted, -n and sorted in decreasing -r

Demonstrate:

$ ./gettopfamily.sh income.txt
7 1696.33
9 1548.75
1 1496.67
4 1179.00
5 1144.50
10 1029.00
6 899.83
2 779.20
3 778.00
8 504.83

Supplement: Previous income.txt were randomly generated. W hen doing some experiments, you often need to randomly generate some data, which we'll cover in more detail in the next section. Here is the income.txt data:

#!/bin/bash
# genrandomdata.sh

for i in $(seq 1 10)
do
    echo $i $(($RANDOM/8192+3)) $((RANDOM/10+3000))
done

Description: The seq command is also used in the above script to produce a column number from 1 to 10, and the detailed usage of this command is further described in the last section of this article.

Random number

The environment RANDOM produces random numbers from 0 to 32767, while the rand() awk produces random numbers between 0 and 1.

Example: Get a random number

$ echo $RANDOM
81

$ echo "" | awk '{srand(); printf("%f", rand());}'
0.237788

Description: srand() the current time as a seed of rand() random number generator when there are no seed

Example: Randomly produces a number from 0 to 255

This RANDOM achieved by scaling the RANDOM variable and rand() awk

$ expr $RANDOM / 128

$ echo "" | awk '{srand(); printf("%d\n", rand()*255);}'

Think: What if you want to randomly generate an IP address for an IP segment? Look at the example: Get an available IP address a friendly way.

#!/bin/bash
# getip.sh -- get an usable ipaddress automatically
# author: falcon <[email protected]>
# update: Tue Oct 30 23:46:17 CST 2007

# set your own network, default gateway, and the time out of "ping" command
net="192.168.1"
default_gateway="192.168.1.1"
over_time=2

# check the current ipaddress
ping -c 1 $default_gateway -W $over_time
[ $? -eq 0 ] && echo "the current ipaddress is okey!" && exit -1;

while :; do
    # clear the current configuration
    ifconfig eth0 down
    # configure the ip address of the eth0
    ifconfig eth0 \
        $net.$(($RANDOM /130 +2)) \
        up
    # configure the default gateway
    route add default gw $default_gateway
    # check the new configuration
    ping -c 1 $default_gateway -W $over_time
    # if work, finish
    [ $? -eq 0 ] && break
done

Description: If your default gateway address 192.168.1.1 configure default_gateway can be route -n ifconfig will be the same as the gateway, causing the entire network to not function properly.

Other operations

In fact, through a loop can produce a series of numbers, but there are related tools why not use it! seq such a gadget that it can produce a series of numbers, you can specify the incremental interval of the number, or you can specify the split between the adjacent two numbers.

Example: Get a series of numbers

$ seq 5
1
2
3
4
5
$ seq 1 5
1
2
3
4
5
$ seq 1 2 5
1
3
5
$ seq -s: 1 2 5
1:3:5
$ seq 1 2 14
1
3
5
7
9
11
13
$ seq -w 1 2 14
01
03
05
07
09
11
13
$ seq -s: -w 1 2 14
01:03:05:07:09:11:13
$ seq -f "0x%g" 1 5
0x1
0x2
0x3
0x4
0x5

A more typical example of seq is to construct links in a specific format and then wget

$ for i in `seq -f"http://thns.tsinghua.edu.cn/thnsebooks/ebook73/%02g.pdf" 1 21`;do wget -c $i; done

Or

$ for i in `seq -w 1 21`;do wget -c "http://thns.tsinghua.edu.cn/thnsebooks/ebook73/$i"; done

Added: Bash version 3, for loop, you can more succinctly produce numbers from 1 to {1..5} (note that there are only two points between 1 and 5) directly after the in of the in loop, for example:

$ for i in {1..5}; do echo -n "$i "; done
1 2 3 4 5

Example: Count the number of times each word appears in a string

Let's start by defining a word: a single or multiple character series of letters.

First, count the number of times each word appears:

$ wget -c http://tinylab.org
$ cat index.html | sed -e "s/[^a-zA-Z]/\n/g" | grep -v ^$ | sort | uniq -c

Next, the top 10 words with the highest frequency of statistics appear:

$ wget -c http://tinylab.org
$ cat index.html | sed -e "s/[^a-zA-Z]/\n/g" | grep -v ^$ | sort | uniq -c | sort -n -k 1 -r | head -10
    524 a
    238 tag
    205 href
    201 class
    193 http
    189 org
    175 tinylab
    174 www
    146 div
    128 title

Description:

• cat index.html Output index .html contents of the file
• sed -e "s/[^a-zA-Z]/\n/g" Replace non-alphabetical characters with spaces, retaining only alphabet characters
• grep -v ^$ Remove the empty line
• sort Sort
• uniq -c Counts the same number of lines, that is, the number of words
• sort -n -k 1 -r Sorted by the number -n in reverse -n of -r -k 1
• head -10 Remove the first ten lines

Example: Count the number of times a specified word appears

Two approaches could be considered:

• Count only those words that need to be counted
• Use the algorithm above to count the number of all words, and then return those words that need to be counted to the user

However, both approaches can be implemented through the following structure. Let's look at Option One:

#!/bin/bash
# statistic_words.sh

if [ $# -lt 1 ]; then
    echo "Usage: basename $0 FILE WORDS ...."
    exit -1
fi

FILE=$1
((WORDS_NUM=$#-1))

for n in $(seq $WORDS_NUM)
do
    shift
    cat $FILE | sed -e "s/[^a-zA-Z]/\n/g" \
        | grep -v ^$ | sort | grep ^$1$ | uniq -c
done

Description:

• if 条件部分 Requires at least two parameters, the first word file, and then the parameters are the words to be counted
• FILE=$1 Gets the file name, which is the first string after the script
• ((WORDS_NUM=$#-1)) Get the number of words, i.e. the total number of $# the file name parameters (1)
• for 循环部分 First produce a series of words that need to be counted through seq shift built-in variable help shift which moves the user's incoming arguments from the $1 that you can $1 by the user (on closer inspection, there appear to be several sets of underlying flavors). You may consider shift the sentence after shift with echo $1 shift use of shift

Demonstrate:

$ chmod +x statistic_words.sh
$ ./statistic_words.sh index.html tinylab linux python
    175 tinylab
     43 linux
      3 python

Looking at option two, we just need to modify the sentence after shift

#!/bin/bash
# statistic_words.sh

if [ $# -lt 1 ]; then
    echo "ERROR: you should input 2 words at least";
    echo "Usage: basename $0 FILE WORDS ...."
    exit -1
fi

FILE=$1
((WORDS_NUM=$#-1))

for n in $(seq $WORDS_NUM)
do
    shift
    cat $FILE | sed -e "s/[^a-zA-Z]/\n/g" \
        | grep -v ^$ | sort | uniq -c | grep " $1$"
done

Demonstrate:

$ ./statistic_words.sh index.html tinylab linux python
    175 tinylab
     43 linux
      3 python

Description: Obviously, method one is much more efficient because it identifies the words that need to be counted in advance, and then the latter is not. In fact, if we grep -E option, we don't have to introduce a loop, we can do it with a single command:

$ cat index.html | sed -e "s/[^a-zA-Z]/\n/g" | grep -v ^$ | sort | grep -E "^tinylab$|^linux$" | uniq -c
     43 linux
    175 tinylab

Or

$ cat index.html | sed -e "s/[^a-zA-Z]/\n/g" | grep -v ^$ | sort | egrep  "^tinylab$|^linux$" | uniq -c
     43 linux
    175 tinylab

Description: You need to sed command can process files cat the pipeline later through the cat command output, which reduces unnecessary pipeline operations, so the above commands can be simplified to:

$ sed -e "s/[^a-zA-Z]/\n/g" index.html | grep -v ^$ | sort | egrep  "^tinylab$|^linux$" | uniq -c
     43 linux
    175 tinylab

So, you can see sed grep uniq sort are, they do simple functions themselves, but with a certain combination, you can achieve a variety of things. By the way, there's a very useful command wc -w which you can use when you need them.

Supplement: The jot command and factor command are factor also mentioned jot Bash-Scripting Guide, which can produce factor of a certain number because they are not available on the machine and therefore are not tested. Such as:

$ factor 100
100: 2 2 5 5

Summary

At this point, the numerical calculation of the Shell programming paradigm is over. This article focuses on:

• Integer operations, floating-point operations, the generation of random numbers, and the generation of columns in Shell programming
• Shell's built-in commands, the differences between external commands, and how to see their type and help
• Several ways to execute a shell script
• Several common shell external commands: sed awk grep uniq sort
• Examples: Digital increment, average monthly income, automatic IP acquisition, counting words
• Other: Related usages such as command lists, conditional tests, etc. are covered in the above examples, please read them carefully

If you have time, please check it out.

Information

• Advanced Bash-Scripting Guide
• shell 13 asked
• Shell Basic Twelve
• SED manual
• AWK manual

• Several Shell discussion areas

• LinuxSir.org
• ChinaUnix.net

Postscript

It took about 3 hours to finish writing, it is now 23:33, it is time to go back to the dormitory to sleep, tomorrow up to modify typos and add some content, good night friends!

On October 31st, change some of the wording to add an example of the average monthly household income, add summaries and references, and use all the codes in the appendix.

Shell programming is a very interesting thing, if you think about it: the above example of M$ Excel to do this work, you will find that the former is so simple and hassty, and give you the feeling of easy use.