bash maths

[flashcoder]

Hi,

I am puzzled and can not figure out whats wrong here. Please someone explain me and preferably give me a reference to the man page where it was explained but I was too dumb to understand.

Here is a small script that perfectly shows the problem:

#/bin/sh
#
# shows the total size of files in given folder ($1)

sum=0
ls -laF $1 | awk {'print $5'} | while read fsize
do
  sum+=fsize
done
echo "$sum"

And the sum shows up ZERO. Why?????

I've tried various counstructions like sum=$(( sum + fsize )) and nothing works..
Its probably because variable fsize is not a scalar but a string and I can not figure out how to force bash to use it as a scalar value..

Thanks.

[Kieren]

Maths in bash isn't that simple I'm afraid. Try the following instead:

sum=0
ls -laF $1 | awk {'print $5'} | while read fsize
do
  sum=$(( ${fsize} + ${sum} ))
done
echo "$sum"

Edit: this still comes up with 0, hold on...

[Kieren]

I'm not sure what's going on here. The code:

sum=0 
ls -laF | awk {'print $5'} | while read fsize
do 
   if [[ -n ${fsize} ]]
   then 
      sum=$(( fsize + sum )) 
      echo $sum 
   fi 
done 

echo "total: $sum"

Prints out:

4096
8192
12288
13477
14637
18733
20356
21668
22240
total: 0

So it creates a running total in $sum but then $sum looses it's value for some reason

[Cabhan]

Ah yes. The number of times that I've seen this exact problem is truly impressive.

This is actually not an error with Bash math. It is, instead, a consequence of the way that Bash handles piping with loops.

When you pipe into or out of a loop, the loop gets run in a separate subshell. This means that all of the work done in the loop works correctly, but when you exit the loop (and therefore the subshell), the work is completely lost.

Try this instead:

sum=0

$fsizes=$(ls -laF | awk {'print $5'})

for fsize in $fsizes
do 
   if [[ -n ${fsize} ]]
   then 
      sum=$(( fsize + sum )) 
      echo $sum 
   fi 
done 

echo "total: $sum"

I believe that this will work.

[Kieren]

Thanks for the info! That works fine but the line $fsizes=$(ls -laF | awk {'print $5'}) shouldn't start with a $:

sum=0

fsizes=$(ls -laF | awk {'print $5'})

for fsize in $fsizes
do 
   if [[ -n ${fsize} ]]
   then 
      sum=$(( fsize + sum )) 
      echo $sum 
   fi 
done 

echo "total: $sum"

[flashcoder]

Ok that was a simple example to show what goes on, in that case its possible to replace the pipe feeding the loop with simple "for" loop.
Here is a piece of the code I am using actually, note that I need to read a line from file in several variables. How about that code, how can I do it w/o pipe?

sum=0
cat ./$idfilename | while read stendname id ip
do
  sum=$(( sum + 1 ))
  ...

I cant figure out really any other way that pipe. Besides pipes are safe to process thousands and thousands of lines but variables are quite limited in size.

[Cabhan]

@Kieren

Ouch, I'm kind of embarrassed that I did that. Thanks for the catch.

@flashcoder

When reading from a file, there is usually a better way to do it than piping the file in.

In this case, the best way is:

sum=0

exec 3< "$idfilename"

while read stendname id ip <&3
do
  sum=$(( sum + 1 ))
  ...

This opens the file and assigns it file descriptor 3, and reads directly from that. This avoids the pipe. You cannot simply do:

while read stendname id ip < "$idfilename"

as this will keep reading the same line. By keeping the file in a file descriptor, you allow the state of the read to be remembered.

For more info, see:
I/O Redirection

As for pipes, I've never tested using a pipe with a file descriptor, but it may be possible. This would allow you to be reading from a pipe without the loop knowing that it's from a pipe (the loop would just see a normal read from a file descriptor).

[flashcoder]

Cabhan!!! You are genius. Srsly... Now this is exactly what I needed..

Seems my lack of experience is lot worse than i thought. Gotta do more RTFM..

[Cabhan]

Haha, no no. I've just encountered all of these problems before you .

The guide that I posted for I/O redirection is actually the best guide to Bash that I've ever seen, and taught me a lot of Bash tricks that I use pretty regularly now. The home page is:
Advanced Bash-Scripting Guide

I highly recommend reading all of it if you're interested in learning more about what you can do with Bash.

[drl]

Hi.

Here are two other solutions. The first is useful with bash: you re-direct into the bottom of the loop:

#!/usr/bin/env bash

# @(#) s1	Demonstrate re-direct into loop.

echo
set +o nounset
LC_ALL=C ; LANG=C ; export LC_ALL LANG
echo "Environment: LC_ALL = $LC_ALL, LANG = $LANG"
echo "(Versions displayed with local utility \"version\")"
version >/dev/null 2>&1 && version "=o" $(_eat $0 $1)
set -o nounset

echo
echo " File sizes:"
ls -laF | awk '{print $5}'

echo
echo " Create separate file, redirect into loop:"

ls -laF | awk '{print $5}' >t1
sum=0
while read fsize
do
  (( sum+=fsize ))
done < t1
echo "$sum"

exit 0

producing:

% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0 
GNU bash 3.2.39

 File sizes:

4096
20480
255
518
495
27

 Create separate file, redirect into loop:
25871

The other is with ksh, which does not create a separate process for this kind of loop:

#!/usr/bin/env ksh

# @(#) s2	Demonstrate ksh not using additional process.

echo
set +o nounset
LC_ALL=C ; LANG=C ; export LC_ALL LANG
echo "Environment: LC_ALL = $LC_ALL, LANG = $LANG"
echo "(Versions displayed with local utility \"version\")"
version >/dev/null 2>&1 && version "=o" $(_eat $0 $1)
set -o nounset

echo
echo " File sizes:"
ls -laF | awk '{print $5}'

echo
echo " Results, ksh:"
sum=0
ls -laF | awk '{print $5}' |
while read fsize
do
  (( sum+=fsize ))
done
echo "$sum"

exit 0

producing:

% ./s2

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0 
ksh 93s+

 File sizes:

4096
20480
255
518
495
27

 Results, ksh:
25871

There's usually more than one way to do things in *nix ... cheers, drl

PS Welcome back Cabhan.