Passing Variables to wget

[Azz]

Hi
I need to download a number of web pages save them to disk and then use gawk to extract info from the pages, the gawk bit is not a problem. I've got a file that lists the web page URL. Each URL is assigned to an incremented variable. I have a FOR loop which should (in theory) pass each variable to wget. The problem is I do not know how to pass the incremented variable to wget -

#!/bin/bash

a1="http://wasd.gre.uk/sd?HTT:IPP:R231"
a2="http://wasd.gre.uk/sd?HTT:IPP:R1445"
a3="http://wasd.gre.uk/sd?HTT:IPP:R891"

nf = 3

for (( i=1; i<=nf; i++ ))
do
wget ${ai}

done

If anyone can shed any light on where I'm going wrong will be appreciated.

Thanks

[bigtomrodney]

If the variables are all working and being generated (test this with echo) then you should try

wget a$i

instead of ${ai}

[Azz]

I've tried your suggestion, and it gives the the desired variable name, but in the process it loses the variable value -

a1="http://wasd.gre.uk/sd?HTT:IPP:R231"

nf = 1

for (( i=1; i<=nf; i++ ))
do
echo $a1
echo a$i
done

> http://wasd.gre.uk/sd?HTT:IPP:R231
> a1

[Franklin52]

Use arrays:

#!/bin/bash

a[1]="http://wasd.gre.uk/sd?HTT:IPP:R231"
a[2]="http://wasd.gre.uk/sd?HTT:IPP:R1445"
a[3]="http://wasd.gre.uk/sd?HTT:IPP:R891"

nf = 3

for (( i=1; i<=nf; i++ ))
do
  wget ${a[i]}
done

Regards

[drl]

Hi.

A solution that is not very general, but works for small sets is:

#!/bin/bash

# @(#) s1       Demonstrate for loop.

a1="http://wasd.gre.uk/sd?HTT:IPP:R231"
a2="http://wasd.gre.uk/sd?HTT:IPP:R1445"
a3="http://wasd.gre.uk/sd?HTT:IPP:R891"

for i in $a1 $a2 $a3
do
  echo wget $i
done

Which produces:

% ./s1
wget http://wasd.gre.uk/sd?HTT:IPP:R231
wget http://wasd.gre.uk/sd?HTT:IPP:R1445
wget http://wasd.gre.uk/sd?HTT:IPP:R891

cheers, drl

[drl]

Hi.

A more general solution, that keeps your loop style is (with corrected assignment):

#!/bin/bash

# @(#) s2       Demonstrate eval: re-scan a command line.

a1="http://wasd.gre.uk/sd?HTT:IPP:R231"
a2="http://wasd.gre.uk/sd?HTT:IPP:R1445"
a3="http://wasd.gre.uk/sd?HTT:IPP:R891"

nf=3

for (( i=1; i<=nf; i++ ))
do
  eval echo wget \$a${i}
done

which produces:

% ./s2
wget http://wasd.gre.uk/sd?HTT:IPP:R231
wget http://wasd.gre.uk/sd?HTT:IPP:R1445
wget http://wasd.gre.uk/sd?HTT:IPP:R891

cheers, drl

[Azz]

Thanks to all that replied.

drl - your a diamond, it works beautifully - Thanks

[drl]

Hi.

You can take advantage of special array syntax to assign contents easily:

#!/bin/bash

# @(#) s3       Demonstrate easy array assignment.

echo " Version $BASH_VERSION"

a=(
"http://wasd.gre.uk/sd?HTT:IPP:R231"
"http://wasd.gre.uk/sd?HTT:IPP:R1445"
"http://wasd.gre.uk/sd?HTT:IPP:R891"
)

nf=${#a[@]}
echo " Obtaining $nf files."

for (( i=0; i<nf; i++ ))
do
  echo wget ${a[i]}
done

exit 0

Producing:

% ./s3
 Version 2.05b.0(1)-release
 Obtaining 3 files.
wget http://wasd.gre.uk/sd?HTT:IPP:R231
wget http://wasd.gre.uk/sd?HTT:IPP:R1445
wget http://wasd.gre.uk/sd?HTT:IPP:R891

Best wishes ... cheers, drl

[Azz]

Hi DRL

You've given 2 versions, which do you think is the best with respect to computer processing time? In reality I have about 700 URL's that I have to process.
Also do you think my approach of using a FOR loop is good, or would you use something different?

Many thanks for your help

Az

[drl]

Hi, Azz.

The time for the processing of the loop in any of the versions is very small compared to the wget, so choose whichever one you like. I like the 3rd one because it would be very easy to paste URLs in, or even to place the URLs in with another script.

If you get ambitious, try them all on the same 10 or so URLs, and I think you will see that the times are dominated by the wget ... cheers, drl