How to convert String to Hex in Shell

[jaigs_27]

Anyone has idea to convert a string to hex.

I have a string containing so many special char and numbers. Due to this I am not able compare them with other strinng in Shell script and my script is failing.

Kindly suggest some way.

[scathefire]

this is how i would do it in PERL, not sure about bash.

#!/usr/bin/perl
use URI::Escape;

# Google
# Encode URL
$url = "http://www.google.com/search?q=book";

$encodedURL = uri_escape($url);

print "Encoded URL: $encodedURL\n\n";

# Decode URL
# www.yahoo.com
$encodedURL =
  "http://%77%77%77%2e%79%61%68%6f%6f%2e%63%6f%6d";

$decodedURL = uri_unescape($encodedURL);
print  "Decoded URL: $decodedURL\n";

Something like that?

[jaigs_27]

Thanks but I have to do it in Bash

[Cabhan]

I guess I'm a bit confused. Why can't you compare them? What are you using right now to compare them? Can you give us a sample of your code and the strings?

[scathefire]

depending on the characters, you could just escape them in the shell using "\", which can be accomplished using sed. In some cases if you can force quotations it can fix some problems. i know quoting it will help when it comes to spaces, not sure about the others. Normally I'd just insert an escape before the offending character.

[jaigs_27]

I guess I'm a bit confused. Why can't you compare them? What are you using right now to compare them? Can you give us a sample of your code and the strings?

VAR1=''abc:123:File:0:4:*:*:1-5:$PATH/bin/test.ksh > $PATH/logs/test.ksh 2>&1
testing testing1 testing2 testing3 $special"

VAR2="abc:123:File:0:4:*:*:1-5:$PATH/bin/test.ksh > $PATH/logs/test.ksh 2>&1
testing testing1 testing2"

echo "$VAR1"|while read VAL;do
if [ -n "`echo $VAR2|grep "$VAL"`" ]; then
echo "Value exist"
else
echo "Value does not exist in VAR2: $VAL"
fi

done


I have two above varibales contaning special chars. Now I want to find out the difference of them, if the values in both variables is not same. How to do that?

[giga_aks]

would you be able to call or come online on yahoo.

sangal_ak04 is my id.

giga_aks

if you can call.. call me, sent you an email, see signature.

[AKS_GIGA]

ok.. here we go. but FLAG variable scope issue is coming due to "|" pipe i think. if someone can help that'll solve Jag's issue.

Lets say file1 is:

[/efare1/home/qabuild/aksutil] $ cat -n j1
     1  dit:abw:fhs:eqs:devqs2m3:Pro Quote Reset:28:01:*:*:1-5:$ET_FS_ROOT/bin/serverCommand "'purgeMarketMaker *'" startOfDay refreshSubscribers
     2  dit:abw:fhs:eqs:devqs2m3:Pro Quote Reset:33:00:*:*:1-5:$ET_FS_ROOT/bin/serverCommand purgeBook startOfDay refreshSubscribers
     3  dit:abw:fhs:eqs:devqs2m3:Pro Iverson File:0:4:*:*:1-5:$ET_INSTANCE_ROOT/bin/readIversonFile.ksh > $ET_INSTANCE_ROOT/logs/readIversonFile.log 2>&1
     4  dit:abw:fhs:eqs:devqs2m3:Get Repair File:15:07:*:*:2-6:etserver getfile spikesymbol.sh
     5  dit:abw:fhs:eqs:devqs2m3:Read Repair File:30:07:*:*:2-6:/etrade/bin/ksh $ET_INSTANCE_ROOT/config/spikesymbol.sh
     6  sit:abc:fhsn:eqs:devqs1m4:Read Repair File:30:07:*:*:2-6:/etrade/bin/ksh $ET_INSTANCE_ROOT/config/testing.sh

Lets say file2 is:

[/efare1/home/qabuild/aksutil] $ cat -n j2
     1  dit:abw:fhs:eqs:devqs2m3:Pro Quote Reset:28:02:*:*:1-5:$ET_FS_ROOT/bin/serverCommand "'purgeMarketMaker *'" startOfDay refreshSubscribers
     2  dit:abw:fhs:eqs:devqs2m3:Pro Quote Reset:33:00:*:*:1-5:$ET_FS_ROOT/bin/serverCommand purgeBook startOfDay refreshSubscribers
     3  dit:abw:fhs:eqs:devqs2m3:Pro Iverson File:0:4:*:*:1-5:$ET_INSTANCE_ROOT/bin/readIversonFile.ksh > $ET_INSTANCE_ROOT/logs/readIversonFile.log 2>&1
     4  dit:abw:fhs:eqs:devqs2m3:Get Repair File:15:07:*:*:2-6:etserver getfile spikesymbol.sh
     5  dit:abw:fhs:eqs:devqs2m3:Read Repair File:30:07:*:*:2-6:/etrade/bin/ksh $ET_INSTANCE_ROOT/config/spikesymbol.sh

Now a script to find all the lines in file1 that are not in file2 (Note: lines contains multiple special characters i.e. ', `, *, so grep command which show its poverty), that why script. if someone else has a solution, pls share.

Script:

[/efare1/home/qabuild/aksutil] $ cat giga.sh
#!/bin/bash

##Global variables
file1="$1";
file2="$2";
file1lines_not_in_file2="";
flag=0;

cat "${file1}" | while read line_of_file1;
do
  ## Set flag variable to ZER0 for every new line from file1.
  flag=0;
  echo "%%% entering newly into loop2";

  cat "${file2}" | while read line_of_file2;
  do
    ##Take the binary form of file1 line in a variable. Binary form conversion is required as string can contain special characters which grep hates.
    f1_line_binary="$(echo "$line_of_file1" | od -b | sed 's/^[0-9][0-9][0-9][0-9][0-9][0-9][0-9] *//;s/$/ /'| tr -d '\012')";

    ##Take the binary form of file2 line in a variable. Binary form conversion is required as string can contain special characters which grep hates.
    f2_line_binary="$(echo "$line_of_file2" | od -b | sed 's/^[0-9][0-9][0-9][0-9][0-9][0-9][0-9] *//;s/$/ /'| tr -d '\012')";

    ## If binary form of line from both files match, Then Set the flag variable to 1 i.e. line of file1 exists in file2 somewhere and Break the inner While loop
    ## as we dont want to process line of file1 to be checked further in file2.
    if [ "giga${f1_line_binary}" == "giga${f2_line_binary}" ];
    then
        flag=1;
        echo  -e "\nOK.. found this line of file1 in file2, flag is set to $flag.";
        echo
        echo "${line_of_file1}" ---------------;
        echo
        break;  ## breaking from inner While loop as this line of file1 exists in file2.
    fi
    echo aks - $flag; read -u 1;

  done


  echo -e "\nfinal flag value after inner loop is $flag.\n";
  if [ "giga$flag" == "giga0" ];
  then
    ## i.e. As this line of file1 doesnt exist in file2 after checking all lines in file2 save it in a variable instead of creating a file.
    ## Sometimes creating a file on a machine IF its a production machine is not allowed.
    if [ -z "$file1lines_not_in_file2" ]; then file1lines_not_in_file2="$line_of_file1"; else file1lines_not_in_file2="`echo "$file1lines_not_in_file2"; echo "$line_of_file1"`"; fi;
  fi
    echo "$file1lines_not_in_file2" ===============;

  echo -e "\n#########################################################################################################";
done

## Show file1lines_not_in_file2 variable on screen (instead of creating a file) showing all the lines In file1 that are not In file2
echo "${file1lines_not_in_file2}" | cat -n ;

exit 0

[/efare1/home/qabuild/aksutil] $

Now, this script logically is doing the stuff. but not generating the output as "flag" variable is loosing its value (if set to 1 in inner loop, i.e. loop for file2)

if someone can solve this, Jags is done with his concern.


Small version of why flag is not setting correctly.. see this script and its output:

[/efare1/home/qabuild/aksutil] $ cat g.sh

#!/bin/bash

##Global variables
file1=$1
file2=$2
export flag=0;


cat "$file1" | while read line_of_file1;
do
  ##flag=0;

  cat "$file2" | while read line_of_file2;
  do
    flag=1;
    echo  -e "Flag is set to $flag.";
  done

  echo -e "\nfinal flag value after inner loop is $flag.\n";
  echo; read -u 1;

done

[/efare1/home/qabuild/aksutil] $ ./g.sh j1 j2

Flag is set to 1.
Flag is set to 1.
Flag is set to 1.
Flag is set to 1.
Flag is set to 1.

final flag value after inner loop is 0.



Flag is set to 1.
Flag is set to 1.
Flag is set to 1.
Flag is set to 1.
Flag is set to 1.

final flag value after inner loop is 0.



Flag is set to 1.
Flag is set to 1.
Flag is set to 1.
Flag is set to 1.
Flag is set to 1.

final flag value after inner loop is 0.



Flag is set to 1.
Flag is set to 1.
Flag is set to 1.
Flag is set to 1.
Flag is set to 1.

final flag value after inner loop is 0.



Flag is set to 1.
Flag is set to 1.
Flag is set to 1.
Flag is set to 1.
Flag is set to 1.

final flag value after inner loop is 0.



Flag is set to 1.
Flag is set to 1.
Flag is set to 1.
Flag is set to 1.
Flag is set to 1.

final flag value after inner loop is 0.




[/efare1/home/qabuild/aksutil] $

[Cabhan]

So you want to check if the strings are equal?

if [ "$VAR1" == "$VAR2" ]; then
    # equal
else
    # not equal
fi

You want to check if each character in $VAR1 is in $VAR2?

for index in $(seq 0 "${#VAR1}"); do
    character=${VAR1:$index:1}
    if echo "$VAR2" | grep -q "$character"; then
        echo "$character is in VAR2"
    else
        echo "$character is not in VAR2"
    fi
done

These require some slightly more advanced Bash, but they are a great deal simpler than re-encoding all of your data.

[GIGAAKS]

I think simple command "comm" solved the request. Thanks to Pawan Sangal.

I should have used arrays in my shell script to squeeze the script. Comm command in Unix/Linux is working well even when there are different lines in file j2 as comm is not showing them, so we are fine as we just need to find lines only in file j1.

 MAN page
 
COMM(1)                               FSF                              COMM(1)
 
 
 
NAME
 
       comm - compare two sorted files line by line
 
 
 
SYNOPSIS
 
       comm [OPTION]... LEFT_FILE RIGHT_FILE
 
 
 
DESCRIPTION
 
       Compare sorted files LEFT_FILE and RIGHT_FILE line by line.
 
 
 
       -1     suppress lines unique to left file
 
 
 
       -2     suppress lines unique to right file
 
 
 
       -3     suppress lines that appear in both files 
  

Ok found that “comm” didn’t work in all cases (without using sort on the files, see below), The red line should not come in the output.

[/efare1/home/qabuild/aksutil] $ cat -n j1; echo; cat -n j2

     1  dit:abw:fhs:eqs:devqs2m3:Pro Quote Reset:28:01:*:*:1-5:$ET_FS_ROOT/bin/serverCommand "'purgeMarketMaker *'" startOfDay refreshSubscribers

     2  dit:abw:fhs:eqs:devqs2m3:Pro Quote Reset:33:00:*:*:1-5:$ET_FS_ROOT/bin/serverCommand purgeBook startOfDay refreshSubscribers

     3  dit:abw:fhs:eqs:devqs2m3:Pro Iverson File:0:4:*:*:1-5:$ET_INSTANCE_ROOT/bin/readIversonFile.ksh > $ET_INSTANCE_ROOT/logs/readIversonFile.log 2>&1

     4  dit:abw:fhs:eqs:devqs2m3:Get Repair File:15:07:*:*:2-6:etserver getfile spikesymbol.sh

     5  dit:abw:fhs:eqs:devqs2m3:Read Repair File:30:07:*:*:2-6:/etrade/bin/ksh $ET_INSTANCE_ROOT/config/spikesymbol.sh

     6  sit:abc:fhsn:eqs:devqs1m4:Read Repair File:30:07:*:*:2-6:/etrade/bin/ksh $ET_INSTANCE_ROOT/config/testing.sh

 

     1  dit:abw:fhs:eqs:devqs2m3:Pro Quote Reset:28:02:*:*:1-5:$ET_FS_ROOT/bin/serverCommand "'purgeMarketMaker *'" startOfDay refreshSubscribers

     2  dit:abw:fhs:eqs:devqs2m3:Pro Quote Reset:33:00:*:*:1-5:$ET_FS_ROOT/bin/serverCommand purgeBook startOfDay refreshSubscribers

     3  dit:abw:fhs:eqs:devqs2m3:Pro Iverson File:0:4:*:*:1-5:$ET_INSTANCE_ROOT/bin/readIversonFile.ksh > $ET_INSTANCE_ROOT/logs/readIversonFile.log 2>&1

     4  dit:abw:fhs:eqs:devqs2m3:Get Repair File:15:07:*:*:2-6:etserver getfile spikesymbol.sh

     5  dit:abw:fhs:eqs:devqs2m3:Read Repair File:30:07:*:*:2-6:/etrade/bin/ksh $ET_INSTANCE_ROOT/config/spikesymbol.sh

     6  GIGA.AKS

     7  dit:abw:fhs:eqs:devqs2m3:Pro Quote Reset:28:01:*:*:1-5:$ET_FS_ROOT/bin/serverCommand "'purgeMarketMaker *'" startOfDay refreshSubscribers

[/efare1/home/qabuild/aksutil] $ comm -23 j1 j2

[COLOR="Red"]dit:abw:fhs:eqs:devqs2m3:Pro Quote Reset:28:01:*:*:1-5:$ET_FS_ROOT/bin/serverCommand "'purgeMarketMaker *'" startOfDay refreshSubscribers[/COLOR]
sit:abc:fhsn:eqs:devqs1m4:Read Repair File:30:07:*:*:2-6:/etrade/bin/ksh $ET_INSTANCE_ROOT/config/testing.sh

 

[/efare1/home/qabuild/aksutil] $

BUT, if we use sort on files as, it works perfectly. Jagbir try karke dhek le… ab.

[/efare1/home/qabuild/aksutil] $ comm -23 <(sort j1) <(sort j2)

sit:abc:fhsn:eqs:devqs1m4:Read Repair File:30:07:*:*:2-6:/etrade/bin/ksh $ET_INSTANCE_ROOT/config/testing.sh

 

[/efare1/home/qabuild/aksutil] $

so the answer to this question is "comm" command with sort. See if "uniq" can take this request to another level.. i.e. what if one of the file has multiple lines which are same i.e. duplicate lines within same file.