Looking for a better faster search and replace text within a file method

[habskilla]

This works but is veeeerrrrry slow. What I'm trying to do is replace each instance of externalId with different unique ID.

 while read line; do echo $line | sed 's/externalId="[^"]*"/externalId="'USR.`cat /dev/urandom | tr -dc a-z0-9 | head -c32`'"/' >> another.txt; done < 2009-07-21-ruleExport2.xml

Anybody got a better faster way to do this?

Edit: Let me explain in more detail of what I'm trying to accomplish.
I have an XML file with system generated externalIDs.
I need to replace each externalID with another unique 16 digit number.

Each time this code is called results in an unique 16 digit number.

cat /dev/urandom | tr -dc a-z0-9 | head -c32

this sed will find externalIDs and replace it with an unique ID:

sed 's/externalId="[^"]*"/externalId="'USR.`cat /dev/urandom | tr -dc a-z0-9 | head -c32`'"/

So all I need to do is:

• Read in a line of xml
• Interrogate the line using the sed
• if an externalID is found, a unique number is generated and replaces the ID
• Read in the next line of xml
• Interrogate the line using the sed
• if an externalID is found, a unique number is generated and replaces the ID. Since we're making another call to /dev/urandom a new unique ID is generated.
• Rinse and repeat.

Example input Text

 <CommandExecutionCaptureRule id="ID70" oid="-1y2p0ij0tfa3k:-1y2p0iizevuy4" class="rule" name="rpm -qa" externalId="USR.ppi0iut9s6iedixxd1440u62eknlpvx8">
        <Description></Description>
        <Severity>0</Severity>
        <RealTime>false</RealTime>
        <Actions />
        <ElementName>rpm -qa</ElementName>
        <CommandLine>rpm -qa | sort</CommandLine>
        <ExcludePattern></ExcludePattern>
        <ExcludeReplace></ExcludeReplace>
    </CommandExecutionCaptureRule>
    <RuleGroup id="ID58" oid="-1y2p0ij32e8cq:-1y2p0iiycq5qo" class="rulegroup" name="Crontab">
        <Description></Description>
        <Children>
            <Child refid="ID71" />
        </Children>
    </RuleGroup>
    <CommandExecutionCaptureRule id="ID71" oid="-1y2p0ij0tfa3k:-1y2p0iiyrpnez" class="rule" name="crontab " externalId="USR.ppi0iut9s6iedixxd1440u62eknlpvx8">

Example output:

<CommandExecutionCaptureRule id="ID70" oid="-1y2p0ij0tfa3k:-1y2p0iizevuy4" class="rule" name="rpm -qa" externalId="USR.g1a1jzuf8hemu0rdwxnu47hbr58xr3vl">
        <Description></Description>
        <Severity>0</Severity>
        <RealTime>false</RealTime>
        <Actions />
        <ElementName>rpm -qa</ElementName>
        <CommandLine>rpm -qa | sort</CommandLine>
        <ExcludePattern></ExcludePattern>
        <ExcludeReplace></ExcludeReplace>
    </CommandExecutionCaptureRule>
    <RuleGroup id="ID58" oid="-1y2p0ij32e8cq:-1y2p0iiycq5qo" class="rulegroup" name="Crontab">
        <Description></Description>
        <Children>
            <Child refid="ID71" />
        </Children>
    </RuleGroup>
    <CommandExecutionCaptureRule id="ID71" oid="-1y2p0ij0tfa3k:-1y2p0iiyrpnez" class="rule" name="crontab " externalId="USR.adc82uzvxapztrbzuifrvmq3a99bprv0">

[secondmouse]

Try running sed a bit differently:

sed 's/externalId="[^"]*"/externalId="'USR.`cat /dev/urandom | tr -dc a-z0-9 | head -c32`'"/' 2009-07-21-ruleExport2.xml >>another.txt

[secondmouse]

In my previous reply I actually dropped the g in the end of the sed line, without it sed will just replace the first externalId's value into a random string. Here's the corrected one:

sed 's/externalId="[^"]*"/externalId="'USR.`cat /dev/urandom | tr -dc a-z0-9 | head -c32`'"/g' 2009-07-21-ruleExport2.xml >>another.txt

[habskilla]

Thanks secondmouse, but your code will replace all externalIDs in the file with one new ID.

[secondmouse]

Apologize for not reading the sample input carefully. Try this python script instead

#!/usr/bin/env python
from random import choice
import re,fileinput,string

def genrndstr(size):
  return ''.join( [ choice(string.letters[0:26]+string.digits) for i in range(size) ] )


if __name__=='__main__':
  p=re.compile(r'externalId=".*"')
  for line in fileinput.input():
    if p.search(line):
      newid='externalId="USR.%s"' % genrndstr(32)
      print( '%s' % p.sub(newid, line) ),
    else:
      print( '%s' % line ),

Name it as sr.py. To run it,

python sr.py 2009-07-21-ruleExport2.xml >> another.txt

On my system it's almost 20 times as fast as the sed method:

user@linux ~/sed_r $ time (for i in `seq 1 50`; do while read line; do echo $line | sed 's/externalId="[^"]*"/externalId="'USR.`cat /dev/urandom | tr -dc a-z0-9 | head -c32`'"/' > another.txt; done < 2009-07-21-ruleExport2.xml; done)

real	0m43.030s
user	0m9.249s
sys	0m40.343s
user@linux ~/sed_r $ time (for i in `seq 1 50`; do python sr.py 2009-07-21-ruleExport2.xml > another1.txt; done)

real	0m2.448s
user	0m1.916s
sys	0m0.452s

[habskilla]

secondmouse, no apology needed. Your script is extremely fast and it keeps the formatting of the xml.

Thanks

[secondmouse]

No problem. Glad it helps