Trying to convert mass of textfiles from "Windows" to Unix encoding.

[Anderen2]

Well, the title says it all
What I am trying to do is to convert a mass of files from standard Windows text encoding to UNIX encoding(462 .txt files). After a bit of searching, I found this bit of code, changed the necessary parts, put in "Sleep" so I could see the output and ran it:

#!/bin/bash
FROM=iso-8859-1
TO=UTF-8
ICONV="iconv -f $FROM -t $TO"
# Convert
find /home/gmod/adv_duplicator/ -type f -print -name "*" | while read fn;
do cp ${fn} ${fn}.bak
$ICONV < ${fn}.bak > ${fn}
rm ${fn}.bak
done
sleep 30

The problem is that it failed
This is the output that I found in console:

/home/gmod/Skrivebord/as.sh: line 8: ${fn}.bak: ambiguous redirect
rm: cannot remove «/home/gmod/adv_duplicator/STEAM_0_1_32704654/bat»: No such file or directory
rm: cannot remove «mobile»: No such file or directory
rm: cannot remove «3000.txt.bak»: No such file or directory
cp: target «complete.txt.bak» is not a catalog
/home/gmod/Skrivebord/as.sh: line 8: ${fn}.bak: ambiguous redirect
rm: cannot remove «/home/gmod/adv_duplicator/STEAM_0_1_11360145/space»: No such file or directory
rm: cannot remove «ship»: No such file or directory
rm: cannot remove «not»: No such file or directory
rm: cannot remove «complete.txt.bak»: No such file or directory
+more of the same

"ambiguous redirect" is after what Ive found out an error because I am trying to use a command which only works on single files, but is "getting more in its input".
Can anyone help me with this?

Thanks in advance

[barriehie]

Well, the title says it all
What I am trying to do is to convert a mass of files from standard Windows text encoding to UNIX encoding(462 .txt files). After a bit of searching, I found this bit of code, changed the necessary parts, put in "Sleep" so I could see the output and ran it:

#!/bin/bash
FROM=iso-8859-1
TO=UTF-8
ICONV="iconv -f $FROM -t $TO"
# Convert
find /home/gmod/adv_duplicator/ -type f -print -name "*" | while read fn;
do cp ${fn} ${fn}.bak
$ICONV < ${fn}.bak > ${fn}
rm ${fn}.bak
done
sleep 30

...snip...

Try this:

#!/bin/bash
FROMM=iso-8859-1
TOO=UTF-8
# Convert
find /home/gmod/adv_duplicator/ -type f -print -name "*" | while read fn;
do cp ${fn} ${fn}.bak
iconv -f $FROMM -t $TOO $fn.bak > $fn
# rm ${fn}.bak  - I'ld do this after verifing everything is okay!
done
sleep 30

And if all that fails I know gawk can do it:

gawk '{ sub("\r$", ""); print }' winfile.txt > unixfile.txt

[Anderen2]

Hi! Sorry for late response, been busy .
Okay tried both your methods, but unfortunatly none worked
#1:

/home/gmod/Skrivebord/sa.sh: line 7: $fn: ambiguous redirect
cp: target «machine.txt.bak» is not a catalog
/home/gmod/Skrivebord/sa.sh: line 7: $fn: ambiguous redirect
cp: target «2.txt.bak» is not a catalog
/home/gmod/Skrivebord/sa.sh: line 7: $fn: ambiguous redirect
cp: target «wagon.txt.bak»is not a catalog
/home/gmod/Skrivebord/sa.sh: line 7: $fn: ambiguous redirect
cp: target «home.txt.bak» is not a catalog
/home/gmod/Skrivebord/sa.sh: line 7: $fn: ambiguous redirect
cp: target «bugs.txt.bak» is not a catalog
/home/gmod/Skrivebord/sa.sh: line 7: $fn: ambiguous redirect
cp: target «kart.txt.bak» is not a catalog
/home/gmod/Skrivebord/sa.sh: line 7: $fn: ambiguous redirect
cp: target <t.txt.bak» is not a catalog
/home/gmod/Skrivebord/sa.sh: line 7: $fn: ambiguous redirect
cp: target «3000.txt.bak» is not a catalog
/home/gmod/Skrivebord/sa.sh: line 7: $fn: ambiguous redirect
cp: target «complete.txt.bak»is not a catalog
/home/gmod/Skrivebord/sa.sh: line 7: $fn: ambiguous redirect
+++

#2:

echo /home/anderen2/Skrivebord/convert
cd /home/anderen2/Skrivebord/convert
sudo gawk '{ sub("\r$", ""); print }' * > ./unixfile
sleep 30

returns:

/home/anderen2/Skrivebord/convert
[sudo] password for anderen2: 
gawk: cmd. line:1: fatal: cannot open file `=Public Folder=' for reading (Inappropriate ioctl for device)

Btw, here is the folder which I am trying to convert:
Convert folder:

anderen2@ubuntu:~/Skrivebord/convert$ dir
=Public\ Folder=    STEAM_0_0_23750699    STEAM_0_1_16187122  STEAM_0_1_31251130
STEAM_0_0_13458808  STEAM_0_0_24605801    STEAM_0_1_17492839  STEAM_0_1_31382123
STEAM_0_0_15217801  STEAM_0_0_25415433    STEAM_0_1_17897803  STEAM_0_1_32141218
STEAM_0_0_15973405  STEAM_0_0_27347352    STEAM_0_1_18845917  STEAM_0_1_32159843
STEAM_0_0_16324415  STEAM_0_0_28364358    STEAM_0_1_20745652  STEAM_0_1_32592959
STEAM_0_0_18472930  STEAM_0_0_28377716    STEAM_0_1_21982955  STEAM_0_1_32704654
STEAM_0_0_18729357  STEAM_0_0_28531472    STEAM_0_1_25910477  STEAM_0_1_33125795
STEAM_0_0_18881452  STEAM_0_0_31886959    STEAM_0_1_26317246  STEAM_0_1_34491390
STEAM_0_0_19144830  STEAM_0_1_11360145    STEAM_0_1_27243895
STEAM_0_0_19407299  STEAM_0_1_14097933    STEAM_0_1_27879973
STEAM_0_0_19907748  STEAM_0_1_14595300    STEAM_0_1_28402009
STEAM_0_0_21746922  STEAM_0_1_14604192    STEAM_0_1_30904764

And inside each "Steam..." folder is a bunch of .txt in the wrong encoding, and some has even more folders in it.

Thanks

[barriehie]

Okay, try this, modified a bit. If you know how many dir levels deep they go you can add -maxdepth to the find options to get all of them.

#!/bin/bash
FROMM=iso-8859-1
TOO=UTF-8
# Convert
find /home/gmod/adv_duplicator/ -type f -print -name "*" | while read fn;
do
filename=$fn
cp $filename $filename.bak
iconv -f $FROMM -t $TOO $filename.bak > $filename
# rm $filename.bak  - I'ld do this after verifing everything is okay!
done
sleep 5

Maybe zip a few of the smaller ones and attach. I don't have any windows type anything on here.

[Anderen2]

Tried it, but added --verbose to the cp and iconv command to see what they returned:

#!/bin/bash
FROMM=iso-8859-1
TOO=UTF-8
# Convert
find /home/anderen2/Skrivebord/convert -type f -print -name "*" | while read fn;
do
filename=$fn
cp --verbose $filename $filename.bak
iconv --verbose -f $FROMM -t $TOO $filename.bak > $filename
# rm $filename.bak  - I'ld do this after verifing everything is okay!
done
sleep 500

Returns this:

anderen2/Skrivebord/convert/STEAM_0_0_28531472/glados.txt.bak.bak»
/home/anderen2/Skrivebord/convert/STEAM_0_0_28531472/glados.txt.bak.bak:
«/home/anderen2/Skrivebord/convert/STEAM_0_0_28531472/glados.txt» -> «/home/anderen2/Skrivebord/convert/STEAM_0_0_28531472/glados.txt.bak»
/home/anderen2/Skrivebord/convert/STEAM_0_0_28531472/glados.txt.bak:
cp: målet «wagon.txt.bak» er ikke en katalog
/home/anderen2/Skrivebord/sa.sh: line 9: $filename: ambiguous redirect
«/home/anderen2/Skrivebord/convert/STEAM_0_0_28531472/rolerkaster.txt» -> «/home/anderen2/Skrivebord/convert/STEAM_0_0_28531472/rolerkaster.txt.bak»
/home/anderen2/Skrivebord/convert/STEAM_0_0_28531472/rolerkaster.txt.bak:
«/home/anderen2/Skrivebord/convert/STEAM_0_0_16324415/faef.txt.bak» -> «/home/anderen2/Skrivebord/convert/STEAM_0_0_16324415/faef.txt.bak.bak»
/home/anderen2/Skrivebord/convert/STEAM_0_0_16324415/faef.txt.bak.bak:
«/home/anderen2/Skrivebord/convert/STEAM_0_0_16324415/faef.txt» -> «/home/anderen2/Skrivebord/convert/STEAM_0_0_16324415/faef.txt.bak»
/home/anderen2/Skrivebord/convert/STEAM_0_0_16324415/faef.txt.bak:
cp: målet «home.txt.bak» er ikke en katalog
/home/anderen2/Skrivebord/sa.sh: line 9: $filename: ambiguous redirect
cp: målet «bugs.txt.bak» er ikke en katalog
/home/anderen2/Skrivebord/sa.sh: line 9: $filename: ambiguous redirect
«/home/anderen2/Skrivebord/convert/STEAM_0_0_19144830/helicopter.txt.bak» -> «/home/anderen2/Skrivebord/convert/STEAM_0_0_19144830/helicopter.txt.bak.bak»
/home/anderen2/Skrivebord/convert/STEAM_0_0_19144830/helicopter.txt.bak.bak:
«/home/anderen2/Skrivebord/convert/STEAM_0_0_19144830/helicopter.txt» -> «/home/anderen2/Skrivebord/convert/STEAM_0_0_19144830/helicopter.txt.bak»
/home/anderen2/Skrivebord/convert/STEAM_0_0_19144830/helicopter.txt.bak:
cp: målet «kart.txt.bak» er ikke en katalog
/home/anderen2/Skrivebord/sa.sh: line 9: $filename: ambiguous redirect
cp: målet «jet.txt.bak» er ikke en katalog
/home/anderen2/Skrivebord/sa.sh: line 9: $filename: ambiguous redirect
cp: målet «3000.txt.bak» er ikke en katalog
/home/anderen2/Skrivebord/sa.sh: line 9: $filename: ambiguous redirect
cp: målet «complete.txt.bak» er ikke en katalog
/home/anderen2/Skrivebord/sa.sh: line 9: $filename: ambiguous redirect
«/home/anderen2/Skrivebord/convert/STEAM_0_1_25910477/asus.txt.bak» -> «/home/anderen2/Skrivebord/convert/STEAM_0_1_25910477/asus.txt.bak.bak»
/home/anderen2/Skrivebord/convert/STEAM_0_1_25910477/asus.txt.bak.bak:
«/home/anderen2/Skrivebord/convert/STEAM_0_1_25910477/asus.txt» -> «/home/anderen2/Skrivebord/convert/STEAM_0_1_25910477/asus.txt.bak»
/home/anderen2/Skrivebord/convert/STEAM_0_1_25910477/asus.txt.bak:
+++

Heres the folder which I'm trying to convert(.zip):
2shared.com/file/Y3mPiR5X/convertq.html (Had to remove "h t t p : / /w w w" because I didn't have enough posts...)
Its an gameserver(GMod) save folder with 588 .txt files with various sizes and folder depth

Thanks ; )

[barriehie]

Gottit, I'll see what I can do.

[barriehie]

Umm, we were overthinking this a bit... (run from the top of the dir tree)

#!/bin/bash
#

find . -type f -iname "*.txt" | while read file
do
  sed -e 's/\r$//' "$file" > "$file".fixed
done

05:06:35 /home/barrie/convert/STEAM_0_1_34491390 $ > wc --lines ./*
    63 ./base1.txt
    63 ./base1.txt.fixed
    65 ./car.txt
    65 ./car.txt.fixed
    66 ./lolz.txt
    66 ./lolz.txt.fixed
    98 ./motel.txt
    98 ./motel.txt.fixed
    53 ./pool.txt
    53 ./pool.txt.fixed
    56 ./prototype2.txt
    56 ./prototype2.txt.fixed
    84 ./prototype.txt
    84 ./prototype.txt.fixed
    39 ./rocket tower.txt
    39 ./rocket tower.txt.fixed
  1048 total
05:06:40 /home/barrie/convert/STEAM_0_1_34491390 $ >

I checked in ghex2 and the line endings are now 0A.

HTH