Piping filenames to sed or bash (longish post)

[qnr]

OK, I have a question for you scripters, but I'm going to make it long and drawn out, so that people can see what I'm doing if they want to experiment.

 

The question is, how do I pipe a directory listing to sed, or a shell script? I have a directory with about 3800 files that I want to modify. The details are at the bottom of the post if you want to get straight to my question. I want to modify them with sed, because I need the experience for some other stuff I'm working on. Now, I can modify them with sed like this:

 

 

First, let's create a text file called base.txt with four lines:

 

sh-2.05b$ echo -e "terry kevin rossnkaren rose lambottnthis line should be deletednboomer and alinak" > base.txt; cat base.txt 

terry kevin ross

karen rose lambott

this line should be deleted

boomer and alinak

For info, the echo -e"

 

Ok, now I'm going to use a little shell script I threw together to duplicate

base.txt 20 times - here's the script:

 

#!/bin/bash

# Useless program to duplicate a file an arbitrary number of times

# $1=basename $2=dupename $3=extension $4=number of iterations



x=0

while [ "$x" -lt "$4" ]

do

       cp $1 $2$x$3

       x=`expr $x + 1`

done



echo



exit 0

 

Now I run it to get 20 different copies of base.txt

sh-2.05b$ ls

base.txt  dupe.sh

sh-2.05b$ dupe.sh base.txt copy .txt 20



sh-2.05b$ ls

base.txt    copy12.txt  copy16.txt  copy1.txt  copy5.txt  copy9.txt

copy0.txt   copy13.txt  copy17.txt  copy2.txt  copy6.txt  dupe.sh

copy10.txt  copy14.txt  copy18.txt  copy3.txt  copy7.txt

copy11.txt  copy15.txt  copy19.txt  copy4.txt  copy8.txt

 

Now, I'm going to create a sed program file named a.sed:

 

sh-2.05b$ echo -e "s/kevin/KEVIN/ns/karen/KAREN/n/delete/dn" > a.sed

sh-2.05b$ cat a.sed

s/kevin/KEVIN/

s/karen/KAREN/

/delete/d

For info: I'm changing "kevin" to "KEVIN", "karen" to "KAREN", and deleting any lines that contain "delete"

 

 

Now, I'm going to create a file named process files:

 

sh-2.05b$ ls copy* | sed 's/.*/cat & | sed -f a.sed > tmp; mv -f tmp &/' > process

sh-2.05b$ cat process 

cat copy0.txt | sed -f a.sed > tmp; mv -f tmp copy0.txt

cat copy10.txt | sed -f a.sed > tmp; mv -f tmp copy10.txt

cat copy11.txt | sed -f a.sed > tmp; mv -f tmp copy11.txt

cat copy12.txt | sed -f a.sed > tmp; mv -f tmp copy12.txt

cat copy13.txt | sed -f a.sed > tmp; mv -f tmp copy13.txt

cat copy14.txt | sed -f a.sed > tmp; mv -f tmp copy14.txt

cat copy15.txt | sed -f a.sed > tmp; mv -f tmp copy15.txt

cat copy16.txt | sed -f a.sed > tmp; mv -f tmp copy16.txt

cat copy17.txt | sed -f a.sed > tmp; mv -f tmp copy17.txt

cat copy18.txt | sed -f a.sed > tmp; mv -f tmp copy18.txt

cat copy19.txt | sed -f a.sed > tmp; mv -f tmp copy19.txt

cat copy1.txt | sed -f a.sed > tmp; mv -f tmp copy1.txt

cat copy2.txt | sed -f a.sed > tmp; mv -f tmp copy2.txt

cat copy3.txt | sed -f a.sed > tmp; mv -f tmp copy3.txt

cat copy4.txt | sed -f a.sed > tmp; mv -f tmp copy4.txt

cat copy5.txt | sed -f a.sed > tmp; mv -f tmp copy5.txt

cat copy6.txt | sed -f a.sed > tmp; mv -f tmp copy6.txt

cat copy7.txt | sed -f a.sed > tmp; mv -f tmp copy7.txt

cat copy8.txt | sed -f a.sed > tmp; mv -f tmp copy8.txt

cat copy9.txt | sed -f a.sed > tmp; mv -f tmp copy9.txt

 

Now, I'm going to execute process:

 

sh-2.05b$ echo "Before process:"; cat copy14.txt; sh process; echo -e "nAfter process:"; cat copy14.txt 

Before process:

terry kevin ross

karen rose lambott

this line should be deleted

boomer and alinak



After process:

terry KEVIN ross

KAREN rose lambott

boomer and alinak

And there we have it, I only compared copy14.txt before and after, but all 20 copy files were changed.

 

=====================================================

 

OK, so that's one way of doing it. Now, I'd like it if someone could tell me how to pipe a list of filenames into a shell script (without using For or somthing of that nature). I vaguely remember remember doing it in the past - unless I'm just imagining it.

 

Instead of using what I did up above and making the process file, I'd like to pipe the filenames into a script like this:

 

sh-2.05b$ cat sedprogram.sh 

#!/bin/sh

# An example program



# First let's ensure that the temporary file is a unique random filename

TMP_FILE="/tmp/`basename $1`.$$.$RANDOM"



# Delete it, just in case

rm -rf $TMP_FILE



# Copy the file we are working on to the temporary file

cp $1 $TMP_FILE



# Now, work on the temp file, then copy it back to the original

sed -e '/delete/d' -e 's/kevin/KEVIN/' -e 's/karen/KAREN/'  $TMP_FILE > $1



# Delete the temporary file

rm -rf $TMP_FILE

 

If I were to execute it manually, for example:

sedprogram.sh copy14.txt

[aru]

OK, so that's one way of doing it.  Now, I'd like it if someone could tell me how to pipe a list of filenames into a shell script (without using For or somthing of that nature).  

 

Why w/o a for loop?

 

The way to go is:

for file in dir; do stuff with $file; done

 

All your above work can be achieved in a couple of lines with a for loop. There is no need to use the ls command at all

 

If you insist in going without a for loop, then you need to put a while loop within your script, something like this:

 

/sbin/ls | script

 

where script would be:

 

while read file; do stuff with "$file"; done

[qnr]

actually, it was to help my feeble memory.... I know how to do it with a for loop, but I'm almost certain I used to be able to pipe the output of ls or dir into sed and gawk And as far as all the work above, most of it was a set up for the demonstration. If you look at it, I can stream edit the whole 3,800 files with just a couple of the lines up above.

 

Thanks for the reply aru, I'll most likely end up doing it that way. Knowing me, I'm probably imagining having done it the other way anyway.... or maybe I did something similar with TOS or something, and I'm getting my platforms confused :shock:

[aru]

...but I'm almost certain I used to be able to pipe the output of ls or dir into sed and gawk ...

I believe that awk can do the job directly because is a full programing lenguage, but I'm almost sure that sed doesn't ;)

[qnr]

I really should have said "bash" I guess in this case, since that's where I want to pipe it to. The files aren't really formatted enough to justify using gawk. I mostly wanted to know about sed I could just experiment to my heart's content (which, admittedly, I can do with for;). For that matter, I could just write a script to simulate piping, just to convince myself that I'm not really senile, and I have been able to do it in the past :)

 

Edit: Just did a wc -l ... turns out there are 18,000 files

[theYinYeti]

Sorry, I don't have time right now, but if nobody else answers the way you want it, and if it is not urgent, feel free to send an email to me (or PM).

 

Yves.