Help with Bash redirection (I don't get it).

[ramin.honary]

I use Bash redirection all the time, usually something like:
this "locate ... 2>&1 | less"
or this "make ... &>build.out".

But I then I got to thinking...

Could I use the "tee" program to get Bash to copy stdout and stderr to separate files, but also write both stdout and stderr to the terminal's stdout without using FIFO's?

# This is not correct:
./simple 2>&1 | tee out err

# this is correct, but I don't want to use FIFO's:
mkfifo .err .out
tee <.err err &
tee <.out out &
./simple >.out 2>.err
rm .err .out

I created a script called "simple":

#!/bin/bash
echo "This is NOT an error."
echo "This is an ERROR." >&2

And here is the command I used to try to make it work:

( # Execute in a sub-shell.
  exec 3>&2    #uses POSIX "dup2()" to copy stderr to fd3
  ./simple \   #writes to stderr (stderr is also fd3), pipes stdout to sub-shell
  | ( #this sub-shell's stdin is "simple's" stdout, and inherits fd3 from parent
      tee err 0>&- 0<&3 \   #close stdin, "dup2()" copies fd3 to stdin
        &                   #fork "tee" to background
      tee out  #takes stdin normally, writes to stdout normally
    )
)

FAIL! It almost worked, but I just can't get that "tee" program to read from file descriptor 3. I did read the Bash manual. According to them:

The redirection operator:

[n]<&word

is used to duplicate input file descriptors. If word expands to one or more digits, the file descriptor denoted by n is made to be a copy of that file descriptor.

I take this to mean that Bash is internally using the POSIX "dup2()" system call, (though I am not 100% sure about that). And I know bash can execute sub-shells using the "fork()" system call, and it forks whenever you execute a pipe, background process, or parenthetical sub-shell. And I know that whenever a "fork()" system call is executed, the child processes inherits all open file descriptors. So I guess everything I know is wrong (because the Bash manual sucks).

I am starting to see that I have made some wrong assumptions of how Bash works under the hood, as I have tried every incarnation of that script I could think of, changing the order of executing the "tee's", starting a sub-process and doing "exec 0<&3" before running "tee", putting everything in a single pipeline, nothing seems to work. Perhaps bash does not allow file descriptors to be inherited like they normally are with the C language.

I am starting to think what I want to do is impossible using Bash's ordinary redirection facilities. Anyone have any ideas?

[Kloschüssel]

*hm* Can't wrap my head around it without using fifos.

Assuming you have a script like that:

$ cat > ./simple
echo "noerr"
echo "error" >&2
^D

Then this should work:

$ ./simple > >( tee out1 ) 2> >( tee err1 >&2)

$ tail *
==> err1 <==
error

==> out1 <==
noerr

==> simple <==
echo "noerr"
echo "error" >&2

The key are the ">( )", which implicitly creates a fifo for the inner process to listen on.

PS: ^D is an alias for CTRL+D keyboard combo

[ramin.honary]

The key are the ">( )", which implicitly creates a fifo for the inner process to listen on.

This was very helpful, thank you!

I was not aware of this syntax at all! I looked it up just now in the Bash man page, and there it is under the "Process Substitution" section. Funny, I've gone all this time without knowing about that!

While this accomplishes what I wanted to do, I am still left wondering, is there any way to get Bash to do "dup2()" and "fork()" in a way that allows a child process to read from a file descriptor that is the stdout or stderr from a parent process?

Another possibility is that I am misunderstanding the the whole functionality of the "dup2()" system call. Is it possible at all for a Linux program to duplicate the stderr file descriptor to fd3, and then fork, then have the child close stdin and "dup2()" the inherited fd3 back down to stdin, then read from it as input? (I should write a C program to test that out.) If it is possible, couldn't Bash also do it?

[Kloschüssel]

While this accomplishes what I wanted to do, I am still left wondering, is there any way to get Bash to do "dup2()" and "fork()" in a way that allows a child process to read from a file descriptor that is the stdout or stderr from a parent process?

You always need a buffer that sits in between two processes, which is the fifo. Can't imagine there's a way without it, but let me know if you find it.

Another possibility is that I am misunderstanding the the whole functionality of the "dup2()" system call.

You might do. dup2 is (as far as I know of it) the code behind replicating one stream to multiple other streams. While it might well use a buffer, the streams themself have to buffer the data until the applications consume the data.

Is it possible at all for a Linux program to duplicate the stderr file descriptor to fd3, and then fork, then have the child close stdin and "dup2()" the inherited fd3 back down to stdin, then read from it as input? (I should write a C program to test that out.) If it is possible, couldn't Bash also do it?

Don't know and I can't wrap my head around that idea easily. That would take me some time. Anyway, my feelings tell me that I should doubt it, but if you find out, share your wisdom with the world!

PS: Before starting an implementation that never works, I would draw the flow of data on some paper. That may clarify things from the start.