Asked  7 Months ago    Answers:  5   Viewed   58 times

Edit: Since it appears that there's either no solution, or I'm doing something so non-standard that nobody knows - I'll revise my question to also ask: What is the best way to accomplish logging when a python app is making a lot of system calls?

My app has two modes. In interactive mode, I want all output to go to the screen as well as to a log file, including output from any system calls. In daemon mode, all output goes to the log. Daemon mode works great using os.dup2(). I can't find a way to "tee" all output to a log in interactive mode, without modifying each and every system call.


In other words, I want the functionality of the command line 'tee' for any output generated by a python app, including system call output.

To clarify:

To redirect all output I do something like this, and it works great:

# open our log file
so = se = open("%s.log" % self.name, 'w', 0)

# re-open stdout without buffering
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

# redirect stdout and stderr to the log file opened above
os.dup2(so.fileno(), sys.stdout.fileno())
os.dup2(se.fileno(), sys.stderr.fileno())

The nice thing about this is that it requires no special print calls from the rest of the code. The code also runs some shell commands, so it's nice not having to deal with each of their output individually as well.

Simply, I want to do the same, except duplicating instead of redirecting.

At first thought, I thought that simply reversing the dup2's should work. Why doesn't it? Here's my test:

import os, sys

### my broken solution:
so = se = open("a.log", 'w', 0)
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

os.dup2(sys.stdout.fileno(), so.fileno())
os.dup2(sys.stderr.fileno(), se.fileno())
###

print("foo bar")

os.spawnve("P_WAIT", "/bin/ls", ["/bin/ls"], {})
os.execve("/bin/ls", ["/bin/ls"], os.environ)

The file "a.log" should be identical to what was displayed on the screen.

 Answers

81

Since you're comfortable spawning external processes from your code, you could use tee itself. I don't know of any Unix system calls that do exactly what tee does.

# Note this version was written circa Python 2.6, see below for
# an updated 3.3+-compatible version.
import subprocess, os, sys

# Unbuffer output (this ensures the output is in the correct order)
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

tee = subprocess.Popen(["tee", "log.txt"], stdin=subprocess.PIPE)
os.dup2(tee.stdin.fileno(), sys.stdout.fileno())
os.dup2(tee.stdin.fileno(), sys.stderr.fileno())

print "nstdout"
print >>sys.stderr, "stderr"
os.spawnve("P_WAIT", "/bin/ls", ["/bin/ls"], {})
os.execve("/bin/ls", ["/bin/ls"], os.environ)

You could also emulate tee using the multiprocessing package (or use processing if you're using Python 2.5 or earlier).

Update

Here is a Python 3.3+-compatible version:

import subprocess, os, sys

tee = subprocess.Popen(["tee", "log.txt"], stdin=subprocess.PIPE)
# Cause tee's stdin to get a copy of our stdin/stdout (as well as that
# of any child processes we spawn)
os.dup2(tee.stdin.fileno(), sys.stdout.fileno())
os.dup2(tee.stdin.fileno(), sys.stderr.fileno())

# The flush flag is needed to guarantee these lines are written before
# the two spawned /bin/ls processes emit any output
print("nstdout", flush=True)
print("stderr", file=sys.stderr, flush=True)

# These child processes' stdin/stdout are 
os.spawnve("P_WAIT", "/bin/ls", ["/bin/ls"], {})
os.execve("/bin/ls", ["/bin/ls"], os.environ)
Tuesday, June 1, 2021
 
IcedAnt
answered 7 Months ago
70

Here's a simpler way of reproducing your issue:

$ cat foo.py
from time import sleep
while True: 
  sleep(2)
  print "hello"

$ python foo.py
hello
hello    
(...)

$ python foo.py | tee log
(no output)

This happens because python buffers stdout when it's not a terminal. The easiest way to unbuffer it is to use python -u:

$ python -u foo.py | tee log
hello
hello
(...)

You can also set the shebang to #!/usr/bin/python -u (this does not work with env).

Monday, August 2, 2021
 
BartmanEH
answered 5 Months ago
28

Just use the date output as a pattern in grep:

$ grep "$(date +"%Y-%m-%d")" file
2013-06-21 00:01:24,915 - INFO   
2013-06-21 00:01:24,915 - INFO

That is, you need to enclose the date sentence to make it be processed. Also, note I used Y instead of your 20%y.


I am looking for a sepcific EmpID in the logs with current date.

Then pipe to another grep:

$ grep $(date +"%Y-%m-%d") file | grep "EmpID#106496"
2013-06-21 00:01:24,915 - INFO  EmpID#106496 
2013-06-21 00:01:24,915 - INFO EmpID#106496 
Wednesday, August 18, 2021
 
Rob W
answered 4 Months ago
67

One way:

findstr WARN log.txt

More complex:

for /f "tokens=1,2,3,4* delims=, " %i in (log.txt) do @if "%l"=="WARN" echo %i %j %m

OUTPUT:
2009-12-07 14:32:52 Sample log
Sunday, August 29, 2021
 
Joe
answered 4 Months ago
Joe
37

A process-substitution-based solution is simple, although not as simple as you might think. My first attempt seemed like it should work

{ echo stdout; echo stderr >&2; } > >( tee ~/stdout.txt ) 
                                 2> >( tee ~/stderr.txt )

However, it doesn't quite work as intended in bash because the second tee inherits its standard output from the original command (and hence it goes to the first tee) rather than from the calling shell. It's not clear if this should be considered a bug in bash.

It can be fixed by separating the output redirections into two separate commands:

{ { echo stdout; echo stderr >&2; } > >(tee stdout.txt ); } 
                                   2> >(tee stderr.txt )

Update: the second tee should actually be tee stderr.txt >&2 so that what was read from standard error is printed back onto standard error.

Now, the redirection of standard error occurs in a command which does not have its standard output redirected, so it works in the intended fashion. The outer compound command has its standard error redirected to the outer tee, with its standard output left on the terminal. The inner compound command inherits its standard error from the outer (and so it also goes to the outer tee, while its standard output is redirected to the inner tee.

Thursday, September 9, 2021
 
RustyFluff
answered 3 Months ago
Only authorized users can answer the question. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :
 
Share