Asked  6 Months ago    Answers:  5   Viewed   32 times

I'm using a python script as a driver for a hydrodynamics code. When it comes time to run the simulation, I use subprocess.Popen to run the code, collect the output from stdout and stderr into a subprocess.PIPE --- then I can print (and save to a log-file) the output information, and check for any errors. The problem is, I have no idea how the code is progressing. If I run it directly from the command line, it gives me output about what iteration its at, what time, what the next time-step is, etc.

Is there a way to both store the output (for logging and error checking), and also produce a live-streaming output?

The relevant section of my code:

ret_val = subprocess.Popen( run_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True )
output, errors = ret_val.communicate()
print output
if( ret_val.returncode ):
    print "RUN failednn%snn" % (errors)
    success = False

if( errors ): log_file.write("nn%snn" % errors)

Originally I was piping the run_command through tee so that a copy went directly to the log-file, and the stream still output directly to the terminal -- but that way I can't store any errors (to my knowlege).

My temporary solution so far:

ret_val = subprocess.Popen( run_command, stdout=log_file, stderr=subprocess.PIPE, shell=True )
while not ret_val.poll():

then, in another terminal, run tail -f log.txt (s.t. log_file = 'log.txt').



TLDR for Python 3:

import subprocess
import sys
with open('test.log', 'wb') as f: 
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    for c in iter(lambda:, b''): 

You have two ways of doing this, either by creating an iterator from the read or readline functions and do:

import subprocess
import sys
with open('test.log', 'w') as f:  # replace 'w' with 'wb' for Python 3
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    for c in iter(lambda:, ''):  # replace '' with b'' for Python 3


import subprocess
import sys
with open('test.log', 'w') as f:  # replace 'w' with 'wb' for Python 3
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    for line in iter(process.stdout.readline, ''):  # replace '' with b'' for Python 3

Or you can create a reader and a writer file. Pass the writer to the Popen and read from the reader

import io
import time
import subprocess
import sys

filename = 'test.log'
with, 'wb') as writer,, 'rb', 1) as reader:
    process = subprocess.Popen(command, stdout=writer)
    while process.poll() is None:
    # Read the remaining

This way you will have the data written in the test.log as well as on the standard output.

The only advantage of the file approach is that your code doesn't block. So you can do whatever you want in the meantime and read whenever you want from the reader in a non-blocking way. When you use PIPE, read and readline functions will block until either one character is written to the pipe or a line is written to the pipe respectively.

Tuesday, June 1, 2021
answered 6 Months ago

You could try setting up your own handler for the onerror event and use XMLHttpRequest to tell the server what went wrong, however since it's not part of any specification, support is somewhat flaky.

Here's an example from Using XMLHttpRequest to log JavaScript errors:

window.onerror = function(msg, url, line)
  var req = new XMLHttpRequest();
  var params = "msg=" + encodeURIComponent(msg) + '&url=' + encodeURIComponent(url) + "&line=" + line;"POST", "/scripts/logerror.php");
Wednesday, June 9, 2021
answered 6 Months ago

The exit code of last command is contained in $?.

Use below pseudo code:

if [ $ret -ne 0 ]; then
     #Handle failure
     #exit if required
Thursday, July 29, 2021
answered 4 Months ago

FileNotFoundError: [Errno 2] No such file or directory: 'command'

command is a shell builtin. subprocess.Popen does NOT run the shell by default.

To run the shell, pass shell=True:

>>> import subprocess
>>> subprocess.check_output('command -v python', shell=True)

To find the full path to an executable, you could use shutil.which() instead:

>>> import shutil
>>> shutil.which('python')
Saturday, August 21, 2021
Jeff Swensen
answered 4 Months ago

This snippet has helped me once in a similar situation:

process = subprocess.Popen(cmd, bufsize=1, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(process.stdout.readline, ''):
    print line,
    sys.stdout.flush() # please see comments regarding the necessity of this line 
errcode = process.returncode
Wednesday, November 3, 2021
answered 4 Weeks ago
Only authorized users can answer the question. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :