Asked  7 Months ago    Answers:  5   Viewed   36 times

I'd like to make the output of tail -F or something similar available to me in Python without blocking or locking. I've found some really old code to do that here, but I'm thinking there must be a better way or a library to do the same thing by now. Anyone know of one?

Ideally, I'd have something like tail.getNewData() that I could call every time I wanted more data.

 Answers

78

So, this is coming quite late, but I ran into the same problem again, and there's a much better solution now. Just use pygtail:

Pygtail reads log file lines that have not been read. It will even handle log files that have been rotated. Based on logcheck's logtail2 (http://logcheck.org)

Tuesday, June 1, 2021
 
treeface
answered 7 Months ago
67

A simple tail recursive function:

unsigned int f( unsigned int a ) {
   if ( a == 0 ) {
      return a;
   }
   return f( a - 1 );   // tail recursion
}

Tail recursion is basically when:

  • there is only a single recursive call
  • that call is the last statement in the function

And it's not "better", except in the sense that a good compiler can remove the recursion, transforming it into a loop. This may be faster and will certainly save on stack usage. The GCC compiler can do this optimisation.

Tuesday, June 8, 2021
 
JohnnyW
answered 6 Months ago
83

There is a tempfile module for python, but a simple file creation also does the trick:

new_file = open("path/to/FILE_NAME.ext", "w")

Now you can write to it using the write method:

new_file.write('this is some content')

With the tempfile module this might look like this:

import tempfile

new_file, filename = tempfile.mkstemp()

print(filename)

os.write(new_file, "this is some content")
os.close(new_file)

With mkstemp you are responsible for deleting the file after you are done with it. With other arguments, you can influence the directory and name of the file.


UPDATE

As rightfully pointed out by Emmet Speer, there are security considerations when using mkstemp, as the client code is responsible for closing/cleaning up the created file. A better way to handle it is the following snippet (as taken from the link):

import os
import tempfile

fd, path = tempfile.mkstemp()
try:
    with os.fdopen(fd, 'w') as tmp:
        # do stuff with temp file
        tmp.write('stuff')
finally:
    os.remove(path)

The os.fdopen wraps the file descriptor in a Python file object, that closes automatically when the with exits. The call to os.remove deletes the file when no longer needed.

Thursday, June 17, 2021
 
Sabya
answered 6 Months ago
63

Use RollingFileAppender.

Wednesday, August 4, 2021
 
keisar
answered 4 Months ago
92

Since you mentioned tail-ing, I'm expecting that you are comfortable with working on the terminal with CLI tools.

You can install awslogs locally and use it to tail Cloudwatch.

e.g.

$ awslogs get /aws/lambda/my-api-lambda ALL --watch --profile production

Aside from not needing to refresh anything anymore (that's what tail is for), I also like that you don't have to worry about jumping between different LogGroups (unlike in the CloudWatch console).

Sunday, October 10, 2021
 
Smandoli
answered 2 Months ago
Only authorized users can answer the question. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :  
Share