Asked  6 Months ago    Answers:  5   Viewed   58 times

I need to read a large file, line by line. Lets say that file has more than 5GB and I need to read each line, but obviously I do not want to use readlines() because it will create a very large list in the memory.

How will the code below work for this case? Is xreadlines itself reading one by one into memory? Is the generator expression needed?

f = (line for line in open("log.txt").xreadlines())  # how much is loaded in memory?

f.next()  

Plus, what can I do to read this in reverse order, just as the Linux tail command?

I found:

http://code.google.com/p/pytailer/

and

"python head, tail and backward read by lines of a text file"

Both worked very well!

 Answers

60

I provided this answer because Keith's, while succinct, doesn't close the file explicitly

with open("log.txt") as infile:
    for line in infile:
        do_something_with(line)
Tuesday, June 1, 2021
 
Freddie
answered 6 Months ago
27

python3 is not Python syntax, it is the Python binary itself, the thing you run to get to the interactive interpreter.

You are confusing the command line with the Python prompt. Open a console (Windows) or terminal (Linux, Mac), the same place you'd use dir or ls to explore your filesystem from the command line.

If you are typing at a >>> or In [number]: prompt you are in the wrong place, that's the Python interpreter itself and it only takes Python syntax. If you started the Python prompt from a command line, exit at this point and go back to the command line. If you started the interpreter from IDLE or in an IDE, then you need to open a terminal or console as a separate program.

Other programs that people often confuse for Python syntax; each of these is actually a program to run in your command prompt:

  • python, python2.7, python3.5, etc.
  • pip or pip3
  • virtualenv
  • ipython
  • easy_install
  • django-admin
  • conda
  • flask
  • scrapy
  • setup.py -- this is a script you need to run with python setup.py [...].
  • Any of the above together with sudo.

with many more variations possible depending on what tools and libraries you have installed and what you are trying to do.

If given arguments, you'll get a SyntaxError exception instead, but the underlying cause is the same:

>>> pip install foobar
  File "<stdin>", line 1
    pip install foobar
              ^
SyntaxError: invalid syntax
Tuesday, June 1, 2021
 
QuantumMechanic
answered 6 Months ago
58

use the following to convert to a timestamp in python 2

int((mod_time.mktime(first_run.timetuple())+first_run.microsecond/1000000.0))

Sunday, August 22, 2021
 
waylaidwanderer
answered 4 Months ago
86

Points :

  • Open file using with. No need to close file.
  • Use zip function to combine two list.

Code without zip with comments inline:

combine =[]

with open("x.txt") as xh:
  with open('y.txt') as yh:
    with open("z.txt","w") as zh:
      #Read first file
      xlines = xh.readlines()
      #Read second file
      ylines = yh.readlines()
      #Combine content of both lists
      #combine = list(zip(ylines,xlines))
      #Write to third file
      for i in range(len(xlines)):
        line = ylines[i].strip() + ' ' + xlines[i]
        zh.write(line)

Content of x.txt:

1
2
3

Content of y.txt:

a
b
c

Content of z.txt:

a 1
b 2
c 3

Code with zip function:

with open("x.txt") as xh:
  with open('y.txt') as yh:
    with open("z.txt","w") as zh:
      #Read first file
      xlines = xh.readlines()
      #Read second file
      ylines = yh.readlines()
      #Combine content of both lists  and Write to third file
      for line1, line2 in zip(ylines, xlines):
        zh.write("{} {}n".format(line1.rstrip(), line2.rstrip()))
Friday, September 24, 2021
 
Dale Zak
answered 2 Months ago
46

You can use get_blob_to_text method.

block_blob_service = BlockBlobService(account_name='myaccount', account_key='mykey')

blob = block_blob_service.get_blob_to_text('mycontainer', 'myblockblob')
print(blob.content)
Sunday, October 24, 2021
 
Manish Das
answered 1 Month ago
Only authorized users can answer the question. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :  
Share