Asked  6 Months ago    Answers:  5   Viewed   28 times

Why is "slurping" a file not a good practice for normal text-file I/O, and when is it useful?

For example, why shouldn't I use these?'/path/to/text.txt').lines.each do |line|
  # do something with a line


File.readlines('/path/to/text.txt').each do |line|
  # do something with a line



Again and again we see questions asking about reading a text file to process it line-by-line, that use variations of read, or readlines, which pull the entire file into memory in one action.

The documentation for read says:

Opens the file, optionally seeks to the given offset, then returns length bytes (defaulting to the rest of the file). [...]

The documentation for readlines says:

Reads the entire file specified by name as individual lines, and returns those lines in an array. [...]

Pulling in a small file is no big deal, but there comes a point where memory has to be shuffled around as the incoming data's buffer grows, and that eats CPU time. In addition, if the data consumes too much space, the OS has to get involved just to keep the script running and starts spooling to disk, which will take a program to its knees. On a HTTPd (web-host) or something needing fast response it'll cripple the entire application.

Slurping is usually based on a misunderstanding of the speed of file I/O or thinking that it's better to read then split the buffer than it is to read it a single line at a time.

Here's some test code to demonstrate the problem caused by "slurping".

Save this as "":

echo Building test files...

yes "abcdefghijklmnopqrstuvwxyz 123456890" | head -c 1000       > kb.txt
yes "abcdefghijklmnopqrstuvwxyz 123456890" | head -c 1000000    > mb.txt
yes "abcdefghijklmnopqrstuvwxyz 123456890" | head -c 1000000000 > gb1.txt
cat gb1.txt gb1.txt > gb2.txt
cat gb1.txt gb2.txt > gb3.txt

echo Testing...

ruby -v

for i in kb.txt mb.txt gb1.txt gb2.txt gb3.txt
  echo "Running: time ruby readlines.rb $i"
  time ruby readlines.rb $i
  echo '---------------------------------------'
  echo "Running: time ruby foreach.rb $i"
  time ruby foreach.rb $i

rm [km]b.txt gb[123].txt 

It creates five files of increasing sizes. 1K files are easily processed, and are very common. It used to be that 1MB files were considered big, but they're common now. 1GB is common in my environment, and files beyond 10GB are encountered periodically, so knowing what happens at 1GB and beyond is very important.

Save this as "readlines.rb". It doesn't do anything but read the entire file line-by-line internally, and append it to an array that is then returned, and seems like it'd be fast since it's all written in C:

lines = File.readlines(ARGV.shift).size
puts "#{ lines } lines read"

Save this as "foreach.rb":

lines = 0
File.foreach(ARGV.shift) { |l| lines += 1 }
puts "#{ lines } lines read"

Running sh ./ on my laptop I get:

Building test files...
ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-darwin13.0]

Reading the 1K file:

Running: time ruby readlines.rb kb.txt
28 lines read

real    0m0.998s
user    0m0.386s
sys 0m0.594s
Running: time ruby foreach.rb kb.txt
28 lines read

real    0m1.019s
user    0m0.395s
sys 0m0.616s

Reading the 1MB file:

Running: time ruby readlines.rb mb.txt
27028 lines read

real    0m1.021s
user    0m0.398s
sys 0m0.611s
Running: time ruby foreach.rb mb.txt
27028 lines read

real    0m0.990s
user    0m0.391s
sys 0m0.591s

Reading the 1GB file:

Running: time ruby readlines.rb gb1.txt
27027028 lines read

real    0m19.407s
user    0m17.134s
sys 0m2.262s
Running: time ruby foreach.rb gb1.txt
27027028 lines read

real    0m10.378s
user    0m9.472s
sys 0m0.898s

Reading the 2GB file:

Running: time ruby readlines.rb gb2.txt
54054055 lines read

real    0m58.904s
user    0m54.718s
sys 0m4.029s
Running: time ruby foreach.rb gb2.txt
54054055 lines read

real    0m19.992s
user    0m18.765s
sys 0m1.194s

Reading the 3GB file:

Running: time ruby readlines.rb gb3.txt
81081082 lines read

real    2m7.260s
user    1m57.410s
sys 0m7.007s
Running: time ruby foreach.rb gb3.txt
81081082 lines read

real    0m33.116s
user    0m30.790s
sys 0m2.134s

Notice how readlines runs twice as slow each time the file size increases, and using foreach slows linearly. At 1MB, we can see there's something affecting the "slurping" I/O that doesn't affect reading line-by-line. And, because 1MB files are very common these days, it's easy to see they'll slow the processing of files over the lifetime of a program if we don't think ahead. A couple seconds here or there aren't much when they happen once, but if they happen multiple times a minute it adds up to a serious performance impact by the end of a year.

I ran into this problem years ago when processing large data files. The Perl code I was using would periodically stop as it reallocated memory while loading the file. Rewriting the code to not slurp the data file, and instead read and process it line-by-line, gave a huge speed improvement from over five minutes to run to less than one and taught me a big lesson.

"slurping" a file is sometimes useful, especially if you have to do something across line boundaries, however, it's worth spending some time thinking about alternate ways of reading a file if you have to do that. For instance, consider maintaining a small buffer built from the last "n" lines and scan it. That will avoid memory management issues caused by trying to read and hold the entire file. This is discussed in a Perl-related blog "Perl Slurp-Eaze" which covers the "whens" and "whys" to justify using full file-reads, and applies well to Ruby.

For other excellent reasons not to "slurp" your files, read "How to search file text for a pattern and replace it with a given value".

Tuesday, June 1, 2021
answered 6 Months ago
  1. To find out where gems are being installed to, run echo $GEM_HOME in a terminal.
  2. When using RVM, gems are installed into your RVM install as it changes $GEM_HOME. Running echo $GEM_HOME now would show a path into your RVM install.
  3. When Bundler is added to the mix, gems will either be installed in $GEM_HOME, or, if you specify a path when running bundle install will be installed to that path. To find out where a gem is through Bundler you can use bundle show gemname to get its full path.
Saturday, July 31, 2021
answered 4 Months ago

Usually, a RuntimeException indicates a programming error (in which case you can't "handle" it, because if you knew to expect it you'd have avoided the error).

Catching any of these general exceptions (including Throwable) is a bad idea because it means you're claiming that you understand every situation which can go wrong, and you can continue on despite that. It's sometimes appropriate to catch Exception (but not usually Throwable) at the top level of the stack, e.g. in a web server - because usually whatever's gone wrong with a single request, you normally want to keep the server up and responding to further requests. I don't normally catch Throwable, as that includes Error subclasses which are normally used to indicate truly catastrophic errors which would usually be best "handled" by terminating the process.

Fundamentally, when there's an error you need to be very cautious about continuing with a particular task - you need to really have a pretty good idea about what the error means, as otherwise you could go ahead with a mistaken assumption about the state of the world, and make things worse. In most cases (not all), simply giving up on a request is better than trying to carry on regardless of a mysterious failure. (It does very much depend on context though - you might not care what went wrong when trying to fetch one piece of secondary information, for example.)

As for catching Exception not catching RuntimeException - that's simply not true. The only odd thing about RuntimeException is that it (and subclasses) are unchecked exceptions, whereas Exception and all other subclasses of Exception are checked.

Monday, August 2, 2021
answered 4 Months ago

I had a similar problem, though I had rvm and ruby installed only for one user. For me, the solution was to check that the application files were owned by the same user for which ruby was installed.

Saturday, August 21, 2021
answered 4 Months ago

If you refer to this is a bug in Ubuntu using O_EXCL to open the file, preventing it from overwriting an existing core.

Sunday, November 14, 2021
answered 2 Weeks ago
Only authorized users can answer the question. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :