Asked  7 Months ago    Answers:  5   Viewed   46 times

In my PHP application I need to read multiple lines starting from the end of many files (mostly logs). Sometimes I need only the last one, sometimes I need tens or hundreds. Basically, I want something as flexible as the Unix tail command.

There are questions here about how to get the single last line from a file (but I need N lines), and different solutions were given. I'm not sure about which one is the best and which performs better.

 Answers

73

Methods overview

Searching on the internet, I came across different solutions. I can group them in three approaches:

  • naive ones that use file() PHP function;
  • cheating ones that runs tail command on the system;
  • mighty ones that happily jump around an opened file using fseek().

I ended up choosing (or writing) five solutions, a naive one, a cheating one and three mighty ones.

  1. The most concise naive solution, using built-in array functions.
  2. The only possible solution based on tail command, which has a little big problem: it does not run if tail is not available, i.e. on non-Unix (Windows) or on restricted environments that don't allow system functions.
  3. The solution in which single bytes are read from the end of file searching for (and counting) new-line characters, found here.
  4. The multi-byte buffered solution optimized for large files, found here.
  5. A slightly modified version of solution #4 in which buffer length is dynamic, decided according to the number of lines to retrieve.

All solutions work. In the sense that they return the expected result from any file and for any number of lines we ask for (except for solution #1, that can break PHP memory limits in case of large files, returning nothing). But which one is better?

Performance tests

To answer the question I run tests. That's how these thing are done, isn't it?

I prepared a sample 100 KB file joining together different files found in my /var/log directory. Then I wrote a PHP script that uses each one of the five solutions to retrieve 1, 2, .., 10, 20, ... 100, 200, ..., 1000 lines from the end of the file. Each single test is repeated ten times (that's something like 5 × 28 × 10 = 1400 tests), measuring average elapsed time in microseconds.

I run the script on my local development machine (Xubuntu 12.04, PHP 5.3.10, 2.70 GHz dual core CPU, 2 GB RAM) using the PHP command line interpreter. Here are the results:

Execution time on sample 100 KB log file

Solution #1 and #2 seem to be the worse ones. Solution #3 is good only when we need to read a few lines. Solutions #4 and #5 seem to be the best ones. Note how dynamic buffer size can optimize the algorithm: execution time is a little smaller for few lines, because of the reduced buffer.

Let's try with a bigger file. What if we have to read a 10 MB log file?

Execution time on sample 10 MB log file

Now solution #1 is by far the worse one: in fact, loading the whole 10 MB file into memory is not a great idea. I run the tests also on 1MB and 100MB file, and it's practically the same situation.

And for tiny log files? That's the graph for a 10 KB file:

Execution time on sample 10 KB log file

Solution #1 is the best one now! Loading a 10 KB into memory isn't a big deal for PHP. Also #4 and #5 performs good. However this is an edge case: a 10 KB log means something like 150/200 lines...

You can download all my test files, sources and results here.

Final thoughts

Solution #5 is heavily recommended for the general use case: works great with every file size and performs particularly good when reading a few lines.

Avoid solution #1 if you should read files bigger than 10 KB.

Solution #2 and #3 aren't the best ones for each test I run: #2 never runs in less than 2ms, and #3 is heavily influenced by the number of lines you ask (works quite good only with 1 or 2 lines).

Wednesday, March 31, 2021
 
Maury
answered 7 Months ago
83

I did it this way:

1. My service configuration

# /src/Tiriana/MyBundle/Resources/config/services.yml
parameters:
     swiftmailer.class:  TirianaMyBundleUtilMailerWrapper

2. Mailer wraper service

It extends Swift_Mailer, because it is passed to different classes expecting mailer to be instance of Swift_Mailer. And it creates Swift_Mailer instance as a field, because... $transport is private in Swith_Mailer (link). Code would be so much better if $transport was protected...

// /src/Tiriana/MyBundle/Util/MailerWrapper.php
namespace TirianaMyBundleUtil;

use MonologLogger;
use MonologHandlerStreamHandler;

class MailerWrapper extends Swift_Mailer
{
    private $_logger;

    /** @var Swift_Mailer */
    private $_mailer;

    public function send(Swift_Mime_Message $message, &$failedRecipients = null)
    {
        $this->_log('BEFORE SEND');  // <-- add your logic here
        $ret = $this->_mailer->send($message, $failedRecipients);
        $this->_log('AFTER SEND');  // <-- add your logic here

        return $ret;
    }

    /** @return Logger */
    public function getLogger()
    {
        return $this->_logger;
    }

    protected function _log($msg)
    {
        $this->getLogger()->debug(__CLASS__ . ": " . $msg);
    }

    public function __construct(Swift_Transport $transport, Logger $logger)
    {
        /* we need _mailer because _transport is private
           (not protected) in Swift_Mailer, unfortunately... */
        $this->_mailer = parent::newInstance($transport);
        $this->_logger = $logger;
    }

    public static function newInstance(Swift_Transport $transport)
    {
        return new self($transport);
    }

    public function getTransport()
    {
        return $this->_mailer->getTransport();
    }

    public function registerPlugin(Swift_Events_EventListener $plugin)
    {
        $this->getTransport()->registerPlugin($plugin);
    }
}

3. Bundle builder

// /src/Tiriana/MyBundle/TirianaMyBundle.php
namespace TirianaMyBundle;

use SymfonyComponentHttpKernelBundleBundle;
use SymfonyComponentDependencyInjectionContainerBuilder;

use TirianaMyBundleDependencyInjectionCompilerOverrideServiceSwiftMailer;

class TirianaMyBundle extends Bundle
{
    public function build(ContainerBuilder $container)
    {
        parent::build($container);
        $container->addCompilerPass(new OverrideServiceSwiftMailer());  // <-- ADD THIS LINE
    }
}

4. And OverrideServiceSwiftMailer class

// /src/Tiriana/MyBundle/DependencyInjection/Compiler/OverrideServiceSwiftMailer.php
namespace TirianaMyBundleDependencyInjectionCompiler;
use SymfonyComponentDependencyInjectionCompilerCompilerPassInterface;
use SymfonyComponentDependencyInjectionContainerBuilder;
use SymfonyComponentDependencyInjectionReference;

class OverrideServiceSwiftMailer implements CompilerPassInterface
{
    public function process(ContainerBuilder $container)
    {
        /* @var $definition SymfonyComponentDependencyInjectionDefinitionDecorator */
        $definition = $container->findDefinition('mailer');
        $definition->addArgument(new Reference('logger'));
        /* add more dependencies if you need - i.e. event_dispatcher */
    }
}
Wednesday, March 31, 2021
 
Palladium
answered 7 Months ago
81

PHP has a built-in function that does exactly what you want: strip_tags

$text = '<b>Hello</b> World';
print strip_tags($text); // outputs Hello World

If you expect broken HTML, you are going to need to load it into a DOM parser and then extract the text.

Wednesday, March 31, 2021
 
ericstumper
answered 7 Months ago
13

If your columns are already defined, what about remove the column iterator?

Try something like this:

foreach ($activeSheet->getRowIterator() as $rkey => $row) {
    $rowIndex = $row->getRowIndex ();
    $parsedData[$rowIndex]['order'] = $activeSheet->getCell('A' . $rowIndex);
    $parsedData[$rowIndex]['ts']    = $activeSheet->getCell('B' . $rowIndex);
    $parsedData[$rowIndex]['summ']  = $activeSheet->getCell('C' . $rowIndex);
    .
    .
    .
}
Saturday, May 29, 2021
 
Raef
answered 5 Months ago
83

If the file should not be directly available via URL, you should put it in App_Data.

For reading it, just use:

var fileContents = System.IO.File.ReadAllText(Server.MapPath(@"~/App_Data/file.txt"));
Tuesday, August 10, 2021
 
keyBeatz
answered 2 Months ago
Only authorized users can answer the question. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :