Asked  7 Months ago    Answers:  5   Viewed   26 times

I'm creating a sort of background job queue system with MongoDB as the data store. How can I "listen" for inserts to a MongoDB collection before spawning workers to process the job? Do I need to poll every few seconds to see if there are any changes from last time, or is there a way my script can wait for inserts to occur? This is a PHP project that I am working on, but feel free to answer in Ruby or language agnostic.

 Answers

53

MongoDB has what is called capped collections and tailable cursors that allows MongoDB to push data to the listeners.

A capped collection is essentially a collection that is a fixed size and only allows insertions. Here's what it would look like to create one:

db.createCollection("messages", { capped: true, size: 100000000 })

MongoDB Tailable cursors (original post by Jonathan H. Wage)

Ruby

coll = db.collection('my_collection')
cursor = Mongo::Cursor.new(coll, :tailable => true)
loop do
  if doc = cursor.next_document
    puts doc
  else
    sleep 1
  end
end

PHP

$mongo = new Mongo();
$db = $mongo->selectDB('my_db')
$coll = $db->selectCollection('my_collection');
$cursor = $coll->find()->tailable(true);
while (true) {
    if ($cursor->hasNext()) {
        $doc = $cursor->getNext();
        print_r($doc);
    } else {
        sleep(1);
    }
}

Python (by Robert Stewart)

from pymongo import Connection
import time

db = Connection().my_db
coll = db.my_collection
cursor = coll.find(tailable=True)
while cursor.alive:
    try:
        doc = cursor.next()
        print doc
    except StopIteration:
        time.sleep(1)

Perl (by Max)

use 5.010;

use strict;
use warnings;
use MongoDB;

my $db = MongoDB::Connection->new;
my $coll = $db->my_db->my_collection;
my $cursor = $coll->find->tailable(1);
for (;;)
{
    if (defined(my $doc = $cursor->next))
    {
        say $doc;
    }
    else
    {
        sleep 1;
    }
}

Additional Resources:

Ruby/Node.js Tutorial which walks you through creating an application that listens to inserts in a MongoDB capped collection.

An article talking about tailable cursors in more detail.

PHP, Ruby, Python, and Perl examples of using tailable cursors.

Tuesday, June 1, 2021
 
Silfverstrom
answered 7 Months ago
90

5 years later we finally have a better solution. Use MutationObserver!

In short:

new MutationObserver(function(mutations) {
    console.log(mutations[0].target.nodeValue);
}).observe(
    document.querySelector('title'),
    { subtree: true, characterData: true, childList: true }
);

With comments:

// select the target node
var target = document.querySelector('title');

// create an observer instance
var observer = new MutationObserver(function(mutations) {
    // We need only first event and only new value of the title
    console.log(mutations[0].target.nodeValue);
});

// configuration of the observer:
var config = { subtree: true, characterData: true, childList: true };

// pass in the target node, as well as the observer options
observer.observe(target, config);

Also Mutation Observer has awesome browser support:

Wednesday, June 23, 2021
 
PHLAK
answered 6 Months ago
59

pyinotify is IMHO the only way to get system changes without scanning the directory.

Wednesday, August 4, 2021
 
Trav L
answered 4 Months ago
76

You don't need the full notation as the placeholder has already moved to that position in the array.

db.junk.update(
    { "commandes.voyagesSouscrits.idVoyage": "123" },
    {$pull: { "commandes.$.voyagesSouscrits": { idVoyage: "123" } }}
)

This part:

idVoyage: { <query> }

is only needed because the positional operator in "commandes.$.voyagesSouscrits" can only match the first array position found in the query.

http://docs.mongodb.org/manual/reference/operator/projection/positional/

Hope that clears it up.

Friday, August 6, 2021
 
dmp
answered 4 Months ago
dmp
83

TL;DR:

Use the async driver if the operations are slow, or use the regular driver in most cases. You shouldn't use the core driver.

MongoDB Regular Driver:

General driver that you can use to search, create, read, update and delete documents. The find(...), updateMany(...), deleteMany(...) and similar methods will hang for as long as the result is not returned or the operation not done (synchronous behavior). This is the driver that most program uses and is good in most cases.

Here is an example for inserting a single Document:

collection.insertOne(doc);
//Do something here.
System.out.println("Inserted!")

MongoDB Async Driver:

Another type of driver that you can use to search, create, read, update and delete documents. This driver offers similar methods than the regular driver (find(...), updateMany(...), deleteMany(...), etc.).

The difference with the regular driver is that the main thread will not hang because the async driver sends the result in a callback (asynchronous behavior). This driver is used when the operations can take a long time (a lot of data to go through, high latency, query on unindexed fields, etc.) and you do not want to manage multiple threads.

Here is an example of the callback when inserting a single Document:

collection.insertOne(doc, new SingleResultCallback<Void>() {
    @Override
    public void onResult(final Void result, final Throwable t) {
        //Do something here.
        System.out.println("Inserted!");
    }
});
// Do something to show that the Document was not inserted yet.
System.out.println("Inserting...")

For more informations, read this.

MongoDB Core Driver

Base layer of the regular and async drivers. It contains low-level methods to do all the operations common to the regular and async drivers. Unless you are making a new API / Driver for MongoDB, you shouldn't use the core driver.

Tuesday, November 2, 2021
 
Industrial
answered 1 Month ago
Only authorized users can answer the question. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :  
Share