Performance: Efficient use of EntityFieldQuery (with node_load OR node_load_multiple) to update thousands of nodes with/without conditions
If you can access the data via SQL then I think the Migrate module can do 80%+ of what you want with relative ease. Your hardest part may be either: 1) getting an SQL dump of the data you want from the remote system, or 2) build an SQL representation of the data from the data you can retrieve from the webservice using a Cron type task.

Assuming you can get an SQL version of your (by the way Migrate and what I'm talking about works on XML, JSON, CSV, text files ... and the framework supports "lists" of data from other sources -- but you'd have to develop some of your own Classes for exotic import sources ... I digress) ... migrate supports querying your objects like "I want to move item(s) from point A to B ... and either update or create them." as of Migrate 2.5+. Most people are comfortable with SQL or CSV files so I'll talk about that here. Being Object Oriented you're free to adjust Migrates behavior of course by extending a classes default behavior.

Once you have the SQL data you can use a Migration to import the data. By using Migrate 2.6's (stable release) track_changes option within a Migration class. The Migrate module will take the fields your migrating from point A to point B of your data and create a hash value of all the data. This hash value is used in subsequent calls to Migrate data to deem whether the given item should be created or updated from the data being fed into the Migration.

An example of this can be seen in this great little blog post:
http://timonweb.com/using-hash-value-trackchanges-to-detect-source-data-changes-in-migrate-for-drupal-7

The key excerpt enabling track_changes from the blog post is:

class ExampleNodeMigration extends Migration {
  public function __construct($arguments) {
    parent::__construct($arguments);
   $options = array('track_changes' =&gt; 1);
   $this-&gt;source = new MigrateSourceSQL($this-&gt;query(), $this-&gt;fields(), NULL, $options);
}


I say SQL would be easiest for this approach because you could easily build a query for your migration that is basically SELECT id, doi, publication source id, last-modified-timestamp where source being pubmed, scopus, etc  FROM my_remote_data and  put $options = array('track_changes' =&gt; 1); and Migrate is basically doing all the work for you -- and should be very little PHP code to consume the data.

I've actually done this process most recently with the Migrate modules default support for CSV files with a Migration that results in 6 different Drupal content types built out from the CSV in dependant-order -- and objects are created/updated on-the-fly using track_changes every time the system does an sync from the remote system ... so it's doable via SQL or CSV easiest I think.


  random tidbit: my CSV file is like 40 columns of data and each
  sub-migration only cares about a limited set of columns in the csv
  file. So I tell each migration to ignore unwanted fields for that
  migration by using PHP's unset function in the prepare for the
  Entity being built. By unsetting the values the hash function only
  hashes the relevant data I care about for that single drupal object
  I'm creating with that migration.


Anyways... I hope this rant helps and offers you a path do this. I've used Migrate on and off for single table or larger migrations of data since the Migrate 1.x days ... it can be complicated to learn if you've never used it.