OOP in python (related to scrapy)
Question: The question is how to share data between objecs in a safe and maintainable
manner.

Example: I've build the scrapy application which spawns numerous spiders.
Although each spider is connected to separate pipeline object, I need to
compare and sort the data between different pipelines (e.g. I need outputs
sorted by different item attributes: prices, date etc.), so I need some shared
data area. The same applies to spiders themselves (e.g. I need to count
maximum total requests). The first implementation used class variables to
shared data between between spiders/pipelines and instance variables for each
object.

    
    
    class MyPipeline(object):
    max_price = 0
    
    def process_item(self, item, spider):
    if item['price'] > max_price : 
     max_price = item['price']
    

(The actual structures are more complex) Then I thought out that having a
bunch of statics is not OOP and the next solution is to have the private class
data for each class and use to store values:

    
    
    class MyPipelineData:
    def __init__(self):
       self.max_price = 0
    
    class SpidersData:
      def __init___(self, total_requests, pipeline_data):
        self.total_requests = total_requests
        self.pipeline_data = pipeline_data #the shared data between pipelines
    
    class MyPipeline(object):
    pipeline_data = None
    
    def process_item(self, item, spider):
      if _data is None:  
         _data = spider.data.pipeline_data  #the shared data between pipelines  
      if item['price'] > _data.max_price : 
       _data.max_price = item['price']
    
     class Spider(scrapy.spider):
     def __init__(self, spider_data):
       self._data = spider_data
      # and the same object of SpiderData is passed to all spiders 
    

Now I have one instance of data shared between all pipeplines (and the same
for spiders). Am I generally correct with this? Should I apply same OOP
approaches in python as in C++ ?


Answer: From what I understand, the approach you are proposing is to keep a reference
from each object to a shared object that captures all of the shared data, and
and I think this is perfectly fine, especially if you name it appropriately so
that its name suggests it's being shared, for readability.

Also, you're hiding the internals of the shared object and encapsulating them
inside methods such as process_item(), which I think is really important for
maintainability (because changes in the internals of the shared object don't
have to affect any other object).

But I'm not sure about the way you are bootstrapping (i.e. initializing) this
shared object. You have these two lines

    
    
    if _data is None:
      _data = ...
    

which are a little surprising. I didn't quite understand what _data is and
where it is defined. Also pipeline_data is assigned to None and never assigned
to anything else, so I'm not sure what you meant there.

If possible, I would prefer to see a function called create_spiders() that
creates the shared object, and then creates the different spiders one by one,
giving them a reference to the shared object. This makes the logic very clear.

* * *

However, in the special case that you want the shared object to be a
singleton, I would consider making it a static object in some module that you
name appropriately, maybe Globals.py. And then inside your Spider code you
would see things like

    
    
    import Globals
    
    class SpiderData:
    
    def update(self):
      self.data.price = 200
      Globals.spiders_data_collector.process(self.data)
    

Inside the module Globals you would initialize the object
spiders_data_collector. I think this requires less code, and this is also
important for maintainability.

