What is the fastest cache for small and rarely changing data to use with Django?
Question: I am in a process of switching from PHP to Python + Django and looking for
equivalent of PHP's "Array cache".

For small data sets from DB like "categories" that was changing very rarely
but accessed very often i was using array cache.

<http://www.mysqlperformanceblog.com/2006/08/09/cache-performance-comparison/>

Concept of it was to generate PHP source with the tree of categories and when
the opcode was turned on it was working like embedding data into application
sources. It was the fastest imaginable cache, very helpful for large load.

Django manual(<https://docs.djangoproject.com/en/1.4/topics/cache/>) states:

> By far the fastest, most efficient type of cache available to Django,
> Memcached..

So the questions are:

  * Would generating a .py file with python dictionaries/lists made any sense?
  * Will this be faster than Memcached? If not why?
  * Are there any known implementations of this?
  * Does Python have anything like var_export() function from PHP?

EDIT:

As pointed in an answer i can use repr() and this can be benchmarked easily so
i have created a simple benchmark:

<https://github.com/fsw/pythonCachesBenchmark>

output of this on my local machine was:

    
    
    FIRST RUN
    get_categories_from_db
    6.57282209396
    get_categories_from_memcached
    (SET CACHE IN 0.000940)
    4.88948512077
    get_categories_from_pickledfile
    (SET CACHE IN 0.000917)
    2.87856888771
    get_categories_from_pythonsrc
    (SET CACHE IN 0.000489)
    0.0930788516998
    SECOND RUN
    get_categories_from_db
    6.63035202026
    get_categories_from_memcached
    4.60877108574
    get_categories_from_pickledfile
    2.87137699127
    get_categories_from_pythonsrc
    0.0903170108795
    

get_categories_from_pythonsrc is simple implementation of PHP's arraycache i
was talking about:

    
    
    def get_categories_from_pythonsrc():
        if not os.path.exists('catcache.py'):
            start = time.time()
            f = open( 'catcache.py', 'wb' )
            categories = get_categories_from_db()
            f.write('x = ' + repr(categories))
            f.close()
            print '(SET CACHE IN %f)' % (time.time() - start)
        import catcache
        return catcache.x
    

this is my simple pickledfile cache implementation:

    
    
    def get_categories_from_pickledfile():
        path = 'catcache.p'
        if not os.path.exists(path):
            start = time.time()
            pickle.dump( get_categories_from_db(), open( path, 'wb' ) )
            print '(SET CACHE IN %f)' % (time.time() - start)
        return pickle.load(open( path, 'rb' ));
    

complete source:

<https://github.com/fsw/pythonCachesBenchmark/blob/master/test.py>

I will later add "Django's low-level cache APIs" to this benchmark to see what
they are about.

So as my intuition suggested caching dictionary in a python .py file is the
fastest way i could get (over 30 times faster than cPickle + file)

As said i am new to Python so probably i am missing something here?

If not: why isn't this solution widely used?


Answer: Python has several solutions that may work here:

  * Memcached (as you already know),
  * pickle (as Blender mentioned) - which of course can be used with eg. Memcached,
  * several other caching (eg. for local memory) & serialization (eg. `simplejson`) solutions,

In general `pickle` is very fast (use `cPickle` if you need more speed) and in
Python you do not need anything like `var_export()` (although you can use
`repr()` on variables to have their valid literal, if they are of one of
primitive types). `pickle` in Python is more similar to `serialize` in PHP.

Your question is not very specific, but the above should give you some
insight. Also you need to take into account that PHP and Python have different
philosophies, so solutions to the same problems may look differently. In this
specific case `pickle` module should solve your issues.

