---
layout: post
title:  "Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks"
date:   2015-8-28 20:00
permalink: dryad
---

The basic idea is fairly straight forward.  Computations on large data sets 
can be expressed as a DAG where vertices are computations, and edges are flows
of informations between them..  Given a DAG, a scheduler can automatically run 
computation in parallel on a cluster of systems.

I can't say I cared for this paper.  I think the contributions are somewhat
thin and uninteresting.  Basically, they built a system whose graph based API
is dramatically more complicated than MapReduce, without sufficiently arguing
that this complexity is really necessary to express interesting computations.
I'd rather deploy a system half as fast[^It's not clear to me if their system
is, in fact, faster that MapReduce as they seem to imply.  And if it is faster,
whether this is because of their graph based programming model (unlikely), or
simply due to more aggressive micro optimizations.] which is "Obviously Correct"
and allows for rapid development by a wide swath of the developer pool -- what
they have here is complex for complexity sake.

Luckily, it seems this idea hasn't taken off widely outside of Microsoft.
Academically cute ideas are typically beaten by less "interesting" ideas that
are pragmatic and accessible.  Seems to be for the best.

---
layout: post
title:  "Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks"
date:   2015-9-1 20:00
permalink: dryad
---

The basic idea is fairly straight forward.  Computations on large data sets 
can be expressed as a DAG where vertices are computations, and edges are flows
of informations between them..  Given a DAG, a scheduler can automatically run 
computation in parallel on a cluster of systems.

I can't say I cared for this paper.  I think the contributions are somewhat
thin and uninteresting.  Basically, they built a system whose graph based API
is dramatically more complicated than MapReduce, without sufficiently arguing
that this complexity is really necessary to express interesting computations.
I'd rather deploy a system half as fast[^1 It's not clear to me if their system
is, in fact, faster that MapReduce as they seem to imply.  And if it is faster,
whether this is because of their graph based programming model (unlikely), or
simply due to more aggressive micro optimizations.] which is "Obviously Correct"
and allows for rapid development by a wide swath of the developer pool -- what
they have here is complex for complexity sake.

Luckily, it seems this idea hasn't taken off widely outside of Microsoft.
Academically cute ideas are typically beaten by less "interesting" ideas that
are pragmatic and accessible.  Seems to be for the best.

Fix the date and the footnote
Fix the date and the footnote
