Eventual consistency is a more natural model than strong consistency for a distributed system, since it is closer to the underlying physical reality. Therefore, we propose that it is important to find a programming model that is both congenial to developers and supports eventual consistency. In particular, we consider that a crucial test for such a model is that it should support edge computation in a both natural and secure way. We present a preliminary work report with an initial solution, called Lasp, which resembles a concurrent functional language while naturally supporting an eventually consistent coordination-free distribution model.
@cite_7: We present Resilient Distributed Datasets (RDDs), a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner. RDDs are motivated by two types of applications that current computing frameworks handle inefficiently: iterative algorithms and interactive data mining tools. In both cases, keeping data in memory can improve performance by an order of magnitude. To achieve fault tolerance efficiently, RDDs provide a restricted form of shared memory, based on coarse-grained transformations rather than fine-grained updates to shared state. However, we show that RDDs are expressive enough to capture a wide class of computations, including recent specialized programming models for iterative jobs, such as Pregel, and new applications that these models do not capture. We have implemented RDDs in a system called Spark, which we evaluate through a variety of user applications and benchmarks.
@cite_12: MapReduce is a programming model and an associated implementation for processing and generating large datasets that is amenable to a broad variety of real-world tasks. Users specify the computation in terms of a map and a reduce function, and the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks. Programmers find the system easy to use: more than ten thousand distinct MapReduce programs have been implemented internally at Google over the past four years, and an average of one hundred thousand MapReduce jobs are executed on Google's clusters every day, processing a total of more than twenty petabytes of data per day.
Applications today that perform data processing from devices at the edge use a MapReduce-style programming model over immutable data @cite_12 and subsequent optimizations for efficient, fault-tolerant processing over streams. @cite_7 These solutions are appealing to the application developer: they present systems for performing efficient computation in a now-familiar programming paradigm.