From Imperative to Pure-Functional and Back: Monads vs. Scoped Continuations - <AUTHOR> http://blog.paralleluniverse.co/2015/08/07/scoped-continuations/ <comment><USER2> The interesting part of this whole thing to me is that you can get composition between your continuations without using something like Monad Transformers[1].

However, the part that I haven't been able to get past yet is that monads are not closed over composition (so there exist some monads that are no longer monads when you compose them). If it turns out that monads and delimited continuations are duals then it seems that there ought to be some continuations that compose only to yield invalid results. (With the alternative being that scoped continuations always compose which suggests something odd is going on.)

I haven't worked through the math yet because it looks suspiciously dreadful, but if anyone knows why scoped continuation composition _is_ closed I would appreciate your input.

[1] - the most disappointing thing about monads to me.

<comment><USER3> Interesting, and disappointing in a way. I assumed that all Monadic compositions would be closed analogous to linear algebra operations and vector spaces. Out of curiosity, do you have a reference to read up on Monadic Composition not being closed? Particularly any lighter references (it's a Saturday after all).

<comment><USER4> Here's a simple counter-example.

The state monad is the following functor

        St s a = s -> (a, s)

if we compose it with itself (using `d` as the second state parameter)

        St s (St d a) = s -> (d -> (a, d), s)

is `F a = St s (St d a)` a monad? The return operation is clear enough

        return a = \s -> (\d -> (a, d), s)

but the join operation fails

        join :: F (F a) -> F a
             :: St s (St d (St s (St d a))) -> St s (St d a)
             :: s -> (d -> (s -> (d -> (a, d), s), d), s)
             -> s -> (d -> (a, d), s)

        join m0 =
          \s0 -> let (m1 :: St d (F a), s1) = m0 s0
                 in (\d0 -> let (m2 :: F a, d1) = m1 d0
                                (m3 :: St d a, s2) = m2 s1
                                (a, d2) = m3 d1
                            in (a, d2)
                    , ??? )

The trouble is the ??? bit where we'd like to place the furthest advanced `s` state named `s2`, but it's not in scope to return at the top since we have to _apply the d state_ to reach it.

This however only means that St does not compose with itself as a functor. It composes just fine in another method. If we install the inner monad "wrapped around" the tuple

        StT s m a = s -> m (a, s)      -- T for "transformer"

then the composition of states looks like

        StT s (St d) a = s -> St d (a, s)
                       = s -> d -> ((a, s), d)
                       = (s, d) -> (a, (s, d))
                       = St (s, d) a

which is obviously a monad.

</comment></comment></comment><comment><USER5> It's an interesting article and something I would like to think deeply about, but I hope the author will forgive me if it isn't immediately obvious to me. Especially the statement that "monads have no place in imperative languages".

For example, I do a lot of Java/Coffee script lately. I don't have access to first class continuations. Don't get me wrong; I'd love to have them. I'm sick as the next person of blowing up in a callback and having my stack completely hosed.

But in Coffee script, monads are nearly trivial to implement. The use of the Maybe monad makes doing repeated transformations and filtering incredibly nicer. It is not obvious to me that a continuation based implementation would be clearer at all.

Perhaps the author is trying to make the case that first class continuations are more important than first class monads for imperative languages. If that's the case, then I tend to agree because, like I said, monads are not difficult to implement even if you don't have explicit support for them. Stack preserving continuations on the other hand... :-P

I think the idea of using composable continuations is a very interesting one and I'd like investigate it further, but unless I'm being very dim-witted (a definite possibility) I don't think the matter is as clear as the author hopes.

<comment><AUTHOR> Author here. I'm not trying to claim that nested scoped continuations (what the literature calls "delimited continuations with multiple prompts") are objectively easy, although I do think they're a lot easier than monad transformers, especially for imperative programmers. In fact, in the post I say that I'm not sure languages should expose this abstraction directly.

It is merely discussed here to show that the theory behind the imperative style is not different from the one behind the PFP style, aimed for those who believe "PFP is just math, while imperative is 'not math'", or to those who are curious to see (as I was) if the imperative style has something like PFP's monad, namely an elegant abstraction that's at the heart of everything.

In practice, however, most continuations should be used sparingly if at all, except one -- the thread (in particular, the lightweight thread). Lightweight threads make the most annoying problems of monads in an imperative language go away, and are easy for imperative programmers to use.

<comment><USER5> Thanks for the clarification. That makes some sense to me.

Also, thank you for the article. It has made me think quite a lot. It's one of those times where my subconscious is saying, "You should pay attention to this", but I'm not quite smart enough to understand what I want to do with it. It will be fun to puzzle it out for a while ;-)

<comment><AUTHOR> You don't need to pay attention to scoped continuations. They're a powerful "general" abstraction (like monads), but in practice all you need to know is just the most common kind of continuations -- the (blocking) thread. You'll rarely need to use other instances of continuations, as imperative languages already has their function built into the language (exceptions and even mutable variables).

</comment></comment></comment></comment><comment><USER6> I like this because it addresses part of the problem of why I've stopped paying any attention to PLT related topics like denotational semantics, monads, category theory, and higher-order/dependent static type systems in general.

In my day to day work the programming language is never the problem because hacking around the limitations of the language to bridge the gap between the problem domain and the computational domain is never the bottleneck. The real problem is when these systems go into production and all static guarantees and beautiful abstractions become basically useless. What I want is something to help me reason about dynamic and evolving large scale software systems. What I get is PLT researchers and library authors bickering over dependent types, monads, and continuations.

Here's a concrete and unsolved problem. How do you specify some kind of topology for your production environment and then have whatever system understands this specification reason about performance, error recovery, failover, and scaling issues. In fact the lower level components don't even matter because you could write it in Haskell or Python or Java or whatever is your favorite language. Right now there is nothing because everyone is arguing about continuations and monads. Basically everyone involved is thinking too small. The one person that I think comes close to approaching the problem from the right angle is [http://www.cl.cam.ac.uk/~srk31/](http://www.cl.cam.ac.uk/~srk31/).

<comment><USER7> Doesn't an expressive and strong type system, as an example, help you reason about software systems?

<comment><USER6> Nope, my problem isn't reasoning about the lower level components. In fact you can assume I have correctness proofs for every piece of the infrastructure. What I don't have is the higher-level piece that can reason about the lower level components and do so when new pieces are added, old pieces removed, new guarantees enforced, old ones relaxed, etc. A static type system no matter how advanced doesn't help with any of it.

<comment><AUTHOR> Yep, that's called the "halting problem", and I hear it's very hard to solve :)

Here's how I generally see it. Writing large software is hard (and a software composed of multiple components is large software, even if you decide using the network stack when calling some functions is a good idea ;)). "Type" people (and I'm certainly for some kinds of static types, I just don't think that they're the solution to everything) would tell you that types help restrict your computation in such a way that reasoning about it is easier. To that I say, but I want to do something specific which it's pretty annoying in PFP, and they answer, just do it a different way, to which I say, if I can do it differently, so can the compiler on my behalf. In short, I don't think that explicit types (i.e. those that can be defined in the language you use) are stronger than implicit types (i.e. anything that a verifier can prove about your program), and at least in theory, they are both "the same" (though, see another comment here on what I think about two things being the same in a mathematical sense). We should use whatever is easier for our minds to grasp.

The more essential problem -- although I don't know if it's more interesting or less interesting -- is what exactly do we want to do in our programs? If we require a Turing complete model, then no matter how many types -- explicit or implicit -- we throw at the problem, we just won't be able to formally prove properties we might care about. Some people, however, claim that we don't really need Turing completeness, and if we don't, that might open a whole new world of possibilities. But once we start restricting the computational model, the representation we use -- say, PFP or imperative -- might become much more important, each being able to compute things that the other cannot (we know that lambda calculus and Turing machines both compute the same functions, but once we restrict each in some arbitrary ways, it gets harder to find equivalences between restrictions in the PFP model and those in the imperative model).

Finally, I think that the notions of what it means "to reason about" or even what it means to be correct (and certainly how important it is to be correct) are themselves vague and require answers from empirical psychological/sociological/economic research than from math.

<comment><USER6> I think the people that say we don't need Turing completeness say that because their models can't really accommodate proper reasoning about exactly the kinds of systems I want to reason about. Proving theorems is nice and all but at the end of the day the temporal and dynamic aspects of the system I'm building are what I really care about. Not some static snapshot of it but the actual evolving gestalt.

<comment><AUTHOR> Oh, absolutely, but there are formal methods for working with that, too -- well, maybe just the "evolving" part if not the "gestalt" \-- mostly based on temporal logic (I can't believe I'm arguing on the side of formal methods now...) See, e.g. the Esterel language[1] used in the industry (directly or through its descendent languages) to write full, safety critical real-time systems. It is an imperative, non-Turing complete language, that is fully verifiable, and actually able to track an evolving gestalt for some restricted definition of "gestalt". Esterel, BTW, has had much more success than Haskell in the industry.

Another example is TLA+, which is used by Amazon to verify their (Java, I assume) web services[2]. Unlike Esterel, TLA+ is Turing complete, and can therefore fail to prove many things. It will also take longer and longer to run the bigger the code is, and therefore only verify algorithms, not entire systems.

Also, see the history of Statecharts[3], a predecessor to Esterel, that was designed first and foremost to assist humans in writing specifications and only later combined with Pnueli's temporal logic to facilitate formal methods.

BTW, the same David Harel who worked with Pnueli on Statecharts and temporal logic, is now involved with Behavioral Programming[4] a very interesting programming style that makes human specifications easier to translate to code. For a usage example, see Harel et al., _Programming Coordinated Behavior in Java_ [5]

[1]: [https://en.wikipedia.org/wiki/Esterel](https://en.wikipedia.org/wiki/Esterel)

[2]: [http://cacm.acm.org/magazines/2015/4/184701-how-amazon- web-s...](http://cacm.acm.org/magazines/2015/4/184701-how-amazon-web-services- uses-formal-methods/fulltext)

[3]: [http://www.wisdom.weizmann.ac.il/~harel/papers/Statecharts.H...](http://www.wisdom.weizmann.ac.il/~harel/papers/Statecharts.History.pdf)

[4]: [http://www.wisdom.weizmann.ac.il/~bprogram/](http://www.wisdom.weizmann.ac.il/~bprogram/)

[5]: [http://www.wisdom.weizmann.ac.il/~amarron/BPJ%20ECOOP%20FINA...](http://www.wisdom.weizmann.ac.il/~amarron/BPJ%20ECOOP%20FINAL.pdf)

<comment><USER6> Thanks for the references.

</comment></comment></comment></comment></comment></comment></comment><comment><USER4> I still honestly haven't given this enough time to digest—my Java-fu is quite rusty so I'm sort of reading between the lines on what I know about delimited continuations generally.

First I want to state that delimited continuations form a monad and (I believe) that in order to get the Java implementation working you need to implement that monad in the "ambient monad" of Java.

Second, I want to dissect the notion of "monad" as used here into two pieces. First, there is the "PFP Notion" of monad which tends to be represented as a data type and some associated operations. Actually, I mean this even more specifically as being aimed _precisely_ at the standard library monads of Haskell. Secondly, there's the mathematical "monad" which is just a qualifier assigned to any structure which satisfies the monad laws.

Delimited continuations are _definitely_ a mathematical monad. They are represented by one PFP monad, but are generally eschewed as a style there. Why? Because delimited continuations are very "large" and PFP tends to favor the idea of having your data "forbid invalid state" so... large types are disfavored.

So now, to the third point: I think this is all just "implementing a specific overlarge monad in Java and Clojure". It turns out that this is really convenient because overlarge things _always are_. They are of course more convenient in PFP, too. But this convenience has the price of making code difficult to treat as data, difficult to analyze statically.

All that said, it doesn't mean at all that this isn't a useful technique for implementation of monads in Java/Clojure.

Four, what about composability? (Specifically, here, we mean functor composition.) Turns out that delcont composes nicely compared to the PFP-style monads. The reason why is again because of lack of restrictions on state space. Non-composability of monads is a _good thing_ because it relates to non-commutativity of certain effects.

You can get better composability in three ways: throw out all the non- commutative effects, work with subsets of effects a particular ambient monad (see Purescript's impl of Eff), split the specification of effects from the implementation of effects and leave the decision of ordering up to the implementer (see Haskell's mtl).

If you don't want any of the above, just realize that non-composition is telling you something and forcing you to make choices. One choice is to build a new kind of composition (monad transformer composition) which does work the way we want by being more sensitive to ordering.

Fifth, and finally, all the above is not to say that there's any lack of merit to using delcont for effects. It's a great technique. It may fit the "algorithmic mind" better, too.

I am trying to say, however, that it's not some grand trick that frees the user from thinking about monads. You still are, you just trade one set of implementation concerns for another. But since things are still monads then the same _mathematical_ techniques will translate. Which is a great thing—it means there are more tools to think about, understand, and become confident in the application of this technique.

<comment><AUTHOR> > I am trying to say, however, that it's not some grand trick that frees the > user from thinking about monads.

This brings us back to the discussion of what it means to be "thinking about monads". When a cashier hands me change, is he "thinking about groups"? Is a Rubik's Cube whiz thinking about groups? Sometimes people are better at something by thinking about it in a completely different way. That is the exact meaning of duality: two things are the same, but different circumstances may make looking at it in one way more useful than the other. A "push" API (of which "PFP monads" or the "monadic style" as I call it, is an example) and a "pull" API are also duals, yet I believe that when writing concurrent code in imperative languages, the pull shape is style preferable.

As discussions about cognition might make some PFP people uncomfortable -- we all should just think "in math", right? (I'm kidding. Mostly ;)) -- we can take this back to theory and say that switching between different representations requires computation. In fact, it can be argued that _all_ computation is simply about changing representations and encoding things. 4 _is_ 2 x 2, but going from representing that same thing as "2x2" to "4" is what computation does (and the lambda calculus computes just by substitution -- namely, by "doing nothing" from a mathematical point of view). All this is to say that going from looking at something one way to looking at it another way actually requires some computational power and is not free.

<comment><USER4> I don't think we're disagreeing at all in principle. I'm mostly trying to point out that your change is a group whether you think about it or not (actually, several). Then, let the best representation win.

Monads aren't a representation, they're a functionality.

<comment><AUTHOR> Oh, I know we're not disagreeing in principle, but I do think monads are a _representation_ of a functionality -- effects -- in an algebra that does not have impure functions. Once effects are introduced as fundamental constructs, there is no need to be "thinking in monads" anymore.

Imagine if we defined our algebra in such a way that there were no functions -- i.e. nothing that can take parameters and return values -- just sequences of instructions that read and assign global variables (and then define -- outside the calculus -- other effects, such as IO, as changing certain variables). We could then define pure functions in terms of this calculus, rather than doing the opposite and defining change to state in terms of pure functions (i.e. monads). We could then say that a pure function is _really_ a subroutine that performs a transformation on variable addresses which we'll call "variable renaming", and writes its output in some well-known address that the "caller" then renames.

In fact, that calculus exists in the Turing Machine, or in our actual computers, and the work of the Haskell compiler is to transform the "high level" concept of pure functions to the "fundamental" representation of instructions and effects... We could then say that even when working with functions you still have to think in terms of instructions (this is actually true if you care about complexity or its earthly form -- performance).

Of course, there is no objective way to decide which of these representations is more fundamental.. They are just different, requiring a compiler -- or a mind -- to switch from one to the other. It is, however, possible that one of them results in more efficient programs on current hardware than the other, and one of them might be more natural for human cognition than the other. Or the same one could be both ;) but actually I think reality is more interesting than that.

I think both human cognition and compilers might not prefer either but a combination of both. Messy from a mathematical perspective, but that's the nature of computation -- in neurons or silicon. When we need to formally reason about the program, the formal verification tool can then transform the messy representation into one or the other "robust" algebras and do its thing there.

</comment></comment></comment></comment><comment><USER8> Uh... What do you think monad transformers do?

</comment><comment><USER9> Very interesting piece. It sounds a lot like how exceptions be used to implement non-error control flow [1]. The one bone I have to pick is the juxtaposition of PFP and imperative as opposites. Certainly it's the case that PFPers can and do implement imperative logic monadically, but it's also the case that imperative programmers often use frameworks that largely treat the business logic, where most of the programming is actually done, as more or less pure functions. Moreover, explicitly hybrid languages like JavaScript, Scala, Rust, Swift, etc. seem to be where a lot of the practical PL excitement is these days. I think the real debate is in the best patterns to use for particular generic problems. That's the core of the blog post of course, I'm just disagreeing with some of the rhetorical framing.

Anyhow, the premise reminds me a lot of _What Color Is Your Function_ [2] and its critique of async programming. I think monads are super useful and intuitive when representing data. Promises/futures blew my mind, once I finally understood them, but I think there are important points about how they can become extremely cumbersome within a large architecture. I have to admit that I'm starting to agree that they may be a dead end, no matter how much syntactic sugar you sprinkle on it.

I think the big issue is that if you can't pull data out of a monad, you can't really encapsulate its logic. For containers, this isn't much of a problem, because the monadic data structure is either inherent to the data itself or you can fold your way down to something more primitive. In those cases, the monadic model seems completely reasonable and very nicely encapsulated to me. I'm not sure I buy the idea that scoped continuations should replace this monad usage.

Connecting back with async programming, I wonder how this jives with the actor model, which has been so widely praised as a model of concurrency that is conceptually simple, easy to reason about, fault tolerant, and imperative. When the author mentioned the thread being the atom of imperative programming, actors immediately came to mind.

Lastly, much like monads, futures and every other abstraction out there, the usefulness of scoped continuations may come down to how easily it can be expressed and comprehended within real programming languages. I found the examples in the original talk and the blog post to both be rather difficult to follow. The former, probably because of the use of = to select the scope and the latter because of the verbosity of Java. It would be interesting to see if there's a syntax that really captures the operation in an intuitive way.

[1] [http://stackoverflow.com/questions/2742719/how-do-i-break- ou...](http://stackoverflow.com/questions/2742719/how-do-i-break-out-of-a- loop-in-scala)

[2] [http://journal.stuffwithstuff.com/2015/02/01/what-color- is-y...](http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your- function/)

<comment><AUTHOR> > The one bone I have to pick is the juxtaposition of PFP and imperative as > opposites.

Well, I most certainly didn't say FP and imperative are opposites; quite the contrary: _most_ FP languages are imperative. Part of the confusion is that there is no definition of FP. PFP, OTOH, _is_ well defined: it is a style of programming where _all_ functions are pure. If you _can_ write impure functions, you're not doing PFP, and if you're doing PFP, you can't write impure functions.

> I'm not sure I buy the idea that scoped continuations should replace this > monad usage.

I'm not entirely sure what you mean by "this usage" (lists?), but, like I say in the talk, I think monads are great for parallel streams, where you explicitly don't care about the stack context.

> I wonder how this jives with the actor model... When the author mentioned > the thread being the atom of imperative programming, actors immediately came > to mind.

Exactly. Threads (and by that I mean any thread implementation, not just kernel threads) are the best use of continuations, and blocking actors like Erlang's (or our own Quasar) make great use of (lightweight) threads, as do Go's goroutines (and Quasar fibers).

> may come down to how easily it can be expressed and comprehended within real > programming languages. I found the examples in the original talk and the > blog post to both be rather difficult to follow.

My opinion is that it's not important for scoped continuations to be directly exposed at all, and if they are, they should be rarely used directly. Using them to create lightweight threads solves almost all of the problems with the monadic style, and the code looks just like any plain-old thread-blocking code (we've had very good success with that in Quasar; Code stays the same, but instead of blocking a kernel thread, it blocks a fiber; the APIs don't even need to change).

> The former, probably because of the use of = to select the scope

I admit that when I gave the talk the precise details of the API and usage weren't yet clear to me, which is why I wrote the post.

> because of the verbosity of Java

Where does Java's verbosity bother you here? (although I'm probably biased, as I like Java's, or Go's, intentional verbosity)

<comment><USER9> Fair point on PFP.

> I'm not entirely sure what you mean by "this usage" (lists?) [...]

By "this usage", I meant containers, generally. Maybe algebraic data types would be more specific.

> [...] blocking actors like Erlang's (or our own Quasar) make great use of > (lightweight) threads [...]

Have you been following the Reactive Manifesto [1] and reactive streams [2]? If so, do you have any thoughts on whether this it builds out design practice in a complementary way to your suggestion here, or is it divergent?

> Using them to create lightweight threads solves almost all of the problems > with the monadic style, [...]

My concern here would be whether you'd then be recreating the old problem you mentioned of having many implicitly monadic constructs in imperative programming but no good way of adding your own. In the analogous case in PFP, even if the continuation monad is the mother of all monads (and I haven't finished reading that post you linked to), I'd have to imagine it wouldn't be a whole lot of fun to use it exclusively. Or else people would do it!

> Where does Java's verbosity bother you here?

I probably should have withheld comment on Java's verbosity. There's probably nothing wrong with it. I think I'm just used to Scala and dynamically typed languages.

Anyway, awesome work. You make a lot of great points.

[1] [http://www.reactivemanifesto.org/](http://www.reactivemanifesto.org/)

[2] [http://www.reactive-streams.org/](http://www.reactive-streams.org/)

<comment><AUTHOR> > Have you been following the Reactive Manifesto [1] and reactive streams [2]? > If so, do you have any thoughts on whether this it builds out design > practice in a complementary way to your suggestion here, or is it divergent?

Reactive streams' design is the dual to threads/fibers/continuations, and I believe that as a _user facing construct_ it is the wrong one (I'll get to that in a minute), but, as Java does not yet have a standard, built-in implementation of continuations/fibers, Reactive Streams is meant as an _interoperability standard_ and does a _very good job_ at that. I like that it exists and I like the spec, which is why Quasar is also RS compliant (turning every stream to a channel and vice-versa).

I don't like the reactive manifesto, though. It conflates some well-known design practices -- which we simply used to call "robust soft-realtime" \-- with a specific implementation (nonblocking), and in the process, overloads a term, "reactive programming", which has a very different meaning (dataflow, i.e. spreadsheet style). Even Erlang, the inspiration to the manifesto, doesn't conform to it (because it discusses implementation). Also, I'm not too fond of manifestos in general.

As to push-based streams, I didn't have time to get into that in the talk (the slide is there, though), but I believe that in imperative languages a pull- based API (like channels) is always superior to a push API for several reasons:

* It is more general -- both are duals, but going from pull to push only requires a while loop. Going the other way requires adding a queue.

* It conveys more information to the developer -- in the push approach, the concurrency is clear: the item is received on the same thread that calls `receive` (the only thing you need is maybe just a tag saying if `receive` itself is thread-safe or not). With a push API, the developer needs to read the documentation to figure out whether the callback can be called on multiple threads concurrently, and, if not, whether it's always called on the same thread.

* It conveys more information to the library -- a pull API has implicit, automatic backpressure. Calling `receive` means we're ready to receive more. With a push API you either don't have backpressure, or, like with reactive streams, you need explicit backpressure (`request`) along with more documentation on when `request` may be called. Look how much of the RS spec is devoted to specifying exactly that, and how much shorter it would be if it were a pull API (like Go or Quasar channels).

> My concern here would be whether you'd then be recreating the old problem > you mentioned of having many implicitly monadic constructs in imperative > programming but no good way of adding your own.

Well, yeah, but that's a secondary issue. I'm not a language purist, and you always need to weigh which is better -- a very general abstraction that is then put to use by everyone to create their own incompatible DSLs, or choose the most useful specific abstractions and make sure everyone can read everybody's code, even if not everything now has the most elegant expression. For large projects I think the latter is preferable.

</comment></comment></comment></comment><comment><USER10> It's pretty interesting that the author did not talk about async/await which gives us back the imperative coding style with futures, effectively solving the problem - it has been adopted by many languages like Python, C# and JavaScript and can be implemented with generators. Generators, are the generalization that gives you the expressiveness you need (not fibers) here. Where are they ineffective here?

<comment><AUTHOR> async/await are a form of _stackless coroutines_ (or _stackless delimited continuations_ although that term is rarely, if ever used). They're a limited form of delimited continuations, and are mostly used as a compromise when implementing full coroutines is hard (or not wanted for some reason). Clojure's core.async's go blocks are also stackless coroutines. Basically, they're delimited continuations that only allow suspension ("shift") at the topmost stack frame of the continuation, and don't allow suspending at deeper stack frames.

<comment><USER11> I think that most languages don't want the complexity of coroutines, as they are a unfamiliar concept to most programmers. It is a shame that they've never gained popularity as they are very general concept.

<comment><USER12> A different take on the merits of explicit yielding (whether by async/await or raw generators), and the perils of full coroutines, by Glyph Lefkowitz of Twisted Python fame:

[https://glyph.twistedmatrix.com/2014/02/unyielding.html](https://glyph.twistedmatrix.com/2014/02/unyielding.html)

<comment><AUTHOR> I'd say that Lefkowitz's claims are mostly relevant when your runtime has no multicore support (like Python). If your runtime is single-threaded, explicit yields _might_ (it's mostly a matter of taste), make explicit reasoning about state-change ordering easier. But then you're using shared-state concurrency again, and unsafe shared-state to boot. If your runtime has multicore support -- like Java or .NET -- all this doesn't matter, because those concurrent changes Lefkowitz is talking about may happen any time -- not just at yield points.

The solution, of course, is not to use unsafe-shared-state-concurrency! Either use threads with messages (like Erlang), or threads with transactional shared- state, like Clojure, or both (in fact even Erlang has both actors and ETS, and Clojure has agents and channels as well as atoms and refs).</comment></comment></comment></comment></comment>


Metadata vs Data: a wholly artificial distinction - <AUTHOR> http://blogs.fluidinfo.com/fluidDB/2009/09/05/metadata-vs-data-a-wholly-artificial-distinction/

<comment><USER2> _At an architectural level, I think that's entirely wrong._

In my applications, I've encountered the opposite. On an architecture level _data_ is usually stored in a relational database, where the nodes of the indexes' B-trees are optimized in size for the disk and processor cache, foreign keys are defined and enforced, and transactions ensure consistency. Meta-data, on the other hand, is significantly more dynamic (the structure changes as an application evolves) and several orders of magnitude smaller in size (hence, no need for indexes), and hence better stored in XML, Yaml, or JSON. I've certainly seen people confusing the two - usually by creating a ton of structured DB tables which should be replaced with a config file.

The fundamental difference I see is this: _Data_ should be optimized for scale. It needs to be well-structured, so your architecture can use that structure to scale well as the amount of data increases. _Meta-data_ should be optimized for change, so the structure can change as your application evolves. Now, there's a lot of ways to structure data, so there's certainly a lot of choices in how (and how much) you want to structure your data, but the trade- off is real - you either spend time defining/maintaining structure to optimize performance, or you ignore performance and structure for the sake of change/development speed.

It's really the same question as with type systems in programming languages - type definitions are meta-data about code which lets compilers/interpreters optimize performance, at the expense of a developer having to write and maintain type definitions.

That said, I agree that for applications which don't require exceptional efficiency, there's a lot to be gained by unifying all data at one point in the structured/unstructured spectrum. But this doesn't mean the spectrum ceases to exist.

<comment><USER3> I don't think you're contradicting the post. He's talking about the meaning of the concepts and you're talking about how to get efficient implementations. Both are important, but if you didn't have to worry about efficiency, it's pretty obvious that the data/metadata distinction is a logical false turn. (To use your terminology, why wouldn't we want everything to be fast, dynamic, and unified?)

The post does hint at this by suggesting that the concept of metadata itself arose from tradeoffs imposed by hardware. Where I think you differ from Terry is that while he implies that it's time we moved beyond all that, you're saying that the tradeoffs (e.g. between scale and dynamism) remain fundamental. That's a legimitate concern. It's not obvious that we can have our cake and eat it too. It will be interesting to see how FluidDB performs at scale over time.

We've been thinking a lot about these issues on our startup too, because we need both speed and dynamicness. The solution we're arriving at is to exploit the static (structured) cases to get as much speed out of them as we can, be fast-ish on the most common dynamic cases (which thankfully are relatively simple), and be correct-but-slow on the complex dynamic cases. What we won't do, however, unless absolutely forced to, is introduce architectural distinctions between these. We want a consistent, regular system (no two types of data, no two types of anything) that happens to run very fast when handed common usage patterns but still allows for arbitrary flexibility. What we're betting is that the usage patterns will be something like 90% static, 9% dynamic-but-simple, and 1% dynamic-and-complex; we want that to map to 90% super-fast, 9% fast-enough, and 1% slow.

<comment><USER2> _To use your terminology, why wouldn't we want everything to be fast, dynamic, and unified?_

My assumption was that this is provably impossible - that well structured data will always be more efficient to interact with, but on second thought, I can't think of a good reason for that. To the contrary, it seems computational speed is unbounded, while the database size/speed we need are (for most current problems) bounded. Hence, "relatively slow" is going to be "fast enough" more and more often.

<comment><USER3> _on second thought, I can't think of a good reason for that_

Isn't it simply that if you know the structure in advance, you don't have to figure out what it is at runtime?

I agree that the tradeoff becomes moot if you can get both sides fast enough. The question is whether present-day hardware has gotten us to that point. That's why FluidDB is an interesting test case of whether a system supporting arbitrary flexibility can perform at scale. I assume Terry isn't being naive about this, which means he must be making some tradeoffs somewhere. The question I'd like the answer to is, what battles is he consciously _not_ trying to fight (at a design level)? Terry Jones, if you're reading this, please speak up :)

</comment></comment></comment></comment><comment><USER4> I'm not entirely sure the author himself understands the difference between data and metadata. It has nothing to do with importance; rather metadata is "data about data". The author seems to think it's got something to do with a distinction between system and user data, which isn't the case. It's an important logical abstraction to make, as "data" is a noun, whereas metadata can be thought of as a collection of adjectives.

Now if you want to start discussing why "data about data" should be considered part of the data itself, that's an argument we can have... but that doesn't appear to be the goal of this article (at least if it is, he doesn't make much of an argument on that level).

<comment><USER3> Huh? He knows perfectly well what the definition of "metadata" is. It's not that hard a concept. Nor is it hard to see why it's muddled, or why systems that make an architectural distinction between the two (data and metadata) are clumsy and rigid. In fact, the whole point is rather obvious (or ought to be), but he makes up for it by offering specific insights. I thought the historical speculation was particularly good.

</comment></comment><comment><USER5> I prefer to think of the distinction as _fluid_.

What one application considers metadata is another application's data.

From the apparently trivial (file sizes, line numbers) to the domain specific (date of joining social network; file server cluster that hosts this image), each tool doing the work has a very different viewpoint.

The motivation for abandoning the distinction is this fluidity, not that there is no distinction.

(Sidenote: I like the Clojure definition of metadata:

"It is used to convey information to the compiler about types, but can also be used by application developers for many purposes, annotating data sources, policy etc.

"An important thing to understand about metadata is that it is not considered to be part of the value of an object. As such, metadata does not impact equality (or hash codes). Two objects that differ only in metadata are equal.")

</comment><comment><USER6> _so hard to find, for example, a file with "accounts" in its name and "automobiles" in the contents ... Consider the UNIX find command which searches based on file metadata and the grep command which searches file contents. Combining the two is not easy._

    		grep -i automobiles `find / -type f -name *accounts*`

_Interestingly, Amazon are currently being sued because they threw away someone's metadata in the process of removing a copy of Orwell's 1984 from a Kindle. You can bet the metadata was removed automatically when the content was removed._

Actually, they are restoring the file to anyone who wants it, including all notes.

[http://www.engadget.com/2009/09/04/amazon-offers-to-give- bac...](http://www.engadget.com/2009/09/04/amazon-offers-to-give-back-your- kindles-copy-of-1984/)

<comment><USER7> This is at the application level. The point is that there's no way to do this as one search without building an index a la Spotlight. You can't write a program called "nameandcontents" that does the above without doing exactly what find and grep do - there is always the data equivalent of NUMA... call it Non-Uniform Data Access. The author wants, at an architectural level, but not application level, UDA.

</comment></comment><comment><USER8> A meta-distinction.

</comment><comment><USER9> It's just data - all the way up and all the way down (??)</comment>


Inadmissible assumptions underlying Internet policy - <AUTHOR> http://www.antipope.org/charlie/blog-static/2012/04/the-inadmissible-assumptions.html

<comment> mike-cardwell If you create a website that users pay for and it becomes popular, somebody will come along and create an Ad funded version and take some or all of your users (assuming everything else is equal).

I have no problem with this whatsoever. I have no problem with advertising. What I do have a problem with is advertisers tracking people _across websites_ , and that all of the major browser vendors are happy to facilitate/encourage this.

If all of the major browser vendors just tied third party cookies to the domain of the top level origin, this sort of cross site tracking would be blocked, with the bare minimum of damage. It wouldn't kill advertising, it would just stop advertisers compiling data that they didn't ask permission for.

There are of course lots of other tricks to track users even if they turn off third party cookies altogether. But lets at least try to get the low hanging fruit first, please.

EDIT: It would probably help if the major browser vendors didn't make large sums of money from this sort of advertising. Talk about a conflict of interest. It would be better for everyone if the evolution of the web wasn't so heavily guided by advertisers.

<comment><USER2> _What I do have a problem with is advertisers tracking people across websites, and that all of the major browser vendors are happy to facilitate/encourage this._

I think everyone has a problem with this to a degree. I'm ok with Google giving me personal search results and Facebook recommending me products based on recent wall posts. What I'm not ok with is my Fiancé searching for wedding ideas one night on her laptop at home, and the next day having all the website ads being shown to me from a particular ad network on my iPhone being wedding related while connected to my home WiFi. This really happened to me 2 weeks ago.

I expect a certain amount of privacy, for both her and myself. Nothing needs to be written in law that says that I cannot be tracked, but there should be a moral code on what kinds of ads are displayed and where they will be displayed. If advertiser networks can't develop a system that isn't invasive then they need to go away, via adblock or otherwise.

<comment><USER3> To add to your point, the problem with cross-website tracking is that it breaks the mental model that people have about these web services.

For instance, one would expect for Google to know everything you searched for. I don't think that anybody is naive enough to not realize it, even non- technical users. So if you're searching for something and you don't want Google to find out about it, then the simplest thing you can do is to NOT search on Google.

And the problem with cross-website tracking is that this model breaks. Now Google / Facebook know what websites you visit, even if you were careful about not searching on Google, or not sharing/following links on Facebook.

It's pretty scary actually.

</comment></comment></comment><comment><USER4>
        The idea that "most people only want to consume" is
        profoundly offensive and serves the interests of abusive
        "producers" who tend towards rent-seeking (see the MPAA
        for a worked example—most notably in how they run the
        film classification system in the USA),

It does not matter if it is offensive, so I will just have to ask the question:

Is it true?

<comment><USER5> It is absolutely true that the majority only wants to consume. I think the author is taking offense at a stronger assumption, which is "Only an elite few want to produce" the number of people which want to produce is much higher than the number of people that have historically been given a wide audience for their productions.

</comment></comment><comment><USER6> I take issue with:

 _b) Funding content via ad sales holds our public arts hostage to a boom/bust bubble economy. Furthermore, there is an incentive for web publishers to prioritize paid ads over editorial content, and to censor editorial content that threatens advertizing revenue_

First, while public arts sounds good, it means entertainment. In a bust, you're likely to wind down your entertainment expenses, in a boom you wind them up. Restaurants are not ad-funded, but still very sensible to the business cycle.

Second, whether ads or not, editors have an incentive to do whatever nets them the most money. For movie producers, it's make a movie that the most people will want to go and see. For authors, it's writing a book that people will want to read.

Correct me if I'm wrong, but this seems to be a euphemistic endorsement of TV Licensing-style financing. However, that runs contrary to point _d) Nobody can be trusted_.

<comment><USER7> I agree.

Other commenters take issue with c), that the majority are consumers. If the art we consume is to be excellent, it will be _rare_. Not just from the supply and demand curve, but with the internet's ease of distribution comes a global "bazaar" of artistic creations that pushes up the threshold of excellence.

Artists then face an essentially limitless supply of "good enough" competitors, even though only a small percentage of internet users are actually content producers. (The rest of us, in my opinion, are those "consumers.")

Advertising works when scarcity does not - Advertising can be shown for all the mediocre works that aren't going to make much money any other way.

For the truly great stuff (AAA Games, Hit music, Movies, and so on) the price of admission will generate a lot of revenue. But for the mediocre stuff, the business model has to adjust.

Personally, I think the real challenge lies in the "below mediocre" art. The market mostly ignores things like flash games, small bands, fanfiction, etc. To me there's a missed opportunity to cultivate the "startup artists" as a sort of Y Combinator.

</comment></comment><comment><AUTHOR> Footnote: after some naval-gazing, I have to conclude that all of these assumptions are based to some extent on the axiomatic belief that market-based approaches are the best way of managing a cultural communications medium.

<comment><USER8> Would you mind explaining what the alternative is? (that is, what non-market- based approach to web publishing is better than the status quo?)

<comment><USER9> As a little-l libertarian, I'd accept that there's a bit of market failure here as the friction of making payments has exceeded the relative ease of "just slap some ads on it". However I suspect this problem is fixing itself over time. Ad inventory is only going to increase, which drops what you can change on average, which is slowly but surely going to take "just slap ads on it" out of the reach of more and more people, and alternate payment methods, while not quite as easy as just shouting "Micropayments!" and walking away with a smug expression, are in fact developing. Kickstarter is one example, and other models like that will develop and return the concept of "patronage" to the field of culture by mixing in a heavy dose of "crowdsourcing". And once we're comfortable with that, who knows what else we might come up with?

And hey, maybe somebody will figure out that whole micropayment thing too. But at least now there are increasingly concrete demonstrations that they were never the only solution, which is fortunate since they don't seem to work.

I'm actually more optimistic now that we can escape from ads as the only choice than I've ever been before. I'm hoping that this is the last year or two that cstross could make this post and have it still seem a reasonable concern.

<comment><USER10> I don't know, it seems like it is going the other way. At least for the buyer, internet ads have become more expensive over time. I imagine mobile ads will become more expensive as well.

</comment></comment></comment></comment><comment><USER11> Advertising has less friction than any other possible solution (so far). That simple fact overrides all other concerns because it means stuff gets paid for where otherwise it wouldn't. Any other solution that creates more friction will fail.

<comment><USER12> There are costs to advertising, but most are hidden. They will start showing on the long run, IMHO. There will be a lot of friction.

<comment><USER13> Like what?

</comment></comment></comment><comment><USER14> It's a bit immature to link to a dystopian fiction as a citation to show that "everyone" can be evil

</comment><comment><USER15> I'm sorry, was there a point? All I see is a series of assertions. (Most of which agree with my ill-considered prejudices, but....)

</comment><comment><USER16> This is only true for certain segments of the internet.

Internet gaming, for instance, has successfully leveraged a subscription based model, as do some specialist interest sites.

Personally, I suspect that the main reason that the first Internet mega-corps have been ad based, is just because that is the easiest industry to implement on what most people are still regarding as merely another media channel.

However, I strongly doubt that it is even close to being the most profitable thing you could do with a global computer network.</comment>