Ask HN: Are macros still an enormous edge in the power of a language? - <AUTHOR> I use Lisp (Clojure) a lot and like it, but it seems like macros, except in specialized cases, aren't quite the game changer they used to be. They're much tougher to reason about than functions, can't be passed around as parameters, and can lead to "magic" DSLs that only their owners understand. Obviously they are sometimes very useful, and the built-in macros (e.g. cond, if-let) are essential, but I don't see them as such a game-changer in modern programming. I feel like over 90% of the edge Lisp has over Blub comes from its functional support, with less than 10% coming from macros. This makes the edge small enough that, in many circumstances, I can see languages like ML and Haskell winning; good static typing (e.g. ML, not Java) becomes a benefit on large projects.
I could be convinced that I am wrong and just don't "get" macros yet, but it seems like much of what's shown off in On Lisp et al is available nowadays in non-macro languages. I'd like to see modern uses for macros that aren't easy to implement in macroless languages. <comment><USER3> I used to like arguing over the Internet about this subject. There are many good technical and management/organizational arguments you can make for and against macros. What I've come to realize is they're all pretty much irrelevant.

The entire point of programming is automation. The question that immediately comes to mind after you learn this fact is - why not program a computer to program itself? Macros are a simple mechanism for generating code, in other words, automating programming. Unless your system includes a better mechanism for automating programming (so far, I have not seen any such mechanisms), _not_ having macros means that you basically don't understand _why_ you are writing code.

This is why it is not surprising that most software sucks - a lot of programmers only have a very shallow understanding of why they are programming. Even many hackers just hack because it's fun. So is masturbation.

This is also the reason why functional programming languages ignore macros. The people behind them are not interested in programming automation. Wadler created ML to help automate proofs. The Haskell gang is primarily interested in advancing applied type theory.

Which brings me to my last point: as you probably know, the reputation of the functional programming people as intelligent is not baseless. You don't need macros if you know what you are doing (your domain), and your system is already targeted at your domain. Adding macros to ML will have no impact on its usefulness for building theorem provers. You can't make APL or Matlab better languages for working with arrays by adding macros. But as soon as you need to express new domain concepts in a language that does not natively support them, macros become essential to maintaining good, concise code. This IMO is the largest missing piece in most projects based around domain-driven design.

<comment><USER4> Excellent analysis!

</comment></comment><comment><USER5> Yes. It's part of good Lisp style not to use a macro when a function would suffice, and yet the source of HN is full of macro calls. If you want to see examples of modern uses of macros, that would be a good place to start.

<http://ycombinator.com/arc/arc3.tar>

If you want a single, self-contained example, here's one:

        (mac zap (op place . args)
          (with (gop    (uniq)
                 gargs  (map [uniq] args)
                 mix    (afn seqs
                          (if (some no seqs)
                              nil
                              (+ (map car seqs)
                                 (apply self (map cdr seqs))))))
            (let (binds val setter) (setforms place)
              `(atwiths ,(+ binds (list gop op) (mix gargs args))
                 (,setter (,gop ,val ,@gargs))))))

How do you implement that in a language without macros?

<comment><USER6> Care to explain it for the Lisp-impaired?

<comment><USER5> It lets you apply an arbitrary function to a variable or a position anywhere within some structure:

        arc> (= flag t)
        t
        arc> (zap no flag)
        nil
        arc> flag
        nil
        arc> (= foo '(1 2 3 4 5))
        (1 2 3 4 5)
        arc> (zap (fn (x) (* x 10))
                  (foo 0))
        10
        arc> foo
        (10 2 3 4 5)

BTW, in more idiomatic Arc you'd say

        (zap [* _ 10] foo.0)

Here's a line in HN that uses it:

        (zap [firstn votewindow* _] (uvar user votes))

This expands into roughly the equivalent of:

        (= ((profs* user) 'votes)
           (firstn votewindow* ((profs* user) 'votes)))

Not quite, though, because you only want to evaluate the code that gets you to the location once, and the operation needs to be atomic. The actual macroexpansion is much more complicated.

<comment><USER7> If that's all it does, then I don't see a problem with implementing it in any language without macros. Let's take simple C. (I didn't test it, or even try to compile - just want to show the idea)

        void zap(void *object, void* (*selector)(void *), void (*action)(void*)) {
            action(selector(object));
        }

Given your explanation: "It lets you apply an arbitrary function[1] to (a variable or a position anywhere within some structure)[2]."

Where object is the structure you're operating on, `selector` is a function taking an object and returning a pointer to the position [2]. The `action` is a pointer to an arbitrary function [1].

Your first example would look like this:

        int foo[5] = {1, 2, 3, 4, 5};
        void *select_first(void *x) {return x;}
        void act_times_10(void *x) {int *i = (int *)x; (*i)*=10;}

        zap(foo, select_first, act_times_10);

Or am I missing something you didn't include in the description?

<comment><USER8> I think you're missing the fact that in the C version you have to write a specialized selector function each time (to find the thing you want to zap), whereas the macro works generically on any place. ("Place" here is an abstraction meaning "settable thing" that is closely related to Lisp macros and is analogous to the left-hand-side of assignment statements in most languages, but far more manipulable by the programmer.)

<comment><USER7> PG asked "How do you implement that in a language without macros?". I responded with a simplest translation of the example, but a more general solution exists too. Nothing stops me from using GObject instead of void*, which can provide named "places" and as much generic functionality as needed. `selector` and `action` can even operate on property tables, which would be a closer translation from lisp.

I was simply surprised that PG asked about something like that - I still think it's pretty simple and the code in C is as close to the macro as possible (with whatever object abstraction you need).

However I have to admit that I thought about other languages now and it seems that the described zap() is not possible in python... and that's interesting / worrying:

        action(object[place])

cannot assign a new value to `object[place]`, while

        object[place]=action(object[place])

has to do the lookup twice.

</comment></comment></comment></comment></comment></comment><comment><USER9> The issue of macro is that it is, by itself, not a power. It's like how knife by itself is not food. However, knife can make you food. That's the main point.

Imagine this Christmas, Perl 6 is out with some killing features. Now, your language (Python? Haskell? ML? Whatever) does not have these features. What would you do?

You have 2 choices, basically: a. forget about that features, write a long blog in which you prove how your language can do just the same thing, albeit "a little" less elegant b. Switch to perl. This is not always possible, btw, since you may sacrifice some OTHER features that your language has.

How about a LISPer? Well, simple, let's implement these features. Keep in mind: All languages are Turing equivalent, meaning they can do roughly the same thing. There are two differences: 1\. some do some task more elegantly (aka easier to maintain) than others. 2\. Some have better support (eg. corporates, community, etc.) than others

Thus, you CAN always write the exact same features in LISP. Now, the definition of the macros WILL be messy. However, as a good practice, you spend 2 days debug that holy mess, and jam it into a file you banish somewhere with a dot precede the directory. Next time, you can use the features, pretending that you are using Perl 6!

This is my strategy. My Scheme code always has for-loop (purely functional, too), foreach-like, OO-like features. Of course, Scheme (remember, it's not even CL) does not provide these, but I can either implement them or find the implement somewhere. I have once even try to implement Icon-like mechanism with Macro + Continuation (okay, the Scheme implementation that time was a bit fuzzy, so I failed. But, with pure Scheme, it SHOULD be possible). That's the main power: you can copy others' ideas extremely fast! Without throwing away your old code, too. Hurray to Macro!

</comment><comment><USER10> Are macros still at the "top" of the features pyramid?

In <http://www.paulgraham.com/avg.html> , PG argues that Lisp is more powerful than other languages because it has macros. However, in the 50 years since the invention/discovery of Lisp, is there now some language that has a new feature X that lisps don't have (and can't easily add)? Or are macros still the "pinnacle" of language features?

<comment><USER5> _Lisp is more powerful than other languages because it has macros_

It's a bit more complicated than that. The key idea is that programs are composed of lists. But that idea is tied to several others:

<http://www.paulgraham.com/diff.html>

<comment><USER10> Fair enough, although I was asking a different question (not very clearly, I suppose). Is there a language X and a feature (or set of features) Y for which the following statement holds?

"Language X is more powerful than Lisp because it has feature(s) Y."

</comment></comment></comment><comment><USER11> In 2006 over 2 months, I wrote an OCR (optical char recognition - <USER2> images back into text) system in Lisp using macros.

First it finds all 'features' by traversing a 1-pixel wide representation of the text character's image. Then you describe which features you want to look for, in a macro language.

So (in layman's terms) , a 'c' is a top curve curving up, a lower curve curving down, and a stem at the left-hand side close to the left-hand of those curves. So 'c' would be about 3 rules, in this macro language.

It was fun but monstrously sluggish and I never got the 1-pixel wide algorithm (called skeletonisation) that I used, to work 100%.

One thing it did have going for it was you could identify really small font chars and even 10-pixel-high truetype fonts, as long as you normalise so each char is 60 pixels high.

I ported the system to C++ also in 2006 and had to write a simple Scheme-ish interpreter to turn the Lisp ruleset into C code.

</comment><comment><USER8> Macros expand into other code, so in the literal, obvious sense, there isn't anything you can do with them that you can't do without them. The question is whether you can do it as briefly, as clearly, or as composably.

Here's another example. I work on a spreadsheet app. A spreadsheet has many operations that apply to either a row or a column (e.g. selecting a row/col or resizing it). The fact that they occur in transposed dimensions make them difficult to abstract over with functions. You probably end up with a bunch of boilerplate that says things like "if this is a column, change the first co- ordinate, otherwise change the second", or most likely would just write the code twice, once for the horizontal case and once for the vertical. In Lisp, we can write such a function for one dimension and then use macros to generate a transposed version of the same function that does exactly the same thing in the other dimension.

<comment><USER12> That's the first use of a macro I can wrap my head around, but it can also be done elegantly without macros - each function that acts on a row or column of data will have a "list of values" argument that could either be a list of column values from a row or a list or row values from a column.

<comment><USER8> Not quite. I wasn't clear enough. These operations are purely UI manipulations. They don't operate on lists of values or even need to know what they are. What they care about are things like: should I set a _width_ property or a _height_ property? To which you might reply: fine, just pass the dimension around and have a _set-size_ function that knows whether to set width or height depending on the dimension, and so on. I say: yeah, you can do it that way, but it takes more code and is harder to read, because now you have to write it in a dimension-agnostic way, which is not how humans (or I at least) like to think about these things. It's easier to write or read the concrete version for one dimension and then say "the other dimension is just like this only transposed", which is exactly what the macro does for us.

Do you need macros to make such a program work? No. Can you use them to make the program shorter, clearer, and more generalizable? Yes.

</comment></comment></comment><comment><USER13> I couldn't live without _iterate_ , a very intuitive, lispy, simple, mostly "backwards compatible" replacement for Common Lisp's clunky, unlispy, ugly _loop_ iteration facility.

<http://common-lisp.net/project/iterate/doc/index.html>

[http://www.lispworks.com/documentation/HyperSpec/Body/06_a.h...](http://www.lispworks.com/documentation/HyperSpec/Body/06_a.htm)

In a language without macros, I'd be stuck with _loop_ and curse every time I need to write some non-trivial iteration code. But thanks to macros, I can use this wonderful iteration facility as a library, and have great integration with CL, just like if it was part of the language.

<comment><USER13> There's also this macro I wrote that I couldn't live without, I call it _fmask_.

Problem: I always hated that _(loop for element in list collect (my-fun element))_ translates trivially to _(mapcar #'my-fun list)_ but _(loop for element in list collect (my-fun element my-constant))_ doesn't. You'd either have to stick with the loop version or use something ugly and inefficient like _(mapcar #'my-fun list (make-list (length list) :initial-element my- constant))_ or write out the lambda: _(mapcar (lambda (element) (my-fun element my-constant)) list)_ (my-constant could actually be an arbitrary form).

None of these solutions appealed to me. So I wrote fmask. With it, the example is simply rewritten as _(mapcar (fmask #'my-fun ? (? my-constant)) list)_

Another example:

      (mapcar (fmask #'list ? (? (1+ (length ?))))
      		 '(3 8 2) '(("some" "items") "A string" (nil "test")))

=>((3 3) (8 9) (2 3))

Here's the implementation:

      (defmacro fmask (function unknown (&rest args))
        (check-type unknown symbol)
        (macrolet ((autotraverse-let (bindings &body body)
    	         (let ((gensyms (iter (repeat (length bindings))
      				    (collect (gensym "TRAVERSED")))))
      		 `(let (,@(iter (for (nil value) in bindings)
      				(for gensym in gensyms)
      				(collect `(,gensym ,value))))
      		    (symbol-macrolet (,@(iter (for (var nil) in bindings)
      					      (for gensym in gensyms)
      					      (collect `(,var (prog1 (car ,gensym)
      								(setf ,gensym (cdr ,gensym)))))))
      		      ,@body)))))
          (labels ((unknownp (arg)
      	       (etypecase arg
      		 (atom (eq arg unknown))
      		 (cons (some #'unknownp arg))))
      	     (literalp (arg)
      	       (or (typep arg '(or array keyword pathname number))
      		   (and (consp arg)
      			(or (and (eq (first arg) 'function)
      				 (cdr arg)
      				 (not (cddr arg)))
      			    (eq (first arg) 'quote))))))
            (let ((unknowns (iter (repeat (count-if #'unknownp args))
      			    (collect (gensym "UNKNOWN"))))
      	    (knowns (iter (for arg in args)
      			  (if (not (or (unknownp arg) (literalp arg)))
      			      (collect (list (gensym "KNOWN") arg)))))
      	    (known-function (and (consp function)
      				 (eq (first function) 'function)
      				 (symbolp (second function))
      				 (second function))))
      	(autotraverse-let ((an-unknown unknowns)
      			   (a-known (mapcar #'first knowns)))
      			  (let* ((unknown-function (gensym "UNKNOWN-FUNCTION"))
      				 (call-function (if known-function
      						    (list known-function)
      						    `(funcall ,unknown-function)))
      				 (main (let ((inner `(lambda (,@unknowns)
      						       (,@call-function
      							,@(iter (for arg in args)
      							     (collect (if (unknownp arg)
      									  (subst an-unknown
      										 unknown
      										 arg)
      									  (if (literalp arg)
      									      arg
      									      a-known))))))))
      					 (if knowns
      					     `(let (,@knowns)
      						,inner)
      					     inner))))
      			    (if known-function
      				main
      				`(let ((,unknown-function ,function))
      				   ,main))))))))

<comment><USER14> Please use Lisppaste to show gorgeous code :-)

<http://paste.lisp.org/new>

<comment><USER13> Good idea!

fmask's implementation with syntax coloring:

<http://paste.lisp.org/display/81488>

I also have a more extended example usage in an annotation at the bottom.

</comment></comment></comment></comment><comment><USER15> You might be interested in what John Wiegley had to say on the subject (in particular, Haskell has no macros; is this bad?) a couple of months ago:

[http://www.newartisans.com/2009/03/hello-haskell-goodbye- lis...](http://www.newartisans.com/2009/03/hello-haskell-goodbye-lisp.html)

The comments below the article also have some interesting and relevant things to say.

<comment><USER16> For somebody with 15 years of Lisp experience he sure managed to mention the most trivial use of macros.

<comment><USER14> Public language changes are usually just trolling for the love of a community; it's like those public "converters" who go from one mainstream, often Abrahamic, religion to another.

The rest of us just rewrite the critical stuff in whatever that's the most efficient, and the non-critical stuff in whatever that's easiest.

Oh screw this! Guys, I just converted from Common Lisp to CSS! The _cascade_ is like OOP inheritance, without the annoying CLOS multiple-inheritance and over-configurable multiple dispatch. It has _keyword_ methods, :hover, :active, etc, I can even implement list processing with getElementById! Take that, you Lisp weenies!

</comment></comment></comment><comment><USER17> Haskell can do almost everything people generally use macros for without them. But there's always some things only macros can do, so for that you have template haskell.

<comment><USER18> As I understand it, if Haskell didn't already have syntax for

      let vars
      in expr

you wouldn't be able to use Template Haskell to create it from scratch. Any domain-specific languages you make have to mimic existing Haskell syntax.

<comment><USER14> Most people think macros are for "compacting" code, or factoring out common parts.

Macros are for introducing new control primitives, syntax and evaluation models into the language.

<comment><USER8> It's both, no? You can always write macroexpanded code by hand, only with much repetition, so in that sense macros are always compacting.

Except, I suppose, for perverse macros which expand into shorter code :)

<comment><USER14> It's for both, but utility functions also serve the same purpose (which is why you see people claiming they could do whatever Lisp macros do with function/methods/classes/monads/templates/makefiles, etc.) :-P

</comment></comment></comment></comment></comment><comment><USER19> Personally, I think compiler macros are far more interesting than 'regular' macros. I would love to see compiler macros even in mundane languages such as python.

Example, in pseudo code:

 defun regex_find(some_string, regex) return regex.compile().find(some_string)

def compilermacro regex_find(some_string, regex) if IS_STRING_LITERAL(some_string) and IS_STRING_LITERAL(regex) SUBSTITUTE exec(regex.compile().find(some_string)) elif IS_STRING_LITERAL(regex) SUBSTITUTE exec(regex.compile()).find(some_string) else DO_NOTHING return

regex_find(MYSTRING, '\w(.+)\b')

A language without compiler macros would compile the regex at runtime, even in those cases that it is a string literal, and could be compiled at compile time. And in the case that both the string and the regex are string literals, the result itself could be computed at compile time. Another alternative, used by lots of languages/libraries, is to maintain a cache of recently compiled regular expressions. Of course, this doesn't help when you have a loop that uses 15 regular expressions but it only caches the last 14, for example.

This is one reason why CL-PPCRE is faster than the C PCRE in benchmarks, it makes great use of compiler macros.

This is a very concrete real world thing, that isn't endlessly debatable, unlike regular macros, and it could be included in languages without any special macro syntax.

One example of a time I _really_ wanted compiler macros in Python was when I was doing some parsing of binary files, using the struct module. The struct module lets you do stuff like struct.decode(data, 'UUUIUH'), where the UUUIUH is some code for Unsigned long, unsigned long, 32bit int, whatever. However, it parses the little code each time you want to use it, which ends up taking way longer than the actual decoding itself. In python 2.x, whatever I was using, they didn't have any way to make any sort of compiled decoder, you had to just use it as above, which is silly, since the code is almost always going to be a string literal and 99% of the work could be done once when the code is compiled.

<comment><USER20> > Another alternative, used by lots of languages/libraries, is to maintain a > cache of recently compiled regular expressions. Of course, this doesn't help > when you have a loop that uses 15 regular expressions but it only caches the > last 14, for example.

Nitpick: it depends on the replacement policy whether it helps. If the cache is MRU, for example, it would work fine.

<comment><USER19> It doesn't matter if each one is used once, say in a loop.

If you have 14 slots in your cache, and you loop over 15 regular expressions, no cache policy (unless it also tracks regexes it has already lost from the cache, which effectively gives it more than 14 slots) will work, except maybe some kind of cache that just remembers the first 14 regexes it sees, and only forgets one randomly every 10 more regexes it sees. Such a cache policy would be really really stupid in most cases not specifically designed to thwomp, say, an LRU cache or a regular deque or something.

</comment></comment></comment><comment><USER21> I used macros a lot in C to initialize structs and arrays with constant information, like keywords for a parser. It would still be useful for languages, like Java, which do not have literals for initializing objects, maps, and arrays. Another use would be DSL's on top of a language like Python. Sure you can do all of this within the languages, but macros would make it more expressive and concise. I really don't buy the argument that we need to be protected from idiots. They will piss in the soup regardless.

<comment><USER14> C "macros" != Lisp macros.

<comment><USER22> Why with the "sarcasm"?

<comment><USER14> Because C macros have no sense of context; they're replaced in the raw character stream while lexer is reading in a file. C macros are mostly stupid string-replacements of the s/foo/bar/g type.

Lisp macros operate on the abstract syntax of the code forms, _after_ it's parsed. They're tree-transforms. They could also be nested, so one macro would operate on the result of another. All of them could also operate on the values read by _reader_ macros, which are Lisp's C-like preprocessor macros: A lisp macro that frobs vectors could, for example, processes code that looks like any of the following: foo[], vector foo, (vector foo), (array 1 ..), etc. if the appropriate reader syntax is defined for them. That's right, ONE macro to do all that.

<comment><USER22> Ah, that makes sense. Thanks for the explanation.

</comment></comment></comment></comment></comment><comment><USER18> Sure, working with code in someone else's domain-specific language is hard. But not as hard as working with a lower-level translation from a domain- specific language which only exists in their head. Even if Clojure were only used for one project in the world, I'm still better off learning Clojure than trying to maintain the JVM bytecode it happens to compile to.

</comment><comment><USER23> Note the reason discussion on programming languages.

[http://gmarceau.qc.ca/blog/2009/05/speed-size-and- dependabil...](http://gmarceau.qc.ca/blog/2009/05/speed-size-and- dependability-of.html)

Lisp and scheme do pretty poorly on the power scale. Even going back to the original data neither finish in the top 10 and Lisp gets beaten by java.</comment>


Launchd chrome css prank - <AUTHOR> https://gist.github.com/nathan/5271225

<comment><AUTHOR> Based on <https://github.com/wesbos/aprilFools.css>.

I wanted a way to have this kick in on it's own on 04/01 without needing root access. You can type 1 line into bash and install the hole thing.

Do this to someone while they are not looking.

Preview [<http://cl.ly/NvQ5>]</comment>


Integrating Amazon Android/iOS Device Farm with Jenkins CI - <AUTHOR> https://packetzoom.com/blog/continuous-integration-with-the-aws-device-farm.html <comment><USER2> I work for a services company and my company deals with atleast 10+ android apps at any point of time. We try to maintain about 20+ office devices plus 20+ employee devices and we could not still feel confident about the device base we are testing. This is very good to know and more good with the post on integrating it with Jenkins. Thanks a lot!

</comment><comment><AUTHOR> It's one of the coolest AWS service. Android on-device testing just got a whole lot cheaper and easier. We use it extensively to test our SDK cross- device before shipping.

</comment><comment><USER3> Really cool. Think they'll start farming some iPhones anytime soon?

</comment><comment><USER4> Thanks for posting, very interesting writeup

</comment><comment><USER5> Thanks Leonid, short and helpful!

</comment><comment><USER6> Nice read.

</comment><comment><USER7> thanks for posting, really helpful guideline.

</comment><comment><USER8> Thanks a lot. It helped me in finding a way to test my application in variety of devices.</comment>


A Case for Upserts - <AUTHOR> http://lucumr.pocoo.org/2014/2/16/a-case-for-upserts/ <comment><USER2>
      > If Postgres would implement a shitty and
      > inefficient version of an upsert statement at
      > the very worst it could be as bad as the current
      > implementation that people write on their own

Wow. please don't do that.

Many people do not understand why an upsert is a hard problem and would blindly use any solution presented without understanding it's limitations. At least with the current situation if a developer finds themselves in need of an upsert they are forced to seek out a solution and in the process understand why it's tricky.

<comment><USER3> You left out a crucial part of that sentence.

        > implementation that people write on their own
        > and then at least, there is an established syntax
        > and a way to improve it further.

That was the point.

Being aware of the problem doesn't mean you'll actually find a good solution. At least this way, any lingering problems will eventually be fixed instead of draining more of your own resources. Chances are, the Postgres team will be able to figure out the subtleties much better than you.

The more domain knowledge your system requires, the worse off everybody is. Here's a story.

At my previous company, I implemented something close to 'upsert'. After hours of researching, implementing, and testing in as many environment I could think of, I wondered why I wasted so many hours trying to get it right. Sure, I now understand the subtleties involved. (Well, I think so anyway. I still don't know what I don't know.) But am I better off now? Is the company? Not really. What we have is a tangled hack.

And worse, I no longer work with this company. So the next guy will have to figure out what's going on when there's an issue. To be safe, I spent another day re-organizing the code for clarity and documenting the complicated workaround. A scrappy syntax shipped today and fixed tomorrow would have been ideal. That way, the new guy could search for the symptom and see that that the current implementation of UPSERT is the cause. They can then use a recommended workaround, or simply upgrade Postgres to fix it entirely.

<comment><USER4> The problem though is that the corner cases of concurrency etc. with upserts need to be hammered out and agreed on at the beginning. Otherwise you will have "well, we improved concurrency handling in upserts and sorry, that broke your application."

Once you have a public API you have a contract, and that contract really shouldn't be broken without good reason. If PostgreSQL offered an upsert syntax with the same limitations as writeable CTE's, and promised to solve the concurrency issues in future releases, I wouldn't use it. I would rather have a hand-written implementation which is guaranteed to work in the future with the same caveats than an implementation I have no control over that may pose backwards-compatibility handling issues if I am supporting multiple Pg versions with my app.

Stability in API's is important. The idea that we will just throw something together now, and then fix it later will result in PostgreSQL getting a bad reputation.

</comment></comment></comment><comment><USER5> I hate this pattern.

People hear, "PostgreSQL is like MySQL except that it is done right." They then go to PostgreSQL and say, "PostgreSQL does not implement my favorite MySQL feature in my favorite MySQL way."

Part of what makes PostgreSQL what it is is that they care so much about getting features right, and making them standards compliant. (Because if you do things differently than the standard, there is a chance that you will find yourself forever maintaining both the way that you did it and the way the standard said to do it.)

I like upserts. I understand the value and the functionality. But please don't add it until you can do it right. And please, please, please make it work like the standard says to. The standard is explicit, flexible, and a standard. Plus if you haven't impressed on the MySQL solution, it really isn't very hard to figure out how to do what you want to do. (I certainly didn't find it hard when I had to do it.)

</comment><comment><USER6> 2226 words and not a single mention of CREATE RULE. Postgres has had something far more general than "upserts" for over a decade.

      CREATE RULE Pages_Upsert AS ON INSERT TO Pages
      WHERE EXISTS (SELECT 1 from Pages P where NEW.Url = P.Url)
      DO INSTEAD
         UPDATE Pages SET LastCrawled = NOW(), Html = NEW.Html WHERE Url = NEW.Url;

<comment><USER7> 1\. Note that there are no plans for postgres to implement a specialised UPSERT. The plan was/is to implement MERGE, which is a fair bit more generic (especially with the 2008 and 2011 extensions). In fact, TFA very specifically disagrees with this route in his conclusion.

2\. IIRC the rule method does not work correctly for multiple-rows inserts including duplicates

<comment><USER8> > 1\. Note that there are no plans for postgres to implement a specialised > UPSERT. The plan was/is to implement MERGE, which is a fair bit more generic > (especially with the 2008 and 2011 extensions). In fact, TFA very > specifically disagrees with this route in his conclusion.

Does the SQL standard actually guarantee that a MERGE is atomic? After all the MERGE statement involves a subselect which seems to be about as concurrency safe as a SQLite replace insert which a join.

<comment><USER9> The MERGE itself is atomic, but it doesn't prevent duplicate insertions without a lock: [http://stackoverflow.com/questions/9871644/is-merge-an- atomi...](http://stackoverflow.com/questions/9871644/is-merge-an-atomic- statement-in-sql2008)

<comment><USER10> If PostgreSQL goes with an implementation of MERGE like that then it would not be a replacement for UPSERT. With UPSERT you can make sure to never get any constraint violations.

</comment></comment></comment></comment></comment><comment><USER11> "The reason for this is that transactions just serialize the execution, they don't guarantee any atomicity of independent row updates. After the delete happens the second transaction gets a chance to run and the update will fail because it no longer sees a row" Umm - I thought everything in a transaction can be treated as atomic wrt to other transactions.. ie they don't see "in between" states?

<comment><AUTHOR> "…it is possible for an updating command to see an inconsistent snapshot: it can see the effects of concurrent updating commands on the same rows it is trying to update, but it does not see effects of those commands on other rows in the database."

[http://www.postgresql.org/docs/current/static/transaction- is...](http://www.postgresql.org/docs/current/static/transaction-iso.html)

<comment><USER12> That only applies for the read committed isolation level. It seems to me that at least serializable should not show inconsistent state in this situation?

<comment><USER8> A row lock is released on a delete.

<comment><USER12> What does that have to do with serializable transactions?

"The Serializable isolation level provides the strictest transaction isolation. This level emulates serial transaction execution for all committed transactions; as if transactions had been executed one after another, serially, rather than concurrently."

If you don't get the right behavior with serializable transactions, as you seem to be claiming, it seems to me that serializable transactions should be considered buggy. In this case they do not provide the guarantees they are claimed to provide.

<comment><USER10> You get the right behavior (a serialization conflict) in PostgreSQL. If you do not in some other database I would consider it to be buggy.

</comment></comment></comment></comment></comment></comment><comment><USER13> I understand that MERGE or something like that would be a nice convenience.

But if I personally had this situation what I would do is (in pseudo code, but think this would be in e. g. Python, or in a stored procedure):

      begin transaction
      does row with unique key x exist?
      if exists: update all relevant fields
      else: insert new row
      commit

How does this have a different concurrency behavior than e. g. an equivalent Oracle MERGE statement or MySQL ON DUPLICATE KEY?

If another process tries to write to this row before the change is committed, it would have to wait (in any case).

If another process tries to read from this row before commit then what happens depends on the transaction isolation level and maybe even on implementation details. But I don't see how this is changed from having a built-in MERGE-like utility.

<comment><USER4> What happens is this: Concurrent transactions marked A and B:

A: Does row exist? Answer no A: Ok, so insert row. A: ....

B: Does row exist? Answer no (A has not committed yet) B: Insert row (locks, waits to see if A commits or rolls back)

A: Commits B: Unique constraint violation error.

Now here's where it gets really tricky: What is the desired behavior? Do we:

1\. Just raise a constraint violation error? That's what we do with LedgerSMB because, frankly, unique constraints are far more likely to be violated by human error than concurrency issues and we can tackle other related errors elsewhere. This is specific to our app. Other apps may differ.

2\. Should the upsert lock, wait, and retry if the the upsert criteria is violated by a pending transaction? That's the cleanest option. However, this requires being aware of _which_ unique indexes are affected by the upsert criteria.

A quick and dirty upsert like yours is good enough for most applications based on human input. It is not good enough, IMO, for applications with lots of direct automation coming in from various sources. So those are very different use cases and they have very different criteria.

<comment><USER13> Thanks for this explanation. I would have expected #1 to happen in the case that you outline.

You're right, since a built in merge utility has more information about the semantical intent of the operation it can essentially repeat the complete operation whereas my naive implementation would not.

</comment></comment></comment><comment><USER14> The UPSERT implementations in this article are simply wrong (he should call them "DELSERT" perhaps).

A simple Postgres function like the following works for me:

      CREATE OR REPLACE FUNCTION upsert_my_table(key text, value text) RETURNS void
      LANGUAGE plpgsql AS $$
      BEGIN
        UPDATE my_table SET value=$2 WHERE key=$1;
        IF NOT FOUND THEN
          INSERT INTO my_table (key,value) VALUES ($1,$2);
        END IF;
        -- add PERFORM pg_sleep(10) for concurrency testing
        END
      $$;

(according to my tests, it has no concurrent UPDATE problems ... duplicate keys are another matter and best handled with restarting transactions / savepoints)

<comment><USER8> > according to my tests, it has no concurrent UPDATE problems

You have concurrency updates with deletes as well as inserts. That's literally the worst way to implement upserts currently. Even CTE's are superior to that.

</comment></comment><comment><USER15> Isn't upsert a bad design smell though?

If you have high concurrent updates, you should be dealing with append-only database design and collapsing into a view to query, like Datomic. All the locking required for atomic upserts would wreak havoc on performance anyway no?

<comment><USER8> > If you have high concurrent updates

When do you not have concurrency?

> you should be dealing with append-only database design and collapsing into a > view to query, like Datomic

True, but postgres is very bad at that. So you need to work with the tools that postgres gives you. Yes postgres currently does not give you an upsert but an upsert is much closer to postgres' design than an append only database that denormalizes on inserts.

<comment><USER15> > When do you not have concurrency?

Highly concurrent _updates_? By that I understand a workload where you have multiple clients updating the same rows.

In my experience it's either rare or the result of doing something inefficiently (e.g., doing analytics as a "view_count" column). Everytime I've faced that I've remodeled the database, both for performance (you avoid invalidating caches for the entire table) and auditing reasons (if you have a bunch of people replacing each other's changes, you might want to retain the history).

</comment></comment></comment><comment><USER16> This article is missing the solution brought by MVCC.

Namely, invalidating any query that comes after the update and asking to redo the transaction. This is done without lock.

<comment><AUTHOR> "Asking to redo the transaction" pushes complexity into the client. Armin did mention that: "Unfortunately this problem does not go away if you tell the client to solve it."

<comment><USER16> Asking to redo the transaction doesn't mean "push the error to client".

In this case, server knows the transaction can pass, so you don't push the error to the client.

The server push errors to the client when:

\- it is sure that the transaction can not pass.

\- it doesn't want to guess whether the transaction can pass.

\- it reached a max retry (because of heavy writes for instance)

<comment><USER8> What you are suggesting is a loop where you try update/insert until it succeeds. Because there is no lock involved there is no guarantee that it will ever finish. That's exactly what's being mentioned in the blog as a workaround until an upsert exists.

<comment><USER17> It is guaranteed to return after either inserting or updating. Most implementors terminate the loop after some arbitrary number because an error elsewhere could prevent both insertion and deletion, and it is nice to get an exception rather than a hang in from your upsert.

<comment><USER8> > It is guaranteed to return after either inserting or updating. Most > implementors terminate the loop after some arbitrary number because an error > elsewhere could prevent both insertion and deletion, and it is nice to get > an exception rather than a hang in from your upsert.

Which is why I said multiple times that this needs database support at which point a lock. That would guarantee that the contention at the lock is fair. With a loop around that section the winner is chosen by network latency and retry speed if high contention happens. Also there is not even a guarantee that it will ever finish.

<comment><USER16> > Which is why I said multiple times

First time read that...

> that this needs database support

Transaction retry is part of mvcc databases

> at which point a lock.

Not necessarily.

> That would guarantee that the contention at the lock is fair.

It's not fair, it's a lock. Locking all the table for one row, is not fair. I don't know what you mean by fair.

> With a loop around that section the winner is chosen by network latency and > retry speed if high contention happens.

Yes and? Anyway, this must be in the database, the database knows the transaction can pass, it doesn't need to send it back to the user, except if there is heavy writes on _this_ row, which would mean the application is under attack or not correctly programmed.

> Also there is not even a guarantee that it will ever finish.

There is a MAX_RETRY, it is guaranteed to return.

\---

Also:

\- You don't state your problem upfront, just say you need this COMMAND. Which is forbidden by netiquette. \- You never state that it's a Write/Write transaction problem \- What about Read/Write? \- There is a solution, use a migration or MYSQL

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> He is incorrect about isolation levels, at least in PostgreSQL. Serializable with DELETE + INSERT will make a concurrent serialiable update fail with serialization error. I just tested this.

        1=# CREATE TABLE my_table (key text primary key, value text not null);
        1=# INSERT INTO my_table VALUES ('key', 'foo');
        1=# BEGIN ISOLATION LEVEL SERIALIZABLE;
        1=# DELETE FROM my_table WHERE key = 'key';
        2=# BEGIN ISOLATION LEVEL SERIALIZABLE;
        2=# UPDATE my_table SET value = 'bar';
        1=# INSERT INTO my_table (key, value) VALUES ('key', 'value');
        1=# COMMIT;
        2 ERROR:  could not serialize access due to concurrent update

<comment><AUTHOR> How is it different from other isolation levels? It looks like it still fails compared to what upsert would do.

<comment><USER10> Other isolation levels will make the concurrent update find zero rows. Serializable gives an error instead which you reliable can retry the transaction on.

They fail in different ways (for serializable a in this case more useful way) which should have been pointed out by the author. It does not seem like he actually tested what isolation levels does or he tested it in a database which does not provide serializable isolation.

<comment><AUTHOR> I think Armin didn't go into isolation levels details because he didn't see a meaningful distinction in the context of emulating upsert.

</comment></comment></comment></comment><comment><USER18> I'm confused. Are we designing accounting software or high concurrency? Upsert is not magic, there are always trade offs.

If this is accounting software then use transactions.

If you're doing a research paper on accounting software then have fun, there are lots of trade offs to play with.

If you are designing for concurrency then drop your constraints and transactions and just make everything an insert. Then when you want to select just look for the first record from the bottom of the table.

No feature, be it upsert or whatever, can ever beat proper design. The best it can ever aspire to do is equal it.

<comment><USER10> How does transactions remove the need for upserts? And things do not get faster if you do not use transactions, in fact they generally get slower.

</comment></comment><comment><USER19> The DBA in me is having a difficult time understanding PostgreSQL's behavior with regard to the delete/insert within a transaction?

In MySQL, I get the exact behavior that I expect with anything above RC isolation level (which is the default for PostgreSQL):

      session 1> create table foo ( k varchar(10) not null, v varchar(10) not null ) ;
      session 1> set transaction isolation level read committed;
      session 2> set transaction isolation level read committed;
      session 1> insert into foo (k,v) values ('a','b');
      session 1> begin;
      session 2> begin;
      session 1> delete from foo where k = 'a';
      session 2> delete from foo where k = 'a';
      -- transaction hangs here as expected since session 1 has a lock on this record
      session 1> insert into foo (k,v) values ('a','c');
      session 1> commit;
      -- session 2 executes the delete
      session 2> insert into foo (k,v) values ('a','e');
      session 2> commit;
      session 2> select * from foo;
      +---+---+
      | k | v |
      +---+---+
      | a | e |
      +---+---+

When I do this in PostgreSQL, I end up with 2 rows in the table, which _seems_ fundamentally incorrect and to me implies that PostgreSQL's implementation of read committed is flawed and should be fixed. The transaction in session 2, at RC isolation, should see the most recently committed result, which is the single record from session 1. I'm not CJ Date, though so am very likely missing some subtlety.

PostgreSQL after the sequence above:

      test=# select * from foo;
       k | v
      ---+---
       a | c
       a | e
      (2 rows)

What am I missing? Why is this considered correct behavior?

<comment><USER20> I'm not an expert, but as I understand it, both deletes see row ('a','b'). Session 2 sees that that row is locked so it waits, and when the lock is released it sees that it's already deleted. However, it doesn't redo the query so it never sees ('a','c'). As far as PostgresSQL is concerned, ('a','c') is a completely different row from ('a','b') and it was created too late to have any effect on session 2.

These transactions aren't serializable but you didn't ask for that.

To get intuitive behavior, a delete followed by an insert with the same primary key in the same transaction would have to be treated as a row update, holding onto a single row lock, rather than as two independent rows with different row locks.

<comment><USER19> This violates the promise of read committed. In RC, all transactions are supposed to see the most recently committed version of a record. The fact that it was a delete, then an insert is an implementation detail and "not my problem" as a DBA. At the time the transaction in session 1 commits, the most recently committed version of k='a' is v='c'. PostgreSQL, in the situation, seems to be operating in read uncommitted isolation level...session 2 is treating the uncommitted delete in session 1 as the most recent version.

I'd like to give PostgreSQL the benefit of the doubt here, but this behavior seems fundamentally incorrect to me.

<comment><USER20> That's the subtle part: they aren't two versions of the same record. They're two entirely different records, so all bets are off. (In your example, they don't even have the same primary key because no primary key is declared. PostgresSQL doesn't know that k is supposed to be a key; it's just another column.)

<comment><USER19> That is an implementation problem. The behavior is still incorrect. MySQL manages to do this correctly, so it's clearly not impossible. I can't come up with any other conclusion except that transaction isolation in pg is fundamentally broken when I see this type of case show up.

</comment></comment></comment></comment></comment><comment><USER21> Before i know about "on duplicate key update" i used to first run an update check the results on how many rows where matched then insert if there was none.

Another way i remember doing was having a unique key and inserting if an exception was thrown the update statement would run update.

Both where slow as hell. Du to the back and forwards stuff required.

<comment><USER22> And worse than that both are subject to concurrency issues.

</comment></comment><comment><USER23> Why doesn't taking a row lock instead of a table lock fix this? Every transaction that wants to read/write to that row id blocks on that lock but you're not blocking the whole table.

<comment><USER16> The row may not exists. OP wants to update a row or create it.

<comment><USER23> You mean that with current semantics if you do a "delete from table where id=X" and X doesn't exist you don't get a lock on that id? So why not just change that or add a "lock my_table in id=X" syntax. Isn't what's needed here fine grained locking by row id?

<comment><USER10> You are correct, and the solutions I have read are based around locking the index entry when doing an upsert. The question is how it is best done. Simplified explanations: one solution is to insert a row and lock the new row, another solution is to lock the index entry itself.

<comment><USER23> Thanks for the answer. I'm sure there's plenty of complexity I'm missing here. It just seems like having an ID-indexed set of mutexes shouldn't be too hard. Does anyone have a good link discussing the design tradeoffs for a performant upsert?

</comment></comment></comment></comment></comment><comment><USER24> I'm a bit confused - can anyone clarify what is the problem in TFA example

      begin;
      delete from my_table where key='key';
      insert into my_table (key, value) values ('key', 'value');
      commit;

? As far as I understand any outside transaction would either see the old value (assuming one existed) or the new value, not the middle result where the row doesn't exist - am I mistaken and this doesn't work that way in Postgresql?

<comment><USER25> I think his point is quite exactly that a concurrent write transaction (starts after the first transaction starts but before it commits) that relies on the existence of that key/value pair will fail because of how Postgres works.

<comment><USER24> OK, I tried this on a random PG server I had and replicated the issue with a concurrent update; and running two such transactions in 'overlapping' way inserted two duplicate entries for the same key.

That is interesting - is the current behavior really considered WAD? I.e., shouldn't then the 'correct' solution be in fixing this behavior instead of extending syntax for 'upserts'?

</comment></comment></comment><comment><USER26> I'm sorry, I only just started learning SQL a few days ago... But I was building a simple web application and I wanted to add a user if the email didn't exist else update that user's fields. And I came across this article: [http://www.the-art-of-web.com/sql/upsert/](http://www.the-art-of- web.com/sql/upsert/)

Is that not an upsert?

<comment><USER27> If you read the whole page, you see that it still requires a full table lock to avoid duplicates. A true upsert would not require a table lock.

</comment></comment><comment><USER28> im not a postgres guy so I don't know how possible this is - couldn't the pythonic "forgiveness not permission" pattern work here?

That is, a proc that just attempts the insert, handles the unique or primary key constraint violation error, and then proceeds to the update as a fallback? Sure you push the concurrency risk to deletion instead of creation, but deletions are generally more rare than creations so a table-lock on deletion is more palatable.

<comment><USER16> posgres is implementing both permission and forgiveness for performance reasons.

<comment><USER16> What is troubling for me, is to have a database that works sometime and is sometime fast. But no database that works all the time and is /slow/.

<comment><USER17> This is a common trouble. It is, however, a structural artifact of data size. You cannot solve this problem with a "code" solution because it is an actual computational complexity issue, not a implementation weakness.

This is why data design is with performance and security as an attribute that you must bake in. You cannot bolt on data design after the fact.

</comment></comment></comment></comment><comment><USER29> I tend to think that UPSERT is the right solution to the wrong problem.

Take the example of a user profile. The problem for me is not typically "the first time the user shows up, I need to create a record, and all other times, I need to overwrite it." In the systems I build, there is almost always a use case for having full access to the history of data. So the better solution is, "I must ALWAYS update whatever record exists AND insert a new record".

That means my UserProfile table also has two extra columns, CreatedOn _and_ InvalidatedOn. Then, creating/updating a user's email address is always the same, two statement operation:

        -- If UserName does not yet exist, this does nothing
        UPDATE UserProfile
        SET InvalidatedOn = NOW()
        WHERE UserName = ?
        AND NOW() BETWEEN CreatedOn AND InvalidatedOn;

        INSERT INTO UserProfile(UserName, Email, <etc.>, CreatedOn, InvalidatedOn)
        VALUES(?, ?, <etc.>, NOW(), MAX_DATE());

In practice I would use default value constraints for the CreatedOn and InvalidatedOn columns. Writing it explicitly is just for this example.

This becomes really important once you've lived with your database for several months/years, and _especially_ when dealing with customers. For example, say you send an important email to a user, then they change their email address, then you send more important emails referring to the first one, which they now claim they never received in the first place. With an UPSERT, that user's email address for all of time looks like the most current version, and you can't figure out why they didn't receive the first one.

It's better to capture more data and realize down the road that you don't need it than it is to not capture enough and realize you need more. Databases are temporal objects. They grow over time and they change over time.

Another example from my past: a client would send us a dump of their data, every morning. We were performing analysis of their data and were supposed to wipe out our database and re-import it every morning. This meant on day N+1 we were re-importing the first N days again. I implemented this as a temporal database, which caught me some hell when the database grew in size. But, after 6 months, we discovered someone had attempted to alter the historical data. If we had blindly taken the data, they would have gotten away with their crime (and it literally was a crime, we had several government regulations we had to fulfill, the project being used in both a health-care and a financial capacity).

I've never regretted implementing my tables temporally. I've sometimes changed the implementation to a data warehouse setup, after learning that the historical data wasn't necessarily important to day-to-day operation, but I've never regretted having the historical data available, and I have almost always learned to regret NOT having historical data.

<comment><USER4> I think you'd need partial unique indexes to make that remotely concurrency safe.

<comment><USER29> I don't think partial unique index would have an impact on concurrency safety. An appropriate one would be useful for making the query for the most recent value faster, but that doesn't mean it would be concurrency-safe without a transaction.

<comment><USER8> How do you implement unique constraints then?

<comment><USER29> The main take-away from my original comment should be "never destroy data". You may write new data, but you may never delete or overwrite data. How you design the database after that is up to you. The particular example I used is of course not going to be good for enforcing, say, uniqueness of email addresses across users.

It has tended to make me really consider the notion of uniqueness within a database. What does the unique constraint mean? I think that people think of uniqueness to mean "at this point in time", but if you enforce it in the database, it means "for all time." In relational algebra, the unique constraint is really about enforcing a candidate key, a field that could satisfactorily be used as a key, but we've so happened to have chosen a different field as the actual primary key. In that context, given that we would want someone to be able to change their email address, I don't think "unique" email addresses make sense.

So many things change over time and so many seemingly "unique" fields turn out to not be unique in the real world that, for the most part, I've given up on the unique constraint. Full names are neither unique, nor are they final (people get married and change their name). Not even Social Security Numbers are necessarily unique, for a variety of reasons ([http://www.computerworld.com/s/article/300161/Not_So_Unique](http://www.computerworld.com/s/article/300161/Not_So_Unique)).

<comment><USER4> That's what partial unique indexes allow you to put in the database. take this example which would work in PostgreSQL:

        CREATE UNIQUE INDEX profile_username_valid_idx_u
        ON profile(username)
        WHERE invalidated_at IS NULL;

The problem with enforcing this in app-space is it doesn't get around the isolation problem. In both cases, you are assuming that the same profile won't get concurrently updated twice by two different transactions. Since only the database can referee this issue, you _must_ enforce the uniqueness there. I.e. this check _must_ happen on transaction commit and your application has no way of knowing if it did.

</comment></comment></comment></comment></comment></comment><comment><USER30> [https://github.com/seamusabshere/upsert](https://github.com/seamusabshere/upsert)

<comment><USER8> Yes. And it's implemented on the client and suffers from contention problems. I already have that :-/</comment></comment>


Google acquires AIMatter, maker of the Fabby computer vision app - <AUTHOR> https://techcrunch.com/2017/08/16/google-acquires-aimatter-maker-of-the-fabby-computer-vision-app/ <comment><USER3> Good for them - machine learning is all the rage these days, but actually turning it into a viable business on your own seems incredibly hard - <USER2> getting data to work on, doing something genuinely useful/valuable with it AND somehow monetizing that is a tall order for a startup (ML generally seems much more suitable as an improvement to an existing product IMO). So my impression is that getting acqui-hired by one of the big players is one of the best outcomes for a team like this - you get a nice signing bonus and get to work at one of the few companies with the scale and access to data to actually make viable products with machine learning.

<comment><USER4> _machine learning is all the rage these days, but actually turning it into a viable business on your own seems incredibly hard - both getting data to work on, doing something genuinely useful /valuable with it AND somehow monetizing that is a tall order for a startup_

Not if you are in manufacturing. (edit: or finance)

</comment></comment><comment><USER5> "from what we understand most of AIMatter's employees will come onboard to Google"

For some reason that i can't fully articulate, and if you want to pedant me on this with a more specific framing then go ahead, but this is very impressive to me and points to the value of the AIMatter team. If 10-30% of people that make it to Google onsites are then approved, what are the odds of finding a group of people who would all pass the process? Quite low and points the value of the team

<comment><USER6> I would agree with this. To imply (as below) that there is some lowered interview bar as part of an acquisition, or other factors, would be, well, un- Googley. Also untrue.

Congrats to us on what looks to be an acquisition of a lot of bright and creative new coworkers.

(Disclosure: I work at Google. I also know nothing about this specific acquisition but will sure mess around with this app tonight.)

<comment><USER7> Do they serve Koolaid at Google by chance?

<comment><USER8> Yeh exactly, they can all pat eachother on the backs about passing the google interview, about joining the elitle club, while modertating youtube comments from there many in-house cafe's.

Why dont tech companies just cut to the chase and start dorms? That way these kids will never have to leave uni, or catch a bus to work again!

<comment><USER9> You joke but there was a time in my life where I would have loved to live in a Google dorm.</comment></comment></comment></comment></comment>


Just Because You Weren't Dancing is No Reason for Us Not to Beat You Up - <AUTHOR> http://www.dailykos.com/story/2014/05/14/1298490/-Just-Because-You-Weren-t-Dancing-is-No-Reason-for-Us-Not-to-Beat-You-Up-and-Then-Throw-You-in-Jail?fb_action_ids=10104551228383710&fb_action_types=og.likes <comment><USER2> Cameras. Every human in any official position of authority of force should be required by law to wear a camera that records at all times video and audio. If the camera ever is disabled for any reason, their testimony for those periods should be considered invalid. The videos recorded should be considered part of the public record.[1]

Law enforcement works for the people, not the other way around. Many need to be reminded of that fact.

[1] Obviously in certain cases this is not desirable; e.g. sting operations. Modern technology is at the point that all faces in a video can be automatically blurred. Individuals who wish their face to be _unblurred_ could specifically request so.

<comment><USER3> Speaking of cameras, doesn't BART have cameras pretty much everywhere on trains and platforms? Surely that would be useful evidence in her defence.</comment></comment>


Eliezer Yudkowsky: That Alien Message - <AUTHOR> http://www.overcomingbias.com/2008/05/faster-than-ein.html <comment><USER2> From the article: We are reasonably certain that our own universe is running as a simulation on such a computer. Humanity decides not to probe for bugs in the simulation; we wouldn't want to shut ourselves down accidentally.

That puts a new twist on the anthropomorthic principle. The only universes that are observed are the ones that have observers - and haven't crashed their universe.

</comment><comment><USER3> Except the universe is probably nothing like we imagine. This story is the modern-day equivalent of tales of sea monsters and sailing to the edge of the flat world. What is the probability that evolution would result in a "life form" anything like us? And by anything like us, I mean anything which can understand any of the concepts we understand? That includes all of the theories of mathematics we impress ourselves so with.

<comment><USER4> "What is the probability that evolution would result in a "life form" anything like us? And by anything like us, I mean anything which can understand any of the concepts we understand?"

Pretty good, in this universe at least. Every intelligent creature in this universe has to be able to do the same basic things: reproduce, manipulate objects, fend off predators, etc.

There's a theory, commonly used in economics and international relations, that competitive pressures force most groups at the apex of a system to think, act, and behave pretty much the same. For example, during the Cold War, the Soviet Union and the United States, despite their radically different concepts of how to run a government, each behaved roughly the same on the international stage. They built coalitions and acted pretty much universally in self-interest.

The same principle probably applies to organisms. The same competitive pressures probably force intelligent organisms to develop similar mental frameworks.

This isn't to say that ET will look anything like us. Just that he'll probably understand patterns, numbers, and the like, because understanding those things are what this universe rewards.

<comment><USER3> _Pretty good, in this universe at least. Every intelligent creature in this universe has to be able to do the same basic things: reproduce, manipulate objects, fend off predators, etc._

That's a very small scope of 'basic things' - it very well might be that it only applies to the results of evolution on earth. What about evolution in our universe on scales that aren't our size, that don't move at our speeds, that aren't made up of carbon/oxygen/etc?

What is there to stop an evolutionary processes from happening inside a star? It doesn't have to be our star, it could be a red giant - with entire complex evolutionary "life" constantly being created and destroyed in a matters of seconds - to them, the star is the accessible universe - it forms the fundamental fabric of their existence.

What is there to stop evolutionary processes to happen on a quantum level?

Most things in the universe don't happen at our speeds, and they don't happen at our size. And get really not-like-this-universe when you start thinking about relatively big things, or relatively little things.

<comment><USER4> I think this is a good point. The number of possibilities is too great to imagine, and there very well could be some type of life I simply can't envision.

"What is there to stop evolutionary processes to happen on a quantum level?"

There are entropy constraints on the conditions under which life can evolve. There must be a persistent storage mechanism that degrades at a certain rate. Too fast, and the mutations pile up and the organisms all die. Too slow, and evolution can't operate. I suspect that most possible life would have to rely upon carbon-oxygen or something similar.

</comment></comment></comment></comment><comment><USER5> "No one - not even a Bayesian superintelligence - will ever come remotely close to making efficient use of their sensory information..."

We can settle that now...

[PDF] Physical limits to computation: [http://puhep1.princeton.edu/~mcdonald/examples/QM/lloyd_natu...](http://puhep1.princeton.edu/~mcdonald/examples/QM/lloyd_nature_406_1047_00.pdf)

Eye: [http://www.eurekalert.org/pub_releases/2006-07/uops- prc07260...](http://www.eurekalert.org/pub_releases/2006-07/uops- prc072606.php)

So if the eye does 10 million bits/sec & the 1 kg ultimate computer does 10^51 oper/sec on 10^31 bits, I think there's a good chance a Bayesian superintelligence can do it.

That computer may not be buildable, & you might have instruments with somewhat more bits/sec, but there's a LOT of room to work with.

<comment><USER6> Isn't there a declining marginal value to the amount of sensory information available and the amount of processing you do on it?

Information and calculation beyond a certain point may simply becoming "boring."

<comment><USER7> Fair point, GavinB. If you defined "efficiency" in terms of achieving the most benefit from an eyeball using the fewest computations, then I'm not sure whether a Bayesian superintelligence could approach "efficiency". Maybe the theoretically optimal program to run on the eyeball's input, would itself require exponentially vast brainpower to calculate!

But I would concede a much higher chance that a Bayesian superintelligence could get bored at exactly the right time, than that it could simulate all possible universes.

<comment><USER8> Do you think it would be a "good" thing if the average intelligence level was bolstered by 40 points? Can you think of any ill side effects from this possibility?

<comment><USER9> increased existential depression...

</comment></comment></comment></comment></comment><comment><USER10> It seems like a sufficiently intelligent intelligence will eventually "escape" all the way up to the "real world". From this example, if the Einsteinians had invented an AI without having the hardware spontaneously melt, it would have escaped into the Einsteinian world. It then would have moved up into the "undergrad world", since it would be smarter than the Einsteinians and they were capable of "moving up".

I can't quite put my finger on it but something seems wrong about this...something mind bending is going on here.

<comment><USER11> You should read Greg Egan.

</comment></comment><comment><USER12> Man, the kind of stuff I read on overcomingbias makes everyday life feel vapid and onerous.

</comment><comment><USER13> \- Shannon limit.

\- Genius: perspiration vs inspiration and how much genetics alone can affect this.

\- Relativity: if we're traveling near c that makes us less intelligent because we think more "slowly"?

\- Tractability: here comes the heat death of the universe.

</comment><comment><USER14> This is mind-bending. Good read, thank you!</comment>


Ask HN: Hacker News Top-Entry Archiving? - <AUTHOR> Is anyone aware of a HN archiver that saves top N entries within an X-hour window?<p>I don't want to miss top news but I also don't need to know about it the day-of.
Before I write something of my own, I thought I'd ask about it and invite like-minded, busy people who want to balance media consumption with weekday responsibilities. <comment><USER2> Algolia's HN search allows you to search stories by popularity in a given time window: [https://hn.algolia.com/?query=&sort=byPopularity&prefix&page...](https://hn.algolia.com/?query=&sort=byPopularity&prefix&page=0&dateRange=custom&type=story&dateStart=1489104000&dateEnd=1489190400)

</comment><comment><USER3> This site does what you want: [http://hckrnews.com](http://hckrnews.com)</comment>


A Brief Glance at How Various Text Editors Manage Their Textual Data (2015) - <AUTHOR> https://ecc-comp.blogspot.com/2015/05/a-brief-glance-at-how-5-text-editors.html <comment><USER2> _This is very efficient, but the consequence is Vi will become slow when working with huge files, because it has to traverse through a bigger linear array_

To put "bigger" into perspective, a modern CPU can traverse and copy memory at over 10GB/s.

This is why I've always found the argument for more "efficient" (and complex) text editor data structures a bit tenuous --- even if you have to move MBs of data with every insertion (as happens with a simple gapless buffer), computers truly are so fast that it wouldn't look any different to the end-user; a 1ms and 1us delay upon each keystroke is, to the user, practically indistinguishable.

That's not to say I'm one of those who preach against "premature optimisation" and don't care about efficiency; far from that, in fact, but the popularity of and lack of speed-related complaints against the small DOS text editors and even syntax-highlighting IDEs on PCs in the 80s through early 90s which used the same "one buffer" paradigm, on machines with a fraction of the memory bandwidth of those today, suggest that the complexity of more "clever" data structures may not be worth it.

...and yet we somehow still manage to make editors that peg a single CPU core just blinking a cursor.[1]

[1] [https://news.ycombinator.com/item?id=13940014](https://news.ycombinator.com/item?id=13940014)

<comment><USER3> > computers truly are so fast that it wouldn't look any different to the end- > user; a 1ms and 1us delay upon each keystroke is, to the user, practically > indistinguishable.

Unless, of course, the 1 ms operation drains the users battery 1000x faster than the 1 us operation.

Efficiency is no longer about speed, it's about noise, temperature, and battery life. Speed is a happy side effect.

</comment></comment><comment><USER4> If you've never interacted with the file saved state of an editor which saves its edit history tree, its a revelation when you try the UNIX _strings_ command on one. It took me a few minutes to realise insert and moves were _not_ the same, because the structure followed insert order, to permit infinite-undo. The actual text position was a subsidiary location tagged on the string.

I think everyone should be asked to write their own simple editor, and then for extension try to convert from the obvious simple tropes in arrays to one which begins to do what these editors do.

<comment><USER5> Or you could implement a gap buffer, and then your complicated editor is using a simple trope.

Emacs is an example of a full-featured editor based on a gap buffer.

<comment><USER4> Sure. You could leap to the best solution first. Probably, you lucked out in the gene pool and your future fate is not to write bad code. The rest of us, who did not acquire sufficiently complex frontal lobes, struggle with concepts and probably dive into stupider solutions first, like I did at uni in the 1980s allocating a fixed size buffer (big initial mistake)_ and maintaining a huge linked list of edits sequence events into it.

I passed the course, but future evidence suggests I didn't learn as much as I hoped!

<comment><USER5> Ha ha, well, I'll confess, I got lucky and discovered the existence of gap buffers before I tried to write a simple editor. And these days, there's a Wikipedia article on quite a few of the fundamental editor data structures.

It's worth noting that no widely-used editor since emacs uses a gap buffer, so it's not like it's won the hearts and minds of implementers.

</comment></comment></comment></comment><comment><USER6> > [Sam] is one of the first editors to separate its UI from the actual editor > - Sam can be used on both the command-line and as a graphical text editor.

Not even close to the first. TECO had both command-line and graphical editing in, ah, 1964: [https://en.wikipedia.org/wiki/TECO_(text_editor)#History](https://en.wikipedia.org/wiki/TECO_\(text_editor\)#History)

<comment><USER7> There were different implementations of TECO (written in the assembly languages of different machines) that had different user interfaces. That does not in any way imply that TECO had an editor process using a defined protocol to communicate with its UI process; in fact, in 1964, TECO ran on two computers, and neither of them had separate processes, or an operating system to run them under. ITS development hadn't started yet.

As far as I know, all the implementations of TECO, even today, glom the screen update logic (if any!) together with the editor-buffer logic in a single monolithic process.

</comment></comment><comment><USER8> Does anyone know what the new text editor Xi [0] uses? What about the GNU Zile [1] library?

[0]: [https://github.com/google/xi-editor](https://github.com/google/xi- editor) [1]: [https://www.gnu.org/software/zile/](https://www.gnu.org/software/zile/)

<comment><USER9> From the previous discussion [1] on HN:

I use ropes in xi editor. I did not find the argument against ropes convincing. Yes, they're not trivial to implement, but in a proper programming language you're not dealing with the data structure directly, you're always going through the interface, so you get the logic right once in the rope library implementation and then forget about it.

In a low-level design, your editing operations would be poking at the data structure directly. There, the simplicity of a gap buffer is a pretty big win. I agree in this environment ropes are too complicated. However, I don't see any good reason to architect a text editor in this way. Use abstractions.

The linked article contains a factual error, the referenced Crowley paper does not consider ropes. Thus it cannot be used in support of the argument that piece tables outperform ropes.

There's one other important concern with piece tables I didn't see addressed. It depends on the file contents on disk not changing. If your file system supported locking or the ability to get a read-only snapshot, this would be fine, but in practice most don't. It's very common, say, to checkout a different git branch while the file is open in the editor. Thus, the editor must store its own copy to avoid corruption. In the long term, I would like to see this solved by offering read-only access to files, but that's a deeper change that can be made piecewise.

[1]: [https://news.ycombinator.com/item?id=15383193](https://news.ycombinator.com/item?id=15383193)

</comment></comment><comment><USER10> > I'm surprised programmers haven't created overlays for Vi or Emacs for the > GtkTextView. Definitely an opportunity for someone there.

[https://github.com/polachok/gtk-vikb](https://github.com/polachok/gtk-vikb) already was a couple of years old by the time of writing this. (Not sure about the current status of this library though, especially considering commit history and no mentions of GTK3 whatsoever.)

</comment><comment><USER11> Previous discussion: [https://news.ycombinator.com/item?id=15381886](https://news.ycombinator.com/item?id=15381886)

</comment><comment><USER12> MicroEmacs, a free text editor from the 1980s, uses a double-linked list of lines. It is fast and memory compact, after all, it worked on a 64K DOS machine.

[https://github.com/DigitalMars/med](https://github.com/DigitalMars/med)

<comment><USER13> In that memory regime (and without virtual memory), code size matters. With a simple approach, Your data structures may take 10% more memory, but if its code is a few k smaller, it's a net win even if you exhaust all memory.

Also, in that tight memory, people would sometimes choose slower approaches if they had lower memory usage. For example, for each line, do you use a gap buffer, or simply move every line? The former may be slightly more efficient, but eats extra memory.

</comment></comment><comment><USER14> > You get three choices, and may choose two:

seems a completely unjustified claim. These features aren't mutually exclusive.

<comment><USER15> I think the claim is that this is true in practice (among the surveyed options) not that it is true in principle.

</comment></comment><comment><USER16> In the comment sections the author claims that Sublime text uses a fork of GtkTextView. Is this true?

Here's the relevant bit:

> Open Sublime in a hex editor, there are gtk debug strings everywhere, along > with gtktextview/edit strings too. It's as "home grown" as a fork can get > really. Some searching should bring up more evidence.

</comment><comment><USER17> Discussed in 2016: [https://news.ycombinator.com/item?id=11244103](https://news.ycombinator.com/item?id=11244103)

</comment><comment><USER18> See also chapter 6 of _The Craft of Text Editing_ by Craig A. Finseth:

[https://www.finseth.com/craft/#c6](https://www.finseth.com/craft/#c6)

</comment><comment><USER19> damn my blog is popular today

any questions let me know

</comment><comment><USER20> How does Neovim do in comparison to Vim with regard to these metrics? Neovim has to look a bit different here because it's async and Vim is not.

<comment><USER21> IIRC Vim 8 now has async features.

<comment><USER20> You're right, Vim 8 does async. I'd still like to know how Vim compares to Neovim according to these metrics.

</comment></comment></comment><comment><USER22> Emacs has a text editor!?

<comment><USER23> It's called evil-mode

<comment><USER24> Hehe, you two made my day! ;-)

</comment></comment></comment><comment><USER25> (2015)

<comment><USER17> Thanks! Added.

</comment></comment><comment><USER26> > You get three choices

I dont agree the you should judge by these criteria.

\- program size: Who cares? A somewhat decent laptop has 500GB+ SSD storage, I dont care if my editor is 5MB or 50.

\- virtual efficiency (RAM usage, and again disk space??): Again who cares if it uses 5MB or 500 RAM? Dev machines have at least 8GB

I care a lot more about stuff like typing lag or autocomplete support.

<comment><USER27> 8GB seems like a lot until you try to run Eclipse and chrome with 3 tabs open.

<comment><USER28> While I don't disagree that 8GB isn't much for a developer machine these days I do also think your claim is a little exaggerated. I certainly don't have any issues running a couple of intelliJ instances + Chrome with _at least_ a dozen tabs open (seriously; I'm terrible for leaving tabs open).

What I find kills my machine is trying to run Chrome and Firefox concurrently. Particularly if I still have IntelliJ open.

<comment><USER29> >with at least a dozen tabs open (seriously; I'm terrible for leaving tabs open).

Come back when you have 150 tabs open. Anyways, somewhere around 200 across 4 windows is where i start seeing about 10GB ram used, with os and other apps.

RAM never seems to be much of an issue unless im abusing things; its cpu/disk that kills me, particularly when some background process goes out of control.

<comment><USER30> I'm horrible about tabs too. But I mostly use it as a "I'll use this page again in a moment". Often that's true, but often I also just end up leaving it around.

My solution was the OneTab extension for Chrome. It gives you a button to click to close all tabs (with some ways of marking exceptions), and adds them to a page, so you won't lose them. It groups it by date, from newest to oldest. It makes it easy to come back to the things I left in tabs "in case", while keeping the tab count down.

In effect I find I come back to quite few of them, but knowing I _can_ makes it a lot easier mentally to press that button.

(And now I'm off to do just that before doing some work)</comment></comment></comment></comment></comment>


Shapeoko forums and wiki hacked – obfuscated PHP found, we're trying to reverse - <AUTHOR> http://www.shapeoko.com/forum/viewtopic.php?f=7&t=7659 <comment><AUTHOR> Is anyone around with experience reversing obfuscated PHP code?

We're trying to figure out what the code implanted by some hacker types in our forum pages does:

      http://www.shapeoko.com/forum/viewtopic.php?f=7&t=7659

Initial reversing of code here by jacob32123, one of our Community members:

      http://pastebin.com/U6qwqhSX
      (line 221+ has more decoded info)

If people are around with interest in this kind of thing, and time to assist... it would be really helpful. :)

      http://www.shapeoko.com/forum/viewtopic.php?f=7&t=7659&start=10#p60550</comment>


Man excavates his basement using nothing but RC construction toys - <AUTHOR> https://mashable.com/video/man-excavates-basement-rc-construction-toys/ <comment><USER2> If someone could make these autonomous, I'd be fine waiting 3 years for my basement if it cost 1/4 as much.

<comment><USER3> I don't think you'd need to. Just expose controls and camera to the internet, and you'll be able to charge people for the privilege of digging you a basement.

<comment> java-man that's a capital idea for a product!

<comment><USER4> tomsawyerjobs.com

</comment></comment></comment></comment><comment><USER5> So, is he just using is basement as a playground, or is he actually completing a project? I'm a little confused after the dump trunk appears to just unload the dirt into another hole on the other side of the stairs (at about the 2:30 mark in the video).</comment>


Scandinavian design porn of the highest order - <AUTHOR> http://boingboing.net/2015/03/27/the-beauty-of-scandinavian-des.html

<comment><USER2> I'm with the submitter's line of thinking here. What those plastic containers are doing in the third image can only be called _ménage à quatre_.</comment>


Japan's VR Girlfriend Is Coming to Creep Us Out in the Real World - <AUTHOR> https://augmented.reality.news/news/japans-vr-girlfriend-is-coming-augmented-reality-creep-us-out-real-world-0183060/ <comment><USER2> This should never surprise anyone. Japan's current generation of 20 something men are so focus on work and gaming that they don't want to deal with complex relations. Whats better then a girlfriend you can launch into the real world and the stow her away when you're done. Its like the adult version of Pixel Pinkie.</comment>