Ask HN: Can I have heterogenous lists in a Lisp while preserving type inference? - <AUTHOR> Trying to see if I can implement type inference on a toy Lisp, but can't figure out what the type of (car '(x y z)) should be in compile time. For those with experience in dealing with types and Lisp, can I keep these heterogenous lists and Hindley-Milner's algorithm W at the same time - or will they naturally run at odds (which is what I'm thinking)? <comment><USER2> I think you need to run dataflow analysis. There is a paper I read which explains the approach, but I lost the link. It describes how to propagate types to optimize code, but the exact same method can be used to type check code too.

Basically the compiler annotates each value it deals with with what it knows about it. So in your example:

        (sqrt (car '(1 2 "hello"))

First '(1 2 "hello") what do we know about it? The value is of type list, the elements types are: integer, integer and string. Since it is a literal, we even know the values: 1, 2, "hello".

We record it somehow. Next we apply car to that value. What do we know about the resulting value? If car was a random function, we wouldn't know anything about the result. But car is not random. It is one of the most common Lisp functions so we have custom code for this "known" function that says that its value is the first item of its first parameter.

Therefore the type of (car '(1 2 "hello") is integer and the value is 1. Since the type of sqrt is an integer N >= 0 (let's ignore negative roots), the expression (sqrt (car '(1 2 "hello")) type checks! And if we have annotated car and sqrt as side-effect free we can fold the whole expression to 1.

Note that if the expression was (sqrt (car '(x y z)) where x, y, z are unknown variables then it would be a "maybe". We can't know if the code type checks or not. Note also that if the expression was (sqrt (car '(-1234 y z)), then the code would _not_ type check as (sqrt -1234) has no type (ignoring complex numbers here). That is strictly more powerful than ordinary type systems which can't determine that (sqrt -1234) is a programming error.

<comment><AUTHOR> I just saw this - but funnily enough, this is how I handled it. Thanks man - appreciate the comment!

</comment></comment><comment><USER3> You want a statically typed Lisp dialect?

Why don't you just declare your list, say, x, as having type (List Any) where Any is the most general type? (car x) would then have type Any at compile time (i.e you don't know its type), so you can't take its square root or cons it onto a list of type (List Int), but you could still print it or cons it onto another list of type (List Any).

(List Any) then has to be distinct from (List ?x) where ?x is an undetermined type. If y is of type (List ?x) and your function contains (sqrt (car y)), you can infer that ?x is a numeric type and y is a list containing only that type of elements.

<comment><AUTHOR> So let's try an experiment:

(sqrt (car '(1 2 "hello"))

How would this work according to you, if the car function will return type Any? And what is the difference between ?x and Any?

Thanks by the way!

<comment><USER3> If that's typed into the listener, it should probably just be interpreted and dynamically typed. If it's an expression in the process of being compiled, it can be simplified to (sqrt 1) and thence to 1.

car should return the type of the elements of the list it is supplied with, whatever that is. E.g. if x has type (List (List Int)), (car x) should return (List Int). I did this for Full Metal Jacket (see [http://web.onetel.com/~hibou/fmj/interpreter.pdf](http://web.onetel.com/~hibou/fmj/interpreter.pdf), section 6).

If something has a variable type (e.g. declared as being of type ?x) its type can be inferred once known. If it's of type Any, its type cannot be inferred more specifically than that. If, instead of the above example, you declared x of type (List Any), the list '(1 2 "hello") could still be bound to it, and the call to car would still be accepted by the compiler, but not the call to sqrt.

<comment><AUTHOR> > If, instead of the above example, you declared x of type (List Any), the > list '(1 2 "hello") could still be bound to it, and the call to car would > still be accepted by the compiler, but not the call to sqrt.

(sqrt (car '(1 2 "hello"))

So in this example, since it's a (List Any), in order to get this to work would we cast the Any to be an Int to then put it through sqrt?

(sqrt (int (car '(1 2 "hello")))

Also checked out out FMJ - super interesting. I do like the way you used Prolog's unification as way to describe successful type matching. How far along is your work with FMJ?

<comment><USER3> > So in this example, since it's a (List Any), in order to get this to work > would we cast the Any to be an Int to then put it through sqrt?

I wouldn't recommend you use casting the way it's done in C. It defeats the purpose of having a strong type system if you can change the types like that. Having a function which checks the type and gives an error if it's not the type you expect is, I think, a better approach.

I'm still actively developing Full Metal Jacket. Glad you find it interesting. An interpreter for the basic language is almost complete. I'll soon add to my tutorials ([http://web.onetel.com/~hibou/fmj/tutorials/TOC.html](http://web.onetel.com/~hibou/fmj/tutorials/TOC.html)) how to do inlining and substitution macros, and how to define new types. Then I'll make my code more robust, possibly add a compiler, and arrange a release on a shared-source licence.

<comment><AUTHOR> Oh yeah, my mistake - you're totally right. So in this case what type does car output (or how do I write the type definition for car), because it can't output multiple types otherwise I wouldn't be able to check its value at compile time?

And yeah, Full Metal Jacket seemed cool (and it's a great name). Do post it here when you start putting those tutorials up!

</comment></comment></comment></comment></comment></comment><comment><USER4> It depends on the implementation, but even if you have proper type inference, I don't understand how it would help with a heterogenous list, since it's the heterogenous nature that will prevent optimization anyway, even if you knew exactly what the types were.

Most lisps, and certainly common lisp, do not depend on type inference for correct behavior in any way, as they are fundamentally strongly but dynamically typed (like Python). The reason for type inference in Lisps is generally for optimization, where knowing that an array is of type X lets you compile X-specific code for functions that take arrays of type X, and to allow efficient memory layout, where your array is made up of contiguous X objects, rather than storing an array of pointers to X objects, and all the cache missing and so on that this causes (as well as overhead of allocating space for the pointers). Just for completeness, I am saying that in the absence of type information, arrays are always actually arrays of pointers.

So lists are always made up of cons cells which contain pointers, so one optimization is out regardless (it is possible that primative types like int or float would be packed into the cons cell itself, but most implementations would achieve this by some kind of bit tagging rather than type information being 'known'). The other optimization is out because it would likely be equivalent to do dispatch via the usual method for untyped objects rather than trying to do somehow use the type information.

<comment><AUTHOR> Thanks man, this is great.

> since it's the heterogeneous nature that will prevent optimization anyway

That was the part I was wondering about. I wanted to see if it was possible to do compile time type checking on a list where I don't know what element would be inside.

It was less for optimization and more for experimentation. Like I was trying to see if I approached building a Lisp with OCaml type checking - how much of the "Lisp" would I have to trade to get proper HM type inference. No purpose other than for play.

Then, although I assumed it was naturally impossible, figured I'd ask.

So car can work for type cons, and then that returns the element regardless of type since the first part of the cons that is pulled becomes just the element?

</comment></comment><comment><USER5> I'm not exactly sure. I think they may be at odds. Scala can handle AnyTypes in lists, but I think you have to decide whether you want compile time type checking or run time dynamic type checking as from my knowledge you can't have heterogenous lists in languages like OCaml (and from the little I know of Haskell, Haskell as well).

<comment><AUTHOR> Hm, that's what I'm thinking as well. Scala's Any list is probably not the same, but I think with Haskell you can do it if you make a union of types. I just want to see if there are ways to allow type inference while keeping the list flexible and heterogenous as otherwise the Lisp loses power if I need to use functions like (car-int '(1 2 3)) to get an int. Do you know how pattern matching works in the case of matching cars to their respective types.

<comment><USER6> A sum type (union type) solves the problem by making the list homogeneous. A `car` against such a list would return a member typed as this sum type and it would then need to be extracted (e.g., with pattern matching) to get at the value.

</comment></comment></comment><comment><USER7> > _can 't figure out what the type of (car '(x y z))_

What? (quote (x y z)) is a constant expression denoting the value (x y z). We know statically that its car is x, of symbol type.

<comment><AUTHOR> > We know statically that its car is x, of symbol type.

I'm saying within a statically typed Lisp or when trying to use HM's algorithm w you don't know what type of car of (x y z). The type of "symbol" is not specific enough as what operations will you allow to be run on symbol if you're working in a statically typed Lisp with type inference.

If symbol is a character you can't run a float addition on it, and any attempt to do so should throw an error at compile time.

In the case above, I'm using x, y, and z as substitutes for characters, floating point numbers, integers, etc. I'm trying to see if I can figure out the specific type in compile time rather than runtime.

<comment><USER7> If we know that L is a list, and that is all we know, we don't know what type (car L) is.

If we know that L holds the value (X Y Z), then we know that the type of (car L) is symbol.

> _The type of "symbol" is not specific enough _

It absolutely is specific enough. It's a fundamental type in the language.

We just don't know what kind of symbol for certain purposes. If we _only_ know that some object O is a symbol, we don't know whether whether it is self- evaluating (nil, t, keyword symbols in CL) or bindable (any other symbol).

In _this particular case_ , not only do we know that (car L) is of type symbol, we know _which symbol_!!!

We know the exact _identity_ of the object: it is X.

If we know this at compile time, because L is a quoted list literal, we have the strongest possible information about what it is, even more precise than most type information.

> _as what operations will you allow to be run on symbol if you 're working in > a statically typed Lisp with type inference_

If I know that O is a symbol, then we can, for instance, compile the (symbol- name O) operation such that O's type tag isn't checked; however, if there is a special representation for a symbol nil, we have to distinguish that. In this case we know that X isn't nil.

> _If symbol is a character_

... then you're talking about a completely different language that isn't one of the mainstream Lisps. At best a very oddball dialect.

What Lisp tutorial have you been following?

Perhaps you have confused symbol objects, and the use of a symbol as a variable name (X having a value binding in a given environment, and the value of that binding being a character object).

<comment><AUTHOR> Oh perhaps I'm not being specific enough. Apologies mate.

> If we know this at compile time, because L is a quoted list literal, we have > the strongest possible information about what it is, even more precise than > most type information.

By '(x y z) I'm assuming that it's not a quoted list literal but a let's say list generated by a function at compile time. Where x, y, and z could be of different types like ('(2.5 7) 6 "hello"). It's very hypothetical, and the contents of the list generated cannot possibly be known at compile time.

> ... then you're talking about a completely different language that isn't one > of the mainstream Lisps. At best a very oddball dialect.

And yes, what I'm talking about is building a toy lisp between an ML language and a Lisp to see how much Lispyness you have to trade off for the static typing and the type inference of an ML language. The definition of oddball!

In OCaml for example which has static typing and type inference you can't just have a "symbol". Even though symbols are a basic part of Lisp, you can't have really strong type inference for a "symbol" despite it being a fundamental type in the language as for strong type inference you would want to do stuff like have separate operators for adding integers and adding floating point numbers. This is very important for HM's algorithm w, similarly - in that case, car needs to be able to take one type, and return one type for it to comply with algorithm w. I'm trying to see if other people had ever tried implementing something like this.

Anyway, great conversation! I think DonaldFisk helped me out already, but thanks!

</comment></comment></comment></comment><comment><USER8> Perhaps it could infer statically typed tuples from heterogeneous types and then apply list semantics to the operations? If the default is immutability, then all lists can be treated that way heterogeneous or not.

SMLNJ and other Hindley-Milner based type inferring languages are good at inferring tuples at compile time so there's probably a little prior art.

</comment><comment><USER9> could be something like this (with some sugar)?

        data Cons a b = Nil | ConsCell a b

        cons = ConsCell

        car Nil = error
        car (ConsCell a b) = a

        cdr Nil = error
        cdr (ConsCell a b) = b

        mylist = cons 'a' $ cons True $ cons "hello" Nil

        mylist :: Cons Char (Cons Bool (Cons [Char] (Cons a b)))

</comment><comment><USER10> AIUI you can't have HLists in languages without higher-kinded types, and H-M becomes incomplete in that case.

</comment><comment><USER11> The actual elements of the lists are typeless basically. The easiest way to do this is that each car of a cons cell is just a pointer to a value (actually both are). Many lisps use tagged pointers where the tag stores the value or even each variable could be a struct with the type and value.

</comment><comment><USER12> Would this perhaps be a better question for StackOverflow?

<comment><USER13> I hope not. This place is already more like Noob News than Hacker News, and the transition has been very gradual.

That was a bit harsh. I'm saying we should try hard to retain the culture of the site.

It was delightful to see this question pop up on the front page. And of course, what was the top comment but a reminder that the old HN is now firmly "the old HN," rather than "the present HN."

<comment><USER12> Ah, to be clear I'm not opposed to this post. I guess it's a combination of a) surprise at seeing this type of technical question here, and b) being used to this sort of question on StackOverflow. I feel like at least for this type of question in particular, StackOverflow is better equipped to produce and sort out a good answer. But I can certainly see why it might fit on both.

<comment><AUTHOR> Just checked back on this post from yesterday. The reason I didn't pose it to SO is that 1. I still do associate HN with a strong Lisp community. 2. It's more of a question of opinion versus a technical question itself and figured the discussion could be interesting for the site. I also didn't give too much thought as to where I was posting it.</comment></comment></comment></comment>


Thousands of rogue restaurant websites diverting customers to OrderAhead - <AUTHOR> http://www.geekwire.com/2015/exclusive-thousands-of-rogue-restaurant-websites-diverting-customers-to-orderahead-deliveries/

<comment><USER2> OrderAhead is a YC company (Winter 2011): [http://blog.ycombinator.com/orderahead- yc-w11-nabs-23m-seed-...](http://blog.ycombinator.com/orderahead- yc-w11-nabs-23m-seed-for-the-faste)

And yes, I can only imagine they're going to get sued very, very hard.

</comment><comment><USER3> There's a similar group in Sydney, Australia doing a very similar thing, creating duplicate restaurant websites and taking a fee in-between.

<comment><USER4> What's it called?

</comment></comment><comment><USER5> They're just disruptively growth hacking inefficiencies in the restaurant space by website seo arbitrage.

</comment><comment><USER6> Sleazy. This has lawsuit written all over it</comment>


GNU sed 4.3 released - <AUTHOR> http://lists.gnu.org/archive/html/info-gnu/2017-01/msg00000.html <comment><AUTHOR>
      ** Improvements
        sed's regular expression matching is now typically 10x faster

        sed now uses unlocked-io where available, resulting in faster I/O
        operations.

</comment><comment><USER2> Fixes a bug introduced between 1998 and 2002 (between the last 3.x and first 4.x). Someone somewhere presumably has written a sed script working around the bug for 15 years!</comment>


Poll: would you visit a "thinkerspace?" - <AUTHOR> http://begriffs.wufoo.com/forms/is-this-a-good-idea/

<comment><USER2> The idea needs a lot of development - and a name better than a 'thinkerspace'.

Math and music? What about philosophy, literature, other branches of science? I can think and write with my flouncy fountain pen at home, so what are the merits of me going to a collaborative space? I'm sure there could be many, but these aren't elaborated upon at all. Conversation? About what? Will it be structured? Themed? How? Or why not?

Tea but no coffee? Bookbinding but no typewriters? Blackboards but no wifi? Why is technology unwelcome in this context? Are these important distinctions? I've no idea.

Sounds like you need a manifesto. And I would suggest you try very hard to make it as unpretentious as possible.

Short answer: Nope.

<comment><USER3> I think he just wants to find people who share his exact hobbies to help fund a clubhouse.

It's not a bad idea, but it's kind of dishonest to assume that this is the sort of thing that interests a ton of people, and that people will flock to it.

<comment><USER4> "Blackboards but no wifi? Why is technology unwelcome in this context? Are these important distinctions? I've no idea." +1

</comment></comment></comment><comment><USER5> Here are some similar existing spaces:

<http://www.theofficeonline.com/tour.htm>

<http://www.paragraphny.com/>

<http://www.sandboxsuites.com/>

<http://www.onealfredplace.co.uk/>

The idea is rather reminiscent of the Diogenes Club:

"There are many men in London, you know, who, some from shyness, some from misanthropy, have no wish for the company of their fellows. Yet they are not averse to comfortable chairs and the latest periodicals. It is for the convenience of these that the Diogenes Club was started, and it now contains the most unsociable and unclubable men in town. No member is permitted to take the least notice of any other one. Save in the Stranger's Room, no talking is, under any circumstances, allowed, and three offences, if brought to the notice of the committee, render the talker liable to expulsion."

(from _The Adventure of the Greek Interpreter_ : [http://etext.lib.virginia.edu/etcbin/toccer- new2?id=DoyGree....](http://etext.lib.virginia.edu/etcbin/toccer- new2?id=DoyGree.sgm&images=images/modeng&data=/texts/english/modeng/parsed&tag=public&part=all) )

<comment><USER6> This is quite cool. I wish we had something like this in Beijing. Also, nice Sherlock Holmes reference.

</comment></comment><comment><USER7> Call it a Renaissance space! Encourage the sciences, writing, philosophy, the arts. Make it the micro-equivalent of going through a liberal arts major for those of us who picked something else or people who can't afford it. Organize discussion, readings. Have a studio setup for painting and the like. I'd sign up in a heart beat.

</comment><comment><USER8> Note that your results will likely be skewed. I don't care about this and I couldn't be bothered filling in the form, and I'm sure I'm not alone.

<comment><USER9> In other words, you decided against one click (the 'submit' button, since the 'nope' radio button was already selected) in favor of several clicks and a hundred keystrokes to tell the guy how uninterested you are? Huh.

<comment><USER8> I didn't do that to tell him how uninterested I was; I did that to tell him that his results are likely skewed, and then explained why I thought that.

Furthermore, discussion on HN is worth a lot more to me than clicking a button in his form.

</comment></comment></comment><comment><USER10> When you mentioned a modern salon I thought about 300 B.C. Back when barbershops / salons were the center of gossip, philosophy, and debates. The stylists / barbers, mostly focused on men, and participated in trimming the beard, hair, and fingernails of other high profile people. Since most of the time they didn't have their own shop, they went to an agora. Agoras were opened to the public, but they had to obviously create a group around high people to allow higher forms of conversations. When men visited these areas, whether it was the agora itself or the shop / salon, they wanted to tackle two things at the same time just like how you're suggesting bookbinding and painting at the same time. Personally, I can't paint and have a decent conversation at the same time, or bookbind-- I won't get much done because I'll have to look at the person from time to time or I'll feel I'm disrespecting them.

I can't think of a being who would want to be in the same room as someone doing bookbinding, painting, and reading. Maybe some readers want a quieter room. Not sure but you should consider topic specific private rooms.

Back then, in an agora there wasn't arts and craft tools because men had their public social fix handed to them already from getting groomed, they did this with their private friends. Now. This was done in the open and in the public, outside in parks or plazas. So taking that idea and making it subscription based and inside. I wouldn't go to it. I would go to a public outdoor space, where I have the option to rent bookbinding and painting materials, or other books so I don't have to bring my own there. I shouldn't be excluded to participate if I wanted to bring my own canvas.

Now I understand why you said, modern day salon, and not a modern day agora. It's indoor based.

</comment><comment><USER11> You basically just described the Common Room of my old university's Maths Department.

</comment><comment><USER12> You are getting a lot of hate in this thread. Just wanted to say that I work in an ad agency, and we have a number of these spaces, and everyone (or at least, the most ambitious people among us) loves them - creatives and technologists alike.

Here's a description of some of them: 1) a small quiet room with a couple couches, soft fengshui-safe decorations, iPad hooked up to overhead speakers playing music, and a guitar. 2) A room filled with magazines, craft supplies, and meditative collages that people have pasted all over the walls. 3) A huge, white room with bare floors, long wooden tables. The walls are white, can be written on with markers and are magnetic for posting up whatever ideas, paper, etc you are banging around, by yourself or with others. There are big screen TVs, and also speakers, scattered that you can hook up your computer to if you need. And of course wifi everywhere (should go without saying)

I use 1 and 3 quite a bit, for collaboration and sometimes just meditation.

</comment><comment><USER13> Is this substantially different from a member's club for technologists? If so, it's a pretty established concept, you just probably want to attract a younger demographic. Wikipedia link is not NSFW, btw.

<http://en.wikipedia.org/wiki/Gentlemens_club>

</comment><comment><USER14> I imagine the clientele would be try-hard pseudo-intellectuals, so no.

<comment><USER15> I had a similar reaction. I often use my local hackerspace (Robots and Dinosaurs in Sydney) as a place to hang out and talk to smart interesting people - I guess because I find people who make stuff more interesting to people who just talk about stuff. (Maybe that makes _me_ the try-hard pseudo- intellectual sometimes… Hmmm…)

</comment></comment><comment><USER16> This type of thing is already present in a number of larger cities. It's usually called a private library, a private club, or an "athenaeum."

</comment><comment><USER17> Why not make a HN poll? <https://news.ycombinator.com/newpoll>

<comment><USER18> He gets better data, and the ability to restrict how people vote. For example, HN polls allow you to upvote several options simultaneously.

</comment></comment><comment><USER19> Almost sounds like a university library outside of a university. I would certainly be in support of that, but as it'll be nowhere near anywhere I go, I'd find it hard to put actual money down.

Now.. I'd totally love one of my _own_ , just need to save up for that bigger house ;-)

</comment><comment><USER20> That's so frightening. I was thinking of this exact thing as a business idea on a road trip this weekend. Comfy chairs, a cash bar, maybe pool/foozball/darts. Companies could rent space out for interviews. You could rent rooms to hang out and bounce ideas around, or maybe have an open space where you can just draw up and chat about ideas and allow others to freely come over and weigh in on them. You might meet a new employee or employer this way.

The names I had in mind were ThinkBar and... uh... "Lounge" something.

Anyway, I just thought it interesting that I'd be mulling this over all weekend and jump on Hacker News, only to see this post.

</comment><comment><USER21> I think your local uni Arts department probably have this one covered...

</comment><comment><USER22> I dunno about that, but something I want is a space that has little one- occupant rooms with no particular furnishings aside from perhaps a beanbag, a frosted window or indirect lighting, walls painted a neutral colour, heavy soundproofing, no electrical sockets, faraday cage shielding, and a door that locks. They should build a few rooms like that into every office. Somewhere that you can go and switch off your external attention and think or meditate.

<comment><USER23> Aside from the faraday cage shielding, that shouldn't be too hard to achieve. Why not just shut off your cell phone?

</comment></comment><comment><USER24> Our hackerspace is largely this during open nights. Too many people to actually do anything practical (www.edinburghhacklab.com)

</comment><comment><USER25> Sounds a little like a university library. Yes.

</comment><comment><USER26> As some-one living in a flat, if there was somewhere with a quality piano I would happily pay gym prices to go play it. Although much like a gym, if I couldn't get at the piano I would soon stop paying.

I live nowhere that would have this unfortunately (140k pop town)

<comment><USER27> I live in Helena, MT (let's say 60k pop). I met a local piano tuner a few months back who had started a program where he would place unwanted pianos in local businesses and maintain them for free. He said the local market just isn't buying pianos anymore and he has a couple dozen sitting in storage. Rather than throw them out, which he is a real, tragic possibility, he gives them away.

I just opened a coworking space in town (grand opening was Friday!) and I'm going to try and get one of these pianos in there.

</comment></comment><comment><USER28> I'm a member of a thinkerspace. It's called a math dept. See also: library, cafe.

</comment><comment><USER29> Isn't it what a cafe is for?

</comment><comment><USER30> I think you'd need to not provide Wi-fi or else people would just be huddled around laptops. But then if you did that maybe no one would come? Or they'd just use their aircard :|

</comment><comment><USER31> I'd like a hackerspace that's not overrun with makerbots.

</comment><comment><USER10> Note: "There was a problem with your submission. Sorry, but this form is no longer accepting submissions."

<comment><AUTHOR> Thanks for pointing that out! I upgraded the account and the form should be back up.

</comment></comment><comment><USER32> Sounds like a "Gentlemen's Club" from an Ian Fleming novel.

</comment><comment><USER33> If it's in San Francisco or Berkeley/Oakland...

<comment><USER34> Mechanic's Institute Library. It's right around the corner from Montgomery BART.

<http://www.milibrary.org/>

<comment><USER33> I would pay a few hundred a month (or more?) if I could hold semi-private events for 20-30 people occasionally in the evenings, but the listed hours of 9AM-5PM are uninteresting to me. Mechanics Institute still sounds neat, I'll check it out sometime. Thanks.

</comment></comment></comment><comment><USER35> Sounds like a hackerspace without computers. I think the intersection of people who love math, arts and sciences, but want to stay away from technology, will be quite low.

<comment><USER36> I used to go to the library with just Kindle to get away from technology (= distractions), so I guess it might be more popular than you assume.

<comment><USER35> But would you pay for that? Besides, a "thinkerspace" would be as noisy as a cafe.

</comment></comment></comment><comment><USER34> I would have said yes until the music part turned out to be non-technological.</comment>


Our take on Derby vs. Meteor - <AUTHOR> http://blog.derbyjs.com/2012/04/14/our-take-on-derby-vs-meteor/

<comment><USER2> It's great to have a liberally licensed alternative to Meteor and Firebase. Many thanks for that.

What I have taken from the article:

1\. Derby is done by competent people 2\. They support server and client code sharing 3\. MIT license 4\. NPM instead of proprietary packaging

I especially like 2). It was one of the killer features of Appjet which had code sections like: /* appjet:server _/, /_ appjet:client _/, /_ appjet:both */ You can guess what "both" did. Appjet was miles ahead for its time. I am sure David will take care of that part at Meteor's.

Last but not least, I don't like the name Derby. It is already used by a popular Java SQL database.

<comment><USER3> Agree on the name. I'm not even sure how to pronounce derby. I have heard it pronounced both "darby" and "derby", with the former being the more traditional, "correct" pronunciation. Kind of like how you are to pronounce grand prix as "gron pree" instead of phonetically. I don't know if it's best to be "correct" or to "dumb it down" to fit in with your peers.

</comment></comment><comment><USER4> Looks like a lenghty review.

Why do all node.js based frameworks prefer nosql? And where are the discussion about automated testing and continuous integration/deployment?

I'd like to see a framework that support traditional RDBMS and makes testing in both server and clinet side easier.

Rails has set the bar higher...

<comment><USER5> The reason we use a document based datastore is that it is easier to think about conflict resolution, and it maps more easily into how JavaScript represents data in the browser. It is common for large-scale eventually consistent systems to use document based stores instead of RDBMS.

We also are big fans of testing, and we will have a better answer to how testing can be done. We currently use Mocha, which is a great test runner for the server and the browser. All of our frameworks are still quickly iterating and adding important features like this.

<comment><USER4> Typically testing client side JS is tricky thus a combination of patterns or good separation is a good enough solution (see GWT MVP pattern).

</comment></comment></comment><comment><USER6> Can we please start talking seriously about security around these frameworks. Exposing your DB API directly to the client is opening yourself to the equivalent of "rm -rf /" or "DELETE * FROM table".

<comment><USER7> I honestly believe that most people ignore that question because they can't convince themselves that someone who can design something like Meteor would create such an obvious security hole.

Edit: If you downvote, please elaborate what I am missing. I really would like to know.

<comment><USER8> Firebase may have major security problems; they plan on having everything client side, and have given a vague "uh, sure, we've got plans - we'll fix it all with ACLs!" answer, which isn't filling a lot of people with confidence. But they seem like cool guys, and they'll probably figure it out.

Meteor and (I believe) Derby simply don't have this this security issue; it's obvious from the design that they don't, it's a major selling point on Meteor's website that they won't, and the Meteor dev's have explained it in detail. They've even explicitly stated what was already pretty damn clear: There is no ability to drop a database (or similar) from a Meteor client.

In short, Meteor and Derby are both client/server frameworks where the security, validation, and authentication lives on the sever; that is a very well proven design (inasmuch as it's the same used in, you know, every webapp on the planet). And contrary to the parents comment, they _DO NOT EXPOSE THEIR DATABASE API TO THE CLIENT_.

So the reason people are ignoring the question (and downvoting you) is that you are talking about Meteor (and Derby), and those concerns obviously do not apply. If you want to argue about Firebase's security, you should feel free, but this isn't really the place to do it. :)

<comment><USER7> I havent claimed that they created an obvious security hole.

I gave the answer why most people dont ask that question while clearly refraining from stating an opinion on the matter, simply because i'm no security expert who can judge if introducing such an api creates additional vulnerabilities even with server side checks(not obvious).

</comment></comment></comment></comment><comment><AUTHOR> I'd be curious to hear any takes on how other similar JS frameworks focused on "live" data binding (particularly Ember, also Knockout) stack up to these two newer players. Clearly Derby and Meteor strive to be more complete end-to-end solutions, but it's my understanding that this (generally speaking, and more long-term) is specifically a goal of Ember as well.

Also, the concurrency models of all the JS frameworks of this nature appear to be pretty immature at first glance. Derby is starting to attach parts of ShareJS, but I don't see much else that looks particularly promising in other frameworks. I'm not sure that you can "partially" implement OT (or mix-and- match parts of different solutions), which perplexes me about some of the code I've seen in the area, even in ShareJS.

<comment><USER5> I can't speak for Meteor as much, but Derby's bindings support pretty much everything that Knockout and Ember can do. The big difference is how you connect the bindings to model data. Knockout has a generic API that you can pass pretty much any JSON data to. This is flexible, but you end up having to write a lot of glue code to hook it up to whatever type of model you use. Ember bindings only work with Ember models. Derby automatically knows to look in its model to bind data.

Knockout is pretty minimal and fast, Ember has not performed very well in update intensive benchmarks. From our testing so far, Derby is somewhere in the middle if you bind everything, but it has granular control of bindings so you can get it to perform about as fast as Knockout.

This is just one small part of synchronizing clients, and Derby takes care of realtime data syncing as well. Synching data with ember requires a lot of manual server implementation. In addition, Ember and Knockout don't have any way of supporting server-side rendering, while Derby automatically renders everything in both the client and on the server.

You are right that you can't partially implement OT. The way that we handle it is by keeping OT operations on defined paths so that they are separately namespaced from other kinds of updates.

<comment><USER9> AngularJS's bindings beat all 3: it works with POJOs, you don't have to write glue code, and views are augumented HTML. Not "real-time", but capable of handling that with, say, Firebase. Furthermore, its testing infrastructure is best of breed, being carefully built-in and using DI. It's incredible how often it's overlooked.

<comment><AUTHOR> Thanks - I've been digging around through the data binding internals of these JS libs and I had indeed forgotten about Angular.

I'm comparing details on deferred updates and ease of integrating an OT-type collaboration layer. Derby's already starting to head this way and appears very promising, but I think there are subtle issues both in ShareJS and Derby's implementation. I'm curious how collaboration would work on some of these other libraries, and Angular seems to be thoughtfully architected, if somewhat large.

<comment><USER10> What subtle issues do you see with sharejs (just curious, I'm in the process of porting it's OT algorithm to python).

</comment></comment></comment></comment></comment><comment><USER2> I just tried the TODO app from the Derby home page and it is just doing xhr polling. I tested on Chrome and Safari.

The Metor and Firebase demos used both websockets.

<comment><USER11> You can turn on websockets with Derby, as it uses socket.io underneath.

<comment><USER2> Thank for clarifying. You should enable it, to demonstrate the cababilities of your framwork.

For high load push driven apps, websockets are a must. I understand that fallbacks are needed for users behind proxies etc. But if the infrastructure supports websockets, they must be used.

</comment></comment></comment><comment><USER12> Trying derby.js out now. Just so that you know you can write in Javascript and/or Coffeescript but the internal code and the examples are written in Coffeescript.

</comment><comment><USER13> Definitely Meteor is out of question given the license. I'm happy that Derby and SpaceMagic are viable MIT-licensed alternatives available today.

</comment><comment><USER14> I didn't know about Derby, thanks for sharing.

I won't even try Meteor, they screwed it up with the license.</comment>


Google May Be Forced to Reveal Its Search Algorithm to an SEO - <AUTHOR> https://www.searchenginejournal.com/google-may-be-forced-to-reveal-its-search-algorithm-to-an-seo/359536/ <comment><USER2> They will settle

<comment><AUTHOR> Unfortunately I agree. It would have been interesting.</comment></comment>


Lousy Pay Raise? That May Be as Good as It Gets - <AUTHOR> https://www.wsj.com/articles/why-the-tight-labor-market-isnt-generating-better-pay-1497441272 <comment><USER2> Same article, not behind paywall: [http://www.foxbusiness.com/features/2017/06/14/why-tight- lab...](http://www.foxbusiness.com/features/2017/06/14/why-tight-labor-market- isnt-generating-better-pay.html)</comment>


Ask HN: Our lead architect says 4GB is enough for a developer machine - <AUTHOR> I work in an alleged data center, building enterprise Java applications. For remote work we're "upgrading" to VDI clients with a single core and 4GB of RAM.  I posted on an intranet forum that this was insufficient, and the reply from our de facto Java guru was: "..physical workstations at sadplace only have 4GB and that works fine for most developers. We have made a request for 6GB - that's under review.."
My reactionary response would be something like "Dude, it's 2016. Eclipse, Chrome, and WebLogic max me out" (Yes, WebLogic, that says it all.)
Should I just leave this place ASAP, or has anyone had success in lobbying their company for better hardware? If so, can you give me some tips on how to pull it off?  Your tax dollars are involved! <comment><USER2> I'd recommend breaking down the time lost due to the slower machine, and spelling out how that equates to dollars down the drain. Here's an example (with arbitrary numbers):

"With a 16GB machine, it takes my test suite 5 minutes to run. With 4GB, that suite runs in 20 minutes. I need to run it on average, 10 times a day. So by giving me the 4GB machine, you're having me sit idle for an unnecessary 15 minutes, 10 times a day. So that's 2.5 hours per day that you're paying me to spin around in my chair and surf HackerNews. That's X wasted dollars every day, which is significantly more than the cost of the upgrade from 4GB to 16GB. So you'll save money by giving me a faster machine".

If there's a way to speak dollars-and-cents to the bureaucrats in your org, then there's a better chance of them approving your request.

</comment><comment><USER3> Sounds like a shortsighted company led by sales people. If there is this amount of penny-fucking at your company, maybe it is best to interview at other places and leave asap.

Things will not get better, as management has costs to cut. Your work equipment is one of those costs. If it was up to them, you'd be shoveling with 3 spoons instead of a shovel, since "it works for most", meaning "we only assign shovels on a 1-1 basis so that we wouldn't need to spend bigger amount now." What are your salary raise prospects at such a company?

</comment><comment><USER4> Yikes, that's pretty low for local development work. If you were mostly doing remote access to a more powerful development server that might work but for doing anything more serious you'll want to consider at least 8GB. Chrome uses a gig by itself so you're already down quite a bit. Most of my dev machines have been 16GB minimum.

<comment><USER5> This. Around 2.8Ghz core and 16GB ram are standard for us. Occasionally we have to settle for 8GB if that is what all that is available in Apple refurbs.

<comment><USER6> I don't care about CPU speed very much, but yeah, 16GB at least for a dev machine. You need something you can run a database on, perhaps an http server + framework, etc.

<comment><USER5> Virtual machines are a killer too. We are primarily web but end up doing more than our fair share of legacy WinForm app updates and that requires a lot of ram if you're going to have at least somewhat of a pleasant experience in VS inside the VM.

</comment></comment></comment></comment><comment><USER7> So, hardware purchases for large organisations (and what I assume might be a government agency or government vendor) seems to be driven more from strategic sourcing relationships and cost as a priority, and actual technical requirements second. Seldom do people go "here's what we need, lets cost it!" More likely its a compromise between cost and service.

That being said, 4GB is stupidly low for a development machine and that should be alarming for the development team (6GB is not much an improvement).

I would argue the "works fine for most developers" point. What works now (the current state) is certainly not going to last. 4GB is already too low and it is only going to get more difficult as the requirements increase (bloat is a constant).

The conversation should be "what is the practical life of this hardware and do we intend to upgrade when it runs out." You should think about performance hits (in the people sense, not hardware sense) if these environments are abandoned by your developers. All hardware purchases have an effective life and what you've described is a purchase which will barely last another 12 months, imho.

<comment><AUTHOR> Yeah it's not a single company making the hardware decisions. Like most government contract situations it's more akin to a 20 company gang bang at a public facility using shared work space and hardware.

I don't even know where I'd begin to petition for a change. This was mostly just my way of ranting somewhere I'd get support, since no one made a peep on the internal comment I posted. I don't think most developers are fine with the 4GB situation. Those that are don't matter, and the others have likely just been beaten into submission or don't want to be seen as fussy.

</comment></comment><comment><USER8> "fine for most developers", sounds to me like the Java guru needs to provider proof of that statement.

Gallup does surveys for businesses and one of the questions they ask is "Do you have the right equipment to do your job".

[http://www.gallup.com/businessjournal/26773/why-employees- ne...](http://www.gallup.com/businessjournal/26773/why-employees-need-right- equipment.aspx)

If your workstation is slow, or you have to deal with OOM issues, and can't get your work done, that is something that needs addressed not by guessing, but my management doing their job and providing you with the right equipment from the start.

If the company isn't willing to provide you the right equipment based on what the employees think and not just on what one person says based on hearsay, than I'd question if that is a good place to work.

</comment><comment><AUTHOR> Oh, I forgot to add: Single Core, 2666 Mhz.

I really wanted to ask if I could donate my old netbook and remote into that.

</comment><comment><USER9> It's very easy for someone on the Internet to tell you to run away from conditions like this. I'm not going to tell you that but consider this: the world is full of places like that but it is also full of seriously _great_ places to work as well. Every minute you work at some crappy job is a minute you could be working at someplace nice instead. When you start interviewing for your next job, be sure to ask them about their development hardware. ;-)

</comment><comment><USER10> We deploy 2 cpu and 4 GB for office 2016 VDI users. That's tight for a developer.

Unless you're offshore and your wasted time is not as valuable, that's a poor VDI use case that doesn't make sense

Fight it with vendor recommendations.

[http://blogs.vmware.com/consulting/tag/vmware-view- storage](http://blogs.vmware.com/consulting/tag/vmware-view-storage)

[http://www.vmware.com/files/pdf/view/Server-Storage- Sizing-G...](http://www.vmware.com/files/pdf/view/Server-Storage-Sizing-Guide- Windows-7-TN.pdf)

</comment><comment><USER11> I second what MalcomDiggs has said, but run some benchmarks on your home box with all the memory. Find an amount at which performance increases level off, & request a just amount based on hard data. If they don't respond after making it perfectly clear as to what you need to do your job properly, then leave. (Assuming you can easily get work elsewhere)

Maybe link this comments thread in your intranet post, a community like HN expressing sane norms might help your case.

</comment><comment><USER12> You'd be surprised what a single, well-scheduled CPU can put up with. Don't want to add too many and deal with CPU Ready, where the hypervisor is trying to schedule all those cores on the host. Smaller VMs are easier to schedule and thus have less opportunity to be forced into idleness.

But yeah you need twice that RAM, minimum.

</comment><comment><USER13> That'll just lead to a dick-swinging contest if your resident Java guru already said 4gb is enough. It's futile. There are many ways to reduce the memory consumption of your desktop machine. Start with those.

<comment><USER14> Tldr: Put up and shut up

</comment></comment><comment><USER15> Put your own RAM in.

I used to have to upgrade my own machines. For years. Monitors and RAM. I haven't had a place that gave me anything less than 16G for years now, though.

<comment><USER12> It's VDI. Virtualized Desktop Infrastructure. Good luck accessing the datacenter.

</comment></comment><comment><USER16> I would expect this from a cost center vs profit center workplace. I have 24 GB on my W530, the more, the better...

</comment><comment><USER17> Laughable. 4 gig was adequate in 2008, maybe.

<comment><USER18> This hits home. I'm working on a 2008 macbook that was (I guess) upgraded from 4gb to 8gb RAM when I bought it off of some guy.

It works for me to develop on though. But I wouldn't recommend it to anyone.

</comment></comment><comment><USER19> Run away

</comment><comment><USER14> Oh dear! Can you BYO laptop?

</comment><comment><USER20> Run

</comment><comment><USER21> Well, if it really is a VDI client and not used for more then the specs would be fine, not much happening locally that would need more.

But then you wouldn't be running Eclipse, Chrome and WebLogic locally on them, so that's probably not the case? And even Chrome alone can easily eat more than 4 GB, so that's kinda laughable for any kind of dev work.

<comment><USER14> Point?

<comment><USER21> Trying to understand why OP talks about VDI clients (basically dumb terminals), but then complains that they don't have enough RAM to run apps locally.

<comment><USER14> Good question. I misread the op the first time. So yeah whats the advantage of stuffing more on the thin client. Unless he is also using the thin client to run stuff he cant run remotely.

<comment><AUTHOR> Just a terminology mistake on my part. I don't know what to call the virtualized OS I'm remoting into. We just refer to it as VDI, but that's more general.

What's the term for the instance I connect to? I guess I should have just called it the VM. My VM only has 4GB of RAM and one core and I'm supposed to write and test (deploy) decent sized applications on it.

<comment><USER14> Run!</comment></comment></comment></comment></comment></comment>


Sorting number strings numerically - <AUTHOR> https://www.arangodb.com/2017/09/sorting-number-strings-numerically/ <comment><USER3> Fun problem, at least abstractly. Here's a similar, maybe more basic approach for positive integers. It uses the idea at the beginning of the article to store numbers in unary, to store the length of the number in unary ... recursively. The idea is to sort by ... the length of the length of the length of the number, then by the length of the length of the number, then by the length of the number, then by the number.

First, imagine we store the number in the form

        aaaaaaaaaaa 92389210184

where the number of a's is equal to the length of the number. This encoding scheme works for the same reasons as discussed in the article. And it only takes 2*(length of original number)+1 characters.

But now, we can do better by applying the same encoding scheme to the 'aaaaaaaaaaa'. That is, suppose the original number is 295 digits long. The number 295 is 3 digits long. So we store the number with 3 b's, as:

        bbb 295 109382388575782352353453....

OK, but what if we want to store a number that's 4,000,000,000 digits long? This would look like

        bbbbbbbbbb 4000000000 109382388575782352353453....

And that's a bit wasteful. If there are more than two b's, then we can recursively apply the encoding to the b's. There are 10 b's, and 10 has two digits, so we get

        cc 10 4000000000 109382388575782352353453....

And so on. (We won't ever have hard drives that can make use of letters e and above, so it can probably stop there.)

So the encoding schemes look like

        a n                            # n is only one digit
        aa n                           # n is two digits
        b log10(n) n                   # log10(n) is one digit
        bb log10(n) n                  # log10(n) is two digits
        c log10(log10(n)) log10(n) n   # log10(log10(n)) is one digit
        cc log10(log10(n)) log10(n) n  # log10(log10(n)) is two digits
        d log10(log10(log10(n))) log10(log10(n)) log10(n) n
        ...

<comment><USER4> Does this have a canonical form? It seems like the same number has multiple possible encodings that don't sort at the same place.

<comment><USER3> You'd definitely have to be more careful than I was in my post! But I think it should be possible to make it work via something like always using either one or two initial letters. The algorithm I'm thinking of is:

1\. Given n, write aaaaaaa n where the number of a's is equal to the length of n.

2\. If the number of a's is 1 or 2, stop.

3\. Let n' be the number of a's.

4\. Write bbbbb n' n where the number of b's is equal to the length of n'.

5\. If the number of b's is 1 or 2, stop.

6\. ...continue until the number of initial letters is 1 or 2.

I think that algorithm is mostly well-defined and only produces one possible output for each input number. So let's see if I can give a good proof-ish argument that it sorts correctly.

Suppose n <= m, then length(n) <= length(m).

Case 1: If their lengths are equal, then in step 1, they are written as

        aaaaaaaaa n
        aaaaaaaaa m

where the number of a's are the same. Now the rest of the algorithm is the same for both numbers, since they have the same number of a's, so they get written as the exact same prefix followed by the number. Since they have the same length, the smaller one sorts first.

Case 2: Now suppose length(n) < length(m), and further suppose length(n) <= 2. Then in step 1, they are written as, for instance,

        a n
        aa m

or something like

        aa n
        aaaa m

Now any encoding of m in further steps will start with the letter b or greater, which sorts below "aa n".

Case 3: Now suppose length(n) < length(m), but further suppose length(n) > 2\. Then in step 1, they are written as

        aaaaaaaaa n
        aaaaaaaaaaa m

Now n will sort before m as long as the encoding of a smaller string of a's sorts before the encoding of a longer string. Let n' = length(n) and m' = length(m). We know that n' < m' and we want to prove that the encoding of n' sorts before the encoding of m'. This is true recursively by the same 3-case argument as above, though I haven't really formalized it well here.

For example, if length(n') = length(m'), then they both get written in the next step as

        bbbb n' n
        bbbb m' m

which is just like Case 1 above. Otherwise, they get written something like

        b n' n
        bbb m' m

which is just like Case 2 above. Otherwise, they get written

        bbbbb n' n
        bbbbbb m' m

which is like Case 3 above, and requires us to recurse again.

</comment></comment></comment><comment><USER5> if you have this problem (or like this problem), there's a whitepaper on this topic (with a different encoding scheme) here: [http://www.zanopha.com/docs/elen.pdf](http://www.zanopha.com/docs/elen.pdf)

I implemented a decent portion of it here for a Go project: [https://github.com/jordanorelli/lexnum](https://github.com/jordanorelli/lexnum)

the whitepaper also discusses an implementation that accommodates floating point numbers, but I didn't implement that portion. This encoding scheme works well for file names.

<comment><USER6> Great. Thanks. I was not aware of this.

</comment></comment><comment><USER7> This looks like a similar problem to providing a lexicographic binary encoding for the integers, and the technique given here reminds me of Exp-Golomb encoding.

A more fun problem is extending this scheme to support rational numbers. [http://www.imada.sdu.dk/~kornerup/papers/lcf.ps.gz](http://www.imada.sdu.dk/~kornerup/papers/lcf.ps.gz) gives a neat scheme using continued fractions. (I've had a go at using it to implement a sort order preserving encoding that works for all the rationals at [https://github.com/NegativeMjark/lexical- binary](https://github.com/NegativeMjark/lexical-binary) )

<comment><USER6> Silly solution: The rationals are countable. Just use any bijective mapping of the natural numbers to the rationals and then use the scheme for the natural numbers. (I know, a mathematician's answer! :-) )

<comment><USER8> That would imply the mapping preserved ordering, which doesn't seem possible.

<comment><USER4> Wait, do the natural numbers and rational numbers have the same ordinality? It seems not because of the whole 'a number in between every 2 rational numbers'. But then what is the ordinality of the rational numbers?

<comment><USER9> The standard argument for showing that they have the same cardinality is to consider the matrix (let's see if I can render this):

        1/1  2/1  3/1  4/1 ...
        1/2  2/2  3/2  4/2 ...
        1/3  2/3  3/3  4/3 ...
        ...

You can traverse each SW-NE diagonal, producing the sequence 1/1, 1/2, 2/1, 1/3, 2/2, 3/1, ...

This will eventually hit every positive rational (several representations, in fact). If you want, you can start with 0 and insert -q when you hit q.

</comment></comment></comment></comment></comment><comment><USER10> The article starts with a discussion of storing large numbers in JSON.

JSON does not restrict the range of numbers. It happens they many languages that parse JSON do, but ArangoDB could try and be different! :)

<comment><USER6> Many JSON parsers will treat long numbers as doubles. Therefore the only safe way to store them without potential loss is in a string. Without the encoding in the article or something along these lines, the sorting done by a database is not compatible with the integer sorting.

<comment><USER11> It remans a fact that the JSON spec does not limit the length of numbers, so there is no need for this kind of kludge.

If Arango would just allow unlimited numbers in JSON data and sort them numerically, developers could use a different JSON encoder and parser in their language of choice, that would encode and parse numbers from the unlimited representation allowed by JSON to whatever bignum library their language has.

<comment><USER12> There is a need for this kind of kludge because systems that you interact with will lose precision with large numbers. If Arango was the only system parsing the json data that would be one thing, but that is not a real-world use-case.

</comment></comment></comment></comment><comment><USER13> Seems much more complicated than just changing the comparator to compare lengths first.

<comment><USER14> Which would additionally preserve trivial human-readability of the stored numbers - one of the reasons of choosing JSON in the first place.

<comment><USER15> This scheme is still human readable, you just have to skip the length prefix and you get a simple number after that. It's not really human-writeable however...

Although if you wanted to save space you could also store the number itself in a bigger base to reduce the number of "digits".

</comment></comment></comment><comment><USER8> Here's a much simpler solution to the problem: your length is no more than ~2^48 since you're using real hardware, so prefix your string with an 8-byte base64 encoding of that 6-byte length.

<comment><USER16> Note that you would not have an encoding for negative numbers then, and you would have to use the alternative base64 table,

      +/0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz

as the normal one is not order-preserving. Otherwise yeah, you could probably do that--and of course there are many ways to patch what you've got to handle negatives, e.g. by noticing that it is slightly wasteful (the base64 prefix ++++++++ is never used) or by using a leading character lower than + (so - <USER2> out but ! for example is available) and then encoding the rest of the string in an "ascii-reversed" way.

However I must object to the idea that "real hardware" cannot handle the petabyte scale. The in-article recursive approach of using the algorithm:

        function encode(number) {
            //calculate some base number_string
            return number < NUM_THRESHOLD?
                number_string :
                sigil + encode(number_string.length) + opt_divider + number_string;
        }

is not a particularly bad one. The only thing that maybe I'd balk at is that the given encoding requires a human editor to use an ASCII table to lookup their string lengths, which is clearly undesirable... one might instead go for the most naive case where NUM_THRESHOLD is simply 10 and have the number 123,456,789,012,345,678,901 be represented as:

        ~~2 21 123456789012345678901

so that it's clearly "there are going to be 2 length strings" (hence two tildes) -- the first always has length 1, it specifies "the next has length 2" and then "the next has length 21."

This is also pretty easy to parse and one can use a leading - sign to encode negative numbers, for example.

<comment><USER8> > However I must object to the idea that "real hardware" cannot handle the > petabyte scale.

By the time I'm talking about integers that don't fit in x86 address space, I'm not talking JSON ;).

<comment><USER16> I guess that makes sense and speaks to a bigger problem here, which is that with a very large number one certainly wants to simply store it in a massive file on disk, and the encoding to do that probably shouldn't be based on the printable ASCII digits--because why.

</comment></comment></comment></comment><comment><USER17> This is a neat trick, but the example code (after fixing a typo, "decodeNonNegatve" to "decodeNonNegative") doesn't sort correctly:

      ["-10", "-11", "0", "1", "-1", "-2", "10", "11", "110"].map(encodeLong).sort().map(decodeLong)

Results in:

      ["0", "1", "10", "11", "110", "-11", "-10", "-2", "-1"]

<comment><USER18> The author used `-` as the sorting character for negatives instead of `!` as stated in the article. If you change his functions to use `!` for encoding negatives then it correctly sorts.

        function encodeLong(s) {
          if (s[0] !== '-') { return encodeNonNegative(s, ' '); }
          return '!' + translate(encodeNonNegative(s.slice(1), ' '));
        }

        function decodeLong(s) {
            if (s[0] !== "!") { return decodeNonNegative(s); }
            return '-' + decodeNonNegative(translate(s.slice(1)));
        }

</comment></comment><comment><USER19> The really interesting property, if you do this right, is that you can concaneate a series of fields, sort them lexicographically, and you end up with the data sorted by the types of the fields. This allows you to create complex composite keys that contain primary indexes directly in the key. I don't know if this is something fundamentally important, or obvious, but when I learned it, it made designing my key schema a lot easier.

</comment><comment><USER20> Related (but different) problem: SQLite4 Key Encoding

[https://sqlite.org/src4/doc/trunk/www/key_encoding.wiki](https://sqlite.org/src4/doc/trunk/www/key_encoding.wiki)

 _Numeric SQL values must be coded so as to sort in numeric order. We assume that numeric SQL values can be both integer and floating point values._

</comment><comment><USER20> Is utf-8 significant here, as opposed to the ASCII subset? It seems like it's an ASCII encoding, and it sorts like ASCII.

I think that utf-8 sorting may depend on the locale. But here you're not depending on that -- you're just using the subset that is ASCII sorting, no?

<comment><USER16> Yes, this usage of UTF-8 is limited to the ASCII subset.

<comment><USER20> Regarding the examples:

        " 0
        " 1
        # 42

Why is the length stored in base 92 while the number is stored in base 10?

Why not just store them both in base 128 or 127 (if you want C strings)? You can't use base 256 because some strings won't be valid UTF-8, but base 127 or 128 seems fine.

Do you want some notion of human readability? That isn't in the problem statement. Your problem statement seems fairly imprecise in a couple ways.

<comment><USER16> I mean, I think a case can be made for something like it, where possibly one adds to the problem statement "the bytes should be contiguous ASCII and should not cause the JSON standard to backslash-encode the characters." The biggest range which does this is 0x23 to 0x5B inclusive. But that only gives 57 workable characters.

If it helps, I am not the author of the original article.

</comment></comment></comment></comment><comment><USER21> I feel like this entry should have a different title. I passed it over at first because it looks like it's just a primer on how to implement ls -v, or something, when it's much more interesting.

<comment><USER22> Not click-baity enough?

</comment></comment><comment><USER23> If you don't need to worry about exponential notation or negative numbers, having a "natural sort" comparator is another way to deal with this.

</comment><comment><USER24> Another name for sign + short representation of magnitude + mantissa is Floating Point, which also sorts lexicographically for positive values.

</comment><comment><USER25> Brilliant!

</comment><comment><USER26> Are there any real life applications for this?

<comment><AUTHOR> Many catalogues of mathematical groups contain things like the monster group, which has 808,017,424,794,512,875,886,459,904,961,710,757,005,754,368,000,000,000 elements. That won't fit in a 64 or even 128 bit integer.

It is common to want to ask a database "tell me all groups of size greater than X, with properties A, B and C". Now, if you have arbitrary sized ints, no problem. If your database (or language) doesn't support big ints, you need to figure out how to do "bigger than X", when you are storing big numbers in some other format, probably strings.

<comment><USER27> Surely if you're storing numeric values as strings, it would be better to use something like base64 encoding to use as much of the available symbol space as possible?

If you can sort a numeric string in base 10, you can sort one in base 16, or base 64, or base 256.

<comment><AUTHOR> I think you could use this method with other bases, although be careful with base 256 -- if your strings are stored in UTF8 or something, then base 256 isn't actually that useful.

However, the important bit (to me) is that you can use your database's sorting function to compare the numbers.

The advantage of base-10 is it's easy to display the number :) Others bases do provide a constant speed improvement of course.

<comment><USER28> I'm genuinely curious, does anyone actually want/need to look at numbers larger than will fit in a 128bit int? I understand there are applications that require use and storage of such numbers, but how often is there are real need to display them?

<comment><USER6> In discrete mathematics this happens a lot. Group orders, sizee of conjugacy classes, semigroup orders, numbers of isomorphism classes, character degrees, etc.

<comment><USER29> "It happens" doesn't imply there's any need for it to happen.

</comment></comment></comment></comment></comment></comment></comment><comment><USER30> Mirror?

<comment><USER31> Google Cache: [http://webcache.googleusercontent.com/search?q=cache:https:/...](http://webcache.googleusercontent.com/search?q=cache:https://www.arangodb.com/2017/09/sorting- number-strings-numerically/)</comment></comment>


Ugandan innovators use smartphones to detect malaria - <AUTHOR> http://www.biztechafrica.com/article/ugandan-innovators-use-smartphones-detect-malaria/9580/#.VNP0GZ3Tm1s

<comment><USER2> Fun, but at best this article and the linked blog are missing critical details. Other organizations (e.g. [http://ianxen.com/](http://ianxen.com/)) have been working on similar technology involving optics, image processing and a blood sample -- and the associated consumables.

Even assuming the tech is sorted the more interesting questions involve how, from a health systems perspective, these things can be deployed in rural health centers at a scale that matters. Some hard training/security/procedure/money questions to figure out, and I haven't heard much about these issues yet. Unclear if it's because the technology isn't ready for prime time or other reasons.

</comment><comment><USER3> From the article it looks like they use a technique similar to the 'Heart rate monitor' apps, but instead of trying to look for pulses they look to quantify the blood oxygen content, perhaps using colour? and from that get a diagnosis on a malarial infection.

</comment><comment><USER4> You've got to love non-invasive tests like this, but the trade-off is lower accuracy.

In this case, for example, it could give false positives for people who have differently shape blood cells - e.g. sickle cell, spherocytosis, elliptocytosis.</comment>


Why would someone use OpenDNS on a per-PC basis? I must be missing something. - <AUTHOR> http://hypecycles.wordpress.com/2009/08/31/opendns/

<comment><USER2> Primarily because client side software that serves a similar function is bulky, harder to configure, and almost never free. If your browser becomes compromised there's a fine chance its built-in phishing/malware filter will be broken or disabled. It's unlikely a browser exploit could change your DNS settings -- if it could you've got much bigger problems.

From a performance point of view many ISP name servers are either oversubscribed or simply not fine tuned. There are many tricks to DNS configuration on busy servers and OpenDNS does a good job keeping their servers fast and responsive. Additionally your ISP may not be giving you the best possible DNS servers. Many ISPs are getting involved in the ISP sponsored custom redirect game. While OpenDNS does indulge in this to a degree their service doesn't seem to take a hit from it which is something I cannot say about other custom redirects.

</comment><comment><USER3> _So, could one really do this OpenDNS thing behind a pay-for-internet- service?_

Yes but it requires the pay-for-internet provider to think about their real usage scenarios and plan for them appropriately. I'm involved in the engineering of a similar system that has nearly a million users, and we've chosen to leave port 53 wide open, even though it can theoretically be used for DNS tunneling or other nefarious use.

Also, I disagree with the comments on the post that claim that ISP DNS is the #1 cause of slowdowns -- DNS on an ISP scale is surprisingly easy to dimension and support in the common case. The important aspects of DNS administration center around debugging misbehaving authoritative servers and management of Denial of Service attacks.

<comment><USER4> I used to find that I'd get slow downs when performing requests because the ISP wasn't answering the DNS query in a timely fashion. This would usually resolve itself in half-an-hour or so, possibly a load issue. But with OpenDNS I've not had those problems, my ISP nameservers are there as backup however if I should need them.

Also OpenDNS optionally does filtering and allows reporting of lookups made. As my kids grow and explore for themselves I think this will be useful.

</comment></comment><comment><AUTHOR> More on this topic and a follow-on post at <http://hypecycles.wordpress.com/2009/09/01/opendns-again/>

</comment><comment><USER5> I used OpenDNS while at Defcon to avoid my DNS servers being spoofed. The minor configuration change was totally worth it.

</comment><comment><USER6> Well, in Turkey, it's used for accessing (most of)the censored sites</comment>


Where Are the PHP Developers in NYC? - <AUTHOR> http://www.centernetworks.com/where-are-nyc-php-developers <comment><USER2> They moved to California and work for Facebook</comment>


Very Sleepy 0.9 - <AUTHOR> http://www.codersnotes.com/notes/very-sleepy-0-9

<comment><AUTHOR> Very Sleepy is an open-source polling CPU profiler for Windows, maintained for many years by Richard Mitton.

As there haven't been any releases in a while, I've been maintaining my own fork of Very Sleepy on GitHub. Today, the fork became one with upstream: Richard blessed the GitHub repository as the official one, and I've changed the name back to Very Sleepy.

Here's to many more releases from today onwards :)</comment>


Why Everyone Missed the Most Mind-Blowing Feature of Cryptocurrency - <AUTHOR> https://hackernoon.com/why-everyone-missed-the-most-mind-blowing-feature-of-cryptocurrency-860c3f25f1fb <comment><USER2> I'm skeptical that it is possible to make a system which is incentive compatible and in which-

Ok well, I guess a major point of concern I have re. this proposal is, determining uniqueness of identity is hard, and I don't really see a fully decentralized way to do it. A federated way to make a scarcity of identity thing, sure.</comment>


The turning point that is HTML5; are you ready? - <AUTHOR> http://www.amundsen.com/blog/archives/1051 <comment><USER2> While not written in any sort of professional manner, this got me thinking. What is the author really suggesting?

Data will be linked to other data, and the client based application will use it?

This sounds a lot like a general-purpose REST(ish) client. For instance, data comes back, with actions on that data tied to other URLs to request to. The client application will decide how to present it (is web design dead at this point?). This is interesting, as the client applications are literally data consumers - and the application developers really data architects. Its possible that there would be no web design, no HTML/CSS/Javascript, and another hollistic, more expressive technology will take its place.

Having data linked to other data is nothing we can't produce with well- designed web services today. Maybe there will be a JSON linked-data API standard. I could be getting the technology wrong but the concept is the same.

I'm not really sure how we would get away from presentation-focused web applications. The client would have to be fairly sophisticated to handle all of the different types of data structures coming in - and 'client data- browsers' would inevitably have different implementations (we'll still need the user agent string, won't we?). Add on top of that every application or 'view' will look the same as no design is added to it. Sounds boring and the masses probably won't catch on. I'm definitely missing something here, but the concept is fun to push around. Imagine how much larger/better/more sophisticated/faster our web applications would be if you only had to worry about data instead of also interface and state?

It doesn't really matter what comes - the mavericks and those willing to change (and not just progress) will jump on, and everyone else will be an outdated developer of yester-year. The concept is only scary to the lazy.

</comment><comment><USER3> Don't talk to me in terms of features, talk to me in terms of benefits. Why would whatever this guy is suggesting benefit anyone ie- the normal person?

Sure HTML5 has some cool features (offline, canvas, location,etc.), but the benefit is simple: get fast beautiful desktop class applications inside of your browser so that you can use them anywhere without any IT headaches.

<comment><USER4> > get fast beautiful desktop class applications inside of your browser so that > you can use them anywhere without any IT headaches.

it must be lovely to live with such strong self-delusion.

</comment></comment><comment><USER5> _the future is not rendering and linking documents on the Web. the future is rendering and linking data on the Web. and there is nothing here to make that experience any better than it is today (which is lousy). in many ways, the HTML5 effort makes it harder to build linked data applications since most will need to continue to rely on heavily scripted Web apps. and "over-scripting" is another sign that the common Web browser is an out-dated product._

Considering that the HTML5 effort was originally called "Web Applications 1.0" I rather think he's missed the point.

HTML5 isn't (just) about rendering documents - it's a full application development technology.

I don't find his statement that _"over-scripting" is another sign that the common Web browser is an out-dated product_ particularly compelling - in fact I don't even understand what it means.

<comment><USER6> Well, I don't fully agree with this guy (I think I don't, actually I'm not fully sure of where he's going). But I've also thought for some time that user-side scripting in modern AJAX apps is getting ridiculous. If you look at the javascript code generated for a GWT app, for instance, it's quite large and hardly understandable.

I don't know, web app development sometimes feels a lot like trying to turn screws with a hammer. HTML4 wasn't really designed to support dynamic applications, yet we've made it work, more or less, by abusing and patching. Nevertheless it was designed for static content, and it shows.

I'm afraid I'm not too familiar with HTML5 yet, so I don't know if it changes this, but from what little I've read it doesn't seem so.

</comment></comment><comment><USER7> I hope HTML5 includes a shift key for this guy.

</comment><comment><USER8> I am so sick of seeing the turn of phrase "The <X> that is <Y>." Someone needs to come up with a new cliched crutch for posts like this.</comment>


Ubuntu Linux Performance Over the Past Six Years on an Intel Xeon Server - <AUTHOR> https://www.phoronix.com/scan.php?page=article&item=ubuntu1210-1810-xeon <comment><USER2> Impossible to look at on mobile. The charts run off the right side of the screen. Viewing in desktop mode doesn't seem to work. I'm guessing they're trying to use a responsive layout, but they didn't test articles like this on mobile. Come on Phoronix.

<comment><USER3> I read and viewed everything just fine on my Android. Maybe user error.

<comment><USER4> The right side of the images are cropped for me on iOS

<comment><USER5> Confirmed on Chrome on iOS

</comment></comment></comment></comment><comment><USER6> Surprised by the decreased socket performance, are sockets not a first-class citizen anymore? Everyone just went to http?

Anyone have clues as to why this has degraded so much over the years?

<comment><USER7> >The socket activity was hit hard by Spectre/Meltdown patches

Did you even read the article?

<comment><USER8> > Please don't insinuate that someone hasn't read an article. "Did you even > read the article? It mentions that" can be shortened to "The article > mentions that."

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

<comment><USER7> I always assumed these guidelines to be more of a joke, as they also say

>Please don't use Hacker News primarily for political or ideological battle.

which HN staff themselves blatantly violate.

<comment><USER9> The Romans had a saying for this: "Quod licet Iovi, non licet bovi". While 20 centuries might have gone since it was coined it still holds true.

Going back to the original assumption that someone did not read the article it should be noted that socket performance in 18.10 is lower than that in 12.10 even when the spectre/meltdown mitigation is deactivated.

</comment></comment></comment></comment></comment><comment><USER10> Wow how come the results are all over the place? I would've expected a 5% - 10% improvement here or there, but look at this!

[https://www.phoronix.com/scan.php?page=article&item=ubuntu12...](https://www.phoronix.com/scan.php?page=article&item=ubuntu1210-1810-xeon&num=5)

</comment><comment><USER11> What is exactly is being measured in, for example, the R script? The same code running in the same modern version of R on Ubuntu 12 and 18? Or is the R version dating to Ubuntu 12? Or is it contemporaneous for both releases?

I really wish these websites would repeat these 100x or something so we could get an idea of the variability in the measurements.

<comment><USER12> The versions used are listed in the benchmark image for almost all of the tests run. Here they are for R.

        Ubuntu 12: R v2.15.1 (2012-06-22)
        Ubuntu 16: R v3.5.1  (2018-07-02)</comment></comment>


Monads Made Difficult - <AUTHOR> http://www.stephendiehl.com/posts/monads.html <comment><USER2> Haskell's Monad is a very, very special case of the mathematical monad. The problem with Stephen Diehl's approach is that it treats the topic in full generality making it much harder to understand how the special case is actually much simpler.

I prefer [http://web.jaguarpaw.co.uk/~tom/blog/posts/2012-09-02-what-i...](http://web.jaguarpaw.co.uk/~tom/blog/posts/2012-09-02-what- is-a-monad-really.html), but that's probably because I wrote it.

<comment><USER3> I think this is for people that already know how to use monads effectively in Haskell and wants to learn more about the mathematical background.

<comment><USER4> This is explicitly not a writeup for beginners ( ergo the title ), it's a side-by-side description of the /very/ general categorical description and how you can model that in Haskell. The three parameter form of morphism composition ( i.e. c y z -> c x y -> c x z ) would be quite unwieldy in day- to-day programming but it does illustrate the underlying definitions better to have categories explicit.

<comment><USER5> I don't mean to nitpick, but the three parameter form is exactly what you use in day-to-day Haskell programming. You simply let c be the infix (->), and you get (y -> z) -> (x -> y) -> (x -> z). We just tend to think of it as a two parameter form because a lot of the time, you can think of (->) as syntax, as opposed to the type that it is.

<comment><USER6> Yup. This is straight cut from Control.Category module:

      instance Category (->) where
          id = Prelude.id
          (.) = (Prelude..)

</comment></comment></comment></comment></comment><comment><USER7> Yes. Thank you. Yes yay yes. I studied mathematics. I've been programming for 15 years, 10-or-so professionally. I've always wanted a "real math" explanation of monads.

They now make sense to me. Seeing why mu needs to go from T^2 -> T and the role it plays concretely WRT the list and IO examples makes it really clear to me.

God bless you for not using some crazy analogy like bacon or whatever.

<comment><USER8> Burritos! Monads are Burritos! ([http://byorgey.wordpress.com/2009/01/12/abstraction- intuitio...](http://byorgey.wordpress.com/2009/01/12/abstraction-intuition- and-the-monad-tutorial-fallacy/))

</comment></comment><comment><USER9> "This assumes you are familiar with Haskell typeclasses and basic category theory."

Where do I learn basic category theory? Anything better than just perusing wikipedia?

<comment><USER10> A college CS or pure math course should cover it.

<comment><USER7> Hrm, not sure about that. I know of no college CS course that _requires_ category theory, let alone having the expectation that a decent CS degree will give you exposure to it.

As an example, the University of Chicago, my alma mater, doesn't expose CS students to Category Theory. Of all the places it might be where you's expect it the most. It has a notoriously "theoretical" computer science department tied closely to the mathematics department and _is where Category Theory was invented_.

Even in a math degree, one wouldn't typically touch on Category Theory in a classroom setting until one studied Algebraic Topology. Before that students might come across it as a neat sideshow, but nothing they'd be interested in using to solve an actual mathematical problem. There are plenty of students who get a BS in mathematics without ever making a serious go at it.

It's also not really _that_ useful as a first-order field of mathematics. It's mostly useful as a way of organizing other mathematical things and as I kind of general vernacular. Even mathematicians sometimes lovingly call it "abstract nonsense".

I see CT as a kind of mathematical interstate system. If you want to go back and forth between Chicago to LA it's fantastic, but most of your work is being done in Chicago or LA itself. The interstate system itself isn't _that_ interesting most of the time.

<comment><USER11> Yes, I agree: you can easily do a bachelor's degree in either mathematics or computer science without ever being taught category theory (though the chances are you'd hear it mentioned a few times).

More specific evidence to go with _jfarmer_ 's: I read mathematics at the University of Cambridge -- like UChicago, an absolutely first-rate institution and not one that gives its students an easy ride -- and there category theory is not part of the undergraduate curriculum. It is one of the courses you can take in "Part III", which is a one-year taught master's degree[1].

[1] Until very recently, the best way to describe it was "a one-year taught master's degree that inexplicably isn't actually a master's degree". But they've fixed that now.

</comment></comment></comment></comment><comment><USER12> I thought Hask was supposed to be the category of Haskell types with functions between them. My Haskell is pretty rusty... how exactly does (->) resolve to this meaning?

Also, on a more theoretical level, since Haskell types themselves belong to the category Hask, by definition, to what extent can Haskell truly model category theory when the objects of a category are evidently being represented here as Haskell types?

<comment><USER13> 1\. Hask has Haskell types as objects, and functions between them as morphisms. "a" is a type, "b" is a type, and "a -> b" is an arrow between them.

2\. Haskell cannot model (the entirety of) category theory. Everything in Haskell is in Hask, and the only way to get to an entirely different category (say Set) is on paper. This great talk gives some insight into the issue: [https://vimeo.com/67174266](https://vimeo.com/67174266)

</comment></comment><comment><USER14> I would like to give a word of encouragement to anyone who, like me, thought this insight into Monads was all greek. Understanding any of the material in this tutorial is not a prerequisite to writing practical computer programs in Haskell.

You don't have to completely understand Monads in order to use them.

Just as you learn adding two numbers together in pre-school and learn about axioms of associativity and commutativity later, you should start by learning IO first and think about the more general ideas later.

Nevertheless, this was quite an interesting article. Maybe I'll try to put some thought into it and see if I can learn something that would help me level up my Haskell skills.

</comment><comment><USER15> I love it! I just wish I understood it.</comment>


Show HN: Foundbite, a sound-based social network for Android - <AUTHOR> http://foundbite.co

<comment><AUTHOR> Hi HN,

I'm the founder of Foundbite, we're a two person team working out of London building an app that revolves around capturing, sharing and exploring sound!

There are tonnes of photo and video apps, but very few for the sounds we hear around us every day even though they play a big part of the atmosphere and feeling of a place or experience.

As well as having the normal feed, profile and following Foundbite also has an interactive soundscape/ map of the world that allows you to find a place you're interested in and listen to the sounds that have been recorded there.

People have been using Foundbite to capture the sounds of nature, sports events and music. Here are some great examples: Street Music in New Orleans: [https://foundbite.co/8408101742](https://foundbite.co/8408101742) Bullet train in Taiwan: [https://foundbite.co/3188702070](https://foundbite.co/3188702070) Lighthouse on Rhode Island: [https://foundbite.co/6846855499](https://foundbite.co/6846855499) A mountain stream in Italy: [https://foundbite.co/9459134084](https://foundbite.co/9459134084)

Yesterday, we launched on Android ([https://play.google.com/store/apps/details?id=com.foundbite....](https://play.google.com/store/apps/details?id=com.foundbite.droid)) but we also have a Windows Phone app ([http://windowsphone.com/en- us/store/app/foundbite/243343f7-3...](http://windowsphone.com/en- us/store/app/foundbite/243343f7-36e4-454d-8bbd-b15d447037ab)) and currently working on iPhone.

Would love to get some feedback, and chat about the idea with anyone who's interested!</comment>