Show HN: EdgeDB – Next generation database - <AUTHOR> https://edgedb.com/blog/edgedb-1-0-alpha-1 <comment><USER2> I'm a pretty experienced Data Engineer working at one of the bigger data companies around. I read your copy and landing page, here's what I took away:

1) New database product that's an alpha. My first thought is "will be interesting to follow for 5 years to see if they make it". database choice is possibly the single biggest decision I make in my role; I don't make it lightly.

2) It looks like it's trying to bridge the NoSQL/SQL gap -- more consistent for SQL fans, more flexible for NoSQL fans. (My judgement: "serving 2 masters is not a great signal for me to choose this DB" but I'm a SQL curmudgeon.)

3) GraphQL is really neat out of the box. Some neat features overall.

And, finally, I'm still not really sure what situation could possibly arise that would make me choose this DB, mostly because that choice isn't something I'd select a <5 year old solution for.

I do like that it's based on Postgres (Citus taught me how powerful the "build on pg" model can be). I really like the graphql and schema introspection features. I don't think I would put this on my vendor/technology comparison matrix.

Hope this is helpful, please don't read too much into the negative stuff -- if I were choosing any other technology pillar besides a db, I wouldn't be nearly as strict. It's an uphill climb for your team, but I wish you the best of luck!

<comment><AUTHOR> Thank you for your insightful comment!

I agree, people do choose a database based on its track record, and this is where EdgeDB will have to prove itself. Overall, I think that the decision to build on PostgreSQL is a great competitive advantage over other new databases: we can confidently say that the data will be safe with EdgeDB. And we can actually spend more time improving the ergonomics of the data model and the query language.

<comment><USER3> Since you actually forked Postgresql, how is it realistic to assume that the data will be safe?

<comment><AUTHOR> We maintain the number of modifications to the minimum, and we don't touch critical paths (there's simply no need for that).

</comment></comment></comment></comment><comment><USER4> I really like the re-imagination of the type system, data model, and the user interface. This is an area where databases could be significantly better than they currently are.

My only concern is that there is nothing about the backend that suggests that this will be scalable beyond tiny data sets. Even on single machines, there is a performance cliff that arrives very quickly when doing these kinds of data model traversals unless you are using something exotic under the hood.

<comment><AUTHOR> > I really like the re-imagination of the type system, data model, and the > user interface. This is an area where databases could be significantly > better than they currently are.

Thank you!

> Even on single machines, there is a performance cliff that arrives very > quickly when doing these kinds of data model traversals unless you are using > something exotic under the hood.

PostgreSQL is used under the hood. We compile EdgeQL to an efficient SQL behind the scenes, similarly to how you compile a high-level language to lower-level language. All our fancy data types fully utilize the relational model, so what's achievable with scaling Postgres will be achievable with EdgeDB. I know quite a few massive deployments of Postgres that work just fine, there's simply no reason why EdgeDB would be different.

<comment><USER5> Can you point an EdgeQL 'server' at, say, an Amazon RDS Postgres cluster?

<comment><AUTHOR> No. Some of the features required us to slightly modify Postgres, but we'll be trying to upstream the changes.

<comment><AUTHOR> > An occasional post tracking the status of that upstreaming effort would be > very welcome.

We've already found a few bugs in Postgres and upstreamed fixes for them. There is a patch or two from us in their review pipeline right now.

<comment><AUTHOR> > May I assume that the slight modifications you were required to make to > postgres in order to support your features weren't just bug-fixes?

Yes. Some of them, like strict parsing of timezone-aware vs naïve datetimes can be upstreamed (as a configurable option) if the Postgres community accepts it. It's in our interest to upstream things like that :) We'll see how it goes.

</comment></comment></comment></comment></comment></comment><comment><USER6> So, EdgeQL is closer (than SQL) to relational algebra as it was originally envisioned? Did Tutorial D have any influence on the language and the system?

<comment><USER7> You got it! We designed EdgeQL from the first principles of the relational algebra and the set theory, with focus on practicality. There are a lot of similar ideas in Tutorial D, but EdgeQL is not a descendant of it.

<comment><USER6> Well, NOW I am curious. And I do a lot data engineering these days ;-)

btw, do you implement any edgeql-specific optimizations? I know that the db shares most of the code with psql, and its optimizer is okay. But being closer to the math foundation opens up lots of opportunities..!

<comment><USER7> > btw, do you implement any edgeql-specific optimizations?

Not on the planner level. Most of EdgeQL-specific low-level stuff is currently built as an extension with helpers. Our strategy is to upstream as much as possible, including any possible improvements to the planner that benefit EdgeDB.

</comment></comment></comment></comment><comment><USER8> > EdgeQL has no NULL.

So how is a missing value in a column represented? A column value is a scalar value, not a "set". As much as I accept that NULL values are a pain to deal with, I don't see a way to represent the absence of information in another way. You can't store an "empty set" in a column.

A NULL _value_ in SQL is something completely different than a NULL reference or pointer in a programming language. Although both have the same name the underlying concept is completely different.

> Functions in SQL can return NULL to signal an error condition.

That's plain wrong. functions in SQL signal errors by throwing an exception

<comment><AUTHOR> > So how is a missing value in a column represented?

Explained in this thread: [https://news.ycombinator.com/item?id=19641060](https://news.ycombinator.com/item?id=19641060)

> > Functions in SQL can return NULL to signal an error condition.

> That's plain wrong. functions in SQL signal errors by throwing an exception

Well, just as an example: take a look at the 'to_number()' function. It will happily return NULL if the format string is an empty string. An empty format is an error condition (it's not documented to return NULL, btw, see [1]) at least in some contexts, so the returned NULL breaking the query can be very unintuitive.

So yes, throwing an exception is what happens in 99% of situations, until you hit a weird case where a NULL is returned unexpectedly for you. And I'm not saying that returning NULL is bad for 'to_number()'; in SQL that's OK. It's just not how we approach things in EdgeQL.

[1] [https://www.postgresql.org/docs/11/functions- formatting.html](https://www.postgresql.org/docs/11/functions-formatting.html)

</comment></comment><comment><USER9> So this is basically a fork of Postgresql? Interesting. Why not just have an extension? I do like the idea of object oriented queries (without having to go through ORM), and simplified JSON usage. Lots of good ideas! I'm excited about this project.

Btw I suggest coming up with a better name than EdgeQL. Maybe just EQL?

<comment><AUTHOR> > So this is basically a fork of Postgresql? Interesting. Why not just have an > extension?

EdgeQL as a language is so intuitive and advanced only because it operates on a very specific data model that can be compiled to a relational model, but isn't really compatible with it. So naturally, EdgeDB wants to be in absolute control of the underlying Postgres backend, which goes beyond of what's acceptable for a Postgres extension.

> Btw I suggest coming up with a better name than EdgeQL. Maybe just EQL?

There are other languages called EQL (at least that's the impression I got when I last googled it), whereas EdgeQL is unique :)

<comment><USER9> Just go with EQL the other EQL people will understand.

</comment></comment></comment><comment><AUTHOR> Hello HN!

This is a blog post about EdgeDB 1.0 Alpha 1. It's a new object-relational database, focused on type safety, usability, and developer ergonomics. We hope you will try it out!

<comment><USER10> I like this a lot! What a great evolution of Postgresl/SQL. I also love what you do for the Python/asyncio ecosystem. Congratulations on the release!

Their is one point I'd like to address, though. It's still a real pain to write queries from within other programming languages. You either write them within strings, which sucks, because you lose all support from the editor (highlighting, auto-formatting, auto-suggestions, etc.), or you use an ORM, which I think is not a good solution to the problem. If you want to boost "developer ergonomics", make editor plugins for EdgeQL first class citizens. My 2 cents. :)

<comment><AUTHOR> Thank you!

> You either write them within strings, which sucks, because you lose all > support from the editor (highlighting, auto-formatting, auto-suggestions, > etc.), or you use an ORM, which I think is not a good solution to the > problem. If you want to boost "developer ergonomics", make editor plugins > for EdgeQL first class citizens. My 2 cents. :)

Yes, that's exactly what we are going to do. We'll obviously add a query builder to edgedb-python and other language bindings. We also plan to reflect the DB schema to your language of choice so that the query builder is fully typed. Then IDEs will have no problem with auto-completing your code.

Another thing we're considering is implementing the Language Server Protocol for EdgeDB.

<comment><USER10> That sounds awesome! :)

</comment></comment></comment></comment><comment><USER3> Congrats on your launch. While I like the idea, I believe that it would take at least a couple of years to convince people trying out in a real production environment. This is such a competitive space and companies prefer staying in the safe zone.

I'm sure you would need to cut off many features but I would start small and release it as a Postgresql extension. Even better, you can sit in front of Postgresql and develop your own client libraries for DDL and queries. That way, you could support RDS and Cloud SQL so companies can confidently try out EdgeDB. Once you prove that the new query language works, you can extend and build a new database from Postgresql.

</comment><comment><USER11> I got lots of question, got bit excited while glancing thru docs.

What is the minimal system requirements? And what about partitioning, replication, event log, etc.

Is there a native way hook to graphql resolvers? I really like it either way, there some tools for composing and stitching schemas.

plans for hosted and managed solution?

Did you consider "pageInfo" type, maybe it is just me, but it proved very useful on frontend.

instead of `Movies: [Movie]`

        Movies {
          pageInfo: PageInfo # { hasNextPage, endCursor }
          nodes: [Movie]
        }

<comment><AUTHOR> > What is the minimal system requirements?

Minimal requirements are similar to that of PostgreSQL. EdgeDB should run comfortably on an average server.

> And what about partitioning, replication, event log, etc.

Tooling for that will be coming in the next few alpha releases.

> Is there a native way hook to graphql resolvers? I really like it either > way, there some tools for composing and stitching schemas.

Not right now. Although I think it should be possible to do what you want via one of the existing GraphQL proxies, like Apollo.

> plans for hosted and managed solution?

Yes, but no ETA right now. This is one of our highest priorities.

> Did you consider "pageInfo" type, maybe it is just me, but it proved very > useful on frontend.

We can add that. One of the first-priority things on our list w.r.t. GraphQL though is to implement Database Views, which will allow selectively exposing your schema (and controlling _how_ it is exposed).

<comment><USER11> This is really good to hear. Thank you

</comment></comment></comment><comment><USER12> Is it correct to say that EdgeDB is an attempt to make Postgres better in terms of query language and abstractions? By "better" I mean the idea of being more flexible and convenient for users and generating efficient SQL for them?

Could you clarify why the benchmarks show lower performance than vanilla Postgres? Is it because of less efficient generated SQL? (I am not a database engineer and probably missing something obvious)

<comment><USER7> Yes, one of the main goals of EdgeDB is to provide improved ergonomics and productivity through a modernized data model and query language. And do that without a significant performance cost.

Lower performance in benchmarks is explained by an I/O bottleneck in the EdgeDB server, the queries are not less efficient. There will be performance improvements in subsequent releases.

</comment></comment><comment><USER13> Congratulations on launching. I've been following EdgeDB after finding about the company behind uvloop. Excited to test this out on a side project.

<comment><AUTHOR> Thank you! Please share with us how it will go.

</comment></comment><comment><USER14> Every time I see a new database announced I expect to see the database of my dreams: a simple JSON store that runs arbitrary computations and store these so they can be queried fast (and reruns just what's needed when some piece of data is updated).

Still nothing so far.

<comment><AUTHOR> You can do this in EdgeDB. The fact that the data is normalized doesn't really matter, because you can request it as JSON.

<comment><USER14> Can you automatically compute and store computed data?

</comment></comment></comment><comment><USER15> Which benefits does EdgeDB have over Hasura which isn't built on top of PostgreSQL but as a layer that runs on top of it and exposes a GraphQL API along with access control, event triggers and other features?

<comment><AUTHOR> The main advantages are EdgeQL and the Data Model.

EdgeQL is a fully featured query language, with support for subqueries, aggregation, transactions, rich datatypes like tuples/named tuples, arrays, JSON, etc.

The Data Model is object-relational, with support for multiple inheritance of object types and links between them. GraphQL really shines when it's used to query EdgeDB, it just feels natural.

That said, GraphQL can only do relatively simple queries, like "fetch me a hierarchy of objects with this shape with some basic filters". EdgeQL, on the other hand, allows you to build queries as advanced as what you can do with SQL.

Finally, there's only one schema in EdgeDB. You don't need to maintain a separate GraphQL schema, like you need to with hasura and other GraphQL solutions.

</comment></comment><comment><USER16> You started out with the claim that your database is fast, but you added almost no info to support this claim. Don't get me wrong but if you remove the initial claim the article would be more trustworthy.

<comment><AUTHOR> The benchmark results are right in the blog post.

</comment></comment><comment><USER17> I wonder how this stacks up against a dedicated graph database like Neo4J.

Are arbitray-depth graph queries a design goal, or are the nested results just for optimizing network traffic?

<comment><AUTHOR> EdgeDB is not a graph database, i.e. we didn't build it for traversing super deep schemaless graphs.

On the contrary, we optimized it for the kind of applications that are usually built with relational or document databases. Those scenarios frequently involve a relatively complex schema with queries that fetch hierarchies of objects 2-3 levels deep. Doing that efficiently in SQL or NoSQL isn't as simple as it sounds (and the blog post makes a point about that).

That said, I expect EdgeDB to perform on par with neo4j on moderately deep object hierarchies. We also plan to add support for recursive queries at some point, although it's not a priority right now.

<comment><USER17> Thanks for the answer.

> queries that fetch hierarchies of objects 2-3 levels deep [...] efficiently > in SQL or NoSQL isn't as simple as it sounds

It doesn't sound simple and it _isn 't_ simple. :-)

</comment></comment></comment><comment><USER18> I love the class of databases that are built on top of Postgres. It goes to show how far people can extend Postgres. I hope others can follow suit.

<comment><AUTHOR> Postgres is an amazing foundation to build on!

</comment></comment><comment><USER19> To return your challenge to Django, do you have an easy way to compose queries as neatly?

``` issues_query = Issue.objects.filter(location__contained=bounding_box) if all_of: for tag in all_of: issues_query = issues_query.filter(issuetag__tag=tag) if any_of: issues_query = issues_query.filter(issuetag__tag__in=any_of) ```

<comment><USER7> We are working on a similar query builder syntax for edgedb-python.

<comment><USER19> How will the api look like?

<comment><USER7> Similar in spirit to Django and SQLAlchemy Core SQL expressions.

</comment></comment></comment></comment><comment><USER20> What are the chances that in the future, EdgeDB can upstream their changes to Postgre so that EdgeQL could be run on top Postgre as an Extension rather than current way of basically forking the whole Postgre?

I really like some body is finally doing something to "SQL" in terms of the language itself. I hope this catches up and other will consider doing something similar.

</comment><comment><USER21> Not sure if this is correct, but seems like only client APIs are Python or hitting the GraphQL API for anything else. Obviously it's an alpha so not sure what baseline one should set but it's a complete non-starter for me even to play with until it proves it can / will interoperate with a broader range of clients (eg: JDBC).

<comment><AUTHOR> We want to provide high-quality bindings for other languages as soon as we can. In the meantime both EdgeQL and GraphQL can be exposed via an HTTP endpoint.

</comment></comment><comment><USER22> For someone looking for a GraphQL solution, which benefits does EdgeDB have over [https://github.com/graphile/postgraphile](https://github.com/graphile/postgraphile) ?

<comment><AUTHOR> Key benefits: easier data model to deal with, integrated migrations, using EdgeQL (not SQL) to write advanced queries and expose them as GraphQL views. In the future we'll have integrated access control layer, but that's a subject for another great blog post :)

<comment><USER23> Regarding the migrations, any more information on that?

<comment><USER7> The reference page has some details [1]. Our goal is to match and exceed the experience of managing the migrations with a high-level ORM like Django. This is one of the reasons behind the SDL/DDL duality. EdgeDB has to grow some tooling around it before we get to that point, though.

[1] [https://edgedb.com/docs/edgeql/ddl/migrations](https://edgedb.com/docs/edgeql/ddl/migrations)

<comment><USER23> Any plans on making migrations being able to rollback as well?

</comment></comment></comment></comment></comment><comment><USER24> How do you think you compare to Cockroach? Besides the query semantics, I'm curious about the depth of the thought around cross region / cross continent and horizontal scaling concerns?

<comment><AUTHOR> EdgeDB is based on Postgres. There're ways of scaling it as Citus Data has shown, and there's a lot of ongoing work in PostgreSQL itself to improve the scalability. We'll be using that as well as actively contributing to further improve it.

</comment></comment><comment><USER25> The database itself looks neat, but wasn't at all what I expected from the name EdgeDB. At the moment, "edge" usually refers to edge computing, i.e. running code in many data centres around the world, near to their users.

I expected EdgeDB to be a database suitable for edge computing, namely able to run globally, near the edge compute nodes, and replicate data with eventual consistency.

<comment><USER26> Me too. If anyone has any pointers to such a database, I'd be interested. I have some devices that are looking like they'll need a custom filesystem based key store, but others running linux with a need to sync the two types of devices.

<comment><USER27> Realm is probably one of the most well known.

</comment></comment></comment><comment><USER28> Is there any plans to support full-text search?

<comment><USER7> Yes. EdgeDB actually used to have it earlier on, but we felt that the design wasn't good enough, so we dropped the FTS support for now. The idea is to have a datamodel/QL UI such that we are not just copying the Postgres FTS design, so things like a different search index backend (e.g. Elastic) are possible.

</comment></comment><comment><USER29> Are there any plans regarding GIS?

<comment><AUTHOR> Yes, we will have it (via abstracted away PostGIS)

</comment></comment><comment><USER30> Free

</comment><comment><USER31> The folks at magicstack have written some seriously cool libraries. Lots of neat cython stuff, postgres, libuv, http parsing. I really like reading their code.

<comment><AUTHOR> Thank you :)

</comment></comment><comment><USER32> Why would you replace nulls with empty sets?

Now you can't tell if it's empty or just not set.

Now I have to track another thing just to tell me if I have touched a thing.

Please don't tell me this is some kind of nulls were a mistake cargo culting?

Nulls are just option types with good usability. Not the languages fault if you decide to put a tripple meaning on it.

<comment><AUTHOR> > Now you can't tell if it's empty or just not set.

Those are the same things in EdgeDB. You can have a "required link", in which case a relationship between two objects is required, and therefore the link cannot ever point to an empty set. Alternatively you can also declare your link as "optional", in which case you have it pointing to an empty set (essentially to "nothing"). You can coalesce that empty set with a non-empty set, if you need.

That is similar to SQL; the key differences here are:

* The boolean logic in EdgeQL is two-valued, it's "true and false", not "true, false, and NULL". Some people find it hard to guess what "true OR NULL" returns in SQL, because unless you are a DBA, this isn't something you deal with on a daily basis.

* Functions/operators in EdgeQL never return empty set to indicate an error condition. This is one of my personal pet peeves with SQL; in some situations it's hard to understand why a big and complex query yields slightly wrong results _sometimes_.

> Now I have to track another thing just to tell me if I have touched a thing.

I suggest you to try writing EdgeQL queries. The experience is usually quite the opposite, and I hope you'll end up liking it!

> Please don't tell me this is some kind of nulls were a mistake cargo > culting?

While I do think that NULL is a hard to deal issue in many languages and I admire languages like Rust without NULL, EdgeQL is slightly different. As you noticed, we do have a concept of empty set, which in some ways is similar to NULL in SQL (with the caveats I explained above).

<comment><USER33> I may not be following correctly, but if a Boolean field doesn't support null, how would you handle a use case like: a survey question that hasn't been answered, vs setting as false by default? Would you use an enum with 3 states vs a Boolean type?

<comment><AUTHOR> This goes a bit beyond what fits in a comment reply, but I'll try to be brief.

\- You can define a `bool` property as "optional" and then it can have an empty set as its value.

\- The 'exists' operator coverts sets into bools; you can use it with the 'if..else' operator to handle empty sets where needed.

\- You can coalesce (the "??" operator) your potentially empty property with a non-empty set, say a "{false}".

\- An empty set isn't some magical value like NULL in SQL. Functions and operators in EdgeQL are strictly defined if they accept sets as their arguments, or not. In the case of logical operators -- they don't; they are defined as element-wise operators, therefore no three-valued boolean logic. You learn the rules once and they apply to all functions and operators in all contexts.

This documentation page (and subsequent sections) explains how exactly EdgeQL is defined: [https://edgedb.com/docs/edgeql/overview](https://edgedb.com/docs/edgeql/overview).</comment></comment></comment></comment>


Advanced Shadowing with Isotope - <AUTHOR> https://www.youtube.com/watch?v=MHHGE9aBJGc <comment><AUTHOR> Shadow testing using traffic mirroring is something we did extensively at Twitter when we built Diffy but many problems remained unanswered about this approach to test automation and observability. e.g. How do we safely replay write traffic? How do we deal with downstream dependencies in general? Can we document our functional interfaces automatically? Can we observe the data patterns flowing through our code in production?

Isotope answers these problems and introduces the most advanced form of shadowing built so far.</comment>


Antarctica warming breaking records - <AUTHOR> https://www.newsweek.com/air-antarctica-hot-suddenly-breaking-records-1464070 <comment><USER2> Antarctica is a beautiful and exotic place. If Earth warming trends continue, perhaps Antarctica will become more hospitable and open up to residential and commercial development.

It would be a _terrible_ thing globally, the situation especially near the equator would be far worse (flooding, storms, heat waves). But at the same time - it's interesting to think the people of the future might have green grass lawns that look out towards unnamed mountains never seen before. A whole new ecosystem and a torrent of discoveries might be found within this landscape.</comment>


Haskell web programming (a simple tutorial) - <AUTHOR> http://yannesposito.com/Scratch/en/blog/Yesod-tutorial-for-newbies/

<comment><USER2> I'm having a hard time understanding the intent behind "You shouldn't need to know Haskell". Do you mean in order to complete the tutorial and understand what Yesod does? That's probably true. Being someone who's been studying and using Haskell in personal projects for the past maybe 10 months or so, Haskell's syntax and fundamental design decisions (while spot on, in my book) are different enough from most programmers' skill sets that I don't think they'll make it very far writing a non-trivial Yesod app without knowing Haskell. I'm fairly proficient/enamored with Haskell now but I'm certain I would have crashed and burned if I got my start trying to learn Haskell from Yesod.

My first dose of Haskell was attempting to write an XMonad config. That turned out to be a frustrating, several hour long ordeal for me that had me write off Haskell as undecipherable, alien hieroglyphics for quite some time after that. I'd urge anyone reading the tutorial who likes what they see to put in the hard time to get a handle on the language, understand what its good for and what it isn't and then return to web programming with it.

<comment><AUTHOR> I am sorry if I didn't made it clearer. Of course, you don't need to know Haskell to follow this tutorial in particular.

But it is clear in my mind, you must know Haskell to do something useful with it. And in particular, make a web application, even if using a web framework.

As you stated, Haskell is not a language you could learn in 3 hours like I learned the bases of Python. It takes a very long time to learn, particularly when you're not used to functional programming.

But an intent of this tutorial was to promote Haskell. Clearly, if someone want to do something a bit more useful, he will very soon realize he must understand Haskell.

I added a specific advice in my conclusion similar to yours and pointing some essential resources.

</comment></comment><comment><USER3> Is there performance benchmark comparing Haskell and Erlang? I hear Erlang is also quite performant when handling a large number of requests, but no hard numbers. Are there similar web frameworks for Erlang as Yesod for Haskell?

Also, what about Clojure? That runs on the JVM but how does it fare on the "correctness" claim of Haskell, i.e., "If it compiles it's close to what the programmer intended".

I'm learning Haskell right now and really loving it. I wonder if I should stick with it for more production projects or make a switch to Erlang/Clojure. I know Erlang has quite a few success stories behind it, CouchDB being one of them.

<comment><USER4> I think it would be uncontroversial to say that optimized GHC Haskell (statically typed, compiled, optimized, native code) is always faster than current Erlang implementations (dynamically typed, interpreted, lightly optimized).

For example, the fastest Erlang programs on the benchmarks game are all slower than all the GHC Haskell programs,

* [http://shootout.alioth.debian.org/u64q/benchmark.php?test=al...](http://shootout.alioth.debian.org/u64q/benchmark.php?test=all&lang=ghc&lang2=erlang)

Median slowdown relative to GHC is 7x.

GHC is aiming for the high performance computing world of late, as well. The Erlang language just doesn't have the facilities for raw number crunching that's needed.

<comment><USER5> Erlang is interpreted. Since this is not about a simple CGI application a benchmark in which you compare the running time of a compiled executable with the interpretation and running time a another program it doesn't make a lot of sense.

Also this performance usually isn't the best way to argument about languages for web applications. Often your bottleneck is somewhere else and i you really have resource intensive stuff why not just interface C?

<comment><USER6> >Erlang is interpreted

No it isn't, it is compiled to bytecode and executed on a virtual machine, just like most scripting languages are.

>Since this is not about a simple CGI application a benchmark in which you compare the running time of a compiled executable with the interpretation and running time a another program it doesn't make a lot of sense.

Read the site, this concern is explicitly addressed. The tests are long running, not short tests repeated a million times. So the byte code compilation step ends up too small to notice. That is why using a PHP bytecode caching plugin doesn't increase PHP's performance on the shootout, and why java isn't dead last from the massive startup time of the JVM.

>Also this performance usually isn't the best way to argument about languages for web applications

This is a myth. Very few web applications actually have one CPU intensive portion that can be moved out into a C module. Most web apps have pretty flat profiles, where overall language speedups make the biggest difference. When 90% of your time is spent in 10% of the code, a C module is a reasonable option. When 90% of your time is spent in 90% of your code, your only option is a faster language.

</comment></comment></comment></comment><comment><USER7> From the bit of Haskell I've managed to learn, it's a great language, and I can see how an awful lot of the errors that you can make in other languages will probably be avoided in Haskell, but I've never understood the (frequently repeated) claim that

"If your program compile it will be very close to what the programmer intended".

There's nothing I've seen in Haskell, or any other language, that will prevent you implementing the wrong algorithm, or the wrong business rules etc. If you're doing a phsyics simulator (to take an example I've played around with), Haskell is not going to stop you badly miscalculating the gravitational effect of multiple objects, or calculating the mass of an object incorrectly.

<comment><USER8> That's the difference between an "intentional bug" and an "unintentional bug".

The former is when you want the program to do X, and X itself is wrong. Sure, defending against this is rarely possible. The latter is in my experience far more common. When you want the program to do X, but you actually told it to do Y.

Haskell catches most of the latter cases.

One of the main reasons is the functional purity. Because return values are the only result of a function, and these values are type-checked against a specification, it is much harder to do the wrong thing.

For example, an imperative program will idiomatically have methods whose sole effect is mutating their arguments or the objects. Even with a type system -- if you just omit the effects, the compiler really has no way to know that you meant the effects to happen, and you will simply have a bug. A purely functional program will idiomatically use a different style. Instead of mutating arguments and objects, it will return a new value. If you forget to return a new value, the compiler will catch that.

Additionally, the types in Haskell are generic by default. The more generic the type of a function, the more _restricted_ is what that function can do. The restrictions narrow the space of wrong things you can do.

Another interesting property of Haskell is where it chooses to be on the "restrictiveness+guarantess vs. powerful+unguaranteed" trade-off.

It seems many don't even notice that the other side of the "power" coin is "guarantees". The more powerful a construct is -- the less you can say about what it will do. In Haskell, the purity and generic types of functions are two things that heavily restrict code. But there are many other mechanisms to _restrict_ rather than empower code. In many contexts, restrictiveness is considered a bad thing, but in Haskell, this restrictiveness is very useful for reasoning about code.

For example, the function of type: (forall a. a -> a) in Haskell can _only_ be the identity function. It cannot print "hello world", and it cannot mutate variables. These restrictions are useful because we now can have useful laws. For example: id . f = f . id = f. And we can derive these useful laws from the type itself, without even looking at the code!

By helping us find the most restrictive sub-language that can express the solution, we rule out even more bugs.

<comment><USER9> The power/guarantee trade off isn't a fundamental one, even though in current languages you often have to choose between the two. Creating languages that combine the power of untyped languages with the guarantees of static languages is an active field of research. For example, dependently typed languages, while currently too unwieldy for everyday use, show promise of having both power and guarantees much stronger than Haskell. Now a third, ill-defined axis of "usability" comes into play.

<comment><USER10> "The power/guarantee trade off isn't a fundamental one, even though in current languages you often have to choose between the two."

It is on the Pareto frontier of the two. Which verges on tautological, but it's still a useful observation. If you can trivially extend the power of a construct while maintaining the same guarantees, the construct was simply broken, and extending the guarantees of a construct without affecting its power means simply that the guarantees were excessively conservative in the first place.

(This can almost be copied and pasted any time anybody claims two properties of a program aren't in conflict, the most common probably being security and ease of programming.)

We may be wrong about where the frontier is... but I doubt we've _very_ wrong.

</comment></comment></comment></comment><comment> t-crayford Spelling corrections, noted as I read:

"point of vue" -> point of view

"shell command are executed" -> "shell commands are executed"

"benchmark are here" -> "the benchmarks are here"

Also, heroku works just fine with haskell, as long as you compile a static binary on an ubuntu machine. I compile mine on virtual box (via way of vagrant), and have 3 haskell apps (none of them really production ready though) running on heroku right now.

Lastly, I don't _really_ think haskell is the right language for a lot of web programming. In haskell, you really want your logic to be as purely functional as possible, but that gets pretty hard to do for web apps that rely on external apis.

<comment><AUTHOR> Thanks! Spell errors corrected!

When I said there is no heroku for Haskell it because it is not official. Furthermore, do you have to use a 64bit architecture when compiling (just curious)? I might try it myself.

And I even think that the argument against the number of Haskellers is in fact a force instead of a drawback. It filter passionate people.

By external APIs, you mean that it would be difficult to write a simple "connect to twitter/facebook" application? Could you tell us about any bad experience you had?

<comment> t-crayford Yeah, 64bit.

I never said anything about number of people...

External apis: nearly all the logic in my app leads to either talking to the db, or calling an http api (and haskell can't use https apis _that_ well, you have to use its bindings to libcurl). In general, it feels extremely difficult to pull my logic out of IO (but maybe I'm just too dumb to).

It's not that its particularly difficult, it's just that it's not a problem I think haskell is well suited to solve. I've used haskell to teach my self automated logic proofs (just resolution refutation), compression algorithms (huffman encoding and lz77), typechecking and breaking encryption (enigma). Every time university exams roll around, I learning them on paper. It's by far and away the best language I've used for stuff like that. It feels like web- apps with lots of IO aren't really a problem haskell is well suited to solve.

</comment></comment></comment><comment><USER11> Is anyone here using haskell or a similar funtional language for any web app?

Would be interested in known your experiences and opinion compared to something like django or rails.

How difficult is it to build something with a complex domain model?

<comment><USER6> >Is anyone here using haskell or a similar funtional language for any web app?

Yes.

>Would be interested in known your experiences and opinion compared to something like django or rails.

Rails is an abomination, so pretty much anything would be better than that. We rolled our own framework in scala. Despite initial resistance from some dynamic language proponents here, it is now the only thing anyone here will use for web development. Even the python fan won't touch django now.

>How difficult is it to build something with a complex domain model?

Easier than in dynamic languages. The problem with complex models is that humans can't hold all that information in their heads. Static typing lets us offload a bunch of important information to the compiler and have it error check for us as we go. Our biggest issue has been that scala isn't strong enough, I'd much prefer to be using haskell.

<comment><USER12> _> Our biggest issue has been that scala isn't strong enough, I'd much prefer to be using haskell._

I hear that. I learned Haskell because I thought it would help me understand Scala better (namely the type system), but Haskell spoiled me and I don't want to use Scala anymore. Only thing keeping me on Scala atm is Lift, a truly excellent rethinking of web frameworks. If I ever find the time I want to start porting Lift to Haskell.

<comment><USER6> I'm not a huge fan of lift, but the portion of it I am ok with (the templates/snippets side of things) is fairly similar to how you use snap. It may be worth it to work on helping to finish snap (or forking it if need be) rather than starting from scratch. Snap currently provides nothing for database access, so you have a blank slate there to copy mapper or record from lift (but those both suck horribly!).

<comment><USER12> Interesting, what aspects are you not a fan of (besides the persistence parts)?

<comment><USER6> The model side of things is certainly my biggest complaint. The other things are minor in comparison, but still annoying. I like the architecture of lift, just not some of the details of the implementation.

I don't like the way templates are forced to use snippets to load data, in particular as it results in snippets being dependent on the database, and thus not suitable for unit testing. The framework should be pulling the data into the view, which then passes it as an argument to the snippet to do whatever transformation on it. That way snippets are entirely self-contained and easy to unit-test. This is the sort of thing that I think would have ended up the way I want had it been written in haskell instead of scala just because writing small, self-contained functions is the typical haskell way of doing things.

I hate the very opinionated nature of the form handling, and the even more opinionated responses from the developers to questions like "how can I make a form that works normally and doesn't require a session". People are not bad for wanting forms to behave normally, and the "security" show the lift devs put on to justify the form handling in lift is insulting.

While I find templates acceptable, they are seriously missing out on type safety. I should get a compiler error if I create a template with invalid html. Ocsigen got this part very right (an ocaml framework/appserver). Not only do I not get this benefit for my markup, but lift itself generates invalid html on me when using virtually any of the included form generation methods. Very annoying.

Ideally what I want is a framework that gives me the type safety of yesod, but with the structure of lift. I hate faux-mvc web frameworks and find they make things more difficult rather than making things easier, and unfortunately yesod copies the typical rails clone structure. Lift doesn't make the mistake of trying to hide http from me, and gives me the simple and correct mapping of template to url, leaves the domain model in the model instead of cramming it into "controllers" that shouldn't be involved.

<comment><USER12> Thanks. Those haven't bothered me too much yet, but good to have on the radar. That's the first real critique of Lift I've seen, other than 'it's not MVC'.

As for Yesod, I was also thinking it would be nice to have the type safety and parallel/concurrent performance of Yesod with the templating and structure of Lift. Don't know Yesod well enough yet though. Too many side projects.

</comment></comment></comment></comment></comment></comment></comment><comment><USER13>
        You shouldn't need to know Haskell.

        Don't pay attention to all the syntax.

        If you are curious you can take a look at Applicative Functor.

This is a nice, whirlwind tour, maybe the opening benchmarks and jab at node are superfluous. All haskell tutorials hit the speedbump of how to bootstrap haskell understanding, where to make forward references and when to say "Trust me" but code shown here: do blocks, ($), <\- and return would be confusing if you didn't know _any_ haskell

<comment><AUTHOR> Should I add a link to this article[^1] for people confused by the Haskell syntax?

The opening is not really an attack. This is the way I myself started to be interested in Haskell web programming.

[^1]: <http://blog.ezyang.com/2011/11/how-to-read-haskell/>

<comment><USER13> Yah, maybe just add links at bottom: LYAH, Applicative Functor wiki page. Or maybe mention one syntax feature, IO's and do-blocks, and say "return" and "<-" just kick things into and out of IO's, loosely speaking. Just don't say IO's are like burritos ;-}

You could also say, for anybody that's looked at lisps, *ML, F#, scala, erlang, the syntax isn't too bad, and for anybody that knows rails, they'll recognize the same REST'y MVC layers/abstractions. I gather the target audience is folks using dynamically typed languages.

</comment></comment></comment><comment><USER14> Finally I know how to draw owls.

</comment><comment><USER15>
      Haskell web frameworks handle parallel tasks perfectly. For example even better than node.js

of course, node.js has no parallelism

<comment><USER16> I think what was meant was concurrency. Haskell's lightweight user-space threading runtime is modeled after Erlang's.

</comment></comment><comment><USER17> haven't finished reading the whole content, but the zoom in zoom out effects are very annoying.

<comment><AUTHOR> Are you talking about the zoom in when you click on some code block? I was thinking to remove them.

<comment><USER17> the Ctrl + , or Ctrl mouse scroll up or mouse scroll down

<comment><AUTHOR> For me, it takes only one to two second animation. After it stabilize. You can disable javascript and there won't be any CSS animation added.

<comment><USER18> for me, in chrome, it takes 8-10 seconds to stabilize. and looks really wonky

<comment><AUTHOR> I disabled animation for chrome.

<comment><USER19> Hey your site is really pretty. Would you mind licensing your CSS for personal use? I'd like to use for my blog (which I will start one of these days).

Very nice blog post. After working on a Python stack during my part time job (check out Apache libcloud), I can safely say that using a statically typed language is a necessity for any non-trivial project. Without a type system you offload the entire framework onto the programmer. To actually grok the project you have to hold the complexity all in your head instead of using the types as a guide and that makes getting started incredibly challenging.

<comment><AUTHOR> Thanks.

The source code of my website is here:

<https://github.com/yogsototh/Scratch/>

And even if not clearly stated, everyone is free to use it and steal from it. Of course, I'd appreciate to be mentioned.

If you want to have the full behavior you should copy not only the css but a bit of js.

More precisely:

      output/Scratch/css
      output/Scratch/assets/css
      output/Scratch/js/init.js
      + jquery

You should have a functioning website which will be iPhone/iPod friendly.

You certainly also need to get the layouts:

      layout/article.html
      layout/default.html

Unfortunately, I didn't finished to switch to HTML5, the current code is XHTML strict 1.0. You can have a preview of my HTML5 transition if you look at the html5 branch. But I'll have to duplicate a lot of CSS modifications since.

If you want any more information, simply mail me, I'll be happy to help.</comment></comment></comment></comment></comment></comment></comment></comment>


Yahoo Weighs Spinning Out Hadoop Engineering Group for $1 Billion Opportunity - <AUTHOR> http://www.readwriteweb.com/cloud/2011/04/yahoo-weighs-spinning-out-hado.php <comment><USER2> Yahoo! has quite a few impressive engineering teams that appear to be saddled down by terrible management. Maybe this can be a new business model for them - a high-end tech incubator.

Of course, they'd first have to stop habitually buying companies and letting them rot on the vine.

<comment><USER3> That's part of my issue with Yahoo. They have so many smart people who do awesome things, but they'd rather spend all day trying to become a media company.

</comment></comment><comment><USER4> Cloudera is pretty much dominating the commercial Hadoop market now. It has a huge marketing head-start and arguably better engineers, since it was founded by Yahoo's employees who thought they can do better by leaving. They also have more experienced support and training organizations, pretty much by definition (Yahoo's Hadoop group didn't do that).

I'm wondering how Yahoo spinoff is planning to compete.

<comment><USER5> >I'm wondering how Yahoo spinoff is planning to compete.

Well if I were the Yahoo spinoff, I'd do exactly what Cloudera is doing, then look for ways to improve on it. It's still a new field, a huge potential industry/problem domain that companies of every size and geography will probably have some demand for, either now or in the foreseeable future. There's plenty of room for multiple competitors, and it's still relatively early.

It's been pointed out on HN before that a reliable way to make money is not to do something new and groundbreaking that people may or may not want, but to look at what's already in demand and making money and find ways to do that competitively or better. Nothing wrong with what Yahoo is doing here in that respect, more power to em.

<comment><USER6> |It's been pointed out on HN before that a reliable way to make money is not to do something new and groundbreaking that people may or may not want, but to look at what's already in demand and making money and find ways to do that competitively or better. Nothing wrong with what Yahoo is doing here in that respect, more power to em.

I think the term is 'fast follower' and they exist in every industry. Remember all those local yogurt shops that popped up when Pinkberry became popular? Or when Red Bull became popular Monster and Rock Star came shortly after. I would also consider Samsung, Hyundai, Huaweii, LG, and a few others fast followers.

</comment></comment></comment><comment><USER7> I'm still a bit of the opinion that Yahoo isn't too far away from a yard sale across the board. I think they might be significantly more valuable in pieces.

</comment><comment><USER8> Original Wall Street Journal article:

Yahoo Mulls Spinoff for Hadoop Software Unit [http://online.wsj.com/article/SB1000142405274870472930457628...](http://online.wsj.com/article/SB10001424052748704729304576287343518337646.html)

</comment><comment><USER9> "We've been covering Yahoo's deepening interest in Hadoop over the past year. The latest development came last week when Yahoo joined the Linux Foundation. It was another signal of its focus on Hadoop and its further commitment to the data analytics technology."

What does joining the Linux Foundation have anything to do with deepening commitment in Hadoop? Supporting Linux makes sense if you're running it on thousands of machines, while Hadoop is a platform-independent (Java-based) stack.

<comment><USER10> Which primarily runs on Linux.

<comment><USER9> correlation != causality. There's probably a zillion other things that Yahoo! uses that also run on Linux. Maybe that has something to do with their support of Linux?

</comment></comment></comment><comment><USER11> What a great idea this would have been 3 years ago.</comment>


How MySpace Music could beat iTunes - <AUTHOR> http://www.thestandard.com/news/2008/04/04/how-myspace-music-could-beat-itunes

<comment><USER2> The idea that MySpace Music _might_ give my email address to any band that I buy a song from terrifies me to the point that I _won't_ buy anything from MySpace until I know that they won't do that. If I want a band to be able to contact me I will sign up for their mailing list.</comment>


How Booking.com Handles Millions of Metrics per Second - <AUTHOR> https://grafana.com/blog/2019/03/21/pro-tips-how-booking.com-handles-millions-of-metrics-per-second-with-graphite/ <comment><USER3> I'd recommend them switching from Graphite to Prometheus in order to resolve problems with high requirements for disk IO and storage space. Prometheus uses less disk IO and compresses time series data points better than Graphite - <USER2> bytes vs 4 bytes on typical data. Prometheus' local storage doesn't scale, but this may be resolved with long-term remote storage systems from this list [https://prometheus.io/docs/operating/integrations/#remote- en...](https://prometheus.io/docs/operating/integrations/#remote-endpoints- and-storage) . I'd recommend trying VictoriaMetrics, since it is the most cost-effective solution for storing huge amounts of time series data.</comment>


Ask YC: What do you use for functional testing? - <AUTHOR>

I've tried Selenium (core, rc and ide), but there were issues with the HTTP referrer not being set properly.
Watir looks nice, but Firefox 3 on linux does not seem to be supported, and I don't want to use Windows/IE just to test.
Anybody have any other suggestions? <comment><USER2> I've struggled with this myself. I've been unhappy with Selenium (perhaps my own setup) and haven't really found a suitable alternative. I don't care if I have to use IE - I'm fine with that - and I use Windows 2008. I've saved this thread hoping that someone will come up with good suggestions :)</comment>


Bezos probe concludes mistress' brother was Enquirer source - <AUTHOR> https://www.chicagotribune.com/business/ct-biz-jeff-bezos-amazon-leak-enquirer-mistress-20190212-story.html <comment><USER2> >Private investigators working for Jeff Bezos have concluded that the brother of the Amazon CEO's mistress leaked the couple's intimate text messages to the National Enquirer.

This is not surprising. People (Bezos) should know better than share information with anyone they don't want revealed. Besties always share salacious material, and given this, I suppose siblings do too.

Question of blackmail remains, I suppose.</comment>


Ask HN: Best place(s) to find datasets to practice ML and DM? - <AUTHOR>

Anyone know of good websites with collections of datasets that are good for practicing datamining and machine learning?
The netflix prize, www.netflixprize.com is one obvious one, but only 1 set.
Project Gutenberg is another where you can get large amounts of text. http://www.gutenberg.org/wiki/Main_Page
Datasets for medicin would be greatly appreciated. <comment><USER2> <http://www.cs.utexas.edu/users/sherstov/pdmc/>

</comment><comment><USER3> Would freebase work for what you need?</comment>


Ask HN: Should there be a monthly meta thread? - <AUTHOR> While I appreciate the fast nature of the front page and general trend towards talking about new things, I think it would be worthwhile to have one thread every month to talk about what went on.
Right now, I guessing it can be used for -
1. Asking questions that otherwise don't require a thread of their own.
2. Revisiting interesting posts of the month.
3. Reflection from community and individual members. (eg - Hey, I am working on porting vscode to ipad) <comment><USER2> Yes. I'm in. Would be fun</comment>


What do people prefer more Wi-Fi or 3G in iPad? - <AUTHOR> http://www.imakerz.com/blog/people-prefer-wi-fi-3g-ipad

<comment><AUTHOR> I think this title explains the significance of the Wifi and 3G Technology. So what do you say about it</comment>


New FAA drone rule is a giant middle finger to aviation hobbyists - <AUTHOR> https://arstechnica.com/tech-policy/2020/02/new-faa-drone-rule-is-a-giant-middle-finger-to-aviation-hobbyists/ <comment><USER2> I'm opposed to the proposed rules. The traditional aviation hobbyist isn't ignorant, incompetent or malicious. And the problem we've seen from some drone hobbyists, resulting in the problems that this rule stems from, are ignorant, incompetent, or malicious. And this rule isn't going to fix that. It might make it very slightly easier to identify the rule breakers, but I think that's specious.

But now, there are all kinds of private for-profit forces at play to become the rent-seeking gate keepers for internet connected, and thus approved, operations. Commercial operators will have no problem with this, whether $50 or $500 a month, it's just a cost of doing business. And getting hobbyists out of the way is A-OK with them.

It's terrible, but also an entirely predictable consequence of decades of FAA defunding, and the neurotic application of politics in aviation causing decades of delays in modernizing the ATC system. Politicians and bean counters have the political capital to make the new system, not FAA enforcement experts, pilots, and user experts (including hobbyists) of that system. It's yet another example of special interests with the money winning.

(inactive CFII)

<comment><USER3> There was a working group of stakeholders that worked hard for months on proposed LAANC systems and solutions that included the AMA and FliteTest and it was disregarded.

Discussion on YouTube suggest the NPRM was mostly written by non-pilot lawyers and lobbyists for Amazon and Verizon, ignoring the findings and suggestions of the working group. So a large segment of airspace users are ignored (fpv racing and freestyle drone pilots, photo drone pilots, RC airplane hobbyists, etc numbering a few million people and all the supporting industry and hobby shops, STEM robotics educators, DRL fpv racing on ESPN, etc). So this is why we are protesting.

Read more in depth on this topic: [https://fpvfc.org/](https://fpvfc.org/) and [https://content.dji.com/we-strongly-support-drone-remote- id-...](https://content.dji.com/we-strongly-support-drone-remote-id-but-not- like-this/) and [https://dronedj.com/2020/02/28/help-save-out-hobby- protest-a...](https://dronedj.com/2020/02/28/help-save-out-hobby-protest-at- faa-headquarters-in-washington-d-c/)</comment></comment>