The HTML5 drag and drop disaster (2009) - <AUTHOR> http://www.quirksmode.org/blog/archives/2009/09/the_html5_drag.html <comment><USER2> Just built a pretty significant business application with a UI based on HTML5 drag and drop. There were certainly a few quirks, but it was nothing compared to web development in "the old days". Without any library the whole DnD song and dance is implemented as part of a React component that's only about 100 lines of JavaScript.

There are two places likely to trip you up. One of them is as mentioned in the original article: it's hard to remember which events need to have their default prevented. At the moment we're calling `preventDefault` in `onDragOver` which is required to identify the element as a drop target, and `stopPropagation` in `onDrop` which I believe is to prevent many browsers' default attempts to navigate to a dropped url-like thing.

The other potential pitfall is that, for security reasons, the data can only be read in `onDrop`, not in `onDragOver`. If you have a string id you don't mind being downcased you can just use the hack of sending that through the data transfer types (eg. as type "id/12345"), with a "text/plain" fallback.

</comment><comment><USER3> > There are no less than seven events associated with drag and drop: > dragstart, drag, dragover, dragenter, dragleave, drop, and dragend.

> This seems rather a lot for a series of actions that can be accurately > described by the mousedown, mousemove, and mouseup events.

Mouse events also include mouseover, mouseout, mouseenter, and mouseleave.

The start/over/enter/leave events serve useful purposes, notably that you can offer UI cues to the user. For instance, if you drag an image onto an image search page that normally accepts text, the search box can become a giant drop target with "drop image here to search" text. And if you drag a non-image onto the same page, the page can check the MIME type and give the user UI that says that won't work.

<comment><USER4> The author gets into that in "dragthat and dragsomethingelse," where they talk about the problem with dragenter and dragleave behaving like mouseover and mouseout, events which fire many times in situations where you generally don't need them to.

</comment></comment><comment><USER5> I still don't get why there it no baseline HTML-only drag-and-drop functionality. It would cover a lot of cases with zero effort from developers:

      <dragarea group="groupX">
        <draggable value="abc">
        whatever you want here
        </draggable>
      </dragarea>

      <dragarea group="groupX" name="variableName">
      </dragarea>

You drag element from first dragarea and get variableName=abc on the server. Multiple values work same was as checkboxes.

Fancy events could be added incrementally, later.

</comment><comment><USER6> Honestly, if I were still a web developer, I would still use a library. The libraries greatly simplify things for the standard use cases.

If I could not use a library, I would look for the unminified code of a library and copy and paste it the relevant section.

Because ultimately, unless things go wrong and I need to dig deep, I have better things to do than mastering yet another complex web api.

<comment><USER7> Nothing you said is incorrect, nor do I behave differently (who doesn't utilise JQuery?).

That being said; it seems like a copout to ignore glaring issues while waving our collective hands in the air and saying "just use a library to abstract it!"

One of the reasons libraries are as popular as they are (particularly JQuery which does "nothing" except provide an abstraction) is because the standards are so weak and cross browser support so incomplete.

A lot of people think we're currently in a great place in terms of web- development. I actually disagree. We're still in a bad place, it just so happens that were were in an even worse place previously (e.g. IE6 era).

Is there any real motive in the standardisation community or web community in general to make the spec' good enough so JQuery is no longer required by 9/10 websites? Why isn't $('#something').click(); built in yet?

<comment><USER8>
        document.getElementById('something').addEventListener('click',function() {})

works on all browsers these days. But we're stuck in our habits.

I don't deny that web development is kind of crappy, but it's just in the nature of it - the reason why iOS and Android are easier to develop for is because there's really only one client for them. Personally, I'll take the multiple, open client approach over the closed one any day, warts and all.

<comment><USER9> One key difference is the jQuery version will gracefully do nothing if "#something" doesn't exist, while yours will throw "TypeError: Cannot read property 'addEventListener' of null".

<comment><USER10> Additionally, that's actually not equivalent to Jquery .on(), which is the recommended way to add event nowadays.

</comment></comment></comment></comment></comment><comment><USER11> If someone was using angular and wanted to use the native html5 drag and drop, I created a directive for it here [https://github.com/ganarajpr/angular- dragdrop](https://github.com/ganarajpr/angular-dragdrop) if you are interested.

There is documentation as well in here : [http://ganarajpr.github.io/angular- dragdrop/](http://ganarajpr.github.io/angular-dragdrop/)

Having read the blog ( from top to end ) I kind of agree with some of the things that the author says. There are a few edge cases that we need to take care of if using native drag and drop. There is also the not-so-slight issue that it doesnt work on mobile browsers....

</comment><comment><USER12> What about this[1] non-jQuery "old script"? Can we use it instead?

[1]: [http://draggabilly.desandro.com/](http://draggabilly.desandro.com/)

<comment><AUTHOR> This may be precisely what I, personally, needed for my web-app. I'll try it tonight and see how it works out, thanks for the link.

</comment></comment><comment><USER13> About prevent default: You need to tell the browser that _this_ element will handle the drop/has handled the drop, or it will perform its default operation. What is it's default operation? To load the file/URL that you just have dropped onto the browser window.

Or maybe you have multiple drop areas stacked on each other. Should the event propagate to the parent element or not?

Ok, why is it done in JavaScript and not simply with dropzone="true"? Because you want to be able to let script logic decide if that drop/dragover event is eaten or not.

I stopped reading at some point, but I wonder if the author has ever implemented D'n'D support in any other language. It works very similar to this in I guess most GUI APIs. Maybe that is another reason why it works like that in HTML5?

<comment><USER14> I sympathize with your argument, but unfortunately it doesn't seem to be correct.

Arguably, you should be able to implement just the 'drop' event and call event.preventDefault() in that event to suppress the default behavior of loading the file/URL. I don't see a clear reason why any other event should matter, the actual action that triggers the load is dropping.

As for propagation to parent elements, that's not controlled by event.preventDefault(), that's controlled by event.stopPropagation().

I do agree that the author doesn't seem to realize that drag & drop is actually rather complex and that other languages look similar (e.g. large number of events). But the quirks around when you have to call preventDefault() do seem to be legitimate issues.

<comment><USER13> Yeah, in other languages you don't use "preventDefault" but e.g. "acceptProposedAction". So they could have added such an method to the event object. I guess it's a bit of a quick fix thing why Microsoft didn't do that and the w3/whatwg just took that.

As to "I don't see a clear reason why any other event should matter". Well, yeah, one could implement it so that only the drop event really matters, but I guess it's done the way it is so it forces the programmer to explicitly handle the dragover event. This way the appropriate mouse icon will be displayed and the user knows what's going to happen when they drop now.

Strange situation where I, a Linux user, defend something made by Microsoft. Usually I spew a _lot_ of hate against Microsoft just for committing the crime called Internet Explorer (especially version 8, which I still have to support, bah).

</comment></comment></comment><comment><USER15> It really should only be used to drag and drop files from another application. If you just want to move elements around on the page mousedown, mousemove, and mouseup are all you need. I don't think we need a new API for this as it is pretty straightforward to roll your own.

<comment><AUTHOR> But future devs shouldn't have to "roll their own" when someone's done it before. Why not just rewrite the standard and make it sane? JQuery shouldn't be the de facto JS library, those features should be (and are being) integrated into HTML already.

</comment></comment><comment><USER16> One of the best and well explained issues with the HTML5 DnD.

The specs for DnD are written very well but the implementation is brilliantly fucked up that the only way you are left is to use a library and enjoy the hardwork of someone who spent hours getting smeared in the dirt.

</comment><comment><USER17> _> Web developers MUST NOT (in the sense of RFC 2119) use HTML 5 drag and drop. They should use old-school scripts instead._

As far as I remember if you do that (i.e. if you use mousedown, mouseover, etc and move your dragged element accordingly), the result won't be nearly as good as with "native" drag and drop, and you will see the dragged element lagging behind the cursor.

<comment><USER18> I've implemented drag and drop multiple times via jquery event handling and never had lagging issues.

<comment><USER13> Indeed. I also implemented it by hand once and once I did a bit of event throttling (which jQuery also does) it worked smoothly enough even on a very old Android 3 tablet (in Firefox). And the dragged element has opacity!

</comment></comment></comment><comment><USER19> @striking: Does this work for you?

[http://www.html5rocks.com/en/tutorials/dnd/basics/](http://www.html5rocks.com/en/tutorials/dnd/basics/)

[http://html5demos.com/drag](http://html5demos.com/drag)

I don't understand why this is an issue, there are tons of great tutorials: [https://www.google.com/search?q=HTML5+drag-and- drop](https://www.google.com/search?q=HTML5+drag-and-drop)

Step 1 to being a web developer, search Google first. More than likely anything you need done has been done 10,000 times and there are at least 100 tutorials.

Don't be afraid to use libraries such as jQuery,which HTML5 drag-and-drop doesn't require, but I see that you mentioned avoiding it in another comment. Libraries many times also have the advantage of increasing browser support transparently, which is generally a good thing.

Worried about loading time with libraries? Use a CDN such as [http://cdnjs.com/](http://cdnjs.com/), most users will have the popular ones already cached in their browser.

Use StackOverflow if you have questions, that's what it's there for. Hacker News is for... news.

<comment><AUTHOR> * I looked into those, but wow wow wee wow are they complex. I'll be working with cloning/moving DIVs instead, especially because HTML5 isn't supported everywhere.

* I prefer to avoid libraries when possible, especially when diametrically opposed ones are in use (I'm making an Angular app, JQuery doesn't always play nice).

* Thank you for the tip about [http://cdnjs.com/](http://cdnjs.com/), I'll be using it in the future.

* Hacker News is news, but it's also about discussion. That and I've personally never heard of how bad HTML5 DnD was, I thought the community might benefit from looking into it too.

<comment><USER20> Since when do Angular and JQuery not play nicely? Angular uses "JQLite" which gets replaced with JQuery if it's already loaded.

I guess if you're trying direct dom manipulation on the same elements from outside of Angular with JQuery and with Angular it could get messy. The community would just say "you're not doing it the Angular way." You're not. You can use the JQuery inside a directive and controller and be fine.

[https://docs.angularjs.org/misc/faq](https://docs.angularjs.org/misc/faq)

<comment><USER21> Ok, to give a specific example:

We use drag and drop for sorting in a few places. Our initial implementation was to wrap jQuery UI Sortable in a directive. This worked to an extent, but for our purpose, we ended up having to read the DOM and update the Angular scope from the DOM after jqUI emitted its event. This logic ended up being somewhat clunky.

We rewrote our directive in terms of HTML5 drag and drop such that when a user drops something, we get an event on the Angular scope, saying which element was dragged, and which element it was dropped on and then our controller rearranges the array in the scope, and only then is the DOM modified by Angular.

This generally isn't possible with jqUI and its plugins, which modify the DOM first and tell you about it later (in such unhelpful low level terms as "moved 100px up" rather than "moved two elements forward").

<comment><USER20> Yeah I can see how that would be a problem. Maybe it'd be better to clarify that JQuery and Angular work great together; however, JQuery _plugins_ (which may have all kinds of selectors, state, DOM manipulation) cannot be guaranteed to do what you want inside of Angular.

</comment></comment></comment></comment></comment><comment><USER22> I made an overview of advanced drag-and-drop features and their support in browsers (with examples):

[http://mereskin.github.io/dnd/](http://mereskin.github.io/dnd/)

</comment><comment><USER23> I didn't read that closely enough to understand exactly what was being said, but it was fucking hilarious.

</comment><comment><USER24> I've seen this naming pattern a couple of times in articles submitted to HN. The "... considered harmful" pattern starts with Dijkstra, "GOTO statement considered harmful"[1]. It doesn't mean "such-and-such feature is implemented incorrectly." It means, "such-and-such feature, even when implemented correctly, gives programmers at large the wrong notions."

So, one would say "singletons considered harmful". The entire notion of singletons is wrong. But the entire notion of drag/drop events is not wrong. Just the implementation.

[1] It's interesting to read some of the counter-arguments, and reflect on programming language design arguments we have today: [http://web.archive.org/web/20090320002214/http://www.ecn.pur...](http://web.archive.org/web/20090320002214/http://www.ecn.purdue.edu/ParaMount/papers/rubin87goto.pdf), and Dijkstra's own commentary on the whole ordeal is--as usual--entertaining [http://www.cs.utexas.edu/users/EWD/ewd10xx/EWD1009.PDF](http://www.cs.utexas.edu/users/EWD/ewd10xx/EWD1009.PDF)

<comment><USER25> > The entire notion of singletons is wrong

Could you expand on that? Haven't heard this argument before

<comment><USER26> They are generally misapplied by developers as a sort of global state, usually with the implied belief that "I'll only ever need one of these".

This has all kinds of nasty downstream effects. It becomes more difficult to trace the dependencies between classes, since any class can invisibly depend on a singleton. It becomes painful/impossible to replace that dependency in a particular instance, which makes both testing and extension difficult.

Dependency injection solves the issue of sharing copies of the same object, without forcing that the object is implemented as a singleton. You can inject a different/extended implementation, and it becomes easy to trace dependencies.

<comment><USER27> How does it make it harder to trace class dependencies? Find all references to the singleton.

Sometimes you just really need some global state. Have you ever done game development? Unless you want to pass a giant GameState godobject to every single thing. Singletons or static classes: NotificationManager.PushNotification(..), DBManager.Instance.UpdatePosition(..)

Singletons are good for when you want a static class but also need some things you can only do with instance classes.

<comment><USER24> I've actually done a lot of "game" development, both as a hobby and in non- game-development fields (lite simulation work increasingly resembles games, and I think in general they should be developed in the same way).

Static classes are a hack for languages that force everything to be an object. You don't see static classes in languages like C++ or Common Lisp or JavaScript. You pretty much only see it in Java or C#. You make global (though certainly namespaced, we don't need to be colliding names here) functions instead.

And the difference between instance classes and static classes is that instance classes encapsulate state. So a singleton is literally nothing more than stateful global functions.

The realization you're starting to need a GameState godobject is what is called a code-smell, i.e. trying to find answers to the wrong problem of "how do we get all of this state around?" The problem should more correctly be "how do we avoid needing to have so much state passed around?"

It's usually a sign that the project is using too much inheritance. It seems natural to have a class "ProjectileWeapon" that has a virtual "fire()" method that returns a "Projectile", from which "Gun" inherits and overrides fire() to return a "Bullet" and "RocketLauncher" overrides to return "Rocket". But the Bullet only has a momentum vector, the Rocket also has fuel, and the Bullet only does kinetic damage, the Rocket includes chemical, and different materials are more or less resistant to different types of damage. The interactions between types that all try to encapsulate their own behavior starts to multiple the number of cases where different bits of code need to know about different parts of the world.

It at first seems to reflect the real world, but in reality a Rocket knows nothing about the air it is flying in and the objects it hits and blows up on. And those objects don't know anything about the Rocket, either, they are equally a'splodey whether they're hit by a Rocket or a Grenade.

The better way is Composition, the "has-a" in "is-a" versus "has-a". A Rocket has fuel. You don't say a rocket "is a fueled thing". A physics engine takes all of the things that have fuel and the things that require fuel (rocket engine) and asks "what is your fuel flow rate" and "what is your fuel consumption rate", respectively. Interfaces and Mix-Ins can be exploited to do this in a type-safe way.

Inheritance is useful when a strict tree-structure can appropriately encapsulate your needs. Unfortunately, very few things are strictly tree-like. The only times I've found a strictly tree-structured problem in the last 15 years has always involved processing language grammars. Inheritance works fine there. But even still, Haskell-style pattern matching works better. I won't go so far to say "inheritance is useless", but I will say my personal experience has been that I've learned to regret having used it in every instance.

<comment><USER27> >The problem should more correctly be "how do we avoid needing to have so much state passed around?"

I work on a large game, and multiple smaller games of my own, which all use a few static classes or singletons. It's never caused an issue, and I've never had such a problem with inheritance. I've never regretted using it in my code. What about Exceptions?

You don't seem to have offered a way to avoid having so much state to pass around. Where does the notification queue go? Where do the settings go? Where does the scene manager go? All the MVC controllers I have are static or singletons.

I don't see what objects have to do with no global variables or functions being allowed. There's no theoretical reason a global function can't operate on or return objects.

<comment><USER24> >> There's no theoretical reason a global function can't operate on or return objects.

That's not the same thing as stateful at all. "function add(a, b) { return a + b; }" operates on two objects and returns an object, but it maintains no state. I said global state was a bad thing, not global functionality. I added a caveat that global functionality should at least be namespaced, though.

You avoid passing state around by either injecting the dependencies or returning the state transitions to some other thing that makes the state changes happen. You make explicit the relationships that the singletons hide.

</comment></comment></comment></comment></comment></comment></comment><comment><USER28> This is from 5 years ago.

<comment><USER29> what he says is still valid,the spec hasnt changed,is broken even or recent browsers such as Chrome ,is complicated and confusing.It's another exemple of half baked spec written by people that are never going to use it,just like app cache.

Frankly HTML5 is a disaster(and no , things like WebWorkers or WebSockets are not part of the HTML5 spec before anyone jumps on me).

It's just hard to build something robust on weak foundations,but that's lot of client side developpers, making these things work even though specs are just not helping.

<comment><USER30> I don't think it's fair to blame the HTML5 drag and drop API entirely on WHATWG; my understanding is that it was originally specced years ago as the sanest useful subset of IE6's drag and drop API. It may not be the prettiest thing ever, but it's prettier than how it started out.

<comment><USER13> Subset? Isn't it more like a super set? IE6 (and above) has no files array that can be read with a FileReader and it only supports the types "Text" and "URL", no support for proper mime types.

<comment><USER30> I think this is what I was remembering:

[http://ln.hixie.ch/?start=1115899732&count=1](http://ln.hixie.ch/?start=1115899732&count=1)

...which is to say, IE6 supported more event types and more configurable behaviour than HTML5 supports, but some of those extra features were just too weird, or outright crashed IE, so they didn't make it into the spec.

</comment></comment></comment></comment></comment><comment><USER31> Ha I remember this article, it's from a few years ago and is when I started to lose my respect for PPK due to all the cursing and carrying on.

<comment><AUTHOR> I agree, his language is incredibly inflammatory and generally unnecessary. But he's done a really good job of writing up the flaws, so I suppose the tradeoff in this case was worth it.</comment></comment>


Grafana Loki – Like Prometheus, but for logs - <AUTHOR> https://github.com/grafana/loki <comment><USER2> I still remember when Grafana first come out because of InfluxDB. It's amazing at that time. I try to setup Graphite and eventually found out Grafana+InfluxDB and go w/ it from there.

Grafana move really fast. Very practical. It first if I remember use ElasticSearch/InfluxDB to store its config itself. Then move to its own SQLite. Then MySQL/Postgres. Eventually add alerting. Add multiple data source.

Nowsday my team even use its with SQL for a cheap version of Periscope/Mode.

Then now they figure out this log thing. ElasticSearch is great but try to run it yourself. On a 50 nodes k8s, I bet your first ES config will down after 10mins the first time FluentD come up and send all the log under the sun since the cluster come up

Given that Grafan guys know how to deal with visualization, I trust them to deliver a great experience again for log.

Look at their history, I'm going to bet hard on this.

<comment><USER3> > Then now they figure out this log thing. ElasticSearch is great but try to > run it yourself. On a 50 nodes k8s, I bet your first ES config will down > after 10mins the first time FluentD come up and send all the log under the > sun since the cluster come up

Tell me about it. I have not one, but multiple k8s clusters (some over 50 nodes), sending data to a single Elasticsearch cluster. A lot of tuning was done, and it's not yet perfect.

Elasticsearch is amazing, but it requires external support, and the tools available to do that are not on par. Curator specifically – it's dumb as a rock and is very limited on what it can do. Even a hot/warm architecture is a challenge to do on curator alone, you should ideally have custom scripts to manage it. Which is a shame as this is one of the "reference" architectures by Elastic. Whatever Curator does should really be part of ES itself.

The Kibana + ELK combo won't give you things like log tailing (logtrail is hackish and hard on the servers). The log forwarders are horrible to work with (be it logstash, filebeat – or worse: fluentd).

Some days it just works and you are happy. Some other days, you wonder why we moved away from syslog senders and text files...

<comment><USER2> I actually have same conclusion as you. ES isn't easy to tune and run well.

A thing many overlook are restart it. You cannot just go and `systemctl restart` it, once that occurs, constant rebalance happenning because it puts the load to the rest of system and eventually bring them down, all around :(. That make it harder to operator in K8S when it needs time to detach/attach volume, plus the overhead of overlay network.

So I myself didn't know about a perfect tunning. But the log search on Kibana is super helpful though.

That's why I kind of sold on this Loki thing. I have a good feeling that software in Go tend to require less config/tunning compare with Java. Eg, InfluxDB.

> Some other days, you wonder why we moved away from syslog senders and text > files...

I wonder the same. I actually like to grep log with tail more than a crappy web ui like Kibana, Sumologic etc.

Tailing log in a webui is clunky.

<comment><USER4> > once [a reboot] occurs, constant rebalance happening because it puts the > load to the rest of system and eventually bring them down, all around

I have this same problem with a 3 node cluster, no Kubernetes. This was using Elasticsearch 2.3 and it worked for a year or two and then all of a sudden, any network interruptions would cause the thing to get split brain and all indices would go red. It would take 8+ hours to go back to green and usually required a reboot of all servers or else it would never recover.

It was happening often enough that the decision was just run a single ES node since the total data was only 25GB :(

</comment></comment></comment></comment><comment><USER5> I'd be interested in replacing an in-house logs filtering/aggregation solution with Loki, but there's a lot that is not clear to me from the initial announcement & materials. Could you please help me understand what are the answers to the following questions?

\- Can I use Loki without Prometheus? I'd like to feed raw logs to it, with custom-generated metadata. I don't want to have to use Prometheus, nor InfluxDB.

\- Can I edit (modify) metadata for some old log line after it was already inserted? Specifically, I need to be able to rebuild metadata later, if I add some new "filters" to my logs (I want to be able to apply them retroactively).

\- Can I run aggregate queries on the metadata (sum, avg, min/max)? If not, what can I do with the metadata? Can I graph the metadata on normal Grafana graphs?

\- Is the text of the logs compressed? If yes, what compression algorithm is used? If not, why?

\- Where can I find the API docs (or at least the API source code) for Loki?

<comment><USER6> 1\. Yes, you can. We initially targeted Kubernetes because that's what we use and packaging for it is simple. We'll soon release packages for all distros, support syslog/journald etc.

2\. Hmm, please open an issue regarding this with the use-case that prompted this? This is not a use-case we have but if this is important for more people, we'd be happy to support it!

3\. You can only select based on the metadata, the metadata is just a set of kv pairs. Like {app=cassandra, namespace=prod, instance=cassandra-minion-0}

4\. Yes, we used gzip. Please see the sheet referenced at the end of the design doc [0] to see what we compared with.

5\. It's mainly protobufs right now [1] over HTTP, but we'll be adding more docs soon. Mind opening an issue for this so that we don't forget?

[0] [https://docs.google.com/document/d/11tjK_lvp1-SVsFZjgOTr1vV3...](https://docs.google.com/document/d/11tjK_lvp1-SVsFZjgOTr1vV3-q6vBAsZYIQ5ZeYBkyM/edit) [1] [https://github.com/grafana/loki/tree/master/pkg/logproto](https://github.com/grafana/loki/tree/master/pkg/logproto)

<comment><USER7> Hm, why is zstd marked as "BROKEN"? zstd would've been my first choice for something like this.

</comment></comment></comment><comment><USER8> Author here! Really excited to release this at KubeCon; happy to answer any questions you might have.

<comment><USER9> I'm curious about this, because at Square we maintain our own homegrown log aggregation system, and it's not really a core competency.

While a _lot_ of our logging needs seem like they would be fulfilled by this system — because we attach trace IDs to log messages, and because (at least in Payments) you can usually find the appropriate trace ID by searching for a Payment ID, which could be annotated too — there are definitely many times I've copy/pasted the text in quotes from a log-generating line of code in a Java or Go file, to find out if it's being executed, or as a handle into a subsection of code/logging.

In the linked design doc, you include a motivating tweet near the top, saying, "just give me log files and grep, I am dying". But unless I'm misreading things, there's no `grep` here. Right?

I'm guessing you could narrow down (using metadata) and _then_ grep, but if the narrowest metadata you have is app name and time range, you're still going to be grepping over a lot of data…

<comment><USER8> It's defiantly something that's missing from the readme, and perhaps not that obvious in the grafana explore view either - but it is there! You can push a regexp match server side and have that distributed to each Loki node, giving you distributed grep.

Will make it more obvious. Davkals has an iteration of the UI that makes it a separate field, which will also help.

<comment><USER10> Maybe I'm not understanding this - the docs say Loki is all about storing compressed log data with metadata, such that only the metadata is indexed. Are you saying you can search the compressed, unindexed data using regex? If so, wouldn't that potentially be incredibly slow?

<comment><USER6> The good thing about this is that the grepping can be parallelised and distributed on to several nodes. Having said that, once you select the relevant metadata right, you should be able to narrow it down enough for the queries to be snappy enough.

While this will definitely be slower than something that indexes the contents, you'll be able to store much more in Loki at much lower costs.

<comment><USER11> What are you using to run the regex? ripgrep could make up for some of the loss from not having it indexed.

<comment><USER12> [https://github.com/grafana/loki/blob/master/pkg/iter/iterato...](https://github.com/grafana/loki/blob/master/pkg/iter/iterator.go#L252:6)

Looks like the Go regex lib, which isn't super performant, so it could potentially be improved if it ends up being an issue.

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> Looks like a good light-weight alternative for setting up a full elasticsearch cluster for some light log insights.

I've been kind of building something similar myself, but this looks much better.

The free cloud demo does look it is getting hammered right now ..

<comment><USER8> >Looks like a good light-weight alternative for setting up a full elasticsearch cluster for some light log insights.

Exactly! Elastic is a really powerful system but I think there is growing sentiment that its overkill for container logs. This is exactly where we see Loki really helping - almost complimenting Elastic even.

> I've been kind of building something similar myself, but this looks much > better.

:blush: thanks! We'd love you input on Loki too...

> The free cloud demo does look it is getting hammered right now ..

Yeah, looks like I'm going to be scaling that all day... Our motivation for over the free service for the next few months is to really battle harden the system - anyone sending us data is really helping us iron out the kinks and improve the open source. Its early days, but expect it to get much better over the next weeks and months..

<comment><USER14> > Elastic is a really powerful system but I think there is growing sentiment > that its overkill for container logs

I don't know much about kubernetes, but Loki looks super interesting for our application logs too.

Could someone maybe ELI5 what the difference is between "container logs" and, well, any other kind of logs? Don't most dockerized application send their stdout to Docker, and aren't those, then, the container logs? What kind of logs _aren't_ "container logs" and therefore a better fit for ElasticSearch than Loki?

Thanks :-)

<comment><USER6> I work with Tom on Loki. When we said we wanted to focus on Kubernetes, we meant we wanted to correlate metrics with logs and focusing on Kubernetes lets us do that well.

Having said that, this will work for any logs as long as you can tag the logs meaningfully. We'll soon be releasing packages for all major distros and journald.

</comment></comment></comment></comment><comment><USER15> Serious question. Why not just put logs in postgres? Rich query language. Indexes. Has support for time-series based indexes. Optionally can support full text search if needed.

<comment><USER16> Apart from the query support, which others have mentioned, there's the sheer scale of data. The most recent time series implementation I worked on had to ingest terabytes per minute from a wide variety of sources in a single factory... And the goal was a single system combining every factory worldwide. Easily measured in exabytes per hour, with thousands of sources. A traditional relational database - even a really good one like postgres - just isn't built with that kind of use case in mind. There really is something to a database specifically designed for a particular kind of query on ridiculous quantities of data.

<comment><USER17> According to my napkin math that's more than the LHC's raw output, do they really need all that for factories? Seems nuts.

<comment><USER18> Need? Probably not. But often you don't know what metrics you care about until something goes wrong - so you over-engineer it and log all the things.

</comment></comment></comment></comment><comment><USER19> So this is something like mtail [1], with the ability to also drill down into the raw logs, and integrated into Grafana? Sounds great!

We're heavy users of Grafana, Graphite, Prometheus, and Elasticsearch, and while the latter started in a BI role it's expanded to take on pretty much anything we can throw at it. However, there's still tons of system and service logs we're not gathering yet, because the effort to get them to Elasticsearch and store/maintain them is not worth it, especially since the value is not always clear until well _after_ the pipeline is set up.

I'm definitely excited to see what loki can do for us, and just upgraded one of our test Grafanas to nightly to start playing with Explore, looks great!

[1] [https://github.com/google/mtail](https://github.com/google/mtail)

<comment><USER20> Grafana Explore author here. It's still in Beta and would love feedback. Simply open an issue at [0].

What we tried to get right is the seamless switch from a Prometheus to Loki where it's retaining the labels of the query to essentially find the logs that come from the same e.g., "job". The assumption is that you need to be consistent with your relabelling rules of Prometheus and Loki.

[0] [https://github.com/grafana/grafana/issues/new](https://github.com/grafana/grafana/issues/new)

</comment></comment><comment><USER21> We have way too many technologies in our industry with names borrowed from culture.

Before clicking to look at the comments, I totally thought this was about some interesting creation mythos with a character that taught humanity how to harness wood from nature to build the first wood dwellings.

I mean, that's totally something that could feasibly get posted to HN and do well.

<comment><USER8> There are only two hard problems in computer science: cache invalidation and naming things.

<comment><USER22> > There are only two hard problems in computer science: cache invalidation, > naming things, and off by one errors.

That's how I learned the phrase.

<comment><USER8> And Loki has all three ;-)

</comment></comment></comment></comment><comment><USER23> We built a system like this similar to this at Datastreamer to store the logs from our web crawl.

We used Grafana + KairosDB and turned the logs into tags essentially.

Our entire productions system had easy 'taps' that you could annotate to monitor everything about our crawler.

For example, number of HTTP requests, their status codes, the language of the content.

We also record intersections of the tags like lang+domain.

The downside of a system like this is that you have to know all your metrics apriori... If not and you need them at runtime you're out of luck.

The UPSIDE is that you use like 1/100th of the total size you would originally need for raw logs.

What we found is that you quickly converge on the tags you need and then you don't end up adding many more.

Everyone says 'disk' is cheap but in our situation our logs outpace the amount of data we would collect. We'd have 1000s of petabytes of logs by now.

<comment><USER24> Is it just me or can no one else find how to actually send the log data to Loki either?

</comment></comment><comment><USER10> From the design doc:

> We will be able to pass on these savings and offer the system at a price a > few orders of magnitude lower than competitors. For example, GCS cost > $0.026/GB/month, whereas Loggly costs ~$100/GB/month[0]

Maybe I'm misunderstanding, but I don't see how Loggly costs anything like $100/GB/month? It seems clear to me that the $99 plan costs $100/GB/ _day_ (or $100/30GB/month). Not exactly cheap, but nowhere near as expensive as the design doc reckons.

[0] [https://www.loggly.com/plans-and-pricing/](https://www.loggly.com/plans- and-pricing/)

<comment><USER6> Yes, you're correct. We'll correct the mistake in the doc.

<comment><USER10> This premise was very far off, and cost effectiveness is key to the design rationale - do you think this impacts upon the utility and attractiveness of Loki?

<comment><USER6> Yes, we have an internal doc where we compared with more providers (including loggly) where we got the pricing right. The line currently mentioned in the doc is a typo.

The rationale still holds and one of the primary reasons we built Loki is to have a an easy to manage scalable "open-source" solution.

I personally have been told to log less at a previous company because of the associated costs of logs and I don't think I utilised anything beyond grep (with a few exceptions). I personally feel the trade-offs are right and a simple greppable/tailable solution that is cheap is missing from the eco- system.

<comment><USER10> A good response, thanks for confirming that you've built this with the correct pricing in mind.

</comment></comment></comment></comment></comment><comment><USER25> Where do people recommend logging infrastructure changes (e.g. database changes, server configuration) for all things potentially not in code?

<comment><USER26> We need more context. Server configuration should definitely be in code/source control these days. Database changes, well, what kind of changes are you talking about?

<comment><USER25> These days, as we have multiple departments and services, it's hard to track down any issues we identify without knowing what kind of changes have happened to potentially cause it. As we don't have a single development platform (multiple acquisitionss) this is hard for us at least until we have unified systems (will take time). I'm referring to any infrastructure or operational change to the stack. For example, deployments/roll outs, scaling servers up or down, port changes, updates to software on key servers, changes to database table structure, manual changes to database data. Any changes at all to the production stack.

</comment></comment></comment><comment><USER27> If you're just tailing logs on the server (or with kubectl) but need a bit more power, angle-grinder [1] is pretty good (and of course doesn't need to be hosted or anything, it's just a CLI app)

[1] [https://github.com/rcoh/angle-grinder](https://github.com/rcoh/angle- grinder)

</comment><comment><USER28> Is this the same technology kausal.co guys had? It's a shame it was just shut down after the acquisition.

<comment><USER8> This is a continuation of our ideas from Kausal, yes! Really glad we joined Grafana as they have given us the time and resources to pursue this.

And not shutdown at all! Everything we did at Kausal is now part of Grafana and Grafana Cloud - David's PromQL-completion UI is in Grafana v6 as the explore view, and Cortex is the backend for Grafana Cloud's hosted Prometheus...

<comment><USER28> Thanks! What about the distributed tracing feature?

<comment><USER8> Its coming :-)

</comment></comment></comment></comment><comment><USER29> if this is like splunk that doesn't make me cry unicorn blood tears every time an invoice arrives then i'm all in. unfortunately it doesn't look like it and honestly i'm not sure what it is. 'prometheus for logs' doesn't quite explain it.

<comment><USER6> Hi please see the design doc for more inspiration as to why we built it: [https://docs.google.com/document/d/11tjK_lvp1-SVsFZjgOTr1vV3...](https://docs.google.com/document/d/11tjK_lvp1-SVsFZjgOTr1vV3-q6vBAsZYIQ5ZeYBkyM/edit)

We're soon coming out with a blog post explaining the motivations and architecture in detail. But yes, the pricing is one of the motivations to build this, and as logs will be stored in S3 or an object store, the cost will several orders of magnitudes less.

</comment></comment><comment><USER30> For Grafana folks: I got all the way through sign up and sending logs from k8s cluster + setting up logging data source to Loki hosted instance, but could not find a way to actually explore the logs in the hosted grafana instance :)

<comment><USER30> I figured that you have to click "Explore" next and then see the Logging tab, although it was not easy for me to find, may be highlight it as a first class link on the left panel?

</comment></comment><comment><USER31> [https://www.youtube.com/watch?v=xJSgf835YRE](https://www.youtube.com/watch?v=xJSgf835YRE)

</comment><comment><USER32> Looks cool, is the source code for promtail available? I can see there's an image available [https://hub.docker.com/r/grafana/promtail](https://hub.docker.com/r/grafana/promtail) (which is used in the readme) but I can't find the source anywhere. Anyone know where it is?

<comment><USER8> Of course! Everything is opensource, all apache licensed:

[https://github.com/grafana/loki/tree/master/pkg/promtail](https://github.com/grafana/loki/tree/master/pkg/promtail)

<comment><USER32> Cool, thanks. No idea how I failed to find it...

</comment></comment></comment><comment><USER33> This looks similar to Graylog ([https://www.graylog.org/](https://www.graylog.org/))?

From Loki's readme, It still stores the full log (but compressed), and indexes values in each logline.

<comment><USER6> Hi, we don't index the values, but rather we index metadata about the logstream, for example, service name, instance ip, and things like that and not the actual contents of the log lines.

Inside kubernetes the metatadata would be podname, namespace, deployement name, container name, etc...

</comment></comment><comment><USER34> What is the purpose of log aggregation without indexing?

<comment><USER8> Thats a really good question. I feel like its pretty well covered in the design doc[0], and you should also read about OKlog[1] which really championed this idea.

To be clear: Loki indexes metadata about the streams, and the streams themselves are indexed by time. We don't full-text index the streams though. We think, when combined with metrics, this represents a nice trade off in terms of complexity vs features. This simplification not only makes Loki easier to understand, but also scale and operate.

[0] [https://docs.google.com/document/d/11tjK_lvp1-SVsFZjgOTr1vV3...](https://docs.google.com/document/d/11tjK_lvp1-SVsFZjgOTr1vV3-q6vBAsZYIQ5ZeYBkyM/view) [1] [https://github.com/oklog/oklog](https://github.com/oklog/oklog)

</comment></comment><comment><USER35> Awesome, can't wait to dig into this. Nice work.

</comment><comment><USER36> I ran the curl ... | kubectl command on my cluster.

What would be the reason that promtail is not tailing log files for all of the running pods?

</comment><comment><USER37> Does this integrate with Prometheus's Alertmanager then for sending alerts based on certain logging events?

</comment><comment><USER38> Is there a way to send logs to Loki without using the tail agent? For example, a REST API for storing messages?

<comment><USER8> There is a REST API, yes [0]. Currently you send snappy-compressed protobufs[1] containing yours log to a HTTP endpoint.

We initially started using fluentd as the agent, but we found its metadata "enrichment" facilities weren't reliable enough - we'd get log lines without the pod tags, for instance. For something like Loki, which depends really heavily on the metadata for index, this was super important. So we wrote promtail.

[0] [https://github.com/grafana/loki/blob/master/docs/api.md](https://github.com/grafana/loki/blob/master/docs/api.md) [1] [https://github.com/grafana/loki/blob/master/pkg/logproto/log...](https://github.com/grafana/loki/blob/master/pkg/logproto/logproto.proto)

<comment><USER38> Awesome, thanks!

</comment></comment></comment><comment><USER39> InfuxDB has this as well [https://www.influxdata.com/blog/writing-logs- directly-to-inf...](https://www.influxdata.com/blog/writing-logs-directly-to- influxdb/)

</comment><comment><USER40> Why not a unixish simple tool to handle metrics? assuming $prog outputs log info to stdout: $prog | collect | action and 'collect | action' could all be done somewhat simply with awk and 1000x more portable.

<comment><USER41> And store it where? And query it how? What will render my NoC dashboards?

With graphite protocol the metric stream is already plain-text with just 3 fields per line (path value time), its pumped over a socket to a collector.. but that's where the hard stuff starts.

<comment><USER40> First I'm not trashing the project, I'm just wondering why simple unix like solutions aren't used.

Store it where ever you want, this isn't a magical datastore that makes things faster, use clickhouse-client, whatever it doesn't matter.

There is a widening disconnect between the unix way and how new projects are created.

</comment></comment></comment><comment><USER42> But I just setup Elasticsearch and Kibana yesterday!

<comment><USER43> A few years ago, I set up centralized logging with Elasticsearch and Kibana on a site with very little traffic (less than 100 requests per minute). After a few months, it apparently fell behind on indexing and worked the logging server's hard drives 24x7. After a few more months, the server died. (It refused to power up and I wasn't interested in going deeper.) I freely admit that I don't know how to configure Elasticsearch correctly, but I feel like it should have been able to handle the indexing load in its default configuration.

I think I'll try out this new logging solution. I only want basic indexing.

<comment><USER44> I try to stay away from long-lived processes written in Java. Maybe it's just me, but I think they consume insane amounts of memory.

</comment></comment></comment><comment><USER45> Very interesting, will check it out!

</comment><comment><USER46> As inmature I'm my first thought was thst the logo looks like a dick to me. :|

<comment><USER47> You cannot unsee it lol

But on a serious note the developers should indeed look into this. I recall this comment from HN [0] which highlights why neutral branding and naming are important.

0: [https://news.ycombinator.com/item?id=14702513](https://news.ycombinator.com/item?id=14702513)

<comment><USER32> I could never unsee it in this old easymock logo [https://www.javacodegeeks.com/wp- content/uploads/2012/10/eas...](https://www.javacodegeeks.com/wp- content/uploads/2012/10/easymock-logo.jpg). They have since changed it to something much more neutral, thankfully!

</comment></comment></comment><comment><USER48> can it just replace ELK with a single golang binary? big if it does

<comment><USER8> If all you use ELK for us simple process/container logs then yes! That's the whole idea.

But Loki is not a replacement for Elastic - we don't have any complex query support, and we don't do full text search. Elastic is great for analytics or BI, but we think Loki is the way forward for your container/pod logs.

</comment></comment><comment><USER49> Lauki

</comment><comment><USER50> its just nvidia was over-valued and the market is correcting itself now alongside all the other tech stocks getting smashed, but I think investors should double down on nvidia now while the getting is good. The company still has the best gpu engineers on the planet $$$.

</comment><comment><USER51> Grafana, are you sure you want to name a piece of software after a guy famous for lying, betrayal, and other forms of unreliability? Besides, I already know two cats, a dog, two children, and a Saturn hatchback named Loki. The Saturn is named appropriately, by the way; let's hope your software isn't.

<comment><USER8> I'm just a really bit Marvel fan though - he did redeem himself in the last film, no?

<comment><USER52> In the actual nordic myths, he is a rather bad guy and in the end fights against the gods when Ragnarök (armageddon) comes.</comment></comment></comment>


Mathematician and comedian test out the Monty Hall problem (video) - <AUTHOR> http://www.youtube.com/watch?v=o_djTy3G0pg <comment><USER2> Massively clearer than a text article.

</comment><comment><USER3> Funny how the guy the guy "backed by math" was also very lucky.</comment>


Evolution of Hacker News - <AUTHOR> http://techcrunch.com/2013/05/18/the-evolution-of-hacker-news/ <comment><USER3> I get the feeling that at the very top of every Hacker News comment thread, there's someone who claims that the author of the article is an incompetent idiot because they found a factual error somewhere in the article. Often this leads to a pedantic discussion of some insignificant details.

With all these extremely knowledgeable people from different fields, I wish I found more big picture insights in the comments here.

<comment><USER4> Yes, this is one of the problems I spend the most time thinking about how to solve.

FWIW, my current theory is that the problem is not the comments themselves, but the upvotes. There are often moderately stupid comments far down the page in any given thread, but they're comparatively harmless, like someone walking down the street mumbling to himself. The problem is that there's a type of stupid comment that attracts upvotes from a certain type of user. If we can recognize either the comments or the upvoters, we can solve the problem. I've been collecting a corpus of them for a while; it may soon be big enough to be useful.

<comment><USER5> I understand why you're against having "advanced" features and flair, but being able to collapse the thread that follows those comments would help enormously. It's almost the same as sending the thread to the bottom of the page.

Once I've decided not to read a subthread, it's a little frustrating to try to guess where to scroll to find the next constructive comment.

<comment><USER6> Thread collapsing helps each person who chooses to collapse a thread, but does nothing to help upvote / downvote the posts in that thread.

If collapsing can be linked to voting or thread positioning it'd be useful. Collapsing a thread means it's not very good[1] and thus should be further down the page.

There is an extension / addon somewhere that does thread collapsing on HN. A web search should find at least one.

[1] or maybe it just means that you're read it and are keeping the page tidy.

<comment><USER7> I'm not sure that's entirely true. If being able to collapse a thread means that more people actually read and upvote threads that are below it, then that in itself will help push the offending thread down the page.

</comment></comment></comment></comment></comment><comment><USER8> I'm curious about the degree to which decreasing quality is a consequence of dilution in which "less desirable" individuals lower the mean quality (or some other holistic measure), versus the social effects due purely to size.

I'm curious because a large community will probably change the behavior of all individuals, regardless of quality. One such effect is they way discussions happen. In a village, town halls can have discussions with every member. An idea is brought forth, multiple people make changes, it gets amended, reworked, changed. In short, topics have some amount of "persistence", and opinions have more dynamism.

In a city, discussions are more like a broadcast: any member might be able to speak, but they're not sparking discussion, they're broadcasting a point of view, and with so many points of view broadcast at any given moment, it's hard to have a discussion that lasts more than a few hours before the next topic of interest is brought forth. In short, topics seem to have little persistence, and opinions have far less dynamism.

It seems much more rewarding in a city to make comments designed to convince the audience of how intelligent I am, because I only have the city's attention for a moment. Perhaps this leads to the preponderance of middlebrow dismissals or other "how can I look smart here?" comments. In a village, by contrast, these comments are much less lucrative. If I walk around and the only comment I can give to others is that their new crop idea might not work, I'm probably not going to be very valued, because people will notice over time that those are the only comments I give.

It's difficult for me to articulate quite what I mean, perhaps someone could help.

<comment><USER9> HN is like democracy, it is flawed but it is the best we've got.

<comment><USER10> HN hellbanning is just infuriating, no upsides.

<comment><USER11> Are there any actual hellbanning tragedies on HN?

<comment><USER12> Imagine the saddest, loneliest, weirdest, most obnoxious kid from your elementary school. Maybe you were that kid. They go up to every little clique and say something, but no one listens. They invite everyone to their birthday party, but no one comes.

Would you wish that on anyone, even if they were annoying or angry or wrong?

<comment><USER13> The jump from complete ostracization to having your HN posts hidden is pretty damn large.

What would you suggest that we do with posters like losethos, if not hellbanning? The guy knows he is hellbanned, and has at various times made new accounts (which also get hellbanned). A regular banning would just have him creating new accounts anyway, so you may as well just let him keep his regular account. In elementary school people like him would be removed from the normal student population and given special attention/treatment, but that is not an option for HN.

<comment><USER14> Isn't losethos proof that hellbanning doesn't really work?

If a hellbanned user is supposed to get frustrated by the lack of replies to their posts and then go away, then it doesn't. You're not supposed to ever know it's there, after all, and you're not supposed to be willing to put up with it.

In the case of someone who might not care about being replied to, or who just doesn't notice they're never being replied to (entirely possible if they don't post often), being hellbanned doesn't work. It's security through obscurity.

And as a result of hellbanning, people can become paranoid about speaking their opinion because they never really _know_ if they're actually participating in a conversation or if the system has decided to make a silent mockery of them.

<comment><USER13> It doesn't matter if losethos gives up and leaves or not because he is hellbanned.

That is part of the beauty of hellbanning, if you realize you have been hellbanned, give up and leave... then great! If you realize you have been hellbanned, don't care and keep on posting... then great! If you realize that you have been hellbanned, come back with a new account and keep it up, you'll be hellbanned again (and quickly. losethos's new account was hellbanned again with its first comment... so great!) If you realize that you have been hellbanned, come back with a new account and clean up your act... then great! If you don't realize that you have been hellbanned... then great!

Hellbanning is not about security; it is about removing people from the conversation. For that it works wonderfully (though I would argue that it is actually too hard to become hellbanned).

<comment><USER15> By default, I have configured HN to show me hellbanned posts - <USER2> someday, maybe I'll be too, and because time to the time I still find some interesting content!

IMHO, hellbanning is both cruel and damaging - the community may lose potential interesting contributions.

For example, I've watched a bit of what losethos wrote for his 64 bit OS - it's an interesting approach. There are not so much at the time, but he seems to submit articles time to time.

I sometimes even read losethos comments, which sound to me like hybrid talk from Galactica. Words linked by references obscure to me, but which still might have a sense (the old greeks did believe in Oracles, so do I. My fault: I shouldn't be a believer, and just accept the traditional medical theory who claims it's just nonsense)

<comment><USER13> He _has_ done some pretty fascinating stuff, but aside from discussions about that particular stuff in particular I think we would gain nothing from permitting him to spam^Wcomment as a normal user.

I also have showdead enabled, mostly because I like to look back in hellbanned posters histories and see why they were hellbanned. In _very_ few cases do I find someone for which I don't understand the hellbanning. In those cases I usually try to give the person a heads up, but in the _vast_ majority of the cases I realize that the person was excessively abusive, spamming their pet project, participating at a stereotypically "reddit" level, or otherwise submitting crap.

(Also, I am pretty certain that losethos's comments are computer generated. He thinks that a god talks to him through a RNG when he initializes markov chains with the bible or something... Any meaning derived from what he says after "god says:" is likely just apophenia)

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER16> _Another subtle feature addition: a flame-war detector. Graham has been consistently deploying and updating proprietary software that determines whether there is a flame war, where people argue heatedly. When these flame wars take place (which Graham says can often get ugly and personal), the story in which the commenting is taking place is moved further down the page._

Remember the above the next time someone claims that a rogue contingent _must_ be flagging stories on a particular topic because it is further down the page than other stories submitted at a similar time and with a similar number of points.

<comment><USER14> It might just be me, but this seems like the kind of feature you wouldn't want automated. How do you algorithmically tell the difference between a flamewar and a spirited debate?

<comment><USER17> It would go a long way to partly explaining why Apple stories get buried so quickly while ones related to Google get to the top and stay there. For whatever reasons Apple engenders strong emotions, positive and negative, while Google does not, at least to that degree.

<comment><USER18> Did you see the stuff related to Google demanding the Windows Phone Youtube app be removed? Massive threads, accusations of shilling all over, along with claims of organized flagging. At least 2 of the major contributors to those threads/flame wars (recoiledsnake and CloudNine) have been hellbanned, and the whole thing was rather unpleasant to read. It also spilled over into most other Google posts on HN, which was quite a few as it was in the middle of Google IO. Along with that there is the inevitable comment on every story about a Google service suggesting that Google will shut it down in 6 months. There is a sizable subset of HN which isn't crazy about Google.

<comment><USER19> His solution was to put flamewars further down the comment list, but if almost all of the comments get flagged as flame war, then it's free to flourish.

Maybe what HN needs, could be some way of giving people who want the heated discussion, a way to express these seperate from rest. Maybe a tab that shows controversial comments, so you can switch between the "calm" and the "heated" discussions.

<comment><USER20> > Maybe what HN needs, could be some way of giving people who want the heated > discussion, a way to express these seperate from rest.

I like this idea. I'd say all who are hellbanned should see other hellbanned users' and they can nuke each other from the orbit in their own little sandbox.

<comment><USER5> You can set "showdead" in your profile to see hellbanned users. But you still can't reply to them (that's kind of the point).

</comment></comment></comment></comment></comment></comment></comment><comment><USER21> I too have noticed pg's growing frustration with HN a little bit. I joined HN almost 6 years ago -- then I was a naive teenager with a scattered view of the world, though with a ferocious appetite for intelligent debate. I used to do light design work then and usually shied away from hardcore backend programming. I have HN to thank for getting me to take the jump. I also have HN to thank for keeping me informed on various fronts so I could make right decisions that were key in seeing my first 'startup' project surviving (thanks patio11 for your instructive thoughts on a/b testing; grellas for legal stuff; potatolicious for thoughts on design, etc. etc. :)).

One thing I want to say is: I think it's okay to be ... well, a little bit mean sometimes. My favorite commenter on this site is rayiner -- one of the reasons I like him is he's never afraid to say what he really believes (sometimes in a slightly mean diction), even if it's going against HN userbase consensus. He's not out to get karma (as some other opportunist high-karma users here are sometimes... complete with amazon affiliate links in their comments, indeed, that is singularly the only thing that annoys me about HN). There's inevitably going to be disagreement when we're discussing important things we have strong feelings on. As long as the discussion doesn't devolve into corrosive name-calling and ad hominems it's fine, and I don't think we go that low very much. I think HN will be fine going forward -- there's a strong identity that is enforced by all users. Thanks pg et al. for keeping this hangout alive and rocking.

</comment><comment><USER22> > _If you are a YC founder, your username will show up in orange to other YC > founders to enable these entrepreneurs to recognize and meet each other._

Silicon Valley is supposed to be this great meritocracy where it doesn't matter where you went to school or who you know, all that matters is how well you can build and "make something people want."

However, it always amazes me how much cronyism (for lack of a better word) there still is in Silicon Valley. Even Y Combinator is not immune to it. To their credit, PG et al are great at pattern recognition and identifying what separates good/great teams from the bad, but I feel like there is still too much cronyism, personal bias, and subjective judgment in their model that excludes a lot of entrepreneurs who don't fit their "type".

Given how influential PG's essays and YC's model have been on other VCs and angel investors, I wonder if it's becoming a self-fulfilling prophecy and/or vicious cycle in the industry where because YC prefers a certain subset of successful founder, it causes many/most investors to also prefer that subset, which makes it easier and easier for this subset of founders to quickly get funding and access to critical resources and networks, while it becomes harder and harder for new subsets of entrepreneurs to break through this walled garden.

DISCLAIMER: I really like YC and I read HN every day. I just think that if e.g. Sequoia and Kleiner Perkins were VC 1.0 and YC was VC 2.0, a new VC 3.0 that addresses the shortcomings of the current model is way overdue.

<comment><USER23> I actually used to think a bit like how you do.

But humans form groups. That's what we do. And this group is pretty easy to join. You don't need a particular skin colour or religion. Just rock up and rock on.

</comment></comment><comment><USER24> I am happy HN is here, even if the small village has become a big city. On a given day, there is a decent percentage of the front page that I don't understand at all, and I enjoy reading the articles and comments. I find it an interesting way to learn new things.

Thank you PG for creating Hacker News. I read it daily.

<comment><USER25> This is exactly what drew me to it as well; I was awed by the opportunity for so much learning. Anecdotally I've found the focus of the front page has shifted a little from hard-core CS (my degree) and maths topics to more startup and web-development oriented projects and articles. However, there is still often one or two articles that give me the hn hit of old.

I don't often comment or submit links, and when I do I don't really care for the karma beyond hoping I've provided something constructive, but despite the shift in focus I continually enjoy the spirited, structured and technical discussion that hn provides. Like gregpilling I read it daily, and would like to thank pg amongst others for continuing to curate such an interesting community.

</comment></comment><comment><USER26> It's sad and indicative of the quality of tech journalism when this line gets written by thenextweb and then quoted by techcrunch without being questioned:

> Having a big audience isn't really the goal. In comparison, Hacker News' > inspiration and the first big YC exit, Reddit has seen as much as 4.4 > million page views in a given day.

If 4.4 million page views in a given day were special for Reddit, it would not be the hugely successful page it is and tech journalists should have a rough idea of traffic figures like these. The same thenextweb post claims further down that Reddit averaged 3 billion page views a month in 2012.

(it was the Obama AMA subreddit that hit 4.4 million page views in a single day)

<comment><USER27> I'm a Brit, so take my token apology for calling you out on this [1], but the way you frame your comment is actually the thing that frustrates me the most about a lot of HN comments. So much negativity, which is often based on some extremely esoteric fact, that actually takes away from the original point of the article, and is just counter to _any_ continuing discussion. Branching off is great. We get some really great discussions on HN that side-step the original point of an article, but when it's just for the sake of bashing, I can't ever support such comments.

[1] : We like to apologise for things

<comment><USER28> I'm not sure I agree with you unless I'm misunderstanding the point of what you mean by what the parent said that was "extremely esoteric".

The point seems to be that we are reading a story about hacker news and someone takes issue with what appears to be an example of lazy journalism.

I could find my own examples in the article. Right off the top there are only quotes from PG and tptacek when it would have been really easy to email other users and get some thoughts to round out the story. (Not to mention the fact that it left out a fairly important point - that tptacek has the most karma on HN other than PG).

</comment></comment></comment><comment><USER29> What bothers me personally the most is the way in which downvotes seem to happen. Don't get me wrong, a lot of users downvote in a good way, so it's not all bad.

Still, I often find grey comments that are brought in a completely respectable way. I feel like these comments are getting downvoted because someone disagrees with them, rather than because they are bad or toxic comments.

The opposite is also true, I often find comments with a positive amount of votes that are in the "internet jokester" style and don't really contribute anything meaningful to the topic at hand.

<comment><USER5> Often if you go back later, you'll see that the downvote has been "corrected" by another reader. I'll sometimes upvote a reasonable comment only because it has negative karma. And I've... uh... in the name of science, made some relatively useless disagreeable comments that had downvotes, then upvotes, then downvotes,... before reaching equilibrium at +1, so other people do too.

</comment></comment><comment><USER30> The article (perhaps ironically) neglected to mention an enormous volume of submissions exist just to harvest pageviews from this community.

There is a constant flow advertisements posing as articles where startups write random fluff to get their otherwise unrelated and uninteresting startups on the front page.

There is a constant flow of mainstream blogs and people meticulously tailoring content for this site.

There have been numerous large companies that have blatantly spammed the site for years undetected, and probably plenty more that still haven't been detected judging by all the 'submissions-only' accounts churning out generic links to mainstream sites.

I don't think routine exploitation fosters a healthy, happy community.

<comment><USER31> your points illustrate some reasons why I do not consider this a community,in a positive sense. there is a lot of interesting discussion, which is good, but it exists to serve and is largely constrained by the need to create a social engine for private investment in tech companies, and the ecosystem that entails.

</comment></comment><comment><USER32> HN is a testament to what happens when you make a thing for a very specific group of people and only them.

It sort of reminds me those companies you hear about every once in a while that say "No" to being sold in Wal-Mart. It seems crazy but there is value in passing on certain kinds of growth.

The only hiccup to this plan in HN's case is that the very specific community it is targeted to is itself growing and growing. So staying "small" might be impossible.

<comment><USER33> > It sort of reminds me those companies you hear about every once in a while > that say "No" to being sold in Wal-Mart.

Speaking of which, I wonder what became of this one?

<http://www.fastcompany.com/54763/man-who-said-no-wal-mart>

Apologies for deviating from the thread's topic, but I re-read that the other day and got to wondering.

<comment><USER34> Snapper/Simplicity ended up being bought by Briggs & Stratton, which apparently decided to back peddle a bit:

Walmart to sell Briggs & Stratton Snapper mowers [http://www.bizjournals.com/milwaukee/news/2013/01/17/walmart...](http://www.bizjournals.com/milwaukee/news/2013/01/17/walmart- to-sell-briggs-stratton.html)

Here is a forum comment apparently from Jim Wier ("The Man Who Said No to Walmart") regarding the Sears deal after his departure: [http://ppecongress.com/showpost.php?s=b79ee2a3f907acf9594e26...](http://ppecongress.com/showpost.php?s=b79ee2a3f907acf9594e2667735c8d06&p=7174&postcount=1)

</comment></comment></comment><comment><USER35> "Around six months ago, Graham brought on someone else, who he chose not to name, to moderate the site. He says the individual is affiliated with Y Combinator and is a 'prudent and thoughtful guy,' and has been doing a great job ever since."

I didn't know this. I think the prudent and thoughtful guy is doing a generally good job. I don't think anyone can read every thread here exhaustively, but there is some good signal:noise ratio here even after all the years of growth.

"I wish I could get people to stop posting comments that are stupid or mean" is pg's summary of what still needs to be fixed. I'm on board with that too.

<comment><USER14> I assumed that once you got enough karma here you started to get mod rights... maybe not though.

<comment><USER36> Mod rights as in flagging articles/users or downvoting? Those are all based on a certain amount of karma (with downvoting having the highest threshold since it can cause the most damage).

</comment></comment></comment><comment><USER37> I think what I love most about HN is that PG (and others?) take an active role in trying to steer the tone of the comments in specific directions, and don't try to make it a super democratic site like Reddit where everything about the community is allowed to just grow organically in a hands-off manner.

I see lots of complaints about supposed censorship or excessive moderation or hellbans, and I understand why those things might frustrate people. Really, I do. But it's not like this is the only place on the Internet where people can go to post comments about tech news. The fact that HN isn't trying to be all things to all people, and isn't trying to be as fair as possible to everyone at all times, is actually what I love about it.

I'm glad that there's places like Reddit where everything is community-driven and the maintainers are totally hands-off as to the content of the site, and I frequently enjoy Reddit as well. But I'm also glad that there are places like HN which are actively molded into being a specific kind of site with specific standards of discourse, even if I don't always agree with it. I wish more news sites had such specific visions for content and comments. Not necessarily the same vision, just _some_ vision other than avoiding even the perception of censorship at all costs.

<comment><USER38> I don't quite know why, but your post made me open up 4chan for the first time in years.

<comment><USER37> Maybe because it's cool to enjoy 4chan even though it is fundamentally effed up in so many ways. :) Just like it's cool to enjoy HN even though it's fundamentally effed up in a completely inverse way. I figure all of these places are great sources of information seen through different prisms, you just have to understand the prism to get the most out of it. I try to make up for it by hitting up a bunch of different places with different approaches to moderation and different kinds of communities.

</comment></comment></comment><comment><USER39> _He worries that Hacker News will become what he calls "an old crumbling building."_

I hope not. This is one of the best & brightest online communities I've ever seen.

I learn so much every single day from the content & insights posted on this site, I'm not sure where I'd be without it :)

</comment><comment><USER40> > With 1.6 million page views a day (...) Hacker News

> Reddit has seen as much as 4.4 million page views in a given day

These numbers didn't seem right to me; reddit gets way more than 3 times hn's traffic.

Last December, reddit had 2 billion page views [<http://blog.reddit.com/2012/01/2-billion-beyond.html>] which works out to over 60 million a page views a day and over 30 times hn's count.

<comment> B-Con I don't get it either. 4.4 is way too low to be Reddit's current page view average. (Now, 44 would be closer.)

</comment></comment><comment><USER41> I for one am glad the community has been diluted to the point that I can participate.

</comment><comment><USER42> I genuinely appreciate HN. I'm a convicted hacker and am nowhere as bright as some of you guys on here (I have my moments).

Being accepted by you guys really does help me get back into tech and do what I know best. THANK YOU!

<comment><USER9> hey Michael,

I've read your story the other day and I thought it was both amusing and clever, even if it was also clearly unethical. 5 years seems disproportional but great to see you on the outside.

If there is any way in which I can help you to jumpstart your career should you need it or throw work your way let me know and I'll see what I can do for you, email in my profile.

<comment><USER42> Thank you! I got 15months, the OP on that thread got 5yrs. Definitely, I appreciate the contact, I'll email you soon (programming right now).

</comment></comment></comment><comment><USER43> HN is becoming my main tech related hub recently. Almost any article worth reading from the big tech sites will find its place here and comments are noting short of amazing sometimes. The depth on some technical topics surpasses even stackexchange.

Also if there is downwards slope in the quality it is very mild - for the two years I have been here it is quite consistent.

<comment><USER44> My main problem with HN recently is that we're getting _way_ too much political crap on the front page. And, along with that, we seem to be getting more a skew to the "left" (in the modern, American sense of the term) which I'm not crazy about (being neither exactly "right" nor exactly "left" I don't want to see HN moving in either of those directions).

I also get the feeling that "groupthink" is a growing problem and that people are getting downvoted for simply having the audacity to go against the local "received wisdom", but that's pretty subjective and I may be wrong.

</comment></comment><comment><USER45>
      On the backend, Hacker News runs on one core, and Graham calls this a "remarkable feat of scaling."

Wow, Really? I'd say that is pretty remarkable, yes. Is there anywhere that goes into more detail about this?

<comment><USER46> That's actually a remarkable feat of _not_ scaling.

<comment><USER47> Scaling is surely about handling demand, which HN seems to do a decent enough job of.

</comment></comment></comment><comment><USER48> _If you are a YC founder, your username will show up in orange to other YC founders to enable these entrepreneurs to recognize and meet each other._

I've been aware of this for a while (a few YC founders I got to meet confided this to me), but it is the first time I've seen it being publicly admitted.

Along with the DMCA[1] link below, I'm happy to see this ever increasing level of transparency on Hacker News. I believe it is a great way to foster trust among the community.

[1] <https://news.ycombinator.com/dmca.html>

</comment><comment><USER49> To those who have been here from the early days, thanks for letting us newer folks crash the party.

I don't have a lot of +50 comments, but I try to self-censor if I'm not getting more +2s than not.

No idea if or when eternal September kicks in here, but... I guess the upside is what pg has proved: anyone dissatisfied with online communities can just build a new one, use a singular purpose and a stripped-down ad-free interface, and people will come (for better or worse).

</comment><comment><USER50> 4 hours per day! Thanks, pg

</comment><comment><USER6> It's interesting to hear that all moderation is done by just a few people. It's easy to knock mods, but I think the moderation is pretty good here.

I like the choice that people make here. People decide to be constructive and useful. It'd be interesting to see how that can be replicated in other places. Reddit has some good subs. Ask science and ask history are great. They have big moderation teams who frequently delete comments. (That's a good thing!). SE is good for their very narrow tasks - they really need to open an ad- supported open discussion version of the sites. At the moment there is a site, its meta site, and its chat site. Having a noodling discussion site would allow people to have that interesting chat stuff.

It's interesting that wikipedia, while being brilliant, is also really toxic and unpleasant for some people.

</comment><comment><USER51> "The idea of a VC having its own news aggregator was a bit outlandish in 2007."

This coming from Tech... crunch/fund?

</comment><comment><USER52> Interesting, the only thing I really learned though was the orange names between YC founders.

</comment><comment><USER53> I might be mistaken but I get the sense that, especially to older members, it's not just "declining quality of discussion" (uninformed/irrelevant, discompassionate, outright rude, inflammatory comments) that is threatening the value of the site. There may actually be a less tangible/measurable quality, that of the early "feel" of the community, that was of great value to those members. It might have come from the particular set of voices that made up the bulk of comments in the early days, or the relative obscurity of the site. While discussion quality can be improved, I doubt some members can ever feel fully satisfied, without the old feel.

Also, there are many innovations in comment systems. Gawker's Kinja system is quite good at surfacing or organizing interesting comments, among other things. Post segmentation and comment filtering are techniques used by Slashdot and Reddit, which dampen the impact of growth on community "feel". But, despite the massive growth, there is strong aversion to incorporating any deep changes here.

I perceive two specific challenges: prominent voices might actually want a feeling that is (or appears to be) impossible to restore, and there's general apprehension about messing too much with the magic formula that brought HN to the point it's at today. With the influx of users, this appears to create an identity crisis for the site.

I wish I could suggest solutions along with these observations.

Edit: I do think it would be useful to understand what members of varying karma levels subjectively/qualitatively perceive to be the values and valuable traits of the site.

</comment><comment><USER54> Man I miss Flashdot.

<comment><USER55> Perhaps she meant ſlashdot.

</comment></comment><comment><USER56> It's not a decline in overall quality, just a decrease in signal to noise ratio. Which is a consequence not of user growth but of the startup culture becoming more mainstream and less nerdy.

Still my favorite site and primary source (directly or indirectly) of new knowledge.

</comment><comment><USER57> I am brand new to HN (and reddit (I'm a noob)). I am a hacker though, I think, and love reading the articles that I would have no other way to read. I am trying to be a more involved member to the hacker/developer community. I think that more people is a good thing overall, it brings more view points and opinion to the site. I do see the downsides to the growth too, though I think that the good far outweighs the bad. I ma not a new member that is a troll or a bad commenter and I think most new users to HN are the same way. Just my $0.02.

</comment><comment><USER58> If you want to see on which date you joined Hacker News, click your login name in the right top of the page (<https://news.ycombinator.com/user?id=Username>) and note the 'created: 1397 days ago'. Then plug in this number into a date calculation:

      $ python
      >>> import datetime
      >>> print datetime.date.today() - datetime.timedelta(1397)
      2009-07-22

Interesting. I joined in the summer of 2009 ;-)

<comment><USER59> Wow...

      >>> datetime.datetime.now() - datetime.timedelta(days=1894)
      datetime.datetime(2008, 3, 12, 16, 24, 16, 996897)

Time really flies.

</comment></comment><comment><AUTHOR> If memory serves me right, there was a lot of discussion in the early days of HN about _not_ being Reddit (when Reddit was much more low quality and getting beaten by Digg).

<comment><USER33> Reddit got a bunch of its initial users via a link or two from PG essays. Indeed, there are some of the early users there who I recognize as early users here, too. It was pretty good at that stage. And then reddit started growing, and linking to all kinds of stuff like Ron Paul, and lots of "outrage" stories, and other flamewar topics, and promptly went to hell in a handbasket.

</comment></comment><comment><USER60> I would like to see an article on The Evolution of TechCrunch.

</comment><comment><USER61> _> Could This Be A Business?_

Without a doubt. In fact, if a prominent hacker in a non-english speaking country set up something similar, it will definitely become influential.

Hatena Bookmark of Japan used to be like the old reddit with predominantly hacker subject articles, but it grew and became mainstream, kind of like new reddit. Now the hacker community there is yearning for something like hacker news; something that is _for them_.

<comment><USER31> It's officially a business! It's called YC.

<comment><USER61> well I mean HN standalone.

</comment></comment></comment><comment><USER62> You can tell that the quality of the comments (and hence of the community) is going down when you can just by looking at the title of a post, predict the type of negative comments that you will see inside.

However the question (which might not have a satisfactory answer) is whether you can keep the same quality standards _while_ still growing. I think that forcibly keeping the community small can be worse than dilution itself.

</comment><comment><USER63> Maybe a de-amalgamation is in order. I'm thinking a sort of inner ring system, where higher karma users contribute but lower karma have read-only. I'm sure this has big issues, just throwing it out there

<comment><USER14> What incentive is there for new users to stick around if they can't contribute to threads though? How do they reach the karma threshold where they can participate? You can't necessarily correlate age and karma with quality.

 _that said_ I did once have a fiendish (and probably terrible) idea for a similar system where you could only post if you had a certain amount of karma, but also, where you could buy karma points outright.

<comment><USER64> Without karma it's very difficult to submit links that anyone will see.

</comment></comment></comment><comment><USER65> Thanks a lot for HN, PG.

</comment><comment><USER66> I'd really like to see the YC alumns. in orange. It would make reading the comments so much more interesting : )

</comment><comment><USER67> I think I might be hellbanned :/

<comment><USER14> Not yet.

</comment></comment><comment><USER68> part of me wants HN to remain somewhat of a "secret" so it doesnt become reddit.

</comment><comment><USER69> Well, there goes the neighbourhood.

</comment><comment><USER70> Is this a press release?

<comment><USER71> How can it possibly be?</comment></comment>


Russia starts blocking LinkedIn website after court ruling - <AUTHOR> http://www.reuters.com/article/us-russia-linkedin-idUSKBN13C0RN <comment><USER2> So store them in Russian WGAS? As long as members can access them there is no reason why user data must be exported. Keep it in Russia and mirror it in various places and synch the mirrors actively. I suspect every country will make similar demands, get used to it</comment>


FastMail adding new features to keep your account even more secure - <AUTHOR> https://blog.fastmail.com/2016/07/18/new-features-to-keep-your-fastmail-account-even-more-secure/ <comment><AUTHOR> It is not clear to me, but it looks like they're removing the "alternate" login option where you could use a weak password you setup along with your two factor token to get access. I liked this because it made logging at at work easy since I don't have my personal password manager on my work machine. I am sad to see it go, but it's nice to get 2FA for regular logins as an option.</comment>


Astroturfing? "Why I'm not worried about Japan's nuclear reactors." - <AUTHOR> http://webcache.googleusercontent.com/search?hl=de&q=cache%3Ageniusnow.com%2F2011%2F03%2F15%2Fthe-strange-case-of-josef-oehmen%2F++&btnG=Suche&lr=

<comment><USER2> Google cache of a story about a google cache? The link raises questions, but I can't tell that it answers any of them.</comment>


Linear Searching 200MB with Ruby, Node, and Go - <AUTHOR> https://www.jonathanotto.com/linear-search-benchmark <comment><USER2> In Go, using strings.Index (from the standard library) will probably out- perform the naive algorithm used in the article — it's well optimised and, for that string length, will use Rabin-Karp search.

\- [https://golang.org/src/strings/strings_amd64.go?s=720:752#L1...](https://golang.org/src/strings/strings_amd64.go?s=720:752#L17)

\- [https://en.wikipedia.org/wiki/Rabin–Karp_algorithm](https://en.wikipedia.org/wiki/Rabin–Karp_algorithm)

One would imagine that similar algorithm implementations also exist for the other languages too.

</comment><comment><USER3> pypy 2.7.13 ~700ms Python 2.7.13 ~63s

[https://gist.github.com/mattbillenstein/39f0399e842c63355c74...](https://gist.github.com/mattbillenstein/39f0399e842c63355c74173e34d49965)

<comment><USER3> string.find takes ~40ms</comment></comment>


Testing Database Transactions in Go - <AUTHOR> https://marvinblum.de/blog/testing-database-transactions-in-go-jEaOGXravM <comment><USER2> This covers one possible deadlock, but there are so many more ways to do this. More common (IME) is a true logic deadlock, particularly if you are using strict serialization (say, your DB stores a ledger of financial transactions).

It's actually really hard to test that your DB is using transactions correctly, since the error cases tend to be races which don't show up under low load.

We built a prototype tool in Python where you can monkeypatch the transaction context manager, and pause the primary thread's execution just before committing, so that you can then do evil stuff like running a competing thread of the same or other DB operations to try to break invariants. But even this won't catch everything; there is a combinatorial explosion of test points and you can't compare them all in a large app.

I haven't seen any folks writing about this, another approach would be to wire up some him thing smarter like Jepsen to direct the anomaly search, Kyle said some folks have reported doing this but nobody has published. I'm interested to know if anyone has had success with this sort of transaction/correctness testing.

<comment><USER3> Most people use queues to do this. At least I do.

</comment></comment><comment><USER4> The easiest way I have found to avoid this error is simply not to pass the `db` around so much. Every data access function (even seemingly trivial ones) should take a `Tx` instead. This also makes your code "composition-ready" if necessary later.

IMO the standard library should provide something that wraps a `Tx` and a `Context` together, as usually I want every statement issued from a transaction bounded in lifetime by the same one I provided to `BeginTx`. This would provide even more incentive to pass around `Tx`s rather than `DB`s, since those functions also often need a context anyway.

<comment><AUTHOR> I agree. I usually have a "model" package abstracting the database access behind functions accepting a transaction. But since I'm using the max open connections for all tests, I can be pretty confident I don't have open transactions somewhere, so I often take the shortcut and just query on "db" directly.

</comment></comment><comment><USER5> Don't forget to measure first. You can metric out the number of connections periodically and see if you are capping out. See [https://golang.org/pkg/database/sql/#DBStats](https://golang.org/pkg/database/sql/#DBStats)

<comment><USER6> Thanks! Thats a really good tip. I didn't know I had easy access to the db client stats!

</comment></comment><comment><USER7> In order to get proper error response when deadlock occurs in testing (not just in Go), use isolation level serializable on every transaction / connection. This is simplest most effective way I have found so far, also you can get in detail why/where it happened. Watch out for different packages test executions since they run in parallel by default, use

go test -p 1

</comment><comment><AUTHOR> Hey guys, this is just a quick tip on how to make sure you don't screw up the database connection pool in Go. This might also apply to other languages and frameworks. I've seen dozens of articles about concurrency and multithreading, but not so many about this type of deadlocks, and I experienced some production bugs because of this.

<comment><USER8> Maybe I'm assuming too much but it wouldn't occur to me to let tx.Commit() depend on the outcome of another connection (here: db.Query()). Either use tx.Query() (which I guess is partly what you're advocating) or put db.Query() in a different goroutine and let tx.Commit() proceed.

It is quite similar to other Go concepts. For example, you don't want to have circular dependencies between channels. The difference is that such a thing would fail very quickly whereas dependent DB connections would only fail after the connections are exhausted.

<comment><USER9> There's a lot of people who have a lot of SQL database experience in a synchronous, one-thread context, who know how to use transactions and have simply never thought about the way the implicit guarantees of a synchronous, one-thread context harmonize conveniently with transactional usage. Once you have interleaved computations, either via some sort of callback system or via threading, suddenly there's some extra concepts to understand, such as the way most database systems (if not all of them, but I'm hedging here) tie transactions to the specific DB socket you are communicating with, and this concept tends to poke its way up to even the highest-level API. APIs like Go's designed to support this by having an explicit concept of "transaction" also tend to be tricky because it's still so easy to use a non-transactional select even in the middle of a transaction just by accident, because it's surfacing a concept that the accidental context of the other APIs people have used always took care of automatically, with no visible signature on the API.

I don't think there's necessarily anything "special" about this, I think it's just a case where there's a lot of people with years or even decades of experience in that context, and it simply doesn't immediately occur to them that in a higher-concurrency world they need to modify these skills.

I suspect there's also rather a lot of database + application combinations out in the real world that, by coincidence and a bit of hacking around problems as they arise, "just happen to work" with the highly characteristic access patterns used by the web pages that can access the DB and the transaction isolation settings in the DB. Using any more concurrent and looser access to the DB is likely to expose a lot of problems that in some sense existed all along, but were just never quite uncovered before with the old access patterns.

(It isn't really Go. Threading an old-school C program will raise the same issues, or going async in a scripting language.)

</comment></comment></comment><comment><USER10> Python context managers are a great way to prevent this type of issues. Anyway feels like a bit strange to not use the current connection (from between the transaction) to query things. Perhaps the problem is between the API, because it's not explicit the borrow/leave of connections from the pool.

</comment><comment><USER11> This site is blocked on my office proxy under "Adult/Mature Content" category

<comment><AUTHOR> Yeah because of my last name :( Some filters seem to be a bit too aggressive. I copied it to Medium:

[https://medium.com/@dekugelschieber/testing-database- transac...](https://medium.com/@dekugelschieber/testing-database-transactions- in-go-12cf617cbe21)

<comment><USER12> What is "blum"? Google is just returning kitchen cabinet parts, etc.

<comment><AUTHOR> Blume means flower in german. But the "urban dictionary" spits out weird translations for it, and it seems like some blockers are configured in an interesting way :P

</comment></comment></comment></comment><comment><USER13> This problem is solved in most languages by wrapping endpoint code in an interceptor that creates and cleans up the transaction for you.

I'm not a fan of Go because it encourages hacks like this. The language is not expressive enough to handle database transactions properly. So it's suggested to try to catch bad situations with a linter instead.

What a timeless Goland solution to the problem. They should put "fix it with a linter" on hats and sell them at Go rallies. You could make billions

<comment><USER4> > an interceptor that creates and cleans up the transaction for you.

I'm not sure about the word "interceptor" here because in Go and every other language I know which does this it is merely a wrapper, but Go does this. From some vantage, the problem only arises _because_ Go does this - if it instead forced you to create a transaction the resource allocation, and so possibility of exhaustion, would be a lot more obvious.</comment></comment>


Open Source "Coca-Cola" - <AUTHOR> http://cubecinema.com/cola/

<comment><USER2> I'm going to buy one of the Open Cola kits. I'll probably set up a tiny static website about it, so if HN is interested I'll post the link here.

I kind of want to investigate adding a bit of cinnamon - I think Pepsi Max uses it and it's tasty but way too sweet.

<comment><USER3> I have always wanted to test the cola without the sweeteners. (no sugar, corn syrup or aspartame like stuff). Can you post impressions - I am definitely interested in the stuff.</comment></comment>