Ask HN: In what areas are NoSQL Databases beneficial over Relational Databases? - <AUTHOR> I have been working with MySQL and SQL Server for quite a long time and thought of trying MongoDB. I realized that I won't be having the ability of joining a lot of tables and all. Preferably what areas would you suggest that NoSQL databases would be beneficial over the traditional relational databases? <comment><USER4> In my opinion, the major reason to leave behind ACID databases is if you have reached the point where they can't scale up any more. If you have reached this point, you're probably a company with 20+ developers. There might also be very specific use cases where specialised databases are useful, such as a search index.

ACID gives you a lot of nice guarantees and it's silly to give it up if you have the choice.

As for MongoDB in particular, I would never use it for anything. It doesn't seem to do anything particularly well. There are better specialist 'NoSQL' databases that have actual benefits to fit your problem.

<comment><USER5> I thought about trying out MongoDB. Which NoSQL are better in your opinion?

<comment><USER6> [https://www.rethinkdb.com/](https://www.rethinkdb.com/) looks very promising and also introduces some very interesting concepts and ideas.

<comment><USER7> RethinkDB got me interested some time ago and I took a look how they differ from SQL and found this page.

[https://www.rethinkdb.com/docs/sql-to- reql/javascript/](https://www.rethinkdb.com/docs/sql-to-reql/javascript/)

Frankly, I cannot understand to write queries that verbose and complex when SQL looks clear and concise.

If I'm missing on how good RethinkDB is, I will gladly like to know.

One part SQL is notorious for is you don't know where it could fail until you send the whole query and receive the error unlike these NoSQL where operations can happen progressively.

While ORM can sort of mimic that behavior, I don't like that it hides SQL which makes crappy SQL alot easier to write and harder to debug.

</comment></comment></comment></comment><comment><USER8> First of all, this question is similar to asking in which cases non-cars are beneficial over cars. Cars cover many use cases but sometimes bicycles perform better in heavy trafficked cities, planes carry you faster to the other side of the world but you need a cargo ship to move tons of containers there. That list goes on and on, up to less common activities like jumping fences (horses) or very rare ones like going to the moon.

Now, there are many use cases for non relational data stores. There are already many answers in the comments but first you have to ask yourself what you need the data store for. In the cars analogy it's where you need to go, how fast, carrying what, etc. Do you need a glorified hash table? Redis is good at that. Do you need very fast queries? You might try Cassandra but be very careful at defining in advance all the queries you're going to run: if you need sorted results you have to plan the columns for them. See [http://www.planetcassandra.org/blog/we-shall-have- order/](http://www.planetcassandra.org/blog/we-shall-have-order/) and remember that NoSQL doesn't always mean schemaless.

Sometimes the choice is about the tradeoffs between the ease of use, installation and planning. MongoDB maybe doesn't particularly excel at anything but you install it with an apt-get (check in advance which version you get) and adding replication is easy. However you better start planning at least a partial schema for your documents quickly or you'll get ton of litter in your database. On the cons side, its native JS based query language is awfully verbose and complex compared to almost anything else, ORMs and especially SQL. Luckily we don't need the console much.

To recap: ask yourself which data you have to manage and why, then google for a database that optimizes your use case and check it against the posts at [https://aphyr.com/tags/jepsen](https://aphyr.com/tags/jepsen) which does a super-excellent job at finding out how dbs fall apart in extreme but realistic cases. Often you'll find that a relational database is good enough. Other times you'll have to choose between relational and one or two categories of NoSQL. There are very few mainstream RDBMs but there are zillions of NoSQLs so the choice might not be obvious. Final car analogy: car + tent or camper van?

</comment><comment><USER9> When clients ask this question, I always suggest starting with a good SQL solution and then figure out why it won't work. So if you're not hitting any major problems with MySQL or SQL Server, then NoSQL solutions will just be incredibly labor intensive for, from your perspective, a loss in functionality. Each NoSQL database is different and designed to approach problems from SQL databases differently. For Mongo, this is having a very flexible denormalized schema with very fast reads. However, companies such as Pinterest and Facebook have found they can accomplish this with MySQL or Postgres through different data modeling. I think the reasons for NoSQL become much clearer when trying to build applications that do not keep a normalized data model.

</comment><comment><USER10> This is the WRONG QUESTION.

The _real_ question is:

In what areas are (recently build in the internet era) NoSQL Databases beneficial over (barely-)Relational _traditional RDBMS_ Databases (build decades ago)?

\----

Scalability, the easy to mutate schemas and all that are artifacts of _our times_.

MySql, Sql Server, etc are _barely_ relational databases, coupled with a almost-decent way to interface them (sql).

For example, in them you can't store a relation inside a field. Schema manipulation is hard. A lot of potential programming power is not possible, without convoluted recent additions the Sql language, or hacking together strings.

And most of them have been made for workloads and scenarios that are at odds to what internet-scale companies need.

Unfortunately the solutions was do "Nosql":

"Any sufficiently complicated database management system contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of a good RDBMS."

\---

Check the relational model. Is far simpler than people give credit, even without get too crazy like some purist want.

In fact, I don't see any reason why the relational model can't be used as flexible how json is applied in the backend.

\---

Despite all the above and more, a traditional RDBMS is solid enough that _most people not need_ to get crazy and move to (full) NoSql. However, could be good to use them in specific cases (like caches, search engines, etc).

But I have experience people using NoSql stores and trying to do the kind of work a RDBMS have solved easier and faster.

</comment><comment><USER11> The benefit: typically near infinite scalability with little loss of performance. It's super fast even when there's a lot of it.

The cost: transactions, relations. These are actually incredibly valuable things to have.

If you're storing billions and trillions of records that don't need relations, NoSQL is great. You'll keep your fast lookups and saves no matter how big things get.

This can matter a lot more if you're a start up with the goal of having trillions+ records some day. Your SQLServer will eventually not be able to scale any higher, and then what? Moving to Nonrelational from relational is so painful.

<comment><USER12> "The cost: transactions, relations. These are actually incredibly valuable things to have"

This is quite arguable. Before the Computer Era, all businesses, including the whole financial sector did just fine without ACID. Still, ACID is used only at a local scope. I can often see ACID requirement used rather as an excuse for poor data modeling than a real need.

<comment><USER13> > This is quite arguable. Before the Computer Era, all businesses, including > the whole financial sector did just fine without ACID.

Can you elaborate a bit? I believe that in the pre-ACID world, business processes were much slower, not online processes like today. When you're only really changing data once a day, backups and manual intervention are acceptable options.

> Still, ACID is used only at a local scope.

What do you mean by "local scope"?

<comment><USER12> ACID is not a way to deal with faster data change. Actually it is totally opposite. The more updates you have and the more online (distributed) you are, the less you want ACID. ACID is extremely latency-sensitive and failure- sensitive. And latency is a problem caused by just plain old physics - <USER2> isn't going to get better unless we find a way to make light travel faster than the speed of light ;) And the more distributed and larger your system is, the more frequent failures (like network partitions or servers going down) will happen.

As for "local scope" I meant "at a single business entity". If you look at a banking or online payment systems as a whole then they are not ACID, they are eventually consistent. The are based on two basic principles:

1\. write everything down at least two times, so you never lose data

2\. updates are incremental, so you never overwrite data

<comment><USER13> ACID sure is a way to deal with data change faster than once a day. Once you stop passing paper slips and order forms through your company, you can get accurate, up to date information at a human timescale, i.e. on the order of seconds.

Of course when you want stuff to happen at a faster-than-human timescale - <USER3> when you just want _lots_ of stuff to happen - an ACID model might not be the right fit.

> As for "local scope" I meant "at a single business entity".

Understood.

<comment><USER12> My whole point is that ACID is not the only kind of consistency and it is not required to do "get accurate, up to date information at a human timescale". It is just a nice and convenient programming model, but it comes at a cost of availability, scalability, latency and throughput. You can design systems with no ACID data store and if you do it properly they can be just as accurate and up-to-date when everything works properly, but can also degrade nicely when some components fail.

[http://highscalability.com/blog/2013/5/1/myth-eric-brewer- on...](http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on-why-banks- are-base-not-acid-availability.html)

[http://www.enterpriseintegrationpatterns.com/docs/IEEE_Softw...](http://www.enterpriseintegrationpatterns.com/docs/IEEE_Software_Design_2PC.pdf)

</comment></comment></comment></comment></comment></comment><comment><USER14> NoSQL is for heterogeneous data that is read in a homogeneous manner, whilst relational excels at homogeneous data read in a heterogeneous manner. If your schema is constantly changing, relational is painful, conversely writing reports against a NoSQL datastore hurts too.

If you come from an RDBMS, you're probably going to dislike MongoDB. I like it for prototyping/MVP but once the schema settles down it's time to move.

However, NoSQL does not start and end at MongoDB, no way. There are lots of different flavours, from huuuuuuuuge key/value stores, to timeseries, to write-one-read-many document stores, to remote syncing mobile app datastores and probably loads of other types that I haven't used yet.

So figure out why you want to use NoSQL (CV building can be a reason too) and play around with some toy installations.

</comment><comment><USER15> I'm still a newbie so I have a hard time articulating this, just a couple years into my career. The "rule of thumb" that I've worked up so far is that document stores are great for records that are useful in isolation but are possibly heterogeneous in nature. (of course there's nothing stopping you from storing foreign keys and doing "joins", or just storing data with a strict schema). Logs and configuration are both simple use cases.

One example from my work is ingesting usage information from a variety of products, like a system that takes in information about all your utilities - power, water, gas, and so on. Each product will have its own attributes that's important to describing it. In a relational world, you might: add columns to your table as new attributes need to be described; come up with some kind of composite attribute system where you try to encode multiple pieces of information in one field; go for a star schema so every product gets its own table; or pivot so that each "record" is represented by multiple rows and the attribute's identifier is a field. Each of those approaches has their ups and downs but they're all present due to hard schemas. In a NoSQL world, a single table can have wildly varying records, each of which is useful on its own without needing to spread metadata into other tables.

</comment><comment><USER16> I find them useful in many scenarios:

\- Bootstrap projects with open or changing requirements. A relational model is usually harder to maintain.

\- Storing and querying objects with complex structures, such as those received from APIs.

\- Having a single programming language across your entire application stack (ie MEAN). Sometimes comes up as a "should" or even a "must" requirement.

\- Fast and simple object storage (ie sessions).

<comment><USER17> > \- Storing and querying objects with complex structures, such as those > received from APIs.

> \- Fast and simple object storage (ie sessions).

Those would also be fulfilled by using a jsonb column in Postgres, right?

<comment><USER18> >Those would also be fulfilled by using a jsonb column in Postgres, right?

Storing JSON is now natively supported in MySQL 5.7

<comment><USER19> Right, the point being that there is no need to choose between key-value and relational DB - since Postgres et al do it all. Which makes the so-called "noSQL" approach completely obsolete.

</comment></comment></comment></comment><comment><USER12> The two things, that Relational Databases typically don't do well: 1\. High availablity and no downtime, ever. 2\. Unlimited scalability of all operations, including writes.

Honestly, many (most?) NoSQL databases also don't really do them well. If you need these two properties, the only valid choice is really a master-less, shared-nothing database system. E.g. Apache Cassandra.

</comment><comment><USER20> The only time you should use NoSQL is when you have no other choice. For example, if your database is too big to fit on one server, or you need a massive number of writes per second. Otherwise SQL will give you better performance, much more security, and better data integrity.

<comment><USER21> And much, much, much easier querying (if you know SQL). For me, the biggest disadvantage of non-sql databases is the difficulty of doing even the simplest aggregates!

</comment></comment><comment><USER22> I would suggest to try using both SQL and NoSQL for some hobbies project. From there you should be able to experience the difference between these db. In a fairly complex application, you will see SQL, NoSQL and Redis, all exist for different purpose.

</comment><comment><USER23> For a personal project I used MongoDB over MySQL for somewhat faster prototyping. I was collecting a lot of data from the X display server during runtime and storing it into a database for querying later. Since there were a lot of different types of messages and I wasn't sure about which parts I wanted to keep, rather than building tables for each of them I converted them to JSON and piped them into a MongoDB collection.

I know this is not a good example compared to a production issue, however I think the reducing the amount of time need to get started with a project is very useful over something that needs a lot of configuring before it can be used.

</comment><comment><USER24> NoSQL is all about missing useful features (such as integrity, transactions, query flexibility), that you unfortunately have to drop if you want to be able to scale in certain ways. Thus, NoSQL dbs are practically worthless until you get to a point where your SQL database won't work any more. At this point, you need to evaluate exactly what is it that your SQL db cannot handle, and switch to a different product accordingly. For example, you would switch to Aerospike if you need to scale writes.

</comment><comment><USER25> Different NoSQL platforms solve different problems. Graph databases can build and query relationships between entities with significant speed. Mongo has been mentioned a few times because it's easy to iterate development with a schema-less database, but it's also fantastic when dealing with class based structures with sparse attribute populations (e.g. a CMDB). Cassandra's notion of an always available database is a key foundational element in high-scale devops environments. Scylla offers incredibly fast transactions (they advertise 1M transactions per second per server). There are XML databases that store and query documents in ways that are easy for developers to translate (i.e. xpath). There are databases like Axibase that are built for time series data. BayesDB is easy to query for statistical inference.

It's very easy to fall into the trap of "everything is relational" or "I could do that with Postgres/Oracle/etc". There are a lot of problems that have good RDBMS centric solutions, but you don't have to look too far to find end users or developers who are unhappy with the RDBMS solutions that they work with on a daily basis.

</comment><comment><USER6> With RDBMS you have to design the schema first, with NoSQL you don't need to. This is nice when you're dealing with very diverse data (or data you don't even know how it'll look like) or are prototyping. One thing I like about this: You can store your data as it is, you don't have to fit it into your model / world view. You can make sense of it later.

At some point you'll probably have to deal with schema anyway. If this comes at a later stage, you'll have to deal with a lot of diversity and inconsistency on potentially many points. This can become much more painful and time consuming than having dealt with it from the beginning.

If you're thinking about NoSQL as something like a simpler, modern SQL alternative, you're probably having a faster start and a lot of problems in the long run.

Scalability is something, NoSQL databases have for them, definately.

<comment><USER26> > With RDBMS you have to design the schema first, with NoSQL you don't need > to.

Not necessarily, if you have a nice framework. Years ago, I was working on an older Perl application with a tiny self-built ORM (a single class of about 1000 LOC). I gave it pluggable storage backend support and added a prototyping backend where the object is stored into a simple SQL-backed key-value store (a table with columns "id", "type", "data"; where "data" contains a JSON document). So when building something with a new data type, you could just start with the prototyping storage, and once your feature was working, switch to table-backed storage without much code changes.

</comment></comment><comment><USER27> At work we use NoSQL for stats collection and aggregations. Our reason for choosing MongoDB is that we can have a rich document structure with multiple levels of nested documents. And with the rich update queries that MongoDB provides, we are able to update all of those nested documents in a single call, without having to update multiple documents.

That's one of the places where it made sense to use a NoSQL solution. Our documents had multiple levels of nesting, but no joins. We also use MySQL for our relational data, so in the end you have to pick and choose. All of the research we did pointed to NoSQL being a great fit for storing nested documents for stats collection. We haven't regretted following that advice yet.

</comment><comment><USER28> I use MongoDB for periphery data to my applications, i.e. to aggregate data such as logs and events. It's also great to store things like reports in it due to the flexible structure. However, for my actual application database I always use SQL (MySQL).

</comment><comment><USER29> Depends, there are many types of NoSQL databases, that all have different focuses.

SQL started getting big because it was the one size fits all approach that worked 90% of the time. In fact it got so big, that universities started to teach only them when they talked about databases in general.

But there are many problems, where SQL is not suited.

If you got hundrets of terrabyte of data you need to analyse, you're probably better of with a Hadoop cluster.

If you need inter process communication, Memcached or Redis is worth considering.

If easy db-sync is your main goal, couch/pouch-db is probably a good choice.

etc. pp.

But these three DBs alone are so different, that they can't be compared like MySQL and PostgreSQL can.

</comment><comment><USER30> NoSQL is amazing when you have non relational data. Think of it as a giant, distributed hash table. Hash tables are awesome and wicked fast. But toss in relations and now you got some interesting problems. RDMS solves that for you but scaling is now harder. At least in my opinion anyway. You can still use NoSQL in these situations but you're typically duplicating data or indexing it yourself so you can refer back to it.

It's all about trade offs. If you're small just pick the best one for the job at hand. When you need scale you can usually hack either into scaling well enough.

</comment><comment><USER31> There are some specific scenarios where nosql solutions simply work significantly better. Probably the premiere example is time series data. You can use sql, but it's a pain. There's a reason why most monitoring systems use some sort of nosql database. A specific business domain example is multi-level marketing. Updates in SQL can take hours as one has to go through the entire tree to make any updates. Graph databases are a much more elegant solution to that problem domain.

</comment><comment><USER32> The relational model is useful for being well understood and part of the reason it is well understood is that the relational model is founded upon relational algebra and another part of why the relational model is well understood is availability in the wild roughly corresponding with the creation of the personal computer.

Search versus query [for some definitions of 'search' and 'query'] seems like a case where business logic suggests an advantage of one approach versus the other.

Good luck.

</comment><comment><USER33> typically the advantages of nosql are along the lines of schema-less, speed of writes, reads should be faster typically as well.

speed of writes, there are no constraints that need to be checked with nosql (typically), it's like writing a value into a hash map. with sql there needs to be a check for multiple constraints: uniqueness, null, datatype, etc. if there's a relation then it needs to check those constraints as well.

the whole thing about rdb's is that they're trying to normalize data where possible, you deal with id's and the fields of the table can change without you needing to propagate these changes to other places. in nosql, if you have relational data then you have to manually make sure all these changes are propagated, pretty annoying if you ask me and quite error prone, you end up writing logic an rdb already optimizes for you. take "cascaded deletes", too, this is manual in nosql.

the best usage of nosql is if you have stagnant data, and you need to write it quickly, take a log file contents, for example, or if your data consistency requirement is not that critical (let's say you implement likes for a web post system, if you're missing 300 here or there, not a huge deal). other cases are things like counters, if you have a game where you're tracking scores, things are changing rapidly , but there's really not a lot of relation with other data, redis is a great example of this type of nosql storage, built in incrementing counters.

most of the time when someone brings up nosql for an application, my initial reaction is that it's a premature optimization, for a lot of data, there are tight relationships and rdb is great for that. but i tend to see that you need both in many cases.

</comment><comment><USER34> You may be interested in this, it's a Jun 2016 paper on the current state of "NewSQL" and describes most of the dbms out there including NoSQL. [http://15721.courses.cs.cmu.edu/spring2016/papers/pavlo- news...](http://15721.courses.cs.cmu.edu/spring2016/papers/pavlo- newsql2015.pdf)

</comment><comment><USER35> In the early days of companies MongoDB has helped me tremendously with development speed because it is schemaless. Since Postgres 9.5 I'm trying to replicate the speed with Postgres binary JSON but most access libraries are still SQL oriented and I haven't achieved the speed of MongoDB with Scala/Rogue yet.

<comment><USER36> I must admit that the term 'schema-less database' sends shivers up my spine. How do you manage that, especially in a large development team. My biggest fear is that someone will make a change to the structure in an ad-hoc fashion which may break something else I am working on.

End of the day, can't most things in the real universe be broken down into a schema? Be it DNA mapping, geological data, customer invoices, product listings etc.? Everything eventually can be contained in a discrete, defined schema?

I've dabbled on and off for several years but have yet to come across one situation where NoSQL would be a better solution than SQL. As others have pointed out - as soon as you start relating table together, NoSQL starts working against you.

I acknowledge that I have 30+ years with SQL, so obviously my viewpoint is biased by what I know. Could the hesitation to adopt SQL by most new developers point to the fact that they find it too intimidating and structured?

<comment><USER26> > I've dabbled on and off for several years but have yet to come across one > situation where NoSQL would be a better solution than SQL.

Full-text search. Just compare the speed of ElasticSearch against "SELECT * FROM logs WHERE message LIKE '%foo%';".

Also other niches: time-series databases, graph databases, etc.

<comment><USER17> Actually, I've done tests with an indexed postgres database of several gigabytes size and tsvector for stemming and co vs. ElasticSearch.

At least until you reach several hundred gigabytes, postgres uses less resources for the same search speed.

This video shows searching through 3 GB of IRC logs, filtering, ranking them based on similarity to the query, creation time, likelyhood that it's the type of message the user wants, highlighting the part of the message with the matching words, and displaying all that:

[https://dl.kuschku.de/videos/2016-09-16_04-03-36.mp4](https://dl.kuschku.de/videos/2016-09-16_04-03-36.mp4)

Postgres is, for smaller amounts of data, more than fast enough.

</comment></comment></comment></comment><comment><USER37> On the topic of joins, I'd suggest if you're bumping into the joining limitation that you've chosen the wrong database for your needs. Having said that, a lot of it can be solved by rethinking your structure to take advantage of embedded documents.

</comment><comment><USER38> Nested data structure.

I'd recommend other NoSql databases over MongoDB like RethinkDB.

Also, SQL like databases support JSON. So Postgres and MySql 5.7.x I think supports it.

The only think I don't like about sql is that I have to know what my data looks like in advance. Set up is a pain

</comment><comment><USER39> For me it's all about not having to define a schema. I can just take my JavaScript object and insert it in a collection. Sure, I know you can do this in PostgreSQL and I might switch.

</comment><comment><USER40> a good place for NoSql is where you don't know what data you are going to receive. e.g. i get weather data from a number of different sources in a number of different formats. the sources and formats change fairly regularly. i am able to extract the data in key/value pairs but don't know which keys will be in any given set of data. with nosql you can extract any key/value pair and save it to the database without needing to parse the data

<comment><USER41> You can just shove json or XML you receive into a SQL database, too.

That can be good strategy when dealing with changing data formats:

1) Import the data "as is" into the database.

2) Process the data, extracting the data you need for querying.

3) Whenever parsing fails, figure out why and fix it (update your source code, throw away bad data, whatever), then rerun the 'process' step for that data.

Reparsing can be made reliable if you use transactions in the step 2. With a NoSQL solution, it may be hard to guarantee that you don't lose a few records ('may' depends on the specific NoSQL solution and the amount of manual work you are willing to do to restart your pipeline). (You can also postpone discovering your data problems, but that's delaying the inevitable)

What NoSQL is particularly good at is running on multiple machines. It achieves that by giving up some letters of ACID.

It also typically makes it easier to store and query unstructured data than SQL databases do, but SQL databases are catching up.

</comment></comment><comment><USER42> No mention of BigTable so far - ACID compliant NoSQL with transactions IIRC. But you have to host with Google. (For at least the data store component)

</comment><comment><USER43> I heard that the are better for the environment and smell nicer too.

</comment><comment><USER44> If you need data integrity you will need a RDBMS.

</comment><comment><USER45> A lot of the entities we deal with in web apps are complex, multi-table structures when stored in a relational database. Think about blog articles (tables: article, author, tag, comment), user profiles (tables: user, friends, follower, interests, etc.), photos, tweets, ads, and such. The beauty of something like MongoDB is that you can store all the data as one "record" in a collection. You can also retrieve it with a simple query. These "records" are structured as something like JSON, so you can just take the data structures you're using in your app's code (particularly Python or JavaScript), and store it as-is with no conversion.

Compare this to an RDBMS-based web app, where to display a blog article for example you'd need to make multiple queries of different tables: first the "article" table to get the text, then the "author" table, then all the comments, tags, images, or whatever. The NoSQL way is much easier to code for this sort of thing. The lack of ACID doesn't matter so much because these kinds of data are usually written just once, read many times, edited rarely, so there are few opportunities for inconsistencies to creep in.

<comment><USER46> >for example you'd need to make multiple queries of different tables: first the "article" table to get the text, then the "author" table, then all the comments, tags, images, or whatever

Haven't you heard of joins?

<comment><USER45> You're just outsourcing the impedance mismatch problem to the database. Yes, we can write complex SQL (and I've written some _really_ complex SQL) to combine multitudes of entities together and generate a result. NoSQL databases eliminate this.

Where it becomes extremely impactful is when a database is distributed or sharded across a cluster. If the various tables (article, author, comment) are stored on different servers, it takes time for the database to reassemble them. Moreover, modern web apps often have content that's written once, read many times: blog posts, photo posts, status messages, news articles, advertisements, catalog listings, etc. Why re-do that work of assembling the content every time people want to look at it? When documents are rarely or never edited, the relational model's defenses against "anomalies" become less relevant. Far better to use a document store like MongoDB, which stores a piece of content as an integrated whole at a single place in the database.</comment></comment></comment>


So Long and Thanks for all the Bits - <AUTHOR> http://jacquesmattheij.com/Tell+HN%3A+So+Long+and+Thanks+for+all+the+Bits <comment><USER2> _The downside for me is that I'm a person that is of an addictive nature and I can't do things half..._

Me too.

I have always struggled with how to allocate my limited time. Every time I have wanted to do something in the past few years, the first question that came to me was, "How much time will this take away from my work?" At first, if the answer was nonzero, I didn't do it. But I have slowly learned that nonzero answers were OK, and in many cases, quite welcome.

 _About a year ago I vowed that I'd quit HN on the day that I'd pass 50,000 karma..._

Funny, some time ago, I vowed that I'd never quit HN, no matter how high (or low) my ROI. I was one of the first ones here and plan to be here to help turn out the lights (perish the thought).

Jacques, please understand that you are part of my addiction to HN. Your contributions have been priceless. I suspect you have reached the point in your life where you have a responsibility to "pass on" the gifts you have received. You probably have many other ways to do this, but none, I imagine, as effective as Hacker News.

Your contributions here have changed lives. I hope you haven't pulled a Matt Maroon and obfuscated your password. I urge you to reconsider. It _is_ possible for an addictive overachiever to get most of his work done and still be part of HN. I am living proof.

Part of being a successful achiever is not giving up. I hope I'm not alone in not giving up on you. There must be a workable solution to your problem. Consider us resources to find it.

What do we have to do to help you continue to be part of this community without sacrificing too much else?

<comment><USER3> Terribly addictive things: drinking, video games, social networking, and online forms.

Is it ethical to prod your alcoholic drinking buddy because he's fun to be around? Maybe the one not wanting to be alone has a problem too and just can't see it.

When somebody admits they found a negative influence in their life, congratulate them, acknowledge you'll miss them, then let them move on.

<comment><USER4> While I agree with the sentiment, I think that the results of an HN addiction can be more productive than the results of alcoholism. Contributing to reasoned debate on HN probably helps to educate a lot of people even if they don't participate themselves.

</comment></comment></comment><comment><USER5> So first thing this morning I scanned HN.

Then I scanned the rest of my 30 startup tabs in my browser.

Coming back to HN, and wanting to get some work done, I finally forced myself to close the tab.

Two hours later I succumb to my inner demons and open HN back up, and there's JacquesM saying he's bailing.

Yay Jacques! Boo Jacques! I respect the hell out of you and I'm going to miss you. I even at one point thought about coming by and visiting, and then it occurred to me: what would we do? Sit in a coffee shop and post comments back and forth on HN? Something wrong with this picture.

So I think you are doing the right thing. My current strategy is to use stopwatches and such to limit my participation -- so far with very mixed results. I am able to use HN as sort of a reverse productivity indicator: the longer I stay on HN any one day the less likely I am to feel like the day was well spent. That's not HN's fault by any means: I think there's just something naturally addictive about interacting with lots of other people who have the same skills and interests as you do. HN is my watercooler. And that's a good thing and a bad thing.

Remember that if you come back we will all forget the fact that you left. Kind of like your friends at the bar forgetting that you went to rehab. :)

<comment><USER6> This may come out as selfish, but what about the rest of us? What if all the high-karma uber-commenters leave? I think you guys have a responsibility to the community to at least dedicate _a little_ time to HN. Please, please don't stop contributing altogether.

<comment><USER7> I upvoted you, because it's an interesting, if misguided, point.

Nobody gets paid for posting on HN or having a high karma. We sure as hell don't have a responsibility to keep donating our time to post interesting stuff... any more than an avid networker has a responsibility to turn up at all the networking events!

<comment><USER6> That you're not paid for commenting on HN is certain; that people sometimes do things out of charity is equally obvious :-) I wasn't intending to imply an obligation, so perhaps "responsibility" is the wrong word. A crude analogy (since it's the first one that pops to mind) is how sad the Ruby community way when _why left.

</comment></comment></comment></comment><comment><USER8> Interesting. I've been working on a blog post entitled "My slow disengagement from HN" about why I had been slowly walking away. But people might as well read yours.

I've been consuming HN through the newsyc50 Twitter account which only shows me news items that have > 50 points. I started out with newsyc20 (> 20) and have been weaning myself off.

I've found that stopping reading the conversations has been the easiest part, but that HN's filter of interesting stories is hard to beat.

<comment><USER9> "I've found that stopping reading the conversations has been the easiest part, but that HN's filter of interesting stories is hard to beat."

(imo) the conversations/comments make HN more than the stories. When I read the comments before the stories, I get a different perspective(than reading stories before comments).

</comment></comment><comment><USER10> I hope you (and everybody else in the community) have peace, prosperity, and happiness, whether on HN or wherever your journey leads you.

</comment><comment><USER7> Sorry to see you go! I hope you're not leaving the IRC channel though :-)

For what it's worth, for those who feel HN is a bit too engaging but still want to read the best, most useful startup articles every day (according to who? according to me!) do follow <http://swombat.com> as a low-engagement substitute. Swombat.com will never take over your life - you can't even comment there! :-)

(Email subscriptions are coming soon, for those who don't even want to open up the browser)

<comment><USER5> I wish I could double-upvote that, swombat. It was a heartfelt message of sadness, and also a plug for your website.

I can't imagine a better way to send off Jacques. True HN spirit.

(And for those of you who think I'm being snarky, I am not. HN is about trying things out to help people. It is fitting that, when presented with a problem, we should share the things we have been working on to help folks with that problem. Outstanding job. I only wish I had something to offer along these lines.)

</comment></comment><comment><USER11> I use all addictive web services primarily on my iPhone:

* Facebook

* Quora

* Twitter

* Hacker News

The interface is slightly more awkward - but not annoying - and I generally check my iPhone when doing something that is inefficient (eg: making tea) so I don't feel bad about using web services there.

I find its made a big difference to my productivity.

<comment><USER12> Same here (on an android). It makes me less inclined to comment, which can be a real timewaster as then you are committed to the thread. Normally I touch type so I don't have the patience for the SW keyboard.

</comment></comment><comment><USER13> Jacques, you are one of my favorite commenters. I'm sad to see you go, but I definitely understand your reasons, this place is way too addictive. I've added your blog to my Google Reader list [1], I hope you'll write there often. I especially love the monthly idea dump.

 _Het ga je goed._

[1] For those interested, the RSS feed url is: <http://jacquesmattheij.com/blog/1/feed>

<comment><USER14> Thanks, needed that. Autodiscovery didn't seem to work on his blog.

</comment></comment><comment><USER15> Two words: Evaporative Cooling.

<http://news.ycombinator.com/item?id=1777665>

[http://searchyc.com/submissions/%2522evaporative+cooling%252...](http://searchyc.com/submissions/%2522evaporative+cooling%2522?sort=by_date)

<http://news.ycombinator.com/item?id=1934367>

<comment><USER16> An alternate name for the result is the Dead Sea effect: [http://brucefwebster.com/2008/04/11/the-wetware-crisis- the-d...](http://brucefwebster.com/2008/04/11/the-wetware-crisis-the-dead-sea- effect/).

However I think that it is too early to get _too_ concerned about hn.

</comment></comment><comment><USER17> Well, I certainly didn't see _that_ coming.

I'm a bit speechless, but thank _you_ for your participation, and all that you've given the community over the years.

<comment><USER18> I just freakin' told you. _Just_ told you.

Edit: this is why I'm always irritated

<http://hackerne.ws/item?id=2161059>

</comment></comment><comment><USER19> My personal solution to the HN addictiveness problem is to use the noprocrast, maxvisit, and minaway that are built into your user settings.

Mentioning here as it was probably 6 months before I realized these existed - from the original thread:

"Like email, social news sites can be dangerously addictive. So the latest version of Hacker News has a feature to let you limit your use of the site. There are three new fields in your profile, noprocrast, maxvisit, and minaway. (You can edit your profile by clicking on your username.) Noprocrast is turned off by default. If you turn it on by setting it to "yes," you'll only be allowed to visit the site for maxvisit minutes at a time, with gaps of minaway minutes in between. The defaults are 20 and 180, which would let you view the site for 20 minutes at a time, and then not allow you back in for 3 hours. You can override noprocrast if you want, in which case your visit clock starts over at zero."

<http://ycombinator.com/newsnews.html>

<comment><USER20> Unfortunately Incognito mode on the browser gets me right back in.

</comment></comment><comment><USER6> What, noo! OK, here's the deal, nobody upvotes this submission so he doesn't go over 50K karma :-)

I never corresponded with jacquesm personally but in my relatively brief time in HN came to enormously appreciate his thoughtful and knowledgeable comments, his calm, anti-flame tone, but most importantly his generosity with ideas.

I am perplexed by this decision, though. I spend one to two hours per day on HN on average, assuming he does more, it's a good chunk of time. Yet, isn't stopping contributing altogether somewhat drastic?

I think HN needs all the mentoring it can get and hope that this does not become a trend. As they say: "Too much of a good thing can be wonderful!"

<comment><USER21> There is a little known bug in the HN code, but karma tops out at 65534 and then rolls back to 1. So, upvote this story ~15000 more times and he'll come back.

</comment></comment><comment><USER22> > About a year ago I vowed that I'd quit HN on the day that I'd pass 50,000 > karma, that should be enough for anybody and that point in time has come.

Downvoted. ;-)

</comment><comment><USER23> I joined reddit when I was supposed to be writing my MSc thesis; my karma soared to a few thousand pretty quickly.

Karma is an odometer, counting up how much time you could have been spending doing something else.

It does pay to take a step back from time to time. Fare well, and I wish you all the best.

</comment><comment><USER24> When I want to get work done, I just set my hosts file to block all of the time-wasting sites.

127.0.0.1 localhost

127.0.0.1 news.ycombinator.com

127.0.0.1 www.reddit.com

Etc,

Then if I want to check those sites I have to go through the hassle of opening the hosts file and commenting the lines.

This definitely of keeps me from checking 'reflexively' throughout the day.

Its a solution that even works on Windows and Mac!

<comment><USER25> There is also noprocrast in your profile.

</comment></comment><comment><USER9> Jacquesm, I never written to you before...I have always procrastinated emailing you. Just wanted to say thank you!.

You have been among the articulate members in here. There are so many 'high' posts from you that not even a * top jacquesm 100 posts * would do justice to your contribution. Thank you for sharing your experiences.

Would it be private to disclose what you would be doing in your "HN time"? Any thing new on your mind? I was wondering how you would hold it back without letting us know :) Please do write about your 30 day de-addiction experience

Good luck to you (and hope you (don't) have a relapse!) :)

</comment><comment><USER26> I look forward to reading the follow up memoir about your time on HN, ' _Mostly Harmless (Assuming Time has no Value)_ '.

It is tempting to downvote some of your old comments to keep you below the 50,000 limit. But as others have pointed out, there will be plenty of other opportunities to connect and learn from you. I'm sure the productivity boost that awaits you will be a boon to us all.

For specifics, I found your sharing the experience with ww.com and video in the early days of the web were incredibly interesting and edifying. Thank you for that.

</comment><comment><USER15> I'm pretty sure I'd prefer to be remembered for most insightful contributions, rather than just as "Best Nick". Still, I'll take what I can.

No doubt we'll still be in touch. I have some follow-ups that I've promised, and are still on the way.

Best.

<comment><USER27> Would you care to fill in a noob on the meaning of your nick?

<comment><USER28> He literally rides giraffes.

<comment><USER15> Yup. I have a giraffe, and I ride it. In suitable circumstances I also ride giraffes that belong to other people.

Email me if you really, really need to know more.

<comment><USER26> _"I also ride giraffes that belong to other people."_

Naturally. Otherwise your nic would be RiderOfGiraffe.

<comment><USER15> Unless I had more than one. Which I don't.

</comment></comment></comment></comment></comment></comment><comment><USER29> Jacques, I'm sad to see you go. I myself remember telling my girlfriend that I was going to quit after 2k karma, and then I was going to quit once I made it to /leaders... and here I am, posting a comment.

Maybe someday I'll have the same degree of self-control you do.

<comment><USER30> The one person who most strongly encouraged me to come back after I initially had decided to bow out of HN participation was none other than jacquesm (<http://news.ycombinator.com/item?id=779116>). I am glad I came back but now, sadly, there can't help be but be some material diminution in quality and sociableness with the departure of one so intelligent, encouraging, and stimulating. Jacques, I can only say what you said to me: "You'll be missed . . . I learned from [you] . . . I hope to see you back here some day!" I wish you well.

<comment><USER29> Well, if jaquesm convinced you to come back, that's just one more great thing he's done that he won't be around to do anymore...

</comment></comment></comment><comment><USER31> It's a bit early for April Fools, Jacques. Nice try. I give it about a week.

If you are in fact serious and able to break the engagement I'm sure I won't be alone in wishing for the speedy return of your contributions.

</comment><comment><USER32> Congratulations Jacques! That's a very good decision. We will stay in touch privately, but I agree, it's best to get hooked on more useful things.

Cheers!

</comment><comment><USER33> This is the kind of message that reminds me that HN is not eternal.

That one day, it will be remembered as a precious community, surprising in the quality of its members and the stability of its standards. A sort of modest, modern day digital Athens if you wish.

So let's enjoy it to the fullest while it lasts, and thank you Jacques, for all the invaluable contributions you brought to the community.

</comment><comment><USER34> Classy exit. Should you find cold turkey not working for you, let me suggest this might be an opportunity to hack yourself and figure out how to participate in a non-addictive manner, which could be a profound positive change in how you operate generally (since you indicate you have "an addictive personality").

Peace and good journey.

</comment><comment><USER35> Jacquesm- You were the first person I ever "noticed" on HN, and whenever I'd scan through comments I'd always stop to read yours. While it's sad to see you go, I understand all your reasons. I've been involved with other communities and had to do the same thing.

Good luck, and like swombat said - I hope you stick around the chat. :)

<comment><USER36> > the first person I ever "noticed" on HN

Same for me. Is there any statistics like average number of posts of the user per day? I'd be really curious to see that number for "high karma" users.

As I've noticed him, I actually had an impression that JacquesM really does only sit in front of HN pages, refreshes, reads and responds.

Afterwards I also found out myself why and how HN is so addictive.

<comment><USER35> Well, by "noticed" I just mean that I noticed that I kept upvoting posts by this "Jacquesm" guy. I had noticed other people existed. :)

</comment></comment></comment><comment><USER37> Wow, 50K karma in just 785 days? That's almost 64/day, quite a feat.

For me, the discussions in the threads are the best part of HN. The submissions are great; but what brings it all together is the discussion.

<comment><USER38> That's certainly some serious evidence in support of his addictive nature. How much time each day would you have to spend on HN to get 50K in karma, even allowing for a person who has a high level of quality submissions and comments? Seems like it would take up a significant chunk of one's life and I could see why he might want to quit cold turkey.

<comment><USER29> While this is true, there's a reason that the mean isn't always used in descriptive statistics: outliers can easily skew this. I've had a few multi- hundred-point submissions, for example, and while 64karma/day is nothing to sneeze at, you'd just need one or two really solid comments or a good submission per day.

Then again, considering the way I'm phrasing this, maybe it's more of a sign of my own addiction rather than something a reasonable person would say...

</comment></comment></comment><comment><USER39> If I had known you'd be quitting after 50,000 karma, I'd have downvoted more of your comments!

<comment><USER40> I'm rather new to HN, so I was wondering, can I buy some of that Karma on E-bay?

<comment><USER40> Mi falta. No offense intended by the tongue in cheek comment. Perhaps not the place....

</comment></comment></comment><comment><USER41> Jacques, I can understand and respect your decision. Thank you for your years of engagement and contributions. Hopefully, we'll stay in touch.

</comment><comment><USER42> Sucks to see you go, you've been one of the better posters on here(if not the best). But I know exactly where you are coming from...HN is just a huge time sink and really hurts your productivity...something new and interesting shows up every hour, so you don't have time to actually get things done.

The best way I found was to eliminate HN from any bookmarks and navigation bars. When you have to type stuff in to get here, you really cut down on your visits.

I think HN itself is a net positive overall since you get to learn a lot of what's happening out there without checking 30 websites for news. The problem are the comments. You check to see what people are saying, then come back and see if there were any new ones or if someone responded to a comment you found interesting.

What pg should do is add a productivity option that doesn't show the comments link. That way you still get the stories, but aren't distracted by the discussion.

A quick and dirty version of this would be to set .subtext { display: none;}

</comment><comment><USER43> So one of the superstars of HN is leaving?

Jacques, it was wonderful to know you and read your thoughts here on HN. HN will miss you dearly. While I respect your decision to leave, I'd urge you to reconsider the decision to leave completely. And if you decide to come back, we will be as welcoming as we always were...

Till then, so long my friend.

</comment><comment><USER44> Well, that's too bad. I don't know if what works for me will work for you, but I find it very hard to check the news sites (or the web) when I'm working on something very interesting.

I'm guessing you wouldn't have to go if that did work for you, so thanks for your help and support over the months I've been here.

</comment><comment><USER45> Quick! Everyone downvote comments from jacquesm and let's see if we can get his karma back below 50,000.

In all honesty, like many others here I completely understand and wish you the best.

The other day I saw the "lists" on HN and wondered what those karma leaders get done in addition to gaining karma on HN.

</comment><comment><USER46> I actually found your blog via HN and have since too become a regular user of the network.

</comment><comment><USER47> It was a pleasure Jacques.

Well gentlemen, this how you make a graceful exit. Round of applause.

(HN Roast anyone?)

</comment><comment><USER48> Jacques, thank you for all the wonderful posts. Really enjoy your insights and the sharing of your life stories. I'm impressed with your desire to continue to learn new stuff and asked questions.

I completely understand your motivation to quit HN. It takes a lot of time and effort to write well thought out posts. Taking a break is good. I hope you do come back for reading. HN is a nice source for news. I found that not logging in helps to stop the urge to post. May be, just may be, after a while we'll see your posts, however sparsely.

Good luck and wish you success on all your endeavors.

</comment><comment><USER49> I'm actually really happy to hear this. While we'll all miss you terribly, every minute you're on HN you're not putting your brilliant mind to much more important things.

<comment><USER50> Hmm, maybe. How relatively important is his direct and indirect mentoring that took place here?

His life, his priorities, and I salute him. But if we're talking about impact, what happens here is significant.

<comment><USER49> I'm sorry I didn't say it better. I'm not saying his mentoring was inconsequential; I'm saying he has even more important work ahead of him.

</comment></comment></comment><comment><USER51> Jacques, I remember you telling me once that karma here at HN was as gameable as it is anywhere. I remain convinced that you've gamed it by making consistently high-quality, thoughtful, well-supported comments and posts helpful to the people likely to vote here. You've certainly earned your HN retirement, if it's that time; but the community will be poorer for it.

</comment><comment><USER4> A little off-topic, but I wonder if it would be easy/hard to write some sort of proxy rule that strips all cookies sent to the news.ycombinator.com domain. That would prevent you from ever being able to login, but would still allow you to maybe browse the frontpage. (Or even better strips all cookies, and only allows access to the frontpage or just the RSS feed)

</comment><comment><USER52> Finally, I can rise to prominence as The Other Jacques!

edit: I can't read this thread, I feel like everyone is addressing me personally :/

</comment><comment><USER53> Have you found a new similar community to join? In the sense a new community with fresh "ideals" you are thinking of joining or already did. It's been more then 6 months I've been trying to find a good place (I need a way to keep track of trends).

I think I've assimilated all HN could give me. And good luck!

</comment><comment><USER54> I feel a lot like how I felt when _why "died". Someone who made great contributions to a community I love is leaving said community.

Best of luck, Jacques. We'll miss you, and we'll always love to have you back, whether to return full force or just ask for feedback on a startup and then leave again.

</comment><comment><USER55> Would HN participation be more manageably casual if the home page kept popular stories around for more than a day? Then you wouldn't feel like you had to visit every day just because you might miss something.

</comment><comment><USER56> Thanks for the kind words about my nic ;), and good luck to you.

Perhaps one of these days someone will succeed in inventing the _optimally_ addictive HN experience, and we'll look forward to seeing you then...

</comment><comment><USER57> Heh, going cold turkey to kick the "addiction", probably the only way. Wishing you luck with it. The place will be weird without you :)

Thanks for all your contributions Jacques, drop me an email any time :)

</comment><comment><USER58> Au revoir, not good bye. I'm sure you'll be back, for old times sake.

</comment><comment><USER59> I am really, really bad with names -- even more so on Hacker News where the names are grayed out...

But jacquesm is definitely one of the few that stuck with me. You will be missed!

</comment><comment><USER60> Sorry to see you go and thanks for all the great insights over the years. Hopefully you stop by a few more times to read all of these appreciative comments.

</comment><comment><USER61> Jacques, thanks for what you've contributed here. You are a voice of reason and capture the spirit of why I lurk and occasionally participate on HN.

</comment><comment><USER62> You have helped me both indirectly through your contributions on HN and personally one on one via #startups on freenode, and for that, I thank you.

</comment><comment><USER63> Jacques, succes met je volgende avontuur ;)

</comment><comment><USER64> Thanks for your great contributions Jacques ... it'll be strange not seeing you around here anymore. Best of luck!

</comment><comment><USER65> I will miss your contributions, which I always find thoughtful and often find enlightening. Best of luck to you!

</comment><comment><USER66> Awww - but, all the best to you, sir. Less traffic and faster pageviews does sound good though.. :-)

</comment><comment><USER67> A very sincere thank you, yours were the comments that would be on the top of my list. Good luck!

</comment><comment><USER68> Cheers, Jacques. We never knew each other but I always appreciated your insights. All the best!

</comment><comment><USER69> i just (almost) quit smoking the <http://www.sql.ru>, a #1 russian database guru community where i have 30k posts, and now you say that hackernews is as addictive?

oh, no...

</comment><comment><USER70> I hope now you can concentrate better on your other endeavours. Good luck!

</comment><comment><USER71> Happy trails, Jacques... enjoyed your commentary and perspective.

</comment><comment><USER72> i totally understand the sentiment. i just set my maxvisit/minaway to 20/1300. one visit per day is healthier. perhaps no visits per day is healthiest for me as well...

</comment><comment><USER73> Thank you, Jacques! We'll miss you :)

</comment><comment><USER74> Your contributions will be missed.

</comment><comment><USER11> You're a good guy. Good luck. :)

</comment><comment><USER75> Will miss you. Farewell.

</comment><comment><USER76> All the best, Sir.

</comment><comment><USER77> Bye!

</comment><comment><USER78> Jacques has been a tremendously valuable member of this community and a true friend to me. Asking nothing in return he has always been there to help.

I remember one day a couple months ago everything that could have possibly gone wrong with my business/life... did. It was one of those moments they always warn you about in the startup literature, and I was not expecting it at all. I didn't even realize what was happening - that this was one of "those" moments - I really, really was ready to quit.

Jacques talked me through it. I was ready to throw in the towel and he was there talking to me on IRC and email and cheering me on.

It was one of those things that you just don't forget. I never forget people who are there for me when I really need them.

Jacques is a friend and the entire community is for the worse from his departure. The plus side is that since Hacker News is now slightly worse, hopefully the downward trend will continue and he will come back again, restoring equilibrium.</comment>


Stealing Bitcoin with Math - <AUTHOR> https://speakerdeck.com/filosottile/stealing-bitcoin-with-math-hope-xi <comment><AUTHOR> Video: [http://livestream.com/internetsociety/hopeconf/videos/130745...](http://livestream.com/internetsociety/hopeconf/videos/130745035)

<comment><USER2> thanks a lot for this link, google didnt reveal it (for me at last) :D <3

</comment></comment><comment><USER3> I see these headlines and I put on my math bib and pick up my math fork and get ready for a hefty meal. But all for naught each time; it inevitably comes down to, "Oh yeah someone doesn't protect against nonce reuse and then lols occur."

It should no longer be the case that platform vendors are able to abrogate responsibility for bad RNGs or RNG bugs. The stakes are too high for people to get it wrong. Even in 2016 we're still seeing repeats of browsers and OS's with "predictable random number generator" bugs even though we have a pretty clear handle in the literature for how to do it well.

</comment><comment><USER4> That point when you realise you're responsible for the data in someones presentation.

<comment> ryan-c Any details you'd be willing to share?

<comment><USER4> The graph displayed of ECDSA duplicate r-value exploits shows 2 prominent "columns" of addresses, the latter of which was in April/May 2014. That latter column was directly related to a commit that I made to the bitcoinjs-lib master branch (which was undergoing major refactoring at the time).

The commit that fixed the issue: [https://github.com/bitcoinjs/bitcoinjs- lib/commit/bc37e65014...](https://github.com/bitcoinjs/bitcoinjs- lib/commit/bc37e650148db2e5d947c84e51167cbd8f37b8a6#diff- dac4a0f35ab99ccdd0234335f29bae2bR63)

The issue itself was that a `Buffer` was being interpreted as `0` by crypto- js's cryptographic hash functions in our implementation of RFC6979, thus creating a case of duplicate `k` values.

The second most interesting point was the majority of the funds (>20k USD) stolen from Counterparty (the only known users of our master branch at that time) was returned by a grey hat.

</comment></comment></comment><comment><USER5> Brilliant work ryan and fillipo! I was at the talk, and very impressed with both the content and your presentation style.

</comment><comment><USER6> I'm curious, would stealing bitcoin like this actually break any laws?

<comment><USER3> Theft is theft when value is converted or transferred without consent. If someone leaves their house unlocked and the door open and a neighbor pinches objects, it is still a crime to take said things. Most jurisdictions recognize this even if the actual mode of theft or the goods are somewhat unprecedented so long as you can demonstrate that it is property.

Bitcoin's classification in the US means that it's SUPPOSED to be subject to all the scrutiny and protections that convertible currencies are supposed to have, although I can't speak to the regulation of that.

<comment><USER7> I think in some juridictions you might be viewed as having provoked the theft. And provoking the theft seems to annihilate the prosecution at least when it is done by police (eg leaving laptop on seat in car under surveillance to arrest thieves), as has been the case in the past in (I think) belgium.

<comment><USER3> > I think in some juridictions you might be viewed as having provoked the > theft.

That's a question of intent and reasonable interpretation. I'm fairly sure "I didn't audit the browser PRNG," will not be looked upon by any court you can name as a reasonable thing to expect people to do.</comment></comment></comment></comment>


Plant Biology Informs the Origins of the Stradivarius - <AUTHOR> http://www.scientificamerican.com/article/plant-biology-informs-the-origins-of-the-stradivarius/ <comment><USER2> tl;dr: Plant morphologist who plays recreational viola applies the approach used to study how leaf shape varies by species and genus over time to the study of the shapes of violins, etc., developed by four major luthier families.

'That the study sets aside differences in sound makes it unique. "This is respectable research, halfway between science and history in terms of academic disciplines... revealing things about the story of the violin that were not obvious before reading it."'

(I'm also a recreational violist. How many more of us are there, I wonder?)</comment>


Facebook officially loses $123B in value - <AUTHOR> https://techcrunch.com/2018/07/26/facebook-officially-loses-123-billion-in-value/ <comment><USER2> So I looked for the FB stock price and found this gem of an article: [https://www.zacks.com/stock/news/312831/should-you-buy- faceb...](https://www.zacks.com/stock/news/312831/should-you-buy-facebook-fb- stock-ahead-of-q2-earnings)

> Based on Facebook's continued importance to advertisers, its growing user > figures across its many platforms, its solid valuation picture, and strong > earnings trends, Facebook looks like it might be a stock worth buying before > it reports its Q2 financial results after market close on Wednesday, July > 25.

<comment><USER3> Stock analysts should have to share what percentage of their portfolio is in any company they write about.

</comment></comment><comment><USER4> > There are barely more people checking Facebook every day compared to > previous quarter. Even worse, Facebook's user base shrank in Europe. > Facebook is still growing, but it's clear that GDPR combined with a > saturated market aren't helping the company.

Facebook is running some _desperate_ ads to try to compensate for this. Literally: "Join Facebook, it's FREE! 2 billion people already have!"

<comment><USER5> The NYC subway ones are straight dystopian:

[https://media.wired.com/photos/5ae2367a7e275409d957991e/mast...](https://media.wired.com/photos/5ae2367a7e275409d957991e/master/w_2264,c_limit/FacebookAds.jpg)

<comment><USER6> War is peace. Freedom is slavery. Ignorance is strength.

</comment></comment></comment><comment><USER7> There is a less well known news coming out recently. Due to a legal case in Washington state, Facebook agrees to not to prevent customers to receive AD based on ethnicity, religion or sexual orientation.

This is mostly making targeting ad illegal and hurt the fundamental revenue model of FB.

[https://www.bloomberg.com/news/articles/2018-07-24/facebook-...](https://www.bloomberg.com/news/articles/2018-07-24/facebook- agrees-to-change-ad-platform-to-prevent-discrimination)

</comment><comment><USER8> It doesn't really make any sense that FB stock should be dropping that much based on slower user growth and revenue.

The main risks to social networks have to do with network topology. If the network starts getting hollowed out then the mathematics show that social networks can implode even faster than they can grow. But user growth isn't a good predictor of that, growth can be flat and then just start accelerating again; this already happened with FB before around ten years ago.

<comment><USER9> >It doesn't really make any sense that FB stock should be dropping that much based on slower user growth and revenue.

Yes, investors and stock holders are typically the most rational people who never panic.

It also doesn't make sense to hold on to stock if you think this is the beginning of a long, slow decline.

</comment></comment><comment><USER10> I guess this means there'll have to be some belt-tightening in the Zuckerberg household.

<comment><USER11> Halt the construction of that wall!

</comment></comment><comment><USER12> $506.2B to go.</comment>


What's the hippest programming language in 2015? - <AUTHOR> http://spiegelmock.com/2015/08/30/programming-with-the-lowest-common-denominator/ <comment><USER2> This article contains a lot of inaccuracies. To name a few, JS wasn't intended to be a competitor to Java. Java was new and exciting when JS was getting ready to go to market, so Netscape got permission from Sun to use "Java" in the name. It was cooperation, not competition.

Second, JS Harmony does have a module/import system, and you can use it in all browsers with Babel.

Third, the assertion that code reuse is the only reason to use JS on the backend is unfounded. Node was built from the ground up to be non-blocking. There are non-blocking libraries in most languages, but in JS they are all non blocking. That's a huge win for certain kinds of workload.

Those errors and unsupported opinions reported as fact are in the first few paragraphs. Maybe the article gets better, but I don't have the patience to read any more.

</comment><comment><USER3> WTF is going on in 2015? In my time, fashionably-dressed & programmer == false</comment>


Americans Strongly Dislike PC Culture - <AUTHOR> https://www.theatlantic.com/ideas/archive/2018/10/large-majorities-dislike-political-correctness/572581/?fbclid=IwAR0BPxtpnc90jI4kBO9HJaWyZ7jyOPnRrayA4Ynuo35yq3QoTJfusuhu8Hg&single_page=true <comment><USER2> previously: [https://news.ycombinator.com/item?id=18215215](https://news.ycombinator.com/item?id=18215215)

</comment><comment><USER3> America though part of wars, the mainland itself has been protected for many years unlike Europe of constant invasion. Europe has been though all that and doesn't want to again, it understands the cost.

<comment><USER4> Europeans don't like PC culture anymore than Americans do. The rapid and sudden rise of nationalism and nationalist political parties across Europe proves you wrong.

</comment></comment><comment><USER5> _" By contrast, the two-thirds of Americans who don't belong to either extreme constitute an "exhausted majority.""_

</comment><comment><USER6> Knn</comment>


Google Announces Plans To Shutter Knol, Friend Connect, And More - <AUTHOR> http://techcrunch.com/2011/11/22/google-announces-plans-to-shutter-knol-friend-connect-and-more/ <comment><USER2> Kinda wish they would keep FriendConnect. It is a nice container to run Javascript apps on 3rd party sites (it proxies your requests so you can call your home server and serve up data). Also, the polling and newsletter apps that are baked into the FriendConnect toolset was sure nice.

Also it kinda sucks that I trusted them to handle the sign-in for my community. 12K users. {Sigh}

</comment><comment><USER3> I see they are shutting down their clean energy program. The renewable energy cheaper than coal initiative showed a unique level of arrogance. We, a software company, can solve a problem in which we have no experience better than people who have devoted their lives to the subject.

<comment><USER4> > We, a software company, can solve a problem in which we have no experience > better than people who have devoted their lives to the subject.

The false assumption here is that Google is only a software company. Any company with lots of machines is at least in part a power management / hardware company, and in the field of datacenters, Google is likely the best in the world. They put a lot of engineering into getting heat from point A to point B.

<comment><USER3> If they were working on data center efficiency, that would be one thing. The RE<C program was an attempt to make solar power more efficient. This involves basic physics, material science, and manufacturing. Nothing that Google had done to that point would qualify them to work in that area.

<comment><USER5> Nothing Google had done in 1999 would qualify them to dominate internet search or scalability either. This is a fallacy. Asserting that the only people who can productively solve a problem are those who have dedicated their careers to it is just plain wrong. It flies in the face of almost every success story, ever.

That said, clearly they didn't solve this problem. If they were making serious progress, they wouldn't have killed it.

</comment></comment></comment></comment><comment><USER6> Thank god they're getting rid of Knol.</comment>


Aaron Swartz hit with 9 more felony charges in MIT hacking case - <AUTHOR> http://www.dailydot.com/news/aaron-swartz-mit-jstor-hack-charges/

<comment><USER2> RANT: Anyone here interested in the underlying issue of academic publishing? To me, the very notion that publicly (or privately for that matter) funded academic research gets locked behind paywalls of organizations that have not contributed financially to the research endeavor published, seems like irony to me. In short, they pay not a single dime to produce the content, charge the researchers to have their submission reviewed and published, publish the content and make money off of the subscription they sell, and sue the hell out of anyone who tries to wrestle it back out of their control.

<comment><USER3> _organizations that have not contributed financially to the research_

 _charge the researchers to have their submission reviewed and published,_

As you point out, they are providing a valuable service. The research is peer- reviewed and published.

You can argue that they aren't needed in the modern peer-to-peer wikiresearch napster world, where anyone can just publish their research wherever they want.

If the researchers don't want their research behind the paywall, they are not forced into the transaction. They must be finding the value-add of the "peer- review and publish" significant enough to give up publishing rights elsewhere.

<comment><USER4> Unfortunately, this is something that has to occur as a systematic change. Individual researchers are not to blame, necessarily. When you are told by your university that you must publish or face not getting tenure, what they mean is that you must publish in a peer-reviewed journal (or conference, since most of what we talk about here is CS and CS is still a conference field mostly). We could realign the review process so that it was something taken on by universities, for example, but this will require a major systematic change.

Peer review is important. It will not stop being important, but unfortunately the service of peer review is currently only being offered by mostly for- profit organizations, which is the primary problem.

<comment><USER5> So true. It's not like publishing your paper on your lab's website will satsify the "publish" requirement for obtaining tenure. Why can't we separate the process of peer review from the process of letting a journal handle distribution of the paper?

My pet peeve when downloading articles from journals is that it is a chain of needless HTTP redirects and elaborate cookies. If you are accessing the network from an approved IP address, is all that really necessary? Why can't it be a simple direct download? Answer: Because they've commercialized the process of reading publicly-funded research results. And with that comes the usual mindless hoop-jumping for even the simplest things.

Many investigators will just post a copy on their lab's website anyway. And that's the link that they will often give to students who need a copy of the paper. So the whole scheme of commercializing the publishing of noncommercial research just looks silly.

Don't question it, just follow along.

</comment></comment></comment></comment><comment><USER6> Just FYI, this is often a prosecutorial bargaining tactic for dealing with an uncooperative defendant (based on personal experience, anyway).

That said, I wish articles like this one would quit making the rounds. It's macabre theater and isn't doing aaronsw any favors.

<comment><USER7> Is the goal to do aaronsw favors? This is pretty clearly newsworthy.

I think you can ding the reporting for missing important subtleties and lacking context (i.e. the fact that the downloaded material wasn't covered by a JSTOR copyright somehow gets skipped!). But arguing that we should plug our ears to news about a case with clear impact to the community because it might look bad for the defendent is just ... weird.

The point about this being a bargaining tactic makes sense though. They want a plea on something to avoid embarrassment, and it wouldn't surprise me if Aaron was refusing to deal.

<comment><USER8> How does ratcheting up the charges minimize the embarassment to the DOJ if Aaron is ultimately vindicated?

<comment><USER9> The DoJ isn't human and is not subject to emotions like "pride" or "embarrassment." Their goal is to win cases and enforce laws, and if the bureaucratic cost of adding more charges improves their chances of having the defendant found guilty on them, they're successful.

<comment><USER7> The DoJ is part of the executive branch, and ultimately answerable to elected officials who _absolutely_ are sensitive to embarassment. No one wants to see a headline like "FBI wastes $2M on failed prosecution of harmless nerd" when it could be "Hacker gets jail time and fine". So they're throwing mud trying to get something to stick. If it looks like anything does, they'll offer a plea again.

You can't really believe that politics have no impact on the case decisions in the DoJ, can you?

</comment></comment></comment></comment></comment><comment><USER10> I'm not sure I want to live in a society where reading too many academic papers too fast is a multi-felony. _"Swartz's program was so powerful…"_

<comment><USER5> As if Python was the most powerful language, heh. :)

Maybe he was going for a world record of most citations in a single paper. Who's to say he wasn't just doing research? How many downloads is too many?

I would be willing to bet that the JSTOR TOS do not give a specific number. e.g. "You may not download more than n papers in 24 hours." And if they don't state a maximum in the TOS, then why shouldn't they, for clarity?

<comment><USER3> It sounds like you are trying to invoke Loki's Wager -- since you cannot define N where downloading N is too many and N-1 is not too many, there must not be such a concept as downloading too many.

People who deal with the law don't have much patience for this.

<comment><USER5> I'm not sure if I am interpreting your last sentence correctly. Can you rephrase it?

<comment><USER3> Courts have a very long history and lots of people have tried lots of really weird things over the years. Swartz will hopefully have a good lawyer, and a good lawyer won't even try to something along the lines of "there wasn't a preset limit on X therefore my client didn't do anything wrong" when his use of X was over a hundred times the combined consumption of all the legitimate users over two months.

Judges are not computers. If counsel presents them with a bad enough argument, they might get insulted that counsel thinks the judge is dumb enough to fall for it. Things that depends on the judge's mood (like purposefully obtuse arguments) are not a good courtroom strategy.

<comment><USER5> You are jumping around a bit. We were talking about TOS and now we're in a court room and using the words "judge" and "dumb" in the same sentence. I was kidding about doing research. Humor. We all know what he was doing. But the truth is I'm serious about these types of TOS. And I'm looking at this mainlly from the end user's perspective. You see the same type of ambiguous TOS language everywhere on the web. Let's stay focused on TOS for a moment, and leave aside the Swartz case. Do you think ambiguous contracts (TOS) are "better"[1] than unambiguous ones? For example, would reducing ambiguity lower the probability of (costly) disputes?

1\. better for who?

<comment><USER3> ToS might be interesting in some cases, but not in this one. JSTOR and MIT kept on denying access to Swartz and he kept on working around their defenses.

<comment><USER5> What if we looked beyond the Swartz case? Then what do you think about these types of TOS?

Maybe another example would be more interesting. Say you have a choice between an API that allows a "reasonable" number of requests in any 24 hour period and one that allows n number of requests in any 24 hour period. Which one would you prefer?

First assume you're an API user. Then assume you're the API provider.

Anyway, this kind of question is what I was getting at. What is reasonable? I don't know what their server capacity is.

I like using automation, I prefer non-interactive to point and click, and I have always found TOS on academic databases, not to mention most websites, interesting. Because they fail to account for anyone who might want to use automation (reasonably, having respect for the resources of the server). But maybe I'm the only one who finds this question interesting.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> EDIT:Make my point more clear

Edited comment:

It seems to me that Federal law is able to be applied to websites created by private individuals or businesses. As long as the ToS has knowingly been broken and the person doing the breaking has benefited materially then he is at risk of federal prosecution. I don't see a lot of comment about reasonable the ToS has to be. This just strikes as being completely irrational.

Original comment:

Riiiigght, so I can put together a website with some strange terms of service and then the FBI will come arrest anyone breaking those terms of service because it is a federal crime.

Just what planet do these guys live on....

<comment><USER8> No, you cannot do that. A prosecutor must demonstrate, first to a grand jury, then to a criminal jury, and simultaneously to a judge, that the accused not only violated the terms of service to a website, but in doing so caused material harm or found material gain, _and_ that they did so knowing that they were violating the terms of the site.

<comment><USER11> The fact seems to remain that any private individual can create a website and have the force of federal government apply to a third party as long as said party has knowingly broken the ToS and gained materially somehow from that.

<comment><USER8> Don't knowingly break the terms of service of websites for profit or to harm others. Sounds straightforward to me.

<comment><USER12> If a restaurant had a 'terms of service' that said no reselling of food that they make, should you be open to federal prosecution if you stopped by to pick up food for yourself and some co-workers? Especially if someone gave you a few dollars for the trouble of running the errand? And even more, if the restaurant has their own delivery service?

Am I missing an important distinction here? Should private companies be able to make binding rules that open people up to criminal prosecution for something that doesn't violate any laws per se? A person breaking a specific law AND breaking a ToS makes sense. A person breaking a law BY breaking a ToS doesn't make sense.

<comment><USER8> I can't reply to this because no part of the example you provided constitutes a federal crime under the CFAA.

On the other hand, the criminal aspect of using a university's noncommercial JSTOR access to scrape a substantial portion of the entire database so you can put it on BitTorrent is not hard to understand.

<comment><USER12> It was unclear if what you were implying was that you shouldn't violate a ToS for your own profit (or at someone's expense) because it was a federal offense. (I can see now that that was not what you were trying to get at)

What are your thoughts on PadMapper vs CL? What is the distinction between scraping that data vs scraping this data that makes one worthy of federal prosecution, but not the other? Considering in both cases it was done for profit or detriment

<comment><USER8> Elements missing from a CFAA case for PadMapper include _at least_ interstate commerce and intent to defraud.

Swartz's prosecution alleges --- credibly, given what Swartz allegedly posted prior to scraping JSTOR --- that Swartz intention was to liberate data from a commercial database onto file sharing networks, making intent a much easier case to prove. Moreover, the indictment is at pains to point out that MIT and JSTOR repeatedly attempted to stop Swartz from continuing his plan, and found themselves in a cat-and-mouse game with Swartz eventually trespassing to maintain access.

PadMapper found itself having exceeded Craigslist's terms, found out by having its access withdrawn and becoming the target of a civil suit, and did not (directly, at least) attempt to evade the countermeasures Craiglist applied to prevent them from obtaining further access.

Whether or not you believe Swartz did something wrong here (I do) or whether you think he should get a felony conviction for doing it (he probably shouldn't), you can see pretty clearly how JSTOR had no straightforward civil remedy to what Swartz was doing. Swartz was playing chicken with them, and he lost --- or rather, his bicycle collided with JSTOR's semi truck at high speed.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> This could be very troubling for scrapers, as they frequently breach the terms of service. If successfully prosecuted it would encourage and empower sites with content that get scraped to push for criminal charges and win given this precedence.

<comment><USER14> Are you suggesting that mere scraping alone would demonstrate sufficient intent to warrant prosecution? Can you elaborate? Various courts have had mixed interpretations on the enforceability of sites' terms of service. How would this impact the body of case law that has been building around automated scrapers such as the many "... vs Google" cases? Also, "precedence" is ranking something in priority; "precedent" is the legal term.

<comment><USER8> The indictment goes into great detail as to how Swartz would have known his actions violated JSTOR's terms, and how he repeatedly took surreptitious steps to continue his plan despite the obvious efforts of both MIT and JSTOR to stop him.

"Mere scraping alone" is unlikely to land you a federal charge; the prosecution needs to demonstrate your intent to act unlawfully. A far more typical outcome for a scraping case is a C&D from the site you scraped.

For obvious reasons, JSTOR can't C&D Swartz once their content hits BitTorrent. Similarly, if you scrape a site and post it to file sharing networks, you might have something to be concerned about.

</comment></comment></comment><comment><USER15> This is potentially terrible precendent essentially allowing private entities to write federal law.

<comment><USER8> What?

<comment><USER16> Step 1: Write a ToS for your private entity

Step 2: Catch someone violating it

Step 3: Sic the Feds (government) on them for violating it (your private, unvetted agreement).

(no comment from me personally, but I think that's what the GP was saying.)

<comment><USER17> I'm pretty sure sneaking into a server closet and patching your computer into their network was already against the law before this incident happened.

<comment><USER18> It was a janitorial closet, not a server room. And connecting to a network is not illegal.

<comment><USER17> You're asserting that it's legal for you to plug your computer into a piece of networking infrastructure that's clearly not intended for general public use, without the permission of that network's owner?

I'm going to have to slap a [citation needed] on that.

<comment><USER18> It's the same network that's available elsewhere on campus, so it's not like he was connecting without authorization. Aside from possibly trespassing, what kind of crime would it be?

<comment><USER8> Are you suggesting that there's a kind of network equipment that you have to trespass to access for which people have a reasonable expectation that they are authorized to use it?

There's no giant sign hanging over the breaker box in my building's elevator room saying "AUTHORIZED USE ONLY", but I'm pretty sure I'd get in trouble if I went in there and started flipping switches.

<comment><USER18> If I'm a student, and there's a switch in the teachers' lounge that connects to the same network that I am allowed to access in the rest of the school, then I can illicitly enter the teachers' lounge _and_ plug in to that network with the same authorization that I would have outside.

Edit: doesn't have to be wifi.

<comment><USER8> I don't follow, but then, we're not talking about wifi networks.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER19> Remember the FREE KEVIN bumper stickers?

FREE AARON.

Just to elaborate... what it seems we have here is a brilliant engineer and idealist leftie that lost his grip on what is reasonable. He seriously fucked up, and then he seriously fucked up by getting caught. He isn't a hardened criminal, he wasn't stealing to make money and he can almost certainly be reformed with a light sentence, community service and probation.

Only the oppositional system doesn't see things that way.

<comment><USER20> I have way more sympathy for aaronsw than I had for Mitnick. Maybe this would change if I looked into Mitnick's case, but my prior is that this analogy would not help Aaron.

<comment><USER21> Same here. The difference is that the motivation that caused Aaron to break the law is a _genuine_ desire to improve the world -- and that's what advocates for Aaron in the arena of public opinion should focus on.

</comment></comment></comment><comment><USER22> It's embarrassing how much of this article is wrong.

</comment><comment><USER23> Anons Unite.</comment>