How To Make An Infinitely Scalable RDBMS - <AUTHOR> http://highscalability.com/blog/2013/11/25/how-to-make-an-infinitely-scalable-relational-database-manag.html <comment><USER2> There was very interesting presentation by one professor. I'm not sure about what university, but he seemed to know his work.

He talked about how databse world is about to change. ACID is really expensive in terms of resources, and so are the more difficult things about relational schema (foreign keys, checks, etc). And architecture of classic RDBMSes is pretty wasteful -- they use on-disk format but cache it in memory.

He talked about how there are basically three new paths for DBMSes to follow. 1) Some drop the restrictions to become faster. This is the NoSql stuff, because you don't really need ACID for writing to Facebook wall.

This is called NoSql database.

2) OLAP, in data warehousing, the usual way to do things is that you load ridiculous amount of data into database, and then run analytical queries, that tend to heavily use aggregation and sometimes use just few dimmensions, while the DWH data tend to be pretty wide.

For this, column store makes perfect sense. It is not very quick on writes, but it can do very fast aggregation and selection of just few columns.

This is called Column store.

3) In OLTP, you need throughtput, but the question is, how big are your data, and how fast do they grow? Because RAM tends to get bigger exponentially, while how many customers you have will probably grow linearly or maybe fuster, but not much. So your data could fit into memory, now, or in future.

This allows you to make very fast database. All you need to do is to switch the architecture to memory-based, store data in memory format in memory and on disk. You don't read the disk, you just use it to store the data on shutdown.

This is called Main memory database.

No, that was the presentation. It was awesome, and if someone can find it, please give us a link! My search-fu was not strong enouhg.

...

What interests me is that we have NoSql databases for some time already, and we have at least one huge (are very expensive) column store: Teradata. But this seems to be first actual Main memory database.

My dream would be to switch Postgres to main memory or column store mode, but I guess that's not happening very soon :)

<comment><USER3> Is this the talk that you are referring to? [http://slideshot.epfl.ch/play/suri_stonebraker](http://slideshot.epfl.ch/play/suri_stonebraker)

<comment><USER4> Note that Stonebraker makes some good points, but there are many ways to build scalability and Stonebraker is too fast to dismiss many.

In particular, his criticism of traditional databases seems based more on philosophy rather than evidence.

I'd advise reading both sides of the story:

[http://lemire.me/blog/archives/2009/09/16/relational- databas...](http://lemire.me/blog/archives/2009/09/16/relational-databases- are-they-obselete/)

[http://lemire.me/blog/archives/2009/07/03/column-stores- and-...](http://lemire.me/blog/archives/2009/07/03/column-stores-and-row- stores-should-you-care/)

[http://architects.dzone.com/articles/stonebraker-talk- trigge...](http://architects.dzone.com/articles/stonebraker-talk-triggers)

[http://gigaom.com/2011/07/11/amazons-werner-vogels-on-the- st...](http://gigaom.com/2011/07/11/amazons-werner-vogels-on-the- stonebrakernewsql-debate/)

[http://dom.as/2011/07/08/stonebraker- trapped/](http://dom.as/2011/07/08/stonebraker-trapped/)

The date on some of those posts in interesting. 2009 is quite a while ago now, and I'd suggest that columnar datastores haven't exactly taken over. Some implementations have made some progress (eg Cassandra), but OTOH many non- traditional datastores have added traditional-database like features (eg, Facebook's SQL front end on their NoSQL system), and traditional databases have added NoSQL features too.

<comment><USER5> Stonebraker is a very smart person but he's also not shy about promoting his own companies/research. You generally get a well-informed but very opinionated take on things from him.

VoltDB, for example, is good for certain complex workloads over large but not- too-large data sets. For a lot of situations it isn't really an alternative to memcache+MySQL or a NoSQL solution.

</comment></comment></comment></comment><comment><USER6> I'm a little skeptical:

\- a bunch of the novel components (the UPS aware persistence layer, for example) aren't actually built yet

\- they're pushing for people to build businesses on it already. I would characterize it as "bleeding-edge with bits of glass glued on", so this doesn't seem entirely honest.

\- there's mostly a lot of breathless talk about how great and fast and scalable it is, but no mention of CAP theorem. To boil down their feature set, it's an in-memory RDBMS using the Actor model.

<comment><USER7> Hi, Alan. Yes, many things are not built yet. Nowhere am I pushing anybody to build their business on it yet, but I am looking for hackers and early adopters/alpha testers. I've gone through pains on every doc that I've created that this is early, alpha, needs lots of work--including the 2nd paragraph of the linked-to article: "InfiniSQL is still in early stages of development--it already has many capabilities, but many more are necessary for it to be useful in a production environment."

Regarding CAP, I'm not addressing multi-site availability at this stage--I want to get single site fully operational, redundant, and so on.

And, yes, to boil it down, it's an in-memory RDBMS using the Actor model.

The most important feature is that it performs transactions involving records on multiple nodes better than anything. This is the workload that keystores functionally cannot do, and which other distributed RDBMS' suffer under. It's also open source, has over 100 pages of technical docs, and is functional enough for people to pound on with some workloads--but not something to put into production yet.

<comment><USER8> When I see a new distributed database, the first thing I want to know is "Is it AP or CP?"

If you haven't answered that question yourself, it's likely you have a partition-intolerant system, and that in real-life scenarios people will lose data.

<comment><USER7> CP in a single site. Between sites, I haven't considered it too much.

</comment></comment></comment></comment><comment><USER9> > UPS systems will stay active for a few minutes, based on their capacity, and > the manager process will gracefuly shut down each daemon and write data to > disk storage. This will ensure durability--even against power failure or > system crash--while still maintaining in memory performance.

How does a UPS ensure durability against system or program crashes, disk corruption in large clusters, and other failures that can affect a simple write()?

> The real killer for database performance is synchronous transaction log > writes. Even with the fastest underlying storage, this activity is the > limiting factor for database write performance. InfiniSQL avoids this > limiting factor while still retaining durability

How do you plan to implement this (since it appears it hasn't been implemented)? What is your fundamental insight about synchronous transaction logs that makes InifiSQL capable of being durable while (presumably) not having a synchronously written transaction log? If your answer is the UPS, please see my first question.

Edit: I don't see any mention of Paxos anywhere. Could you explain what you're using for consensus?

<comment><USER7> Hi, yid. UPS protects against multiple simultaneous system crashes. Single system crash gets failed over, no problem. If both UPS systems detect their upstream PDU's as being out, then the InfiniSQL management protocol will initiate graceful shutdown, including persisting to disk. For write() issues, at least intially, I think that stuff in commodity hardware (such as ECC memory) is sufficient protection in most cases. Attaching a high end storage array, or using ZFS, also protects against low level disk problems. I don't see those problems as needing to be solved for a 1.0 relase, but am very much open to contributions that address those issues any time you want!

The fundamental insight about not needing transaction logs is pretty simple actually: if the power is guaranteed to either stay on, or to allow the system to quiesce gracefully, then the cluster will not suddenly crash. That's the motivator for transaction logs--to make sure that the data will still be there if the system suddenly crashes. Get rid of the need for transaction logs, get rid of the transaction logs.

Regarding consensus, I expect that there will be a quorum protocol in use amongst an odd number greater than 2 of manager processes, each with redundant network and power. But the specific protocol I haven't ironed out. If there's something I can grab off the shelf then it may be preferable to implementing from scratch, but I haven't gotten there yet.

This stuff hasn't been implemented yet, but the core around which it can be implemented, has been.

Do I sense a volunteer? ;-)

<comment><USER9> I feel like you're banking a little too heavily on external measures for error protection/durability like UPSes and ECC memory, rather than embracing the inevitable fact that corruption and failures will occur. In short, I'd really need to see a white paper on why you're not reinventing the wheel before I'd use InifiSQL.

Kudos for engaging the community though; please do keep us posted as you progress.

<comment><USER7> Actually, there's precious little that an application can do if a memory chip fails, or if ECC gets too many corrupted bits. If it gets too many corrupted bits, the kernel will generally do something like halt the system. I am not familiar with any application which does a write-read-write (or similar) to memory, but would be curious to learn about them. I'm sure such an algorithm can also be used in InfiniSQL.

I am familiar with applications such as IBM WebSphere MQ which does a triple- write to disk for every transaction, to overcome corruption problems. But even IBM will recommend turning that parameter off for performance reasons if the storage layer performs that kind of verification, such as HDS or EMC arrays. So, if an InfiniSQL user wants that level of storage protection, they can buy it.

Regarding InfiniSQL's planned use of UPS systems, that's pretty much the identical design to how the above-mentioned storage arrays protect block storage from power loss. I'm just moving the protection up to the application level.

Thanks for the conversation and thoughtful comments. Please go to [http://www.infinisql.org/community/](http://www.infinisql.org/community/) and find the project on Twitter, sign up for newsletter, hit me on LinkedIn.

\----

Hi, yid. For some reason I can't reply to your last comment. I think I didn't explain things well enough--I do plan to implement synchronous replication to protect against single replica failure. I describe the plan somewhat in [http://www.infinisql.org/docs/overview/](http://www.infinisql.org/docs/overview/)

Much of the code for the replication is in place (and it actually worked a few thousand lines of code ago) but it's currently not functional.

I have a big backlog to work on, but n-way synchronous replication is definitely in there.

<comment><USER9> > Actually, there's precious little that an application can do if a memory > chip fails, or if ECC gets too many corrupted bits.

That's where replication and (distributed) consensus comes in, usually at the application level.

<comment><USER10> Or Resilient Distributed Datasets [https://www.usenix.org/system/files/conference/nsdi12/nsdi12...](https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf)

</comment></comment></comment></comment></comment></comment><comment><USER11> GNU AFPL?

Clause 13 is a real pain to deal with when exposing this over the network.

I guess the developer wants to sell a license (like the mysql java client GPL'ing).

Can't blame him, he needs to get paid.

<comment><USER7> Hi, gopalv. What samspenc said. My understanding of the AGPL is that only the modifications made to the source code itself of the covered project would need to be opened up (or have a commercial license). Meaning that merely using InfiniSQL won't require you to open source your app. MongoDB has the same license BTW, and lots of people use it without being forced to open their code. And, yes, I want to get paid somehow--but the AGPL won't stop anybody from using my work how they see fit. But if they modify it and distribute it, then they'll have to comply with the license (or contact me directly for an alternate arrangement).

<comment><USER11> I fully understand what this means and I hope you do get calls about alternate licensing, but remember that people like me do not make these decisions.

I thankfully don't have to - this means I don't need to talk to lawyers about this.

Because AGPL took away the most important bit of unassailable ground I had to argue with when it came to deploying GPL - "Using this code implies no criteria we have to comply to, only if we distribute it".

Clause 12 and 13 - basically took that away from me completely.

Look, I'm not going to tell you what license to use.

But leave me enough room to complain that I have had trouble convincing people that we can use AGPL code in a critical function without obtaining a previous commercial license by paying the developer.

<comment><USER7> Hi, gopalv. I'm glad to talk to you further privately if you wish. You can go to InfiniSQL's site to send me your email, connect on LinkedIn, or whatever: [http://www.infinisql.org/community/](http://www.infinisql.org/community/)

I'm not too religious about licensing--if I can get early adopters/contributors, and so on, I'm willing to consider changing the license terms.

I'm looking for open doors.

<comment><USER12> Personally, I think the choice of AGPL is good. It enables you to give to the community and get useful community involvement, while allowing commercial companies to have proprietary forks (for a cost) as well as commercial support.

All the best!

<comment><USER7> Thanks!

</comment></comment></comment></comment></comment></comment><comment><USER13> In-memory distributed database? VoltDB is already way past 500Ktx/sec on a 12-node cluster.

On their site though, it says no sharding and that it can do these 500Ktx/sec even when each transaction involves data on multiple nodes. Does this performance degrade directly in relation to the number of nodes a tx needs to touch?

A simple, straightforward, wire-level description of how things work when coordinating and performing transactions across would be very useful. There's a lot of excited talk about actors, but nothing that really examines why this is faster, or any sort of technical analysis.

<comment><USER7> Hi, Michael. Yes, VoltDB is very fast, but they self-admittedly _do not_ perform well if transactions contain records spanning across multiple nodes. That is the key feature difference between InfiniSQL and VoltDB (along, of course, that their project is functionally much further along).

If you want more details about how things work when performing transactions, I think that the overview I created would be a good starting point. It probably doesn't have everything you'd ask for, but I hope that it answers some of your questions: [http://www.infinisql.org/docs/overview/](http://www.infinisql.org/docs/overview/)

And to answer your question about performance degradation pertaining to number of nodes each transaction touches, I have not done comprehensive benchmarks of InfiniSQL measuring that particular item. However, I do believe that as multi- node communication increases, throughput will tend to decrease--I expect that the degradation would be graceful, but further testing is required. The benchmark I've performed and referenced has 3 updates and a select in each transaction, all very likely to be on 3 different nodes.

I'd like to invite you to benchmark InfiniSQL in your own environment. I've included the scripts in the source distribution, as well as a guide on how I benchmarked, as well as details on the specific benchmarking I've done so far. All at [http://www.infinisql.org](http://www.infinisql.org)

I'd be glad to assist in any way, give pointers, and so on, if there are tests that you'd like to do. I also plan to do further benchmarking over time, and I'll update the site's blog (and twitter, etc) as I do so.

Please communicate further with me if you're curious.

Thanks, Mark

<comment><USER14> So basically, VoltDB acknowledges that cross-partition transactions are Hard and has put a lot of effort into minimizing them. (This is basically the entire point of the original HStore paper.)

InfiniSQL says don't worry, we'll just use 2PC. But not just yet, we're still working on the lock manager.

I look forward to your exegesis of how you plan to overcome the well- documented scaling problems with 2PC. Preferably after you have working code. :)

<comment><USER7> I'm not doing 2PC. I'm curious what gave you that impression, and I'll clarify any documents that seem to give off that impression.

I'd like InfiniSQL to handle hard workloads as well as it can, to minimize the amount that developers need to redesign their applications.

Please go to [http://www.infinisql.org/community/](http://www.infinisql.org/community/) and follow the project on Twitter, and sign up for the newsletter, to keep apprised of progress.

Thanks for the input!

<comment><USER14> [http://www.infinisql.org/docs/overview/#idp37097184](http://www.infinisql.org/docs/overview/#idp37097184) gave me that impression.

<comment><USER7> Oh, I thought you said 2PC (two-phased commit).

Yes, InfiniSQL uses two phase locking. The Achilles' Heel is deadlock management. By necessity, the deadlock management will need to be single- threaded (at least as far as I can figure). No arguments there, so deadlock- prone usage patterns may definitely be problematic.

I don't think there's a perfect concurrency management protocol. MVCC is limited by transactionid generation. Predictive locking can't get rolled back once it starts, or limits practical throughput to single partitions.

2PL works. It's not ground-breaking (though incorporating it in the context of inter-thread messaging might be unique). And it will scale fine other than for a type of workload that tends to bring about a lot of deadlocks.

<comment><USER15> HAT ([http://www.bailis.org/papers/hat- vldb2014.pdf](http://www.bailis.org/papers/hat-vldb2014.pdf)) looks pretty promising in terms of multi-term transactions. It turns out that you can push the problem off to garbage collection in order to make transaction id generation easy, and garbage collection is easier to be sloppy and heuristic about. The only problem is it isn't clear yet that HATs are as rich as 2PL- based approaches, and that nobody's built an industrial strength implementation yet.

</comment></comment></comment></comment></comment></comment></comment><comment><USER16> Looking at the "About and Goals" section of their docs [http://www.infinisql.org/docs/overview/#idp37033600](http://www.infinisql.org/docs/overview/#idp37033600)

I can't seem to find the word "Reliable" or any variation thereof anywhere in there.

In fact, that word is no where to be found on the blog post or on the entire InfiniSQL page (not in the Overview, Guides, Reference or even FAQ). I find this quite remarkable since reliability is the true virtue of an RDBMS, not speed or even capacity. At least that's what PostgreSQL aims for and this being another RDBMS, and is also open source, I see it as InfiniSQL's only direct competitor.

It's nice that this is scalable, apparently, to ridiculous levels, but if I can't retrieve what I store in exactly the same shape as I stored it, then that's a bit of a buzz kill for me.

Can we have some assurance that this is the case?

There's a note on "Durability" and a shot at log file writing for transactions, and presumably InfiniSQL uses concurrency and replicas, to provide it. In the Data Storage section, it mentions that InfiniSQL is still an in-memory database for the most part [http://www.infinisql.org/docs/overview/#idp37053600](http://www.infinisql.org/docs/overview/#idp37053600)

What they're describing is a massively redundant, UPS backed, in-memory cache.

Am I wrong?

<comment><USER7> Hi, eksmith. I talk a bit about plans for durability in that overview document.

I promise that I have every intention of making InfiniSQL a platform that does not lose data. I have a long career working in environments that demand 100% data integrity. If I de-emphasized it, it was not intentional.

PostgreSQL doesn't scale for OLTP workloads past a single node. There are a handful of products similar to InfiniSQL (google for the term NewSQL for a survey of them).

And yes, a redundant UPS-backed in-memory cache. I have some ideas on how to do regular disk backing as well (which I'm sure you've read).

And if a more traditional log-based storage layer is added, InfiniSQL will still scale nearly linearly across nodes horizontally. Multi-node scale and in-memory are not dependent on one another. Though I believe that redundant UPS systems managed by a quorum of administrative agents, and provide durability just like writing to disk.

Are you familiar with high end storage arrays, such as from HDS or EMC? They write to redundant memory, battery backed and managed by logic in the arrays. I'm just moving that type of design to protect the database application itself, up from the block layer.

And some people trust their datacenter power--they use pure in-memory databases without UPS already, or they do things like asynchronously write transaction log, which also sacrifices durability. For those groups, InfiniSQL ought to be just fine, without UPS systems.

</comment></comment><comment><USER15> The write bottleneck for traditional databases has never been the write-ahead log, with group commit and a battery-backed RAID controller you'll have a hard time saturating the disk with log writes. The bottleneck has always been random I/O induced by in-place updating indexes based on B-trees. You don't need to be in-memory if you use better data structures. TokuDB and TokuMX are proof of that.

<comment><USER7> Hi, Leif. It's not hard to get to the throughput limits of a single log device, even on a fast array. I've done it on Sybase, WebSphere MQ, Oracle, MySQL, basically on enough platforms that I assume it to be the general case. The log writes don't saturate the array itself--but the log file has a limit to how many blocks can be appended--even on fast arrays. But imagine getting rid of the transaction log entirely--the entire code path. That will be faster even than a transaction log write to memory-backed filesystem.

But I agree that other write (and read) activity going on in the background and foreground, also limits performance--and in fact, I've seen the index write bottleneck that you describe in real life, more-so than simple transaction log writes. So, you're correct.

I've read about Toku, but I really doubt that it writes faster to disk than writing to memory. Are you really trying to say that?

I think it would be great for InfiniSQL to be adapted to disk-backed storage, in addition to memory. The horizontal scalability will also apply, making for a very large group of fast disk-backed nodes.

I think your input is good.

<comment><USER15> I'm not talking about the speed of the array, rather a battery backed controller. With one of those, as long as you're under the sequential write bandwidth of the underlying array, it pretty much doesn't matter how much you fsync the log, so that bottleneck (commit frequency) goes away.

If you're planning to write data faster than disk bandwidth, then you have no hope of being durable and we're talking about problems too different to be worth comparing, and in that case I retract my comment.

I don't understand what distinction you're trying to make between the "array itself" and the "log file has a limit to how many blocks can be appended". Can you clarify what limit you're talking about?

<comment><USER7> Well, array=battery backed controller with a bunch of disks hanging off of it. Actually, there is latency associated with every sync. What I've seen on HDS arrays with Linux boxes and 4GB fibre channell adapters is about 200us per 4KB block write. That is very very good for disk. It's also slower than memory access by many orders of magnitude. This was about 3 years ago. Things are bound to be faster by now, but still not as fast as RAM.

I don't think it's unreasonable to want to write faster than an I/O subsystem can handle. Maybe it's not for every workload, but that doesn't mean it's for no workload.

The distinction I wasn't being clear about was that the storage array (the thing with the RAID controller, cache and disks hanging off of it) is not being saturated if a single transaction log file is being continuously appended to. But that headroom in the array does not translate to more throughput for the transaction log. I don't know if it's an important distinction.

</comment></comment></comment></comment><comment><USER17> An in-memory RDBMS hardly seems to be "infinitely scalable". How would this work with DBs in the terabyte size or larger?

<comment><USER13> A terabyte of RAM is pretty cheap. Around $12K for the RAM. Last I quoted out a system for VoltDB, the total cost (complete servers with CPU, disk, RAM) came to ~$17/GB to $22/GB.

If you actually have transaction processing at this scale and need that performance, the RAM cost is not a major issue.

</comment></comment><comment><USER18> This is what Clustrix (YC company) claims to do.

<comment><USER7> Hi, amalag. Yes, Clustrix is very similar to InfiniSQL (not to mention having been around longer). I believe that InfiniSQL has _vastly_ higher performance at least for the type of workloads that InfiniSQl is currently capable of. InfiniSQL is also open source.

I hope there's room for competition in this space still.

<comment><USER19> What do you base your performance claim vs Clustrix on?

<comment><USER7> Here is some back of napkin analysis:

Starting with this benchmark report: [http://www.percona.com/files/white- papers/clustrix-tpcc-mysq...](http://www.percona.com/files/white- papers/clustrix-tpcc-mysql-benchmark.pdf)

Basically, InfiniSQL does not currently support complex indices, so it can't do a TPC-like transaction.

The maximum throughput on 9 nodes is 128,114 per node per second. I don't know if that's 4 or 8 core nodes. If roughly 10% of transactions are multi-node transactions, then 12,811/node/s for multi-node, and 115,303/node/s for single-node transactions.

I don't know if full redundancy for Clustrix was configured, or a hot spare, so I don't know how many actual usable nodes were configured, but likely fewer than 9. But I don't know the precise number.

Roughly 10% of those transactions are contain records on multiple nodes. Based on 9 nodes, that means about 12811/node/s for distributed transactions combined with 115303/node/s for single node transactions.

InfiniSQL maxsed at over 530,000 multi-node transactions on 12 x 4-core nodes. [http://www.infinisql.org/blog/2013/1112/benchmarking- infinis...](http://www.infinisql.org/blog/2013/1112/benchmarking-infinisql)

That's 44,167 per node.

\</comment></comment></comment></comment><comment>---

These were not apples-apples benchmarks, but Clustrix performed about 12,000 multi-node transactions per node per second, along with a whole bunch more single-node transactions.

I don't know how it would perform on the benchmark I used. And I intend to do a tpcc benchmark once InfiniSQL is capable of complex keys (among whatever else it it currently is missing).

<comment><USER19> Several problems here:

1\. Unlike your dataset, the tpcc dataset for the benchmark was not memory resident. Total dataset size was 786G. Just shrinking the dataset to fit in memory would substantially change the numbers.

2\. The tpcc workload is much more complex than your benchmark. It doesn't make sense to compare a tpcc transaction, of which there are multiple flavors, to your workload.

3\. All Clustrix write transactions are multi-node transactions. There's a 2x data redundancy in the test. We do not have a master-slave model for data distribution. Docs on our data distribution model: [http://docs.clustrix.com/display/CLXDOC/Data+Distribution](http://docs.clustrix.com/display/CLXDOC/Data+Distribution).

\-----

Now we've also done tests where the workload is a simple point updates and select on memory resident datasets. For those kinds of workloads, a 20 node cluster can do over 1M TPS.

</comment></comment><comment><USER20> What's up with the weird coding standards? Include files named infinisql_*.h and #line statements... strange.

<comment><USER7> Oh, the infinisql_*.h is because I deploy all header files as part of "make install", when what I really should do is boil it down to just the api header. The api is for stored procedure programming. Yes, I have it on backlog to fix. I give them all that name in case somebody installs to /usr/local (which you probably oughtn't) it's clear what application they all belong to. Yes, I could create a subdirectory, too. But the fix will be when I clean up api.cc to only have to pull in the one header instead of several of them.

#line statements because I get compiler messages from time to time putting things on the wrong line after having imported headers.

</comment></comment><comment><USER21> What is the difference to Teradata or Netezza except this is open source and lack the burden of universality yet?

<comment><USER7> Those are analytics databases, also known as data warehouses. Optimized for batch reporting. InfiniSQL is geared for operational/transactional (OLTP) kinds of workloads.</comment></comment>


Mac OSX Leopard: Worth The Wait - <AUTHOR> http://www.techcrunch.com/2007/10/26/mac-osx-leopard-worth-the-wait/

<comment><USER2> _"Don't Click The Time Machine Button_

 _"... if Time Machine hasn't made a backup yet to visit, clicking the Time Machine button completely borks OSX...least it did for me ... I'm sure now it's making backups that it will be a great feature, but just be warned: don't hit that button on your first day."_

That is the _first_ button I would have pressed!

<comment><USER3> "New OS X backup software ironically destroys all your data" would be an even more hilarious PR disaster for Apple than all those 'bricked' telephones.</comment></comment>


NIST announces approval of Secure Hash Standard - <AUTHOR> http://csrc.nist.gov/news_events/index.html

<comment><AUTHOR> Here's the text of the standard: [http://csrc.nist.gov/publications/fips/fips180-4/fips-180-4....](http://csrc.nist.gov/publications/fips/fips180-4/fips-180-4.pdf)</comment>


Ask HN: What would you do with free images of everything on Earth? - <AUTHOR>

We've just launched a web site that provides RDFa metdata and a simple JSON API for programatically looking up pictures of things in the web of linked data (for instance,  in Freebase or Wikipedia)
http://ookaboo.com/
Right now we're really looking for somebody who can use our photographs to build applications and we're willing to adapt our API and metadata to their needs.  We're also trying to build an integrated search and browse interface that will help people find pictures that they're looking for.  Everything is PD or creative commons,  so our content is reusable and remixable.
Any thoughts? <comment><USER2> The pictures may be used by the OpenStreetMap project because OSM is missing lots of details and with exact date when this photo was taken, it would be great to have these photos available.

Aerial images of Dortmund, Germany for example were published for OSM use and the map quality in Dortmund exploded, it was a huge benefit for OSM.

It would be great to see photos to be used with OpenStreetMap.

</comment><comment><AUTHOR> The link is <http://ookaboo.com/></comment>


Rezoom.SQL: Statically typed SQL for F# - <AUTHOR> https://github.com/rspeele/Rezoom.SQL <comment><USER2> This is really cool. Yet another example of how Type Providers are such a game-changer for people.

Relevant for folks:

FSharp.Data: [http://fsharp.github.io/FSharp.Data/](http://fsharp.github.io/FSharp.Data/)

SqlProvider: [http://fsprojects.github.io/SQLProvider/](http://fsprojects.github.io/SQLProvider/)

FSharp.Data.SqlClient: [http://fsprojects.github.io/FSharp.Data.SqlClient/](http://fsprojects.github.io/FSharp.Data.SqlClient/)

Azure Storage Type Provider: [https://github.com/fsprojects/AzureStorageTypeProvider](https://github.com/fsprojects/AzureStorageTypeProvider)

<comment><USER3> > Yet another example of how Type Providers are such a game-changer for > people.

I've seen this statement twice on HN over the past few days, so I'm getting curious about F#. I would like to understand, however, exactly how type providers change the game.

"An F# type provider is a component that provides types, properties, and methods for use in your program."[1]

Is there an essential difference between an F# type provider and a Haskell module (which also exports types, properties and methods)?

[1] [https://docs.microsoft.com/en- us/dotnet/fsharp/tutorials/typ...](https://docs.microsoft.com/en- us/dotnet/fsharp/tutorials/type-providers/)

<comment><USER4> I've never used F#, so I might get this completely wrong, but I think they are a practical form of dependently typed programming: a type provider builds new types that depend on values in your program. If you have some sort of schema for data, a type provider could parse it and build appropriate types to hold data that conforms to the schema. A type provider for a database, say, could take a string that specifies a database connection, connect to the database, and then query it to figure out what sorts of tables it holds and then construct F# types capable of holding a record from one of the tables.

<comment><USER5> SqlProgrammabilityProvider of [http://fsprojects.github.io/FSharp.Data.SqlClient/](http://fsprojects.github.io/FSharp.Data.SqlClient/) basically does just that. I usually prefer the SqlCommandProvider part of this project, as the actual queries are transparent in your F# code.

</comment></comment></comment></comment><comment><USER6> Thank you for this! I think SQL is a great language, and I've always wanted to write straight SQL with compiler guarantees, as projects like Esqueleto almost let me do.

For rapid prototyping, I think ORMs are still the best solution. However, their benefits start to diminish when scaling the team and product start becoming priorities. Working /with/ the database becomes a lot more fruitful than trying to wrangle it to your format. After all, web applications are just database skins.

<comment><USER7> > ...I've always wanted to write straight SQL with compiler guarantees...

SQL Server Data Tools does this exceedingly well. It's a Microsoft-built addon to Visual Studio.

Catching errors during compilation of SQL files is certainly very helpful! They also provide a very nice diff tool where you can diff between your project SQL files and a given server/database or between 2 different projects or 2 different databases... It also has a nice data-diff tool. (All for SQL Server only of course.)

This tool helped me very much to replace heavy ORMs (such as EntityFramework and so on) with just Dapper and a bunch of query procedures that are kept in the database itself (known as Stored Procedures in SQL Server). More detail on my setup in this comment - [https://news.ycombinator.com/item?id=14852425](https://news.ycombinator.com/item?id=14852425)

<comment><USER8> One of the things I was really excited about with the Drizzle MySQL fork a long time ago was how they were planning to make the server side SQL language supported through plugins. Being able to use the same language on the front- end and for stored procedures would be _really nice_. For me, that's Perl, but for a lot of people it's probably Python, Ruby, Java or some JVM, derivative, etc. To be sure, there are downsides (not the least of which is making it easier to load a lot of logic on the SQL server which may be much harder to scale if you aren't careful), but it would also make it much easier to do some complex SQL operations.

</comment></comment></comment><comment><AUTHOR> Some discussion on reddit: [https://www.reddit.com/r/programming/comments/6qizpb/statica...](https://www.reddit.com/r/programming/comments/6qizpb/statically_typed_sql_for_f/)

<comment><USER8> LOL at how a chunk of the comments ended up being the author or this and an author of Diesel for Rust discussing cool capabilities of Diesel.

Personally, I prefer my ORM/query builders to be more structures and adhere to the parent language more, with sane escape hatches for small bits of raw SQL as needed. This does look like it would be cool for a project that took the inline SQL or SQL phrasebook approach that wants to add a bit of structure, as it might be easier to put this in place over that?

<comment><USER9> You might prefer: [http://fsprojects.github.io/SQLProvider/](http://fsprojects.github.io/SQLProvider/)

</comment></comment></comment><comment> default-kramer This might be a dream come true for me. That Visual Studio integration is awesome - I did not know about F# Type Providers but I'm clearly overdue to learn about them.

</comment><comment><USER10> Hi, I am the author of this project.

The choice to typecheck a SQL dialect directly instead of using an embedded DSL like LINQ is uncommon enough that it gets the majority of the focus in my own documentation. However, it wasn't what motivated me to create the project.

I wanted to have an ORM that could take full advantage of my resumption monad library, Rezoom. This library is similar in concept to Facebook's Haxl or Dataloader, in that it can combine independent requests into batches and cache ones that have already been resolved.

The main difference (besides being F#) is that it is intended to be used for mutations, not just queries. This is a bit tricky because when you mutate the stuff in the database, that may invalidate the cached results of previous queries. The problem isn't too bad since, as with Haxl and Dataloader, the cache is per-request so it's very short lived, but it could still cause problems. Contrived example:

      // domain layer line item logic, includes business rules
      // returns the updated invoice
      let addLineItem invoiceId lineItem =
        plan {
          // query for the invoice
          let! invoice = DB.getInvoiceById invoiceId
          if invoice.IsSent then
            failwith "Can't edit this invoice because it has been sent"
          // run INSERT/UPDATE statement(s)
          do! DB.addInvoiceLineItem invoiceId lineItem
          // re-query for the invoice to get its current state
          return! DB.getInvoiceById invoiceId
        }

`DB.addInvoiceLineItem` needs to invalidate the cached result of `DB.getInvoiceById`. Otherwise this code won't return the updated invoice, but the invoice before updating. Again, contrived example, but when the "reload the invoice" line is in a whole other function somewhere else it could be a real problem.

The way Rezoom solves this is to have "errands" (its word for a batchable data dependency) specify 2 128-bit masks. One represents its dependencies, the other represents its invalidations. When you run a command with a non-zero invalidation mask, all the cached results whose dependency bits overlap with the invalidation bits get evicted.

The caches are local to a "category" which is pretty much just a tag for which library created the errand, so that the errands wrapping, say, a web API don't stomp on the caches for the errands talking to a SQL database (actually the connection string is involved too so you can deal with multiple databases from one app). For the errands generated by Rezoom.SQL, each bit in the mask represents one table, so running an UPDATE/INSERT/DELETE invalidates the cache for all commands that read from the affected tables. If you have more than 128 tables in your database the IDs wrap around so you'll get some false invalidations possibly hurting performance, but never a false cached result hurting correctness.

This all could make it very easy to write a GraphQL server backed by Rezoom + Rezoom.SQL.

<comment><USER11> This is super awesome. Unfortunately in my company I haven't converted the mindset fully to F#, so I'm wondering, is there a good way to use this in C# somehow?

<comment><USER10> You can create an F# library which references the Rezoom.SQL.Provider NuGet package, defines your migrations and all your queries (`type MyQueryExample = SQL<"select whatever">`), and not really write any F# code. Because this project is a generative type provider, it makes real .NET types you can then reference from C# or VB.NET or whatever.

If your goal is just to run queries, either synchronously or as `Task<T>`s, that'll do just fine. If you want the batching/caching stuff I described above, you'll want to build `Plan`s, which currently requires an F# computation expression.

However, in C# 7 I think it's possible to write plans with `async/await` syntax. I just need to write the AsyncMethodBuilder to support it. I've done it for F# asyncs here: [https://github.com/rspeele/FSharpAsyncMethodBuilder](https://github.com/rspeele/FSharpAsyncMethodBuilder)

This would let you get the full batching/caching/transactional benefits of Rezoom even when writing a project that's 90% C#.

</comment></comment></comment><comment><USER12> That's really impressive. Seriously.

I think type providers are a really fantastic feature. I just wish that I had more cause to write f#.

</comment><comment><USER13> How production ready this is and how could you compare this to SqlClient (addition to wider DB support)? Is there any possibilities to compose queries?

I am currently with SqlProvider but raw SQL would be interesting option =)</comment>


Free Apache Kafka as a Service with Confluent Cloud - <AUTHOR> https://www.confluent.io/blog/kafka-made-serverless-with-confluent-cloud <comment><USER2> From the post:

"*With try free promotion for Confluent Cloud, receive up to $50 USD off your bill each calendar month for the first three months. New signups only. Offer ends December 31, 2019."

Free trial for a managed Kafka service in the cloud has been available from various other vendors including us ([https://aiven.io](https://aiven.io)) for a while.</comment>


Reminder: "Broken gets fixed, but shitty lasts forever" - <AUTHOR> http://jamesgolick.com/2011/2/15/verify-none..html <comment><USER2> <http://twitter.com/#!/tenderlove/status/37665907648110592>

I think the responses here, and especially this blog post, are completely out of line. Yes, he's a community leader, but he still has the right to write lazy code to solve problems quickly. It's not like he published it as a gem, blogged about it, and tried to convince a bunch of people to use it. It was just a tweet about a fun hack.

<comment><USER3> I agree. Being "a leader" doesn't mean you aren't allowed to have fun anymore.

The blog post reads as, "I have some pet issue that I am not important enough to solve; so someone more important than me should solve my pet issue for me". Well, sorry. The world doesn't work like that.

(Maybe you would become a leader if you didn't spend hours of your day thinking about how one-off scripts could be hacked by a malicious server?)

</comment></comment><comment><USER4> I think it's a perfectly valid criticism.

One of Rails' biggest influences on the Ruby community (aside from making you feel like a jerk if you don't write tests) is that there is one "best" way to do common things and you should only deviate from the convention when necessary.

This is great most of the time, because it allows people to jump right into code they've never seen before (or haven't looked at in a while) and know exactly what's going on without having to follow a bunch of code paths and random classes. Having trouble with a poorly documented gem? Just jump into the code because all gems are laid out the exact same way.

However, often when you know something is pretty standard functionality (like doing an SSL request with net/http), you know it's been solved many times before and just find the first article on Google about it and do it that way. And you're about 100 times more likely to just blindly copy and paste the code when it comes from the guy who wrote the most popular Ruby HTML parser.

I see the same thing all the time in iOS development. Everyone just uses the same patterns that Apple puts in their documentation, which is why Joe Hewitt's Three20 framework was such a big deal.

<comment><USER5> _..which is why Joe Hewitt's Three20 framework was such a big deal._

Three20 wasn't a big deal because it was _different_ from Apple. The noise behind Three20 is that it was (and still is, to a large degree) _a cluttered mess of a framework_ that made it near impossible to use one part without importing the whole thing. I believe that Joe Hewitt has basically said as much on Twitter as well.

If you were to instead point out Apple's template code as a good example of code that is widely-used and not very good, you would have a much better analogy..

<comment><USER4> Oh yeah I agree that Three20 is a cluttered, monolithic mess. What I should have said was that Three20 was the first major attempt (that I know of) to break away from the strict nested-controller way of making data-driven apps by implementing a pretty nice URL-based application model.

Matt Gallagher has also done a good job of deconstructing the Apple Way and try new approaches: [http://cocoawithlove.com/2009/03/recreating- uitableviewcontr...](http://cocoawithlove.com/2009/03/recreating- uitableviewcontroller-to.html)

<comment><USER5> Does recreating UITableViewController to be slightly more flexible really constitute a new approach?

<comment><USER6> It's not slightly more flexible, it's hugely more flexible. As someone who has apps that consist of a complicated web of tableviews, Matt Gallagher's work is amazing and helped me immensely.

Although, I think this link is better: [http://cocoawithlove.com/2010/12/uitableview-construction- dr...](http://cocoawithlove.com/2010/12/uitableview-construction-drawing- and.html)

<comment><USER5> If you remove "to be slightly more flexible" from my previous post, you'll find the real point I was trying to make.

</comment></comment></comment></comment></comment></comment><comment><USER7> I find it repulsive that poor diligence on the part of a programmer can be blamed on some 'community leader' setting a bad example. As a programmer, all code that goes into _your_ project is _your_ responsibility. When things go well you're the only one who gets paid, therefore when they go wrong you're the only one who should get the blame.

See also: [http://enfranchisedmind.com/blog/posts/fyi-my-open-source- us...](http://enfranchisedmind.com/blog/posts/fyi-my-open-source-users/)

<comment><USER8> Your approach is unpractical and I doubt you follow it yourself. Almost nobody that uses opensource libraries reads all of the code. As with journalism, science and many other things in life, you have to trust upon others to deliver work that conforms to certain standards. Even though every newspaper article and scientific paper that you accept is 'your responsibility', you would be pretty miffed if an article in the free local paper was completely false.

If you give away toilets for free, it's your responsibility to make sure they aren't lined with sodium and will blow up after the first flush. If you give away a toilet for free, you are an asshole if you know it has a huge hidden hole and you don't warn takers about it. If you write a piece of software that claims to be doing something, but it actually contains malware, then you are violating the law (as much as when you give me a nice statue that contains a spy camera).

It has never been possible or allowed to give just anything you want away for free, without you being liable in any way. That hasn't changed with software and no license in the world can exempt you from certain basic responsibilities concerning your 'gift'.

<comment><USER9> Are you speaking morally? From my understanding the warranty disclaimer in many licenses (GPL, BSD, MIT) does exempt you from responsibility...

<comment><USER8> Claiming you are exempt from any responsibility does not actually legally exempt you from any responsibility. You can't legally bring a spy camera into my home by presenting me with a free gift that contains a hidden camera. You can't legally blow up my home, even if I accept a free device from you and press the buttons you told me to press, unless you were very clear about what the device would do (and no, claiming in the license that it is not meant for anything does not nullify the website and the examples of what the code is obviously supposed to do). Going from the realm of the obvious to the realm of the somewhat less obvious: if you give me a free bicycle, knowing it's fron fork is starting to give way, you are responsible for my skull fracture if you don't tell me about it. The practical problem, usually insurmountable, is proving you knew about it. More broadly: the consequences of known defects that obviously have disastrous consequences are the responsibility of the provider. This also holds for software and doesn't depend on the license.

Now my point is not that this is the case here or that such cases are common or even important. The point is that the blanket statement of my parent is plainly false.

Of course, IANAL, but you don't need to be a lawyer to understand jurisprudence. There are plenty of known cases where people intentionally harmed others and tried to plead guilty based on not actually performing the action that lead to the damage. It doesn't fly.

</comment></comment></comment></comment><comment><USER10> Here's a reference/cheat sheet to using SSL with verification with Net::HTTP:

[https://github.com/augustl/net-http-cheat- sheet/blob/master/...](https://github.com/augustl/net-http-cheat- sheet/blob/master/ssl_and_https.rb)

It shows different ways of providing Net::HTTP with verification certificates.

<comment><USER11> _Thank you._ This is the best example I've seen of how to make Net::HTTP do the _right_ thing with SSL certificates.

The reason the VERIFY_NONE fix got so popular is that, by default, if you turn on SSL support in Net::HTTP in an otherwise working codebase, your calls fail with certificate errors. The docs ([http://www.ruby- doc.org/stdlib/libdoc/net/http/rdoc/index.ht...](http://www.ruby- doc.org/stdlib/libdoc/net/http/rdoc/index.html)) have absolutely no helpful information except for mentioning the default of VERIFY_PEER. If you're noodling around on your own you'll discover VERIFY_NONE works well before you discover how certificate stores work.

</comment></comment><comment><USER12> This isn't the first time I've heard this gem (also repeated as broken gets fixed, shoddy lives forever). Anyone have the original attribution for it?

</comment><comment><USER13> While true, it has to be balanced with the fact that you are tasked with creating a product to earn (or save, etc) the company money. Perfection can never be reached, and you have to say 'Good Enough!' at some point and just ship.

And really, if it's crappy code and lasts forever... Isn't that good enough? If you never have to touch it again, then it was as perfect as it needed to be. If you DO have to touch it, it was broken and will get fixed.

I'm not advocating writing crappy code. I'm just saying that you have to draw a line somewhere. I've met amazing coders that will fiddle with the code forever and never finish because they just keep making it better, more readable, prettier, more extensible, or some other thing forever.

</comment><comment><USER14> The point in general is valid, but picking someone out and showing them up in public was unnecessary.

</comment><comment><USER15> Using sensible wrappers around Net::HTTP would resolve the issues with the unreasonable defaults.

Wrest (<http://github.com/kaiwren/wrest>) would be a good-fit here: it defaults to VERIFY_PEER for SSL connections, has client-side caching support, comes with a fluent API, redirect support, callbacks, patron (curl) back-end etc. etc.

The project is being actively maintained, and we're working to add async support using EM for now.

</comment><comment><USER16> The very best criticism of code is written in code.

</comment><comment><USER17> Example code should be exemplary.

That's harder, of course when the library has unreasonable defaults.

</comment><comment><USER18> DO we have a term for this in the small?

Those small utilities you wrote a decade ago in perl when your company was tiny, over a couple of days that end up a staple of daily use in a company 10 years later and any attempt to re-factor or replace them 10 years way is met with extreme resistance? What do we call that - Crappy-enough software? (I say this as the author)

(Actually, going back and looking at said code, it's not badly written at all, it's entirely readable, a few perl scripts, no spaghetti, and it's clear how they interoperate just by reading them. So it's not crappy or badly written - it's just overly simplistic.)

</comment><comment><USER19> reminds me of something gabe newell (co-founder, managing director of valve software) once said: "a game won't be delayed forever, but a game can suck forever."

<comment><USER20> The fact that you can't delay "forever" is a cheap semantic trick. You can delay for so long that you run out of money / investor patience.

There's a vast graveyard of game companies which can attest to it.

<comment><USER21> I suppose one has to excuse Gabe Newell for not understanding this, because his company has been unsuccessfully trying to go out of business in exactly this fashion for most of its existence. Their failure to go bankrupt during any of their long delays is as notable and impressive as any of the games they've actually shipped.

<comment><USER20> Valve is an extraordinary company. That quote does not apply to non- extraordinary companies and hence is idiotic to repeat as a general rule.

It is as meaningless as if you quoted Feynman saying "think about a problem long enough and you'll solve it".

The implicit assumption in either case is one of an underlying quality process and _that is one helluva an assumption to make_.

</comment></comment></comment></comment><comment><USER22> Frankly it's instructive to have a high profile snafu.

And how can you get upset with someone with that twitter pic?

</comment><comment><USER23> If James Golick considers this to be important enough to criticize other people and even write a blog post about it, why does he write a fix?

<comment><USER24> He did: [https://github.com/jamesgolick/always_verify_ssl_certificate...](https://github.com/jamesgolick/always_verify_ssl_certificates)

<comment><USER23> Cool!</comment></comment></comment>


Revelator raises $2.5M to build a business platform for digital music - <AUTHOR> https://techcrunch.com/2016/08/29/revelator-series-a/ <comment><USER2> More extruded blockchain product for sucker VCs. There's four or five of these at the moment, all using the same buzzwords. I wrote some up here: [http://rocknerd.co.uk/2016/09/01/fools-gold-rush- blockchain-...](http://rocknerd.co.uk/2016/09/01/fools-gold-rush-blockchain- initiatives-for-everybody-especially-the-artists-our-eternal-and-only- concern/)</comment>


The amazing page 46 of the iOS 7 terms and conditions - <AUTHOR> http://pikoa.com/pik.php?id=64

<comment><USER2> it's a joke:

[http://grahamcluley.com/2013/09/page-46-apples-new-ios- agree...](http://grahamcluley.com/2013/09/page-46-apples-new-ios-agreement- funny-fake-makes-serious-point/)</comment>


Ask HN: Who are your (programming) heros/heroines? - <AUTHOR> I am enamored with john carmack and cliff click. I watch all of videos and read all of their blogs. Who do you idolize as leaders in programming? <comment><USER2> I don't really like idolizing developers that much, especially calling them heros - because it puts a big mental divide between you and them, making you think you could never reach their level.

I do however never miss a post from Tom Dale and Yehuda Katz (when they are not ranting about politics), and Patrick McKenzie (when he's not ranting about cryptocurrencies).

</comment><comment><USER3> Maybe not hero, but I thoroughly enjoy Andy Wingo's writings and talks. Every time I forget that programming is fun, I go to his blog.

He really likes what he does and is a very inspiring individual, and does a lot to bridge the gap between academia and us regular folks.

</comment><comment><USER4> Zed Shaw, the programming motherfucker.

[http://programming-motherfucker.com/](http://programming-motherfucker.com/)

</comment><comment><USER5> D.J. Bernstein

<comment><USER6> Seconded: [https://cr.yp.to/djb.html](https://cr.yp.to/djb.html)

</comment></comment><comment><USER7> @DHH</comment>


Xervmon – Integrated Hybrid Cloud Management - <AUTHOR> http://gigaom.com/2014/02/17/xervmon-serves-up-multi-cloud-monitoring-and-management-suite/

<comment><USER2> This is a great coverage.</comment>


Former NSA Director Says Terrorists Love Using Gmail - <AUTHOR> http://www.washingtonpost.com/blogs/the-switch/wp/2013/09/15/former-nsa-and-cia-director-says-terrorists-love-using-gmail/

<comment><AUTHOR> So evidently capturing the 99.99999999% of non terrorist gmails is ok because of this?</comment>


Autonomous Zone forms around emptied East Precinct in Seattle - <AUTHOR> https://www.capitolhillseattle.com/2020/06/welcome-to-free-capitol-hill-capitol-hill-autonomous-zone-forms-around-emptied-east-precinct/ <comment><USER2> I wonder what the Ferrari dealership down the street is doing..</comment>


6 Principles to get people to do what you want  Notes on Influence by R Cialdini - <AUTHOR> https://medium.com/startup42-stories/a-short-summary-of-influence-the-psychology-of-persuasion-c3dab4009d10#.i5q7t18nu <comment><AUTHOR> Wouldn't you like to be able to get people to do what you want from them? For instance make them buy or use this awesome tool you built but that they don't realise it's awesome until they actually use it?

Or wouldn't you like to know how to optimise a message so that people prefer the higher pricing rather than the lower one?

Well, every time these kind of issues come up during group sessions lead by the very good Jean-Xtophe Ordonneau where entrepreneurs share their successes and problems, he points out to the book "Influence — The psychology of persuasion" by Robert Cialdini.

So this time I decided to dig into the topic and, at least, watch one of his videos. I have to say that was SUPER insightful, so I wrote down ready to use notes. These are the 6 principles that lead relationships and how to leverage them.</comment>


Process Partitioning - <AUTHOR> https://gist.github.com/2374465 <comment><USER2> I particularly like the reliance on crc to help in situations where a numeric identifier might not be possible / desired. It's organic.

This gave me enough of an 'aha!' to move forward with an experiment I was conducting.

Thanks!</comment>


Intel HTML5 Development Environment - <AUTHOR> http://html5dev-software.intel.com/

<comment><USER2> Sad to see Intel failing big time with their HTML approach. I have used XDK long before this acquisition and it is a terrible piece of software. Spend 30 mins with it and this fact will be obvious.

The IDE creates nothing of value. The simulator just puts fancy device pics on the screen and doesn't emulate anything truly.

The debugger opens up dev tools and inspects the _whole_ IDE and not just the app you made.

The build service has no API and I have no idea why I would use this over build.phonegap.com which actually has a proper API.

Add to this, the UI is filled with bloopers - black icons which are impossible to decipher and gray/disabled icons that glow up on hover. There seems to some popup or the other that shows up with warning even with their own samples.

I will stop since it's obvious that this is all just a marketing ploy. appMobi was great at this - getting their stuff published in all sites (they were at their heart a marketing company). And appMobi seems to have succeeded in finding suckers at Intel.</comment>


OpenBSD vs. FreeBSD - <AUTHOR> Nobody wins, when the family feuds. ~ Sean Carter. OpenBSD has all the right ideas and a proven track record. The same argument can be made for FreeBSD.
And I get it. Differences are what make us unique and who doesn't like having options? However a small demographic of users only care about the best possible outcome. I never understood "fanboydom" and while I use FreeBSD I see it as only a tool. In that regard like most engineers I only care about  refining my tool set.
I'm curious about other peoples thoughts on this? Is anybody else sick of playing wack-a-mole when it comes to features... for fucks sake I just want all the dope tech in one place. lets just put all this shit in one OS with the option to choose specific features. Lets keep it free of GNU-ware and the shitty GPL... <comment><USER2> Ultimately, even if you ignore FreeBSDs and OpenBSDs history of cooperation (or the lack thereof, either of which I won't get into), the projects have different values that don't necessarily mix - so I'm not sure it's possible. It's okay to have different values, we don't all need to think a certain way.

Personally, I'd also begin worrying about mono-cultures and a lack of competition, if they were to be forced together. In the market-place of ideas, it's good to have multiple approaches to a single problem, because it lets you shop around, evaluate, and pick whatever solution fits you and your requirements best.

</comment><comment><USER3> What is this in relation to?</comment>


How do I find the longest palindrome in a string? - <AUTHOR> http://codegolf.stackexchange.com/questions/16327/how-do-i-find-the-longest-palindrome-in-a-string

<comment><USER2> For anyone interested in how to actually do this efficiently, without building a suffix tree, here's a good explanation: [http://www.akalin.cx/longest- palindrome-linear-time](http://www.akalin.cx/longest-palindrome-linear-time)

<comment><USER3> Cool, but unfortunately also a great example of when excessive commenting actually makes the code less digestible.

<comment><USER4> In addition, the use of cryptic variable names like s, e, d or j with big block comments describing what each one stands for is particularly irritating.

<comment><USER5> Thanks for the feedback! In my defense, I can only say that I was a less- experienced coder when I wrote that. :)

The comments were intended to extend the preceding prose explanation rather than being comments per se. The actual code is not much longer than the naive version, though, so if I get around to writing an updated version, I'd probably find some other method of conveying that info.

Some other things I'd do differently if I were to update it today:

\- I'd write it in Javascript and provide a little interactive applet to play with it. (Similar to my posts on primality testing: [http://www.akalin.cx/intro-primality-testing](http://www.akalin.cx/intro- primality-testing) and [http://www.akalin.cx/primality-testing-polynomial- time-part-...](http://www.akalin.cx/primality-testing-polynomial-time-part-2) .)

\- I'd upload the code to GitHub.

\- I'd add unit tests.

Cheers!

<comment><USER4> No offense was meant, no defense is necessary. Putting your code out there for HN to critique is indeed an admirable action. Your humility is even more admirable.

Your links on primality testing are particularly relevant to me as I just finished that section in SICP - thanks for the links!

</comment></comment></comment></comment></comment><comment><USER6> "generate a set of every single possible Palindrome as large as the string you're testing"

This kind of memoization could well be the most efficient solution in the real world, albeit not one that will pass your CS 101 exam.

More generally #codetrolling is like putting the creative hat on and will probably help people to come up with out-of-the-box tricks that could in some cases be quite useful.

</comment><comment><USER7> Wasnt this the old Cue/Greplin hiring challenge from years back?

</comment><comment><USER8> Finally, a job for LISP.

</comment><comment><USER9> suffix tree. same as longest subsequence problem. create suffix tree, reverse string and find longest subsequence.

</comment><comment><USER10> Just search for "notlob".</comment>


Supermassive black holes may be lurking everywhere in the universe - <AUTHOR> http://phys.org/news/2016-04-supermassive-black-holes-lurking-universe.html <comment><USER2> Here's a classic video that helps grasp (if that's even possible) the size of a supermassive black hole [https://www.youtube.com/watch?v=QgNDao7m41M](https://www.youtube.com/watch?v=QgNDao7m41M)

<comment><USER3> Thanks, it was short, but informative!

If I may, I would like to point to another kind of supermassive black hole: [https://www.youtube.com/watch?v=Xsp3_a- PMTw](https://www.youtube.com/watch?v=Xsp3_a-PMTw) :)

</comment></comment><comment><USER4> Could someone please explain to me in plain English, how is it that the black holes "emit radiation at many frequencies" and, at the same time, "the light is unable to leave their pull"??

<comment><USER5> There are several effects.

Accelerating black holes, or actually any accelerated matter, emits gravitational waves. This are disturbances of space-time itself and unlike electromagnetic waves mediated by photons moving through space-time. At least in a classical picture of gravity, not sure how that would work out within quantum gravity when gravitational waves are mediated by gravitons. But I guess you can just think of the gravitational waves being emitted at the event horizon.

Then there is electromagnetic radiation emitted from matter falling towards the black hole but still outside of the event horizon. The gravitational forces of the black hole heat up this matter to the point that it emits electromagnetic radiation in a wide spectrum.

Finally there is Hawking radiation which is a suspected electromagnetic radiation due to quantum effects near the event horizon. The associated black- body temperature is tiny, two billionth of a Kelvin for a 30 solar mass black hole and decreasing with increasing mass. The temperature of the cosmic microwave background with 2.7 Kelvin is in comparison gigantic and therefore black holes absorb much more cosmic microwave background radiation than they emit Hawking radiation. The universe has to become a lot cooler before Hawking radiation can start to evaporate black holes.

<comment><USER6> Gravitational waves (GWs) propagate a _change_ in the gravitational field. Consequently, GWs are only shed by accelerating matter where the change in motion is not rotationally or spherically symmetric.

Spherically symmetric gravitational collapses (of dust or gas, for example) are certainly systems of accelerated matter, but do not shed gravitational waves. Rotating spherical bodies also do not shed GWs. To first order, this means the rotation of stars and planets do not shed GWs, although a planet revolving around the star in an essentially Kepler orbit will shed GWs, as will a pair of stars in close orbit around each other.

GWs solve the apparent removal of momentum-energy from a system where there is no removal of mass-energy by non-gravitational radiation. For most systems the energy loss is negligible. That's why the first GW detection involved a late stage inspiral of two compact massive objects with no way to ditch their angular momentum electromagnetically or via neutrino (or other dark matter) emission.

GWs themselves have spin 2 symmetry; in theories with gravitons, those are spin-2 gauge bosons, typically massless (because they are long range), and you get large numbers of them forming the classical GWs, just as you will find large numbers of photons forming a bright flicker of light, although you can detect tiny numbers of photons because electromagnetism is relatively strong -- tiny numbers of gravitons are not going to be detectable by humanity any time soon.

It's not curvature (and certainly not gravitational forces) heating up accretion disks. The matter in the disks move on intersecting geodesics and occasionally collide. Such collisions tend to ditch the particles' momentum into emitted photons.

Hawking radiation is an effect of accelerated observers seeing more (and more energetic) particles than non-accelerated ones, and gravitational collapses producing an acceleration between past and future observers. Where a past observer sees a tiny number of low-frequency particles in the "vacuum" outside the collapsing object, a future observer (after the horizon has formed) will see a larger number of higher frequency ones. The acceleration is curvature dependent, and curvature just outside the horizon depends on black hole mass -- more massive black holes have less curvature just outside the horizon (which is further away from the gravitational singularity or whatever is at the centre of a black hole). These particles near but outside the horizon can escape, which takes mass-energy away from the dynamical spacetime just outside the horizon. That in turn causes the horizon to retract and that exposes stronger curvature. That in turn produces a stronger acceleration for further future observers, and the process continues -- that's black hole evaporation.

Hawking radiation is always emitted, for SMBHs that only grow by incidental interception of CMB radiation, and even for stellar mass black holes. Infalling matter simply replaces (and then some) the outgoing "fossil" infrared radiation from the space just outside the black hole as the horizon was forming.

</comment></comment></comment><comment><USER7> Related reading, URL says it all [https://aeon.co/essays/is-the-black-hole-at- our-galaxy-s-cen...](https://aeon.co/essays/is-the-black-hole-at-our-galaxy-s- centre-a-quantum-computer)

Imagine a civilization building black hole computers in their solar system. That's pretty badass :)

<comment><USER8> Off-topic query: The above linked aeon.co article has these opening lines:

 _After you die, your body's atoms will disperse and find new venues, making their way into oceans, trees and other bodies. But according to the laws of quantum mechanics, all of the information about your body's build and function will prevail. The relations between the atoms, the uncountable particulars that made you you, will remain forever preserved, albeit in unrecognisably scrambled form – lost in practice, but immortal in principle._

Can anyone point me to the article or wiki of this theory/phenomenon? I would like to read up more about it.

<comment><USER9> If you think of the world as being deterministic then the way your atoms will react with other atoms (their place in time and space) will be a direct consequence of where they were before, which again is a consequence of their previous state and we can track this all the way back to your current life.

<comment><USER10> That assumes that the universe is both deterministic _and_ reversible. SHA2 is deterministic, but good luck reversing it. Essentially, to work backwards, time, when going forwards from state A would need to move to some unique state B.

Consider two hills with a valley between them. In one universe, I roll a ball down one hill. In the other, I roll it down the opposite hill. In both, the ball ends up in the valley, and the information about which hill the ball rolled down is lost.

You could perhaps argue that other effects (me disturbing the air on the hill I walked up) might still imply which hill the ball rolled down, but the question really is if such a scenario could exist, probably in a more simple form.

<comment><USER11> Right, and as far as we know, quantum mechanics evolves the wave function in a time-reversible fashion.

</comment></comment></comment></comment></comment><comment><USER2> A couple of great YouTube channel for anyone interested in this stuff. The videos range from surface explanations to actual math equations.

[https://www.youtube.com/pbsspacetime](https://www.youtube.com/pbsspacetime) and [https://www.youtube.com/user/sixtysymbols](https://www.youtube.com/user/sixtysymbols)

</comment><comment><USER12> Would our brains even be able to detect the moments the earth entered one?

We'd probably get a little warning from amateur astronomers as they freaked out months before it happens.

<comment><USER13> What if we entered one and came out on the other side without even noticing something happened?

<comment><USER14> there is no other side. its not a physical hole. its a gravity hole.

Think of a moon that drags you onto it. You can't ever have enough power to reverse moving towards it. Light isn't quick enough to leave the pull. thus it is dark.

<comment><USER15> >"there is no other side. its not a physical hole."

Isn't a black hole more like an edge of the universe? My understanding is that you end up with a Zeno's paradox situation where no matter how close you get you can never actually reach it.

<comment><USER16> Not quite -- in your reference frame, you cross the blackhole's event horizon and go inside, but an observer (who is outside the event horizon) will only see you reach the event horizon and slowly "fade away."

<comment><USER17> Which is why black holes makes no sense.

Nothing can ever fall into them, so they can't ever form, or grow. (Obviously speaking from the POV of Earth.)

Since they can't form, and can't grow, how can you see a "supermassive one"?

<comment><USER18> Black hole has zero gravity at it center. ;-) Moreover, gravity can stop light and other electromagnetic forces, but cannot stop gravity, so time is not stopped.

<comment><USER18> WTF? I have a follower-downvoter? Every body has zero gravity forces at their center of mass. We also can clearly see that mass continue to generate gravity forces even after fall to black hole, so time is not stopped, at least for gravity.

PS.

Every black hole has second event horizon in the center of black hole, and I am very curios is it filled with mass or not. :-/</comment></comment></comment></comment></comment></comment></comment></comment>


Google, Facebook quietly move toward automatic blocking of "extremist" videos - <AUTHOR> https://www.yahoo.com/news/exclusive-google-facebook-quietly-move-toward-automatic-blocking-002529206--finance.html <comment><USER2> One man's "extremist" video is another man's political views.

Where does it stop? Does a video supporting Trump become an extremist video if enough people complain about it? What about a video that shows you how to take apart a pistol for cleaning?

<comment><USER3> Agreed, there is a fine line and it should not be decided by companies.

<comment><USER4> If it's that company's site then I don't see who else has any business getting involved. Do you want someone telling you what (legal) content you can or can't show? Why?

<comment><USER2> Youtube, Facebook, Twitter all loudly trumpeted their attempts to assist "free speech", especially during the Arab Spring.

And they were only doing that when what they were doing was going along with the cultural democratic zeitgeist at the time.

If they were more open about "Hey, we're providing you a free service, we can shut you down if you don't conform to public opinion" that's fine also. But you can't have it both ways.

Either you support the free expression of unpopular opinions, or you reserve the right to censor anything at all you find objectionable. You can't say "I support expressing opinions, as long as I approve of them, and I support freedom!"

<comment><USER5> Of course you can, and don't be ridiculous.

You can absolutely run a company that allows users to communicate in interesting ways, and say you support freedom but also refuse to show content you don't want to.

Beheadings. Child Pornography. Instructions for making chemical weapons. Snuff films. Animal abuse. Recruiting extremists to kill innocent people from your country.

I'm really tired of idealists who refuse to accept pragmatism. If you would really run a video site that willingly hosts those things, then I have no respect for you. At all.

<comment><USER6> > You can absolutely run a company that allows users to communicate in > interesting ways, and say you support freedom but also refuse to show > content you don't want to.

You can say you support free speech and ban free speech you don't like. What?

<comment><USER7> >You can say you support free speech and ban free speech you don't like. What?

Actually most countries that have free speech do actually ban certain things that fall under hate speech laws.

Some of the things in the parent comment aren't hate speech. However I think it's certainly possible for a company or government to decide between what should be allowed or not allowed, and to do that in a way that isn't morally objectionable.

Your argument seems to be something along the lines that companies and governments cannot possibly censor free speech in a morally acceptable way. It's a slippery slope argument: "ban ISIS and you'll eventually ban Trump supporters" (that seems to be jimmywanger's argument).

<comment><USER2> No, no slippery slope argument here. My argument is that hate speech is in the eye of the beholder, and is not well defined.

If we had well defined guidelines, such as pedophilia defined as nude people under 18 years old, that would be one thing. Although ridiculous (see all the teenagers who are put on registries for sending naked pictures to each other), at least it can't be twisted to make something else illegal.

Go ahead, define "hate speech". And if the laws already ban hate speech, why wouldn't we just punish the people who violate laws instead of censoring them?

And since hate speech laws vary from society to society, do we have to create a separate site for each country, where the allowable videos shown differ, and the comments are pruned differently?

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> What's "extremist"? Dabiq?[1] Stormfront? Donald Trump? The National Rifle Association? Fox News?

It's worth reading Dabiq to hear what ISIS has to say. Know your enemy. They're quite clear about wanting to kill everyone who doesn't support their strict interpretation of Islam. (ISIS cannot have allies; they consider that religiously prohibited.) The latest issue also includes a long article about why the Muslim Brotherhood in Egypt has abandoned strict Islam and is now an enemy. None of this is likely to recruit anybody who isn't already a fanatic of some kind. Unfortunately, there are those in search of something to be fanatical about. Even if they can turn only 0.004% of the population into fighters for their cause (that's roughly their recruiting success rate in the Western world), that's a success for them.

Dabiq mentions their 10 videos for this month, and searches for them turn up articles about them, but not playable versions of the videos. Even the one linked to "archive.org" isn't coming up. Unclear if this is censorship.

[1] [https://www.clarionproject.org/docs/Dabiq- Issue-14.pdf](https://www.clarionproject.org/docs/Dabiq-Issue-14.pdf)

<comment><USER9> _What 's "extremist"? Dabiq?[1] Stormfront? Donald Trump? The National Rifle Association? Fox News_

I think its pretty simple. If a given group has ever claimed responsibility for, expressed the intention to commit, or publicly praised, a terrorist act against civilians, that's an extremist group. Whatever can be done to block their communications with the rest of the world should be done. It's no mystery as to who they are - they're very public about it.

The same kind of analysis should be done for individuals that desire to enter the US or non-residents already visiting the US. _" Oh, you were on Facebook saying how the guy in Orlando was a hero? Sorry, time to go home."_

<comment><USER10> People protesting against austerity and gentrification have broken windows of banks and stores. Is this terrorism since in the end civilians own those? If it is where do you draw the line on censorship of the people who want the same things but were never involved in those crimes against property?

<comment><USER9> I consider it terrorism when the intent of the act is to hurt or kill people. I think most people, beyond those being deliberately pedantic, define it the same way.

<comment><USER11> First, that distinction is not so clear for many because the word "terrorism" is constantly banalized by conservative politicians that use it to attack protesters of all kinds. We have seen a lot of this during the last few years, where austerity protesters, mobs preventing unfair evictions, etc. have all been called terrorists by right wing politicians and media.

The other problem with your definition is that, while it may help distinguishing terrorist _actions_ from other actions, it does not help drawing the line on what kind of communications "are terrorist" or "promote terrorism" to censor? There are many tricky questions, for example:

\- Do you censor everything that a terrorist group says, or only whatever explicitly advocates terrorist tactics?

\- Do you censor sympathizers with the terrorist group?

\- Do you censor people that share the ideas of the terrorist group? Even if they do not defend them with violence? But what if they do not promote violence but they do not publicly condemn the terrorist group?

Etc, etc.

As you can see, the question is tricky. In Spain this debate has been alive for years with regards to the basque independentist group ETA. This was often a mess, with even certain independentist newspapers being prohibited at times. Sometimes these actions made ETA recruiting easier indeed--it's easy to argue that violence is your only alternative when your political voice is censored. In most absurd cases, people that are not even independentist or anything alike are being prosecuted for posting jokes involving ETA on Twitter.

So, it's not about being pedantic. Discriminating what kind of media promotes terrorism and not is hard. The political implications of censoring are huge, and there can be unintended consequences. It basically means drawing a clear political line on the limits of freedom of speech.

I would be afraid if these lines are drawn by private entities, secretly, without a public debate on what is tolerable or not, and with the public not even knowing whether they are being censored or not.

<comment><USER9> You raise some good points, but again I think _most_ of your concerns can be addressed using generally accepted common sense. Yes, I think that everything that a terror group says should be blocked, and yes I think sympathizers should be treated as members of the group when it comes to communications.

Let's also remember that in the context of Google and Facebook, we aren't talking about censorship by the government, or private organizations setting laws that all citizens must abide by. These are private organizations that are free to censor anything they wish, and to define their own terms of service. What they are doing seems to be a socially responsible move. People that are unhappy with these services are free to publish their stuff elsewhere.

<comment><USER12> Then those private organisations shouldn't advertise those services as pro free speech when it comes in handy.

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> Google, Facebook, and especially Twitter are used by terrorist organisations for recruitment and propaganda. These videos brainwash young people into hatred of the west, incite them to leave their countries to go join a war that is not theirs, polarize, and are directly responsible for suicide attacks.

Sure, information _wants_ to be free, and all that cyberpunk jazz. At what cost? Apparently at the cost of innocent lives. These companies are directly responsible for giving these terrorists a platform. The free speech-apologists are indirectly responsible for fostering fertile ground for these terrorists.

Yes... blocking this information will also block innocent information. But I rather see that, than see more innocent lives lost. I say: chase them underground. This makes more clear the difference between a real jihadist with bad intentions, and a pious person slowly being brainwashed, because their feeds show nothing but dead children being pulled from rubble, Anwar al-Awlaki propaganda, and anti-Semitic hoaxes.

<comment><USER14> interesting. I would never make this trade and find it horrifying.

> This makes more clear the difference between a real jihadist with bad > intentions, and a pious person slowly being brainwashed

While I respect you view it this way, I have trouble believing this is even slightly reminiscient of reality. You seem to imply that if not for terrorist propaganda on twitter, "pious" people would not be terrorists. I suspect this is an outlandish strawman devoid of any substance.

> At what cost? Apparently at the cost of innocent lives.

This has always been the cost. Many people have knowingly given their lives for such ideals. Certainly private corporations have the freedom and latitude to make their own decisions, however, there is a reason freedom of speech was built into the bill of rights.

edit: it was indicated that driving them underground was considered positive, I would also like to respectfully disagree. If we accept this is where the "real" terrorists are congregating (online communities) having them remain in plain view and analyzing their recruitment and propaganda techniques would likely be a high leverage option.

<comment><USER13> When "normal" citizens can find instructions on how to create IED's and pipe bombs on the surface internet, something is very very wrong. This is what causes naive easily-influenced youth to radicalize. They do not even have to seek it out on the deep web, their social feeds are filled with such extremist content.

Then when the police confiscates their computer and finds Inspire magazine on there, they are branded a terrorist and face jail. While there is a difference between curious rebellious youth and hardcore jihadists, it's very murky when there is such easy access to real damaging stuff.

Driving them underground would make anyone seeking them out a solid suspect. Right now innocent bystanders are confronted with their propaganda and may give in (whereas normally they never would).

I know these are American companies, but they are used world-wide. The bill of rights and American standards of free speech do not have to apply everywhere though. Europe faces the consequences of this unbridled free speech. You want free (hate) speech? Fine. Create your own website and host your own content. Don't (ab)use these big companies who hide behind our-algo's-could-not-find-it and do not take their responsibility, because they have so many users they can never police them all.

Real terrorists use both the surface web (for recruitment and propaganda) and the underground web (for planning attacks). No smart terrorist uses the surface web for planning. The damage of surface web propaganda far exceeds the damage of having no intel on surface web propaganda.

<comment><USER14> > When "normal" citizens can find instructions on how to create IED's and pipe > bombs on the surface internet, something is very very wrong.

I disagree. In the extremely unlikely event I have to disarm an IED I would like to be able to google this from my mobile. I also own a copy of the anarchist cookbook, and I have for about a decade (acquired around age 16). To date, I have not committed any acts of terrorism.

> This is what causes naive easily-influenced youth to radicalize. They do not > even have to seek it out on the deep web, their social feeds are filled with > such extremist content.

If being aware of (or actually downloading) torbrowser was the only impediment to terrorism with an IED I would be very uncomfortable. Luckily, I do not accept your appraisal of the situation.

> Driving them underground would make anyone seeking them out a solid suspect.

If we accept that interacting with terrorist online should make you a suspect, surely having them be easily accessibly would help lop off the low hanging fruit. I don't actually believe this would be a great course of action, but the logic of keeping terrorists (and potential terrorists) in plain sight is largely pretty sound. Also, how would we verify who was seeking out these terrorists if channels of communication are largely asynchronous and offline as well as executed with varying degrees of obfuscation?

> The bill of rights and American standards of free speech do not have to > apply everywhere though.

Correct. I indicated they do not have to follow the bill of rights, it is a bill of rights garunteed to be provided by the U.S. Government to its citizenry. However, the point I was making is that many sovereign nations allow for free speech because it is fundamentally important. So important, it is at the cornerstone of the culture of a young and extremely prosperous nation: America.

edit: I respectfully disagree with almost every comment you have made. I don't want to levy an accusation that you are somehow complicit (or actively involved) in perpetuating myths about terrorism, free speech and the tornetwork but if you are not, I find your rhetoric painfully ignorant.

</comment></comment></comment></comment><comment><USER14> I think an important point which is related is that these companies control information. DMCA requests and the companies issuing them have become much more sophisticated. The joke is that Bing is much better for porn. However, I personally have noticed many files, images and other media have completely disappeared from google results.

The amount of information on the internet is of course staggering, but no one seems concerned that it is accessed through a very tightly controlled pipeline.

edit: not sure how anyone could disagree that the english speaking internet is almost exclusively controlled by google and (to a lesser extent Microsoft). Chrome and Edge/Explorer have ~67% of browser market share and google has ~75% SE market share while bing has ~7%. 2 companies collectively control the environment and on ramp to the english speaking internet. Facebook (and now microsoft with the purchase of linkedin) as well as google control the bulk of people's online identitites.

</comment><comment><USER15> As others have noted, the issue is that so many political terms are relative (e.g., the way 'terrorist' is applied to Middle-Easterners far more swiftly and casually than Americans who are equally violent for very similar reasons.)

An organization doing what Google and Facebook are doing must therefore (a) clearly articulate the position on which their judgments are based (b) openly assert that their position is fundamentally superior and preferable to the moral order that accepts the practices that are being suppressed (c) provide a open and defensible argument for selecting one set of values while forcefully rejecting another and (d) be willing to modify this position in the event that fatal contradictions appear, or excessively negative consequences result.

In short, these platforms must not just govern, but do so from a well- considered and articulated philosophy of liberal governance.

Fortunately, they need not start from scratch. There is an abundance of data to support the idea that nations that have broadly inclusive economies with widely and equitably representative policy making bodies are the most prosperous, stable, and desirable places to live. (For details, see "Why Nations Fail" by Daron Acemoglu and James Robinson [https://www.amazon.com/Why-Nations-Fail-Origins- Prosperity/d...](https://www.amazon.com/Why-Nations-Fail-Origins- Prosperity/dp/0307719227/ref=sr_1_1_twi_pap_1?ie=UTF8&qid=1466900793&sr=8-1&keywords=why+nations+fail))

By adopting for themselves the governing principles that underpin the most successful societies, Internet platforms can become greater forces for good. At the same time this empirically grounded approach will better shield them from maliciously self-serving people who want to see political and economic power concentrated in the kinds of small and closed sets of hands (specifically, theirs).

As any number of examples from Egypt to Russia to Venezuela to Zimbabwe have shown, those who dominate closed economies and the political systems that reinforce each other invariably make life a living hell for everyone. To the extent that these systems are maintained by way of retrograde social norms, any platform committed to a more open, just society should have no qualms about disarming those who view civil society as anathema.

</comment><comment><USER16> I have never came across any extremist video on facebook, youtube or even twitter. Maybe if I was actually searching for it I could find it? What sort of terms would one use?

</comment><comment><USER17> Equivalent to pretending that Hitler never existed.

</comment><comment><USER18> Not sure if this a good thing they would just move further underground vs. allowing it to continue and flag these accounts/send notice to law enforcement and the FBI. They supposedly track us anyway hopefully they are tracking these degenerates more then sally sue saying how much she loves joe weiner.

</comment><comment><USER19> I wonder what it is they hope to accomplish by blocking this type of content. One could argue that what _actually_ happens is a Streisand effect that promotes the views these companies are attempting to squash. My belief is that movements like Trump are a direct effect of overreaching censorship by the left which has essentially created a new right wing movement due to people backlashing against left wing media shaping.

In short, if the ideas are truly terrible they should be left viewable to everyone since they'll only turn people off. It's when you hide things that people become curious as to why the content is powerful enough to require censorship.

<comment><USER4> If by "extremist content" you mean religious fanatics beheading people, or calling people to arms, or giving tips on producing bombs or whatever, then you are entitled to treat this as "unpopular ideas" which "need defending in a free society" or whatever, but don't be surprised if sane, rational people take a somewhat different view and decide they don't want to spread hatred, nor leave themselves open to lawsuits for the honour of hosting this content for free. All companies like to talk about fostering freedom or whatever, but as organisations which exist to make profit it's partly because censorship costs them money, and also the more content they have, the more of a surface they have to make money from. They don't really give a shit about you or your freedom, and there's precious little internal conflict about whether to host this months crop of terrorist atrocities.

<comment><USER19> I'm not talking about videos of beheadings (although there's a case for keeping things like that around too to help educate people). I'm talking about suppression of everyday right wing viewpoints. Things like this: [http://gizmodo.com/former-facebook-workers-we-routinely- supp...](http://gizmodo.com/former-facebook-workers-we-routinely-suppressed- conser-1775461006)

For context, I'm not right-wing myself but absolutely object to censoring content based on political viewpoints such as this.

<comment><USER4> From reading that, the only criticism that seems sane and fair is that they should have chosen a different string to "trending"; "recommended" perhaps, or "curated". My expectations that i'm going to receive a much-needed education in some topic from occasionally clicking on the "trending" link on facebook aren't especially high; perhaps yours are a little misplaced if present?

Also, I note that that article states that other "curators" denied such a bias exists. Facebook have denied such activities. Ok, being sceptical, they would say that. But it could be bullshit spewed by an ex-exmployee So, where's the study? The data? How do we show whether or not there is a bias, assuming you care enough to investigate? It's just more conspiracy theories and "typing" isn't it.

</comment></comment></comment></comment><comment><USER12> "Boohoo, I want to feel safe in my safespace and don't feel icky feelings" \- modern Internet user</comment>