Ruby on Rails and the importance of being stupid - <AUTHOR> http://blogs.law.harvard.edu/philg/2009/05/18/ruby-on-rails-and-the-importance-of-being-stupid/

<comment><USER4> Judging by some of the comments here, it seems people are giving Greenspun a free pass because he's apparently getting at deeper point. However when I read this article, it is chock full of straw men. The comparison between a competent Microsoft programmer vs a complete bumbling fool labeled as an MIT Genius is at best intellectually dishonest. I wrote a lengthy response which I'll post here in case his moderator decides he doesn't like it:

I read the moderation policy where it's suggested that reviews of the post are not valued. However I feel an obligation to point out the factual errors in this post. There are dozens of nonsensical assertions and could potentially be very misleading to anyone who doesn't understand Rails or web development in general.

My first general critique is that there is no real comparison going on here. It says the business guy called up Microsoft and they recommended buying a bunch of hardware, but there's no discussion of who developed the site or how they got up and running. There's no discussion of the price of the hardware, which clearly looks to be well into the 5-figures, or the price of the fiber connection at home, system administration, backups, etc. To get into some specifics:

 _The programmer, being way smarter than the swaptree idiot, decided to use Ruby on Rails, the latest and greatest Web development tool. As only a fool would use obsolete systems such as SQL Server or Oracle, our brilliant programmer chose MySQL._

This is a caricature of an "MIT Genius" that doesn't jive with reality. Anyone who was actually that smart would know better than to dismiss Oracle in favor of MySQL. They may prefer using Ruby on Rails and be more productive than if they used .NET, but they wouldn't go around calling people idiot's for such superficial reasons. Therefore you're not describing an actual genius, just someone who thinks they are a genius, but is actually a fool. Using such a person as the basis for an argument of why Microsoft's recommendations are better than Rails is intellectually dishonest.

 _How do you get scale and reliability? Start by virtualizing everything. The database server should be a virtual "slice" of a physical machine, without direct access to memory or disk, the two resources that dumb old database administrators thought that a database management system needed._

The reason that virtualization is done in the web deployment world is so that you can get access to fast and reliable hardware if you need less than the cost of the full resources. A degenerate example would be that if your capacity requirements could be met by a 250mhz processor, you would get better throughput by using 1/8th of a 2Ghz server. The reasoning for this is that the vast majority of sites don't need dedicated hardware, which you seem to imply as being cheaper, but clearly it is not if you are leasing server capacity.

 _Ruby and Rails should run in some virtual "slices" too, restricted maybe to 500 MB or 800 MB of RAM. More users? Add some more slices!_

I'm going to assume you are talking about EngineYard here, since that is the managed Rails hosting provider I am most familiar with and is somewhat inline with your pricing figures below. First, the 500 or 800 MB is just a base amount of RAM that is good for most small Rails apps. When that starts to run out, the solution is NOT to add more slices, you simply commission more RAM. EY can do this without even restarting your slice. Incidentally you can also commission more CPU if you need it. The reason they start with two production slices is for redundancy. One of your slices goes down for some reason? That's okay because there's a backup.

 _The cost for all of this hosting wizardry at an expert Ruby on Rails shop? $1100 per month._

What you described above is a very poor description of what you are paying for at a managed hosting provider like EngineYard. I will describe managed hosting in a minute. But to compare to your unmanaged Microsoft example, I currently pay $8/month for a 256MB of unmanaged hosting that is plenty to server significant traffic on a well optimized app. This is an order of magnitude less than the Verizon FiOS line _alone_ , and provides much better network connectivity (ie. multiple tier-1 connections, lower latency to more endpoints).

With managed hosting at EngineYard, you are not just paying for the server. You are basically paying for a fulltime system administrator. They have people all over the world ready to help you at a moment's notice any time of day or night. They proactively monitor your server and contact you if they notice any abnormalities. They provide a large suite of finely tuned recipes and standard software installations that they can install on a moment's notice, and will tie into their monit-based server monitoring setup. The individual machines in the cluster are optimized for their specific tasks. The network hardware and topography is optimized for real world usage scenarios. They continuously tune the machines for throughput and move clients around to avoid bottlenecks. They will even take significant steps towards helping the client tune their own application, above and beyond their contractual obligations for server adminstration. In short, you've completely ignored 95% of what they do, and painted it as extremely expensive without even providing a comparison against the overhead costs of buying and managing your own servers.

 _For the last six months, my friend and his programmer have been trying to figure out why their site is so slow. It could take literally 5 minutes to load a user page. Updates to the database were proceeding at one every several seconds. Was the site heavily loaded? About one user every 10 minutes._

If a request on an unloaded server takes 5 minutes to load, and the programmer can not figure it out in 6 months, then that programmer is incompetent plain and simple. Laying this at the feet of Rails is just plain ridiculous.

 _I began emailing the sysadmins of the slices. How big was the MySQL database? How big were the thumbnail images? It turned out that the database was about 2.5 GB and the thumbnails and other stuff on disk worked out to 10 GB. The servers were thrashing constantly and every database request went to disk. I asked "How could this ever have worked?" The database "slice" had only 5 GB of RAM. It was shared with a bunch of other sites, all of which were more popular than mitgenius.com._

Are you implying that you need enough RAM to keep the entire database in physical memory? That is patently false. In a worst case scenario, yes it could take performance down quite a bit, but disc access is not nearly as slow as implied above. I've served tons of sites on pure shared hosting (not even virtualized) with much higher load and orders of magnitude better performance than you are describing here.

 _How could a "slice" with 800 MB of RAM run out of memory and start swapping when all it was trying to do was run an HTTP server and a scripting language interpreter? Only a dinosaur would use SQL as a query language. Much better to pull entire tables into Ruby, the most beautiful computer language ever designed, and filter down to the desired rows using Ruby and its "ActiveRecord" facility._

This is nonsense Philip. Please don't take this as an ad-hominem, because there's no other way to put this. What you described here is 100% pure nonsense. ActiveRecord, like any ORM component, abstracts away some SQL in order to simplify common database interactions. The lion's share of ActiveRecord code is all about constructing efficient SQL. When you are developing with Rails it shows you all the SQL running the development log, and you can quickly spot n+1 errors. If you need something more efficient, it offers plenty of levels of access right down to pure SQL.

 _In reviewing email traffic, I noticed much discussion of "mongrels" being restarted. I never did figure out what those were for ... What am I missing? To my inexperienced untrained-in-the-ways-of-Ruby mind, it would seem that enough RAM to hold the required data is more important than a "mongrel". Can it be that simple?_

I'm shocked that a programmer would speculate so wildly as to say something like this. A mongrel is an application server. I don't understand what you seem to think it is, but it's simply the process serving up Rails requests to the web server and passed through to the client. Typically you run more than one so you can serve multiple requests concurrently, but for a well-optimized app usually no more than 3 or 4 are necessary. Rails uses a non-threaded share-nothing architecture which means you can scale horizontally across unlimited servers. Note that I am not talking about virtualized servers. I'm talking about when you have more traffic than the biggest server in the world can handle, Rails will let you scale out painlessly at the web server level until your database can not be served by a single box. At that point you need to look at database sharding, or alternative data stores using Map-Reduce or some other scalable database solution.

None of this is to say Rails doesn't have its warts. Ruby is memory hungry, leaky, and relatively slow. Deployment has traditionally been very complicated compared to something like PHP (although it's much improved with Phusion Passenger aka. mod_rails for Apache/Nginx). There are many reasons why you would be well-advised not to use Rails, however this article doesn't touch on any of them. Rails, just like Oracle, .NET, Java or many other technologies is a proven platform with pros and cons. In this article you pit an apparently competent programmer developing swaptree.com against what can be described as nothing less than a complete bumbling idiot using Rails. You insist the cost of Rails is high without any justification or direct comparison against the costs of swaptree.com.

I've read your blog in the past and found it to be pretty interesting, which is why I've taken the time to write this response, and suggest politely that you retract this article.

<comment><USER5> You are glossing over a lot of sarcasm, I think.

<comment><USER4> Fair. However it's really hard to isolate the sarcasm because there's no real factual meat to the article.

<comment><USER6> My short reaction: Good lord, you know all that and still feel like the article is targeted at you? Or at Rails as a technology?

Long reaction: You missed the point of the article, which is that keeping on top of the latest and greatest technologies is almost never necessary, and it is _never sufficient under any circumstances_. You don't have to know what a mongrel is. You do have to understand the orders-of-magnitude difference between different levels in the memory heirarchy. (RAM is much better than disk -- a simple, stupid fact that people ignore all the time.) There are lots of people running around with credentials and hot technologies who don't know what they're doing, and there are lots of young people who worship those guys and spend their time running after trendy stuff because they haven't yet figured out the difference between learning technology and deciding what to wear. (Which might not be as bad as relying on engineering principles to choose your wardrobe. Hmmm, personal food for thought.)

Sure his article isn't particularly original in intent or execution, but the need for this article is perennial. You have to keep updating it because it's aimed at people who only pay attention if you talk about the current latest and greatest. That's why Rails was the perfect victim -- that's where his target audience _is_ right now. (And the Microsoft stack is the perfect frumpy foil to Rails.) Not that the Rails community doesn't contain other kind of people; it evidently does, or posts like yours wouldn't exist. But it is also The Trendy Thing and is therefore cursed with attracting the naive my- favorite-band-is-better-than-yours types who think "follow the buzz" is _the_ successful strategy for _all_ domains of life.

Fast-forward ten years, and I'm sure he'll have written the same article with the blanks filled in with another hot technology. Which is a good thing.

</comment></comment></comment></comment><comment><USER7> Yawn. Yet another misinformed "Ruby/Rails can't scale" article.

Seems to me the main problem is that his "MIT trained" friend had no experience building a scalable web service. He would've botched it up in PHP, Python, Java, whatever. There's nothing about his main mistakes- running your database on a shared server, naive (ab)use of SQL- that is unique to Rails.

And this bit: "pull entire tables into Ruby, the most beautiful computer language ever designed, and filter down to the desired rows using Ruby and its "ActiveRecord" facility" is _completely_ incorrect and makes it obvious Philip Greenspun knows less than nothing about what he's ranting about.

<comment><USER8> I didn't read "Rails can't scale" at all. Did you even finish it? He recommended still using Rails at the end.

<comment><USER9> well it would be bad advice for the programmer to simply throw away all of his work at that moment and start from scratch - regardless of what you think of the technology he used

"Rails can't scale" was implied

<comment><USER8> I don't think it was implied at all, especially since he recommended using it at the end. It's implied only if you read the title and nothing more.

What was both implied and directly stated was that a cloud-based architecture is often not the best idea for a lot of people, despite the modern mania for it.

<comment><USER10> Phil specifically addresses the idea that he is dissing RoR in a comment.(emphasis mine)

"Angry Rails Enthusiasts Whose Comments I Deleted: A lot of the comments were of the form "Your assertion that it is impossible to build a responsive Web site with Ruby on Rails is wrong. Rails is in fact great if programmed by a great mind like my own."

The problem with this kind of comment is that _I never asserted that Ruby on Rails could not be used effectively by some programmers._

 _The point of the story was to show that the MIT-trained programmer with 20 years experience and an enthusiasm for the latest and greatest ended up building something that underperformed something put together by people without official CS training who apparently invested zero time in exploring optimal tools._

Could some team of Rails experts have done a better job with mitgenius.com? Obviously they could have! But in the 2+ years that our MIT graduate worked on this site, he apparently did not converge on an acceptable solution.

My enthusiasm for this story has nothing to do with bashing Ruby or Rails. I like this story because (1) it shows the fallacy of credentialism; a undergrad degree in CS is proof of nothing except that someone sat in a chair for four years (see [http://blogs.law.harvard.edu/philg/2007/08/23/improving- unde...](http://blogs.law.harvard.edu/philg/2007/08/23/improving- undergraduate-computer-science-education/) for my thoughts on how we could change the situation), (2) it shows what happens when a programmer thinks that he is so smart he doesn't need to draft design documents and have them reviewed by others before proceeding (presumably another set of eyes would have noticed the mismatch between data set size and RAM), (3) it shows that fancy new tools cannot substitute for skimping on 200-year-old engineering practices and 40-year-old database programming practices, and (4) it shows the continued unwillingness of experienced procedural language programmers to learn SQL and a modicum of RDBMS design and administration, despite the fact that the RDBMS has been at the heart of many of society's most important IT systems for at least two decades."

That is exactly what I understood from the article.

I don't see any rails bashing in the original article and you would have to cherry pick phrases to get that idea. I read the HN comments first and thought phil had gone off on a rant against RoR t judge from some comments here.

that will teach me to read HN comments before reading the original article!

<comment><USER4> It doesn't show any of that because it's all made up. He just slapped together a story that would appeal to someone like you based on your preconceptions, but there is no actual argument. The whole thing could be reduced to "idiots can't write software" and it would lose no substance.

Even some of these points that are supposed common sense engineering wisdom are specious. Do you need to draft design documents to build a workable product? Of course not! Is the first thing you should do when you start a new website to buy $20k worth of hardware? No! Do you need enough RAM to hold your entire database? Maybe it's the best optimization you can do, but it's far from a foregone conclusion.

Why I am I so vitriolic? Because the article is not truthy. The quote above says "MIT-trained" in the same sentence as "without official CS training." Uh, it doesn't get much more official than MIT. Suggesting that a programmer with 20 years experience couldn't get a single web page to load faster than 5 minutes is a flight of fancy plain and simple.

I might as well write a long-winded story about how Microsoft hired a Chimpanzee to program the next version of Word, and failed, therefore mammals make terrible programmers and furthermore decided to use C based on ill- informed simian whimsy.

</comment></comment></comment></comment></comment></comment><comment><USER11> "Bad programmer writes bad code"

This is hardly a new idea. The fact that a bad programmer used Ruby on Rails, or Django, or PHP, or C++ to write bad code and implement it on a shoddy system is no reflection on anything. This is essentially a story of someone who took good advice for a hosting environment, and someone who took bad advice for a hosting environment.

Learn your tool, don't buy into the hype. Make sure you are aware of the reason behind everything that you do (because they I read it on a blog is not a reason). Don't be a bad programmer.

<comment><USER12> Simply knowing particular languages has been pointed out on sites not too far away from here as a sign of a Good Programmer. This counts as evidence to the contrary. But not absolute evedence, I guess we all still have to think for ourselves.

<comment><USER13> I could understand an argument that knowledge of certain languages is correlated with being a Good Programmer.

This guy seems to have heard about the cool tools of the day, jumped in, and failed mostly due to a lack of experience.

One great lesson of experience is that there is often a simple solution to complex problems.

</comment></comment></comment><comment><USER14> "Configure the system with no swap file so that it will use all of its spare RAM as file system cache."

That will do EXACTLY the opposite of what he wants. Give it a swap file and unneeded parts of memory will be swapped out to disk, freeing memory for use as a file system cache.

There is _never_ a reason to configure a system without a swap file - <USER2> if it's a laptop and you don't want the disk to spin up.

Don't want the system to use swap space? Don't allocate more memory than you have (or buy more memory), but disabling the swapfile never helps. In some cases disabling it doesn't hurt anything, but it never helps.

<comment><USER15> Why should it ever be acceptable for someone to sit down in the morning to a machine with 4GB+ RAM and have things like a volume OSD take tens of painful seconds to swap in because the night before the machine ran 'updatedb' and the system decided to swap out a few bits of "unused program memory" with a useless cache of the entire disk index?

<comment><USER7> Since you know its useless, why not just turn off the updatedb cron job?

<comment><USER16> Because updatedb isn't inherently useless, but rather that file system caching (especially for pages read once) shouldn't eject application pages from RAM to disk.

Being able to run locate and have it quickly return an accurate result _is_ useful at times. I just don't want it paging my entire session out to disk every night in order to do that.

<comment><USER14> "I just don't want it paging my entire session out to disk every night in order to do that."

It doesn't.

</comment></comment></comment></comment></comment><comment><USER17> This is a slam on virtualization, not Ruby/Rails. Bad title, but the overall point remains: When you buy a "slice" of something, you have no idea what you are really buying. If you buy a piece of hardware with 32GB of RAM, then that's what you get. And if you know what you are doing, it is going to be much cheaper than buying "slices." In other words, the whole pizza is always cheaper than if you were to pay for 8 separate slices.

<comment><USER18> Most virtualized infrastructure providers give you some guarantee of how much RAM, CPU, and diskspace you will have available. You may be able to exceed the limits from time to time, but the minimums should be clear when you sign up.

<comment><USER19> I haven't seen any that guarantee disk access latency.

<comment><USER20> Probably because it would be impossible without having a dedicated physical disk for each slice on the machine, or some kind of really fancy network storage array.

Maybe that will change once SSDs become the default hardware. Without the bottleneck of seek time, latency should be much easier to quantify. Also, since you wouldn't be penalized for "context switching" (I know this term doesn't really apply to disk IO, but I mean switching disk jobs often, which requires a head move on HDs) you could maybe someday slice up the SSD time like a CPU and guarantee it directly. (For instance, if the SSD is capable of 200mbps, you're slice could be guaranteed 10mbps. Or something more technically realistic, I am but an amateur.)

</comment></comment></comment></comment><comment><USER21> Ram is King for most workloads; and he's right, it's ridiculously cheap. If you need more than 16GiB ram, it makes a lot of sense to buy your own hardware and co-locate it.

Now, I disagree about keeping it in your basement, at least once you have users on it. co-locating a 2 cpu box is going to cost you around $100/month, and you get much better connectivity than DSL at that price. DSL isn't much cheaper, if you get a reasonable uplink speed (at least around here) and I don't know about you, but the power in my house (I live in California) isn't exactly /enterprise grade/.

But yeah, I see a largely untapped market for renting high-ram otherwise-cheap servers; I've rented out one 32GiB server with a bunch of drives (and some slow CPUs) to a guy for $1200 setup and $175/month... once my next load of servers is up and built, I'm thinking about chasing that business model again, if I can build the servers faster than I get new vps signups, anyhow.

</comment><comment><USER22> The problem is not that you need better hardware for this scenario, you just need a better programmer.

"Not helping matters was the fact that the sysadmins found some public pages that went into MySQL 1500 times with 1500 separate queries (instead of one query returning 1500 rows)."

Looks like someone forgot to use :include on some finders. Let's say you have 1000 users with an address each. This will produce 1000 SQL queries:

      User.all.each do |user|
        p user.address.street_name
      end

This, however, will only issue two queries:

      User.all(:include =>:address).each do |user|
        p user.address.street_name
      end

<comment><USER23> Well, if you know your way around RoR, the first thing you do after most development is done, is to load the query_reviewer plugin to get an automated profiling on each page, and take steps to limit the number of queries. Either by memcached, or by adding indexes. It's not unusual to go from a couple of hundred queries per page to less than 10 per page in a day of optimizing.

</comment></comment><comment><USER24> Interesting read. I work with Ruby on Rails myself but I'm not that knowledgeable about scaling it.

It seems to me they didn't really know what setup they were running if they are wondering what a 'mongrel' is.

I hope they weren't trying to serve the site on only a couple of mongrels.

My first thoughts,

Benchmark a bit, use a tool like fiveruns to find out what's really happening. I wonder what the real bottleneck is.

Of course they shouldn't use a shared database server, but I'm wondering if they are they using caching? What I'm reading about that site I think they could cache the hell out of it. I've only used the builtin Rails cache methods but a tool like memcached should help out on all the database requests.

<comment><USER25> Thankfully there are numerous parties out there that will handle the scaling issues for you (heroku.com, engineyard.com) so you can concentrate on what's important, the app itself.

<comment><USER26> If the application is performing idiotic operations such as "[querying] MySQL 1500 times with 1500 separate queries (instead of one query returning 1500 rows).", merely moving to a better provider or using more powerful servers won't help. A modern web-based application is very complex -- the programmer _has_ to understand how the database and application communicate, how HTTP works, how to cache data, etc.

<comment><USER27> As a RoR developer I have often been hired into optimize various services. One of the biggest issues I have seen is this exponential growth of queries.

I have routinely seen requests at clients with 2-3000 queries (even with query caching). Most of them are small, but at that amount it doesn't matter how small and efficient each query is.

I love AR to death as an OR library, but it is extremely easy to get into these kinds of issues when you iterate over a large dataset and then call associations of associations without thinking too much about it. I don't think it is only an AR issue, conceptually I think it is true for all ORMs.

They can be a PITA to unravel and are often very hard to do in a clean AR like way in more complex data models. Normally you end up doing some fairly un AR like pre loading like you would do in a pre ORM app, which while ugly works.

<comment><USER28> Agreed, I do all my dev with query trace in order to make sure I'm not looping back into individual queries.

to those that don't know, Rails doesn't load the children of AR objects by default, so if you do something like

Select * from books

and then iterate through the books and get books.author_name where the author data is a relational table, you're going to get a separate query for each row.

<comment><USER29> Usually the point where you could get into trouble would look more like: book.author.name in the given example.

</comment></comment></comment></comment></comment></comment><comment><USER30> Ruby on Rails and the importance of being COMPETENT.

</comment><comment><USER31> Shouldn't the lesson to take away from this rather be that it's important to _not_ be stupid? Clearly, the "MIT trained" programmer don't know what what he's doing, at all.

I'd expect similarly disastrous results had he used the same 100% ignorant approach to any other language or platform.

</comment><comment><USER32> Umm, it sounds like some tweaking of your ActiveRecord::Base.find calls with the ":include" parameter might improve performance by 10,000x as well.

</comment><comment><USER9> this is a little funny. This isn't the first time Greenspun has been evangelizing the benefits of MS ASP (and now ASP.NET) vs x technology (back then he was deriding Java). The structure of the post is the same: some sensationalism with some missing context (since he doesn't usually do his homework on what he is criticizing).

I guess some things don't change (much)

<comment> patrickg-zill I think he is using the MSFT technology as the worst case or "caveman" approach and then contrasting it with situations where the person outsmarted themselves.

</comment></comment><comment><USER33> Classic case of overengineering the problem. I'd venture that it's a common enough class of mistake for those who have recently learned the hip/new/cool way of doing a thing. Thing = scaling a webservice in this example.

</comment><comment><USER34> seems to be about ruby and rails by coincidence.

</comment><comment><USER35> This post spoke poorly about two concepts: * query optimization * robust hosting Both of which are generic and have nothing to do with Ruby on Rails in particular.

</comment><comment><USER36> I'd been puzzling why the name was so familiar and it's because this man is the source of the ultimate Smug Lisp Weenie quote...

Greenspun's Tenth Rule of Programming: "Any sufficiently complicated C or Fortran program contains an ad-hoc, informally-specified bug-ridden slow implementation of half of Common Lisp."

</comment><comment><USER37> ORM is rarely effective unless it is just used for CRUD. Most OOP programmers using ORM rarely use it efficiently.

<comment><USER38> ORMs are extremely useful. Many extensions/plugins/gems/whatever can be developed for an ORM. With ORMs not being tied to one database technology extensions to the ORMs can reach a wide audience. Look at Rail's ActiveRecord. There are many extensions based on that ORM that tie into Rails allowing for developers to not have to recreate the wheel over and over. When using ActiveRecord I have seen how a simple over-site can lead to inefficient database use, but it makes that inefficiency pretty obvious and can usually be easily corrected. ORMs can also take care of query caching and other optimizations for you. If you pay attention to what your ORM is doing for you, an ORM can be efficient and a giant time saver. BTW, what do OOP programmers have to do with things anyway? Do functional or procedural programmers use ORM more efficiently?

</comment></comment><comment><USER39> just built a service in PHP that can process 15,000 rows of data, with a complexity of about O(1000*(10|100)n) and it runs in 4.5min. I was looking to beat 15m. I expect to get that down further.

right now i use a LIMIT of 100 to minimize sql queries, and memory. i might try going to 1000 rows or about 10k in data, but in my experience when the DB has to return that much data you are not really gaining that much.

what do you think? any other optimizations i should look to?

<comment><USER40> One optimization is to write big O notation in minimized form. It looks more impressive to write O(n).

And to try being helpful: do aggregation in the db where possible, be sure indexes are good, play around with the number of rows you process at a time, use joins if appropriate to minimize queries, if your queries are expensive at all use EXPLAIN, depending on the database, look into using a cursor (maybe?).

<comment><USER39> Cursor?

I was using LIMIT and OFFSET, did you mean something else?

update: I processes 200k-300k rows in a few hours, and since the DB is on the same box, as the php app, there is no need to make this more complicated.

</comment></comment></comment><comment><USER41> Troll bridge. Pay troll.

<comment><USER7> Trolls are usually anonymous.

</comment></comment><comment><USER42> Hm. It seems the linked article is somewhat over-engineered for general consumption.

</comment><comment><USER43> Choice appropriate tools for your task, not the task for available tools. MS fans, Delphi fans, Java fans and now RoR fans are thinking second way - <USER3> love their tools.

The altrernative approach is about using mix of technologies and tools to complete actual tasks. Today's linux distributions, which are actually mix (if not mess) of applications and tools written on every possible scripting language is a good example.

In the area of web-development the same approach works the same way - you can build different [sub]services and [sub]systems with different tools which are more appropriate for some particular task. REST-JSON-Key-Value-Storage here, classic SQL back-end there and so on.</comment>


Sorry Gartner – NABD customer engagement solution has done it - <AUTHOR> http://www.nabdsys.com

<comment><AUTHOR> According to Gartner:"An important evolution in the market is the as-yet- unsuccessful attempt to create a single business application that joins two sets of capabilities, such as the following: A customer record, the database and business process rules that enable agents The communications and context routing (for example, email, chat) that enable customers"

NABD has made a good progress in this direction with its integrated BMP engine, routing rules engine, collaboration framework, and multi-channel unified experience.</comment>


Live map of London Underground trains - <AUTHOR> http://traintimes.org.uk/map/tube/ <comment><USER2> Matthew Somerville who made this has a bunch of other awesome things on his website - <http://www.dracos.co.uk>

One I used just yesterday was for finding postboxes in the UK given a postcode - <http://www.dracos.co.uk/play/locating-postboxes/>.

</comment><comment><USER2> Source code at <https://github.com/dracos/underground-live-map></comment>


Reasons to get excited about KDE in 2018 - <AUTHOR> https://dot.kde.org/2018/01/16/reasons-get-excited-about-kde-2018 <comment><USER2> Is 'Wayland only' result of valid reasoning? Is Wayland successfully replacing X11 on Linux? I am not using Linux for some time now so I am not sure.

<comment><USER3> I don't know about the other distros, but it's now the default on Ubuntu, which is one of the most popular.

<comment><USER4> Default on Fedora Gnome since 25.

</comment></comment></comment><comment><USER5> There are no reasons to get excited by KDE in 2018 because there isn't a single stable distro that supports it running (maybe opensuse, I did not try that). Mint KDE was amazing, but out of the dozen or so others I've tried, none has been stable and some haven't even installed. Hardware is an 2013 old retina laptop so all the drivers exist and are mature. Big distros like Kubuntu and KDE Plasma were the worst here, almost unusable after install. Fedora simply didn't install. As a long time KDE user, I've decided it's finally time to throw in the towel and switch. Clearly, no one cares enough to actually make a stable system out of this amazing desktop environment.

<comment><USER6> > maybe opensuse, I did not try that

Yes, openSUSE, you should try that.

openSUSE has a massive test suite, called openQA, with which they automatically test their builds. And among other things, they also maintain two builds with the latest KDE commits on top of their more stable version openSUSE Leap and one on top of their rolling release openSUSE Tumbleweed (those builds are called Argon and Krypton respectively).

So, they have integration testing all the way through, prior to an official KDE release already. This also means that they can essentially always ship the latest KDE Plasma LTS on openSUSE Leap without any problems.

And, KDE gives back for all of that effort and then schedules LTS releases to be relatively close to openSUSE Leap releases.

So, openSUSE is that distro that you're looking for, which takes this amazing desktop environment and makes a stable system out of it. Like, honestly it's hard to remember the last crash that I had (when I was not dicking around with random KWin scripts, that is).

The aforementioned Manjaro, I've also heard good things of, in that it at least does not treat KDE as a second-class citizen, but it's still miles away from all that extra effort that openSUSE puts in.

</comment></comment><comment><USER7> Did kde sandbox the file preview thing yet?</comment>


A discussion about the breaking of the Internet [KIK head of messenger] - <AUTHOR> https://medium.com/@mproberts/a-discussion-about-the-breaking-of-the-internet-3d4d2a83aa4d#.ljlbfuyc0 <comment><USER2> dupe, submitted by the author before, discussion there: [https://news.ycombinator.com/item?id=11346845](https://news.ycombinator.com/item?id=11346845)

</comment><comment><USER3> Anyone else start off siding with Azer, but actually think Kik weren't that unreasonable (though could have worded things better)? They even offered to buy the name off of him.

I think this guy is doing OSS a disservice. Trying to be like Linus in terms of temperament but not having the years of respect required to pull that off...</comment>


Ask HN: How should I build a simple marketplace website today? - <AUTHOR> I think I'm over-engineering so I could use a second opinion. Let's assume that nothing off the shelf met your needs, so you had to build the website from scratch, what programming language, frameworks, databases, hosting platform, etc would you use to build your application.
Requirements:
* 500 daily active users
* 1,000 product listings
* Very low maintenance
Features
* Browse items for sale
* Search, filter products
* In app messaging (Buyers can message sellers, sellers can respond)
* CRUD interface for sellers listing products
* User authentication
It feels stupid asking the question, but it seems like none of the "modern" web frameworks fit such a simple use case. <comment><USER2> > it seems like none of the "modern" web frameworks fit such a simple use case

Why is that? To me it looks like Django or Ruby On Rails would fit the bill perfectly.

The only things you'd need to do by yourself is the searching & filtering (possibly integrate with ElasticSearch since conventional databases aren't really good at search, especially not full-text search where you want a certain fuzziness to account for typos).

Authentication comes for free thanks to the framework, and Django even gives you the admin-side for free so you have a basic CRUD interface for adding/deleting products.

<comment><USER3> For these low numbers, Postgres will do full text search just fine. So as mentioned, just use Rails or Django with Postgres.

<comment><USER2> Is it fuzzy though? My concern was more about ES being able to handle "approximate" matches; where as I assume Postgres would be limited to regex- like matches.

<comment><USER4> Postgres can definitely do fuzzy matches!

[0] [https://www.rdegges.com/2013/easy-fuzzy-text-searching- with-...](https://www.rdegges.com/2013/easy-fuzzy-text-searching-with- postgresql/)

</comment></comment></comment></comment><comment><USER5> No need to build this - Sharetribe or Arcadier provide this functionality as a SaaS product and can be customised down the line.

[https://www.arcadier.com](https://www.arcadier.com)

[https://www.sharetribe.com](https://www.sharetribe.com)

</comment><comment><USER6> If the point is the marketplace itself, rather than the software, I would start with something that already has marketplace functionality.

I'm sure there are others, but Prestashop has a user-contributed marketplace extension that's relatively cheap, and comes with the source code. Prestashop itself is open source. It's all php, but I'd argue that might be a plus if you're going to iterate quickly on look/feel/functionality.

That way, you could see how well the concept works before investing any time rewriting it to fit your needs.

Also, see Stripe connect as a way to get your "cut" from marketplace sales. It's supported by the module mentioned above: [https://stripe.com/connect/use- cases](https://stripe.com/connect/use-cases)

</comment><comment><USER7> I used Laravel PHP framework to build App from scratch quickly. Also used Linode VPS service to host my App. You can choose that fit to you by reading this link.

[https://joshtronic.com/2018/08/19/vps-showdown- august-2018-d...](https://joshtronic.com/2018/08/19/vps-showdown- august-2018-digitalocean-linode-vultr/)

</comment><comment><USER8> I'd just write it in regular JavaScript. The time you spend learning x framework can be used to write the 5% of it's features you will actually use.

People invoke framework names like they are casting +100 intellect buffs on themselves but you just don't need them. Be a crazy person and make a webpage without any frameworks. Come to the dark side.

</comment><comment><USER9> Wes Bos has recently released Advanced React course. It's really good, and it walks you through building a store you're describing(minus messagig) with Node, React, and GraphQL.

If I had do keep it REALLY simple, I'd just make use Django or Node/Express with minimal javascript.

</comment><comment><USER10> you could use bubble.is or airtable with jotforms or something like that if you wanted do it with out code. Otherwise use anything. I would use what ever your most familiar with. or something you want to learn.</comment>


The Not So Easy Oculus Rift Constellation Teardown - <AUTHOR> https://www.ifixit.com/Teardown/Oculus+Rift+Constellation+Teardown/61128?utm_content=bufferc43a9&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer <comment><AUTHOR> Palmer says: "Come on, @iFixit, you can go further than that! :)"

[https://twitter.com/PalmerLuckey/status/715269888948854784](https://twitter.com/PalmerLuckey/status/715269888948854784)</comment>


Nutritional psychiatry is the future of mental health treatment - <AUTHOR> https://www.independent.co.uk/life-style/health-and-families/nutritional-psychiatry-mental-health-treatment-future-depression-anxiety-vitamins-a8236106.html <comment><USER2> Looks like yet another fad. Everyone already knows we should eat better, and everyone knows that it has positive impacts on our energy levels, mental capacity and so on. That they're attempting to couple it with mental health in a more disciplined way seems to intentionally ignore the core causes of things like depression, usually stem from life circumstances. This then looks like yet another attempt to treat the symptoms, not the cause... much like the drugs the field is better known for.</comment>


How start-ups are like stubble. - <AUTHOR> http://www.brendanpickering.com/?p=35

<comment><USER2> >>Your parents notice. "That sounds great honey, can we help?"

My parents frequently ask to help me grow my beard :)

I'm not a big fan of the beard analogy, I feel like there must be many more relevant ones (a garden, for instance)</comment>


Who's afraid of Solar PV? - <AUTHOR> http://theconversation.edu.au/whos-afraid-of-solar-pv-8987

<comment><USER3> (I live in South Australia, and I've previously worked on electricity energy trading systems here)

This is a pretty good article, but misses a little context that maybe helpful to non-Australian readers.

1) Power bills in Australia have been rising rapidly over the last couple of years. The reactionary response has been to blame the new Carbon Tax, but the real causes are much more complex[1].

2) In South Australia at least it gets really, really hot. For example in February 2009 we had 6 Consecutive days over 40 °C (104 °F) and a maximum of 45.7 °C (114.3 °F)[2], and then in November 2009 we had 6 Consecutive days over 38 °C (~100 °F)[3]

These peak temperatures occur earlier in the day than the typical power spike (12pm-4pm instead of 5pm-7pm), and caused huge power spikes (from air conditioner usage), and this often causes power companies to have to shut off power (they have a policy of doing rolling blackouts when they don't have enough capacity).

 _This_ peak demand is much, much higher than the average peak demand shown on the linked article, and these are the peaks the power companies invest to meet. If these peaks are reduced by solar panel usage (which they should be, since the come during the best time for solar production) then it should reduce the requirements for larger power stations.

[1] [http://www.heraldsun.com.au/opinion/power-play-between- feder...](http://www.heraldsun.com.au/opinion/power-play-between-federal-and- state-govenments-is-just-a-costly-farce/story-e6frfhqf-1226454453982)

[2] [http://en.wikipedia.org/wiki/Early_2009_southeastern_Austral...](http://en.wikipedia.org/wiki/Early_2009_southeastern_Australia_heat_wave#Adelaide.2C_South_Australia)

[3] [http://en.wikipedia.org/wiki/Late_2009_southeastern_Australi...](http://en.wikipedia.org/wiki/Late_2009_southeastern_Australia_heat_wave#Adelaide.2C_South_Australia)

<comment><USER4> As another South Australian I have to say this article makes me a little concerned about further price rises of mains power to make up for the shortfall. Although the long term benefits of Solar are amazing the short term situation for power infrastructure here could be a problem.

*Edit: I can see a flatter usage profile should probably mean cheaper power, but still I think there is some justifiable paranoia in worrying about power prices increasing in South Australia again.

<comment><USER5> Another South Australian. Power is just the next thing to be disrupted by technology changes.

I think we need smart meters and a more market driven approach to retail pricing. Peak electricity costs can go through the roof and people will respond by shifting their usage to alternatives or different times when there is less demand and prices are lower. Unfortunately there is no market transparency and we are paying for retail electricity without any real understanding of the underlying costs. We are paying ridiculously increasing amounts for electricity so we can build infrastructure to handle peaks that many would avoid if there were market pressures on consumers. Changes like rooftop PV and wind farms are only a part of the picture.

<comment><USER3> Unfortunately, a _more market driven approach to retail pricing_ is often very consumer-unfriendly because it means your bills will be very unpredictable.

Power prices vary dramatically during a day, and are often gamed by generators. Exposing retail customers to those swings directly is dangerous IMHO.

Having said that, there are many intermediate steps that help. For example, many power companies do partition prices by time of day, which makes a lot of sense[1].

In South Australia, ETSA is trialing "Direct load management", where they can shut off air conditioners for 30 minutes using smart meters when there is high demand, in return for much lower tariffs[2].

[1] [http://www.savepower.nsw.gov.au/Portals/0/docs/news/Media071...](http://www.savepower.nsw.gov.au/Portals/0/docs/news/Media07121202.pdf)

[2] [http://www.etsautilities.com.au/centric/our_network/demand_m...](http://www.etsautilities.com.au/centric/our_network/demand_management.jsp)

</comment></comment></comment></comment><comment><USER6> An interesting problem is highlighted on that graph: solar seems to be peaking around mid-day, but energy demand peaks at 5 - 7 pm when solar is pretty weak. The curves showing pre- and post-solar have almost identical peaks; solar hasn't yet successfully reduced peak load. (The California energy system posts a similar graph, with a similar shape: <http://www.caiso.com/Pages/TodaysOutlook.aspx> , made worse by the fact that the California wind farms apparently perform best in the morning and worst in the afternoon.)

The difficulty is that, unless storage gets dramatically better, we'll still need a huge conventional electricity base. And it will still burn a huge amount of fuel: most generators are not "instant-on" -- it can take weeks to spin up a nuclear plant, days to spin up a coal plant, and hours [1] to spin up a natural gas plant. Of course, there are some savings, but there is huge waste if you have to keep the whole conventional energy infrastructure spinning during the day just to fuel that 6 pm peak load.

Advice to entrepreneurs: finding a cost-effective way to store solar energy for 4 hours will be worth more than another 2% increase in efficiency. And inventing instant-on conventional-fuel plants will also make a huge difference in GHG emissions.

[1] - Page 8 of [http://www.euec.com/getattachment/euecjournal/Paper_3.pdf.as...](http://www.euec.com/getattachment/euecjournal/Paper_3.pdf.aspx) gives times between 1.2 hours for a warm start to 6 hours for a cold start.

<comment><USER7> We don't need any breakthroughs. Electric cars could handle the entire peak load with _today's_ technology, the right incentives, and capital flowing in the right place.

The peak load for South Australia was 1.8 gigawatts. There are 1,275,041 motor vehicles registered in SA.

If you could get all of them turned into electric cars plugged into the grid, it would be a 1.4 kW power drain per vehicle, which is less than the power drain of an ordinary kettle here (240V, 10A, ~2.4 kW).

The cheapest Tesla Model S has a 40kW hour battery meaning it could sustain that power drain for over 28 hours (a total of 51 GWh). It's got a 10kW charger standard, which is more than enough.

So if everyone had the cheapest Model S, kept it plugged in with the ability to send power back into the grid, and didn't mind that their car was sometimes down to 20% charge, you could power the _entire grid on solar alone_.

That's obviously a set of unrealistic assumptions, but it does indicate that it is feasible to solve the problem like this.

A more realistic set of assumptions:

      * 1/10 cars are turned into a Model S base model equivalent
      * Through financial incentives, the owners are willing to keep them plugged in
        during the day and using up to 20% of the battery to push back into the grid.
      * Cars need to shave 0.2 gigawatts off of peak load for four hours to bring
        it in line with the rest of the day

That's 800 MWh of power to satisfy peak demand, at 200 MW.

We have 127,504 vehicles, can can use up to 8kWh and 20kW each, giving:

1 GWh of capacity and 2.5 GW maximum power draw.

So it's just enough. Alternatively, 5% allowing 40% usage works, etc. The power draw is insignificant.

Owners can be _heavily_ compensated for pushing energy back into the grid. I won't run the numbers here for length and time reasons, but knocking out peak power usage is incredibly profitable. You're literally decomissioning a large percentage of power plants. It would be feasible to make all the energy your car uses free; likely the lithium ion battery packs, too.

The question is; would 5% of the driving population buy a $50k car if they no longer had to pay for fuel or battery packs? Financially that would probably put it closer to a BMW. I think it's feasible. What about in 10 years when the cars are $20-30k? Undeniably. In 20 years when you only need 2% of the population to be in the scheme due to battery increases and the cars cost $20k? No brainer.

The system would require a smarter grid (so you can plug your car in at work and have it all taken care of), 10 years of Moore's Law for batteries, etc.

But I think the numbers check out, and it means we could go crazy with solar (the explosion in solar must continue to charge these vehicles during the day). What's needed is the will to make it happen.

<comment><USER8> >So if everyone had the cheapest Model S, kept it plugged in with the ability to send power back into the grid, and didn't mind that their car was sometimes down to 20% charge, you could power the entire grid on solar alone.

The OP said cost effective, not pie-in-the-sky.

Even if you took out the battery packs and sold them separately, the cost would exceed 10 years worth of electricity supply.

>Owners can be heavily compensated for pushing energy back into the grid.

You're asking people who cannot afford the expensive technology to subsidise those who can. This is the exact reverse logic of most progressive taxation regimes.

>The question is; would 5% of the driving population buy a $50k car if they no longer had to pay for fuel or battery packs? Financially that would probably put it closer to a BMW.

So you want to subsidise the purchase of expensive cars to the point where it is financially a good deal. This magic money no doubt comes from other taxpayers.

And for what end? Just so you can have some type of boutique distributed power generation system?

>What's needed is the will to make it happen.

No, what's needed it pots of _other peoples money_.

I've spent the best part of 5 years trying to hose down the jetsons fantasies of people pushing ridiculous schemes like this as not only unworkable, but inequitable for forcing up a basic cost input of life - energy - <USER2> effectively vanity purposes of a small subset of the population. I usually cop a pile of flamebait and downvotes each time, but I do so because there seems to be a mass delusion going on, and this has become one of those things you can't say.

<comment><USER9> _So you want to subsidise the purchase of expensive cars to the point where it is financially a good deal. This magic money no doubt comes from other taxpayers._

The proposal isn't a subsidy. It's a straight market payment for an economic benefit - the difference in cost between off-peak and peak electricity is real, and this means that storing off-peak electricity and releasing it at peak is an activity that's worth money.

I'm not sure I agree that the economics work out exactly as described, though - if it would be worth paying for everyone's lithium car battery packs to do this, then it would be even more profitable to build a giant lithium storage battery in a central location.

<comment><USER7> Well, the benefit is that it combines the capital costs of owning a vehicle and storing energy. Many people will be purchasing a 160+ mile range car, but day to day use only 20-30 miles of that - an unused capacity large enough to power an average house for over _24 hours_.

This is in contrast to dedicated lithium ion batteries, which just don't work out economically. Otherwise you're right - the utilities would do it themselves at scale.

<comment><USER9> I can see how that's true if the car owner's continue to pay the capital costs of the battery - the part I'm taking issue with is where you say that it would likely that the car owners would have the capital cost of their battery packs paid for by taking part.

<comment><USER7> My back of the envelope calculations elsewhere in the thread were $1571/yr savings per car (10% overall reduction in grid cost); the Model S battery replacement insurance is $12k per car, which is a ~7.5 year payback (by which time, they claim the pack will be at 70% effectiveness).

Elsewhere someone corrected me that the peak grid usage in a year is actually significantly higher - the figures I was looking at were average daily figures - which may make grid savings quite a bit higher than 10%.

I don't think it necessarily is a clear financial win now; I'm just saying, if you squint just right it kind of makes sense now, and with the march of battery technology and PV, not to mention the inevitable adoption of EVs whether or not this scheme exists, it's going to look better and better from here on in.

<comment><USER9> No matter how you slice the numbers though, the fact remains that if it's economical to pay for someone's car-optimised battery pack, then it must be even _more_ economical to build a giant fixed battery.

(Since we know that the latter isn't economical, then that also indicates a problem with your numbers - I suspect that for one thing, insurance couldn't be economically provided at $12k/pack if all of those packs were being used for daily peaking storage).

<comment><USER7> I suspect you're right, and it may well mean it doesn't make outright financial sense today.

The key is probably that the $12k replacement program is an investment in Tesla; they're cash starved, and are willing to sell the batteries at a loss in return for the cash advance. Also, they're betting on the batteries being ~half price in 10 years.

From further reading, they'd be $30k - 2.5 times the price - to buy upfront today.

But that doesn't mean that the subsidy doesn't exist - if 1% of consumers would buy a Model S due to its other advantages, maybe 2-3% would if that purchase returned $1500 a year back to them.

So while you're right that it's not outright going to pay for itself in 2012, it probably halves the gap between a Model S and a BMW in the same class, and it's only going to get better as battery/PV technology improves.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> TL:DR; version: Utilities make most of their profit during 'peak' times (noon to 5pm) and SolarPV systems are reducing the requirement during those times, putting price pressure on electricity.

Here in California we had a similar challenge but with Watar. We periodically go through drought cycles and during the last big incentives were put in place to get people to use less water, rebates for toilets, reduced cost if you were 20% below your non-drought average, no watering during the day, drip irrigation, etc etc. Then the utility needs a rate increase because they aren't getting as much water usage. It is a hard sell though to tell people "You have to use 20% less but we're going to charge you the same amount"

So electricity, like water, is a blended cost where the scarcity unit is priced to cover the physical plant costs of delivering the unit. We wholesale adoption of Solar PV it will require power utilities to come up with a different formula to recover their costs. The end result is that it will shift the cost from business (who pay the biggest power bills during the day) to non-businesses.

[1] [http://sanfrancisco.cbslocal.com/2011/04/19/east-bay- custome...](http://sanfrancisco.cbslocal.com/2011/04/19/east-bay-customers- could-face-water-rate-hike/)

</comment><comment><USER11> The article doesn't focus much on spot pricing, which is really important in what is going to happen in the future. They showed a big drop in utility revenue during peak hours. This revenue used to be caused by high usage multiplied by really high prices. Now its high demand multiplied by moderate prices.

If solar adoption continues along the path described (which is inevitable at this point, considering steadily dropping solar PV equipment prices) then the midday spot price will continue to drop as well.

We'll find out some answers to the question, what do you do when the spot price of electricity approaches 0? Water desalination. Pump water uphill. Charge electric car batteries. Electrolysis to convert water to hydrogen, for later use in fuel cells or combustion engines.

We will certainly see some innovation in short term energy storage, since the price of electricity just before the late afternoon peak will be much lower.

</comment><comment><USER12> Related:

[http://reneweconomy.com.au/2012/why-generators-are- terrified...](http://reneweconomy.com.au/2012/why-generators-are-terrified-of- solar-44279)

[http://grist.org/article/2011-11-23-the-problem-with-the- ren...](http://grist.org/article/2011-11-23-the-problem-with-the-renewables- are-reaching-cost-parity-meme/)

[http://www.recycled-energy.com/newsroom/publication/the- batt...](http://www.recycled-energy.com/newsroom/publication/the-battle-over- centralization/)

<http://www.ilsr.org/solar-grid-parity-101/>

[http://www.greentechmedia.com/articles/read/how-will-the- cal...](http://www.greentechmedia.com/articles/read/how-will-the-california- system-operator-cope-with-33-renewables/)

</comment><comment><USER13> >> The unspoken fear of all utility managers is the "Death Spiral Scenario".

>> In this nightmare, a utility commits to build new equipment.

>> However, when electric rates are raised to pay for the new plant, the rate shock moves customers to cut their kWh use.

We have a "death spiral" in water prices in my local (southern California) water district.

There was a drought. People were asked to conserve. People conserved too much. Drought ended. Water use did not return. Water companies then need to raise prices to meet their distribution costs. Which lowers demand further.

It can only end ugly.

<comment><USER14> It only end ugly for the water companies. On the contrary, it ends beautifully for the customer.

<comment><USER13> Not really. The water company is raising prices on everyone because we're not using as much, but they still have same overhead. If the whole community hadn't conserved, we'd have gotten more water for the same or lower cost.

<comment><USER15> > _The water company is raising prices on everyone because we're not using as > much_ //

If they're like the UK water companies then they'll be moaning they're suffering but still be making vast profits.

It's like "we can't fix the pipe network without increasing water bills and we already put them up 10% this year; oh and we made 20% more profit this year so we must be serving our customers well ....".

</comment></comment></comment></comment><comment><USER16> A shift in energy production is good news, so now we can use our coal generation capacity to desalinate water. We get new sunlight every day, but the fresh water that we have is all we have.

<comment><USER17> "We get new sunlight every day, but the fresh water that we have is all we have." Actually, we get new fresh water every other day - it's called rain.

<comment><USER16> If we actually got new water every day it'd be arriving on meteorites.

Of the water present on the planet, 97% is salt water which is unusable for drinking or agriculture. Of the 3% freshwater, 68.7% is locked in glaciers. So, 1% of the water on the planet is liquid and most of that is underground. We don't really have a lot and things like fertilizers damage what we have.

It turns out that the amount of fresh water is relatively constant. If we want more in places that have less then we have to manufacture it from seawater, steal it from our neighbors by cloud-seeding, or rely on the weather cycle convert seawater into fresh water and then distribute it.

If you live in Dubai, you aren't waiting for it to rain. You're building desalination plants and processing seawater.

<comment><USER3> In South Australia we are building a wind and solar powered desalination plant too: <http://en.wikipedia.org/wiki/Port_Stanvac_Desalination_Plant>

</comment></comment></comment></comment><comment><USER18> Sounds like technology driven disruption to me.</comment>


Understanding Image Virality [pdf] - <AUTHOR> https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Deza_Understanding_Image_Virality_2015_CVPR_paper.pdf <comment><USER2> I'd be impressed if they could find an algorithm to determine the dankness of a meme.</comment>


CBTC: Communications-Based Train Control [video] - <AUTHOR> https://www.youtube.com/watch?v=Mjx3S3UjmnA

<comment><USER2> Great video. I'm amazed that so much of the system is still run on old, highly manual technology.

By comparison, the London Underground is all computerized, to the point where you can actually monitor the entire system in real time: [http://traintimes.org.uk/map/tube/](http://traintimes.org.uk/map/tube/). The Tube is much older than the NYC subway, too, although admittedly smaller (11 lines compared to NYC's 34, 270 stations vs 468; on the other hand, the annual ridership is about 70% of NYC's).</comment>


Court: Native American tribe can't be a "sovereign" shield during patent review - <AUTHOR> https://arstechnica.com/tech-policy/2018/07/court-native-american-tribe-cant-be-a-sovereign-shield-during-patent-review/ <comment><USER2> What would happen if more parties start proxying their unlawful or at least dubious behaviour through Native American tribes like this? Some tribes seem to be willing to sell their immunity for enough money.

I'm not from the U.S., what kind of actions could the U.S. government take?

<comment><USER3> This idea was one of the strands of the biggest, hugest conspiracy theory of the 90's variously known as the Inslaw scandal, The Octopus, etc., which pulled into itself events surrounding a murder involving an obscure (23 member, iirc) group of Native Americans, the Cabazon:

> Some believe Alvarez had discovered money-skimming by outsiders helping the > tiny Cabazon Band of Mission Indians manage its fledgling casino. Others > suspect, however, that he had stumbled onto plans for a top-secret weapons > deal on reservation land.

> Witnesses and court documents diverge considerably, however, on whether a > secret partnership between the Cabazon tribe and the private security firm > Wackenhut Corp. was a deal to provide security services, build a munitions > arsenal or sell weapons to the Nicaraguan Contras, a U.S.-backed rebel > group. [1]

Trouble is, as far as your question goes, various government entities including the Justice Dept. and the CIA, etc. were the ones taking advantage of the territorial immunity of the tribe concerned (in various of the conspiracy scenarios, that is, which I in no way endorse, though I still read up on it when possible)

If you're interested, for an overview of the wider ramifications, [2] & [3] are not-bad entry points.

1: [http://www.sandiegouniontribune.com/sdut-murder-case- dropped...](http://www.sandiegouniontribune.com/sdut-murder-case-dropped- in-1981-ca-tribal-slaying-2010jul01-story.html)

2: [https://www.muckrock.com/news/archives/2017/may/16/FBI- promi...](https://www.muckrock.com/news/archives/2017/may/16/FBI-promis- part-1/)

3: [https://www.muckrock.com/news/archives/2018/mar/15/danny- cas...](https://www.muckrock.com/news/archives/2018/mar/15/danny-casolaro- primer/)

<comment><USER4> > Trouble is, as far as your question goes, various government entities > including the Justice Dept. and the CIA, etc. were the ones taking advantage > of the territorial immunity of the tribe concerned.

I just had the scary thought of the US setting up Guantanamo-like prisons on reservations. If it was legally possible, they probably would have done it already.

<comment><USER5> The plenary power doctrine holds that US tribes aren't sovereign with regard to the US federal government, which is why they're controlled by laws instead of treaties. This is also why you can't get tribal protection against federal criminal charges.

But the flip side of that is that tribal sovereignty can't protect government actions; as the superior power both Constitutional rules and international treaties would remain in force.

(I can't work out what's up with the Inslaw case. Tribal territory definitely doesn't legalize CIA arms deals, so there isn't even a motive for a conspiracy like that. But even Wikipedia suggests that case could "drive a sane man to madness".)

</comment></comment></comment></comment><comment><USER6> I figured it wouldn't fly because of some of the earlier rulings with casinos.

Well, I cannot blame the tribe for trying to get in on some of that tech money. Its not like there has been much outreach in this era of diversity initives. The tech companies list Native Americans in the Other column. Their prefered universities to recruit from don't really recruit from the reservation schools, and those same universities give scholorships and positions slated for Native Americans to non-natives. I look at Apple's initiave to get community colleges programming in Swift and see no tribal community college. Google blocked a tribal community college on Google Voice (thanks AT&T for finding that one). One PC company, who has since changed their ways, refused to treat tribal community college as an educational instutions.

<comment><USER7> Native-Americans have a diverse background in the last hundred years, including military service, sports and film making.. generally not big on the Academic side, if that is possible to say.. the programs you allude to are fraught with difficulties, for all kinds of reasons.. not simple, and certainly not solved by urgent, well-meaning people..

<comment><USER8> This is an issue that I'm regularly amazed urgent, well-meaning people are not up in arms about. That lack of academic success is largely to blame on the Department of the Interior's ineptitude. Reservation schools are often so bad, parents send their children to _still_ existing Residential Schools, where Church protected groups make millions a year to set kids up in foster homes and enroll them in the local public school.

The entire situation is a disgusting.

<comment><USER6> I'm just amazed that the whole of America is not offended by the Department of the Interior's ineptitude. Our internet got cut for two weeks because they were required to be removed from the internet because of how sloppy their IT security was. Made for some problems with our distance learning classes. Even the colleges who paid for their internet from a private provider were affected. Hell, BIE still thinks a T1 is good enough for a community college.

North Dakota has open enrollment, so there are a lot of buses from other schools on the rez each morning (which I find good for the students). The tribes runs the local school not the BIE, and I think that is much more common these days. We don't have any of the old BIA schools, and the most common boarding school in our area is a world of difference from the old government crap (it is now run by Native Americans and does wonders for troubled children).

<comment><USER9> I Grew up near a rez in ND, my dad worked at the tribal college for as long as I can remember. Open enrollment does help. A sizable chunk of the local public school population lived on the rez. The tribal highschool looks more like a prison (not a single window on the outside) and teachers rarely last more than a couple years.

On the note of the sad state of diversity initiatives, I recall a particular quote from a local restaurant owner: "If their last name has a verb or an animal in it I throw it in the trash"

<comment><USER6> I think I know which one you mean (opened in 1985, it does have a sky light, one window behind the concrete towerish thing, and a door on the 3rd floor boiler room). Amazingly, its a "celebrated" design from Texas. The "how do we vent a gas leak" was hilarious, not to mention how fast strep throat traveled.

Oh yeah, that place was a horrible place since it was half public (high school), half BIA (elementary). I am told they have improved, but it was not great when I went there. They finally got rid of the BIA.

Yeah, some of the "border" towns have some really, flat out racist people. The "safety inspections" on the first of the month positioned to stop all cars coming off the rez were some serious BS. Talk about hitting people when they are already down.

</comment></comment></comment></comment></comment></comment><comment><USER10> The EFF has a good write-up on this also:

[https://www.eff.org/deeplinks/2018/07/federal-circuit- reject...](https://www.eff.org/deeplinks/2018/07/federal-circuit-rejects- pharmaceutical-companys-attempt-dodge-review-its-patents)

</comment><comment><USER11> Really trying to use any other sovereign in a patent case is just plain silly. It is US patents that are relevant for granting barring treaties. They are out of jurisdiction by definition - leaving only diplomatic consequences which are usually pretty mild. France can't stop the US from calling Californian white wine champagne.</comment>


10 things I wish I was serious about before starting a startup - <AUTHOR> http://www.swaroopch.com/blog/serious-before-startup/

<comment><USER2> Too late. Most of us already started. But we will take advise on how to grow profitable.</comment>


PCell technology demonstration at Columbia by Steve Perlman - <AUTHOR> http://www.youtube.com/watch?v=5bO0tjAdOIw

<comment><USER2> Steve Perlman is about to learn a really ugly fact about modern capitalism. Scarcity = $$$. I hope I'm wrong, but given how certain people are reacting...

<comment><AUTHOR> Perlman is going to learn that he hasn't actually created anything new and the telecom industry will prefer to use the standards that are implementing similar solutions.</comment></comment>


Network Coding in P2P Networks - <AUTHOR> https://adlrocha.substack.com/p/adlrocha-network-coding-in-p2p-networks <comment><USER2> I did some research on network coding a few years ago (see [1] if interested), and I was very excited by the potential until I read the paper by Li, Li, and Lau that proves the maximum gain in throughput we can expect in undirected networks is 2x. [https://iqua.ece.toronto.edu/papers/zp- transit.pdf](https://iqua.ece.toronto.edu/papers/zp-transit.pdf) This makes me think there is no point in introducing all this complexity... just stick to packet networks and routing, and you're at most 2x less efficient than then optimal netwrok coding scheme...

[1] A short literature review paper on network coding dating back to 2009: [https://minireference.com/static/tmp/network_coding_review.p...](https://minireference.com/static/tmp/network_coding_review.pdf)

<comment><USER3> When you've made all other optimizations and your performance is bottlenecked by that last bit of efficiency, then perhaps it is interesting! 2x is nothing to sneeze at.

If your application is costing users two million dollars a year in bandwidth ... then halving it would potentially save a million dollars a year-- and could well be worth an impressive engineering effort.

<comment><USER4> Real world incremental complexity is very high. To say nothing about the lack of an ecosystem.

I fondly recall interviewing at one of the faang companies years ago, when they ran out of things to ask me, and the guy turned to me and said: "Let's talk about network coding". My response, "Why would anyone want to do that?", probably didn't help land me the job.

</comment></comment></comment><comment><USER3> Network coding is very interesting but the maximum gains from the network coding itself are fairly limited... a small constant factor. In attempting to apply network coding I've found that vulnerability to (maliciously) corrupt data significantly limits the utility.

If this is an area that interests you, you might also be interested in efficient set reconciliation-- where a party can communicate to another party a set using only exactly the size of the set-difference to the other parties set without having any idea of what the other parties set has.

This can result in fairly large bandwidth savings in P2P gossip applications without reducing robustness-- and could be combined with network coding in places where network coding is applicable.

I worked on an efficient implementation of setrecon [https://github.com/sipa/minisketch/](https://github.com/sipa/minisketch/) and an application of it to Bitcoin ( [https://arxiv.org/abs/1905.10518](https://arxiv.org/abs/1905.10518) ).

</comment><comment><USER5> Seems pretty weird to waste substantial (1000x) CPU and disk I/O to figure out what you are missing, and if you have enough pieces.

Why not just use a reed-solomon based approach and turn 1000 pieces into 1100 and then you know you can recover the file with any 1000 of those 1100 pieces? That enables you to actively search for the rare pieces to improve the health of the network and as a side benefit you use 1000x less CPU and I/O.

Adding significant complexity so that sometimes you get a missing piece one time step sooner seems like a huge waste.

<comment><USER6> Out of curiosity I compress a 3 hour lecture one time to 70 MB and was totally surprised by how watchable it still was. It got me wondering, out of that 10 GB ultra uber HD XXL video there is about 50 MB that you must have, 100-200 MB of data that would be really useful but the rest quickly drops to "nice to have but not at all necessary". (my numbers are mostly imaginary)

I strongly think that packaging and delivery methods should be specifically designed for the goods being shipped. (Sand is not glass)

Poor TV and radio signals use to gradually degrade the picture and audio. Even with partial reception you could still figure out useful things like if your city was going to get bombed.

Today on p2p you get a multi GB blob of data with which you cant do anything until you got all of it. Streaming will only give you the beginning which could be worse than not watching something.

But if we take say 100 fold and do 70 GB of reads for a 700 MB video... (I guess a few more numbers here and below)

From the 80's to 90's memory went up from 16 kb to 1 MB. From 90's - 2000 from 1 MB to 512 MB. From 2000 to 2010: 512 MB to 48 GB. 2020, 128 GB? (rather disappointing compared to the previous jump) Is 512 GB a reasonable guess for 2030? I don't think we will see the 32 000 fold increase from 1980-2000 but 2 or 4 times seems reasonable? 70 GB of reads wouldn't be such a big deal.

Monitors will grow of course but houses are getting smaller. We can do all 4 walls, then the ceiling and even the floor, increase frame rate and resolution but it has to stop eventually. (haha)

<comment><USER7> Progressive images - see demo images a bit down the page here: [https://blog.codinghorror.com/progressive-image- rendering/](https://blog.codinghorror.com/progressive-image-rendering/)

It's pretty cool on the rare occasion when you stumble across a _huge_ image on the web (think astronomy); first a low res version pops up and then you can see the top-to-bottom loading as it gets gradually replaced by higher resolution versions.

</comment></comment></comment><comment><USER6> I never knew! What fascinating stuff. The idea that anyone who has the entire file could generate a block that fits as the final piece seems totally magical. You could do a final even block or odd, group every 3rd, every 4th, first half second half... and then the "last" block just magically fits... for everyone... mind blown!</comment>


If Crowdfunding is the New Day Trading, Look Out - <AUTHOR> http://blogs.hbr.org/2013/09/if-crowdfunding-is-the-new-day-trading-look-out/ <comment><USER2> Crowdfunded equity investments should be treated as gambling, don't do it unless you're willing to lose money.

Over half of traditional angel investments don't return the investors money and 90% of the returns come from 10% of the companies. These are traditional angels who are well connected, do due diligence, etc.

With crowd-funding from the start you face an adverse selection problem, a lot of the "best" looking startups are going to take money from well respected angels and are going to fill up their seed rounds long before they need to turn to crowd-funding. So with crowd-funding you're left trying to find the good companies (which there are significantly less) which aren't obviously good from a surface analysis.

So rather than 10% of the companies returning 90% of the returns with crowd- funding it might be closer to 5% or 1% of companies (only time will tell us the exact numbers).

<comment><USER3> Exactly, and it is really hard for me to see a scenario in which this whole thing ends well. It's basically inviting a huge pool of suckers to the table, gambling on terms set by the dealers, with payouts engineered and controlled by the dealers.

Just about the only way for this to work, without causing a ridiculous speculative bubble, would be to force increased transparency of the financial documents of private companies (thus blurring the lines between private and public, which is opening a can of worms in its own right).

Success in angel investing is predicated on two things: 1) access to the best deals on the best terms, and 2) enough capital to absorb losses in search of outsized hits. Retail investors have neither of these things going for them, so what they're doing amounts to little more than playing roulette.

<comment><USER2> I still think it should be legal. After all you can take $10k and spend it at a Casino, there's no reason you shouldn't be able to gamble it on a company.

Although there are other reasons for investing other than financial returns, for example you believe in the mission of the company or you want to be a customer of the business.

<comment><USER3> I'm not arguing against its legality; I'm arguing that it's dangerous as written. It needs more enforcement of transparency on the side of the companies. The difference between gambling and "investing" of this sort is that at least gambling is 100% transparent. You know exactly what the odds are at the roulette table. And most people who gamble (barring those with addictions, of course) _realize_ they're gambling; most of the new people who will "invest" probably won't believe they're gambling.

 _" Although there are other reasons for investing other than financial returns, for example you believe in the mission of the company or you want to be a customer of the business."_

Sure, but that's what services like Kickstarter are for. Let's not kid ourselves into thinking that most Mom & Pop retail investors are going to be day-investing in small businesses out of high-minded principle. They're going to see dollar signs and dream of getting rich quickly. And there will be plenty of people ready and waiting to sell them those dreams.

Maybe it'll take a bubble or two to get people to realize the dangers involved, and eventually, it'll normalize into sort of what day trading has become. But there will be pain before then.

</comment></comment></comment></comment><comment><USER4> The nice thing about Crowdfunding is that, like traditional trading, motivations vary.

For example, I recently backed the Infrared Human Vision study on Microryza [1] simply because it sounded like cool research. I backed the Ubuntu Edge even though I knew it would most likely fail to reach its funding goals because I wanted to send a message that there's a market for a "pocket to desktop" convergence device. I sent a few dollars to Oculus on Kickstarter because I wanted to see that become reality.

I'm not naive... I know that as this gets bigger there is going to be room for technical trading, abuse, and an emergence of some myopic thinking similar to the "quarter-by-quarter" tunnel vision that plagues public companies on Wall Street. But the thing I like the most about crowdfunding is that it takes this long-standing idea of "principled investing" and really lets you put your money where your mouth is.

[1] [https://www.microryza.com/projects/can-we-biologically- exten...](https://www.microryza.com/projects/can-we-biologically-extend-the- range-of-human-vision-into-the-near-infrared)

<comment><USER5> From the article: "To be clear, Malik isn't talking about Kickstarter where funders make a donation that acts like a pre-order. He's talking about the public buying stock in private companies, something that may soon be legal thanks to the JOBS Act..."

<comment><USER6> I think breckinloggins's point was that people may be willing to make _equity_ investments for reasons other than profit.

(Now I'm wondering whether startups will be allowed to offer rewards in exchange for funding, leading to more "X is not a store" backlash.)

</comment></comment></comment><comment><USER7> I see an interesting parallel of "crowdfunding" to MOOCs. The original (private capital / high-prestige colleges - or their synthesis in Harvard, a hedge fund with a university attached for tax purposes) is a clubby self- dealing arrangement designed to preserve a certain level of oligopoly profits. The cheap substitute gets held up as a way for the average person to buy into that system, when it's really a way to dump an inferior product on the public with the same brand and none of the private advantages that make it worthwhile in the first place.

<comment><USER2> That's a straw man arguments, I don't think anyones claiming that doing courses on Coursera is the equivalent to a Stanford degree. Certainly the universities aren't.

<comment><USER8> But they are charging plenty of money for University of Phoenix Online courses.

The statistics, that a disproportionate number of graduates from Phoenix and places like that are unable to make student loan payments, tell you that they are not returning on investment for those students.

If you are poor, and study very hard, it is possible to win a scholarship at a reputable college, and achieve social mobility that way. Pretending that people who have not worked hard enough to win a scholarship can become middle class by taking out a loan is a dupe. They would do better to join the army for that purpose.

<comment><USER2> MOOC != University of Phoenix

<comment><USER9> MOOC == Hey, those guys might be on to something?

</comment></comment></comment></comment></comment><comment><USER10> I disagree that just because most people who day trade lose money, that wider access to the markets is a bad idea.

I'd rather have the option to enter a market when the odds are against me than not have that option at all.

<comment><USER11> Well, I'm not sure if the author is saying that the access period is a bad idea, or that availing oneself of it is a bad idea.

I would certainly agree with the latter. As for the former, we already have a lot of ways for people to throw their money away, so why not one more. Literal actual gambling is legal in many places, perhaps even most, so I don't think I could argue against this form of gambling being available too.

</comment></comment><comment><USER12> This disdain for "common" investment is over the top.

A much bigger concern than the potential for poor decisions by individuals is the creation of financial instruments that are, by government fiat, only available to an elite group of institutional investors and the otherwise- wealthy.

<comment><USER13> > This disdain for "common" investment is over the top.

Completely agree. After pointing out that the institutional investors are clueless and lose to the market, he then says that only the little guys should stay out of the startup space. Honestly, many of the reliable indicators for Venture Capital are available to the public (eg past success). No reason to keep the little guy out.

<comment><USER13> Expanding a bit, HBR argues:

> pouring more capital into VC has historically led to lower returns

Why this is unexpected is beyond me. More money should mean more competition. He's essentially arguing that we should allow VCs to keep exclusivity on the market because historically their returns have gone down as exclusivity has decreased.

There may be argument that the startup market may have more scams than the public markets and so on, but the author at HBR is not arguing that. He's arguing that a natural function of opening up a market is a bad thing because it provides less returns to the existing players.

</comment></comment></comment><comment><USER14> Non-acredited investors are only allowed to invest 5% of their income. If grandma aint't got no income she ain't losin' alot of money. As long as all transaction go through marketlaces and the marketplaces are forced to validate the limits. Then no one is losing their life savings. Personally, The better option is to force the state lottos to inact the 5% limit and worry about the crowdfunding.

</comment><comment><USER15> If you don't know who the sucker at the table is, it's you.

<comment><USER16> Exactly. Crowd funding is an ideal source of capital for con artists.

"We are a team of Nigerian Bankers, and with your donations we will raise enough money to give you double what you contribute!"

"We are a team of scientists who have discovered this one wierd trick that the medical community doesn't want you know about. If we meet out goals, you can receive the new wonder pill that adds 3 inches to your penis!"

The future's gonna be interesting...

<comment><USER17> It'll be much like the past.

</comment></comment></comment><comment><USER18> Wow, this editorial exactly parallels my concern. Basically people chasing outsized returns getting fleeced by the bottom of the barrel type pump-n-dump types.

Here is my thesis, there exists a significant number of people who attempt to exploit people's desire four outsized returns using illegal "pump and dump" schemes where they pump up the value of a penny stock and them dump their shares to the unwitting victims. This is illegal and they get hunted down and sometimes caught. Now we'll have a "perfectly legal"[1] way for these same folks to create the illusion of an outsized opportunity. And that will create dark versions of wefunder type companies which are focused not on enabling the next generation of entrepreneurs but instead in fleecing the next generation of technology "investors".

[1] "Perfectly Legal" \- code for a way to take money from someone where they cannot get it back, and yet had they known all of the facts would never have handed it over in the first place.

</comment><comment><USER19> I'm a little surprised at HBS, this is not a very well thought out commentary.

People like to say startup investments are "like betting" and it's true to a certain degree -- it's very risky and odds are against you. But what separates it from more popular forms of betting (casinos, lotteries) is...

You have to hold on to your investment for many years.

The timeframe on returns is a huge difference between day trading and early stage investments. Day trading was a very close approximation of off-track betting. It had similar psychological dynamics driving it. You win some, you lose some, you keep coming back for more, you think you can beat the odds tomorrow if you just get some money together.

Those addictive dynamics don't play as well over a timeframe of 5-10 years. It's like buying a lottery ticket for 2020. Not as addictive.

I'm not saying there are no downsides to unsophisticated investing. Just that day trading is a poor comparison.

</comment><comment><USER20> I'm worried that crowdfunding will lead to serious malinvestment, with large numbers of naive investors throwing money at a thousand Fakeblocks and Colors.

</comment><comment><USER21> What they are describing doesn't sound like crowd-funding at all. It looks more like advertized private equity with dramatically lowered quality standards.

It does not sound as if "accredited investors" would necessarily include a lot of people, especially when people know that the risk is worse than in the usual stocks.</comment>