The Novelty Curve - <AUTHOR> http://darmano.typepad.com/logic_emotion/2008/02/the-novelty-cur.html

<comment><USER2> _"... Something tells me that this visual shouldn't require additional explanation. :-) ..."_

The graph looks screwy. None of this curvy point to point business. It would look more like a spike followed by a

\- a) a plateau if you can't scale

\- b) a dip when users exercise the back button

\- b) an upward curve if your site changes with user demands</comment>


The Mayflower Autonomous Ship Project - <AUTHOR> https://newsroom.ibm.com/then-and-now <comment><USER2> Wave Gliders [1] have been doing that for years.

The tiny Wave Gliders are quite useful. They're small, smaller than a surfboard. They're considered "marine debris" for vessel safety purposes. Electronics is so small today that they carry compute power, GPS, cameras, and Iridium for communications, plus other small oceanographic instruments as desired. They're tough enough to survive hurricanes. The control center has AIS info and steers them away from vessel traffic. Propulsion is mostly passive, using wave action to propel the glider forward. They're slow, but make steady progress.

It's a quiet, useful and profitable technology.

[1] [https://www.liquid-robotics.com/wave- glider/overview/](https://www.liquid-robotics.com/wave-glider/overview/)

<comment><USER3> I wonder how the Wave Glider deals with fouling of the umbilical from seaweed and fishing gear?

</comment></comment><comment><USER4> If you like autonomous ships, check out Saildrone

> Saildrone designs and manufactures wind and solar-powered autonomous surface > vehicles called saildrones, which make cost-effective ocean data collection > possible at scale. We are building the world's largest high-resolution ocean > datasets, working with governments and private companies around the globe.

[https://climatescape.org/organizations/saildrone](https://climatescape.org/organizations/saildrone)

</comment><comment><USER5> I feel Mary Celeste would have been a better name.

[https://en.wikipedia.org/wiki/Mary_Celeste](https://en.wikipedia.org/wiki/Mary_Celeste)

<comment><USER6> Yes! Not only does Mary Celeste have a way cooler name and backstory than the Mayflower, it also carries fewer weird colonial/imperialist connotations.

</comment></comment><comment><USER7> Looks like they have found their new 'Watson' for the Ocean. At least this one probably does something useful. Everything IBM does using buzzwords seems to be to keep their consultancy business thriving and staying in the news.

</comment><comment><USER8> Sorry for the tangent, but I am surprised to see a dedicated section to Blockchain on their site. Its right alongside Cloud, IoT and Security. Wonder what they do with it.

EDIT: I need to clarify a bit. I clicked on the link yes. However there are pages of it, wasn't sure if its aspirational or they actually make money off it

<comment><USER9> It was all the rage a few years back; everyone had an aspirational blockchain project. It being IBM, they'll probably vaguely keep at it long after it's passé.

</comment></comment><comment><USER10> I know this is a dumb thing to complain about, but why is their CGI so bad?

</comment><comment><USER11> An unfortunate name.

</comment><comment><USER12> Hope it can follow Col Regs.

<comment><USER13> For those who aren't avid boaters: [https://en.m.wikipedia.org/wiki/International_Regulations_fo...](https://en.m.wikipedia.org/wiki/International_Regulations_for_Preventing_Collisions_at_Sea)

Looking at that tiny wing sail, it's gonna be sloowwww. Maybe that'll help?

<comment><USER14> 20 knots is not all that slow. According to google average container ship speed is about 24 knots. Even the navy destroyer I used to sail on topped out at 30 knots using its gas turbines, which were only to be used for "serious purposes".

<comment><USER15> Very rough scale with respect to sailing vessels around 30-55 foot length... multihulls (fast = 20 knots, common = 15 knots, many = 10 knots), monohulls (fast = 12 knots, common = 8-10 knots, many = 6-8 knots). For an ocean-going trimaran, their 10 knot speed is not fast. It is quite likely the vessel's apparently near-fixed "wingsail" represents a very conservatively-rigged configuration that is biased for stability instead of speed.

<comment><USER16> Thinking about it in the context of the colregs, 10kts is above the hull speed of a 40ft sailing yacht, and is probably plenty fast enough to sink that yacht in a collision.</comment></comment></comment></comment></comment>


IBAnimatable – Design App Store Ready Apps in Interface Builder - <AUTHOR> https://github.com/JakeLin/IBAnimatable <comment><USER2> this looks really cool. I'm tempted to start a side project just to try this out

<comment><AUTHOR> Please let me know your progress, thanks.</comment></comment>


Installing Joomla 1.5.6 On A Lighttpd Web Server (Debian Etch) - <AUTHOR> http://www.howtoforge.com/installing-joomla-1.5.6-on-a-lighttpd-webserver-debian-etch <comment><USER2> Up-voted for the mere fact that HowtoForge is an absolutely awesome resource and Falko Timme is some kind of super-robot who posts tutorials for every imaginable sysadmin task (and helps in the forums, too!)

Thanks for years of great stuff, Falko.</comment>


Parliament Shooting - <AUTHOR> http://www.bbc.com/news/live/uk-39355505 <comment><USER2> I can't seem to reach people in the area. Has the mobile network been taken down or is it just overloaded?

Apparently, threads on the topic are being flagged down...

<comment><USER3> yea i submitted a related article. never went through...</comment></comment>


This Weekend in Ghent, BE: Newline - <AUTHOR> https://0x20.be/landing/newline.php <comment><AUTHOR> If you happen to be in Belgium this weekend or know someone who is: Newline is the annual conference by the local computer club. This year's programme is pretty stellar ( [https://hackerspace.gent/newline](https://hackerspace.gent/newline) ). Everyone is welcome, the entire event is free.

edit: fixed link</comment>


The potato-salad guy should keep every penny - <AUTHOR> http://www.newyorker.com/online/blogs/culture/2014/07/laughing-all-the-way-to-the-potato-salad.html

<comment><USER2> I saw earlier today that the New Yorker had written about the potato salad guy, and I skipped it because I wrongly assumed it was going to be more whining about the state of society and cynicism of our new tech-savvy overlords (cynicism which I kind of share, to be honest)...It's nice that they took a non-do-gooder stance.

I chipped in $3. That's about how much I'd tip a good street performer in New York, and this Kickstarter made me laugh. The comedic timing, as much timing as you can convey in a plain written description, was just right...and I'll happily admit, I'm very curious to see how the hell he's going to deliver a "bite of potato salad" (and a photo) to me...I'm also curious if this will be the first Kickstarter project I've backed that has actually delivered its reward on time. Good for the potato guy for making the kind of joke lots of people may have had in their heads, but actually executing on it.

The most annoying protests, IMO, are from the people who ostensibly represent a good cause, like a homeless shelter or cancer research, complaining that a potato salad is getting $45K while they can't even raise $4K. Well, what does that say about your cause? Hopefully, it says _nothing_ about your cause, but maybe it says something about your process. If some random guy from Ohio can raise $45K for something that _no one should care about_ , I think the proper attitude should be, _damn, just think about how much money I could raise for something people DO care about_ and then, get started on figuring out how to go about it.

The jealousy of goodhearted people towards this goodhearted potato fan is a bit unbecoming. And sad, because if they're unhappy about a guy making a shitload of money while being completely honest about what he was doing, how angry will these people be when they learn about the many, _many_ people who raise a shitload of money while being _dishonest_?

</comment><comment><USER3> Related: [http://www.ted.com/talks/amanda_palmer_the_art_of_asking](http://www.ted.com/talks/amanda_palmer_the_art_of_asking)

The people's responses to the comment are telling. The Kickstarter said he'd make a potato salad. He probably has. He asked, and people gave, why do people rage about this? It just so happened that lots of people thought it was funny / wanted to give money / whatever. Who cares? The proverbial _you_ could have thought of it too.

I don't understand the stink people have over this :/

<comment><USER4> > The proverbial you could have thought of it too. It's here (imho) that the > problem lies. People get jealous around money, especially when (from their > perspective) it's unearned, and this often leads to irrational anger, which > is the bread and butter of internet comments.

<comment><USER5> By what measure is it unearned? The guy said he would make potato salad, the market decided that was worth $70k. Assuming he follows through and makes the salad, he's earned his money.</comment></comment></comment>


how to optimized images to make it sharp on ipad3? - <AUTHOR> http://www.mobify.com/blog/ipad-3-and-retina-screen-what-it-means-for-your-mobile-commerce-site/

<comment><AUTHOR> The simplest answer is just to double the dimensions of the image then use CSS to make it the right size.

So if you want to create an HD image at 200 x 200 on a Retina display, create the image at 400 x 400 then use CSS to control the HD image down to 200 x 200: img { width:200px; height:200px; }.

The HD images are still 72 dpi resolution.

The tricky part is then preventing non-HD devices from unnecessarily downloading the larger images.</comment>


Ask HN: What Is Your Favorite Lesswrong Diaspora Blog - <AUTHOR> What is your favorite individual, public intellectual or blogger who had been a lesswrong fan? (opinion ahead: I think LW splitting has been the best for rationality community) <comment><AUTHOR> Mine is SSC and ribbon farm</comment>


Ask HN: Are ride-sharing pay rates sustainable? - <AUTHOR> I recently did an UberPool.  The passenger and I each paid $6 for the entire ride.<p>Between pickups and dropoffs, it lasted 45 minutes.  The driver explained it would take him 15 minutes to return to an area with customer demand.  After taking into account Uber's take, plus the other costs of being an Uber contractor/driver, how long can we expect these rates to last before either a driver revolt or significant price increase to customers? <comment><USER2> Mechanically, the answer is until what ever is pouring money into the system runs out of money.

But figuring out all the money flows can be quite complex.

So from the driver's perspective there will always be people available for whom "some money" is better than "no money." That is the part of the labor pool who have multiple jobs which each contribute some part of their overall income. Uber is good for those people because the hours are _very_ flexible and so they can be "fit in" around a lot of other more scheduled economic activity.

So an interesting question is this, what is the supply of these laborers and is there any other activity that can get them the more income with the same flexibility? Is that the critical resource here? "remainder" laborers?

If that is the case, then that shortage will drive the price up in order to meet demand for Uber rides. And if the price for Uber rides goes up, it will mean that some Uber customers for whom it was marginally useful at the previous lower price will drop out.

And that summarizes to there will neither be a driver revolt nor a significant price increase. There may be fewer drivers which leads to a gradual price increase or there may be higher driver turnover and stable prices. Time will tell though, regulation could step in that changes the balance significantly in favor of one party or another.

<comment><USER3> Mechanically, it will last until all the first generation Uber driver cars give out. I very much doubt many drivers are properly accounting for wear and tear on their vehicles and still make minimum wage.

<comment><USER2> I would push back on that thought a bit. Consider that in our existing history are door to door sales people who would use their own car to sell goods on a commission basis, the canonical example of that is the "Fuller Brush Salesperson".

Also, assuming these people file tax returns, if they are independent contractors then they are writing off their car mileage at 55 cents a mile against any income they make. Hard to know for sure without talking to actual drivers though.

<comment><USER4> There is a big difference between driving while selling and selling your driving.

</comment></comment></comment></comment><comment><USER5> Remember that Uber subverts the traditional employee model. Drivers are contractors, responsible for their own hourly wage. Sometimes they will earn below the minimum wage, and at other times, well above it.

In the areas where Uber is prominent, it's hard for low-skilled workers to find any employment at all. Consequently, some of them turn to Uber. This means that no matter how bad the wage, Uber has a ready supply of drivers.

It's extremely important for Uber to make Uber a cost-effective alternative to driving or public transit. The drivers are less important to Uber -- both in the short and in the long run (automated cars), the drivers are the expendable part of the equation.

(However, do note that under certain circumstances, Uber subsidizes the drivers: sometimes they are paid more than the passenger is charged.)

<comment><USER6> _The drivers are less important to Uber -- both in the short and in the long run (automated cars), the drivers are the expendable part of the equation._

Oh bullshit. The drivers are the face of the company. As in clean cars, professionally dressed, courteous. The only thing separating uber from a DC taxi is the app and the driver. I've had many good rides with uber drives who all said it paid more than their last gig. But already they were shopping around for other ride sharing apps that paid even more. Uber loses its drivers people use lyft.

And what's with the automated cars crap? It seems everyone is an expert and has a timeline on when they are going to rollout. 3 years, years, etc. is it an eventuality? Yes. But I wouldn't make business plans around it just yet.

<comment><USER5> > Clean cars, professionally dressed, courteous. The only thing separating > uber from a DC taxi is the app and the driver.

No. People will _always_ trade personal comfort for convenience and price. Look at airlines. Back in the 70s, a flight was a great pleasure. Then the airlines were deregulated. As it turned out, people didn't care nearly as much about their legroom and on-board meals as they cared about price. That's why nowadays, flying coach is generally uncomfortable, but also cheap.

And in that regard, Uber has a huge advantage: UberX is _much_ cheaper than a taxi. In Chicago, a taxi+tip costs about 3x more than UberX (with Pool). It's similar in many other cities. That's the big advantage that UberX has on taxis.

Indeed, Uber's pricing is getting competitive with public transit -- in Chicago, UberX (Pool) is only 2x or 3x more expensive than public transit for a given distance, but much more comfortable (regardless of the niceness of the car) and much faster than public transit.

For UberX, the drivers and cars could be awful. It wouldn't matter. What matters is that UberX occupies a sweet spot in terms of pricing and speed that neither taxis nor public transit can match. Consumers value pricing and speed far above comfort.

<comment><USER6> I don't deal in absolutes and most people don't. People will chose the best option based on a range of factors. Some people will put up with cheap uncomfortable flights for a few hours, <4\. They are more concerned with getting to their destination. Some people won't fly with Easyjet or Ryanair at all. You'll notice there are no long haul low-cost airlines. The cheap airlines would have to supply things like food and water. The sort of things that cost money.

I've never taken UberX in Chicago. I have used it in London, Paris, NYC and they were no cheaper than a taxi. In fact they were a little more in Paris. But I knew that going in because I chose comfort over price (and speed, UberX isn't allowed to use the bus lanes, yet).

 _Consumers value pricing and speed far above comfort._

I'm sorry, you keep coming back to this conclusion but none of the evidence you've presented holds up. Except in rare cases, Uber is more expensive than a taxi yet people chose them anyway. So people value comfort just as much as price. In fact when people talk about why they use Uber price is rarely the factor. typically they've had bad experiences with taxis.

<comment><USER5> > I have used it in London, Paris, NYC and they were no cheaper than a taxi.

I used Uber extensively in London this January. I actually found that for long distances, taxis were about 3-4x the price of an UberX (with Pool). London taxis are very expensive.

In NYC, Uber had a long-standing $5-for-anywhere-in-Manhattan offer. I lived there for a few months, and found Uber and Lyft to generally be significantly cheaper than taxi+tip, even disregarding promotions.

Finally, there was a good article on this topic by BusinessInsider at [http://www.businessinsider.com/uber-vs-taxi-pricing-by- city-...](http://www.businessinsider.com/uber-vs-taxi-pricing-by- city-2014-10). It indicates that Uber is generally cheaper than taxi, but that's as of 2014. With UberPool, prices have dropped significantly further yet.

I conclude that UberX/UberPool is generally significantly cheaper than a taxi, and more convenient except in areas where you can easily hail a cab off the street. Looking at the history of consumer goods and transportation, I think that those factors significantly trump comfort.

</comment></comment></comment></comment></comment><comment><USER7> When I talked to drivers, I came to the conclusion that the drivers are paid more than the passengers are charged, which makes sense. Uber makes huge losses every year.

The hope is that this can last long enough before the drivers can all be fired and replaced with automated cars, or at least that the system will get better (i.e. better at stuffing cars full of people, more demand by training people to order the carpools, etc.).

<comment><USER8> "When I talked to drivers, I came to the conclusion that the drivers are paid more than the passengers are charged"

That isn't true in general for an Uber driver. What you might have been seeing is that when Uber goes into a new city they guarantee a minimum amount of pay per hour for a driver during certain times if they are logged into the app accepting fares. It is only a temporary measure until they get established in a local market. Once they stop subsidizing earnings for drivers drop precipitously.

<comment><USER9> That minimum exists in SF, which is not a new market. Source: my last Uber driver.

</comment></comment></comment><comment><USER10> We can't keep expecting "employers" to solve problems like this because fewer and fewer people will have employers. Coase's theory of the firm explains employment as a way to reduce transaction costs—it's hard to find the right person for each task and negotiate a deal with them each time, so just pay people to obey orders and use them to get things done instead.

The technology we build is driving transaction costs closer and closer to zero. Fewer tasks will need firms to coordinate them—they'll be coordinated by the market instead. More people will sell their work directly to the people who want it.

The basic income approach can solve the living wage problem without relying on the employment relationship. Any energy we spend trying to get Uber to solve the problem will be wasted when we build ride hailing marketplaces that don't require an Uber to connect the rider and the driver.

</comment><comment><USER11> You used to have a much smaller amount of cab drivers making a pretty good living due to unions blocking competition and stifling the entire industry.

Now, you have many more people making a smaller amount of money. This is essentially what happens in any competitive industry.

</comment><comment><USER12> Even if they aren't, Uber will likely be able to survive the burn rate until the cars drive themselves. Uber just placed a "long term" order for 100k S-class cars.</comment>


How DigiCash blew everything - <AUTHOR> https://cryptome.org/jya/digicrash.htm <comment><USER2> Date: 1999

Attitudes toward failing startups seem to have changed a bit since then...and toward founders' behavior under stress.

Probably easier to raise big capital today too.</comment>


Extracting campaign finance data from gnarly PDFs using deep learning - <AUTHOR> http://jonathanstray.com/extracting-campaign-finance-data-from-gnarly-pdfs-using-deep-learning <comment><USER2> My personal bet is that you would have probably gotten better results by simply feeding your data into xgboost or lightgbm.

This current fad of focusing on neural networks just seems sorta silly when we have simpler and (often) better performing models. Just look at what sorts of models win kaggle competitions.

<comment><USER3> Maybe but I think 90% of the work is wrangling the data and understanding the problem, it wouldn't be that hard to substitute gradient boosting for the NN

</comment></comment><comment><USER4> Deep learning (and indeed any kind of machine learning) is really not necessary here. I wrote a very simple baseline that estimates the total gross amount as the biggest numerical value present (favouring ones that begin with a dollar sign and/or occur more than once). This achieves 93% accuracy (code is here [https://gist.github.com/rlmacsween/166b7c1c1b0c5f466a0fe9b46...](https://gist.github.com/rlmacsween/166b7c1c1b0c5f466a0fe9b467d61537)) in comparison to the 90% for the deep learning model.

<comment><USER5> Nice! An excellent baseline, something to beat.

</comment></comment><comment><USER6> This is a problem that fascinates me as well. How humans effortlessly look at a document with tables and columns Etc. and effortlessly extract all of the 'factoid' bits from the document and often their relationships. Back at IBM there had been some work on extracting tables from PDFs but it turns out to be a pretty challenging problem to generalize.

</comment><comment><USER7> Looks like a very shallow article that hasn't actually solved the meat of the problems. Am I reading it wrong?

<comment><USER5> Heh. It's my work, so maybe I can clarify the goals. This is meant to be a proof of concept. I took a week and was able to show that relatively simple deep learning techniques are capable of generalizing over unseen form types with high accuracy. I also showed that tokens-plus-geometry is a viable format, and that hand-crafted feature engineering is still necessary (and still used in SOTA approaches).

I also believe that preparing and cleaning this data set, and bringing a challenging investigative journalism problem to the attention of other researchers, would be valuable even if I hadn't done any work on this baseline solution. This is a problem that journalists currently expend a huge amount of time and money on, which reduces the effectiveness of transparency around political ad spending information.

<comment><USER8> I'm curious if you considered or attempted to use pdfplumber's table extraction methods to separate tabular from non-tabular text. That would be my starting point on a problem like this, as picking the relevant rows of a table is far easier than picking from the set of all tokens. By the way, when you say tokens, you're referring to non-whitespace characters separated by whitespace? How reliable have you found pdfplumber to be in picking out words/tokens?

<comment><USER5> I didn't try separating out tables because the total field isn't actually "inside" the table in many cases. Certainly the other fields I want are not.

pdfplumber seems mostly ok at extracting tokens. Sometimes it seems to combine tokens that should be separate. I suspect a few percent of the error is actually problems earlier in the data pipeline, as opposed to the model proper.

</comment></comment></comment></comment><comment><USER9> Accuracy may not be the best metric, if your dataset is imbalanced.

<comment><USER5> It's not a binary classifier. Every invoice in the dataset has a total amount written somewhere on it. Accuracy here is whether the network chooses the correct token from each PDF.

</comment></comment><comment><USER10> > This project aimed to find out, and successfully extracted the easiest of > the fields (total amount) at 90% accuracy using a relatively simple network.

Why does this page not even say what this field was? For all we know, OpenCV would have worked better.

<comment><USER11> As it states, the field in question is "Total amount"

</comment></comment><comment><USER12> how does the performance compare to a simple ocr reader?

<comment><USER3> It operates on ocr output. It's job is to identify the total expenditure by looking at ocr output.</comment></comment>


Google has patented a way to hunt down potholes - <AUTHOR> http://www.businessinsider.com/google-has-patented-a-way-to-hunt-down-potholes-2015-8

<comment><USER2> That's not the issue. Potholes are reported by many people. The issue the getting it repaired (properly).

I'm not knocking road crews filling potholes. There just aren't enough of you to keep up.</comment>


A Primer on the Doomsday Argument - <AUTHOR> http://www.anthropic-principle.com/?q=anthropic_principle/doomsday_argument <comment><USER2> Other than the fact that you can't coherently reason based on your own existence (existence is not a predicate), one should rightfully be wary of an argument that would have been valid _and false_ at every point in the past.

The Doomsday Argument is not an argument in favor of its conclusion -- it's a reductio ad absurdum that demonstrates why this type of reasoning is invalid.

<comment><USER3> Is this not a weakness in many (all?) deductive arguments? i.e. if we throw out this argument then we have to throw out much of science.

<comment><USER2> In a general sense, I don't believe so. While "there exists an experimenter" is an inherent assumption in many deductive chains, there's no attempt to use the existence of the experimenter as a prior in a Bayesian calculation. The logical flaw in the DA comes when you say "I, the reasoner, am likely to exist at a random point in the history of humanity", because the reasoning could not take place without your existence and therefore the hypothesis is not- falsifiable (no matter when in the history of humanity the reasoning occurred).

</comment></comment></comment><comment><USER4> >Now we modify the thought experiment a bit. We still have the hundred cubicles but this time they are not painted blue or red. Instead they are numbered from 1 to 100. The numbers are painted on the outside. Then a fair coin is tossed (by God perhaps). If the coin falls heads, one person is created in each cubicle. If the coin falls tails, then persons are only created in cubicles 1 through 10.

>You find yourself in one of the cubicles and are asked to guess whether there are ten or one hundred people? Since the number was determined by the flip of a fair coin, and since you haven't seen how the coin fell and you don't have any other relevant information, it seems you should believe with 50% probability that it fell heads (and thus that there are a hundred people).

The conclusion in the last sentence is incorrect is an incredibly subtle way. Since 10 times more people are in cubicles in the heads case, the probability that you find yourself in a cubicle at all is ten times higher in that case, which affects your prior. By Bayes' theorem:

P(HEADS| WAKE UP) = (P(WAKE UP|HEADS) * P(HEADS)) / ( P(WAKE UP |HEADS) * P(HEADS) + P(WAKE UP|TAILS) * P(TAILS) )

= (10 * P(WAKE UP|TAILS) * P(HEADS)) / ( 10 * P(WAKE UP|TAILS)* P(HEADS) + P(WAKE UP|TAILS) * P(TAILS) )

= 10 / (10 + 1)

= 10 / 11, or about 91% chance of heads given that you find yourself in a cubicle.

<comment><USER5> > the probability that you find yourself in a cubicle at all is ten times > higher in that case

No. That you are in a cubicle is a given fact, so the probability that you find yourself in a cubicle is unconditionally 1.

<comment><USER6> >>No. That you are in a cubicle is a given fact, so the probability that you find yourself in a cubicle is unconditionally 1.

But it's not 1 that you would be in a cubicle at all. The paradox disappears if the problem was formulated like this: "God shuffles DNA and there is 0.0000000000001 chances tha it shuffles up you and he does that for every cuebicle". Or: "God create you first, then flips a coin and put you at random in one of the created cubicles".

Formulation of the problem should mention which one is it. If it doesn't we are back to guessing what God does (similarly to 2 envelope problem which comes down to guessing what the sponsor's preferences for amounts are). As there are infinitely many ways God could decide to create humans in cubicles you can't answer that without giving some guess (priori) for probability distribution over those choices.

<comment><USER4> There are only finitely many ways to arrange matter in a finite volume under a finite temperature. If it's safe to assume that anything we could classify as "human" will fit inside a sphere of radius 1 light-year and has a temperature under 10^20 K, there are finitely many humans.

<comment><USER6> Yeah but it's not given that God chooses at random who is going to cubicle. He might have for example created you first and then put you in a cubicle regardless if it's 100 of them or 10. I agree with your resolution of that paradox though. It's reasonable to assume - having no information about the process - that God creates humans randomly and it's 10x more likely that you are created in a world with 100 cubicles than in a world in 10 cubicles. In my mind there is no paradox at all there.

</comment></comment></comment></comment></comment><comment><USER7> Is there a strong, formal version of the Doomsday argument not based on cubicles and such? The obvious flaw in that version is that it is based on finite possibilities: that is, finding oneself in this or that kind of universe out of so many possible ones.

For starters, we have no idea of how likely our universe is; we know what numerator is 1, but what goes on the denominator? Is there even such a number? If so it is something so huge that the probability is vanishingly close to zero.

Then what about the parallel universes hypotheses: what if the universe is constantly fragmenting into multiple futures. In one future, I will submit this comment and close the tab, then check my e-mail. In another, I keep the tab open and go make a coffee.

This proliferation of multiple futures from any moment would tend to provide a way for doomsdays to be circumvented.

If you are cloned into N futures in this moment, and you're killed in N-1 of them, then you don't know it; by the anthropic principle itself, only the surviving future matters; there is no consciousness in the others. The suriviving future has no idea about the size of N, either.

<comment><USER8> The big gaping hole I see is that the blog post at least doesn't treat the earlier epochs with the same question. If a person in 1000 CE took up the same question, would they predict Doom Early or Doom Late? If everyone for the last ten thousand years would predict Doom Early, that seems like a piss poor predictor.

<comment><USER9> That's just a special case of a general feature of probabilistic arguments: a small but positive fraction of the times they are used, they will give wildly misleading predictions. If we assume that everyone uses the doomsday argument with a uniform improper prior on the total number of people who will ever live, then all of them will conclude with 95% probability that they are not one of the first 5% of people to ever live (that is, that no more than 20 times the past population will be created in the future), and 5% of them will be wrong, which is exactly how probabilities are supposed to work.

</comment></comment></comment><comment><USER10> This is a much better discussion of the Doomsday Argument, and the self- indication and self-sampling assumptions:

[http://www.scottaaronson.com/democritus/lec17.html](http://www.scottaaronson.com/democritus/lec17.html)

The whole course is also very good, and will probably be interesting to many HN users.

<comment><USER6> Very nice read, from this lecture:

>>So you're in the room. Conditioned on that fact, how worried should you be? How likely is it that you're going to die?

        A: 1/36.
        Scott: OK. That would be one guess.

>>One answer is that the dice have a 1/36 chance of landing snake-eyes, so you should be only a "little bit" worried (considering). A second reflection you could make is to consider, of people who enter the room, what the fraction is of people who ever get out. Let's say that it ends at 1,000. Then, 110 people get out and 1,000 die. If it ends at 10,000, then 1,110 people get out and 10,000 die. In either case, about 8/9 of the people who ever go into the room will die.

But it's not really a problem. If you flip a coin, then 10 coins, then 100 coins and continue until you get all heads then vast majority of flipped coins will be heads while still any particular coin have 1/2 probability of landing heads. There just isn't any paradox or even anything surprising. Similarly here it's in my opinion obvious that chances of dying are 1/36 and they are the same for all people in the situation. The fact that most people die isn't a paradox at all.

<comment><USER10> I think it's still a "problem." Which one is the "correct probability" that you're going to die? If, instead of dying, a red light in the room might turn on after a minute, how would you place a bet about whether the red light would turn on?

</comment></comment></comment><comment><USER11> So, there are a few problems here:

The argument assumes that the only possible end-state for humanity is doomsday; that is, there are a finite number of humans, and when there are no more doomsday has occurred. The article touches on this with the infinite humans / transhuman stuff at the end.

The argument assumes exponential growth in number of humans until the terminal state, but this is not a foregone conclusion. While this doesn't change the reasoning in a strict sense, it may put the "soon" cutoff millions of years in the future.

Finally, the prior probability is chosen arbitrarily. It does matter quite a bit. Take the exponential argument for evidence as to why: For any given probabilty p that doomsday will occur on a given day, it is more likely today than on any single other day! This is simply because if doomsday occurs today, it cannot occur on any future day, making it less likely on each successive day.

Except that if p=1E-10, doomsday is fantastically unlikely to occur this year. The Bayesian argument has the same problem, you just start seeing it at a different value of p.

</comment><comment><USER12> Let's go back to step II:

Imagine that the coin that we're flipping isn't fair, and comes up heads with prior probability p. Further, suppose that the ratio between the small population and the large population is R (in the article this is R = 10 / 100 = .1). In this situation, upon discovering that you're inside the small population, the posterior probability of heads ends up as

        p / (R - p * (1 - R))

So what if the coin is very unlikely to be heads instead of 50/50? If p = .001 (and we leave R = .1), our estimate of the probability heads after we observe that we are in the small population only comes up to about 1%.

Thus, the real fallacy in the argument is that the choice of prior is unimportant. (If anyone tells you the prior is unimportant in any situation, they are wrong.) With a suitable estimate of the prior likelihood of Doom Soon, the posterior likelihood of Doom Soon is still low enough.

<comment><USER13> I played around with the model a bit and based on a couple assumptions it can be scaled to

        P(Doom Soon) = x/(1+x) where
        x = B/b * d

where B is the number of humans born in the Doom Late scenario, b is the number of humans born so far, and d is the prior probability of Doom Soon.

Assumptions: a) the number of humans born between now and Doom Soon is negligible and b) the Doom Late scenario has many more humans than Doom Soon (B + b ~= B).

Notice I made no assumptions about the prior, d. Of course d does matter, but the point is that as long as the number of humans in Doom Long is assumed to be large enough, the probability will go close to one even for very small d.

For example, if we use a Doom Late population of 200 trillion like in the post, then we have 95% probability of Doom Soon even if d ~= 0.0001.

That being said, I am still fairly unconvinced by this argument. It would have every intelligent species continually concluding that they are about to go extinct, right up until the moment that they either _do_ go extinct or they achieve immortality and stop reproducing.

<comment><USER12> As B -> \infty, also t -> \infty.

I'm reminded of the Fight Club quote: On a long enough timeline, the survival rate for everyone drops to zero. :)

<comment><USER13> Haha nice. Although, in this case we are looking ahead, basing our short-term survival on what a long-term survival would theoretically look like. So the more people there are in the hypothetical Doom Late, the more likely Doom Soon becomes. The more I think about this the more absurd it seems.

<comment><USER12> Yeah, all this analysis (both what we've done and in the original article) are facile--a proper Bayesian treatment would have continuous priors/posteriors that would be a little more informative than an either-or.

The problem you're seeing here is that if you make your possibilities "Humans live forever (even past the heat death of the universe)" or "All humans die in the next 10 minutes", you'll find that the chance that humans die in the next 10 minutes is really absurdly high.

</comment></comment></comment></comment></comment><comment><USER5> Here's a simpler way to see what is going on. Number all the humans that will ever live in order of their birth from 1 to N. The odds of your birth order number (call it B) being in the range of 1..M<N is M/N. The larger N is, the less likely it is that your B is as small as it is. So the odds of N being large (relative to B) are small.

Another way to look at it: half of the humans who ever live will have fewer people born after them than before them. Your a priori odds of being such a person are exactly 1/2.

</comment><comment><USER14> "If the Doomsday argument is correct, what precisely does it show?"

That philosophers are not statisticians?

<comment><USER15> That philosophers take their arguments too seriously?

I mean, this is fine as an interesting thought exercise. To the degree that you take it seriously as applying to the real world, to that degree you need... something. Perspective? To not take fine-sounding argument so seriously? To take a long walk? To get a life?

Not everything that cannot be logically disproven is true. Not everything that cannot be logically disproven should be acted upon as if it were true. It is useful to know when to smile at the philosophical earnestness and then just go on with your life.

</comment></comment><comment><USER16> Is this a restatement of the Fermi Paradox?

[http://en.wikipedia.org/wiki/Fermi_paradox](http://en.wikipedia.org/wiki/Fermi_paradox)

<comment><USER17> No, it attempts to quantify how many humans are alive at a point in the future.

[http://en.wikipedia.org/wiki/Doomsday_argument](http://en.wikipedia.org/wiki/Doomsday_argument)

</comment></comment><comment><USER18> Ok, this may be mathematically naive, but...

In the cubicle example discovering that you are in cube 1-10 makes the likelihood of the 1-10 scenario much more likely than it was before you discovered you were in one of the first 10 cubes using a simple application of Bayes theorem.

With the 100 billion or 100 trillion people example, you are person 60 billion. Theoretically making 100 billion much more likely than if you didn't know where you fell. That probability approaches one if work off the assumption that doom(late) means hundreds of millennia of humans spreading across the galaxy at our current growth rate. the higher the Total possible humans in doom(late) the higher the probability that being in the first 100b indicated that there will only be 100b. (or similar numbers)

One difference I see between the two scenarios is time. The 100 cubes are not filled in sequence. In the 100billion/100trillion example, every single person that ever lived is in the 100billion, until they're not, then everyone is not in the 100 billion. I don't know that it affects the math, but it affects my thinking about the problem.

</comment><comment><USER19> Perhaps I'm ignorant of some deeper concept... but it seems that this argument is deeply flawed on the basis that they are presuming that finding yourself in cubicles 1-10 somehow indicates that cubicles 11+ are probably vacant.

In both doom soon and doom late, cubicles 1-10 are occupied. There's no aspect of doom late that would be made less likely as a result of cubicles 1-10 being occupied.

The reverse works well, though. Being in one of cubicles 11+ /does/ eliminate the possibility of doom soon. So only passing the cubicle threshold gives any meaningful information about which of the two scenarios are true.

<comment><USER20> It's probably not worth worrying about the fate of humanity till you've figured out the rooms example. Since you're on this site I'm sure you can trivially knock up a script to simulate what happens. Or just draw the probability tree. In any case the relevant question is,, out of those who find their room number is less than ten, what is the probability that the coin toss meant only ten rooms are occupied. The answer is 90.9% - we could go round in circles trying to choose words that everyone understands, but the numbers speak for themselves.

<comment><USER19> Okay, here's a simplified probability tree.

Scenario 1: Doom soon

10 people have a room number less than or equal to 10.

Scenario 2: Doom late

10 people have a room number less than or equal to 10.

\---

Given that each scenario has an equal probability in isolation, there are 20 possible positions for the the <=10 individual to find themselves in (10 in doom soon, 10 in doom late).

Of those 20 possibilities, 10 of those are in the doom soon scenario. The remaining 10 are in the doom late scenario.

10 doom soon to 10 doom late is a 50/50 probability.

Edit: writing up a script as well, right now. Will add to link to it in my original comment when complete.

<comment><USER20> You're wrong. Not being mean, you've just arrived at the wrong numerical answer to the maths question.

(Let's leave aside mentions of doom and stick to the rooms: according to the article no-one has quite nailed down yet whether there are subtleties about unbounded future populations). Rather than try to draw a tree I'll write it out flat:

10 rooms (0.5) & you're in 1-10 (1.0): Probability 0.5

10 rooms (0.5) & you're in 11-100 (0.0): Probability 0.0

100 rooms (0.5) & you're in 1-10 (0.1): Probability 0.05

100 rooms (0.5) & you're in 11-100 (0.9): Probability 0.45

Work out any relative probabilities using the right-hand column. For instance, when you're in room 1-10, the chances of the coin having been heads is 0.5/0.55. Even if this bugs you, this approach of using the tree will let you get the right answers to tricky teasers about medical tests with false positives and false negatives.

<comment><USER19> Just to confirm my understanding...

The situation you gave me was "out of those who find their room number is less than ten, what is the probability that the coin toss meant only ten rooms are occupied".

In my example, I composed my tree of solely those who with a room number less than or equal to 10, since in your question, we're only concerned about those who are in rooms 1-10.

In your example, it seems you've included those who are in rooms 11-100... If we are only concerned with rooms 1-10, what relevance does the probability of 11-100 play in this role?

More specifically, your seems to be what the odds are of /being in/ room 1-10, rather than the odds of each scenario occurring, assuming you are in room 10.

</comment></comment></comment></comment></comment><comment><USER21> Imagine you have 100 ordered rooms. All initially are filled with gold and ponies. God tosses a coin and depending on the result of the flip God strips 10 or 100 rooms of ponies and gold.

You are in one of the rooms, open your eyes and see that there is no gold and ponies around. You take a look at your room number and see 7. Then you know that best decision is to operate on assumption that there is 91% chance that only 10 rooms were stripped of gold and ponies. So lots of gold and ponies should await you very soon as you move to further rooms of increasing numbers.

Reassured with that bulletproof argument we may look into the future with hope that rather sooner than later there will be times where we won't be confined to this dull rock we evolved on, to this lame rapidly decaying bodies with machines that push those clunky electrons around that can barely add 0 to 1 while overheating immensely.

Because near future is (likely) full of gold and ponies.

I hope this illustrates why purely philosophical arguments are empty of any utility. If you reason without experimental data to anchor your reasoning to something real you end up with reasoning that works just as well when applied to the thing you had in mind when it wandered as to thing opposite to your intention giving opposite conclusions and teaching us absolutely nothing about world.

Ah. First two dowvotes and no comment from fans of realworld importance of philosophy in this day and age. How classy!

<comment><USER22> The Doomsday Argument works because the result (Doom Soon or Doom Late) affects the number of observers, and we can work backwards to update our beliefs on which one will be.

That is why replacing the words with ponies and gold won't work unless you are a pony or sentient gold.

<comment><USER21> For my argument I just replaced "existing" with "not having gold and ponies" and drew parallel not between "specific rooms being occupied" and our civ existing at specific time but between "specific rooms not being emptied of gold and ponies" and "our civ having decent tech at specific time".

It works. Use your own substitutions.

Phisolophical reasonings often latch onto inexplicit meaning of words. If you replace them with different words that fullfill same role in language but have different meaning argument becomes much less apparently profound or even false.

</comment></comment></comment><comment><USER23> "Corresponding to the prior probability (50%) of the coin falling heads or tails, we now have some prior probability of Doom Soon or Doom Late."

Nice to just magically know this probability. I get that the actual number doesn't matter, but I worry about a predictor that doesn't care about any real probabilities based on observation.

</comment><comment><USER24> My favourite answer to the doomsday argument: The many worlds hypothesis is correct and we haven't created our "world ending" p=np computers yet. (Literally halt if randomly generated solution is incorrect and catch fire to every sentient being in the world).

Another possibility is that we're very close to building realistic simulations, and by induction, it makes sense that we're at this stage of development.

</comment><comment><USER25> > The last step is to transpose these results to our actual situation here on > Earth

Yeah, that's the trick, isn't it? Is the metaphor truly valid? This how philosophers differ from scientists: they see no need to test their hypothesis with real-world measurement. This is also why philosophy needs to go the way of alchemy and astrology - as an amusing but archaic belief.</comment>


Piano emulator in HTML5 - <AUTHOR> http://mpdreamz.github.com/html5-piano/

<comment><USER2> What about this requires canvas? Seems like it could be done very simply with just divs (and audio elements, of course).

</comment><comment><USER3> Here is another amusing alternative made using youtube: <http://www.youtube.com/watch?v=oD-sSolVDiY>

</comment><comment><USER4> Crashes my Chrome OS (beta channel) browser tab, though when it loaded once there was no sound.

<comment><AUTHOR> Thank you, I'm using 12.0.742.122 myself but i doubt its chrome rather this thing not being memory profiled at all just yet.

<comment><USER5> No probs with it on FF Nightly (6.0a1 2011-04-22).

</comment></comment></comment><comment><USER6> This is cool, but I don't know on what planet "C, Eb, F#" is a major chord. Seems to be off by a half-step (i.e. should be C, E, G)?

<comment><USER7> Yes, the major should be (C E G). (C Eb F#) looks like a C dimnished, though I think it would be better written as (C Eb Gb).

<comment><USER6> True - for some reason I have this affinity to certain notes being sharp and certain ones being flat (i.e. F# > Gb, but Eb > D#), even though the rule is that it should be consistent based on the key.

</comment></comment></comment><comment><USER8> An amusing alternative:

<http://sweetmusicapp.com>

</comment><comment><USER9> Didn't freeze for my Opera 11.50. Great job, I like musical tools this this a lot!

</comment><comment><USER10> Completely froze Opera 11.50 for me.</comment>


Eating our own dog food (for lunch) - <AUTHOR> http://p.ota.to/blog/2013/09/eating_our_own_dogfood/

<comment><USER3> I agree that getting up and moving is good for you, so having lunch delivered to your desk is probably not the best idea.

That said, it reminds me of a funny story. At Sun (Mtn View) we would go to La Cosentena[1] and get burritos, you could fax in an order for 'will call' and pick it up which made things a bit smoother. An engineer decided to show off the distributed object technology Sun was building with an example app called 'burrito tool' that you could select your burrito and it would fax it to La Cosentena for pickup.

It was great, and fun, and we bragged about it. Then some enterprising Sales Engineer made a copy and took it with him for use at a trade show. We showed up for lunch and none of our burritos were made, they explained they had been getting hundreds of faxed in burrito orders all morning and stopped making them when nobody came to pick them up. Sure enough, Burrito Tool, even from the floor of the conference in New York city, found the Object request broker back at HQ and forwarded Burrito requests to the real object back in Mtn View. Holy bogus burritos!

[1] [http://www.yelp.com/biz/la-coste%C3%B1a-mountain- view-2](http://www.yelp.com/biz/la-coste%C3%B1a-mountain-view-2)

</comment><comment> jon-wood Office assistants bringing food to your desk. So you don't have to leave your desk ever, for any reason. Call me crazy but I quite like standing up and leaving my desk to get some lunch.

<comment><USER4> Leaving your desk and going for a stroll (outside or inside) is great for mental peace. Standing in a long queue with a horde of other people for 15 minutes is not. This is an elegant solution to the problem :)

(Disclaimer: I am a Potato.)

<comment> jon-wood Fair point, I spend most of my week working from home, and when in the office we have a rota for people making lunch which we thence sit down to eat, so I'm a little out of touch with the lunchtime rush!

I'm totally behind the idea of lunch at work, I just think its something that can be handled far better than dumping a meal at your desk

</comment></comment></comment><comment><USER5> My old^3 workplace has had an equivalent for years, but running via IRC/SMS (it's been around since before phones were good at complex webpages) - <USER2> small, tight Java codebase. Sometimes I despair at how many extra layers of cruft seem to be spent making everything webby (memcache for an app this simple? really?), when the end result is the same level of functionality.

<comment><USER6> I also raised an eyebrow at the mention of memcache - but hey, if the outcome is that the intern learns a thing or two, why not.

</comment></comment><comment><USER7> Please add some left-padding to the content...

</comment><comment><USER8> How long did it take to build the app with all functions?

<comment><USER9> Just as important, how much did the intern learn while building it? Sounds like they got to touch a number of different APIs, libraries, and concepts.

<comment><USER10> A stupendous amount. Before starting this project I had never done any complex frontend stuff. Pretty much just HTML CSS and a bit of jQuery.

With this one project I learned about backbone, marionette and the limits of what can be done on the frontend. It was quite a big learning curve, and even now I'm still discovering where the correct balance of frontend-back end stuff is.

Overall a great experience, really opened my eyes to the power of JavaScript, before I had adamantly stuck to nothing more exciting than HTML forms with JS validation.</comment></comment></comment>


A Business Idea for Anyone Who Wants It - <AUTHOR> http://freakonomics.blogs.nytimes.com/2007/09/12/a-business-idea-for-anyone-who-wants-it/ Steen Levitt, co-author of Freakonomics, provides the algorithms he used to detect cheating on standardized tests by teachers at public schools. He suggests that with the No Child Left Behind act there might be a solid business idea in developing software that looks for statistical anomalies in test scores in order to detect cheating. Teachers might cheat by inflating the scores of their students in order to secure more funding for their school. <comment><USER2> That is not a very good idea. If money given by government is tied to test scores, schools have more incentive to let students cheat than otherwise, for the high scores. You'd be trying to sell them something that at the very least, lowers overall test score averages.

<comment><USER3> So you sell this on the federal level, to let the DoE make sure the state level schools are honest. What's even better, you can charge a lot more because the DoE has a bigger budget.</comment></comment>


Chinese schools enforce smart uniforms with tracking system to monitor students - <AUTHOR> https://www.abc.net.au/news/2018-12-28/microchipped-school-uniforms-monitor-students-in-china/10671604 <comment><USER2> Don't forget that Chinese school students also need to wear their school uniform faces because of the facial recognition monitoring.</comment>


695k TPS with Node.js and VoltDB - <AUTHOR> http://voltdb.com/company/blog/695k-tps-nodejs-and-voltdb

<comment><USER2> I'd like to add an independent voice too. I tried VoltDB out around a year ago. Ran it on 3 servers, with k=1 (all data replicated on 2 nodes). Servers were Dell PowerEdge 860 - only 8GB RAM, and a Q6600 quad-core processor. They cost maybe $500 or so. Even with a low-end buildout, I was able to hit around 110k TPS. This wasn't one of VoltDB's sample apps; it was a small proof-of- concept for a telecom application. They don't seem to be riding the hype train.

As far as open source: I was using the .NET client, and fixed a few issues with it. The folks at VoltDB were really responsive about working with me and accepting my patches. They also took time to explain the inner workings, sorta give me a launching point to investigating the core code. Pretty good experience for a commercial open source project.

<comment><USER3> > it was a small proof-of-concept for a telecom application.

would you describe what telecom application this was ? was it billing, ocs, pcrf or something else ? thanks !

<comment><USER2> Yes, very basic realtime billing/CDR storage. With traditional DBs, I have to resort to batching balance: Read N messages from queue, compute overall balance updates, then apply at once. With VoltDB, I can just fire off records as they come in.

</comment></comment></comment><comment><USER4>
        TITLE DISCLAIMER: 695k using 20 fairly large machines

I was, at first glance, pretty impressed with the 695k TPS title claim. Upon reading the article, it became apparent that this was based on running 8 8-core EC2 m2.4xlarge instances for node.js in addition to 12 8-core EC2 m2.4xlarge instances for VoltDB. In terms of server architecture, this is quite a large setup.

Just for an idea of how big of a machine we're talking for each of these instances:

        High-Memory Quadruple Extra Large Instance
        68.4 GB of memory
        26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each)
        1690 GB of instance storage
        64-bit platform
        I/O Performance: High
        API name: m2.4xlarge
        $1.800 an hour running Linux on demand
        $864.00 a day baseline for 20 of these

This isn't meant to discredit the author or work, just to make sure it's apparent that this solution is for serious businesses only with deep pockets and boatloads of traffic. You'd be spending _$26,784_ for the servers alone over a 31 day period if using them on demand.

The results are definitely impressive for the right target audience.

<comment><USER5> I'm not sure if I am supposed to be impressed or not. On my macbook pro with 1 redis master and 1 redis slave I can do about 300k updates / second to the master replicated to the slave. Comparable? Who knows?

These benchmarks don't mean much to me -- what matters is performance for your application.

<comment><USER6> I'll start by saying I dig Redis. I wish we had a storage engine that offered some of the kinds of flexibility that Redis does. If I were making Volt from scratch I would embed Redis or something with flexible schema and secondary indexes and spend less time working on SQL and more time working on distributed functionality.

>These benchmarks don't mean much to me -- what matters is performance for your application.

Very true. Think about how you would implement an application like Voter in Redis. Voter is an event counting app that applies business logic server side to do what is basically fraud detection.

Let's take a look at what each of these transactions is. [https://github.com/VoltDB/voltdb/blob/master/examples/voter/...](https://github.com/VoltDB/voltdb/blob/master/examples/voter/src/voter/procedures/Vote.java) 3 statements and an entire round trip to validate data is kosher. 1 insert/update statement that is actually updating two different summary views

You would need to implement your own locking to get the isolation necessary to implement the business logic and there would be at least two round trips for each vote. You would also have to write your own code to maintain the materialized view used to print the summary. Sure you could implement the validation logic client side, but then you lose the ability to transactionally update said logic and the state it depends on.

With the new scripting functionality in Redis you would be able to do it in one round trip but you would still end up writing your own code for the view and rollback. How you will attach a debugger to debug your server side logic is also worth considering. I mistakenly wrote that Redis doesn't cache Lua scripts, but it actually does.

Now that you have server side logic you will also probably want to start maintaining replicated state to make some of your transactions single shard transactions. With Redis you are on your own for updating replicated state in a way that stays consistent across the cluster even when restoring different shards.

Also, no matter how fast Redis is you may reach a point where you have run out of disk/network IO, or even CPU. Now you are looking at sharding Redis. You are probably going to want to maintain replicated state to make more of your transactions single shard transactions. With Redis you are on your own for updating replicated state in a way that stays consistent across the cluster even when restoring different shards.

Don't forget re-sharding and backups which are now your problem. You definitely want to be able to roll back to before that last application update that corrupted your data. Don't forget that you will need to bring the replicas up in the correct state as well. Preferably in parallel with bringing up the master since you are down right now, and it would be nice if it resharded at the same time since you took this opportunity to add capacity. Redis cluster is coming, but it isn't here today and depending on the application you may not want to risk it not being ready when the time comes.

You should at least double the number of ops (where an op is roundtrip to the DB) done by Volt if not more. That said, doing half as much work inside the procedure doesn't make it twice as fast and doing twice as much doesn't make it twice as slow. The dominant cost for such a trivial operation is not the operation itself. Redis is much better at this because it is single threaded and written in C. My bet is that the number of instructions and cache misses to accept and execute a command in Redis is much smaller because the execution engine isn't dealing with stored procedures, schema, undo logging, SQL, and code expecting to deal with distributed transactions even if none are in play.

>I'm not sure if I am supposed to be impressed or not.

If you haven't tl;dr-ed already I'd call it good.

</comment></comment></comment><comment><USER7> So voltdb is an in memory SQL Db with some durability and replication functionality. Is there any good documents on how it compares with some of its competitors/ who are its competitors?

<comment><USER2> I think durability should not be defined in terms of actual backing hardware. VoltDBs transactions commit and stay committed. If you restore from a backup, just like any other database, you'll have data loss until the backup point. What if my hard drive crashes when it loses power? At that point I'm in the same position as when I lose RAM.

VoltDB is often compared to other in-memory NoSQL systems, although I'm not aware of any that actually offer cross-server transactions. VoltDB is best when you have a high volume OLTP workload and need to really scale beyond what traditional RDBMS can do.

Edit: Here's a benchmark using VoltDB as a key-value store: <http://voltdb.com/company/blog/key-value-benchmarking>

<comment><USER8> > What if my hard drive crashes when it loses power? At that point I'm in the > same position as when I lose RAM.

Oh sure, except that the probability of losing your hard drive when you lose power is about 1 in a _very_ big number. Whereas, the probability of losing the contents of your RAM when you lose power is 100%.

Now, MySQL (and in fact, most RDBMS outfits) provide tools for fixing corrupt data. But you actually need corrupt data to fix. That's the entire point of a hard drive.

I'm sure VoltDB would be perfect for a chat application or something of that nature. It would be terrible for anything that involved the transfer of $$$ (banking, ecommerce, etc).

<comment><USER9> VoltDB Community Edition uses synchronous intra-cluster replication, as well as rolling on-disk snapshots for durability.

The Enterprise Edition adds a synchronous or asynchronous command log (a logical journal) as well as asynchronous multi-cluster replication.

The VoltDB blog has a series of posts covering command logging in detail: <http://voltdb.com/company/blog/intro-voltdb-command-logging> <http://voltdb.com/company/blog/voltdb-command-logging-replay> [http://voltdb.com/company/blog/voltdb-command-logging- option...](http://voltdb.com/company/blog/voltdb-command-logging-options)

In short, VoltDB offers tunable durability from single-node purely in-memory, all the way up to synchronous disk storage on multiple local nodes and multiple asynchronous copies in another data center. All options are fully transactional. All options are extremely performant.

We've worked very hard to make durability and performance not mutually exclusive. We have internally achieved six-figure TPS numbers using synchronous writes (requires a disk controller with a BBU, or a decent SSD).

</comment></comment></comment></comment><comment><USER10> If you think node.js is fast try <http://vertx.io>.

It's like node but isn't single threaded so scales over multiple cores without having to fork. Also it's polyglot so you can don't have to use JS if you don't want.

I'm hoping to publish some performance results vs node.js before our 1.0.final release in the next few weeks.

<comment><USER11> How does vert.x compare to to Apache Deft?

</comment></comment><comment><USER12> Is EC2 really the right way to do these tests? I would have thought it would introduce unwanted variables related to network congestion and server load caused by other EC2 users.

<comment><USER9> You might get a better number on bare metal, and you certainly won't get a worse one, so in a sense it's not ideal.

On the other hand, a big chunk of the web is run from AWS. Demonstrating performance there implies performance in many places. It's also an easily reproducible environment, which is helpful for benchmarking.

</comment></comment><comment><USER13> Sounds nice. Some questions:

\- I assume you did the tests with direct SQL statements without any ORM? How would an ORM affect the test's performance (with an Node.JS-ORM of your choice, e.g. sequelize)?

\- How does Node+VoltDB compare to i.e. Node+PostgreSQL or Node+MySQL

\- What does in memory SQL exactly mean? How is data made persistent?

<comment><USER2> VoltDB doesn't really support ad-hoc SQL. It can execute ad-hoc by flipping a flag, but performance is achieved by using stored procedures. You write the sprocs in Java, and their compiler looks at the SQL statements and optimizes.

VoltDB smokes other RDBMS (Postgres, MySQL, MSSQL, etc.) for certain OLTP workloads because it stays purely in-memory, _but removes all locking_. Normal databases spend a huge amount of time dealing with locks, even if the memory is in RAM. VoltDB was based on the H-Store project; you can read more about the ideas at [1].

Making transactions execute serially[2] adds latency to an individual transaction, but the point with VoltDB is to send tons of transactions so that there's always work queued up. You should be able to get average latency of, say, 10ms for the type of app mentioned here. I'm surprised it's not mentioned in the blog post. (Maybe on EC2, the latency numbers were less-than-ideal, but throughput was fine.)

SQL and ACID have nothing to do with the backing store. Using RAM is just as valid as using disk. Practically, with VoltDB you're going to set the "k factor" - number of replicas you want. This allows you to lose k servers without suffering data loss (put batteries on the servers; that should be pretty good). There's a snapshot backup feature, and the enterprise edition has a disk logger to further minimize the loss window in case of catastrophic failure.

1: <http://hstore.cs.brown.edu/publications/> 2: More or less. Servers are sync'd via NTP, allowing one to know when it's safe to run a specific transaction. There's also some ways Volt might know when one tx is independent and safe to run before another.

<comment><USER14> Well, it removes all locking and achieves scalability only if you partition (shard) the data and don't have cross-partition transactions.

<comment><USER15> VoltDB does support cross-partition transactions. They're not as fast as single-partition transactions, of course, because you're doing them over a network. But distributed transactions have been a supported feature since 1.0.

<comment><USER14> Right, that's what I meant. You only get scalability as long as you don't have cross-partition transactions.

</comment></comment></comment></comment></comment><comment><USER16> > that's beneficial to the scaling architectures of both Node.js

That's not very accurate (unless you run multiple Node.js processes). See [http://stackoverflow.com/questions/2387724/node-js-on- multi-...](http://stackoverflow.com/questions/2387724/node-js-on-multi-core- machines)

</comment><comment><USER17> looks very interesting. If I architect a tracking system again, I'd look at this setup.</comment>


The Facebook Privacy Fiasco Begins - <AUTHOR> http://www.techcrunch.com/2009/12/09/facebook-privacy/ <comment><USER2> _The vast majority of users will ignore this information the same way 80% of them have ignored privacy settings in the first place. But Facebook will still be able to claim that it did its best to get their informed consent._

When I went over to FB, it threw a huge overlay window at me (that I couldn't really close with the Esc key or an obviously large Close button) that walked me through the new, more easily configurable privacy settings, all of which hadn't changed from my previous choices of "Friends Only".

I really fail to see what the sinister bluster is all about. What more does TC want Facebook to do?

</comment><comment><USER3> I just logged into facebook and while the option for "everyone" was there, the default was to preserve my old settings.

What's the issue here?

<comment><USER4> Same here. Maybe Facebook was scared of Twitter at some point in the last 6-8 months but I don't see any incentive for them to risk a user revolt. They're doing fine -- Twitter is a different type of service. Who quits Facebook to use Twitter exclusively? I'm guessing almost no one. They compliment each other nicely.</comment></comment>


Is TechCrunch just about provoking headlines? - <AUTHOR>

Probably not, but I'll find out for sure if you vote for me to get an internship at TC for this summer.  About.me is sponsoring a competition for internships, go vote for me if you need some extra karma.  Just two simple clicks (and you can do it again after waiting 24 hours).<p>http://about.me/ericjamesjohnson <comment><USER2> Ha, I see what you did there. You get a vote, best of luck.

</comment><comment><USER3> In the words of a fictional genius, 'bazinga'

Goodluck

</comment><comment><USER4> Haha - you got me</comment>


The Families Funding the 2016 Presidential Election - <AUTHOR> http://www.nytimes.com/interactive/2015/10/11/us/politics/2016-presidential-election-super-pac-donors.html?hp&action=click&pgtype=Homepage&module=photo-spot-region&region=top-news&WT.nav=top-news&_r=0 <comment><USER2> Quoting the article—please take this away if nothing else:

        But regardless of industry, the families investing
        the most in presidential politics overwhelmingly
        lean right, contributing tens of millions of
        dollars to support Republican candidates

<comment><USER3> This article is about direct contributions from individuals. This is just a fraction of the donations to political campaigns. For example, you don't find any mention of labor unions which provide more money than individuals and donate almost exclusively to democrats. And we haven't mentioned corporations.

Imagine a giant hammer and sickle and a McDonalds arch so large that you can't see the pile of monopoly pieces or even the White House behind them.

<comment><USER2> 2012's biggest individual contributors: [https://www.opensecrets.org/orgs/list.php?cycle=2012](https://www.opensecrets.org/orgs/list.php?cycle=2012)

The first two items in the list are both Sheldon Adelson

ActBlue is primarily individuals donating to democratic candidates they support.

#4 backed republicans almost exclusively. Interestingly, their founder funded Oliver North's legal defense in the Iran-Contra scandal. More on his 2012 campaign funding: [https://en.wikipedia.org/wiki/Harold_Simmons#2012_presidenti...](https://en.wikipedia.org/wiki/Harold_Simmons#2012_presidential_election)

Only when you get to #5 do you see union money. But, the interesting thing about that money is

        Most of the political work of the Service
        Employees International Union is funded
        by about 300,000 janitors, nurses' aides,
        child-care providers and other members
        who voluntarily contribute on average $7
        per month to SEIU's Committee on Political
        Education (COPE).

[http://www.washingtonpost.com/wp- dyn/content/article/2010/10...](http://www.washingtonpost.com/wp- dyn/content/article/2010/10/20/AR2010102004912.html)

\---

So, the top 5 list for 2012 literally consists of _2 people_ on the Republican side and hundreds of thousands or millions of people on the Democratic side.

</comment></comment></comment><comment><USER4> Random thought, do people quantify investment in politics, if not in pure numbers, at least in some ball park figures ? Like, markets are down, we are better off investing 10 million in presidential elections.

<comment><USER5> Congress and the President oversee roughly 3-4 trillion dollar per year budget. Even in best years throwing billions in elections will yield enviable ROI.

</comment></comment><comment><USER5> The article is pretty thin on substance.

> in a nation that is being remade by the young, by women, and by black and > brown voters.

While this is a certain class of activist wet dream, the reality contradicts it. Big business is becoming increasingly good at doing what is best for them. They also learned to use treaties for overriding/bypassing a nation's laws. Power is getting increasingly concentrated.

</comment><comment><USER6> We should be happy that these kind benefactors are investing so much to produce free entertainment for the rest of us.</comment>


Dogs prostate cancer detection 98% reliable - <AUTHOR> http://guernseypress.com/news/uk-news/2015/04/10/dog-cancer-detection-98-reliable/

<comment><USER2> Why can't we build something that's as sensitive as a dog's sense of smell?

[Update]

Guess they're working on it: [https://vimeo.com/53900492](https://vimeo.com/53900492)</comment>


Show HN: MoonQuery.js Mongo like querying of arrays in JavaScript - <AUTHOR> https://github.com/pmoon00/moonQuery.js/blob/master/README.md <comment><USER2> How is this different from using an Array.filter method?!

<comment><AUTHOR> It's a logicless version of the Array.filter function. You can define the rules with just JavaScript objects.</comment></comment>


Going with the flow of ideas - <AUTHOR> https://www.youtube.com/watch?v=nwXxicUjU_0

<comment><USER2> Nice introduction for flows.</comment>


Multiple vulnerabilities in RPM – and a rant - <AUTHOR> https://blog.fuzzing-project.org/52-Multiple-vulnerabilities-in-RPM-and-a-rant.html <comment><AUTHOR> Some more discussion:

[https://lwn.net/Articles/698453/](https://lwn.net/Articles/698453/)</comment>


Examples of "Lazy Registration" - <AUTHOR> http://webjackalope.com/lazy-registration/ <comment><USER2> It's not just about how long the signup form is, I believe it's simply the language "account" or anything that connotes joining something. The form can be three fields, and people still hate it.

The language "save my info for later", and versions of that, are much better. It flips it around in the user's mind: "oh, now they're not asking me to give something up, they're helping me out"

<comment><USER3> _> The language "save my info for later", and versions of that, are much better._

While I understand what you're saying, I believe that would trip me up. I'm very much used to scanning for "Register", "Sign up", "Join" or "Create account".

I'm also unsure how much people really hate giving up their information. If anything, people are _looking_ for places to hand over their information: Facebook, MySpace, LinkedIn, etc.

I think the problem is really that the more steps needed to convert your users, the more likely it is a part will give up before the end.

<comment><USER4> At Soup.io (tumblelogging/lifestreaming) we initially had content creation buttons right on the front page. We figured that the best first action for new users would be to actually post something, not filling out a sign-up form.

But we had exactly the problem you're describing: People are so used to scanning for "register"/"sign up" that some even assumed we were in invite- only beta.

We eventually changed that to a "Try it now -- no sign up required" button, which works better. The wording and UI is still tricky though: "Try it now", "trial account" etc. carry the connotation that it'll cost something later.

We've been meaning to A/B test a straightforward sign-up form for a while and measure effects on both conversion & retention, but unfortunately haven't gotten around to it yet.

</comment></comment></comment><comment><USER5> Stack Overflow's registration is awful because it only supports OpenID, a service that is spotty at best. I tried to ask a math question there and just couldnt get it to log me in.

<comment><USER6> That's your Open IDs provider's fault, not Stack Overflow's. Yeah, they may suffer a little but not enough to be responsible for a shoddy provider. But with how many providers are out there, it should be easy enough for you to use another one.

<comment><USER7> No, it's Stack Overflow's fault for using such a terrible authentication system. OpenID has pretty much proven itself to be a failure in user experience.

Facebook Connect has accomplished most of the goals of OpenID, anyway. They should use that instead.

<comment><USER8> This makes absolutely no sense to me. OpenID does not dictate anything about the user experience. The site that is using OpenID places a log in textbox, decorate as you wish, which takes you to your OpenId provider...which is the experience you chose, given by the provider that you liked....

Facebook connect took off because of the site itself, how many people use it, and people like those who constantly tell me that I should have Facebook logins for my sites and apps (and no, I shouldn't).

so...

\- stop insulting openid unless you got a better idea that isn't proprietary

\- or if you cant think of a better solution than openid, help us get it out there because it beats the old solutions, and we can at least start improving until we can think of something better (i can't think of anything better...i gave it a shot for an afternoon, its theory is sound, and its very flexible and solves the problem just great)

<comment><USER7> OpenID is a horrible user experience because it forces the user to set up an account externally to the site they want to visit. This is unintuitive, potentially insecure, and annoying.

OpenID is worse in every way to an internal authentication system. The only advantage is the mere dream of single sign-on, which has yet to come to fruition.

</comment></comment></comment></comment></comment><comment><USER9> "Really clever sites have even found a way to bypass the signup form altogether. They slowly ask for data along the visitor's path of discovery, and pretty soon she's a member of the site, without having to fill out a form!"

Be careful with this, it can backfire. Sometimes you come across a cool site that apparently doesn't require signup, then you spend some time creating something, and then at the last minute you get a message: "Ok, now to actually save your thing you just spent 10 minutes creating you JUST have to signup"...

It's so frustrating I usually just walk away in disgust. Don't pretend no signup is needed to use a feature if it's not true.

</comment><comment><USER10> Luke Wroblewski's "Web Form Design" <http://www.rosenfeldmedia.com/books/webforms/> makes this case well. He calls it "gradual engagement." It's also one of the best books on a web design topic I've ever read.

</comment><comment><USER3> I'd love to see some numbers on "Lazy Registration". How many users does it save? How much effort goes into implementing it?

I've seen the idea before and I'm not really convinced. Sure, it sounds like a great idea, but let me offer some thoughts.

1\. I have never been to 11 of the 12 sites mentioned in the article. Before I read this article, I knew exactly one site where lazy registration is actually practised: Stack Overflow. The idea has been floating around for some one or two years now. It hasn't really catched on. To me this suggests that many authors find it difficult to implement.

While difficulty in itself is not a problem, the time and effort invested in something needs to pay off, and it needs to pay off more than investing it into another feature. Opportunity costs apply here too.

2\. How many potential users really care about registration? Does a "registration reddit-style" really put off that many people? In my experience, people will just pick their normal username and mash a laughably simple password into the appropriate box. There isn't much cognitive overhead there. This will, in my opinion, really only be a problem to people who have never registered an account before, anywhere.

Sure, you need to convince them that your product is useful. They won't explore further if they aren't convinced that it may provide some value to them. But it's probably a lot easier to put a tour on your website than to implement lazy registration.

3\. If it does indeed give more users, do the extra users remain users for the same amount of time, or do they switch more easily to competitors? What happens to the users that you would've gotten with registration as well? Do they switch more easily? It seems to me that that's a very real possibility.

Currently, my intuition tells me to just use a registration form if you need it. In places where you don't need your users to be logged in, don't require them to be. Just let them do as much as possible without an account, but I wouldn't go for "unregistered accounts". That's too much effort for something of which you don't know how it will turn out.

<comment><USER11> Sometimes it's not the fact that you have to sign up but the way registration is done.

"people will just pick their normal username and mash a laughably simple password into the appropriate box"

It's not as easy as it sounds.

1) My username might be taken 2) Often, I have to go to my email and activate the account. 3) Entering and remembering a password is not as simple as it sounds. New York Times is a good example on how not to do it: <http://twitter.com/vpdn/status/1620213111> 4) Most of the time I don't see why I have to sign up.

<comment><USER3> The NYT is a site where registration is unnecessary[1]. Registration was harder to do for them than it is to not have registration at all.

I was talking about situations where you store data on a user and link more data to a user. Sure, you can do that using some kind of token in a cookie, but if you want to give your users a name at least some of the time, registration is easier. At that point "unregistered users" add more work.

Your other examples are more a problem with the way registration is implemented, than with registration _per se_.

[1] From a technical point of view. It probably makes business sense to them, although I'm unsure how.

<comment><USER12> _makes business sense to them_

It lets them publish a rate card which includes something to the effect of "46% of our pageviews come from US zip codes with median household incomes in excess of $80,000"

</comment></comment></comment></comment><comment><USER7> The best example of this is Posterous. You just email something to email@example.com and you instantly have an account and a blog.

You can then just keep using it or you can add a password and "claim" it later to get access to other features.

</comment><comment><USER13> This is a helpful article, but I think there's a difference between easing someone towards a registration and not requiring a registration at all. Some of the companies listed seem to be the latter.

</comment><comment><USER14> With 280slides.com you can create an entire presentation and export it to PowerPoint and a number of other formats without ever registering. Only if you want to save a document will it prompt you to login or register.

</comment><comment><USER15> I really agree that this approach of gradually engaging the customer is the best way to go. The last thing a startup needs is to lose customers just because they can't use the product without going through a ridiculously long signup form.

It just makes sense: don't take the information until you need it. I found another article on A List Apart that may of interest to people who liked this article. <http://www.alistapart.com/articles/signupforms>

</comment><comment><USER16> Very good approach and something that I'd like to work toward in my projects ...

From a technical standpoint, what's the best way to tackle this? Let's say you're using Rails or Django, you let a visitor create [some-widget] and then ask if they'd like to register ... do you store a reference to the new [some- widget] in a cookie until they do?

<comment><USER17> I would treat users that have not yet registered as closely as possible to normal users. So create a user object in the database for them and keep track of the last access date. You can either mark the account as temporary, or just store an empty password and rely on that to identify accounts that are not fully registered. You can purge inactive accounts after a certain number of days. If it's only keyed by session ID then there's no point to keep this data around past the session expiration date.

Then in a few places in your app you can check if you have the necessary information on file for the current user acocunt, and prompt the user to enter it if not (things like email address, a picture, real name, whatever it is you need). When they finally select a password they become a normal user of the site.</comment></comment>


Module Counts: Languages module growth chart - <AUTHOR> http://www.modulecounts.com/

<comment><AUTHOR> Good to see Node.js and Go to be the fastest growing languages.</comment>


Hyperthreading under scrutiny with new TLBleed crypto key leak - <AUTHOR> https://arstechnica.com/gadgets/2018/06/tlbleed-a-new-way-to-leak-crypto-keys-on-hyperthreaded-processors/ <comment><USER2> Did you like the party? The bill is due.

(Most per-core speed gains in the last decade or two have been due to more speculation and cleverer caching. Now we pay for it)

<comment><USER3> Just curious, where to from here for the hardware companies like Intel/AMD/etc?

<comment><USER4> More cores, maybe. AMD is already planning to sell a 32 core chip. Even if you turn off hyper threading, it still has 32 threads.</comment></comment></comment>


IKEA Product Names Matched With Their Google Image Results - <AUTHOR> http://www.geekosystem.com/ikea-image-meme/ <comment><USER2> I'm sitting on a Moses right now.</comment>


Show HN: Marker – open source hand-drawn illustrations - <AUTHOR> https://usepastel.com/marker-illustrations <comment><USER2> I don't want to sound mean or anything, I don't think the creator of this owes anyone anything more (or at all).

I do wonder when I visit that site... is it just 10 free illustrations?

Nothing wrong with that, I just wonder when I visit if I'm missing anything.

<comment><USER3> A few similarly sparse sites for illustrations have done well on product hunt and now it's more common. Someone recently created an aggregator because a bigger library is definitely needed

<comment><USER2> Do you have a link to that aggregator?

<comment><USER3> Can't find it but it's one of the many, many, many links on this page: [https://www.producthunt.com/search?q=illustrations%20](https://www.producthunt.com/search?q=illustrations%20)</comment></comment></comment></comment>


The importance of stupidity in scientific research - <AUTHOR> http://jcs.biologists.org/content/121/11/1771.full <comment><USER2> Previous HN discussion:

<http://news.ycombinator.com/item?id=430495></comment>


Unix toolbox - <AUTHOR> http://cb.vu/unixtoolbox.xhtml <comment><USER2> The author's website is incredibly clever: <http://cb.vu/>

It's a fake terminal but it's surprisingly fully functional (lots of commands work, tab autocompletion, etc.)

<comment><USER3> "sudo is for wimps"

<comment><USER4> no sandwiches :(

</comment></comment></comment><comment><USER5> Brilliant work — my only suggestion would be to add a few more sections for "modern" tools like Git, Mercurial, Nginx, Redis, OpenVPN, Python easy_install, Ruby gems, LXC, etc.

And perhaps also a section on ancient, but widely used, systems like autotools — even something as simple as ./configure --prefix would be useful to have known...

Hmz, I wonder if the author is aware of this thread on HN?

<comment><USER6> Perfections seems to be the enemy of good. Actually I really like site format as ~50 page reference seems to be really useful, or something that you could take on a long car trip and casually look into.

I think that maybe programming and Make examples were unnecessary -- they don't really give any idea about what to do with the language and don't present any paradigms or quirks that you might need and there are much better short tutorials on that topic out there. All in all, this wouldn't save you if you got to write something in C++, but this would definitely help if you need to remember some unix command that's just slipped your mind.

Though they might have included stuff for LVM to 'file system' section, LVM commands are more or less intuitive as they are.

</comment></comment><comment><USER7> WOW. If I could, I would vote for this twice! I wish I had found this years earlier; it would have made acclimation to the command-line much quicker.

Fantastic.

</comment><comment><USER8> Interesting, although I'd have probably also put some examples for Python or Perl in alongside the C and C++ examples.

</comment><comment><USER9> Is it only me, or is this website not loading for anyone else? I've tried over the last 30 minutes of so??!?

<comment><USER10> [http://downforeveryoneorjustme.com/http://cb.vu/unixtoolbox....](http://downforeveryoneorjustme.com/http://cb.vu/unixtoolbox.xhtml) is your friend.

<comment><USER9> Ah, that's useful, thanks ;)

</comment></comment></comment><comment><USER11> i dont understand how this is a toolbox as much as it is a command line reference.

</comment><comment><USER12> Very nice overall. Thank you and kudos!

</comment><comment><USER13> thank you. thank you. and thank you.

</comment><comment><USER14> This is awesome! bookmarked!

</comment><comment><USER11> good stuff, thank you</comment>