SciPy 1.0: fundamental algorithms for scientific computing in Python - <AUTHOR> https://www.nature.com/articles/s41592-019-0686-2 <comment><USER2> > Our policy for evolving the API over time is that new functionality can be > added, while removing or changing existing functionality can only be done if > the benefits exceed the (often significant) costs to users and only after > giving clear deprecation warnings to those users for at least one year. In > general, we encourage changes that improve clarity in the API of the library > but strongly discourage breaking backward compatibility, given our position > near the base of the scientific Python computing stack.

This is one of the nuggets out of it. I have been using Python with scipy and more extensively with it's sci-kit library since 2014. Started with it by solving discrete traveling sales man problem for solution of sitting arrangements and was astonished with its ease of use of matrix algebra with these libraries and got hooked to it. It helped me to do scientific computing and create a user friendly user interface on web to help end-user to just focus on siting arrangement and conference Organization. Can solve it with Matlab or GNU octave, but couldn't have developed easily the whole system with user interface in them.

If Swift wanted to be used in scientific computing it needs to follow the similar policy of managing its scientific stack.

Julia is good for scientific computing, but it's UI and web development libraries are isn't that powerful. So will see how it develops.

I constantly see on HN criticism of Python and something rewritten in Rust as something revolutionary or ground breaking that it will change the whole world. Earlier this fad was with go language. Probably in time to come Rust will be used for some core library development which brings substantial benefits like scipy, not something rewritten as wrapper around C library with unsafe code which can be done with Python much better.

<comment><USER3> That discrete traveling salesman problem of seating arrangements ... how did you model that? I have a similar problem with organizing committees where committees have members that are in other committees and so committees with overlapping members can't meet together etc etc.

<comment><USER4> Not exactly TSP, but something similar on solving seating arrangements: it's with simulated annealing; it might help with formulating your optimization problem.

[https://news.ycombinator.com/item?id=16548012](https://news.ycombinator.com/item?id=16548012)

<comment><USER3> Danke!!!

</comment></comment></comment></comment><comment><USER5> Shameless plug:

Here is a grand tour of scipy as a jupyternotebook me and a colleague used as intro to scipy for PhDs in astrophysics:

[https://github.com/maxnoe/scientific_python_notebooks](https://github.com/maxnoe/scientific_python_notebooks)

You can try it without installing the software by clicking on the binder button.

<comment><USER6> There is also [https://scipy-lectures.org/](https://scipy-lectures.org/) :-)

</comment></comment><comment><USER7> SciPy, and especially numpy: the "free drugs"of the scientific computing world. In the best possible way.

<comment><USER8> > (...) In the best possible way.

I do not see it that way. A whole generation of students is growing with the wrong impression that writing a "for" loop is inevitably inefficient. Also, they believe that it is OK for large arrays of numbers not to be a core language construct, requiring an external library like numpy. These two incorrect beliefs are incredibly damaging in my view.

In a sane programming environment (e.g., with jit compilation), writing a matrix product using three loops should be just as efficient than calling a matrix product routine. The matrix product should also be rightly available without need to "import" anything.

<comment><USER9> Well, you got to think about optimisation - loop unrolling, parallelisation on all available cores, efficient data layout that works well with caching, doing it on GPU, adapting to the type of GPU you have. It's not as easy as dumping three for loops.

<comment><USER8> The fact that you write three nested loops on your program does not mean that the compiler/interpreter cannot do all these nifty things when running the code.

<comment><USER10> You're talking about the wrong language, then. This is not c-python (or python too perhaps) if you want such ridiculous compiler-level optimizations. If you want the languages for loops to be almost on-par with the hyper-optimized implementations being utilized in numpy and the layers below it then you should use something like C++ and hand-roll those for-loops yourself.

The reason python is so amazing in this regard is because that has all been done for you on some level and you just need to use them. Sure, could the numpy/scipy interfaces into lower-level code be more closely aligned to plain for loop implementations of the algorithms they represent? Perhaps.

</comment></comment></comment></comment></comment><comment><USER11> I remember going to the first SciPy conference at Caltech in 2002. IIRC Most of the talk was about needing better package management. What happened since is just amazing.

</comment><comment><USER12> Congratulations to all involved with this paper, an awesome accomplishment for the SciPy community-- pretty wild that it's almost 20 years old.

<comment><USER13> Good that they got a paper in Nature. It's important such accomplishments get rewarded and acknowledged in the traditional academic way. I hope these people pile up citations. The engineers who build the foundations of modern scientific research get far less credit than they deserve.

<comment><USER8> Indeed! This publication is very good news, and I am happy that free software acquires such a prominent place in the academic citation space.

On the other hand, notice that scipy is in large part an interface. The underlying solvers (e.g., numpy.linalg.solve) are often routines written in Fortran several decades ago, and already duly cited and academically recognized.

<comment><USER14> There is tremendous value that often goes unrecognised in taking best of brand routines and packaging them so real people can use them, consistently and reliably over decades.

<comment><USER8> Sure, that's why people keep doing that: taking the same exact Fortran routine and giving the world access to it from whatever different languages and frameworks are fashionable.

</comment></comment></comment></comment></comment><comment><USER15> [https://web.archive.org/web/20200207035109/https://www.natur...](https://web.archive.org/web/20200207035109/https://www.nature.com/articles/s41592-019-0686-2)

</comment><comment><USER16> I learned about SciPy via Numerical Python (Apress/GitHub: [https://github.com/apress/numerical- python](https://github.com/apress/numerical-python)). That was a ton of fun, especially after I recovered from freshman calculus flashbacks.

</comment><comment><USER17> Great read!

I'm ignorant of numerical computing, so I have to ask: is Fortran still used because it is that much faster than C or any other specialized language? I assume that is the case because I'm sure someone would have rewritten those routines by now otherwise.

<comment><USER18> Fortran still is the language for numerical computing, most algorithms in this area have been implemented in Fortran. And this is very specialized code that has been test by experts in their areas. That's why a project like SciPy cannot simply replace it with something written in C or Python.

</comment></comment><comment><USER19> May I ask what the status of parallel computation is with SciPy?

<comment><USER20> OpenMP threaded blas-routines

</comment></comment><comment><USER21> Isn't Scipy on 1.4.1?

<comment><USER22> Because it took half a year for the paper to get accepted. You can see that the manuscript was sent to Nature Methods on July 28, 2019.

<comment><USER23> For the record, July 2019 is still at least 18 months after the release of SciPy 1.0 (late 2017)

[https://docs.scipy.org/doc/scipy/reference/release.1.0.0.htm...](https://docs.scipy.org/doc/scipy/reference/release.1.0.0.html)</comment></comment></comment>


I've been snailmailing people abusing the recycling - <AUTHOR> Since most people throw their trash (styrofoam, other non-recycleables) in Amazon boxes or fail to break down their boxes in the first place... their address is right on the package! You can mail them and I do. It's funny to see the habits change. <comment><USER2> When you send these letters, I hope that you are doing the responsible thing. Are you including your return address?</comment>


See most voted/commented news at a glance on Hacker News and Reddit - <AUTHOR> https://chrome.google.com/webstore/detail/jaapjannjfdcklcgadbhiddhbelhhnkp

<comment><AUTHOR> Code available at <https://github.com/aercolino/Hype-Amplifier></comment>


Goblin Fights in D&D Are the Worst - <AUTHOR> https://kotaku.com/goblin-fights-in-d-d-are-the-worst-1822301602 <comment><USER2> This seems like its really a fault of a poor DM; at the very least you can fudge some rules to make simple combat simpler (ie goblins share initiative), and of course, avoid summoning them when the moods not right.

And you hardly have to go to the length of _explanation_ to make them more interesting: give them scouts, a simple power structure, basic tech (stick+stone spear, carts, etc) and you have more than enough justification to pull more interesting fights out them. A goblin runs, presumably informs some leader, and soon you get a swarm out. You don't need to explain this; rather its what you'd _expect_ to happen, if you saw a goblin with a scope or something.

I mean, it doesn't really matter. Its trivial to fudge the lore on the fly, and to skip dull encounters entirely. I think they still hold value in terms of flavor (territorial control and general threat of the area), and for giving players a way to acknowledge growth (compare lvl 1 fights vs lvl 10 with a horde).

Actively dreading them, I think, is the result of an overly mechanical DM. You're basically operating like a CRPG would, incapable of gauging whether a horde _should_ attack, stuck with whatever the dice rolls.

And hell, fudge the spawn roll. Interrupt the fight with a dragon. Force tactics onto them, by "accident": a group of goblins was ahead of the players, and a group behind, when they stumbled into the players. An accidental surrounding, and you keep your braindead goblins.

Worst case scenario: Claim new species and beef up their stats/int and make them as threatening as you want, while keeping general goblin lore.

They're too damn simple a monster to hate. You can do almost anything you want with them and get away with it

</comment><comment><USER3> Not to overthink it too much, but the idea that goblins are hardly more intelligent than rats (despite the game stats) doesn't really make sense in- universe. They can use weapons! Maybe they aren't as intelligent as humans/elves, but they are more than dumb animals.</comment>


Detailed analysis of Federal Circuit hearing: copyrightability is certain - <AUTHOR> http://www.fosspatents.com/2013/12/detailed-analysis-of-federal-circuit.html

<comment><AUTHOR> Fair use is still likely to be on the table however although in this particular case (unlike WINE or mono) I don't really see it as there was no attempt (or need) to be compatible with Java.</comment>


New website lets you trade Starbucks gift cards for Bitcoin - <AUTHOR> http://www.digitaltrends.com/cool-tech/card-for-coin-trade-starbucks-gift-cards-bitcoin/ <comment><USER2> I actually built a very comprehensive site that did this exact thing back in 2011, even hiring a very good designer to help me with the graphics (paid with Bitcoins -- about 60 BTC for his work if I recall). The mobile web frontend for the Bitcoins->Card side worked well enough that I would load it up and use it to buy coffee at my local Starbucks via the PDF417-code that Starbucks uses.

However, I never released due to all the regulatory issues. Very unfortunately, FinCEN still considers this type of transaction to be an exchange of value, and thereby regulated. At the time, FinCEN still hadn't released their guidelines on digital currencies, but they were increasing their regulatory stance on gift cards. The situation is still not completely clear, even now, but I'm pretty sure that FinCEN will try to claim that this type of operation a money transmitter, with all the headaches that this brings.

Also, I wasn't sure if Starbucks would try to do something crazy like sue me to take down the site. But it was primarily my fear of FinCEN that caused me to drop the idea and put the code in storage. Regulation is taking a heavy toll on this country's innovation; this is just one very small example.

Interestingly, I had no problems getting a business bank account for the idea back then. I think it'd be much harder now.

Despite spending 6 weeks part-time on a project that I ended up scraping, I don't regret it since it was my first major Python project and eventually led me to learn lots of implementation details of the Bitcoin protocol itself (though that certainly wasn't necessary for my project).

Edit: clarity

<comment><USER3> It's a shame FinCEN regs are so onerous, they're really stifling startups and ensuring that innovation will happen outside the US.

It's pretty vague, but it's easy to interpret the latest guidance to say that exchanging one digital currency for another requires a license, customer identification, suspicious transaction reporting, etc. Even if you never touch dollars, euros, yen or any other "real" currency, are you still a "money transmitter", no matter how small the amount?

[http://www.fincen.gov/statutes_regs/guidance/html/FIN-2013-G...](http://www.fincen.gov/statutes_regs/guidance/html/FIN-2013-G001.html)

"If the choice for regulators is to permit money laundering on the one hand, or to permit innovation on the other, we are always going to choose squelching the money laundering first."

[http://www.bloomberg.com/news/2014-01-29/new-york-state- regu...](http://www.bloomberg.com/news/2014-01-29/new-york-state-regulator- promises-tough-bitcoin-rules.html)

Good luck exchanging stored-value for digital currency in this environment.

<comment><USER2> A business that buys/sells gift cards for Bitcoin is a bit of a grey area I think, but I'm absolutely _sure_ that FinCEN considers exchanging Bitcoin for alts to be money transmission, regardless of the amount involved, and once they catch up with the technology, they'll start looking to make a precedent.

And yes, these businesses will simply take place outside of the U.S., taking all those jobs with them.

<comment><USER4> I have not looked into these details being discussed at all, but the "sending jobs elsewhere" angle sounds nutty. I have to assume that money transfers are also regulated in the places you're thinking of, especially if it is a developed country or a desirable place to live. I mean, you can't reasonably expect to just set yourself up as a bank without jumping through a few hoops.

<comment><USER2> You realize that some of the largest Bitcoin businesses, such as the now defunct Trade Hill, and the still operating moneero, have moved overseas to more open Latin American economies, right? Uraguay and Costa Rica seem like pretty nice places to live.

These businesses employ/have employed dozens of people, and that could easily grow to hundreds.

Also, by shutting out any chance of an complying American exchange, we de facto sent jobs to Bitstamp in Slovenia, which is growing in to a very nice- sized business, I hear.

</comment></comment></comment></comment></comment><comment><USER5> My mom was one of the few unfortunate buyers from target that had her card number stolen. The thief has purchased $500 worth of Starbucks gift cards in the last two weeks.

I don't think it is a coincidence that the thief purchasing these cards. I'm not speaking for the creators, they could easily be caught in the crossfire. That being said, this is very obviously an exploitable method of transaction specifically for a thief. No matter the intention of the project. [https://news.ycombinator.com/item?id=6175294](https://news.ycombinator.com/item?id=6175294)

</comment><comment><USER6> Considering buying Starbucks gift cards with stolen credit cards is already an activity of scammers, this seems a bit dangerous. Let alone the FinCEN issues.

</comment><comment><USER7> It looks like Starbucks gift cards sell for a roughly 20% discount on eBay†. The discount seems to be proportionally higher for higher value cards (who really wants a $400 Starbucks card?). So I guess you get a better deal _buying_ cards on Card for Coin at a 30-40% discount to face value but you get a better deal _selling_ them on eBay...

† [http://www.ebay.com/sch/Gift- Cards-/172009/i.html?_from=R40&...](http://www.ebay.com/sch/Gift- Cards-/172009/i.html?_from=R40&_nkw=starbucks+gift+card&LH_Complete=1&LH_Sold=1&rt=nc)

<comment><USER8> > who really wants a $400 Starbucks card?

Why wouldn't you? At $4/visit you should be good for 4 months if you go nearly every day.

<comment><USER7> Something something time value of money something. I'm not really sure. I was just pointing out a feature of the sale prices. I was only eyeballing at that...

</comment></comment></comment><comment><USER9> This is a nice workaround. I've been long thinking about what could be used instead of dollars to trade for Bitcoin. Of course, starbucks cards are not as widespread and their amount is probably limited (in a sense that there's not enough of them to satisfy demand in the case this seriously takes off), but it's nevertheless great to see people finding ways to bypass the stupid regulatory system, because of which there are no real Bitcoin exchanges in the US.

<comment><USER10> "it's nevertheless great to see people finding ways to bypass the stupid regulatory system"

Keep saying that when your "Freedumb Über Alles" exchange takes your USD, gives you 0 BTC, and tells you "tough shit, sucker".

<comment><USER9> But that's my choice. See, that's the problem with advocates of any regulation. Regulating != informing. If all government did was saying "This business is not reliable" I'd be perfectly fine with it. But then they actually prevent businesses from operating without licences. This creates a lot of opportunities for corruption. On what grounds do governments think they have the right to tell every person what to do with their money?

<comment><USER11> > On what grounds do governments think they have the right to tell every > person what to do with their money?

On the grounds that it's the law; those are pretty good grounds.

<comment><USER9> Slavery was also legal not so long ago. Being gay was illegal not so long time ago. Alcohol was illegal at some point. Weed is still illegal in most places these days.

Something being a law doesn't make that something justified or moral. It just allows certain groups of people to benefit from it while lying to others that it's for their own good.

<comment><USER11> > Something being a law doesn't make that something justified or moral.

Strawman, I never said it did. Grounds != moral.

</comment></comment></comment></comment></comment></comment><comment><USER12> Very clever. I just forked over a $10 card for $6 in bitcoins. I've been reading about bitcoin since its inception and finally have my own slice of the pie...

</comment><comment><USER13> Whats stopping people selling their gift cards and using them before the buyers? Or am I missing something?

</comment><comment><USER14> Starbucks gift cards strike me as an odd choice. I can't help but think this choice was made to assist in a certain type of scam: [https://news.ycombinator.com/item?id=6175294](https://news.ycombinator.com/item?id=6175294)

<comment><AUTHOR> Nah, the Starbucks choice is explained in the article. It's a remnant from the original project to help indie coffee shops accept Starbucks cards.

EDIT - I just read the article you referenced. That scam only applies to registered cards (cards attached to a Starbucks account). I don't accept registered cards, or have anything to do with that.

</comment></comment><comment><USER15> This service goes in the opposite direction, but if you're looking to cash out dogecoins into Starbucks, Visa or Amazon giftcards, check out [https://treats.io/](https://treats.io/).

</comment><comment><USER16> tried out the site (had a card with 10 bucks) and tried creating a bitcoin wallet. no luck having the bitcoins show up in my wallet yet. I have a feeling I did something wrong...

<comment><USER2> What wallet are you using? If you're using a full wallet (Bitcoin-qt), then it could take up to 12+ hours to download the full blockchain and your transactions show.

As a beginner, you'd be better off with a wallet like Electrum or Multibit that will show your transactions almost immediately.

You can also view them here on blockchain.info, by putting your _public key_ (the one that starts with a 1) into the search bar. Your transactions should show up, assuming nothing is going wrong with this site:

blockchain.info

Otherwise, just contact the owner, I'm guessing he'll be eager to help.

1\. [https://electrum.org/download.html](https://electrum.org/download.html) 2\. [https://multibit.org/](https://multibit.org/)

<comment><USER16> I was just using an online based wallet through blockchain.info.

Tried doing a search for my public key 18fyCm3vkftJ9nNgZ4SwoJkCuGVZQ2kZcE and had no luck finding the transaction.

When I get home tonight I need to check out these other wallets and get in touch with the owner. If I'm only out 10 bucks though on a gift card that I got for xmas. No sweat off my back.

*Oh, and thanks for the help!

<comment><USER2> No problem. Just in case you don't get those BTC, I just sent you some to that address to play around with or buy a little something with. But hopefully, you'll end up receiving both transactions.

Check out electrum or multibit, they're both fairly easy to use. Be sure the password-protect your wallet with a long password (write it down somewhere!). If you do continue to use blockchain.info be sure to enable 2-factor authentication and use a strong password.

<comment><USER16> I did get the BTC. Thank you so much for you generosity. I have downloaded the Bitcoin-Qt and is downloading the block chain now.

I'm going to check out electrum and multibit next. Thanks!

<comment><USER2> Great! Glad they both came through.

I need to make one correction to what I wrote above. In my haste, I accidentally called your Bitcoin address your "public key". Actually, your public key is a related but different concept. Properly said, the string that starts with "1" is just a Bitcoin address.

Anyway, just wanted to correct myself that those are two different but related concepts. This has no impact on your use of Bitcoins -- just use the one that starts with "1" to give out to receive Bitcoins. Most programs won't even show you the public key I'm talking about, so no need to worry about it.

</comment></comment></comment></comment></comment></comment><comment><USER17> How do they make money? They buy lots of coffee for themselves?

<comment><USER18> They can sell the SBUX gift cards on a gift card exchange (Cardpool) or eBay for ~25% discount and make the spread.

</comment></comment><comment><USER19> Would Amazon giftcards work?

<comment><USER20> They would but easy to fraud, because you could use stolen cards to buy them online. Then the guy who trades you for Bitcoins screwed when amazon cancells all the cards. If you can also buy starbucks cards online with visa/mc too then this whole venture is doomed as well.

<comment><AUTHOR> Yeah I'd like to try Amazon but the more liquid cards seem to be more fraudulent.

<comment><USER20> Create vouchers in 711 you can instantly redeem for bit/altcoins. Bitinstant did this but was badly implemented. Unsure if msb required. Some millionaire prince could make it happen. Probably issues with it being able to use for online gambling too considering paysafecard is forced to sell a crippled version in the US</comment></comment></comment></comment>


HyperDex: A Searchable Distributed Key-Value Store - <AUTHOR> http://hyperdex.org/performance/

<comment><USER2> From the FAQ <[http://hyperdex.org/faq/>](http://hyperdex.org/faq/>):

"So, the CAP Theorem says that you can only have one of C, A, and P. Which are you sacrificing?

HyperDex is designed to operate within a single datacenter. The CAP Theorem holds only for asynchronous environments, and well-administered datacenters enable us to sidestep this tradeoff entirely."

I'd like to see how they pull that off when a node goes down. I guess in "well-administered" data centers, nodes don't go down.

Sounds like they're sacrificing "A" to me because they're doing synchronous replication.

<comment><USER3> It's a bit of a middle ground. Yes, the replication is synchronous, which impacts availability. However, the master can remove a failed replica from the chain fairly quickly. In principle, with proper tuning, a node failure would merely cause a brief hiccup. This would feel more like a period of increased latency than a full-blown outage. So there really needn't be much sacrifice of availability.

However, there's also a sacrifice of partition tolerance. If the master is unable to communicate with any replica, the system can't serve requests. Also, the master is implemented as a collection of Paxos nodes; if these nodes are partitioned from one another, the entire system would grind to a halt.

Since this is intended for intra-datacenter use, one could argue that a full network partition might be unlikely. (Depending on what sort of data center you hang out in.) But in CAP terms, it's possible, of course.

(I base all this on the value-dependent chaining paper cited below.)

<comment><USER4> This sounds like a CP system to me. There's nothing wrong with that btw, I don't know why people are so reluctant to admit this.

AP systems have some useful properties, but they're also (typically) more difficult to reason about.

The "hiccups" you describe are periods of unavailability. The increased latency is caused by an element of the system waiting for the data to become available again, a totally valid strategy for coping with transient failures/partitions.

Your argument about intra-datacenter partitions being unlikely are true, but they do happen. You also make a good point about such partitions also affecting client applications. Both of these are indicative of CP systems and, like I said: there's nothing wrong with that.

Personally, I think both AP and CP distributed systems are equally interesting. What I consider a red flag is attempting to rationalize how a system "beats CAP".

<comment><USER3> To be precise, I would argue this is actually a CA system.

"C" because there are consistency guarantees, which are upheld even in the face of failures / partition.

"A" because the system will continue making progress even after a node failure. What I called a "hiccup" can be made arbitrarily short, in principle at least. The system can work around failed nodes, it does not need to wait for them to be repaired.

Not "P", because if the network falls apart sufficiently, the system will generally not be able to progress.

<comment><USER5> There is no CA.

You cannot guarantee consistency and availability simultaneously in the face of network partitions. Once the line of communications is cut or overloaded (slow enough = a partition), you have to pick one or the other. It's basic physics.

If two entities can't communicate, they can't synchronize state, so one (or both) of them have to quit acting like they have a consistent view of the data.

<comment><USER6> It's not exactly clear from the paper what CA should mean.

I've seen people claim it means "you guarantee both consistency and availability, as long as there are _no_ network partitions (you don't have to handle those because you haven't chosen P)". That's a supportable claim. So you can do that, but it's kind of a useless choice, because as long as there are no network partitions, both CP and AP systems can _also_ guarantee full consistency and availability.

I lay the CAP family out thus:

* CP: on network partition, lose availability

* AP: on network partition, lose consistency

* CA: on network partition, lose both

It seems to be a common thread among distributed systems engineers who claim to have beaten CAP: "network partitions don't matter for whatever reason, so therefore I can always guarantee both availability and consistency and so CAP must be wrong yaaay!!"

Sorry, no, nice try.

</comment></comment></comment></comment></comment></comment><comment><USER7> This is a pretty nice implementation of an old design concept that has merit for modern distributed problems. It is worth studying as a model for distributed systems.

I am aware of a couple commercial distributed systems that use it (not this implementation but the underlying algorithm). Several organizations seem to have reinvented it over the last five years.

Organizing data in this way was studied in the 1980s but was poorly suited to the computing systems of the time. Relatively little was written about it because it was viewed as a dead end and modern literature has all but forgotten about it. Most of the research was done by companies rather than academics. Designs like this have fallen into the blackhole of "if it is not on the Internet then it doesn't exist". Back when I was studying these models I found more crusty old patents related to these types of models than relevant papers on the Internet. It will be valuable to have some modern literature pertaining to these designs.

</comment><comment><USER8> For those that want to take a quick look at the source code without physically cloning it, there's a GitHub link buried somewhere in the site (I forget where I found it): <https://github.com/rescrv/HyperDex>

Their Python client seems to be using Cython for extra speed: [https://github.com/rescrv/HyperDex/blob/master/hyperclient/p...](https://github.com/rescrv/HyperDex/blob/master/hyperclient/python/hyperclient.pyx)

</comment><comment><USER9> I'm looking for a good paper or a book detailing patterns or strategies of storing complex data structures and relations, or designing complex schemas on top of key/value stores, possibly with real-world use cases / success stories. A bonus would be the same on top of key/value/range(search) stores, just like HyperDex is. Any suggestions HN?

Technically, it should be even possible to implement SQL on top of a store like HyperDex (there's the Tenzing paper from Google). But I'm not looking specifically at SQL, but at more general scope.

</comment><comment><USER10> It looks like they trade the ability to scan ranges of keys for the ability to get single objects via multiple attributes. The value-dependent stuff is also a neat way of solving the consistency issues with multiple node updates. Interesting stuff.

If you were to swap this with your Cassandra cluster, you'd be losing multi- datacenter replication. Although partitions within a data center are pretty rare, you'd also lose some availability there as well. However, Cassandra is usually hash-partitioned so it needs to do broadcast for a scan (AFAICT, it even needs to do a broadcast for a lookup on a single secondary attribute), so you'd probably gain quite a bit of performance with HyperDex.

I can't tell if it's possible to dynamically change the set of secondary attributes being indexed without rebuilding the entire data set. Or how value- chaining works with missing attributes.

Also, apparently consistency has some... gaps... when you search via a secondary attribute:

"The searches are not strongly consistent with concurrently modified objects because there is a small window of time during which a client may observe inconsistency."

</comment><comment><USER11> I'm reading through the paper, and I'm curious if I understand the implications of key subspacing and value dependent chaining correctly - do all reads and writes for a given key get forced to a single node? I understand how the replication that's described allows for failover when the point-leader fails, but does it also allow for scaling writes and key lookups?

<comment><USER12> Reads and writes for a given key always go to a single node. As you add more machines with the same workload, you are handling fewer keys per machine. The flip side is, if the number of keys per machine stays constant, more machines directly translates into the ability to handle more keys.

HyperDex scales linearly as you can see in our scalability graph.

<comment><USER11> I think you're missing the point of the question. I'm not asking how you scale in general, I'm asking whether your replication scheme is only for failover or if it contributes to your scaling story - i.e., if a given key goes hot, does everything have to run through the point leader, or can each replica take writes/serve requests? It sounds like the former, which is fine, just wanted to clarify.

<comment><USER12> It's the former. I'd rather not sacrifice our strong consistency by using replicas to serve GET requests.

</comment></comment></comment></comment><comment><USER13> How does it compare to elasticsearch?

<comment> k-a-r Elasticsearch is a Fulltext search solution not a K/V Database.

<comment><USER14> Technically yes, but you can use elasticsearch as a distributed key-value store. I use it in production and it works amazingly well for me. Yes, searches may return stale data (one second old by default) but that is not much of a problem in my app.

<comment><USER15> That sounds both interesting and non-intuitive. Can you go into more detail about how you have been using ElasticSearch as a k/v store?

EDIT: Color me surprised. I didn't see it mentioned at all in the ElasticSearch docs or in their Github repo, but apparently this is a fairly popular thing to do.

"Elastic search as a database?" [http://stackoverflow.com/questions/6636508/elastic-search- as...](http://stackoverflow.com/questions/6636508/elastic-search-as-a- database)

"ElasticSearch, datastore for user and social data?" [http://stackoverflow.com/questions/8190288/elasticsearch- dat...](http://stackoverflow.com/questions/8190288/elasticsearch-datastore- for-user-and-social-data)

"Elasticsearch and NoSql database" [http://stackoverflow.com/questions/8026880/elasticsearch- and...](http://stackoverflow.com/questions/8026880/elasticsearch-and-nosql- database)

</comment></comment></comment></comment><comment><USER16> Did you compare the performance with (local) BerkeleyDB?

Also, is there a locking mechanism?

I'm the author of Torrus (torrus.org), and BerkeleyDB stability and non- network nature are quite painful. But I'm relying on its speed, concurrent locking, and some ways to acquire an exclusive lock on a table. It would be interesting to offer an alternative backend for torrus.

<comment><USER12> I'm not sure exactly what you're looking for, but Berkeley DB is embedded in your application on a single host, while HyperDex is distributed. It sounds like that's exactly what you want.

What do you use exclusive locks for?

<comment><USER16> I use exclusive locks where multiple processes concur for data access, and these locks guarantee the data consistency.

For example, the GUI engine may eventually start cleaning its old cache, and the exclusive lock prevents other GUI processes (which are mod_perl or fcgi processes) to do the same.

another example is when the database is big enough, and when multiple collector processes start at the same time, I use the exclusive lock to let them initialize and slurp the data sequentially: this prevents from excessive disk seeks.

but as you mentioned in another thread, your solution is locked to a single architecture, and I cannot afford that.

</comment></comment></comment><comment><USER17> Is it open source? I browses around the site and FAQ and didn't spot a mention of the license.

<comment><USER7> The code is open source. However, they applied for patents on the algorithm:

<http://www.wipo.int/patentscope/search/en/WO2011137189>

<comment><USER9> While waiting for the answer, what does this generally means if the patent is granted? Since the code is open-sourced under BSD, can the algorithm be used in other open-source BSD-licensed products, and can corporations then use these products to store data, without needing to pay the royalties on the patent?

<comment><USER18> The BSD license does not contain a patent grant, so it's technically possible that you'll have to pay to use this software in the future even though it's under the BSD license. I'd really like to see a switch to the Apache license, which does contain a patent grant.

<comment><USER9> So if I release my code under Apache license, and later it is found that it happens to infringe on a patent, am I liable to pay the damages on behalf of anyone using my code and getting sued based on the patent infringement?

Or, to put it another way - what is the best "free" license for a developer to release his/her code under? I don't really even care about mentioning my name in derivative works.

<comment><USER18> I'm no expert, but as far as I'm aware the Apache grants the user a license for your relevant patents. The GPL3 goes a step further and grants the user a license for any patents you have a license for whether its your own or one you licensed from somebody else. Since neither situation likely applies in your case, the BSD and Apache licenses are basically functionally identical. However, by choosing the Apache license you are telling potential users that you don't have any "sleeper" patents, so you might get better uptake.

If somebody later asserts a patent against your code, both you and your users are screwed no matter what license you choose. But the choice of license might influence who might help you out if that happens...

</comment></comment></comment></comment></comment></comment><comment><USER19> I think maybe only certain versions of Ubuntu are supported? It won't install because I don't have a repo that provides libgoogle-glog0

<comment><USER12> The packages are built for Ubuntu 11.10. What version of Ubuntu are you running?

<comment><USER19> Thanks for letting me know the target version. I am running 10.4.4 LTS

</comment></comment></comment><comment><USER20> Since they compare it with redis, I wonder if this can handle data bigger than memory. The other properties seem nice :)

<comment><USER12> We compared it to Redis as it is one of the many key-value/document-database systems that provide the high throughput and low latency properties that make it comparable to HyperDex. We are expanding the systems we benchmark against, aiming to compare against as many systems as possible.

HyperDex is designed for bigger-than-memory data.

<comment><USER21> The comparison with Redis is a bit strange at a first glance, what I mean is, if in Redis I replace the actual fetching of data with "return 'foo'" at every query it still returns 150k/requests per second for every operation. The same will do memcached _per single core_. So either in your code there is some good trick to improve speed in the networking layer, or you are comparing multiple cores/nodes to single core/node.

About benchmark "E", this is a primitive that Redis lacks, so it is in my opinion a bit misleading.

Another question is, in the benchmarks with Redis, does the data set fits inside Ram also in HyperDex DB (or OS cache)? In such a case you are comparing two in memory databases in practical terms, but at the same time you say that HyperDex is suitable for datasets bigger than RAM (in a portion of text _near_ the Redis benchmark), but the numbers that the users will experiment in actual production setups will be different if the data set does not fit in RAM.

Finally, in the benchmark it is not clear that Redis can perform a lot of work in a single operation, for example a variadic LPUSH can insert 10 elements inside a list in the same time an LPUSH does this with 1 elmenet, more or less. If you multiply this for N cores in a stand alone Linux box you get 1 billion operations per second or alike, but I would be not comfortable with writing this in the Redis home page. I mean, if you don't specify _very_ well the methodology this unfortunately is just marketing material without real interest for the field.

<comment><USER12> Both systems were single process per host.

I agree that Redis does not have good support for "E", so anyone looking at the results should look at A-D,F.

Both systems performed entirely out of main memory to avoid touching disk.

The Redis benchmark results only came in last night. We'll be writing up the methodology soon. We used the YCSB Redis binding from branch master. I'd be interested in your opinion on how well the bindings map to what is truly best for Redis.

<comment><USER21> Redis uses a single thread, not just a single process, so if Hyperdex is multi-threaded you are comparing single core vs multiple cores. As you can see with memcached that instead is able to use multiple cores (a feature that Redis is going to implement soon) this leads to a big performance improvement in this kind of benchmarks.

EDIT: (I checked that Hyperdex actually uses threads and multiple cores) If you want a fair comparison you should run Hyperdex on a single core as well, or you can run N instances of Redis (one per number of core) and write the benchmark so that it uses all the instances.

I'll check the YCSB Redis bindings, I never looked at them before. Thanks for the reply.

EDIT2: there are also problems with the YCSB Redis bindings:

1) It basically forces an object-store data model on Redis, so only uses hashes to store objects, and every time an object is stored or deleted, a sorted set is updated as well. This is a possible use case of Redis but not a very idiomatic / representative one.

2) Even for benchmarks not involving searching, the sorted set is anyway updated.

3) There is no pipelining used to alter the object and store the sorted set. Every operation pays 2x the Round Trip Time in the Redis bindings.

4) Even worse, there is no pipelining in the "search" operation, so you may the RTT a lot of times when you do a scan operation with this bindings.

The minimal change to the YCSB bindings is to modify the bindings to use pipelining when possible (almost always, actually). Still I think that an intermediate layer to turn Redis into an automatically-indexing object store does not make sense. Another big problem is that you are comparing multi-cores vs single-core.

So if you really are interested in a comparison between HyperDex and Redis you should pick an use case and model it accordingly with the best tools of both the databases.

What I would recommend is to use the following use cases.

1) Populate the two DBs with 10 millions of hashes, then write a benchmark where 50 clients simultaneously get and set specific fields.

2) Like "1" but increment a field by 10 at every write query.

3) Simulate a leader board where 50 clients simultaneously update the scores of the different "players" in one operation, and ask the top 10 users in the leader bord with another operation.

4) Simulate a capped collection where you always add the latest news in a web site, and you can get the top-10 to show in the home page. Every of the 50 clients should write a single item and fetch top-10 items.

And so forth. Always use 10 million objects and mixed reads and writes with 50 clients at the same time. Write the best code for both the DBs (I can help with Redis).

This time you are truly comparing the two DBs in a real world scenario.

<comment><USER22> I think YCSB is meant so that experts in each technology can tweak it for their system. So, it looks to me like you're in a great place to make that happen, and I'd love to see it!

Also, It's extremely difficult to get speedup of 4x on 4 cores, so I'll believe that argument when I see it. It seems to me that with the current implementation of Redis, you'd run into some serious problems with memory management if you run 4 redis nodes on one 4 core machine.

Also, you say: "This is a possible use case of Redis but not a very idiomatic / representative one." Can you elaborate? What is idiomatic? From the front page of the Redis site: "Redis is an open source, advanced key-value store."

Also, on a slightly different topic, I can't seem to find any real documentation about the consistency guarantees of Redis, so I thought you might be able to point me in the right direction. It appears that the master/slave replication scheme in Redis just backs up onto the slaves eventually, but the master immediately returns. Is that true? If a master goes down (disk and all), could a write have been confirmed to the client which is missing on a slave? What replication protocol does it use? I find the documentation lacking in this regard and it'd be great if you could point me to some actual technical specifications. Thanks!

</comment></comment></comment></comment></comment></comment><comment><USER23> This reminds me of <http://xanadu.com/zigzag/> .

<comment><USER12> HyperDex is different. ZigZag focuses on data visualization. HyperDex provides you the same key-value interface of other systems, while also providing an efficient search primitive.

Check out <http://hyperdex.org/tutorial/> for examples of the Python API.

</comment></comment><comment><USER24> Is this better than hadoop?

</comment><comment><USER16> the FAQ mentions that you target x86_64 architecture. Does the server work on i386?

also, I hope it's doing proper memory alignment and endiannes independence? (because Mongo sucks on that)

<comment><USER12> We use many lock-free datastructures which rely heavily upon the x86_64 architecture. Further, the expanded virtual address space enables us to mmap everything.

All network traffic is packed, and in network byte order.

<comment><USER25> > the expanded virtual address space enables us to mmap everything

This sounds like a disaster waiting to happen when a node's working set is larger than RAM. Have you considered the impact on performance due to excessive I/O resulting from this kind of overcommit?

Redis performance, for example, suffers tremendously when its database size exceeds the available RAM, which is why the authors advise implementors to cap its database size. The only real difference I can see here is that Redis allocates its storage anonymously, while Hyperdex uses file-backed pages; in either case, performance under memory pressure will largely be governed by how the VM chooses to move pages in and out of its backing store -- behavior the application has no control over.

Optimizing the performance of working sets larger than RAM is _hard_. Redis had a (thankfully) aborted attempt to do so (the VM is gone in the most recent stable release); and the InnoDB buffer pool in MySQL has been refined for many years and is subject to quite a bit of tuning for specific workloads. (See also <http://blog.kennejima.com/post/1226487020/thoughts-on-redis> for some thoughtful discussion on the subject.)

You claim on the one hand that Hyperdex was designed to work on data sets larger than RAM, but on the other hand you admit that all the benchmarks were performed with a working set smaller than RAM. I'd like to see comparative benchmarks where the working set is larger than RAM to be convinced.

<comment><USER26> > This sounds like a disaster waiting to happen

Well, define "disaster". A memory-mapped dataset will cause pathological performance only when there is thrashing: If you are accessing a small percentage of the entire dataset, then unused data will be paged out and remain paged out. If the bulk of data being accessed exceeds the amount of available physical RAM, then you will get I/O trashing.

Memory-mapping is a good alternative to static allocation or a home-grown paging system because it lets the kernel handle the dynamics of allocation, letting your application transparently and gracefully handle RAM tension situation by relinquishing memory space to other apps. Kernels (including Linux and Windows) and CPUs are extremely efficient at paging I/O, much more efficient than a hand-written paging system because there's no need for the application's code to check whether a page is in physical memory — that's handled by the CPU itself.

Of course, any I/O incurred by a too-large dataset will drastically reduce performance compared to in-memory speed. But paging in itself does not necessarily lead to "disaster".

<comment><USER25> Did you read my entire comment? I specifically said, "when the working set is larger than RAM."

Your observations, while true in the abstract, don't reflect real-world behavior under these conditions.

<comment><USER22> Working set larger than ram doesn't imply thrashing. It all depends on the page replacement algorithm employed by the OS, and the applications that are running.

</comment></comment></comment></comment></comment></comment><comment><USER27> If you're building a distributed KV stores, you should benchmark against other distributed KV stores. Mongo and Cassandra aren't really. But Riak is.

Plus, since its "distributed" here's the benchmark I'd like to see:

1\. Set up a cluster of 8 nodes. Set data replication to 3. 2\. Load 3TB of data into the cluster, across 1M documents (or some data set of that order) 3\. Run your tests. Optimize each of the DBs for the best way to access them (Eg: link walking vs. map reduce on Riak if that's faster, or secondary indexes if that's faster, or Riak Search if that's faster-- there are many ways to search Riak.) 4\. Throw out the results of #3. 5\. Pull the plug on 2 nodes. EG: Shut down completely, no longer on the net work at all. Poof, gone. Pick the two nodes by rolling the dice. If you have SPFs, and a bad role of the dice would have brought the whole cluster down, remove the word "distributed" from your marketing. 6\. Let the cluster sit for 30 minutes. 7\. Run your benchmarks.

The benchmarks developed in #7 are the ones I want to see.

I have no clue how HyperDex would perform in this situation. It could kick Riak's butt (but then, small cluster performance is just one of the criteria that is important to me). I just wish people did benchmarks like this (though I know its a PITA to do it this way.)

<comment><USER28> Why would a competitor build this Riak benchmark? How are they supposed to know which Riak configuration performs best?

That's the point of YCSB. Each vendor can submit the optimal configuration for their system, and they all run the same benchmark.

At the end of the day, each vendor is going to publish benchmarks that show their system performing better than all others.

It's your job, not theirs, to verify those benchmarks for your particular work load.

Edit: Also, while Mongo wouldn't classify as a distributed store, Cassandra definitely would. In fact, it's more "distributed" than Riak, since you have to pay Basho for multiple DC support (unless that's in the open source version?), whereas Cassandra is completely free.

<comment><USER29> If things haven't changed in the past year you are right about not being able to do replicated multiple DC support.

</comment></comment></comment><comment><USER30> Has it been proven in a pr0n environment? If not, I'll wait and see.

</comment><comment><USER31> Interesting.. keep it coming.

We need a unified nosql language.. Basically what SQL is.

<comment><USER32> What's wrong with SQL?

<comment><USER33> * COBOL-inspired syntax that is neither human or machine parseable (easily).

* Strange nested query rules

* Language encourages Cartesian products for virtually every non-trivial calculation. This makes join order crucial for reasoning about performance.

* Humans don't have the brain capacity to reason about these joins, forcing the logic onto the query planner.

* Humans don't have the brain capacity to reason about the query planner reasoning about the joins.

* End result: bizarro-english syntax that's supremely hacked-up to twerk the query planner to do your bidding.

<comment><USER34> What? SQL is virtually _plain English_ and is efficient across a range of database architectures for moderate-to-complex join situations. Its just an implementation of Boolean Algebra with a thin abstraction layer that uses words like SELECT, JOIN, FROM, WHERE, etc.

If you have performance issues, its invariably because the structure of your data is less than optimal or you have many linked external tables. Or -- you are _really_ bad at efficient query design. Even highly experienced SQL-ers always look at old queries and realise that they could be three times faster and half as big, and if there is a performance issue then that's the first thing they'll review.

Humans have been 'understanding' SQL very well for nearly forty years -- without modern refinements like graphical query designers, either.

Not all data is a suitable candidate for an RDBMS solution, I am the first to agree, but there's not actually much wrong with SQL when what you need _is_ a relational table store.

There are a few bad implementations of it, though, it is true, and a lot of people who can't seem to grasp SQL for what it is.

Edit: I am myself a _huge_ fan of NoSQL solutions and always seek to rationally justify a non-relational datastore solution where _practical_ or _possible_. NoSQL isn't a replacement for SQL (with all its imperfections) for the kind of slicing and dicing where it _is needed_.

<comment><USER26> SQL is a very, very poor implementation of relational algebra. For example, the output of an SQL query is not even a valid relation variable (in E. F. Codd's terminology).

If you want to look at how SQL might have looked if it had been relational, take a look at "Tutorial D", the language proposed in a book called _The Third Manifesto_ , by Hugh Darwen and C. J. Date: <http://thethirdmanifesto.com/>. There are several experimental implementations.</comment></comment></comment></comment></comment>


Covid Germany Anomaly, Why? - <AUTHOR> Why does Germany have such a low number of serious/critical covid patients? The last number I saw was 23 which is nuts compared to a country such as Spain with a similar case count. Water, beer, schnitzel? What's the magic there? <comment><USER2> German resident (md) here. The 'official' explanation is that Germany did lot of testing early in the course of the epidemic. We have many cases in the healthy population resulting in less severe symptoms. The older population is still not so much affected. A significant rise in death-counts is expected in the next two weeks. Hospitals are planing for the worst. Stay safe!

<comment><USER3> I keep seeing that "next couple of weeks" phenomenon. At what point can we be assured that we crossed those ever-critical couple of weeks? It keeps pushing out.

I am thinking that Corona is killing the same people who would have died from regular flu for the most part, i.e. it's not in addition to normal deaths. I understand that this is a minority opinion, but data may bear it out eventually.

<comment><USER4> Look at the 'worldometers' that someone else posted. You will see sloping lines there. You can judge for yourself when it has actually hit.

Because of a lack of ICU facilities, governments are hoping to push the critical point out as far as possible.

<comment><USER5> except virus != ill != death

</comment></comment></comment></comment><comment><USER6> The cases and deaths counts aren't informative enough to really say anything useful about what's going on. When a "case" is reported, we don't when the symptoms started occurring, or even when the original sample was collected. So you can't really draw any conclusions from this "consumer grade" information. You'd have to study it properly, and look a sample of individual case progressions.

I think the CFR stat is especially misleading, and we shouldn't be trying to compare it. For instance, if a country simply has a longer lag between testing and reporting the tests, the CFR will look higher. CFR will also look lower if the growth rate is high.

I made a little hypothetical to illustrate this.

Imagine a disease where your skin turns bright green on day 8 and you drop dead on day 30, without fail for all infections. It's growing 25% a day. What's the "CFR" at day 100?

In this hypothetical, all infections are symptomatic, all symptomatic cases are detected, and all symptomatic cases are fatal. But people don't drop dead immediately -- it takes 22 days, and at 25% a day the number of cases grows 135x in that period.

So even though this fictional disease is 100% fatal, the "case fatality ratio" will stay under 1% all the way through the growth phase.

In summary: those stats aren't really evidence that Germany's outbreak is progressing any differently from anyone else's. The information is much too vague to say one way or another. Instead the baseline assumption is, Germany has the same virus as everywhere else, and is applying the same treatments. So for any individual patients, the outlooks will be the same here as anywhere else that still has ICU beds and ventilators available.

</comment><comment><USER7> From what I understand, countries like Germany are only reporting deaths with no co-morbidity, whereas Italy anyone that was positive with Covid19 in their system and dies is being reported as a death.

The lack of standardized reporting is odd considering the impacts. It definitely seems to be contributing to the general confusion on decision making.

<comment><USER8> So the Italian numbers are inflated (do we know how much?) and the Germany numbers are more accurate?

<comment><USER9> Hard to say, but the issue (the one OP talked about) is BEFORE counting deaths.

Right now the same source (for Germany):

[https://www.worldometers.info/coronavirus/country/germany/](https://www.worldometers.info/coronavirus/country/germany/)

Has 23 "Serious or critical" people out of 41,324 cases (and the remaining 41,301 are in "Mild condition").

In Italy:

[https://www.worldometers.info/coronavirus/country/italy/](https://www.worldometers.info/coronavirus/country/italy/)

3,612 critical out of 62,013 total (6%).

In Spain:

[https://www.worldometers.info/coronavirus/country/spain/](https://www.worldometers.info/coronavirus/country/spain/)

3,166 critical out of 46,406 total (7%).

In France (seemingly much worse than Italy or Spain):

[https://www.worldometers.info/coronavirus/country/france/](https://www.worldometers.info/coronavirus/country/france/)

3,375 critical out of 22,511 total (15%).

Now France's anomaly (compared to Italy and Spain) could be an undercounting (by 2x) of total cases, but the anomaly in Germany could only be explained if the actual number of cases in Spain and Italy are undercounted by 100x!

There must be _something else_.

<comment><USER3> No, you have arrived at the answer.

The counts are wrong by orders of magnitude.

Now follow through with what that means.</comment></comment></comment></comment>


Millions to be made in Unbundling Reddit - <AUTHOR> https://realrealgood.substack.com/p/unbundling-reddit-how-was-discord <comment><USER2> Reddit has the problem where the most popular subreddits (r/gaming for example) have generic content upvoted the most, and so there often isn't actually anything worthwhile there if you have an actual interest besides memes or general industry news. Then on the other extreme, you have overly niche subreddits for specific video games, where most of the content is just fan art, overly in-depth subreddit drama, or similar memes now just re-skinned for that specific game. There isn't really a middle ground for a lot of these categories, for the people who want "just enough content for the games I like". I've found either subscribe to r/gaming and get a broad amount of nothing, or subscribe to a list of the games you enjoy and get a very in-depth lot of nothing.

</comment><comment><USER3> I guess the trick here is how do you make a site that is so much better than a subreddit that anyone is willing to manually click over to your site instead of consuming the subreddit's content along with the rest of their reddit feed.

<comment><USER4> There are many ways. If you take the pc hardware subreddits for example, providing tools to easily build a pc spec and share it would already make it more useful than Reddit.

Goto some of these subreddits, and you'll see the hoops these people jump through to take part in the community. Definitely need to start solving stuff like this.

<comment><USER5> I think you just described [https://pcpartpicker.com/](https://pcpartpicker.com/)

<comment><USER4> How do I do that in about 60 seconds and share it on a Reddit-like site, with the possibility of people recommending alternative parts in a easy way?

I guess the point I'm making is if you want to improve on the subreddits, simple stuff like this is low hanging fruit.

Edit: I'll add, pcpartspicker is exactly the kind of hoop I'm talking about. A large amount of those hardware subreddits are built on that tool, so why funnel all the traffic to Reddit at all?

<comment><USER6> two major reasons:

reach: get enough upvotes in Reddit and you can attract a lot of people including prominent figures in the space to look at your build

friction: people passing by might not want to create a profile just to comment on your build, but if they stumble on the build on a site where they already have registered it's so much more likely for them to speak their mind

a mass of users, even from unrelated topics, carry non linear value in social interaction based sites, and so it happens grouping topic together wins over fragmenting over multiple small sites

</comment></comment></comment></comment></comment><comment><USER7> Interesting article. Definitely something to think about.

</comment><comment><USER8> Thanks for sharing!</comment>


Ask HN: Feedback on our concept?  "Instant Chat" when you're in a retail store - <AUTHOR>

We're trying to solve the problem of a) needing help in a store and b) not being able to find an employee.
Specifically, we want the service to handle both a) where is x? and b) what do I need to accomplish y or to make z?
http://www.helpping.co/
You're lost in the store, but there's a "magic" button in front of you.
Would you use it? <comment><USER2> I don't mean to sound cynical, but are you sure this is a problem worth solving?

Yes, it's a problem, but it's along the lines of my microwave getting dirty over time, or needing to fill a car tire occasionally. I don't need or care about a service that only cleans microwaves or only fills tires.

If you want to improve on brick-and-morter, feet-in-the-store retail problems, why not, say, show me a big pinterest-style list of everything in every grocery store around me? That would save me some time instead of perusing up and down aisles and not being sure whether I should visit that farmer's market that _sometimes_ has interesting stuff.

<comment><AUTHOR> You make a good point, and candidly, we're not sure. The crux of the problem as we see it is that:

\- If you enjoy shopping, you're probably pretty good at it, and don't need a lot of hand-holding.

\- If you don't enjoy shopping, then a store is a bad solution. A better solution is "don't go to the store at all, we'll ship you the stuff," aka Amazon

What we're really trying to do is find the segment of the market that cares a lot about this? i.e. Willing to go shopping, if stores can just make it bit easier when you get there.

Thoughts?

<comment><USER2> > What we're really trying to do is find the segment of the market that cares > a lot about this?

Yes. Go do that. Like, right now. Find people you don't know who you think are your target demographic and ask them. They're probably in the grocery store right now.

You'll come back thinking (A) that nobody really this problem, (B) that everybody has this problem, or that (C) nobody thinks they have this problem and you need to show them that it is a problem and that, once you do, you'll change their life forever. B is probably a little delusional. C sounds really time-consuming and expensive.

<comment><AUTHOR> Yeah. Thanks for that. We've done some of this and we're seeing (a) or (c).

We've kept going because we "feel" that there's something to be "fixed" inside a store.

For example, discovering new products inside a store is really painful. Typical "big box" stores are >100,000 sq ft with >100,000 products.

Want to find a "healthy snack that taste like wheat thins and goes well with cheese"? Well, we have a wall of a 1,000 products. What you're looking for might be in there somewhere.

What do you think of this related problem?

<comment><USER2> I'm no expert, but I'm very sure that the difficulty in finding items in a retail store is by design.

If every customer walked in, grabbed exactly what they needed without looking at anything else, paid and left, the stores would probably make half as much money. It's common knowledge that "big brand" items in a grocery store go on the eye-level shelves because the distributors have paid more, and I remember reading that the ends of the aisles ("caps" maybe?) are super-primo real estate - that's why the ends always have Pepsi and Tostitos and big-name items. It's probably the same reason there are "impulse items" in the aisle when you check out. Oh, hey, yeah, I _do_ need some gum, why not buy a pack.

<comment><AUTHOR> Yeah -- It's definitely by design.

It's also by design that milk and bread are in the back corners of the store.

But, ecommerce makes this disdain for the shopper unsustainable. 10 years ago, if you're store sucks, you don't really have a choice. Now, if you're store sucks ... well I can buy a majority of the stuff I need through Amazon, Diapers.com, etc.

So, could HelpPing be a timing play? Helping change stores right when they need to change ... so that they can survive to 2020?

This sounds too theoretical even as I type it

<comment><USER2> I don't think the grocery store will change anytime soon. :)

</comment></comment></comment></comment></comment></comment></comment><comment><USER3> There's no app/service for this already? This is a pretty good idea.

I like the "chat" aspect especially since (on the backend) it could be taken care of with machine learning & automation. Most of the time looking for a store employee I just want to know what aisle product X is in.

As a third party service for smaller stores and a white label solution for larger chains, it sounds like a nice business model.

<comment><AUTHOR> There are apps that handle different parts of this, e.g. Aisle411 for product location, AisleFinder, etc.

But, in our early tests, what we found is that shoppers quickly move from "where is x?" to "what do I need for y?"

Our general hypothesis is that you need both machine learning / automation + human agent to create enough value for the shopper that they want to come back

<comment><USER3> _> But, in our early tests, what we found is that shoppers quickly move from "where is x?" to "what do I need for y?"_

I figured that would be the case. I imagine that there would be a "HelpPing administrator" (similar to live chat support on websites) that could a) jump in to assist on more complex questions and b) assist in mapping those complex questions into the system algo?

I like the idea. Get out there and do it. I hope, though, that the button isn't gonna look like the "easy button" from Staples ;)

Edit: Also, check patents and/or patent asap

<comment><AUTHOR> The "prototype" you see in the video is actually a Staples Easy Button with a piece of paper taped over it.

You don't like the shape?

</comment></comment></comment></comment><comment><AUTHOR> One of the problems we're running into is that this problem (need for location and advice) does not seem to be an acute problem for users. In other words, most people are bothered by it, but not bothered so much that they pull out their phone to download an app.

What's HN's thoughts on this?

Should we keep searching for users for whom this is "a hair on fire" problem? or, try to find a different related problem that's more painful (for more people)?

<comment><USER2> If you're thinking of making this an app, consider two processes:

Mine:

1\. I realize I need something and don't know where to find it. 2\. I look around for a person who looks like they work here. 3\. I ask said person.

Yours:

1\. I realize I need something and don't know where to find it. 2\. I magically and suddenly remember, "Oooh, there's an app for this that I downloaded last night when I was looking for apps that would improve my shopping experience." 2\. I pull out my phone from my pocket. 3\. I unlock my phone. 4\. I navigate around my phone looking for the app which takes a while because I can't remember the name of the app or the icon. 5\. I open it and wait for it to load. 6\. I wait for geolocation to warm up while it tries to figure out what store I'm in. 7\. I finally get a button that says "Ask now." 8\. I click the button and wait. 9\. A squawky voice attempts to help me, but by now I've forgotten what I'm looking for.

I'm really not trying to shoot you down, but I think you could find a bigger problem to work on. :)

<comment><AUTHOR> Ha! I like the second "story."

Btw, I think it explains the lack of user traction of apps like Aisle411, ShopKick, etc. They take too long to use and they don't "fit" well with a shopping trip.

What keeps us churning on this space is the magnitude of the $ spent. $1.4T spent annually in low-service stores (e.g. Walmart). These formats have minimal shopper-facing technology (lots of great supply chain stuff) and have evolved slowly.

So, that goes us thinking ... so far, still thinking

</comment></comment></comment><comment><USER4> A lot of stores already have this "magic" button. It simply pages an employee and brings them to you. And yes, I would use it.

<comment><AUTHOR> We've seen this to. Target has been particularly good about doing this.

Have you used the current solutions? What did you think of them?

</comment></comment><comment><USER5> I would get rid of your use of "OnStar".

<comment><AUTHOR> Thanks. We've dropped the "OnStar" adjective.

What do you think about the "Instant Chat" analogy?</comment></comment>


How to setup Microsoft Outlook custom domain for free email hosting - <AUTHOR> http://www.blogsynthesis.com/setup-microsoft-outlook-custom-domain/ <comment><AUTHOR> Outlook.com now supports IMAP [http://blogs.office.com/b/microsoft- outlook/archive/2013/09/...](http://blogs.office.com/b/microsoft- outlook/archive/2013/09/12/outlook-com-now-with-imap.aspx) as posted here: [https://news.ycombinator.com/item?id=6375770](https://news.ycombinator.com/item?id=6375770)</comment>


Moved to BackboneJS and JQuery Templates - <AUTHOR> http://www.akshay.cc/2010/10/moved-to-backbonejs-and-jquery.html

<comment><USER2> There are a couple of questions about Backbone towards the bottom of the article, and this seems like a fine place to answer them:

akshayr mentions that in the context of an event handler in a View, "this" is bound to the instance of the view -- which differs from jQuery, where "this" is bound to the element that the event originated from. This is by design: it's awfully nice if methods on a View maintain their notion of "this", regardless of if they're called directly, or triggered by a DOM event. If you want to know which element was clicked, that's what "event.target" is good for, where "event" is the object that jQuery passes as the first argument to all DOM events. Here's an example that makes use of both "this", and the specific element:

        var NoteList = Backbone.View.extend({

          events: {
            "click .note .title": "openNote"
          },

          openNote: function(e) {
            var el = $(e.target);
            var note = this.notes.get(el.attr('data-id'));
            note.open();
          }

          ...

        });

The other issue raised is the lack of JSONP support in the default implementation of Backbone.sync. It would be great to make this a configurable option, without having to drop down into jQuery for simple fetches, as akshayr did. Hopefully this will be a part of the next release.

<comment><AUTHOR> The "event.target" approach is no good when the click handler is attached to a parent container with children. 'event.target' in that case will be the children instead of the parent. An example is here - <http://gist.github.com/652212#file_hn> comment

<comment><USER2> Ah, but that's what "event.currentTarget" is for.

<http://api.jquery.com/event.currentTarget/>

<comment><AUTHOR> Excellent. Will use that now.

</comment></comment></comment></comment><comment><USER3> I love the concept of using JQuery Templates! But does anyone have experience on how using this affects SEO and search engine crawling?

<comment><USER4> Presumably quite negatively if you use them to build everything. But there are valid places, the obvious one being whenever you'd construct a complicated HTML fragment manually using JavaScript strings.

I'm currently using knockout.js with jquery templates to generate a 'feed' component that has unique content for each user. The content also changes very rapidly so there is little point in a search engine trying to index any single version of the page. I was quite hesitant to go with the JavaScript templating approach at first (ie would normally have gone with generating the content on the server), but it has made quite a few things significantly easier.

As long as you're mindful that search engines won't be able to index any javascript-generated content, you should be able to consider the use of all these new client-side templating tools.

<comment><USER3> Yeah those were basically my thoughts as well. Thanks

</comment></comment></comment><comment><USER5> Can you please spell check city names. s/Bangaluru/Bengaluru/g.

<comment><AUTHOR> Sorry, where is that again ?

<comment><USER5> <http://directory.myminister.info/>

The addresses of the MPs of Karnataka state, many of whom reside in Bengaluru.

Also, using javascript everywhere breaks linkability, as evidenced by my inability to link to the exact page in question :)

<comment><AUTHOR> 1> All the data is straight up coming from official government records. I've noticed quite a few typos coming through. At this point, its a bit unreasonable to go fixing them. Since this data is refreshed every month, chances are that it might eventually get fixed in the government records. Also this is just a past time pet tech spike app, I don't think anyone is actually looking at it.

2> Linkability and searchability (search engine bots and SEO) are certainly a problem in these kind of apps. In this particular case, I've been thinking about providing a way for other websites to embed and link to mp information.</comment></comment></comment></comment>


Ask HN: What are some good online CS Master's programs? - <AUTHOR> Hi! I'm trying to find a good online program to get a Master's degree in CS.
I'm a competent self-taught web developer, but I have a non-technical Bachelor's degree, which procludes me from getting the work visa and makes job search more difficult.
So I'm looking for a high quality program that will teach me some solid fundamentals of CS I may have missed (algorithms, data structures, etc), and give me a proper CS diploma I can use in my job/visa applications.
Can you recommend some good options? What criteria should I use to pick the best one? What should I keep in mind?
I would really appreciate any help! <comment><USER2> No personal experience with this, but GaTech OMSCS [1] seems very compelling. Great testimonials, and impossible to beat price.

[1] [https://omscs.gatech.edu/](https://omscs.gatech.edu/)

<comment><AUTHOR> It seems like their admission requirements demand a technical (math/CS) bachelors, and my diploma is from completely unrelated field.

<comment><USER2> I've read that you can take some courses from WGU[1] to qualify, again no personal experience with them.

[1] [https://www.wgu.edu/](https://www.wgu.edu/)</comment></comment></comment>


My life as a NATO collaborator (1989) [pdf] - <AUTHOR> http://guppylake.com/~nsb/WarSpy/SpyInHouseOfWar.pdf <comment><USER2> The author's page:

[http://guppylake.com/~nsb/](http://guppylake.com/~nsb/)

"You Might Know Me From... My standards work. Among other things, I'm co- creator of MIME, the Internet standard multimedia data format. My old barbershop quartet is featured in the first-ever MIME message."

And the page where he introduces the article here discussed:

[http://guppylake.com/~nsb/WarSpy/](http://guppylake.com/~nsb/WarSpy/)

<comment><USER3> That logo of the Bulletin of the Atomic Scientists is chilling.</comment></comment>


Ask HN: Is anyone else tired of reading the same articles? - <AUTHOR>

I have spent the last hour reading articles from HN I've saved over the last few weeks. They are all the same. Lean startup this, A/B test that, Design is important, I failed, Build a community then revenues, etc.
After years of reading the same articles, I have learned very little. Everyone is repeating the same stories/advice, and, as a community, I don't see any progress.
Are we doomed to continuously repeat the same sad stories over and over again? Can we finally kick this habit? I don't know- but I am tired of reading the same articles written with slightly different adjectives. <comment><USER2> I've been a user longer than you have, and I still learn things.

I think the solution, as with any news site, is not to assume that everything on it will be stuff you don't already know. Which seems pretty obvious, really...

<comment><AUTHOR> I think the problem isn't about learning new things - it is repeating information which may not even be as good as the original article.

I'm going to work on building a site that will be the Best of HN. Somewhere people can go to see the best article on AB testing or Design. If a better article comes along, it will receive more votes and become the new #1.

<comment><USER3> "If a better article comes along, it will receive more votes and become the new #1."

More votes than what? A previous article's votes on HN?

As time goes on, there are more people reading HN, so even shitty articles posted today may get more votes than stellar articles posted two years ago.

</comment></comment></comment><comment><USER4> I think you're suffering from confirmation bias. Certainly there's a lot of repeating content, but there's tons of new stuff on HN constantly.

An unfortunate habit I once noticed in myself was skipping over articles whose titles I didn't understand, even though if I want to learn about _new_ things that's almost certainly the wrong way to go about it. You'll notice (and remember) what you're familiar with much more than what you're not.

If you really want to increase the diversity of content on HN, spend more time on the new page. There are ~25 links submitted in the past couple of hours (on a Sunday). Probably 80% of submitted links receive fewer than 5 upvotes, but there's tons of great stuff in there that simply doesn't get upvoted quickly enough to get a front-page vote-boost.

<comment><USER5> I like the new page better than the front page- sure there's some crap but there is tons of great stuff there.

</comment></comment><comment><USER6> You could try the "new" link up at the top, and up-vote things you find interesting. There are a LOT of worthy submissions that never make it to the front page.

<comment><USER7> Completely agree, often I find the best articles on /new and not on the front page.

I think a great experiment for pg would be to set /new as the front page for HN and see what happens.

<comment><USER8> I like your experiment of making /new the homepage for a trial run and see what happens

</comment></comment></comment><comment><USER5> If you don't like it- write something new and post it!

</comment><comment><USER9> You make a valid point. But, those answers have to do with the questions that startups ask all the time:

1\. how do you launch? answer: "lean startup this" 2\. how do you make convert more users? answer: "A/B test that" 3\. is design important? answer: "design is important" 4\. what happens if you fail? answer: "that's okay, I failed" 5\. what do you do first, community or revenue: "build a community, then revenue"

We're limited by the subject matter, which is what makes this community so valuable. Moreover, our answers are similar because we've all developed a very similar startup ethos over time.

Are there other questions we need to be asking as startups, which produce different answers?

</comment><comment><USER10> Is basically the problem of threads-based forums repeated in the blogo/HN- sphere...

It would be cool if the best blog-posts could be somehow preserved, and shown when somebody is interested in them as if they were new, + personalizing the news/reading experience (Thoof comes to mind here).

In the past years I built this (<http://www.logilogi.org>), as a hobby/passion-project, but naturally it has not taken off either, as threads and/or "expiring the attention" that is given to posts, have the advantage that they focus readers on a limited number of posts, increasing the likelyhood of attaining critical mass in discussing/replying to them.

So there is a dilemma here...

</comment><comment><USER11> This site isn't set up to preserve knowledge. A newcomer hasn't really got anywhere to go to see all the accumulated knowledge, and they can't tell which old stories they should read to get that knowledge quickly.

As such, things will continue to be reposted over and over.

So yeah, the site is doomed to that scenario.

Instead of reading every story, read the first few comments. They'll tell you if anything new or interesting has been posted.

</comment><comment><USER12> What would you like to read about ?

<comment><AUTHOR> It isn't a question of 'What' but 'What's next?'

Articles regurgitating AB testing homepages, green buttons, and, my favorite, 'Follow me on Twitter here' aren't progressing the community.

What's next? What can we do with this information that advances the discussion?

I don't have the answer. I'm raising the question.

</comment></comment><comment><USER13> Because of the emphasis on a particular business model by YC and by extension HN, ritualized repetition of the conventional wisdom isn't surprising. On the other hand, suppose someone had a radically different and successful business strategy. Would blogging about it at length be likely to raise or lower returns?

</comment><comment><USER14> There is a lot of articles on the same themes... which is reasonable, people aren't so unique that there won't be tons of overlap. You can choose to not keep reading A/B articules et al. Spend the time you have wasted reading the same things over and over again learning a new skill...

<comment><AUTHOR> I think you are hitting at exactly what I'm so frustrated with - the goal of these articles is to help people learn a new skill/ learn from their mistake. However every few weeks, the same meme reappears. The Startup version of Groundhog Day.

<comment><USER15> Merlin Mann got very frustrated with this same problem as it pertains to the GTD/Lifehack community. (where he felt that he was part of the problem.) He wrote this essay about it: <http://www.merlinmann.com/better>

</comment></comment></comment><comment><USER16> Maybe a simple keyword filter would help? Could be done as fancy 'realtime' search which would actually be cool/useful giving instant feedback. It would make it easy to screenout posts that are uninteresting to the user.

</comment><comment><USER17> Maybe there just aren't enough awesome articles out there for you?

Otherwise, if you see something you'd like to read on HN, submit it! If that isn't successful, then you can really start complaining.

</comment><comment><AUTHOR> Why is this post falling? 7 pts in 28 minutes should be front page, and now its on the second page? I thought it was a valid argument to raise...

<comment><USER18> Since you ask, personal opinion only ...

Articles without links have a penalty applied. This site is about finding cool stuff, not about musing over the site itself. If it isn't a link to somewhere else then it's less likely (although not certain) to fall strictly in line with the site's original intent.

Yes, it's a valid point to raise, but a better way to raise it would be to write a proper article, discussing the issue, proposing possible solutions, weighing their merits, then making specific suggestions, and finally, submitting a link to the item to wrote. This is much to be preferred over just complaining.

<comment><AUTHOR> Thanks- I am doing that right now +1

</comment></comment></comment><comment><USER19> There's plenty of new stuff on here all the time. Get an rss reader and scan through titles. Makes it easier to filter the repeats.

</comment><comment><USER20> I just skip over the repeats. I'm still new here so I probably haven't read all of them. And, after all some things bear repeating.

</comment><comment><USER21> Even this post is a duplicate sentiment.

</comment><comment><USER22> One issue you run into is that, in any field, the tried and true gets repeated a lot (for newcomers and for other reasons). Anything 'really new' tends to not be embraced because it's untested and no one knows for sure it will work...etc. Those are valid concerns. Even for areas where people are doing cutting edge stuff, you pretty quickly develop a few respected "experts" and anything which doesn't readily agree with their view will be dismissed, actively attacked, etc -- even in cases where it is a non-competing alternative which addresses, in effect, a different issue/niche. Also, those folks who already know all the tried and true stuff will tend to be fairly small in number. Trying to get a good conversation going among that small group will tend to be tough and not well-suited to a very public discussion.

If you can think of a way around some of those issues, more power to you. But it is an issue that tends to crop up to some degree on every forum I have ever belonged to, regardless of the topic in question.

</comment><comment><USER23> its impossible to find your perfect news feed unless you are running it... I would just start skimming over a few other feeds routinely and you will be able to pick up a bigger variety of stories.

treehugger, techmeme, betanews, techcrunch, and even ted will be good places for a wider inventory of stories.</comment>


Ask HN: What can I do professionally to combat climate change? - <AUTHOR> I've seen a lot of recommendations for what I can do in my personal life to help with climate change, but I've also seen that lot of smart people think that my lifestyle contributions are a drop in the bucket. It seems to me (and please correct me if I'm wrong) that I could have a much larger impact by focusing my software development skills/career on combating climate change.
That being said, I'm looking for resources to help me decide what direction to take professionally in order to work on projects that a) have a larger impact on climate change than personal lifestyle changes, and b) are interesting problems technically.
Some of the good resources I've found include Bret Victor's "What can a technologist do about climate change?" [1], and Steve Easterbrook's "Climate Change: A Grand Software Challenge" paper [2], both of which are great resources. I'm interested to hear if any HN'ers know of other great resources in this domain, or specific companies/organizations with which I could work on this extremely important problem.
[1] http://worrydream.com/ClimateChange/ [2] http://www.cs.toronto.edu/~sme/papers/2010/Easterbrook-FSE2010-wkshp.pdf <comment><USER2> Figure out a way to stop people from flying unnecessarily: [https://www.washingtonpost.com/news/theworldpost/wp/2017/11/...](https://www.washingtonpost.com/news/theworldpost/wp/2017/11/02/plane- pollution/?utm_term=.dafc4c375ef5)

</comment><comment><USER3> I would say make training videos or do comparisons of products like reusable drinking bottles etc.

Educating people on what they can do is probably the single most effective way to leverage. YouTube and other video platforms are becoming the dominate way to consume content. So figure how to educate on those platforms.

</comment><comment><USER4> Convincing everyone its a real problem and we must deal with it now is the first step. We need to tackle this problem as a society, not as individuals.

</comment><comment><USER5> Lets make such a resource together, a forum or list of things to do, and invent/work on?

<comment><USER6> A resource for finding jobs for "green" companies would be great. At least here in Canada, my searches that intersect "software" with "green"/"renewable"/"wind"/etc yield next to nothing.</comment></comment>