HAproxy in the era of Microservices - <AUTHOR> http://47ron.in/blog/2015/10/23/haproxy-in-the-era-of-microservices.html <comment><USER2> My current project consists of ~7 microservices, which has been wonderful for many reasons but exposed us to substantial complexity on this score. At present ~2 of them have to receive communications internally or externally over HTTP, so we have those fronted by a proxy to make service discovery easier. (It is, regrettably, not an off-the-shelf proxy, since one of them required application-specific routing to a sticky host(s) depending on the contents of the HTTP request).

The thing which has made our lives MUCH easier from an orchestration, discovery, and routing perspective is NSQ, an OSS message bus. It's an event model rather than a request model -- producers of events say "Hey world, by the way, EventType just happened." and consumers can register to say "Apropos of nothing, please tell me about any EventType" without producers or consumers having to be aware of each other or aware of what a consumer intends to actually _do_ with regards to the event.

Events go into topics ("a type of event") and each topic can have an arbitrary number of channels ("one thing which should happen for each event in this topic"). If you're clever about where you raise events, you can stitch together arbitrarily complex systems by just adding new consumers on new channels, without having to make the producers aware of the new functionality at all.

Concrete example: We wanted a log of all orders made to the stock exchange. We have a little utility, nsq-to-s3, which takes all the events on a topic/channel, persists them in memory/disk for a while, and periodically flushes that log to S3 for long-term storage. This trivialized the logging feature. (I was going to write our own nsq-to-s3 but an OSS project of that same name existed and was adequate to our needs, so we didn't have to go to 8 microservices... yet.)

This was much easier than making a generic S3 logging service and then making edits to seven microservice codebases to make a HTTP request out to the logging service, with e.g. retry/failure recovery/monitoring/etc, everywhere in those codebases we wanted something logged to S3.

<comment><USER3> This sounds similar to kafka, how does it compare?

<comment><USER2> I have heard from several folks that they use Kafka where we'd use NSQ, but have insufficient experience with Kafka to give an informed opinion on the tradeoffs. I will say that few technologies I've used in my career were as easy to adopt as NSQ was.

</comment></comment></comment><comment><USER4> The most significant issue with HAProxy and large SOAs is the lack of really good dynamic re-configuration. You don't want to have to manually change your configuration every time someone wants to add or change load balancing to a service, and if you do it automatically you run into HAProxy reloads being a reliability problem as they drop traffic on the floor due to Linux being goofy in how it implements SYN handling.

It looks like there is a lot of good advice on this thread for how to mitigate the configuration problem (bamboo, consul template, etc ...) but then you run into reloads, and even if you use zero downtime restarts as I described in my HAProxy deep dive ([http://engineeringblog.yelp.com/2015/04/true-zero- downtime-h...](http://engineeringblog.yelp.com/2015/04/true-zero-downtime- haproxy-reloads.html)), you still end up with periodic ~50ms blips in your timings.

We've found that Synapse ([https://github.com/airbnb/synapse](https://github.com/airbnb/synapse)) is best in class at managing HAProxy for us in a fairly large SOA. It is really intelligent about doing as much as possible dynamically over the HAProxy stats socket as opposed to reloading for every configuration change, and we can finely tune how often we are reloading HAProxy. The best part is that it is fully pluggable. Use DNS, marathon, zookeeper, etcd, etc...: Synapse can manifest that service registration information into HAProxy configurations.

<comment><USER5> Another problem with the configuration reloading is that all the backend/service metrics counters that HAProxy exposes on its metrics endpoint get reset on every reload. That makes for really noisy metrics gathering in situations where you have rapid successive changes to the config, such as gradually scaling up hundreds of instances of a new version of a microservice and then scaling down the old version. When you then scrape the metrics via Prometheus using an bridge like HAProxy Exporter ([https://github.com/prometheus/haproxy_exporter](https://github.com/prometheus/haproxy_exporter)), you lose a lot of counter increments due to the frequent resets, so you get lower and noisy rates for that time period.

<comment><USER6> I hadn't heard of Prometheus before. We use nagios for alerting/alarms, and graphite/grafana separately for metrics. Would you recommend looking into prometheus as an alternative?

<comment><USER5> As a Prometheus author, I'm biased, but yes :)

See also [http://prometheus.io/](http://prometheus.io/) and [https://www.youtube.com/watch?v=HjN23GgCzQY](https://www.youtube.com/watch?v=HjN23GgCzQY) for an intro.

</comment></comment></comment></comment><comment><USER7> HAProxy is a great fit for most microservices, but I'm not sure it has the feature set for most organizations that are deep into microservices - at least in the way the author describes it. The problem is more business-driven than technical; so let me explain why (and feel free to correct me if you think I'm wrong!)

Most companies don't need microservices, so I'm making an assumption that any company using microservices has both the volume and complexity of business partners that make a microservices approach a good business idea (this assumption is assuredly false, but still useful). And if you've gone this route, the next step is generally organizing your business around an SOA where business units can be represented by an API (microservices are useful from a business perspective because you have many stakeholders within a company that want to make small changes frequently but independently). Once you've done that, you start wanting to encapsulate your relationship with your customers in an API as well, and the eventual destination (if your company survives long enough and is successful) is that you just operate a set of APIs that power your customer-facing UX product in addition to the API-based services you provide to your high-volume clients.

My point is this: the end state of a microservices architecture is an API gateway. API gateways provide essential services such as authentication, access control, and connection throttling. You still need something like HAProxy to provide you with load balancing, health checks, etc. across many nodes - but you're going to have a very simple instance of HAproxy for each microservice and rely on your API gateway to organize everything under a single domain.

With microservices, your configurations can get incredibly complex very quickly, so it's often better to maintain independent components with very simple configurations (that can be easily updated dynamically) than to try to have a single component in the middle of your stack upon which many other components are dependent.

<comment><USER8> I think that HAProxy can work, but it needs assistance. As it doesn't allow you to use resolvable hostnames, you can't rely on DNS load balancing for microservices. As I said in my comment on the article, Consul + Consul- template works well for this.

<comment><USER9> As of 1.6 it _does_ allow resolvable hostnames [1]. This is the example given in the release announcement:

        resolvers docker
         nameserver dnsmasq 127.0.0.1:53

        defaults
         mode http
         log global
         option httplog

        frontend f_myapp
         bind :80
         default_backend b_myapp

        backend b_myapp
        server s1 nginx1:80 check resolvers docker resolve-prefer ipv4

[1] [http://blog.haproxy.com/2015/10/14/whats-new-in- haproxy-1-6/](http://blog.haproxy.com/2015/10/14/whats-new-in-haproxy-1-6/)

<comment><USER7> That's pretty new; but still doesn't remove the need for an orchestration layer (though it does make it a lot easier since I often rely on DNS with short TTLs for global routing).

</comment></comment></comment></comment><comment><USER10> I'm sorry to be that guy, but the lack of contrast makes the text difficult to read.

<comment><USER11> It's good enough

<comment><USER12> I'm sorry to be that guy, but the lack of contrast makes this comment difficult to read.

</comment></comment></comment><comment><USER13> Folks should take a look at Baker Street (bakerstreet.io) which is a HAProxy- based system for routing traffic between microservices. Baker Street is designed to be really easy to get up and running (the install is super quick) and is based on the Yelp popularized SmartStack. However, one of the beautiful aspects of Baker Street is that unlike SmartStack it does not require running and maintaining a ZooKeeper cluster which can be a PITA for teams just starting to work with microservices.

Full disclosure: I am the lead developer of Baker Street :)

Website: [http://bakerstreet.io](http://bakerstreet.io)

Source: [https://github.com/datawire/bakerstreet](https://github.com/datawire/bakerstreet)

<comment><USER14> Hey there!

Just a small nitpick: Running Zookeeper is insanely simple. We deployed it in 2-3 days, and it has had literally 0 issues over the last year.

I highly recommend [https://github.com/mbabineau/docker-zk- exhibitor](https://github.com/mbabineau/docker-zk-exhibitor) to get started (Docker container that wraps together Zookeepher and Netflix's Exhibitor manager for it).

Remember, pick boring technologies that just work.

<comment><USER15> zookeeper is not a boring technology that just works:

[http://blog.cloudera.com/blog/2014/03/zookeeper- resilience-a...](http://blog.cloudera.com/blog/2014/03/zookeeper-resilience- at-pinterest/)

[http://arstechnica.com/information-technology/2015/05/the- di...](http://arstechnica.com/information-technology/2015/05/the-discovery-of- apache-zookeepers-poison-packet/)

[https://tech.knewton.com/blog/2014/12/eureka-shouldnt-use- zo...](https://tech.knewton.com/blog/2014/12/eureka-shouldnt-use-zookeeper- service-discovery/)

Here's ibm and netflix with some advice about how non-boring running a ZK cluster can be:

[https://www-01.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/...](https://www-01.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html)

[http://techblog.netflix.com/2011/11/introducing-curator- netf...](http://techblog.netflix.com/2011/11/introducing-curator-netflix- zookeeper.html)

</comment></comment></comment><comment><USER6> We use a single monolithic HAProxy to route to all of our applications. We have about 8 different kinds of applications we host. Of those, we have a few separate instances of those applications available at similar (but different) URLS. Think: `[https://sub.domain.com/<prefix>/appname/path`](https://sub.domain.com/<prefix>/appname/path`).

We route based on `sub`, `prefix`, `appname`, and even sometimes `path`. Managing the config is literally going into this relatively big file, adding in ACLs, adding backends, and setting up the routing rules. We manage the file with puppet, but it's static - not generated.

I feel that it's probably time to start deploying many HAProxy instances, generating the file per service, and then maintaining a frontend haproxy that routes according to subdomain.

Can anyone comment on similar deployments? Have you found it easier to manage many haproxy instances rather than a single monolithic one?

</comment><comment><USER16> We're using a home baked solution using our own config server ([https://github.com/niko/goconfd](https://github.com/niko/goconfd)) which evaluates templates POSTed to it. Combined with some trivial shell loops over a blocking POST and a HAProxy restart this does the job sufficiently well. If anybody is interrested in the details I'm happy to elaborate.

</comment><comment><USER17> I heard AirBNB installs HAProxy on each of their nodes, and then manages the configuration for each service on each node. Then the service just needs to access 127.0.0.1 instead. This sounds compelling to me because you eliminate the single point of failure of the HAProxy goes down.

<comment><USER18> It's called Smartstack and is also entirely open-source - [http://nerds.airbnb.com/smartstack-service-discovery- cloud/](http://nerds.airbnb.com/smartstack-service-discovery-cloud/)

Having been a long-time user, I can tell you the added complexity of every service having to allocate its own port, then tracking that port across every other service is a lot of overhead on its own.

Not to mention you also have to run a zookeeper cluster and two additional daemons on every instance.

All in all it's a good concept, but slightly too complex for my needs. Recently made the switch back to using internal ELB's for every service and not looking back.

</comment></comment><comment><USER19> I'm moving away from HAProxy. The lack of zero downtime config reloads is consistently causing problems, especially in a high reload environment (i.e. using bamboo [0]). One team deploys an application that goes into a restart loop, changing our routing table constantly? Well there goes our 99th percentile out the window.

That said, would prefer not to be using nginx, so if anyone has any recommendations for high quality zero-downtime-reloadable HTTP proxies, I'd be very interested.

[0]: [https://github.com/QubitProducts/bamboo](https://github.com/QubitProducts/bamboo)

<comment><USER4> Check out the deep dive I wrote into how you can help prevent HAProxy reloads from dropping traffic without a large 99% hit over at [http://engineeringblog.yelp.com/2015/04/true-zero- downtime-h...](http://engineeringblog.yelp.com/2015/04/true-zero-downtime- haproxy-reloads.html)

Looks like bamboo encourages folks to use this strategy via a custom reload command: [https://github.com/QubitProducts/bamboo/issues/152](https://github.com/QubitProducts/bamboo/issues/152)

<comment><USER19> That's egress traffic :) And yeah, [https://github.com/QubitProducts/bamboo/issues/143#issuecomm...](https://github.com/QubitProducts/bamboo/issues/143#issuecomment-139306818) is currently my best idea (with some additional nfq stuff). Doesn't help when requests span the entirety of the reload though.

<comment><USER4> I mean, I wouldn't encourage anyone to use HAProxy for microservice load balancing unless they're running it on every node, at which point clients are always talking to localhost and everything is egress.

That being said, you can turn ingress into egress with an ifb. A coworker of mine created a proof of concept for using our strategy with an external facing load balancer but I don't think he ever tried it in production so I'm not sure how well it works.

To be fair I'm scared of both the iptables and tc solution on external LBs because you might never know that you're refusing connections accidentally. Linux 4.4 is coming with some patches that should help with this and afaik BSDs have done it right since the start.

</comment></comment></comment></comment><comment><USER20> This is exactly why we built out conf-builder ([https://github.com/radiantiq/conf- builder](https://github.com/radiantiq/conf-builder)). None of the existing solutions matched what we needed and so far it's worked out pretty well for us. If you use docker there is also a neat tool called registrator that we've played around with [https://github.com/gliderlabs/registrator](https://github.com/gliderlabs/registrator)

</comment><comment><USER21> Using consul ([https://www.consul.io/](https://www.consul.io/)) in conjunction with HAProxy is another great way for service discovery and routing.

<comment><USER8> Yep, Consul-template + HAProxy is great for a dynamic HAProxy solution.

<comment><USER22> [https://github.com/eBay/fabio](https://github.com/eBay/fabio) might also be an option for you.

Disclosure: I'm the author.

<comment><USER8> That looks interesting. I'll check it out. Thanks!

</comment></comment></comment></comment><comment><USER23> If you're using marathon, check out bamboo, it can generate haproxy configuration files from templates which can be triggered by changes in marathon. Cool stuff!

</comment><comment><USER24> What a non-article! tl;dr you should use run HAproxy on a known host that forwards to your services.

Why not nginx?</comment>


Ayu: A theme for modern web development - <AUTHOR> https://github.com/dempfi/ayu <comment><USER2> Would love to use this on Atom.

<comment><USER3> ditto. But make it Visual Studio Code for me, please!

(any guide on converting themes between IDEs?)

<comment><USER4> IIRC, Sublime Text themes are based on TextMate grammar, so I believe Atom's documentation on converting from TextMate would work (YMMV):

[http://flight-manual.atom.io/hacking- atom/sections/convertin...](http://flight-manual.atom.io/hacking- atom/sections/converting-from-textmate/)</comment></comment></comment>


Francis Bacon biography researcher finds friend's unpublished diaries - <AUTHOR> http://www.theguardian.com/artanddesign/2016/jan/25/francis-bacon-biography-researcher-finds-friends-unpublished-diaries <comment><USER2> No, not that Francis Bacon [https://en.wikipedia.org/wiki/Francis_Bacon_(artist)](https://en.wikipedia.org/wiki/Francis_Bacon_\(artist\))

<comment><USER3> I just started reading and was 100% let down!!!!! Came to post the same thing.

People who up voted didn't read I bet.

<comment><USER4> The painter is also interesting.

<comment><USER3> (I use to work in the art world as a teenager) While he is interesting the implications of someone having a detailed journal from the 16th and 17th Century and the 20th is huge.</comment></comment></comment></comment>


BadPower attack corrupts fast chargers to melt or set devices on fire - <AUTHOR> https://www.zdnet.com/article/badpower-attack-corrupts-fast-chargers-to-melt-or-set-your-device-on-fire/ <comment><USER2> > The bad news is that the research team also analyzed 34 fast-charging chips, > around which the fast charger models had been built. Researchers said that > 18 chip vendors did not ship chips with a firmware update option, meaning > there was no way to update the firmware on some fast charger chips.

Isn't this a good thing? If you can't update the firmware you can't infect it. Conversely, if you still _can_ infect it via some vulnerability then the same technique can be used to "infect" it with fixed firmware.

</comment><comment><AUTHOR> > The Tencent team said they verified their BadPower attack in practice. > Researchers said they selected 35 fast chargers from 234 models available on > the market and found that 18 models from 8 vendors were vulnerable.

I would rather throw my charger out if it had a bug and buy a new one than risk someone intentionly or accidentally (compromised phone) turning it into an electric match.

</comment><comment><USER3> Whether firmware can be updated or not, how will any of these charges ever be found and replaced/repaired?

What are the logistics of executing such a recall _successfully_ , such that people aren't vulnerable?</comment>


Ask HN: What's the best open source alternative of a software/services you know? - <AUTHOR> <comment><USER2> Firefox to Chrome (only slight sarcasm).</comment>


Can you help? I'm working on new mobile app and I've 3 questions? - <AUTHOR> http://feedback.nessfile.com/

<comment><USER2> Brewster does something similar to this if I am understanding the landing page correctly. You should check out their app if you haven't already.

<comment><AUTHOR> Yes, I know Brewster</comment></comment>


Intro to statistical data analysis in Python – frequentist and Bayesian methods - <AUTHOR> http://ipython-books.github.io/featured-07/ <comment><USER2> I also recommend Probabilistic Programming & Bayesian Methods for Hackers [http://nbviewer.ipython.org/github/minrk/Probabilistic- Progr...](http://nbviewer.ipython.org/github/minrk/Probabilistic-Programming- and-Bayesian-Methods-for-Hackers/blob/master/Prologue/Prologue.ipynb)

<comment><USER3> agreed - i think this is the best applied bayesian book on web - in python anyways

</comment></comment><comment><USER4> Most helpful thing I ever learned about Bayesian statistics came from Kant: all of a sudden the "prior" and "posterior" were easy to remember. In his introduction, he discusses the origin of synthetic knowledge, and sets about distinguishing between _a priori_ and _a posteriori_ knowldge: that which one had before, and that which one has after. Of course we all know about "a priori" but I had never associated "a posteriori" with the same line of thinking.

[https://en.wikipedia.org/wiki/A_priori_and_a_posteriori](https://en.wikipedia.org/wiki/A_priori_and_a_posteriori)

</comment><comment><USER5> Sweet! Bookmarked. Thanks for sharing. I'm just getting started with Python and this will indeed serve as a great resource once I start delving in Data Science stuff.

<comment><AUTHOR> If/when you start delving into Data Science stuff, you should check out Data Science Weekly [http://datascienceweekly.org/](http://datascienceweekly.org/) (I'm co-editor) which is a free newsletter of data science articles, blog posts, job postings, resources, and interviews with data scientists.

<comment><USER6> Data Science Weekly is great. Definitely worth looking into for anyone interested in data science.

</comment></comment></comment><comment><USER7> I love how every day there is a new post about R and statistical data analysis. It's really a hot topic. I hope somebody could upload a course using measure theory for the ones like me interested more in the abstract probability concepts.

<comment><USER8> 1) "R _or_ statistical data analysis", since this is most definitely python.

2) enjoy: [http://ocw.mit.edu/courses/mathematics/18-466-mathematical-s...](http://ocw.mit.edu/courses/mathematics/18-466-mathematical- statistics-spring-2003/lecture-notes/)

</comment></comment><comment><USER9> I use R for my statistical programming mostly, but I use Python for a lot of other things. It's nice to have this as a reference when I don't feel like moving back to R.

</comment><comment><USER10> Awesome to have this page. I was wondering about this only few days back, if I could find a website where I could learn Statistical Data Analysis in Python. :)

<comment><USER11> For those interested in this topic, I just finished and highly recommend [http://shop.oreilly.com/product/0636920023784.do](http://shop.oreilly.com/product/0636920023784.do). It covers IPython, NumPy, Pandas and matplotlib. It doesn't cover algorithms and analysis so much, but it's great for learning the tools if you already know the statistics.

</comment></comment><comment><USER12> Consider using good Python style (like consistent use of whitespace) when trying to teach people to use Python (in any fashion).

<comment><USER13> semicolons oh my goodness

</comment></comment><comment><USER14> Are there similar resources for social sciences, preferably marketing/micro- econ.? And also preferably in R.</comment>


Thousands Violate SF Housing Laws Using Airbnb, Few Face Penalties - <AUTHOR> http://www.nbcbayarea.com/investigations/Bay-Legal-Thousands-violate-SF-housing-laws-few-face-penalties-380218621.html <comment><USER2> > Since February of last year, 1,281 other hosts have also registered with the > city. However, that number is a far reach from the 7,046 hosts that Airbnb > acknowledges are listing more than 9,000 properties in San Francisco. That > means at least 82 percent of hosts are breaking the law.

I agree that Airbnb hosts not following regulations are probably contributing to the supply crisis. But 9000 properties? Isn't SF in need of 10s of thousands of units if not even more than that to match demand?

I think SF's ineptitude around permits, zoning, and the "not in my backyard" mentality is a way bigger issue.

</comment><comment><USER3> I'm conflicted on the SF AirBnB issues because on one end I'm part of the "any more housing is good housing" crowd but at the same time I'd like AirBnB to succeed (as a user when I travel from SF) and I'm afraid of downstream consequences of heavy regulations in its host city.

I've more or less accepted I'm a hypocrite on the issue, or at least selfish on it.

<comment><USER4> What percent do you estimate of AirBnB landlords would withdraw from AirBnB if they were further encouraged by AirBnB to abide by the law?

</comment></comment><comment><USER5> If SF is going to be a NIMBY city, actively discouraging new development, violating property rights, and banning short-term rentals in a manner akin to some sort of socialist state, I guess that's just the shit we're used to here. But to restrict supply, implement all sorts of price controls, and then engage in this sort of selective (or nonexistent) enforcement feels so utterly corrupt. Though I suppose expecting any sort of consistency on the matter is asking too much.

</comment><comment><USER6> I registered with the office of short term rental close to six months ago. Still haven't heard back from them. Ditto for a few other people I know.</comment>


What do you say to people when they say that programming is not demanding? - <AUTHOR> http://programmers.stackexchange.com/q/37316/2673

<comment><USER2> The top-rated answer (by the submitter I think?) is excellent: [http://programmers.stackexchange.com/questions/37316/what- do...](http://programmers.stackexchange.com/questions/37316/what-do-you-say- to-people-when-they-say-that-programming-is-not-demanding/37334#37334)

<comment><USER3> I had seen a post by Bruce Eckel conveying the same thought; and it certainly puts it correctly.

 _Writing Software is Like ... Writing_ <http://www.artima.com/weblogs/viewpost.jsp?thread=255898>

And in most of the cases, IMO, not to forget those awesome co-workers who (have a forget-stuff-in-60-mins syndrome and) would need a lot of help on grammar or vocab, every single day. Blessed life, I'd say!

</comment></comment><comment><USER4> Try explaining being a DBA or a sysadmin - if we're doing our jobs right, no- one will even know we're there.

</comment><comment><USER5> 'Yeah, you're probably right.'

</comment><comment><USER6> Don't you hate when your theoretical phisicist, mathematician, molecular biologist and rocket engineer friends do that.

</comment><comment><USER7> I'd like to know what type of jobs the responders work. Would be interesting to see a breakdown of answers to "how demanding is programming" by users of different languages/frameworks or by what type of company they work at (startup, big corporation, etc).

</comment><comment><USER8> If you get full control of the code base from the ground up, you can often make life easier for yourself (it takes work and continual refactoring, though). What's difficult is when you have to work through other peoples' messes.

</comment><comment><USER9> I fire up Terminal, then stand up, offer my chair to them, say I'll be back later.

</comment><comment><USER10> Making a superfluous statement is a red flag. In most cases I'd nod and move on.

</comment><comment><USER11> "It depends" or "that's not entirely true"

It various, I know people who do little html and consider it programming. And obviously there are those in charge of huge software developments.

</comment><comment><USER12> I say: "Shhh! Don't tell anyone, then they'll all want to do it!"

</comment><comment><USER13> you do it

<comment><AUTHOR> Isn't that what caused all those business critical excel spreadsheets ;)

</comment></comment><comment><USER14> Of course, you ignore them!

<comment><AUTHOR> Unless it's your wife... Then you say 'Yes, dear'

</comment></comment><comment><USER15> The same thing as when people say that teaching is not demanding.

</comment><comment><USER16> All the problems in the first answer are people problems. If you work in a software firm that doesn't treat its employees like code monkeys, or better yet, work for yourself, then programming isn't demanding. It's fun, relaxing, and awesome.

Sales is demanding. :S

</comment><comment><USER17> Many times when I hear the "programming is very easy" claim it comes from someone who is out of work: "Oh, he say programming hard? Not so! Programming very easy! I do this job for you cheap!"</comment>


Colorado Startups - <AUTHOR> http://www.coloradostartups.com <comment><AUTHOR> This is one of David Cohen's blogs, he is one of the founders of TechStars, a program similar to YC.</comment>


Why is almost half of Google in beta?  - <AUTHOR> http://royal.pingdom.com/2008/09/24/why-is-almost-half-of-google-in-beta/ <comment><USER2> If gmail is still considered 'beta', the term has little significance anymore. I believe it persists mostly because of the PR bump derived from declaring the 'beta' is over.</comment>


If Planet Nine Is a Tiny Black Hole, This Is How to Find It - <AUTHOR> https://www.discovermagazine.com/the-sciences/if-planet-9-is-a-black-hole-this-is-how-to-find-it <comment><USER2> This is both a very obvious and a very interesting proposal: locate an apparent source of gravitation by observing the gravitational effects it has upon a fleet of small (almost point-like) craft.

However, there's something that niggles me.

If it's a primordial black hole that has been captured by the sun, why is it (as far as I understand) apparently in the same plane (ecliptic) as all the other major bodies in the solar system? It makes sense for all planets _& cetera_ to be in the same plane because they all originated in the same swirling (flat) planetary nebula... but a captured object could potentially orbit at any inclination (and, considering how massive it is, I doubt it would be coerced into being coplanar by comparatively minor intersections with light bodies such as those present in the Oort Cloud, which are arranged roughly spherically anyway).

<comment><USER3> If SpaceX can afford to put a car into orbit around the sun as a publicity stunt during rocket testing, I wonder how much this would cost.

Edit: Probably around $100MM: [https://www.spacex.com/about/capabilities](https://www.spacex.com/about/capabilities) for the launch, several $100MMs for the (space based?) accelerator and spacecraft, and then a few more for operations and research. Yeah, $1 billion as described in the article seems like a good estimate: [https://arxiv.org/pdf/1805.01306.pdf](https://arxiv.org/pdf/1805.01306.pdf)

Since knowledge arising from this experiment will probably only be useful as a common good, there aren't really incentives for this to happen, unless governments decide to go through with it.

NASA's 2020 budget is only $22.6 billion: [https://en.wikipedia.org/wiki/Budget_of_NASA](https://en.wikipedia.org/wiki/Budget_of_NASA)

The Apollo program cost $153 billion: [https://en.wikipedia.org/wiki/Apollo_program#Costs](https://en.wikipedia.org/wiki/Apollo_program#Costs)

Okay, this thing is ten times further away from the sun (400 AU) than Pluto (40 AU), 80 times the distance of Jupiter from the sun (5 AU), very far. Still, it's much closer than our next closest star, Proxima Centauri, which is 265612 AU away, or over 660 times the distance of Planet 9.

<comment><USER4> If we get actual evidence supporting that there could indeed be a black hole within our solar system, that seems like a true opportunity to advance our knowledge of physics. The potential to study a real black hole up close seems worth massive investments given what kind of breakthroughs could he possible from it.

<comment><USER5> Yes and no. I mean, how would you study a black hole of radius ~5cm without getting really close? Consider that:

1\. Time dilation dictates that it would take ages for any instrument to get near the black hole, conduct measurements and send its results back to Earth.

2\. Time dilation also means you would never see anything fall into the black hole. So studying the inside of the black hole is out of question, too. But arguably this is what would be required to make any breakthroughs in our understanding of black holes.

3\. Tidal forces near the event horizon of such a tiny black hole would have catastrophic consequences[1] for anything (human or object) that gets too close to it.

[1] [https://en.wikipedia.org/wiki/Spaghettification](https://en.wikipedia.org/wiki/Spaghettification)

<comment><USER3> Wouldn't this mean that these tidal forces may (have) cause(d) highly accelerated matter to shoot through our solar system?

<comment><USER5> I assume you're referring to astrophysical jets[1] that are common with SMBHs. I'm afraid, though, I'm not familiar enough with jet physics to answer your question. A priori, the tidal forces only cause infalling matter to get shredded (and heated up). Whether a proper accretion disk together with a (significant) jet can form, given the size of the black hole, I don't know.

On a related note: It'd be interesting to see in how far such a small BH would _actually_ be capable of "sweeping [its] local neighbourhood clear of objects" like planets are (as was claimed in some comment above). Only then would I expect a significant accretion disk and, possibly, a jet. The reason the answer to this question is not obvious (at least to me) is once again the tiny size of the BH: How would anything manage to fall into the black hole, given that it's so tiny and, thus, very easy to miss?

[1] [https://en.wikipedia.org/wiki/Astrophysical_jet](https://en.wikipedia.org/wiki/Astrophysical_jet)

</comment></comment></comment></comment></comment></comment><comment><USER6> When the original paper about planet 9 being a black hole came out a year ago, everybody thought it was a cool-sounding but just completely wild idea -- both physicists and pop science articles. Not impossible by any means, but not within the top 250 new physics ideas in plausibility. Then the discussion died down. I figured it would end up being one of those forgotten but cool gems.

Then the most authoritative string theorist in the world, who hasn't worked on anything relating to experiments in decades, suddenly writes a very short, simple, single-author paper to use this idea to promote Yuri Milner and Mark Zuckerberg's space probes. Now it's the conjunction of _two_ crazy ideas, and it's really taking off. I just don't know why this is happening, or what to make of it.

<comment><USER2> Of course if there was enough dust in interstellar space to reflect a laser beam (as happens when we see the traces of glowing beams shone through carbon dioxide gas) we could just prove the heavens with laser beams until suddenly we saw a beam bend. Unfortunately that isn't the case so we have to fling stuff at it and see if they veer off track.

<comment><USER7> You can't easily linearly scan space for a moving target, among other show- stopper problems.

<comment><USER2> No, but since using satellites on trajectories you hope will be perturbed is essentially 'scanning' anyway, it would be preferable to do it with a light- beam than with physical objects.

</comment></comment></comment></comment><comment><USER8> A well characterized black hole in our solar system would be our (one way) ticket out of here. The velocity we could gain would be limited to the vessel integrity. And thanks to math, being able to choose approach trajectories means we could go anywhere.

Well worth looking real hard just in case it does exist.

<comment> T-A Assuming that you're thinking of the Oberth effect, it's no miracle worker. In the (non-relativistic) impulsive burn approximation you gain a speed factor sqrt(1 + 2*v_escape/delta_v) [1].

With v_escape = c and delta_v = 10 km/s that's a factor 245, for a final speed of 2450 km/s. That will get you to Alpha Centauri in 535 years.

[1] [https://en.wikipedia.org/wiki/Oberth_effect](https://en.wikipedia.org/wiki/Oberth_effect)

<comment><USER8> Alas I am not well enough educated on the subject to have ever come across that particular term. (I do wish KSP was available on *nix) And I do see your point as it pertains to solid matter.

How ever as a layperson I am at least vaguely aware that there are hyper velocity stars, some observed w/delta-v close to 10% of C if memory serves ...

Which would indicate to me that having a gravity well we can delve arbitrarly deep into has profound implications compared to gravity wells where it you go too deep you run into clouds or dirt.

For example the vessel could be paired, co-orbiting with a sacrificial mass that is dumped into the hole while the vessel leaves at at factor greater than a few hundred.

<comment> T-A The fastest hypervelocity star I can find with a quick bit of googling is S5-HVS1, at 1700 km/s [1], slower than the hypothetical black-hole-accelerated ship example.

You can't delve arbitrarily deep into a black hole's gravity well if you want to get something back; you need to stop at the horizon, where the escape velocity equals light speed.

Dumping sacrificial mass into the hole is built into the Oberth effect scenario: the ship's extra speed (relative to the hole) is matched by a lower speed for the ejected reaction mass, which therefore falls toward the hole.

I suppose you could imagine something like a circular accelerator built around the hole and powered by dropping masses into it.

[1] [https://newatlas.com/space/fastest-star-hypervelocity- ejecte...](https://newatlas.com/space/fastest-star-hypervelocity-ejected- milky-way/)

</comment></comment></comment></comment><comment><USER9> The article talks about a scattergun approach and that detection would require getting within a few dozen AU of the object. That would require a huge number of probes and a means of coordinating beams spread across multiple continents. Wouldn't it be easier to determine the objects rough position ahead of time?

</comment><comment><USER10> Could a black hole orbit our tiny yellow dwarf sun? Even if it were tiny for a black hole it would be hard to imagine that it would have less mass than our sun. If it had more mass than our sun wouldn't our system be a binary system, and if so it would suggest a black hole is not really a satellite of the sun.

How small can a black hole be and still qualify as a black hole? I guess a better question is what is the least massive black hole observed and what is the smallest current theories allow?

I have no idea what I'm talking about so all of that is probably wrong. I am just not able to wrap my head around it.

<comment><USER11> > How small can a black hole be and still qualify as a black hole?

There's no theoretical lower bound on the mass of a black hole. Once matter has ceased to repel other matter, it will continue to fall inward. A violent impact could generate the forces necessary on a small scale to create micro black holes. Black holes do seem to slowly lose energy though, so small ones would not live very long[0]:

[0]: [https://en.wikipedia.org/wiki/Micro_black_hole#Hawking_radia...](https://en.wikipedia.org/wiki/Micro_black_hole#Hawking_radiation)

<comment><USER5> > so small ones would not live very long

Actually, this is completely unknown. While Hawking radiation is a well- established effect in the semiclassical limit (read: when gravity is weak), we don't really know what happens when the Schwarzschild radius reaches scales where tidal effects become noticeable already near the event horizon and, eventually, quantum gravitational effects (supposedly) take over.

In fact, this whole question lies at the heart of the black hole information paradox. Do black holes evaporate completely? Or will evaporation stop at some point, leaving behind a black hole remnant?

</comment></comment></comment><comment><USER12> From the paper:

>However, the .01 c mission sketched in [10] has a spacecraft mass of only 6.6 mg, divided between the sail and the payload. Multiplying this by 100 while keeping the sail mass fixed leaves a payload mass of barely .65 grams.

Oof. That's not a lot.

</comment><comment><USER13> Wouldn't a black hole that small evaporate due to Hawking radiation?

<comment><USER14> No. At even 1% of Earth's mass, a black hole's already down to 2 Kelvin of output, across a surface as big as a grain of very fine sand. It would take more than 10^44 years to evaporate on its own, but since the cosmic microwave background is 2.7K it would actually grow over time.

For some other data points, a thousand ton black hole would explode in 84 seconds. A million ton black hole would last 2600 years as an unimaginably hot point five hundred times smaller than a proton. A billion ton black hole would last 2.6 trillion years as an unimaginably hot point twice as big as a proton.

</comment></comment><comment><USER15> Discovering a mini black-hole at the boundaries of our solar system would also mean discovering a lovely power source just waiting to be tapped, right ?

<comment><USER16> Black holes themselves do not generate power. If they have an accretion disk disk - i.e. they're feeding on matter - then that can be quite good at converting mass into radiation which you can then harvest. But you could also do that with the radiation of a more easily accessible mass-energy converter: the sun

<comment><USER17> You are both talking about the _Penrose Process_ , or perhaps only one of you is?

[https://en.m.wikipedia.org/wiki/Penrose_process](https://en.m.wikipedia.org/wiki/Penrose_process)

</comment></comment></comment><comment><USER18> It's such an unimaginable thing. 5cm across, yet "super" massive (not SMBH).

</comment><comment><USER19> Planet Nine is a Mass Effect Relay

</comment><comment><USER20> >>> about 5 centimeters across. Consequently, it is almost impossible to spot with a telescope.

If it is 5cm across, then it should be rather bright (Hawking radiation). While still tiny, it wouldn't be black.

<comment><USER21> This calculator[1] indicates that the temperature for a 2.5 cm radius black hole should be 0.007 K. So below the temperature of the cosmic background (2.7 K).

[1] [https://www.vttoth.com/CMS/physics-notes/311-hawking- radiati...](https://www.vttoth.com/CMS/physics-notes/311-hawking-radiation- calculator)

<comment><USER22> Does this mean that the cosmic background could be adding energy to the blackhole (Maybe helping with stability?)? I'm not sure how temperature and heat transfer would work in this context.

Edit: For those interested, there is an entire article on "Black Hole Thermodynamics" on Wikipedia ([https://en.wikipedia.org/wiki/Black_hole_thermodynamics](https://en.wikipedia.org/wiki/Black_hole_thermodynamics))

<comment><USER2> You are correct: a black hole whose Hawking temperature is less than that of the cosmic background radiation is "colder" than the space it is immersed in and consequently absorbs energy (and mass-energy equivalence) from the heat bath it is immersed in. The easiest way to envision it is that the black hole swallows up photons of the microwave background radiation and is emitting less energy as Hawking radiation, leading to a net imbalance in the black hole's favour.

Of course eventually the expanding universe will drop the temperature of the cosmic background radiation further and the process will reverse.

</comment></comment></comment></comment><comment><USER23> So could something like Starlink be used to detect things like this?

<comment><USER2> No. Who implied that?!

Starlink will be a network of very near earth communication satellites. This black hole, if it exists, is far far further out than Pluto.

The probes wouldn't be even remotely reminiscent of Starlink satellites either.

<comment><USER23> It was a question. I am not an expert and so I asked.

Also I said "like this".

If you have a mesh of satellites around the earth and used that data along with other astronomy tools, I'm sure there are other things we can learn.

</comment></comment></comment><comment><USER24> Does anyone else feels like that projects like these should be avoided unless we are certain that our corner of the galaxy is not part of a larger dark forest? The consequences of such actions can be catastrophic. If we do send out probes they should be designed to disintegrate after some fixed period of time.

<comment><USER25> I too choose to not use anything that radiates electromagnetic waves on the off chance aliens detect them.

<comment><USER24> Electromagnetic waves will die off and become indistinguishable from background radiation after a certain distance. Not so much for physical objects.

<comment><USER26> The famous Douglas Adams quote is relevant: "Space is big. You just won't believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it's a long way down the road to the chemist's, but that's just peanuts to space."

The odds of an alien civilization picking up one of our probes, even if we send out thousands of them, is preposterously small. Imagine trying to such an object if it had been left floating at an unknown point in the ocean; now consider how much bigger interstellar space is than Earth's oceans.

<comment><USER17> "It's like trying to find your car keys in a field using a pair of binoculars."

I cannot remember where this quote came from, but I think it was something to do with a missing Russian cargo vessel in the Atlantic.</comment></comment></comment></comment></comment>


Own Your Data - <AUTHOR> http://www.zeldman.com/2011/01/10/own-your-data/ <comment><AUTHOR> The comments are where the meat of the article is. You can also get some info here: <http://tumblr.tomhenrich.com/post/2675663088/own-your-data>.

I find Tantek's implementation fascinating but I think this is a good critique of it. I also agree with <http://twitter.com/#!/glenda/status/24216200326152192> that the ideas of personal data ownership and permanence is one of the next big things to tackle.</comment>


Show HN: Gibbon Tabs – A powerful and light Chrome tab manager - <AUTHOR> https://github.com/macadev/Gibbon-Tabs <comment><AUTHOR> Hi there! I'm the creator of this little project. It's a Chrome extension to help you manage your tabs in Chrome. I'm sure some of you can relate to how awful it is having 50+ tabs open all the time across multiple windows (I'm looking at you StackOverflow).

This light weight extension lets you find tabs quickly through fuzzy-search. It also lets you store collections of tabs so you can open them later.

I'm happy to have finally completed ONE side project! Thanks for checking this out!

<comment><USER2> Thanks for sharing. Having tried a bunch of these, I like it a lot'.</comment></comment>


Open-sourcing F14 for memory-efficient hash tables - <AUTHOR> https://code.fb.com/developer-tools/f14/?r=1 <comment><USER2> I've recently started using Folly's F14 hash map implementation (specifically, F14ValueMap--my keys and values are relatively compact) and it's the fastest unordered associative container I've yet found. This is for an application that repeatedly adds to and removes from the map. In particular, it's faster than a custom solution using open addressing that I had written that is pretty much optimal for this particular application (for an open addressing implementation, that is).

I do wish it could be used header-only out of the box, but still a great library.

<comment><USER3> Have you compared to ska::flat_hash_map or khash? I've found the latter to be ideal if my keys/values were POD and the former otherwise. I'm not yet convinced that it's actually better based on their website. std::unordered_map is infamously slow.

<comment><USER2> I did look at klib about a year ago. Honestly I don't recall whether I looked at khash in particular; I know I was also considering some of the other klib stuff for other uses. I did use Google's dense_hash_map for a time.

For my application, the problem with implementations that use more traditional tombstone algorithms is that removing items from the table doesn't decrease the load, so that at some point the entire table has to be recreated. For my application, worst case performance is far more important than average performance.

dense_hash_map definitely behaves as described above, and (from a quick look at the code) I believe khash does as well.

I haven't looked at ska::flat_hash_map but thanks for the tip.

<comment><USER4> If I am right, if you delete all elements in F14, there will be no tombstone. However, if you delete some, there may still be some tombstones left. You may still need rehashing due to those remaining tombstones. In that sense, F14 is not eliminating the worst case; it makes the worst case happens less often. Then the question is how "less often". This probably depends on the access pattern.

What is your access pattern? Are insertions and deletions frequently interleaved? Or do you insert a batch and then delete another batch? It will be interesting to have a micro-benchmark to measure how the F14 strategy works in practice.

> _I believe khash does as well._

Yes, khash uses the traditional tombstone solution. It rehashes in the same space when there are too many tombstones.

<comment><USER5> It would be nice to have a deeper understanding of how/why this works, but in practice it seems fine. Under a continuous insert and erase workload the number of chunks with an overflow fluctuates a bit but stays low enough to keep probe lengths short, so F14 never needs to rehash to clean them up. Batched adds and then removes might let you bias a bit toward having longer probes for the remaining keys, but it never gets too bad. A truly adversarial workload could make things pretty bad, but they could also just insert lots of keys that all have the same hash code.

See the steady_state_stats test at [https://github.com/facebook/folly/blob/master/folly/containe...](https://github.com/facebook/folly/blob/master/folly/container/test/F14MapTest.cpp#L797)

</comment></comment></comment></comment></comment><comment><USER6> Performance is very similar to my header-only version of the Abseil's hash tables available at [https://github.com/greg7mdp/parallel- hashmap](https://github.com/greg7mdp/parallel-hashmap).

</comment><comment><USER7> > Those options are slow, risky (clients can change the key), incompatible > (maybe we can try again after proxy iterators work), and ugly, respectively. > After careful deliberation, we chose ugly, using const_cast to make the key > mutable just prior to its destruction during rehash.

You can't win them all, I guess…

Also, TIL that invalidating iterators does not actually invalidate references to keys and values:

> The standard guarantees reference stability: References and pointers to the > keys and values in the hash table must remain valid until the corresponding > key is removed.

</comment><comment><USER8> [https://github.com/sparsehash/sparsehash](https://github.com/sparsehash/sparsehash) was for a decade+ hands down the most memory efficient map implementation, originally written by Craig Silverstein of Google. I am not sure how the more recent SIMD stuff impacts it.

<comment><USER5> sparsehash is still more memory efficient, but it is quite slow in comparison. Also, in practice it doesn't reach the ultra-low space overheads claimed by the documentation. It allocates memory blocks of many different sizes, so the dominant space overhead becomes internal fragmentation in the allocator. For sparsehash using JEMalloc's default allocation classes (spaced about a factor of 1.2 apart) the memory used relative to useful data varies from about 1.3 for small values to 1.1 for large values.

</comment></comment><comment><USER9> This seems like an implementation similar to the one Google engineers have given talks about (w/vectorization, etc) so it's cool to have more options to use a modern performant table like that.

</comment><comment><USER10> Since the hashes uses open addressing wouldn't they be susceptible to hash poisoning attacks? Java plugged that hole in HashMap in Java 8 by reallocating lists over a certain length as balanced trees. But that strategy isn't possible with open addressing. [https://www.nagarro.com/en/blog/post/24/performance- improvem...](https://www.nagarro.com/en/blog/post/24/performance-improvement- for-hashmap-in-java-8)

<comment><USER5> F14 supports stateful hashers, so in situations that need to be resilient to hash poisoning we can use a hasher designed for that purpose.

<comment><USER10> I recall that that was what was suggested for Java's HashMap too, but they found it to be insufficient. By careful analysis, an attacker could still figure out how to cause hash collisions.

<comment><USER5> I haven't seen those discussions, but is it possible that they were trying to rehash the results of Object.hashCode rather than pass all of the bytes of the original value through a new hash algorithm?

If you haven't seen it before you will appreciate [https://accidentallyquadratic.tumblr.com/post/153545455987/r...](https://accidentallyquadratic.tumblr.com/post/153545455987/rust- hash-iteration-reinsertion)

<comment><USER10> Afaik, they augmented the hash function with a random seed, which didn't work out. See: [http://emboss.github.io/blog/2012/12/14/breaking-murmur- hash...](http://emboss.github.io/blog/2012/12/14/breaking-murmur-hash- flooding-dos-reloaded/) It appears that SipHash would be enough (for now) to prevent hash poisoning. Most open addressing hash implementations doesn't yet use SipHash though.

</comment></comment></comment></comment></comment><comment><USER11> There was a discussion thread on here recently about Google's Hash Table implementation. I wonder how the approaches and design considerations compare to each other.

<comment><USER5> I'm one of the authors. Seeing the Google presentation at CPPCon convinced us that the potential wins were worth the effort to productionize, but the similarity in the designs is a case of convergent evolution. Since then Google and Facebook have collaborated on a microbenchmark ([https://github.com/google/hashtable- benchmarks/](https://github.com/google/hashtable-benchmarks/)), which shows that the algorithms have slightly different tradeoffs but neither dominates the other.

</comment></comment><comment><USER12> Hasn't this been open-sourced since at least last March?

[https://github.com/facebook/folly/commit/93d49bcf9a44d6b2d9c...](https://github.com/facebook/folly/commit/93d49bcf9a44d6b2d9ce2c0a1f309457b6132709#diff-01399ee4f56d48b1e6847210ecf4c80b)

<comment><USER5> You're right, it's been available for a while now. This is the first time we've made an announcement about it.

</comment></comment><comment><USER13> It's funny how they bring up same birthday probability without referring to the "birthday problem" even if it supports their case for chunking even better.

For a hash table implementation, it's even more relevant that out of 180 people in a room, chances are > 99.999% that two people share a birthday (meaning collisions), whereas the chance that 15 people in that group are born in the same fortnight is still very small.

<comment><USER14> I realized that I misunderstood this comment (I assumed same fortnight meant "same fortnight number in a year" \- when it should be same 14 day span) but I thought it was worth calculating this example anyway. The oldest person alive was born in 1903. Since then 3035 fortnights have passed. Assuming an even distribution of birth fortnights, there'd be a 3.4058% chance of two people having the same birth fortnight (1 - (P(3035,15)) / (3035^15), where P is for Permutations).

It's still fairly likely, but far from the ~1 in 10^23 odds of having NO shared birthdays between 180 people.

</comment></comment><comment><USER15> how comes that F14BasicMap (the base class of F14FastMap F14ValueMap and all the others) is extending std::unordered_map ? [https://github.com/facebook/folly/blob/master/folly/containe...](https://github.com/facebook/folly/blob/master/folly/container/F14Map.h)

they seem to be calling the table_ member for every method in the book. Still weird by they are deriving from std::unordered_map in the first place.

F14Table<Policy> table_;

the implementation class F14Table - [https://github.com/facebook/folly/blob/master/folly/containe...](https://github.com/facebook/folly/blob/master/folly/container/detail/F14Table.h)

<comment><USER5> That's a fallback implementation for platforms without SIMD. The real code is lower down.

</comment></comment><comment><USER16> Curious how this compares to brownhash, which is on its way to stable Rust on July 4th.

</comment><comment><USER17> Someone should port this to C and add it to the next version of Lua, given that Lua's main datatype is pretty much a hashtable with some extra magic.

</comment><comment><USER18> I'm curious what Rust's implementation looks like in comparison now...

<comment><USER5> [https://github.com/rust-lang/hashbrown](https://github.com/rust- lang/hashbrown) is a port of Google's SwissTable algorithm (abseil flat_hash_map). It also uses SIMD filtering and has the potential to be quite good.

<comment><USER19> Which has been recently upstreamed to the standard library, replacing the old hash map implementation. (This sped up rustc a noticeable amount, incidentally.)

<comment><USER20> Is `hashbrown::HashMap` now completely ported/identical with `std::collections::HashMap`? I just compared their performance (on yesterday's beta channel) on a hashmap with over 200k keys and didn't notice any relevant speedup.

<comment><USER21> Yes, as of a couple of days ago in nightly (pre-1.36): [https://github.com/rust-lang/rust/pull/58623](https://github.com/rust- lang/rust/pull/58623)

</comment></comment></comment></comment></comment><comment><USER22> Can this easily be ported to Rust?

<comment><USER16> Fb probably open sourced the work after the engineers realized that hashbrown is superior to it. Just a hunch. Need to see benchmarks..

<comment><USER23> F14 has been open source for over a year already...</comment></comment></comment>


Ethereum and Computer Language - <AUTHOR> https://bitflate.org/post/2019/11/16/ethereum-and-computer-language.html <comment><AUTHOR> Author here. I write this article to clear up the discussion about Ethereum prospect. There's a lot of hype and attack. I hope this analogy can help people find common ground.</comment>


Open-source, DRM-free Kindle alternative - <AUTHOR> https://www.vice.com/en_us/article/7x5kpb/anyone-can-build-this-open-source-drm-free-kindle-alternative <comment><USER2> This isn't really equivalent to a kindle though. The fonts are basic, there's no enclosure. Fonts are a huge thing for eReaders and Amazon has spent a lot of time perfecting them.

It's also important to remember that it's perfectly possible to use the Kindle without DRM. You can just put your own books on it with a cable, though you'll have to convert them to MOBI as it doesn't do ePub. The same can be done with other brands like Kobo, which can take raw ePub. Also, the Kobo's can be pretty easily modified as they just run plain Linux, I'm running PyGame on an older one. The entire OS was simply on an internal SD card so really easily modified (though not sure if the new ones can be modified too).

I think the problem with DRM and ebooks is not the readers. It's the availability of books without DRM... Basically the "GOG" of eBook stores. Even if you have an open eBook reader, where are you going to get the books from?

I think for the reader, I would prefer to buy commercial hardware as look and feel is an important thing for a device you will interact with a lot. Similar to the way us Open Source aficionados don't build our own laptops, but do use free software :)

<comment><USER3> > Even if you have an open eBook reader, where are you going to get the books > from?

There's a huge ebook pirating scene, so you could always do that, buying the book beforehand if the ethics disturb you.

What is sad to me is that there's no way to check out an ebook from a library in a way that doesn't involve Amazon.

<comment><USER4> > there's no way to check out an ebook from a library in a way that doesn't > involve Amazon

Isn't this what Libby - [https://libbyapp.com/](https://libbyapp.com/) \- does?

<comment><USER5> But then you can't use it on an e-reader. It remains in the app.

<comment><USER6> Libby (and by extension Overdrive) supports checking out books onto supported e-readers. They must support either the Kindle or Adobe DRM standards though.

[https://help.libbyapp.com/en-us/6059.htm](https://help.libbyapp.com/en- us/6059.htm)

</comment></comment></comment></comment></comment><comment><USER7> I'd buy one if:

1) It was prebuilt, plug-and-play.

2) It comes as a kit. I can do it as a project with my kid. Once assembled, it's as above. Kit is consumer-friendly.

Digikey parts list and PCB is a little over my laziness tolerance.

People confuse free-as-in-freedom with free-as-in-not-making-a-profit. I've done free-as-in-freedom businesses, successfully. I wish others did too.

A lot of manufacturers underestimate the power of 100% open. I'm not price- sensitive. I'll pay for it. If my cheap Chinese [tablet/keyboard/mouse/webcam/etc.] comes with a PCB schematic, parts list, and open source firmware, I'll probably pay triple and prefer it to a Logitech/Razor/etc. I'll give nice online reviews too.

And the designs are simple enough no competitive edge is lost.

On the other side, projects like this, lacking a business model, rarely get mature enough for me to use. Make 'em nice and sell 'em, I say.

And yes, there will be cheaper clones, so profit margins can't be insane, but people will pay extra for the original branded version, AND profit margins can't be insane in competitive markets either way.

<comment><USER8> Digikey parts list + pcb can be a reasonably convenient way to put your own kit together - you can upload a csv to digikey and have them fill your shopping cart automatically, and it works pretty well. They even suggest alternatives like if your list had 8x an item, but it is actually cheaper to buy 10x the item it will suggest that.

<comment><USER7> One big difference is that if I order a parts list and PCB based on your web site, if you've made an error or have a part missing, I'm stuck debugging. If you're putting together a kit, you've (presumably) gone through and made sure it's all there and all works together. That includes the auxiliary stuff, like firmware, or having a case.

Another big difference is if I place three orders, and one comes up out-of- stuck or delayed by months, I get stuck with a useless 2/3 of a kit. I can't return the useless 2/3, since those vendors didn't mess up. If you're selling me a kit, you're doing logistics for me.

Even with things with standard parts, like putting a computer or bicycle together from standardized parts, I've had issues. With DIY, this thing can explode.

<comment><USER9> As someone who has several pcb + digikey projects, I don't buy this.

It's literally 2 clicks instead of one click. 3 clicks to add the 3d-printed part fromShapeways.

It IS already exactly, a kit. You get a box in the mail that has everything in it, it just says DigiKey on the box instead of mydumbproject.org

That's 3 fully pre-loaded urls all together in one spot. No uploading a csv. Just click the link and up pops a pre-loaded shopping cart. If anything is out of stock, you see that before buying not after they ship 1/2 of it.

I'm exactly the same amount likely to "forget to include some parts" in my painstakingly crafted digikey cart as in any kit. It takes days sometimes to hunt down and figure out just what all should go in the cart because there are 500 versions of everything. You don't do that much work to leave things out.

I want to share my efforts for whoever wants it, or just for reference, not run a mail order business. I have a day job. The stuff is free, the knowledge and directions are written up into a followable recipe, in a wiki and github which _I_ don't even have to be the only one who can improve over time, and you only have to pay the actual material suppliers, and fill out 3 order forms instead of one. And when I lose interest or die, it's all still there without me to keep providing it.

If you think it makes so much sense to provide that last little bit of service, and charge for it, feel free to start up and run that business. You can take all my stuff and produce and sell those kits. I bet you have no interest in that though because you have better things to do. Yeah, me too, and this ereader guy too. If you want anything more, you want a Kobo or any of the other 50 rootable no-names. Those already exist. You don't need this or a kit version of this for that.

Figuring out what all to put in that digikey cart was the 900x hard part, but clicking 2 links instead of one is just over your laziness threshhold? OK.

You have to buy the pcb from a different link? Well you have to buy batteries from somewhere else even for a complete finished product let alone a kit, and a protective cover, and a memory card, and a car charger, and headphones, any number of extra bits like that. The book reader didn't already come with the books I hate to surprise you with that outrageous ommission. This argument just doesn't wash on so many levels.

<comment><USER7> I'm not complaining about what you're doing. I'm describing what I want. You can accuse me of being unreasonable, but people said the same thing about remote controls. Who'd be lazy to not just want to walk over and switch a channel on the TV? And countless other conveniences.

I'm also not asking YOU to do anything differently. You have a day job, and with however many kits you've made, no way in heck this would compete with YOUR day job. I also want a blueberry plant in my back yard. Neither you, nor this ereader guy, nor even the local garden store have any obligation to give me that. It's a free market.

But yes, I do think there's a pretty good business in providing that last little bit of service. I'd pay for it, and I suspect many others would as well. You can view this as an integration of Digikey+Shapeways+PCB. Or you can view this as an expansion of AdaFruit/SEEED/etc. to be more open. Or you can view this as moving cheap Chinese manufacturing up-market.

And I'd buy an open source Kobo too, for that matter. There's a world of difference between a rootable device and an open source device too. Most rootable devices I have are now sitting in storage somewhere, unused, when the fun ran out and the maintenance overhead kicked in. One or two were never used, as rooting became a pain-in-the-butt, some feature I needed was missing, or there was some hardware change between the rooted version and the one I bought. Most open source devices I have are in active use, some with a few tweaks.

I'm glad to support open. I value my freedom. I also like having things which keep working, and open mitigates risks (if the manufacturer goes under, the community sometimes takes over). I'm glad to pay for that.

However, I'm busy, and I'm not glad to buy a headache. That goes double for people less technical than you or me, ones who want projects for kids, etc.

</comment></comment></comment></comment></comment><comment><USER10> The kindle doesn't lock you into Amazon's ecosystem though... It supports many different formats, Calibre allows you to convert your books to a kindle- specific format for convenience and I've yet to find an ebook DRM that can't be easily cracked...

<comment><USER11> Same experience here. I have literally never purchased a single book from Amazon, though I have read lots of their free "classics editions". I've jailbroken my paperwhite so I can read/do whatever I want, but I still get great battery life and the ability to do things like send to my kindle over e-mail. I get the free-as-in-freedom guys' argument, but I'm perfectly happy with my device and like the convenience of a simple book-reading device.

<comment><USER10> Exactly. Don't get me wrong, I love open source hardware as much as the next guy but fixing something that isn't broken is a plain old waste of time.

</comment></comment></comment><comment><USER12> I want an e-ink web browsing tablet.

I want to be able to go to nytimes.com, or Vox, or Hacker News, and read the articles and comment threads on a high-contrast screen that doesn't hurt my eyes. I don't want to use some read-later service that sends specific articles to my e-reader, I want to just navigate to the sites directly. There would be trade-offs for sure, but it seems like the benefits would outweigh the downsides.

I've been waiting for ~ 10 years now. Am the only one? Is anyone ever going to make this?

<comment><USER13> Those exists now, in the form of eink Android phone/tablet. The refresh rate of eink makes their experience suboptimal, but some find it usable enough.

You can search for Boox tablet, or Hisense A5 phone.

<comment><USER12> Oh that's interesting, I missed this one: [https://www.techrepublic.com/article/review-onyx-boox- max3-a...](https://www.techrepublic.com/article/review-onyx-boox- max3-a-13-e-ink-android-tablet/)

I know I shouldn't be complaining about price now that I've heard it exists at all, but $840 is _really_ steep! I was thinking something more like $400. I know it's a niche product, but even so.

<comment><USER14> The Max3 has a giant screen (larger than the biggest iPad Pro). I'm still getting used to mine, but I mostly like it.

The web browsing experience is OK, but it's slow and monochrome.

<comment><USER12> Monochrome is expected. Do you say it's slow because of the screen's refresh rate, or do pages take a long time to render?

I'm kind of eying the BOOX Nova2, which is in my price range at $340. The screen is 7.8", which is ~ 2" smaller than what I'd ideally want, but not by much.

<comment><USER14> It's mostly the refresh rate, I think. The processor is fairly decent.

</comment></comment></comment></comment></comment></comment><comment><USER15> PocketBook is a reasonable DRM-free all-format alternative to Kindle.

<comment><USER16> Agreed. I've used their devices since the Pocketbook 360 days (5" screen with a hard cover - features that were uncommon).

It eventually stopped working this year, and a new review of the market place suggested they're one of the better options - so I picked up the HD3.

I can not understand how anyone could accept a vendor (especially one like Amazon) limiting or remote controlling my device's contents.

</comment></comment><comment><USER17> I vow for open standards and used to upload epub files to Google Play Books and read in my devices (Kindle Fire 8 - with Google Play sideloaded, iPhone, iPad) with decent Sync. In december I got a Kobo Aura 2, my eyes are more comfortable have less distractions but I lost ubiquity and sync, also E-reader feels slugish and dictionary font is very small even with glasses. It's a take the blue/red pill situation.

<comment><USER18> I'm so sad that _WikiReader_ project, debuted in 2009, was abandoned in 2014.[0]

Hopefully, _The Open Book_ looks more promising & would be able to fully replace _WikiReader_.

JFTR, Need write an article about _The Open Book Project_ [1] on Wikipedia.[2]

[0] [https://en.wikipedia.org/wiki/WikiReader](https://en.wikipedia.org/wiki/WikiReader)

[1] [https://github.com/joeycastillo/The-Open- Book](https://github.com/joeycastillo/The-Open-Book)

[2] [https://en.wikipedia.org/wiki/Open_Book](https://en.wikipedia.org/wiki/Open_Book)

</comment></comment><comment><USER19> Hoping for good things for this product but will put in an plug for the Remarkable- open source software, well designed and capable hardware, solid product vision.

<comment><USER20> Are you sure it's accurate to say the Remarkable uses open source software? Their website says it runs on "Codex — A purposely designed Linux-based operating system for low-latency digital paper displays", so at least the Linux kernel is open source, but look at their EULA: [https://support.remarkable.com/hc/en- us/articles/36000028275...](https://support.remarkable.com/hc/en- us/articles/360000282757-reMarkable-End-User-License-Agreement)

"You are not entitled to modify or distribute the Software."

Doesn't sound like open source software to me.

Also, in this day and age, I believe that devices should be open hardware as well (as the Open Book appears to be). Open source software is a good start, but it isn't good enough.

Another Kindle alternative that is open software + open hardware is the Inkplate 6, I've pre-ordered one: [https://www.crowdsupply.com/e-radionica/inkplate-6](https://www.crowdsupply.com/e-radionica/inkplate-6)

<comment><USER19> I take the point, but two responses-

1) in practical terms, what you get is essentially an open, hackable platform. The CTO is a former KDE dev, you get root on the device, much of the product is being developed in the open-

[https://github.com/reMarkable](https://github.com/reMarkable)

And there are recipes for getting UI and background components running on the device. I have not done this, but do use the API in my own workflow to get content on the device.

2) Yes, they are building a _product_. A, say, religious commitment to open source, and/or open hardware, is a commitment to limit the range of decisions that may help the business to a much narrower set that adhere to the religion. Sometimes strict open source is good for business. Often it is not.

As someone who in his younger years has bought quite a large number of "open hardware" products, the software for which was always only barely usable, the business model hypothetical, and therefore never went anywhere and now litter the graveyard- I am _very_ happy for this team to be doing the right things for the _business_.

It's a really good product, with a really bright future, and it also happens to be pretty open and hackable.

Kind of best case, in my opinion.

Cheers.

<comment><USER20> I'd also like to push back on your cynicism about open hardware products. I must admit that history is full of open hardware failures, like the adorable Chumby (what went wrong?), but times have changed and I think you will find that the open hardware is building significant momentum today, I'd encourage you to take another look.

The purchase that turned me into an open hardware evangelist was the Planck EZ mechanical keyboard: [https://ergodox-ez.com/pages/planck](https://ergodox- ez.com/pages/planck)

It's a joy to use. The layers feature of the open QMK firmware blew my mind and changed how I think about typing. The keyswitches take seconds to replace so it's extremely easy to repair/maintain, which is great for the planet. It's tiny and portable, which was perfect for my needs before the pandemic hit and I stopped going outside. And perhaps most importantly, the company seems pretty successful, they seem to be a competent business that is in it for the long haul.

I am also very impressed with my Pinebook Pro: [https://www.pine64.org/pinebook-pro/](https://www.pine64.org/pinebook-pro/) The year of the Linux desktop may never come, but this shockingly cost- effective open hardware laptop convinced me to quit Apple products after a decade of loyalty. For my needs, which are mostly browsing the web, editing code, and SSH-ing into servers, it's good enough for like 1/10 of the price. I'll grant you that Pine64 is more of a hobbyist project than a business, but it seems extremely successful on its own terms and seems sustainable despite selling stuff basically at-cost.

I've now ordered a System76 Thelio desktop, also open hardware (although closer to Apple prices, performance costs money I guess). [https://system76.com/desktops](https://system76.com/desktops)

I don't say all of this to suggest that Remarkable is bad, it seems pretty good. I'm just saying that it can be _better_ on open source software and hardware, without it hurting the business or the product. It might even help.

</comment></comment></comment></comment><comment><USER21> This is not a replacement.

I love my Kobo but tend to use my phone because I can buy books using the Amazon android app. I hate the UI (both navigation and the suggestions on which are never what I want to read next) but it works ok for buying books.

A replacement would be a book market place where I can buy DRM free books and use clients on any ebook reader to download them to native format supported by the reader. Wirelessly and without fear of someone keeping track of what I read.

Kobos can be hacked to some extent, but no replacement firmware exists to my knowledge. Their readers already run Linux but I'm sure that they don't release drivers for their display or UI source code.

<comment><USER22> I really wish the publishers/distributors supported OPDS. Either add in a user/password prompt that's not to speed, or just put tokens in the URLs. I'd buy far more ebooks from any distributor willing to do this.

</comment></comment><comment><USER23> Imagine an open-source iPod that won't have access to iTunes. It likely would be a success, but not a replacement.

I see the power of Kindle in the ease of buying books. I see the device as a storefront to Amazon's e-book collection.

To be more massively successful, such a device should come with access to a sync service (or several), and access to e-book stores. IDK if Amazon has an API for buying books, but definitely some e-book sellers with APIs exist.

And yes, of course it should allow uploading DRM-free books of your choice, and maybe integrate nicely with Calibre.

The software part of the project looks much bugger and gnarlier than the hardware part.

</comment><comment><USER24> who would want to read anything on 4.2" display?

I've had bigger (4.5") display in smartphone 8 years ago and I would not call even that suitable for reading books

why is nobody discussing this and why they don't show the device in hand to show how ridiculously small it is?

> Main features: > 4.2" inch e-paper display with partial refresh, driven over > a dedicated SPI bus.

[https://github.com/joeycastillo/The-Open- Book](https://github.com/joeycastillo/The-Open-Book)

<comment><USER22> I did all my (non-school) reading for my first half of college on a Nintendo DSi, before I got a tablet. High resolution, large screen, good backlight, fast display refresh rate--all of these improve the experience, but none of them is required, and if you've never used them then all you know is _this is so much better than reading on my laptop_. Sure, having a small screen is a little frustrating, but having easy buttons to change pages makes up for it (the DSi RL buttons were better than all tablets I've ever had) and once you get into your reading, you stop noticing.

Tips:

\- set the background color as close as you can to the bezel color, and remove the margins if you have a small screen \- Set the font size a little large if you have a low resolution screen; antialiasing makes it much easier to read, even if your screen is also small.

Tips if your device has a slow processor and little ram (I had all or most of this automated at the time, through a batch file and a custom calibre conversion profile)

\- resize or remove images. I stored mine alongside the book rather than in it, saved as smaller 8bit PNG files. \- split your ebooks into multiple files (~100kb/file without images allowed pages to load in <1/2 second) \- Make sure the files in your epub are actually split by chapter (calibre can do this automatically, I think) \- if you have control over file compression (I use a script to generate epubs), don't use high compression--especially if there's images.

<comment><USER24> you are ruining your eyesight by using such small display/letters and fixed focused distance, it may not look like that when you are young, but give it few years and you will see results

4.2" is OK for temporary reading for very short periods, but ebooks are usually used for long reading sessions and one would want to use it for years before throwing away

</comment></comment></comment><comment><USER25> Is there any text-to-speech system that could run on hardware like this with good results? I still use my old Kindle Keyboard with sideloaded books because it has pretty decent text-to-speech (which Amazon removed from later revisions.) I've tried some open source TTS systems like festival and could never get any to work well enough to actually use (insufficient voice quality to work at faster reading speeds.)

</comment><comment><USER26> My question is whether this device can download and read ebooks from Amazon? I don't have a Kindle and the small number of Amazon ebooks I have (many of them free), I read on Amazon's Reader page. My take on Amazon is they aren't going to make it easy to bypass their controls, both DRM and otherwise.

<comment><USER27> Calibre strips Amazon's DRM and most book reading software supports you fetching books directly from Calibre.

<comment><USER28> Note: not out of the box, but there's a plugin for that. Keywords that will lead you to it: Apprentice Alf.

<comment><USER11> Is there some ban against linking drm strippers on HN? I've seen countless links to scihub or to paywall bypasses on articles.

</comment></comment></comment></comment><comment><USER29> Looking at the GitHub page, I like how the board appears to be laid out like an equivalent of a literate program.

<comment><USER30> Yeah this stood out to me too! It costs nothing to add more silkscreen to the PCB and it serves as an excellent learning tool.

</comment></comment><comment><USER31> If there is not a kick starter for this yet I hope there is soon. I would back this immediately.

<comment><AUTHOR> There isn't a kickstarter but he does have a Patreon.

[https://www.patreon.com/joeycastillo](https://www.patreon.com/joeycastillo)

</comment></comment><comment><USER32> previously on HN: [https://news.ycombinator.com/item?id=21246417](https://news.ycombinator.com/item?id=21246417)</comment>


Equifax was breached using a two month old vulnerability - <AUTHOR> https://arstechnica.com/information-technology/2017/09/massive-equifax-breach-caused-by-failure-to-patch-two-month-old-bug/ <comment><USER2> So we know how - my immediate question is how many other instances are out there, unpatched, tangled up with things like Payment Portals, Auto Reg. Portals. Gah.

</comment><comment><USER3> You wonder really just what goes on behinds the scenes leading up the a big breach. Is this on anyone's minds ? It damn well should be.</comment>


Setting up pre-commit tests with Git - <AUTHOR> http://dduvnjak.pw/adding-pre-commit-tests-with-git/

<comment><USER2> I'm not sure if this is a good practice. As project grows, running tests takes more time - remember that tests are not only unit tests which should run quickly, there are acceptance tests which can take some time to run. Waiting for commit to finish can become painfully slow.

Additionally, I like to commit partial solutions to feature branch as I work. Later I will fix the mess with rebase - this workflow wouldn't work.

You need CI anyway, so I'd leave tests there, and let the developer decide when to run the unit tests on his machine.

<comment><AUTHOR> This example runs unit tests only, so it doesn't take too long (integration and other tests would still have to be executed on CI). It's more of a convenience than a practice, so you basically don't have to think about running unit tests.

<comment><USER2> Sure, if you find it convenient - go for it. However, I'd not enforce it as a policy for whole organisation for the reasons outlined above.

</comment></comment></comment><comment><USER3> IMO we shouldn't have to jump through hoops to be able to commit. Let people commit whatever they want and then run tests to decide if you want to merge in their branches.

The workflow should encourage people to commit early and often to safeguard the work they've been doing as much as possible. It's very frustrating to lose work because it wasn't in the git object store.

<comment><USER4> True, although I've used something similar that compiles the code and runs tests, if they failed it asked if I still wanted to commit. It was just a quick check that I was probably committing what I thought I was.

</comment></comment><comment><USER5> Wouldn't it be a lot better choice to make this a post-receive hook on your Git server? E.g. run the tests before code gets accepted to master.

I often intentionally commit code that doesn't work (pass tests or even compile), but not to master branch. One of the big advantages of Git is that you can share code with your team mates even if that code doesn't work yet.

<comment><USER6> You could always set the script to only run the tests on certain branches on pre-commit. (i.e. if branch name is not master, exit 0)

</comment></comment><comment><USER7> At my work I setup a pre-commit hook that my team uses to run Uncrustify, a source-code beautifier. If the style of the commit doesn't match our rules for that language it will spit out a command for you to run to fix it. I think overall it has been a big win for our team. Uncrustify runs almost instantly and having a uniformly styled code-base is worth the small inconvenience.

<comment><USER8> I've been doing this from my make file every build. That way, vim will take me right to the error.

</comment></comment><comment><USER9> I can see this working if you're collaborating without Github's pull requests.

Personally I'd want to scope it to only run the test suite for merge commits onto the master branch since it's reasonable in development branches to commit changes with failing tests.

</comment><comment><USER4> Warning, this won't check the code you're committing, it'll check the code you have in your working directory. You need to stash all non staged changes before running the tests.</comment>


Ask HN: Why is IE8 usage growing? - <AUTHOR> It might not be much, and it might be slow, but the last 3 months the IE8 usage is growing at a slow rate[1]. While I expected to see it dead by now [2] from the last year's decline, this is not the case.<p>The second part of the question is, how do we stop this or should we just ignore it? Although this is a particularly more difficult one...<p>[1] http://gs.statcounter.com/#desktop+mobile+tablet-browser_version_partially_combined-ww-monthly-201501-201505<p>[2] http://ux.stackexchange.com/a/64361/19209 <comment> jay-saint Has anyone considered a tax season effect on these results? I used to work for a computer manufacturer that also did re-seller / refurbishing of used computers. December was always when the seasonal tax preparation companies (Jackson Hewitt Liberty Tax), would pop up buying loads of cheap semi-reliable used systems. These tax preparers would then rent whatever cheap strip mall space was available, stick 20 PCs and a few used HP laser printers and connect them all to a decent terminal server.

The small company I worked for would sell a couple hundred of these tax prep systems a year just locally, multiply this out across the US and this could be a correlation. Many of these systems get reloaded, updated and then pulled back off the web to be resold after tax season. On company we sold them to would even let people use a part of their return to buy one of the systems.

<comment><USER2> > and connect them all to a decent terminal server

Why would those systems be running IE8?

<comment><USER3> Windows XP

</comment></comment></comment><comment><USER4> It's because Windows 2003 support ends on July 14th, 2015, and Microsoft is enforcing a browser upgrade to the latest available version (IE8) even for Custom Support Agreements that take effect after that date. So in order for any customer to continue to get 2003 OS support they have to upgrade to IE8.

On the server side, we will see some similar micro-effects as we get closer to January 1, 2016, where IE7 will go down and IE9 will go up due to 2008 having to upgrade, and IE9 will go down and IE11 will go up due to 2008 R2 having to upgrade.

<comment><USER5> How many people actually browse from server editions of Windows? I remember that I skipped XP because I could not stand the Play-Doh interface, so I went from 2000 to 2003.

<comment><USER4> You would hope zero, but there are many Citrix or Terminal Services applications and virtual desktops that give remote users a limited (at least as much as IE6/IE8 allows) browsing experience.

<comment><USER6> Why would you hope zero?

</comment></comment></comment></comment><comment><USER7> You may not believe it but there are a lot of enterprises that are still running Windows XP and just now in the process to upgrade to Windows7. IE8 is the default browser in Windows 7.

Just a guess, but for example, i know for a fact that the german telecom is doing this XP->Win7 migration in the coming months...

<comment><USER8> Yup, my rather large workplace (in the Fortune 100) switched from XP to Win 7 just about a year ago. We have IE8 but they will also let us use an ESR version of Firefox. Other business units than mine get to use Chrome, too.

</comment></comment><comment><USER9> IE8 is coming out of the Trough of Disillusionment and finally reaching the Plateau of Productivity. Obviously.

</comment><comment><USER10> When I follow [1], IE8's numbers are (from Jan) 2.56%, 2.61%, 2.13%, 2.19%, 2.26%. So they don't look to have grown at all.

If anything they've declined a little, although this may be within the margin for error for the sample.

<comment><AUTHOR> 2.13%, 2.19%, 2.26% is what worries me. They dropped dramatically in the same period of last year. From March, they are slowly creeping up.

<comment><USER11> I think the most important part of Petefine comment is _" within the margin for error the sample"_. I would be surprised if the margin of error was inferior to 0.13%.

It's an Human bias to see patterns when there are only random events.

</comment></comment></comment><comment><USER12> IE8 is growing because people are moving off of IE5/6/7\. Soon they'll have to move off IE8 as well, but I imagine that IE8 is the logical stopover since apps that only ran in older IE versions don't require a lot of work to run in IE8. Getting them up to modern browser standards, though, will take time.

Example: many many years ago I worked on a project that used XML Data Islands as an early form of AJAX-like functionality. You could fairly easily modernize it up to IE8 without major changes. But to go beyond that would require heavy rewrites.

</comment><comment><USER13> It's an interesting trend that I have seen among my company's user base. It could be real, but it could also be an increase in false positives.

A possible explanation for the false positives:

Many large companies are seeing more and more machines moving to IE11, but have websites/tools that require IE8. Ideally each web site should explicitly declare compatibility/standard modes they require in their HTML. But for older enterprise software, this is not an option, so companies take the 'easy' approach of enabling compatibility mode for all sites and/or all intranet sites. A better approach when you can't edit the html of the web app is to use the group policy editor and set compatibility mode on a URL by URL basis. But many companies take the easier route. This is my theory that may explain the false positives.

If they aren't doing so, tools that aggregate browser usage should be doing additional analysis on the user agent string to get a better sense of the 'true' IE version. Perhaps using methods like ie-truth [1]. When I see trends like increased IE8 usage, it doesn't make sense to me... so I have doubts that these tools are testing the browser type accurately.

[1] [https://github.com/Gavin-Paolucci-Kleinow/ie- truth](https://github.com/Gavin-Paolucci-Kleinow/ie-truth)

</comment><comment><USER14> Browser stats are noisy – if you look at a larger timescale, you'll see this is actually somewhat lower than December:

[http://gs.statcounter.com/#desktop- browser_version_partially...](http://gs.statcounter.com/#desktop- browser_version_partially_combined-ww-monthly-201405-201505)

The first thing I'd do is check to see whether the apparent trend is supported by other data sources, particularly traffic to the sites you care about. It's worth poking around somewhere like [http://stats.wikimedia.org/archive/squid_reports/](http://stats.wikimedia.org/archive/squid_reports/) or [http://akamai.io](http://akamai.io) to see if other they're seeing the same trend.

</comment><comment><USER15> Much of the corporate utilities in my company seems to only work in IE. There is a whole ecoculture of business software that ignores any other browser. I have bring up a VM to complete some of required paperwork.

</comment><comment><USER16> It must be IE gathering its numbers to face off against Spartan in the IEpocalypse.

Another theory is that corporates/governments are switching over from XP to Windows 7 (with Internet Explorer 8 as default) because XP was declared "dead" (end of life on April 8, 2014 - no more security updates) and the existing PCs aren't capable of running UEFI required for Windows 8...

Windows 7 and Windows 8 is going to get a free upgrade to Windows 10; it's not far fetched to believe some people are installing a fresh copy of Windows 7 on a clean machine to make use of the free upgrade.

<comment><USER17> Windows 8 will run on anything that ran Vista and does not require UEFI. Placing a sticker proclaiming Windows 8 certification or something like that on a new computer requires UEFI.

<comment><USER16> Good to know :)

</comment></comment></comment><comment><USER18> Stats of one of my bigger clients show an almost 50% drop in IE8 in the last year (UK traffic only).

Don't believe everything you read. Also, who actually uses statcounter?

<comment><USER19> I picked a client at random (typical small business in the US) and see a 54% drop in IE8 and a 32% drop in IE9 usage year over year.

IE11 represented the most traffic (for IE users) and it grew by 346%.

Sessions from IE users overall were down 15% for the same period.

Source: Google Analytics

</comment></comment><comment><USER20> IE 8 gets, for the most part, to end of life next January. So it should go back into shrink mode as those IT departments get with the times.

<comment><USER21> Thanks for that info. Verified it on the MS support site and emailed the link to work.

I am working on a project to be delivered to CA Dept of Public Health around January. We were told to target IE 8 as our minimum browser (as well as FF/Chrome/Safari). Seems a waste, given this news. Time to push for IE 9 or better.

</comment></comment><comment><USER22> You can see from this there is a general trend downwards in usage, but there are sudden, periodic, upswings in usage such as December last year.

[http://gs.statcounter.com/#desktop+mobile+tablet- browser_ver...](http://gs.statcounter.com/#desktop+mobile+tablet- browser_version_partially_combined-ww-monthly-201205-201504)

</comment><comment><USER23> Its growing because of China and India large populations of primarily young people whoh every 6 months add tens of millions of internet users who become of age for middle class parents to buy them a computer, or in other cases scrape enough money to purchase the cheapest clone (the very machine which would have a pirated os and ie8).

</comment><comment><USER24> According to NetMarketShare (which has much better stats than statcounter which excludes most corporate users, hardly any of which use Chrome), IE8 was down to around 15% share of desktop users back in November but spiked up to 19% December through Feb. It's now trending downward again at 17% in March and 16% in April.

</comment><comment><USER25> It is most likely because you're including May and this month is incomplete. In June, it's possible when you run the numbers again it will show a different picture, similar to if you review the months past. That is what I've observed at least when using gs.statcounter. Hopefully this is the case.

</comment><comment><USER26> Bots like the IE8 user agent.

</comment><comment><USER27> Many organizations, still use IE8. IT Support in those organization either don't care to upgrade or their internal products just don't work on any new versions.

</comment><comment><USER28> No one's mentioned China yet? Perhaps they're finally updating from IE6 over there.

</comment><comment><USER29> This is a pure guess - people downgrading back to Windows 7. Maybe new laptop buyers install Windows 7 because they don't like 8.1, IE8 is default on Windows 7.

Could be a lot of things, but it is a good chance that this has something to do with Windows 7.

<comment><USER30> It seems unlikely to me that enough people are tech savvy enough to downgrade a version of windows but can't (or don't want to?) switch from IE8 - which we can probably all agree is not a particularly nice browser

<comment><USER31> You don't have to be technically savvy to take advantage of downgrading though, resellers will do it for you. If you go to CDW, TigerDirect, NewEgg, etc. they all are selling PC's now, new and referbed, with Windows 8 licenses but with Windows 7 pre-installed. All you have to know from a technical standpoint is that you prefer 7 over 8.

<comment><USER32> My gut feeling says that a consumer that isn't savvy enough to upgrade a browser is most likely shopping at a big box store, or maybe Amazon.

</comment></comment></comment></comment><comment><USER33> That's because IE is unpredictable in every aspect ! :D

</comment><comment><USER34> Probably many filing up Taxes</comment>