How to steal a developer's local database - <AUTHOR> http://bouk.co/blog/hacking-developers/ <comment><USER2> These have a fairly simple fix that you can implement yourself as a developer. Don't let your services listen on (AKA bind to) 0.0.0.0 or 127.0.0.1.

The entire 127.0.0.0/8 block is dedicated to the loopback interface [1]. That's 2^24 - 2 unique IP addresses you can choose at random. This basically eliminates the feasibility of the DNS rebinding component, as it would take prohibitively long to find the actual loopback address that your services have bound to.

It's important to note that this is much more effective than not using the default port. It's _much_ faster to iterate all 2^16 ports on the same IP address than it is to wait for DNS TTL to expire so you can rebind to another IP address.

As a bonus, you don't have to worry about port collisions when nobody's allowed to listen on 0.0.0.0. Everybody can use 8080 if they want.

[1] [https://tools.ietf.org/html/rfc5735#section-3](https://tools.ietf.org/html/rfc5735#section-3)

<comment><USER3> I'll point out for the benefit of others that not all operating systems treat 127.0.0.0/8 as belonging to the loopback interface.

Linux does, so this will probably work for the majority of users here, but, I believe, it will not work for FreeBSD, for example (I'm on an iPad and don't have an SSH client at the moment so I can't verify). In those cases, however, you can add "aliases" to the loopback interface, with specific IP addresses in the 127/8 subnet, and then use them.

<comment><USER4> On OSX (and, I suspect, vanilla FreeBSD), this looks like (replacing 127.1.2.3 with the address you want to use; 0xff000000 must stay the same, since netmasks of all addresses on an interface must not confict if they're on the same subnet):

        sudo ifconfig lo0 alias 127.1.2.3 0xff000000

And then, when you're done:

        sudo ifconfig lo0 -alias 127.1.2.3

to tear it back down.

<comment> X-Istence Recommendation for aliases on an interface is to use 255.255.255.255 as the netmask on FreeBSD.

For example to add extra interfaces to lo0 on FreeBSD you may use:

      ipv4_addrs_lo0="127.0.0.2-5/8"

in rc.conf

And now those addresses are automatically assigned to lo0, the netmask will be set to 255.255.255.255 automatically for all but the first in that range (.2). Since 127.0.0.1 is unconditionally assigned to lo0, you could also use:

      ipv4_addrs_lo0="127.0.0.2-5/32"

This is what I do on FreeBSD.

<comment><USER4> The ifconfig man page just says the netmask must be "non-conflicting" if on the same subnet as the first network address for the interface (that'd be 127.0.0.1/8 for lo0). I'm presuming the narrower netmask qualifies as non- conflicting?

<comment> X-Istence Yes indeed!

</comment></comment></comment></comment></comment></comment><comment><USER5> In a past life I had to write some DNS rebind attacks for some CPE testsuite software that is out there.

It was very easy to write some javascript that hangs out in the browser, gets the updated DNS host as the 192.168.0.1 address (sure sure, you can go crazy guessing other addresses) and then about 60% of everyone was on admin:admin or something as common; the first 12 or so bits of an ethernet address are the vendor identifier, which makes the process even easier to assume. Then you just start posting data to well-known web admin interfaces and update the router password.

I have no idea how well this works, three or four years later...

<comment><USER6> I wonder there's a solution by requiring a PTR record pointing at each A record which should have Same Origin Policy applied.

<comment><USER7> This would break shared hosting, load balancers, or any other architecture without a unique IP address per hostname. I don't think this is feasible unless it's applied to IPv6 only due to IPv4 scarcity.

<comment><USER6> You can have multiple PTR records.

</comment></comment></comment></comment><comment><USER8> _> The attack depends on multiple software products all making very reasonable decisions about how they should work, but the way they interact with each other leads to a vulnerability._

I'm sorry, but I disagree. A browser allowing externally loaded scripts to access private ip ranges is not a reasonable decision.

PSA: To protect yourself from this, and some more bad browser defaults, use NoScript with "allow all scripts globally." Keep the JS, but filter out some bad stuff. Also enable ABE (application boundaries enforcer, made to solve exactly this problem) for good measure.

<comment><USER9> _A browser allowing externally loaded scripts to access private ip ranges_

There are plenty of legitimate systems in the ecommerce world that rely on doing this.

Take Ariba Punchout, for example. The idea is for the store to pass the user's shopping cart back into the customer's internal ERP system to be turned into a purchase order and sent for approval. The way this is handled is that the initial request sets up a session including a URL to a customer-internal service that's intended to handle the transaction. When the user completes checkout, a page is returned to the user containing an XML payload, and script to submit that XML to the specified internal service URL. That way, the store is able to provide the PO data to what is otherwise a completely internal system.

<comment><USER8> Which can all be handled using an (automated) whitelist, à la CORS.

Too many things run on localhost (or intranet) and assume that localhost=safe. I know, "that's wrong" and "don't listen on 127.0.0.1" and all that, but that horse has left the barn, emigrated, founded a family and died happy. We can't put the 127/192/10 genie back in the bottle.

Prevent access to all private resources from the outside and whitelist on a need-to-access basis. Or be ready to keep monkey patching your system against these exploits forever.

</comment></comment></comment><comment><USER10> Now I'm a little scared.

A few months ago there was a post [0] by antirez about how dangerous it is to leave a redis instance open to the world, in that an attacker could, for instance, authorize an SSH key on your machine and gain remote connectivity.

While the average workstation is not usually reachable from the outside network, you could probably combine some variant of that attack (the first thing that comes to mind: overwrite .bash_profile) with the attack of this article, causing a lot of fun.

[0] [http://antirez.com/news/96](http://antirez.com/news/96)

</comment><comment><USER11> Running your browser in Red Hat's SELinux sandbox [1] [2] limits the ports you can connect to and thus limits this type of attack to those ports (80, 81, 443, 488, 8008, 8009, 8443, and 9000 in the default configuration).

[1] [http://danwalsh.livejournal.com/31146.html](http://danwalsh.livejournal.com/31146.html) [2] [http://www.bress.net/blog/archives/195-Firefox-in-a- sandbox-...](http://www.bress.net/blog/archives/195-Firefox-in-a-sandbox-with- Fedora.html)

<comment><USER12> This is the only advice in this threads that actually solves the issue.

Doesn't have to be SELinux, any of the frameworks will do. Or run it in a new network namespace.

<comment><USER13> Sort of. It solves this specific attack.

If you were attacking a local webapp interface instead of a non-http daemon like redis, you would need your browser to be able to access the web service. At that point, this kind of attack would still allow an attacker to also access that web service.

</comment></comment></comment><comment><USER14> I have seen this story posted and discussed in several locations. It boggles my mind that everyone is talking about DNS filtering and/or browser security models, when it's painfully obvious that the actual problem is the fact that the targeted services (redis, memcached, elasticsearch, etc.) apparently do nothing whatsoever to authenticate incoming connections (at least in their default configuration).

Yes: remote DNS servers have no business serving up loopback addresses. Yes: browsers shouldn't let remote scripts access resources on the local network.

But WTF are you guys doing running services bound to network ports (even if only accessible from the local machine) that apparently have no authentication whatsoever? Have none of you ever used a multi-user machine?

When I was in university we had just three SunOS boxen shared amongst all undergrads in my faculty, and all three were directly accessible from the whole of the internet - there was no firewall of any kind. Even back in those rather more innocent days you learned real quick not to put up services which didn't authenticate every incoming connection.

A good firewall is not a substitute for having individual machines be secure.

A machine having only one (intended) user is not an excuse to run services that are not secure against local users.

</comment><comment><USER15> Interesting attack. A far more feasible one is just to throw nmap around your next conferences WiFi network and try common postgres/mysql combinations. You'd be surprised how many developers have such services exposed, often with 'developer passwords' and production dumps loaded.

<comment><USER16> I think it'd be very common to "protect" these services by making them bound only to localhost. The fact that this attack bypasses that protection is pretty interesting.

<comment><USER17> It's also common to open these up so that team members can grab a copy of your database. I haven't done that, but I can think of a case in the past few months where a developer had done so.

Edit: Now that I think of it and especially with containerized dev environments and VMs, I'd bet it quite common. I'm sure I've opened up a DB or search container more than I needed to just because I couldn't get the damn things to talk. I still would have a firewall, but not everyone does.

<comment><USER18> Yeah, I'm pretty sure I've bound services to 0.0.0.0 just to avoid the trivial annoyance of checking an IP.

</comment></comment></comment></comment><comment><USER19> Question: could DNS rebinding be used to tap into 1Password inter-process communication? They use localhost websockets for IPC; it's authenticated through the request origin and then through verifying the PID is in fact the browser [1].

DNS rebinding could definitely get around the PID check, but could it spoof an origin to something like "safari- extension://com.agilebits.onepassword4-safari-2bua8c4s2c"?

[1] [https://support.1password.com/mini-extension- security/](https://support.1password.com/mini-extension-security/)

<comment><USER20> I don't see how this would work, seeing as the safari-extension protocol doesn't get the location to go to based on DNS.

<comment><USER19> Well first off, I honestly don't know if it's possible; if I knew, I'd just shoot an email off to 1password.

That being said, I don't think we're suggesting the same thing. I'm saying one could write a webpage that uses DNS rebinding to make requests on localhost, like OP. Then, the webpage, completely bypassing the browser's built-in extension system, makes a request to 1password over localhost (which they're already using for IPC).

The reason that DNS rebinding is relevant is twofold; first, you need the request to hijack existing IPC between the browser extension and the standalone desktop app, and two, because you need to have the request come from the browser's PID. That should all work.

The question here revolves around 1password's verification that the origin is coming from, for example, something similar to

> safari-extension://com.agilebits.onepassword4-safari-2bua8c4s2c

So then your webpage would also need to be able to spoof the origin of your localhost requests to look like they were coming from that origin. I don't know if that's possible or not, but if it is, it would imply that this technique could get you illegitimate access to 1password.

</comment></comment></comment><comment><USER21> Browsers could pin DNS responses when a page finishes loading so any further requests for that domain will use the cached IP instead of doing name lookups but that would be a PITA because they generally rely on the OS DNS subsystem. It might also break long-running pages that won't failover anymore.

It would probably be easier to simply keep a IS_LOOPBACK flag for every DNS name resolved and kill any connection attempts if the flag changes while the page is loaded. Then you can keep using the OS DNS resolver logic.

DNS might legitimately return a different CDN but it sure as hell won't flip between private IP spaces and the public internet.

<comment><USER22> Captive portals [0] often return short-lived DNS responses to private IP space until you pay or login, at which point their DNS servers start returning proper responses (and stop intercepting DNS requests). Although not "legit", this is a case which is very common - you wouldn't want your browser to suddenly stop working because you connected to a captive portal that started modifying DNS responses.

[0] [https://en.wikipedia.org/wiki/Captive_portal#Redirection_by_...](https://en.wikipedia.org/wiki/Captive_portal#Redirection_by_DNS)

<comment><USER23> You're both correct. I hope they find a solution to fix the utterly broken garbage that captive portals are: Every "portal" is different and complete prevents automated internet access. I hope this gets fixed somehow, although I don't have high hopes: [https://datatracker.ietf.org/wg/capport/charter/](https://datatracker.ietf.org/wg/capport/charter/)

</comment></comment></comment><comment><USER24> Recently Redis aliased POST and Host: to QUIT for this reason.

<comment><USER25> It seems to me that they should quit on reading any input that they can't parse. At best silently ignoring bad input leads to software not doing what it's supposed to do. At worst it leads to attacks like this.

I realize this is contrary to "be conservative in what you do, be liberal in what you accept from others", but I never thought that was a good way to write software.

<comment><USER26> That's at the very least a much harder change to roll out, since if things are broken after a new version of Redis comes out people will blame Redis for breaking things.

<comment><USER25> I'm not very familiar with Redis, so I might be missing something here.

But if the choice, after a Redis update, is between: a) my software breaking and, hopefully, an error message saying "redis failure parsing 'XYZ'" or b) my software /maybe/ continuing to function, while passing commands to Redis that it's ignoring

I would always pick (a), and I think most programmers would think likewise.

<comment><USER26> Let's say you're using a cache server as a best-effort cache, and you take advantage of its ability to store complicated data structures. Your client implementation has a small bug with one of them, and ~1% of the time it sends something to the server that's not to spec. Right now, the server returns an error for that specific request, but doesn't drop the connection and continues processing later requests on the connection. You know about the errors, but they're not worth fixing.

Now the cache server has a security update, so you apply it right away. But now when it gets your invalid command it not only returns an error but it drops the connection. Your client doesn't handle this well, and now your caching is fully broken and your server falls over from the load.

<comment><USER24> That's exactly my point of view on the matter.

</comment></comment></comment></comment></comment></comment><comment><USER27> When you hear about someone jail breaking an iphone trough the browser, this is how. The fact that the browser works as a window to all tcp-sockets running on a device, it's the perfect way to exploit buffert overflows on a device that lacks a terminal.

Also remember this with all your IoT appliances running on your local network. Even if it has a local IP-address, as long as you have a computer with a browser on the same network, you might as well consider your devices being publicly accessible from the rest of the internet.

<comment><USER28> what? I haven't heard of a single jailbreak that worked like that. You got some links?

edit: to people linking me safari CVEs (can't reply to you all) - these are memory corruptions. Not accessing ports on localhost.

<comment><USER29> Since no one has provided you with a link, here's one such approach: [http://www.redmondpie.com/how-to-jailbreak- ios-9.3.3-using-s...](http://www.redmondpie.com/how-to-jailbreak- ios-9.3.3-using-safari-no-computer-required/)

I think one of the first sites to do it was this one: [https://jailbreakme.qoid.us/](https://jailbreakme.qoid.us/)

<comment><USER22> Your first link appears to require the installation of an app on the phone (after installing a dev certificate). The app, running arbitrary native code, performs the actual jailbreak (most likely using a chain of kernel exploits).

The devs seem to have gotten around the App Store by using an approach like TestFlight, which allows apps to be deployed for development and wider testing purposes without going through the App Store.

</comment></comment></comment></comment><comment><USER30> Cool to see DNS Rebinding getting more attention lately :D just a couple months ago I used DNS Rebinding to attack Ethereum wallets: [https://medium.com/@rhodey/walking-past-same-origin- policy-n...](https://medium.com/@rhodey/walking-past-same-origin-policy-nat- and-firewall-for-ethereum-wallet-control-30c29b73a057#.ew1ir61kg)

<comment><USER31> Nice one! Not surprising to see cryptocoin implementers make all the classic security mistakes

</comment></comment><comment><USER32> Do developers often run things on localhost? I mean sure, you'll have things running on your dev machine, but for me at least, [http://127.0.0.1/](http://127.0.0.1/) will just show the default webroot, with its placeholder index.html. All my actual sites listen for custom hostnames (since otherwise you only get one site per machine or have to do silly things with port numbers on the url).

So unless somebody has crafted a page specifically targeting me and my naming convention for local sites, this wouldn't be an issue. And of course, once you hit a site, you'd still need to deal with the same security that the public facing version sees. You certainly wouldn't go out of your way to disable that on your local machine.

Databases are named, and often live within named database server instances, so they'd need to be specifically targeted as well. And, again, they have authorization to deal with. It's not like you'd leave that open either.

Is it common to do it any other way?

<comment><USER33> Yes, I've always run everything on localhost and use different ports, I think most devs are the same. I've never even thought of doing anything else

> All my actual sites listen for custom hostnames (since otherwise you only > get one site per machine or have to do silly things with port numbers on the > url).

It's not just web servers though, every from databases to caches servers.

>Databases are named, and often live within named database server instances, so they'd need to be specifically targeted as well

I think most would stick with the defaults. With mssql server it's "." or "sqlexpress" or whatever.

>And, again, they have authorization to deal with.

I've always used windows authentication, but that now seems like a terrible idea.

</comment></comment><comment><USER34> In what universe does it make sense for a DNS server to serve records that point to 127.0.0.1??

This DNS Rebinding hack should be easy to eliminate: only etc/hosts can make records that point to localhost.

I don't care if a HUGE swath of applications are broken by this, DNS just seems broken to allow remote DNS servers to make localhost records.

Please excuse my awkward use of words, I am a combination of sick and delightfully ignorant about DNS.

<comment><USER22> DNS rebinding can also be used to attack private addresses, for which it would be perfectly legal to have real DNS names (think DNS entries for a company's intranet). Blacklisting localhost only offers a bit of security against silly devs, but arguably pivoting an attack from a public server to a private network is much more valuable and harder to stop.

<comment><USER34> Well, that sounds super bad too!

There should be some central network policy in the enterprise that says a DNS record coming from outside the company can't point to addresses on the 10.xx.xx.xx or 192.168.xx.xx spaces. I'm surprised that isn't already the common configuration.

<comment> X-Istence Unbound has that as a possibility in the recursive DNS resolver.

Super simple to set up too:

            # Enforce privacy of these addresses. Strips them away from answers.
            # It may cause DNSSEC validation to additionally mark it as bogus.
            # Protects against 'DNS Rebinding' (uses browser as network proxy).
            # Only 'private-domain' and 'local-data' names are allowed to have
            # these private addresses. No default.
            private-address: 10.0.0.0/8
            private-address: 172.16.0.0/12
            private-address: 192.168.0.0/16
            private-address: 169.254.0.0/16
            private-address: fd00::/8
            private-address: fe80::/10
            private-address: 127.0.0.0/8

</comment></comment></comment></comment><comment><USER35> This problem doesn't just apply to localhost, although it's most straightfoward to exploit that way. You could also use this technique to scan the user's LAN or, in a more targeted attack, bypass IP address restrictions on specific servers.

Scripts from the public Internet shouldn't be able to access private or local networks as a matter of policy.

Similarly, in a high-security environment, scripts from a private network shouldn't be able to access the public Internet - to help prevent exfiltration of private data.

<comment><USER36> > Scripts from the public Internet shouldn't be able to access private or > local networks as a matter of policy.

I agree, and it's encouraged some pretty stupid practices where it is used for things _other_ than espionage or malicious intent.

Lenovo has a driver detection / update tool on their website, to run it you download a helper application that opens up a HTTP endpoint on localhost, then their website uses it to scan your system and (hopefully) shuts it down afterward. Why was this done in the first place, forcing users to download and run an executable (which has to be restarted each time you scan) that has no UI except for the web browser tab you already have open is dumb.

</comment></comment><comment><USER18> I always kinda wondered when the other shoe was gonna drop on "why do databases even have passwords"...

<comment><USER37> I prefer key-based auth or local user based auth.

</comment></comment><comment><USER38> I'm sure I've seen browsers (maybe Opera?) which wouldn't let a website on a "routable" IP address make any requests at all to anything on a "non-routable" IP address; I assume that 127/8 is included in the latter range. That approach basically eliminates the DNS rebinding attack, I assumed that was normal practice in all browsers - obviously not, though.

</comment><comment><USER39> Another good reason to develop w/in a virtualized server on your dev machine?

<comment><USER40> No.

If you are going to go to that much effort, better to just spend a little bit of time on security.

<comment><USER41> The gain by using VMs and provisioning sofware is not limited to security. There are a lot of good reasons to do that.

<comment><USER42> Correct. Like developing to a common deployment target, provisioning a development VM with the same toolchain you use to provision production VMs, isolating your VM from the system state of your development machine. I cannot imagine developing any other way.

<comment><USER43> I did this for a while, but it falls apart with constant care and feeding. I had a vagrant box that could be fully provisioned with Ansible, and checked everything in to the repo. I paid for Parallels because virtualbox is too slow to do any real work (unit tests take 2x or 3x longer to run for example).

I couldn't convince anyone else to use this stuff. They were happy to just run some homebrew commands every now and then.

I was away from the project for about six months. When I got back I thought "great, I'll just `vagrant up` and be back in business."

Parallels had upgraded itself to some version that wasn't compatible with vagrant. I was eventually able to find an old parallels installer and downgraded.

My laptop had a newer version of ansible, and it couldn't run the old ansible scripts. There was no easy fix to something that should be easy (ansible could no longer create postgres users when ansible was running through a non-root account. With vagrant you log in as the `vagrant` user and that user has sudo).

I deleted all that stuff, and now I do all my development locally. I do run my DB's in docker containers, but that's it. Now it is super easy to add any version of any database to any project. Just a few lines in docker- compose.yml. But my docker-compose.yml was written for an older version of docker-compose. I tried to upgrade it to the latest syntax and nothing worked. I reverted that and things are still running. It is only a matter of time before some docker update renders my db and redis useless.

At that point I'll just `brew install redis` and `brew install postgres` and be done with it. Everything will run, and run at native speeds. Yay!

<comment><USER42> I think this is overstated. This only falls apart if you cannot get buy-in from your development team, and are not applying proper version management to your tool chain.

First, we use Ansible, not just for development, but for deployment. Because of this, if the time comes to upgrade Ansible, everybody upgrades, and any incompatible playbooks are immediately addressed.

Second, Parallels is a bad choice. They do not have a sufficient enterprise focus, and their product is updated too often. It's great for desktop virtualization, but not for much else.

We use VMWare Fusion, have no performance issues, and have never experienced the issue of hypervisor bit rot. On the one occasion when we upgraded VMWare, we also upgraded the corresponding Vagrant plugin, and everything worked fine.

<comment><USER3> At least we won't have to worry about any future VMware Fusion upgrades breaking anything, since there quite possibly won't _be_ any future upgrades.

</comment></comment></comment></comment></comment></comment></comment><comment><USER44> My takeaway from this is that binding to localhost doesn't give you as much security as you think.

</comment><comment><USER45> Looks like the best way to protect against this is filtering private IP addresses from DNS responses. Is there a reason why ISP DNS servers in general would ever need to serve out a private IP?

<comment><USER46> The only problem with that is all ISPs will have to configure their DNS systems to do so (not going to happen soon). The only reasonable fix would be applied at the browser level as they are updated reasonably often.

A better mitigation than that mentioned in the article is that browsers should ignore a DNS update if it goes from a public IP to a private IP range. DNS pinning as suggested would cause havoc with most wifi captive portals, especially for those not computer savvy or the badly configured/implemented captive portals.

<comment><USER47> I think you may be able to discover actual IP addresses with WebRTC. If the host is on IPv6 then it may well be a public adress. Hence it would be impossible to define "local" in this setup.

<comment><USER46> As far as I'm aware[0] there are private IPv6 addresses that should be treated the same as 169.254 and 10/172/192 in IPv4. It shouldn't matter what IP the host has, just what IP the address of the site you are communicating with goes from the public range to private range.

[0]: [https://tools.ietf.org/html/rfc4291#section-2.5.6](https://tools.ietf.org/html/rfc4291#section-2.5.6)

</comment></comment></comment></comment><comment><USER22> This is an interesting, albeit well-known attack vector. A similar attack was used to attack Avast [0].

The author notes that write access could be used to inject dangerous objects (e.g. malicious pickles) into the database. This is arguably a much more serious bug because it does not require DNS rebinding (such a request can be performed cross-origin) nor can it be mitigated by refusing to read the response (as Chrome is proposing to do).

In short: the database modification attack is potentially much more severe, but as of yet no precise attack chain has been identified. However, I think it's very likely that some server software uses e.g. pickles in the database.

[0] [https://bugs.chromium.org/p/project- zero/issues/detail?id=67...](https://bugs.chromium.org/p/project- zero/issues/detail?id=679&redir=1)

</comment><comment><USER48> Hmm, Little Snitch, if configured properly (ie. you allow the browser to only connect to ports 80 and 443) will alert you if a site wants to connect to something weird like 3306, 9000 etc. Then you can kill the packet and nothing happens. Like on OPs PoC. Still, it's super interesting PoC.

<comment><USER49> This kind of thing is exactly why I put up with the hassles of running Little Snitch.

</comment></comment><comment><USER50> I run my real databases on non-standard ports in docker and put honey pots on the standard ports. In those I fill it with dialog from love scenes in popular movies, it's not a ton of data but it's certainly interesting.

<comment><USER51> That's cute. Do you happen to have it on GitHub? Would love to see what comes through.

</comment></comment><comment><USER52> similar problem with intellij: [https://news.ycombinator.com/item?id=12292148](https://news.ycombinator.com/item?id=12292148)

</comment><comment><USER53> I love this class of attack. I wonder if you could use a similar technique to get German Trasmission users to download fine-inducing torrents, etc..

</comment><comment><USER54> I've become increasingly complacent and often allow NoScript to "temporarily allow all javascript on this page", but will stop doing it, having just tried the PoC. It found Redis (which runs in a container, but with the port exposed).

The PoC failed to work when using TorBrowser (with the security slider set to High) and letting NoScript temporarily allow.

</comment><comment><USER55> We are designing a solution in gngr here: [https://github.com/UprootLabs/gngr/issues/219](https://github.com/UprootLabs/gngr/issues/219)

In essence, the resolved address of a request will be checked if it lies in a reserved block. If so, further policy checks will be made for the resolved address, and the IP address will be pinned for that HTTP request.

Would appreciate feedback here, or on the issue.

</comment><comment><USER56> "How to steal some developer's local database"

</comment><comment><USER57> This reminds me of a similar vulnerability in webhooks [1]. I never thought of throwing a POST request at Redis to muck with keys but I tried it just now and it totally works. Geez.

[1] [http://blog.fanout.io/2014/01/27/how-to-safely-invoke- webhoo...](http://blog.fanout.io/2014/01/27/how-to-safely-invoke-webhooks/)

</comment><comment><USER58> Routers like Fritzbox also protects agains DNS Rebinding: [https://en.avm.de/service/fritzbox/fritzbox-7390/knowledge-b...](https://en.avm.de/service/fritzbox/fritzbox-7390/knowledge- base/publication/show/663_No-DNS-resolution-of-private-IP-addresses/)

</comment><comment><USER47> With WebRTC local ip discovery [1] it can be easily extended to work against a whole local subnet. Looks very dangerous.

Probably best to attack this on the DNS rebind level. Encapsulating the browser network context somehow and firewalling this might help mitigating this attack too.

[1] [https://www.browserleaks.com/webrtc](https://www.browserleaks.com/webrtc)

</comment><comment><USER51> Seems to me that containerizing your dev environment with something like a well-constructed _docker-compose_ YAML should mitigate this.

By "well-constructed" I mean that the backend services for a given project should only be available on the container network, and not be exposed to the host network.

</comment><comment><USER59> Out of curiosity, what data do you have in your development databases that this becomes such a grave concern? I mean I'm all for security and love to see how creative people can get but we are talking about dev environments and not some part of the infrastructure (automated test machines, production, etc).

<comment><USER60> If you had read the whole article, you would have the answer to your question. Es. It can get you to code execution if you poison python pickled data. Ecc..

<comment><USER61> I actually thought the opposite. Surely the database is more valuable than the ability to execute code. Assuming its a copy of the production database.

</comment></comment></comment><comment><USER62> The crazy thing to me is that people here look for solutions at lower OSI level (DNS, interfaces, IP address,) where to me the problem is that there are these services that run with zero security.

Fix the services, require authentication and permission enforcement and the problem is gone.

<comment><USER33> >Fix the services, require authentication and permission enforcement and the problem is gone.

If the attack is coming from a browser it would authenticate with the users credentials.

</comment></comment><comment><USER63> I noticed that I am running redis-server in the background on my mac...but I have no idea what started it. I'm not doing any relevant development at the moment, and haven't in months.

How can I trace the source and ensure it doesn't restart on reboot?

</comment><comment><USER64> It's very concerning considering Homebrew's popularity and its habit of running stuff as your local user. Compromising any application that runs as you with as much access to your computer as yourself is pretty bad.

</comment><comment><USER65> Perhaps a server, when running in development mode, should require a custom HTTP header? This would be a non-simple request, and the browser will refuse. Would this be a reasonable counter-measure?

<comment><USER66> A secret value, whether it's called a 'password', a 'key', a 'token', or comes in an 'Authorization' header or 'X-CustomHeader' is always a good countermeasure.

<comment><USER65> Right, but a custom HTTP header does not even have to contain a secret. I just has to have a non-standard name. Firefox will refuse setting it, right?

</comment></comment></comment><comment><USER67> Can this be solved by configuring the local system (e.g. Debian?) to blacklist any DNS resolution that ends up being a private IP address? Is this is possible to configure at the firewall level?

<comment><USER47> How about local IPv6 addresses? I can easily see this being left out due to sheer complexity.

<comment><USER67> I'm guessing that even in a future in which we have 100% of IPv6 deployment, we would still run our loopback/LAN interfaces with IPv4 for simplicity...

<comment><USER68> It's too late. I have already run into issues on Linux distros with some daemons binding to 127.0.0.1, but localhost resolving to ::1.

</comment></comment></comment></comment><comment><USER69> Create a docker container, mount -v /tmp/.X11-unix:/tmp/.X11-unix in there and run your browser that way. It's VM lite and mitigates this attack.

</comment><comment><USER70> This is why I use Vagrant whenever I can. All my services live within a VM and are only accessible via a specific hostname.

</comment><comment><USER71> The PoC website works on my redis even when using Chrome canary 55 which should have removed HTTP 0.9.

</comment><comment><USER42> This is a non-issue for those of us who are using Vagrant or similar development VM technologies.

<comment><USER72> Not necessarily since it's not that uncommon to expose the ports from the VM. Not saying that's a good idea, but it happens fairly often.

<comment><USER42> These exploits rely upon the ports being exposed on localhost. Even if you are not (as you should) using ssh tunneling to the VM, any exposed ports are on a separate IP address, not localhost. However, if one is forwarding ports to the host machine, that's obviously a problem.

<comment><USER73> These exploits rely upon the ports being exposed _on a predictable IP_. A lot of setups are going to give you very predictable IPs for the virtual machines.

<comment><USER74> Thanks to WebRTC, it's even easier - you can enumerate all local IP addresses and then you only have to scan one or two subnets.

Local VMs are _definitely_ vulnerable to this as well.

<comment><USER75> Not if you actually work in the VM, and do not expose database ports to the host (why in hell would you do this anyway). No amount of ip scanning is going to open ports.

<comment><USER76> > why in hell would you do this anyway

Integration with IDEs.

<comment><USER75> > _Integration with IDEs._

That's a fair point, but Rsync or shared folders should have you covered for this.

<comment><USER76> Actually I meant something like how Intellij has DB integration, so you could put in your host & port and connect to the DB to help you write queries etc.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER77> Is there any similar attack for data in postgres / mysql local databases?

</comment><comment><USER78> Looks like Qubes OS is the only ultimate solution.

</comment><comment><USER79> The DNS rebind seems weird, any sensible DNS forwarder should ignore local ip (127.0.01/8, 192.168.0.1, etc). This attack doesn't seem feasible if you can't hinack local adresses.

</comment><comment><USER80> I think that running elastic search on a different port (other than 9200) will go undetected? (just a thought)

</comment><comment><USER81> If the box doesn't beep any longer - how can I find where I stored my speaker- less server?

</comment><comment><USER82> Oh no, my development database! What ever will I do if 10,000 entries of Lorem Ipsum get leaked!? In the wrong hands, all of my bunk data from trying to get a PUT right could be really dangerous.

<comment><USER83> you joke, but I have to _constantly_ tell our devs "no, you can't have actual prod data to test your queries against"

<comment><USER84> Fine, as long as you can produce replacement data with the desired qualities.

<comment><USER85> it's called a test fixture.

<comment><USER86> Frequently data produced by hand (by developers writing the tests) or randomly generated (by something like quickcheck) does not have all the edge cases you would encounter in real data.

So there is some justification to having real data available - this is why in the finance space at least developer machines have relatively stringent restrictions placed on them compared to a lot of other organisations.</comment></comment></comment></comment></comment>


Why I Despair for Computer Literacy in the UK - <AUTHOR> https://artplusmarketing.com/why-i-despair-for-computer-literacy-in-the-uk-b5b82383fd2b <comment><AUTHOR> Yeah, I don't usually post an article I've written myself here, since most aren't relevant to Hacker News. But in this case, I think it's important for as many people to know about the state of the UK education and computer science teaching's terrible quality as possible.

We cannot afford to see kids leave school at 16 (or even start university to learn computer science at 18) with no knowledge of how to program or develop anything.

What's worse, this isn't even ancient history. I mean, I'm 26 at the moment, and this is literally all I was taught about IT and computer science through high school and sixth form college. We've got a whole generation (or more) of young people who could be getting started in the tech industry but don't anything other than how to type out a few documents in Microsoft Word. I actually suspect it might even be one of the reasons why London doesn't have as many successful startups as the US and Silicon Valley. Because the skillsets necessary to succeed with a web business just aren't here.

Either way, it needs to be fixed pronto. Otherwise we will continue to have a situation where most of the UK knows nothing more than how to use Microsoft Office programs and the odd web browser.

<comment><USER2> If it was anything like my school (although we did have proper IT course in the final 2 years) they don't even teach kids to use word well. It's just rote memorization of a specific version of word. We learned much more about computers trying to get games to work on DOS because we had to do stuff like enable extended memory.

<comment><AUTHOR> Yeah, it was all based on one of the Windows XP versions of Microsoft Office when I was at school. So the skillsets likely wouldn't transfer perfectly into the newer ones with the ribbon UI.

And playing games on DOS? If that was in school, you had a far more enjoyable IT education than I ever did. Don't think we ever got to play more than a few flash games in class, and even that was only when the teacher wasn't looking.

If you mean at home... then yeah, that's a possibility lost for kids who grew up with Windows XP or later. Very few kids now would even see why they'd want to play a game (or use any program for that matter) built for MS DOS.</comment></comment></comment>


Ask HN: Best setups to avoid availability outages on AWS - <AUTHOR> Using only AWS services, what do you put in place to help prevent disruptions when a single availability zone goes down?
The most simple would be to set up your instances in a multiple AZ's and then configure the ELB to round robin requests until the health of one of the instances is poor.
Any other thoughts? <comment><USER2> I hate to sound like a simpleton, but for a small operation you're best off putting all your eggs in one basket.

I'm in one of the us-east zones and I haven't had a failure in at least a year. They retired one machine I was using and dealing with that was as simple as starting and stopping -- at a time I chose.

With five zones in U.S. East, the probability of a zone failure affecting a single zone systems is 1 in 5.

If you're a busybody who spreads your system across five zones, the probability of a failure affecting you becomes 1.

You're spending more money, and dealing with a lot more complexity, all to increase the probability that hardware failures will affect you.

Now, you're hoping that a zone-distributed system will be able to recover from failures, but that's tricky to do and it's quite unlikely that this will work if you haven't tested it. Add the fact that all the other "cool kids" will be trying to recover their systems at this time and make AMZN's control plane go down.

In the meantime, with probability 4/5 I'm sleeping through the disaster and the first time I hear about it is on hacker news.

<comment><USER3> This is correct; probability works against you if you spread your instances over multiple zones without also compensating with some very solid failover. If you are not willing to put the effort into truly going fully active/active/active across THREE availability zones, you might as well just stick with one and weather the outages.

<comment><USER4> I think that's a bit of an exaggeration. My old company (Acquia) kept a lot of customers up through the various outages with mere two-zone redundancy.

You do need to actually engineer for failover and recovery, though. That point is correct: It doesn't necessarily help you to have half of your servers survive an outage if that still kills your service or requires painful amounts of manual intervention. And it really is a lot easier to just shrug, keep your backups up to date, and pray for your zone to come back ASAP. Do your cost- benefit analysis.

</comment></comment></comment><comment><USER5> My thought: Have a very nice screen for your mobile app/website that says "We are down for maintenance, please stand by."

Sorry to be fatalist, but it's a hard problem. This last outage was more than just an AZ failure. Region-wide API usage was affected, so operations like static IP reassignment and ELB changes were not taking effect. This means you are hanging out in the wind should there be something unusual that requires manual intervention (as was the case with us).

Route 53 is a good service but I don't know how its control plane works, and it could be that problems in a single region would disable the ability to update DNS records (I would guess that DNS reads are a lot more available than writes). And in any case DNS is not a very good failover mechanism due to upstream caching.

Unless your business model requires higher reliability than Instagram, Netflix, and Pinterest, I'd suggest going multi-AZ, crossing your fingers, and doing everything else right.

<comment><USER6> I think Route 53's TTL is 60 seconds.

<comment><USER5> TTL is not always honored -- some caching nameservers peg it to a lower limit, and Android has a bug (just recently fixed) that pegs it to 10 minutes: <http://code.google.com/p/android/issues/detail?id=7904>

</comment></comment></comment><comment><USER7> _Mindless rambling ahead; I love this topic_

Richard,

AWS is an amazing tool and you have a few options here, but the downside is the more options you use to be highly-available (HA), the more expensive AWS gets (as you would imagine).

Your first option is to be HA across a SINGLE region; to do this you make use of the elastic load balancers (ELB) + auto-scaling. You setup auto-scale rules to launch more instances in different availability zones (AZ) either in response to demand or in response to failures (e.g. "always keep at least 3 instances running").

You compliment that with an ELB to load-balance incoming requests automatically across those instances in the different AZs. This is all fairly straight forward through the web console (except auto-scaling is still done via CLI for some reason)

If you want to be HA ACROSS regions you can't just use ELBs anymore, you have some added complexity and an additional AWS feature you will likely want to use: Route 53.

Route 53 is Amazon's DNS service which offers a lot of slick DNS-features like removing dead points from DNS rotation, latency-based routings, etc. There are also something like 29 deployments of Route53 (and CloudFront) around the globe so you'll hopefully never have Route53 become a point of failure for you even disaster strikes.

In this scenario you would setup the HA configuration for a single-region as mentioned above, but you would do it in multiple regions. Put another way, 2+ servers in multiple AZs in _each_ AWS region. Then a Route53 DNS configuration setup to point to each ELB in each region representing those individual pockets of servers.

Ontop of that you would use Route53 to manage all routing of client requests into your entire domain; you can leverage the new "latency-based routing" (effectively why everyone was asking for GeoDNS for years, but even better) and monitor capability to ensure you aren't routing anyone to a dead region.

SIMPLIFICATION

\</comment><comment></comment><comment>--

Here is what I would recommend given the size of your budget and need to stay up in the AWS cloud, in-order of expense:

1\. Launch a single instance in a region with acceptable latency that has never had an outage before (e.g. Oregon has never completely gone down but Virginia has -- yes yes I know VA is older, but you understand my point). This solution will be cheaper than multiple instances in _any_ region.

2\. Launch multiple instances using the web console, in multiple AZs in US- EAST (cheapest option for multi-instances) and front them with an ELB. You skip any auto-scaling complexity here but you need to keep an eye on your servers. I think ELB fixed the issue where it would effectively route traffic into the void if all the instances in an AZ went down.

OPTIONAL: If you didn't mind spending a few $ more, you could do this strategy in the region that has never gone down for added piece of mind.

3\. Launch single instances in multiple REGIONS and front them with Route53. This isn't really a recommended setup as entire regions will disappear if you lose a single instance, BUT I said I would list possibilities in order of price, so there you go. You could mitigate this by setting up auto-scaling policies to replace any dead instances quickly in the off chance you wanted to do exactly this but not babysit the web console all day.

4\. Launch multiple instances in each region, across multiple AZs fronted by ELBs and then the entire collection fronted by Route53.

NOTE: The real cost comes from the additional instances and not from Route53 or the ELB; so if you can use smaller instances to help keep costs down (or reserved instances also) that might allow you to provide a larger HA setup.

What about my data?

\</comment><comment></comment><comment></comment><comment></comment><comment>-

Yes, yes... this is an issue that someone already touched on (data locality below).

You will have to decide on a single region to hold your data; in this case I would recommend using DB services that aren't based on EC2 and have never experienced outages (or rarely) -- this includes S3, SimpleDB and/or DynamoDB. AWS's MySQL offering (RDS) are just custom EC2 instances with MySQL running on them, so any time EC2 goes down, RDS goes down.

The other DB offerings are all custom and except for SimpleDB a long time ago, have never experienced outages that I am aware of.

Making this choice is all about latency and which DB store you are comfortable with (obviously don't choose SimpleDB if everything you do requires MySQL -- then use RDS); you'll want your data as close to your web tier as possible, so if you are spread across all regions you'll just want to pick a region with the smallest latency to MOST of your customers (typically West coast if you have a lot of Asia/Aus customers and East coast if you have a lot of European customers).

Want to Go to 11?

\</comment><comment></comment><comment>-----

If you have the money and desperately want to go to 11 with this regional- scale (which I love to do, so I am sharing this) you can combine services like DynamoDB and SQS to effectively create a globally distributed NoSQL datastore with behavior along the lines of:

1\. Write operation comes into a region, immediately write it to the local DynamoDB instance, asynchronously queue the write command in SQS and return to the caller.

2\. In 1+ additional EC2 instances running daemons, pull messages from SQS in chunk sizes that make sense and re-play them out to the other regions DynamoDB stores; erase the messages when processed or if the processing fails the next dameon to spin up will replay it.

3\. On reads, just hit the local DynamoDB in any region and reply; we trust our reconciliation threads to do the work to keep us all in sync _eventually_.

NOTE: If you prefer to do read-repairs here you can, but it will increase complexity and inter-region communication which all costs money.

The challenges with this approach is that you pull up a lot of DB concerns into your code like conflict resolution, resync'ing entire regions after failure, bringing new regions online and ensuring they are synchronized, diffs, etc.

There is a reason AWS doesn't offer a globally-distributed data store: it is a really hard problem to get right once you make it past the 80% use case.

Your data will determine if this is an option or not; some data allows for certain amounts of inconsistency in which case this strategy is awesome and works great; while other data (e.g. banking data) cannot allow a single wiggle of inconsistency in which case pulling all this DB logic up into the application is a bad idea. Your failure scenarios become catastrophic (e.g. your conflict-resolution logic is wrong and wipes out the balance from an account; or keeps re-filling the balance on an empty account... something bad basically)

It is all a trade-off though; if you managed your own Cassandra cluster though, Cassandra does all this and much more for you automatically; but then you just put your time into Cassandra administration instead of developing the logic around DynamoDB (or SimpleDB, or MySQL, or whatever); just pick which devil you feel more comfortable with.

I am not aware of a services company yet that offers cross-region AWS datastore deployments yet; Datastax and Iris Couch will setup things for that like you via a consulting/custom arrangement, but there isn't a dashboard for launching something like that automatically.

Hope that helped (and didn't bring you to tears of boredom)

<comment><USER8> The above is why I love to read HN...

However, it seems that at some point here you are better off going back to getting dedicated hosts at a couple different distributed data centers and dealing with the complexity yourself. Surely by the time you "get to 11", you are spending more to be on AWS than doing it yourself would cost?

<comment><USER4> If anything, I suspect that the opposite is the case. If you're planning to set up servers in data centers on opposite sides of the globe, AWS is great. This is where their abstraction layer pays off. All their regions work the same way, accept the same API calls, are backed by effectively the same hardware. Get your setup working in one region, then copy a few S3 buckets over, change the --region argument to some calls in your script, and suddenly you're up in Singapore.

Plus, Amazon is building tools, like the above-mentioned Route 53, to support just this use case.

Now, those of us who have worked a lot within one AWS region will complain and moan about the horror of trying to work cross-region, but that's not because AWS makes cross-region unusually hard. It's because they make your work within one region unusually easy. We're all spoiled by, e.g., the magic of cloning EBS volumes from zone to zone via snapshots. (Well, okay, it's magical when it's working, anyway. ;)

</comment></comment></comment><comment><USER9> Decide which part of the CAP Theorem <http://en.wikipedia.org/wiki/CAP_theorem> you want to give up on. Presumably you decided that Availability was not it, so you need to program around lack of consistency and/or partition tolerance. Essentially that means there is no "master database", and you will need to reconcile differing views. This can get quite application specific, and you need to understand your data well.

</comment><comment><USER10> FWIW I initially went into ELB assuming it would solve a lot of my redundancy problems. And while it has helped a lot (I spread my frontend across 3 zones), I've suffered through a number of ELB failures or disruptions, including this latest one, which is one of the worst. Even with fully functioning servers that I can connect to individually, ELB was intermittently rejecting connections and failed to reregister instances. There's no silver bullet! Just prepare for failure and attempt to handle it gracefully, learning from each one. I suppose you should also think hard before you launch into a greater AWS budget to increase availability. Most of us are tempted to do that after each major incident--which is why Amazon can walk away from these events in a better position than before (until they have a genuine competitor that is).

</comment><comment><USER11> We run a few decent sized social games and we have survived all the major AWS region outages in the past year. He's what we do and what I would suggest.

1\. Use Rightscale. You can get away with the free edition, but for $500/month the basic paid edition will allow you access to arrays and all the excellent scripts available on the marketplace.

2\. The front end. I would strongly suggest moving away from ELB. We are using it and are about to get rid of it. The main problem is what exactly happened last night. If a whole AZ goes down, the ELB for that zone can get screwed and the DNS was not updating the CNAME to remove the bad zone. Instead of ELB, we have our own LB solution we are going to roll out that will use Rightscale server arrays and will handle the updating of the DNS names itself. We also aren't going to use Route53, because we learned last night that the API for that can go down and you can get stuck with bad DNS records.

3\. Application servers. Use at least 3 AZ and have them evenly spaced. This is easy to do in Rightscale with sever arrays. Make sure your voting ration for scaling isn't 50% because you might not scale correctly if you loose 2 AZ. Keep the vote to 30% and you will be happy (if one zone votes to grow, let it grow).

4\. Database. This is the fun one. We have been using MongoDB with pretty good success. Our multi-shard DB has 3 servers per replica set and has them distributed equally between AZs. We use 4 EBS drive RAID-0 drives for storage which have had problems in the past due to the outages that EBS sometimes has. Our best bet has been a watcher process that will kill the mongod process if there's any problems writing to the drive array. By doing this, the replica set will automatically failover to the next server and we won't get stuck with a primary node that can't write back to disk. For backups, we just freeze the writes on the secondaries and do EBS snapshots even 15 minutes. Rightscale has some great EBS tools for managing this for you. If you loose a server, we can deploy a new server in a matter of minutes and it will rebuild the RAID array from the last backup so we have a warm spare.

5\. Monitor, monitor, monitor. Rightscale has some great tools for monitoring everything. Use them, and use more monitoring on other infrastructure (ie Pingdom)

Doing something like this will cost a lot more that just sticking to a single AZ, but you should be able to survive one, if not two complete datacenter outages.

<comment><USER12> > If you loose a server, we can deploy a new server in a matter of minutes and > it will rebuild the RAID array from the last backup so we have a warm spare.

Unless you can't actually get a new server provisioned because the now-fragile API (like last night) is under such load from people trying to mitigate their downtime... We use rightscale, and rightscale won't solve this issue. If AWS is being clobbered by people trying to get new boxes up, using a third-party api abstraction service doesn't help.

Also, rightscale themselves were affected last night, throwing invalid alerts about servers being inaccessible when they were actually still operating normally.

<comment><USER11> That's also the beauty of Rightscale. If you do everything correctly you should be able to provision a new server in another region if need be or even another cloud (ie Rackspace).

Most outages of AWS don't last more than a few hours. The real goal is to make sure your infrastructure can hobble on one leg for those few hours until help arrives and you can cleanup the mess once the outage is over.

When the API goes down it sure isn't fun. Just try to project yourself the best you can. I had no problems with Rightscale last night other than anything trying to reach EC2.

<comment><USER11> And also,

If shit really hits the fan, its not a bad idea to make sure you have a OpenVPN tunnel ready on each server.

This will allow to get connectivity between old and new instances if you can't update the security groups due to the API being down.

</comment></comment></comment></comment><comment><USER13> Sending traffic to different zones isn't the challenge, the challenge is deciding where your master data will live. In fact, this has always been one of the biggest challenges of building a fault-tolerant systems. If your master data store lives in one zone then you've got latency issues, but if it lives in multiple zones then you need to find a logical way to shard. You could also replicate across zones and then turn off writes if the zone with the master fails. You could even change masters in that case, but there's risk of data loss there.

Anyhow, sorry I don't have a simple answer - I'm not sure a simple answer exists.

<comment><USER14> multi-az in RDS does the replication and automatic failover for you, no sharding necessary.

<comment> mh- except when it doesn't -- [https://forums.aws.amazon.com/thread.jspa?threadID=98376&...](https://forums.aws.amazon.com/thread.jspa?threadID=98376&tstart=0)

</comment></comment></comment><comment><USER15> I am on PHPFog for my front-end and with an AWS RDS back-end. I managed to survive this incident without an outage (I am on U.S East as well), although I did get some horrendous response times from RDS for about an hour there.

PHPFog are on AWS and I pay them to make sure they have the redundancy worked out. If they don't, I would yell at them until I got some money back.

I am considering configuring RDS for Multi A-Z, but need to research it a little more first. From what I can tell you just click a button to turn it on, but there were a lot of people complaining yesterday that the fail-over didn't work at all when it was supposed to.

I also have a bunch of EC2 VMs that do back-end processing and have a load of CRON jobs on them that need to run once every 24 hours. If these go down for a couple of hours then there is no noticeable impact to my customers, they can still log into my service and access their historical data.

I have considered spreading across multiple regions etc but at the end of the day it's just too expensive for the small increase in reliability.

</comment><comment><USER16> Assuming we want to minimize latency and maximize reliability, we want to create a stack that:

1) Has AutoScaling Groups & Elastic Load Balancers in two regions (and only two availability zones; let's keep front-end instances in the same AZ as your local/region-specific DB)

2) Has Databases in two regions and uses Master Master replication

3) Instances talk to their local DB. If they detect their local DB is down, they failover to the remote DB (ie, the far region). If they failover, they notify you.

4) DNS does geographic load balancing (pre-ELB). You'll need to use a provider like DynDNS or UltraDNS to give you Geo Load Balancing & Failover. Or, you could pair a monitoring service like CatchPoint with Route53

5) Application caching (Memcache, Redis, etc). Let's not put more load on the DB's than necessary.

That's a good start, at least.

</comment><comment><USER17> I haven't tried this (I use single EC2 deployments, some Heroku, also have a Hetzner server) but it is something that I have been thinking of: have the web services that back up your web app on a single server, and yes that will fail on hopefully rare occasions. Host the Javascript+HTML5+CSS front end on S3 with Cloudfront CDN. The home page of your app will almost never go offline and you control what to report to your users if your backend services are offline. Sure you lose core functionality, but you still have static content and a friendly message about temporary lack of services.

Going beyond that at a cost of slow response times when trying to access a downed backend, you could deploy back end web services to two different hosting providers, perhaps running something like CouchDB replicated on each provider. The Javascript on your UI could switch to an alternative back end after a timeout. For "one page" style apps, you could maintain the state information that a backend host is down in the browser.

</comment><comment><USER18> Start here: <http://aws.amazon.com/architecture/>

I don't think they show how to do ELB across Regions, or diversity against single-ELB problems (although I haven't seen ELB fail yet). You'd probably have to build this yourself.

<comment><USER19> Today's status update (<http://status.aws.amazon.com/>) indicates partial ELB failure, still unresolved:

 _Jun 30, 12:15 AM PDT [..] Elastic Load Balancers were also impacted by this event. ELBs are still experiencing delays in provisioning load balancers and in making updates to DNS records._

 _Jun 30, 12:37 AM PDT ELB is currently experiencing delayed provisioning and propagation of changes made in API requests. As a result, when you make a call to the ELB API to register instances, the registration request may take some time to process. As a result, when you use the DescribeInstanceHealth call for your ELB, the state may be inaccurately reflected at that time. To ensure your load balancer is routing traffic properly, it is best to get the IP addresses of the ELB's DNS name (via dig, etc.) then try your request on each IP address. We are working as fast as possible to get provisioning and the API latencies back to normal range._

</comment></comment><comment><USER20> From what I've heard, you're on the right track. However, I'd want it to not round-robin but go to the nearest working node. I don't use AWS, so I don't know how to configure the ELB, but I would assume that this is possible.

</comment><comment><USER21> Super low TTL and Refresh combined with replication to a DR provider. High Availability placed exclusively on a single provider - even Amazon (albeit different AZ's) is of zero value if all of Amazon itself is offline...

</comment><comment><USER22> I'm shocked no one has mentioned <http://www.cedexis.com/> yet.

</comment><comment><USER23> Try a multi-infrastructure PaaS like <http://appfog.com/></comment>


Privacy Incident Involving DHS OIG Case Management System - <AUTHOR> https://www.dhs.gov/news/2018/01/03/privacy-incident-involving-dhs-oig-case-management-system <comment><USER2> Not just employees, but also "subjects, witnesses, and complainants". Yikes. This is way worse than the last one.

<comment><USER3> > _the last one_

I've lost track now.. which one do you mean?

The OPM breach of the SF-86 database is still the worst and most catastrophic that I'm aware of. That was larger (millions), was the result of background checks (aka blackmail fodder), included full read/write access (tainted?), and likely every position that required a clearance outside the CIA.

<comment><USER2> That's the one I meant. You're right, that one does seem worse. I didn't realize that the previous one had information on so many people.

This one seems especially terrible because it had information on people unrelated to the US government.

<comment><USER3> During a background check, they start by investigating the subject and interview friends, family, employers, colleagues, classmates, significant others, roommates, etc.. and may investigate them to some degree. They only have detailed info on the subject but would still have lots of info on the others.

*I had a Secret 10+ years ago.

</comment></comment></comment></comment><comment><USER4> Does anyone have experience with the credit protection firms always hired to attempt to make right in these cases? It seems like the onus is still on the individual to fix their credit if something bad does happen, so they could conceivably effectively be doing nothing.

> "...unauthorized unauthorized transfer of data."

Am I being dense or is this unintended?

<comment><USER5> Probably a typo, but in the CIA world, there is a lot of 'authorized unauthorized' "off the books" activity.

<comment><USER6> >Probably a typo, but in large and slow moving bureaucracies, there is a lot of 'authorized unauthorized' "off the books" activity.

FTFY.

It's actually a pretty good way to explain speculative execution. You know you might need to so something so you just do it using the slush fund and if it isn't useful you toss the result and if it is useful you keep doing it while papers are shuffled around and authorization happens. Certainly not ideal but not unheard of either.

</comment></comment></comment><comment><USER7>
      Investigative Data: Individuals associated with DHS OIG
      investigations from 2002 through 2014, which includes
      subjects, witnesses and complainants who were both DHS
      employees and non-DHS employees.  The PII contained in this
      database varies for each individual depending on the
      documentation and evidence collected for a given case.
      Information contained in this database could include names,
      Social Security numbers, alien registration numbers, dates
      of birth, email addresses, phone numbers, addresses, and
      personal information provided in interviews with DHS OIG
      investigative agents.

Most of that is business-as-usual for a DB leak, but there's something that catches my eye:

      personal information provided in interviews with DHS OIG
      investigative agents.

Would this only be information provided by people about themselves, or does the inclusion of "subjects of investigations" mean that this is a colossal dump of life-ruining hearsay that could destroy anyone that DHS has ever touched?

Speaking of which, this bit at the end disgusts me:

      What is DHS doing to better secure employees' PII?

Why does this read as anything other than "What is DHS doing to better secure _american citizens '_ PII"? DHS employees aren't the only people affected by this. I'd expect "subjects, witnesses and complainants of DHS investigations" to number in the millions.

<comment><USER5> Put away the pitchforks.

OIG is internal affairs. The people they investigate are employees, not the public.

[https://www.oig.dhs.gov](https://www.oig.dhs.gov)

<comment><USER8> Employees /are/ the public.

You don't lose the right to protection from someones incompetence just because you're employed by them.

</comment></comment></comment><comment><USER9> And here we go again.

</comment><comment><USER10> Does anyone know if this includes SF-86 and related investigation data?

<comment><USER3> The OIG case data would have nothing to do with SF-86 data which is kept by OPM.

The OIG investigates complaints against DHS (and child agencies) officials, employees, and policies brought by Congress, the public, and other agencies. It's similar to the stereotypical Internal Affairs department in every police movie.

<comment><USER10> OPM keeps that data even for contractors?

</comment></comment></comment><comment><USER11> Can someone from USDS chime in? I thought these occurrences were mitigated against by having tech professionals providing guidance to internal development teams.

<comment><USER12> AFAIK, each agency has to request USDS/18F assistance.

<comment><USER13> And it's unrealistic to expect that assistance to prevent breaches, especially when there are hundreds of government agencies separately running insecure legacy systems. The USDS/18F has on the order of < 100 employees AFAIK. That is hardly enough manpower to secure the systems of every government agency.

</comment></comment></comment><comment><USER14> Homeland "Security"

<comment><USER6> Sounds like a great name for a cinema in the DC area.</comment></comment>


For Stanford Class of '94, a Gender Gap More Powerful Than the Internet - <AUTHOR> http://www.nytimes.com/interactive/2014/12/23/us/100000003252545.app.html <comment><USER2> Discussion from yesterday: [https://news.ycombinator.com/item?id=8784926](https://news.ycombinator.com/item?id=8784926).

</comment><comment><USER3> Very good article. I must admit, the more I read about Thiel, the less I like him.

It's unclear to me why they think that SV is more meritocratic than anything else. Sports still seems much more meritocratic than SV. Any industry that depends so much on getting funding hardly seems like it would be especially meritocratic -- at least at first blush.

<comment><USER4> The more I read about Thiel, the more I wonder how someone so smart can be so dumb.

<comment><USER5> Most people don't question views that justify their wealth.

<comment><USER6> Which wouldn't be so bad, but Thiel, Andressen, et al, seem determined to foist their flavour of "smart stupid" on everyone else.

<comment><USER5> Admittedly, I can kinda-sorta understand becoming a libertarian if you grew up in Northern California: its local/municipal/regional governments are a _special_ kind of stupid that you only get when combining lower-class reaction with landowner chauvinism with new money elitism.

</comment></comment></comment></comment></comment><comment><USER7> That title is a bit misleading.

I read this yesterday; it's pretty jumbled. It's an article that jumps around a bit describing how a couple of the people from the Stanford Class of '94, specifically David Sacks and Jessica DiLullo Herrin, made a lot of money using the internet.

It doesn't really get into the reasons there are fewer female than male entrepreneurs from that class, at least not with any depth, and then it concludes by saying that people from that class are getting into internet startups now.

<comment><USER8> I agree, it wanders all over the place like they had to meet a quota of words. I don't think the article really communicates what it was like for the "computer nerds" off in a weird, unproven, tiny little corner of the world nobody cared about or believed in. I think if you explore that, you'll start to understand why women weren't as much a part of that weird subculture. It wasn't all bad, I worked closely with women at IBM in 1994 (working on OS/2) that was also the year WIRED started to go mainstream. TV ads started to show URLs. Java went mainstream in 1994. But before 1994, almost nobody knew what the Internet was. Once the Internet went mainstream, people already familiar with the technology had a huge advantage: nerds.

In 1994 when I was at the University of Florida, only one dorm on the entire campus had "Internet" access. But it wasn't web access, we only had telnet. The moldy dorm was 100+ years old and barely had air conditioning. They called it "Fletcher Island" because we were cut off from everyone else--there was maybe 6 of us guys on the whole campus that had Internet. And there was only one guy that knew how it worked--he never spoke, was very pale, and he lived like a hermit back in the dark corner of this ancient building. The air quality was so bad that I got sick and had to move out. Maybe 2-3 women lived under us but we never saw them because the hallways were locked to keep the sexes apart.

</comment></comment><comment><USER9> How exactly are they measuring this "gender gap"? At one point it seemed their argument was that female graduates didn't found as many companies as male graduates. (Though it seems like there could be a lot of different explanations for that.) The quote about how women "played a support role instead of walking away with billion-dollar businesses" comes from the founder of a web site for female writers, which made me wonder if it was more advocacy than analysis.

I am concerned about the role of women in Silicon Valley, so it was nice to see that the Times' reporter notes that few women "described experiencing the kinds of workplace abuses that have regularly cropped up among women in Silicon Valley." It would've been nice if this article had more statistics about what specifically happened to the class of '94

</comment><comment><USER10> Why do articles like this always divide it by gender and never combine gender and race? There are more Asian women featured in this article and probably included in the class than black (and potentially hispanic) men, despite the fact that the population of black men is and was in 1994 larger than Asian women.

<comment><USER11> I've always wondered why the tech industry is so obsessed with the gender gap and yet almost completely ignores the race gap.

Is the gender gap really a bigger problem, or is it because men in tech think they are more likely to socially benefit from championing gender equality?

<comment><USER12> The racial gap is present in almost all faucets of American life. That suggests a society wide issue.

The gender gap is fairly unique to the tech industry and it is pretty extreme. My graduating class at Illinois for EE was 95% male.

<comment><USER11> >That suggests a society wide issue.

The gender gap is every bit a society wide issue. When I was in middle school and high school, I didn't know a single girl who was interested in technology. This says that this issue begins when girls are young and is either something imprinted on them by society or something that is innate.

I haven't seen anything that suggests that our industry is more able to solve the gender gap than the race gap. Both of these problems have complex causes that extend far outside of the scope of what one industry can fix, but I don't see any reason to fixate on one at the expense of the other, which is exactly what we're doing.

>The gender gap is fairly unique to the tech industry...

This isn't unique to the tech industry at all. The gender gap is present in numerous blue collar professions--some of which pay very well (electricians, plumbers etc...)

>...and it is pretty extreme. My graduating class at Illinois for EE was 95% male.

And I'm sure you can find many equally extreme examples that demonstrate the race gap.

There are other gaps as well. Rural vs urban, rich vs poor, single parent vs 2 parent households. Yet none of these receive anywhere the press that the tech gender gap does.

<comment><USER13> Certain aspects of male/female difference are definitely innate. For example, among young infants, females are more interested in faces and people, while males are more interested in mechanical objects. I believe a study showed this effect was present & statistically significant only a short time after birth (too soon to be influenced by culture).

There are well-known differences in brain structure, such as the portions of brain used for language. Women have an edge in verbal reasoning, while men in spatial.

Male IQ has more variability than female IQ, resulting in more men in both the top and bottom percentiles. This may explain why more men are executives, and why more men are homeless (that's just my suspicion). Studies have also found that male IQ is higher on average by (variously) 3-5 points. From my personal observations, men are more likely than women to focus intensively on one activity, to the exclusion of all others, like skateboarding, or surfing, or computer programming, or gaming.

Men weigh more than women, have more muscle mass, have 40-60% more upper body strength, and 25-30% lower body strength, though with training the gap narrows to 0-8% (so perhaps this difference is more cultural). Men outperform women in most athletic areas, though perhaps not in distance running. Women live years longer than men and have lower rates of mental illness.

Women and men earn equal pay when you control for the job worked, the hours worked, and the amount of experience. I believe a study showed that single male and female professionals with no dependents, equal time in the workforce, working equal hours, earned equal amounts. The pay gap arises because slightly fewer women work high-paying jobs, they work fewer hours per week / take more time off, and drop out of the workforce at higher rates (e.g. stay at home parent).

A good summary of research and sources are available in Wikipedia articles like "Sex differences in intelligence" and "Sex differences in psychology". Certainly do not take my word for any of this, and please correct me if I'm wrong.

Personally, I think we should make sure that every person is judged based on individual merit, and be encouraged and supported to pursue what they're interested in with no preconceptions. That's what equality means to me.

</comment></comment></comment></comment></comment><comment><USER14> I always find it quite indicative of American attitudes that this article took "number of companies founded" as a metric of success, and ham-fistedly conclude that the men are "ruling".

Why not take "happiness with current quality of life" as one? Incorporate that into your survey, and see how many men and women report back as being satisfied and happy with their lives.

</comment><comment><USER15> It couldn't possibly have anything to do with more men than women being into computing? They cite examples of women who were in computing dropping out. But surely lots of men dropped out, too. But the women made more news. And since there were more men, more remained to go on and get rich on the internet. It's just bad, bad science. Please apply some proper statistics instead of stirring up emotions with anecdotes.

<comment><USER16> So take that thought one level deeper: why are proportionally fewer women entering computing and proportionally more of them dropping out?

<comment><USER15> I knew this would come up. You probably think because computers weren't marketed to women? What if electronics companies only marketed to men because it was mostly men who were into electronics? Then rinse, repeat, why were fewer women interested in electronics, and so on...

I don't have all the answers, but one answer is: because the women were doing other things. There seems to be this idea that women who don't go into engineering become "desperate housewives". In reality they become physicians or bankers or lawyers. The article actually mentions that.

The reality is that computing is a pretty anti-social activity. Yes, there are meetings and clients, but your main job is being glued to a computer screen, alone. Frankly, I don't think it is a recipe for happiness, and women are smart enough to recognize it. Also, women have more options - a major one becoming mothers. So less incentive to try hard, maybe? Even the article mentions that the famous female entrepreneur eventually dropped out to become a mother. Although of course in the next sentence they have to reframe it as "went to Austin for her husband's job" instead of "to become a mother". Can't have that.

I recently met a young woman who was torn between studying mathematics or becoming a social worker. I did not try to stir her towards maths, even though I myself have a maths degree. What argument could I have used, except the prospect of higher income? Frankly I think social workers are probably happier than mathematicians on average.

<comment><USER17> You completely missed the "proportionally more dropping out," therefore expending three paragraphs saying nothing.

<comment><USER15> I didn't miss it. If they find it is not as much fun as they thought and they have other options, they drop out.

I'm pretty sure, for example, that women tend to be less concerned about the potential income of their profession.

I wouldn't be surprised if women are on average less antisocial than men, for the simple reason that women are desired by the world. Men wonder how to talk to women, women wonder how to avoid being talked to (great simplifications of course). So computing would on average be less attractive (in the days before facebook anyway).

</comment></comment></comment></comment></comment><comment><USER18> Heard her talking about this piece on NPR. Unimpressive.

</comment><comment><USER19> Too much hype. What this article does not mention is that all of these people are now in the sunset of their careers. Even the ones that achieved a lot are already declining from their peaks. I'm much more interested in what the class of 2015 is going to do.

<comment><USER5> Age 42 is the _sunset_ of their careers? Really?

</comment></comment><comment><USER20> Front page of NYT? Must be a slow news day... Interesting collection of stories, though.</comment>


EU Plots against the Pirate Bay - <AUTHOR> http://torrentfreak.com/eu-plots-pirate-bay-ban-and-piracy-clampdown-090201/

<comment><USER2> "It's proposed that ISPs should disconnect subscribers who share copyrighted content, based on information provided by the entertainment industry."

Because the entertainment industry is completely fair and balanced.

</comment><comment><USER3> Breaking the back button should be a capital offense.</comment>


Using Parse for IoT to Create an Order Button - <AUTHOR> http://blog.parse.com/2015/04/01/using-parse-for-iot-to-create-an-order-button/

<comment><USER2> Spark has some interesting stuff in this area.

$19 for the upcoming Photon board, and $39 for the cellular-connected Electron.

[https://store.spark.io/](https://store.spark.io/)

Arduino seems like overkill here.

</comment><comment><USER3> The interesting part of the Dash isn't the backend (something that can handle API calls), but the hardware, and how cheap they can make it.

<comment><USER4> As I mentioned in the other thread, the ESP8266 is a Wifi chip that you can buy for $5/each on an order of 1, and can run custom code, so you'd only need one of those plus a case, battery and button. You could probably build a button for less than $10, without even ordering large quantities.

<comment><USER5> Don't forget your power regulation.

</comment></comment></comment><comment><USER6> Does anyone knows the tech behind cloud code? how do they execute javascript safely on the server ?

</comment><comment><USER7> The title led me to believe that Amazon had teamed up with Parse for this which I would have found surprising.

This has nothing to do with Amazon Dash other than using a button trigger an HTTP request.

<comment><USER8> I think dash announcement is why facebook is releasing this blog post now.

<comment><USER7> The title has changed and it is now clearer, the title was originally "Amazon Dash built on Parse".</comment></comment></comment>


TechCrunch is hiring - <AUTHOR> http://jobs.37signals.com/jobs/6659

<comment><USER2> I'd like to point out that Mark Hendrickson, CEO of Plancast, was previously a TC hacker. If you have a half-decent idea, expect the TC brand to help you get networked. Oh, and you're guaranteed to get multiple hits of positive coverage if you leave on good terms.

I'm sure jkincaid will pipe in here at any moment to sell you all on this amazing opportunity.

<comment><USER3> as a side note is plancast down.. i just tried going there, and nothing happened.

<comment><USER4> We were just down for a few minutes...back up and chugging along now. :)

Sorry for the inconvenience.

</comment></comment></comment><comment><USER5> "Relentlessly Resourceful hacker" -- Reminds me of something PG would say :)

<comment><USER6> <http://www.paulgraham.com/relres.html>

</comment></comment><comment><USER7> _We're looking for a Relentlessly Resourceful hacker_

Why would someone fitting that description want to work at TC?

<comment><USER8> Why wouldn't they?

<comment><USER7> Because relentlessly resourceful hackers do their own thing.

</comment></comment></comment><comment><USER9> Am I the only one that thought this was some parody post by Jason or DHH (when you see the 37signals.com in brackets), before realizing it was just on the jobs page?

</comment><comment><USER10> I was really hoping this was a joke.

Incidentally - <http://friedisms.tumblr.com/>

"We don't listen to music in the office. We listen to US Army audio transcripts from the front lines of the Vietnam War. The power of these bits of history really help us focus on fixing bugs in Basecamp."

</comment><comment><USER11> The upside is that working for TechCrunch can get you networked into the right group of self-righteous windbags in case you want one of your ideas to get hyped up one day.

The downside is that Arrington has a history of making emotional business decisions, attacking people in a public forum, and throwing his own men to the wolves to boost his self image.</comment>


Mathematicians Illuminate Deep Connection Between Classical And Quantum Physics - <AUTHOR> http://www.sciencedaily.com/releases/2008/10/081010081650.htm <comment><USER2> <http://www.aimath.org/news/que/>

Link to more pictures and links to their papers

</comment><comment><USER3> Woah.

Well, I did a physics degree a few years ago, and all this stuff flew about a mile over my head.

<comment><USER4> You're not alone.</comment></comment>


WePay Grows Revenue By Ten-Fold In 2011; Will Launch Mobile Apps This Year - <AUTHOR> http://techcrunch.com/2012/02/13/online-payments-startup-wepay-grows-revenue-by-ten-fold-in-2011-will-launch-mobile-apps-this-year/ <comment><USER2> Maybe it was because he went on Millionaire Matchmaker.

Just kidding. Good job Bill!</comment>


Sociologists invade World of Warcraft, see humanity's future - <AUTHOR> http://arstechnica.com/tech-policy/news/2010/05/sociologists-invade-world-of-warcraft-and-see-humanitys-future.ars

<comment><USER2> Jane McGonigal (Alternative Reality Game Expert) says that it took 100 million mental hours to create and produce Wikipedia and that that effort is expended in only 5 days of playing World of Warcraft. She sees gaming of that sort as a resource to be exploited.</comment>