Stop using low DNS TTLs - <AUTHOR> https://00f.net/2019/11/03/stop-using-low-dns-ttls/ <comment><USER3> I operate authoritative name servers for almost 10.000 domains. Originally, I used a default TTL of 2 days, as recommended by RIPE-203¹ (which is also compatible with the recommendations of RFC 1912²), but this was not accepted by users, who didn't want to wait two days. Therefore, for all records except SOA and NS records, I changed the default TTL to one hour, which I still use as the default value unless a change is scheduled and/or planned, in which case I lower it to 5 minutes. I do not want to lower it any more, as I've heard rumors of buggy resolvers interpreting "too low" TTLs as bad, and reverting to some very-high default TTL, and thereby wrecking my carefully planned DNS changeover. I have, however, not seen any real numbers or good references on what numbers are "too low", and would like to hear from anyone who might have some information on this.

1\. [https://www.ripe.net/publications/docs/ripe-203](https://www.ripe.net/publications/docs/ripe-203)

2\. [https://tools.ietf.org/html/rfc1912#page-4](https://tools.ietf.org/html/rfc1912#page-4)

<comment><USER4> Would you mind sharing the service?

<comment><USER3> The service? What service? What do you mean?

<comment><USER4> You said you run authoritative servers- I assume you provide DNS hosting. I'm curious which provider you run.

<comment><USER3> I am hesitant to say; we only target the local area, and our home page isn't even available in English. Our main role is as a domain name _registrar_ , also providing, in increasingly tangential order, domain name strategy planning, some trade mark strategy, DNS hosting, HTTP redirects, E-mail, and web hosting. Our main value proposition is _support_ ; call us and talk to us directly, or send an e-mail, and get an answer more or less immediately. We only very reluctantly provide self-service control panels, and we don't mention its availability unless people directly ask for it, and we generally discourage its use, preferring that people simply tell us what they want done in their DNS. Some people, including some very large companies, prefer this arrangement, and if you are one of them, and you are part of our local market, I'm sure you'll be able to find us.

<comment><USER5> Probably way better to not say than to say. Little upside versus who knows what downside.

</comment></comment></comment></comment></comment></comment><comment><USER6> The irony of all of this is that those TTLs are almost meaningless as a server operator anyway. Even if you set your TTL to 5 minutes, there are a whole lot of clients that will ignore it.

When I made a DNS switch at reddit, even with a 5 minute TTL, it still took an hour for 80% of the traffic to shift. After a week, only 95% had shifted. After two weeks we still had 1% of traffic going to the old IP.

And after a month there was still some traffic at the old endpoint. At some point I just shut off the old endpoint with active traffic (mostly scrapers with hard coded IPs at that point as far as I could tell).

One of my friends who ran an ISP in Alaska told me that they would ignore all TTLs and set them all to 7 days because they didn't have enough bandwidth to make all the DNS queries to the lower 48.

So yeah, set your TTL to 40 hours. It won't matter anyway. In an emergency, you'll need something other than DNS to rapidly shift your traffic (like a routed IP where you can change the router configs).

<comment><USER7> (Weeping in agreement.)

The lesson I took away from hosting operations is that the implementation of internet standards has a long tail of customization; it's part of the job to accommodate them graciously. :)

<comment><USER8> Do you mean gracefully?

<comment><USER9> No reason one can't be both graceful and gracious.

</comment></comment></comment></comment><comment><USER10> > Why are DNS records set with such low TTLs?

The author seem to be missing one of the big reasons ridiculously low TTLs are used: it lets passive eavesdroppers discover a good approximation of your browsing history. Passive logging of HTTP has (fortunately) been hindered as most traffic moved to HTTPS, but DNS is still plaintext.

Low TTLs mean a new DNS request happens (apx) every time someone clicks a link. Seeing which domain names someone is interacting with every 60s (or less!) is enough to build a very detailed pattern-of-life[1]. Remember, it's probably not just one domain name per click; the set of domain names that are requested to fetch the js/css/images/etc for each page can easily fingerprint specific activities within a domain.

Yes, TTLs need to have some kind of saner minimum. Even more important is moving to an encrypted protocol. Unfortunately DOH doesn't solve this problem[2]; it just moves the passive eavesdropping problem to a different upstream server (e.g. Cloudflare). The real solution is an encrypted protocol that allows everyone to do the recursive resolution locally[3].

[1] [https://en.wikipedia.org/wiki/Pattern-of- life_analysis](https://en.wikipedia.org/wiki/Pattern-of-life_analysis)

[2] [https://news.ycombinator.com/item?id=21110296](https://news.ycombinator.com/item?id=21110296)

[3] [https://news.ycombinator.com/item?id=21348328](https://news.ycombinator.com/item?id=21348328)

<comment><USER11> > The author seem to be missing one of the big reasons ridiculously low TTLs > are used: it lets passive eavesdroppers discover a good approximation of > your browsing history.

I operate DNS for hundreds of thousands of domains. I've tried to reassemble browsing history from DNS logs, and I can tell you it is damn near impossible. You have DNS caches in the browser, the OS, broadband routers, and ISPs/public resolvers to account for - and half of them don't respect TTLs anyways.

The reason people set low TTLs is they don't want to wait around for things to expire when they want to make a change. DNS operators encourage low TTLs because it appears broken to the user when they make a change and "it doesn't work" for anywhere from a few hours to a few days.

<comment><USER12> The problem is that your ISP can log and mine your DNS requests, regardless of the servers you use. They definitely do this and one can only assume they then sell it after some sort of processing.

<comment><USER13> The comment you're replying specifies caching at the browser, OS, and router level. Not one of three would show up as DNS refreshes with the ISP because the DNS is not being refreshed.

<comment><USER14> Don't browsers and operating systems mostly respect ttls?

So if some things are cached, you won't get a complete picture, but the picture you get might be enough.

<comment><USER15> I can't tell. I run Firefox at home, and set up my own DoH server (mainly because I saw the writing on the wall and and if Mozilla/Google are going to shove this down my throat, I want it shoved down on my terms, but I digress). If I visit my blog (which has a DNS TTL of 86,400) I get a query for my domain not only on every request, but even if I just hover over the link. It will also do a query when I click on a link to news.ycombinator.com (with a TTL of 300) but not when I hover over a link. It's bizarre.

</comment></comment></comment></comment></comment></comment><comment><USER16> I seem to remember a paper a few years ago that (IIRC) tested this by setting a very low TTL (like 60), changing the value, and seeing how long they continued to receive requests at the old value... and most updated within the TTL, but there were some that took up to (I want to say) an hour. I'm probably getting bits of this wrong though..

I did find this paper: [https://labs.ripe.net/Members/giovane_moura/dns-ttl- violatio...](https://labs.ripe.net/Members/giovane_moura/dns-ttl-violations- in-the-wild-with-ripe-atlas-2)

The violations in that paper that are important are those that have increased the TTL. Reducing the TTL increases costs for the DNS provider, but isn't important here. The slowest update was about 2 hours (with the TTL set to 333).

Of those that violated the TTL, we don't know what portion of those would function correctly with a different TTL (increasing the TTL indicates they're already not following spec). So I wouldn't assume that increasing the TTL would get them to abide by your requested TTL. They're following their own rules, and those could by anything.

Considering how common low TTLs are... you're worrying about a DNS server that's already potentially causing errors for major well known websites.

<comment><USER17> It is important to note that this study used active probes asking selected recursive resolvers around the world.

From my own experience when changing records and seeing when the long tail of clients stops calling the old addresses (with the name), it is a really long tail. An extreme example that lasted almost six months was a web spider that just refused to update their DNS records and continued to request websites using the old addresses.

Is there a lot of custom written code that does their own DNS caching? Yes. One other example is internal DNS servers that shadow external DNS. There is a lot of very old DNS software running year after year. Occasionally at work we stumble onto servers which are very clearly handwritten by someone a few decades ago by people with only a vague idea of what the RFCs actually say. Those are not public resolvers of major ISPs, so the above study would not catch them.

Naturally if you have a public resolver where people are constantly accessing common sites with low TTLs then issues would crop up quickly and the support cost would get them to fix the resolver. If it's an internal resolver inside a company where non-work sites are blocked then you might not notice until the company moves to a new web hosting solution and suddenly all employees can't access the new site, an hour later they call the public DNS hosting provider, the provider diagnoses the issue to be internal of the customer's network, and then finally several hours later the faulty resolver gets fixed.

<comment><USER18> > An extreme example that lasted almost six months was a web spider that just > refused to update their DNS records and continued to request websites using > the old addresses.

It may have been Java client that was not restarted. At least for older versions of Java the default was to cache result forever.

<comment><USER19> Yep, older java versions had some ridiculous caching of both positive _and negative_ DNS responses. That was some weird problem to troubleshoot. We ended up writing our own caching then, back in Java7ish. And the first version of our DNS caching was broken and promptly triggered load alerts on 2 DNS servers of our operations team by issuing ... a lot of DNS queries very very quickly :)

</comment></comment></comment></comment><comment><USER20> > Of course, a service can switch to a new cloud provider, a new server, a new > network, requiring clients to use up-to-date DNS records. And having > reasonably low TTLs helps make the transition friction-free. However, no one > moving to a new infrastructure is going to expect clients to use the new DNS > records within 1 minute, 5 minutes or 15 minutes. Setting a minimum TTL of > 40 minutes instead of 5 minutes is not going to prevent users from accessing > the service.

Note that you can still get the benefit of a low TTL during a planned switch to a new cloud provider, server, or network even if you run with a high TTL normally. You just have to lower it as you approach the switch.

For example, let's say you normally run with a TTL of 24 hours. 25 hours before you are going to throw the switch on the provider change, change the TTL to 1 hour. 61 minutes before the switch, change TTL to 1 minute.

<comment><USER21> Wouldn't you be canarying your switch over a period of longer than 24 hours anyway?

I can still imagine a benefit to short TTLs in the sense that you can maybe roll out your canary in a more controlled way. But that's a lot more complicated than the issue of quick switching.

<comment><USER22> If it's planned, yes.

If your cloud provider does an oopsie (e.g. [https://news.ycombinator.com/item?id=20064169](https://news.ycombinator.com/item?id=20064169)) and takes down your entire infrastructure, or you have to move quickly for some other reason, or you're recovering from a misconfiguration, the long TTL can add 24 hours to your mitigation time.

If you're just playing around with your personal project/web site, you just added a giant round of whack-a-cache to your "let's finally clean up my personal server mess" evening.

<comment><USER23> As most who's ever worked with web hosting can confirm, small business customers often have no idea of what they're doing, and I've talked to _many_ people who switched providers after seeing an ad for cheap hosting, without realising that they have to a) wait for the DNS changes to propagate, b) that they have to actually move their web site from one provider to another.

Subsequently, my previous employer lowered the default TTL simply because it got rid of all the bad Trustpilot ratings about customers being "prevented from leaving", and started offering a "move my WordPress site for me" service to profit from all the panicking new-comers who had no idea about how to do trivial things like importing/exporting a database and transferring files.

</comment></comment></comment></comment><comment><USER24> It would have been interested to see actual delay rather than qualitative results of the nature "<x>% wasn't in cache so this is horrible!". Admins and users don't care if it's in cache, they care what the impact to operations and load time is. [https://www.dnsperf.com/dns-speed- benchmark](https://www.dnsperf.com/dns-speed-benchmark) says lookup times for my personal domain results in 20ms-40ms. Ironically the same dns test for 00f.net is taking 100ms-150ms.

99% of apps will gladly trade a 30ms increase in session start (assuming the browser's prefetcher hasn't already beaten them to it) to not have to worry about things taking an hour to change. Not all efficiency is about how technically slick something is.

<comment><USER17> I just tested 00f.net and got as low numbers as 6ms. Latency is a question about network traffic between the client and the server, and unless you use anycast you will get different latency depending on what place in the world the client and server reside in, and if you use anycast it depend on how good the contracts and spread the anycast network has.

<comment><USER24> Very true, looks to be hosted out of Europe. The point about 00f.net optimizing for ease of operation vs optimizing milliseconds of performance only holds doubly true with this information though.

</comment></comment></comment><comment><USER25> Hug of death by the looks of it. Maybe they need to quickly change their DNS entries to point to a better server :-)

Edit: [https://outline.com/CrL5gf](https://outline.com/CrL5gf)

<comment><USER3> Doesn't work for me. This does:

[https://web.archive.org/web/20191103211525/https://00f.net/2...](https://web.archive.org/web/20191103211525/https://00f.net/2019/11/03/stop- using-low-dns-ttls/)

</comment></comment><comment><USER26> > I'm not including "for failover" in that list. In today's architectures, DNS > is not used for failover any more.

I mean, my company does this for certain failure scenarios involving our CDNs. Can anyone tell me why we're idiots, or is this just hyperbole?

<comment><USER27> You aren't idiots if you're using it where there are no better alternatives - it's preferable to use load balancers etc where available, but there are places where it's very much "DNS or nothing".

<comment><USER28> How would I use a load balancer to fail traffic between, say, London and Amsterdam with no fiber in place between them? Where would the load balancer physically exist in that scenario and how would it fail to the other when power is lost in one location? Would I make a third PoP to isolate it? What would then be my redundancy story for that PoP? How would I relocate traffic to my backup load balancer PoP number four?

Within a single network, sure, load balance all you want. That's not the scenario low TTLs go after.

<comment><USER3> > _How would I use a load balancer to fail traffic between, say, London and > Amsterdam with no fiber in place between them?_

What people use in those sitations is Anycast.

Of course, DNS itself, or e-mail, don't need this kind of redundancy, since the NS (or MX) records themselves provides a list of failover servers. The corresponding alternative for HTTP, SRV records, has been consistently stonewalled by standard writers for HTTP/2, QUIC, etc.

There is an interesting draft RFC which I am keeping an eye on, but I don't want to get my hopes up:

[https://tools.ietf.org/html/draft-nygren-dnsop-svcb- httpssvc...](https://tools.ietf.org/html/draft-nygren-dnsop-svcb-httpssvc-00)

<comment><USER29> > What people use in those sitations is Anycast.

This requires that you blow a publicly-advertisable prefix for every unique combination of services you would want to fail-over.

E.g., if you wanted to be able to have independent fail-over between your customer-facing self-service portal and your webmail interface (each relying on specific state that you can't replicate synchronously, and can't guarantee replicate consistently with each other), you would need to /24s, one dedicated to anycast for the webmail interface, one for the self-service portal, and separate from any services which are active-active.

Whereas using DNS, you could use your other existing public /24s that you are already using for your active-active services.

In the last days of IPv4, an extra 2 /24s just for this is quite an expense.

</comment></comment></comment></comment></comment><comment><USER30> That can be acceptable price for minimizing the impact of accidental DNS misconfiguration. Which probably happened to every sysadmin.

Or is there a better way to quickly invalidate DNS caches in case of emergency?

<comment><USER28> No, there isn't. The specification as implemented requires no invalidation mechanism, which means no such mechanism across _all_ caches exists, nor will it ever. The long tail kills you in such a failure scenario, and remember, people who make kitchen appliances write DNS resolvers.

</comment></comment><comment><USER31> "The urban legend that DNS-based load balancing depends on TTLs (it doesn't)"

So whats the solution? We are using AWS ALB/ELB and it states in docs, that we should have low TTL, and it makes sense. Servers behind LB scale up and down. What is the option B?

<comment><USER32> In fact, if you use Route 53 with an alias to an ELB, the TTL is hard-coded at 60s -- it is not even configurable. If it were, we'd follow the practice of lowering it prior to changes, and raising again once things or stable, but as it is, that's not an option (moving DNS off AWS would be a hard sell, not cause it's terribly hard but afamic, there's not really any value to doing it).

</comment></comment><comment><USER33> I would maintain that if you are experiencing poor performance for a web site, there are MUCH more fruitful places to look than DNS latency. Third party objects, excessive page sizes, lack of overall optimization based on device are just the tip of the iceberg.

<comment><USER34> For many apps I've worked the DB connection setup was always the slow part (use PgBouncer). Then, the part was the queries. DNS, gziped CSS/JS - <USER2> a red-herring.

<comment><USER33> Yeah definitely. A poorly crafted SQL query can wreak havoc on performance, especially at scale!

</comment></comment></comment><comment><USER35> > Here's another example of a low-TTL-CNAME+low-TTL-records situation, > featuring a very popular name:

> $ drill detectportal.firefox.com @22.214.171.124

Is captive portal detection not a valid use case for low TTL? The entire point is to detect DNS hijacking of a known domain, which takes longer when you cache the DNS results...

<comment><USER24> Captive portal detection involves more than just checking for DNS hijacking. The browser tries to load [http://detectportal.firefox.com/success.txt](http://detectportal.firefox.com/success.txt) and acts based on how that goes. Having a short TTL does not help.

<comment><USER35> If your captive portal is implemented by intercepting your DNS queries, then having a short TTL should ensure that the captive portal actually has a query to intercept.

But sure, there are other implementation approaches (e.g. injecting HTTP redirects), which I imagine is one reason why Firefox doesn't literally inspect the DNS reply.

</comment></comment></comment><comment><USER36> I run authoritative DNS for a very busy domain - 30B queries per month. Originally we had 6 hour TTLs, but now I use 60s. We have had no problems. Uptime and fast failover comes before anything else.

</comment><comment><USER37> There was a dns record looked up primarily by large supercomputers that had a 0 ttl. It was used for stats via a UDP packet (because it was non blocking, nevermind that the dns query was blocking). This was set to 0 for "failover" but it hadn't changed in years. I worked out that our systems alone had caused billions of queries for this name.

After I complained I think they upped the ttl.. to 60.

<comment><USER24> Reminds me a of a server pair at the last healthcare place I worked. Between the two of them they'd generate something around 1200 DNS lookups per second (about 60% of the load on the DNS servers) of their own name. I think the logic was if the name stopped responding then server A was primary. If the name was responding the server that owned the IP it was responding for was primary. If the servers wanted to swap primary/secondary they would issue a DDNS request.

After about 8 years we were were restructuring our DNS infrastructure for performance and I rate limited those two to 10 or so queries per second each. In that time there must have been 300 billion or so requests from those two boxes alone.

<comment><USER37> In my experience that sort of thing is from the local hostname not being present in /etc/hosts and (of course) a caching resolver not in use.

Some process on the system wants to connect to itself, which then causes a dns lookup. Add a high transaction rate on top of that, and 1,200/second is easy.

The funniest one I remember seeing was thousands and thousands of lookups for HTACCESS. Turned out apache was running on top of a web root stored in AFS and not configured to stop looking for .htaccess files at the project root, so it would try to open

      /afs/domain.com/app/.htaccess
      /afs/domain.com/.htaccess
      /afs/.htaccess

When it hit /afs/.htaccess then it would try to contact the AFS sell called .htaccess, which would do a bunch of dns lookups.

This would be done something like twice for every incoming web request.

</comment></comment></comment><comment><USER38> Just anecdotally, from running PiHole, looking at the logs, I have some sites being resolved 12K times over 11 days... That's over a thousands requests a day.

<comment><USER39> Running

      $ echo min-cache-ttl=300 | \
          sudo tee /etc/dnsmasq.d/99-min-cache-ttl.conf

will likely cut down on the number of forwarded queries by a large amount. Adjust the value (in seconds) to your needs.

Don't forget to run

      $ sudo pihole restartdns

afterwards.

</comment></comment><comment><USER40> Can anyone explain why ping.ring.com needs to have such a low TTLs?

      =-=-=-=-=
      $ drill ping.ring.com @220.127.116.11
      ;; ->>HEADER<<- opcode: QUERY, rcode: NXDOMAIN, id: 36008
      ;; flags: qr rd ra ; QUERY: 1, ANSWER: 2, AUTHORITY: 1, ADDITIONAL: 0
      ;; QUESTION SECTION:
      ;; ping.ring.com. IN A

      ;; ANSWER SECTION:
      ping.ring.com. 3 IN CNAME iperf.ring.com.
      iperf.ring.com. 3 IN CNAME ap-southeast-2-iperf.ring.com.

      ;; AUTHORITY SECTION:
      ring.com. 573 IN SOA ns-385.awsdns-48.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
      =-=-=-=-=

I've been trying to find out from Ring support for a few days, and while the support layer has been trying to find out, not much information seems to be getting back. To put this in perspective, in my house with two Ring devices (a doorbell and a chime), I am getting 10,000+ DNS un-cached requests a day, which easily is 20x more than the second most requested domain.

<comment><USER41> That's actually a pretty long TTL by Amazon standards.

Amazon.com:

        ;; ANSWER SECTION:
        amazon.com.             60      IN      A       184.108.40.206
        amazon.com.             60      IN      A       18.104.22.168
        amazon.com.             60      IN      A       22.214.171.124

Or some AWS services

Glacier:

        ;; ANSWER SECTION:
        glacier.us-east-1.amazonaws.com. 60 IN  A       184.108.40.206

S3 has even shorter:

        ;; ANSWER SECTION:
        s3.ap-northeast-1.amazonaws.com. 5 IN   A       184.108.40.206

Or, say, DynamoDB:

        ;; ANSWER SECTION:
        dynamodb.us-east-1.amazonaws.com. 5 IN  A       126.96.36.199

The main reason to do so is to be nimble, it's to be able to react to incidents as fast as you can and change, and to make potential deployment patterns possible.

From time to time, you need to do something with customer facing infrastructure: Remove the DNS entry, watch the traffic drain over the next 5-10 minutes, and then do what you need to do on the device, test, and then add it back in the DNS again, from which you can watch traffic return to normal levels and verify everything is good.

<comment><USER42> You're looking at the SOA (573) record for ring.com, but he's asking about the CNAME (3) for ping.ring.com.

</comment></comment></comment><comment><USER43> Thankfully this is one of those things, that you don't need to respect. TTLs are just suggested values in the end (the standard may disagree).

I just checked, and I actually have TTL forced to 1 day in dnscrypt-proxy. My internet experience is fine. I guess I never noticed in the last 2 years or so.

</comment><comment><USER44> Why does DNS cache expiration need to be in the critical path?

Instead of a browser doing

1\. Local DNS lookup (resulting in expired entry)

2\. DNS query

3\. DNS response

4\. HTTP request

why not do

1\. Local DNS lookup (resulting in expired entry)

2.1. DNS query

2.2. HTTP request

3\. DNS response

4\. If DNS response changed and HTTP request failed, HTTP request again

Maybe use two expiration lengths, one that results in flow 2 and a much longer one that results in flow 1.

<comment><USER45> Ya, this is roughly what the fb apps do. Dns rarely blocks, and changes are seen quickly.

</comment></comment><comment><USER46> Well, in my case it makes sense, I think: I host my server at home, and have a dynamic IPv4. I don't know when it could change, so I just set the TTL to something low.

Since the traffic is low, though, I can afford to check for an IP change every ~5min, and although I set a TTL of ~15min on most services, the main CNAME (ovh-provided dynamic dns service, TTL set by them) is set to 60s.

My IPv6 record was set to 1h, but I'll look into increasing it. It is true that my mobile phone often pings my server, so I imagine that it could reduce the battery usage.

</comment><comment><USER47> Please excuse any ignorant use of terminology, I am not a DNS expert like others on here, but I can share some experience in the smaller business world.

A company I worked with a couple of years ago was using Dyn as their DNS provider, and one day we got a notifcation that we had passed the usage limits for our account. This seemed impossible considering our site was getting a couple of hundred unique visitors a day. A few things came out of the analytics.

1) A short TTL on an A record had been left on from a website migration project. The majority of the requests were coming from our internal website administrators. I moved it up to a couple of hours and this went a way.

2) We were getting a huge amount of AAAA record hits. I think most modern browsers/OS try quad A first??? We didn't have IPv6 configured, and therefore the negative resolution had a TTL set to the minimum on the SOA record, which was 1second! A change of this to 60 caused a huge reduction in requests. I suppose I should have set up ipv6, but I didn't.

3) When we sent out stuff to our mailing list the SPF (or rather TXT) records saw a peak that was off the chart. We had a pretty settled infrasructure, so I moved that TTL to a day (I think from memory) and it flattened the peak somewhat.

4) There was a large peak in MX request around 9am. I put this down to people opening their email when they got to work and replying to us. I had to set the TTL to a couple of days (of course) to smooth that one.

I like to think it was worthwhile and improved things for users. I at least had a nice warm glow that I had saved the internet from a bunch of junk requests, and it just felt tidier.

</comment><comment><USER48> I'm wondering what it would take to make a DNS caching service with updates based on reliable notifications rather than polling? After all, every cellphone does it.

<comment><USER49> This already exists (NOTIFY), but it's only used for master-slave setups (ie. a bunch of DNS servers serving some authoritative zone who want changes to be transmitted to all slaves ASAP)

It would be interesting to (ab)use this mechanism in the way you suggest. A recursive DNS server could ask to be NOTIFY'ed of changes in the zone they are querying...it would, of course, add load to the server, and it would need strict limits to avoid DoS, but it seems an interesting idea.

<comment><USER45> The big problem, to the extent there is one, is between the client and the recursive server. Not as much the recursive and the authoritative. Cost is highly amortized between recursive and authoritative for busy names.

</comment></comment></comment><comment><USER50> The author says low TTLs are bad because of latency but never attempts to quantify how much latency we are actually talking about. Its hard to know how outraged I'm supposed to be without actually seeing the numbers.

It seems that a lot of sites are ok with slightly higher latencies if it means greater operational flexibility.

<comment><USER43> Latency is dependent on many things. DNS server location - accessing my website from australia will take 500ms for DNS lookup (or twice as much if I'm using cnames). If this is not cached somewhere, that's 500ms every few seconds with those <1minute TTLs. If I'm on GPRS, or similar, that will add a bunch more hundreds of ms to every useless DNS resolution, incl. unpredictable variability.

So there's no single latency to report.

</comment></comment><comment><USER51> "It was DNS" is the root cause of enough postmortems to justify low TTL values, in my opinion.

</comment><comment><USER52> I run my own dnsmasq server on an old laptop and force really long TTL caching regardless of what the records come back with. I even cache nxdomain. It works great, except once or twice a month I have to flush the cache because Slack seems to not handle it well.

<comment><USER53> I'm doing the same but "only" three hours and it's working just fine. Not a slack user though.

</comment></comment><comment><USER54> I'm honestly not sure what this author is complaining about. If the infrastructure can handle it and the zone owner is willing to pay for the excessive traffic, and DNS cache operators are fine with it, then this seems like a call for premature optimization.

</comment><comment><USER55> The missing data in the article that has many graphs is how often did the records truly change

</comment><comment><USER56> I have worked in a place using GTM to fail over from a bad data center to a good data center. Maybe few minutes TTL. I worried about it but availability is much higher this way, especially combined with a only change one data center at a time.

<comment><USER57> This.

I'm kind of suprised that I can't see any other comments talking about GTM's (I assume you mean F5 Global Traffic Managers)

Where I am at the moment GTM's are used everywhere, and everywhere the TTL is set to 30s.

The only part of this that really annoys me is that the global default configuration, rather than serving up a subset of the list of IP addresses, only a single IP address is returned when you resolve down to the A record.

When I've pressed the issue that _at least_ on our internal GTM's we should just return a bunch of IP addresses every time someone resolves the address, I've been told that it would break load balancing... which blows my mind because who on earth is relying on DNS to load balance traffic with a 30s TTL, I would have thought that the normal thing to do, if you actually wanted load to balance, would be to result a subset of IP addresses in a different order and with a different subset each time. That way other DNS servers which resolvers that will cache that record can at least be returning multiple addresses to all the clients it serves, as opposed to everyone using that resolver getting stuck to a single address for 30 seconds...

But all of that being said, it would make perfect sense to me to just return like 4 IP addresses publicly for every resolution and rotating setting the TTL to like 30s so that clients could spend 30s iterating through the A Records they have cached, then hit your resolver up again and get a different sites addresses back if your site had gone down...

</comment></comment><comment><USER58> Obvs. The authors server isn't seeing longer TTL's because there's no need for clients to keep querying his server for them?

</comment><comment><USER59> To avoid delay when migrating a website IP, what I usualy do is to first migrate on a HAProxy (like 2 days before switching) so all ISP DNS are update and on the D-day I switch my backend to the new website/VM.

And then I change my DNS again to the new IP.

You have to tune a bit to get the right IP in your logs but so far it works.

</comment><comment><USER60> >> The urban legend that DNS-based load balancing depends on TTLs (it doesn't - since Netscape Navigator, clients pick a random IP from a RR set, and transparently try another one if they can't connect)

Unless you do not return an RR set and what you return is based on geolocation and data center health.

<comment><USER61> Uhm, how does this work with "global" DNS services which people tend to use more and more? (Eg. Cloudflare's 220.127.116.11 or Google's 188.8.131.52/18.104.22.168)

Basically, your request is coming from them and wherever their servers are (US, I guess, though they probably have several data centers) and they route it to the final user.

I think using DNS-based geolocation sounds like a really bad idea: what am I missing?

<comment><USER62> The EDNS0 client-subnet extension exists for this exact reason.

<comment><USER61> Thanks. It seems, unfortunately, that only Google DNS and OpenDNS (Cisco iirc) include the data as of now. Older articles even mention how you have to have your website (well, nameservers) whitelisted for them to forward client subnet as part of DNS queries, not sure if that is still the case.

Of course, caching gets more complicated and less useful with this.

</comment></comment></comment></comment><comment><USER63> DNS Caching: Running on Zero

[https://archive.nanog.org/meetings/nanog50/presentations/Tue...](https://archive.nanog.org/meetings/nanog50/presentations/Tuesday/NANOG50.Talk64.bhatti- nanog50_dns.pdf)

</comment><comment><USER64> Should DNS providers have a setting that increase TTLs over time automatically? I.e the longer I leave my dns entry pointing to the same IP the longer my TTL is?

Obviously it would be possible to opt out for situations where you genuinely need a low TTL on a domain.

<comment><USER65> This is definitely a feature I've also thought would be useful to have in DNS providers.

I've worked on managing thousands of (sub)domains and the administrative overhead of changing the TTLs for everything manually would be considerable. I'd certainly like an automated way to say "These records should gradually increase TTL up to <X> time over <Y> time" (e.g., gradually raise TTL to 2 days over 2 weeks if there are no changes).

There are downsides to high TTLs though: (1) you need to remember to preemptively lower them ahead of any planned changes (if you want those changes to take effect quickly), and (2) you can't change the records quickly in an emergency. But, fortunately, lots of record types are ones that you probably don't need to change in an emergency -- and for ones that you do, you can use a low TTL.

Anyway, I'd personally like to see automated TTL management as a feature in DNS software.

</comment></comment><comment><USER66> For those running CloudFlare, proxied DNS records have an unchangeable TTL of 5 minutes.

</comment><comment><USER61> This entire analysis is plain out wrong.

They've collected data on DNS queries "for a few hours". By definition, clients who have DNS cached (iow, most clients, since browsers and resolv calls in operating systems will do that for you), will not issue DNS requests for any records that have a TTL that has not yet expired.

So, they've caught _all_ (well, all that were re-requested) the TTLs shorter than whatever "a few hours" is, and only those longer ones that expired exactly during the experiment and were re-requested.

To run a proper experiment testing for "short" vs "regular" (let's say 1-3 days), you need to collect data for days (eg. at least 7 days, preferably at least 30), but even that would not report most TTLs longer than 7/30 days.

Articles like this are bad because they can easily confuse even the knowledgeable people like the HN crowd.

</comment><comment><USER67> I hate low DNS TTLs. They are a stupid way to do load balancing.

However they wouldn't be quite as bad if web pages didn't load useless crap from 60 different domains.

</comment><comment><USER68> What if you accidently make changes to your authoritative nameserver you want the recovery to be as fast possible because it's a complete outage

<comment><USER69> Nothing precludes you from upping the TTL after the change. Traditionally DNS admins progressively drop the TTL prior to a change to reduce the time an RRSet is in flux (so if your TTL is N, N + 1 seconds prior you drop it to half N, and again and again until its your preferred window size) and cautious ones slowly ramp back up again to the regular value.

</comment></comment><comment><USER70> Am I missing something, or is the reason most of the queries observed have low TTL because, well, they have a low TTL? IOW, the higher TTL responses would be cached downstream and so you'd see them less often. If that is the case, the distribution shown is not all that surprising.

<comment><USER61> It's weird how people are not understanding this: perhaps it's the way you phrased it. Or perhaps you missed to mention the core part from the article: the experiment was only run "for a few hours". This means that many a DNS record (well, most) with TTL greater than the experiment duration would not show up in the data.

FWIW, I've learned in the past that while there are plenty of people who claim to want communication to be as succint as possible, majority are unable to understand when somebody is really terse (while still saying exactly enough). I've learned to follow up such a terse statement with examples and longer explanations for the majority that does not get it.

But maybe it's just that people don't expect the mathematics-level precision on the internet :)

</comment></comment><comment><USER71> Maybe DNS servers should support push updates rather than relying on polling.

<comment><AUTHOR> They do!

[https://tools.ietf.org/html/rfc1996](https://tools.ietf.org/html/rfc1996)

[https://tools.ietf.org/html/rfc1995](https://tools.ietf.org/html/rfc1995)

<comment><USER24> I don't think they meant to authoritative servers.

</comment></comment></comment><comment><USER72> Seems like DNS TTL was a big issue before HTTP 1.1.

Connections are cached and reused.

<comment><USER24> If you're talking about keep-alive this times out far before most of the DNS TTLs mentioned in this article and don't persist after a connection is closed anyways.

<comment><USER72> That's not true. It all depends on the implementation...

Also what are you talking about? If the connection is closed, how would it be used?

Connections should only be closed due to? Inactivity. If a connection is closed, don't you think you would probably want to do another DNS request?

Also if your doing proper layer 4 load balancing using BGP, DNS is a moot point...

Magic...

<comment><USER24> > Connections should only be closed due to? Inactivity.

Or if the user closes the browser of if the server/proxy restarts. But yes, mostly inactivity on the order of a couple of minutes.

> If a connection is closed, don't you think you would probably want to do > another DNS request?

That's the whole point of the DNS TTL, to say how long to go before doing another lookup rather than doing it each time you reconnect.

> Also if your doing proper layer 4 load balancing using BGP, DNS is a moot > point...

BGP load balancing operates on layer 3, and is irrelevant as you still need to DNS lookup an anycast address. EDNS client subnet is better anyways.

<comment><USER72> An anycast address doesn't change. I mean come on. And it actually operates on layer 4. It uses layer 4 to actually work?

I hope you didn't pay for your education.

<comment><USER24> Anycast addresses change all the time. Ask Google, Microsoft, Amazon, Akamai, Cloudlfare and so on if you don't believe me. About the only anycast IPs that don't change are public DNS resolvers but that's also true of unicast resolvers as well.

By that logic BGP is a layer 7 load balancer since it has an application layer. BGP only exchanges layer 3 reachability information to update route tables therefore you can only load balance layer 3 with it.

Personal attacks and other things in your comments are against the HN guidelines. The goal is to talk about DNS/TTLs and their impact on performance not insult each other. [https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

</comment></comment></comment></comment></comment></comment><comment><USER73> Cloud providers actually use low TTLs to route traffic globally away from regional failures. You're not seeing that go away anytime soon, there aren't other options.

</comment><comment><USER74> Uh? I always used 24 hours TTL for DNS. I reluctantly move it to 1 hour for some tests, then quickly set it back to 24 hours. What are these people thinking?

<comment> throwaway-9320 One use case where short TTL-s make sense is running a service on a residential network where a power outage or router reboot can trigger an IP address change. If the IP address changes then you won't be offline for too long.

Yes, it is not exactly great, but at least it works well enough for self- hosting services.</comment></comment>


Do you need that comment? - <AUTHOR> http://rile.yt/thoughts/do-you-need-that-comment/ <comment><USER2> Riley missed the "you're really proud of what you did there" reason... Unless that's acceptable...</comment>


IDEA to Business for Dad-trepreneurs - <AUTHOR> Hi all,
I would like to know your opinions/thoughts/experiences how to bring an idea to market. To help you advising me  I share below few details about my background and current situation.
I have been working for last 7 years in the finance industry as a quantitative developer and got a use case that I would love to work on.
I have read a lot in the last few years about product development and startups. I have even found a company and worked on it for 18 months a few years ago (not the current use case).
Currently, I have got family and 2 fantastic daughters, therefore, can't afford to work few more years with no income at all.
As senior software engineer, I have been working as consultant/ contractor/freelance and work on some other ideas as side projects. Unfortunately, I have learned that is quite hard or almost impossible to have a full-time job and then work at nights or early mornings (not sleeping much with babies) on a side project.
As a lesson learned, I have tried to work part-time but in the area where I live, it's very difficult to work part-time besides there might be some conflicts with intellectual property.
As currently, I have a very well paid job, my current strategy is to save money and in 1 or 2 years work on the idea. This is a good strategy but, it might be too late for my current use case.
I'm just looking for a way to bring the idea further!
Appreciate your comments and experiences.
Best, Scugnizzo <comment><USER2> What is your use case? Where are you located?

<comment><AUTHOR> I'm located in Zurich. My use case is in the financial index business as it requires transparency and independence. Smart contract can really help in this kind of business.

<comment><AUTHOR> I can give you more details if you want.</comment></comment></comment>


The day Google founders asked Steve Jobs to be CEO - <AUTHOR> http://www.pocket-lint.com/news/36534/google-bosses-wanted-apple-boss <comment><USER2> This "article" is misleading. The video which this link-bait inaccurately describes states "they visited Steve Jobs just to meet him because he was a hero of theirs and they said to John Doerr after the meeting 'Why can't he be our CEO?'" (The section of the video pertaining to the CEO hiring starts around the 24 minute mark.)

The video makes it seem as if that question was basically a joke and also that it wasn't ever directly posed to Steve Jobs. I resent being lied to by the puff piece, but it is interesting to think about how such an outcome would have changed things (bmelton's thoughts being an example).

</comment><comment><USER3> Wow, how flattering to Schmidt.

I'm not terribly fond of him, personally.

Not surprised that Jobs was unwilling to give up a hardware business. The profit margins might be worse but it's less volatile.

<comment><USER4> _Not surprised that Jobs was unwilling to give up a hardware business._

I'm not surprised Jobs was unwilling to give up a business he built (twice!) for a business someone else built.

</comment></comment><comment><USER5> Google would, quite obviously, be a VERY different company with Jobs on board, and I honestly doubt that Google would be anywhere NEAR as close to 'cataloging the world's information' as they are now with Jobs in charge.

Google has been quite happy to acquire when it made more sense than building it in-house.

Google has been quite happy to spread into entirely new markets. Apple shares this, sort of, but all their growth has gone into media markets, and Google has gone into some VERY un-sexy spaces.

Google is more than happy to throw together a minimum viable product and release it, as they have with Wave, GMail, their numerous other 'beta' programs, etc. I can only imagine how much Jobs would have slowed this process, making sure that everything was just perfect.

Mostly though, Google has way too many pots for Jobs to have had his hand on everything, which likely means that they wouldn't be shipping a tenth of what they are.

Apple's release schedule is slow, but generally worth the wait -- Google 'ships early, ships often', as so many of us advocate, sometimes to great success, and sometimes to early failures.

In many ways, Google still operates like a start-up, and Apple, in my opinion, does not.

<comment><AUTHOR> I agree with everything you said except the fact where you said that Apple doesn't act as a start-up, see here: [http://www.businessweek.com/innovate/content/jun2010/id20100...](http://www.businessweek.com/innovate/content/jun2010/id20100610_525759.htm)

Apple has refocused their strategy on core products, in contrast to Google. Encouraged openness and debate. And finally, created cohesion across business units. Apple is a different beast than Google, but I don't think that because they operate in differing ways that makes one more of a start-up than another.

<comment><USER5> Perhaps I misspoke, but I essentially meant it in regards to the way they launch new products. In more ways than not, Google is ultramegacorp. They're buying companies left and right, striking deals with Telecoms and sitting on who-knows-how-many companies' boards.

But, when it comes to launching something new, they definitely have no qualms about throwing the proverbial pasta against the metaphorical wall, and seeing if it sticks, hypothetically.</comment></comment></comment>


Buy this Heartbleed T-Shirt, and support OpenSSL Foundataion - <AUTHOR> http://teespring.com/iheartbleedopenssl

<comment><USER2> Not bad, shares info about OpenSSL and Heartbleed, a way of supporting foundation and it's a nice geek t-shirt. Why not.

True, it could be, somebody is trying to make money. If the t-shirt cost $30 and they would support only $5 then I would suspect they're concerned more about profit.

It doesn't cost much more than a regular t-shirt, so as long as they give $5 from each to OpenSSL I'm in. It's extra $1000 for foundation, if the goal is reached.

</comment><comment><USER3> Talk about trying to make money off of a negative incident.

<comment><AUTHOR> Trying to create a win-win scenario. The majority of net proceeds will go to the foundation.

</comment></comment><comment><AUTHOR> Cool or what?</comment>


Python Web Framework Advice - <AUTHOR> http://marcuscavanaugh.com/blog/python-web-framework-advice/

<comment><USER2> django is on the way to conquer the world <http://code.google.com/p/google- app-engine-django/> one of the project owners: gvanrossum</comment>


Analyze how well your resume matches a job - <AUTHOR> http://jobscan.co <comment><USER2> There seems to be a checkbox just _after_ the "Analyze" button, which might not even be on the screen if someone tries this out. It is checked by default and seems to mean that the site will share your submitted data with unknown recruiters. There also isn't a privacy policy on the site to clarify what the implications are.

To the site operator: If this is a genuine attempt to help people and you're not out to screw anyone, the above makes you look bad. Also, talk to a lawyer, before someone's lawyer talks to you much less pleasantly. Depending on where you are, you could be in all kinds of trouble if anyone didn't see that box and then objected to the consequences.

<comment><USER3> Could be a reverse honeypot for bots?

<comment><USER2> I'm genuinely wondering at this point whether we should flag this post and submit the site as a suspected phishing exercise to the major search engines. There are no verifiable contact details, terms, privacy policy or other expected information on it. It's presumably collecting a lot of sensitive personal information in a dubious way. The guy allegedly submitting this post seems to have an HN account that he only ever uses for self promotion and it appears he has never replied to any comments or otherwise interacted with anyone here. About the only information I can find that looks anything like real data about this site is that it's trying to raise real money at a $3M valuation via AngelList, and the guy supposedly behind it worked for Microsoft and Groupon, the latter claim not exactly boosting confidence in the legitimacy of this new site. I don't want to go all knee-jerky and cause trouble if it really is a genuine project, but this looks really shady.

<comment><AUTHOR> Hi there,

I'm the creator of Jobscan and I built the app in an attempt to solve the problem of qualified candidates getting weeded out by ATS because they don't have the right keywords on their resumes.

My previous start-up Wodache (feel free to Google it) won numerous awards.

This is a MVP in an attempt to solve a large problem. I prioritized product features over privacy and thank you all for the feedback. I'm working on a privacy notice and ways to make the site more trustworthy.

I'm also an avid reader of HN through Zaker News app so I don't tend to comment or share links. But I'll get on that also. :)

</comment></comment></comment></comment><comment><USER4> Genuine question : outside of software, are job listings also terrible reflections of the needs of the position? I feel like I am often contacted regarding positions for which I am not qualified on paper.

Recently, for instance, a recruiter contacted me regarding an opening which required the following (among others) :

 _7+ years experience, C, C++, Python, Ruby, Flash, Matlab, "network programming", RPC, sockets, "database programming" [1]._

I don't have 7 years experience, and would never advertise myself as having meaningful professional expertise with all but 1 or 2 of the actual skills listed (although I've worked with each at one point or another).

Yet, I would assume they thought I was a viable candidate for the position since it was sent my way.

[1] position title : "Ruby on Rails developer"

</comment><comment><USER5> That's a hard (and beautiful) problem, but this silly web app is definitely not the answer.

<comment><AUTHOR> Hi, I'm the creator of this app. I'd love to hear how you might think this tool can improve?

Thanks!

</comment></comment><comment> alok-g Bright.com tried to do similar thing [1], and shortly after sign up, I had a bridge construction job sent my way as a great potential fit.

[1] Heard this directly from the founder about a year back.</comment>


"I came across a cool pixel shader for simulating CRT phosphor look" - <AUTHOR> http://www.retronator.com/post/150868840323/while-preparing-my-next-retronator-magazine <comment><AUTHOR> More at "Designing a Large-scale Phosphor Filter/Shader" [https://news.ycombinator.com/item?id=12574396](https://news.ycombinator.com/item?id=12574396)

source: [https://twitter.com/retronator/status/779723309638299652](https://twitter.com/retronator/status/779723309638299652)</comment>


Dave Winer Says Google (Google+ Api) Doesn't Get It - <AUTHOR> http://scripting.com/stories/2011/09/15/googleDoesntGetIt.html

<comment><USER2> Strongly disagree. They may do that later, but it would be dumb to do it now. They let people post tweets to Google Buzz, and then Google Buzz was overrun by tweets, and nobody was interested in actually using Google Buzz anymore. They just used Twitter and automatically posted tweets to Google Buzz.

It would be a grave mistake on Google's part to do that so early in the product's life, when people are still figuring out how to use it best.</comment>


Instagram's UX Problems - <AUTHOR> http://jakeculp.me/blog/2012/12/8/instagrams-ux-problems

<comment><USER2> While these are certainly improvements that Instagram could make to improve their app, this article is more attitude than helpful critique. Absent the cruft, there is little substance here.

Unworthy of HN, IMO.

<comment><AUTHOR> Well, its on here, I guess you can just choose to ignore it then.

</comment></comment><comment><USER3> I actually don't get one point that I hear often and that's made in this article. Namely that Instagram used to have higher quality of photos. And now that it's more popular, the quality has gone down.

But what does it matter to YOU, what the average user is putting up? I have a list of people that I follow - who I consider good or am interested in. The fact that there is a million or billion other users snapping ugly photographs is irrelevant. Would you judge a photo-lab based on the service they offer or how good the photos of their other users are?

<comment><AUTHOR> My criticism of their user base was not suppose to be a given point in my argument and so most of that statement dealt with personal details instead of things that would effect others as well.

<comment><USER3> Right - I understand that this was a tangential point.

But I just see it brought up consistently (that Instagram is getting worse because of decreasing average photo quality) - which I don't get.

</comment></comment></comment><comment><USER4> Right now, the UX problem I am faced with is the sheer non-readability of your blog. First, I trust the fonts look beautiful on a Mac, and it certainly is not your fault that font rendering on Windows sucks big time, but unless you want to lock out a rather large share of potential visitors, you should really test your design on a Windows machine. Your fonts are simply unreadable on Win7/Chrome. Second, light gray text on salmon background? Seriously? That is virtually zero contrast!

<comment><AUTHOR> I just recently converted from Windows to Mac and when on Windows the fonts looked fine.

</comment></comment><comment><USER5> Another one: if I know someone's user name I can't add them through the "Find & Invite Friends" page under options, I can only do it through the explore page.

Also, the forced cropping of pictures taken outside of the app is by far the most annoying UX thing from my perspective.

</comment><comment><USER6> "people taking pictures of their weed and their 18 children with their baby daddies"

...

<comment><AUTHOR> It's a joke, take it or leave it.</comment></comment>


Ask HN: Is anybody else getting Facebook phony tagged-photo spam in their feeds? - <AUTHOR> I've seen this using multiple, unrelated Facebook users in my feed:
There's a fake Ray-Ban "Official Site" (rbdxxx.com, not the true Ray-Ban site, but a scammer) that used to generate fake recommendation posts from friends, presumably ones with weak permissions and/or passwords.
In the past 2 weeks, I've seen multiple cases of a fake Ray-Ban site now generating fake "1-day sale" ads as photos in somebody's Facebook photo set and then broadcasting ads by "tagging" each of their friends as being in the fake-photo-ad, which then generates an ad in each of their friends of the form "(Your friend name) was tagged in a photo."
The only flagging mechanism I see is to:
- click on the ellipsis
- select "Give feedback on this photo"
- click Spam
- click Send
... but it's built around being disciplinary to the Facebook user, and they lack a means of calling attention to the underlying fraudster. <comment><USER2> Doesn't this scenario usually indicate that the user in question has had their account security compromised? (i.e. they've been hacked)

</comment><comment><USER3> Yes, I've seen the exact same Ray-Ban scam multiple times over the years.</comment>


The Pedigree in Silicon Valley Privilege - <AUTHOR> https://medium.com/@puppybits/the-ingrained-biases-in-hiring-that-are-killing-meritocracy-and-diversity-de721316830#.bfd2g3zdh <comment><USER2> What I see in this essay is a dissonance resulting from a collision of what is necessary for a software engineer (or even webdev) vs required for a computer scientist. Which is reasonable; our pop culture, academic institutions, and hiring managers routine conflate the two. However, the truth of the matter is that the union between the two skill sets can be vanishingly small, depending on the specialization.

Some huge percent of open dev jobs can be done by just about any applicant after ~1 month of training; I think the market is starting to realize that and salaries will eventually correct. However, just because your average javascript code monkey or java CRUD app dev is overvalued doesn't mean there isn't value in a CS degree for actual context sensitive applications (data analytics, machine learning models, firmware dev, etc).

*This coming from a CRUD app dev who's paid way too much for how trivial what I write actually is purely on account of the network effect.</comment>


Ask HN: Career advice, please - <AUTHOR>

Hey everyone, at a point where I'm really not sure what to do with myself so I'd like some advice. I'm 22, a year out of college, been working for a (very) large tech consulting firm since graduation doing systems implementation (design to development, the whole shebang). It's definitely been a great experience so far, but I've come to realize I am not really interested in doing consulting long term so I've begun to evaluate different options. Long term, I'd like to move over to some form of product development. As such, I've really set my sights on Google's APM position for now.
Here's what I see as my options right now:
1. Spend the remainder of the summer preparing for the APM position and apply in the fall. I've got a solid background in design and programming, but I've already laid out an aggressive learning plan for myself this summer that I hope would prepare me in time. Despite this, I recognize it's a very competitive position and there's no guarantee of anything ever materializing.
2. An ex-coworker and friend recently transferred out of a team at Google where he was working as a Tech. Associate and offered to send my resume directly to the team. This doesn't seem all that glamorous of a position, but it'd get my foot in the door, get me some more hands on experience with Google products, and allow me to network within the company. Now, I'm fairly sure I'm qualified for the gig, but I don't know how Google is with people transferring internally and if I'd ever have a shot at landing a position in product development (and/or as a program manager).
3. Ignore Google, shoot for applying for an established (have already had two rounds of significant funding) start-up doing exactly what I had tried to do with a start up of my own, a field I'm definitely passionate about. They're offering a few interesting positions, but they're only internships which makes me incredibly nervous.
4. Disregard switching, acquire currency. Stick with the consulting gig, toughen it out for a bit, and wait until my prospects are better and I have a bit more cash tucked away. I'm not crazy about this option at all, though I recognize it being the safest. It could be a direct path to a MBA from a top-tier school which could let me move into any field I'd like.
5. ???
I really appreciate any advice anyone has. I know I'm only 22, but I don't think age or lack of experience should be keeping me on a path I'm not interested in.
TL/DR: 22, first year tech consulting looking to change. Options: 1. Spend summer readying myself (and eventually apply) to Google's APM position. Unsure if qualified.
2. Apply for Tech. Associate position at Google. Foot in the door, qualified, unsure if I would be able to transfer out into product development.
3. Apply to interesting start up.
4. Disregarding switching, acquire currency until I'm better prepared in a few years.
E: Added TL/DR ;) <comment><USER2> Seems like we are in the same boat; also 22 working as a senior network admin for my local telco, will be keeping an eye out for other peoples replies, would also enjoy the prospect of working somewhere like Google :)

My current experience says just go with whatever feels right for you at the time, people put too much pressure on the long term plan, basically there is no point in been miserable for a few years hoping that it might open a door later on...

If you do what feels right, in the future it's a lot easier to admit your mistakes (everyone will make them) and people/employers will understand more.

Also a few side projects while your working always help ;)

<comment><AUTHOR> Thanks. It's seriously comforting to know others are in the same boat as me.

I do have side projects on (and other, non-tech related interests) that keep me going regardless. Life is good, and it's not the end of the world regardless of what I do - thanks for keeping me grounded.

<comment><USER2> Another thing I would mention is don't assume your restricted to apply for the 'student' jobs at Google because of your age, I have a friend that got past 3 phone interviews, and the personal interview.

He was 22 at the time and only had 2 years of 'real world' experience within the area, it's all about your written/verbal/personal presentation and of course CV.

In the end he didn't make the cut, however that was more because they noticed he was interested more in the salary than the challenge of working at Google :)

\---

Just remember "keep calm and carry on"

</comment></comment></comment><comment><USER3> How do you know you'll love the product management position? It may be fun for a few months, but if it's not something you're passionate about you'll get bored. You'll fall in this groove of making really easy money and just be comfortable. There's nothing wrong with that if that's what you want!

When I was 22 all I wanted to do was find a high paying job. America kind of brainwashes you to think that's the logical next step after college. Spend some time thinking about what makes you happy and what you REALLY want to do with your life. Then just go after it. No need to wait and prepare yourself. Life is short man.

<comment><AUTHOR> Do I know for sure? No. Currently though, it looks like what I'd be most interested in.

I chased the money after graduating, and it hasn't worked out the way I wanted it too - honestly, at this point, as long as I can afford rent, student loans, PBR and the occasional concert ticket, I'm a happy camper. I just want a position that allows me to work with great people, learn, be creative, and grow as a person.

</comment></comment><comment><USER4> Go to Google, build your resume and get more experience under your belt. Save up. Then chase your dreams via your own startup or another that you're passionate about and have a little cushion. If things go awry, rinse and repeat.

<comment><AUTHOR> Thanks for the advice. Again - worth waiting for the position I want (potentially a long shot) or getting my foot in the door (and potentially transferring)?

<comment><USER4> Obviously I don't know all the details but you've got two disparate but important things to take into consideration here:

(1) A bird in the hand is worth two in the bush. To have the job now even though it's not exactly what you want is important for the aforementioned reasons. Not to mention the possibility of transfer. I would lean this way.

(2) HOWEVER, I am the type of person who won't shine unless I'm doing what I love and am passionate about. If you are the same and think that might have a hinderance on your performance then hold off. If you have the discipline to buckle down even though you don't like your job then refer back to #1. By judging your opportunities, I would surmise you have the discipline that I lack.

Good luck.

<comment><AUTHOR> Again, thanks. Definitely something to think about. I very much to have the capacity to 'buckle down' and do something that isn't my number one priority, if it has the potential to pay off in the future. At this point, thanks to your comments, I'm leaning to just applying for both positions and just seeing what happens.

<comment><USER4> Email me in the future if you want any advice (in profile). I'm interning at LivingSocial in DC right now and love the hell out of it.

</comment></comment></comment></comment></comment><comment><USER5> Join Google.

"Experience is the name everyone gives to their mistakes." --Oscar Wilde

<comment><AUTHOR> Obviously my goal - however, should I wait and prepare for the APM or go for the Tech. Associate role and try to transfer after some time?

</comment></comment><comment><USER6> I'll go for the Google role. Social proof of the name "Google" counts.

</comment><comment><USER7> One advice I would like to give as a 31 year old to 20 year old kids is: Save up. This may not seem like having any connection with career or startups etc at first. But only if I had saved up last 10 years, I would have a lot better safety net to quit my job and start something of my own. Not that I would not do it now but it would have been nicer to have a better padding. It is simple. If you have entrepreneurial itch as a 20 year old, you cannot have the lifestyle of a regular 20 year old who spends most of what they make (not generalizing necessarily but you get my point). I bought a BMW convertible 4 years ago since I could afford it. I did not think about saving that extra money instead and keep my old car which was btw a 2006 Honda. So you see the point. Today, I only drive my wife's car since I can commute to work using public transportation. Just my 2 cents.</comment>


Flavors of Comment Spam - <AUTHOR> http://tatiyants.com/?p=564

<comment><USER2> I love Markov Chains. See also the Usenet hoax / experiment "Mark V Chaney".

</comment><comment><USER3> Spot the 'bayes killing' spam :)</comment>


Ask HN: Mailbox alternatives for OS X? - <AUTHOR> I'm looking for exactly mailbox, without bugs.
I love the swipe to archive/schedule. I use those features all the time.
Outlook on iOS is the perfect replacement. But what about OS X?
Prediction: Unless a startup comes out of nowhere and does this, MSFT will end up replacing Mailbox.app on OS X (hopefully!) <comment><USER2> I switched from Mailbox to Inbox a few months ago. Glad I did!

There is no desktop app, and the settings aren't as customizable but the bundling feature makes up for that.

</comment><comment><USER3> Right now I am using Airmail and I'm pretty happy about it. I have about 4 email accounts and it works well.

The shortcuts are Gmail compliant so you can use whatever you are used for with motion/archive/delete and more.

I've used Postbox, MailPlane3, Mailbox, Mail and a lot of others, Airmail is the best one so far.

</comment><comment><USER4> This is a bummer. Despite how incredibly buggy Mailbox is on OS X, it was still my favorite client since Sparrow. I might give Nylas[0] a try.

[0] [https://www.nylas.com/n1](https://www.nylas.com/n1)

<comment><AUTHOR> The (current) lack of 'swipe to snooze' is unfortunate with Nylas. Looks great otherwise though.

</comment></comment><comment><USER5> I've basically tried them all, and the latest Airmail client works the best for me. It's fast, no lag in the animations, does what I want it to do, and even looks decent.

</comment><comment><USER6> I use Airmail and I like it, though my usage doesn't require anything out of the ordinary, beyond having different mailboxes for different accounts.

</comment><comment><USER7> I use Postbox but I'd be interested in hearing from anyone who has used both Postbox and Airmail to see if they prefer one over the other?

</comment><comment><AUTHOR> This seems especially relevant now that Mailbox is officially being shut down on Feb 26.

<comment><USER8> That's why he asked...

</comment></comment><comment><USER9> I still miss sparrow. It felt so much lighter weight than airmail on OSX.

</comment><comment><USER10> Airmail 2</comment>


How Your Camera Works - <AUTHOR> https://www.objc.io/issues/21-camera-and-photos/how-your-camera-works/ <comment><USER2> The right sidebar should be able to be hidden or outside of the content. It currently covers up a majority of the images on the right

<comment><USER3> Assuming you're on a desktop, and you're currently scrolled over an image, put your mouse cursor just to the left of it. It'll hide itself after a half second or so.

<comment><USER4> Yes but depending on the size of your screen (or if you zoom in the text) it keeps reappearing every time you scroll down the page a little :/

<comment><USER5> [http://www.readability.com/m?url=https://www.objc.io/issues/...](http://www.readability.com/m?url=https://www.objc.io/issues/21-camera- and-photos/how-your-camera-works/)

problem solved.

</comment></comment></comment></comment><comment><USER6> Why does every mention of a phone camera have to be iPhone? Can some people not separate phone from iPhone any more?

<comment><USER7> Because it's the most used phone camera, the most common point of reference.

See Flickr's stats: [https://www.flickr.com/cameras](https://www.flickr.com/cameras)</comment></comment>


Jemalloc 1.0.0 released: the malloc used in FreeBSD, NetBSD, and Mozilla Firefox - <AUTHOR> http://t-t-travails.blogspot.com/2010/04/stand-alone-jemalloc-100.html

<comment><USER2> Rather than linking to a paragraph on someone's blog, this ought to link directly to:

<http://www.canonware.com/jemalloc/>

It looks like in addition to jemalloc serving FreeBSD, NetBSD, and Mozilla, it is also used for Facebook servers:

 _Since 2009 I have adapted jemalloc to handle the extreme loads Facebook servers commonly operate under. Facebook uses jemalloc in many components that are integral to serving its website._

<comment><AUTHOR> The blog entry was written by Jason Evans, the author of jemalloc.

<comment><USER2> Okay, I understand that, but judging from the briefness of the post and its lack of detail it looks like Jason Evans must have simply intended to direct his readers to the official site. Therefore it would seem to me that this post should have linked straight to the official site as well.

<comment><AUTHOR> The official site doesn't have an announcement, though. It's not exactly "news" to link there.

</comment></comment></comment></comment><comment><USER3> Promoting it as "the malloc used in Firefox" isn't a good PR strategy. ;-)

</comment><comment><USER4> Before considering changing the allocator to improve performances, review the way you consume memory.

<comment><USER5> While you can try to not do anything stupid with regard to your allocations, there's often very little you can do.

If you try to implement your own allocator you're often just wasting time. You either use the best allocator library you can find or end up having a half- baked allocator of your own on top of a poor default allocator library. A good allocator is essential for performance.

Writing a good allocator is not trivial so your home-made allocator is likely to be slower, consume more memory than an optimized generic allocator, and exhibit all kinds of weird random hiccups when a critical malloc() or free() happens exactly when the library can't avoid housekeeping anymore and goes into a costly cleanup mode. An optimized allocator has loads of knowhow in guarding against these corner cases, both algorithmically and by having empirical special cases to help amortize the cost of maintenance.

I've seen many programs that couldn't possibly have predicted their allocations any better but were still stuck until an alternate allocator was bolted in and the juices began to flow smoothly again.

I expect a modern allocator to adapt to various use cases, from managing large variable-size chunks of memory to intelligently pooling a thousands and thousands of small objects of one size. And much more. Automatically.

<comment><USER6> I think the point was that a "home made" allocator is going to be able to exploit structure in the data that won't be visible to a general purpose malloc -- c.f. the pool allocators used in Apache, which know a priori that they'll be freeing everything at the end of the request. Likewise objects with the same lifetimes can live in the same block, etc...</comment></comment></comment>


Facebook's Canvas Encryption Proposal - <AUTHOR> http://developers.facebook.com/docs/authentication/canvas/encryption_proposal

<comment><USER2> It seems that this is just a PR move by Facebook... that will hopefully be equally as misunderstood by the media and general public as the original problem is in itself. I don't see how the sharing of UID's is any different/worse than cookies that are persistent on users computers other than the fact that users may be less aware of them, and i guess they're not associated with a store of personal information.

I think that the perception is that if a business gets my Facebook UID, they have access to my information - which is not true, they can't retrieve any more information than you publicly make available.

Maybe I'm just misunderstanding what the big issue is all about though.

</comment><comment><USER3> This privacy problem is trivially solvable. Facebook should generate a surrogate key for user/app relationships, and only share that with the apps. Then my ID cannot be tied back to me by anyone but facebook, and I cannot even be tracked across applications by the same developer.

<comment><USER4> We have talked a lot about solutions like this, but most of them end up more trouble than the encryption proposal. For example, developers need to link back to your profile page and your friends' profile pages in their UI - do we change all those URLs individually for all applications, too (since they contain a user ID)?

I think it is a clever idea, but it ends up breaking down in a lot of ways, and it has the bad quality that it is not backwards compatible with existing applications.

<comment><USER5> Why not create a new URL handler that takes an app-specific user ID that redirects to the correct profile page _only if_ the logged-in user has permission to view the page?

That way, you would normally only be able to see what the world would see; but if you try to view your or your friend's profile, it's fluid?

</comment></comment></comment><comment><USER6> This sounds like major overkill. If the only problem is the data in the iframe url, couldn't they POST the data into the iframe?

Create the iframe with some JS that POSTs a form containing the data to the canvas app. The resulting iframe url is the same but without any query parameters.

<comment><USER7> Yup. Also it's not solving the problem of app developers sharing the UID's intentionally. After the decryption at app's server side, they can share uid as usual.

I am not sure if fb has rules for not pushing uid's with 3rd party services - other than that, I don't see how this solution avoids the problem.

<comment><USER8> Exactly ... this encryption is absolutely useless for intentional sharing. Facebook does have rules against sharing ANY data with third parties, however people are clearly violating that. Update ... I just blogged about it here: [http://www.allfacebook.com/is-facebooks-proposed-user-id- sol...](http://www.allfacebook.com/is-facebooks-proposed-user-id-solution- sufficient-2010-10)

</comment></comment></comment><comment><USER9> The risk of sending the UID of the user to ad networks is that it's possible an unscrupulous ad network will store and use or sell the data.

However, if an ad network wanted to get the UIDs they can still easily get them. Just about all Facebook iframe applications use the Facebook Javascript SDK, which provides methods that can easily be used to collect the user's current ID or other information. Most ad networks require developers to load a remote javascript file on their site, and this javascript file can simply make calls through the SDK to fetch whichever information it wants.

<comment><USER10> That only works if the ad javascript is embedded directly in the page. All you have to do for this not to be an issue is host the ad in an iframe, and fortunately most ad integrations tend to be this way already.

</comment></comment><comment><USER11> Not that it matters given the problem domain, but the Python example code here has a timeable HMAC. Don't use native string compare to check MACs!

</comment><comment><USER12> Maybe it would be better not to waste time on this and instead work on educating the public. How about a comic ("Facebook UIDs and You")? :D

Actually the problem is how Facebook is working its way into all parts of the internet with its widgets and Facebook logins.

</comment><comment><USER5> This is a sad attempt to reconcile with the community.

It's hardly a proposal when they already seem intent on following through with this plan. It's not exactly an RFC.

Not to mention the flaws present pointed out by the other commenters...

</comment><comment><USER13> So, how does this affect [https://chrome.google.com/extensions/detail/ejpepffjfmamnamb...](https://chrome.google.com/extensions/detail/ejpepffjfmamnambagiibghpglaidiec)</comment>


He went to an in-network emergency room. He still ended up with a $7,924 bill - <AUTHOR> https://www.vox.com/2018/5/23/17353284/emergency-room-doctor-out-of-network <comment><AUTHOR> > Kohan called for a nurse, who explained that he would need jaw surgery that > night. In the meantime, he tried to check whether the hospital — Dell Seton > Medical Center — was in his insurance network.

> "I was on my iPhone lying there with a broken jaw, and I go on the Humana > website and see the hospital listed," Kohan says. "So I figured, okay, I > should be good."

Except he wasn't: While the emergency room where Kohan was seen was in his insurance network, the oral surgeon who worked in that ER was not. That's how Kohan ended up with a $7,924 bill from the oral surgeon that his health plan declined.

> Kohan called his insurance plan, which assured him that nothing was wrong — > that the doctor had likely coded the visit wrong and would just need to > resubmit the claim.

> Kohan's doctor did resubmit the claim, and increased the price of the > surgery by about $2,300, billing records show.

> I reached out to the doctor who saw Kohan, an oral surgeon named...

> himself did not respond to my request for an interview. But the day after > Canizales and I spoke, Kohan said she called him — and let him know that the > entire $7,924.13 charge was being dropped.

> I also reached out to Dell Seton hospital to ask why it contracts with an > oral surgeon who does not accept any medical insurance.

> The hospital declined an interview but provided a statement noting that this > is a common practice across the country.</comment>


Roofstock: Buy rental homes with tenants in them - <AUTHOR> https://www.roofstock.com/ <comment><USER2> Being a landlord requires time and money and energy, and it's not something you just dive into without looking carefully. It's certainly not something you should jump into if you can only put 20% down. If you are not willing/able to be on call every day, then you need to hire a management company and pay them a large cut of the rent each month. There may also be HOA dues and there are certainly taxes. There could be expensive maintenance, etc. Each thing takes a slice out of your rental income.

On top of all that, the renter may get angry at the world and trash the place causing tens of thousands of dollars of damage, or they may up and leave because they can no longer afford the place. Worse, they may stick around and stop paying rent until you forcibly evict them after months of no rental income. You may be suddenly forced to spend money updating the place to comply with some new law.

I'm all for making things easier, but don't fool yourself into thinking you're making free money, or even making money at all for sure.</comment>


Allan J. Lichtman's election prediction system - <AUTHOR> https://www.washingtonpost.com/news/the-fix/wp/2016/10/28/professor-whos-predicted-30-years-of-presidential-elections-correctly-is-doubling-down-on-a-trump-win/ <comment><USER2> Man fits linear discriminant classifier on small, noisy set of data, finds it has 100% accuracy on entire set. Did not do any cross-validation to figure out robustness of fitting procedure or magnitude of generalization error.

News at 11.

<comment><USER3> To be fair he claims to have developed the system in 1980 and then used it subsequently to predict elections. But you're right that the data is noisy and small. And he did get it wrong in 2000 by predicting an Al Gore victory. So 7/8 correct on the "test set."

</comment></comment><comment><USER4> And that's why we're all worried about this thing called "overfitting".

I generally hate it when people post XKCD comics in response to more serious posts, but this one contains a compiled list of presidents who were elected despite some precedent saying they wouldn't: [https://xkcd.com/1122/](https://xkcd.com/1122/)

<comment><USER5> The Washington Post should be embarrassed for publishing this silliness... I'm tempted to flag the OP.

PS. The Washington Post's editors may want to read this Washington Post article from 2012, titled "the problem of overfitting elections, in one brilliant cartoon:" [https://www.washingtonpost.com/news/wonk/wp/2012/10/18/the-p...](https://www.washingtonpost.com/news/wonk/wp/2012/10/18/the- problem-of-overfitting-elections-in-one-brilliant-cartoon/)

</comment></comment><comment><USER6> I have a bachelor's degree in political science* from AU, and Lichtman was my prof for one class. Suffice it to say that prolonged exposure to him is more than sufficient to cure a person of any notion that he is a genius who can see into the future.

There is a tiny kernel of a good observation buried in his model, which is that the everyday "horse race" events that the political media focus on have very little to do with who wins and who loses in a presidential election. In fact it's pretty safe to say that _campaigns themselves_ don't have a whole lot to do with that.

In the modern era at least, presidential elections are generally just a referendum on the performance of the incumbent. If the economy is at least stable, and there are no disastrous wars going on, voters almost always re- elect the incumbent or elect the designated successor from his party. If the economy is in the tank, or a war we're involved in has taken a particularly bad turn, they throw the in-party out and vote the out-party in. That's more or less all there is to it.

Which means that the victor in presidential elections is almost never a total surprise; by looking at the economic indicators, you can make some fairly solid predictions which party will win months before they even get around to choosing their candidate, no elaborate model required.

Lichtman's innovation was take that basic idea and then hang a bunch of bells and whistles on it, the entire purpose of which is to impress credulous reporters by taking a rule of thumb and presenting it as some kind of mechanical fortune-teller. Which, since reporters need things to report on and he provides a kind of pre-packaged story that requires no research and only one interview, invariably gets picked up.

So: every four years he gets to do a round of press, applying his "method" to the current election, which is good for his career and for the profile of the university. And of course none of the reporters circle back after the election to measure how well his Prediction Machine really performed; and even if they did, he's left enough subjectivity in his "keys" to give himself an out if his predictions turn out to have wrong.

Everybody wins, I guess? Except the people who read those stories and think there's actually something more to them than some basic historical knowledge and a dollop of PR razzle-dazzle.

* _Well, actually an interdisciplinary degree in "communications, legal institutions, economics and government." But the path I chose through those disciplines was heaviest on the poli sci._

<comment><USER7> So GWB won because of the dot com crash?

<comment><USER6> 2000 is a fascinating case. You may recall that GWB did not win the popular vote.

</comment></comment></comment><comment><USER8> An interesting article and "keys to the White House", but nowhere does he "double down" on a Trump victory. If anything he's saying "anything is possible" for this election.

Glad the headline here is not the same as WaPo's suggestive one.

</comment><comment><USER9> Not defending his overfitting, but it's worth pointing out that recent polls are suggesting Johnson won't reach his 5% after all. In which case (if Clinton wins) this professor can still claim accuracy.

</comment><comment><USER10> Statistical bullshit aside, even if his stats were on point, you'd have to say Trump is a Black Swan of a candidate, breaking models generated by past data.

</comment><comment><USER11> List of keys, for those who (like me) that the article failed to provide said list: [https://en.wikipedia.org/wiki/The_Keys_to_the_White_House#Th...](https://en.wikipedia.org/wiki/The_Keys_to_the_White_House#The_13_Keys_to_The_White_House)</comment>


Working with Apache Spark: Highlights from projects built in three weeks - <AUTHOR> http://www.insightdataengineering.com/blog/Working-With-Apache-Spark.html

<comment><USER2> Really Nice set of projects. Would have also like to see a starter application with a mini tutorial on how someone can experiment with some of these technologies.

Github references on these projects would also have been useful. Thanks for sharing.

</comment><comment><USER3> Did you consider Summingbird? Seems like a lot of what you are doing might have been simplified by using Summingbird rather than building separate speed and batch layers in Hadoop / Spark.

<comment><AUTHOR> That's a great point for our next session - Summingbird would be perfect for everyone trying to implement Lambda Architecture without having to produce separate code for streaming and batch.

</comment></comment><comment><USER4> Is the code available somewhere ? I'd love to see the one related to fantasy sports

</comment><comment><USER5> Would love to try this! Thanks for sharing !

</comment><comment><USER6> Thanks for sharing! This is awesome!!</comment>