Show HN: URL Canary – Get an alert when someone finds your secrets - <AUTHOR> https://urlcanary.com/ <comment><USER2> You might think about embedding bounties in crypto blockchains. For example, create BTC wallets that can be unlocked using a secret sitting next to (or steganographically embedded in) the secret you're trying to protect. This gives the person uncovering the secret an incentive to activate the canary. {RI,MP}AA are apparently doing this with their music and movies, so they know when they are showing up on pirate sites.

<comment><USER3> As skeptical as I am about crypto currencies, this is a really interesting application. Basically exploiting human greed. Thank you for sharing it.

<comment><USER4> That's some awfully pessimistic and dehumanizing language. I just consider it to be paying someone for the trouble rather than jumping straight to "exploiting greed."

<comment><USER5> Since the person profiting is someone who obtained the secret through shady means - either breaking into an insecure system, or taking advantage of their access to a system which doesn't encrypt passwords - I don't consider it pessimistic or dehumanizing to describe the behavior as greed.

<comment><USER6> Greed is a subjective term. The company who is charging money could be described as greedy for making a profit when they could have lowered the price to remove any excess profit. Why is the person who understands and uses the system in a way it wasn't intented greedy?

<comment><USER5> Because they're stealing secrets from people? I confess I'm confused why this is confusing. The exercise here is to know when something is public that you want to keep secret. The proposed solution is to make the secret something have value independent of its secretness, to tempt those that stole the secret to obtain that independent value. I'm fine attaching "greed" to profiting from stealing secrets.

</comment></comment></comment></comment></comment></comment><comment><USER7> Similar idea than [https://uriteller.io](https://uriteller.io). This is nice way to check whether your end to end encrypted chat is really secure or not.

<comment><USER8> This is brilliant. They recommend using a url shortener, but I want to see if anyone is parsing comments and visiting urls from HN.

[https://uriteller.io/7S6asCJSwrOzApjG84hIRA](https://uriteller.io/7S6asCJSwrOzApjG84hIRA)

Edited to add:

Here's the view key if you're interested. Just append it to the end of uriteller.io:

ZBt0gGoUHtIsyQ7KFwikYg

A crawler on AWS hit it two minutes after I posted it.

<comment><USER9> Lots of Mac users with out of date OS :S [https://uriteller.io/ZBt0gGoUHtIsyQ7KFwikYg](https://uriteller.io/ZBt0gGoUHtIsyQ7KFwikYg)

</comment></comment></comment><comment><USER10> FYI - I signed up and the email confirmation page showed me someone else's canary URL

<comment><AUTHOR> That's not good! Taking a look.

EDIT: I believe it's because the CSPRNG state ( [https://metacpan.org/pod/Bytes::Random::Secure::Tiny](https://metacpan.org/pod/Bytes::Random::Secure::Tiny) ) was created before the process forks, so they shared the initial state and generated the same token. I've reduced it to 1 worker pending an actual fix.

Sorry about that, and thanks for pointing it out.

<comment><USER11> Given this kind of disaster potential, wouldn't it be a rather attractive option to use /dev/urandom?

<comment><AUTHOR> Bytes::Random::Secure::Tiny seeds itself from /dev/urandom, I just need to make sure to initialise it on first use, instead of when my program first starts (which I've now done).

In general I suspect if anything you'd be _more_ likely to mess it up by reading bytes from /dev/urandom manually than by using a library.

<comment><USER12> How exactly can one mess up reading bytes from `/dev/urandom`? Serious question.

Open the file. Read from it. If no failures on open or read, you have random bytes. In essence, there _is_ already a library for this: `open` and `read`, which seems to be the same API surface area as this library.

<comment><USER11> Here's all the ways this can possibly go wrong:

[https://insanecoding.blogspot.com/2014/05/a-good-idea- with-b...](https://insanecoding.blogspot.com/2014/05/a-good-idea-with-bad- usage-devurandom.html)

<comment><USER12> Most of these are ridiculous.

First, the author mentions that a `read` from urandom can be interrupted. I am unaware of any system where this is actually possible. And _even if it were_ , the author's original code (and my description of an implementation) already works! The `read` call will return an error, and that error is handled. His "improved" code is simply an optimization around retrying from this device, but it's not an improvement in safety.

His second argument is that /dev/urandom might not have enough randomness in it. This is, quite simply, not a concern for anyone not writing code for specific embedded devices or for extremely early in the kernel boot process. Anyone who is writing code for these environments is almost certainly already aware of these limitations. And even then, using a library like the one the GP is using doesn't actually help, since it's virtually guaranteed to just be reading bytes from `/dev/urandom` for its seed in the first place.

The rest go into situations that — quite frankly — border on ludicrous. If someone has replaced your `/dev/random` with `/dev/zero`, _you have already lost_ and there is nothing you can or should reasonably do besides nuke the machine from orbit.

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> I don't understand this. Someone just visits the URL and I get a email notification? If it's supposed to be secret, how does the script knows if the visit is OK or not, sending me the notification only when it's OK? Are the good guys supposed to know this is a sort of "honeypot" URL and not visit it?

<comment><USER14> Put a URL in a firmware image that is never called by your device/app.

Monitor the URL for access. You then receive an alert that someone accessed the URL.

This gives you a real-time notification that a reverse engineer has looked at your firmware. It also gives you an IP address. So you now "know" that someone might try to hack your device. And you also have an IP address.

This is a billion dollar security play!

Jokes aside. It is a good concept for IoT firms who have a security advocate, but no budget. Would help persuade people that there are hackers targeting their devices/apps with quantifiable data of degrading value.

<comment><USER15> Huh. That's...a good idea. The idea has to be developed further though. This has to be deployed on different domain names and with full content control on the web pages.

I guess that's all doable with the "private server with root access" under enterprise pricing. What a great way to precisely measure cover time. You could inject arbitrary URLs into an application to see if your API has been reverse engineered.

<comment><USER14> Hackers worth their salt work in air-gapped environments.

This is a signal, but not a game changer for security pros.

<comment><USER15> I'm speaking specifically about the case where someone is trying to reverse engineer a private API from an application. Then interacting with an API endpoint will necessarily trigger the canary.

Having retrieved API secrets offensively, and overseen secret rotation defensively, I'd say it _would_ be a game changer. It's an excellent idea to automate this discovery with an alarm. The current discovery system is either an internally developed, half-baked version of this that comes from sophisticated logging, or manual oversight.

</comment></comment></comment></comment></comment><comment><USER16> Semi related asking the wizard of HN. I recall reading a similar trick whereby one can embed a github key/token/api key/??? In a source file in one's repository and then if your code ever is stolen and pushed to github then you will receive a notification because github will revoke the token. Does this or similar ring a bell for anyone?

<comment><USER17> [https://blog.github.com/2015-02-05-keeping-github-oauth- toke...](https://blog.github.com/2015-02-05-keeping-github-oauth-tokens-safe/)

</comment></comment><comment><USER18> Great project! I just found out that one creepy Safari extension is crawling every URL I visit.

Also Yahoo Slurp is crawling my email URLs. Sigh.

<comment><USER19> What's the extension?

</comment></comment><comment><USER20> This is neat.

For added security, maybe better to hide the canary URL in a bit.ly link? Someone might know your 3 URLS.

<comment><USER21> You can preview bit.ly links by appending a + to it.

<comment><AUTHOR> But if you are already suspicious enough that you want to do that, you were never going to click on the original URL anyway, so it's no worse than not using bit.ly.

<comment><USER22> Not true. I know bitly and googl have those info pages, so I'll check them out when I am curious but am not sure whether I want to alert whereever it leads. I know I'll get more info at best, and don't lose anything at worst. For random URLs, I guess I could try whois lookups, but I'd be much more likely to just check it out than with a short URL which is easily checked out.

<comment><AUTHOR> I think you missed my point.

If seeing that the bit.ly URL redirects to a known-urlcanary domain would put you off visiting the URL, then seeing the raw known-urlcanary domain (not behind bit.ly) would _also_ be enough to put you off visiting it.

<comment><USER22> Ah, yes, if that is the alternative, then of course. I thought it was between some shortened link and an unknown domain.

</comment></comment></comment></comment></comment></comment><comment><USER23> A quick test using Zapier's webhook functionality tells me I can duplicate this with their platform. Doesn't seem to work with link shortener, but I have the sense more time invested on that will yield a result. So, I think the concept has merit but you're off center of the target and much more value is required.

<comment><USER24> I imagine there's a lot of value to be found in a dashboard, or generally in managing these canaries. Which canary was deployed where, how often are they triggered (with the ability to mark some instances as false alarm) etc.

But of course that heavily depends on the use case

</comment></comment><comment><USER4> So what happens when the three offered domains become widely known to be fake? Can this service be federated and use custom domains or is it just another game of whack-a-mole?

<comment><USER25> From the landing page: "...and it's easy to setup a URL Canary on a custom domain name."

<comment><USER26> Not to be overly negative, but if you're going to the trouble of pointing a custom domain to this service, couldn't you just write a quick PHP script that mail()s out when it's accessed? As far as I can see, the only advantage of using this third-party site is convenience, a custom domain makes it much less so.

</comment></comment></comment><comment><USER27> Will the canary URLs always be to the same domain? If so, can't I circumvent this just by using custom dns locally to map that domain to somewhere else, like localhost? If it's not a domain, but an IP address, can't I just route that IP somewhere else?

<comment><AUTHOR> There are currently 3 domains available. If this takes off I could do something like mailinator (offer the user a small selection, but never reveal the full list at once).

You can also register your own domain and point it at my server and your canary will work just fine on that domain.

If you're playing at a high enough level that you've specifically blackholed URL Canary traffic by IP address, then you're a worthy adversary. And additionally, that is a splendid problem for my project to have.

On the enterprise tier you even get your own server with its own IP address, so there should be nothing linking it to URL Canary at all. (Although the enterprise tier is extremely expensive, and if anyone buys it I will probably panic).

</comment></comment><comment><USER28> This is neat.. Its currently a single type of token (ie. a URL).

If you check out [https://canarytokens.org](https://canarytokens.org) you will notice the ability to create several others (be notified when someone resolves an IP address, be notified when someone opens a file, be notified when someone views a QR code, etc)

</comment><comment><USER29> how does this differ from [https://canarytokens.org/generate](https://canarytokens.org/generate) ?

<comment><AUTHOR> Very little!

Although the generated URLs don't have "canary token" in them.

<comment><USER28> Canarytoken allows you to download a docker instance, so you can host the server on any domain you like.

</comment></comment></comment><comment><USER30> I'd really like to see a generalisation of this idea to any personal/private data stored in any database. Any time a piece of 'your' data (e.g. a medical record) is accessed, you get an alert. There could be an industry of alert brokers that decide if the alert is important or not - you might employ one, or write your own, or choose to look at every alert. While it would require a big change to how we store data, I suspect that the changes required to be GDPR compliant start going in this sort of direction.

</comment><comment><USER31> Is following a URL common practice for someone who accesses documents? I mean if I come across a repository, my first inclination isn't to find all of the hidden URL's and chase them.

<comment><USER32> It's common practice for robots and vulnerability analysis

</comment></comment><comment><USER9> How does a potential customer know how effective this is? How do you even know that it's effective for your personal use case as described [https://urlcanary.com/about](https://urlcanary.com/about) ?

How many attackers are going to click that link or _any_ link for that matter? Seems the value prop of the product is based on the assumption that folks will click. Maybe it's a solid assumption. I just can't see the evidence for it.

<comment><USER33> The value lies in sometimes demonstrating a channel is insecure. It won't always do so. Obviously the converse--proving a channel is secure--is much harder.

I don't think it has commercial value really. But for social awareness, showing that mail, or notes, or storage providers aren't always as private as you'd hope, that's where the value is.

<comment><USER9> But it demonstrates nothing if no one clicks. The channel may be insecure, compromised and no one knows - not the author or the parties who are supposed to have the secrets.

I see what he's trying to do, but this isn't the way I don't think. Mathematical proofs that verify that a payload is observed, opened, or accessed work. They are deterministic. They are also way more complex. I think this is trying to solve a problem in a simple way but it's still just as nondeterministic as without this solution IMO.

<comment><USER33> Absolutely. But if you have the choice between a) not knowing 100 pct of the time, and b) knowing for certain it's not private X pct of the time, for unknown X probably not 0, you'd want b. You will have more information by trying.

</comment></comment></comment></comment><comment><USER34> This looks neat. What is a good use-case for your average techy? What about your average non-techy?

<comment><AUTHOR> It can be useful anywhere you have sensitive information that you wouldn't want to fall into the wrong hands, or more specifically that you would want to find out if it did.

It could go in backups, in your git repository, bug tracker, internal wiki, etc.

For an average non-techy, I don't know... they might want to put one in their diary?

<comment><USER35> For any web/cloud service, I think this might be easily triggered when the service is trying to be helpful. I.e. when fetching previews, displaying in/active links, etc...

<comment><AUTHOR> You're right, and this does happen.

If that's a problem for you then you can mitigate it a bit by checking the User-Agent and IP address in the alert email you receive.

</comment></comment></comment></comment><comment><USER36> Do you have domain diversity? Organizations will blackhole urlcanary.com

</comment><comment><USER37> Nice! I've often considered implementing a service like this at the DNS level. DNS would be ideal because it would work for more services, not only HTTP.

</comment><comment><USER38> given the options are 3 domains to choose from:

      1. emotionalrec.com
      2. factwisdom.com
      3. tdurl.uk

Smart thief simply won't follow the link.

<comment><AUTHOR> Even then you're no worse off than if you didn't have it at all.

</comment></comment><comment><USER39> This is neat. Reminds me of the MR robot episode where he puts a link to track the FBI in season 3

</comment><comment><USER40> Maybe I don't understand the concept but why would you ever put a secret public?

<comment><USER41> You don't. You put it in a thought to be private place but you'll find out if it gets public at some point that way.

<comment><USER40> But why host it somewhere reachable at all?

<comment><USER41> It's a canary, it should ideally never be triggered but in case it does you want to know about.

Another way would be to put a fake user in your users database and then watch password leaks and see if it shows up. Then you know you've been breached. It should never happen but if it does it's good to know.

</comment></comment></comment></comment><comment><USER42> I like the simplicity of this.

</comment><comment><USER43> fail2ban[1] can be set up to do this trivially.

[1] [https://www.fail2ban.org/](https://www.fail2ban.org/)

<comment><USER44> ok I'm like 8 pages into the configuration wiki and still have no idea how to set this up

<comment><USER43> I'd recommend just installing it and turning on a couple of the built-in filters by editing /etc/fail2ban/jail.local to see how it goes together. See the comment at the top of /etc/jail.conf as well and check out 'man jail.conf'.

Essentially to do the same thing as URL Canary you'd set up an action that only emails and trigger that with a custom filter that scans your web server's access log for accesses to a particular URL.

</comment></comment></comment><comment><USER45> this is a really interesting application. Basically exploiting human greed. Thank you for sharing it.

</comment><comment><USER46> Hah, that enterprise pricing is insane.

<comment><USER47> Indeed, at £1k (€1136, $1403) one might as well run an internal application. At a very basic level, it can be achieved with three lines of bash:

        while echo -en "HTTP/1.1 200 OK\r\n..." | nc -l $IP $PORT; do
            cat $MESSAGE | sendmail -i -t
        done

More complex configurations could still be worked out in <50 lines of Python.

<comment><USER48> Why would anybody buy this (dropbox), I can recreate this with a simple ftp server and 30 minutes of coding!

<comment><USER47> Dropbox is not a devs-only product, given that eg. the CEO might want a personal Dropbox folder, or one might need a shared folder between the marketing team and the developer team (and even then, at the enterprise level it is not unheard of to use simpler alternatives like SFTP/NTFS shares).

Canaries on the other hand are exclusively used by technical people, who won't mind developing and spinning up a tiny honeypot.</comment></comment></comment></comment>


Basel III: The good, the bad, and the ugly - <AUTHOR> http://banksimple.com/blog/2010/09/17/basel-iii-good-bad-and-ugly/

<comment><USER2> Very thoughtful post. Everyone who wants "more" banking regulations should look more towards the capital reserve and other leverage measures instead of trying to nit-pick things like banning X fee, Y technique or Z instrument. Markets work when risk is priced at its real level - as opposed to the governments bail-out moral hazard and bankers' group-think.

His conclusion that there are still problems with how risk is weighted is certainly true as well - though I'd say that a lot of this could be solved using super-transparency.

Let require releasing a programmatic model for every instrument that will determine pricing for the underlying assets (i.e. all the mortgages in a MBS would be individually modeled with real data inputs). Then we can have a real idea of the risk from the sum of the underlying assets.

<comment><USER3> The new requirements on loan level data are fairly extensive and will certainly facilitate micro-analysis of portfolios of asset backed securities. The problem is that this will result in even more financial engineering. The government basically doubled the amount of informaiton that loan underwriters have to take when writing loans. It even requires information such as how far the recipient lives from their primary work place (in the case of home loans).

That kind of information is not easy to come by and it's certainly not easy to incorporate into an analysis. It raises the cost to the loan underwriter, to the loan issuer, to the securitization underwriter, which ultimately goes full circle back to the end consumer (credit card rates are already at 15 year highs due to the Dodd bill). There is also another side effect, this legislation, pariticularly as it pertains to securitizations which fund the majority of home loans and consumer debt, will facilitate further complex financial engineering (the more data you have, the more you're likely to use it). It will also give certain funds the opportunity to discern demographic information, which is currently not "technically" possible but will be a lot easier to do once the flood of data comes in from issuers. What i'm hinting at is that institutional speculators and investors will be able to discriminate against pools by analying income, location of workplace, years of employment (another factor), income history etc.

I may be ranting so I'll cut short here. Long story short, the costs of banking are about to sky high for everyone. I am not sure the American public would support this legislation if they realized just how much it would effect their lifestyle, access to capital, cost of capital, and consumption.

<comment><USER4> "What i'm hinting at is that institutional speculators and investors will be able to discriminate against pools by analying income, location of workplace, years of employment (another factor), income history etc."

Isn't this a good thing? Aren't these the factors that a bank should consider when deciding to lend money? Wasn't it the lack of such analysis that led to over-valuing mortgage securities that turned out to be worthless?

I'm confused.

<comment><USER3> You would think so but the reality is that these factors, when enforced, place certain demographic groups at distinct disadvantages. That disadvantage can be augmented when a bank can no longer hide (for all intents and purposes) who they are lending money to.

For example, I could analyze a pool of loans by a Michigan based auto lender. Under current guidelines, i basically get id numbers, fico score, stats on seasoning, outstandnig balance, etc. Who knows where these people live. With new data i could start guessing with reasonable accuracy where these people live (never on an individual basis, but you can get an idea of what counties or towns). I could discern that Bank B is lending to a "riskier" demographic than Bank A. While Bank B cant legally discriminate in their lending policies, the capital markets can certainly discriminate on which notes they want to buy. This would raise the cost of capital to banks lending to "disadvantaged" groups in society (minorities, immigrants, etc).Now a quick fix is for people to analyze the data and make sure that there is no humanly way that analysts like myself can figure narrow down where people in a pool live (which allows us to make assumptions on makeup), but from what i've seen that isnt going to be the case.

Now you're in a predicament. African Americans are over twice as likely to default on a home loan than their white counterparts. Who stands to lose here?

Look its just a thought. Finance is so complex that it's virtually impossible to figure out the second and third layers of unintended "consequences" of monumental legislation like the Dodd bill. Things are still playing out and there are armies of lawyers trying to predict how it will affect capital markets. So much is still uncertain (which explains why the markets are behaving the way they are). One thing isnt, the middle and lower classes are going to get hit hard. Not because the rich arent being taxed enough, but because these regulations are going to make the cost of funding consumption noticeably higher across the board. My group is predicting that within 2 years we will see an end to "free" credit cards that come with rates less than 9%. With risk-based pricing out the window, issuers need to find a way to imbed reserves into their loans or recoup higher potential for loss.

<comment><USER4> "My group is predicting that within 2 years we will see an end to "free" credit cards that come with rates less than 9%."

I still see this as a good thing. More people cutting up their credit cards is one of the best things that can happen for long term financial stability, in my opinion.

</comment></comment></comment></comment></comment><comment><USER5> I have a banking background, and the articles greatest argument is that 2015 is too far away. Although I agree it's far away, banks rely on so much leverage that unwinding assets can take some time. Mainly mortgage and other long-term loans make 2015 relatively short notice.

Mildly off topic, but still looming large is the bonus system within banking. I know that across the board bonuses are centered around short-term goals. From the CEO all the way down to the lowest teller, the majority of employee and management incentives have to be aligned with the long-term viability of the company. Until this is fixed, the financial sector will continue the boom and bust cycle.

</comment><comment><USER6> When I read the Black Swan by Taleb the crucial point was to realize how weak we are when we rely on financial experts. Be careful, bad regulations can be opening the door for new creative thinking. For example in Spain the new accounting system gives more freedom to imagination when giving value to assets.</comment>


Having a Cat Does Not Cause Mental Illness, New Study Suggests - <AUTHOR> http://www.huffingtonpost.com/entry/cats-mental-illness-toxoplasma_us_58adc06ae4b03d80af714072 <comment><USER2> If you look at the abstract, all of the correlations actually show that having a cat does predict mental illness at all ages considered. But the effect sizes are relatively low (albeit consistent with the more direct studies looking at people actually infected with toxoplasma rather than people who might be infected), so with a few hundred or thousand people, it technically doesn't satisfy p<0.05. Nevertheless, this is further evidence _for_ toxoplasma having bad effects.

This seems to be a trend with toxo papers - take a too-small sample, do no power or Bayesian analysis, find all bad correlations, but declare evidence of safety anyway.

<comment><USER3> The jump from "we could not detect a difference" to "there is no difference" is so frustrating. (And this is a mistake the authors make in this case, not just the journalists.)

<comment><USER4> They _did_ say "no evidence of a link" in the title and abstract, though the conclusion at the end of the abstract unfortunately says the study strongly suggests there is no link. Given the quoted odds ratios are > 1, this is a strange statement to make.

</comment></comment></comment><comment><USER5> Let me guess... study sponsored by cats?

<comment><USER6> Obviously you didn't read the article. The humans that come in contact with cats have altered minds; they sponsored the article, with no direct links to the original feline overlords.

</comment></comment><comment><USER7> Is there a simple, accurate and cost efficient test for toxoplasma? I got a cat 2 years ago... it never leaves the apartment on the 7th floor and never comes in contact with other cats. However it is a rescue cat so could have been infected before adoption.

<comment><USER8> Your pet is not even remotely close to the cats risk group.

Only the 0,7% on average of the cats drop oocysts in its faeces and it still will take 1 to 5 days for this oocysts to activate and be able to transmite the disease. If you clean daily the litter box of your cat with gloves and you are not pregnant or inmunocompromised the problem is greatly minimized.

<comment><USER9> Is washing your hands not sufficient enough? I have a cat and clean the litter box every other day without gloves.

<comment><USER8> Should be, but gloves will not do any harm. Toxoplamosis is not the only disease that your cat could host.

</comment></comment></comment></comment><comment><USER10> Oh thats good. This was something I've been wondering lately.

</comment><comment><USER11> Having a cat may not cause mental illness, just that in my experience having a cat is a sign of mental illness

<comment><USER12> i realize you're joking, but it made me think: i wonder if there are any correlations about "not having a pet" with egocentrism.

<comment><USER11> I feel people who own pets don't see things from a pet's perspective. But that's probably just me. Having someone love you based on sort of sort weird stockholm-syndrome based relationship is weird. The cat comment, as much as it's a joke I have seen most folks around me who own cats have this need for constant drama. And Goodness, no matter how much the cats randomly lashes out at these owners, the owners still love em, where's the bloody logic in that? I guess everyone on here is a S&M lover. But anecdotal evidence doesn't count for much haha

<comment><USER13> If a cat is "randomly lashing out" at their owners, then yeah, that's a pretty good sign that the owners don't see things from the cat's perspective.

Cat social behavior has its own internal consistency though. Usually the problem starts when people expect cats to behave like dogs. They don't have the same motivations as dogs at all.

Learn cat body language, and understand that cats see us as big, weird, slightly dumb, hairless cats that belong to the same "cat colony" as them, and nothing a cat does will seem "random" anymore.

<comment><USER11> With all due respect, I will never understand how that statement differs from trying to understand the ways of an abusive ex and trying to thread carefully based on it? pretty sure the abusive nature comes from somewhere and then for some darn reason I'm walking on egg shells when I could pretty much invest my time in a more nicer manner. Unless you think that putting up with complete crap is something people ought to aspire to do (don't mean to come up as an asshole, but seriously mate?)

<comment><USER13> Well now I'm confused. My cats have never done anything that could be likened to an abusive partner.

I respect my cats as individuals with their own needs and desires and they respect me back. We're like roommates.

If you had a roommate that didn't speak English as their first language, would you make an effort to understand them so you could get along? Or would you scream or try to hit them or spray water in their face (as a lot of people do to cats) anytime he did something you didn't like? And if you did, and he reacted violently in turn, would you really be surprised?

<comment><USER11> The question is why would you want to be stuck with someone who doesn't speak your language and gets pissed off at everything you do because they see you as dumb by default and their culture is the stark opposite of yours?

<comment><USER13> I kind of feel like we're speaking past each other here. I'm not sure what sort of dysfunctional cat/owner relationships you've observed to make you so vehemently opposed to the idea that they can be mutually beneficial.

I assure you, my cats actively enjoy and seek out my company, attention, and affection, and I likewise enjoy and seek out theirs.

When I get home, my cats run to greet me, with the straight upraised tail (with slightly curled tip) that indicates "friendly excitement." They sniff me and rub themselves against me with the full length of their bodies, which is a common greeting ritual for cats that belong to the same colony. They come and sit next to me on the couch, and when they want privacy, they will go and sleep somewhere else.

If they seek out my attention, and I don't have time or the desire then to give it them, a few firm setting them on the ground or shifting them out of the way, and they'll settle and not bother me (unless something is genuinely wrong and I haven't noticed, like the water dish being empty/dirty).

When I seek out their attention, if they don't feel like giving it right then, they indicate so through body language or even just leaving, and I respect their desires and don't force myself on them (unless it's in the case of something for their health, like giving them medication or taking them to the vet).

They can't speak English, though they understand a handful of words, such as their names, "treat", "dinner", "come here", "no", "gentle", and "please" as well as tone of voice. I can't speak Cat, but I understand a handful of words via body language and tone of voice when they meow.

I don't ever get angry at my cats for being cats, though occasionally I do get frustrated. And in return, though they sometimes get frustrated when I am failing to understand one of their requests, they don't get angry and lash out either.

I think we communicate fairly well for beings of two completely different species, and find each other's company mutually agreeable.

<comment><USER14> Don't bother, it's obvious they're not a pet owner and don't understand the sort of companionship one can have with pets.

When I first got my cat (for mousing) I was shocked at how quickly I grew attached to her. I had her about a week and she found a way out of the house, I searched and searched and searched and was about to let her go emotionally when I found her, and I was extremely happy.

And I had only had the cat a month, and no I didn't live by myself either.

You could use the same logic this person is using to describe a parents relationship with their child as well. It's just someone's opinion who has never owned a pet.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER15> Fine! Guess I'll have to go back to blaming my parents or eating lead paint chips.

</comment><comment><USER16> Why the cat hate on HN?

<comment><USER17> HN hates everything, cats included. It's bewildering.

<comment><USER18> > cats included. It's bewildering.

Only to those possessed by cats.

Q.E.D.

</comment></comment></comment><comment><USER19> My cat told me so a long time ago.</comment>


Harvard didn't reject you because you're Asian - <AUTHOR> https://gettingintohypsm.com/2018/12/08/you-didnt-get-rejected-because-youre-asian/ <comment><USER2> In the hypothetical, with 100 asians and 1 black all applying with identical credentials, if the school picks the 1 black student and 1 of the 100 asians, then the school certainly is being racist. They may be saying that they are going for a diverse student body, but they are choosing the black student because that student is not like the others in 1 category - race. So they are using race as a factor, and in this hypothetical, the determining factor. The fairest thing would be to put all 101 applicant names into a hat and pick 2.</comment>


Smaller Cities Are Luring High-Tech Talent - <AUTHOR> https://www.wired.com/story/how-smaller-cities-trying-plug-brain-drain/ <comment><USER2> The title on the article is "How Smaller Cities Are Trying to Plug America's Brain Drain". Why did this post get a headline rewrite, a rewrite that arguably changes the meaning of the original headline, and does not match the article's content at all.

A quote from the article: "Between 1940 and 1980, the wage gap between poorer cities and richer cities in the US actually shrank, according to research by Penn State economist Elisa Giannone." Unfortunately, the article fails to follow up on what happened around 1980 (say, 1975-85) that caused this. What policy(s) changed? What technological or sociological developments contributed to this? It seems like this article concentrates on the outcomes that follow from workers making rational economic decisions: taking high-paying jobs, or demanding more money for in-demand skills, rather than trying to give a context for workers entirely rational, market-driven response to the context. Seems like an odd gap.</comment>


Ask HN: Where Can I Get Some Free / Royalty-Free Music Tracks - <AUTHOR>

Hey guys,
I'm writing a couple of games using XNA. One is a personal project, and one is for school. Where can I pick up some decent background music which I could include in my game? Note that I'm probably going to distribute through XBOX Live and maybe the Windows Marketplace for $1-$5 if I ever get around to completing them both.
Preferably looking for a electronic/techno/ambient style of music.
Thanks <comment><USER2> Archive.org has an "open source audio" section that contains a lot of CC licensed music. <http://www.archive.org/details/opensource_audio>

</comment><comment><USER3> Take a look at Jamendo < <http://www.jamendo.com/en/> > \-- it offers music licensed under Creative Commons licenses, and it's relatively high-quality.

</comment><comment><USER4> There are a lot of musicians who advertise themselves on iDevGames in the Classifieds section of the forum (<http://idevgames.com/forum/forumdisplay.php?f=4>). A lot of them will work for free, or under _very_ reasonable terms, because they want to experiment with games or they're just feeling generous. And they really are responsive. I've worked with a couple.

</comment><comment><USER5> shameless plug: You're welcome to use any of the music from my 2008 album, it's all CC - providing you can find anything worth using, I make no guarantees :) Just let me know if you're interested.

<http://snugglytako.rpmchallenge.com/>

</comment><comment><USER6> If you find a track licensed under Creative Commons, be sure that it's not licensed Non-commercial. It's bar far the most common CC provision.

Also try Rumblefish's music licensing store: <http://musiclicensingstore.com/>

</comment><comment><USER7> Whatever you end up with, make sure to double check it's legality. People like to put copyrighted music on free music websites.

I too recommend archive.org and jamendo.com. Also check out <http://www.freesound.org/>

</comment><comment><USER8> <http://musopen.com/> is a pretty decent archive of classical music. Not quite what you're looking for, but a useful resource if others are interested

</comment><comment><USER9> you might be able to find something you like at <http://www.youlicense.com/> they have a lot of bands and if you can't find something you like for free you'll still definitely get a good licensing deal.

</comment><comment><USER10> thesixtyone has a CC section with a very good collection. Here is their CC electronic section: <http://www.thesixtyone.com/cc/electronic/> There are some other relevant genres too.

<comment><AUTHOR> Wow, I'm actually going to use this site as a source of music for myself as well. Good find.

</comment></comment><comment><USER11> Check out iamusic.com.

</comment><comment><USER12> ccmixter is good.</comment>


Ask HN: Domain dynamics? - <AUTHOR>

Should the home page of a domain be a portfolio or should it be a blog? I have a really short and sweet domain (reggi.com) and I don't know what to do with it. Should I use subdomains like this? http://portfolio.site.com/ and http://blog.site.com/ . Should I use root folders like this? http://site.com/portfolio/ and http://site.com/blog/ . Examples would be nice!!<p>Is the best way to do this with multiple wordpress installs or is there a better way? <comment><USER2> Indeed, examples would be nice!! What's the domain ? Since you already have it you don't need to be shy about it.

If you do an 'Ask HN' it helps if you provide enough information to actually answer the question ;)

<comment><AUTHOR> It's my name reggi.com

<comment><USER2> Cool, ok in that case I'd go for the blog, it immediately ties your blog content to your person and that's more or less a natural match.

I can't easily figure out an association where 'reggi' can be made in to a stand-alone item either. On top of that a blog can be moved to a section of a product site later, so you don't actually lose much if you decide to change tack later on.

To prepare for that you could do:

<http://reggi.com/blog/>

And redirect the homepage there for the time being. If you change your mind later on all you do is drop the redirect and provide a link to /blog/ .

Hope that helps!</comment></comment></comment>


Leave only footprints: how Google's ethical ignorance gets it in trouble - <AUTHOR> http://arstechnica.com/gadgets/news/2012/05/googles-street-view-engineer-knew-data-collection-was-questionable.ars <comment><USER2> >Whether it was a matter of time pressure or miscommunication, the apparent lack of advice and supervision resulted in an alarming misstep for Google.

Then there is also the question of how much baby sitting / bureaucracy / micro management can a company like Google should/could afford.

For instance, where I work, we have mandatory training for handling ethics issues without supervision. It is pretty decent in terms of coverage and clarity. Then there is a relatively modestly staffed ethics department that can assist in resolving any issues that cannot be resolved using existing training and guidelines. There is no supervision or management involvement here. It's not management's job to burden the Engineers with supervision.

Perhaps Google could learn from this and develop a "when in ethical doubt, don't do it" culture amongst the Engineers and have a Ethics compliance department handy if issues arise that cannot be resolved using the standard training.

Obvious things aside, did the Google management actually encourage the Engineers to collect plain text data and what did Google actually do with that data - that we will never know. But what _could_ they possibly do with people's browsing data, from a management/corporation perspective? Certainly they have no use for people's passwords to increase their profits? (The "rogue employee" can do harm with that data but that is different territory.)

<comment><USER3> > _"Then there is also the question of how much baby sitting / bureaucracy / > micro management can a company like Google should/could afford."_

This wasn't a small side project.

Streetview is a major initiative which required the sustained deployment of significant resources on a worldwide scale and an vast ongoing commitment to the bandwidth necessary to serve the data collected.

<comment><USER2> Yes, but for any company with multiple major initiatives the smarter thing to do is to decentralize the decision making and let the individuals handle it at the initiative level. Otherwise you end up with heavy middle management that becomes hindrance and liability for the most part.

</comment></comment></comment><comment><USER4> I'm going to go ahead and play devil's advocate; I think concern over this is really overblown. There are two things about this case that I don't really get:

1\. If you're blasting your data over an unencrypted wifi connection, do you have a reasonable expectation of privacy? This seems to be equivalent to someone screaming the contents of their email, and then getting angry at you for eavesdropping. Also, who the hell transfers that kind of data over anything but SSL?

2\. What was Google trying to get out of these packets, besides wifi-GPS data? This seems like more of a simple overstep of collection than anything else. Do we seriously think that this data was collected maliciously? They already have the majority of people's data, what more would they be trying to get? After all of my DNS lookups, my GPS coordinates, my email, and my social connections, what more is there?

<comment><USER5> > _1\. If you're blasting your data over an unencrypted wifi connection, do > you have a reasonable expectation of privacy?_

Yes, you have a reasonable expectation of privacy. Connecting to someone's unsecured wifi is a criminal offence; scooping data from that connection maybe in some jurisdictions.

People may be stupid for not securing their wifi. But no-one at Google is stupid. They don't have that excuse.

> _Do we seriously think that this data was collected maliciously?_

Not being evil isn't a magic pass for being daft. Google should have known better. The engineer raised concerns, so at least one person knew it might have been a problem. Google has lived through years of people being concerned about privacy. How can they get something so simple so wrong?

<comment><USER6> >Connecting to someone's unsecured wifi is a criminal offence

Not necessarily. Check your local laws, and be aware that they will almost certainly differ from those elsewhere. <http://en.wikipedia.org/wiki/Legality_of_piggybacking>

<comment><USER5> True. I should have added "[...] in the UK".

</comment></comment></comment></comment><comment><USER7> Google is not to be trusted. Thats the hard truth. From the article:

"Outlets like the New York Times noted that Milner did raise the [privacy] issue with superiors before the code was implemented (managers claim to never have read his reports)".

Right. Nobody at Google knew anything about this engineer collecting data for 3 YEARS? Thats just not reality. Corporations doesnt work that way.

And the fact that Google lies about it shows exactly what they are like.

<comment><USER8> It's extremely interesting (read: troubling) to watch Google transform from a collection of geniuses to a bog-standard corporation under substantially the same management. Perception is everything, sometimes.

</comment></comment><comment><USER9> Was it deliberate, though? The impression I gained was that the engineer made the software dump the packets and extract the SSIDs, but forgot to remove the dumps.

<comment><USER10> That was Google's old story. The new story is that Milner discussed the idea of running analytics on the captured traffic.

</comment></comment><comment><USER11> This creates the appearance that every single person involved with actual responsibility (managers, etc) decided to throw Milner under the bus at the first sign of trouble. Disgusting if true.

Claiming not to have read his reports makes them shitty managers. Claiming that everything is his fault despite their complete failure to actually do their jobs makes them terrible people.

</comment><comment><USER12> [http://www.theregister.co.uk/2007/04/18/uk_war_driving_arres...](http://www.theregister.co.uk/2007/04/18/uk_war_driving_arrests/)

Situation in UK not clear, most police action is around cautioning people doing strange things with laptops in parked cars. Responding to complaints.

Given current security levels in UK coming up to Olympics, any use of communications kit like this in a moving vehicle would need to be really upfront, or you had better have a good story when stopped.

<comment><USER13> War-driving is no longer about laptops in cars.

A couple of years ago I spoke at a New Zealand hacker con (<http://2010.kiwicon.org/the-con/talks/#e41>) about "Wardriving in the age of Arduino".

My final demo was an Arduino-based War-walking rig that easily fit into a stylish Trilby hat and logged SSIDs with GPS locations as you walked.

You could then easily display the result on a map with indications of the WEP/WPA/etc security in use.

Is that dapper gentleman out for an afternoon constitutional or casing your network?

(Besides, by now, most (possibly rooted) smart phones can probably do all that's required anyway.)

<comment><USER12> Would respectfully advise that enhanced hat wearing people have a very clear explanation ready should the police stop them for any reason! They will need chin straps on windy days. I take the point about smart phones.

What I was getting at is that street law often depends on whether someone has made a complaint to the police or not. Wardriving/warwalking may not _automatically_ be breaking the law in the UK, except through the 'theft' of network or 'nuisance' issues.

</comment></comment></comment><comment><USER14> If your premise is that Google is a big bad company, this is your case? I've always found this story to be excruciatingly disappointing.

So, besides cameras and lasers, they also had a wifi sniffer on board that just captured all the data in the right frequency band. That is, after all, the whole point of this exercise: gather data and see what you can do with it later on. The case for doing this for WiFi is obviously that matching SSID to GPS locations and a little bit of triangulation through signal strength gives you a fast, precise enough location with a simple WiFi scan without having to wait for a one minute cold GPS fix. That you are not even going to get if indoors.

The big problem here seems to be that they captured raw data instead of just running an actual wifi scan like your network card would. But if you've ever had to manually trigger one, you know that these take time, and then they only represent a single snapshot. So capturing raw data over the whole timespan solves both the timing problem and it allows you to get much better results given that you can now pretty exactly tell when a network comes out of reach. These things however require analysis that is easier done at some backend, not on the same machine occupied with freaking 3D lasers.

And yes, it turns out, that if you were sending an email at the exact same time the google car was in reach of your network and that your network was unencrypted, they could possibly collect single isolated frames or fragments of your message. The reason we don't use WiFi for time critical stuff after all is of course that data corruption is very frequent and might require many turnarounds for one package to arrive in whole and correct.

If you want to prove that Google is evil, find that 'Chiquita colombia killings' case. This one is just.. stupid.

</comment><comment><USER15> It sounds like Milner has a conscience.

But the conduct of his managers when faced with the issue is troubling.

OK, so if wardriving (sniffing wireless networks other than your own) is legally in the clear, then should everyone start collecting data? If Google does it, then it must be a business opportunity, right? As long as people don't do anything illegal with the data[1], what's the harm?

This is why paying serious attention to encryption, and making it easy for people to use, might not be the mere fodder of "tinfoil hats" that some portray it to be. Maybe it's just plain common sense.

1\. How many of you read about the TJ Maxx case?

<comment><USER16> > This is why paying serious attention to encryption, and making it easy for > people to use, might not be the mere fodder of "tinfoil hats" that some > portray it to be.

Setting up WPA2 on your Wifi router is not very hard. Most routers now will do it at the push of a button. (I personally don't do it that way, I set everything up by hand through the router's web interface, but that's because I'm a techie.)</comment></comment>


'Stereotype Threat': Why Women Quit Science Jobs - <AUTHOR> http://www.npr.org/2012/07/12/156664337/stereotype-threat-why-women-quit-science-jobs <comment><USER2> I imagine this article being read in 100 years like we might read about black bile causing melancholy.

<comment><USER3> Why? I think this is a real psychological effect. Humans are not Vulcans. I think a similar psychological effect also works with minorities. Or really anyone about whom stereotypes exist.

This is interesting in terms of game theory. Where the very existence of a stereotype makes ending that stereotype harder.

</comment></comment><comment><USER4> Yet we've heard recently that those who're reminded how good they are at a subject perform less effectively ... wouldn't that be an opposite result for men who were aware of a gender based performance stereotype?

Also, I'm hoping they looked at situations with opposite performance stereotypes to confirm their results?</comment>


Web startups are just attempts at social engineering - <AUTHOR> http://www.startuptrekking.com/2010/09/web-startups-are-just-attempts-at.html

<comment><AUTHOR> This is related to the recent "Why the Fuck?" thread (<http://news.ycombinator.com/item?id=5088815>) and is something I wrote a couple of years ago</comment>


Smalltalk, just take a look - <AUTHOR> https://hackernoon.com/smalltalk-just-take-a-look-d0a0052ed5e4 <comment><USER2> For anyone interesting in trying out, here are the open source Smalltalk implementations available:

Pharo [1] Squeak [2] Cuis [3]

Note that these implementations have FFI and which you can modify / build shared libraries for. The creator of Cuis, who works at a satellite company, did just that by integrating OpenCL because he wanted the liveliness of Smalltalk. Cuis is also the most minimal of the lot.

[1] [http://www.pharo.org](http://www.pharo.org) [2] [http://www.squeak.org](http://www.squeak.org) [3] [https://github.com/Cuis- Smalltalk/Cuis-Smalltalk-Dev](https://github.com/Cuis-Smalltalk/Cuis- Smalltalk-Dev)</comment>


Farming hard drives: 2 years and $1M later - <AUTHOR> http://blog.backblaze.com/2013/11/26/farming-hard-drives-2-years-and-1m-later/ <comment><USER4> Why do they need hard drives when they don't even back up data?

Seriously, you can find reports of this all over the internet. It's on reddit, twitter and elsewhere- these guys do not back files up properly, and their systems are so shoddy that they'll report they have files even when they don't.

My girlfriend brought her computer in for repair and they changed the hard drive. She specifically checked before she left that the backups were set, and the dashboard said they were. When we went to restore, however, none of the files could be.

We contacted support and they kept insisting we did something wrong. Finally they admitted that they had erased the files (yes, actually erased them). Due to a "bug" their dashboard reported we still had the files, even though they were gone.

They did everything they could to hide their mistake. They lied to people on twitter about what I was saying (since the only way to get real support was to hound their twitter account). They lied about all sorts of things. I had to take screen shots of everything and post it online before they'd even admit to screwing up. Even at that point their dashboard was still saying it had all of these files, when it in fact had nothing.

Backblaze is a completely untrustworthy company. If they spent half as much effort on backing up files as they do on marketing that may be a different case. Anyone who uses them for backups might as well not have backups at all.

<comment><USER5> Hey Tedivm! Yev from Backblaze here -> It's true that the data was removed from your account. After your trial lapsed and was not paid for, we removed the data after 7 days as is our standard practice. Unfortunately in your case the overview page did not update to show that you had no data associated with your computer, so when you purchased the license, there was no data on the machine that it attached too, so when you erased data from the machine while performing maintenance, you assumed the data was on Backblaze, when it was in fact purged from our system a few weeks prior.

We've since fixed that visual glitch (if you have a trial that expires and is not paid for, once the data has been deleted, that will be reflected on the overview page) AND we're currently working on some other notification systems to help people avoid losing data. As always, I apologize that you had this problem, but I'm happy to inform you that we've since fixed that visual bug on our end.

<comment><USER4> So everyone else is clear, we did pay for the account after the trial ended. So we were paying customers, this wasn't an incident where a trial expired and we tried to get our data back.

Also, to be clear, the customer service experience was horrible. The support team repeatedly ignored what we were saying and send us form responses back which were just not relevant. The twitter team claimed we misread the retention policy. The whole thing was kind of ridiculous.

Even more to the point, and the main reason why I tell this story, is that people should know how this company operates. People should know that they don't have backups of backups, that there is no margin for error should your account lapse, and that their systems in general are not robustly programmed in the sense that there are ways that your data could be lost and you wouldn't know it.

From my perspective Backblaze only has one thing going for it over competing companies, and that's it's marketing department.

<comment><USER6> If I understand it right, you purchased a license after your trial expired and the backup was purged. And you didn't backup anything new upon getting the new license. So the GUI status bug was a real and unfortunate bug on Backblaze's side, but it was only a GUI bug and in fact they didn't lose your data in any unexpected way. This situation seems to be about 2/3rds BackBlaze's fault and 1/3rd your fault.

You also have bonus fault on your side for not testing the BackBlaze backup. Never ever trust a backup, even if it's right in front of you on your own disk, unless you can successfully restore it.

<comment><USER4> Yup, there were serious flaws in our backup plans here. In my defense, we hadn't been dating long enough at the time for me to really get things set and we were on our way to burning man ;-)

I am not claiming to be without fault, but I do think that Backblaze is on the lower end of the reliability spectrum when it comes companies that do this, which is why I bring up these issues. I've switched over to Crashplan and have external harddrives doing local backups.

I should also point out that this isn't the only issue that's come up with this company. Apparently their client side GUI is also not completely truthful about when it's backed things up, and will claim to have backed up files when it's really only backed up the indexes. It then spreads the real backups out in order to increase performance. Although this is a cool feature, it's one people should be aware of and not hidden. I have less knowledge of this though, as I only saw it in a few blog posts (google around instead of taking my word for it).

<comment><USER7> "Lower end of spectrum" \- of precisely two companies that I know of, CrashPlan and BackBlaze, that do "Whole Disk Backups at a fixed rate".

I'm a big fan of using a combination of Arq (Love It), Dropbox, BackBlaze, and SuperDuper, but - crashplan/backblaze are both pretty similar, and both have their pros/cons, and, likewise, hundreds of anecdotes from the user community of each.

For what it's worth, I'm in my fifth year of Backblaze, and I've always been able to recover all my files - but I've also kept my subscription paid up.

And, I don't believe I've ever seen your "real backups happening spread over" claim - I am frequently in the field, so I am _very_ aware of exactly when backblaze is sending data (frequently over my iPhone using Roaming Data) - <USER2> when backblaze says it's done backing up - that's pretty much exactly when data is no longer sent out.

<comment><USER4> Yev, the same dev who is commenting here, has admitted to the claims about back up times. You should read through this reddit thread, it shows that Backblaze does not do things as you'd think-

[http://www.reddit.com/r/YouShouldKnow/comments/1q14ca/ysk_th...](http://www.reddit.com/r/YouShouldKnow/comments/1q14ca/ysk_that_backblaze_may_not_be_backing_up_your/)

> If you boot up real quickly and want to do a scan one thing you can do is > open up the control panel and hit "Alt" \+ "Restore Options", we'll do a > small-file rescan immediately and schedule a large-file scan.

>Small is files under 30MB or so, it's a quick index, the larger files take a bit longer and we try to spread those out over the course of a few hours so as not to be too heavy on your system.

<comment><USER5> Yes, that's accurate. The program was designed to be light on systems so we are not constantly scanning and indexing your hard drives. In our best practices we tell people to leave their machines on for a few hours. We cannot run when the machine is off, so if the machine is constantly being turned on and off it'll disrupt our scans. The "Alt + Restore Options" method forces a scan, but you'll still need the machine to be turned on/connected to the internet in order for us to transmit the data. Just pressing "backup now" will start pushing the data that has already been queued for upload during a previous scan (if there was anything left over), but will not necessarily initiate a scan (unless there was nothing left over from the previous uploads).

<comment><USER4> So, to be perfectly clear, it is more than possible for you to sync the photos from your computer, hit the "backup now" button, have Backblaze respond back that a backup is complete, and then find out that all of those photos you thought you had backed up were not there.

All of this to make it look like your application is performing better than it actually is.

On top of that you keep doing that infuriating thing where you ignore what people are saying and respond as if they're idiots or by trying to deflect the issue. No one expects you to run when the computer is off, and pretending like that's what's being said is downright insulting. What they do expect is that hitting the "backup now" button, and having your application respond that their files have been backed up, should actually mean that the files are backed up.

The more you talk the more you try to twist this around into something it's not- a visual glitch, misreading your policies, or people turning their computers off when they shouldn't- rather than trying to understand why your customers are upset and actually deal with it. That's why this issue has grown, and why we keep having these conversations- you guys have zero concept of responsibility and would rather insult your customers than actually make a viable product.

<comment><USER5> No, we should display the last time you were fully backed up in the console. If you turn on your computer, move photos to it, and turn it off, even if you press "backup now" likely they won't be included in the scan. In our best practices and introductory email we say that if you want at least a "daily" backup, you have to let us run for at least 3 hours so that we can fully index your new/changed files and back them up. If you added a lot of files it'll take longer, if you only add a few, it can be pretty quick.

If your complaint is that the Backup Now button does not do a full-system scan, that is totally valid and after the reddit thread our engineering folks are looking in to changing the "backup now" button's behavior. One of the reasons it doesn't do that now is because a full-scan will hang your system, where uploading the remaining files is very light and unnoticeable. Like with any functionality decision it's tough to say what is the best answer, hanging someone's system each time they press a button, or kicking off a remaining files upload and gradually scanning the drive over the next hour.

I sincerely apologize if I am not communicating well though, I am not trying to talk down or assume you or anyone else is an idiot in any way. I've tried to address everything that you bring up on here and on reddit. As far as expecting Backblaze to run when the computer is turned off, you'd be surprised at how many of our support tickets ask, "If my computer is off, are you still working?" so we do see that quite a bit. I don't bring it up to try and dig at anybody. I also think we've taken responsibility for the bug that mislead you in to believing you had data on our system when it had already been removed. Once we realized what had happened in your case we offered a refund and have since fixed it so that it does not happen to anyone else.

We're in the business of backing up data. When customers lose data, whether it's something they did or something that occurred on our end, we feel badly about it and try to make it right. We do have a viable product. We've restored over 4 billion files for the customers that have accounts with us. We take it very seriously.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> I've been trying to reduce my data hoarding. I had a larger back-up hard drive go down and take a lot of data with it. I felt sick knowing I potentially lost years of funny images, videos, music, writings, etc.

What I found is I had back ups of back ups of the things I actually cared about and the things I lost I hadn't truly looked at in a long time. A lot of the music I hadn't listened to in years.

I have found that I can fit on small SSD drives and as a result of the things I do wish I hadn't lost, I'm just keeping up with the practice of keeping multiple back ups.

TL:DR: I have made an effort to keep a smaller data footprint instead of hoarding.

<comment><USER9> If the only instance of a thing is on your "back-up" drive then it isn't a back-up.

<comment><USER10> Data that isn't backed up (as in a second copy) doesn't really exist.

Likewise: data that isn't backed up off site isn't really backed up.

<comment><USER11> > Data that isn't backed up (as in a second copy) doesn't really exist.

An IMO more realistic statement: Data that is not backed up is implicitly classified as "nice to have" \- in many cases a perfectly rational choice, if done consciously.

> Likewise: data that isn't backed up off site isn't really backed up.

You may as well argue that no data is ever "really" backed up because there is always the possibility of simultaneous failure. It's always a choice of how much risk you're willing to accept vs. how much money and effort it would cost to reduce it further.

<comment><USER10> Yep. 1 copy of data is not enough, because it can be lost. Now, assuming that N copies of data is not enough, it's easy to see that N+1 copies is also not enough, because that extra copy can be lost, thus reducing to N copies, which is not enough.

This proves by induction that no matter how many backups you have, it's not enough.

</comment></comment></comment></comment></comment><comment><USER12> The original "Drive Farming" link is at least as interesting as the new article:

[http://blog.backblaze.com/2012/10/09/backblaze_drive_farming...](http://blog.backblaze.com/2012/10/09/backblaze_drive_farming/)

<comment><USER13> And here's the discussion of that story:

[https://news.ycombinator.com/item?id=4631027](https://news.ycombinator.com/item?id=4631027)

I browsed the comments with two things in mind: was anyone uncomfortable with shucking and farming on this scale, and were the drives of equivalent quality?

Regarding the latter, user rsync of rsync.net claimed that "the drives inside these enclosures are the worst spec'd, highest failure rate drives." User budmang of Backblaze, however, said, "The drives were literally the same inside the external enclosures." User atYevP of Backblaze posted a similar comment.

<comment><USER14> High failure rates of external drives can also be attributed to the USB controllers - they're better than their IDE predecessors (which failed a lot, sometimes taking the drives with them - only low level utilities could revive them). Otherwise, I don't see how or why the hard drive manufacturers would bother to send the worst batches (which aren't _that_ much worse) to external hard drive assemblers.

<comment><USER15> I repair computers for friends and families and the majority of the time I get a broken external to look at, it's the enclosure that's at fault and the drive is absolutely fine.

I encourage people to buy their enclosures and drives separately - it's a little more expensive, but this way you can RMA just the enclosure if it fails. With an all-in-one, you either void your warranty to get the data back, or ship the whole unit back for a replacement.

<comment><USER14> That's an interesting point, since I have a Samsung hard drive I took OUT of an enclosure and out of curiosity I checked the warranty on it - it's a 3 year warranty that is still active and the web page said I could initiate a return.

So, technically, I could RMA the drive myself if it fails... or would they notice it and not process the return?

</comment></comment></comment></comment></comment><comment><USER16> Sadly I read this as a precursor to shortages in water, fish and fruits/vegetables that are coming. The populations of monarch butterflies, bees and other insects are dropping like a stone year over year. The fish near asia's costs are disappearing. The culprits include pesticides, overfarming, overfishing, and pollution. I am not even as worried about the garbage anymore -- scandinavian countries have figured out ways to burn the garbage and "contain" the waste gases. However, the main problem of capitalism is the numerous externalities that are exploited until it's too late to fix. It's not just global warming. It's raping the ecosystems for profit. No futures are going to save us from that. We need sustainable practices but I really don't see a way that'll happen at this point.

<comment><USER17> Too fatalistic. In 19th century they predicted that Earth can't sustain population over 1 billion. In comes oil and agricultural revolution. I agree what we are doing with ecosystem in general is a crime, but it won't kill us. At least not anytime soon.

<comment><USER16> Just because advances were able to stave off collapse and allow billions of people to consume resources at an ever greater rate does not mean that the same can be done with the effects of overfarming, overfishing, and pollution. The fallacy is thinking "since we did it once we can always continue doing it." The probability exponentially rises that you can't, since this is basically the same as the probability of always succeeding in time to avert a disaster.

Actually garbage / messing up ecosystems is a different type of problem than obtaining resources. It is somewhat related to the second law of thermodynamics. Once you introduce so much entropy into a system the genie is out of the bottle.of course this isn't exactly thermodynamics but I can say it this way -- putting humpty dumpty together again is different than figuring out how to make humpty give you more food for decades.

<comment><USER17> I really agree with you on the point that we are doing all those things. I am a fan of the [http://physics.ucsd.edu/do-the-math/post- index/](http://physics.ucsd.edu/do-the-math/post-index/) blog.

What am I arguing for - that while we do consume more resources, progress is exponential as well. For example if we seriously begin to run out of fossil fuels, amount of effort that is put into other energy sources would increase by order of magnitude and solution will be found. It might not be right away, it might not be silver bullet, but when push comes to shove and there are no alternatives, human ingenuity is frightening.

In the end progress stacks multiplicatively - you add great battery technology to great energy generation technology and you got more than sum of the parts. So more people there are doing research - faster it goes _.

_ This point is only true from certain point of view. Consider genius starting from nill, how far could he take technology within his lifetime? Electricity, probably. Microprocessors and genetics, probably not. Now consider million of people working, each discovery builds on an earlier one, each tool makes other people researching more efficient or enables them to make new discoveries. Now think about multiple generations of people building discoveries on top of each other while sharing information.

</comment></comment></comment></comment><comment><USER18> Why didn't they touch on why the Hitachi drives weren't the same quality as the other brands with this statement:

"Switching hard drive manufacturers from Hitachi to Seagate and Western Digital. The assumption here is that all drives are the same and as we've learned that's not really the case."

Where is the footnote. Why aren't they the same?

<comment><USER19> In the parent article they said they switched because of cost: "When Hitachi drives got too expensive we switched to Seagate and Western Digital. When internal drives got too expensive we turned to external drives. When hard drive availability became an issue we turned to Drive Farming."

From their article on v2 they said: "The Western Digital and Seagate equivalents we tested saw much higher rates of popping out of RAID arrays and drive failure. Even the Western Digital Enterprise Hard Drives had the same high failure rates. The Hitachi drives, on the other hand, perform wonderfully." [http://blog.backblaze.com/2011/07/20/petabytes-on-a- budget-v...](http://blog.backblaze.com/2011/07/20/petabytes-on-a- budget-v2-0revealing-more-secrets/)

There are other articles where they mention that they do testing and some brands perform better than others.

<comment><USER20> We used to make PODs like this at work for netbackup storage.. started with WD drives ~128 WD drives, at about two years we were at ~8 WD drives still working. At least once a week we had people in there swapping drives. I have yet to see a WD not fail.

<comment><USER21> On the other hand, I've got a WD Caviar Black that's been running strong for more than five years. None of my WD drives have ever failed (knock on wood!). I don't run 100+ drives though, so I'm probably just hiding in the statistical margin.

<comment><USER22> Also your thermal environment is going to be quite different, which could be a significant factor. How often do you power-cycle that drive BTW? I've had more drives go bad on power-up (just failing to spin up and move the heads correctly) than through surface errors and such.

<comment><USER21> I try not to restart that machine more than once a month.

<comment><USER22> That si better than running the drives 24/7 - if one is going to go then that way it is more likely to go alone. If every in an array (anything I even half way care about get s the RAID1 or RAID5 treatment) has been spinning for years then a power cycle can be quite scary. Restarting once month or so increases the chance that if they fail they'll not all fail at the same time.

Having said that, my main little server at home hasn't had a power cycle for somewhat longer than a month as there have been no kernel updates or that like that have required it...

</comment></comment></comment></comment></comment></comment></comment><comment><USER23> The extrapolations in the article were pretty optimistic to begin with.

I mean, when you look at the dotted trendline in this graph: [http://blog.backblaze.com/wp-content/uploads/2013/11/blog- co...](http://blog.backblaze.com/wp-content/uploads/2013/11/blog-cost- trend-1.jpg) you'd think by 2016 drives would cost $0 per gigabyte and by 2020, drive companies would be paying you to take drives off them.

<comment><USER24> Well, at that point, if the current graph charts cost per gigabyte and approaches zero, then you would change the scale of the graph to show costs in terms of terabytes and eventually petabytes.

Ultimately, after traversing enough benchmarks, the scale would go logarithmic, and measurements would have to take orders of magnitude into account.

<comment><USER25> Indeed, if you click through to the article the historical data is drawn from, you will see it plotted on a semilog plot. The trend is well described by an exponential function.

</comment></comment></comment><comment><USER26> Aren't they big enough at this point to cut deals directly with the vendors? 10,000 drives at a discount, that sort of thing.

<comment><USER13> In the previous story, user brianwski of Backblaze said, "We have been told that until we buy blocks of 10,000 drives, we cannot deal directly with the drive manufacturers. We're not there yet."

[https://news.ycombinator.com/item?id=4633213](https://news.ycombinator.com/item?id=4633213)

<comment><USER27> Cool.

I know a few Backblaze people are reading/posting in these comments, so I'll ask here:

Do you guys know the approximate price of when you _can_ purchase them direct in 10,000 blocks?

<comment><USER28> I'm not at Backblaze, but when I was at Google I did know what Google was paying for drives direct from the manufacturer and it was not as much lower than the distributor price as you might expect. Margins are pretty thin in the disk drive business.

</comment></comment></comment></comment><comment><USER7> I have a colleague who works in the industry and purchases a few petabytes of spinning disk a quarter. I love reading the backblaze blogs, and constantly forward them to him. He thinks the backblaze guys are pretty crazy with their "shucking strategy", finds it hard to believe seagate/WD guys will honor warranties on those "shucked" drives, but if they will, all the more power to backblaze.

He does note that his costs per drive didn't go up dramatically after the Thailand floods, and they were pretty close to what backblaze was playing for their "shucked drives" \- though, they didn't go _down_ signficantly either.

I'm guessing Backblaze is still at the volume levels where they can't negotiate directly with Drive Vendors - I'm not sure where that level is - <USER3> Petabbytes/Quarter? 100 Petabytes/quarter?

Regardless, I continue to be very impressed by how backblaze manages to circumvent what others might consider insurmountable hurdles. Truly hacking the system in a good way.

<comment><USER5> Yev w/ Backblaze here -> Yea unfortunately when we shuck the drives the warranties are voided, but we add that factor in to the costs that we are willing to pay for the hard drives. Truth was, it was cheaper for us to buy the externals and shuck them, even with the voided warranty, than pay the larger fees for internal drives.

Part of the is price sensitivity. External drives are geared towards consumers, who simply will not buy the drives if they go up too much in cost. The internal drives we were buying are targeted at business' who have a choice to make, either buy at the higher price, go out of business, or get creative. We try to go the go creative route because we really don't want to go out of business, and paying the higher prices for drives, especially in this case, would have gotten us there.

As for the negotiation, BrianWski mentioned it in another reply on here, but we keep getting weird answers from vendors and manufacturers. Typically they want a minimum order of about 10,000 hard drives in order to work directly with them and we're simply not willing to buy that many at a time, especially since normally the price goes down monthly, meaning carrying inventory is leaving money on the table.

Love that you enjoy your blog posts, stay tuned ;-)

</comment></comment><comment><USER29> I personally (for home use) held off buying drives for two years, found out I did not really need so much space and discovered SSDs would do the trick.

<comment><USER21> Same here. I bought my last 2 TB for $80 shortly before the crisis, and was able to scrape by at 90% capacity until 4 TB drives became cheap enough.

</comment></comment><comment><USER30> > When the Drive Crisis started, industry pundits estimated that the hard > drive market would take anywhere from 3 months to 1 year to recover. No one > guessed two years. Was the delay simply an issue in rebuilding and/or > relocating the manufacturing and assembly facilities?

This is a typical response in all markets that are commodities (or approaching commodity status). Once makers have had an excuse to move prices up, they want to hold onto the extra profit for as long as they can, so prices are fast to rise, but slow to fall.

The most direct effect can be witnessed in gasoline prices. Notice how a news report that a camel farted in the desert of Saudi Arabia results in the price of gasoline at your local station rising by 25 cents a gallon (or more) nearly instantly. Even though the gas in the underground tanks was the same gas (at the same cost to the station) that it was 15 minutes ago.

But then, it takes weeks before that 25 cent increase goes away when no more camel farts occur over those same weeks. Same effect here, just with hard drives rather than gasoline.

<comment><USER31> Six months after the Thailand drive crisis, drive prices "peaked" at about double their earlier prices and began falling. Just about then, an article came out featuring a bunch of panic inducing quotes from a Seagate VP loudly claiming it might be a year before they could get enough drives made, shortages would be terrible. It felt so disgusting to me, none of it was true, the VP was just trying to encourage pre-buying and drive hoarding to artificially pump sales.

</comment></comment><comment><USER32> I wonder why external drives are cheaper than internal drives. One would think that they would be more expensive, since the cost to the manufacturer of an external drive includes not only the drive itself but the adapter contained in the enclosure, the enclosure itself, cables, possibly a power supply, more elaborate packaging and the additional labor required to assemble the external drive. I assume that this consumer cost differential arises from differences in the markets for internal and external drives, but naively it seems like the internal drive market would be more price-sensitive and thus lower margin.

Does anyone know why external drives are typically cheaper than internal drives?

<comment><USER33> Maybe, with the advent of laptops becoming more popular than desktops, there is just a bigger market for them. Cost is usually due to demand constraints.

<comment><USER34> I doubt it. Server farms consume an unreal quantity of drives. Desktops consume 1 drive per person.

<comment><USER35> Server farms store data for actual people. So sit back and think about how much unique data you have on servers somewhere. Add it up and I dont't think the total data storage at all data centers everywhere in the US add up to more that 20GB per person in the US as we are just not that interesting even at 5x redundancy your still at 1/30th a drive per person. Don't forget at 300 million people that would still be 10 Million drives which is a lot of servers.

PS: The NSA could be recording every phone call ever made and it's still not all that much actual data.

<comment><USER34> The dropbox model is not the only place for server farms.

How about render farms & their associated data stores. How about Google Maps & the associated scads of satellite imagery.

You could be right, of course, but I'm not sure we can say with certainty.

<comment><USER35> Google Maps stores a 'tiny' amount of data per person. The surface area of the earth is a little less than 200million square miles or a little less than 2/3 of a square mile a person. at 1 foot accuracy that would only be two 10 megapixel pictures per person not that we actually have maps anywhere near that good of most small towns let alone the oceans or arctic. The only thing that really spit's out data is raw scientific data, but even Nasa just dumps well over 99% of what they collect so it's more of a short term problem.

</comment></comment></comment></comment></comment></comment><comment><USER34> I don't know a whole lot about backblaze; as they are a backup service, is there a good reason why they couldn't ramp up usage of tape drives? Perhaps keeping recently uploaded backups on-disk, and shuttling older ones to tape?

<comment><USER36> Their backups are progressive, so a single disk image might be spread across 100 tapes. A restore would be a nightmare. Tapes really only work if you are doing a full image each time.

<comment><USER37> or a program of partial backups interleaved with full backups, say 3-to-1. You balance a reasonable restore time with a reasonable storage requirement.

</comment></comment></comment><comment><USER38> This seems like something that should be part of their business instead of waiting for catastrophes.

<comment><USER21> Since the price of storage is consistently declining, buying any more than necessary is wasteful over the long term.

<comment><USER39> yeah, but it seems like you'd give yourself more than a few weeks of runway. Don't have to buy 5 years worth, but perhaps 3-6 months would be reasonable. The peace of mind has got to be worth the ever so slight cost increase.

</comment></comment></comment><comment><USER40> Still... no linux.</comment>


A Change to Trello Teams - <AUTHOR> https://blog.trello.com/trello-teams-at-work <comment><USER2> I guess the fears about Atlassian's acquisition of Trello were real.

By limiting free teams to ten boards, they've effectively knee-capped it as a viable business tool (card-based task management without the bloat of Jira).

What's especially insidious is the short notice and high price tag of Business Class. Are there any good alternatives out there?</comment>


Human exoskeletons: full metal jacket - <AUTHOR> http://www.economist.com/news/science-and-technology/21684117-battlefield-factory-floor-orthopaedic-clinic-artificial <comment><USER2> Exoskeletons are covered in this Breakthrough episode:

[http://channel.nationalgeographic.com/breakthrough- series/vi...](http://channel.nationalgeographic.com/breakthrough- series/videos/paul-giamatti-on-human-engineering/)

</comment><comment><USER3> Can't wait for somebody to come along with an ingenious solution to the power problem. I feel like there's a solution in front of us that hasn't been 'discovered' yet.

<comment><USER4> I suspect the biggest part of that will be reducing energy requirements by using springs, or equivalents.

<comment><USER5> Semipowered and unpowered enhancing prosthesis are very hard to actually implement. Here's an ARS article about a lab actually succeeding: [http://arstechnica.com/science/2015/04/an-unpowered- exoskele...](http://arstechnica.com/science/2015/04/an-unpowered-exoskeleton- decreases-the-energy-required-for-walking/)

<comment><USER6> Allot of the planned exosuits are semi-powered they allow you to carry more weight buy actually distributing it to the ground through the boots rather than it having to hang on your shoulders and lower back.

This is effectively lowering the waist suspension belt which is built into every high weight backpack. Effective weight distribution is just as important as core strength, a good backpack system with 60kg in it can feel lighter than a crappy duffle bag across your shoulder.

The system it self can be recharged using the wearer motion to give the additional speed and strength when needed.

Most combat engagements aren't long the actually power output of the suit will be concentrated into a few short bursts rather than long active periods (one of the reasons many prototypes us a capacitor bank rather than a battery).

A suit designed like some of the supercap hybrids (Mazda 6 for example) isn't that far away and while there are quite a few engineering challenges current power density isn't a deal breaker. So there's a good chance that the first deployed suits will be very video game like and will have a cooldown, you need to jump over a 10ft wall? no problem.. you need to break a door in? easy.. just need to wait a bit before one stunt to another, that's doable, and not a deal breaker at one any level.

</comment></comment></comment></comment><comment><USER7> Exoskeletons is a cool concept, but given all the problems with supplying power, why would you bother if it's easier to put soldiers in an armored vehicle?

<comment><USER8> Vehicles can't enter most buildings and climb stairs.</comment></comment>


De-Risking Your Startup (SaaStr 2017 Talk) - <AUTHOR> https://codingvc.com/saastr-2017 <comment><USER2> Great sharing - the slides are really insightful on some of the strategic aspects of company building. Even having been in the startup world for quite some time - I hadn't seen some of these points articulated so clearly.

<comment><AUTHOR> Thanks! If you have any questions I'm happy to try to answer them.</comment></comment>