Canon's cloud platform has lost users' files and can't restore them - <AUTHOR> https://www.digitalcameraworld.com/news/canon-websites-held-to-ransom-by-hackers <comment><USER2> This sounds to me like the classic "replication is not backups" situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were "big" - that is actually the one thing they should have backed up because all of the smaller files can be regenerated from the raw ones. I would not be surprised if all of this was running under someone's desk.

<comment><USER3> What's the difference between replication and backups? Is the distinction that backups must be stored on separate infrastructure, whereas replication might still be 1 or 2 points of failure?

<comment><USER4> Replication is about having data level redundancy to protect from drive failure. Backups are about having point in time snapshots of the system state, and about having them tiered from a location perspective. The 3-2-1[1] principle says to have 3 total copies, 2 of which are local but on different devices, and 1 of which is offsite. This gives you tiers of recoverability.

It's important from a backup perspective that it's point in time as well, otherwise as soon as you get ransomware that encrypts your file you now have replicated those changes everywhere.

[1] [https://www.backblaze.com/blog/the-3-2-1-backup- strategy/](https://www.backblaze.com/blog/the-3-2-1-backup-strategy/)

<comment><USER5> I would add to that, that the offsite one should also be off _line_.

<comment><USER6> That's a lot harder to pull off. What methods do you use to accomplish this?

<comment><USER7> S3 bucket with object lock is as offline and convenient as it gets

<comment><USER8> I don't know much about S3, but isn't S3, by definition, online, not offline?

<comment><USER2> I've always wondered if Amazon backs up S3. I don't think they explicitly say but I get the impression that it is the user's responsibility to replicate to a second region to guard against data loss so I am guessing not. Object Lock wouldn't protect against an S3 failure.

<comment><USER9> They say it's designed to provide significant durability : [https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurabili...](https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html) (99.999999999% durable over a year per object, and able to sustain data loss in 2 facilities).

Given the peeks that amazon has provided into the scale of S3, I don't know if you CAN 'back it up'.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> This is all because someone thought it would be "easy" to add a stream of recurring revenue for cloud photo storage but didn't take the time to design the service from a technical perspective to be resilient enough to not crash and burn.

<comment><USER11> This can be just calculated risk.

Incredible damage for some users can be negligible damage to the company. In this case just 10TB of data was lost, so maybe thousands of users for the long term storage option.

Losing customer data has potential for big reputation or brand damage, but surprisingly often the damage is relatively small. Putting in too much effort when potential damage to business is minimal may not be worth of the cost.

<comment><USER12> This comment reads as "This is normal; nothing to see here move along".

Which is more of an indictment on the state of things than on the comment itself.

If you can't protect people's shit, don't offer to hold onto it for money!

<comment><USER13> No, that comment sounds like the insurance job described in Fight Club[1]

> Narrator: A new car built by my company leaves somewhere traveling at 60 > mph. The rear differential locks up. The car crashes and burns with everyone > trapped inside. Now, should we initiate a recall? Take the number of > vehicles in the field, A, multiply by the probable rate of failure, B, > multiply by the average out-of-court settlement, C. A times B times C equals > X. If X is less than the cost of a recall, we don't do one.

[1] Video link: [https://www.youtube.com/watch?v=SiB8GVMNJkE](https://www.youtube.com/watch?v=SiB8GVMNJkE)

<comment><USER14> Or that actually happened, in the known Ford Pinto Memo "affair":

[https://en.wikipedia.org/wiki/Ford_Pinto#Cost–benefit_analys...](https://en.wikipedia.org/wiki/Ford_Pinto#Cost–benefit_analysis,_the_Pinto_Memo)

<comment><USER15> And a little further down the same page you linked....

[https://en.wikipedia.org/wiki/Ford_Pinto#Retrospective_safet...](https://en.wikipedia.org/wiki/Ford_Pinto#Retrospective_safety_analysis)

Turns out the Pinto was within the realm of normal for a car of that type and era and if you're building something normal it's hard to justify recalling it at great expense unless everyone else is recalling it to (in which case that would be normal).

<comment><USER14> Yep, but that is an analysis from a Legal viewpoint.

What - I believe - made the Pinto case memorable was not that in a given subset of accidents it was (or was not) slightly more dangerous, it is the fact that the matter was looked into by the manufacturers and that it was waved off on a mere cost/benefit basis (with a supposed cost for increased security per car of US$ 11).

Mind you this is all in all normal, any safety norm or related technical specifications is (or should be) based on the "reasonable costs for society to obtain a reasonable increase in safety".

</comment></comment></comment></comment></comment></comment></comment><comment><USER16> Japanese camera companies are really bad at cloud software - Sony Playmemories probably the worst offender. Also the in-camera "mini-apps" tend to be terrible.

They make good hardware but horrible software. They should really buy a computational photography company that also knows how to make good web and native apps, like [https://skylum.com/](https://skylum.com/)

<comment><USER17> We were actually on NTT (Nipon Telegraph and Telephone) cloud for many many years up until very recently. At the time we started using them AWS, Azure, Google Cloud, etc. did not exist. But over the past few years the quality of service has dropped considerably. My guess is that their best engineers were being poached by AWS, Azure, etc. and the leftovers couldn't keep it up.

</comment></comment><comment><USER18> I take nightly images of my computer's primary drive and replicate them offsite, with alerting if it fails and automated scheduled backup restore testing.

It blows my mind to think that some random schmuck like me has better backup procedures than these multibillion dollar global corporations.

Companies that don't invest in their IT end up paying for it tenfold down the track.

<comment><USER19> On topic topic, can I get HN's advice on my backup strategy?

All important files on Synology Ds218+ NAS, which has 2x12TB helium HDDs in mirror.

Daily HyperBackup to Google Drive. I test restoring files on a monthly basis.

Email alerts on failure.

Anything I can improve?

<comment><USER20> Google, as a company, is not reliable - I refuse to host anything critical on Google Drive (even G Suite) because there's zero recourse or accountability if a bug in Google's software or if Google arbitrarily decides to ban you for life from their services (e.g. due to misidentifying you from a dodgy YouTube user or repeated Google Play Store policy violations) then they will delete all of your personal and business accounts and you're SOL.

What you have is fine for short-term recovery - but I'd make sure you have a long-term and/or cold-storage option set up. It doesn't need to be anything particularly fancy: I'd get a rugged+ portable+external single-drive USB enclosure with a single 16TB drive and have the Synology do a backup of your most critical data onto that drive and store it in your bank's safety deposit box or better yet: leave it with a trusted friend who lives in rural Minnesota or similar.

It goes without saying to encrypt that drive as you won't have full custody of it: but use a simple, proven encryption scheme with a large ecosystem that you know that you'll be able to decrypt in 10-20 years' time.

For backup drives that I keep local to me, I refuse to encrypt them because (in my experience) the possibility of being unable to decrypt data in a desperate or urgent situation just makes me wince.

<comment><USER21> What do you make of comparable services from other providers, say, Amazon Glacier?

<comment><USER22> I've used glacier. For stuff you don't need to restore in a rush, it's cheap.

I did however have a recent health scare and it made me wonder how my non tech wife could possibly have restored the files as the interfaces are all heavy.

Not a factor I'd previously considered when assessing my backup/restore.

<comment><USER21> > For stuff you don't need to restore in a rush, it's cheap

That was my thinking. It seems a good fit as a last-resort backup. Low month- on-month storage costs, high retrieval costs. So we're essentially betting that we'll never retrieve the data. Which seems fine.

Also, it apparently has strong assurances against data-loss. Lots of nines. [0]

> how my non tech wife could possibly have restored the files as the > interfaces are all heavy

It's all web-API-based, right? Is there a a decent FOSS GUI to navigate it?

[0] [https://aws.amazon.com/glacier/features/](https://aws.amazon.com/glacier/features/)

<comment><USER22> If you retrieve the data slowly it's cheap. It's expensive if say you are a retailer who needs their database restored asap.

I use a Linux perl client!

<comment><USER21> I suppose the biggest risk is failure-to-pay, as with all cloud backups/storage. If I allow my payment card to expire, Amazon aren't obligated to continue to store my data. If I drop off the grid for an extended holiday, that could be a real risk.

To my knowledge, Amazon offer no means of prepaying.

<comment><USER20> > If I allow my payment card to expire, Amazon aren't obligated to continue to > store my data.

Business idea: cloud archive storage where you pay when you upload data and optionally pay a modest monthly fee for real-time access to stored data, but they'll guarantee to keep your data for you if you stop paying: you'll just need to pay to retrieve that data.

As the long-term archival data wouldn't need to be stored in a data-center: just a commodity tape-library box in a basement in a farm somewhere near a freeway I imagine it would be kinda cheap to run as a business. You could set- up a Foundation or other entity to ensure long-term continuity of operations and have it self-sufficient through an endowment. E.g. a $1m endowment would easily pay for something like this into perpetuity.

<comment><USER23> There was Permanent.org, three months ago: [https://news.ycombinator.com/item?id=22943620](https://news.ycombinator.com/item?id=22943620)

<comment><USER21> I agree with this comment on how that project completely fails to provide the necessary assurances of dependable longevity: [https://news.ycombinator.com/item?id=22944681](https://news.ycombinator.com/item?id=22944681)

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER17> "We will contact affected users shortly and offer..." I'm not sure how I expected this sentence to end, but it certainly wasn't anything as useless as "our deepest apologies".

</comment><comment><USER24> Lesson for those designing cold-storage solutions--design your system, as much as reasonable, to not support deletion operations. The reason why files get lost is usually due to software bugs, bad configurations, and operator errors. Design your system to protect against these things.

<comment><USER25> Both AWS S3 and Google cloud storage buckets have an option that makes it impossible to delete stored objects for some period of time. The option was added for some legal compliance reasons, but I find it useful as an extra safeguard that important service data is not accidentally or maliciously deleted.

<comment><USER26> I suppose that feature would be incompatible with user data in a GDPR world ??

<comment><USER27> Git is also incompatible with GDPR, you can't simply delete a file from all history.

<comment><USER28> Git is not really intended for personal data

<comment><USER29> Yeah, I don't really understand why you would like to stuff personal data into a git repository (unless we're strictly speaking about author data). It's really not the tool for it.

Now for those who decided that a blockchain is the perfect solution for storing personal data however...

</comment></comment></comment></comment></comment></comment><comment><USER30> So today I learnt:

1\. Canon don't have any backup procedures in place for their cloud platform. Any hacker now will be salivating at the idea of pulling a ransomware hack I imagine.

2\. Canon developers follow the 'test in production' methodology of continuous integration.

<comment><USER31> How do you know that? Gitlab had 5 backups and could restore none.

<comment><USER32> Is it a backup if it can't be used like one?

<comment><USER31> It is and is not at the same time until you try to restore it.

<comment><USER30> Quantum mechanics goes over my head

</comment></comment></comment></comment></comment><comment><USER33> There is one somewhat reassuring lesson here -- No matter the company, no matter the brand, no matter their worth: software development is wrought with challenges, and major mistakes for seemingly obvious things are made by everyone, all the time, at every level.

<comment><USER34> This is not one of them. Ensuring proper data loss protection is something both Amazon and Google are very good at. A company of Canons size could easily get the proper guidance to avoid it. But it would not have been as cheap as storing the files on your own servers. So this happened because Canon did not want to spend the money on doing it right with offline backups or append only storage in a cloud with enough nines.

<comment><USER35> Companies of Canon's size can be constrained for resources just as much as any other company. Canon is largely a legacy company at this point, with significantly declining revenue the last several years. [https://i.redd.it/lrlfen2x7sm31.jpg](https://i.redd.it/lrlfen2x7sm31.jpg)

<comment><USER34> Even more reason to spend a little bit more to make this right. It's not expensive to get a proper expert on the matter to figure things out. The last thing they want is bad publicity for things that could earn them recurring revenue.

<comment><USER33> I don't think they deliberately thought they were doing something inadequately, these problems are always obvious in hindsight. Issue is that eventually most companies will have this moment seeing through the lens of hindsight, the reasons are always different.

</comment></comment></comment></comment></comment><comment><USER36> Everyone should be looking for a backup strategy that involves at least a hot backup and a cold backup, with some sort of offline validation. However this is easier said than done (because now you've to deal with multiple file systems, decisions about cryptography, lack of interest in creating something that you hope you'll never need, and so on). I wish there was a service (and don't tell me about Backblaze because it's incomplete and doesn't work that well - from my own experience) that would do all these things for you in a reliable and trustworthy way (including giving you snapshots in physical media from time to time - delivering to different addresses, possibly in different jurisdictions, and so on).

<comment><USER37> I have a multi-dimensional backup strategy, which includes "cloudy" backups of my most important assets, using services like GitHub, local hourly NAS (Synology double-redundant RAID-like) incremental backups of my whole system, and running "hot" backups of my system and development volumes; automatically updated every four hours (or on-demand. It's a bootable CCC clone).

I seldom need to use anything more than my "hot" backups, but have occasionally needed to restore individual files from NAS.

Whether or not I have a backup is never even something I think about at all, which is a big weight off my shoulders.

But photos and videos are a different matter from code (my main assets). They require a _lot_ of space. It's easy for me to be smug about backups; but photographers have a much more intense set of assets.

<comment><USER38> A problem with digital media as well is that it encourages you to just save everything but it's easier that way. But as I'm discovering well into the digital era, I wish I had more curated photos and video--though that comes with its own problems.

<comment><USER39> I dread the day I decide to go through all my photos and organize them. It's one reason why I'm so reluctant to move off of google photos, it's sooo easy to find stuff and it groups photos together nicely.

But I really want to move my photos off of google.

</comment></comment></comment></comment><comment><USER40> From time to time, I have a perisistent feeling that iCloud might be misplacing my files. It's just a feeling, but I realized that it's quite challenging to be sure it's not happening.

<comment><USER41> It's not so hard. A few command line commands on a Mac. Store the md5 of each file, throw it into a JSON file. Load it all up in a dict. I do something similar every month. Gotta do something with those cores overnight :)

It's very brute force and inelegant, but also simple and quick enough.

<comment><USER40> One day, I'll do it. Honestly, I'm just afraid to find out the answer… :)

</comment></comment></comment><comment><USER42> wow this is bad. I know you should keep local backups too but if you shoot RAW then that quickly becomes cumbersome, and only using cloud backup looks very convenient.

If I were a professional photographer missing client material or someone who lost potentially irreplaceable memories I'd want a hell of a lot more then just "[Cannon's] deepest apologies".

<comment><USER43> The software and service might have been ... provided "as is", without warranty of any kind, expressed or implied, including but not limited to...

<comment><USER20> > provided "as is", without warranty of any kind, expressed or implied, > including but not limited to...

IANAL, but when a service is advertised as being ideal for storing your work or photos, isn't that directly implying that there is a warranty that their service is fit-for-purpose? You can't advertise something in big lettering and then countermand that in the small-print - so those magic words certainly don't shield the company from liability at all.

I understand that the "without warranty of any kind, expressed or implied"-line we see in software-licenses and EULAs is when software is distributed _without consideration_ (e.g. open-source software), but when there is consideration (i.e. people paying Canon to host their photos...) then there's a liability if Canon lost peoples' data - so I understand they will be sued for this if anyone lost anything of value. At the very least a 100% refund...

They only way I can see Canon getting out of this is if they had prominent warnings displayed throughout their service's UX advising their users that their service was not suitable for long-term storage of valuable data.

Again, IANAL - can anyone is is a lawyer chip-in?

<comment><USER44> Also NAL, but I would say it very much depends on the law in which-ever jurisdiction it gets tested in - which might be the one you are in, the one Canon are in, or somewhere else entirely.

Some legal concepts such as "fit for purpose" as defined in the UK's Consumer Rights Act (2015) certainly seem relevant here, but that states that companies should offer replacement or refund. Note the word refund, not recompence, is used: Canon may be required to repay you in full everything you paid for the service. All £0.00 of it.

 _> i.e. people paying Canon to host their photos_

Is this the case though? Several other comments have mentioned it being a free service. [https://image.canon/st/en/faq.html](https://image.canon/st/en/faq.html) states that too. Unless there is a non-free option too.

 _> but when a service is advertised as being ideal for storing your work or photos_

I'm not sure what it is advertised as, but that doesn't seem to be what the service is intended for.

Reading other parts of the FAQ it seems that the service is intended as a transfer agent, with the convenience of online storage being a useful side effect. Quoting the FAQ: _" Image.canon is designed to ease your imaging workflow – whether you are a professional, enthusiast, or casual user. Wirelessly connecting your camera to the service allows seamless forward of images not only to your computer and smartphone devices but ..."_. The implication I'm making being that they can just argue that their service is designed to move images around your devices and the user should have been backing them up from there.

Users might try to argue false advertising if "fit for purpose" doesn't fly because of the purpose being defined differently. But good luck funding such as case against the lawyers that Canon can afford. It would have to be a class-action or similar, unless some government body takes the issue up (which from the users PoV will effectively be the same, and the best they'll get is a small voucher for a few £ off future Canon offerings). That is what a lot of things like this boil down to: legally enforceable sometimes doesn't exactly mean legal, it sometimes means "can be enforced by having a better legal team than the little guy"!

</comment></comment></comment></comment><comment><USER45> I was wondering what those weird messages from [https://image.canon/](https://image.canon/) were about.

IIUC the idea of their new platform was to be ephemeral in any case. Just to give a pipeline to stream in to other accounts.

I had no idea it was offering long term storage. Better off with S3 Glacier for that!

</comment><comment><USER46> On-site and at least two off-site.

It sucks for their users who maybe don't know the golden rule above. I certainly don't blame non-tech folk who paid a known photography company to handle a complex problem on their behalf - they did the right thing in handing this responsibility to people who they believed could be trusted.

To the greater point - this is incredibly damning of Canon's cloud storage going forward. As others have said, the amount of data lost overall is not much in the greater scheme of things, but that's not the concern. What's worrying is that they were able to lose any data at all. How much redundancy do they have? How are permissions managed?

When I backup photos and videos at home I have a script to _chattr -i_ all the files independently as they're stored on top of the redundancy and backups. You need to protect your data from yourself, too.

</comment><comment><USER47> There is no cloud, it's just someone else's computer.

</comment><comment><USER48> They're paying for the negligence against software developers. Those two incidents come down to the design flaws after all. The CEOs of Canon Inc and Canon USA are too old to admit the fact that software is more powerful than what it used to be when the hardware was selling well simply owing to its pure performance and features and the software was just a driver. You would be surprised if I tell you how many skilled software engineers have left the companies in the last three years. What makes things worse is that the old men aren't aware of why they left.

</comment><comment><USER49> The cloud is just another backup option and you should have 3. That being said, you expect more from a paid service.

</comment><comment><USER50> I can't access the article, but there was a recent ransomware attack on Canon [1]. I wonder if this could be related to it.

[1] [https://news.ycombinator.com/item?id=24185734](https://news.ycombinator.com/item?id=24185734)

<comment><USER51> I think they claim that this was a coding error and not a ransomware case.

The timing is convenient though.

</comment></comment><comment><USER52> Ouch, I am guessing they are using some on-premise or none major cloud provider? I feel for them though, without the right resource or budget they are doing their best. I doubt they intentionally decided to not make a good product.

</comment><comment><USER53> Canon is no software company and doesn't have the right mindset (also see most other camera manufacturers mobile app reviews). Probably outsourced with a focus on having low development and operating costs.

</comment><comment><USER54> Anyone who relies on any software camera companies produce deserves to lose their data IMO. Just look at Canon DPP. Almost 20 years old, and it's still a primitive, unusable, steaming pile of shit.

</comment><comment><USER55> I'm redirected into a refresh loop with this url: [https://www.digitalcameraworld.com/cc.html](https://www.digitalcameraworld.com/cc.html)

</comment><comment><USER56> I think people would be surprised at the amount of backup or file storage services that keep everything in one data center. Your files might be redundant across machines but it's not like they have a second data center somewhere with another copy of all your data, and they certainly don't backup everything up to tape or something.

</comment><comment><USER57> The camera companies are so bad at software that the software companies are building better cameras than them... in the few millimeters thickness of a phone! I salivate for the day that Google or Apple buys Nikon or Leica or whatever and shoves all of that computational photography goodness under a 24x36mm camera sensor.

<comment><USER20> As of 2019, Apple sources their iPhone camera sensor components from Sony and the camera lenses from Largan Precision (and other companies). The software that Apple contributes is important - yes (after-all, this is software- defined-photography now, with Portrait Mode and that fancy but fake depth-of- field stuff in the single-lens iPhones) - but let's not pretend Apple is "a camera company".

<comment><USER58> I expect Apple sold more camera than Canon last year.

</comment></comment></comment><comment> Funes- Saving your shit on "the cloud": not even once.

<comment><USER59> I store my data on the nextcloud server sitting in the same room. I guess for added safety I could also set up automated encrypted backups to backblaze or something.

</comment></comment><comment><USER60> With spinning rust so cheap nowadays - anything could be backed up easily both in the cloud and locally.

</comment><comment><USER61> No backups? Wut?

</comment><comment><USER62> People call me crazy for having local backups, cloud backups, and offline backups for my critical assets.

<comment><USER63> 3-2-1 backup is not crazy and also is an IT standard.

</comment></comment><comment><USER64> I'm utterly shocked at how many people assume that "The Cloud" is both run by people infinitely more intelligent and more competent than they are and that everything there is backed up.

What a crock of shit.

</comment><comment><USER65> Has GDPR compliance gone too far?

<comment><USER66> No.

Losing data is just as bad from a GDPR perspective.

<comment><USER65> Oh, really? Can you actually be fined for destroying disks containing customer information?

<comment><USER67> Yes! Accidental loss is a data breach, see page 7ff of [http://ec.europa.eu/newsroom/document.cfm?doc_id=47741](http://ec.europa.eu/newsroom/document.cfm?doc_id=47741)

<comment><USER65> Interesting, they seem to call this an "availability breach"; of course, they require disclosure if it affects people. Do data storage providers need to essentially SLA, then?</comment></comment></comment></comment></comment>


Why It's Hard to Get a Business Off the Ground [video] - <AUTHOR> http://www.youtube.com/watch?v=YQscE3Xed64

<comment><AUTHOR> I have just come back to Australia from 2 months in SE Asia and one of the most striking difference is the lack of rules - in particular how easy it is to get a business going if you want to. As countries develop, they introduce rules which tend to slow down economic growth. Some of these rules are good (it feels a lot safer here due to better road rules, construction etc), but many are pointless and introduced for the wrong reasons as the video indicates.</comment>


Artificial Intelligence Software Is Booming, But Why Now? - <AUTHOR> http://www.nytimes.com/2016/09/19/technology/artificial-intelligence-software-is-booming-but-why-now.html?ref=technology <comment><USER2>
        Date		Approximate cost per GFLOPS inflation adjusted to 2013 US dollars
        </comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>---
        1961		$8.3 trillion
        1984		$42,780,000
        1997		$42,000
        April 2000		$1,300
        May 2000		$836
        August 2003		$100
        August 2007		$52
        March 2011		$1.80
        August 2012		$0.73
        June 2013		$0.22
        November 2013	$0.16
        December 2013	$0.12
        January 2015	$0.06

<comment><USER3> I believe the NN/ Deep learning renaissance started around 2011 -- iirc Alex net was released in 2012 and the famous Google's paper on detecting cat on youtube also around that time.

It seems like the drop of costs from 2007 to 2011 is higher than the previous period. What happened in 2007? Or was it just a slow down during the start of 2000s ?

<comment><USER2> The drop 2011 was due to GPUs taking over. (See my source: [https://en.wikipedia.org/wiki/FLOPS#Cost_of_computing](https://en.wikipedia.org/wiki/FLOPS#Cost_of_computing) ) I believe the gaming industry has (indirectly) heavily subsidized AI's progress.

Nevertheless prices still fell by a factor of 16 between just 2000 and 2007. So the 2,000's weren't stagnant.

</comment></comment></comment><comment><USER4> TLDR from article which supplement's eli_gottlieb's comment:

"Much of today's A.I. boom goes back to 2006, when Amazon started selling cheap computing over the internet. Those measures built the public clouds of Amazon, Google, IBM and Microsoft, among others. That same year, Google and Yahoo released statistical methods for dealing with the unruly data of human behavior. In 2007, Apple released the first iPhone, a device that began a boom in unruly-data collection everywhere."

The combination of smartphone + cloud created a virtuous (for AI, that is) cycle of unending data collection, which fed the improvements in theoretical research models which needed data at a larger scale to be validated.

<comment><USER5> AI is moving fast. Faster than it has in a long time. In research circles, "supervised learning", or data classification problems, are considered solved. Consider the magnitude of that statement: If we have sufficient labeled data, we can predict those labels accurately on data we haven't been exposed to: fraud, faces, diseases, you name it.

AI is moving fast for three reasons, which Andrew Ng summarizes neatly: 1) We have more data than ever before (some of which has been organized in fabulous datasets like ImageNet), much of which is being generated online or by sensors. 2) We have more powerful hardware than ever before -- this is the combination of distributed computing with GPUs. You could say that AI advances at the speed of hardware, or at least is limited by hardware advances. Brute force AI like deep learning is the main beneficiary. 3) We have seen a steady stream of algorithmic advances for years. Specialists despair of keeping up with the literature, research in AI is so feverish.

Others have pointed out additional factors: open-source projects that lower the barrier to entry for the algorithms; cloud computing services that open up access to the hardware. The choke right now, and what's slowing wider adoption, is skills. Many companies don't have the teams to implement AI well.

Some of those companies are also the noisiest about selling AI. I have deep doubts about some vendors to deliver on the hype they're encouraging. And I believe that will lead, not to an AI winter, but to a poisoning of the well that the AI sector will have to address for years to come.

But overselling and hype are inevitable symptoms of real advances in tech, and in some ways, the tech is advancing faster than the hype can keep up with, precisely because the people hyping it, whether salesmen or journalists, don't understand the true extent of what's going on.

AI is just math and code. In a sense, you could say we've entered the age of Big Math, which is the next stage after Big Data. The math is necessary to process the data and determine its meaning. The math takes the form of massive matrix operations that can be processed on the parallel calculators known as GPUs. To call it statistics, as the reporter of the piece does, is only partially true. The math involved in AI comes from probability, calculus, linear algebra and signal processing. It's more than fancy linear regression. And it's definitely more than just making predictions about customer behavior, however attractive that is for industry.

Asking why now about AI software is like asking why now about cars after the Model A came out. Because it's there, it's faster than horses, and it makes you look cool. Like cars or any other powerful technology, AI is part of a race, and that race is taking place between nations and companies. You can decide not to adopt AI, the same way newspapers decided to ignore the Internet, or the way the French decided to fight German panzers with mounted cavalry in WWI. There really is not choice whether or not to adopt AI-driven software. It's a question of when, not if. And for many companies, the when is now, because later will be too late.

To give one example of how fast it's moving: Deep learning has been widely thought to be uninterpretable, or without explanatory power, but that is changing with cool projects like LIME: [https://homes.cs.washington.edu/~marcotcr/blog/lime/](https://homes.cs.washington.edu/~marcotcr/blog/lime/) Which is to say, for some problem sets, we'll be able to combine the impressive accuracy of DNNs with the reasons why they reached a given decision about the data.

On the hardware front, NVIDIA and Intel are racing to build faster and faster chips, even as startups like Wave Computing or Cerberas come out with their own, possibly faster chips, and Google works on TPUs for inference.

<comment><USER6> "the way the French decided to fight German panzers with mounted cavalry in WWI"

Quick note: I think this is historically inaccurate. In World War One, the Germans had essentially no tanks, unlike the British and French. In World War II, there's a popular rumor that Polish (not French) cavalry forces charged German tanks at Krojanty, but this is itself a myth (the cavalry attack was against infantry, tanks arrived afterwards).

<comment><USER7> >> In World War II, there's a popular rumor that Polish (not French) cavalry forces charged German tanks at Krojanty, but this is itself a myth (the cavalry attack was against infantry, tanks arrived afterwards).

Not just any cavalry- Polish winged hussars:

[https://en.wikipedia.org/wiki/Polish_hussars](https://en.wikipedia.org/wiki/Polish_hussars)

Obviously, they could totally take a tank single-handedy, but they have now ascended into heaven to party with the Valkyries.

(It is known.)

</comment></comment></comment></comment><comment><USER8> Because it's been just about 10 years since they figured out how to:

1) Use convolutional layers, ReLUs, and a few other tricks to ameliorate the vanishing gradient problem,

2) Perform continuous, high-dimensional stochastic gradient descent on graphics cards, and

3) Apply these things via stochastic grad-student descent to sufficiently massive datasets that even the most brute-force models and training methods (backpropagation of errors on a loss functional) can actually work.

In the meantime, the hardware for doing it has become commoditized and the software has consolidated and become standardized. So now it's A Thing in industry.

There are lots of "smarter" algorithms that almost definitely come closer to human cognition, for instance probabilistic program induction. But _those_ aren't _fast_ , and don't always neatly separate training from prediction: you're just not gonna be able to train those models ahead-of-time on a 10,000 image corpus inside a single week with today's hardware and software.

We need to find ways to make machine learning fast even when it's not just a bunch of matrix multiplies. Otherwise, every time we make our algorithms more interesting, we cripple ourselves computationally.

<comment><USER9> I'm concerned about something similar, where a lot of AI techniques seem to be rushing toward a local maximum:

* AI researchers do things that get good results faster out of Nvidia cards

* Nvidia makes their cards faster at the things AI researchers are doing

It's getting good results. We sure are going up this gradient quickly. But I don't think it's going to get us to a global maximum.

<comment><USER10> I agree w/ you about moving up a gradient quickly w/ the "GPU manufacturing <-> deep learning research" feedback loop. I think it could last a while though. One really important area of research is figuring out how to take better advantage of greater capacity. Also, how to do more with fewer training samples (0 shot, 1 shot, etc learning). Then there's reducing the precision of the units you're using to increase capacity through software. Applying these algos to video, audio, media generation, and others will eat up all the resources you can throw at it; the algos today could take advantage of larger capacity when applied to time-series. There's so much going on that I don't see it slowing down for at least 3-5 years.

Also, I'd like to point out that we've seen some big breakthroughs in the past 10 years. But for the past 10 years, the whole field of deep learning has been looked at with skepticism and has been very niche. Over the past couple of years, a lot of money and resources have been put in place to pursue this area of research. More money doesn't necessarily mean more results, but there are many many more people working on these problems than ever before. A lot of them are legitimately brilliant researchers in the prime of their careers. I think there's still more to see.

I am concerned about an Nvidia monopoly around deep learning hardware. They give away tons of free cards to deep learning research groups, but at some point they'll want people to start buying. I assume they expect that will be the enterprise set, but if suddenly they manage to move all their capable deep learning cards to Teslas only (which have a huge markup), it will put the hobbyist deep learning developer at a big disadvantage. The only check and balance on that is the fact that Nvidia makes cards for gamers too, and gaming card competition is still somewhat robust, so any technology that gives enterprise a big boost will probably make its way to their flagship gaming cards quickly. Nvidia's only real competition, AMD, is so far behind that they might as well not be in the business. As someone who usually roots for the underdog, it pains me to see AMD fumble so badly in this whole area.

Quantum computing could be a new hill, but I think that's a ways out and I don't know enough about the topic to speak with any real confidence.

<comment><USER11> As someone who does (non-NN) ML on AMD, I might ask why you think that AMD is so far behind. In my experience, their GPU hardware is excellent, maybe even better than Nvidia, especially if you factor the price in. Where they ARE lagging heavily is:

1\. GPU as a service: While all major providers (AWS, Azure) offer Teslas on their servers, there is no AMD on the cloud (that I know of)

2\. Key libraries: Nvidia comes with matrix libraries and cuDNN out of the box, while for AMD, there are only open source offerings that are a bit difficult to manage.

But, If you write your own software or rely on open source, AMD is quite performant and affordable. The problem is that it is really obscure. So, yes, they are really bad at marketing and if you are looking at them as a user instead of as a developer, they are invisible.

<comment><USER12> The key libraries part has kept me wondering. How much would it cost for AMD to assign a handful heavy duty engineers to this task (writing AMD optimised kernels for convolution etc)?

Their management has been fast asleep for at least 2 years

<comment><USER11> That is a part of the problem: they assigned people for the task, and produced _open-source_ libraries for matrices, FFT, maybe even something for DNNs. But, those are not polished much, and you have to hunt them down and install them yourself. And, they do a really bad job at marketing.

On the other hand, finding and installing those libraries is nothing compared to actually developing GPU computing software, so, as I said, if you want to program GPUs, even such scattered state of AMD platform is not that worse than Nvidia. Because, in Nvidia, you install CUDA and you get everything set up. And then - what do you do? You still have to learn a not-so-easy black art of optimizing kernels for the actual hardware.

<comment><USER12> I haven't seen CuDNN equivalents (in terms of perf) for common machine learning frameworks from AMD. I don't think they exist, if they did, people would shift to using AMD.

For NVidia I have seen some faster kernels than the ones supplied by NVidia - [https://github.com/NervanaSystems/neon](https://github.com/NervanaSystems/neon) \- though CuDNN introduced Winograd kernels too in their last update

<comment><USER11> I do not know about the quality of this since I do not use NNs, but there is [https://github.com/hughperkins/DeepCL](https://github.com/hughperkins/DeepCL) and I think I have seen others.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> I'm still not comfortable calling the thing that is booming "artificial intelligence". It is mostly pattern recognition and classification. Intelligence is something else.

<comment><USER14> Yes, "signs of intelligence" include _creativity_ , _pedagogy_ , ..., and of course, getting bitten by the bug of existential angst.

Accept no subsititute.

<comment><USER15> It seems pretty likely that we will start to see "General AI" discovering problematic things like the unsolvable nature of ethics questions, the ungroundedness of truth claims, the immense silliness of religions and ideologies, etc.

Which might be a good thing. Terrifying visions of AI are all about certainty and the authoritarianism it creates.

If existential depression is a mental friction from the lack of certainty about what to do, then some measure of it is probably necessary...

<comment><USER8> >Which might be a good thing. Terrifying visions of AI are all about certainty and the authoritarianism it creates.

Indeed, may the gods protect us all from some things actually being true and other things actually being false. That would be terrible!

<comment><USER14> (I sense/assume a missing /s in your post, Eli.)

The objection here would be that entertaining that we can assert T/F of _all_ propositions, given results of _halting problem_ [computation], _incompleteness_ [formalism], and _uncertainty_ [physics], is unreasonable.

<comment><USER8> >The objection here would be that entertaining that we can assert T/F of all propositions, given results of halting problem [computation], incompleteness [formalism], and uncertainty [physics], is unreasonable.

This displays a radical misunderstanding of the phenomena mentioned. The Halting Problem and logical incompleteness are the same thing underneath, and while they do hold in all formal systems, this never actually matters for non- meta-level mathematics. Basically any theorem about a structure we actually care about will be sub-Turing-complete, and with modern type theories, we can tear down inductive types and rebuild them with stronger axioms when we need to. As a result, some self-referencing theorems are true-but-unprovable, but we never actually need those theorems.

Uncertainty in physics is either just probabilistic (in the case of typical experiments), or, in the special case of Heisenberg uncertainty... no actually, that's just probabilistic imprecision too. That's what Heisenberg's inequalities _actually say_ : "the product of the standard deviations of these measurements must always be at least this much."

Nowhere are we encountering the kind of _radical, existential_ "we can't know anything and had better give up" uncertainty without which /u/mbrock seems to think we will all fall into political authoritarianism. He's been reading too many liberal philosophers of World War II, or maybe just watched that BBC thing "Dangerous Knowledge" and took it seriously.

<comment><USER14> It seems _irrelevant_ that, as of now, "any problem we care about" is sub- Turing-complete. We're still taking baby steps as a species and the relevant timelines in context of this thread are futuristic. Agreed?

The point is that to maximally insist that "we can't know anything", and/or, "we can know everything" are both equally unreasonable positions.

> Nowhere are we encountering the kind of radical, existential "we can't know > anything and had better give up" uncertainty without which /u/mbrock seems > to think we will all fall into political authoritarianism. He's been reading > too many liberal philosophers of World War II, or maybe just watched that > BBC thing "Dangerous Knowledge" and took it seriously.

S/he can address your concerns -- that was not my intent, but hopefully you agree that unreasonable insistence on maximalist positions inherently carries the danger of "political" authoritarianism.

Your _practical_ position reminds of a purported exchange between Wittgenstein and Turing, per Hewitt [ref]. I am sympathetic to it. But reminder again that my initial comments here were in context of intelligent machines. In fact as I am writing this little note, I am entertaining the thought that a veritable intelligent machine may in fact review the 3 (methodological) constraints noted, and shrug it off as practically unimportant. Or it may become alarmed. :)

[edit/p.s. ref:[http://lambda-the-ultimate.org/node/4302](http://lambda-the- ultimate.org/node/4302)]

<comment><USER8> >It seems irrelevant that, as of now, "any problem we care about" is sub- Turing-complete.

It's not irrelevant, it's a question of how you view the Church-Turing-Deutsch thesis: can hypercomputation occur in the physical world? If it can, then why and how are we somehow blocked from utilizing its physical manifestation to reason about the relevant questions of real-world events? If it can't, then why aren't systems with finite, if large or even growing, Kolmogorov complexity thus sufficient for all reasoning about the real world?

>S/he can address your concerns -- that was not my intent, but hopefully you agree that unreasonable insistence on maximalist positions inherently carries the danger of "political" authoritarianism.

My objection has been precisely that to insist on radical ignorance has led to an obnoxiously enforced liberalism.

>Your practical position reminds of a purported exchange between Wittgenstein and Turing, per Hewitt [ref].

I think Turing is just plain wrong here. Real mathematics did not originate by receiving ZFC with first-order logic as a revelation at Sinai and extrapolating theorems and structures from there! It began by formalizing ways to reason about the real world around us. When those original informal methods became insufficient in the foundational crisis of the 19th century, _then_ mathematicians started inventing foundations to unify everything without paradox.

Notably, someone in the comments thread then mentions Curry's Paradox, which I looked up and found _surprisingly_ underwhelming. Curry's Paradox is a _perfect_ example of what Wittgenstein called a meaningless language game! Classical implicature X -> Y isn't always equivalent to the existence of a causal path "X causes Y", but natural-language implicature _mostly_ talks about causal paths, so conflating the two in symbolic logic derives a "paradox".

>In fact as I am writing this little note, I am entertaining the thought that a veritable intelligent machine may in fact review the 3 (methodological) constraints noted, and shrug it off as practically unimportant. Or it may become alarmed. :)

I don't think that a machine limited to reasoning within a single mathematical structure or foundation can really qualify as "intelligent" in the human sense. Logical foundationalism is the wrong kind of thing to constitute a capacity to think.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER16> We don't marvel over the ATM reading a handwritten paper check correctly. That's a considerable achievement.

<comment><USER17> From the first time it started doing that, I've been amazed that that works so well in a production system. Every. Time.

<comment><USER16> The first time I saw it, I thought they had people in some call center doing the reading. But they don't, at least not often.

The US Postal Service used to have 55 centers where humans tried to read envelopes that the machines couldn't. They're now down to one. "We get the worst of the worst. It used to be that we'd get letters that were somewhat legible but the machines weren't good enough to read them. Now we get letters and packages with the most awful handwriting you can imagine."

[1] [http://www.nytimes.com/2013/05/04/us/where-mail-with- illegib...](http://www.nytimes.com/2013/05/04/us/where-mail-with-illegible- addresses-goes-to-be-read.html?_r=0)

<comment><USER18> very interesting

>> equipment that can read nearly 98 percent of all hand-addressed

that number must be pretty outdated

</comment></comment></comment></comment><comment><USER19> 1\. Availability and _accessibility_ of large amounts of training data. Without this training and validation is expensive if not impossible. Now if you don't have the data you can acquire it yourself. Leading to ...

2\. Computational speed & storage upgrades. This applies largely to physical, time-critical things like automated driving. The self-driving car could have had all the data it needed in 1980 to do its thing, but required fast computers and lots of data storage to do it safely in real time in a feasible commercial product.

3\. Advancement of algorithms. Fervor and excitement around AI/ML has been on a slow but perhaps exponential burn. This has led to the refinement of algorithms that largely sat dormant from the late 80s (and earlier) until fairly recently. This also means lots of open source libraries for people who wish to implement without caring about the underlying mechanisms behind the algorithms.

These things are leading more people to dabble recreationally and commercially.

</comment><comment><USER20> Ready access to high-quality usecases and training data, along with shared knowledge of the methods that work well on these helps: [https://www.kaggle.com/competitions?sortBy=numberOfTeams&gro...](https://www.kaggle.com/competitions?sortBy=numberOfTeams&group=all&page=1&site=main)

(disclaimer: I work at Kaggle)

</comment><comment><USER21> I thought we are already over that cliff of becoming forgotten. Yes, we didn't call it AI 2 years ago. But has there been such a big technological change since then with the googles, facebooks and twitters of this world? I think the problem is that it is actually so transparent that you really can't see it when it is applied to you.

</comment><comment><USER22> "Democratizing AI" is the word being thrown around a lot by CEOs, PR, et al. But only time will tell whether it's true democracy.

<comment><USER23> As far as I know, the concept of "democratizing AI" comes specifically from the OpenAI initiative, which intends to make new developments in AI technology broadly available and not tied to any single company or entity. This in order to ensure that no single entity has control over how this technology can be used.

Given that Google, Apple, Facebook and others have vastly more data than any independent project, and therefore have stronger AI (currently limited to e.g. speech recognition, image recognition and other low-impact applications), the state of democratization of AI by this measure is poor today.

</comment></comment><comment><USER24> Some how machine learning got rebranded and now it's important... and major products are putting prediction into the ui more.

</comment><comment><USER25> Such vacuity

</comment><comment><USER26> So, the influx is due to a plethora of things, which are not all mutually exclusive:

1) The Internet has indeed produced large data sets which allowed statistical AI approaches to flourish opposed to logical AI approaches.

2) Somewhat better algorithms. However, I'd say that the algorithmic development hasn't had as much progress as the comments elude. We've only had a few notable innovations like NMT in the past ~20 years.

3) Computational Power to run the algorithms, so we can perform more experiments on large data sets, more computationally intensive algos, and induce better hypotheses.

4) Libraries like scikit-learn and keras have "democratized AI". Grad students used to implement algorithms themselves in the 2000s; now middle-schoolers are doing ML with the tooling already available.

Those are basically it. I think (2) could even be taken off, because again: learning algorithms have barely changed IMO.</comment>


Ask HN: What tool do you use to keep track of all passwords? - <AUTHOR>

What tool do you guys use to keep track of all of your startup accounts and passwords? <comment><USER2> My brains alone. Over the years I've developed a pattern I can remember easily, and apply another pattern to it based on the site or app's name. It's a simple math formula that changes the entire password instead of just applying a suffix or prefix to an existing strong password.

Ironically, the only site this fails on is my banking site which has wish-it- was-2-factor authentication and a limit of 8 characters for the password. Stupid.

<comment><AUTHOR> Right on! But what if you have a team of employees and interns that need access to different accounts. You can't expect them all to memorize unique passwords.

<comment><USER2> True enough. But all they'd have to memorize is one hard password (or even a pattern like 102938), and one modifier pattern based on the URL of the site.

A simple example, using 102938 as the base password would be taking the main name part of the url (eg. www.google.com would just be "google"), converting each letter to a number (g = 7, o = 15, o = 15, g = 7, l = 12, e = 5), adding them all together (61), divide that by the number of characters used (61 / 6 = int(10.16) = 10), then add that result to every pair of numbers in the original password (10 + 10 = 20, 29 + 10 = 39, 38 + 10 = 48), giving you a password of 203940 for that site :)

EDIT: forgot to make this hard to crack. Again, the pattern approach can work - 203948 for "google" could easily and predictably become "go20og39le48@mycompany" or some such.

</comment></comment></comment><comment><USER3> Lastpass. It's awesome. I use it for passwords, but also as a digital lockbox for random account information where I don't want to keep the cards. A lot better than evernote, which is where I used to keep this type of stuff.</comment>


Preferred Women's Waist-to-Hip Ratio Variation over the Last 2,500 Years (2015) - <AUTHOR> https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0123284 <comment><USER2> If the cover chart is anything to go by the variance in any time period about the mean is immense.

I could believe there is a trend but at any point in time that looks like the last and next 100 years is almost entirely reflected in contemporary state.

The paucity of ancient references also stood out in that.

<comment><USER3> Also the methodology for estimating ancient WHRs seems suspect.

</comment></comment><comment><USER4> My preference seems to be 1 stddev from the mean</comment>


Don't make me pick a unique username ever again - <AUTHOR> http://fredandrandall.com/blog/2011/09/23/dont-make-me-pick-a-unique-username-ever-again/

<comment><USER2> unique usernames are needed for nice urls like youtube.com/user/johndoe, flickr.com/photos/johndoe, etc. that's what they are for.

from the examples above, flickr username url is optional and you can stay with the ugly one, that's probably the solution you're looking for, isn't it? now you must decide which use case is easier.</comment>


Abandoning the Work I Hated - <AUTHOR> http://opinionator.blogs.nytimes.com/2015/08/20/abandoning-the-work-i-hated/ <comment><USER2> I know the feeling; I've had some great jobs and some bad jobs in my career (albeit "only" 10 years as a professional). The most fun ones were energizing and are (to be honest) the things I seek out when looking for a new opportunity. The #1 thing that I ask when I look at a new job now is "Will I have fun?" #2 is "Is this good for me?" I'm still chasing after 2 different feelings:

1\. Working with a couple of colleagues to finish an integration, finishing it, then booking a last-minute tee time to go play golf in celebration. It was a rewarding moment.

2\. Getting together with teammates every couple of days for "Halo-thirty," where we'd break, get together in the afternoons for some halo3 in our teamroom, goof off for a bit, then get back to work. That team was remarkably- fun to be in and around.

I've actually found it a bit liberating now that I know what i'm looking for. In certain contexts at work I speak my mind more freely, decline or accept certain assignments, etc, and Nothing Bad Has Happened (yet).

</comment><comment><USER3> I have a kid. And that is the only reason I do not abandon my pointless work.

Anyone like this do it, and survive? I would love to see more motivational stories.

Those who transitioned to professional clown work appreciated!

<comment><USER4> This is a huge issue. One you have kids, it becomes far more difficult to walk away from your income, even for a transitional period. Another complicating factor is health care.

I'm always interested in these stories, kids or none, but I think it changes the equation dramatically. Like you, I'd really like to read more stories about people who did this with the additional obligation of supporting children.

<comment><USER5> Perhaps having a kid should motivate you more. Can I myself in the mirror in the morning? But also, can I look my kid in the eye and tell her to do the right thing, while not doing so myself? And what are you teaching your child by financing better education or better healthcare for her through a job that may be increasing inequality for other children, thereby reducing the quality of life, education and healthcare for them?

I'm not saying those are the choices you are making. But on the other hand, a lot of people are but cognitive dossonance prevents them from seeing that.

Lookup "The Banality of Systemic Evil" and "Matthew Effect" for more thoughts.

<comment><USER6> Achieving yourself and seeking fun in work comes after you put food on the table and provide safe home to your dependant. The question for many is not whether providing better teaching, better healthcare, but to provide teaching or healthcare at all if you loose or give up your job.

</comment></comment></comment></comment><comment><USER7> So how does this jive with [https://news.ycombinator.com/item?id=10095776](https://news.ycombinator.com/item?id=10095776), which states that suffering can lead to a more meaningful and selfless life?

<comment><USER8> I think it's implied in this OP that the author did not see any meaning in his lawyer job, there's nothing "bigger than him" he's working for, just making a lot of money keeping actual criminals out of prison. There's no "bigger" purpose in that.

As a party clown/musician there is purpose - he can make kids happy, he knows that "something in his future is expected of him", his work is extremely meaningful to the kids, he catches glimpses of that (whether he actually understands that is hard to say, he'd have to be able to 100% become a child to see the child's point of view).

<comment><USER9> "just making a lot of money keeping actual criminals out of prison".

If the defendants were not found guilty then they are not "criminals". Unless you were in court then you really ought not to judge. (Though I am aware the the American justice system is probably way less perfect than it claims to be).

<comment> saiya-jin person committing a crime is a criminal, no matter if justice system finds him guilty (I mean come on, we all know similar cases). reciting some law textbooks seems a bit ridiculous in face of reality

<comment><USER9> Sure but in the absence of other evidence, only one person really knows the truth. How can you be sure that a person you think is criminal is actually one? If you have the evidence to know that, then you should be able to convict them.

</comment></comment></comment></comment></comment><comment><USER10> Thank you, this was a good read. I really needed it right now.

</comment><comment><USER11> While i wouldn't personally become a clown (matter of preference), i really admire this guy's courage. I frequently think about upping and leaving the security of a cushy job, and it's a lot easier said than done. Freedom is scary!

<comment> saiya-jin nah, it isn't. just the first step towards it is hardest. each next one is more and more simple, but we tend to create mental barriers to protect our comfort zone, maybe remnant from times when life was more about plain old survival

</comment></comment><comment><USER12> An inspiration.

</comment><comment><USER13> In case the end of the article got you as curious as me, this is the guy's page:

[http://www.movingtothebeat.net/about- robert.html](http://www.movingtothebeat.net/about-robert.html)

<comment><USER14> The picture at the top right in the block of pictures on the left side says it all.

</comment></comment><comment><USER15> Hmmm... believe I'd actually rather be a lawyer than be in a clown suit surrounded by bratty spoiled germy kids....

Guess it takes all kinds.

</comment><comment><USER16> Following in the footsteps of Herschel Krustofski... good for him.

</comment><comment><USER17> If this is true, he is one of the most courageous guys in Americas.

<comment><USER18> Becoming a lawyer is different than let's say becoming a doctor. The lawyers I know barely make a living. The profession is not the ticket to "the good life", like it was in the 50's.

I don't know what school he graduated from. Why he ended up as a criminal attorney? How good/ethical of an attorney he was? (I throw in ethical because it's not judged by the typical client, like I feel it should? Maybe, their really isn't a good way to judge a person ethics?)

Anyway, he made a life change. He can always go back to law, so in reality it's not that heroic of an act. He still has his degree, and passed the bar in his state. He definetly has a fallback. A fallback most of us don't have? It's very common to lawyers to pass the bar, and never practice law.

<comment><USER17> He cannot go back. Competition will immediately portray him in local press as a clown.

</comment></comment></comment><comment><USER19> >Robert Markowitz is working on a memoir.

Thriller of the century.

<comment><USER20> Wisdom and happiness are often boring (try reading the meditations of Marcus Aurelius, I loved it but occasionally yawned). "Success" at any cost, extremes, pain, murder, 168 hour code sprints are more thrilling.

<comment><USER21> > _.., 168 hour code sprints are more thrilling._

"So then I downed the fifth Red Bull that hour and wondered whether I better change that for loop into a while loop because the first operand was empty anyway."

Can't wait for the novel!</comment></comment></comment>


Mysterious Repeating Radio Burst from Space Is Back, Right on Time - <AUTHOR> https://news.yahoo.com/mysterious-repeating-radio-burst-space-175100403.html <comment><USER2> Its rotating. Its periodicity is now well understood. It has focal beam qualities. The "Mysterious" word is over-used. All astronomical objects which are smaller than the resolution limit of the telescopes used, are "Mysterious" because we cannot state what morphology they conform to.

Another word is "interesting" -If we can type this source into a class which is radically different by size, weight, speed of rotation, density, chemical composition, interior structure, number of black holes on a wednesday at 5pm, we can then apply that knowledge to the structure of the universe.</comment>


Transgender Cyclist Wins Female Cycling World Championship - <AUTHOR> https://www.nationalreview.com/news/transgender-cyclist-wins-female-cycling-world-championship-claims-only-objections-come-from-losers/ <comment><USER2> "Transgender rights" in sports are inconsistent with having separate sports competitions for males and females. In Connecticut, girls' track events are being won by "transgender girls".</comment>


My story with App Review - <AUTHOR> https://twitter.com/keleftheriou/status/1274356729224892416 <comment><USER2> My app has been repeatedly rejected because I do not offer free in-app trials (but offer money-back guarantee on my website) and also do not offer in-apps purchases.

I asked if I offered in-app purchases, would the app be approved?

No, since I offer 3-day money-back guarantee on website-based purchses, I must ALSO offer a free trial in the app.

But I don't want to offer free trials. I'm ok offering a 3-day money-back guarantee, but Apple says that's not an option for apps.

They insist that I (1) change my business model from 3-day money back guarantee to free trial and (2) purchase a shit ton more infrastructure to handle the audience that free trials attract (I've tried free trial business model and it created a lot of use but no more revenue than 3-day money back guarantee). The money-back guarantee seems to weed out a lot of free loaders, at least in this domain — not necessarily others.

<comment><USER3> You say that Apple is demanding you change your business model but actually you're demanding that Apple accomodate yours.

If someone buys your app and doesn't like it then how do they know about the money back guarantee ? Because depending on how visibly you communicate that to the user at best you could make it hard for users to know about it and at worst you could be flat out being deceptive.

Either way Apple takes on the significant chargeback risk for your app which is far more concerning for them than whatever revenue they get.

<comment><USER4> > Either way Apple takes on the significant chargeback risk for your app which > is far more concerning for them than whatever revenue they get.

Gee, if only there was a way for the person to signup directly on the app maker's site, where they would assume all the risk...

<comment><USER3> You are still placing all of the reputation risk on Apple though.

If this guy builds an app which is forcing users into paying upfront and then hiding a money back guarantee in some hidden text then that is deceptive. And Apple suffers since they allowed the app to be sold on their store.

<comment><USER5> > And Apple suffers since they allowed the app to be sold on their store.

Well, they could allow the app to be sold outside their store, and then they wouldn't face any reputational risk from it.

I don't really see how this is deceptive either - if the app clearly states that you're paying for it upfront, and what the price is, then how are you being deceived? If you aren't willing to pay for it without trying it, then you can just not use it.

<comment><USER2> > Well, they could allow the app to be sold outside their store, and then they > wouldn't face any reputational risk from it.

The service is sold outside the store (website). The apps to use the service are free on all platforms.

</comment></comment></comment></comment></comment></comment><comment><USER4> It seems like a simple law could fix this:

Phone device makers that provide an application store must also allow the user to load non-approved applications with equivalent API access to approved applications, and must make it possible for the user to load a third-party application store with equivalent access to the first-party application store.

<comment><USER6> I would like this to be extended to game consoles and any other kind of personal computing device, too! Although phones are clearly the most egregious example, game console manufacturers create a weird two-party market where they subsidize the purchase of the console so they can rent-seek off the larger playerbase by taking a cut of game sales.

<comment><USER3> That would completely destroy the gaming console market.

Nintendo would largely be fine. Sony and Microsoft however would be forced into anti-consumer practices like putting advertising everywhere and selling your data. Not to mention a complete consolidation of the studios where most games would be made by Sony Studios and only sold on the Sony app store.

<comment><USER7> > like putting advertising everywhere and selling your data

Ever used Xbox Live on the Xbox you paid for and seen ads? We're already there.

<comment><USER3> Well it would go to a whole new level if you want to take away their only revenue source.

Microsoft may even just decide not to build consoles at all.

<comment><USER8> Sounds good to me. Let somebody who wants to play ball fairly build the consoles.

</comment></comment></comment></comment></comment></comment><comment><USER9> It's a very nice example of Stockholm syndrome: "They are the nice big brother inflicting pain to devs and users and restricting their rights, and it's probably my fault if they are not nice with me"

<comment><USER10> I think despite gathering his courage to speak out, he is still afraid of Apple's reprisal, so he doesn't want to speak too harshly of Apple, lest Apple decides his apps are in violation and his account must be banned.

</comment></comment><comment><USER11> > But getting it on the store has been one of the most difficult, non- > technical obstacles I've had to overcome, and this was only one of _many_ > other hoops I've had to go through.

We were very careful in our app to not push the boundaries at all. Nothing even remotely outside the norm for this very reason.

Now... contrast this "code chilling effect" with how great innovations have happened. Apple is clearly preventing any App company from ever having the same opportunity that Apple themselves had.

Bad moves I think.

</comment><comment><USER12> I assume the policy is based on the fact someone at Apple has decided that keyboards on the Apple Watch can never be designed to have a satisfactory user experience, therefore all watch keyboards are banned.

However OP's app is grandfathered in, but when he tries to make other similar apps, they aren't allowed.

<comment><USER13> Apple felt this way about web pages on Apple Watch also. But I believe they have since relented.

</comment></comment><comment><USER14> > Pushing for an answer on how one of my apps can have a watch keyboard, but > the other cannot, they could only "discuss via phone".

What would Apple's response be if you started with "This call may be recorded"? Is recording without notice legal in any state?

<comment><USER15> Limiting the question specifically to "can you record your own phone calls for your own use", and ignoring questions about what you can do with the recordings (which is where things like CCPA or GDPR would apply), there are two regimes which cover the vast majority of telecommunications law: one-party consent and two-party consent. (India doesn't allow recording at all, but that's because they're stuck in the Ma Bell era of telecom law.) One-party consent means one party - i.e., yourself - needs to consent to recording for the recording to be legal. The latter means all parties need to consent to recording. (Evidently "two-party consent" was coined before conference calls were a thing.)

About three quarters of the US is one-party consent but a quarter are two- party consent. California is part of the latter group and if any party to the call has California residence, then you need consent from all parties to the call before you can record. Yes, this has gone up to California supreme court. Yes, this includes all of the FAANG companies. So you would need some kind of notification that you were recording and for the person on the other line to consent to the recording, regardless of where you live, simply because Apple employees live in California.

(Depending on jurisdiction, there are exceptions for a party to a call to record the call if it has something to do with specific crimes, but "violating antitrust law" isn't one of them.)

<comment><USER16> I don't see the Federal Supreme Court weigh in on this, and without that I don't see how California can impose it's laws on people out of state. It's no different from any other law-- take open carry gun laws as an example. A state that prohibited open carry would not be able to assign liability to someone that open carried in a state where it was legal. It really doesn't matter what the California Supreme court says, they don't have authority outside of California. I suppose they could make laws about how such recording could be used if they entered in California, but not much else.

<comment><USER17> The problem I see is if you're recording the call for legal reasons, you're probably going to court in California where it wouldn't be admissible even if they wouldn't be able to charge you.

<comment><USER16> I'm thinking more about this specific situation of getting a formal record of what Apple tells developers. My suspicion is that if lots of people could compare notes there'd be some inconsistency. Certainly they don't bully folks like Netflix around because loads of people would suddenly become interested in Android if apps like that disappeared. Also California law would be irrelevant in a Federal class action suit.

</comment></comment></comment></comment></comment><comment><USER18> The more people speak up about these situations the faster they will be fixed permanently to avoid a neverending flow of PR nightmares.

</comment><comment><USER19> I am not sure that this app is _for_ blind and visually impaired users, or for everybody but also useful enough that blind and visually impaired can use it (which would be pretty darned useful, for a keyboard)

If it were explicitly an assistance app, would Apple be in violation of any discrimination laws by refusing to have it in the app store in that way?

</comment><comment><USER20> The moral is, don't relies on Platform vendor's official distribution system as a sole way to distribute your software. If that platform doesn't allow you to distribute the software outside of it, don't bother to support such a platform. You have been lost already. Don't waste your time on dealing with such hideous non-free platform.

<comment><USER21> I understand the sentiment but there's a reason developers still flock o develop for Apple and Google even under these conditions. Despite the restrictive nature of the app stores they're the only chance most small developers have of ever reaching a wide enough _paying_ audience.

Many apps are pretty much just a feature bolted onto the OS and could easily be shipped by Apple or Google as an integral part of the OS thus taking the whole reason foe that app's existence. So for many devs the whole income stream depends on Apple or Google far more than just by using their app store.

<comment><USER22> I'm not sure why you included Google there, as you can install on Android without going through their store. You just have to confirm that you're really sure you want to.

<comment><USER21> I was pretty clear why I included Google. Because using their app store is the single best chance for a dev to monetize their app on Android, or at least to get it noticed. Sure, they can use another store like F-Droid but while the possibility is there, they'd _severely_ limit the audience. And most people are discouraged (for good reasons) from installing APKs from random places on the internet.

Few Android apps are distributed exclusively outside of the Play Store and even fewer get any traction with users, not to speak of being a financial success for the dev.

</comment></comment></comment></comment><comment><USER23> Well said

</comment><comment><USER24> These kind of tweets kinda defeats the true purpose of Twitter, why don't the people write a blog instead of making dozens of tweets? I mean, you can write a blog with exact o more words, you can express yourself more and make the article much richer, and then you can link it to your twitter account, it makes absolutely no sense to do what he did.

<comment><USER25> Engagement. The majority of the people that such a blog would be shared with won't read it because it requires clicking an extra link. Tweets can be read in-line. Also, tweets in a tweet thread can be commented on specifically rather than on the blog post entirely.

<comment><USER26> _Also, tweets in a tweet thread can be commented on specifically rather than on the blog post entirely_

So it's like a college lecture where the professor stops after each sentence and has to answer questions from the audience. Pretty much the opposite of communication.

<comment><USER25> Not really. The tweeter in this case isn't stopping between each tweet to address comments. It is asynchronous versus the synchronous example you gave.</comment></comment></comment></comment>


HY4 - Zero emission electric air taxis with hydrogen fuel cells - <AUTHOR> http://hy4.org/ <comment><AUTHOR> An interesting aspect of these is that they are very quiet. Maybe they will allow flights past midnight, when the noisy planes are not allowed. With a range of 750-1500km, that puts most of Germany within range.

However the speed (200km/h) is not as fast as trains or even fast cars on the autobahn.</comment>


Why don't you shop on eBay anymore? - <AUTHOR> https://www.reddit.com/r/AskUK/comments/6px6iw/why_dont_you_shop_on_ebay_anymore/ <comment><USER2> I can't trust sellers. They put their own interest way over buyers. I get it, you need to make a profit, but it's way too uneven. It results in overcharges in postage, incomplete to false descriptions and more expensive products. The few bargains aren't worth the trouble. It's just easier to go with Amazon. At least there the shipping is fast and there's someone to complain to if things go wrong.

</comment><comment><USER3> I haven't been using ebay much anymore, except for some tech related stuff mostly because it's flooded with overseas sellers or domestic sellers sourcing products from overseas. Not that I have anything against it but most of the time I don't want to wait weeks for something to get delivered with open ended delivery dates or deal with trying to return something overseas.

</comment><comment><AUTHOR> I thought it was interesting to see people who still like Amazon and think it's better than eBay.

I still use both, but for what I buy eBay tends to be easier and better.

</comment><comment><USER4> for cheap knockoff electronics from china I use aliexpress.com usually cheaper. although sometimes ebay is cheaper.</comment>


Abstract visualization of biological neural network - <AUTHOR> http://nxxcxx.github.io/Neural-Network/ <comment><USER2> Really Impressive. I wonder if you could use this to visualize a neural net that's solving actual problems.

<comment><USER3> This doesn't look like a neural network that's solving any actual problems beyond visualizing firing patterns. Most ANNs don't have any concept of time built into the model. When you're in feed-forward mode, the underlying computation is simply a bunch of dot products.

However, you might be able to make some cool visualizations if your ANN was a Spiking Neural Network.

[1] [https://en.wikipedia.org/wiki/Spiking_neural_network](https://en.wikipedia.org/wiki/Spiking_neural_network)

<comment><USER4> Well it's thresholded so you could visualize the "activated" neurons in a manner such as this.

Would it be useful though? Maybe. Probably not as the fact that a neuron fires isn't nearly as useful as why it fired (what it represents).

But I like that you mention time. It always seemed odd that NN's today completely ignore that aspect. Sure whether a neuron fires is binary. But the accumulation of spikes is not binary and highly temporal. yet we completely ignore this aspect.

<comment><USER5> I'm confused by you guys saying NN's today completely ignore time? If I have a network of Hodgkin-Huxley neurons and I integrate the equations, I'm integrating over time. I'm not sure how that is ignoring it? Disclaimer: it has been about 4 years since I've written any code to integrate H-H neurons, so I might be forgetting something really obvious. :) But I can't imagine EVERYONE is ignoring time when they simulate networks of neurons.

<comment><USER6> They are talking about NNs used in machine learning, which are atemporal, unlike H-H or other models of human cognition.

In a machine learning neural network there is no integration; they are basically just nonlinear data transformations that can (usually...) be trained.

</comment></comment></comment></comment></comment><comment><USER7> Is it based on an actual model of neural activity (Izhikevich, H-H, etc.) or is it an abstract visualisation in the sense that it just looks like neurons firing?

<comment><USER7> OK, it's the latter. Pretty pictures, moving on.

<comment><USER5> There is a serious amount of heavy lifting in the visualization department done here for us. I imagine forking this repo and adding a simple model to govern the neuron and axon behavior would be doable. Writing this off as "pretty pictures" is a little over-the-top. Seems like a significant amount of work went into producing these real-time visualizations.

</comment></comment></comment><comment><USER8> Shameless plug for a little visualization I made some time ago of a neural network in a developing fruit fly, which performs an interesting probabilistic algorithm to elect a maximal independent set [http://jberryman.github.io/fly- mis/](http://jberryman.github.io/fly-mis/)

</comment><comment><AUTHOR> Be sure to zoom in!

<comment><USER9> Good tip! Also left click and drag to rotate. Right click and drag to pan.

</comment></comment><comment><USER10> Very cool, what's the backstory? Just a fun side project?

</comment><comment><USER11> Is a neural network all that you would need to model in order to simulate a human's thoughts?

How far are we from mapping a real brain to a simulation like this and hitting play?

<comment><USER12> It somewhat depends on what you think of as a neural network. For simple models no, state is vary important, as is signaling delays, hormones, and blood flow etc. Which means each neuron would individual be far more complex than what is generally used in computer science. Further brains are far from random so you would need to either copy someone or simulate the various developmental stages a person goes though. Finally, a neural network is pointless without some output so for anything beyond pretty pictures you need some sensory input's and some way to handle outputs.

However, I have seen estimates that somewhere around a yottahert class computer working for a few years might be capable of simulating a brain sufficiently complex to hold an arbitrary conversation. Building the software capable of running such a simulation is another story.

</comment></comment><comment><USER13> This looks absolutely stunning. I wish I could train the brain by myself and see how it operates in as an artificial neural network. Anyway, great job!

</comment><comment><USER14> This is awesome! [http://imgur.com/t/science/0eZe3RK](http://imgur.com/t/science/0eZe3RK)

</comment><comment><USER15> Can someone tell me anything about this project? I'm curious.

</comment><comment><USER16> Wow this is amazing. Great work.

</comment><comment><USER17> Everyone complaining or fussing about how primitive this model is on a browser, ya'll need to chill out and stop taking beautiful work like this for granted. Some of you may say this is easy to do or why didn't he also "do this" or "do that" \- how about you help out and make it better ;)

@sethbannon Thanks for the visuals and hard work!

<comment><USER5> I agree, a really cool project which I would imagine could be extended to a "real" model without a ton of work. I imagine the amount of work to fork a javascript library which integrated a network of H-H model neurons to make it output this type of visualization would be far more work than adding the H-H piece to this code. I could be wrong but at first blush, that's how I see it.

<comment><USER6> I think it would be quite a bit of work to integrate the H-H model or any model of a biological brain with this visualization.

This is because the visualization is basically just a graph, with axons as edges and neurons as nodes, and then signals are propagated with extremely simple graph-traversal style logic. On the other hand, the H-H model tries to deal with the physical reality of the human brain so it is much more complicated and relies on some data (like axon width) that simply doesn't exist here.

It would be possible and potentially interesting, for sure, but probably more work than you might think.

</comment></comment></comment><comment><USER18> I like the other title better, "Abstract visualization of biological neural network"

<comment><USER19> Or at least "Biological Neural Network Simulation in JS"

<comment><USER20> oh the power of buzzwords

</comment></comment></comment><comment><USER21> Is it based on anything or just some random connection and nice visualization (which then isn't really impressive)</comment>


There still aren't enough tech workers, and enterprises are paying the price - <AUTHOR> http://www.cio.com/article/2982976/there-still-arent-enough-tech-workers-and-enterprises-are-paying-the-price.html <comment><USER2> As others have pointed out, there isn't really a "shortage" in the general sense. There's a "shortage of people will to do this for the amount I want to pay". If you want to attract more talented people to IT, pay more.

And on a related note: more businesses should "play the long game" and hire junior level / inexperienced people, and provide serious mentoring, on-the-job training, and help groom them to advance. The refusal of firms to help their staff grow and develop is, IMO, a big part of the reason why we have the move to the "gig economy". It would also help if companies treated employees as _people_ not some fungible commodity, where developers can be swapped in and out like cogs in a machine.

<comment><USER3> _It would also help if companies treated employees as people not some fungible commodity_

Large companies can't do this: they're too wedded to the idea of "human resources", where any job can be filled by any human with the appropriate SAP title, and managing by drug tests, and other non-human-judgement processes.</comment></comment>


IPad user survey - <AUTHOR> http://technologizer.com/2010/06/30/ipad-survey/

<comment><AUTHOR> These results are realy impressive. This product is spot on ! Beeing a happy user of the iPad for a month now since its release in France, my experience matches the survey results.

I can now do things on Internet, like browsing or commenting on Hacker News, or checking and responding to mail in situations I couldn't imagine before. I'm for instance currently writing this comment on my iPad lying in my bed just after I woke up.</comment>