The sad state of personal data and infrastructure - <AUTHOR> https://beepb00p.xyz/sad-infra.html <comment><USER2> I've been working on a project along these lines recently. I've called it Dogsheep - the basic idea is to have scripts that export all manner of my personal data (from Google, Apple HealthKit, Twitter, LinkedIn etc) into SQLite database files, then use my Datasette web app to browse them and run interesting queries.

More about that here: [https://dogsheep.github.io/](https://dogsheep.github.io/)

The tools I've built so far are under [https://github.com/dogsheep](https://github.com/dogsheep)

<comment><USER3> I went similar route after discovering Google Checkout refuses to return my YT comments. I wrote quick&dirty tempermonkey script logging submitted HTML forms (addEventListener "submit").

<comment><USER4> Did it work the way a scraper does? Curious to know how you did this.

<comment><USER3> I was only interested in stuff I posted on the internet for indexing/easier search, [https://developer.mozilla.org/en- US/docs/Web/API/HTMLFormEle...](https://developer.mozilla.org/en- US/docs/Web/API/HTMLFormElement/submit_event) and [http://www.meekostuff.net/blog/Overriding-DOM- Methods/](http://www.meekostuff.net/blog/Overriding-DOM-Methods/) covered form submissions, result goes to tempermonkey localstorage (GM_setValue).

</comment></comment></comment></comment><comment><USER5> There's a lot of things in this article that I agree with whole heartedly. A few weeks ago I posted this comment [https://news.ycombinator.com/item?id=21650908](https://news.ycombinator.com/item?id=21650908) which shares some of the same frustrations.

Another thing that annoys me right now (and this is a problem of my own making...) is earlier this year I started taking notes with an app on my iPad called Notability. It works great with the logitech crayon stylus/pencil and is useful for jotting down notes when doing online courses etc.

Except I've shot myself in the foot a bit because those notes are now bound to the app. Yes there is the Notability app for OSX, and yes I should have anticipated this problem sooner, but that's beside the point, my notes are locked into the Notability ecosystem. They support this half assed solution to export them as RTF files or PDFs but you lose stuff like handwriting recognition.

One project on my TODO list is to see if I can reverse engineer the proprietary Notability file format, which includes the text recognition and all the things needed to render the lines that make up your notes. I know there have been attempts to do this e.g. [https://jvns.ca/blog/2018/03/31/reverse-engineering- notabili...](https://jvns.ca/blog/2018/03/31/reverse-engineering-notability- format/) I just need to put the time aside to make it work

<comment><USER6> And one more thing was why doesn't my browser store my history - it just the URLs of where I have been but the html, the images, the text etc. Then let me search that.

Someone raised this on here a few weeks ago and it was a gobsmacking moment - my hand flew to my forehead and I realised yes - that would be so useful but the big tech firms find it more profitable to have that data on their hard drives not mine.

<comment><USER7> > Then let me search that.

Like Firefox's Awesome Bar?

<comment><USER8> The Awesome Bar doesn't search the _content_ of pages you have visited.

I believe that's the exciting idea the GP is talking about.

</comment></comment></comment></comment><comment><USER9> This is why I think scraping rights is so valuable.

As the article says:

> Best case scenario is if the service is local-first in the first place. > However, this may be a long way ahead and there are certain technical > difficulties associated with such designs.

> I'm suggesting a data mirror app, that merely runs in background on client > side and continuously/regularly sucks in and synchronizes backend data to > the latest state.

Here are a few premises:

1\. It's a fact that only a small portion of users care heavily about centralizing and truly owning their data

2\. As such, it's reasonable for companies to not focus on exporting data. That's not how they get value out of their data

3\. That being said, companies should at least not punish that small group of users for taking matters into their own hands

Scraping is our solution to this problem, and the least companies can do is allow well-behaved (rate-limited) scraping.

<comment><USER10> I have thought about these issues a lot; especially lately. With regards to being able to scrape your data or just get your data back from 3rd party's, I think that's a losing battle. You need to be in control of your data before it gets to them. Web sites and APIs are constantly changing and sometimes just disappear. This idea of polling for changes seems very brittle and would never be up to date.

What I picture is an a program that you use to store your own microblogs, blogs, contacts, comments, etc. and then you publish to whoever from that app via their API or crawling.

Imagine you just created a new microblog entry. You can now either post to your Twitter, Mastodon, etc. accounts with the click of a button. You would have to poll for replies though and it would be up to you to store them if you wished (you probably want to if you are storing your replies). As an added benefit you could see the replies in one place instead of bouncing between two sites.

The point is, when you create the data, it's yours first. Then if you want to, you can post it other places. Tools like this are abundant for businesses, but we don't seem to build tools for actual people anymore.

<comment> shaki-dora > With regards to being able to scrape your data or just get your data back > from 3rd party's, I think that's a losing battle.

Both Google and Facebook have tools that allow you to export your data in both (usable) JSON and HTML.

<comment><USER10> That's fine but how often do you export? How do you merge it with other exported data? Are you exporting your entire history every time? What about when Google/Facebook breaks their API or get rid of it?

<comment><USER11> That is why we are creating PrivicyPal -- to set a schedule to download and merge data from many sources and monitor over time. Will be adding beta testers soon.

See [https://www.privicy.com/privicypal/about](https://www.privicy.com/privicypal/about)

</comment></comment></comment></comment></comment><comment><USER12> As a counter-point (I'm playing devils advocate because I actually agree with the author in general): at one time I found an old laptop that I hadn't used in quite some time. Excitedly I booted it up and dug around the file system to find ... nothing of interest. I thought there were treasure troves of stuff there but actually there was very little of interest at all.

It made me think about those hoarder reality TV shows and how there is a digital equivalent of that. In some ways I'm glad all of my digital data isn't amassing and following me around. I'm not sure I really care about what I posted to MySpace pages back in 2003 or usenet in 1998.

This is really a matter of ambivalence to me. On the one hand I would love detailed data about my history, even down to the exact GPS coordinates of my location every moment of my life. On the other, I am not sure this data would truly improve my life in any way. It may be that access to such data could make my life worse.

<comment><USER13> I have a location trace over the last year and I am using it for time tracking (how long was I in the office). Also I have a heatmap of all the places I have been that is really nice to look at. Also it has helped me in cases where bike sharing companies said: where did you leave the bike on 27th of September? We can't find it. Or the doctor claiming I didn't appear on the previous consultation and I should pay the appointment out of pocket. One look at the timestamped map and I can recall the situation.

<comment><USER12> That is really cool. Can I ask where you are storing that data? Is it something you cooked up yourself or a service you are using?

One of my back-of-the-mind ideas is to combine GPS data like this with some kind of basic activity-type id. Like "working", "practicing guitar", "socializing with friends", etc. Then I could get reports on how I spend my time.

<comment><USER13> I use the owntracks app with a selfwritten http server that dumps it into a postgres. Then I use grafana and leaflet for visualization. Regarding your activity tracking: I already also track pulse and smart watch movement, but plan on adding more inputs like the travel mode detection on Android (walking, cycling, driving) That together with my calendar data should give sufficient data for time reports :)

</comment></comment></comment></comment><comment><USER14> The Solid Project [0] (AFAIK which is led by Tim Berners-Lee) is made to tackle this exact problem.

It is about defining a standard/protocol to store personal data in a 'pod' and give minimal/granual access of my pod to web apps. Apps are separated to pods, and it allows the decoupling of data & functionality. You can switch(upgrade) from text message to instant messengers without losing any chat history for example.

It also has an advantage that it prevents lock-in, since one can move their data around trivially.

Looks like the OP would greatly benefit from this.

[0] [https://solidproject.org](https://solidproject.org)

<comment><USER15> Has anyone built an actual, used-in-production app on top of Solid yet?

<comment><USER10> Here are some examples: [https://solidproject.org/use- solid/apps](https://solidproject.org/use-solid/apps)

The problem though is the same with all self-hosting: maintaining a server. It looks like you're just on your own if you want to use a "solid" app.

</comment></comment></comment><comment><USER16> Or how about just letting your data vanish, except for items you've made a specific effort to save (e.g. your own original writing or musical compositions)?

Technologists seem preoccupied with creating perfect recollection, seemingly without realizing that people who have this ability innately often find it burdensome.

Given my druthers, I wouldn't retrieve my data from a service, I'd purge it.

<comment><USER17> In either case, you control the destiny of your data, either forcing it to an archive or to /dev/null.

The problem is exactly the lack of such control. Frankly, if you let the data out, there's no guarantee a copy of them does not linger somewhere.

Having companies have an incentive to get rid of your data after some time, for profit reasons, might be helpful. For instance, work email is usually purged promptly after the mandatory retention period expires, so that it could not be used in litigation.

</comment></comment><comment><USER18> This got me thinking about personal data storage. I think step 1 in owning our data is having a place to store it. That storage should be abstracted from the actual provider(s) so we can migrate and/or replicate our data. It should also be available from multiple devices. A personal data warehouse like this should be easy to create, a la 'deploy to heroku'.

It's shocking to me how few people I've met take their personal data storage seriously. Most folks I know treat Dropbox/Drive like a landfill.

<comment><USER19> Agreed. I feel as much as we've seen NASs grow into devices that are vaguely approachable by enthusiasts, there is still a huge amount of ground to cover before they would be the sort of thing that one could easily deploy in their parents' house.

There has been piecemeal progress in swinging the pendulum back from cloud- everything to easier to use edge computing. The Helm email server is one example. The slightly more plug-and-play approach to modern NASs is another. And there are others. But you can tell that the vast allocation of R&D is not going here yet. I do think investors will eventually wake up and realize that user demand for data control means better edge devices and avoiding reliance on the centralized cloud.

What I have envisioned for PAO [1] is federated encrypted backup. I would like to see NASs allow me to basically allocate a percentage of my capacity to various peers to store encrypted-at-rest duplicates of their data. And vice- versa. Basically a federated mesh. No need for blockchain or other crypto-hype nonsense. Just straight authenticated and encrypted file storage.

My opinion is that cloud dominance really traces back to the advent of and self-reinforcing power of asynchonous Internet connectivity. When Internet connectivity was often synchronous (think the very early days of DSL), peer- to-peer networking remained very common. As the number of users using asynchronous connectivity increased, it became reinforced as more services centralized data and content. Peer-to-peer is effectively a relic of the past now. Only today have we started to see some resurgence of symmetric connectivity (e.g., 1Gbps symmetric fiber). I believe deployment of symmetric connectivity will be a decentralizing force as more people realize it's possible to just access your file system and data directly between devices rather than use an intermediary. And as vendors realize this is an opportunity space to offer interesting technology (e.g., the likes of Zerotier) to consumers.

[1] [https://tiamat.tsotech.com/pao](https://tiamat.tsotech.com/pao)

<comment><USER18> _I would like to see NASs allow me to basically allocate a percentage of my capacity to various peers to store encrypted-at-rest duplicates of their data_

This is something I've been thinking about, too. We have so many Internet- connected devices with increasingly cheap storage-- Some universal protocol for distributing data across this network would be really cool. (I understand this could sound like blockchain. I have no horse in that race.)

<comment><USER15> I would worry about the liability implications of such an approach. If one of those peers is storing (say) child porn, and some of that porn ends up on my hardware via automatic duplication, does that implicate me legally in their crime? If it does, does it matter whether it's on my hardware with or without my knowledge? If the police identify my hardware as part of that peer's storage network, is that hardware at risk of being searched and/or seized? That sort of thing.

</comment></comment></comment></comment><comment><USER20> The answer to why the author of this article can't do any of the things they list is simple. It is hard to write the software. The data is there but some program has to use it.

The author asks why he needs some start up to take his data and then trust them to do things. It's because that start up spent a lot of money to build the software to run on their servers and now they don't want to give it away for free.

In my opinion, this is the root cause of most of the issues with our data and control on the web in general. There was a time when the web itself disrupted the centralized networks of the day like America online, MSN, and CompuServe. The centralized services we use today such as Facebook, Google, Amazon, Twitter and others could have never been built unless a permission less platform like the web existed and disrupted centralized platforms. The centralized services we use today such as Facebook, Google, Amazon, Twitter and others could have never been billed unless a permission list platform like the web existed disrupted America online and the others. They had invested a lot in the infrastructure that the web leader replaced. They were extracting rents and controlling the platform.

Today we need something like that but the infrastructure we need to disrupt was built by Facebook, Google, etc. Wordpress and OpenStreetMaps is just one example.

That's what I have been putting my money into for 8 years and open sourcing:

[https://qbix.com/platform](https://qbix.com/platform)

[https://qbix.com/token](https://qbix.com/token)

Take it, use it, it's free. Build on it Whatever solutions you want to manipulate whatever data you want. And perhaps more importantly, host software for entire communities of people who want to collaborate and connect with each other. Not just manage their personal data.

Feedback welcome on how to make it better. We are planning to officially launch later next year to be like a Wordpress of 2020.

<comment><USER21> Hey, I just want to say your post generally resonated with me and definitely made me want to check out qbix. I've also spent years on developing software that I'm giving away for free - most of it anyway, but not the parts that would enable others to offer commercial hosting at scale. My software is "just" implementing standards (ISO and OASIS) though, so users don't have to spend their time on proprietary languages, APIs, markup, protocols, etc. With your initial statement on non-free software by start ups and your conclusion to give it all away, I thought you might be interested in this standard- oriented approach as kind of a middle ground between proprietary vs free/open SW.

</comment></comment><comment><USER22> Interesting to compare to article that came through yesterday about giving up on semantic web, where the whole movement got a sound drubbing in the HN comments.

These are infrastructure problems, they should be treated as such, i.e., maintained by tax dollars.

The failure of the semantic web and the sad state of personal data are primarily failures of the free market to solve these problems, imho.

[https://news.ycombinator.com/item?id=21823869](https://news.ycombinator.com/item?id=21823869)

<comment><USER23> This doesn't seem like a failing of the free market. The free market has already created products that solved this problem, and they are some of the most popular products in history, but people are moving off of them, e.g.:

\- People are moving from Microsoft Office to Google Docs

\- Designers are moving from Sketch/Photoshop/Illustrator to Figma

\- People are moving from Evernote/Text Files/Whatever to Notion

\- People are moving from HTML files to Webflow

\- People have already moved from native mail clients to Gmail

There's a problem here, but it isn't that the free market hasn't solved this problem, it's that people are choosing other features (mainly collaboration) as more important than data ownership.

Note that this is a transition mainly driven by tech people, none of these products have gone mainstream yet (excluding Gmail of course), and the products that offer data ownership are still far more popular overall. But if the mainstream follows tech peoples lead, that won't be for long.

<comment><USER22> > people are choosing other features (mainly collaboration) as more important > than data ownership

If it were treated like infrastructure, we could have both.

Pointing to a product that got replaced by another product doesn't inherently prove anything.

And seriously, who uses Gmail voluntarily? The complete non-existence of a suitable SMTP implementation is a pretty good example of a _clear_ failure of the market to properly allocated resources.

edit - thanks for pointing out Figma though, might have to check that out

</comment></comment></comment><comment><USER24> Pretend we didn't have computers and people wanted to store all the facts in their lives - how many people could manage a library / filing system rich enough to catalog the level of information we're expecting to keep here?

<comment><USER10> > Pretend we didn't have computers and people wanted to store all the facts in > their lives

This is a pretty moot point since if we didn't have computers we wouldn't have things recording a lot of the data in that article to begin with. For example, with location, you could write down where you are every minute of the day, but that isn't very practical. Luckily, we have computers to automate that. Does that mean you should have to give that data away to a third party?

> how many people could manage a library / filing system rich enough to > catalog the level of information we're expecting to keep here

Nobody could do that manually. That's the job of computers. I suppose you could keep a journal and store boxes of pictures and a lot of paper. It would be a pain, take up a lot of space, and take forever to search through. Luckily we do have computers and they happen to be really good at searching. So you should be able to just store this stuff on your computer and have it assist you with owning that data.

Instead what we have is a world where you upload everything to the cloud so someone else owns the data and you have no idea what's happening with it. They also get to choose how the data is presented to you. Since your data is spread among so many companies it hard to get the aggregations mentioned in the article. Usually the only way that happens is companies agreeing to share your data with each other. Most people are okay with this since it's "free".

<comment><USER24> My point is, people's minds aren't caught up to the scale of data, and I don't think there is a technology solution for it. We now have more data and more technology but it's not solving the problem.

Hell I don't always know where I'm going to put all the groceries I take home.

EDIT - to add - too much data isn't really useful. When we are talking personal data collection it's basically a librarian's job, which is non- trivial

<comment><USER25> > My point is, people's minds aren't caught up to the scale of data, and I > don't think there is a technology solution for it.

I totally agree with the general thrust of this argument. I'd like to hear more about use cases for this kind of personally-owned, aggregated data store. Once this article started talking about searching over, say, notes and highlights from articles and blog posts, I started to see specific use cases that seem totally compelling. However, it's not clear to me how this part of the data ownership conversation matches up with the seemingly more principles- driven data ownership conversation.

Theoretically there's nothing stopping you from building some of these more specific implementations (which the author has done--btw those projects looks really cool).

</comment></comment></comment></comment><comment><USER25> Are there any other interesting articles on good use cases for this kind of centralized data ownership. The syndication concept and the discussion about data rights aside, why else might I want to have all the data I ever create or interact with in aggregate like this? The author has referenced a few really interesting projects they worked on, curious if there are more good ones.

<comment><AUTHOR> I'd be interested to know too. One reason I started writing this all up is to harvest more of other people's setups, tools and workarounds -- I'll be sure to link them later.

<comment><USER25> Sounds good, thanks. Cool blog btw, I poked around for a while. Nice work!

</comment></comment></comment><comment><USER26> > (typically) the more something machine friendly the less it's human > friendly.

I believe I have solved some of that problem. Hode (the Higher Order Data Editor), a kind of generalization of graph databases, lets a user enter data in a manner which I believe is as similar as possible to natural language. To encode the fact "cats kill birds" as a "kill" relationship between "cats" and "birds", you just write "cats #kill birds". Relationships involving other relationships can be represented with similar ease, and relationships involving any number of elements. The query language is not much more complicated -- for instance, everything that kills birds would be "/e /it #kills birds".

[https://github.com/JeffreyBenjaminBrown/hode](https://github.com/JeffreyBenjaminBrown/hode)

</comment><comment><USER27> This worries me too - a lot. So much so that I'm working on a video series documenting the creation of a personal photo & video storage system from scratch. My belief is that one way to extricate our data from big tech's catacombs is to spread more widely the skills needed to build out personal data stores.

I'm starting with photos & videos because my own collection is trapped in Flickr, and it may not be long for this world. I intend to build out such a system and publish instructional videos on how to do it as I go along.

Imagine a world in which more or less everyone can do this. It's not such a long shot, don't you think?

</comment><comment><USER28> I can't help but think that this glut of resources we are burning on deep library/platform stacks and spying on users might be better spent on attaching more metadata to all information flowing through our systems. Some provenance might be good.

I've seen a bit of how Boeing tracks parts and doing something simpler but similar for data might be tractable now, except that i think it would take APIs that worked substantially different than conventional code. Except perhaps in Ruby and Node (thinking in particular about htmlsafe tagging in Rails)

</comment><comment><USER29> I actually do share many of the author's thoughts on how maddening it can be to collect specific pieces of data that seemingly should be collectable but aren't because of some stupid reasons xyz. To play devil's advocate, though - Is it really malice, or just that people rarely think about these particular use cases in the first place?

It seems like a lot of the things and products mentioned here, if released by an independent dev or small team, could similarly be overlooked. I can't imagine most of the engineers I know (and I suppose especially not rich megacorps) to really ever consider the .01% of people (the kind of demographic you'd find on HN, I guess) saving ALL browsing history across browsers according to some universal standard or LinkedIn statistics or YouTube text history.

OT-ish: I can see how this would be relevant for most people living well enough, say, like middle-upper class America, who can afford these technologies and to care about the multitude of examples presented, but is there a conversation about how much data we should or need to be collecting (to say nothing of handing off to 3rd parties) at all in the first place?

I've always felt that relying and interacting with less technology (or at least making efforts within reason to) was better for my own quality of life (less tracking, less worrying about posting regrettable stuff, sticking to basic principles like "move more, eat less" instead of obsessively counting stuff on my old MyFitnessPal and Fitbit) - surely I'm not alone?

<comment><AUTHOR> Great to hear we resonate!

This is a valid point, I kind of admit these are sort of first-world problems. But my main motivation for raising this issues in the first place is to learn better, process information more effeciently, have better memory, and this is something I wish to use to work, learn and reason about things that really matter, like climate change or solving poverty, etc.

Regarding tracking less -- I guess people are different, I do know people who are happy to just stick to 'move more, eat less'. For me personally such maintenance is boring, and looking at stuff like workout/sleep data etc really motivates me to learn more about it and keep going. I really hate going for another run but at least I'll have a datapoint after it!

I feel like the actually stressful bit is having to think about tracking. If it was done automatically and you didn't have to think about it, then why not? You'd always find people who are obsessed about doing (or not doing) things even without counting I guess.

</comment></comment><comment><USER30> Isn't, overwhelmingly, the problem the economic model all software is made under? Any attempt to come up with a solution to this that ignores this fact will never treat the root course of the issue.

<comment><AUTHOR> Maybe! But I'm certainly not feeling in capacity to challenge the current economic model :)

</comment></comment><comment><USER31> One reason this 'sad state' exists is because content providers (for example websites, apps, etc) need to generate revenue and the most prevalent method is advertising.

Micropayments would go a long ways to shifting providers away from advertising and towards pay-per-use. Many users would not object to paying a fraction of cent for reading an article - especially if it would enable the provider to remove the tracking and invasion of privacy all in order to make a buck.

</comment><comment><USER32> The reason I started / made cloudcmd [0] back around 2009 is that I saw this coming. My goal was to build a decentralized storage system with search capabilities and smart enough that you could rebalance storage based on cost and convenience.

Is now the time to restart this?

[0] [https://github.com/briangu/cloudcmd](https://github.com/briangu/cloudcmd)

</comment><comment><USER33> I've been using Tiller for a few months:

[https://www.tillerhq.com/](https://www.tillerhq.com/)

Totally happy to pay for someone to maintain connections between bank APIs and a Google Spreadshert. Curious how long they'll last.

</comment><comment><USER34> I agree it's sad, but there's an incentive to keep your data captive. This problem isn't really a technical issue.

</comment><comment><USER35> Wonderful post!

> Why can't I search across watched youtube videos even though most of them > have subtitles hence allow for full text search?

This blows my mind every time I'm on youtube... so much potential, and yet.

> Often, a friend recommends you a book so you want it to add to your reading > list.

Yep. For a while I was collecting them in a spreadsheet. After about two years I've realized it's actually a lot more about the _context_ of why/where/when I added a book rather than it simply existing in a long list. Even though I had "Source" and "Date Added" as columns, I (still) have no way of grouping them by topic cross-referenced with my notes.

Also, the conversation in which I received the recommendation likely has valuable context I haven't included, and good luck deep-linking to a message. (Telegram handles this OK, but Gmail? Or god forbid iMessage).

> Why can't I see what was my heart rate (i.e. excitement) and speed side by > side with the video I recorded on GoPro while skiing?

Another angle I've considered: the past four (text/email/GPS) interactions with (some person) has resulted in higher stress levels ... this is an insight I typically extract from writing about my day. Would be interesting to have it suggested to me. Yes, lots of privacy implications here.

> It's just a matter of regularly fetching new stories/comments by a person > and showing new items, right?

Not sure if you've seen fraidycat [0] and the discussion [1]. Basically, a fetch-and-consume model for blogs, Twitter, etc with frequency/priority levels.

> Why am I forced to manually copy transactions from different banking apps > into a spreadsheet?

Plaid [2] looks promising, but I haven't built anything noteworthy with it yet.

> Why can't I easily share my web or book highlights with a friend?

This is ridiculously hard. I think my favorite solution to date is to copy- paste the whole article into a Google Drive doc and annotate it. Not a good solution, I know.

> I wonder what computing pioneers like Douglas Engelbart (e.g. see Augmenting > Human Intellect) or Alan Kay thought/think about it and if they'd share my > disappointment.

I imagine they would be/are very upset.

0 - [https://github.com/kickscondor/fraidycat/issues](https://github.com/kickscondor/fraidycat/issues)

1 - [https://news.ycombinator.com/item?id=21802952](https://news.ycombinator.com/item?id=21802952)

2 - [https://plaid.com/](https://plaid.com/)

</comment><comment><USER36> Just a quick grammatical note:

> Monzo API only allows to fetch all of your transactions within 5 minutes of > authentication.

The referenced link states:

> After a user has authenticated, your client can fetch all of their > transactions, and after 5 minutes, it can only sync the last 90 days of > transactions. If you need the user's entire transaction history, you should > consider fetching and storing it right after authentication.

So I would think the sentence would be:

"Monzo API only allows to fetch the last 90 days of your transactions after 5 minutes of authentication."

...which actually seems worse.

<comment><USER37> I don't understand. The two sentences both seem to match the referenced link -- the first says that it is only possible to fetch all transactions in the first five minutes, while the second says that after five minutes, you can only fetch the last 90 days of transactions (so, can't fetch all). Are you saying it is worse that you can still fetch transactions after five minutes?</comment></comment>


Dealing with Android fragmentation – experiences from our startup - <AUTHOR> http://blog.instabridge.com/post/57424346106/dealing-with-android-fragmentation-how-bad-is-it-for <comment><USER2> The behavior of the Android Telephony/Signal APIs which OpenSignal depends on varies widely across phones. Many Samsung phones don't implement these well at all, bummer considering they are very popular handsets.

One of the things that makes Mobile development interesting to me is the mobile context, unlike the static context of desktop computing. But being smart in the mobile context requires using sensors, and this hardware boundary is where we see a lot of behavioral differences among Android phones (fragmentation). The existing CTS doesn't do a good enough job here. Its where the fragmentation is the worst - in the interesting APIs.

Also, the fragmentation problem for me is also about Google being able to deliver features to users at a reasonable speed. They can with the Nexus, but the Nexus was and is not likely to ever be a commercial success. Its mainly HW for the Google engineers to dev on, and they make a few more for us.

Bluetooth low energy was announced in 4.3. How long will it take for that feature to arrive for a sizable number of users?

What about the limitations of dvm? For example, its 65k method limit for APKs. How can Google fix this when they can't get everybody on the same page in a reasonable amount of time? Its this kind of fragmentation that hurts.

If you're building some basic CRUD app, don't worry. If you want to make interesting nontrivial apps that touch many APIS - especially HW APIS, that when you have to watch out - and have a very good testing plan.

<comment><USER3> I think several big selling phones had Bluetooth BLE before it was baked into the platform, so it really depends on what you mean by sizable number of users. The fact that it's hardware dependant too, somewhat limits the adoption.

<comment><USER4> They were advertised, but only the S4 and Note2 had BLE that came close to working...

though "Close to working" is as good as I can say for my wife's Note2... it was extremely flakey and loses connection all the time.

</comment></comment></comment><comment><USER5> Interesting that they declare phones with less than 4" diagonal screens to be barely smartphones, when that's the size of all iPhones before the 5...

<comment><USER6> I was recently dragged into a Verizon store by a nontechnical friend to help with buying a second smartphone. I let them browse for a while then asked them what they thought about what they saw. I was actually thinking of recommending an iPhone since they really do offer a simpler interaction model over the rest.

After a couple questions I realized that she couldn't figure out which ones were the iPhones from the Android from the Windows Phone. So I walked around with her to the phones that she liked and answered a few questions about each (storage, battery, if it was a popular phone that sort of thing). I realized that she hadn't even checked out the iPhones at all. They were kept over in a special Apple section of the store, and from a distance looked like the small budget smart phones they had elsewhere in the store. So she had skipped them entirely. All of the phones she liked were in the larger screen sizes.

I took her over to them and the conversation went something like this, "and these are the famous iPhones, give them a try, they're really easy to use." She messed with it a bit, "isn't there a bigger one, like the other phones?" I pointed out that it's designed to be easier to use in one hand, for people on the go. "But it's so small, it's hard to see things on the screen, I have to hold it so close" She pointed to the iPad mini they had there "something in- between that one and this one"

And that was that, it came down to a decision between the Droid MAXX, the Galaxy S4 and the Galaxy Note II. She walked out with an S4.

<comment><USER7> Ooh ooh, if we're playing anectodal data game day I'll pitch in too.

One girl I knew was complaining about how big her phone is and asked me about my 4s at the time so I let her try it out. She decided she'd get the 5 right after it was out because her current phone was "too damn big to text with easily". I pointed out she could install a different launcher/keyboard but her phone had never been updated.

This is the problem with anecdotes. The plural is not data.

I also know a fair amount of women primarily that have iphones because they don't like the giant android phones. But again, could just be my friends, where I live, the alignments of whatever influencing buying decisions.

I'm getting more annoyed with all of these "my anecdote X which reinforces view Y means company FOO should BLAH" comments. They don't really contribute to actual discussion of the technical problems at hand, which are interesting. Hopefully that didn't come off too harsh as its just intended to spark more of a discussion of can we please stop with anecdotes in these stories in future? Pretty please? If you come to the twin cities I'll even buy you beer, or if we're ever in the same spot. I'm not above bribing.

<comment><USER8> You might be reading too much into the comment.

For both your and parent's anecdote, screen size was the primary determinant for this user. I wouldn't be surprised if that's common.

It's a good reminder that many people don't buy a device based on the topics we discuss a lot like openness, lock-in, carriers, and fragmentation.

<comment><USER6> Right. She actually like the way the Windows phone hardware looked the best -- it's colorful and different looking than everything else. But I think she would have bought an iPhone to be honest just based on name recognition _if_ Apple would offer competitive sizes. With the way it was presented in the store, it had as much sales sex appeal as the aging netbook display to one side of it and the feature phone display behind it.

The iPhone 5 just absolutely mystifies me, they had a 4 and 4s for sale there too but unless you hold the phones right next to one another and see the extra row of icons on the screen, they're virtually indistinguishable as products to most people, yet the 5 costs much more. To a nontechnical buyer it seems inexplicable that this extra row of icons costs $100-$200 difference in price. Even digging into the specs, I couldn't really make a case for the 5 over the 4 series without getting into technical specs she didn't care about. Her comments were "the 5 is the new one, but I'm getting price gouged for it, and the 4 is on the way out...and these other phones are really popular too"

In the store, with all of the important context clues that stores put around flagship products, the iPhone section felt like it was selling to a market that was quickly disappearing. Like selling incredibly nice horsewhips at a car dealership.

Even Nintendo sells two sizes of their portables.

</comment></comment></comment></comment></comment><comment><USER9> Dealing with fragmentation? Null check!

I'm seeing a lot of apps that assume too much. They assume I've got GPS. And even if I had GPS they assume it is turned on. Same for Wifi or any data connection. Therefore a lot of crashes occur because of null pointer exceptions.

The simple fix is ofcourse to null check or to use a little more of try and catch.

<comment><USER10> Most apps aren't really useful without a data connection of some sort, games might be an exception but then most of them would need advertising revenue to continue development and ads require an internet connection.

<comment><USER11> I'd argue the complete opposite: that is, most apps that don't work without a data connection aren't useful.

Specifically, most apps should be able to perform their functions off-line. Why? Because mobile data is expensive and unpredictable in a lot of countries. You can't expect that 3G will be on and working, and you can't expect that someone will be inside wifi.

FWIW, anecdata and all that, the majority of people I know in NZ prepay for data in chunks that roll over multiple months, and so they only turn on 3G when absolutely necessary.

</comment></comment></comment><comment><USER12> Android fragmentation - do you want to support 2.3 or not? After that it's about as annoying as responsive design is - possibly less for certain apps.

The challenge is of course backporting an app developed for 4.0 or frontporting an old app in the new API. That is definitely a pain.

</comment><comment><USER13> The biggest problem I see from experience is memory limitations impacting design.

I have a republic wireless android phone that ships with very little free memory until you root it. Have to be Very careful and pick and choose which apps you install based on size. Is it worth uninstalling evernote to fit this new app? Or is your new app worth the $100/month it would cost to switch to a carrier with a more advanced phone where space wouldn't matter?

I also have a 32 gig nexus7 tablet where I pretty much don't have to care about app size. 500 meg game, well OK whatever.

The article advises fooling around with different UI depending on device size. Users actually hate that. I can't stand how baconreader renders differently on my phone and on my tablet, when they should be the same. gmail is another offender. I want the phone gmail on my tablet and the tablet baconreader on my phone (or was it the other way around?) Users can understand screen and finger size issues but can't tolerate arbitrary foolin around with UI design.

<comment><USER14> So, by "memory", you mean internal storage, not RAM?

They are two different things, with their own problems,so it is better to try to be precise.

<comment><USER13> Yes, although I imagine there are devices that are RAM limited.

I have enough ram not to be ram limited, but for example per my "running" tab, tunein radio pro burns up about 20 megs continuously. No idea why and I don't think I'm getting 20 megs worth of battery draining "anything" out of it. Too bad, because when I'm actually using it, its an awesome app. Google Goggles burns a bit less than 6 megs continuously, even if you're not using it, to do... what exactly?

This all costs battery, if nothing else. I'd rather have an extra hour of runtime rather than goggles and tunein starting up slightly faster when I use them.

There should be a configurable android setting where "piggish" apps can ask to start a battery sucking and ram sucking 24x7x365 service, but this specific app is simply not allowed to do so because its a pig-app.

</comment></comment></comment><comment><USER15> "Versions before 2.2 are ancient by smartphone standards and can therefore also be ignored, just like most developers ignore iOS 3.0 and earlier."

In fact, the current Xcode version only enables developers to target iOS 4.3 and above. So even if a developer would want to support 3.0, he would not be able to do so.

<comment><USER16> On the other hand, iOS3 can also be safely ignored because it's a completely insignificant part of the userbase: early june studies showed an install base of 0.1% on phones (0.6% on tablets).

By comparison 2.5% of the android userbase was still on 2.2[0]. Though things look good as it's been falling fast (2.2 was over 8% in february)

[0] [http://www.gsmarena.com/android_in_july_41_jb_now_more_popul...](http://www.gsmarena.com/android_in_july_41_jb_now_more_popular_than_23_gingerbread- news-6507.php)

<comment><USER17> This still doesn't change the fact that phones using 2.2 are ancient and users are probably uninterested in downloading apps, so you can safely ignore them.

It's a high percentage, but mainly because of negative selection. People who don't care about apps bought cheapest Android phones.

<comment><USER18> I wish Google disclosed more statistics than "worldwide active users" for the Android versions.

If they published additional details, for example:

* filtering by country/city,

* filtering by age or demographics,

* "total money spent on apps" instead of active users,

then developers could be more confident in supporting 4.0+ only (and convincing their clients or employers to do so).

</comment></comment></comment></comment><comment><USER19> I only deal with iOS, so excuse my ignorance, however the following statement from the article is puzzling:

"While Android fragmentation is increasing, the development time for developers is not."

Because earlier in the article it states this:

"Whenever we have a new release we spend about four hours testing the new build on all the devices. In total we estimate the extra effort of fixing device or vendor specific crashes to add 10% to our development time."

So it seems a bit inconsistent. If there is more fragmentation, then that 10% development time figure would necessarily have to increase.

I would be interested in the data demonstrating the cost-benefit analysis of supporting the high fragmentation as opposed to just supporting the the OS version with the highest penetration. I'm not arguing either way, but it would be interesting to see the effect on revenue. My hypothesis is that older OSes would likely result in less revenue (than newer OSes), so the question would be if the revenue would exceed the percentage of development cost dedicated to the particular OS in question. I really have no idea.

<comment><USER20> i think they are saying there is fragmentation, but they can effectively group the different fragments into this much smaller list of buckets by the 5 major versions and the 3 screen size ranges. that leads to 15 possibly combinations, ( although i doubt an android 2.2 phone will have a big screen size and so on... ) So until a new major category comes out they still have there 10-15 "groups" of devices to test...

</comment></comment><comment><USER21> > (I use the numbering scheme since the numbers are more precise than the > associated names. "Jelly Bean," for example, refers to both 4.0, 4.1, 4.2 > and 4.3).

4.0.X is Ice Cream Sandwich, not Jelly Bean.

<comment><USER22> Which proves his point beautifully. I have no idea what version the silly names refer to being an iPhone user. I do know them by their numeric identifiers though.

<comment><USER23> They are just fun code names. Nominally textual rather than nominally numeric.

I think they're all supposed to be desserts. Akin to how all releases of OSX have cat names. Until the last one. Combo breaker!

</comment></comment></comment><comment><USER24> While the headline talks about Android fragmentation he is only discussing Android _phone_ fragmentation. If you want/need to support tablets the problem gets a lot harder.

</comment><comment><USER25> What are common vendor specific issues that cause crashes?

<comment><USER26> Crappy OpenGL drivers are a common device-specific problem.</comment></comment>


Microsoft announces new Surface devices - <AUTHOR> https://blogs.windows.com/windowsexperience/2018/10/02/empowering-a-new-era-of-personal-productivity-with-new-surface-devices/#_ftnref5#jRt2FPRKhrfYwbZ4.97 <comment><USER2> Microsoft comes closer and closer to Apple's premium look & feel in hardware.

However $349 headphones might be too much.</comment>


Study: Microsoft Saw Bugs Decrease 60-90% From TDD With 15-35% Time Cost (2008) - <AUTHOR> https://www.microsoft.com/en-us/research/wp-content/uploads/2009/10/Realizing-Quality-Improvement-Through-Test-Driven-Development-Results-and-Experiences-of-Four-Industrial-Teams-nagappan_tdd.pdf <comment><USER2> In my experience, there's companies where it's clear that TDD solves less than hiring competent developers and having _anyone_ writing requirements. Testing requires being able to think in depth about maintenance and verification of software. These are not easy tasks and often get distorted into 100% coverage or else - which really just encourages unmaintainable and usually useless tests.</comment>


Ask HN: What are the projects/apps I can build to learn Go Language? - <AUTHOR>

I am a seasoned Java developer, starting my Go journey. I am sure that doing project is the right way of learning it. <comment><USER2> I built this to learn Go: [https://cryptowat.ch](https://cryptowat.ch).

It was the perfect project because it involves a lot of concurrency and networking. I'd think about projects that involve those two things.

<comment><AUTHOR> That's just awesome! Can you share more details about it? Looks like you are consuming some api and producing the graphs and data realtime.

<comment><USER2> It's a couple of Go services communicating between each other over TCP. Data persists in Redis, and is streamed live to clients over websockets.

I'm planning on breaking it up into smaller services. Go is a great language for SOA.

</comment></comment></comment><comment><USER3> A friend and I built [http://usefixie.com/](http://usefixie.com/) with it to learn Go. Like artursapek's project, Fixie was a good fit because it involved concurrency and networking, and latency was critical. It was also nice that we could leverage a well-developed open source project (GoProxy).

</comment><comment><USER4> I'm not finished yet, but I'm writing a trading bot for Steam so I don't need to manually confirm trades if the offers are exactly what I'm looking for.

<comment><AUTHOR> Wow! Sounds pretty intense.</comment></comment>


Online Reputation Startups - <AUTHOR>

We all know the power of online reputation, AirBnB verifications, etc. Are there any startups out there attempting to serve as a central clearinghouse for collecting and managing individual online reputation? <comment><USER2> Klout?</comment>


HN Feedback: Startup Idea - Sports Twitter Network - <AUTHOR> http://www.sportytweets.com This is a network of twitter accounts I've created.  I'm looking for input and advice on the idea.  The twitter accounts are about 1 month old, and already have about 3000 followers across the network, and growing fast, every day. <comment><USER2> You may get into a bit of trouble using all those logo marks without permission. They are huge sticklers on that stuff.

</comment><comment><USER3> Nice. I was thinking about a similar idea recently (judging from the other sites that have been posted, many have had this idea). I wanted to focus on specific games though. If you go to a forum for a college football team, such as buckeyeplanet.com, hornfans.com, or texags.com, you will notice that for each game they create a "Game Thread". I wanted to build a system for gamethreads, since using phpBB, vBulletin, and other forum software doesn't seem to be the best solution for a series of one sentence commentaries on what is happening in the game. If you could nail down this feature and promote it on message boards for each team, I'm sure people would use it. I think many message boards have trouble managing the volume of traffic and small posts on game day. Good work!

</comment><comment><USER4> I like the concept. I don't think I would ever visit the website again (nothing appealing there). But I did follow for the Titans NFL twitter account. That's a real service and will definately add value, especially if you can aggregate from non-traditional news sources.

Not sure how you can monetize the twitter feed aspects. Neat idea though. Good luck!

</comment><comment><AUTHOR> Thanks everyone for the comments. If you'd like, please follow your favorite sports team, and feel free to provide me feedback via our site.

Are there too many feeds? Does the account update too often or not enough? Are we missing vital feeds?

Thanks! Andy

</comment><comment><USER5> I was tossing this around in earnest with a group of friends late last winter. We essentially were planning what something that heavily resembled ingamenow... now.

Now that someone realized the idea, I'm kind of glad we didn't invest much time in it...

</comment><comment><USER6> See also: <http://ingamenow.com/>

<comment><USER7> And <http://fanchatter.com> :)

</comment></comment><comment><USER8> I'd like to see international soccer scores/news.</comment>


Mark Zuckerberg Doesn't Know What the First Amendment Is For - <AUTHOR> https://www.newyorker.com/news/our-columnists/mark-zuckerberg-doesnt-know-what-the-first-amendment-is-for <comment><AUTHOR> "Facebook is not an anomaly in the American media system—it is precisely the result of rampant profit-seeking, lazy thinking, and a lack of civic responsibility. Of course Zuckerberg tells Buttigieg whom to hire. Of course he sees Warren, and not Trump, as an existential threat. Of course Facebook allows Trump to run false ads. The company doesn't know what the First Amendment is for—and we are not making it learn."

</comment><comment><USER2> to be fair, most people don't</comment>


New Google Translate languages - <AUTHOR> https://techcrunch.com/2017/03/06/googles-smarter-a-i-powered-translation-system-expands-to-more-languages/ <comment><USER2> The neural network translation is so fickle, it can fail to accurately translate a three digit number. It gets worse with larger numbers, with errors by orders of magnitude.

[https://translate.google.com/#en/ja/six%20hundred%20sixty- si...](https://translate.google.com/#en/ja/six%20hundred%20sixty-six)

[https://translate.google.com/#en/ja/eleven%20thousand%20one%...](https://translate.google.com/#en/ja/eleven%20thousand%20one%20hundred%20eleven)

Interestingly I wasn't able to reproduce it with English/Russian translation, maybe they hardcoded it somehow, or maybe neural network for Russian hasn't rolled out yet.

<comment><USER3> I've been using it for English/Russian for the past couple years, haven't noticed any change... it's still pretty awful and makes the same mistakes.

<comment><USER4> They are supposed to transfer to neural translations today might not have transfered over yet. I used to follow a Russian forums because the developer of the software I used was on them. Would be interesting to see if there are any changes to the translations.

</comment></comment></comment><comment><USER5> This title is misleading. The languages Google "added" have already existed; what Google has done is run translation sentence-by-sentence instead of by phrase-by-phrase.

</comment><comment><USER6> Is the algorithm (including e.g. network topology) published somewhere?

Alternatively, is there some open/published work with comparable performance?

<comment><USER7> Well, yeah: [https://arxiv.org/abs/1609.08144](https://arxiv.org/abs/1609.08144)

</comment></comment><comment><USER8> "In addition, the Neural Machine Translation system learns over time and improves, resulting in better and more natural translations the longer it works."

Dumb question (maybe?). How is it learning since it's not getting any feedback on what is the right and wrong answer?

<comment><USER9> It is getting feedback, you can correct its results.

<comment><USER8> But who does that on any scale?

<comment><USER10> Occasionally, I submit my translations/validations via Community page: [https://translate.google.com/community#en/ky](https://translate.google.com/community#en/ky)

<comment><USER11> I would, if they'd not store my contributions locked away and assume ownership. Same with Google Maps, they have a contribution feature which I'd use if you could download Google's map under a similar license to OpenStreetMap.

For Gmail and stuff I get it, but translate and maps are data projects, not software projects, which take hard work by thousands of people to build. It's extremely hard to build a competitor, and so I refuse to contribute to the existing de facto monopoly.

<comment><USER10> I am really hopeful, that some day they will give a link to download your contributed translations. But anyway, this service is one of the ways to help your language to prosper.

<comment><USER11> > give a link to download your contributed translations

Well, that wouldn't really help anyone. They would need to publish all translation data to help anyone make a competitor, which is not in Google's interest.</comment></comment></comment></comment></comment></comment></comment>


An update on Truecrypt - <AUTHOR> http://blog.cryptographyengineering.com/2013/12/an-update-on-truecrypt.html <comment><USER2> > I usually take a pretty skeptical attitude on this blog when it comes to > Internet security. For the most part we do things wrong, and I used to think > most people didn't care. The fact is that I was wrong. If the response to > our audit call is any evidence, you do care. You care a lot.

I used to feel the exact same way the author did initially. It wasn't until this year, between teaching a class on security (whose demand I still can't even remotely fathom) and becoming more outspoken about my concerns around the way we handle security, that I really started to realize that people do care deeply about security. That said, most people don't know enough to positively impact security, but that's a problem of education; it's something I'm hoping to put a serious dent in over the next year.

<comment><USER3> > Education

Make sure to cover responsible disclosure.

<comment><USER4> This is subjective, and it's rather presumptuous to claim one particular philosophy as the "responsible" one. Microsoft rejected this term for that reason, despite the fact that it would benefit them.

</comment></comment></comment><comment><USER5> I cannot explain exactly why but something about this project rubs me the wrong way.

So much time spent doing bureaucratic things, organizing, raising money. Have meetings, setup a board.

Did the authors, the people who wrote the code, that did all that work, receive such ample payment for their role? Should not some of the money raised be offered to the developers? (Some say that they are anonymous I dont know if that is true, but they do take donations on the site) Lets give them 70% of the money raised to work on TrueCrypt and make it even more amazing.

They developers did all that work for free (?) , but now these people have to get paid really well to see if what the devs did is correct.

Seems to me most of the money goes towards hiring a for profit consulting company. Great way to drum up business I guess.

Meanwhile: From the official TrueCrypt FAQ:

Q: TrueCrypt is open-source, but has anybody actually reviewed the source code?

A: Yes. In fact, the source code is constantly being reviewed by many independent researchers and users. We know this because many bugs and several security issues have been discovered by independent researchers (including some well-known ones) while reviewing the source code.

So the reviews and audits have been going on for a long time by many individuals around the world. Anyone can do it.

As far as I know, Linux has never been subjected to a formal audit. it has been gawked at by thousands and thousands of individuals. None of them read the whole thing for sure, but parts.

<comment><USER6> Some people are raising funds to review truecrypt code,meaning they will always be one step behind truecrypt and there are others who are working on truecrypt compatible implementations[1][2][3].

It is my opinion that some of this money will be better off if it went to supporting these independent projects that allows management of truecrypt formatted encrypted volumes without using truecrypt.It is possible to manage truecrypt volumes without using truecrypt binary from truecrypt people.

[1] [https://github.com/bwalex/tc-play/](https://github.com/bwalex/tc-play/)

[2] [http://code.google.com/p/cryptsetup/](http://code.google.com/p/cryptsetup/)

[3] [http://code.google.com/p/zulucrypt/](http://code.google.com/p/zulucrypt/)

<comment><USER7> The money was donated to the Truecrypt audit project. It cannot go to other projects.

</comment></comment></comment><comment><USER8> > And finally, the most exciting news: we've signed a first contract with iSEC > partners to evaluate large portions of the Windows software and bootloader > code. This review will begin in January.

That's huge. I assume you're not referring to having access to the Windows source code, though.

Here's a crazy idea. After the whole NSA stuff, many governments are going to require Microsoft to give them access to the source code, if they want them to continue using it, or buy the new versions of Windows. Any chance you could contact such a government, to allow you to do the audit on their behalf, or to work together with them on it?

That would be a win-win for everyone. They get a team of experts to review the Windows code base, and you get to know everything about Windows. They probably won't be very eager to get Americans to do this for them, though, so make sure you flaunt all of your credentials.

<comment><USER9> > After the whole NSA stuff, many governments are going to require Microsoft > to give them access to the source code, if they want them to continue using > it, or buy the new versions of Windows.

That's interesting to hear. Got any more information on that?

<comment><USER10> Microsoft has made source code available to governments for quite a long time now. However I don't think Microsoft has ever made _buildable_ source code available, nor does it intend to.

<comment><USER11> Can you explain what you mean? How is it the source code if it's not buildable?

<comment><USER12> Think about the largest, most complex codebase you've ever worked on. Then, remove any architecture documentation, wikis, etc. used to document how it's all tied together. Then, remove the build scripts and any lib folders.

Imagine how much fun it'd be trying to get that to build. Multiply the terror that inspires by (at least) 100x.

<comment><USER13> I think everything you say is likely to be accurate. However, given that we're talking about a nation state in an adversarial context, remember the relevant comparison is "Express the pain of black-box reconstructing the Windows build process in terms of ACEs (aircraft carrier equivalents)."

I very much doubt that building Windows is a 10 ACE problem, or a 1 ACE problem, or a 0.1 ACE problem.

<comment><USER14> Where can more info be found on the "ACE" metric?

<comment><USER15> [https://en.wikipedia.org/wiki/Jane's_Fighting_Ships](https://en.wikipedia.org/wiki/Jane's_Fighting_Ships)

Or if you're lazy you just Google up this Wikipedia entry:

[https://en.wikipedia.org/wiki/List_of_aircraft_carriers_of_t...](https://en.wikipedia.org/wiki/List_of_aircraft_carriers_of_the_United_States_Navy)

The USA has ten active _Nimitz_ -class aircraft carriers. Wikipedia claims that the most recent, the USS _George H.W. Bush_ , was completed in 2009 at a cost of $6.2 billion. A bargain compared to the projected cost of the USS _Gerald R. Ford_ , now under construction with a budget of $15.5 billion.

These numbers are ridiculously overlarge in this context, of course, which is the point. Access to the Windows build toolchain is not even a $10M project. It's not a technical problem. You start by just offering the money to the company. A few million dollars, plus a history of enforcing your TOP SECRET security clearances, plus credible assurances that you're not going to launch a competing product, is probably enough to convince the company to just _give_ you what you want. Quietly, of course. No need to spook the other customers.

If that doesn't work, there's always espionage.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER16> This, in a nutshell, is the underutilized value of open source - the ability for the general public to conduct a trustworthy third party audit and validate security claims of software creators.

<comment><USER7> Not really. This is the value of crowdsourcing. The audit itself is being conducted by a professional software security firm (our sister company, as it happens) and their portion of it might only have cost a small factor more had complete source code not been available. It's the money and expertise that are making this possible, not the source code.

(Don't get me wrong: I strongly prefer open source software to closed.)

<comment><USER17> I have a question.

Can government issue gag order to security firm doing audit and prevent them from releasing backdoors or intentional weaknesses they discover?

<comment><USER7> No.

<comment><USER18> I'm surprised you're so confident considering how many experts were shocked to discover how far the US govt had been able to go.

<comment><USER7> Comes from an understanding of how service providers actually get gagged: the courts _actively issue a gag rule_ based on orders _they themselves issued_. That's can't be true of basic science.

<comment><USER19> What if the powers that be decide to "classify" the results/reports? I would guess that would be a big legal hurdle. But if things get to that point, a copy the report would probably show up on wikileaks... I guess?

<comment><USER20> That's not how classification works.

<comment><USER7> It's also not how research works. For something to be restrained from publication, the government needs to know it exists. But with new research results, they can't know what to restrain until it's too late.

They could try passing a law outlawing security research, but: good luck with that.

<comment><USER21> Don't they now know what to restrain? (due to this post, at the very least...)

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER22> Tip for gaining more donations: once they get 501c(3) status, many larger companies have established charity matching programs. Usually it just takes some employee initiative to ask for an organization to be added (/ paperwork verified) and then companies will match donations. In addition, philanthropy departments can be petitioned for grants. Given all the recent news, that might generate a decent amount of grants.

</comment><comment><USER23> This is extremely awesome and I support it. I have sensitive files with tax info and some software keys I like to keep protected. NSA aside I don't want some malware or vulnerability in my system allowing intruders to take my stuff.

<comment><USER24> Can't malware just keylog your passphrase?

<comment><USER23> Sure it could but would it be smart enough to find the crypt file. Hopefully McAfee can help with the key logger.</comment></comment></comment>


Ask HN: How should I handle making money in the App Store? - <AUTHOR>

Hey HN, forgive me if this has been asked a million times before - I assume it has, but I've been Googling for the last couple of days and haven't found much of anything. I've got an iOS app that I've been working on for some time, and I'm finally getting close to ready to push it to the app store and hope for the best. I've decided to make the app free (and iAd-supported), but I'm not really sure how to go about setting up iAd in the first place. I don't mean the technical stuff; that all seems fairly straightforward; I mean the business stuff. Apple wants banking and tax information - should I just use my personal info? Or should I do this as a company, and start a single proprietorship, or a one-man LLC, or something like that? Is there a decent guide to this stuff anywhere out there? What are the things that you wished you had known before you started making money (or at least trying to) on the app store? <comment><USER2> You are already a sole proprietor. That's the default designation in US tax law [1]. I'd at least start a one-person LLC. There are liability benefits, and probably some tax benefits (I'm not an accountant). In my case, I started an S-corp.

Either learn enough about bookkeeping and accounting, or hire an accountant.

I remember having to fax some company info over to Apple. There are online services where (for about $2) you can submit a PDF and have the service fax Apple for you.

You can start by setting up iAds. But in my next update, I'd also add AdMob. iAds fill rate is terrible outside of the USA. Its fill rate is "meh" inside of the USA. AdMob fill rate is about 99%.

Good luck!

[1] [http://www.irs.gov/Businesses/Small-Businesses-&-Self- Em...](http://www.irs.gov/Businesses/Small-Businesses-&-Self-Employed/Sole- Proprietorships)

</comment><comment><USER3> I have registered three developer accounts on the App Store (two for companies and one as an individual). Disclaimer: This was some time ago though.

If you have already registered a developer account, you will have already decided on whether it is for an individual or a company, in which case, your tax/banking need to follow that.

If you want to register a new dev account for a company, you should give yourself a few weeks since you will need to do some back-and-forth paper work with Apple. It's not that complicated but it takes time. Plus you will need to actually do the incorporation stuff for your government.

The following are things that I wish I had done from the start. If you go the company route, I would recommend that you consult a lawyer to get all of the incorporation paper work done correctly, and consult an accountant to learn about your financial reporting responsibilities. It will give you peace-of- mind down the road. Also, be on-top of your accounting. Again, for peace-of- mind down the road. It's easy to put off.

One more tip. Don't rely on iAd alone. Do backfill to something. Admob for example.

Also, have a realistic expectation for a purely ad-supported app. They do work, but require significant activity. E.g., let's say you get $1.50 per 1000 ad impressions (cpm) on 60s ad rotations. How many total user minutes per day do you expect? You can use this to get a very rough estimate.

Best of luck to you!

<comment><AUTHOR> Thanks, this is great stuff! I have a developer account registered as an individual. Does that mean that I can still charge/add ads, as long as I provide my own personal banking and tax information?

Also, yeah, I have no illusions about making a ton of money from this app. It's mostly just something I started putting together while I was learning iOS, and then kept polishing once I was comfortable with iOS because I liked the idea. If I can't get the money stuff squared away I'll probably put it up for free anyway; I just figured that if I'm gonna release it to the world I might as well see if I can get some beer money for doing so.

<comment><USER4> I should also mention from experience with a niche app which has a few thousand users, I ended up making $1.50 in revenue over the course of a year.

I released an update this December, and am now charging $1.99 and doing thousands of percent better, and I haven't put any effort into promoting the app yet.

</comment></comment></comment><comment><USER4> If you start an LLC/LLP/Corp, you will be paying some amount of money in fees and taxes each year to keep it in existence. Where I am in Rhode Island, USA, this is $500 for minimum corporate tax (applies to LLC/LLP as well), and a $75 filing fee for an annual report.

You will also may want to hire an accountant to prepare your corporate / LLC tax return / schedules, especially in your first year.

In short, if you are going the LLC/Corp route, expect to add $1000 per year in overhead, plus at least a dozen hours in time.

I personally would wait (you are already a sole proprietor by default) and then if you start making some decent money look into it again. It is possible to transition an app from a individual to a company account, but there is some footwork involved.

Keep in mind, you will not be reaping any tax benefits of incorporating until you are making actual money. Your decision to incorporate / form an LLC at this point should be driven only by the amount of risk you are personally taking on in potential liability. I would suspect for most free applications, you won't have much to worry about. But if you're sitting on a pile of money and are writing an app that some how might inspire someone to sue you, you might want to think about it.</comment>


Silicon Valley's Secrets Are Hiding in Marc Andreessen's Library - <AUTHOR> https://www.wired.com/2016/09/marc-andreessens-book-collection-explains-silicon-valley/? <comment><AUTHOR> Here are high resolution images of the bookshelves: [http://imgur.com/a/eoDng](http://imgur.com/a/eoDng)</comment>


College economics classes - how useful are they? - <AUTHOR>

I'm a college student who's debating the value of college economics classes. I didn't like the AP economics class I took in high school. It seemed too idealized and non-applicable.<p>I'm wondering what you guys think about the value of economics classes in college, especially for pursuits related to entrepreneurship. If it matters, I'm attending a top 10 school. <comment><USER2> You'll get some subject matter information from the teacher that you'll be required to barf back on an exam. That's useless (as is almost every subject matter I studied in college/law school, in my experience. Statistics, a philosophy course in symbolic logic, and calculus excluded.)

What you gain (if you're lucky) is the meta-lesson of how to think like an economist. Meaning that you'll understand the reasoning methodologies. This will give you an ability to understand real life in ways that mere mortals cannot/do not. I can't say whether "thinking like an economist" is better or worse. But it will give you perspective and an understanding of how large groups of people behave.

Same deal with law school, in my experience. They teach you to think like a laywer. That's the useful part. Hint: If you are trying to figure out an answer, spend your time working out the correct question to ask. Then the answer comes almost trivially.

Second hint: law is binary. The answer is either "yes" or "no". Phrase your question to elicit an answer that is "yes" or "no".

I'm sure the same holds true for engineering and other disciplines. Yeah sure you'll learn some "stuff" but more important you'll learn how to ask a question and solve a problem.

I haven't looked at khanacademy.org's economics stuff. But I'll bet you won't learn the meta-skills there.

Go take the class. And demand a lot out of your professor.

(Disclaimer: econ major in college.)

</comment><comment><USER3> It will similar to your high school class in terms of being idealized and non- applicable.

I think they are interesting courses to take and there are some great conceptual takeaways. I'm not sure how valuable for entrepreneurship economics would be. I would personally go for microeconomics for it's applicability to business. However, macroeconomics is still interesting in a broader (even political and global) sense though.

I don't know what you'd be taking as an alternative to economics, but I don't see why you wouldn't take them. I took both macro and micro in college. I enjoyed the courses and found them useful. As philphodgen pointed out... what you will gain from taking the course is the ability to think like an economist.

</comment><comment><USER4> It is true that modern Economics is not just too idealized but also too mathematical. Economics has math envy toward Physics so they mimic the supposed "rigor" of mathematical analysis to give it a veneer of precision and respectability. That said, there are crucial principles you probably should know (Supply and Demand, Division of Labor, Comparative Advantage, etc). The college classes may be better than your HS AP course as it depends on the teacher more than anything. The intro classes tend to avoid the more advanced and abstract and do try to tie it to real life examples, especially Micro- Economics.

<comment><USER5> I don't know if I would call it envy. It's an attempt to try and model human behavior - as (ir)rational as it may be. Using math to try and get a better understanding seems like a pretty good idea. No model is perfect, but they constantly try to adjust.

<comment><USER4> This is the exact rationale and hope in the field but it a false hope. I'm not opposed to using math but as your comment implies, humans are not atoms or particles like in Physics and the subject requires different methods than the physical sciences. Economics is more closely allied with the methods of Psychology or Ethics, identifying and validating basic principles such as the examples I mentioned which require no math to understand. The proof of these principles are validated via logic and observation of human action not mathematical proofs. This methodology was dismissed as non-scientific early last century and, in the context at the time with very successful mathematical Physics, the mathematical economists took over the profession to counter the charges of not really being a science since there was no math equations.

</comment></comment></comment><comment><USER6> I think taking an engineering economics class would be incredibly useful (something like: [http://ardent.mit.edu/real_options/ROcse_CMI_latest/index.ht...](http://ardent.mit.edu/real_options/ROcse_CMI_latest/index.html)). I'm a fan of the Austrian school of economics (Rothbard, Menger, Mises, Hayek), but micro/macro are good classes to see how to address complex system analysis - though it may be the wrong approach.

</comment><comment> alan-crowe Joel Spolsky has a reputation as a programmer, blogger, and entrepreneur. His recommendation of microeconomics

<http://www.joelonsoftware.com/articles/CollegeAdvice.html>

is striking for its clarity and strength.

<comment><USER7> excellent article!

</comment></comment><comment><USER7> I would say take statistic's classes and read some econ books on the side. Also from my personal experience I found micro to be much more engaging and applicable. This is all my personal opinion of course.

</comment><comment><USER8> you can meet loads of high-fliers on their way up and tag along for the ride

</comment><comment><USER9> not much , khanacademy.org has a bunch.</comment>


HTML5's time is back - <AUTHOR> http://lists.w3.org/Archives/Public/public-html/2011Nov/0011.html <comment> ryan-allen All these specialised tags seem so completely redundant. Why do we need a sanctioned header and footer and time tag? Why wouldn't you just make tag names open ended and control the presentation with CSS? Why on earth can't we use arbitrary attribute names, instead we have to use this clunky data attribute. It's truly ridiculous.

This is how I hear W3C announcements on such things:

"A specialist committee of 500 have reviewed a sample of so-called 'internet sites' as found on the so-called 'internet'. We have noticed that often times people like to have so-called 'headers' and 'footers' in their website. We have decided then that in your best interests that we have officially sanctioned a <header> and <footer> tag. Thank you, thank you. We also have noticed that occasionally people put times and dates in their web pages. Therefore we have sanctioned a <date> tag for your convenience. We are glad to contribute to the evolution of the web."

It is so arbitrary and detached from real world web development. This isn't 'bringing the web forward' in any sense. I don't know what the answer is but it certainly isn't this.

<comment><USER2> If everything were arbitrary then there would be no agreed upon standards and semantics.

By having an agreed upon structure you allow the internet to get smarter. Data mining, search engines, usability, etc...

E.g <phone> tag that is always expected to have a phone number. That way when it's rendered on certain devices it will always have a phone number that can be interacted with.

Sure you can achieve that now with text parsing but there's better ways to evolve the web and that's what standards are for and the introduction to new elements that are semantically accurate.

<comment><USER3> Pardon the devil's advocate question, but to what extent is semantic markup (other than with microformats) relevant to today's web? I mean, how is it used by anyone? I've tried to use semantic markup and it's not really a very effective abstraction for many kinds of data and UX.

<comment><USER4> Ask anyone who accesses the web through a screen reader and they'll tell you it's extremely useful. Not to be flippant, but well structured markup allows a screen reader to:

* Emphasise headings

* Allow users to skip past header sections automatically

* Ignore information unrelated to the content (aside sections)

* Allow users to navigate by heading structure

* Voice certain parts differently - links are an obvious candidate, but screen readers could put on different voices for quotes

* Etc etc etc.

<comment><USER5> Hang on a sec, that's what CSS does for the sighted, we don't expect the HTML to tell the browser how to layout the page, so why are you expecting the HTML to be able to tell a screen reader how to present it.

Semantic web is suppose to elicit meaning from content, not presentation.

The whole concept of semantic markup is extremely fishy to me. On the one hand you're supposed to remove presentation directions but on the other it's supposed to give presentation clues to screen readers?

<comment><USER4> When you build a site with CSS you're over-riding the browser's built in stylesheet. If you don't provide your own CSS then the site is readable without, because the browser lends basic visual meaning to semantic markup.

Think of screen readers as having internal styling. Links are read in a different voice because the screen reader is applying a default audio style to the page.

Audio styling is part of CSS2.1 [1] but isn't widely supported or widely written (classic chicken/egg problem). If you want to you can (at least theoretically) over-ride a screen reader's default presentation of your markup.

[1] <http://www.w3.org/TR/CSS21/aural.html>

<comment><USER5> I know, although I disagree that almost any site without its css stylesheet is really readable, but the original question was how is semantic HTML useful.

If usability is the only answer and it's doing precisely the opposite of what semantic HTML is _supposed_ to achieve, namely separation of content and presentation, it appears to me that we've not really decided what semantic HTML is, it just sounds kinda cool and has a happy side effect of making it easier for screen readers to parse.

So many of these new tags seem too narrow, a victim of the old fashioned thinking of the w3c where everything, to them, is still a document.

Why use article instead of div? Will it really make mining easier or will you get loads of false positives? If my site creates hourly weather reports, are they articles? Why can't I define my own tag of weatherreport? Is there any point?

To quote ryan's original comment:

 _This isn't 'bringing the web forward' in any sense. I don't know what the answer is but it certainly isn't this._

<comment><USER6> HTML shouldn't need CSS + JavaScript to incorporate basic functions. The point of these new elements is to DEFINE semantics that we can take advantage of TOMORROW. This goes far beyond screen readers.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> I'm so glad about this response from W3C (I do criticize the general bureaucracy of W3C though). In many aspects, I felt that the recent moves of the HTML living standard were much like that of XHTML2:

\- Why do keep the redundant <a> element while we can add the href attribute to all elements?

\- Why do keep <br> while we can generalize the line structure via <nl>? (NB: I'm not sure about the exact element name)

Is removing <time> in favor of <data> really different from these statements?

<comment><USER8> The problem with HTML is that it tries to simultaneously solve two completely different problems:

1) How do I provide data in a meaningful, semantic, machine-readable format?

2) How do I display data in a web browser, on a phone or in print? Do I use columns? What type of navigation options should I provide to the user in this instance?

XML solves (1). XSLT/HTML (as a UI layout description format only) can solve (2).

Attempts to merge the two concepts together will end in certain failure.

<comment><USER9> Yes, certain failure. That internet thing will never amount to anything :)

There's a saying about the perfect being the enemy of something, can't remember it though.

</comment></comment></comment><comment><USER10> Can someone translate this from bureaucrat-ese to English for me? I really don't understand what this is saying and what its importance is.

<comment><USER11> Dear Mr. Editor,

You have until next Tuesday to put the <time> element back in the spec. If you don't, we'll do it for you.

Lots of love,

HTML WG Chairs

<comment><USER10> Ah, ok. I parsed the title as meaning "HTML 5's time has come again".

<comment><USER12> I did same, and was left here thinking "Where the frak have I been - when was HTML5 on hiatus from being current?"

</comment></comment></comment></comment><comment><USER13> I like the followup message:

 _I respectfully request that the <time> rollback happen at 2am on Sunday._

[http://lists.w3.org/Archives/Public/public- html/2011Nov/0012...](http://lists.w3.org/Archives/Public/public- html/2011Nov/0012.html)

(Probably only funny if you have had to do timezone handling in your code before)</comment>


Xpra: (GNU) screen for X applications - <AUTHOR> http://partiwm.org/wiki/xpra Xpra is 'screen for X' -- it allows you to run X programs, usually on a remote host, direct their display to your local machine, and then to disconnect from these programs and reconnect from the same or another machine, without losing any state. (see the FAQ: http://partiwm.org/browser/README.xpra also, for more) <comment> smoof-ra freaking awesome.</comment>


Students are demanding the facts about coding bootcamps - <AUTHOR> https://techcrunch.com/2016/06/25/students-are-demanding-the-facts-about-coding-bootcamps/ <comment><USER2> People should not pay to learn programming. There are so many free materials online.

The best way to learn is with documentation, which is free, and examples.

A lot of coding 'lessons' are extremely dull, they show you how to make Hello World! then they tell you to try it and oh look, it works! Of course, or it wouldn't be in this book right?

Sometimes documentation is lacking. Sometimes examples are not readily available. These things should be solved. People should pay to fix these things, if they're gonna pay for anything.</comment>


Finding a Murderer in a Victim's Eye - <AUTHOR> https://daily.jstor.org/finding-a-murderer-in-a-victims-eye/ <comment><USER2> I was _really_ hoping this would be a paper describing how to "enhance" the image a la the CSI TV show[0]

[0]: [https://www.youtube.com/watch?v=3uoM5kfZIQ0](https://www.youtube.com/watch?v=3uoM5kfZIQ0)

<comment><USER3> My favourite parody of csi style enhance.

[https://m.youtube.com/watch?v=2aINa6tg3fo](https://m.youtube.com/watch?v=2aINa6tg3fo)

<comment> King-Aaron That's fantastic.

I must say though, this whole discussion makes me think of Will Smith here: [https://www.youtube.com/watch?v=tw5kRvCZ_Ng](https://www.youtube.com/watch?v=tw5kRvCZ_Ng)

</comment></comment></comment><comment><USER4> I remember seeing this in the Wild Wild West[0] movie from 1999. It's curious seeing the history behind it.

[0] [https://www.youtube.com/watch?v=6LFHZDXhUI8](https://www.youtube.com/watch?v=6LFHZDXhUI8)

<comment><USER5> Seriously though, doesn't it bother you that the primary gag is to repeat the same five words at each beat in the rhythm of that scene? It's such a lowest common denominator bit. This is sort of an odd crux to juxtapose, as a premise of art immitating life.

On the one hand, you have a plot device delivering some interesting sci-fi exposition to move the story forward, and on the other hand, you've got this squeamish every man wringing his hands from the vantage point of some supposed moral high ground, each ignoring the other, distracted by their own fixation.

Really makes you think.

</comment></comment><comment><USER6> Identifiable Images of Bystanders Extracted from Corneal Reflections

[https://journals.plos.org/plosone/article?id=10.1371/journal...](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0083325)

</comment><comment><USER7> Also used as a plot device in one of the amazing _Lord Darcy_ stories, by Randall Garrett. A detective and his Watson in a medieval setting where magic is not particularly magical, but actually forensic science. They obtain a very clear image of the murderer from the victim using a spell, but it doesn't resemble any of the possible perpetrators.

</comment><comment><USER8> I recall many years ago reading an article about how, with sufficiently powerful computers, it would be possible to reconstruct what a person in a photograph is looking at by enhancing the pinpoints of light reflected off their pupils.</comment>