Android platform engineer on application architecture - <AUTHOR> https://plus.google.com/105051985738280261832/posts/FXCCYxepsDU <comment><USER2> Unfortunately, the contents of this post are no surprise to any seasoned Android developer. It's very clear the Android engineers haven't given a lot of thought to how applications are to be developed.

I'm all for them not enforcing one methodology over another. However, in order to build an Android application (following whatever pattern) we do need core building blocks that have been well thought through on how they will be used.

In general performing Async tasks (in particular networking) is excruciatingly convoluted on Android - well, at least to get it right. _Many_ inexperienced Android developers simply get this wrong, and quite frankly I can't blame them. Such a common task should be trivial to perform and not require hours of background reading.

The issue largely comes down to the fact the "system framework" has been designed to be simple (supposedly[1]) for system framework maintainers, and not developers writing applications. Destroying and recreating activities (so you can load different resources) may be simpler from AOSP's perspective, but it's not at all logical from the perspective of an application. Sure, there are several ways to work around this, but they're all stupidly convoluted.

 _Frameworks should be written for consumers, not maintainers._

[1] Okay, the system frameworks definitely aren't actually well designed / simple to work with for framework maintainers either, that was just the (missed) goal. If you want to see world's worst state machine (a stupid amount of booleans), check-out: [http://stackoverflow.com/questions/17847775/nested- fragments...](http://stackoverflow.com/questions/17847775/nested-fragments- illegalstateexception-can-not-perform-this-action-after-onsa)

<comment><USER3> Not only didn't they give a lot of thought on how apps are to be developed, they made a bunch of really shortsighted design choices that continue to harm Android as a platform today.

First is the stupidity of Activities. The basic idea was that 1 screen = 1 activity and we can remove all non-active activities from memory and restart them as needed. This was clearly designed with an eye on memory constrained systems. Nowadays even the cheapest phones have shitloads of RAM but we're still stuck with this retarded model. Worse, now we have support for tablets and phones with big screens the 1 activity per screen no longer makes any sense anymore either.

Second, the decision to go with Java as a language. After they make us jump through ridiculous hoops to fit our applications into their stupid activity model just to save RAM, they go waste it by chosing a GC'd language, combined with the performance hit you take for the VM this this has really hurt Android app performance. In return we get an OS that can run in multiple CPU architectures, unfortunately no one runs it onanything other than ARM. The 0.01% of non-ARM devices can't run half the software anyway because they included ARM JNI components to work around the slowness of the VM.

But, I hear you say, you can supply JNI binaries for multiple architectures. Sure you can, if you can fit them in your APK, which you can't because your APK is limited to 50MB because it has to be downloaded to a temp partition before installing. This brings us to the thirds major fuckup: the file system on Android, or specifically the fact that there are several filesystems instead of just the one. But hey, at least now we get to support removable storage, which is both a PITA for developers, extremely confusing for end- users (why can't I install this app when I have gigabytes of free space on my SD card ?) and after all that trouble hardly any phone has removable storage anymore.

This seems to be the big problem with Android's development: a complete lack of vision by Google. They never designed for tomorrows hardware, instead they designed for last years hardware. Android is an OS built around hardware limitations that no longer existed by the time it gained momentum. It's like they were completely surprised by the speed at which mobile hardware evolved.

<comment><USER4> > First is the stupidity of Activities. The basic idea was that 1 screen = 1 > activity and we can remove all non-active activities from memory and restart > them as needed. This was clearly designed with an eye on memory constrained > systems. Nowadays even the cheapest phones have shitloads of RAM but we're > still stuck with this retarded model. Worse, now we have support for tablets > and phones with big screens the 1 activity per screen no longer makes any > sense anymore either.

Shouldn't fragments solve this?[0]

[0] [https://developer.android.com/guide/components/fragments.htm...](https://developer.android.com/guide/components/fragments.html)

<comment><USER5> Uh, fragments introduced just more of the same problems. A fragment in an activity has an insane, confusing lifecycle, then if you nest them, you basically summon Cthulhu. The more experienced part of the Android dev community is slowly transitioning into using views instead of fragments.

</comment></comment></comment></comment><comment><USER6> Eschewing the traditional application architecture has made for both a terrible development experience and poor quality applications in general. The application lifecycle is convoluted with even the Android developers joking about how confusing it is. Everything about the model has made it far more difficult to port existing software to, and native development for me personally has been an absolute nightmare. No clean exit strategy for applications, half the methods in the lifecycle don't even get called, etc. I can't imagine the completely naive and oblivious thought process that led to the mess they created. Good thing Android is backed by Google though, so no matter how terrible it is it'll still be popular.

<comment><USER7> I agree. Android architecture is a mess. Activity life cycling is actually impossible. There are certain actions the OS takes that will cause crashes no matter how well architected the app is.

My main quibble with Android is that since everything is tied to the activities it is impossible to build an MVC style app. You are better off using the NDK or an engine. Then you can build your software correctly.

<comment><AUTHOR> Do you mind expanding on some of your statements? Especially:

> Activity life cycling is actually impossible. There are certain actions the > OS takes that will cause crashes no matter how well architected the app is.

<comment><USER7> I work on a very popular app in Latin America. That means we get to experience every single edge case in the eco system. My favorite quirk has been that for some reason I will never understand when an app goes into a full screen ad it can make your main activity eligible for GC. That means that you don't get any of your Activity life cycle events. Just a lone finalize and your app is gone. Then when the video ad is done you get a truly spectacular crash!

Sadly, I have a half dozen different ways activities have found to explode themselves. It makes me sad!

<comment><AUTHOR> Sounds like the video ad is a new activity? Since it's full screen, your main activity is no longer visible so you should be seeing onPause and onStop fire. Are you saying you don't see them?

Also, I'd imagine video ads are also quite memory intensive, so if a low-RAM device is playing one I wouldn't be surprised if Android has to kill some processes (like your activity that was sent to the background) to make space.

Perhaps you should look into what these video ads are doing and what sort of resources they eat up?

</comment></comment></comment></comment></comment><comment><USER8> One thing platform engineer should do is to rotate into app engineer role and experience it for a couple month at least, without doing it you don't really understand all the pains the app engineers are suffering due to platform design. At very least you can make your doc so much better by knowing which part you absolutely want to cover in details.

This is especially true for Android platform team, to be honest.

<comment><USER9> What a platform engineer needs to do is to listen user's feedback early and often in order to improve the API. An API is a product,and like every products there should be usability concerns. I wouldn't say Android SDK is bad, it's just so convoluted it doesn't feel like it was written in this century.

It's like someone wanted to impose something regardless of its usability, a lot of design decisions seem to have been taken unilaterally because of one engineer's ego.

</comment></comment><comment><USER10> After developing a few Android apps as an agency dev, we settled on a common structure that helped keep our code relatively clean.

Each activity would have an associated fragment, the idea was that the activity would be responsible for loading the fragment as the current view and other lifecycle functions as well as being the entry-point for any external data-loading (through creating and calling of services).

The fragment would initiate with a reference to the parent activity as a context, and would be responsible for view-centric things like displaying content and adding event handlers, which would more often than not use the context reference to call some remote API.

I suppose you could think of it in terms of Model/Controller/View-Controller but I'm sure there is a more correct model for thinking about it (and it probably is more straight MVC anyways).

The problem I've found with Android is that Google has created almost it's own language when it comes to Android app architecture, which creates a lot of confusion, especially for developers who just assume that all frameworks use the MVC concept (laugh if you will, but this is a thing).

Even though google is quite behind when it comes to the app market, I think there is still time for them to come out with a new set of documentation which outlines the "correct" (the "Google-blessed" one at least) structure for Android apps so that we as a development community can start moving away from the "everything-and-the-kitchen-sink goes in the current activity" style of Android development.

<comment><USER11> What you described sounds a lot like what I used to do for years. It works fairly well, and you can tack a headless Fragment onto it for things that you don't want directly connected to an activty's lifecycle (rotating the screen mid user sign-up shouldn't restart the HTTP call, etc).

This works well, and I'm at the point where I develop it without thought while creating new apps, but I've been playing with a less constrained DI flow instead using Dagger 2.

The more business logic and multiple copies of shared UI elements in different Fragments are directly connected to their parent Activity, the harder it is to re-arrange and re-use them. I've been working on migrating all core elements of our application into a shared library project, and that would've been impossible with a closely tied codebase.

<comment><USER12> > rotating the screen mid user sign-up shouldn't restart the HTTP call, etc

This was the very first thing I struggled with when making my first (and to be honest, only) Android app. I was making an HTTP call in a background thread and when the screen rotated, the app crashed. Making this kind of basic functionality harder to program is such a big turn-off for the platform, and seriously lowers my confidence in the correctness of what I've written.

I ended up writing this gross code ([https://github.com/bschwind/skritter- android/commit/6fdffab1...](https://github.com/bschwind/skritter- android/commit/6fdffab111629cb79d732789b83a088e9c335eea)). That's probably not the recommended way to solve it now, or maybe it is. I don't know because everyone seems to have their own special solution to the problem, and we're lacking solid, idiomatic solutions to basic problems an app developer faces.

For the small Android work I do now, I just stay in my own little NDK world.

Edit: It also seems I was pretty frustrated with the android API at the time, lots of colorful print statements :)

<comment><USER11> I used to use AsyncTask way more, I now stick to other ways to doing background processing. What you have is pretty close to what I described as a "headless fragment", it's a pretty common (but ugly, imo) paradigm, from the looks of it you either made something similar to or followed this: [http://www.androiddesignpatterns.com/2013/04/retaining- objec...](http://www.androiddesignpatterns.com/2013/04/retaining-objects- across-config-changes.html)

Nowadays I've pretty much drunk the RxJava kool-aid. For this kind of thing, I would make the HTTP method exist inside of an Observable which would be observed on a shared single instance Looper provided via Dagger.

<comment><USER12> Yep! I think that's the exact article I settled on after running into that problem. It's been 2.5 years since I wrote that code and I haven't done any significant Android programming since then. Next time I do, I"ll give RxJava a try.

</comment></comment></comment></comment></comment><comment><USER13> I worked with Dianne fifteen years ago, and she was good. However, hearing from others who were involved in early Android, and then left in disgust, the designers who set out the initial form did boot have extensive application framework/developer experience, and we're still hiring to this day because of it.

Let's say it all together: Implementation inheritance is a mistake, and interface based design is a clearer option.

<comment><USER14> It's not just that. Hackborn spent many years denying that Android was janky as fuck, or making excuses for it (e.g. [https://plus.sandbox.google.com/105051985738280261832/posts/...](https://plus.sandbox.google.com/105051985738280261832/posts/2FXDCz8x93s)).

<comment><USER15> You must be mistaking me with someone else.

<comment><USER14> Deleted, then. Perhaps I have conflated the memory of being at javaone and being told all about the wonders of javafx, and the contemporaneous wonders of "filthy rich clients".

</comment></comment></comment></comment><comment><USER16> This makes my head spin.

Plenty of examples on developer.android.com use Activities, Services, BroadcastReceivers for things that go beyond what Dianne Hackborn is recommending here.

It's like the Android SDK team want to have their cake and eat it too:

"We've built a tremendously powerful and useful SDK. It really helps you build great apps quickly and easily."

vs.

"Oh, and, by the way, be careful not to misuse our main components. They were never designed to help you build great apps quickly and easily."

<comment><USER17> Well, d.android.com exists to showcase the core apis. They can't possibly show you how to create an app with 3rd party components on top of a single activity. Just take the bear necessities from there add your own stuff on top.

<comment><USER16> I guess I've been reading that documentation from the wrong perspective. I thought they were trying to show me the right way, or at least, a good way, to create apps and add features to them.

</comment></comment></comment><comment><USER18> The fact that an Android platform engineer doesn't know of design patterns beyond MVC, and only knows that vaguely by reputation, explains why it's so difficult to build well designed apps for Android.

Yes, technically the platform APIs don't "care" about what's going on in an app. But when the APIs are designed in such a way as to (inadvertantly) work against MVC or other design patterns, they make developer lives more difficult. And you can expect that to happen if your platform engineers aren't familiar with these concepts.

<comment><USER19> Actually I'd attribute the difficulty to the culture of "Java-ism" that emphasises huge, complex, design-pattern-filled "extensible" architectures which happen to be extensible only in very specific ways.

I wanted to take up casual writing of Android apps as a hobby, mainly because I think the little portable computer I keep in my pocket could become so much more useful, but the unusual API combined with the complexity and general baroqueness of the architecture was a big turn-off. I don't get the whole "design patterns" thing either, but I do have extensive experience with writing Win32 apps, and a little bit of X (which is not all that different.)

<comment><USER20> 26min into the 2014 Google I/O Android fireside chat: [https://youtu.be/K3meJyiYWFw?t=26m](https://youtu.be/K3meJyiYWFw?t=26m)

Q: So I was wondering if you guys at any point considered the support -- the official support -- of the Scala programming language. I'm asking this question especially now that we all saw that Apple released Swift after 4 years of work. And I think that Swift allows for things for iOS developers that we can't do in Java with the Android SDK. So my question was: have you ever thought about it before -- the release of Swift? Have you thought about it after that? Is there any plan?

Android answer 1: Well, so, I like Cocoa, but Objective-C is based on C, which is 40 years old, so I think Apple sort of had to, like, update it a bit. Umm, so, yeah [laugh], I don't know. Scala -- I don't know. Anyone want to take that? [Android team laughs]

Android answer 2: [loudly] No. [room laughs, claps] Alright, seriously, it's -- Java is the programming language for Android. I don't really think there's a lot of benefit to -- the entire framework is built around the Java programming language and I don't really -- I don't think there's much benefit for us to try to support another whole -- another different language.

Person from answer 1: What about Swift? [room laughs]

Person from answer 2: [jokingly] I'll think about it.

Person from answer 1: [mockingly] It's got optional semi-colons.

Person from answer 2: You have the NDK so you can throw something there. But the NDK -- it doesn't have access to the framework, and that's kind of the challenge. If you want to do a different language, if you want to have it as a first-class language equivalent to the current framework you've got to have either a whole new framework or some bindings to it, which is going to make it a really bad experience because it's going to be Java and this other language.

Q: I want to work with Scala because Scala is compiled to the JVM, so it can run on Android. So there are currently ways to do that -- to work with Scala -- but it's using third-party libraries. So you have classes -- that's what I meant by people to wrap their Java classes, but then it becomes tricky to integrate that cleanly with Android Studio. That also evolves quickly. I'm not sure if it's as complicated as creating a whole new language like Apple did, but my question was just if you guys were considering any...

Android answer 3: From the tool perspective, probably later this year or early next year, there are some changes going on inside Gradle right now that will make things easier to you to just have a module that's originally written in Scala that you compile down to bytecode and then we can just dex that and package that with your application. But it's not going to help you access the activity API through Scala code. It would be -- if you have some code that -- your business logic or whatever that you want to write in Scala because it's easier for you, you should be able to do that. Because as you said, it's compatible. It just generates regular bytecode and then just dexing it. But that's very different from saying "you can code Android in Java" where you have access to all the framework APIs.

<comment><USER21> I gave up on watching those Android Fireside chat sessions because of the horrible jokey way in which so many valid concerns end up getting dismissed, every time.

<comment><USER16> It's been interesting to see the emergence of the Android Developer-Advocate- Comedian in Fireside chats and at Google I/O sessions.

Maybe some of these folks are bored with their jobs, or just tired of trying to do the impossible: convince the developer community that the Android SDK is a great and well designed thing that makes creating great, robust, secure apps easier and faster.

"We know we can't give them world-class, visionary, insanely great engineering, but we can try to give them something entertaining."

Bread and circuses ...

</comment></comment></comment></comment></comment><comment><USER22> Disclaimer: I am not an Android developer.

I cannot seem to reconcile these two statements:

> Should you use MVC? Or MVP? Or MVVM? I have no idea. Heck, I only know about > MVC from school and had to do a Google search to find other options to put > here.

And

> In Android, however, we explicitly decided we were not going to have a > main() function, because we needed to give the platform more control over > how an app runs.

So this person claims to not know or care about design patterns; but then explicitly enforcing a design pattern?

She does say:

> Android could feel like it has strong opinions on how apps should be > written. With its Java language APIs and fairly high-level concepts, it can > look like a typical application framework that is there to say how > applications should be doing their work. But for the most part, it is not.

But after reading through the post, it does seem like a particular set of design patterns is being propagated.

<comment><AUTHOR> Seems many people are missing the overall point of the post: The APIs Android provides that are listed here are intended to be used as the glue between your application and the rest of the system. However you want to build your application from that point on is up to you.

In particular, Android Activities aren't simply a main() function because they are designed from a multitasking standpoint; users are going to be going in and out of apps, changing the state of the device (such as rotating the device), etc. In other application environments, you might register callbacks to fire when the window is hidden or gets resized. In Android those callbacks already exist and are set in stone to encourage apps to behave as a good citizen on a potentially resource-constrained mobile device. But as the post details, you don't have to do anything beyond properly managing heavy resources (like camera access) and saving and restoring state at the appropriate points if you don't want to.

</comment></comment><comment><USER23> You need only look as far as the large collection of flags involving the Intent object to realize how badly things went wrong.

<comment><AUTHOR> Most Android developers I know (and even some iOS developers) consider the Intent system one of Android's diamonds in the rough.

Could you perhaps explain how you would've designed it?

<comment><USER2> The _concept_ may be a diamond in the rough i.e. co-operative multi-tasking. The implementation, not so much.

<comment><USER24> Speaking as someone who has been writing Android software for like 4 years now as my job (but who has been in the industry for much longer than that, as a 42 year old who has programmed in just about every major environment), I agree with what you wrote here.

Intents are great in concept and they work out well for users, but the interface/API for using them to do anything can quickly become painful if you go even a step off the beaten path of "perform this operation with these parameters".

A common recurring theme with Android (and this applies to all the major bits -- Activities, Fragments, Services, etc) is that you'll easily hang yourself if you don't have a really clear picture of how they actually work but you attempt to use them. Android almost completely fails, IMO, when it comes to the concept of "keep the easy things easy". There's a huge upfront cost to understanding the major components before you can start really using them at all safely, in my experience. As a bare minimum, if you don't have an intimate understanding of the Activity and Fragment lifecycles and a full understanding of what needs to be on the UI thread when (and when different callbacks from the OS are/aren't on the UI thread) you are likely to create a terrible mess of an app that kinda sorta works on most devices, usually, but is broken in some fundamental ways you aren't aware of.

I don't personally know any day to day Android developers who haven't spent a considerable amount of time in the AOSP code trying to figure out exactly how some of these systems work because failing to understand them internally combined with lack of any sort of clear direction from the platform team (extremely open case of there being more than one way to do anything, which makes authoritative answers to the question "How should I do this?" nearly impossible without a long list of caveats) is the path to writing Android code which mostly works but has some fundamental mismatches with the operating system and becomes a complete mess to maintain.

</comment></comment></comment></comment><comment><USER25> The great "Advocating Against Android Fragments" post on the Square Corner blog by Pierre-Yves Ricau highlights how horrible the Android lifecycle is.

[https://corner.squareup.com/2014/10/advocating-against- andro...](https://corner.squareup.com/2014/10/advocating-against-android- fragments.html)

Also, if you are doing any substantial Android development, you are bound to run into many issues (say, for instance, it's horrible soft keyboard management system). Hackborn can be found on many of these issues trashing developers asking those questions.

[https://groups.google.com/forum/#!topic/android- platform/Fyj...](https://groups.google.com/forum/#!topic/android- platform/FyjybyM0wGA)

Non productive vitriol is the norm for this person.

</comment><comment><USER26> Android noob here. I've been teaching myself Android programming for the past couple of months and I've found it overall a messy and unpleasant experience for the reasons many of you have already mentioned. So much, in fact, that I'm looking for alternatives.

So what's the best option?

Kivy? Python is slow, adoption seems non-existent, not suitable for every kind of app

Cordova? Web apps are slow, JavaScript sucks big time

Qt? Google discourages use of the NDK for developing apps, can't figure out anything from their crappy website

Xamarin? C# doesn't look like a significant step up from Java, from Microsoft

Delphi? Proprietary, Windows-only and costly

Clojure/Kotlin/Scala? It seems you need to be a master Android developer to use them, still doesn't get you out of the horrible Android architecture.

<comment><USER27> Android is a messy process. Since you've effectively implied that you're excluding all of the options, you only have message Android.

Web apps don't suck that much as long as each screen does one thing well. Xamarin is for getting a slightly more elegant language (since Java 1.8 isn't supported) into the mix while providing a reasonable cross-system platform. Clojure/Kotlin/Scala do require you to master Android first because they are built on top of it. Each provides is own abstraction, even if it's just a language.

In the end it's about trade offs and acceptance. Raw Android is like working in a better assembly. It's got decade's old abstractions. It's powerful, but annoying. For everything else, there's the complexity of working with the various abstractions over a system that does not want to be abstracted.

</comment></comment><comment><USER28> If this is so terrible, why isn't there some Java framework on top of this that's designed to make sense for developers?

<comment><USER29> They do exist.

Check out rxjava from netflix, dagger and butterknife and okhttp and retrofit and picasso (and and....) from square, MVP libraries, kotlin and anko from jetbrains

What is truly holding back many developers is the above cargo cult that the path mentionned in android.developer.com is always the right one.

There have been some harsh comments on Dianne Hackborn, but I find that she is actually doing us a great service here by dissiping this myth. I find she does a great job at explaining what is the scope they really care about, and the one where you are free to discover better alternative that do not have the "Google" label on it.

<comment><USER30> I agree with your last two statements, but the idea the RxJava and Dagger make things _easier_ to understand for developers (I'm inferring this relates more to newcomers to Android as they are the once which have a hard time grokking this stuff) is a joke. RxJava and Dagger only increase ramp up time and the amount of stuff to learn, regardless of how useful they are. If someone came to me confused and I pointed them in that direction, Im sure they would get lost in the woods somewhere and Id never see them again.

<comment><USER29> Fair point. They make things simpler, not easier

</comment></comment></comment></comment><comment><USER31> Rails is built on top of Ruby language, designed to be opinionated, following MVC. You like it, you use it. You don't like it, try another framework.

Using the same analogy, Android is build on top of Java language, is 'opinionated': instead of designing with Model, View, Controller components, it's designed with 'unusual' components (as in not following any classic architectures) e.g. Activity, Service, ContentProvider. You like it, you use it. You don't like it, unfortunately you still have to use it. The real issue here is a lot of developers depend on Android framework, and many of them seem to struggle with the way it's structured, with no direct replacement (not taking into account hybrid frameworks here).

The way I see it, architecture is an opinionated topic. Sure classic architectures like MVC, MVP, MVVC are proven ones, but they serve as a guideline at most. There is no right or wrong if you follow this and not follow that. You are not happy with any of them, you make your own architecture/pattern. Hell it's opinionated, you are allowed to have an opinion here! If you think MVC or some other pattern is the right way to go, then the author has the right to think her design is the right way. It just keeps going on.

<comment><USER32> > You like it, you use it. You don't like it, unfortunately you still have to > use it.

You always have the option of falling back to OpenGL-rendered widgets and using a framework with MVC if you want to. Qt supports this, for instance.

</comment></comment><comment><USER33> Recently I was looking for one book desperately , a book that shows android (and its apps) architecture, form low-level to high-level, design decision which you should take when you design your app architecture. I hate how all books are kind of reference manual. If I want to see what an API doing , I will look at it documentation I don't need another book.But sadly most of books written about android is just talking about API details.

p.s. If you know such book I would be grateful .

<comment><USER34> You might consider this book: [http://www.amazon.com/Android-Programming- Ranch-Guide-Guides...](http://www.amazon.com/Android-Programming-Ranch-Guide- Guides/dp/0321804333) I've read some of their books before, and they have good understanding (and presentation) of architecture.

</comment></comment><comment><USER35> One of the biggest challenges for building on Android is its complexity, and the team must successfully integrate with a lot of surface area conceptually (the fundamental building blocks Dianne explains is just the start). This unusually negative comment thread (for HN, I think) is probably illustrative of how frustrating Android development can be.

On the other hand, it's been clear to me that Google is investing heavily in making Android an easier platform to develop for.

First, the tooling: Moving to Gradle for the build system was a huge improvement. Making a repeatable, extensible build is the vital with teams moving towards the DevOps world with CI/CD. Android Studio has also been a welcome change; all of the separate components of the build being integrated together makes for a much better development experience. The best example of this is Instant Run, which is a great accomplishment for the tooling team.

The Support Libraries are also a solid investment for implementing good patterns - particularly patterns that can be tricky to implement yourself like the separately sliding appbars. Getting more developers to adopt the support libraries also allows more compatibility, since Google can release the fixes independently from platforms.

YouTube has also been a great resource for Android developers as Google has been constantly releasing content about how to build better apps. Jo and Ian are really great.

Anyway, I would watch this space. It might be a coincidence that Dianne made this post, or the most recent videos ("concurrent tasks with multiple documents" and "tasks and back stack", respectively) were also about basic Android features. Maybe Googlers are reflecting on what Android is due to the current news of Oracle v Google. Or maybe it's because IO is this week. I'll be watching!

<comment><USER16> I agree that moving to AS+Gradle was a net improvement (even though Gradle is not always so quick) because it let us exactly reproduce the individual developer's build on the CI server, as you say.

OTOH, I wonder how many devs really grok the Android gradle plugin and feel comfortable just writing tasks themselves without carefully copying/pasting/modifying some magical incantation from somewhere.

I feel like the verdict on Instant Run is still out though. We need to get used to the complexity of IR's choices to restart the Activity or stop the entire app and reinstall the apk etc and gain solid confidence that we can just trust IR to _truly_ reproduce our code changes in its special, optimized way.

<comment><USER36> > how many devs really grok the Android gradle plugin and feel comfortable > just writing tasks themselves without carefully copying/pasting

That plugin uses Apache Groovy for its syntax, though in reality only a tiny non-Turing Complete portion of Groovy's syntax is used in virtually every build script out there. The main difficulty with Gradle is really the complex API it exposes to Groovy, e.g. not knowing whether to use an = like in

        depos = {
          //stuff
        }

or

        depos {
          //stuff
        }

</comment></comment></comment><comment><USER37> I think it's fine for the Android team to take this approach like so many framework authors before them. I think the problem is that Android developers are seeking patterns and prescriptivist documentation because that's what many other platforms offer these days.

On iOS, the message has always been clear: "Use MVC, delegates & data sources, storyboards, etc. /We use them internally/ and you should too. Don't go re- invent UIViewController!"

</comment><comment><USER38> Tldr: that's how the system interacts, you are free to do what you want and just use the framework for system interactions if you need them.

I guess this is a late reply but I feel the need to add in here considering I began my career with Android devices and because of that have looked at everything from that perspective(I write code for web apps, analytics, devops too). What everyone seems to be focusing on is how broken the android platform is. But no one sees the main theme of the post being that your code is really not necessarily a part of the design. Take the entire Android environment , Android currently works on at least a million distinct devices which vary massively in configurations (I include tv/wear/auto and maybe upcoming platforms here too). This with a 6 month major release cycle (if you want to argue about the necessity of this, it's a another topic). You really cannot break everything in the next api level and neither can you make a simple framework for the comfort of the newer programmers or people from other environs. If you really want to get to the main bit of development, there are several starter packs which can help you in the same. This would also explain the sheer number of Android libraries available. this how the system works,not your app. And unless you are building a system app you are free to do what you want and interact with the system as it dictates, because more than the Android system is designed to cater to the end user. It's not a product for developers (I look at the ios development as a a product for devs and end users).

Android development is frustrating and I have had my fair share of pointless arguments on bugs which the Android team refuses to recognize. But when you go through the AOSP code you see how much of can be done with the system. The bitmap provider implementation for example is my go to code for whenever I have to perform intensive processing.

The new memory manager is also an interesting piece of code which Android devs should read. And all these were written while interacting with the same framework.

</comment><comment><USER39> I remember her name. Wasn't she the one who vehemently was defending activity lifecycle and other broken multi-threaded API, something every Android developers I talked to would consider totally broken?

To be fair, developing on Android has become so insane releases over releases that her crazy write up about no engineering design hardly comes as a surprise...

Shame on Google for forcing us to use that POS.

<comment><USER40> I met her once at Google I/O, and it wasn't a pleasant interaction. I was lamenting how the recent (at the time) switch from USB Mass Storage to MTP made file transfers to and from computers very iffy. (And it's still pretty iffy today.) I was asking why the change was made, and if there was a chance at the option for it being restored, as devices used to allow both.

A lot of times, files on the Android device don't show up properly in Windows Explorer. She asked what I was doing, and I said copying APK files. (I regularly used to backup APKs so I could revert software updates which degraded the experience.)

She immediately stated that they "didn't support piracy" and walked off. As if the process of sideloading APKs or transferring them to and from a PC was somehow automatically illicit activity, rather than... something a lot of developers probably do regularly.

<comment><USER41> I see Android guys lack commonsense and I am not surprised. How did they think you are pirating apk's after attending I/O event that costs fortune to enter ?

</comment></comment></comment><comment><USER42> They really should have stuck with int main. What's unfortunate is the quantum leaps they've been taking in terms of complexity to just get an app up and running on their platform. But I guess that's the only way their engineers can justify their salaries/bonuses. Same as with that horrific nightmare of an app Facebook puts out.

<comment><USER43> What would staying with the main entry point solve, aside from being familiar?

Your application would still have to directly manage binder callbacks and register for lifecycle events. Re-using the old Java/POSIX entry point actually increases the complexity significantly.</comment></comment>


Canine, an easy Ruby "binary" library - <AUTHOR> http://gist.github.com/211445

<comment><USER2> This is the best command line utility library for ruby that i've used: <http://visionmedia.github.com/commander/>

<comment><USER3> This looks good -- thank you!

I actually hacked up a half-baked version of this the other month, and it's been sitting around waiting for me to polish it or to find something better. And now here's the better, polished version ready for adoption.

</comment></comment><comment><USER4> Oh, this is nice. I've felt that other "command line utility" libraries for Ruby have been way more complicated than they needed to be. I'm converting some of my simple Ruby scripts and utilities to this to give it a try.

</comment><comment><USER5> This rules. I've been looking for something like this for ages, but never felt satisfied with rubigen or thor.</comment>


How America Can Rise Again - <AUTHOR> http://www.theatlantic.com/doc/print/201001/american-decline <comment><USER2> Wow, this is a great read. It's long, but full of gems and interesting talking points.

 _> > "I think it would be easy for common-sense Americans to draw up a list of big things that would seem to demand concerted effort. Deficits are too big. Health costs are unacceptable. Oil. And yet we have a political system that seems to be constantly consumed with trivial things. We cannot seriously grapple with the big issues. Tactics consume strategy._

Probably true, but the question is just how many common-sense Americans do we have? I live in Boston now, but I'm originally from a suburban town in Georgia. When I visit home, people commonly express to me their fears of impending terrorist attacks, their theories about how drugs and atheism are corrupting society, etc. Then these same people hop in their SUVs and drive 90MPH to the nearest McDonald's.

Average people don't like to worry about reasonable long-term problems. It's just too boring. They'd rather fret over the remote possibility that some Islamic extremists will fly planes into their homes. Thus, we spend trillions of dollars and thousands of lives fighting in the desert while our economy tanks.

 _> > the most worrisome symptom was the relative shortage of a jeremiad theme under Presidents Clinton, George W. Bush, and now Obama. This he attributed to Ronald Reagan, "who managed to equate criticism with anti-Americanism, and render unintelligible bad news about America."_

I can't count how many times I've heard, "If you don't like it then you can find another country," in response to criticism of American government. Yuck.

<comment><USER3> Yes, because driving an SUV and eating at McDonalds are really where our concern should be. The truth is you're no better than the anti-drug, terrorism mongers.

Rush Limbaugh tells them to fear terrorists so they line up to do it while NPR tells you not driving an SUV is bad so you line up to condemn it. Same sheep, different Sheppard.

In the end their not realizing the low statistical chance of a terrorist attack is just like you not realizing the scientific theory you've deified might not be correct and if it's not we'll have done nothing to find out how to actually deal with the problem.

<comment><USER4> That's a pretty aggressive reply, almost to the level of flaming.

The point is that random theories of drugs, atheism and terrorism are made up problems that average people can't do anything about. Discussing security is important and being aware of potential threads is necessary and important, but to a limit.

Taking care of the environment, eating healthy, saving money, improving things in your surroundings are things you can do to improve your life and your community.

The sheep/shepherd argument is too simplistic and while it's something to think about, it was formulated in a way that it stops the conversation, instead of moving it forward.

</comment></comment></comment><comment><USER5> The United States needs to learn to function as one power in a complex system that it can neither escape nor dominate. It also needs to view other nations successes in pulling their citizens out of poverty not as a threat but as an opportunity and it needs to ask a lot of questions.

        - Why are we in Afghanistan, Iraq?
        - What we learned from Vietnam?
        - Why is the Palestinian problem not solved?
        - How can we stop large oscillations in our Economies?
        - Why are our friends trusting us less?
        - Why is our infrastructure aging?
        - Why did we export manufacturing to cheaper labor overseas and not imported
           the labor?
        - How much of our large Service Industry can go the way manufacturing did?
        - Why is Health Services and Insurance so expensive?

I can fill a few pages with questions. America certainly has the brains to answer those questions, but not the will to implement the answers.

<comment> sid- "- Why did we export manufacturing to cheaper labor overseas and not imported the labor?"

That is a very interesting suggestion. But how would we get rid of minimum wage,health care costs,unions,cost of living to actually lower the cost ? Other countries have an advantage as the minimum wage is lesser and human rights activists are not as vocal. Maybe if people think more deeply about it they will have a solution.

</comment></comment><comment><USER6> While there are some interested points about the national psyche, to me, you just can't ignore the deficit problem. Endlessly spending more money than you ean will eventually cause problems, whether you're an individual, business or government. I don't know if the problem will be this year or in 50 years, but something has to give somewhere along the line.

<comment><USER7> If the deficit is spent on infrastructure improvements that support greater prosperity in the private sector, then in the long run the country's tax base expands and that debt can be paid off.

(On the one hand, in theory, if the people buying our debt are _convinced_ that the government is using the borrowed money for that purpose, then they won't bid our interest rates into the stratosphere as happened with Argentina et al. On the other hand, I'm not 100% sure how well that theory applies here, because a lot of the national debt-holders are foreign institutions that may be making geo-strategic calculations as well as profit-making calculations. On the other other hand, if our economy goes tits-up it's not going to be good news for China, either.)

<comment><USER8> _If the deficit is spent on infrastructure improvements..._

Sure, if. Is it being spent that way? Not even remotely.

</comment></comment></comment><comment><USER9> So, we can "rise again" by fixing some nit-picky things like filling potholes and giving California (a bankrupt entity) more US Representatives. Thanks for the tips. Actually, the roads around where I live are pretty good.

 _the absolute pillars of American strength: continued openness to immigration, and a continued concentration of universities that people around the world want to attend._

Whatever you think of immigration, there's really a logical problem with the above thought. America is a place that consists of people, who are solely responsible for America's greatness...NOT theoretical future inflows of other people. His assertion can't be tested. It's a guess, no different than saying: "My upcoming policy of betting red on the roulette wheel is the source of my wealth. Because it worked before."

China and India are the countries on the ascendancy right now [1] and they are not rapidly replacing themselves with other nations, and would probably not appreciate the suggestion that doing so would be the "pillar" of their strength. This writer, of course, had to say it because it's ideologically required thinking.

[1] The US's ascendancy can be roughly marked from 1920 to 1965, a period of very low immigration.

<comment><USER10> Just to pick on one point: what conceivable moral logic leads one to choose democratic representation based on the financial health of their existing government? I just can't see where that comes from. What's next: people with net credit debt are worth 3/5 of a non-slave?

<comment><USER11> > Just to pick on one point: what conceivable moral logic leads one to choose > democratic representation based on the financial health of their existing > government?

Wasn't the idea to give each state to senators so that populous states do not encroach on less populous states?

If California gets many senators, they will surely vote to get federal money to fix their deficit.

</comment></comment></comment><comment><USER12> While the article gives you the feeling of a very in depth article, it looks at certain very important things in a very shallow way, and in the end doesn't offer much.

The article forgot technology.Technology plays a huge role in the economic changes affecting America, and the world. It enables shifting jobs around the world, automating jobs , and driving rapid change everywhere.It seems that these abilities would be only getting stronger in the future.

It also focused strongly on the value of america's world leading education sector as an economic engine. But the fact of the matter is , that most workers are average, and in order to replace them , china and india's workers might be just good enough.

</comment><comment><USER13> Saw another similar article recently: [http://www.newgeography.com/content/001296-dont-give-up- on-t...](http://www.newgeography.com/content/001296-dont-give-up-on-the-us).

</comment><comment><USER14> America's been failed by its leaders. It isn't a partisan thing because the problem has been with both parties.

The assasinations, riots and corruption at the highest levels in the sixties and early seventies tore at the very fabric of our country.

But we found great leadership and came back. We will do it again, stronger than before.

</comment><comment><USER15> Contrarian reads such as this lead me to believe that we've already made the turn.</comment>


Kim Jong Un's Uncle, Formerly North Korea's No. 2, Is Executed - <AUTHOR> http://www.npr.org/blogs/thetwo-way/2013/12/12/250556545/kim-jong-uns-uncle-formerly-north-koreas-no-2-is-executed

<comment><USER2> I feel so sorry for the rank and file North Koreans every time I think about that country.

As far as one of the NK 'inner circle' being executed though ... I only hope it hurt.

<comment><USER3> I'm less certain about this being the usual shakeup. There was a lot of talk that he was very well traveled and had stronger connections with the outside world than most within their government. He'd have been hard pressed to make the current situation worse so what if he was trying to do something different? Obviously just supposition, but food for thought.

</comment></comment><comment><USER4> George Orwell's book "1984" (written 1948 and prior to the formation of North Korea) outlines a state eerily similar to North Korea with its erasure of people's past, punishment for not clapping enthusiastically and generally for "dreaming different dreams", as this article puts it. It's disturbing to see these kind of states still managing to transfer their totalitarian systems from generation to generation with no foreseeable hope of a positive change.

<comment><USER5> Most Western nations seem to be moving that direction as well.

</comment></comment><comment><USER6> This is like a modern day Game of Thrones.

<comment><USER7> His crime was being capable of displacing Kim Jong Un. Almost every monarch in history had a purge of uncles and male cousins.</comment></comment>


How Imgix Built a Stack to Serve 100k Images per Second - <AUTHOR> http://stackshare.io/imgix/how-imgix-built-a-stack-to-serve-100-000-images-per-second#stackshare-weekly <comment><USER2> I love imgix to death, really the only web service I use daily that constantly amazes me.

I've been using them for a few clients now and it's been a fundamental shift in how we think about media management in the CMS. Rolling out new features with different image size requirements using existing media doesn't require us doing anything other than changing a few parameters in the URL. Before, we'd have to run processing tasks on a fleet of EC2 instances and wait a few days.

We compared a few similar services, but none of them really got the flow right like Imgix does. Some require you to setup the parameters beforehand or define everything in their web interface or via a REST api. And quality wise, none of them are near as good as Imgix.

I can't sing their praises high enough honestly.

<comment><USER3> > Before, we'd have to run processing tasks on a fleet of EC2 instances and > wait a few days.

Few days? Why?

<comment><USER4> 2 ways to do thumbnails. 1 Precompute(what they did) and 2 Compute on demand(imgix)

Of course the best is to do something in the middle.

<comment><USER2> Imgix is the middle, they process the parameters once and cache the result.

<comment><USER4> That is just caching.

Middle is: You precompute some fixed presets(like him, lower latency) but also have the ability to generate on demand if a preset doesn't exist(used when adding new presets, no reason to wait *days to have the new preset).

<comment><USER5> Once you've built a system that works fast enough to handle image rendering work on-demand, then adding batch or precomputational features is largely just creating requester logic to ask for the image before the real demand arrives.

</comment></comment></comment></comment></comment></comment><comment><USER6> This company repeatedly sets off my B.S. detector with its technology story. Here's a previous article where they didn't have their schtick quite so polished:

[http://www.wired.com/wiredenterprise/2013/05/imgix- graphics-...](http://www.wired.com/wiredenterprise/2013/05/imgix-graphics- card-internet/)

Believe it or not: Google already had awesome, high-performance image tech. YouTube re-encoded every _video_ in 2010 as a favor to Apple. Meanwhile these guys are saying Google couldn't make _thumbnails_? So they had to do a startup using the GPUs in Mac minis?

"We just couldn't crunch those images down to a smaller size," Zacharias remembers. "It would have taken a significant amount of Google's entire processing power just to do that."

Preposterous, dishonest claim.

I looked at this back then and discovered:

Google provides it through App Engine. Adobe provides this through Akamai. [http://cloudimage.io/](http://cloudimage.io/) does it through Rackspace CDN. [http://cloudinary.com/](http://cloudinary.com/) does it through CloudFront.

That said Imgix seems to provide a stable service people enjoy. Perhaps they should sell that instead of these oddball tech claims. Because what they're doing isn't hard or novel. They sure are making it complicated, though.

<comment><USER7> > They sure are making it complicated, though.

Seems like a large degree of it is marketing-related.

"Any intelligent fool can make things bigger, more complex, and more violent..."

</comment></comment><comment><USER8> > An early Facebook photos engineer mentioned he had seen remarkable > performance and quality coming out of the Apple Core Graphics stack .

What does the Apple stack have, which can't be done on a Linux box with a high-end GPU like a K20 or a Tesla?

<comment><USER6> [Talking purely CPU] You can resize 100k images a second on a 2013 laptop. [Source: the Google guy who serves your images.] A couple ARM cellphones could meet this entire company's [CPU] image processing needs.

Think of what YouTube is doing for videos in the time you read this sentence if you want to put their silly claims in perspective.

Anybody remember when Steve Jobs said the 400Mhz PowerPC G4 Altivec was a supercomputer? But the only code that showed up was a Reverb that took a weekend to run and a couple FFT libraries that had accuracy problems? Good times.

<comment><USER9> 100k images a second is definitely more than my laptop can even read off a disk in a second. Unless they're already small images.

</comment></comment></comment><comment><USER10> My middle-click is broken on this site and doesn't open things in a new tab. : (

</comment><comment><USER11> > "In most cases, your images will traverse the entire stack without ever > touching a hard drive."

> "Our fetching and caching layers are largely custom built, using MogileFS"

Could you specify the use case of mogilefs in your project, is it some sort of "cold storage" for the fetched image data? Can you provide more details - amount of trackers, storage-nodes, storage-size, ... ? Do you also consider using a ceph cluster in future?

<comment><USER5> Commodity CDN's will always have some amount of cache misses. For most services that's fine; there isn't too much additional overhead in serving a static Javascript file a few times. If you're doing computational work for each request, you want to try to avoid unnecessary work as much as possible.

It's helpful for us to hold onto master images after fetching -- these images can be large and may not be served to us from a particularly fast source. The speed of light does come into play, so anything we can do to shorten distances is beneficial. This cache layer respects the Cache-Control headers of the original objects, so the behavior there is under our user's control.

There's a similar need to cache rendered output ("derivative images"), because while most operations are very fast they do still add a little bit of latency. This helps with our rendering capacity, but it's really about improving time- to-serve latency wherever we can.

We evaluated a few open source object stores and decided on MogileFS. It largely came down to the simplicity of what Mogile does and how it does it. We figured that we could rip pieces out and adapt it to our needs with a lot less work than more elaborate systems (like ceph). With that said, we didn't actually do an implementation test with ceph, so it may in fact be great for our use case.

Not sure off-hand about the number of servers deployed for use with Mogile, but it isn't a gigantic deployment. We aren't storing anything that can't be re-acquired, so we can adjust our storage to hit particular cache hit ratios rather than storing the enormous long-tail forever.

The first generation of storage servers are 16 core / 32gb / 6x4T systems -- more like what I would use for computationally related storage like HDFS than a large object store. We could stick with this hardware spec, or we might move to SSDs if latency becomes an issue (I personally doubt it will). We might also move to more dense systems, with more like 20-30 drives. That's all based on our metrics and how things perform in practice.

</comment></comment><comment><USER12> I wonder if one could slash costs by building a stack on top of Imgix that concatenates several "master" images into a single "grandmaster" image and then just serve them up individually with cropping operations ...

</comment><comment><USER13> How does the cost of Mac Pro's compare with more traditional servers?

<comment><USER14> Cheaper than re-writing and re-testing everything to work on Windows/Linux + whitebox servers.

<comment><USER6> They're using code that they cannot license elsewhere. They're making OS calls to process the images.

</comment></comment></comment><comment><USER15> Somehow disappointed by the title. This is just a list of software they use. IMHO, that's not a "stack" article in my book.

>>>> The load balancing and distribution layer is based on custom C code and a LuaJIT framework we created called Levee. Levee is capable of servicing 40K requests per second on a single machine

Is "Leeve" open sourced ? It would be nice to see this source code and it would had been great to include some discussion on why they chose to implement this instead of using some other off-the-shelf LB solutions (like HAProxy or Openresty which are also mentioned in the article). This is the kind of things I like seeing in a "stack" article.

<comment><USER5> This is pretty typical in my experience for StackShare posts. For example: [http://stackshare.io/500px/how-500px-serves-up-over-500tb- of...](http://stackshare.io/500px/how-500px-serves-up-over-500tb-of-high-res- photos)

imgix is very interested in open sourcing things we create, and there are a few smaller projects at [https://github.com/imgix](https://github.com/imgix) now. Hopefully Levee will join them in the future, once we have reached a certain point in development.

</comment></comment><comment><USER16> From the article: "Levee is capable of servicing 40K requests per second on a single machine."

How are concurrent connections simulated for testing?</comment>


Making ubuntu.com responsive - <AUTHOR> http://design.canonical.com/2014/03/making-ubuntu-com-responsive/ <comment><USER2> Plain prerendered html5 with no JS was not available?

<comment><USER3> Where's the framework for that?

<comment><USER2> the great text files framework.

<comment><USER3> And its IDE, vim.

<comment><USER2> I prefer nano, as it reminds me to keep the files short.</comment></comment></comment></comment></comment>


Le myope, a short-sighted Raspberry Pi camera - <AUTHOR> http://saladetomateoignon.com/Wordpress/a-short-sighted-raspberry-pi-camera/ <comment><USER2> The related layercam project: [https://news.ycombinator.com/item?id=8211473](https://news.ycombinator.com/item?id=8211473)</comment>


Australian retailer Jaycar ripping off Arduino kit from local company - <AUTHOR> https://www.youtube.com/watch?v=SW8K9D9u5aI <comment><USER2> Sad to see the last electroncis retailer treating the already decimated Australian hobbyist market. Especially when he is already supplying Jaycar with his kit.

</comment><comment><USER3> Well this sucks. Jaycar is one of the last retail electronic hobbyists left.

<comment><USER4> Was thinking exactly the same thing.

</comment></comment><comment><USER5> Oh cry me a river. Someone else put an Arduino in a box? It turns out the retailers held the power? Well, no kidding. I thought it was remarkably decent of them to not make it yellow with blue writing.

<comment><AUTHOR> Copying the manual word by word and stealing original diagrams is okay too?</comment></comment>


Internet Data Is Rotting - <AUTHOR> http://theconversation.com/your-internet-data-is-rotting-115891 <comment><USER3> This needs to be solved on the protocol level. Of course, the players who have control over our protocols are exactly the people who don't want this to be solved at all.

The next best thing would be to redefine what "bookmarking" is. When I bookmark a page, I want it to be permanently stored on my local machine and full-text indexed. In fact, it's rather ridiculous that after 25 years browsers don't have anything of this sort. Unfortunately, the most popular browser in the world is controlled by the same people who control our protocols.

If I ever get the energy, I will attempt to write a browser extension for this.

<comment><USER4> wget -r ?

<comment><USER3> No. There are some cases where it is useful to download many pages in a batch, but what I am talking about is, effectively, partial local replication. Bookmarking I describe should create a tiny (but useful) version of the web on your computer.

It needs to be seamless. It needs to be searchable. It would be incredibly useful if it would capture relationships between pages (links) in addition to pages themselves to navigate offline.

I can see many use cases for a kind of "super bookmark mode" where the browser automatically stores all the pages you visit within a certain domain.

<comment><USER5> I would loooove this. Open one of your bookmarks and you can't tell it's the local copy unless you looked at the local path sitting in the URL bar. Maybe make an option to fetch a live copy though for sites like the front page of HN that are in constant turnover, but it would be great. Storage has never been cheaper, let me hoard!

You can even take it a step further and create a viewable timeline of all the pages you've visited in case you have something at the tip of your tongue, but need to retrace your steps and logic to get there. Browsing history is kinda lackluster for this. My last ten entries on firefox are three hackernews articles with "Add Comment" dispersed half a dozen times in that list. If I had something along the lines of zoteros timeline for papers I could probably find stuff way easier.

</comment></comment></comment></comment><comment><USER6> > As of last fall, its Wayback Machine held over 450 billion pages in 25 > petabytes of data. This would represent .0003% of the total internet.

> Universities, governments and scientific societies are struggling to > preserve scientific data in a hodgepodge of archives, such as the U.K.'s > Digital Preservation Coalition, MetaArchive, or the now-disbanded > collaborative Digital Preservation Network.

Like any conservation work, the benefits are incredibly easy to ignore - <USER2> something goes awry / stops getting funding and suddenly it's too late. Consequently it's easy to have a myopic view of the issue.

These organizations are doing very important work, and I hope that internet users and governments don't take them for granted.

</comment><comment><USER7> We should aim at browsing the Internet _by date_. We're moving everything there ignoring the fact that as it is there is no built-in permanence. We're accustomed to that after Gutenberg: it wasn't that easy losing every copy of an important document. Now it is, things disappear and we're getting into a drifting cultural bubble impossible to trace back.

The Internet Archive is doing God's work, but it's not enough. If you don't have the URL of a site that is gone, you probably won't find any reference to it after every online hyperlink to it has disappeared as well. It might become then inaccessible after a while, stored yet gone anyway.

<comment><USER8> "Browsing by date" is pretty much Brewster Kahle's ideal, and is what the Internet Archive's Wayback Machine approximates, thanks in large part to WARC storage.

I'd also like to see a distinction between the idea of web servers, which are really _publishers_ , and where _archives_ are kept. Ideally _not_ all in one single store, a/k/a the Internet Archive, but replicated fairly widely.

</comment></comment><comment><USER9> Things got far more worse in China right now.

Baidu Tieba, which could be considered as Reddit for China, just made all posts before 2017-01-01 inaccessible. And a number of other online forums are doing the same thing due to political reasons.

<comment><USER10> would you elaborate on the political reasons?

<comment><USER11> If you want to rewrite history it doesnt help to have lots of old copies lying around. Any data not under government control is a threat.

<comment><USER12> Isn't Baidu under government control anyway?

<comment><USER13> Yes, but the effort to go through and scrub that much data to ensure only the ideas you want out are getting out would be massive. Seems easier to just shut the door and board up the room than to try and clean it.

</comment></comment></comment></comment></comment><comment><USER14> Back in the mists of time, I used to use wwwoffle proxy. It was great for low- latency links, but also had the benefit of keeping an offline archive of whatever you'd browsed.

Project's still there, although not sure how well it does with the modern web.

[http://www.gedanken.org.uk/software/wwwoffle/](http://www.gedanken.org.uk/software/wwwoffle/)

There are a bunch of more modern variations too:

[https://archivebox.io/](https://archivebox.io/) \- "Your own personal internet archive"

[https://getpolarized.io/](https://getpolarized.io/) (as seen on HN previously)

[https://github.com/kanishka-linux/reminiscence](https://github.com/kanishka- linux/reminiscence)

[https://github.com/fake-name/ReadableWebProxy](https://github.com/fake- name/ReadableWebProxy)

<comment><USER8> Sadly, a lot of old-school proxies (squid, privoxy) are stymied by SSL/TLS connections.

I think we're due for the idea that a proxy can be designated as a trusted intermediary, most especially if it's run on a personal basis. I'm sure this presents security issues, but it also avoids some.

<comment><USER15> > I think we're due for the idea that a proxy can be designated as a trusted > intermediary, most especially if it's run on a personal basis.

We have that idea now; you designate the proxy as a trusted intermediary by accepting its certificate. The chain looks something like this:

        You: browser, take me to https://youtube.com
        Browser: proxy, get me https://youtube.com
        Proxy: YouTube, get me /
        YouTube: I'm youtube.com -- here's a certificate signed
                 by the government of Egypt that proves it. And
                 here are the contents of /
        Proxy (to browser): I'm youtube.com -- here's a
                            self-signed certificate attesting to
                            that. And here are the contents of /
        Browser (to user): SECURITY ALERT! SECURITY ALERT!

Configure your browser to accept that certificate, and your proxy can handle its own connection to youtube and just pretend, to your browser, that it is youtube.

<comment><USER8> Does Chrome (and other browsers') MITM attack defeat prevent this? That's my understanding.

[https://comodosslstore.com/blog/google-chrome-63-will- warn-y...](https://comodosslstore.com/blog/google-chrome-63-will-warn-you-of- man-in-the-middle-attacks.html)

<comment><USER15> I'm answering based mostly on having read that link. It looks like the protection applies only in the case where an error is being surfaced. The problem Chrome wants to address is that users will click past the SECURITY ALERT.

If you properly configure your own CA, then the TLS error triggering this behavior won't occur, and there is no security problem for Chrome to put its foot down on -- your proxy is providing a _valid_ certificate for whatever domain, as far as Chrome is concerned, not an invalid one.

Compare [https://support.portswigger.net/customer/portal/articles/178...](https://support.portswigger.net/customer/portal/articles/1783085-installing- burp-s-ca-certificate-in-chrome) .

> The Chrome browser picks up the certificate trust store from your host > computer. By installing Burp's CA certificate in your computer's built-in > browser (e.g. Internet Explorer on Windows, or Safari on OS X), Chrome will > automatically make use of the certificate.

> When the Burp CA certificate has been installed for your built-in browser, > restart Chrome and you should be able to visit any HTTPS URL via Burp > without any security warnings.

<comment><USER8> Thanks. This is something I've got some plans on.

</comment></comment></comment></comment></comment></comment><comment><USER16> Yeah it's annoying that links get broken. But maybe it's better this way. There's something about modern tech that has turned all of us into digital horders. I (we?) have backups and backups of backups and redundant RAID servers with every version of every file so that no byte shall ever perish. I have essays still that I wrote in highschool nearly 20 years ago. To what end? I'm partial to material minimalism. Why not data minimalism?

<comment><USER17> My common response is that there are real world costs to having too much stuff, data is pretty small to keep around. You can fit a RAID 5 box with 30 TB of space in a shoebox for $15/month, that is enough to keep pretty much any content that you ever consume so that if you want anything it still exists. My parents and grandparents hoarded files and documents to no end and I have been going through some of them and there is mostly garbage but there is some real gems in the rough. Willingly disposing of the internet through our own negligence is something I don't advocate for because there is some sort of value preserved in what we save for the next generation.

<comment><USER18> There are real-world costs to information hoarding as well, when it's done by people who are not you and whose incentives are not aligned with your own.

I'm glad that data is rotting on the Internet. In fact, I'll go one step farther and say that data should rot more quickly on the Internet. There's no reason that some scummy marketing algorithm should have access to my high- school social media posts. There's no reason that governments or private individuals with a grudge ought to be able to through the history of everything I've written, no matter how off-the-cuff, in order to dredge up something that makes me look like an undesirable when it's taken out of context.

If an individual wants to save a particular piece of information, that should be their choice. Otherwise, by default, information ought to disappear from the Internet.

<comment><USER17> I think that in terms of what you are talking about bitrot is good. People should have their stuff deleted after a while but there is a sense of permanentness in internet culture and that should change to one of fleetingness which more bitrott may actually bring about causing more people to back up their own stuff instead of leaving it to these companies.

</comment></comment></comment></comment><comment><USER19> > Then there is also a problem of software preservation: How can people today > or in the future interpret those WordPerfect or WordStar files from the > 1980s, when the original software companies have stopped supporting them or > gone out of business?

This issue in particular we have great solutions for (open formats / text), but they are of course less profitable than only-my-app-can-read-this formats.

<comment><USER20> FWIW those particular formats are widely understood even if they are proprietary (well, at least in WordStar's case). And as long as the software runs (be it natively or via an emulator or VM), you can always open and convert/print the files (e.g. you could use vDOS to run WordStar or whatever and use its printer emulator functionality with Windows' PDF printer to create a PDF from the WordStar files).

</comment></comment><comment><USER21> I read somewhere that the lifespan of the average hyperlink is only about two years.

I count myself lucky I was introduced to the HTTRACK archiver program many years ago and thus have complete offline copies of many of my favorite websites of the early 00's.

<comment><USER22> Can you give some examples of these 'favorite websites'? I'm interested in knowing what kind of website would be so interesting that I would want an entire offline copy of it. (Besides maybe Wikipedia)

<comment><USER21> Mostly defunct webcomics but also some of the small personal sites that documented and collected resources for particular events or strange people. A lot of those arose out of the SomethingAwful forums. For example, there was a guy named Brian who wrote batshit insane fanfiction about himself. One of these sites archived the fiction, interviews with Brian, videos, recordings of collaborative reading skype parties, etc, all neatly on one site and now safely tucked away on my drives. Now I have a little piece of nostalgia from 2004 I can step back into.

<comment><USER23> Are you able to navigate through the sites using the original links? I notice that on the Wayback Machine, internal site links only work if that particular page was also archived.

<comment><USER21> Yes. It also lets you designate a particular number of "steps" outside the original site it will also archive. So I give it a site and a 1-step limit, I'll get the site and also any individual other webpage linked to somewhere on the original site. It doesn't do so well with modern sites that full of CDN- hosted content and pages that depend on data from two dozen different domains to function properly but it's great for old pre-web2.0 stuff.

</comment></comment></comment></comment></comment><comment><USER24> I'm okay with internet rot and you should be too. I'm not sure where we got the idea that "our data must be preserved forever". This can be especially harmful for teens and young adults whose indiscretions now follow them forever.

Think of the privilege you had when you were younger. You could do something stupid and nobody could whip out a high def camera to record it and make it part of your history forever.

Let it rot.

<comment><USER25> I'm OK with it because otherwise you are whitewashing history.

For example I have recordings of the Colbert report going back to ~2005. Some of his skits released during that time would be classified as "hate speech" in 2019. Of course he, and mainstream broadcasting companies would love it if you didn't think about that. There are plenty of news clips and interviews where mainstream politicians (on Left AND Right) casually dismiss gay marriage. Powerful tech influencers like Mark Zuckerberg would love it if his IMs would disappear from the Internet. The examples go on and on.

<comment><USER26> I think if the last 10 years have taught us anything, it's that preserving the past does nothing to impede the changing moral zeitgeist. More records simply mean more people to attack for holding an opinion that has simply gone out of fashion. If the past decade wasn't characterized by tribalism and moral hysteria I'd be more inclined to worry about stringent preservation.

At this point, I'm not really comfortable with what we're preserving.

<comment><USER5> It's simple. Don't say ignorant things on the internet, especially when it's tied into your real life identity. Lots of people struggle with this simple rule, because they have a compulsion to share their own myopic opinion (usually the same viral opinion as everyone else in their echo chamber) and contribute to the noise. If I was a public figure I wouldn't be on any social media platform at all, at best it wastes your already limited and highly valuable time.

That being said, there's a lot of valuable information on the internet that is absolutely worth preserving. Scrape off the layer of social media and the internet is still a place of learning and problem solving. I do my own repairs on my car and the amount of times I've found a well written photo essay on a particular fix in a random honda forum, only to find the linked pictures broken, is astounding and a shame to say the least.

<comment><USER26> Who gets to decide what's ignorant? What if I'm being sensible about what's ignorant, but I can't predict how society will feel in the future?

</comment></comment></comment></comment></comment><comment><USER8> Among the handy tools that can be used to save and access present, at-risk, and/or rotted data, are bookmarklets.

I've recently added two to my browser, "open in Wayback Machine" and "Save in Wayback Machine". Respectively:

        javascript:void(window.open('https://web.archive.org/web/*/'+location.href));

        javascript:void(window.open('https://web.archive.org/save/'+location.href));

This makes opportunistic archival and reference easy. There are also Wayback Machine / Internet Archive browser extensions.

(These are from the Internet Archive, not my work.)

For bulk archival, lists of URLs can be submitted to the IA's save address:

        https://web.archive.org/save/<URL>

(Used in the bookmarklet above as well.)

This can be automated with a simple shell script using any console or script- based HTTP agent, such as curl, wget, lynx, etc.

</comment><comment><USER27> I actually have a different problem -- not sure it is one that I can legally solve.

I have 10 years of lovingly curated YouTube videos playlists, which, now when I look into the older ones, are a barren wasteland of "Video removed" or "Video not available". It is heartbreaking. Is there any way I can prevent this from happening?

<comment><USER28> I'd download and store the videos locally with youtube-dl.

<comment><USER29> I concur, youtube-dl is what I've been using during the last few years: whenever I find a YT video I might want to watch again, I now immediately download it. Learned the need for that the hard way.

Check out its options here: [https://github.com/ytdl-org/youtube- dl/blob/master/README.md...](https://github.com/ytdl-org/youtube- dl/blob/master/README.md#options)

With --add-metadata you can embed the YT video description in the video file. The downloaded video file name will contain the YT identifier so you can still match them back if needed.

There is another option to save the metadata to a separate JSON file if you prefer that.

To download your playlists, give it each playlist's URL instead of the video url:

        youtube-dl --add-metadata --ignore-errors 'https://www.youtube.com/watch?v=8GW6sLrK40k&list=RDQMc4l8l2aQrNo'

That example URL includes a specific video from the list, but will download all of them. It works just the same if you only give it the `list` parameter, but all links to playlists I've seen point to one of their videos.

The option --ignore-errors will jump over the unavailable videos instead of stopping.

Edit to add: If you want to download your playlists as separate directories, with each video file name including its original index in the playlist, see these examples in youtube-dl's documentation: [https://github.com/ytdl- org/youtube-dl/blob/master/README.md...](https://github.com/ytdl-org/youtube- dl/blob/master/README.md#output-template-examples)

</comment></comment></comment><comment><USER30> I think it's good that data is lost . Only items where someone gives enough of a duck to save it should be preserved. It's not as if physical paper content, which ends up recycled or in a landfill 99.999 of the time is any different. It's true that digital formats change but fighting that is the cost of preservation. A museum of software needs to also preserve the context on which software was run in order to save it from the mists of time, albeit temporarily.

<comment><USER31> I feel like you've never had to look for information or how to do something, and the only decent source is entirely gone or consists mostly of pictures, which are also mostly gone. Maybe somewhere at some point in time somebody saved it, but that copy gets lost and never makes it's way online again so you can find it. A lot of information is being lost in this way and I'm not sure why we should be fine with that.

</comment></comment><comment><USER32> This seems like a good time to plug running a storage server on your local network. You can pick up old workstations off eBay for 100$. Stick a couple of drives in it, load it up with data to preserve and then put encrypted backups in the cloud. Back blaze B2 is something like .001$ per gigabyte.

It's a fun experiment with clear, practical use.

<comment><USER33> Not quite that cheap 0.005$ per GB. USD 6 per month for personal unlimited backup though.

Trying the trial now.

Thanks for bringing it to my attention.

</comment></comment><comment><USER34> I hate to be this guy, but isn't this why printers and physical books exist?

<comment><USER35> If books are all we need, why did anyone bother creating the Internet?

<comment><USER36> Computers were a mistake.

</comment></comment></comment><comment><USER8> The recent shutdown of Google+ was another case of this.

As one of the people helping coordinate information among those still using the platform and hoping to migrate off of it, discovering the Archive Team's GoogleMinus project this past January was a huge boost. That ended up being the largest archival project undertaken to date, 1.6 PB, and succeeded in capturing 98% of all G+ profiles, now stored at the Internet Archive.[1]

While it had long been obvious that the project was ill-stared, the shutdown announcement came as a surprise, and Google's tools, communications, and support for both individuals, and far more importantly, _groups_ , looking to continue their existence off the platform, was abysmal.

I don't fault Google for killing the service -- I was suprised it survived as long as it had. I _do_ fault Google for _how_ they did so. And that episode was hardly the worst in history.

One of the lesser-known parts of G+ were its Communities. In the process of the shutdown we came to realise that there were over 8 million of these, about 50,000 with 1,000 or more members, of all descriptions. Many frivilous or worse, but many also not. And all stuck in a very hard spot by Google's actions.[2]

Even preservation of _individual_ data does very little for groups, and is one of the issues we're considering in the post mortem of the G+ mass migration, intended to be of use to others.[3]

________________________________

Notes:

1\. For those preferring not to have their content archived, the IA WBM respect DMCA requests, and as Google+ posts are all listed under the user's account, requesting removal is exceedingly straightforward.

2\. Characteristics of number and size are collected here, compiled by me, based in part on on data provided by Friends+Me: [https://social.antefriguserat.de/index.php/Migrating_Google%...](https://social.antefriguserat.de/index.php/Migrating_Google%2B_Communities#Google.2B_Community_Characteristics_and_Membership)

3\. Discussion at Reddit and elsewhere. Compilation at the PlexodusWiki. [https://old.reddit.com/r/plexodus/comments/boa97x/g_migratio...](https://old.reddit.com/r/plexodus/comments/boa97x/g_migration_post_mortem_what_went_well_what_went/) [https://social.antefriguserat.de/index.php/G%2B_Migration_Po...](https://social.antefriguserat.de/index.php/G%2B_Migration_Post_Mortem)

</comment><comment><USER37> Another problem of the present-day WWW is, even archiving all the data is far from enough to preserve the history! That's because the Web has a dual-role, (a) as a protocol, or a medium of communication, and (b) as the software, or the user-interface.

A good history preservation should allow you to somehow "browse" it, as if the historical system is still alive. How the website worked, how it was used, that's all parts of the history. If old operating systems and programs are preserved, there is no reason _not_ to preserve websites in this way.

Back in the old days, many systems are federated and/or distributed, which means the software and the protocol are two separate entities. You use a newsreader, which talks the NNTP protocol to obtain news from a Usenet newsgroup. If you want to preserve history, you can (a) archive the newsreader program with source code, and (b) archive all the data on the NNTP server. That's exactly what has been done already, if you load a Usenet archive to your newsreader, pretty much you would have the experience similar to how Larry Wall browsed the Usenet back in the late 80s, at worst you need to rewrite a compatible "mock" server, but that's all. On the other hand, little of the early BBS systems have been preserved, once the server is gone, everything is gone.

The transformation to the web, means now the platform (a web community) = protocol (backend database format) = user interface (HTML/CSS), they're all tightly coupled together. It creates several problems:

(1) The "internal state" cannot be archived. A website is a system with constantly updating parameters, and often they are not stored. Simple examples: (a) On Hacker News, I cannot see what was shown on the frontpage yesterday retroactively, (b) A user changes his/her avatar, now we had no idea how the old avatar used to look like, and (c) an early user has been banned from the forum, now his/her personal profile is inaccessible, (d) on some social media platforms, sometimes a old post may be raised from the dead by renewed interests ( _look, how stupid this comment was!_ ), and now suddenly it's flooded by new posts, leaving no trace of how it used to look like.

(2) The "reader/user interface" cannot be archived. You must have seen something like this: You changed the website frontend, superficially, lots of "conservative" users complained, but the point is: now the old frontend and its "look-and-feel" is lost. If it was a simple CSS file, there are chances to bring it back, but if it was a major rewrite of frontend code, now history is gone forever. And in the lifetime of a website, the design and architecture is likely to be changed many times.

As a result, _even if a website and all its content is still alive, it may already be a shadow of its past for a long time_ , don't even mention to preserve it! And currently, there are two ways to archive the web, both are flawed:

(1) Preserve the HTML at the surface. It's good for single pages, but you cannot browse a website in this way at all. None of the button on the website would work.

(2) Preserve the database. For example, using the API to save posts, or dumping the database - the frontend and reader are not preserved. Using Hacker News as an example, now every single post is archived, but it's far from a full experience, at least you should be able to click someone's username and see all the posts.

Now more and more websites are powered by JavaScript, makes the problem even worse. You are now literally running a program on your computer without any control over it. Once the platform is gone, no archive can save you.

What is the solution? I guess there's no full solution, but there are some possibilities:

(1) Wikipedia-like websites already have builtin version control, but it's very difficult to browse the historical version of the entire website. Systems like this can improve the frontend / user interface to allow a user to "lock on" a historical date.

(2) When building an all-Javascript website, spend some energy to build a plain HTML version as well, it may help avoiding the upcoming digital dark age.

(3) If you are going to close a website, perhaps it may be a good idea to make your internal backups of database and codebase at different years publicly available with sensitive information removed, and allow everyone to setup and run a replicated version. It's infeasible for a big website, but it may be a workable idea for a small community.

And I can imagine the archeologist from the 22th century digging into the old backup tapes of Reddit and attempt to rerun the system.

But ultimately, it's a problem that is needed to be addressed by the protocol and software with archive and preservation in mind.

\---

BTW: A few weeks ago, I've written a lengthy comment on the fundamental conflict of history preservation and personal privacy, using the Usenet as an example, you may find it interesting.

* [https://news.ycombinator.com/item?id=19562650](https://news.ycombinator.com/item?id=19562650)

</comment><comment><USER38> And that s a great thing! There is no reason to maintain everything, in fact the entire function of our brains is to filter out useful information from a cataclysm of sensory input. The internet figures out what to keep and what to throw away, the hard thing seems to be to willingly make it _forget_ stuff.

</comment><comment><USER39> If it is rotting, then it will be a great time for scavengers and carrion feeders--maybe even the 'era of'.

</comment><comment><USER40> I don't care. In fact, I prefer it this way. Death in species is an evolutionary advantage. So too with culture. We mustn't let it ossify.

For the whitewashing thing, it will happen anyway. Only vigilance can protect against rewriting. Websites can be altered. There is no provenance.

I'm not convinced infinite recall is useful.

<comment><USER7> I don't think it's for us to decide.

We've been mourning the loss of the Library of Alexandria for circa 1,500 years and I can't notice that we're getting radically wiser during the last 25 ones.

Letting all go is definitely the cheaper and convenient attitude... for us. But we might be leaving nothing to build upon for the future generation. They should have the same opportunity to ignore what they want that we have.

<comment><USER41> And this is an important era in the history of our species. We just invented an almost-free communication medium capable of communication across the globe. This is as big as the discovery of fire.</comment></comment></comment>


Similar sites like HN - <AUTHOR>

I am a late bloomer when it comes to finding sites like HN. I think HN is a pretty good source. Other than having to filter and it being a little hard to find the useful articles, I think this is a great place to stay up to date on the happenings.<p>Are there other sources like this? In light of a few HN is dying articles, what are some other sites people are going to? <comment><USER2> techstartu.ps reddit.com digg.com</comment>


PhD Model Canvas (adopted from business model canvas) - <AUTHOR> http://cs.joensuu.fi/~seivazi/

<comment><AUTHOR> We thought why not to adopt BMC for researchers and PhD guys like myself. Here what we come up with: PMC Model for researcher to make plans more concise, easy to keep track on the goals and modify them. I would appreciate any comments regarding to the Model and elements.

</comment><comment><USER2> I think the canvas is a great tool for discussing research. One can easily use it to plan a research proposal or article

<comment><AUTHOR> I think we should think how we we adjust the model for any scientific paper as you discuss that in the first point of idea.</comment></comment>


React Interview Questions - <AUTHOR> https://tylermcginnis.com/react-interview-questions/v <comment><USER2> You might want to resubmit, the correct link is: [https://tylermcginnis.com/react-interview- questions/](https://tylermcginnis.com/react-interview-questions/)</comment>


Why does apartment hunting still suck? - <AUTHOR>

There are so many apartment hunting websites, but they're all so terrible. Most are just covered with ads and redirect to listings created on other sites. Rent.com is decent, but it's still a little "meh" - it just feels like it has no heart. Padmapper is decent, but it just feels like a pet project that isn't innovating any more.
I feel like this is a market with no real great tool for this. I can't say that anything that I've used has given me a "wow what a great experience" reaction.
I don't know what the "best" way of doing this would be, probably very similar to how other sites do it, but just with minor tweaks to make them more relevant or something.
Just some thoughts from somebody currently looking for an apartment! <comment><USER2> Have you explored MapThatPad (www.mapthatpad.com) ? It's not your everyday listing site, but an apartment hunting organizational tool. It takes 2 steps to easily pull in and map as many listings from Craigslist, streeteasy, Oodle, NYtimes and other sites. You can share your maps and invite friends to collaborate on your hunt. Not to mention, that you can see all important nearby amenities such as restaurants, groceries and transit (for NYC and SF).

</comment><comment><USER3> Simple: almost all listings are from agents / agencies. Their incentive is to rent as many apartments as possible, as often as possible, at as low a risk as possible.

This is why I only rent directly from private owners who have 1-3 units total. Then the negotiation can proceed among equals, and there's a far higher likelihood of getting treated like a human being.

Of course, finding apartments becomes much more difficult under these constraints, but it's been worth it to me.

</comment><comment><USER4> I agree with you 100%. I went to an agency that said they help you find an apartment free and turns out there is a total of $400 in fees, so much for free. I'm going to go back to the old school way, looking in the newspaper at least they are somewhat straight forward!

</comment><comment><USER5> it sucks because the data they all aggregate sucks. landlords are lazy and agents purposely withhold information so you have to contact them directly.

house hunting sites are usually much better because most of their data comes from a standard source (the mls) with defined fields.

</comment><comment><USER6> padmapper.com helps relieve a lot of the pain, but not all of it.</comment>