Developing a mobile cross-platform library in C++ - <AUTHOR> http://www.skyscanner.net/blogs/developing-mobile-cross-platform-library-part-2-c

<comment><USER2> Every time I think about what Android _could_ have been, my frustration mounts. Google chose to use Java for the top layer on top of their native code libraries (as in on top of C libraries like surface manager, opengles, openvg, sgl, etc.). Why they chose Java instead of a language that integrates well with C, I just don't understand. It would have made the ecosystem much more open.

I wish they had considered just making the top layer in Luajit, as well as exposing a C API. Luajit would have been just as efficient battery-wise as Dalvik, and perhaps even better-performing. Best of all, people then could have much more easily built libraries for Python, Ruby, and any other language with good C interop.

Instead, we're left with Java. It's not a bad language, but it is very bad at integrating with other languages and platforms.

I prefer Android to iOS, but the iOS architecture is definitely more open to third-party platforms. Unfortunately, Apple is not so welcoming and puts up roadblocks.

<comment><USER3> Yeah I think Python would have been a good choice. Something that looks like kivy perhaps.

[http://kivy.org/#home](http://kivy.org/#home)

Java is just too popular and too average. Has enough people knowing it, works well with an IDE, compiles (and thus hides the source better).

The apps I wrote for example, was more about the API and editing XML files. I've used Java before but in this case it felt different, I would say easier. Not a lot of FactoryFactoryManager type mess many stereotypically associate with Enterprise Java back-ends. So it wasn't too bad.

Another interesting point. At one of the PyCons someone asked Guido publicly (he was still at Google back then), why doesn't Android use Python more? And Guido replied it was a "political decision". Moreover the tone he replied it was betraying that he was a bit upset or annoyed at it. So I imagine there was some internal discussion, perhaps more than one and more than once, and one camp one and one lost.

<comment><USER4> I would love it if Tcl were a first class option. W/o getting into Tcl vs. Lua vs. Python vs. Ruby vs. Perl vs. Chicken Scheme vs... if the underlying platform really were C, all these things would be viable options. C would have let all these flourish.

</comment></comment></comment><comment><USER5> I have written a good chuck of common code in C++ for my diagramming app "Lekh Diagram"([http://www.avabodh.com/lekh](http://www.avabodh.com/lekh)). The shape recognition, shape management etc all done in C++ code. The app is available for Android and iOS.

Integrating the C++ code with Objective C is much smoother experience than the Java in android.

As the Objective C (more precisely objective C++) is superset of C++, I can directly call C++ code from objective C and for other way communication (calling objective C code from C++), I had defined an C++ interface. The C++ code calls on interface only and the interface is implemented in objective C.

For Android, little more work was required. I had write wrappers code. I also had to hit many stupid issues like local reference table overflow etc.

<comment><USER6> Objective-C also runs on Android with Apportable. Less of a headache. :-)

<comment><AUTHOR> It might, but unfortunately, you can't build a library with it. It only transforms whole apps. Apportable was investigated in the first part here [http://www.skyscanner.net/blogs/developing-mobile-cross- plat...](http://www.skyscanner.net/blogs/developing-mobile-cross-platform- library-part-1-exploring)

<comment><USER7> Not once did the article talk about the C++ ABI which is one of the most important things a library developer needs to be aware of (as opposed to an application developer).

The C++ ABI is unstable. You can't effectively deploy 3rd party libraries with that either.

Android ships with multiple versions of the compiler, each one of those are incompatible with each other's standard library. On top of that, Android ships multiple C++ standard libraries for each compiler, all of which are not- interoperable either. Then Android updates the compiler with each NDK drop introducing more potential incompatibilities.

The end result is you can't deploy prebuilt C++ libraries (a prickly issue for commercial/proprietary code) unless you can force users to use your exact toolchain. But library writers usually can't fairly dictate those terms, especially if the users want intermix libraries from other sources which may use a different toolchain.

</comment></comment></comment></comment><comment><USER6> Or you know you can use Apportable and write Objective-C which is entirely cross-platform. :-) Don't have to sell your soul to C++ if you don't want too and you don't have to deal with the NDK.

If really want to write C++, Apportable also is the only place you will get full C++11 support (using libc++) and the same clang as iOS to make your life easier with just a little bit of constancy.

It will seriously make your life easier (the big thing is not having to maintain Makefiles for Android).

<comment><USER8> If you can use libc++, what's stopping me from doing the same? Why should I use Apportable instead of just heading over to clang's website and downloading/building/linking it myself?

<comment><USER6> It won't work out of the box. There is actually a project in the AOSP world that attempted to get it working but it's labeled experimental and it fails half the tests. There are lot of missing features in Bionic (Android's libc) that libc++ needs that we provide in Apportable and doing so we are able to pass all the tests.

Apportable is more than just C++ though. It's about building a better NDK or rather a platform that makes things consistent across the board with other native environments like iOS. Native in Android is an afterthought which we aim to change. We fix things like libc, C++, and provide you with a lot of low level libraries you expect would be included.

Apportable also is the only place you are going to find a native profiler today (baring awful hacks) or will get the latest version of clang that run on Android either.

The idea is to save you time with inconstancies and make it easier to deal with building native code for both (you can even use the same Xcode project and target both from there).

We also support Objective-C and everything that you get from iOS so you can even bring that code over.

<comment><USER8> Thanks for the detailed response. I'll definitely give you guys a try.

</comment></comment></comment></comment><comment><USER9> I made a painless way to integrate native libraries into Java (Android NDK support too) with some automatic JNI generation [1]. It uses GWT's javascript comment syntax (edit: see [2] for examples).

[1] [https://bitbucket.org/wetherbeei/acceljava](https://bitbucket.org/wetherbeei/acceljava)

[2] [https://bitbucket.org/wetherbeei/acceljava/src/4140d721c11b5...](https://bitbucket.org/wetherbeei/acceljava/src/4140d721c11b564c46c415e8cbc9cbda1ef380f8/testproject/src/edu/illinois/crhc/acceljavatest/Test.java?at=master)

</comment><comment><USER10> Most game engines are built as C/C++ and wrapped with Objective-C, Java, etc wrappers. It is really easy with games because you just need an OpenGL context and some basic views, everything else can happen in the render. When you have to add native UI and integrate with libs like StoreKit, GameCenter, audio etc those are also wrapped.

Still the best underlying tech is C/C++ for games. Even top engines like Unity export to this format. It would be entirely limiting to build an entire engine in Objective-C or Java for mobile multiplatform. C/C++ can be used everywhere with a little bit of wrapping.

</comment><comment><USER11> I'm not quite sure I understand his objections to Xamarin. He says, "It can only output to intermediate Xamarin libraries, not native ones."

On iOS Xamarin does compile down to native code and on Android is compiled to dex format. They are not interpreted on those platforms.

Xamarin can utilize PCL assemblies, so non-ui code could be written once and used multiple places without recompilation. This is actually very handy since those libraries can be built on Windows, where the toolchain is better (IMO VS is better than XS) and the system is more powerful.

<comment><AUTHOR> I'm sorry if this wasn't clear. The purpose of building this library was to have something which could be distributable to any mobile developer, not just the ones using Xamarin. If you did it through Xamarin, you could only give it to Xamarin developers, which is very limiting. We need something that could be importable by most developers, especially native SDK ones.

</comment></comment><comment><USER12> We are developing a game engine in C++ that compiles and runs on iOS, Android, Windows and OSX. We use the Marmalade SDK and though there are some issues with it we are actually running the builds on all these environments without issues which is pretty cool. And if Marmalade goes under we can still port the lower layers to OpenGL.

</comment><comment><USER13> Just because support for multiple platforms is the goal doesn't mean that it has to share the same code base.

Actually, for the author's simple use case (REST API), writing per-platform would probably have taken as much code as the wrappers themselves. Not to mention improved maintainability and a slew of other factors.

Diving down to C++ for cross-platform libraries starts to make sense when there are 4-5+ platforms (certainly not 2), or if there are complex algorithms (certainly not just a REST API), or if there's a very wide range of functionality (i.e. 100+ API calls).

<comment><USER14> C++ is currently my language to go on mobile development, even if I nowadays prefer GC enabled languages, because of:

\- experience with C++ since the ARM was published

\- only language common to all mobile OS SDKs and I not want to rely on third party toolchains

\- C++11 makes it feel refreshing (except for compile times :\ )

\- I am only doing portable graphics related stuff as an hobby

\- RoboVM would be nice, but it is still kind of work in progress

In the OP case, you are right, but it could also be compensated with third party libraries like Qt, instead of the DYI approach that was taken.

However with bigger teams with disparate skillsets, maybe something like Mono would be a better approach.

<comment><USER15> I hope you also support non ARM Android devices like Samsung Galaxy Tab 3 (Anrdoid 4x on Intel Atom x86) and there are also devices with MIPS CPU.

<comment><USER14> So far only hobby stuff, so I only target my own devices, but it is quite easy to support multiple devices.

I am a bit older than C++, so I do know how to write portable code before we had VMs everywhere. :)

</comment></comment></comment></comment><comment><USER16> I think if you want to write xplatform c++, Qt is the way to go. Qt also has QML that solves all the pains of flexible dynamic UI framework.

</comment><comment><USER17> I have a decent code base written in C/C++. Now as I am thinking about mobile apps, I wonder if it is worth all the hassle on Andriod to go through all of this or if it is just better to do a rewrite in Java. Does anyone have any experience about this?

<comment><USER18> That depends on what your "decent code base" is, what it is meant to accomplish, how much coverage of your library is going to exist in your mobile app, whether (and how often) you will actually re-use the core libraries, etc.

Realistically, what the OP is doing is pushing the maintenance and platform- specific code into a different part of the stack. In either case, code in different languages, with different native-interface specifications and idioms, is required to be written and maintained. Unless you have some certainty that the maintenance of the multiple different platform wrapper code bases will be less laborious than simply writing platform-specific code, it's probably not worth it.

In the worst case, for example, a new function in C or C++ that you write may have to have one (or more!) new functions written in the platform-specific wrapper code in order to utilize it.

</comment></comment><comment><USER19> Why use explicit memory management in the iOS wrapper (delete in dealloc) when ARC in conjunction with std::shared_ptr would take care of it automatically?

<comment><AUTHOR> In iOS, things are simple. The deallocating code is very concise and safe. Why having the performance overhead of smart pointers? On the other hand, it could be useful to have them in Java and avoid manual destruction. However, this wasn't investigated :(

</comment></comment><comment><USER20> link to Google cache...

[http://webcache.googleusercontent.com/search?q=cache:Y2AE0GA...](http://webcache.googleusercontent.com/search?q=cache:Y2AE0GAywcYJ:www.skyscanner.net/blogs/developing- mobile-cross-platform-library-part-2-c+&cd=1&hl=en&ct=clnk&gl=us)

<comment><AUTHOR> it's back up

</comment></comment><comment><USER21> Great! Let's do the same thing for Browser Plugins now :)

</comment><comment><USER22> Interesting.

</comment><comment><USER23> this is not open sourced?

<comment><AUTHOR> No, unfortunately, it needs polishing before open-sourcing it. If there's enough demand, I might consider it...</comment></comment>


Pixie: A sweet Clojure-ish language - <AUTHOR> http://blog.goodstuff.im/pixie <comment><USER2> I like the side-note.

> As a side note, expressing configurations in Clojure S-expressions makes a > ton of sense. Clojure S-expressions provide a superset of JSON and Clojure's > eval allows you to define a function to compute certain values. It's an > escape hatch so that you're configuration files don't need to become > accidentally Turing complete. The Turing complete nature of your > configuration files is well defined as Clojure.

I had posted a while back wondering why more formats do not just derive from sexp or SXML (yes, say that out loud in the office). As I think some Lispers (I am even below beginner) cannot help but notice that if sexpr are coincidentally (I am sure it can be done without, but still, I am not sure if McCarthy and company just started with sexp or choose it specifically and held their ground beyond it was their choice) core to the lisp's homoiconic power- features, why more people do not just want sexp as the core data definiton, keep the data and program as close as possible, and just macro the data back, tossing back and forth between code and data as the division is limited.

Anyway, I like that far more intelligent people than me not only like this idea, but are encouraging it and pushing it forward.

(Yes, flame away. I know some people love Lisp and hate, I just thought it is an interesting premise; I am ready for you to throw shoes at me, HN.)

<comment><AUTHOR> Once you use Clojure's EDN, you start to cringe when you see JSON.

Spec - [https://github.com/edn-format/edn](https://github.com/edn-format/edn)

Walkthrough - [http://www.compoundtheory.com/clojure-edn- walkthrough/](http://www.compoundtheory.com/clojure-edn-walkthrough/)

<comment><USER2> I was excited to hear someone's Guile Scheme answere to Sinatra, Artanis (yes, I know, cute), led me to the Summer of Lisp winner list, including this of course and the Learn You a Datalog guy. Naturally, that led me to EDN.

Thanks for reminding my to finished the EDN docs.

[http://www.learndatalogtoday.org/](http://www.learndatalogtoday.org/)

</comment></comment></comment><comment><USER3> This seems like bit of a short sighted move not that I disagree with it.

Project Jigsaw which will be a big part of Java 9 aims to make the JVM more modular which will reduce the memory footprint substantially. Personally I would like to see a version of Clojure that targets this JVM specifically and abandons backwards compatibility.

Leaving the JVM means you abandon the decade of libraries many of which you simply can't get on any other platform (in particular for the enterprise). Given that Clojure has been gaining a lot of ground in these large companies it seems like a missed opportunity.

<comment><USER4> In fact JVM is pretty fast. If you run a simple java app without dependencies and only displaying "Hello world" it will run in about 1s on modern hardware. The problem with Clojure is AFAIR related to parsing and loading a big number of namespaces which is quite slow.

<comment><USER5> 1s to print hello world is an awful lot I would like to point out.

<comment><USER6> Upvoted you because I agree. JVM has many strong points but I don't think anyone is disputing that startup time is a weakness. I've used some java tools that you invoke from a command line and I'm always slightly annoyed by how slow they are.

As an aside - I was trying to reduce -Xmx to improve HelloWorld startup (lol idk...) and Xmx smaller then 1024k and it couldn't start up with less then 1024k so there's that.

On my windows computer (also I barely know what I'm doing so probably even timing it wrong).

      [Mon Mar 09 03:22:31 zebra@ZEBRA:~ ]
      $ time a.exe
      Hello, world
      real    0m0.098s
      user    0m0.015s
      sys     0m0.015s
      [Mon Mar 09 03:22:34 zebra@ZEBRA:~ ]
      $ time java Hello
      Hello, world!

      real    0m0.285s
      user    0m0.000s
      sys     0m0.031s

<comment><USER7> Worry about -Xms rather than -Xmx for startup times.

I suspect there are ways to make it faster (e.g. there's a nashorn runner that starts up appreciably faster than "proper" java), but the main use case for java is server-side programs that run for days or weeks between restarts, so it's optimized for that use case.

<comment><USER8> _the main use case for java is server-side programs that run for days or weeks between restarts, so it 's optimized for that use case._

Unfortunately for the folks who still have to use Java desktop software written in Swing.

</comment></comment></comment></comment></comment></comment><comment><USER9> > And the JVM has the slowest startup time of any runtime I've ever > encountered.

Blame Clojure not the JVM.

[https://nicholaskariniemi.github.io/2014/02/11/jvm-slow- star...](https://nicholaskariniemi.github.io/2014/02/11/jvm-slow-startup.html)

Besides if 0.04s is still too slow, there are quite a few (commercial) AOT compilers to native code available.

However, Pixie does look quite cool.

What I am missing in Clojure is the ability to take advantage of type metadata to compile it AOT to Android Dalvik/ART friendly bytecode.

Apparently not even 1.7.0 will fix the performance issues.

<comment><USER10> .04 is user time. The elapsed time was .12 - this is how long you have to wait. Still it's a fair point: 120ms is noticeable but not painful.

<comment><USER9> Right, I looked into the wrong place.

</comment></comment></comment><comment><USER11> > There are some things that I would like to see from/in Pixie...

> Some form of package distribution support... could this be piggy-backed on > Clojars?

There is Dust :

[https://github.com/pixie-lang/dust](https://github.com/pixie-lang/dust)

which pulls packages from Github.

</comment><comment><USER12> I'm already replacing my project's ancient Go and Rust based services with Pixie powered end points!

<comment><USER13> Could you elaborate some points that motivated you to switch from go? What would be next?

<comment><USER14> This week's hottest language is SVIRFNEBLIN. It's got everything: privilege escalators, compile-to-malgeboge, mini-hdmi, and that thing where your types are all fragments of Sapphic erotic literature!

<comment><USER15> Oh, Stefon, don't ever change.

</comment></comment></comment></comment><comment><USER16> Language and implementation both look very promising! I was looking for small footprint and fast startup modern lisp for a long time for my embed hobby projects. And finally it's here! I'm really excited about this language and where it stands now compared to when I first discovered it.

<comment><USER17> You should check out Scheme implementations like Guile and Chicken.

</comment></comment><comment><USER18> I want to see a clojure clone embedded as a Go scripting language. The immutability means that the interpreter would be mostly threadsafe, and thus would support goroutines well.

<comment><USER7> Embedding an immutable language in a mutable one is a recipe for awkwardness - how would you expose host-language things to the inner language? Better to do it the other way around - mutable scripting language embedded inside immutable host.

<comment><USER19> Clojure is an immutable language built on top of mutable hosts. Interop is handled beautifully. A go hosted Clojure (gojure?) would probably be fine.

<comment><USER7> Calling Clojure from Java is not remotely what I would call beautiful; it's a lot of Strings and pain (especially when you compare to calling Scala from Java). Calling Java from Clojure is substantially more elegant (it requires a structured FFI but Clojure is good at those).

</comment></comment></comment></comment><comment><USER20> Direct link to pixie's github [https://github.com/pixie- lang/pixie](https://github.com/pixie-lang/pixie)

</comment><comment><USER21> Can anyone verify if Pixie has concurrent multithreading or is it limited to greenthreads as described in the article? Based on the fact that atoms are implemented and they would not be necessary in a single threaded environment, I would guess that support for concurrent multithreading would be at least in the works, if not already possible.

</comment><comment><USER22> A quick video if you want to see pixie startup time in action [https://www.youtube.com/watch?v=LlDQTLRrcZI](https://www.youtube.com/watch?v=LlDQTLRrcZI)

</comment><comment><USER23> Dynamic typing and parentheses are what keep me away from Clojure or Lisp like languages. How can I get over them?

<comment><USER24> There aren't many languages that have radically fewer parentheses than Clojure / LISP.

Clojure:

        (defn blub-extra [a b]
            (blub (inc a) (inc b)))

8 parens + 2 brackets

Scala:

        def blubExtra(a: Int, b: Int): Int {
           blub(inc(a), inc(b))
        }

8 parens + 2 braces

Java:

        Integer blubExtra(Integer a, Integer b) {
            return blub(inc(a), inc(b));
        }

8 parens + 2 braces

Ruby:

        def blubExtra(a, b)
            blub(inc(a), inc(b))
        end

8 parens

Python:

        def blubExtra(a, b):
            return blub(inc(a), inc(b))

8 parens, one colon

C:

        int blubExtra(int a, int b) {
            return blub(inc(a), inc(b));
        }

8 parens + 2 braces

It's roughly the same numbers of brackets (or equivalent) in Clojure, Scala, C and Java. A bit less in Python and Ruby.

<comment><USER25> Scala example is wrong, wouldn't even compile; should be:

        def blubExtra(a: Int, b: Int): Int = blub(inc(a), inc(b))

The original would need an equals thrown in there, otherwise it's procedural syntax (IIRC, has been deprecated or will be in the next release) which has a return type of Unit, thus not compiling when specifying a return type of Int.

        def blubExtra(a: Int, b: Int): Int = {
          blub(inc(a), inc(b))
        }

<comment><USER24> Thanks for the typo spot. I've not written Scala for about 5 years. Unfortunately I can't go back and edit my post.</comment></comment></comment></comment>


Curiosity Sends 1st Color Panorama of Gale Crater - <AUTHOR> http://www.nasa.gov/images/content/675227main_pia16029-full_full.jpg

<comment><USER2> It's kind of strange how it looks like it could be a picture of somewhere on earth but it's really much, much colder</comment>


October 21 post-incident analysis - <AUTHOR> https://blog.github.com/2018-10-30-oct21-post-incident-analysis/ <comment><USER3> The timeline of events was interesting (and much appreciated), but the root cause analysis doesn't really go much deeper than "we had a brief network partition, and our systems weren't designed to cope with it", which still leaves a whole lot of question marks.

Of course, without detailed knowledge of how GitHub's internals work, all we can do is speculate. But just based on what was explained in this blog post, it sounds like they're replicating database updates asynchronously, without waiting for the updates to be acknowledged by slaves before the master allows them to commit. Which means the data on slaves is always slightly out-of-date, and becomes more out-of-date when the slaves are partitioned from the master. Which means that promoting a slave to master will _by definition_ lose some committed writes.

If "guarding the confidentiality and integrity of user data is GitHub's highest priority", then why would they build and deploy an automated failover system whose purpose is to preserve availability at the cost of consistency? And why were they apparently caught off-guard when it operated as designed?

(Reading point 1 under "technical initiatives", it seems that they consider intra-DC failover to be "safe", and cross-DC failover to be "unsafe". But the exact same failure mode is present in both cases; the only difference is the length of the time during which in-flight writes can be lost.)

<comment><USER4> I believe you're right. They go into their design in their MySQL HA post:

[https://githubengineering.com/mysql-high-availability-at- git...](https://githubengineering.com/mysql-high-availability-at-github/#semi- synchronous-replication)

> In MySQL's semi-synchronous replication a master does not acknowledge a > transaction commit until the change is known to have shipped to one or more > replicas. It provides a way to achieve lossless failovers: any change > applied on the master is either applied or waiting to be applied on one of > the replicas.

They only require one other replica (as opposed to a quorum) to be reachable from the master for the master to continue acknowledging writes. If a new master has been elected on the other side of the partition, both will continue acknowledging writes.

They noted this as a limitation that they were working on (but unfortunately a bit too late in hindsight):

> Notably, on a data center isolation scenario, and assuming a master is in > the isolated DC, apps in that DC are still able to write to the master. This > may result in state inconsistency once network is brought back up. We are > working to mitigate this split-brain by implementing a reliable STONITH from > within the very isolated DC. As before, some time will pass before bringing > down the master, and there could be a short period of split-brain. The > operational cost of avoiding split-brains altogether is very high.

<comment><USER5> I'm by no means an expert on any of this stuff, but:

> They only require one other replica (as opposed to a quorum) to be reachable > from the master for the master to continue acknowledging writes (from your > comment)

> Orchestrator considers a number of variables during this process and is > built on top of Raft for consensus... (from the article)

Doesn't quite make sense to me. Doesn't raft require the master to wait for a quorum before committing writes? I understood it as a pretty important aspect.

> A candidate must contact a majority of the cluster in order to be elected, > which means that _every committed entry must be present in at least one of > those servers_ (from the raft paper [1])

I understood that the bold is only true if commits are acknowledged by the quorum

Edit: I'm not implying your post is incorrect, just trying to understand how the two fit together

[1]: [https://raft.github.io/raft.pdf](https://raft.github.io/raft.pdf)

<comment><USER6> This was my first thought, you beat me to it. It seems like they contradict each other. Is it Raft with quorum or is it a single replica node that is caught up with master? You can't have both (unless you only have three nodes).

<comment><USER4> > unless you only have three nodes

Interestingly enough this seems to be how managed sql solutions do it. You just have one primary and one failover replica in another zone. So a quorum write is just a synchronous write to both (2 out of 2). You don't have the split-brain problem because the original primary can't make progress if it can't contact the replica.

[https://cloud.google.com/sql/docs/mysql/high- availability](https://cloud.google.com/sql/docs/mysql/high-availability)

[https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Conce...](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html)

Github's problem is that they were trying to be too smart and allow any of their replicas to be candidate masters without increasing their quorum size. In theory this is higher availability than the managed sql solutions (they can be available even if their entire coast gets nuked) but they do it at the cost of consistency in the more common failure scenarios.

</comment></comment></comment></comment></comment><comment><USER7> > With this incident, we failed you, and we are deeply sorry.

I'm always impressed when people actually apologize instead of dancing around an almost apology.

Overall this is one of the best post mortems I've seen - great tone, very well written, super informative, has all of the steps (apology, information on issue, real steps to prevent it happening again) that hurt customers generally want to see. Really impressive timelines too - 43s reconnect after initial issue, 15 minutes to change status.

Gitlab overall seems to have more incidents like this and I really like their custom of working through them in public Google docs. Definitely seems a better idea for incident communication than relying on your own tech (github pages) for incident communication.

<comment><USER8> Definitely checkout Gitlab's post mortem from last year [https://about.gitlab.com/2017/02/10/postmortem-of- database-o...](https://about.gitlab.com/2017/02/10/postmortem-of-database- outage-of-january-31/)

It's very informative and also dives deep into their "Whys" pertaining to the outage.

<comment><USER9> Thanks for sharing that. We are glad that you appreciate our transparency which is one of the core values at GitLab.

</comment></comment></comment><comment><USER10> Disclosure: I work on Google Cloud.

I'm a little confused by this part:

> While MySQL data backups occur every four hours and are retained for many > years, the backups are stored remotely in a public cloud blob storage > service. The time required to restore multiple terabytes of backup data > caused the process to take hours. A significant portion of the time was > consumed transferring the data from the remote backup service. This > procedure is tested daily at minimum, so the recovery time frame was well > understood, however until this incident we have never needed to fully > rebuild an entire cluster from backup and had instead been able to rely on > other strategies such as delayed replicas.

At first, I had assumed this was Glacier ("it took a long time to download"). But the daily retrieval testing suggests it's likely just regular S3. Multiple TB sounds like less than 10.

So the question becomes "Did GitHub have less than 100 Gbps of peering to AWS?". I hope that's an action item if restores were meant to be quick (and likely this will be resolved by migrating to Azure, getting lots of connectivity, etc.).

<comment><USER11> This was an interesting point, but prefixing it with a call to authority was both needless and distracting. Why did you include it?

<comment><USER12> I didn't read the Google Cloud part as an assertion of authority, just as a disclosure - if they're talking about competitors (and especially how the choice of competitor negatively impacted GitHub) I appreciate it.

(disclosure - I work for a competitor, not on cloud stuff)

<comment><USER13> I also appreciate it. Its very common for owners/employees to criticize/attack competitors online anonymously. While the GP wasn't attacking, its just nice to know he works for a competitor.

</comment></comment></comment></comment><comment><USER14> In my career, the worst outages (longest downtime) I can recall have been due to HA + automatic failover. Everything from early NetApp clustering solutions corrupting the filesystem to cross-country split-brain issues like this.

Admittedly, I don't recall all the incidents where automatic failover minimized downtime, and probably if a human had to intervene in each of those, the cumulative downtime would be more significant.

But boy, it sure doesn't feel like it.

<comment><USER15> In cases where you can rely on the system to self-repair, automatically moving writes quickly seems reasonable. But otherwise, it seems like you want the system to cope with the situation where writes are unavailable -- clearly a lot of things will be broken, but if writes fail fast, reads are still viable.

Assuming you have that, it's OK to rely on a human to assess the situation, make sure the dead master is really dead, salvage any partially replicated transactions, and crown a new master. With the right tools, it could take only a few minutes -- a bit longer if you have to wait for the old master to boot to see if it had locally committed transactions that didn't make it to the network. If it takes 5 minutes to resolve this (including time to get to the console), you can do this ten times a year and still have three nines.

For the more likely case where it's a network blip, the situation resolves itself (in a nice way) by the time the operator gets to the console.

<comment><USER16> Indeed, given recent history I'd almost suggest it is better to take the site down for a few minutes than let the automatic failover systems put you into a 24-hr degraded service situation.

</comment></comment></comment><comment><USER17> Reading this post-mortem and their MySQL HA post, this incident deserves a talk titled: "MySQL semi-synchronous replication with automatic inter-DC failover to a DR site: how to turn a 47s outage into a 24h outage requiring manual data fixes"

[https://githubengineering.com/mysql-high-availability-at- git...](https://githubengineering.com/mysql-high-availability-at-github/#semi- synchronous-replication)

> In MySQL's semi-synchronous replication a master does not acknowledge a > transaction commit until the change is known to have shipped to one or more > replicas. [...]

> Consistency comes with a cost: a risk to availability. Should no replica > acknowledge receipt of changes, the master will block and writes will stall. > Fortunately, there is a timeout configuration, after which the master can > revert back to asynchronous replication mode, making writes available again.

> We have set our timeout at a reasonably low value: 500ms. It is more than > enough to ship changes from the master to local DC replicas, and typically > also to remote DCs.

[https://blog.github.com/2018-10-30-oct21-post-incident- analy...](https://blog.github.com/2018-10-30-oct21-post-incident- analysis/#2018-october-21-2252-utc)

> The database servers in the US East Coast data center contained a brief > period of writes that had not been replicated to the US West Coast facility. > Because the database clusters in both data centers now contained writes that > were not present in the other data center, we were unable to fail the > primary back over to the US East Coast data center safely.

> However, applications running in the East Coast that depend on writing > information to a West Coast MySQL cluster are currently unable to cope with > the additional latency introduced by a cross-country round trip for the > majority of their database calls.

</comment><comment><USER18> > Connectivity between these locations was restored in 43 seconds, but this > brief outage triggered a chain of events that led to 24 hours and 11 minutes > of service degradation.

Network computing is rough. (I am not being at all sarcastic).

<comment><USER19> 2000x multiplier between connectivity loss and service degradation.

<comment><USER20> This is rather common; too many HA technologies really aren't designed for wide-area failover, or anything besides the total and complete failure of a node.

I once saw a ten-second "grey failure" in the network cause an entire SAN cluster to go read-only for a day while it checksummed all replicas.

If you don't Chaos-monkey test these things in production, they really won't work when needed. And you can't just "unplug a box" in your testing, you need to test slowdowns and intermittent errors under production loads.

Few enterprises are willing to risk or invest in that sort of testing. So we all hope for the best and point fingers upstream at vendors when the HA doesn't work as advertised.

<comment><USER21> I've suffered more outages at the hands of HA database technologies (that were administered in-house) than such have prevented. There are very few organizations I trust to run them properly.

<comment><USER20> I've always stuck with log shipping to a warm standby wherever possible for the same reason. Well-managed single instance databases can do 3+ nines easily; very few applications actually require more availability than that. Especially at 5x the cost in opex for a theoretical-but-not-really-wink-wink extra 9.

<comment><USER18> In github's case, of course, they can't do a single instance db for performance reasons, even before you get to HA. If I understand it right.

<comment><USER22> Depends on how your architecture, github as a whole likely has volume of writes greater than a single server instance, however either components could be split in separate dbs ( seems to be their approach) or shard / partition the data

</comment></comment></comment></comment></comment></comment></comment><comment><USER23> I find their incident management times pretty impressive. 2 minutes to detect & alert, ~2+ minutes response, 10 minutes to initial triage, 15 minutes to internal change control, 17 minutes to escalation & public communication, 19 minutes to major incident escalation & broad engagement, 73 minutes to remediation start & further public communication.

Initial triage inside 10 minutes then change control, escalation, major incident, and public communication within another 9 minutes. That's hard to beat with humans in the loop.

<comment><USER24> Most places I've seen have similar times to that, all except the public communication bit.

Lots of companies are wary about letting engineers declare downtime directly - it could have costs down the line giving customers refunds etc.

Getting legal, PR, and 3 levels of management to all agree on a message before it gets published is a recipe for the downtime notice taking hours to be published.

<comment><USER23> Do you have a NOC or similar dedicated group? Both initial engagement and their escalation path had a ~2 minute response time. Paging oncall team member, vpn connection, etc, takes ~5-10 in my experience. Similarly their incident manager apparently assessed the situation and further escalated in ~2 minutes as well. Thats pretty dang fast for coming up to speed on a as yet to be diagnosed major incident.

WRT to public communication, yeah. Overly onerous and adhoc seems to be the norm. But we do have "analysts" publically using status post count as inverse evidence of reliability. Creates some real bad incentives for the actual customers.

<comment><USER24> For a core service, github is sure to have a 5 minute human SLA from dedicated SRE's who have an on call rotation etc. That means someone is already at their computer, logged in, and ready to go. They'll typically be investigating within 30 seconds.

Probably within another 30 seconds that human sees many alerts coming in that don't look like false alarms and hits the 'escalate' button.

If it's within office hours, anything that looks major will usually have a bunch of SRE's on the case within 3 or 4 minutes. Usually in various roles from 'I acknowledged this page, it's my job to fix it!' to 'I'm just laughing at your misfortune, but paying a little attention just incase the broken thing turns out to be that change I made yesterday and I have to take over mitigation efforts'.

</comment></comment></comment></comment><comment><USER25> It's great to see how open they are about what happened and how it got fixed. I also appreciate the decision to prioritise _data integrity_ over _site usability_ too.

Looks like Github could be OK after the Microsoft acquisition.

<comment><USER26> > I also appreciate the decision to prioritise data integrity over site > usability too.

That actually goes against what happened; they seemed to have implemented an automated failover which resulted in a split-brain situation rather than letting one DC completely degrade to read-only.

However mistakes happen and hindsight is 20-20.

</comment></comment><comment><USER27> Offtopic, but this reminds me: did google ever issue an incident report on why youtube went down earlier this month? A quick search didn't turn up anything.

<comment><USER28> I haven't seen anything disclosed. Best I've found are articles from 5+ years ago about previous YouTube outages.

</comment></comment><comment><USER29> There is often a trade-off between a large distributed central store and several independent ones. The primary con of the former being incidents like this one, whereas the con of the latter is separate systems to perform eventually consistent aggregation to support some centralized features adding complexity. I wonder if there is any value in GitHub decentralizing the metadata pipeline. So many of the actions are namespaced that this could be reasonable theoretically, at a large practical cost.

On a related note, I often reach for Cassandra when starting projects knowing that building my application around its limited access approach has data replication benefits in the future. For all the flexibility benefits to devs given by SQL/RDBMSs, there are flexibility downsides.

<comment><USER30> An interesting point. Namespaced data would include anything related to a repo, github pages domain or organization, ie. almost everything. This would therefore all be readily shardable. Non-namespaced are basically just user credentials and settings. However, looking forward in a post-GDPR/China data law world, "network political bloc" based sharding of user data may also be either a regulatory requirement or a worthwhile forward-looking protective measure. (We just had this conversation yesterday at Infinite Food.) One organization I believe global user data sharding goes on at already is Google.

</comment></comment><comment><USER31> Sounds like a ton of complicated, fragile work done under crunch times and probably lots of stress. Hats off to the teams at GitHub.

</comment><comment><USER32> If I read things correctly, they made a fairly... interesting... tradeoff: 954 still-as-yet-unreconciled DB writes in exchange for 24 hours of site downtime.

I think I'd have made a different choice, but cool that they were upfront about it.

<comment><USER33> > we captured the ... writes ... that were not replicated ... For example, > __one of __our busiest clusters had 954 writes

Their wording ("one of") makes it sound like they had up to perhaps one order of magnitude more (1k-10k), but they do not actually give us a useful number, merely say "it wasn't too much" and "one of an unknown number of total clusters had 1k".

> I think I'd have made a different choice

I think you might misunderstand how these things go and the tradeoffs.

At the beginning of the incident, they find themselves with lots of writes in the west-coast master which aren't in the east-coast one, and some in the east-coast one that aren't in the west-coast one.

Orchestrator cannot promote a working master because they have diverged.

Your choices are:

1\. Lose data (large unknown amount) by dumping an hour of west-coast data and going back to east-coast data, can be done in maybe 3 hours total by just deleting west-coast and serving all traffic from east-coast for as you rebuild the west-coast cluster.

2\. Lose data (small amount) by rebuilding east-coast from a backup so it can be promoted to, replicating west coast data to it, promoting to it (what they did, 24 hours of time), and then try to manually fix up the small amount of lost data after-the-fact (ongoing)

3\. Develop tooling to automate the reconciliation of data while the site is down such that the east-coast side can be merge-promoted without a rebuild, probably takes at least 3 days to build and might break everything, but if it works it probably merge-promotes in under an hour.

4\. Keep the site down until east-coast data is manually reconciled, still requires rebuilding east to promote west to it, but then requires manually handling some lost writes... probably about 30-40 hours total downtime.

5\. Update the application servers to work fine when the us-west DC is the master, probably about 2 months of development with the site down for the duration.

Which choice would you have made instead? I'm willing to bet they went with either 2 or 4 (and if 4, changed to 2 when it took longer than expected). They probably assumed it would take about 4 hours, and then it simply took much longer than expected because computers are complicated, it turns out.

<comment><USER32> I'd have opted for 1 personally (losing 30 minutes of data) assuming I knew the alternative was 24 hours. I'll grant you they probably figured it would be much faster than that.

Alternatively, why not just let the West-Coast replicate to an East-Coast slave, and switch em. Surely the peering bandwidth between West-Coast/East- Coast is higher than whatever they were doing pulling full backups from the cloud?

<comment><USER33> > I'd have opted for 1 personally (losing 30 minutes of data) assuming I knew > the alternative was 24 hours [of read-only access to prod]

With no offence meant, I'm glad you don't get to make decisions at github then. I'd lose faith in github for that since their primary purpose is to store other people's data (git blobs+metadata, comments, etc). If it was their data, it could be fine, but since it's user data, I don't think losing it is acceptable, even if that means being read-only for an entire week.

> why not just let the West-Coast replicate to an East-Coast slave, and switch > em?

At the time of the incident, east-coast has already diverged, so it's not possible to replay logs from west-coast without moving east-coast back in time to some point prior to west-coast (but a point recent enough that west-coast still has full replica logs to replay).

It's not possible to simply "rewind" a mysql database typically without restoring a backup.

It's not possible in many mysql clustering solutions to catch-up to a peer unless you're already within a recent time window where the replay logs are still available.

As such, the only option is to restore from backup and then catch up, which is what they did.

I assume if taking another backup of west-coast, transferring it to east-coast over the DC link, and restoring it was faster, they would have done it, but I would be totally unsurprised if that was about the same total time.

I'd also say that if you have a practiced procedure (restoring from a scheduled backup in the cloud) vs an ad-hoc procedure (custom backup, custom storage and transfer), the former is probably much safer to do when you're under fire and want to minimize risky moves stressed engineers have to make.

<comment><USER32> The source code is paramount, no argument there. But this wasn't about that. I find the metadata less important, but as you say, I don't make the decisions. The factor you aren't considering is opportunity cost given productivity loss. In your hyperbolic example where you claim you'd rather they were read-only for a week than lose 30 minutes of data? I'd rather get things done during that week personally, and if the price I have to pay is resubmitting a comment or re-approving a PR, I guess I'd pay it.

Also, MSSQL has had quick delta snapshot/restores forever (assuming you have them turned on and going every hour or so), mysql really does not have a similar feature?

<comment><USER34> MySQL does not have any kind of native snapshotting capability, no.

It is becoming somewhat common to run large MySQL deployments on filesystems which do (like ZFS) to get a similar effect, albeit without the ability to restore a snapshot on a host while that database host is online. There is even some experimental work going on in the community to use ZFS snapshots as a state-transfer mechanism for laggy nodes in a Galera MySQL cluster, which seems promising.

</comment></comment></comment></comment></comment></comment><comment><USER35> I'm not super versed in MySQL failover, but am I correct to conclude the Orchestrator and RAFT also did them in? And isn't that architectural components right to existence to exactly prohibit such a situation from happening? Genuine question.

<comment><USER36> Orchestrator and RAFT did not help them, but mostly it was their configuration. The setup allowed an automatic failover to the west cost data centre, so when the connectivity failed the tools did their job and voted in the west coast.

The solution applied is to prohibit cross-country failover in Orchestrator, but allow it to continue doing failover within regions.

</comment></comment><comment><USER37> Postmortems are always really interesting, if you're preparing for an interview where system design will be discussed you could do worse than incorporate reading a couple to help in preparation.

<comment><USER38> I wondered "is there a list of interesting postmortems, like there is for 'falsehoods programmers believe' posts?", and found one at [https://github.com/danluu/post-mortems](https://github.com/danluu/post- mortems)

<comment><USER39> a good halloween read for sure!!!

</comment></comment></comment><comment><USER40> >At 22:52 UTC on October 21, routine maintenance work to replace failing 100G optical equipment resulted in the loss of connectivity between our US East Coast network hub and our primary US East Coast data center. Connectivity between these locations was restored in 43 seconds

lesson: a technician replacing a switch must be able to do it faster than the leader heartbeat timeout of the consensus protocol. (reminds me how in high school we trained to very quickly disassemble/assemble Kalashnikov machine gun - something like under 20 sec. total - the whole choreographed sequence of movements was learned and practiced like a samurai sword kata :)

</comment><comment><USER41> >> It's possible for Orchestrator to implement topologies that applications are unable to support, therefore care must be taken to align Orchestrator's configuration with application-level expectations.

So basically Orchestrator acted correctly but application layer is not nicely integrated with it? Sounds like something not well designed on application side, which offests the whole point of global consensus. Not much detail provided for what exactly went wrong on this though.

They mentioned that they'll take extra care of this but I'm still very concerned about the reality that two systems (orchestrator & app) are so loosely coupled.

<comment><USER42> I don't think that's the right interpretation of it. The "unable to support" in this case seems to have been "can't reliably work due larger latency between components". A tighter coupling between the two would likely not have encoded this knowledge either.

</comment></comment><comment><USER21> It seems like the obvious play here is to fail-over all workloads to the west coast, so they don't incur cross-regional latency. Do they explain why this wasn't possible? If so I cannot find it.

<comment><USER43> > All database primaries established in US East Coast again. This resulted in > the site becoming far more responsive as writes were now directed to a > database server that was co-located in the same physical data center as our > application tier.

All their application servers were in the east coast DC. There wasn't anything to fail over to in the west coast DC. Why they had this fail over config for their DB is a bit unusual given the topology, it seems like they probably meant to use these replicas purely for DR.

<comment><USER21> The quote does not address my question. It just says that everything was fine once their masters were in the east again with their application servers. The question of why there are no application servers in the west to fail-over to is what I'm curious about.

</comment></comment></comment><comment><USER44> This is a great write up - making these public with all the gory technical details helps us all be better at our jobs.

I'm curious whether shutting down the east coast apps entirely and running off of the west coast was considered? Not enough capacity or some other problem?

edit: I guess the third technical initiative strongly implies that it was just a capacity issue: "This project has the goal of supporting N+1 redundancy at the facility level. The goal of that work is to tolerate the full failure of a single data center failure without user impact."

<comment><USER45> In the article they said that both regions consumed writes and got into an inconsistent state during the process.

> This effort was challenging because by this point the West Coast database > cluster had ingested writes from our application tier for nearly 40 minutes. > Additionally, there were the several seconds of writes that existed in the > East Coast cluster that had not been replicated to the West Coast and > prevented replication of new writes back to the East Coast.

<comment><USER44> As I read it their solution to the east coast write conflict was to just put those aside and reconcile them later (and they are still working on that) - <USER2> don't think that would impact their ability to turn off east coast app servers too.

</comment></comment></comment><comment><USER46> It's better to go down hard than to have a crappy fix that tries to keep things alive.

</comment><comment><USER47> On a side note, does anyone know if anything special was used to create the descriptive images in this blog post? They look great and describe connections between related regions really well.

Eg [https://blog.github.com/assets/img/2018-10-25-oct21-post- inc...](https://blog.github.com/assets/img/2018-10-25-oct21-post-incident- analysis/recovery-flow.png)

<comment><USER48> The engineering team sketched them out by hand and were later translated into vectors by the design team.

The vector graphics were made in Sketch ([https://www.sketchapp.com](https://www.sketchapp.com)). Another popular alternative at GitHub is Figma ([https://www.figma.com/](https://www.figma.com/)).

</comment></comment><comment><USER49> Basically

"..we will also begin a systemic practice of validating failure scenarios before they have a chance to affect you. This work will involve future investment in fault injection and chaos engineering tooling at GitHub."

</comment><comment><USER50> The first diagram labeled "Normal Topology" shows Master in East and no other master. Later they acknowledge that lots of stuff doesn't work if Master is not in East because of latency. So then there's all this Orchestration, and it never could have worked in the first place?

That seems incredible - what am I missing?

</comment><comment><USER51> The CAP theorem strikes again.

</comment><comment><USER52> Untested RPC timeouts strike again. Every service needs integration tests that exercise timeouts. Some config and code can't be exercised in automated tests and needs regular disaster readiness testing.

Service client libraries need unit tests that show the library returning expected errors for all the failure scenarios: missing config, name lookup failure, unreachable, refusing connections, closing connections, returning error responses, returning garbage responses, refusing writes, responding with high latency, and responding with low throughput.

</comment><comment><USER53> This makes me wonder why GitHub is effectively using a single database (though replicated with HA considerations and CQRS-ized).

From the article, it appears they are are partitioning based on function (commits in this DB, PRs in this cluster)... but I just don't see a strong business need to glob all commit data together into one massive datastore. Perhaps it's an economy-of-scale driver?

<comment><USER54> This comment... I've thought about this a lot. I think asking this question is probably the most meaningful insight into this failure that has appeared.

What GitHub has done is create a giant single point of failure in the form of a globe spanning under-tested mysql database to manage a vast number of unrelated git repos, most of which are dormant. At any moment only a minuscule fraction of these repos are actually mutating.

One can imagine a system that doesn't conflate every GitHub repo into a monster distributed mysql database. Each repo could be coupled with an independent SQLite database, for instance, that spins up in a few microseconds and synchronizes with its distributed copies only when necessary. Otherwise these lay dormant and available, safe from the vagaries of some under- specified planet scale mysql instance.

I imagine testing such a design would be vastly easier as well; one need not replicate the conditions of this mighty distributed mysql construct. Whatever aggregation is ultimately necessary (analysis, billing, etc.) could be performed with asynchronous ETL into some database with minimal coordination and no risk to availability for customers.

</comment></comment><comment><USER55> No reflection on the root cause? I am not into hardware but shouldn't there an redundant network connection in place or at least set one up?

</comment><comment><USER56> Very impressive complexity. Really appreciate the transparency. Much has been a couple of stressful days.

</comment><comment><USER57> Didn't lose data perhaps, but sounds like information was definitely lost and/or corrupted.

</comment><comment><USER58> I didn't learn any of this in school. Any MOOCs for cloud architecture?

</comment><comment><USER59> So they geographically replicated their mysql in order to survive such a partition and instead they destroyed their entire database and now have no reason to believe any of it is consistent at all.

</comment><comment><USER60> Just five days before Microsoft completed the GitHub acquisition so they had to lower their QoS to meet the expectations of everyone using Azure...</comment>


Aminal: Golang terminal emulator from scratch - <AUTHOR> https://github.com/liamg/aminal <comment><USER2> I hadn't heard of SIXEL ([https://github.com/saitoha/libsixel/blob/master/README.md](https://github.com/saitoha/libsixel/blob/master/README.md)) before. It's cool to see a standard for bitmap graphics in the terminal catching on.

<comment><USER3> I think currently one of the most popular Linux termemu is kitty which also supports this. So its definitely catching on.

<comment><USER4> KiTTY is useful for Windows users but are any significant amount of people using it on Linux? I would imagine that aside from people using the default terminal emulators that come with their distro or desktop environment like xterm, gnome-terminal and konsole, the most widely used terminal emulator on Linux would be urxvt.

Personally I use terminology which I am satisfied with. [https://www.enlightenment.org/about- terminology.md](https://www.enlightenment.org/about-terminology.md)

<comment><USER5> OP is talking about this Kitty ([https://sw.kovidgoyal.net/kitty/](https://sw.kovidgoyal.net/kitty/)), not this one [http://www.9bis.net/kitty/](http://www.9bis.net/kitty/).

<comment><USER4> Ah, I see. Thanks.

</comment></comment></comment></comment></comment><comment><USER6> While a GPU accelerated terminal emulator sounds great on paper, do keep in mind that GPU acceleration doesn't really give you that much in terms of raw performance. You still have to render glyphs individually, almost always on CPU. So, the reality is probably more that GPU acceleration gives you more flexibility, rather than giving you performance. Anecdotally, I just ran this and it takes roughly twice as long to slog through my dmesg as gnome-terminal.

That said... pretty cool. I don't know if there already existed VT100 emulation written in Go, but it doesn't hurt to have that. Plenty of applications might want to embed a terminal or otherwise have terminal emulation.

<comment><USER7> With a fixed width font in a single size surely caching glyphs goes a long way? Most of the characters probably come from a small set of characters, and obviously a cache can support full unicode. I guess you couldn't do subpixel hinting but an alpha blended character can be tinted any colour for FG/BG.

<comment><USER6> You can do caching with any renderer, GPU or not. Almost any text renderer caches glyphs to some degree. Of course, even fixed width fonts can have things like ligatures and composite characters which complicate matters. Rather than caching individual code points, you'd really need to cache something more like individual grapheme clusters.

Subpixel rendering is tricky, but should be possible to do with shaders. You could just render to a normal single channel texture at 3x the horizontal spatial resolution (basically, stretched 3x horizontally) then when rendering move 1 pixel across the glyph rendering for each subpixel, alpha blending individually.

<comment><USER7> Ah true. I guess I personally do not use a font with ligatures so it didn't really come to mind.

Subpixel renderering with shaders is a neat idea, is it something that has been done before?

<comment><USER6> I don't know; I assume yes because it seems possible and I doubt I'm the very first person to think about it.

</comment></comment></comment></comment></comment><comment><USER8> it's great to see more terminal emulators, especially ones that have good unicode support and are trying to be faster than the average garbage.

There's at least one other terminal emulator that does things akin to the image-catting and viewing colors when you click them, namely terminology [0].

There's also at least one other that uses opengl for faster rendering (alacritty [1]).

I don't know of any other that do both these things; use opengl for speed and also innovate in interactivity.

[0]: [https://www.enlightenment.org/about- terminology.md](https://www.enlightenment.org/about-terminology.md)

[1]: [https://github.com/jwilm/alacritty](https://github.com/jwilm/alacritty)

<comment><USER9> Also: iTerm2 can use macOS' Metal graphics API to great effect: [https://gitlab.com/gnachman/iterm2/wikis/Metal- Renderer](https://gitlab.com/gnachman/iterm2/wikis/Metal-Renderer)

</comment></comment><comment><USER10> Sixel is great. But you could also interpret the headers of common image file formats as terminal escape sequences, so that you can directly "cat lena.png" and see the image. Now, _that_ would be a real hack!

<comment><USER11> The terminal couldn't do that as it wouldn't know the difference between a user typing `cat lena.png` into Bash from typing the same sequence of characters into vim (for example).

This is why there are separate commands like `icat` which will output ANSI escape sequences for rendering a PNG. Unfortunately a great many of these are terminal specific (iterm, terminology, kitty and VT200/VT300 (sixel) all have their own unique escape sequences. The closest to a standard we have is sixel and, frankly, I don't think it's that good on modern systems[1]). So you end up with different command for each "standard" you want rendered.

The approach I've taken in my own $SHELL is to have a builtin command like `cat` (I call it `open`) and that auto-detects which terminal emulator you're running and picks the right encoding to match what is supported by the terminal (if no standard can be auto-detected, then it falls back to ANSI art using coloured blocks). However the issue with that is now people need to run new custom shell instead Bash / Zsh / whatever. But the logic behind the auto- detection is really quite simple so that could be written it's own standalone tool.

[1] sixel re-encodes images in character blocks of 6 pixels. Where as iTerm, Kitty and Terminology send the image as is over the terminal via a base64 encoded string. All those solutions are inlined via ANSI escape sequences though.

<comment><USER10> > The terminal couldn't do that as it wouldn't know the difference between a > user typing `cat lena.png` into Bash from typing the same sequence of > characters into vim (for example).

This is not what I am talking about. I mean the binary headers inside png, jpg, tiff files, etc. These binary sequences can be easily interpreted as terminal control sequences, and they are very unlikely to appear otherwise. For pnm files I agree that the situation is a bit more delicate, as these sequences are likely to appear in the wild (e.g., in the pnm man page for example).

<comment><USER11> Ahhh. That's an interesting idea. I like it

<comment><USER10> This is an inevitable feature of the perfect terminal. The alternative is showing the binary contents of the image file in ASCII, which is always undesired.

<comment><USER11> Not just undesired; it can sometimes play havoc with your terminal session. I've had tmux status bars break; bash prompts go "weird" and all sorts of other glitches before when I've accidentally piped binary data to the STDOUT.

I'm very tempted to fork Aminal and have a play with your idea.

<comment><USER10> That would be lovely! I do a lot of image processing using unix pipes, and when I forget the "|display" at the end it breaks everything.

Notice that you do not even need to use sixels in your case. Since you control the rendering, you can simply put the pixels of the image on a rectangle (that allows paning and zooming, for example), and move the cursor below it.

<comment><USER11> It wouldn't be until I'm off for Christmas holidays (at the earliest) though. I don't have a whole lot of time left between work and family; and have a few projects on the go as it is already.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER12> This is neat, and it's important to iterate on things we take for granted like what terminal emulator we use.

But a larger question, at what point do we focus on the novelty of the project, not that it was written in a specific language?

<comment><USER13> Once we've rewritten everything in go. More realistically, once the language is stable and known enough that these things become uninteresting.

<comment><USER14> Could you elaborate? I don't understand what you are implying

</comment></comment></comment><comment><USER15> It's great that there's a hotkey to web search for the current selection, but I'd not be surprised to see people requesting that it not be google. Know your audience and all that.

<comment> yjftsjthsd-h Seems like a config option would be easy enough.

</comment></comment><comment><USER16> Looks very cool. What's the timeline for it being released for Windows?

<comment><USER7> It would be great to see this working with the new ConPty API.

Note: I work for Microsoft but I have never touched anything to do with ConPty.

</comment></comment><comment><USER17> > Retina display support

I want to know what is difficult to support this.

<comment><USER18> This may be referring to running under Linux with a high DPI monitor - a lot of programs which are naïve about this don't scale properly and are teeny- tiny.

</comment></comment><comment><USER19> Isn't OpenGL deprecated now? iTerm recently switched to Metal.

<comment><USER20> Since it's Linux compatible it's much simpler to support a global standard like open gl and mostly work everywhere rather than use whatever apple is pushing now and also support open gl anyway for Linux.

I assume they're open to contribution if someone wants to add Metal support for mac osx.

</comment></comment><comment><USER21> ETA on tmux -CC support?

</comment><comment><USER22> I knew this would be created eventually but I was thinking it would just be a shell. This looks impressive at first glance, implementing everything with OpenGL.

<comment><USER23> I would have loved to see someone with a plan9-golang background creating a modern 9term-like "dumb" terminal. One that is mostly an editable text buffer with mouse plumbing capabilities.

Nowadays, terminals within vim/emacs is what's closer, feature wise. Not quite dumb though.

</comment></comment><comment><USER3> Does it not use VTE? If so that would be great.

Also, to the other commmentator- be glad its not written in "modern web technology". I keep 20 terminals open usually.

<comment><USER24> Edit: it's GTK that provides multithreading functionality, not VTE itself.

VTE based terminals are actually multithreaded globally, so you only ever have one process regardless of number of tabs/windows open.

But I agree, VTE is something I avoid ever since the scrollback fiasco. (For the uninitiated: the library wrote all scrollback to disk, which is not something end-users might expect security wise. Scrollback is encrypted now, but it appears you still cannot disable writing to disk, as e.g. the output of dmesg, consisting of 62210 bytes of output on my machine, causes 5-digits of bytes to be written each time on any VTE-based terminal I've tried even with "infinite" scrollback disabled. And none appear to honor TMPDIR).

<comment><USER25> I wish I understood it enough to throw it into GDB - it looks like it uses O_TMPFILE, but I don't think that quite gives you what you want.

[https://github.com/GNOME/vte/blob/65d67f6f814df4f4ab800898bb...](https://github.com/GNOME/vte/blob/65d67f6f814df4f4ab800898bb7f4b1bc9f135b0/src/vteutils.cc#L50)

<comment><USER24> The O_TMPFILE flag is just an anonymous file as far as I understand. From that source file, VTE uses glib, particularly g_get_tmp_dir() and g_file_open_tmp() (which uses the latter), which claims to honor TMPDIR according to the docs: [https://developer.gnome.org/glib/stable/glib- Miscellaneous-U...](https://developer.gnome.org/glib/stable/glib- Miscellaneous-Utility-Functions.html#g-get-tmp-dir)

Perhaps a terminal emulator is unable to read variables exported in .bashrc, as the terminal is started prior to executing the shell.

</comment></comment></comment></comment><comment><USER3> Cannot edit previous comment so adding new-

Can you please add some details on Unicode support? Most terminals claim to have it but fail miserably on Asian (Asia continent) fonts.

1) RTL for west Asia

2) Indic fonts and dependent vowels

3) Double width cjk.

I genuinely believe that without all 3, no terminal emulator deserves to claim Unicode support.

<comment><USER26> And I am not sure how many of them support Arabic right to left either..

<comment><USER3> Arabia is west Asia :)

<comment><USER27> Or rather, Middle East.</comment></comment></comment></comment>


The Ping-Pong Theory of Tech-World Sexism - <AUTHOR> https://medium.com/matter/the-ping-pong-theory-of-tech-world-sexism-c2053c10c06c

<comment><USER2> It's not so much a theory as a collection of vignettes based on one individual's situation in their company.

I do agree with getting good at ping pong or whatever the office choice of communal activity is.

Playing foosball / Mario Kart / basketball with the whole gang in various roles definitely helped with gaining acceptance and enabling better dialogue during working hours - guess what, it's called team building and spans across sexes.

</comment><comment><USER3> "If there's not a penis or an MBA involved, a decision can not be made."

So she not only reduces men to their genitals and implies that men rely on them when making decisions, but considers it equivalent to, say, their college degrees, in assessing relevancy. We're supposed to listen to this person on matters of sexism?</comment>


CC500: a tiny self-hosting C compiler - <AUTHOR> http://homepage.ntlworld.com/edmund.grimley-evans/cc500/ <comment><USER2> Nice hack! The next step would probably be a CC which compiles to a minimal virtual CPU instruction set so that we just need some Assembler macros to port it to a native platform (x86, ARM, etc.), plus a library of formats (ELF etc.) to create applications for those platforms.

<comment><USER3> This was here just 3 days ago: [https://news.ycombinator.com/item?id=8558822](https://news.ycombinator.com/item?id=8558822) ; Now, YOU just need to write those assumbler macros. But why? Are you in some shortage of working compilers?

<comment><USER2> > Are you in some shortage of working compilers?

I am just interested in a small maintainable C compiler which can _easily_ be adopted to bootstrap a new platform or cpu (on FPGA). I consider such a tiny C compiler just a convenient assembler to implement basic things on new platforms.

I know GCC and Clang/LLVM but those tool chains are really big, and I guess it requires a lot of work to support a new platform and to polish all things so that everything works properly.

A _portable_ tiny GCC/LLVM replacement for small embedded systems would be really nice. GCC and Clang are great for powerful systems but I consider them overkill for small systems.

<comment><USER4> Do you know the "portable c compiler" PCC?

[http://pcc.ludd.ltu.se/](http://pcc.ludd.ltu.se/)

<comment><USER2> Thanks! I will study it.

</comment></comment></comment></comment></comment><comment><USER5> Nice project, but is it just me or is calling this a "C" compiler a bit of a stretch? Couldn't the title read _CC500: a tiny self-hosting subset-of-C compiler._

I realize it looks like C, but then so does anything with the words if, while, and braces. (It doesn't seem to have the word "for".) Since it says the compiler "does not even parse the types", that part seems like window-dressing and not part of the language.

I bet you could write the complete reference for the language this compiles in a paragraph.

</comment><comment><USER6> Does 767 LoC / 17KB really get to be considered tiny? I mean, sure, it's _small_ and efficient.. but there are other contenders to consider, such as Fabrice Bellard's [0] tiny obfuscated self-hosting c-compiler in under 500 lines.

How can we unambiguously distinguish between what qualifies for the classification of a "tiny" c-compiler versus just a "small" one?

[0] [http://www.bellard.org/otcc/otcc.c](http://www.bellard.org/otcc/otcc.c)

<comment><USER7> I propose the tinyness of a compiler is directly proportional to the number of comments spontaneously arguing it isn't.

<comment><USER8> That means it can be gamed

</comment></comment></comment><comment><USER9> I am playing with your code. But all my inputs end with error in the program function. Could you provide one meaningful example, which cc500 compiles completely?

</comment><comment><USER10> And now, to check how port it to TR3200 cpu...

</comment><comment><USER11> what does he mean by "self-hosting" ? I mean it generates a binary.

<comment><USER12> It means it can compile itself.

<comment><USER13> And the significance being, this compiler can handle code complex enough to create a compiler.

<comment><USER14> And it also means (thought this used to be more significant) that porting it to a new platform "just" means re-targeting the code generator and recompiling.</comment></comment></comment></comment>


Macro quantum mechanics - <AUTHOR> http://www.foxnews.com/scitech/2011/12/02/scientists-link-diamonds-in-strange-quantum-entanglement/

<comment><USER2> The actual article is published in Science, which I think means there's a moratorium (of a few months?) on releasing the full text. That's apparently why the article's not available on the ArXiv.

The article is here: <http://www.sciencemag.org/content/334/6060/1253.abstract>

Entangling Macroscopic Diamonds at Room Temperature

K. C. Lee, M. R. Sprague, B. J. Sussman, J. Nunn, N. K. Langford, X.-M. Jin, T. Champion, P. Michelberger, K. F. Reim, D. England, D. Jaksch, and I. A. Walmsley

ABSTRACT: Quantum entanglement in the motion of macroscopic solid bodies has implications both for quantum technologies and foundational studies of the boundary between the quantum and classical worlds. Entanglement is usually fragile in room-temperature solids, owing to strong interactions both internally and with the noisy environment. We generated motional entanglement between vibrational states of two spatially separated, millimeter-sized diamonds at room temperature. By measuring strong nonclassical correlations between Raman-scattered photons, we showed that the quantum state of the diamonds has positive concurrence with 98% probability. Our results show that entanglement can persist in the classical context of moving macroscopic solids in ambient conditions.

</comment><comment><USER3> Fwiw, this appears to be a reblog of: [http://www.livescience.com/17264-quantum-entanglement- macros...](http://www.livescience.com/17264-quantum-entanglement-macroscopic- diamonds.html)

which was previously submitted here: <http://news.ycombinator.com/item?id=3302904>

and also covered in the New Scientist: [http://www.newscientist.com/article/dn21235-entangled- diamon...](http://www.newscientist.com/article/dn21235-entangled-diamonds- blur-quantumclassical-divide.html)

</comment><comment><USER4> Question: Since we know that quantum entanglement does not provide for FTL communication, then what _are_ some practical uses for it? Why does it continue to get so much attention in research? This answer might be, "to learn more about quantum mechanics," but my impression is that entanglement is fairly well-understood in physics.

<comment><USER5> Imagine if this could be used for communication (probably not, but you never know). That means you could have communication between two mechanical entities at the opposite sides of the earth without worrying about pesky radio wave things like line of sight and increased signal strength requirements over a distance.

But a more appropriate answer is that there probably could be some sort of application, but unless we start building stuff we won't know what we're capable of and will be unable to engineer cool gizmos that use it.

<comment><USER6> I'm just skimming comments and haven't fully absorbed the context, but even without FTL, being able to communicate "directly" (e.g. "through" the Earth) would significantly reduce signal travel time, compared to running around the Earth on fiber or, longer yet, via bounced radio waves.

For one, financial market traders would have a field day with such.

</comment></comment></comment><comment><USER7> IANAP and the article is lacking on details where I most want them, but how does this show that the diamonds themselves are entangled, and not just the split light beam (à la the double-slit experiment)?

<comment><USER2> I believe the idea is that the photons are entangled with a certain vibrational mode in the diamonds. When you do an interference experiment with the photons, it demonstrates that the photons are in a coherent superposition. If the vibrational mode in the diamonds had decohered, the photons would have automatically been decohered as well (since they were entangled with the diamonds).

It's hard to tell for sure without being able to read the journal article, though.

Also, it's important to note that the center-of-mass position of the diamonds has _not_ been put in a superposition, which is what people usually mean when they say "macroscopic superposition". This experiment only shows that a certain vibrational mode in the diamond is superposed. (For all I know, that vibrational mode might just involve a few atoms.) Unfortunately, this type of link-bait is quite common in the quantum information community.

</comment></comment><comment><USER8> Crazier the Physics becomes day by day. What happened to the Masses being inversely proportional. How did QM get into diamonds!!</comment>


Indian IT firm accused of discrimination against "stupid Americans" - <AUTHOR> http://arstechnica.com/information-technology/2013/08/indian-it-firm-accused-of-discrimination-against-stupid-americans/

<comment><USER2> I thought it was weird that all the contractors I met with Infosys were of South Asian decent. Usually in IT, teams are a mix of different nationalities.</comment>


Mari0 – Super Mario Meets Portal - <AUTHOR> http://stabyourself.net/mari0/

<comment><USER2> It's blocked by Eset Nod32, I suppose it's a false positive ?

<comment><AUTHOR> I suppose so, because it's a pretty awesome (free) game :)

PS. Not related to it</comment></comment>


Introducing the Common App for Programming Jobs - <AUTHOR> https://symni.com/blog/a-new-way-to-get-a-job/

<comment><USER2> This is an awesome concept. Today's process is extremely biased and we are often passing on the "right" candidate because their resume doesn't hit all of the relevant keywords.

</comment><comment><USER3> Great solution for all the companies who complain about their inability to hire/evaluate quality technical talent!

</comment><comment><USER4> This approach makes so much sense, a real-time saver for the recruiters as well as those looking for a job!

</comment><comment><USER5> This is a really cool idea to help people save time when looking for jobs!

</comment><comment><USER6> I wish this existed the last time I was looking for a job!</comment>


Startup Horror Story - What to do? - <AUTHOR> https://gist.github.com/jackp/5d5a390c0bdd749c67b8

<comment><USER2> You didn't mention your location, but I'm going to guess "U.S.".

Find a few consulting gigs - then worry about dealing with the drama. Here's what I would do: First, for the consulting gigs: 1) post that you're available for a for weeks on facebook 2) email a few connectors & founders in the startup community that you're available for short gigs 3) email marketing agencies in your city the same thing

Try to buy yourself 1 months worth of paid gig time - then use that time to look for positions. Don't jump into a new job simply bc you need one.

</comment><comment><USER3> Sounds like their broke, so even if you did sue and win, you likely not get paid. As CIO, ask yourself if going off the grid for a week after launch was a good idea? I'm surprised it didn't come up during your discussions before you left. Regardless, I'd chalk it up to lessons learned and move on with your immediate need to secure future income. Good luck to you!

</comment><comment><USER4> Condolences.

But didn't you pick up on the brewing "smell" when Jim became CEO? I think you were very naive to believe either Bob or Jim with their reassurances. This is exactly what the Jims and Bobs of the world are like. Full of it!

In startup-land your story is just one of hundreds. Of course, you could get a lawyer but it is going to cost you lots of money and massive amounts of stress on your part. Even if the judgement (years later) goes your way, you are unlikely to even recover your legal expenses, yet alone be compensated for your losses.

It was an interesting learning experience! Just make sure you don't fall for a variant of this drama again. Good Luck!

</comment><comment><USER5> Get a lawyer.

<comment><USER6> Read this answer [https://news.ycombinator.com/item?id=6673208](https://news.ycombinator.com/item?id=6673208)</comment></comment>


If Edgar Allan Poe wrote JavaScript - <AUTHOR> http://blog.anguscroll.com/if-edgar-allen-poe-wrote-javascript

<comment> qu4z-2 I like this.

The only line that tripped me up was "There I sat engaged in guessing the meaning of each cursed expression". The half-rhyme is fine, but the second half inevitably comes out as "THE meanING of EACH cursed EXP ... What now?" I see how it's supposed to be read, but I just always end up reading it the "wrong" way. Maybe something like "There I sat engaged in guessing values for each cursed expression" would work? I don't know.

Anyway, it pretty much made my day. I've always enjoyed The Raven, and it manages to keep very close to the style. I especially like including Underscore.

EDIT: Hm... It occurs to me that the submitter might not be the author.</comment>


How YouTube Serves as the Content Engine of the Internet's Dark Side - <AUTHOR> https://www.buzzfeed.com/josephbernstein/youtube-has-become-the-content-engine-of-the-internets-dark <comment><USER2> They're really getting angry in the comments.

<comment><AUTHOR> It always strikes me as odd when people who constantly publish their stuff publicly on the Internet want to be contacted before someone publicly refers to what they do publicly on the Internet.</comment></comment>


JQuery File Upload - <AUTHOR> http://aquantum-demo.appspot.com/file-upload <comment><USER3> The number of jquery file upload plugins is astounding. This looks like a decent one, though makes the job of picking one that much harder...one of the drawbacks of jquery's open plugin ecosystem.

Who wants to write the blog post reviewing and testing them all in various browsers? I will donate $5!

<comment><USER4> Here is a community idea, not just for jQuery plugins but also CPAN, curse.com, and other code/add-on/plugin communities.

(1) Plugins must be submitted with a list of tags/categories indicating their purpose. For example, a plugin for uploading multiple files might take the tag "uploader". If it has file preview features, it might have the tag "file- preview".

(2) People rating the plugins may add tags. These will be marked specially as rater/community tags.

(3) For a particular tag/category, a rater may arrange plugins in order of preference (preference voting -- see <http://selectricity.org/> for an example of a working system).

(4) People looking for a plugin can see how the community rates it in each category compared to other plugins.

And if you believe what the community has to say, this rating system can give you a decent idea of which plugin may best do the job for you.

</comment></comment><comment><USER5> I've been pretty happy with <http://valums.com/ajax-upload/>

</comment><comment><USER6> This was on the front page 2 weeks ago: <http://news.ycombinator.com/item?id=2130501>

Unfortunately that news item linked to a blog post and not to the project itself.

<comment><USER7> mmm... we could use some feature to mark posts as duplicated here on HN, right?

</comment></comment><comment><USER8> We've been working with uploadify, but I like the fact that this comes readily built-in with image preview/delete options and requires no flash. Has anyone tested this on an iPhone?

<comment><USER9> AFAIK, Mobile Safari does not support file uploads, so this (or any similar plugin) will not work.

<comment><USER10> Hell, the iPhone doesn't even have the concept of 'files'.

</comment></comment></comment><comment><USER11> Did some hunting for a good jQuery upload plugin a week ago for (self promotion warning) imgscalr.com.

valums ajax-upload seemed like it would be the best fit, but I couldn't get a full-screen Drag and Drop target working and had a hard (impossible) time customizing it to behave differently.

Wanted to use uploadify, but as mentioned it didn't work inside a browser with Flash-blocking enabled which turned me off immediately.

I actually wrote an HTML5 DnD + File API tutorial a few months ago: [http://www.thebuzzmedia.com/html5-drag-and-drop-and-file- api...](http://www.thebuzzmedia.com/html5-drag-and-drop-and-file-api- tutorial/)

and decided to get down and dirty with the whole process since everything else out there seemed relatively new to the scene anyway. Some things I found out:

1\. Opera and Mozilla (thank the gods) seem to behave almost identical to each other... at least with the HTML5 DnD behavior/events and File API functionality.

2\. Safari 5 (on Win 7) doesn't seem to understand Drag and Dropped _native_ elements at all. I also scrubbed through the Apple HTML5 site and found no mention of support for the File API anywhere... so even if DnD was made to work, it wouldn't know what to do with the resources.

TIP: There is a great tutorial from 2009 online, and seems to be for Safari only, where you can hack the styling of an <input type="file"> dialog and MAKE it a drop target while hiding the default file box here: [http://www.borismus.com/wp- content/uploads/2009/02/drag_drop...](http://www.borismus.com/wp- content/uploads/2009/02/drag_drop_upload.html)

3\. Opera, as much credit as it gets for being so great, doesn't seem to want to have anything to do with some of these native HTML5 APIs -- I believe I read somewhere (this has all become a blur) that they are sitting out from these 2 APIs until the dust finishes settling... probably the smart move, but still annoying my app won't work in their browser.

4\. IE 8/9... what can I say. I don't think I expected any of this to work, and probably plan on a Flash-fallback for IE only as some longer term solution, but damn, I sort of hoped it would have.

5\. There is a massive post on all the things wrong with the HTML5 DnD API at quirksblog that made me want to give up HTML5 programming and go back to embedded Applets: [http://www.quirksmode.org/blog/archives/2009/09/the_html5_dr...](http://www.quirksmode.org/blog/archives/2009/09/the_html5_drag.html)

My point being (12 paragraphs later) that from a week of working on this problem there doesn't seem to be a sure-fire solution out there yet for someone wanting to go the "pure HTML5" route... there are some awesome starts to it as linked below, and all the JavaScript from imgscalr.com is heavily commented if you want to take a look at it, but it seems we are still a far way away from some drop-in/jack-of-all-DnD-upload experience that we were hoping for.

Made me realize HTML5, at least in the aspects I'm most interested in, is still a long way away from giving me exactly what I need.

</comment><comment><USER12> One feature Uploadify still retains over this plugin: The Flash upload dialog allows you to restrict what file types (by extension) are selectable, which can prevent some people from uploading the .JPG when you only accept audio files, etc. I don't think anything in the HTML5 DnD API has accounted for this.

</comment><comment><USER13> This is the first one I've ever seen that supports D&D from Mac/Safari. Very nice.

</comment><comment><USER14> Similarly, <http://www.uploadify.com/>

I really appreciate that this doesn't require Flash as Uploadify does. Very cool, I'm probably going to see if we can get this in our project - <USER2> upload is an important story coming up next week!

<comment><USER11> It may not require Flash but with NoFlash in Chrome I can't get any of the demos to load - not sure why.

<comment><USER14> Uploadify requires flash, the submission's script does not. Sorry if that was confusing.

Uploadify uses Flash's multifile upload whereas the submission uses the new File API (I believe, I'm guessing, I haven't looked at the source)</comment></comment></comment>