A rant about PHP compilers in general and HipHop in particular. - <AUTHOR> http://blog.paulbiggar.com/archive/a-rant-about-php-compilers-in-general-and-hiphop-in-particular/ <comment><USER2> Aside from the interesting technical details, the interesting part of this article is how much difference a brand name makes. He had difficulty getting people interested in PHP compilers, then Facebook announces one, and it's the new hotness.

This is a problem many startups have. Because people use brands as a shortcut to determine whether something is of sufficient quality to be worth taking the time to investigate, being a new unknown is a bit of a hurdle to overcome.

<comment><USER3> it's not really so much a brand name as just getting the concept out there. php developers don't realize that there are option when it comes to the runtime.

i would guess that the majority of them wouldn't even know to look for phpc or any alternative. there just hasn't been any news of something like HPHP before HPHP. if you want to run php, you go to php.net or install it from a package and go from there.

combine the above with very little actual need, very little public information on how to build a large site, and lack of maintainer interest and phpc just won't get major traction.

I would have loved to have found phpc several years ago when i was still working on a large site that had performance problems.

<comment><USER4> It wouldn't really surprise me if a lot of the people freelancing in PHP wouldn't know enough about the interpreter so see any advantage in compiled code.

<comment><USER3> a freelancer wouldn't know and if they did they wouldn't care. (i am a freelancer.)

we're talking about websites that are in the top 5% of all sites (by traffic) that would even need to think about using something like phpc or hphp. the number of those that would be willing to use a freelancer to handle their scaling issues are probably in the single digits at best.

<comment><AUTHOR> On a tangent, I wonder why lots of people call it something other than "phc". I said phc a lot, its on the website as phc, but I've heard phpc, phpcompiler, and tons of variations. I'm curious why.

</comment></comment></comment></comment></comment><comment><USER5> I love this kind of meaty thinkpiece from someone who clearly knows what he's talking about. So rare.

I wonder if LLVM isn't getting mature enough. Yes, the Unladen Swallow folks are having trouble with it, but it sounds like the most important troubles are related to dealing with Python runtime realities, so those wouldn't necessarily apply to another runtime system.

<comment><AUTHOR> Its amazing how close all of the scripting language run-time are, in terms of design and implementation. I think using LLVM with PHP would be worse, since the PHP interpreter is so badly written.

I expect that LLVM will be mature real soon now.

</comment></comment><comment><USER6> There were a few really interesting technical observations in this that made it worthwhile from my perspective.

</comment><comment><USER7> _PHP doesn't really need a JIT. Server side programs in PHP don't do a great deal of dynamic stuff, and it would be incredibly rare to load some random code at run-time, so a JIT wouldn't be all that useful._

Uh, what? JIT compilers even improve the performance of _C_. Loading code at runtime and having a JIT are totally orthogonal.

If you're a language like PHP where you can't guess if a value is going to be a hash or the number 42 until runtime, JIT techniques are essential to get any reasonable performance. (Ah, here comes the argument, "PHP apps are never CPU- bound.")

It is odd that the article claims that JITs for dynamic languages are never successful; there is V8 and TraceMonkey, and Parrot optimizes Perl 6 pretty well. Java is fast because of its JIT. The author claims that LLVM's JIT compilation doesn't work, but that's untrue; LLVM's JIT works just fine for C. LLVM's C runs faster than GCC's statically-compiled C, anyway.

<comment><USER8> LLVM isn't used as a JIT for C in any existing compilers (clang, llvm-gcc, etc) as far as I know; It's used to emit static object code. (If I'm wrong, of course, I'd appreciate a pointer to the projects)

<comment><USER7> Dunno what the defaults are, but I'm fairly sure I've compiled C applications with JIT enabled.

<comment><USER9> I'm not sure what that would even mean. JIT is an optimization for virtual machines. It compiles VM code (such as Java bytecode) or interpreted code to machine code at runtime. C already compiles to machine code.

In order to do anything resembling JIT, you would have to insert a runtime where one does not already exist.

<comment><USER7> LLVM's compiler suite compiles C to LLVM bitcode, and the LLVM (low-level virtual machine) runs that code.

<comment><USER10> I don't think LLVM provides any such virtual machine, despite its name. It has a code generator, but no VM as such.

<comment><USER11> I'm almost positive that LLVM provides both. You always target LLVM bytecode with your frontend, but then you can choose either to:

\- run the bytecode using the LLVM with or without JIT.

\- compile the LLVM bytecode into x86 bytecode.

This is what clang does (afaik) and is a reasonably nice use case because you write a static language that targets LLVM bytecode and end with a _fast_ program because LLVM will output optimized x86 binaries.

However, I'm pretty sure that the efforts like unladen swallow attempt to use the LLVM's virtual machine in the hope that it will become very fast as they optimize it (especially the JIT part).

<comment><USER12> Yeah, you just instantiate an ExecutionEngine and feed it your LLVM bytecode, and it'll give you back C function pointers that you can call. The ExecutionEngine is either a JIT (which'll compile your bytecode into native code and give you that back) or it's an interpreter (where the function pointer you get back is an entry point into the interpreter).

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> I believe pretty much anything Paul Biggar has to say about dynamic language interpreters / compilers. His talk at StackOverflow DevDays London about dynamic languages was a great eye-opener for me and a fantastic experience.

<comment><USER14> I was at the same talk, but I got the impression that because his perspective comes from trying to do static analysis and optimization, he doesn't fully appreciate some of the capabilities of dynamic languages. The overall effect was a static typer missing the point.

But as he says in this post, PHP code doesn't necessarily take advantage of these capabilities in practice.

<comment><AUTHOR> Yeah, PHP isn't Python or Ruby, which are properly dynamic. I think those languages are too far gone in their particular dynamic direction to be saved. But I'm convinced you can build a language that feels dynamic, but whose meta- programming is largely compile-time. Even the funky Python I see (metaclasses and decorator) could largely be compiled if the language rules were slightly tweaked.

I'm interested: which capabilities don't I appreciate? (BTW, my background is very much being a static-typer, missing the point, and slowly realizing it).

<comment><USER14> First off, I'm pretty much a static typer myself, so I'm not necessarily the best advocate for the dynamic case. But by the same token, I can argue against some of the problems of static typing.

First up: I'm talking about object-oriented approaches, with polymorphism and dynamic dispatch being the core abstraction tool.

If you try to introduce more static typing into a system with polymorphism, you're trying to push type decorations so that they can follow where the expressions go a little further, so that you can reason more. Parametric polymorphism, or generic types and methods (hence forth called generics to avoid confusion with OO polymorphism), are a key tool. They let you better reason about values of which types cross method boundaries. With generic methods, you can usually infer the type parameters. With generic types, the type arguments can serve as extra documentation. A Java ArrayList<E> seems clearly superior to ArrayList at first glance.

But all sorts of problems start coming up when you mix polymorphism with generics. Type inference isn't so simple anymore, as ad-hoc choices need to be made. Instantiations of generic types have no necessary subtyping relationship. So you have to reintroduce dynamicism in the back door via base classes or interfaces etc. that use a top type (java.lang.Object or whatever) to re-unify types; or you dabble in variance. And you want to do more with your arguments of type parameter types, so you bring in type parameter constraints, or type classes, or other means of generically handling values.

So perhaps you have use-site variance or declaration-site variance combined with type classes combined with polymorphism. Suddenly using or declaring generic types has gotten quite a bit harder. Getting type information to flow into the right places starts requiring confusing tricks, particularly when you want e.g. the methods of your ancestor to return a type related to the concrete instantiation type, for example something like this in C# syntax:

        // so you want a GetThis method that returns a correctly
        // typed thing for the current instance
        class Base<T> where T: Base<T> { public T GetThis() { ??? } }
        class Desc: Base<Desc> {}

When the static typing advocate has tangled himself up in a knot like this (or deeper, e.g. when you're passing other type arguments around across multiple classes and hierarchies, just to get the damn types right), the dynamic typer shakes his head, and says "don't you see - it's all just objects!".

One of the paradoxes is that declaratively specifying, at compile-time, all the types that may reach different parts of your program at runtime, is harder than writing the program to build the object graph at runtime in the first place, and treating the occasional runtime type error as a bug that will be eliminated.

Now to the other side of the story, meta-programming. I agree that you can do lots of meta-programming at compile-time, but there are two problems, as I see it. One: compile-time is too early to be making decisions. What do you do when you can't afford to restart the process, but you still need to change the running code and how it works? This isn't a theoretical concern; I've architected such a server system in the past. It could keep on handling requests relating to client sessions associated with a previous versions of the server-side behaviour, but new client sessions would use the latest version of the server-side behaviour. Rolling over the server farm from one version to the next didn't require tricks with load balancers and restarting app servers, etc.; it was all built into the system.

(Actually, the architecture was really interesting, and I should write it up in more detail one day. Another key fallout of this approach was that the session data, which was (usually) stored on disk and was a bit like a Lisp or Smalltalk image only typically 20K in size, had a pointer (a URL) inside it telling the server where to get the behaviour for this session. This meant that problem sessions could be post-mortemed quite effectively: the old session could be resurrected, stepped through, etc.)

And the second problem of compile-time metaprogramming: the formalisms you use to munge with the types, particularly static types which the compiler will later infer further things from, may in practice be quite different to the imperative constructs programmers are used to using to implement behaviour. When they want to add a method to a class, they want to do it using the same tools (or close) to that which they'd use for adding an item to a list or hash table.

Perhaps the meta-programming is implemented as a plugin to the compiler, or a macro system, where the code is modifying the compiler's own definitions of the types etc. But now we've introduced an abstraction level into the chain of source code -> executable text that most programmers are not familiar with, and may be intellectually ill equipped to deal with - not because they're stupid or incompetent, but rather because they simply don't need to know. Consider e.g. hygienic macros and the problems of which symbol refers to which level of abstraction, the compilation level, or the runtime level; the distinction between the level you're at when you're quoting and unquoting, in Scheme terms. Consider the confusion of this poor fellow whom I helped out:

[http://stackoverflow.com/questions/326321/how-do-i-create- th...](http://stackoverflow.com/questions/326321/how-do-i-create-this- expression-tree-in-c)

When you consider the tangled mess of static typing you can get into when you mix polymorphism with generics, and then add in the ability to munge with code and types such that the compiler sees the modified code / types, and hopefully you can see it's easy to add more complexity than the benefit you get out.

</comment></comment></comment></comment><comment><USER15> > I've heard the argument "you don't need a compiler, since PHP is rarely the > bottleneck" for many years. I think its complete bollox. ...

> Unless your PHP server is sitting there idling (which is probably the case > for many PHP servers out there) ...

Don't these two statements (abbreviated in my quote above, but directly one after the other as quoted) directly contradict each other? My personal experience is also that the CPU is never the bottleneck in (well written) PHP apps. You're usually always waiting for the DB or the network/io latency of something else like memcache.

In fact, I know at my last job (handling ~100s of millions of dynamic PHP requests/day) that PHP's gzip compression was turned on specifically because we had gobs of spare CPU, so using it to save bandwidth was a win both for speed and for the bill.

<comment><USER14> Here we go again.

If you have lots of spare CPU on machines in a web server farm, you have too many machines. Even though I/O is the bottleneck for any single request's response time, CPU is generally the bottleneck on total load.

<comment><USER3> that's a double edged sword though. if you don't have enough spare cpu, you won't be able to handle any sort of spike in usage. say from an advertising campaign that works a little too well or if a server goes down.

and once you start maxing out boxes, the load comes even harder since the site loads slower. you very quickly run out of resources and suddenly you're SOL and everybody's mad that the site won't load.

honestly, it's hard to point at any one source of problems when scaling a site. cpu, ram, and i/o are all equally problematic. and if you under plan any one of those it can take your site down into a spiral of doom.

you really must allocate your resources for peak load, not for average or nominal load. and then allow for some failures.

<comment><USER14> Sure. But under max load, that gzip will be adding to your problems.

And of course, graceful failure is an art in itself. You have to start turning down requests before you actually hit peak capacity, or you risk getting stuck in a thrashing pit.

</comment></comment></comment></comment><comment><USER16> The guy seems petty. The name of the tool is hphp, if you drop in an i and an o, you have hiphop, which is a perfectly fine name considering Django and all the other music named tools. It is my belief that the dislike of the name comes from a dislike of the type of music.

And rewriting a negative comment on your blog? That's just not done. Delete it, but don't rewrite it ever.

<comment><AUTHOR> > And rewriting a negative comment on your blog? That's just not done. Delete > it, but don't rewrite it ever.

Why not? Its my blog. Its not like I hid it. Ultimately, to cultivate a good community, you have to have strict rules. Now I'm not exactly curating a massive community here, but it seems like an effective way of discouraging the kind of dickheadedness that was his original comment.

<comment><USER16> It's your blog, but it's not your comment.

<comment><AUTHOR> That's an interesting perspective. Does the poster "own" the comment on my blog? Do I? Since he has no control over it, is it mine? Is it still his if he posted anonymously? Does the fact that he put my email instead of his in the email box hand control over to me?

All interesting philosophical questions. However, since I have editorial control, and determined that he was acting anti-socially, it seems perfectly fine to me to scribble over it (though I could see a problem if I wasn't transparent about it).

<comment><USER17> People own their own creations. The power owns (the copyright on) his comment even if he posts it to your blog, unless you've made an agreement otherwise (e.g., in your blog's terms of service).

You control what your blog displays, though. There is probably a technical legal difference between deleting a comment (definitely right and legally permissible) and modifying a comment (thus creating a derivative work of a copyright work which you may not be licensed to derive from).

</comment></comment></comment></comment></comment><comment><USER18> I think this person is not really informed about JIT.

Knowing the code at the beginning of the run doesn't make you know its runtime profile, at all. But it's something the LLVM generation is not aware of.

I think that the people who saw only the suits in front of java and C# didn't see what happent on the JVM/CLR side. And a lot of things where shaken there.

<comment><AUTHOR> I'll admit I haven't ever written a JIT, but I wouldn't say I'm not informed about them. Its a question of tradeoffs: JITs can help you get good run-time performance if your types are static for some large portion of run-time, but not really known at compile-time. I would allege that they are available at compile-time. I think my research showed I could prove a type for about 60% of variables/fields/etc, and tie it down to {int,float} or {object,null} in most other cases. I guess I forgot to put that in.

<comment><USER18> You are speaking about monomorphic specialization, in the JVM benchmarks they are at 80% of specialized code (correctly guessed concrete type), moreover, the JVM keeps the code for various specializations if it needs too. And the good part is that they don't do all this stuff if it's useless.

The entry into JIT is stiff, but you already spent 4years on your static compiler, so basically you had this time to make/reuse/spacialize a JIT interpreter that would just be a drop in replacement.

<comment><AUTHOR> I would understand the term "monomorphic specialization" to mean "can be many dynamic types, but in practice only has one dynamic type". What I'm implying is that the vast majority of values only have one static type, if your static analysis is good enough.

The question of "which is better" wasnt really a factor in my decision to build a static compiler instead of a JIT. Building a JIT as a solo effort for a complex language like PHP would probably have prevented me doing enough research to get a PhD. More importantly, I wasn't interested in JIT, I was interested in static analysis and (static) optimization.

</comment></comment></comment></comment><comment><USER19> > The PHP interpreter is also quite memory hungry, as interpreters go. Any PHP > value in your program uses 68 bytes of overhead [6]. An array of a million > values takes over 68 MB.

Well, that is obscene.

<comment><USER20> Python (and probably every other dynamically typed language that does use the tagged pointers trick) has the same issue, even worse on 64-bit. Allocating [sys.maxint-1 for x in xrange (1000 * 1000)] takes up 36 megabytes of memory.

Allocating dict((x, float(sys.maxint-1)) for x in xrange (1000 * 1000)), which is closer to PHP's combined arrays-dictionaries and values which I believe are always floats (unless I'm thinking of Perl or Javascript?), takes up 97 megabytes.

Of course if you really need 1 million floating point values, that are not going to be None and that are contigously indexed 0 to 999999, you could do: array.array('d',(float(sys.maxint-1) for x in xrange(1000 * 1000))) which has no overhead.

But for a random collection of objects with attributes and lists and dictionary, the overhead exists. I've got 15 gigabytes of Python objects on this box across a couple dozens of processes and sometimes I dream of a time where every bit was accounted for, where for every bit I could trace back to where it was set and why and accounted for it.

As for the "tagged pointers" trick -- basically taking advantage of that pointers to objects are going to always have the last bottom bits set to 0 and saying that if they are set to 01 for example, that the top 32 or 62 bits are an immediate integer -- I think someone tried it out in Python a few years ago and the result were that it was no win overall, though it might have decreased memory usage occasionally.

<comment><USER19> I was thinking more about the overhead of building arrays of small strings. Depending on how you measure memory usage, Python is 3-4x more efficient at building lists of one-byte strings than PHP is wrt memory usage.

</comment></comment></comment><comment><USER21> Is there anything particularly novel about HPHP that would be patentable?

<comment><AUTHOR> I don't know. I don't think there is enough information released about it.</comment></comment>


Zynga Says Offers Were Just 10% Of Revenues - <AUTHOR> http://www.businessinsider.com/exclusive-zynga-says-offers-are-just-10-of-revenues-2009-11

<comment><USER2> Zynga also said that they stopped doing offers, when in reality they just turned them off for the SV area. When you have a record of lying, why should people believe you

<comment><USER3> Is there any proof that they just used geolocation to disable ads in a certain area? I'd like to give you the benefit of the doubt, but I am interested in seeing the proof.</comment></comment>


Graphics Analysis of an NES Game's Select Menu - <AUTHOR> http://www.dustmop.io/blog/?p=123

<comment><USER2> Love these types of articles.

Also, if you develop for, say, the Apple Watch, there's a goldmine in looking at classic console games and how they made use of constrained resolution, memory, colors.

<comment><USER3> I feel like mobile developers and the creators of the hardware have given up on heavy optimization to work on minimal hardware and instead just gone with more expensive, top of the line hardware so they can continue current programming practices.

<comment><USER4> I work on console targeted projects, we still have to optimise for very tight hardware constraints!

<comment><USER3> The reality is that the phones of today are as powerful or more so than the laptops of yesterday.

<comment><USER5> Yet my phone battery used to last a week and now it doesn't even last a day. Mobile CPUs may be able to handle all that bloated code fast enough, but not long enough.

<comment><USER4> When your phone battery lasted a week, what did you do on your phone, compared to what you do on it now? If I leave my laptop on standby the battery lasts at least 3-4 days, but if I use it continuously it lasts under 7 hours. Similarly with my smartphone, if I leave it on standby I get 2-3 days, but with "normal" usage - (moderate browsing, emails, texts, phonecalls) I have to charge it either daily or every other day.

<comment><USER6> I'm not the OP, but when _my_ feature phone battery lasted a week, I made some phone calls every day or two, and used the phone as a daily alarm clock.

If Google Play Services decides to not burn all of my battery in busy waiting or whatever, and if I make _zero_ phone calls and have a daily alarm, my smartphone battery will last between two and three days. When my phone was new and I was commuting with it, I would typically get between eight and ten hours on a charge.

I would _happily_ double the thickness of any modern non-ruggedized smartphone in order to double its battery life.

<comment><USER4> > When my phone was new and I was commuting with it, I would typically get > between eight and ten hours on a charge.

Sounds like you got a dud phone.

but being serious, It's one of the trade offs of modern smart phones. THey're designed to last roughly a day of usage. If you use them like you used your old phone, you'd get 2-3 days out of them, but then you might aswell just run with an Alcatel[0] and get the super long battery life.

[0]: [https://www.o2.co.uk/shop/phones/alcatel/1040/#contractType=...](https://www.o2.co.uk/shop/phones/alcatel/1040/#contractType=payasyougo)

<comment><USER6> I'm confused. If _this_ was what you were going to reply with, why on Earth did you bother asking the question?

Anyone who's even remotely techy knows _why_ their smartphone battery lasts far less than a day on average. That doesn't mean that they're happy with the (often fashion driven) tradeoffs made by phone designers, or that they don't actually want a vaguely-smart-phone that also has battery life of a week or more.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> I would imagine the limitations would make the UI design itself very effective aswell. Every extra sprite you include bears a serious performance cost so you would pare things down to whats absolutely necessary

</comment><comment><USER8> It's interesting how much this page relies on the browser behavior of rendering all animated GIFs on the page in sync with each other, such that the palettes rotate in sync with the images.

</comment><comment><USER9> Got me to search for NES overclocking and surprisingly people did swap clocks to pump the cpu faster:

[https://www.google.com/?gws_rd=ssl#q=nes+cpu+overclock](https://www.google.com/?gws_rd=ssl#q=nes+cpu+overclock)

I already fried my old SNES so I won't touch my NES but still worth reading.

</comment><comment><USER10> Got me thinking about the Goonies 2 pause map.</comment>


Mac OS X Isn't Safe Anymore: The Crapware / Malware Epidemic Has Begun - <AUTHOR> http://www.howtogeek.com/210589/mac-os-x-isn%E2%80%99t-safe-anymore-the-crapware-malware-epidemic-has-begun/ <comment><USER3> As an aside: Everyone who read this article, please keep in mind that the process injection model used by these pieces of adware is exactly what your typical "keylogger" uses also.

No malware literally logs keys typed anymore. I cannot stress that point enough. Instead they log form submissions (e.g. POST requests) which give the malware author much more useful information they can data mine in an automated way (e.g. URL, named parameters, etc). This works even on a "secure" page (e.g. HTTPS with extended certificate).

I'm super tired of supposed power users or "geeks" telling others to copy/paste in their username/passwords to improve security. That's not how this works, it isn't how any of this works. Nobody reads raw key-streams, they're completely useless because they fail to contain CONTEXT (i.e. where you typed what).

Sorry, just a pet peeve of mine. The term "keylogger" is largely a misnomer. A more accurate name would be "credential hijacking" or "form submission theft." A lot of malware actually use standard injected JavaScript to add event hooks to a page, to fire the data back to a evil browser extensions.

<comment><USER4> Don't a lot of login forms hash passwords with JS before sending it over the internet? Wouldn't it then be useless to anyone listening?

<comment><USER3> I cannot think of a popular site which does this.

If that provides security really depends on what the "bad guys" are hooking. If they're placing event triggers straight onto text box/button/form elements themselves (either through JavaScript or grabbing something akin to Win32 messages) then that wouldn't do anything at all.

Even if they did grab the raw POST request (which is somewhat common) a hash would only provide security if it was merged with an anti-forgery token sent from the server, otherwise the "bad guy" could just re-post the exact same hash and login anyway.

I think it really boils down to how popular your site is. If for example Facebook did that, because it is popular enough with the "bad guys" they're going to spend the time circumventing any JavaScript-based security you could implement.

<comment><USER5> > I cannot think of a popular site which does this.

LiveJournal used to do it. No idea if they still do. I don't think it ever caught on.

</comment></comment></comment></comment><comment><USER6> > _Mac OS X Isn't Safe Anymore: The Crapware / Malware Epidemic Has Begun_

Yeah, not really. Like it hadn't began all the other times in those last 14 years that such articles appeared.

I've used Windows for decades (still do ocassionally), and had lost count of malware, adware and viruses I had to battle. So, don't tell me about "malware epidemic" on OS X with a straight face...

<comment><USER7> >adware and viruses I had to battle

This is largely a user problem now. I haven't caught a single problematic download on my Windows 7 box.

<comment><USER8> See recent Lenovo issue. It isn't a user problem but an ecosystem problem starting with the OEMs and the general discovery and distribution of software.

</comment></comment></comment><comment><USER9> Part of the problem is that a lot of legitimate freeware / open source software is not signed. I assume because people don't want to pay the $100 a year just to support OS X. So people get used to installing unsigned software, and then end up installing malware.

<comment><USER10> CNet's downloader app looks like it's signed. It warns that it was downloaded from a website (which it was), but you don't have to do the right-click-open song and dance.

<comment><USER11> I'd love to see Apple take a stand and revoke their certificate. Usually I strongly support developer freedom / openness, but these apps are straight-up malware. Having a decline option somewhere doesn't matter if essentially all users who accept do so unintentionally.

</comment></comment></comment><comment><USER12> I've seen a lot of Mac users with adware in the last few months. I've found Adware Medic[1] to remove nearly all of it pretty easily.

[1]:[http://www.adwaremedic.com/](http://www.adwaremedic.com/)

<comment><USER13> Seconded, AdwareMedic is quick and painless.

Power users might also appreciate Little Snitch[1] to see what their Mac is connecting to.

[1]: [http://www.obdev.at/products/littlesnitch/index.html](http://www.obdev.at/products/littlesnitch/index.html)

<comment><USER12> I forgot to mention, after running AdwareMedic, make sure to check browsers' search engine and homepage settings.

</comment></comment></comment><comment><USER14> How about a non-Apple App Store: something like homebrew with a friendly GUI that's easy to navigate? I started using Homebrew Cask recently, and it seems like a perfect workflow for the average user who just wants to download VLC or whatever.

I'm imagining Grandma pulling up the "Application Warehouse", let's say, and clicking a download button under a VLC icon. It gets downloaded from a trusted source over HTTPS, gets checked against a hash, symlinked and Gran's ready to go, all without the hassle of shady installers from the search engine shitpile.

<comment><USER15> Microsoft really should consider making something like Ninite ([https://ninite.com](https://ninite.com)) a native component of Windows 10.

It skips all the garbage and installs the application.

<comment><USER3> They are/have. In Windows 10 it is called "OneGet." It is a Linux-like package manager to complement their Windows Store (app store) which isn't going away.

</comment></comment></comment><comment><USER16> I cleaned some crapware off an acquintances computer. She is around 70 and didn't know why the computer was not behaving correctly. It was really easy compared to windows crapware. When my dad's Windows computer had malware I had to reformat . But with OSX I deleted a plist or two and it was done.

<comment><USER17> I have to disagree with this. At my current job I deal with a lot of Mac malware/adware, and fully removing it is complicated process. After clearing the applications folder and removing browser extensions, you have to check a lot of folders, and you kind of have to know what you're looking for. In /Library/ for example, you have to check Application Support, Extensions, Frameworks, LaunchAgents, LaunchDaemons, PreferencePanes, and StartupItems. I like OSX, but it definitely needs a MalwareBytes equivalent.

<comment><USER18> [http://www.thesafemac.com/arg/](http://www.thesafemac.com/arg/)

<comment><USER17> Yeah, that site has been really useful. That's where I discovered the large number of folders in which malware can hide.

</comment></comment></comment></comment><comment><USER19> "Examining further comes up with something curious… the person who wrote this malware wanted to give special thanks to his mom."

That's the old default Credits.rtf.

</comment><comment><USER20> Now that Mac OS X has hit about 7% of internet users, it's profitable enough for adware/malware folks to target.

Most of the infections on Windows aren't due to some huge security issue on Windows that Macs are magically immune to. They are due to the users themselves installing adware or malware-infected software from sites online. Now that there are more Macs out there, the reward is greater. So, there is more revenue to be made form adware-laden software and a better return for the time investment/risk of creating malware for Macs (to send out spam, be used in DDoS attacks, sniff for and steal financial info and passwords, etc).

<comment><USER21> This is a common (and tired) response but it's really not entirely true - Windows does in fact have a lot of potentially catastrophic holes that are innately tied to higher privileges for users.

Most of what Windows has implemented since 7 with UAC, MSSE and now integrated with Defender is a layer on top that introduces some failsafes. I won't argue that it's been a massive and much-needed improvement to Windows, but Java and Flash still provide viable vectors to bypass it and infect a Windows machine.

Designing actual viruses - stuff that has the ability to read and modify the filesystem - is still harder to pull off undetected on OSX. This article intimates as much. Most of what's included here is either bundled applications you don't want - but you still have to actively find and then agree to - <USER2> browser modifications. Neither of those is within 500 sqmi of, say, CryptoLocker.

<comment><USER20> I never claimed that Windows or Mac OS X are more or less secure than the other. I very specifically said the following:

"Most of the infections on Windows aren't due to some huge security issue on Windows that Macs are magically immune to. They are due to the users themselves installing adware or malware-infected software from sites online."

This is 100% accurate and what most home users have to deal with in terms of issues on Windows. The vast majority of Windows issues that end users experience and get frustrated over have nothing to do with Java or Flash flaws or needing to compromise a system. The users themselves give the apps permission to install and do their thing.

It's also worth noting that Java and Flash don't provide much of an attack vector for the majority of Windows users you and I know anymore either. Firefox won't permit outdated versions of the Java or Flash plugins with security issues to run and will direct you to update. Chrome has its own version of Flash built in and automatically updated with the browser and disables Java by default. Even Internet Explorer blocks outdated ActiveX plugins like old and insecure versions of Flash and Java these days.

<comment><USER21> I still don't think that's true - visiting a malicious site without any action still provides far more of a risk on Windows than it does on OSX.

Are there improvements on the browser and OS side that are helping? Sure. Do those impact the vast majority of Windows users? Probably not. Look at browser & OS version usage and you'll see that the "users you and I know" are probably not indicative of the majority of users in general. At least not yet.

<comment><USER20> All major browsers on Windows block outdated Flash and Java by default. All major browsers on Windows are automatically updated to the latest version by default. So, for the vast majority of Windows users, the attack vectors you're mentioning simply don't apply anymore. That means users you and I know and most users we don't.

What I'd meant by that line was that this doesn't apply to users in other countries where the majority of users are still using hacked (and completely insecure) versions of Windows XP. Sadly, Windows XP still represents about 19% of online users. Thankfully, most of those users are using a 3rd party browser as IE 6 is down around 1%.

</comment></comment></comment></comment></comment><comment><USER22> > It wasn't that long ago that you could install almost anything for OS X from > almost any website, and you didn't really have to worry about what you > clicked on.

Full stop. That's a ridiculous statement to make. Are we really pining for a return to such an oblivious mentality? Good riddance.

<comment><USER9> It's a true statement though. OS X users avoided malware by there not being malware, not by being smarter or being protected.

<comment><USER22> It's along the lines of "Not long ago I could back out of my driveway without even looking, but now my new neighbor's kids play in the street."

I will stipulate it's a true statement. Still foolish.

</comment></comment></comment><comment><USER23> Breaking news: Free download sites like Download.com are shit. More at 11.

</comment><comment><USER24> "We'd love to see Apple fix some of the App Store issues and make everybody use it."

I agree with the first part of this but not the second.

</comment><comment><USER25> That's a really damning Yahoo screenshot and it matches my experience pretty well. Yahoo is not an acceptable search platform and I'm really confused about why Mozilla thinks otherwise.

<comment><USER26> Money

</comment></comment><comment><USER27> I click on all MacKeeper ads I see, repeatedly

Let the fuckers pay for that

<comment><USER28> You see ads?

<comment><USER27> Sometimes when I run without Ad blockers...

</comment></comment></comment><comment><USER29> I've been a Mac developer since 1984 and the last time I ever saw a virus was 1988 I think. It's not impossible to get irritation-ware if you download random crap from these download sites but genuine malware is extremely difficult to produce. Saying "Mac OS X Isn't Safe Anymore: The Crapware / Malware Epidemic Has Begun" is beyond stupid.

<comment><USER30> By "difficult to produce", do you mean difficult to find? If you mean difficult to develop, it's actually very straight forward, and no different from Windows.

</comment></comment><comment><USER31> As a Mac user who migrated from Windows, I had no doubt that it was only a matter of time before Macs became more lucrative targets. Anyone who thinks that their OS of choice is unassailable is fooling themselves.

</comment><comment><USER32> "If you do stick to the App Store, you have nothing to worry about. We'd love to see Apple fix some of the App Store issues and make everybody use it."

Yeah, that'd be just awesome.

<comment><USER3> Devil's advocate here:

That could just be the default, and the user could disable it. On Windows 8/8.1 the default is for the "SmartScreen Filter" to block "unrecognised" applications from being run or installed. See their FAQ [0]. It can be disabled however.

If someone is smart enough to be installing applications from third party sources themselves, then they're smart enough to flip a switch in a Preferences panel to enable it.

However this does protect the lowest common denominator who these malware are actually targeting (i.e. computer illiterate individuals who will click ads in search results).

[0] [http://windows.microsoft.com/en-us/windows7/smartscreen- filt...](http://windows.microsoft.com/en-us/windows7/smartscreen-filter- frequently-asked-questions-ie9)

<comment><USER32> If it can be disabled, I'm less bugged by it. It's still a lot of friction for the acceptance of open source apps and other "not really appstore-compatible" projects, though.

</comment></comment></comment><comment><USER33> Apple was never good at security, they just weren't a big target. Now they're both bad at security AND a big target.

</comment><comment><USER34> OSX will become walled garden just as IOS. I am personally happy for that, many developers will return to Linux.

<comment><USER35> Apple knows that their computers are a favorite amongst developers. No way would they do that.

Apple already made their move and it was a nice compromise called Gatekeeper.

</comment></comment><comment><USER36> Recently was surprised to discover that the official uTorrent distribution, downloaded straight from utorrent.com, has some Spigot stuff in it. Was I tricked somehow or pre-hijacked already into downloading a non-authentic installer, or do they make money that way now—not yet completely sure.

</comment><comment><USER37> Pretty sure the "Allow apps downloaded from:" has been set to "Mac App Store and identified developers" since 10.8? Maybe 10.7?

</comment><comment><USER38> This is a big problem. Are there browser extensions that can block crapware?</comment>


How Germany re-introduced border controls (FIA) - <AUTHOR> https://blog.zoller.lu/2020/07/introducing-border-controls-in-schengen.html <comment><USER2> Thanks for sharing the information. I find it saddening that there is no broad discussion on the measures taken and their effectiveness. This would be rather important as many different groups of people suffered disproportionally during the lockdown. (Everywhere)

While I'm sure that this void of dicsussions will be filled by researchers, I'm not sure that the derived knowledge will arrive at the decision makers.

<comment><USER3> How is there no broad discussion? There's dozens of papers on the topic, both scientific and of the pseudoscientific "lockdown is worse than COVID, which is just a flu. here's a linear regression model"/"let's look at data that wasn't available to policy makers at the time" variety, and German media doesn't go a week without discussing the political or societal impact of one measure or another for a few months now.

<comment><USER2> I would not rate the current news reporting as broad discussion. The discourse in the media is biased and not helpful with regard to the future.

It's 80% parents complaining, bashing of food production plants, cooks going crazy and driving to the right edge of the spectrum. Reports of teens and young adults bashing glass on the week ends. All these reports burry the helpful discussions.

With regard to the scientific progress I have nothing to say as I'm not up to date. However producing high quality research takes some round trips to settle to an stable state. Nothing wrong with that process but the gained knowledge must be distributed back to the people. Currently this does not happen. Or: it happens and nobody is listening.

</comment></comment></comment><comment><AUTHOR> FIA Access to documents showing on what basis Germany reintroduced bordercontrol during the period of March 2020.</comment>


O2 (UK's original iPhone carrier) drops unlimited data with iPhone4 - <AUTHOR> http://shop.o2.co.uk/new-iphone/tariffs.html

<comment><USER2> I've been an O2 iPhone customer since the iPhone 3G launch. I'll happily switch to another network to not have to care about my data usage.

<comment><USER2> Sorry, I was eating an apple (oh the hilarity) while I wrote that with a spare finger...

I want unlimited data, but not because I want to make use of an absurd quantity of data every month - I want it to be unlimited _because I don't want to have to think about it_.

I realise that I'm ahead of the curve compared to a typical consumer, but I don't care - to me the Internet is a constant. It's just always there and I don't have to think about how much I'm using. It's like the air I breathe.

O2 have said that 97% of their customers will fit into the basic 500MB/month plan, so only 3% of people will really be inconvenienced by this. My response to that is - in that case why not simply invoke the "fair use" policies of your unlimited data plan to move those 3% of people onto a more appropriately priced plan and leave everyone else as "unlimited"?

</comment></comment><comment><USER3> How can people afford these increasingly expensive smart phones?!</comment>


Ask HN: How many of you actually use and rely on intellisense/autocomplete? - <AUTHOR> I used to think that autocompletion facilities like Intellisense and its variations were a key component to productivity when writing code. But I only thought this because I've typically used IDEs that provided that. I recently moved to emacs because it offers so many other conveniences, but the one item I found hard to get going reliably and usefully was autocomplete for my particular situation (which is not relevant).
But, I discovered how much I actually didn't need this kind of tool. I'm curious what others' opinions and experiences are?
In emacs if I really want to save typing, I go much further than any normal IDE and use yasnippet to customize much larger chunks of code for me, so I can still lean on that. But using intellisense as an "inline function lookup" doesn't appear as useful as I at first thought. <comment><USER2> For me it helps getting a feel for new languages / environments so I can get up to speed and tinker without losing too much momentum. Also camelCaseMethod completion (e.g. via ccm) can speed up progress dramatically if you know the exact method you want to call.

It's all nice as long as those features don't get in to my way (slow autocomplete, method and block templates that are harder to navigate than writing without help, massive method lists like Xcode shows for every possible message you might want send to that object etc).

I always liked the IntelliJ IDEA autocomplete for Java code as well as e.g. Spring/Hibernate Annotations; it was way more responsive than Eclipse and indeed a productivity feature saving time, alongside its impressive refactoring tools.</comment>


One OS to Rule Them All - <AUTHOR> http://blog.travisbloom.me/post/7725147378 <comment><USER2> "One OS"?; wouldn't that be Linux? Linux already runs on everything from smartphones and embedded systems to supercomputers. In fact, Linux is probably the most common OS in the world, _except_ on the desktop (and I don't think that is likely to change much).

</comment><comment><USER3> Somebody just got back from one of those cool, all expenses paid, junkets to Seattle! I wonder which superstars of the Windows world he got to eat lunch with at the Redmond campus! MMM mmm good!</comment>


Don't Use Booleans - <AUTHOR> https://www.luu.io/posts/dont-use-booleans/ <comment><USER2> That clearly depends on the expressivity of your programming language.

Python is an example where named arguments preserve context, readability and simplicity (especially for people not aware with language features, bitmasks, etc):

       fetch(
         accountId,
         includeDisabled=true,
         fetch_history=true,
         fetch_details=true
       );

Similar languages (Perl for instance) recommend passing dictionaries (hash maps), which roughly reads the same way. Typically, these ways are easily extendable without breaking the API.</comment>


Controlling the Police Cameras - <AUTHOR> http://www.technologyreview.com/view/539631/controlling-the-police-cameras/

<comment><USER2> Those seem to be good ideas. A nice balancing act, I guess that was what may have made it interesting to analyze for Richard Stallman.</comment>


Firefox Developer Edition 42 Release Notes - <AUTHOR> https://www.mozilla.org/en-US/firefox/42.0a2/releasenotes/

<comment><AUTHOR> "Indicator added to tabs that play audio with one-click muting" This is really useful. In my case, I usually have music playing in my browser and this allows me to quickly mute it while I watch a video. I just whish it wouldn't switch tabs when unmuting.

Asynchronous call stacks sound promising as well :)

</comment><comment><USER2> "Indicator added to tabs that play audio with one-click muting"

Awesome ! This is the only feature that could have me switch to Chrome. :)

</comment><comment><USER3> > "Process separation enabled by default in Developer Edition"

We are finally getting there...!</comment>


Ask HN: How do you manage all your 301 redirects? - <AUTHOR> As your site grows and product marketing changes, how does your team manage 301 redirects?<p>Are there any tools that make this simple or identify redirect chains? <comment><USER2> > Are there any tools that make this simple or identify redirect chains?

I have a Chrome extension you can use to audit the internal redirects on a site that might help that checks 100s of pages at a time:

[https://www.checkbot.io/](https://www.checkbot.io/)

It crawls the pages of site to identify redirect chains (can cause SEO + speed problems), 302 redirects (can cause SEO problems) and redirecting JS/CSS links (can cause speed problems). You can also pull up information like which pages are linking to the same redirecting URL to help you untangle/eliminate them.

I find it pretty common as well that sites will have unintentional redirecting internal links e.g. one page might link to the homepage as example.com instead of www.example.com, triggering a redirect from example.com to www.example.com which slows down browsing for users (similar for http vs https, and tailing slash vs non-trailing slash links).

<comment><AUTHOR> I signed up for the free account and this feels like a great product.

Definitely we're looking to identify redirect chains so this looks like it could be a big help.

Thank you for the help here.</comment></comment>


The Joy of Startup Design. And Sex. - <AUTHOR> http://thehappyplug.com/post/413017539/the-joy-of-startup-design-and-sex

<comment><USER2> the article is really an introduction to how our 'lizard brains' react to i/o, and specifically in the web. But the author finishes by recommending 'the joy of sex' as reading for your start-up. I'm sure there are better books out there which explain directly how to design 2 dimensional interfaces to entice people push and chose.

What books do you recommend for this sort of thing? Anybody read the joy of sex, and agree that it's the best?</comment>


Negative mood enhances creative problem-solving  - <AUTHOR> http://www.informaworld.com/smpp/content~db=all~content=a713751102

<comment><AUTHOR> Only the abstract is available for free, but the conclusion is interesting.

Two of the best hackers I've worked with were in a negative mood a lot of the time. Hmmm. Maybe I shouldn't be so cheerful.</comment>


Future AI: The Explorer and the Philosopher - <AUTHOR> https://www.centauri-dreams.org/2019/01/29/future-ai-the-explorer-and-the-philosopher/ <comment><USER2> I have yet to read a compelling formal engineering argument as to why humans are the best vehicle for a long term space exploration system. I've long argued that increasingly complex probes, eventually leading to a generally intelligent stellar exploration system, would be the most efficient way to understand the universe.

I wish people would just admit that human space exploration isn't really about optimized scientific exploration to increase knowledge, but about adventure and excitement.

<comment><USER3> I thought the idea was always more about colonization. Diversify the species extinction risk. Spread through the galaxy.

<comment><USER4> I think one of the most interesting questions of the next few [hundred|thousand|?] years will be whether humans move on to new planets (or moons, etc?), or whether humans will construct stations or habitats of some sort in Earth's orbit?

Or both, maybe? It could be that we try to, for example, go live on Mars, find out that it isn't worth it or feasible or something, and then go with Earth- orbit habitats.

Or maybe, neither.</comment></comment></comment>


Serverless image resizing at any scale - <AUTHOR> https://cloudonaut.io/serverless-image-resizing-at-any-scale/ <comment><USER2> I saw a popup appear and immediately went away.</comment>


Hack for Good - <AUTHOR> https://docs.google.com/document/d/1mpJt-kP_QlWxvnBORoMgySS6LNmHhq-Tyzt60zpQuS0/edit?usp=sharing <comment><AUTHOR> Hello.

Please help us find any possible errors or malfunctions before we officially launch the project.</comment>


Evolutionary System Development (pdf) - <AUTHOR> http://cs.gmu.edu/cne/pjd/PUBS/CACMcols/cacmDec08.pdf

<comment><AUTHOR> Since this is a PDF, I will put abstract here:

 _Large systems projects are failing at an alarming rate. It's time to take evolutionary design methods off the shelf._

 _Many critical large systems are failing. The replacement FAA air traffic control system, the FBI virtual case file, and the Navy Marine Corps Internet (NMCI), are a few of the many billion-dollar systems that could not deliver the functions needed. In stark contrast, the Boeing 777 aircraft, the Global Positioning System (GPS), and the U.S. Census database system have been outstanding successes. Why do some systems fail and others succeed?_</comment>


Brain waves can be used to detect potentially harmful personal information - <AUTHOR> http://sciencebulletin.org/archives/6145.html <comment><USER2> Technology can be used for good and for evil. To assure that EGC will only be used in authenticating users and not extracting personal conditions, there's a long road down the valley.

</comment><comment><USER3> Occultists know this for milenniums.

<comment><USER4> Occultists hate her! This one weird trick to read brain waves they don't want you to know!

<comment><USER5> HAHAHA! That was great.</comment></comment></comment>


Why the gig economy was doomed from the start - <AUTHOR> https://mashable.com/2018/06/14/gigged-book-gig-economy-mashtalk/ <comment><USER2> The article's title I see is "The gig economy crash, with 'Gigged' author Sarah Kessler". I can't see the word "doom" or phrase "from the start" in the article.</comment>


bcal v1.9 storage expression calculator gets REPL mode - <AUTHOR> https://github.com/jarun/bcal/releases/tag/v1.9 <comment><AUTHOR> _bcal features_

      - evaluate arithmetic expressions involving storage units
      - convert to IEC/SI standard data storage units
      - REPL mode with the last valid result stored for reuse
      - show the address in bytes
      - show address as LBA:OFFSET
      - convert CHS to LBA and vice versa
      - show binary, decimal and hex representation of a number
      - custom sector size, max heads/cylinder and max sectors per track
      - minimal dependencies</comment>


Java Tuning in a nutshell - <AUTHOR> http://randomlyrr.blogspot.it/2012/03/java-tuning-in-nutshell-part-1.html

<comment><USER2> These are heavily used bookmarks

[http://nerds-central.blogspot.com/2009/09/tuning-jvm-for- unu...](http://nerds-central.blogspot.com/2009/09/tuning-jvm-for-unusual-uses- have-some.html)

[http://marxsoftware.blogspot.com/2011/10/javaone-2011-defini...](http://marxsoftware.blogspot.com/2011/10/javaone-2011-definitive- set-of-hotspot.html)

[http://marxsoftware.blogspot.com/2011/10/javaone-2011-every-...](http://marxsoftware.blogspot.com/2011/10/javaone-2011-every- last-nanosecond.html)

<http://www.umbrant.com/blog/2012/twitter_jvm_tuning.html>

</comment><comment><USER3> This is excellent advice. I already discovered some of it (the hard way), but much is new. Many thanks to the author for posting such a compendium.

Many people don't care, but when you run code (written in Clojure) on the JVM, use multi-gigabyte heaps _and_ require API response times below 20ms, this type of tuning becomes critical.

</comment><comment><USER4> One of the best uses of a mindmap I've ever seen.

</comment><comment><USER5> they've managed to run linux on a potato, but java in a nutshell ?!?!?!?!</comment>


Some rambling conversations I've had on moving from MacOS to Debian - <AUTHOR> http://www.oblomovka.com/wp/2012/01/16/some-rambling-conversations-ive-had-on-moving-from-macos-to-debian/

<comment><USER2> Thanks to the submitter. I have to admit to liking articles like this. I guess I must be a real geek, because I also enjoy reading about computer security, encryption, and looking through people's screenshots (sadly, there is no screenshot thread here on HN).

</comment><comment><USER3> I weep when I see a European (Irish) web-priced X220 with SSD at €1,700.</comment>


React+Flux can do in just 137 lines what jQuery can do in 10 - <AUTHOR> http://swizec.com/blog/reactflux-can-do-in-just-137-lines-what-jquery-can-do-in-10/swizec/6740 <comment><USER2> Kinda linkbaity. As commenters in the article pointed out, the jQuery code omits the HTML and the React port has some functionality that is not present in the jQuery version, so the LOC comparison in the title is pretty meaningless.

The conclusion is that for a tiny one-off tool that can be banged out in 20 minutes, a big framework like React/Flux is completely overkill. But for large codebases where maintainability is a big issue, then React/Flux can be extremely useful (considering that's exactly the problem React/Flux try to address).

Am I the only one who thinks this stuff ought to be obvious?

<comment><USER3> This is more like what I'd expect a 20 minute React version to look like: [http://bl.ocks.org/insin/raw/b2f4901d4173677908a8/](http://bl.ocks.org/insin/raw/b2f4901d4173677908a8/)

<comment><USER3> And a Redux version because why the hell not: [https://gist.github.com/insin/2f9a4cf810eeed88da89](https://gist.github.com/insin/2f9a4cf810eeed88da89)

</comment></comment></comment><comment><USER4> I think this blog post is pointless. Of course every abstraction adds some overhead at first. If your problem is simple enough to fit in a few lines of code you don't have to think a lot how you structure it.</comment>


Ask HN: Laptop for Software Engineering student? - <AUTHOR> Hi HN! I bought a MacBook Pro the summer of 2017, the Skylake version and figured I'd be able to use it for a long time, since that had been my experience with their laptops. Unfortunately I'm not satisfied with the computer and I'm looking at getting a new laptop for dev. I don't play games, I just use my computer for writing code, general browsing and watching movies/shows every now and then. I'll be dual-booting Windows and Debian. I'm looking for a laptop that's reliable and has good support for drivers.<p>Does HN have any good recommendations? <comment><USER2> What about the MBP are you dissatisfied with?</comment>


The Discoveries of Continuations (1993) [pdf] - <AUTHOR> http://www.cs.ru.nl/~freek/courses/tt-2011/papers/cps/histcont.pdf

<comment><USER2> Changing topic a bit, the other day I tried looking for some paper on John Reynold's webpage and the ftp server is apparently offline: ftp://ftp.cs.cmu.edu/user/jcr

Made me think about how everything eventually comes to an end...

<comment><USER3> Or is it : [http://www.cs.cmu.edu/afs/cs/user/jcr/ftp/](http://www.cs.cmu.edu/afs/cs/user/jcr/ftp/)

seems like the same paper [http://www.cs.cmu.edu/afs/cs/user/jcr/ftp/histcont.pdf](http://www.cs.cmu.edu/afs/cs/user/jcr/ftp/histcont.pdf)

</comment></comment><comment><USER4> A good paper and a fairly easy read. Reynolds is diplomatic: While he mentions in the abstract that "In the early history of continuations, basic concepts were independently discovered an extraordinary number of times. This was due less to poor communication among computer scientists than to the rich variety of set- tings in which continuations were found useful" the truth appears to be more textured. In the body of the article he cites a number of occasions in which seemingly relevant papers were declined from journals for what seem to be petty reasons.</comment>


At 90, She's Designing Tech for Aging Boomers - <AUTHOR> http://www.npr.org/blogs/alltechconsidered/2015/01/19/377702882/at-90-shes-designing-tech-for-aging-boomers

<comment><USER2> please, please more of this. silicon valley seems determined to produce more things that fall in my bucket of "stuff my mum used to do for me" \- uber = mum used to drive me places, disruptive laundry, shopping for food, buying clothes... i'm not even sure if i'm half-joking.

</comment><comment><USER3> Makes sense. As more of the population ages, needs technology, lives longer but is also still interested in using technology, catering to this market will be both necessary and useful to companies.</comment>


The limitations of sampling profilers - <AUTHOR> http://danluu.com/perf-tracing/ <comment><USER2> > we need something that does more than just show aggregates, like a sampling > profiler would.

This isn't really a limitation of sampling profilers. This is really a limitation of the front-end nearly every sampling profiler stick their data into. Basically they just aggregate sampling data and stop there.

For the gecko platform we use a sampling profiler but we're able to correlate samples with unresponsiveness events. For instance you might have only 0.3% of your samples running Garbage Collection or X but that 0.3% was all consecutive and was scheduled during a VSync event and prevented the app from displaying the frame on time. You want your UI to highlight things that impact latency even if in aggregate the function was fast, because it still managed to make you skip a frame on the screen.

Tracing Vs. Sampling is just a data collection method. The issue here is the UI and how displaying the data in a strictly aggregate/bottom-up view isn't enough.

Here's a demo of our open source cross-platform multi-process _sampling_ low level profiler we use:

[http://people.mozilla.org/~bgirard/keyboard_timeline.mp4](http://people.mozilla.org/~bgirard/keyboard_timeline.mp4)

<comment><USER3> The ability to show something more than aggregates is, sure, a UI/front-end issue. (And an issue with intermediate storage, obviously).

But it's misleading to say that tracing-vs-sampling isn't a factor, unless you believe that the most-performant sampling and the most-performant tracing are roughly comparable. TFA claims otherwise; the claim is that the maximum resolution of tracing is much better than the maximum resolution of sampling.

<comment><USER4> That's nonsense. If anything the maximum resolution of sampling is higher, since it has lower overhead, so it perturbs the measured thing less.

However most sampling profilers only report CPU-time spent in a task, not wall-time. So, the things he discusses like time spent waiting for a task to complete would be obscured. However it would be perfectly possible to report the number of samples in which a thread was blocked in a method call (though it could not be said with certainty it was the same invocation, it would be enough to see that a majority of time blocked was elapsed there), it's just that they do not typically do so.

<comment><USER5> _maximum resolution of sampling is higher, since it has lower overhead_

The article claims otherwise.

This is apparently mostly related to the costs of preemption and resulting icache replacement.

 _though it could not be said with certainty it was the same invocation, it would be enough to see that a majority of time blocked was elapsed there_

But that's not quite what's being discussed. The main interesting thing here is tracking what oddness causes outlier worst-case times, which very much does require tracking individual invocations.

<comment><USER4> And I'm calling that aspect of the article nonsense. The cost of preemption is incurred only infrequently, so even if it perturbs the point after it measures slightly, it only does so infrequently and the point _at which it measures_ (assuming it has not been affected by the prior sample) more accurately represents a system without any instrumentation.

Assuming each invocation has a unique stack trace, each call site can still also effectively be tracked through sampling. Looking at his examples, this seems reasonably likely, as they all have quite different behaviours.

What tracing does do is permit a clear sequential analysis of an arbitrary granularity of macro behaviours.

If the macro behaviours are chosen with sufficiently _low_ resolution that they are large enough for instrumentation to be an immeasurable overhead, then you obviously get a very clear and accurate picture of the system behaviour at that reduced resolution.

Tracing RPC calls and other similar behaviours as done in the blog are a good example, but it isn't down to increased resolution; quite the opposite.

</comment></comment></comment></comment></comment><comment><USER6> Hi, cheap plug here for my really, really low-overhead event tracer: ppt.

[https://github.com/lally/libmet](https://github.com/lally/libmet) (BSD license)

Basically, you define structs that you then fill in your code. Then give them to a ppt-generated library routine. It's a bit of a pain, but you get two nice features:

\- The storage is completely non-blocking: you're just writing to shared memory.

\- Overhead is linear to write rate: if you have a spare CPU core on the machine, the overhead on your active cores is a constant + the cost of filling & copying (once) the struct.

\- It'll prefer to lose some data in case the disk falls behind in writing -- it won't slow down your app. Auto-generated sequence numbers tell you what/if any data was lost.

\- You can leave the code on in production: the listener attaches via ptrace() and injects a destination buffer into your process (constant time), and if there's no buffer, the data gets harmlessly dropped.

I've left it alone as I haven't needed it for myself in a while, and there hasn't been interest (I don't really advertise it, though). If anyone's interested, just ping me or file an issue.

</comment><comment><USER7> Sampling and instrumenting profilers both lie, in different ways. Sampling is incomplete; instrumenting has potentially strong observer effect. Neither are the ultimate or necessary future. They are both useful.

</comment><comment><AUTHOR> In retrospect I probably should have used the second half of the title: "glimpses of tracing tools from the future".

</comment><comment><USER8> The most interesting bit about that, to me, was the diagram of a google search's RPC tree. That was awesome, and I imagine these days it's a more more so.

</comment><comment><USER5> There's also a brief mention in [http://joeduffyblog.com/2015/11/19/asynchronous- everything/](http://joeduffyblog.com/2015/11/19/asynchronous-everything/) about making async/remote calls not break stack traces. That feels like it ought to be somewhat related to the distributed performance profiling discussed here.

<comment><USER9> [https://github.com/cocoonfx/causeway](https://github.com/cocoonfx/causeway) seems to be the current version of the early work that must've influenced the Midori work you link to.

</comment></comment><comment><USER10> sampling is useful to detect where to direct inquiry later, creating flame graph of the whole application is quite taxing and sampling can help restricting the research area

</comment><comment><USER11> Google could open source the profiler at some point.</comment>