Observations running 2M headless sessions - <AUTHOR> https://docs.browserless.io/blog/2018/06/04/puppeteer-best-practices.html <comment> fake-name I have a python project to manage headless chromium, which exposes basically the entire dev-tools UI:

[https://github.com/fake-name/ChromeController](https://github.com/fake- name/ChromeController)

It also supports some nice bits - there's a tab pool interface, it does execution lifetime management, and there's actual parameter checking for arguments generated by parsing the `protocol.json` file the browser supplies.

I use it as a mixin for some high(ish) volume web archiving I do as a hobby. It does a nice job poking through things like buttflare and other bullshit WAFs.

<comment><USER2> How do you archive? Screenshots?

I haven't found a way to do mhtml with chrome headless which would be so convenient..

<comment> fake-name Archiving backend: [https://github.com/fake- name/ReadableWebProxy](https://github.com/fake-name/ReadableWebProxy)

Stores the scraped content in a postgres database, with external blob storage for binary content.

Additionally, historical records of the scraped content are kept, so I have something that acts kind of like the internet archive too.

<comment><USER3> Thanks for your effort, this is an awesome project.

</comment></comment></comment></comment><comment><USER4> Excellent article and tips! I learned a lot from Joel's knowledge on Puppeteer and its intricacies.

I'm running a somewhat different business of which Puppeteer is a pretty big part. All points made in the post are valid, I could only add the following after having run ~400k Puppeteer sessions.

\- Race conditions happen. This issue [0] is causing roughly 3% of all Puppeteer runs to fail in my case. I had to bake in a retry mechanism.

\- Memory matters, CPU not so much. Just looking at my Librato and AWS stats, the CPU is mostly idle when running multiple concurrent sessions.

\- One way to establish compartmentalisation is to actually run each scraping session in a separate, one-off Docker container. You pass in the code via a disk mount or such. No hassle with too many tabs, or shared context between runs. Each container is destroyed after running.

[0]: [https://github.com/GoogleChrome/puppeteer/issues/1325](https://github.com/GoogleChrome/puppeteer/issues/1325)

<comment><USER5> I'm amazed that one must go as far as creating different containers - duplicating the browser, dependencies and whatnot - to achieve tab isolation and browser resiliency in Chrome, when Firefox does this with as little as container tabs.

<comment><USER4> Well, in my specific case it is also a security measure. In the context of a Saas you want very strict separation between user sessions. This was my main concern.

</comment></comment></comment><comment><USER6> Belated note: my work colleague has recently released an Elixir based solution to supervise Chrome, see:

[https://github.com/holsee/chroxy](https://github.com/holsee/chroxy)

[https://slides.com/holsee/deck-5](https://slides.com/holsee/deck-5)

<comment><AUTHOR> Very nice! It's amazing to me how many developers just run headless wo any sort of service-layer. You're gonna have a bad time :(

</comment></comment><comment><AUTHOR> Hey folks! I'm Joel, entrepreveloper behind this browserless.io service, happy to answer questions or chat!

<comment><USER7> Hey Joel–awesome service and post. Your tip about parallelizing via browser instances instead of tabs resonated with me as I've had a lot of headaches with one bad request taking down a full instance. That said, it does seem like the constant launch/close of puppeteer browser instances does have some sort of memory leak – Here's what happened when I switched from parallelizing via pages to browsers: [https://imgur.com/a/rCz3oow](https://imgur.com/a/rCz3oow)

Have you seen any similar behavior in your system? I initially thought I was forgetting to close the browser instance but triple checked my code and it does seem like I'm calling close() everywhere.

<comment><USER8> We have found 2 related issues:

\- Sometimes Promise returned by page.close() never resolves so it's good to call Promise.race() on that together with a Promise that resolves after some timeout period (30s?)

\- Sometimes Chrome process doesn't get killed so we are also manually killing remaining Chrome process after browser.close()

<comment><USER7> Good tips. How are you killing remaining zombie Chrome processes? Just polling the processlist and running "kill" commands?

</comment></comment></comment></comment><comment><USER9> Some of the puppeteer notes are effectively in the docs and quick starts. I've found puppeteer to be an absolute joy to use after years of phantom, custom scrapers, common scraper libraries, etc.

At a certain scale things obviously get harder but making your "sessions" ephemeral handles a lot of the resource issues.

The cat and mouse with people looking to block puppeteer access may heat up a bit too. But for the most part puppeteer makes doing this stuff 100x easier.

<comment><AUTHOR> Right - probably repeating some stuff already out there. The main issue I've been seeing on puppeteers slack channel is folks who are totally new, and don't read the docs, so trying to help all devs effectively

</comment></comment><comment><USER10> Very interesting to see this. I've been running a similar infrastructure rendering sometimes up to 500k pages a day, although often without images. I'm also running on Digital Ocean, but using nightmare.js ([https://github.com/segmentio/nightmare](https://github.com/segmentio/nightmare)), which runs on top of Electron, which in turns runs on top of Chromium.

The CPU and RAM patterns I see are different, with fixed CPU usage at near max and memory oscillating between 65% and 80%. I believe this is due to the different usage pattern, I basically _always_ have at least 20/30 jobs running concurrently on each machine, and they're usually fairly long (up to 10 minutes or so).

Contrary to what you mention, I've never had an issue with pages crashing and bringing the whole browser down. Maybe it has happened, but it's definitely negligible compared to the benefits I get by running say 5 pages in parallel. For some tasks I've also had some luck overriding the cross-content policy and using dirty ol' iframes to render multiple webpages in the same session.

I've considered migrating to puppeteer, so it's encouraging to see large scale project sharing their experience with it.

<comment><USER11> Same experience here - we run a docker instance of Chrome using tabs for multiple pages rather than multiple browsers, and they regularly run for days without issues. Of course their RAM usage gradually expands but it is easy enough to systematically stop & start the container, thanks to some built in error and fault handling to retry any requests which failed.

I can't say I've seen one tab bring down the entire browser, but I'm sure thats feasible, but thanks to docker and the fault handling above, it'd restart the instance and be up within seconds.

<comment><AUTHOR> This is what I've initially noticed as well (app will run for several days and eventually the account will cancel due to unavailability ... you'll get paged).

The screenshot I showed in that post is an instance that's been running for _months_ under high load. I can't stress enough that using tabs/pages will always result in frequent restarts which can be really tricky if there's other sessions that need to gracefully finish.

</comment></comment></comment><comment><USER8> Problem with fixed concurrency is that required memory per Chrome process varies a lot.

We ([https://www.apify.com](https://www.apify.com)) are solving this by autoscaling number of parallel Puppeteer instances based on memory and CPU. Our open source SDK ([http://github.com/apifytech/apify- js](http://github.com/apifytech/apify-js)) implements this using class PuppeteerCrawler ([https://www.apify.com/docs/sdk/apify-runtime- js/latest#Puppe...](https://www.apify.com/docs/sdk/apify-runtime- js/latest#PuppeteerCrawler)) which internally uses AutoscaledPool that provides autoscaling:

\- [https://github.com/apifytech/apify- js/blob/master/src/autosc...](https://github.com/apifytech/apify- js/blob/master/src/autoscaled_pool.js)

\- [https://www.apify.com/docs/sdk/apify-runtime- js/latest#Autos...](https://www.apify.com/docs/sdk/apify-runtime- js/latest#AutoscaledPool)

Sadly this feature is currently limited to our platform because it's mainly build for running in Docker containers. And as Docker container don't know about it's CPU consumption vs limits it requires a notifications about reaching its CPU limit from underlying platform. We have solved this using Websocket events. But we are currently working on extending this to work anywhere/locally.

<comment><AUTHOR> I've definitely noticed this as well and have been working on a "auto-scale" feature as well (to tone-down concurrency when under load). It's not an easy task, but you can see my first pass here: [https://github.com/joelgriffith/browserless/blob/master/src/...](https://github.com/joelgriffith/browserless/blob/master/src/Chrome.ts#L556)

</comment></comment><comment> squeaky-clean Great article! I've got several apps that maintain long running Selenium jobs (requires a real JS engine for the pages it hits) and ruthlessly killing containers after a each job and with a timer is the only solution I found. Zombie processes are a b __ __!

My team is in the planning phase of rewriting one, and we're going to use Puppeteer this time. Definitely sharing this with them!

<comment><USER12> Would Tini help?

[https://github.com/krallin/tini/blob/master/README.md](https://github.com/krallin/tini/blob/master/README.md)

<comment> squeaky-clean I may have misused the term zombie process. Seems that's when an entry for a terminated process doesn't clear it's entry in the process table.

I'm my experience Selenium will appear to be functioning normally from the outside, but the driver completely locks and doesn't proceed at seemingly random times. I couldn't think of a sane way to detect this, killing containers seemed easier and has worked reliably so far. I'm definitely open to hearing suggestions though.

That aside though Tini looks like a good starting point for containers like mine anyways. So thanks, it does seem helpful!

<comment><USER13> It runs out of entropy? Not sure if this applies to containers, but had something like this happen in a virtual machine we used for builds (Jenkins). I made the mistake of instantiating a cryptographically secure PRNG during _build_ time; the initialization needed entropy to seed the CSPRNG with, and this would lock up the whole virtual machine in the middle of the build. It seems that headless VMs have no possible source of entropy, unless explicitly configured to use host's source. Maybe something similar happens with containers?

</comment></comment></comment></comment><comment><USER14> One thing that I noticed is that if the website is behind Distil Networks, they will block on the first request and make cumbersome, if not impossible, to automate some task.

I get that is important to protect the information of their clients (which seems to be content aggregators), but there are legitimate use cases for allowing at least some scrapping to happen - one being when the information is about the person/company who wants to read it in an automated form so it can be used for further processing.

</comment><comment><USER15> At Ahrefs we are running between 130 and 170 millions sessions a day. The project started before the release of puppeteer. I can tell you the release of puppeteer saved me a lot of time. It's way more easier to keep a correct browser state using it than what was previously available. It also interacts well with lighthouse. I wouldn't call chrome headless stable or bug free (it doesn't even handle https correctly). But it's a good thing to have it available.

<comment><USER11> What issues do you have with HTTPS/TLS? I can't say I've come across any issues so far - is it a specific usecase to Ahrefs?

<comment><USER15> [https://github.com/GoogleChrome/puppeteer/issues/1159](https://github.com/GoogleChrome/puppeteer/issues/1159)

I don't think we have a fancy usage of puppeteer. But still we get between 2% and 5% of errors depending on the version used. It's a small ratio, but at this scale it's a lot of pages.

</comment></comment></comment><comment><USER16> I'd like to know, what are people on HN using headless browsers for?

<comment><AUTHOR> You'd actually be surprised by the applications, I definitely was. Some I've seen:

\- Screenshots

\- PDF generation

\- Functional tests

\- Regression tests

\- Scraping SPAs

\- Crazy stuff like printing user-generated content for personalization products..

<comment><USER17> Among some of these, I've also built an internal microservice for analyzing URL redirect chains. Input a URL and it will provide me with all of the redirects it goes through and information for each one including method (html meta, 301, etc), timing, etc.

<comment><USER18> I've done some hacky work redirect analysis but just the final url. Thought of open sourcing?

</comment></comment></comment></comment><comment><USER19> I recently played around with a headless Firefox instance to test a web app on Travis and so far it has been working incredibly well. Here is a sample .travis.yml file for those interested: [https://github.com/dom96/geckodriver- travis/blob/master/.tra...](https://github.com/dom96/geckodriver- travis/blob/master/.travis.yml)

</comment><comment><USER20> I'm running ~600k to ~800k chrome sessions a day, each in a new browser process (no tabs/pages) because of cookie interferences.

The magic is hidden in chrome's command line parameters which can significantly speed up the browser start.

I'm not using headless because I need an extension but most command line switches also apply to headless I guess...

A prepared profile/user directory also helps speeding up things.

<comment><USER21> Can you share the command line switches that help?

</comment></comment><comment><USER22> I think there is no need to manually `ADD dumb-init` when Docker provides an init (tini) process via `--init` flag.

[0]: [https://docs.docker.com/engine/reference/run/#specify-an- ini...](https://docs.docker.com/engine/reference/run/#specify-an-init-process)

<comment><AUTHOR> The reason I baked dumb-init into the image was in order to offer better dev UX (since I'm assuming some/lots/most folks out there will just `docker run` without much context on what's going on). I actually like specifying with the init switch better, but it opens up new devs to a world of hurt

</comment></comment><comment><USER23> At TestingBot [1] we run about 50k sessions / day in real Chrome browsers (each session is a new Virtual Machine with Chrome freshly launched).

Compared to the other browsers we offer, Chrome is by far the most stable.

[1] [https://testingbot.com](https://testingbot.com)

<comment><USER24> Hi, could you elaborate as to why TestingBot runs one browser per VM, as opposed to running multiple browsers on a beefier VM?

<comment><USER23> Because of security, we want to make sure each test can not be affected by a previous test.

If you'd run multiple tests on 1 VM, then one customer could change a property on the VM or alter a state that might cause the next test on the VM to fail.

By providing a pristine VM per test, we guarantee that each test cannot be affected by any previous test.

<comment><USER24> Thanks for the info!

</comment></comment></comment></comment><comment><USER25> Do people generally agree with `3. page.evaluate is your friend` over using puppeteer's methods?

<comment><USER26> the biggest annoyance with `page.evaluate` is transpiling javascript which adds helpers outside of the `page.evaluate` closure that in turn are not serialized and sent to the browser.

<comment><AUTHOR> That's a fantastic point: I'll add that as a gotcha</comment></comment></comment>


Advice to myself when starting out as a software developer - <AUTHOR> https://blog.pragmaticengineer.com/advice-to-myself-when-starting-as-a-software-developer/ <comment><USER2> > The best way to learn something is to teach it...Not only will you learn > something by teaching, but you'll also help and inspire others.

I especially like the last point about teaching what you learn and how you use accountability mechanisms like signing up to do a public talk to force yourself to learn the details. Great hack!

I think more of us should share what you learn and document our learning process</comment>


Academic Journals: The Most Profitable Obsolete Technology in History - <AUTHOR> http://sasconfidential.com/2015/11/06/obsolete/ <comment><USER2> "Gershman lays out his vision for the future of academic publishing...". The vision appears to be based on the concept of an 'overlay journal', such as the mathematical journal 'Discrete Analysis', which was launched in September 2015. See: [http://t.co/JmJQVNMUEd](http://t.co/JmJQVNMUEd)</comment>


LeanChess: The smallest chess-playing program at 328 bytes - <AUTHOR> https://leanchess.github.io/ <comment><USER2> This is really cool.

One trick I used to use to keep code small, was to take advantage of the fact that CX always equals 00h upon exiting a loop (when there are no short-cutting JMP related statements before it, of course).

Same goes for when you know what the registers will be set to after a particular interrupt has been triggered, such as if it has counted characters or loops or whatever. Not all of the return values are standardized, but a good lot of them are.

I've noticed sometimes people preserve registers a lot more often than they have to, like this:

        enter:
          push ax
          push bx

          mov  ax, ?  ; whatever value
          mov  bx, ?  ; whatever value
          int  21h    ; for example....

          pop  bx
          pop  ax

        exit:
          mov  ax, 0h
          mov  ax, bx
          ...

... to me this is really stupid code, but I see it a LOT in assembler. Instead, they could do something tinier like this:

        enter:
          mov ax, ?  ; whatever value
          mov bx, ?  ; whatever value
          int 21h    ; for example....

        exit:
          ; now stuff the regs to what you
          ; want them to be, keeping in mind
          ; the known exit values from the
          ; interrupt

I used to have a lot of tricks, since programming on the 8086 pretty much ensured you needed to know at least a few of them.

It's a rabbit hole, but a fun one to follow if you are trying to shrink your code. And if AX is already zero, use INC AX instead of MOV. But those kinds of things I'm sure you already do in your code. I haven't reviewed it all that thoroughly.

<comment><USER3> Thanks.

Indeed, the very first instruction is

      int 10h  ;BIOS display mode 0

that relies on AX being 0.

As for CX, there aren't that many loops in LeanChess (precisely 5, but who's counting :)). However, CH being always 00h allows to save a couple more bytes by doing

      mov cl, 8

instead of

      mov cx, 8

<comment><USER2> Yup, I actually got to the end of reading the code and noticed you do things like that. You have to read slowly to get all the nuances, so it was totally worth the effort.

The weirdest DOS "shortcut" I ever did was to remap INT 21h onto INT 3, giving me a single-byte INT 21h with a debug-killing side effect. Of course you have to have enough INT 21h's to convert to make that worthwhile, but it is a neat trick to have about if you have a lot of the same INT repeated throughout.

        sub     ax,ax
        mov     ds,ax
        mov     es,ax
        mov     si,21h*4
        mov     di,3*4
        movsw
        movsw

Of course if you aim it just right, you don't need the sub ax,ax. For that matter, it's an ideal place to use cx if you just exited a loop. And even then, you could probably find other short methods to make similar change. You don't even need to cli/sti because the chances of an INT 3 happening in the middle of the change is of no consequence, and all _your_ INT 3 calls will only happen after this part runs.</comment></comment></comment>


A Sad Irony: The Federal Judiciary's PACER Pricing Is Illegal - <AUTHOR> http://www.aarongreenspan.com/writing/essay.html?id=84 <comment><USER2> >Not surprisingly for government IT projects, PACER looks like most web sites did around 1994. It's actually not one web site at all, but more like one hundred, spread across different court districts and circuits in the country. The district-level courts each have their own custom version because judges demand custom features, and no one can refuse their demands. The appellate- level courts have an entirely different and separate infrastructure, written in a different language. (Technically-inclined readers, it's a Java applet. But really twelve different Java applets, depending on the circuit. No joke.) The Supreme Court, which can do whatever it wants, opts out of PACER entirely. Each level of PACER has separate login requirements, not to mention that PACER is separate from CM/ECF, an unbelievably badly-named system that actually lets you file documents, but under a separate login. What this means is that it is essentially impossible to follow a case from court to court, and if you even try, it is very, very expensive.

What. The. Fuck.

<comment><USER3> PACER was designed in the early 1990s, before the internet was popular, i.e., when AOL _was_ the internet for most of America. Indeed, PACER is probably as old, if not older than, most HN users.

It has not been upgraded since then, largely because Congress has not provided sufficient funds to the courts for them to do so (appropriations in the past two decades have largely been reserved for operations or for courtroom construction, but not technological upgrades. Much of the money from PACER access fees goes toward meeting the shortful in the operation budget, because Congress in its infinite wisdom has underfunded the federal courts for most of the last two decades.

(It's wierd that I have spent most of the last two days defending the American court system...)

<comment><USER4> This is such a load of self-serving bureaucratic nonsense. Pacer is an awful site that rips off the American people. Even if it is a cash cow, there's NO justification from keeping copyright free documents that are essential to democracy behind a paywall.

Increase the fees that Lexis-Nexis pays. Or charge more to submit documents. There's lots of solutions that could work to create a site that's usable and free to the American people.

This blame Congress crap has gone on long enough. When will the U.S. court system take its responsibility to the American people seriously? The answer: when people like you stop parroting its rationale.

<comment><USER5> Does 'rprasad work for the court system? Does he have some financial stake in it? If not, how is this "self-serving"? If it's just the idea that you find "self-serving", why did you target it at him specifically with "people like you"? What incentive does he have to "parrot" anything from the courts?

The US Courts are underfunded and have crippling case loads. Beyond that, is it even clear that any court has the authority to "increase the fees that Lexis-Nexis pays"? Or to charge more to submit documents?

<comment><USER4> Lexis-Nexis pays bulk rates to Pacer via contract. Attorneys pay to file documents. Those are set by the U.S. court system, not Congress.

He's making the same argument the U.S. court system has made, sub-rosa, for years, as it's _raised_ fees for accessing court documents. It used to be 8 cents a page. Jumped to 10 cents earlier this year.

</comment></comment></comment></comment></comment><comment><USER6> I'm not sure why RECAP (<https://www.recapthelaw.org/why-it-matters/>) is not mentioned more prominently in TFA.

"What does RECAP do? RECAP is an extension (or "add on") for the Firefox web browser that improves the PACER experience while helping PACER users build a free and open repository of public court records. RECAP users automatically donate the documents they purchase from PACER into a public repository hosted by the Internet Archive. And RECAP saves users money by alerting them when a document they are searching for is already available from this repository. RECAP also makes other enhancements to the PACER experience, including more user-friendly file names."

<comment><USER7> Well, to state the obvious, it's because recap is a system that is competition to the author's plainsite project. Petty squabbling much like the article itself.

<comment><AUTHOR> Actually, PlainSite depends upon RECAP, and I think it's a great product, which is why I mentioned the Princeton team. The essay was about PACER's pricing scheme, not about RECAP, but I fully encourage everyone to use it.

<http://www.recapthelaw.org>

Not only that, but I've pledged $5,000 to anyone who can extend RECAP to Chrome:

<http://www.aarongreenspan.com/writing/essay.html?id=83>

<comment><USER7> Fair enough, seems I rushed to a conclusion.

</comment></comment></comment></comment><comment><USER8> According to one commentator, that's not the only error in the article;

There are a couple of factual errors in this post:

"When you further consider that in the course of prosecuting one's own court case, one must refer to many, many other cases because of the nature of legal precedent, legal research can suddenly cost thousands of dollars—and that's without even hiring an overpriced lawyer."

Aaron misunderstands the nature of legal research. Legal research is done at the appellate level. In the course of prosecuting a case, legal research is necessary, but you won't be conducting legal research with PACER--it's not that kind of system. District court cases are not precedential with respect to other district court cases and it is rare that there is a usable opinion deciding the case. At the appellate level things are different, and there are both free and paywalled systems containing those precedents, such as Google Scholar (free) and Lexis/Nexis (paywall).

Putting aside legal research, it is not impossible to follow a case from court to court. Upon filing a notice of appeal, a district court case will be given an appellate court number which can then be followed on the appellate court's version of PACER.

<comment><USER9> > _District court cases are not precedential with respect to other district > court cases and it is rare that there is a usable opinion deciding the > case._

Lawyer and former litigator here. It's true that district-court decisions have the status of binding precedent authority only within their own districts. Such decisions, though, can be _and regularly are_ cited elsewhere as "persuasive" authority.

For example, in a California case, a lawyer might argue, in effect, "Your Honor, in a similar situation in New York, Judge So-and-So took Action X, for the reasons explained in her published opinion; we submit that you should take Action X too."

<comment><USER8> I never said that district court decisions had no value; simply that you're not going to find them easily in PACER. And there's nothing wrong with citing caselaw in F.Supp.; but my point is that lawyers do not normally do case law research using PACER. The way the US legal system should work is that once the parties see that Judge So-and-so is right, litigation should cease, right? But it doesn't work that way. Especially at the federal level--if Main Justice doesn't like a decision they'll keep shopping until they get one they like--as will any other litigant who has to litigate in multiple jurisdictions. And my friend, since when do district judges follow the decisions of their district colleagues? Those decisions are only persuasive as well. Look at my SDFla comment. The only binding precedents are those from the Supremes and those that come from your circuit (or the 5th, if you're in the 11th and there's a need to cite a pre-1982? case). Or the Fed. Cir. for matters within their jurisdiction. Or the FS Ct., but now we are really far afield.

</comment></comment></comment><comment><USER10> I agree with this essay, but I want to point one thing out.

      > Any startup could design a system better than this
      > for $10,000. (For a frame of reference, PlainSite,
      > which is roughly as complex as PACER, has cost Think
      > about $1,000 so far.

This is shortsighted. I think hosting a government website is quite complex. They have very strict security requirements and maintenance requirements, with very little economy of scale. $25M does seem a bit high, but I can easily see their costs above $1M.

<comment><USER11> I doubt that $10,000 is anywhere close to the cost of designing the system. $10,000 is not even enough to hire one senior developer for a month. And the designers would have to have domain knowledge of the legal system, which very few web developers have; you'd probably need to hire lawyers as consultants. Plus, the article mentions that different courts and judges have differing requirements, so just figuring out what the system needs to do seems non- trivial.

Also, even if the system _could_ be designed for $10,000, the yearly cost of operating it (hardware costs and personnel) would be significant, since all the data from the various courts would need to be entered daily, and someone would need to verify that the data in the system was complete and accurate. Plus, the system would need to be backed up and have redundancy and disaster recovery plans. You can't afford to shut down the legal system across the entire country because one data center loses power. We're not talking about Twitter here; we're talking about critical national infrastructure.

<comment><USER12> I'm a senior developer and I cost about that, at freelance rates. I could definitely design and build the things you mention in a month, providing the core functionality (a document storage system) is not overly complex.

And who said anything about the yearly cost of operating it?

<comment><USER13> (Slightly off-topic.)

> I'm a senior developer and I cost about that, at freelance rates.

For a month? Do you live in a place with really low cost of living?

<comment><USER14> For any place that's not New York, LA, or Silicon Valley, $120,000 is pretty solid money for a working professional.

<comment><USER15> How is 120k not considered good money in New York, LA, or Silicon Valley for a working professional? I live in New York and make in this ballpark and I consider it pretty decent money. Those cities are expensive, but I don't think so expensive that anyone wouldn't consider a 6 figure salary as somehow "not solid".

</comment></comment></comment></comment></comment></comment><comment><USER16> Note, the PACER incident is not the direct reason why Aaron was in the legal trouble he was in, that was his downloading of JSTOR's material, which under the law clearly illegal.

However there is widespread speculation that the PACER incident was behind why prosecutors may have wanted to throw the book at Aaron.

<comment><USER17> Just a note, but it's possible what Aaron did actually was legal. There was an article [1] by an expert witness in the case posted earlier today. So I think the trial would have been revealing. Admittedly the article doesn't prove anything, but I think everyone has so far been assuming Aaron's guilt. And while there is no denying the acts that he committed, they may have been legal.

1\. [http://io9.com/5975592/aaron-swartz-died-innocent-++-here- is...](http://io9.com/5975592/aaron-swartz-died-innocent-++-here-is-the- evidence)

<comment><USER18> A bit of a tautology there: If the expert witness that was on Aaron's side wasn't planning to testify he was not guilty... he wouldn't be the expert witness on Aaron's side.

<comment><USER19> But that doesn't mean the expert witness is wrong.

<comment><USER18> No, it just means he's not the impartial expert on the facts that some seem to be imagining him to be. I'm sure the prosecution had an expert witness who could have written a post saying "Aaron would have been found guilty."

<comment><USER20> For what it's worth, the expert we're talking about stated in his post that he usually testified _against_ hackers. He explained what he thought was different in Aaron's case in his post.

</comment></comment></comment></comment></comment></comment><comment><USER21> The article cites PlainSite as a cost reference. That surely can't be the cost of a developer, server, bandwidth etc? Free labour? Anyone know more? I know that an enerprising hacker can do a lot, but setting up a system better than another that has a $25 million budget? And would anyone care to guess what it should cost per page (ideally with some sort of cost justification)? 1c? .01c?

<comment><USER22> It should cost $0 per page. The infrastructure and maintenance for a document database of our country's tax-funded court proceedings should be so vanishingly small in relation to what we already spend on the justice system that it should approach 0.

</comment></comment><comment><USER23> FYI, Interesting bit from the PACER TOU:

>"The per page charge applies to the number of pages that results from any search, including a search that yields no matches with a one page charge for no matches. The charge applies whether or not pages are printed, viewed, or downloaded. There is a maximum charge of $3.00 for electronic access to any single document."

It is also interesting to note that pricing is decoupled from actual bandwidth usage, or any other metric that could be correlated directly with the cost of providing the service.:

>1\. We use a formula to determine the number of pages for an HTML formatted report.

>2\. Any information extracted from the CM/ECF database, such as the data used to create a docket sheet, is billed using a formula based on the number of bytes extracted (4320 Bytes equals one page).

>3\. For a PDF document, the actual number of pages are counted to determine the number of billable pages.

</comment><comment><USER24> Lawyers use PACER quite a bit, but it's not because they love it or make money from it (in fact, clients are increasingly rejecting "research" charges). It's just that PACER is often faster, easier, and more certain than using the firm's internal document management system. PACER is set up for litigation - DMSes, while they let you store anything, don't understand what the content is.

Our startup is basically tackling that problem - and one of the side benefits will be dramatically reduced PACER bills. There is no reason firms should ever be looking up their own case documents on PACER, since they get them for free in the first place.

</comment><comment><USER25> For once I find myself agreeing with Aaron Greenspan, except AaronSW's latest legal troubles were with JSTOR, not PACER. The article should have at least pointed that out.

<comment><AUTHOR> Tom,

A lot has been written by many people about JSTOR, so I wanted to focus on PACER and its pricing, but I do mention JSTOR here: <http://www.aarongreenspan.com/writing/essay.html?id=82>

Aaron

</comment></comment><comment><USER26> It is by no means clear from reading this what the meaning of "necessary" is in context. _Maybe_ it means "necessary to cover costs," but I can't tell from this post.

<comment><AUTHOR> See Senator Lieberman's 2010 letter to Senate Appropriators for clarification (which I've updated my essay to reference):

"As you know, Court documents are electronically disseminated through the PACER system, which charges $.08-a-page for access. While charging for access was previously required, Section 205(e) of the E-Government Act changed a provision of the Judicial Appropriation Act of 2002 (28 U.S.C. 1913 note) so that courts "may, only to the extent necessary" (instead of "shall") charge fees "for access to information available through automatic data processing equipment." The Committee report stated: "[t]he Committee intends to encourage the Judicial Conference to move from a fee structure in which electronic docketing systems are supported primarily by user fees to a fee structure in which this information is freely available to the greatest extent possible… Pursuant to existing law, users of PACER are charged fees that are higher than the marginal cost of disseminating the information."

Since the passage of the E-Government Act, the vision of having information "freely available to the greatest extent possible" is far from being met, despite the technological innovations that should have led to reduced costs in the past eight years. In fact, cost for these documents has gone up, from $.07 to $.08-per-page. The Judiciary has attempted to mitigate the shortcomings of the current fee approach in a variety of ways, including limiting charges to $2.40-per-document and the recent announcement that any charges less than $10-per-quarter will be waived. While these efforts should be commended, I continue to have concerns that these steps will not dramatically increase public access as long as the pay-per-access model continues."

<comment><USER8> Keep in mind that the US court system is not funded through filing fees. Courts generally have struggled with funding. Minimum mandatory penalties and the drug wars caused Miami to end up with four downtown federal courthouses and a (relatively) new federal prison. These facilities were not inexpensive. To get back to the principal issue, the Government gets free access to PACER (and even free photocopies at the courthouse) whereas a private defendant, like Swartz, has to pay for these out of his own pocket. The government gets a case agent, the defendant gets none. The defendant may get a public defender, but only after he has exhausted his funds--and this happens.</comment></comment></comment>


Nomads at last - <AUTHOR> http://www.economist.com/surveys/displaystory.cfm?story_id=10950394 <comment><USER2> this falls in line with the idea that computers of tomorrow might be what a keyboard is to a computer today.

any individual computer will be just a tool used to connect to your online virtual desktop.

can this already be done with windows virtual machine?

<comment><USER3> Sure, but I think there's more to it than that-- if you take the promise "ubiquitous computing" seriously, there's a lot of potential input/output devices available besides a traditional keyboard/mouse/monitor.</comment></comment>


Market share trends for SSL certificate authorities for websites - dane-pgp https://w3techs.com/technologies/history_overview/ssl_certificate/ <comment> dane-pgp IdenTrust is now the biggest CA, according to these statistics, largely due to being the cross-signer of the Let's Encrypt intermediate certificates:

[https://letsencrypt.org/certificates/](https://letsencrypt.org/certificates/)</comment>


Zeplin: Collaboration app for UI designers and front end developers - <AUTHOR> https://www.zeplin.io

<comment><USER2> I am not very familiar with iOS and Android development workflows, but this seems like a nice integrated set of tools. I also like the UI very much (from what I've seen in the video)</comment>


How Silicon Valley is Hollowing Out the Economy (And Stealing From You To Boot) - <AUTHOR> http://business.time.com/2013/05/07/how-silicon-valley-is-hollowing-out-the-economy-and-stealing-from-you-while-theyre-at-it/?xid=rss-topstories&utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+time%2Ftopstories+%28TIME%3A+Top+Stories%29 <comment><USER2> Well, this article is retarded, where shall I begin? Kodak killed by Internet companies, what a joke, Kodak died of its incompetence to transition from film to digital tech -- Nikon, Canon & co. are still doing pretty well selling expensive camera gear. If anything, lots of people are buying these cameras to shoot their Instagram photos -- or at least, buying expensive smartphones that contain optics from traditional camera companies (Carl Zeiss is around since 1846... I guess old dogs can learn new tricks when they are smart).

Translation software from Google/Bing "stealing" work from human translators?? That work was paid at some point, and if it was reused and repurposed, that's because it was made freely available by their copyright owners. For one thing, many of the training materials used by machine learning-based translation are official documents, e.g. from the United Nations and EU, because these are often long, complex, and their translation is both high-quality and focused on precision (which often doesn't happen with literature -- "traduttore, traditore").

</comment><comment><USER3> Very interesting discussion and timely in the sense that I spoke with a co- worker yesterday about how Google is obtaining free market research on businesses through the Google Online Marketing Challenge. Essentially, it's a competition where students consult with businesses, produce a report on the company, industry, etc and implement an Adwords campaign for them. No doubt I gained a lot of experience from doing this but submitting all the reports to Google and with them being able to aggregate everything, I'm sure they would find some value in it.

</comment><comment><USER4> The problem is even deeper: _information is non-scarce_. The marginal cost of production for any unit of information is _zero_ , and thus the natural market price for a piece of information is _zero_.

The problem being that while the creation and copying of _new_ information is _obviously_ massively valuable, we've not yet invented any effective and sensible way to trade information for physical goods like food and land and medicine.

The more information-based our economy gets, the more blatantly neo-feudal it gets, _because our property laws for information are blatantly feudal and have been for decades._ Popular examples of the problem include patent trolling and Mickey Mouse's regular copyright-extension bills.

Free Culture was a marginal movement in the past couple decades. It's going to quickly become the next big question of the world economy. When data is what makes the world go 'round, how do you compromise between compensating the original creator of the data, making use and extension of the data as widespread as possible, and not allowing "intellectual property" to expand into feudal-style veto or rent on everyone else?

<comment><USER5> > The marginal cost of production for any unit of information is zero, and > thus the natural market price for a piece of information is zero.

The cost to reproduce an existing piece of information is very low (not zero; you still need computing technology).

But the cost to produce an original piece of information is often still quite high. Look at how much it cost to make Star Trek Into Darkness, or for George RR Martin to write his next "Game of Thrones" novel, or Apple to produce iOS 7. These take huge investments in time and technology.

Even the premier product of free culture, the Linux kernel, receives millions of dollars of investment every year in salaries and supporting technologies.

<comment><USER4> Yeah, but the point is that the resulting product (like the movie, code, whatever) is a tradeable product. The act of creation itself is not something we can sensibly trade. People are trying with stuff like Kickstarter, but the core problem is still there: how much should you pay to _maybe_ get something later that _might_ match the description you _thought_ you liked?

Market economics doesn't work without excludability.

<comment><USER5> The solution to that core problem is to move your transaction to after the act of creation, so that you can see the final product before you decide whether you want to invest in its creation.

And that is where the concept of copyright came from in the first place.

</comment></comment></comment></comment><comment><USER6> Is "stealing" really the right word for this?</comment>


Unpowered RC gilder can reach 500mph using dynamic soaring [video] - <AUTHOR> https://www.youtube.com/watch?v=nv7-YM4wno8 <comment><USER2> I wonder if this dynamic soaring principal can be applied to wind turbines.</comment>


Workstation Show and Tell - <AUTHOR> http://warvo.com/2009/11/workstation-show-and-tell/

<comment><USER2> Boy and I thought I was cool with my two differently-sized monitors.

</comment><comment><AUTHOR> Haha yeah i reckon ay. You should post that comment onto my blog rather than on here :) I am more likely to reply.</comment>


Police admit they're 'stumped' by mystery car thefts -- can you help? - <AUTHOR> http://www.today.com/news/police-admit-theyre-stumped-mystery-car-thefts-6C10169993

<comment><USER2> There's no mystery. The thieves monitor the radio frequency car remotes use and record the remote's signal. Then they process what they've received to be able to mimic the remote adequately to fool the car's receiver.

All the thieves need to do is be near the car when the owner activates the legitimate remote, to pick up the remote's radio signal. Once they have the handshake between the car and the remote, they're in.

Elementary, my dear Watson.</comment>


Sites designed with System Fonts - <AUTHOR> https://github.com/mrmrs/system-fonts <comment><AUTHOR> Web fonts are awful. Would be great to get a long list of sites that don't use them.</comment>


Big Brotheresque App Kills Your Automotive Anonymity - <AUTHOR> http://www.wired.com/autopia/2010/12/new-app-marks-the-end-of-automotive-anonymity/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+wired%2Findex+%28Wired%3A+Index+3+%28Top+Stories+2%29%29&utm_content=Google+Reader

<comment><USER2> I'm sorry, but this is bullshit. Their privacy policy states they share personally identifiable information with third parties only with consent. Their entire business model is the exact opposite. They reason the site and app exists is to share personally identifiable information with third parties without consent.

This is nothing but vigilantism.

<comment><USER3> I assume they mean the private personal information of the _user_, which is different than the PII of the target.

Also I feel I must again correct this misconception that mere reporting of violations of the law is somehow equivalent to unauthorized meting out of punishment for lawbreaking. Vigilantism and reporting crimes are two different things.

<comment><USER2> Crimes should be reported to the police, not the public. A public report of a crime _is_ a punishment (see the Biblical story of Cain, public markings for adultery in colonial New England, etc.).

The sex offender registry in the U.S. reports crimes to the public, and at best, it's nothing but a source of FUD ("Find the sex offenders on your block!"). At worst, it's a way of branding offenders for life, keeping them from holding even completely innocent jobs. Don't forget you can get put on the registry for something as trivial as urinating in public.

Give me any reason to believe this system will lead to a better end than the sex offender registry and I'll change my mind.

<comment><USER3> The problem with the sex offender registry is that its implied claim ("this person has at some time in the past committed a crime that would lead us to believe they are a sexual danger to others") is often false. That is, the actual crime would not lead a reasonable person to believe that the offender present a danger to them. In the event that it were always true or the burdens added by the label were always justified, there would be no wrong in reporting it to the public.

> for "life"

You mean for the duration of their stay in the registry.

> Give me any reason to believe this system will lead to a better end than the > sex offender registry and I'll change my mind.

Ends are all well and good. But to me, a more important issue is that the public has a right to state facts about publicly committed acts in any forum they desire and which allows it. If people believe that it is beneficial to air their concerns about other people's driving in this forum, no tentative evaluation of the quality of the ends can be permitted to preempt that right.

</comment></comment></comment></comment><comment><AUTHOR> I would be willing to bet that most people who use the app to flag other drivers are themselves worse than average drivers. Does the app send flagger's info to the insurance company as well as the flagee's info?

<comment><USER4> I would hope so, because I'm betting most people would be using this while driving.

<comment><USER5> Very few of them are likely to have launched the app prior to getting in the car as well. So even with the voice entry of the license plate (how accurate is that going to be) you'll still have to get the app open, then pay attention to whatever prompts it gives you before submitting to make sure you are reporting the right license plate. This seems like a horrible app and very distracting.

</comment></comment></comment><comment><USER6> If, instead of reporting drivers to the DMV and their insurance companies this sent a message to a mailbox accessible by the driver, it would be pretty cool.

As it is, there are too many drivers so unaware of road rules and best practices that they people driving correctly are the incompetents. This will fail to improve road safety, although it may succeed at raising insurance company revenue by providing an excuse to raise rates.

</comment><comment><USER7> What's to keep 4chan from wrecking your insurance record if they decide they don't like you?

<comment><USER8> Basic human decency, covered with an icing of crossed fingers?

<comment><USER9> Read his post again, and pay close attention to the part that says _4chan_

<comment><USER7> Ding!

</comment></comment></comment></comment><comment><USER10> I can't wait until my parked car gets flagged by my neighbor that doesn't like me and my insurance rates go up.

</comment><comment><USER11> There are processes in place already to keep one accountable while driving. We can debate the effectiveness or lack thereof, but this is not the solution.

Generally, traffic citations and tickets are only reported to the DMV and insurance companies _after_ someone has been convicted in court or paid a fine. There is almost always the ability to show up in court and contest the citation.

DriveMeCrazy seems to circumvent this due process.

They recognize they have 'noise' in their data. That noise is going to cost time and/or money to sort out if an innocent driver has their license plate entered by mistake or by some jilted/angry/petty person.

<comment><USER12> it circumvents the 6th amendment of the constitution, this part in particular:

       to be confronted with the witnesses against him;

<comment><USER3> This right only applies to a citizen charged with a crime in a court of law. It doesn't give you the right to know what people were saying about you behind your back at the water cooler, or know which people are complaining about how you cut them off using this app.

<comment><USER12> when you speed you break the law, which is why you have the right to go to court to fight a ticket. If I get charged with speeding, and get 4 points on my license because some asshat thought I was speeding then I should have the right to cross examine the guy who reported me in court, like I can do with cops.

<comment><USER3> You seem to be confused. There is no reason to believe the police will charge you based on an anonymous comment from a user of this app. If they did, I am sure he would be called to the witness stand if you decided to contest the charge. But honestly I cannot see this happening at all.

I really don't understand what you are so riled up about considering in another thread not five minutes ago you claimed that the police never charge anyone based on traffic violations reported by other citizens. As I recall, you had this on good authority from your friend in the NYPD.

<comment><USER12> my problem isn't with the DMV aspect, since they can't do anything with it....my problem is with the insurance companies, who might just decide to factor this into their rate calculations.

And since the rates seem to be taken from thin air, they could raise your rates without you ever knowing that someone reported you.

</comment></comment></comment></comment></comment></comment><comment><USER13> Is it just me but, even though the information is sent to the DMV and insurance companies, it's completely unenforceable? How is this any different then me just calling your insurance company (assuming I have this knowledge) and saying 'Hey, this guy was driving like a tool on such-and-such a date'. What proof is there? If my insurance company or the DMV did anything with any of these reports against me, I would immediately challenge the veracity of the data.

</comment><comment><USER14> I look forward to reports of accidents being caused by drivers fiddling with their iPhones in an attempt to flag other drivers.

</comment><comment><USER15> While, I don't really like that these "offenses" are stored permanently, I would like a way to anonymously notify a driver they're doing something wrong.

I can't stand smokers that throw their (sometimes still lit) butts out the window. It would be nice to have some way to vent other than road rage.

<comment><USER6> Or even to leave neutral or positive comments:

\- Your left brake light is out

\- Thanks for being aware I was on a bicycle to your right and waiting before making that turn

\- Nice RomRaider decal on the side of your WRX, is there an open-source ECU tuner's group in this city?

\- I saw a black Mustang, license plate XXXX, clip the side of your parked vehicle and drive off.

</comment></comment><comment><AUTHOR> I have a friend who writes down license plates on his bicycle commute, and phones them in to the cops regularly. This app is no different from that. You should have no expectation of anonymity when driving a multi-ton vehicle on public property at speeds that can kill people.

<comment><USER12> so your friend ties up the 911 line over some petty bullshit like someone doing +5mph? Just fyi, the cops hate your friend, and I'd bet that they've never acted on a single one of his tips

<comment><USER16> "ties up the 911 line"

Where did he say he's using the emergency line?

"the cops hate your friend"

you don't know this obviously

"I'd bet that they've never acted on a single one of his tips"

even if that were true, the cyclist is going to feel better so long as the probability of them acting sometime is positive. Think revenge lottery ticket. Humans don't process small probabilities well you might as well use it to feel better after getting cut off in the bike lane.

<comment><USER12> -because the sort of guy to report people as often as was implied in the original post, is the sort of guy who uses 911.

-actually I do, my friend is with the NYPD, and they hate people like that since they waste their time, they can't give you a ticket unless they've seen you break the law

-he said/she said doesn't apply to traffic tickets, so no, there is literally 0 chance of any positive result

<comment><AUTHOR> If there are several witnesses to a felonious driving offence such as dangerous driving, the cops will most definitely track you down and charge you.

<comment><USER12> that's an old wives tale, ever called the cops yourself? I did.

Was involved in a hit and run, reported the guy right away, took them 2 hours to show up.

Had a tree fall on power lines, took them and the fire department 40 minutes to show up...granted there were no sparks flying. But still.

Had 2 accidents, first time had to wait for 4 hours for them to show up. The second time 30 minutes.

The point being, is that it takes a while for cops to actually act, and by the time they do, the guy you wasted precious time reporting, already left the jurisdiction

<comment><AUTHOR> I've called the cops 3 times, and they've never taken more than 5 minutes to show up.

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> reporting to the DMV and the insurance companies is over the top...those 2 entities specifically ignore vigilantism and your record can only be affected if a police officer who actually knows the law issues a citation.

all this will do is overwhelm the resources of the DMV/insurance companies, costing them money, which they'll transfer to their customers.

and seriously, the only people who'd do this, would be someone who does 60mph in the left lane, while everyone else is doing 80. And in fact, the guy is the one who is breaking the law, since he is impending the flow of traffic.

i look forward to having 4chan track this guy down...snitches get stitches.

<comment><USER3> > those 2 entities specifically ignore vigilantism

Reporting violations of the law is not the same as taking enforcement of the law into your own hands. The latter is vigilantism. The former is this app.

> all this will do is overwhelm the resources of the DMV/insurance companies, > costing them money, which they'll transfer to their customers.

I don't buy the line of argument that insurance companies and the DMV are incapable of filtering their email, and that somehow this is going to cost significant enough amounts of money that consumers will be affected.

> and seriously, the only people who'd do this, would be someone who does > 60mph in the left lane, while everyone else is doing 80. And in fact, the > guy is the one who is breaking the law, since he is impending the flow of > traffic.

Since we're going with juvenile arguments, I'll just say that when you assume, you make an ass out of u and me.

> i look forward to having 4chan track this guy down...snitches get stitches.

Jailhouse moral philosophy! Great finish.

<comment><USER17> Speaking of 4chan. This app could be their next uber prank. All they need to know is David Miscavige's license plate number, and I bet hundreds of Anonymous people will be targeting him all over California.

How about high school kids? Piss of a popular kid and all his/her friends could start targeting your license plate.

This app has huge potential for misuse, and the fact that he's hoping insurance companies may use it for keeping track of customers is very scary indeed.

<comment><USER3> > very scary indeed

I guess the hacker community is just as susceptible to fear-mongering as the average Republican, because dozens of reasons not to be concerned about this are screaming out at me:

1\. No one uses it yet.

1.1. We're not sure anyone ever will.

2\. There is zero evidence that it will cause any harm.

3\. There is zero evidence that, if it does cause harm, that harm will be irreparable.

4\. There is at least some reason to believe that it might have good effects as well.

5\. Many of the harmful things that people in this comment area are complaining about could already be accomplished through other means today. The actual increase in attack surface is essentially nil.

6\. Many of the commenters, while complaining about vigilante justice, are also calling for vigilante justice against the app's author. On the one hand, turnabout is fair play. On the other hand, if you embrace this sort of vigilantism, why not embrace the sort the app's author is promoting? Also, have you no shame, you hypocrites?

If you think that this app is "very scary indeed," I really believe you need to work on your fear tolerance because there are things in the world worth being afraid of and this is not it.

</comment></comment></comment></comment><comment><USER18> Can I use this app to report people who text while driving?

</comment><comment><USER19> This actually doesn't bother me too much, as long as the flaggees are informed of the flaggers identity. The responsibility should cut both ways.

</comment><comment><USER20> The whole point of _having_ a license plate (or "tag") on your car is to make you accountable. Any anonymity drivers may have had up till now has been entirely incidental. You are responsible for your driving, and you always have been since you got your state-issued license.

<comment><USER18> The license plate makes the vehicle's _owner_ accountable, mainly to ensure that the vehicle is fit to drive. But it doesn't identify the driver. I'd hate to see my insurance rates go up due to this app because I lent my car to a friend, or because someone behind us got aggravated when the car stalled while I was teaching my daughter to drive.

</comment></comment><comment><USER21> I can't see how this [reporting flagged subjects to dmv and insurance companies] is going to work. First of all they need a proof of violation that would be difficult to obtain. Also this company will get sued by flagged users into oblivion for violation of privacy. Awesome monetization idea though.

</comment><comment><USER22> I suspect this app will cause more bad driving than it solves. That's right. You. Put down that cell phone and watch the road.

<comment><USER23> Driving alone is at odds with any sort of mobile computing.

</comment></comment><comment><USER24> Not all bad: It's be cool if someone would report to me when my tail lights are out. Or that I forgot to turn my headlights on. Or if my gas cap is hanging out.

<comment><USER9> tail lights = your responsibility to make sure your car is in operable condition.

forgot to turn on your headlights = usually other drivers will flash or quickly flip their lights on/off to warn you

gas cap = again, your responsibility.

</comment></comment><comment><USER25> _the end of road-going anonymity can only improve safety... Drivers can ... even flirt with other drivers_

Can he have this both ways?

</comment><comment><USER26> Wow, this is horrible for too many reasons.</comment>


2nd Law of Thermodynamics Doesn't Prohibit a Decrease of Entropy - <AUTHOR> http://thinkinghard.com/blog/SecondLawDoesntProhibitEntropyDecrease.html <comment><USER2> This article starts from a completely wrong premise: a beneficial mutation is _not_ a decrease of entropy in a closed system. There's no closed system involved.

Organisms are not thermodynamically closed, because they take in both material and some sort of free energy (whether together in the form of food, or separately as in water, air, sunlight, etc), and also emit both waste heat and waste material. Organisms don't even have any (nontrivial) closed subsystems.

Most of the interesting things organisms do -- grow, reproduce, evolve progressively, maintain their states -- run counter to the general thermodynamic tendency for systems to approach equilibrium. This does not violate the second law of thermodynamics because organisms' inputs have lower entropy (and higher free energy) than their outputs. As long as there's more entropy leaving than coming in, internal entropy decrease is allowed by the second law (up to the difference between influx and efflux).

(Actually, I'm cheating a little bit here. Entropy is a property of things, not a thing itself, so talking about it entering or leaving isn't really right. But it _acts_ like a thing, and entropy flux is (usually) well-defined and acts as you'd expect flow of a physical thing to work.)

(BTW, there's a bit of inconsistent terminology here. Physicists call systems that don't exchange either energy or matter with their surroundings "closed". Chemists call those systems "isolated", and use "closed" for systems that exchange energy but not matter. Which physicists call "open". Yeesh. Anyway, this version of the second law applies to physicist-closed/chemist-isloated systems. Organisms exchange both energy and matter, so they're "open" in either terminology.)

<comment><USER3> > Organisms are not thermodynamically closed

Organisms are thermodynamic machines.</comment></comment>


Key lessons from CloudLock's $293M acquisition - <AUTHOR> https://amitkarp.com/2016/06/28/key-lessons-from-cloudlocks-acquisition/ <comment><AUTHOR> _What turned out really well for the company and the investors was that CloudLock was more scrappy than many of its competitors. [...] when the large incumbent security players all realized they need to acquire startups in this space, their best options were the startups that haven't burned a lot of money_</comment>


Anyone here feels that he/she doesn't have any knowledge depth or weight? - <AUTHOR> Doesn't matter how much you study or get to know new things, you have this feeling that you don't have any weight. (and this is not related to imposter syndrome, it is different) <comment><USER2> What is your interest of the topic? There's things you are passionate about and then there's things you have to do to get to what your passionate about.

</comment><comment><USER3> Why do you think this is different from impostor syndrome?

<comment><AUTHOR> You don't feel like a fraud, you are not "attributing your success to luck, or interpret it as a result of deceiving others into thinking you are more intelligent" [https://en.wikipedia.org/wiki/Impostor_syndrome](https://en.wikipedia.org/wiki/Impostor_syndrome). It is something else.

It is like you always feel that there is a huge amount of detail in every topic, and whatever you do, you don't get the feeling that you really know the topic.</comment></comment>


The structure of the 'THE'-multiprogramming system (1968) - vezzy-fnord https://www.cs.utexas.edu/users/EWD/transcriptions/EWD01xx/EWD196.html

<comment><AUTHOR> Is this the early days of unit-testing? I love the way he talks about the tests. A pleasant reminder that a well-formed test suite is akin to mathematical proof of functionality, as long as you've covered all the "appropriate" cases.

<comment><USER2> I know the Burroughs (1961) and Apollo teams did tests thorough enough to be compared to unit tests. Probably started there. Concept has been independently invented many times. The smarter one, though, was Cleanroom methodology's usage-centered testing designed to find all the bugs users would encounter in practice. Idea is they'd appreciate 30 bugs they never experienced more than even 1 that they did. That plus inspections/verification led to very low- defect software.

Cleanroom Software Engineering [http://groups.engin.umd.umich.edu/CIS/course.des/cis376/ppt/...](http://groups.engin.umd.umich.edu/CIS/course.des/cis376/ppt/lec21.ppt)

Empirical evaluation of Cleanroom methodology [https://www.cs.umd.edu/~basili/publications/journals/J33.pdf](https://www.cs.umd.edu/~basili/publications/journals/J33.pdf)

Agile is like a watered-down version of things like Cleanroom with unproven fads added. Check out Cleanroom to see what software _engineering_ looks like. One of few. One or two people even wisely combined it with Python given its readability and low-defect level. Worked very well. :) Ada or Ocaml would be the ideal, next project for Cleanroom or another engineering methodology (eg Praxis Correct by Construction).

</comment></comment><comment><USER2> "Select a project as advanced as you can conceive, as ambitious as you can justify, in the hope that routine work can be kept to a minimum"

I like that. Not sure it's practical lol. Would be nice. One I found true, though, was this:

"A less qualified young man, originally included, found our activities beyond his mental grasp and left the group. I mention this explicitly, because at least in Holland, the intellectual level needed for system design is in general grossly underestimated. I am more than ever convinced that this type of work is just difficult and that every effort to do it with other than the best people is doomed to either failure or moderate success at enormous expenses."

It seems especially true. Languages, compilers, OS's, verification of the above... the stuff that seems to be the best has taken some bright people to come up with. Old papers on Orange Book A1 systems said they usually got done because they picked geniuses to build them. Google uses the same model to come up with things like MapReduce and F1 DBMS far as I can tell. However, as A1 papers said, geniuses are in short supply. Oh, what to do...?

I think the prior work and 80/20 rule give the answer: leverage geniuses to build tools and methods that make it almost trivial for the rest to apply them correctly. Good examples are the capability-secure OS's making POLA easy, languages such as Ocaml knocking out tons of issues, parsing systems like LANGSEC, middleware like ZeroMQ, web runtimes like Opa (or Cornell's SWIFT), processors like SAFE (crash-safe.org), fault-tolerance like NonStop, and so on. Each is easy enough to learn to use day-to-day while knocking out most of the tough problems users will encounter. And each could be designed, implemented, and verified/validated by elite developers.

The success of tools such as Ada, Oberon, Java, Eiffel, and Python in eliminating entire classes of problems shows us the approach works. Just gotta make it even better, put the brightest on it, and absolutely make sure it's marketable to the rest. Ada, LISP, and Wirth languages' adoption showed last part is critical. Plus... (rolls eyes)... probably needs a bullet-proof, usable C/C++ or Java FFI. Can always have geniuses in metaprogramming convert all that shit later. ;)</comment>


These Aren't the Patent Remedies You're Looking for (No, Really) - <AUTHOR> https://www.eff.org/deeplinks/2015/12/these-arent-patent-remedies-youre-looking-no-really <comment><USER2> Seems to me like NPEs shouldn't be awarded extra damages for someone bringing the actual thing to market and unknowingly using a patent. Maybe we can call it Bob's Law. on a side note it would be pretty cool if you could just post an idea on a forum and if enough people agreed it became a law.</comment>


Facebook names the 20 people who can overrule Mark Zuckerberg - <AUTHOR> https://www.wired.com/story/facebook-names-20-people-overrule-mark-zuckerberg/ <comment><USER2> "This is the first time a private transnational company had voluntarily assigned a part of its policies to an external body like this."

Ayn Rand wrote a book about this too...</comment>


Pixel 4 has a flaw that no software update can fix: It Bends - <AUTHOR> https://bgr.com/2019/11/05/pixel-4-xl-bend-test-vs-iphone-11-google-has-to-fix-a-design-flaw/ <comment><USER2> Every phone, actually every solid, bends/breaks if given enough pressure. I can see the video of him bending the phone but I have no idea how much pressure he applied other than my seeing his thumbs shaking under the pressure. What would really be interesting would be for those reviewers to put all of the phones in a hydraulic press and measure how much force was needed to bend the phone.

</comment><comment><USER3> >iPhone 6 buyers discovered an annoying problem. Apply enough pressure on those first-gen thin aluminum frames, and you could bend them without damaging the structure of the phone. Apple was quick to fix its "Bendgate" problem and made sure it never happened again.

1 you do get plenty of "damaging the structure of the phone" with touch dying first

2 "Apple was quick to fix" by _never_ admitting any problems and charging users $150 repair fee [https://www.extremetech.com/mobile/239746-apple-will- fix-tou...](https://www.extremetech.com/mobile/239746-apple-will-fix-touch- disease-iphone-6-plus-not-free)

</comment><comment><USER4> How does this even happen? Do they not stress test this stuff before shipping?

<comment><USER5> It makes for a dramatic YouTube video, many add impressions will be sold with breathless headlines, but look at how much force he needs to apply and ask how often you do that to your phone? These flaps will lead to someone at that company making sure it's harder to repeat but it won't make a difference for almost anyone in normal life.

("Antenna gate" was similar: lots of heated prose while affecting many orders of magnitude fewer people than AT&T skimping on their network provisioning but the "look at those tech company engineers thinking they're so smart" angle distorts things)

<comment><USER6> > ask how often you do that to your phone?

Put your phone in your back pocket and sit down. This is exactly how bending became a big issue with the iPhone 6.

<comment><USER7> I've never understood putting your phone in your back pocket. Why sit on an $800-$1000 device? For me, it is not comfortable, and I would be paranoid about it falling out when sitting down or getting pick-pocketed. Can someone help me understand? I wonder if my front pocket preference stems partly from growing up with early cell phones that were much too thick to put anywhere but your front pockets (if even that).

<comment><USER8> As an example: women's jeans typically don't have front pockets large enough to hold contemporary phones. The solution is often back pockets, or carry it in hand (or in a bag).

</comment></comment></comment></comment></comment><comment><USER9> I wonder if this is a fake bending problem, like iPhone 6, or actually a real issue?

<comment><USER3> Are you saying Apple faked its internal documents stating 6 is ~3x more likely to bend than previous models?

<comment><USER9> Bends _more_ easily doesn't mean bends _too_ easily.

Of course what's too easy is a grey area. I do know as an owner (who's generally rough on his stuff) of a 6 plus for four years that they don't bend easily.

<comment><USER3> According to Apple internal documents "iPhone 6 Plus was 7.2 times more likely to bend" than iPhone 5. X times more likely means X times weaker. Aligning all the structural weak spots (sim tray, volume button, empty space on both sides of PCB) was a real stroke of genius.</comment></comment></comment></comment>


Linus meltdown on a Git pull - <AUTHOR> http://lkml.iu.edu/hypermail/linux/kernel/1510.3/02866.html <comment><USER2> Linus's Rant Rewritten [http://catcode.com/comments/2015/cf20151101.html](http://catcode.com/comments/2015/cf20151101.html)

<comment><USER3> Too weak. Uses weasel words. ie: _" Consider this replacement:"_ Linus' version is obviously too strong for some peoples' tastes, but the rewritten version doesn't send a clear message. A lot of things can be said about Linus' typical comments, but the messaging being ambiguous is not one of them.

<comment><USER4> > "Consider this replacement:"

Those aren't weasel words, that's direct feedback.

The rewritten message clearly indicates a rejection of the submitted code, so directing the submitter to consider the example replacement seems a constructive and appropriate response.

The rewritten version neglects the criticism that the submitted code is too fiddly relying on the short-circuit of the || operator, but Linus failed to mention that as well :p

<comment><USER3> I disagree. _" Consider this replacement"_ is practically an invitation to debate; or worse, possibly misunderstood as _" I agree your way works, but I like my way better, so let me be capricious about it and reject your PR because I can."_ Another poster re-rewrote it in a way that I feel is at least unambiguous.

<comment><USER4> Maybe. But imagine if Captain Jean-Luc Picard said it to you as a direct order. You'd probably get busy considering.

<comment><USER5> I'm a nerd, so sue me. I'd rather have the order from Captain Janeway.

</comment></comment></comment></comment></comment></comment><comment><USER6> Yes, simplicity... what a concept. Something the software engineering discipline seems to either loathe, be afraid of, or just want to avoid at any cost.

For example, why something as _simple_ as 'git pull' has had so much confusion and discussion. Why there are millions of views on S.O. on just the differences between pull and fetch?

[http://stackoverflow.com/questions/292357/what-are-the- diffe...](http://stackoverflow.com/questions/292357/what-are-the-differences- between-git-pull-and-git-fetch) ==> Viewed: 1080104 times

Good code doesn't just pass unit tests. It should be know-able, think-able, make some level of sense and generally be easily understandable.

<comment><USER7> We are still in the first century of software development, I doubt any code written today will be relevant by the time we enter the second. The computer science class of 2065 will enjoy seeing all these chaotic things we made, running in virtual machines hosted by museums that are yet to be built :-)

<comment><USER8> I sure hope no one will have to write a computer code in 2065.

</comment></comment></comment><comment><USER9> Don't call it a meltdown for goodness' sake. Linus only gives this treatment to the prima donnas at his level to keep them in place. It may sound impolite to say so, but some of them act as if they're God's gift to Earth.

Linus is doing what he needs to do to keep everyone in their place. Some of these programmers don't like taking no for an answer. I've been that guy before. I've been a real butthead to people with my code. So I respect what Linus is doing. Because I had it coming, and so did the recipient of this rant.

<comment><USER10> "No problem, I'll revert it all." \- direct quote from the so-called "prima donna".

<comment><USER9> Try this email [http://lkml.iu.edu/hypermail/linux/kernel/1510.3/02919.html](http://lkml.iu.edu/hypermail/linux/kernel/1510.3/02919.html)

>> Get rid of it. And I don't _ever_ want to see that shit again.

> I don't want to give up on that this easily

</comment></comment></comment><comment><USER11> If you think that's a meltdown, this must be the first Linus thread you've read.

<comment><USER12> Really the title should be edited to "Linus is uncharacteristically nice to a contributor."

</comment></comment><comment><USER5> The response can be found here:

[http://lkml.iu.edu/hypermail/linux/kernel/1510.3/03200.html](http://lkml.iu.edu/hypermail/linux/kernel/1510.3/03200.html)

      From: Rasmus Villemoes
      Date: Wed Oct 28 2015 - 10:30:41 EST

      On Wed, Oct 28 2015, Hannes Frederic Sowa <hannes@xxxxxxxxxxxxxxxxxxx> wrote:

      > Hi Linus,
      >
      > On Wed, Oct 28, 2015, at 10:39, Linus Torvalds wrote:
      >> Get rid of it. And I don't *ever* want to see that shit again.
      >
      > I don't want to give up on that this easily:
      >
      > In future I would like to see an interface like this. It is often hard
      > to do correct overflow/wrap-around tests and it would be great if there
      > are helper functions which could easily and without a lot of thinking be
      > used by people to remove those problems from the kernel.

      I agree - proper overflow checking can be really hard. Quick, assuming a
      and b have the same unsigned integer type, is 'a+b<a' sufficient to
      check overflow? Of course not (hint: promotion rules). And as you say,
      it gets even more complicated for signed types.

      A few months ago I tried posting a complete set of fallbacks for older
      compilers (https://lkml.org/lkml/2015/7/19/358), but nothing really
      happened. Now I know where Linus stands, so I guess I can just delete
      that branch.

      Rasmus
      --
      To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
      the body of a message to majordomo@xxxxxxxxxxxxxxx
      More majordomo info at http://vger.kernel.org/majordomo-info.html
      Please read the FAQ at http://www.tux.org/lkml/

</comment><comment><USER13> If anyone wrote a code review this way at a workplace, they would likely be reprimanded, do it again and likely fired. I'm not sure why we can't expect basic levels of professionalism out of project maintainers, much less decency.

<comment><USER5> That's not the point of code reviews. If you fired someone for this, then you are a remarkably bad manager.

Please don't ever manage people. Thanks.

<comment><USER14> No point in being less aggressive (compared to Linus) when doing code reviews?

I think when you hire someone the least you can expect from them is to not cuss out a co-worker for a mistake.

<comment><USER5> Ah, drat. I misread the comment. Forget what I wrote!

</comment></comment></comment></comment><comment><USER15> The title of this post is editorialized. I do think the topic is worthy of discussion though, so I upvoted it.

In spite of the tone and language, Torvalds manages to keep the issue fairly impersonal (The reference to 'braindamage' sort of crosses the line). Nowhere does he specifically call any specific individual out, even though he easily could with git blame on the specific line as well as grabbing the author, committer(s), and reviewers (individuals responsible for signing off the commit) information from git log.

Still, if my code received that sort of treatment, I would probably be in tears.

<comment><USER16> > if my code went that sort of treatment, I would probably be in tears.

I think the reason Linus feels justified in being an asshole is that he feels you can't accidentally write code this way. Rather, he thinks it's someone trying to make themselves look smart at the expense of the project, which is itself assholish behavior. I'm not saying he's right, but if you look at it from his perspective it's at least logically consistent.

</comment></comment><comment><USER17> FWIW, here's the author's response to linus's criticism:

[http://lkml.iu.edu/hypermail/linux/kernel/1510.3/02919.html](http://lkml.iu.edu/hypermail/linux/kernel/1510.3/02919.html)

meh, it's just another day at lkml, not a meltdown.

</comment><comment><USER18> " Get rid of it. And I don't _ever_ want to see that shit again.

Linus"

And this is why Linux dominates practically all computing platforms everywhere.

</comment><comment><USER19>
        if (overflow_usub(mtu, hlen + sizeof(struct frag_hdr), &mtu) || mtu <= 7)
            goto fail_toobig;

Any code which relies on the short-circuiting behavior of a logical operator in such an obscure way _is_ bad code. Linus may have gone a bit overboard in his critique, but, again, it was almost all criticism of the code, not the person. The only criticism of people is him calling people who like that code "out to lunch".

In short, I'm not seeing a meltdown.

<comment><USER20> Relying on the short-circuiting behavior of || and && in C is a long-accepted idiom. It's practically the _purpose_ of those operators.

[https://en.wikipedia.org/wiki/Short- circuit_evaluation#Commo...](https://en.wikipedia.org/wiki/Short- circuit_evaluation#Common_usage)

<comment><USER21> The obscure part is relying on a side effect of the first clause in the second clause.

<comment><USER22> That is pretty common and readable for '&&', but I agree it is too subtle and obscure for '||'.

</comment></comment></comment></comment><comment><USER4> If I maintained a large project and saw a pull request like that I'd lose my cool as well. Linus clearly needs better and more thorough subsystem maintainers so code like that gets corrected (or rejected) earlier.

</comment><comment><USER23> How much if any of this is just bike shedding?

I'm all for not pulling in useless broken code, but why the public rant?

<comment><USER24> The rant is about being overly fancy with the code and making it hard to read. If one scans the line, the thing that stands out first is the function being used. Not only is it a gccism that has never been used before, but it's a gccism that is used in a poor manner. By copying the "safe" subtraction to the same variable, you've now changed the behavior of the function. Before, if the mtu was too small, it would goto the error handler and not alter the mtu. The new function, if the resulting value is between 0 and 7, will overwrite mtu, match the second conditional, and be shipped off to the error handler with a new, invalid, value.

The second part of cleverness is in fact even more damning than that. The conditional || mtu <= 7 is a siren's call for anyone looking for a cheap optimization. To someone not paying attention, this looks like it could be a nice patch -- take out an or that couldn't possibly be true under any circumstance, and save a few ops on every incoming packet. Bundle it in with other optimizations, and you've got an overflow just waiting to go off.

So no, it's not bikeshedding, it's being too clever for one's own good, and in one line adds a new function call that's just barely been proposed for a standardized version of C, and has a pair of logic errors to boot. In short, ticking off pretty much everything you don't want to do in a line of code.

<comment><USER25> Thank you: the most comprehensive (and damn near only) comment on the page that talks about the technical decision being represented here.

</comment></comment></comment><comment><USER26> On a tangent: [http://blog.dilbert.com/post/129215160011/why-trump- insults-...](http://blog.dilbert.com/post/129215160011/why-trump-insults- people-part-of-my-trump)

Also: Linus rarely insults people, he criticizes the code, albeit harshly. [http://blog.codinghorror.com/egoless-programming-you-are- not...](http://blog.codinghorror.com/egoless-programming-you-are-not-your- job/)

</comment><comment><USER27> To be fair, Linus is the boss. He is the one who controls the Linux kernel development and just like at any firm, if the employees aren't doing a good job, they get shouted at. The difference here being that Linus' rants are in the public because of lkml while the scalding meted out by other bosses remain in the confines of their respective offices. Just another day at work here.

</comment><comment><USER28> Am I the only one who finds this fine and completely appropriate?

</comment><comment><USER29> > sh.t ... f.cking

Self censoring the swear words - how civil :D

Asserting dominance with aggressive behavior is fine I guess (with in boundaries) - it makes calling out someone much more powerful but at the same time makes you look like a tool if you're wrong and you lose respect faster - double edged sword.

</comment><comment><USER30> What a horribly editorialized 'clickbait' title.

</comment><comment><USER31> I would guess this discourages some good programmers who would have strong positive impact from trying their hand at helping out with the kernel.

</comment><comment><USER32> Pleasant fellow

</comment><comment><USER33> Linus is really incompatible with the way Software Engineering (safe spaces, toxic, problematic) is going, it is going to be a very enjoyable show to see the two sides butt heads constantly as they disagree with how he deal with people.

<comment><USER34> What leads you to believe that it's actually going that way? A couple high profile winers with too many twitter followers and no horse in the race does not a trend make.

</comment></comment><comment><USER35> it was indeed a piece of shitty code. Linus has right to protect his baby.

</comment><comment><USER36> Time for the quarterly "Linus said mean things to someone" post already?</comment>


Outlier Detection at Netflix - <AUTHOR> http://techblog.netflix.com/2015/07/tracking-down-villains-outlier.html <comment><AUTHOR> One of the authors here, I'll be around to answer any questions if anyone has them. I'm sure my colleague will be around as well.

<comment><USER3> At the risk of sparking off-topic debates about Netflix's content selections...are the same event streaming/data processing frameworks also used to detect trends and make actions on the content/frontpage side? Such as, detecting an anomalous spike of users watching some obscure movie in the last 24 hours, and then updating the "Trending Now" queues for users systemwide? Or is that something that happens (relatively) slow enough not to require an elaborate real-time framework?

<comment><AUTHOR> I don't work on the recommendations side of things, but this is actually the ONE technical topic at Netflix that I don't believe I am allowed to discuss. Sorry!

</comment></comment></comment><comment><USER4> Have you heard of Numenta? Their machine intelligence algorithms are great for anomaly detection in streaming data, including a product "Grok" for IT analytics ([http://numenta.com/grok/](http://numenta.com/grok/)). And all open source in NuPIC: [https://github.com/numenta/nupic](https://github.com/numenta/nupic)

<comment><AUTHOR> This was something I looked into when i performed the initial investigation for this project. It was a bit difficult to locate supporting academic material on the algorithm though. The white paper on the page seemed more like marketing material than an academic paper, which I imagine serves their business purposes better.

I will say the fact that nupic produces an outlier score and confidence score are things that would have been incredibly useful by the time this was brought to its end users. Definitely worth a look for anyone looking to do realtime stream processing for anomaly detection.

</comment></comment><comment><USER5> This is very interesting. Looking at the data though it is classified as "Errors per server". It isn't really disclosed what variables this figure entails but I'd have to imagine adding more information than simply error counts would improve the separability of the data?

<comment><AUTHOR> It would certainly help separability to observe the data in a higher dimensional space, however when you're taking automated action against the results its sometimes pertinent to know which specific metric is causing the server to be an outlier.

For example if network tcp retransmits are throwing it off we probably just want the system to kill it and let the autoscaling group bring up another server. If its memory usage we probably want to page someone.

<comment><USER6> Yes, which is a reason why all non-linear clustering techniques can be a challenge when investigation follows outlier detection.

</comment></comment></comment><comment><USER7> This is very interesting and a source of frustration I have with New Relic and the other alerting services we use. New Relic uses a 5-minute rolling average for error rates, and alerts when that average goes above some threshold. However, that means that it takes ~5 minutes from a spike occurring to an alert being created - even if the error rate has increased to 50%.

It would be much better for it to be doing this sort of outlier detection - <USER2> gradual increase in error rate to 3% should not trigger a critical alert, whereas a big jump in error rates should trigger an alert quickly.

Has anyone implemented a system like this?

</comment><comment><USER8> Would be interesting to try and fit a distribution to error rates (some type of counting process) and then monitor the probability of having the occurred number of errors (with in some period of time). Then low probability events might indicate an outlier.

<comment><AUTHOR> We have another component of the same outlier detection system that does this type of fitting, identifying low probability events using a Bayesian model + Markov Chain Monte Carlo. It hasn't gained nearly as much traction internally (yet) as the clustering approach here.

<comment><USER8> Cool. Have you found the MCMC approach to be less robust than clustering? It seems like this may be the case due to probability model assumptions.

</comment></comment></comment><comment><USER9> Why not take this a step further and have another system try to diagnose the issue with the identified outlier server before handing it over to the alert system? Seems like you'd be generating a lot of alerts otherwise.

<comment><AUTHOR> Our central alerting gateway (CAG) does much more than just send emails and pages. When the system was fist built we hooked into CAG directly as it is capable of taking many of the desired actions on its own.

</comment></comment><comment><USER10> [https://www.reddit.com/r/MachineLearning/comments/3dbrum/out...](https://www.reddit.com/r/MachineLearning/comments/3dbrum/outlier_detection_at_netflix/) reddit discussion on the same thing.

Note my thoughts are included and thought they might be of interest to anyone looking into this problem.

</comment><comment><USER11> The data looks like it could lend itself to an approach where you model the error rate based on the prior data (at simplest, get a mean and variance out of it) and then use a Chi-square critical range check to see if the last n (degrees of freedom in the check) measurements are likely to have come out of the modelled distribution. Is that something you've considered?

<comment><AUTHOR> Hey there,

We have considered modeling the distribution from which the data is typically drawn and then calculating likelihood of newly observed data. Some of the approaches that we use to detect anomalies on stream starts per second (SPS) now depend on these services. Same software package, slightly different solution.

A colleague of mine (Chris) implemented a data tagger which allows users to annotate data that is typically fed into this system. We have plans to have the backend automatically swap out the algorithm based on performance against their tagged data.

We've written about SPS here: [http://techblog.netflix.com/2015/02/sps-pulse- of-netflix-str...](http://techblog.netflix.com/2015/02/sps-pulse-of-netflix- streaming.html)

</comment></comment><comment><USER12> You could also have the client run some tests and auto report problems such as the bad server's id.

You could also have a manual report button the user can click on. And if you want to get really advanced, have a system that learns from those manual reports so that it can later warn when there is a high probability of a "problem" occurring.

<comment><AUTHOR> One of my colleagues designed and implemented a telemetry data tagger that allows service owners to annotate the data reported into our primary telemetry system Atlas [0] and has further worked on mechanisms for service owner feedback to automatically create training data sets for this implementation.

Great question, and I'm glad others are thinking along these lines -- we fully believe that this is important for bridging the gap between service owners and our analytics.

[0] [https://github.com/Netflix/atlas](https://github.com/Netflix/atlas)

</comment></comment><comment><USER13> I think the graphs would be better presented differently.

Instead of a line graph, try different opacities. So you have one line per server, going left-to-right, shading from white (for no errors) to whatever color (for the maximum number of errors overall). And perhaps a dot per server and a single line indicating overall status.

</comment><comment><USER14> Regarding devops alert spam, on the one hand it's possible to setup rules and filters and tweak notifications just right, but that is often an initial hassle.

Possibly a startup opportunity to offer an easier option with error logs + machine learning.

</comment><comment><USER15> In the audio world we would have run the data through an FFT, not sure if you can do that with non-audio data.

<comment><USER16> You absolutely can. The FFT is one cycle of a Discrete Fourier Transform, and since this data is both discrete and in regular intervals, it should behave identical to just sampling an audio signal.

<comment><USER17> I am curious why you say "The FFT is one cycle of a DFT", since the FFT is quite literally a DFT, only an algorithm in a better complexity class.

</comment></comment></comment><comment><AUTHOR> I should note that the team responsible for this (Insight Engineering) is hiring a manager[0] and engineers[1]. If you're interested in solving this kind of problem come check out the job post as I'm really excited to be building out our team.

[0] [https://jobs.netflix.com/jobs/2406/apply](https://jobs.netflix.com/jobs/2406/apply)

[1] [https://jobs.netflix.com/jobs/2259/apply](https://jobs.netflix.com/jobs/2259/apply)

<comment><USER18> How does someone get into doing work like this? This sounds like it would be pretty cool. I started my first job working with infrastructure/ops for a larger scale app and it's been really interesting.

</comment></comment><comment><USER19> For those of you on computers, here's the link with better formatting - [http://techblog.netflix.com/2015/07/tracking-down- villains-o...](http://techblog.netflix.com/2015/07/tracking-down-villains- outlier.html)

<comment><USER20> Ok, we changed to that from [http://techblog.netflix.com/2015/07/tracking- down-villains-o...](http://techblog.netflix.com/2015/07/tracking-down- villains- outlier.html?utm_content=buffer26802&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer&m=1).

<comment><AUTHOR> Thanks so much dang!</comment></comment></comment>


There's a Tool for That: 13 Essential Tools for the Ninja Web Developer - <AUTHOR> http://blogs.telerik.com/blogs/posts/12-05-08/there-s-a-tool-for-that-13-essential-tools-for-the-ninja-web-developer.aspx <comment><USER2> For the front-end web developer, LiveReload is a fantastic tool for minimizing context-switches while crafting CSS (plus variants) and JS.

</comment><comment><USER3> And not one mention of throwing stars.</comment>