The 1975 Buffett memo that saved the Washington Post's pension - <AUTHOR> http://finance.fortune.cnn.com/2013/08/15/warren-buffett-katharine-graham-letter/ <comment><USER2> The title is incorrect because Buffett doesn't say playing the market is futile. In fact, he clearly states he believes a few will outperform due to skill (but that you can't distinguish which outperformers did so on skill versus luck).

What he does say in the quoted part is that a large fund of say 20 billion likely can't outperform due to it's size. That is a big difference from saying that it's futile to play the market (since not everyone is trying to invest 20 billion). It's also similar to what other investors like Peter Lynch believed.. that smaller investors have an advantage over the large funds.

This isn't made clear at all in the linked page. As far as I know, Buffett has always felt efficient market theory is wrong and that a few could beat the market on skill. There's also nothing in the linked quote that contradicts this (just the title that suggests it).

So the title is clearly deceptive. It suggests that Buffett did not believe in investing skill at a younger age and only changed his stated beliefs after becoming a very successful investor.

<comment><USER3> > _Buffett has always felt efficient market theory is wrong_

People often refute the EMH without understand that it is a family of hypotheses, from the Strong to the Weak form, with a great deal of subtlety in their concept and meaning.

Buffet's argument in the linked excerpts is actually pretty close to a weak form of the EMH: all participants start with broadly similar information and capabilities, so performance naturally regresses to the mean.

Here's my favourite discussion of the EMH:

[http://skepticlawyer.com.au/2013/05/29/bubble-trouble-all- in...](http://skepticlawyer.com.au/2013/05/29/bubble-trouble-all-information- is-not-equal/)

<comment><USER2> That's a good point. Many people who don't believe in the strong version do actually fall into the weak form.

<comment><USER4> But there is a very significant difference between the two. The strong form leads you to believe that it is impossible to expect to do better than the average even with a Herculean effort. The weak form allows that it is possible to beat the market, even by a lot, if you have insight or capabilities that most of the market lacks.

This leads to wholely different conclusions. In (1), a cheap index fund is the only investment option that isn't equivalent to gambling. This is comforting to believe, because you could feel completely comfortable in not spending any effort on researching your investments. In (2), it is okay to buy a modest position in Google in 2004 because your knowledge of the untapped potential of the Internet hints that Google could be much more important than the (median) skeptic believes.

Perhaps most importantly, if you are in camp (2) you will consistently have your investment choices denounced by those in camp (1), which gets very annoying.

<comment><USER3> It's important to note that the EMH doesn't suppose that all actors agree on the price. It just means that the price represents the summed outlook of participants.

Those who think a stock is underpriced will bid it up to get more; those who think it is overpriced will sell out at progressively lowering prices.

Because the movement in prices creates an opportunity to profit for those who have information that is not yet revealed, those people will enter the market and create pressure on the price.

In some mathematical forms, this is assumed to happen instantaneously and universally. Thus: all available information is factored into the price. Essentially, you can't arbitrage because you have instantaneously driven up the price already by arbitraging. (It's weird, yay calculus).

The looser forms basically say that this is what happens in the long run, on the average, even without instantaneous adjustment of prices. And when you compare market time series to pure randomness, they have similar characteristics. So in a sufficiently large group of agents, some will profit and some will lose _merely by chance_. Then you're back to taking an average and being unable to beat it in the long run, because in a game of chance outcomes converge to the long-run probabilities of the game.

<comment><USER4> Okay, so would I be correct to assume that you claim it is in the long run impossible to beat the market except by dumb luck?

I don't see how this follows from your reasoning. You've allowed the possibility of beating the market if an individual trader has some knowledge or insight which the average of the rest of the market does not. So presumably an investor which only acts when he has such knowledge would beat the market also in the long run.

<comment><USER3> The EMH grew out of the empirical observation that prices wiggle around randomly (and the different forms look at different ways available information could be responsible for that).

Buffet himself, in the letter, has already said that most investors start with comparable amounts of information and capability. So speculating by buying and selling according to predictions of future values of stocks will eventually cause a regression to the mean.

Buffet disagrees with the EMH but he agrees with the conclusion that you can't beat the market by speculating on the _movement of prices_. He basically lets the random wiggle happen, and when a price dips to what he thinks is a bargain, he buys.

The problem is that we struggle to test the null hypothesis. We can't create 1,000 parallel 20th centuries with 1,000 Warren Buffets to see if he comes out significantly ahead a statistically meaningful number of times. We can only compare him to chance. In a sufficiently large sample, long streaks of perfect performance can emerge purely by chance.

<comment><USER4> So what you're saying is that it may (or may not) be possible for an individual inevstor to consistently beat the market through skill, but that it is impossible to verify whether an investor who beat the market is in fact skilled or just lucky.

<comment><USER3> I guess so. Naturally I secretly suspect in my heart of hearts that _I_ am smart enough to out-perform the market ...

<comment><USER4> If you believe Buffett's thesis that skilled value investors who do their homework consistently beat the market, it stands to reason that there should also be other strategies that consistently beat the market (let's say over 20 years or so). It also stands to reason that value investing, since it is such a heavily publicized strategy, probably _does not_ beat the market today, since it is such a widely published and acknowledged strategy.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER5> His point is that playing the market is "speculation", not investing. He differentiates this with "investing", where you simply try to assess the intrinsic value of a stock, and if the stock is trading at a significant discount to it, then you buy it.

Further, once you've bought it, you keep it. You don't try to time the ups and downs of the market, buying moving and out of the position quickly. Since this is how Buffett likes to invest, one of his key principles is investing in owner-operators or people that are behaving as such. You want the executives of the company to treat it as if they owned all the stock and could never sell it. Because when you have the people running the company worrying about next quarter, you're not worrying about next decade.

Also related, Buffett's two key rules to investing:

Rule #1. Never lose any money

Rule #2. Never break rule #1

</comment><comment><USER6> I remember reading a quote from a top trading executive which went something like the following:

"Only a fool plays the stock market without insider information."

</comment><comment><USER7> This is why investing in an index fund is a great decision. When you rely on more active management, you pay higher fees, and your returns are likely to be about average (or worse). With an index fund you can take a healthy return, mitigate risk by diversifying well, and pay a fraction of the fee (which equates to hundreds of thousands of dollars over 30-40 years from the added growth).

<comment><USER8> Better yet, be the one charging the fees.

</comment></comment><comment><USER9> Meta: QZ is utterly unreadable to me on both desktop and mobile. Fortunately there's Readability.

<comment><USER10> More meta: the article uses lowercase l to represent the digit 1.

<comment><USER9> Perl can do that in Underwood compatibility mode (obscure Perl Journal cover reference...).

</comment></comment></comment><comment><USER11> Quoting from page 13:

 _A further problem is that in no case were the superior records (returns) I have observed based upon institutional skills which could be maintained despite changes in the faces. Rather, the good results have been accomplished by a single individual or, at most, a few, working in fairly specialized areas in which the great bulk of investment money simply had no interest._

So... he seems to predict a bleak future for Berkshire's returns after his departure as well?

<comment><USER12> He's been planning for his retirement/demise for a very long time. He's pretty good at playing the long game, so I wouldn't bet against him.

</comment></comment><comment><USER13> A quote from Warren Buffet: "In addition to the ones benefitting from short- term luck, I believe it possible that a few [stock portfolio managers] will succeed—in a modest way—because of skill."

Yes, but with an appropriate degree of scientific skepticism, the "skill" assumption could actually be chance. And Occam's razor argues that chance is the more likely cause, not skill.

For 100 stock fund managers buying and selling stocks, statistics tells us that 50 of them will do better than the market averages (and 50 will do worse). And a few of the managers will do very well indeed, but not because of skill. In fact, one can design a blind computer model that buys and sells stocks randomly, using pseudorandom numbers to make the decisions, and half the accounts will come out ahead of the averages, and a small number will do much better than the averages, just as in real life -- all because of chance.

More here: [http://arachnoid.com/equities_myths](http://arachnoid.com/equities_myths)

<comment><USER14> What is the market? A bunch of fund managers. So your critique about not beating the median like saying half of all hockey players are worse than the median and so hockey is just a game of chance. You have to fix your prior. Track performance for some period of time. Then track the winners. This will show whether it's a game of skill or chance.

<comment><USER15> The market is NOT a bunch of fund managers. Its a representation of the state of assets being debt/credit or company shareholding.

The market does not do better because there are fund managers, it does better if the underlying assets increase in value. All the other fluctuations are based on expectancies of those assets and arbitration.

It is entirely possible for 100% of fund managers to do below the market average, as fund managers dont own 100% of the market. It is also likely they have a losing bet against a market, as active fund managers employ resources in their operations, by trading and actively managing a portfolio, usually in the form of fees, and potential downsides on extra taxation as well.

Tracking "the winners" is the most fallacious and common attempt at gauging how good a fund manager or a stock is doing. Bernie Madoff was one of the winners for decades. Past results give no expectation to the future, which is so misleading given that the most common tool to show the value of a stock is a chart of how it did before, as opposed to how the company is doing (which require reading the statement).

I suggest reading "The Intelligent Investor". It is by far the best book on this topic out there , and it has enough layman terms and concepts that are very easy to understand once you expose yourself to them.

<comment><USER14> While I think The Intelligent Investor is a worthwhile book, I think it got it wrong for the very reasons I mentioned above.

Anyway, I'm not making any argument here for how the market works, other than pointing out that saying that half of fund managers do worse than the median does not imply that they are unskilled or not providing value. Nor am I saying that they do provide value, but rather the original theory is both flawed and insufficient to make a conclusion.

In games of skill, past results certainly give expectations of future results. Going back to the hockey analogy, I would expect the top 10% of performers over five years (determined by some metric) would as a whole outperform a random sample of 10% of the league in a sixth year. Madoff is a bad example here - he was breaking the law, but nobody knew. If you catch a top sports player doping, it doesn't suddenly mean that the game is a game of chance, or that all of the top performers are cheaters (although repeatedly catching cheaters may indicate systematic problems) - rather it means that someone cheated to fake performance. They were playing a different game, but nobody knew.

Going back to the original point, if you want to determine whether something is a game of chance or skill, you have to fix the prior.

Let's say we had coin flipping tournaments, and people were claiming that it's a game of skill. Looking at past statistics, some people seemed to perform far better than the expected 50-50 split. You however claim that coin flipping is just a game of chance. How do you prove it? You look at results for some period of time, say five years. Take the "winners" from that group - say the top 20% of performers. Under the hypothesis that it's a game of skill, these should be a selection of the most skilled people.

Now track their results for a sixth year. If it was a game of skill, the expectation would be that as a whole, these individuals would continue to perform at a high level. However, as a game of chance, you would expect performance to be completely random among these high performers - and by doing this, you can determine whether coin flipping is a game of chance or skill.

</comment></comment></comment></comment><comment><USER16> Warren Buffet also said:

"I'd be a bum on the street with a tin cup if the markets were always efficient"

<comment><USER17> I think one particular problem with the strong-form efficient market hypotheses is this:

A stock should already be priced for all of the publicly available information. However, that information has to be analyzed and interpreted. That's the catch.

"Today company XYZ announced their purchase of Chinese company ABC."

There are probably many different ways to interpret that information. Maybe the purchase will result in increased profit, maybe it will bankrupt the company. Nobody knows.

If all public information were easily interpreted, i.e. if X happens the stock will drop by 5%, then the markets probably would be efficient.

<comment><USER3> The EMH doesn't propose that participants have future knowledge. It says that current prices reflect current information and that in the long run, all participants regress to the mean.

<comment><USER18> Well yes, but he's still correct. Not all participants accurately gauge what the available information means for their positions, and even for those who do, figuring it out takes time.

</comment></comment></comment></comment><comment><USER19> There is no evidence that mutual funds (or any other active managers) can outperform the stock market.[1] This can be explained by the strong form of the efficient markets hypothesis; "In strong-form efficiency, share prices reflect all information, public and private, and no one can earn excess returns." [2]

[1] [http://faculty.chicagobooth.edu/john.Cochrane/teaching/35150...](http://faculty.chicagobooth.edu/john.Cochrane/teaching/35150_advanced_investments/Luck%20versus%20Skilll%20in%20the%20Cross%20Section%20of%20Mutual%20Fund%20Returns.pdf)

[2] [http://en.wikipedia.org/wiki/Efficient- market_hypothesis#Str...](http://en.wikipedia.org/wiki/Efficient- market_hypothesis#Strong-form_efficiency)

<comment><USER20> The market is definitely not strong-form efficient. If it were, you wouldn't have any price movement attributable to earnings surprises, because the private information would already be priced in. More importantly, markets are only efficient because some people are researching and trading to the true underlying value. The whole reason markets are supposed to be efficient is because if they are not, there is an opportunity to make riskless arbitrage profit, which someone will exploit.

<comment><USER19> Strong form efficiency does not imply perfectly predictive of all future events. Such a claim would imply that no durable good could increase in price at a rate higher than the federal funds rate plus some risk premium (haircut).

</comment></comment></comment><comment><USER21> If what he says here is true, the parents of every kid on the planet should test their child for some aptitude in this area--at least a little--because learning your kid was preternaturally adept at, basically, "Making a Lot Of Money" would be the most important thing that ever happened to the kid. Also wouldn't it have been one of the most sensational "news events" of all time, if nothing else, when someone honestly reported Someone Has Found the One Weird Trick To Becoming Really Rich, And It Always Works If You Do It Right. The rise in billionaires I surely would have heard about, at least.

So what's the catch? Why can't I just go and study this book and make a lot money if I think I'm smart, now that I've read this article?

<comment><USER13> > So what's the catch?

The catch is that testing people for an unexplained adeptness at making money in equities is essentially certain to produce a handful of "successes" that arise from chance, not talent.

Given a pool of ten million investors, all playing the market, and all making random picks, it is certain that a handful will succeed spectacularly _because of chance_. I know because I have modeled this sort of thing with computer programs, each of which produce "winners" simply by making random picks.

With a large enough population, some will "succeed" spectacularly, through chance. Those successes will find it hard to accept that their success arose from chance rather than genius, just as happens in real life.

> Why can't I just go and study this book and make a lot money if I think I'm > smart, now that I've read this article?

If there really was a book that contained secrets to beat the market, secrets that worked, that were reliable, that anyone could put into practice, businesses would abandon equities as a way to raise operating capital. After all, why should they line the pockets of a bunch of non-productive speculators? What possible incentive would they have?

The only reason the equities market works is because investors get a fair return on their investment, consistent with the risks they take, and businesses get operating capital at a reasonable rate as well. Both parties to the transaction get a fair deal. All this talk about making a killing in stocks using "secrets of the winners" overlooks the fact that, if it were possible, if someone could really make a fortune by something other than blind luck, reliably, deterministically, the market would collapse.

And for the life of me, I can't understand why people find this so hard to understand.

</comment></comment><comment><USER22> The title is misleading, but Warren Buffett is right.

The more money you manage, the harder it is to beat the market as a trader. However, if you're playing with a modest amount of money, you can play the market, since moving small amounts of money around is much easier than say, 20 billion...

</comment><comment><USER23> I read all this 1975 dated letter and while I enjoyed every piece of it, this phrase contained in the very last page is what caught my attention the most:

"Conventional approaches to money management should not be expected to produce above average results."

It basically means that if you want to perform better than your competitors you logically should act differently from them. As in distancing yourself from the status quo.

I can see this advice being spot on for startups as well, it has everything to do with thinking outside of the box and redefining the rules of the game.

Good ole Warren nailed it decades ago.

</comment><comment><USER24> I had a funny thought reading the letter (the actual letter at Forbes, not the vacuous qz.com summary).

The letter is type-written, with nice tables and footnotes. That probably means that it was either hand-written by Buffet or by a secretary (or stenographed or whatever), and then given to a typist to format.

In turn that means that some random lowly typist got to snoop on all this premium investment advice, that 20 years later would be becoming common wisdom. I wonder if she (I assume a typist in the 70s would be female) noticed and appreciated it, perhaps acted on it.

<comment><USER4> Probably not. It's a sad fact that women are on average disinclined towards risk-taking, and investment is a classical risk-taking activity. This effect must have been even more pronounced in the 70s.

</comment></comment><comment><USER25> and yet he himself is an example of exactly the opposite mentality. He has been an outlier manager for decades and would have been a great choice of manager for anyone he would take money from. He effectively takes new money whenever his holdings pay dividends and reinvests it successfully. Roughly speaking he's returned 100% (vs 50% for the Sp500) in 10yrs giving about a 7.2% return rate. He consistently beats the market and then claims that "managers" cannot.

<comment><USER26> Not quite - I think his point was that money managers _with very large sums to invest_ cannot beat the market. He made that claim almost 40 years ago, and more recently his own record clearly contradicts it. So perhaps he no longer holds such a view...

Edit: It looks like Buffett had changed his view by 1984 [1]: "Size is the anchor of performance. There is no question about it. It doesn't mean you can't do better than average when you get larger, but the margin shrinks."

[1] [http://www.tilsonfunds.com/superinvestors.html](http://www.tilsonfunds.com/superinvestors.html)

<comment><USER27> It's simple math. A money manager controlling 100% of the market by definition can't beat the average ;) So the potential for good performance must increase along that spectrum.

</comment></comment></comment><comment><USER28> The factor I think he missed: All those managers timing trades in their random way has a randomizing effect on the system overall.

But I'm not an economist, any ideas if or how that influence manifests?

<comment><USER29> In aggregate, it wouldn't much matter since randomness would cancel itself out. So broad index funds wouldn't see that volatility.

At shorter scales, however, randomness like that would add volatility to individual stocks, so you'd see more random results for returns among small portfolios -- thus creating quite a few genuinely lucky fund managers.

</comment></comment><comment><USER30> It's interesting that this article states "Buffett, then _just_ 44 years old" while in the tech industry 44 years old is considered...

<comment><USER31> I would expect similar language to be used if this were instead an article about something Donald Knuth said when he was 44. The "just" reflects the public familiarity with the _82_ year old Buffett.

</comment></comment><comment><USER32> I love this line:

> Wall Street abhors a commercial vacuum. If the will to believe stirs within > the customer, the merchandise will be supplied -- without warranty.

</comment><comment><USER33> Tell that to the hedge fund billionaires. Oh wait, knowledge networks, I guess it's all just insider trading.</comment>


How the hell does Wolfram Alpha work? - <AUTHOR> http://www.techradar.com/news/internet/how-the-hell-does-wolfram-alpha-work--599561?src=rss&attr=all <comment><USER2> Summary: a large database + Mathematica + natural language processing that doesn't always work.</comment>


An analysis of reported cases of 2019-nCoV related deaths outside Hubei province - <AUTHOR> https://sinopsis.cz/en/an-analysis-of-reported-cases-of-2019-ncov-related-deaths-outside-hubei-province/ <comment><USER2> [https://www.worldometers.info/coronavirus/](https://www.worldometers.info/coronavirus/)</comment>


EINTR and PC Loser-Ing: The "Worse Is Better" Case Study (2011) - <AUTHOR> http://blog.reverberate.org/2011/04/eintr-and-pc-loser-ing-is-better-case.html <comment><USER2> Wow. I am the author of this, but I forgot all of this after 8 years and it was like reading an article written by someone else. :)

I think the only thing I would add now is that SA_RESTART does seem like inherently the wrong design for this. The code that registers the signal handler might be totally unrelated to the code making the system call. They might be from different libraries, written by different people, completely unaware that the other exists. The right place to specify whether the system call should automatically retry is at the place where the system call is made. So I don't think Unix actually evolved into the "right thing" here.

<comment><USER3> Yes, it's fraught. Unix didn't evolve to the "right thing", if indeed there is a right thing to evolve to.

An important consideration here is that Unix, for much of its history, ran on machines with only a few kilobytes of memory -- a really big one might have 256k. So unnecessary complexity in the kernel translates to it being too big to be useful at all. Extra code in user programs wasn't great either, but couldn't make the whole system unusable, so it was less bad.

System calls that might return EINTR could have been wrapped in a standard library call that would take, say, an extra argument that says what to do -- maybe even a function pointer to call. But there also wasn't really a standard library, yet, and not everybody wanted that much junk going on around their system calls. You got less than a million instructions per second, and system calls burned often a millisecond or hundred. Anyway, what would a library pass to it? Something it got from its caller?

But if you want a loop, you know how to write one, and you can put in it exactly what you want. So the only problem is what happens in some library you use that does system calls. At the time it was considered good form to return to the caller if you got EINTR, and let the caller decide what to do, instead of looping in the library. Then your caller could have a loop calling you, instead.

Then BSD did their thing, and then library callers that might need to break out had to have a setjmp to break out to, and generally had to shut down immediately, afterward, because the library was probably left in a corrupt state.

So things are still not right, by any defensible definition.

But there's no evidence Richard Gabriel really understood these tradeoffs, or cared, really. It was really all a metaphor for LISP vs. C, where C was "worse" and drove out LISP, and Unix and PDP-11s drove out TOPS-10 and then the Lisp Machine, and then the PC came along with DOS, and the world went all to hell.

And here we all are!

</comment></comment><comment><USER4> There is a genuine complication that this post seems to have missed. If a system call is interrupted and returns to user space (due to a signal, most likely), then it is in one of three states: nothing happened, the call finished, or it is part-way done. If nothing happened, then PC can point back to the syscall or EINTR can be returned. (EINTR is a promise that no progress was made.) If it's all done, it can return success. But, if it's only part way done, then either it needs to restart transparently or the user code needs to be prepared to handle this.

With recv(), for example, the return value can indicate that some but not all bytes were read. When read()one from a file, most programs aren't prepared for partial success.

Linux has some ERESTARTSYS mechanisms to handle some of these cases. The code and the semantics are a bit gross.

<comment><USER5> > most programs aren't prepared for partial success

But read() and write() return the number of bytes actually read or written. So actually they are, or should be...

<comment><USER4> On a normal file, unless it's NFS with a special mount option, read() and write() never return partial results, which means that many programs can't handle it.

Even modern async/await/promise/future systems frequently have no reliable way to express partial success.

</comment></comment></comment><comment><USER6> The Unix side of this comparison is talking about signals, but what is the ITS side talking about? A technical description of that is available at [0].

>Another way to state the PCLSRing modularity principle is to say that when a process looks at the program counter of another process, it must always see a User mode PC, never an Exec mode PC. When one process wants to access the state of another process, and the target process happens to be executing in Exec mode, then something must be done in order to put the target process into User mode.

[0] [https://hack.org/mc/texts/pclsr.txt](https://hack.org/mc/texts/pclsr.txt)

<comment><USER7> And to complete the circle, the Unix way for one process to look at another's program counter, ptrace(), works via signals.

I recently had to implement a version of that behavior when writing a GDB stub for an embedded microkernel. The system in question originally had neither a debugger nor an equivalent of signals, or any other way for system calls (message sends and receives) to be interrupted – so the programs written for it naturally assumed they wouldn't be, and didn't have any code to retry interrupted operations. I could have gone through and changed those programs, but it would have been a lot of work and easy to get wrong. Instead, just as described in that link, I had the GDB stub set the PC to point back to the system call instruction, so it would be restarted after continuing the thread, and adjust the argument registers if necessary (e.g. if a combined message send+receive operation was interrupted after the send but before the receive, the arguments had to be changed to request only a receive). This was made more difficult by the fact that the microkernel originally threw away some of those register values, in the name of efficiency, assuming they wouldn't be needed once the system call had started…

</comment></comment><comment><USER8> Discussion on EINTR wrt close in Linux: [https://lwn.net/Articles/576478/](https://lwn.net/Articles/576478/)

Glibc shows use with TEMP_FAILURE_RETRY: [https://www.gnu.org/software/libc/manual/html_node/Opening-a...](https://www.gnu.org/software/libc/manual/html_node/Opening- and-Closing-Files.html)

Using TEMP_FAILURE_RETRY when close always deallocates the fd will at best get EBADF, at worse you close a random file descriptor.

<comment><USER9> This is the best example of how bad EINTR is as a design, in my opinion. I was painfully aware of how hard it is to write correct code when dealing with interruptible system calls, but even I didn't know that close() could potentially be interrupted.

<comment><USER10> It's not hard to write correct code in the face of EINTR. You can simply try again. Or just use libc (or another language's wrapper).

There is a real need for interrupts during a "slow" system call. I much prefer EINTR to sth. like completion ports. EINTR is simple and usable. (I don't understand well enough the problems around interrupted close())

<comment><USER9> One cannot simply try again. One has to take the byte count returned by read() and write() and deduct that from what was expected. One has to find out what time it is after returning prematurely from sleep() and nanosleep(). For a lot of the interruptible system calls, one has to write defensive code.

<comment><USER10> > For a lot of the interruptible system calls, one has to write defensive > code.

Errrr, duh. Remove "interruptible" and that sentence still holds very true.

If you don't care about being interruptible, write restarting wrappers doing a simple subtraction. (Or use fread(3)/fwrite(3)). It's not hard.

nanosleep(2) gives you back the remaining time, it's about as easy to use as read(2)/write(2). Very simple. sleep(3) doesn't, but due to seconds resolution it's not a generally useful API anyway. Still useful for quick debugging. Not a system call, but a libc call, by the way.

<comment><USER9> I think we are in complete agreement that it is not hard to write defensive wrappers to work around the fact that the API is a mess.

</comment></comment></comment></comment></comment></comment><comment><USER11> "The "worse" system (Unix) did indeed do "the right thing" eventually, even if it didn't at first. "Worse is better" systems incrementally improve by responding to user needs. Since users got tired of checking for EINTR, the "worse" system added the functionality for addressing this pain point. The whole thing did leave a rather large wart, though"

Which of course is exactly what RPG predicts about 'worse' systems: after a lot of pain, they will gradually evolve into doing the right thing, but probably never as well as if they had been correctly conceived from the start.

<comment><USER12> And the "better" systems don't get a chance to evolve into the right thing, because while you can kludge together a fix for something that's broken, it's much harder to remove an unnecessary or extraneous feature.

That, and the "worse" systems have been on the market for nine months by the time the better systems are available. I try to err on the side of underengineering systems, as long as there are no lives on the line.

<comment><USER13> I think I understand what you're trying to say, it the UNIX/New Jersey approach came much later than the MIT/ITS approach. The conversation happened (if I recall correctly) because the ITS approach required more architecture- specific changes (to revert/restore state), while the later UNIX approach was quickly ported to new architectures. The ITS people were effectively asking "how'd you manage to handle this gnarly thing so quickly, so often.", and the UNIX people's response was "we punted".

</comment></comment></comment><comment><USER14> [http://wiki.c2.com/?DanWeinreb](http://wiki.c2.com/?DanWeinreb) Dan Weinreb was the MIT guy; Bill Joy was the Unix partisan. Before he passed away, Dan's blog had his side of this story; unfortunately, I can't find it now. Fortunately WikiWikiWeb knows a little!

Edit: found it! [https://web.archive.org/web/20121107034606/http://danweinreb...](https://web.archive.org/web/20121107034606/http://danweinreb.org/blog/the- worse-is-better-idea-and-the-future-of-lisp)

<comment><USER15> I'm confused why the guy from Berkley (in CA) originally from Michigan is suddenly referred to as "the New Jersey guy".

<comment><USER16> The essay being quoted compares two design philosophies, the MIT approach:

 _> I will call the use of this philosophy of design the ``MIT approach.'' Common Lisp (with CLOS) and Scheme represent the MIT approach to design and implementation._

and the New Jersey approach:

 _> Early Unix and C are examples of the use of this school of design, and I will call the use of this design strategy the ``New Jersey approach.''_

(Unix came out of Bell Labs, which is in Murray Hill, New Jersey.)

When the anonymized Bill Joy is introduced, it is written that he is _" from Berkeley (but working on Unix)"_. That parenthetical is important because it establishes him as working under the New Jersey approach (and possibly working with people in New Jersey), and therefore as being a "New Jersey guy", despite not physically being located there.

</comment></comment></comment><comment><USER17> "worse is better" seems to be born out again and again and again. In fact it's fairly hard to find cases where "do the right thing" won. Here's some examples where a clearly flawed technology became totally dominant.

      * Unix
      * The C language
      * DOS and Windows
      * Javascript
      * The x86 instruction set.

And here are some cases where dominance was achieved in a more narrow market, or in which one might argue about which technology was "worse", or where its not totally clear which was the winner.

      * Linux/SysV vs any of the BSDs
      * Ethernet vs ATM, SONET, etc.
      * Android vs iOS
      * MySQL vs Postgres (Postgres is having a renaissance, but for a long time mysql was very dominant)

What is also especially striking is how often "The Right Thing" in programming languages seems to lose, whereby "lose" I mean "have a very small fraction of market share despite almost everyone agreeing that the language is better in almost every way to more widely-used languages like C/C++, Java, etc."

      * LISP and Scheme
      * The whole ML family
      * Haskell
      * Rust (although its soon to tell on this one)

Meanwhile, languages that aren't obviously terrible, but have major flaws, like Java and Python, are ubiquitous, and the dumpster-fire languages Perl and PHP have enjoyed massive success.

<comment><USER10> My impression is that there are various, even contradicting understandings and opinions in the wild what "worse is better" actually means. Rarely do people who interpret it in a cynical way demonstrate a good understanding of the total set of qualities that matter for the success of a technology.

If you think LISP or Haskell are without flaws (despite all the years of hype), then... you're wrong.

<comment><USER18> It's not that they are without flaws, it's that it is a lot easier to fix/ignore/work-around the flaws than in other languages, where the flaws are unchangeable (except by the standards committee) and constantly in your face.

<comment><USER3> No. You are just used to your flaws, and not used to theirs; and have learned not to complain about yours, but complaints about theirs are welcomed in your circle.

In other words: other systems really are crap, but so is yours, really, but we all get our stuff done, despite it all. Sometimes things get better (Moore's law has forgiven all manner of evil) and sometimes they get worse (Java).

<comment><USER18> > You are just used to your flaws, and not used to theirs

This is a testable theory: name me a flaw in Common Lisp that cannot easily be worked around. I'll bet that for any flaw you name, I'll be able to show you an easy way to work around it.

<comment><USER10> Of course you will be able to give a brittle, half-assed work around for any issue that someone throws at you. But you know, not all people are obsessed with LISP, not all people are ok with half-assed workarounds.

I think you are being very dishonest to yourself if you think that LISP is universally the best just because you can theoretically turn it into anything. You can't, really. Good tooling matters. Error messages matter. Standardization matters. Syntax matters (to some extent). Mindshare matters (a lot). There's a lot of subtle qualities that you can't just fake.

When I write performance sensitive code I'll personally just pick C. I'm sure there's a Common Lisp solution available to specify machine types and compile efficient code. (Probably there's many, which is another problem). But what's the point in putting lipstick on a pig when I can just use the real deal that works perfectly for what I'm doing?

<comment><USER18> "Half-assed" is in the eye of the beholder. If you appoint yourself as the ultimate arbiter of half-assedness then your position becomes unfalsifiable.

But I think you might be surprised at how un-half-assed these things can be.

<comment><USER10> > But I think you might be surprised at how un-half-assed these things can be.

You might be surprised how little I care, because I'll just use a solid, battle-tested tool instead.

Are you suggesting we should rewrite compression or imaging or ... software in Common Lisp because it will be so much better?

<comment><USER18> > You might be surprised how little I care

You obviously care enough to have taken the initiative to engage me on this.

> Are you suggesting we should rewrite compression or imaging or ... software > in Common Lisp because it will be so much better?

No. It's generally never worthwhile to _re-_ write _anything_. But one should not fall prey to the sunk cost fallacy either.

<comment><USER10> If you think we should not "fall prey to the sunk cost fallacy" and write future software in Common LISP, then I recommend you to start working on improving the Common LISP systems-level coding situation. Because I'm not willing to replace this

       unsigned long update_crc(unsigned long crc, unsigned char *buf,
                                int len)
       {
         unsigned long c = crc;
         for (int n = 0; n < len; n++)
           c = crc_table[(c ^ buf[n]) & 0xff] ^ (c >> 8);
         return c;
       }

with something like this ([https://github.com/pmai/Deflate/blob/master/deflate.lisp](https://github.com/pmai/Deflate/blob/master/deflate.lisp))

        #-lispworks
        (defun update-crc32-checksum (crc buffer end)
          (declare (type (unsigned-byte 32) crc)
                   (type (simple-array (unsigned-byte 8) (*)) buffer)
                   (type fixnum end)
                   (optimize (speed 3) (debug 0) (space 0) (safety 0))
                   #+sbcl (sb-ext:muffle-conditions sb-ext:compiler-note))
          (let ((table (load-time-value (generate-crc32-table)))
                (cur (logxor crc #xffffffff)))
            (declare (type (simple-array (unsigned-byte 32) (256)) table)
                     (type (unsigned-byte 32) cur))
            (dotimes (i end)
              (declare (type fixnum i))
              (let ((index (logand #xff (logxor cur (aref buffer i)))))
                (declare (type (unsigned-byte 8) index))
                (setq cur (logxor (aref table index) (ash cur -8)))))
            (logxor cur #xffffffff)))

        #+lispworks
        (defun update-crc32-checksum (crc buffer end)
          (declare (type (unsigned-byte 32) crc)
                   (type (simple-array (unsigned-byte 8) (*)) buffer)
                   (type fixnum end)
                   (optimize (speed 3) (debug 0) (space 0) (safety 0) (float 0)))
          (let ((table (load-time-value (generate-crc32-table)))
                (cur (sys:int32-lognot (sys:integer-to-int32
                                            (dpb (ldb (byte 32 0) crc) (byte 32 0)
                                             (if (logbitp 31 crc) -1 0))))))
            (declare (type (sys:simple-int32-vector 256) table)
                     (type sys:int32 cur))
            (dotimes (i end)
              (declare (type fixnum i))
              (let ((index (sys:int32-to-integer
                            (sys:int32-logand #xff (sys:int32-logxor cur (aref buffer i))))))
                (declare (type fixnum index))
                (setq cur (sys:int32-logxor (sys:int32-aref table index)
                                            (sys:int32-logand #x00ffffff
                                                              (sys:int32>> cur 8))))))
            (ldb (byte 32 0) (sys:int32-to-integer (sys:int32-lognot cur)))))

Have a good day.

<comment><USER18> That is not exactly an apples-to-apples comparison. You've taken a very simple C function and compared it to Lisp code written in two different ways to take advantage of features available only in specific compilers. Equivalent C code would be chock-full of #ifdefs and look just as ugly. I could produce an equally contrived example where to do the same thing as two lines of Lisp would require pages and pages of C.

Second, these code snippets are not comparable in their functionality. The Lisp code generates the crc table, the C code assumes it has already been done. If you wanted to add bounds checking to the C code you would have to make major changes. If you wanted to add bounds checking to the Lisp code, all you would have to do is change the optimization settings. (This is one of the reasons that today's computing world is a swiss-cheese of security holes.)

Third, there are any number of ways to make the Lisp code look significantly prettier. Comparable code would look something like:

        (defun crc (crc l)
          (dolist (item l)
            (setf crc (logxor (aref crc-table (logand 255 (logxor crc item))) (ash c 8))))
          crc)

It's even easy to embed an infix parser [1] in Lisp so that you could write:

        (defun crc (crc l)
          (dolist (item l)
            (setf crc infix(crc_table[(crc ^ item) & #xff] ^ (crc >> 8)))
          crc)

if you wanted do. You could also embed all this in a little DSL so that the resulting semantics and even the generated code would be exactly the same as the C code.

Fourth, most Lisp systems have a foreign function interface, so if you want to you can call C functions directly and so get all the benefits of C (such as they are) from within Lisp.

\---

[1] [http://www.cs.cmu.edu/Groups/AI/util/lang/lisp/code/syntax/p...](http://www.cs.cmu.edu/Groups/AI/util/lang/lisp/code/syntax/parcil/parcil.cl)

<comment><USER10> > The Lisp code generates the crc table

I don't think it's generating a table, just calling generate-crc32-table. Let's remove that line and the difference is still ridiculous (even if you look at only one of the two implementations).

> Comparable code would look something like:

Can't see any machine types here.

> (setf crc infix(crc_table[(crc ^ item) & #xff] ^ (crc >> 8)))

You really don't see how ridiculous that is? You are chasing exactly the target that is unholy in your eyes. Except, you will never reach it (the lisp version is really ugly, probably has really bad tooling, poor error messages, poor standardization and mindshare, and so on)...). While you could just use the right tool for the job.

> and compared it to Lisp code written in two different ways

question is, WHY was the Lisp code written in two different ways, but not the C code? Go figure.

> If you wanted to add bounds checking to the C code you would have to make > major changes.

If you want automated and _perfect_ bounds checking, you need a few more flavours of GC/OOP (because you need some automated notion where the "length" field is, and that length field needs to be authoritative), and you will end up with less modular, less portable code (because you can't just pass sub- arrays / smaller lengths, but always need to pass object handles. Can't support sub-allocators, etc). It's a tradeoff. If you go for automated bounds checking, that's a total different game. Personally I don't like to do GC, and I like to manually insert explicit "bound" checks at strategic locations. But YMMV.

<comment><USER18> > I don't think it's generating a table, just calling generate-crc32-table.

Well, yeah. What do you suppose a function called generate-crc32-table could plausibly do?

> Let's remove that line and the difference is still ridiculous (even if you > look at only one of the two implementations).

OK, lets' do that experiment:

        (defun update-crc32-checksum (crc buffer end)
          (declare (type (unsigned-byte 32) crc)
                   (type (simple-array (unsigned-byte 8) (*)) buffer)
                   (type fixnum end)
                   (optimize (speed 3) (debug 0) (space 0) (safety 0)))
          (let ((cur (logxor crc #xffffffff)))
            (declare (type (simple-array (unsigned-byte 32) (256)) table)
                     (type (unsigned-byte 32) cur))
            (dotimes (i end)
              (declare (type fixnum i))
              (let ((index (logand #xff (logxor cur (aref buffer i)))))
                (declare (type (unsigned-byte 8) index))
                (setq cur (logxor (aref table index) (ash cur -8)))))
            (logxor cur #xffffffff)))

That's 14 LOC, and 8 of those are declarations. For an example that you cherry-picked to be a perfect match for the kind of task that C was designed to do.

> You really don't see how ridiculous that is? You are chasing exactly the > target that is unholy in your eyes.

You are completely missing the point, which was:

> It's not that they are without flaws, it's that it is a lot easier to > fix/ignore/work-around the flaws than in other languages, where the flaws > are unchangeable (except by the standards committee) and constantly in your > face.

And specifically, you were responding to this:

> I'll bet that for any flaw you name, I'll be able to show you an easy way to > work around it.

You haven't actually done that, you've just exhibited some nice clean C code and some ugly CL code. I was just showing you some of the possible ways that the Lisp code could be improved. I threw the infix in there not because I was advocating it, just to show you that it was possible.

So what exactly is it about that code that makes it "ridiculous" in your eyes? Whatever it is, I'll bet I can fix it with very little effort.

For example:

> Can't see any machine types here.

That's right, I left them out (because you can do that in CL if you want). Is that what you want to see, a cleaner syntax for type declarations? Because that's trivial, a few DEFTYPEs and a two-line macro.

> WHY was the Lisp code written in two different ways, but not the C code?

Because the Lisp code was trying to leverage system-specific optimizations while remaining portable. Do you seriously doubt that I could find an example of ugly #idef-laden C code that does the same?

> If you want automated and perfect bounds checking, you need [a bunch of > stuff]

That's right. And why is it that you don't see that in C? Because adding that to C is _hard_. Really really hard. So hard that in 47 years it still hasn't been done in any standardized way. Yes, it's true that CL is not a particularly good systems programming language out of the box. But turning Lisp into a good systems programming language (if that's what you want) is relatively easy, almost an elementary exercise. And it can be done at the user level. No need to wait for a language design committee.

You want to know what is ridiculous? A language that after 47 years still doesn't have a linked list as part of its standard library because the language design makes it impossible.

<comment><USER19> C doesn't have linked lists in its standard library probably because there is a myriad of ways of doing linked lists with different performance, stylistic and other trade-offs.

Should it be macro-based, like that <sys/queue.h> thing from BSD? Or just pure declarations?

Many C programs that maintain lists favor the "intrusive container" approach: mixing in the link node into their own payload data structure and then using that structure itself as a list node.

C++ certainly provides lists. C++ is opinionated: they pick a representation (or at least API) and stick it in.

<comment><USER18> > C doesn't have linked lists in its standard library probably because there > is a myriad of ways of doing linked lists with different performance, > stylistic and other trade-offs.

No, that's not why. The fact that there are myriad ways to implement linked lists is a fact independent of any programming language. And yet Lisp somehow manages to offer linked lists as a native part of the language.

The reason C cannot be extended in this way is that its memory model precludes it. You can't add GC to C because of the way that pointers work.

> C++ certainly provides lists.

Yes. Many languages do. Nonetheless, my original claim, that it is easier to work around Lisp's limitations than those of other languages, stands.

<comment><USER19> C's memory model also precludes decent string handling also, yet it has a string library. But that's it; the only other data structuring provided is the _qsort_ function for sorting arrays and _bsearch_ for searching them. I think it's deliberate. It looks like ISO C tends to avoid specifying new library features that can (and likely would be) be written entirely in C.

<comment><USER18> > it has a string library

But not a decent one :-)

<comment><USER10> Making the point. C is not a language for convenient canned solutions. But I'm not too sad about that. There are a myriad ways of doing strings, and most high-level programming languages are still stuck in the 90s with UCS-2 encodings or such, and offering only solutions that don't scale well for, say, dozens of megabytes of strings. C isn't stuck, at the cost of convenience.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER20> A decent number of theoretically interruptible system calls don't actually get interrupted (or automatically retry) regardless of the signal settings because no one bothers to check for it and the calls are cheap.

Honestly only IO really has any excuse for supporting this sort of behavior.

</comment><comment><USER21> In general I think that programs using the POSIX API to deal with system calls, threads, and signals is unnecessarily hard to do _correctly_. Especially if you want good performance.

I'd welcome some completely new operating system API that guides programmers towards correct and efficient solutions, rather than leaving numerous pitfalls that may only be discovered after the software is put into production.

</comment><comment><USER22> This discussion seems to ignore the fact that being blocked in a "long-running system call" is the normal state for many (most?) Unix services.

If you look at `ps ax` on your system, you'll likely see about a hundred processes. But if you look at `top`, you'll see only a handful of processes having non-zero CPU usage. Why? Because most processes are just waiting (in a system call) for something to do. A web server is blocked in a select/poll/epoll() call waiting for a connection. Your shell is blocked in a read() call waiting for you to type something. This is just the normal way that a main loop is implemented on Unix.

When you kill one of these processes, they need a way to break out of their loop and with the EINTR approach, they get a chance to break and exit.

I'm far from convinced that a "majority" of services want to just catch signals and carry on.

<comment><USER2> Correct me if I'm wrong, but I think if you kill a process normally, it will invoke a signal handler which will exit the process (maybe writing a core file first) without ever returning to normal program flow.

Recognizing EINTR at the program level isn't required for this kind of shutdown. I think the system call will only return if the signal is ignored or if the signal handler returns, but you would only do this if you thought you had recovered from the error.

<comment><USER22> This is true, but if anything it reinforces my point that continuing past a signal is the exception, not the rule.

In the general case services are not at liberty to just exit(), they need to perform some kind of active cleanup action before exit. So the signal handler would set an "exit flag" somewhere and the EINTR would be an indication for the main loop to check this flag before continuing.

The only common case I can think to continue past signals is SIGHUP, which some services interpret as a command to re-read their configuration file. In this case, you are essentially doing a shutdown and startup sequence anyway, only in a possibly more efficient way. E.g. the case of a web server, if you were previously listening on port N there's no reason to believe that the new config file won't ask you to instead listen on port M. So you will be closing down most connections anyway, and catching SIGHUP is mostly an optimisation as exiting and restarting would have a similar effect.

<comment><USER23> For continuing past signals, what about SIGCHLD? I don't think anybody wants that to kill their process.

Most programs are also going to want SIGSTOP, SIGTSTP, SIGTTIN, SIGTTOU, and SIGCONT to be handled transparently with the process continuing on as if nothing happened. I've actually been bitten by this. I had written a program that worked fine on Linux. While it was waiting for input on stdin you could suspend it with Ctrl-Z and then resume it with `fg`. When my friend tried this on his machine (a Mac, IIRC), it would suspend fine. Then resuming it would cause it to immediately die with "Interrupted system call". I hadn't written any code to handle EINTR. I'd been getting away with it because something in glibc and/or the kernel were helping me out and hiding this detail for me.

After a SIGWINCH or a SIGINFO you probably want to continue on too. If you've registered a handler for SIGUSR1 and/or SIGUSR2 you usually don't shut down after receiving them.

SIGPIPE is arguable. If you've decided to handle broken pipes by looking for errno == EPIPE then you likely want SIGPIPE to be ignored entirely, as if it had never happened. But you have to opt into this behavior by explicitly setting SIGPIPE to be ignored. The default behavior is to terminate your process.

For SIGALRM you probably do want a system call to be interrupted and return EINTR. But that usually doesn't indicate that your program will quit but that you wanted a timeout on some operation and that you intend to continue on with other work in the case that it didn't complete quickly.

The issue here is that signals have a bunch of different behaviors shoehorned into them. There are things like SIGSEGV and SIGBUS that need to be handled synchronously because they related to the current instruction. There are things like SIGINT, SIGINFO, and SIGTSTP that come from user action on the terminal. There are things like SIGCHLD that are generated by the kernel in response to events on the system. And there are things like SIGTERM and SIGUSR1 that are generated by a different process calling kill(2). Trying to make blanket statements about all of them is tricky.

</comment></comment></comment></comment><comment><USER24> That you can interrupt long running system calls with a signal provides a way to add a timeout to system calls that may (or probably will) block, by scheduling a timer signal with alarm or setitimer in advance of the possibly time consuming system call. Which is more clear is certainly up for debate, of course. Additionally, that break sends an interrupt signal also allows the user to bail out of long running system calls and at least potentially get back a responsive system.

<comment><USER25> No, you can't, because that strategy has a race codition.

<comment><USER26> You should really explain what that race condition is.

<comment><USER27> I think they are referring to the fact that the signal could be triggered before the system call is executed, in which case you will miss the signal and the system call will never time out. For example:

      alarm(1);
      // alarm could theoretically be triggered here before read() syscall is executed
      read(...);

This sort of problem is what the self pipe trick [https://cr.yp.to/docs/selfpipe.html](https://cr.yp.to/docs/selfpipe.html) is made to solve. I believe it is not really necessary anymore because there are functions like pselect() which solve this by accepting a sigmask parameter.

Anyway, most system calls that one would want to have a timeout for have alternatives that accept a timeout parameter.

</comment></comment></comment></comment><comment><USER28> It's sad, but worse is better is so pervasive, that even many of you very smart people have difficulty conceiving of what the problem is, much less what would be a better way, not because it's particularly hard, but because of technology culture. It seems trivial. Just return an error, right?

I've had to deal with the problems of EINTR for so long. SA_RESTART seems nice, but you can't rely on it, and it only begins to scratch the surface of what would actually be "better".

How about you can call anything in a signal handler?

How about any code, user or kernel, be safely re-entered?

Of course you can't magically solve every concurrency and resource contention problem, but having functional style, and things like unwind-protect, language support for safety, etc. goes a surprisingly long way.

How about you can have arbitrarily many signal handlers for any condition and have arbitrarily many places to continue on the stack, which can receive arbitrary data?

How about I can call a system call from inside a signal handler, from inside the debugger, from inside a signal handler, inside a system call, and pop up a UI to ask the user what they want?

How about every piece of code can be moved around and called from anywhere?

How about you can pass around an objects that can resume a function or a system call in the middle?

How about the thing reading trashy network packets doesn't have absolute power, while my user process doesn't have permission to close the lid or play audio?

Not just Lisp code, but a large fraction of modern languages that have things like GC and bignums, have to make sure it's safe to do anything, even in the middle of adding a number. So why can't we have these things that have been well known for over 40 years? But I'm sure you're familiar with the relatively small set of crews that designed most the stuff we use. vmlinux, kernel32.dll, and even shabby xnu, might be cute, like the jerq is a cute joke on the perq, if we didn't have to actually rely on them for real stuff.

Unfortunately, it's not hard to see that the worse-is-better culture is a result of larger human culture, which is pushed around by the same typical things. But the good thing is that culture being intangible, it only takes the mostly mental acts of skepticism, education, and personal communication, to change it.</comment>


Brunsli: Practical JPEG repacker (now part of JPEG XL) - <AUTHOR> https://github.com/google/brunsli <comment><USER2> Preemptive disclaimer: I don't want to belittle the work the authors did here in any way, and are actually excited especially about the reversible, lossless jpeg<->brunsli coding and that google's buy in and network effects will most likely mean this comes to a viewer/editing software near you in the not so distant future (unlike lepton, which never got out of it's tiny niche).

But seeing the 22% improvement figure reminded me that the typical JPEG file on the internet is rather unoptimized even on write-once-read-many services like imgur or i.reddit.com which transform files (stripping meta data etc) and do not preserve the original files. Just using the regular vanilla libjpeg encoder you can usually save 5%-10% just by lossless recoding of the coeffs and the better-yet-more-computionally-intense mozjpeg coder can even get you a bit further than that.

Then again, the imgur single image view page (desktop) I just randomly opened by randomly clicking an item on their homepage transfers 2.9MiB of data with ads blocked (3.9MiB deflated), 385KiB of which was the actual image, and that image can be lossless recoded by mozjpeg to 361KiB (24KiB difference, a 6.2% reduction), so the 24KiB (0.8%) reduction out of 2.9MiB of cruft hardly matters to them I suppose and may be cheaper in bandwidth and storage cost to them than the compute cost (and humans writing and maintaining the code).

Using brunsli, that same 385KiB imgur file went down to 307KiB so roughly a 20% reduction, but still only 2.6% reduction of that massive 2.9MiB the imgur site transferred in total.

<comment><USER3> I get what your saying but when your storing millions of images and transferring them frequently those small reductions in size are very significant when you get your storage and/or transit bill. A.lot of the fluff and filler you mentioned is cached, so your not bringing it down every request. Even if it wasn't you can assume it's deemed becc

<comment><USER4> I think that's his point - you can _already_ reduce JPEG size a bit by using fancier libraries, but even huge sites like Imgur apparently don't bother.

<comment><USER5> Maybe Imgur doesn't bother, I don't know. I work for Cloudinary, and we do bother. A lot.

</comment></comment></comment></comment><comment><USER6> Sorry, can someone here clarify --

Does this reduce the size of JPEG files, maintaining JPEG format?

Or is this about repacking JPEG files into JPEG XL files (while reducing size), while maintaining the ability to losslessly revert to JPEG?

The page never explicitly states what format the "repacking" is _into_ , and it has so many references to JPEG XL that it no longer seems obvious that it's into just JPEG?

<comment><USER2> It does not maintain the JPEG format, tho it will be part of the JPEG XL format.

It is repacking/recoding jpegs in a lossless and reversible manner, so that clients supporting brunsli can be served directly with the optimized version (their apache and nginx modules seem to serve files with a image/x-j mime), and clients without support can be served with the original jpeg file (or served with a brunsli file and decoded with a wasm-brunsli->native jpeg decoder pipeline if wasm is an option), while you only have to keep the brunsli (or original jpeg) file around.

Since JPEG XL isn't finished yet, there still might be minor differences in the end that make the current brunsli format incompatible to the JPEG XL format, so I wouldn't mass-convert your files just yet.

</comment></comment><comment><USER7> This could be what provides the 'activation energy' to get large Web sites working with JXL; it avoids the situation where you don't want to touch a new standard until everyone else switches too. If Google ships a decoder in Chromium, I would give JXL decent odds of getting a lasting niche.

Also, 20% off JPEG1 sizes is better than it may sound; you only save about 30% more by switching to any of the new codecs (JXL's VarDCT or the video-codec- based ones) that apply a ton of new tools. Given JPEG1 was published 1992, that just confirms to me Tim Terriberry's quip that it was "alien technology from the future."

Working against JPEG XL: it's four modern image formats in one (VarDCT, Brunsli, FUIF, and a separate lossless codec) so a ton of work to independently implement. Also, video codecs _already_ have ecosystem support, and have or will probably get hardware support first.

Further out: "JPEG restoration" is something I mostly see experimental work about but could also take the old format a bit further. The idea is to use priors about what real images should look like to make an educated guess at the information that was quantized away, so you get less of the characteristic blocking and ringing from overcompresed images.

(For example, look at Knusperli, JXL's "edge-preserving filter", or AV1's CDEF. The "quantization constraint" supported by the first two of those is, to me, what makes it "restoration" and not just another loop filter: it can always return pixels that could have compressed to the exact DCT coefficients in the file.)

<comment><USER8> The separate lossless codec and FUIF and some additional ideas have been integrated into the modular mode. The modular mode and VarDCT can be mixed in the same image.

JPEG XL decoder no longer performs 'quantization constraint', it is a classic loop filter, just with heuristics and control fields to maintain detail much better than loop filters in video codecs.

If you start with sharp high quality originals that don't have yuv420 in them, you usually will see 65 % savings with JPEG XL, i.e. more than 30 % on top of brunsli. You can think of VarDCT-mode as guetzli(-35 %) + brunsli(-22 %) + format specific changes(-30 %) like variable sized dcts, better colorspace, adaptive quantization and loop filtering.

<comment><USER7> Thanks, I appreciate the corrections and detail.

I was mostly going by the committee draft on the arXiv ([https://arxiv.org/abs/1908.03565](https://arxiv.org/abs/1908.03565) ), which still mentioned the quantization constraint (J.3) and a "mathematically lossless" mode (annex O) distinct from modular. "Four formats" was just imprecise wording on my part.

The material already public about JPEG XL's design rationale, testing, and so on has been really fun to follow as an outsider. I selfishly hope there's more of it as the work on it and rollout continue. Even with the standard itself public, a lot can be mysterious to those not immersed in this stuff.

<comment><USER9> Yes, quantization constraint was removed post-CD and Annex O was recently merged into the modular mode to simplify the standard. JXL's page count is 1/2 to 1/3 of contemporary video codecs.

<comment><USER7> Thanks. If there are places the public can follow progress post-CD, I'm curious. (I see [https://gitlab.com/wg1/jpeg-xl/](https://gitlab.com/wg1/jpeg- xl/) was updated late Feb, yay.)

<comment><USER9> :) There are updates every ~3 months at [https://jpeg.org/news.html](https://jpeg.org/news.html).

</comment></comment></comment></comment></comment></comment><comment><USER10> Does anyone know how this compares to other projects such as Lepton?

[https://github.com/dropbox/lepton](https://github.com/dropbox/lepton)

The goals and results appear similar. Is the primary difference that brunsli will likely actually be part of a standard (JPEG XL)?

<comment><USER11> And how does it compare against HEIF?

<comment> 0-_-0 HEIF can't compress JPEG images losslessly

<comment><USER2> Indeed, HEIF/HEIC is basically a slightly dumbed down HEVC (h.265) i-frame (full frame) (HEIC)[1] and new container format (HEIF)[2], similar to WEBP being VP8 i-frames in a RIFF container. So they are used as full blown codecs in practice, usually not in a lossless mode, so shifting JPEG to HEIC or WEBP will lose some quality.

[1] Decoding HEIC in Windows (10) requires you to have installed a compatible HEVC decoder. Which is 99 cents (and the hassle of setting up a store accounts and payment processing with MS) or an alternative free one which will use the HEVC codec that is shipped with hardware such as newer Intels (QSV) or GPUs. Thank you patent mess!

[2] HEIF the container format can contain JPEG data, but in practice does not or only as a supplementary image (previews, pre-renders, thumbnails, etc)

</comment></comment></comment></comment><comment><USER12> If someone is wondering where the name comes from: [https://www.saveur.com/article/Recipes/Basler-Brunsli- Chocol...](https://www.saveur.com/article/Recipes/Basler-Brunsli-Chocolate- Almond-Spice-Cookies/)

Not sure if it is the best recipe - the ones I use are usually written in German.

<comment><USER13> Brotli, another compression algorithm created by Google is also named after a Swiss baked good: [https://en.wikipedia.org/wiki/Spanisch_Br%C3%B6tli](https://en.wikipedia.org/wiki/Spanisch_Br%C3%B6tli)

<comment><USER14> Zopfli also comes from a Swiss bakery product. Developers must like carbohydrates :)

<comment> spider-mario And let's not forget Gipfeli and Knusperli ;)

[https://github.com/google/gipfeli](https://github.com/google/gipfeli)

[https://github.com/google/knusperli](https://github.com/google/knusperli)

<comment><USER15> How about grittibanzli?

[https://github.com/google/grittibanzli](https://github.com/google/grittibanzli)

</comment></comment></comment></comment></comment><comment><USER16> Huh, I always just used good ol' jpegtran:

      jpegtran -copy none -optimize -progressive -outfile "$image" "$image"

I have a wrapper script around this to allow bulk optimization as well as calculating stats.

Time to switch I guess.

<comment><USER17> You should use mozjpeg instead. It's a jpegtran fork and drop in replacement which optimizes the Huffman table even better.

<comment><USER5> Does it really optimize the Huffman trees better? I wouldn't think there's more you can do to optimize Huffman trees than make them reflect the actual histogram.

Mozjpeg does try different progressive scan scripts – different scripts lead to different compression behavior. That helps a bit, though I don't like it when it separates the Cb and Cr planes, becauses that leads to "flash of weird color" when decoding progressively. See also: [https://cloudinary.com/blog/progressive_jpegs_and_green_mart...](https://cloudinary.com/blog/progressive_jpegs_and_green_martians)

</comment></comment></comment><comment><USER18> I like the check list for features... can't wait for

"Nginx transcoding module"

</comment><comment><USER3> I understand what your saying but when your storing millions of images and transferring them frequently those small reductions in size are very significant when you get your storage and/or transit bill at end of month. A lot of the fluff and filler you mentioned is compressed and cached, so your not bringing it down every request and not in raw form. Even if you are sending a huge amount of uncachable stuff with each request it doesn't mean savings wouldn't be appreciated. It's funds for a team lunch if nothing else!

</comment><comment><USER19> Could someone explain what a "repacker" is?

What I know: In Photoshop, when I save an image as JPEG, I can decide the "quality" (Low, Medium, High, etc). The lower it is, the smaller the file size but the image will have (more) artifacts. The resulting image can then be opened in any image viewer including browsers.

Also, I was told to save the "master" copy in a lossless format (e.g. TIFF or PNG) because JPEG is a lossy format (like MP3 to WAV).

So how does a "repacker" come to play?

<comment><USER20> JPEG compression works in two phases:

1\. Discrete Cosine Transform, discarding insignificant information.

2\. Compression of bitstream.

Step 1 is lossy in practice because it throws away things that seem insignificant. The "quality" control determines what counts as insignificant. Step 2 is lossless, and just tries to make the data coming out of Step 1 take up as little space as possible.

A repacker redoes Step 2 better: it makes the file smaller without reducing the quality, by changing how the data is compressed, not changing which parts are kept.

</comment></comment><comment><USER21> > Brunsli has been specified as the lossless JPEG transport layer in the > Committee Draft of JPEG XL Image Coding System and is ready to power faster > and more economical transfer and storage of photographs.

I thought the lossless part of JPEG XL was done by FUIF, am I misunderstanding something?

<comment><USER9> Lossless transcoding of JPEG bitstreams is done by Brunsli, there is also lossless storage of pixels, based on tech from FUIF plus an adaptive predictor by Alex Rhatushnyak.

<comment><USER8> Plus the 'select' predictor from WebP lossless :-)

<comment><USER5> Plus tons of cross-pollination from the PIK/VarDCT mode, e.g. the (MA)ANS and Brotli entropy coding and the Patches intra/inter block copy mechanism :)</comment></comment></comment></comment>


PCILeech - <AUTHOR> https://github.com/ufrisk/pcileech/ <comment><USER2> Two related links previously on HN (no comments):

 _Introducing the Memory Process File System for PCILeech_ [http://blog.frizk.net/2018/03/memory-process-file- system.htm...](http://blog.frizk.net/2018/03/memory-process-file-system.html)

 _Using your BMC as a DMA device: Plugging PCILeech to HPE ILO 4_ [https://www.synacktiv.com/posts/exploit/using-your-bmc- as-a-...](https://www.synacktiv.com/posts/exploit/using-your-bmc-as-a-dma- device-plugging-pcileech-to-hpe-ilo-4.html)

<comment><USER3> And from the other side of things, securing the host OS from devices, I found this article a few years ago interesting:

 _Fuzzing PCI express: security in plaintext (2017)_ [https://cloud.google.com/blog/products/gcp/fuzzing-pci- expre...](https://cloud.google.com/blog/products/gcp/fuzzing-pci-express- security-in-plaintext)

</comment></comment><comment><USER4> I've had intel_iommu=on in my boot cmdline on my personal machines (ones I do my banking on!) for 5 years without issue

(board+CPU for all machines both support it, dmesg confirms it is indeed on)

<comment><USER5> From the article:

> _Does not work if the OS uses the IOMMU /VT-d. This is the default on macOS > (unless disabled in recovery mode). Windows 10 with Virtualization based > security features enabled does not work fully - this is however not the > default setting in Windows 10 or Linux._

For PCI (shared bus) the device could just spoof packets, but for PCIe, there are dedicated lanes for each device. I wonder if a physical MITM device is practical.

</comment></comment><comment><USER6> Roughly 10 years ago, I knew a guy who sold various exfil/border-extraction products to state actors including FireWire dongles similar to Inception (they may even be effectively the same thing, IDK). He specialized in memory dumping attacks on several platforms including Windows, FreeBSD, Linux and macOS.

The core problem is that buses should be authenticated, authorized, encrypted, and selective-/least-privileged channels. Exposing a memory and expansion bus to the outside world in the name of "convenience" is insane. A trusted set of components in the OS and in hardware should:

0\. Be able to do a HwIDS checksumming of all firmwares to detect tampering.

1\. Limit devices' ability to connect unless they are authorized by the user, much like a "hardware firewall" UI.. vaguely similar to say VMware Workstation/Fusion's dialog when plugging in a new USB device mixed with something like Little Snitch'es dialog for a process wanting to connect to a particular port.

2\. Authenticate devices with public/private key certs that are burned in, a function where the device can answer challenge requests, and Signal protocol- like construction properly modified for PKI. Then, and only then, can a host talk securely to a device over an encrypted channel.

<comment><USER7> We never sold anything for it, but for some demos back around 2004/2005 we flashed some firewire iPods with custom firmware that performed the attack described here:

[https://web.archive.org/web/20071011191205/http://md.hudora....](https://web.archive.org/web/20071011191205/http://md.hudora.de/presentations/firewire/PacSec2004.pdf)

(To be clear, those slides aren't mine, but I can no longer find the firmware we based ours off of, and never did get permission to post source to our mods, which were never distributed.)

I know a couple of our customers took entirely the wrong lesson from our demonstrations and banned mp3 players from their office buildings entirely afterwards :)

<comment><USER6> Hahaha. That's awesome. :)

Speaking of things that look silly dept.:

Many moons ago around 1998 in uni, I had an HP 48 graphing calculator which was both programmable and had a very powerful IR LED that was usually used for serial transfers between similar devices. It so happened that some enterprising soul made a customizable, preset and _code-learning_ IR remote control app for it that worked, so I put the brand of the lecture hall's 4 TVs in it. From the back of the room, some 15 meters away, I subtly turned on all of the TVs with a phreak'n calculator. Disbelief, confusion and hilarity ensued.

<comment><USER7> TV-B-gone before its time.

That's great. I hope I'd have done exactly the same.

</comment></comment></comment></comment><comment><USER8> So an Action Replay for x64

[https://gamehacking.org/wiki/Action_Replay_(Amiga)](https://gamehacking.org/wiki/Action_Replay_\(Amiga\))

<comment><USER9> Also for PC:

[https://www.youtube.com/watch?v=usaioMbE8EQ](https://www.youtube.com/watch?v=usaioMbE8EQ)

</comment></comment><comment><USER10> How widely deployed are IOMMUs these days? I thought they became a standard thing a few years ago.

<comment><USER11> The problem is that, used properly, IOMMUs are horribly expensive.

Consider a NIC driver where you're mapping an outgoing packet for DMA. What used to essentially be a virtual to physical translation becomes a virt to phys + entering the phys in the iommu + removing the mapping when the transmit is complete. This is expensive for hardware and software reasons. At one point I benchmarked a 100g setup on linux, and with the IOMMU enabled, we lost about 90% of the bandwidth and most of the CPU time was spent in lock contention over the red-black tree that managed the IOMMU tables. This was 5-ish years ago, so perhaps things have gotten better.

So that makes people want to just enable the IOMMU for SR-IOV (and full device) pass-thru to VMs. This is cheaper, since you just set the mapping up when you allocate phys mem for the guest, and tear them down when freeing phys mem.

MacOS used to use a really cool trick where they pre-mapped all mbufs into the IOMMU. That made network traffic transmit and receive comparatively fast. However,it also prevented lots of optimizations that modern operating systems use for zero-copy IO (like attaching pages from sendfile directly to mbufs, similer to skb_frags).

<comment><USER12> A problem on the hardware side is that Intel's IOMMU TLB is tiny (64 entries), so using huge pages for all DMA-accessible memory is absolutely required to get a good performance out of it.

We've done some benchmarks here: [https://www.net.in.tum.de/fileadmin/bibtex/publications/pape...](https://www.net.in.tum.de/fileadmin/bibtex/publications/papers/ixy- writing-user-space-network-drivers.pdf) (Figure 9 on page 10)

Only a very basic benchmark, working on more...

<comment><USER11> Nice paper, and thanks for the reminder of how small the IOMMU tlb is. We never hit this because we were testing full-sized packets (and really bigger, because of TSO) and hit host IOMMU management overheads at ~100K to 200k TSO sends/sec.

<comment><USER12> Interesting, did you use huge pages?

I think ~100k to 200k TSO "packets" per second should be doable with the IOMMU. But I guess it depends where the data is coming from. Could be one of the odd cases where copying data is faster than doing zero-copy, e.g., just copy everything into the same small set of small-ish buffers to keep the number of pages that need to be present in the IOMMU small?

</comment></comment></comment></comment></comment><comment><AUTHOR> Related: [https://github.com/ufrisk/pcileech- fpga](https://github.com/ufrisk/pcileech-fpga)

</comment><comment><USER13> Weren't these used in a famous counterstrike cheating scandal in Norway?

<comment><USER14> Yep, they're also pretty popular with people who cheat in ESEA/FaceIT third- party ladders. One of the guys, ra1f, eventually outed himself [1]. The software-based anticheat is fairly decent, pushing people to hardware-based cheats built off of PCILeech to avoid detection.

[1]: [https://twitter.com/rra1f/status/1067518342595006466](https://twitter.com/rra1f/status/1067518342595006466)

</comment></comment><comment><USER7> Does DMA over PCIe work using USB gadget mode with a Linux device? i.e. could a Pi be used easily and inexpensively to build an acquisition device for this?

Edit: Bleh. Nevermind. I saw this photo:

[https://gist.githubusercontent.com/ufrisk/c5ba7b360335a13bba...](https://gist.githubusercontent.com/ufrisk/c5ba7b360335a13bbac2515e5e7bb9d7/raw/2df37be67047e19ea2c3f73be67a0ba06fea203d/_gh_mbp.jpg)

with a pcie adapter connected over what looked like USB3 and forgot that it's thunderbolt on the macbook. I was not quite to the middle of my first cup of coffee when I asked that.

<comment><USER15> Well USB is not PCIe and doesn't do DMA, so no.

<comment><USER7> Aren't most of the devices on the linked page doing PCIe over USB3?

Edit: Bleh. Nevermind. I saw this photo:

[https://gist.githubusercontent.com/ufrisk/c5ba7b360335a13bba...](https://gist.githubusercontent.com/ufrisk/c5ba7b360335a13bbac2515e5e7bb9d7/raw/2df37be67047e19ea2c3f73be67a0ba06fea203d/_gh_mbp.jpg)

with a pcie adapter connected over what looked like USB3 and forgot that it's thunderbolt on the macbook. I was not quite to the middle of my first cup of coffee when I asked that.

<comment><USER16> USB 4!

</comment></comment></comment></comment><comment><USER17> An interview with Ulf Fritz:

[https://www.youtube.com/watch?v=MIfY8g73xms&feature=emb_rel_...](https://www.youtube.com/watch?v=MIfY8g73xms&feature=emb_rel_end)

<comment><USER18> Another interview with him: [https://www.youtube.com/watch?v=W5Yb3q9iJao](https://www.youtube.com/watch?v=W5Yb3q9iJao)

</comment></comment><comment><USER19> Doesn't this change everything wrt cheating?

<comment><USER20> Not really.

A few of the better anti-cheat products have had detection vectors for these for a while now, from simple things like detecting the driver to outright probing the device.

<comment><USER21> Surely you could combine a cheating chip with a legitimate device like a network card so to the OS it looks just like a network card.</comment></comment></comment>


Carphone Warehouse fined after failures put customer and employee data at risk - <AUTHOR> https://ico.org.uk/action-weve-taken/enforcement/the-carphone-warehouse-ltd/ <comment><USER3> I once got my daughter a phone from CW. As part of the process I needed to give them my mobile number - it was a show stopping requirement. About 3 months later I was getting spammed by an autodialler 3 or 4 times a day. The dialler never connected when answered. Calls were always different numbers, never blocked, but always UK based mobile phones. I phoned back and discovered that the recorded message I was put through was for a Telemarketer. But there was no way to stop them. I was already registered with the telephone preference service (TPS) and so should not be getting unsolicited spam marketing calls.

One day, out of the blue I answered the number and there was someone on the other end of the line. I then went through a difficult conversation where I attempted to get out of them why they were phoning me. I found out, (1) the company was called something ridiculous like the "phone delivery depot for your provider" and they were trying to sell mobile phone contracts by using that name when answering calls "Hi this is the 'hone delivery depot for your provider'", I kid you not. He even claimed to be from my mobile provider at one point (no he wasn't, I'd already checked that when I attempted to get them to block the calls.) (2) the guy on the other end of the phone knew quite a bit about me, so had access to data from a third party. (3) it was a total phishing exercise and they were trying to get fees for reselling the customers new fixed term contracts to them after they had become rolling monthlies.

After laying in to him I finally got the route of the data. "We get a lot of our leads from Carphone Warehouse." BING!! The moral of this story is, always read the fine print. I probably in haste missed a tick box or ticked the wrong box - or the girl serving me "helpfully" did it for me on their computer system without asking. Carphone Warehouse are a scummy company, with bad practices, and best advice is to avoid at all costs!

<comment><USER4> Put yourself on the do-no-call list (if you're in the UK) and never receive a spam call again. And on the very very rare occasion you do, just say "I'm on the do-not-call list by the way" and see how fast they hang up.

[http://www.tpsonline.org.uk/tps/index.html](http://www.tpsonline.org.uk/tps/index.html)

Takes 5 minutes.

<comment><USER5> TPS only applies to 'unsolicited' marketing calls

If you've inadvertently agreed to marketing by overlooking some opt-out section of a contract then that counts as consent and TPS means nothing:

[https://ico.org.uk/for-organisations/guide-to- pecr/electroni...](https://ico.org.uk/for-organisations/guide-to- pecr/electronic-and-telephone-marketing/#consent)

<comment><USER3> The issue is - if you don't understand why the marketers are calling you, it can seem like an unsolicited call. As I mentioned, they never answered the calls till the very last one and that took a good 2 - 3 months. I think they were using an autodialler with PAYG sim cards and refilling it every couple of weeks as I blocked each caller.

The problem was, I needed to answer my phone as I was working away from home at the time. Any of the calls could have been a valid call about my real life. I often got calls from people not in my contact list.. from random clients, my kids' schools, data centres or related to work.

</comment></comment></comment></comment><comment><USER6> Section 13 onwards makes for a great read as to how this all went down. The attacker used a common security scanning tool Nikto, which would have told them that WordPress and its plugins were horrifically out of date. Any tool could have found this (WPScan, Nessus.. a manual check through HTML source), but Nikto was likely identified due to some common defaults such as the User- Agent within the requests etc.

Once they identified the vulnerability to exploit, it seems they used genuine credentials to then place webshells on the server which gave them persistence - a low-level reverse shell if you will, allowing them to traverse the file system, execute commands as if they had genuine access. Next up, a quick grep for plaintext credentials that they can then pivot with. There are still questions around how those credentials were obtained, but they'd don't specifically call them out as being default.

The report then jumps to the person accessing databases (including payment card info.. whoops) - the question here is why did a Wordpress instance have access to such data? I can't imagine it would have needed it, so I suspect that they either had one huge DB server containing the backend for this WordPress instance, plus their customer data & payment info databases/tables. Or, the malicious actor traversed the internal network, using the WordPress as a pivot point.

Last point worth calling out, is that they are unsure how much of the data was actually exfiltrated - "[...] the transaction/payment card information referred to above was located and accessed: it cannot be ascertained whether or not some or all of that information was indeed exported, but that is a very realistic possibility." This is a common problem with breaches, and an area what active monitoring can help. By planting unique details/tokens into the database (users, payment information etc.), you can start scanning for them on the clear and dark web. As and when they appear, you can confirm if the data was exfiltrated (and with many honeypots, pinpoint time of the breach). This is one of the many reasons I put BreachInsider[0] together.

[0][https://breachinsider.com](https://breachinsider.com)

<comment><USER7> possibly the WordPress install did not use a dedicated db user/password - <USER2> possible that the root user and password was used.

<comment><USER6> Good point, the report calls out that the root password for 40+ servers was the same, and that a large number of employees knew or used it. Seems sensible to transfer that same mentality across to other sensitive accounts.

<comment><USER7> Ouch having worked at BT our internal security would have had afield day with that - but Mobile telecoms where always considered much weaker technicaly.

</comment></comment></comment></comment><comment><USER8> Peanuts. The GDPR strategy of fining a percentage of revenue is the only way to make companies take this seriously.

<comment><USER9> I'd be interested to know which company thinks a £400,000 needless expense isn't a problem.

I jump through hoops to get a £7 train ticket paid.

<comment><USER10> The company is doing £10B in sales, do you think they're going to restructure their IT to be more protective of user data because they got a fine that's 0.1% of their £500M gross profit?

If they let a £10 train ticket slide for each employee it would be more than this fine.

</comment></comment></comment><comment><USER11> A thousandth of the top-end of their expected profits for 17-18. That'll teach them.

<comment><USER12> It's not supposed to harm them, it's supposed to make it economically sensible for them (and companies like them) to employ one or two people with the sole responsibility of making sure stuff like this doesn't happen.

<comment><USER13> I don't agree, the ICO is a total joke and hopefully GDPR will allow them/some other body to give proper fines.

There have been cases where spam companies have done 100m SMS/phone calls and got a £100K fine (which they can just prepack away). It's probably a tenth of their telecom cost per call. Just a cost of doing business.

<comment><USER14> That's a different issue, though. In your example, the company is actively doing something malicious, and accepting the fine for getting caught as part of their costs. Other than the threat of a really huge fine (or worse sanctions), they're unlikely to stop.

In this case, it's a company that's not actively trying to be malicious, but rather due to [cost-saving/incompetence] has underfunded IT security to an excessive degree. For them, and others like them, to change their ways it's a lot less self evident that the fine needs to be huge. The fine (and accompanying reputational damage) just needs to be enough to make them, and similar companies, take IT security seriously.

Very different situations, really.

</comment></comment></comment></comment><comment><USER15> The short version is that Carphone Warehouse had a very outdated copy of Wordpress, a database with no encryption of passwords, a generally sloppy infrastructure security-wise, and you can pretty much guess the rest of the story.

While I agree with many others that this is a small fee all things considered, it does put other companies on notice: secure your shit or you'll be held liable (to some degree at least).

<comment><USER10> They shared passwords - how does that happen in a tech company? UK ICO needs more teeth, these sorts of failings should have punishments so severe that the company can't afford to pay dividends or bonuses.

</comment></comment><comment><USER16> Put that into context, fine me £3 for doing something bad...

</comment><comment><USER17> That's a joke. It would take them a lot more to fix than 400k. Just the pentests + risk assessments would cost more and they would also need to educate their personnel (developers, testers, DevOps, managers), and implement new policies and procedures, SDLC etc. The price of all of that would be well over a million.

400k is not even a slap on the wrist. I'm disappointed.

</comment><comment><USER18> I'm in two minds about the actual penalty notice given here.

On the one hand, it's good to see an organisation receiving a real financial penalty after a serious breach, and apparently there were many serious failings in security on the part of Carphone Warehouse.

On the other hand, reading the Commissioner's views in the notice, there is a disturbing amount of commentary about measures not in place at Carphone Warehouse that asserts apparently without evidence that such measures are widely accepted security standards, either acknowledges that these measures may not have made any difference to this kind of attack anyway or implies again apparently without evidence that they would, and then considers the lack of each such measure to be a contravention in its own right.

I'm not sure this is a good thing, because if we took the principles indicated by those views and applied them more generally, I'm not sure many if any organisations would meet the required standards here.

For example, how many organisations that use WordPress to run their web sites have measures in place to detect unauthorised use of legitimately issued WordPress credentials? What would that even look like?

How many small organisations have a dedicated WAF box installed, or the money to run regular independent pentesting or hire suitably qualified in-house staff to do it? (Obviously we're not talking about a small organisation in the case at hand, so maybe the ICO would apply a more liberal standard in other situations.)

There is a reference to transaction data being encrypted but the encryption key being present in plain text in the application source code. Well, OK, but that key has to be present _somewhere_ accessible on the system or you can't access the data at all even for legitimate purposes. If you've got an attacker who apparently already has access to both your database and your runtime system, I'm not sure what alternative they would have preferred to see that would have mitigated the damage materially here. Does it really matter whether the attacker looks at source code or somewhere else like environment strings, if the end result is the same either way?

Compared to obviously terrible practices like not updating externally accessible software for many years so the live version has multiple known vulnerabilities or storing full credit card data including things like CVCs for all historical transactions, some of the issues raised here seem both much more dangerous and much more practically mitigable than others.

<comment><USER19> > For example, how many organisations that use WordPress to run their web > sites have measures in place to detect unauthorised use of legitimately > issued WordPress credentials? What would that even look like?

Install the free Wordfence plugin and you'll get an e-mail whenever someone logs in using an admin account.

<comment><USER10> Or you can block admin use from unauthorised IP addresses (or whitelist if you own the IP).

I limit SSH access to my ISPs net blocks, that's just a hobbyists personal computer on which I have no legal obligation to protect the data.

</comment></comment></comment><comment><USER20> What do people make of ICO's recommendation for production servers to have antivirus software installed? Is this a standard approach to Windows? Is it generally accepted?

Please excuse my ignorance.

</comment><comment><USER21> Carphone Warehouse, a total scam organisation, just like Trustpilot. I wrote about them here:

[https://www.linkedin.com/pulse/day-31-write-every- carphone-w...](https://www.linkedin.com/pulse/day-31-write-every-carphone- warehouse-sucks-why-win-zubair-quraishi/)

</comment><comment><USER22> All of this data was leaked through a WordPress plugin uploaded to an old installation; WordPress is one of the easiest platforms to secure (IMO) so can't understand why organisations can't take the short amount of time needed to secure it.

</comment><comment><USER23> Stewart Lee on the values of Carphone Warehouse: [https://www.youtube.com/watch?v=W2firijxQOo](https://www.youtube.com/watch?v=W2firijxQOo)

</comment><comment><USER24> Wordpress is like a gift that keeps on giving for hackers and blackhats. All it takes is missing one vital upgrade or using an outdated plugin.

<comment><USER19> Do you have a suggestion for a platform where missing a vital upgrade won't introduce security holes? I'd love to know what it is.

<comment><USER24> I have a feeling that if you tally up all the times a server was hacked, Wordpress will be the reason for the bulk of them. Yes many platforms when not kept upgraded will have holes, but nothing in the magnitude of the WordPress universe.</comment></comment></comment>