Creating LFortran, an interactive Fortran compiler built on top of LLVM - bauta-steen https://lfortran.org/blog/2019/04/why-we-created-lfortran/ <comment><AUTHOR> There seems to be a missing question in the FAQ: Why is Fortran worth saving? What are the advantages of writing code in Fortran? Why not just allow the ports to C++ to happen?

There might well be good answers to these; but it seems that if you want to make Fortran cool again, you have to provide a vision for what makes it special.

<comment><USER2> > Why not just allow the ports to C++ to happen?

I see two reasons why C++ is not a suitable language for domain specialists in numerical computing.

1\. Programming in C++ requires a certain discipline to understand what is going on, for example to not accidentally make useless copies of huge arrays. In fortran you do not need to learn such a discipline because everything is explicit.

2\. Fortran natively supports multidimensional numeric arrays. In C++, not really; you need to use "libraries" whose evolution is independent to that of the language and may become unsupported. A few years ago, everybody said to use "blitz", somewhat later it was "eigen", now it probably is something new. The C++ numerical code that you may write today will probably be obsolete in a few decades. Yet the fortran code will be alright. It is "eternal".

<comment><USER3> Eigen is still the best-in-class C++ library for linear algebra as far as I know. It's been around for a while, is still actively developed, and I don't think there's a risk of it becoming obsolete anytime soon.

That being said, the learning curve for someone not familiar with C++ to write code using Eigen is much higher than writing the same code in Fortran. If you're not already a C++ programmer, you just want to do some math, and Numpy/Matlab are too slow or constrained, then Fortran is a solid choice.

<comment><USER2> > That being said, the learning curve for someone not familiar with C++ to > write code using Eigen is much higher than writing the same code in Fortran.

Moreover, the algorithms implemented inside eigen are hidden behind dozens of onion-like layers. Once you peel all these layers you find code such as this:

[https://eigen.tuxfamily.org/dox/JacobiSVD_8h_source.html](https://eigen.tuxfamily.org/dox/JacobiSVD_8h_source.html)

I can read and write C++ and I teach numerical linear algebra for a living, but mother of god, this SVD implementation is horrendously unreadable to me. Most of the lines of code are stupid bureaucracy necessary just to add and multiply a few vectors. The same algorithm in linear algebra textbooks requires about fifteen lines of pseudo-code, and similarly for a fortran implementation.

<comment><USER3> Yeah, libraries that make extensive use of templates are basically dark magic only comprehensible to C++ experts (see also: almost everything in Boost).

The positive side of this is that it enables nice interfaces for the library, and makes a lot of the abstractions basically "free" (since the cost is paid at compile time rather than via pointer indirection at runtime). But it definitely makes those libraries inappropriate for students who want to look at an understandable implementation of the algorithm.

<comment><USER2> > ...makes a lot of the abstractions basically "free"

Maybe it is just me, but I do not really see the need for "abstractions" in a linear algebra library. Numbers are already an abstraction, you do not want to hide them! Ok, maybe you want to choose between "float" and "double", but this does not merit all that overcomplication.

<comment><USER4> The abstractions inside Eigen are there to enable powerful compile time optimisations, such as compiling the addition of three vectors `Eigen::VectorXf x = a + b + c;` in a single loop instead of naively allocating a temporary to evaluate `Eigen::VectorXf tmp = b + c;` then evaluating `Eigen::VectorXf x = a + tmp;`.

This logic is omnipresent in Eigen and is crucial to its performance. It is also used to have specialized versions of some decompositions when the matrix's size is known at compile time.

</comment></comment></comment></comment></comment></comment></comment><comment><USER5> This looks very interesting! I hope that maybe this project can include a "refactoring parser", that is, for each token in the parse tree, store where in the source code file that token originated from. Also, white space and comments have would have to be included, such that the round trip (source file -> AST -> source file) is possible without loss. I've long wanted to write some refactoring tools for modern Fortran, but not having such a parser is a big blocker—there's only so much you can do with regexes.

<comment><USER6> Wouldn't writing an Antlr grammar handle this?

<comment><USER5> Does Antlr support an annotated parse tree that would allow for round- tripping? I didn't see anything in the documentation, at a quick glance. Even then, formulating a grammar (especially one that also catches whitespace and comments) seems like a pretty daunting task.

<comment><USER6> Catching whitespace and comments is no problem in Antlr[1]. I'm not exactly sure what you mean by round-tripping. In terms of how daunting it is, it's much easier than trying to use Flex/Bison or lex/yacc, that's for sure.

1: At the bottom of the grammar, there are rules to intentionally skip over whitespace and comments, but it could have just as easily captured them and parsed them further. [https://github.com/antlr/grammars-v4/blob/master/c/C.g4](https://github.com/antlr/grammars-v4/blob/master/c/C.g4)

<comment><USER5> What I mean by "round-tripping" is that I would have to be able to read a .f90 file into an AST structure, then write out the tree to a new file, and end up with the exact same file content as the original. This would be a prerequisite for a refactoring tool, which has to preserve comments and code formatting.

</comment></comment></comment></comment></comment><comment><USER7> You can try it online: [https://mybinder.org/v2/gl/lfortran%2Fweb%2Flfortran- binder/...](https://mybinder.org/v2/gl/lfortran%2Fweb%2Flfortran- binder/master?filepath=Demo.ipynb) !

</comment><comment><USER8> Using Fortran in Jupyter CNA be extremely handy to learn certain algorithms by interactive exploration. Despite all the nice and modern alternatives, Fortran is so popular in HPC world that still generations of students will have to learn it.

</comment><comment><USER9> This post talks a lot about creating a community. Are they doing that anywhere in particular?

<comment><USER10> I am one of LFortran authors. We just open sourced it recently. Do you have any recommendations if we should create a mailinglist, or rather use something like Discourse? Or both.

Right now if you have any questions, you can open an issue at our GitLab repository:

[https://gitlab.com/lfortran/lfortran](https://gitlab.com/lfortran/lfortran)

</comment></comment><comment><USER11> Question:

What is a good Fortran compiler/implementation these days?

I know they have Intel and IBM and SimplyFortran as commercial software. GFortran works via GCC. Is it as good? Has anyone on here used this LFortran.

<comment><USER12> Gfortran is very nice, but I recommend a recent version (at least ver >= 7) rather than old versions (e.g. ver 4.4, which may still be the default on CentOS 6 etc). For performance, I guess Intel fortran is probably better (but also depends on calculations).

<comment><USER11> Thanks!

</comment></comment></comment><comment><USER13> Not sure what they mean by lack of support for GPU - Intel Fortan has this and I am sure others do.

<comment><USER10> I am one of the LFortran authors.

Can you please point me to some documentation or examples where Intel Fortran is used to offload to, say, NVIDIA GPUs? I don't think it supports it yet.

PGI and IBM support CUDA Fortran, other compilers don't. PGI/Flang has an experimental support for offloading "do concurrent". Some compilers also support the openmp/openacc pragmas and GPU offloading. One can also call CUDA C from any Fortran compiler via iso_c_binding. None of this is ideal.

I think it's still a research problem how to best integrate GPU support in the language itself. CUDA Fortran is nice --- one thing we want to do down the road is to implement CUDA Fortran and to allow LFortran to transform code it understands into a Fortran dialect that all Fortran compilers can compile, so that one can use any Fortran compiler and still use CUDA Fortran.

The other path is to parallelize "do concurrent", but part of this is also means to extend Fortran to allow to specify the array layout, as Kokkos ([https://github.com/kokkos/kokkos](https://github.com/kokkos/kokkos)) allows.

That is one of the biggest problem with Fortran: people want to be assured that their code will run on modern hardware. When using, e.g., C++ and Kokkos, then there is some assurance that the code will run.

<comment><USER14> AFAIK Julia uses patched LLVM's PTX output, which I think should be done by all languages to work towards a common optimization platform. Also CuArrays uses multiple higher level NVIDIA libraries, like CudaBLAS and CuDNN.

The goals look similar to me, so it's worth to take a look at them.

<comment><USER10> Yes, I was planning to start with what NumBa ([http://numba.pydata.org/](http://numba.pydata.org/)) is doing, they also use the LLVM PTX backend.

There is a really promising new project by Chris Lattner (the original author of LLVM) called MLIR: [https://github.com/tensorflow/mlir](https://github.com/tensorflow/mlir). That might be the best intermediate representation that all the compilers (Julia, Fortran, ...) could target.</comment></comment></comment></comment>


When a Phone App Opens Your Apartment Door, but You Just Want a Key - <AUTHOR> https://www.nytimes.com/2019/03/23/nyregion/keyless-apartment-entry-nyc.html <comment><USER2> There are arguments for and against, I think it's good that for it's just in the lobby and the oldies have been given keycards. I don't see a problem with that.

Where I see the big benefit is when someone loses their key and the landlord has to weigh up whether or not they need to install a brand new lock and recut all of the keys. This would be alleviated using an electronic system.</comment>


Nestle's California Water Theft Censored by Reddit - <AUTHOR> http://pontiactribune.com/california-water-nestle-reddit-censors-anti-media/

<comment><USER2> Is "censored" the correct term or were rules actually broken to warrant the removal of the post? Reddit definitely faces a real challenge when subreddits are established with a name like _/ r/news_ but are self-moderated and enforce their own strict rules regarding what qualifies as news. There are tons of cases of rogue moderators but also very fair reasons for posts to be removed.

I appreciate one commenter's response:

 _Nick Bernabe - while I respect your work and what you 're accomplishing here, not being an opinion piece isn't just about citing facts. It's also about maintaining a certain professionalism in the writing. Let the facts be persuasive, don't editorialize with statements like "Because profit, of course. Or perhaps more befittingly, theft. But you get the idea." That just diminishes the strength of what is being presented by making the author look obviously partisan and therefore easier to disregard._

It's way too easy to shout "censorship" and start coming up with conspiracy theories where mega corporations are taking control of peoples' freedom. In reality, I don't think Nestle actually cares about what anyone on Reddit's saying when we're all pseudonymous anyway. Twitter might be a different story though.</comment>


A Time Machine time bomb - <AUTHOR> http://rondam.blogspot.com/2009/09/time-machine-time-bomb.html

<comment><USER2> how could that guy hook up an eSATA drive to his mac? As far as I can see, none of the currently available machines provide eSATA support.

I thus assume that he was using OSX on a custom-built machine and it's very well possible that OSX doesn't have built-in eSATA support (due to the lack of official hardware) and thus doesn't recognize the external drives as external ones.

You see, when I plug a USB or Firewire drive into my Mac, Time Machine doesn't try to back them up.

<comment><USER3> Presumably using either an ExpressCard (in a MacBook Pro) or PCIe (Mac Pro) eSATA controller. Not exactly what I'd call "custom built". Interesting though, that it recognises USB and Firewire as external, but not eSATA. I wonder if it supports hotplugging/ejecting of eSATA drives at all?

<comment><AUTHOR> I use an ExpressCard. It didn't even occur to me that a drive connected this way might appear to the OS to be an internal drive, but that actually makes a certain amount of sense.

</comment></comment></comment><comment><USER4> When the backup drive get full Time Machine is going to delete the oldest backups, that's how it works. Using it for a permanent backup solution isn't too smart.

On my systems I also backup to a BackupPC system for longer term storage.

<comment><USER5> Exactly. Time machine is for those "oh shit, my disk is toast/I really didn't mean to delete those files/I'm switching to a new mac" moments.

It's not intended to be used to retain a complete history of your machine and as such if used for this purpose you're eventually going to get a nasty surprise.

</comment></comment><comment><USER6> On my Time Machine preferences pane it says 'Notify after old backups are deleted'. I read this to imply if TM decided it needed a monstrous amount of extra storage and decided it needed to delete a _lot_ (or all) of the old backups, there wouldn't be any notification until they were gone. I don't see any option to notify _before_ deleting them; perhaps a confirm option there would have been useful?

<comment><AUTHOR> That's how it was in Leopard. It changed in SL.

</comment></comment><comment><USER7> I'm becoming less and less comfortable with hard drives as long-term backup media. The other day I plugged in my external to back up some family photos, but the drive wasn't coming up and while I was investigating (this was Linux so it could have been any number of reasons) this faint noise was bugging the crap out of me. "What is that, a metronome?" It took a few minutes before the "oh shit" set in.

Time machine is nice because its nearly impossible to remember to do backups regularly (unless it's your job) and the backups are highly available, but really its more like an extended trash bin, not an archive. Storing your backups on-site is one thing, but keeping them plugged in and mounted is just asking for it.

Whatever happened to tapes? They're like comically expensive now. Another option is to copy your Time Machine out to S3 once a month. (Would probably take that long) but I don't know how you'd preserve the aliases / hard links.

<comment><USER8> Just have two external hard drives. I have one at home I use daily, and then one I keep in my desk drawer at work I use weekly [1]. The chances of both failing at once are pretty unlikely. You _could_ do the cloud, but I'm not comfortable having all my data in the could. If you _have_ to use an external provider, use Rysnc [2] . They seem trustworthy [3].

[1] <http://jwz.livejournal.com/801607.html>

[2] <http://rsync.net/>

[3] <http://news.ycombinator.com/item?id=702247>

<comment><USER9> Does it have to be "the cloud" as in S3? What about good old space on a traditional host? I have a Dreamhost account that claims to offer "unlimited" space for a flat monthly rate, and they support traditional *nix rsync/scp/etc. Seems like a good fit to me.

<comment><USER10> Unlimited space for _web content_. Backups don't count as web content, and will get you smacked down hard.

<comment><USER9> Very good to know :)

</comment></comment></comment></comment></comment><comment><USER10> >> Newly mounted external drives should be excluded from backup by default.

Hey! Let's solve a backup trimming problem by _potentially never backing data up_ unless the user digs around in their preferences! This seems like a cure that's worse than the disease.

<comment><AUTHOR> I wouldn't call irretrievably wiping out _all_ the backups of my primary hard drive in order to make room to back up a media drive that I just mounted a "backup trimming problem."

And by the way, I had that volume on my exclude list, but TM forgot about it. (That's a problem that has been around since TM first appeared.)

</comment></comment><comment><USER11> >> almost bordering on _legally actionable negligence_

Good luck with that, and please do call us when every single kernel contributor gets sued because someone accidentally overwrites their Windows partition when installing Ubuntu.

<comment><USER8> Linux disclaims all liability in the license. Seems like Apple might be bound by a few more laws because they are actually selling a product.

<comment><USER11> Check the OSX license, available at <http://store.apple.com/Catalog/US/Images/MacOSX.htm>. Excerpted:

YOU EXPRESSLY ACKNOWLEDGE AND AGREE THAT USE OF THE APPLE SOFTWARE IS AT YOUR SOLE RISK AND THAT THE ENTIRE RISK AS TO SATISFACTORY QUALITY, PERFORMANCE, ACCURACY AND EFFORT IS WITH YOU.

and

TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT SHALL APPLE BE LIABLE FOR...LOSS OF DATA.

and for good measure,

THE APPLE SOFTWARE IS NOT INTENDED FOR USE IN THE OPERATION OF NUCLEAR FACILITIES, AIRCRAFT NAVIGATION OR COMMUNICATION SYSTEMS, AIR TRAFFIC CONTROL SYSTEMS, LIFE SUPPORT MACHINES OR OTHER EQUIPMENT IN WHICH THE FAILURE OF THE APPLE SOFTWARE COULD LEAD TO DEATH, PERSONAL INJURY, OR SEVERE PHYSICAL OR ENVIRONMENTAL DAMAGE.

Bottom line is that these licenses don't mean much anyway - they just make it slightly harder to sue someone, in the same way that those personal liability forms you sign before doing anything are just to scare you. In the absence of an explicit contract - as in Apple guarantees "this will not cause any data to be deleted" - they've got a very good case that while it's sad you lost your data, it's not their fault.

Paying for something may create more of an expectation, but does not (in and of itself) guarantee you anything or increase liability.

<comment><USER8> Obligatory: IANAL

I knew that these disclaimers were in both licenses. However I think that the fact that Linux is given away for free and that OS X is a product makes a difference in the clause _TO THE EXTENT NOT PROHIBITED BY LAW_. I mean, you can't buy an Apple computer without OS X on it. Are they really allowed to disclaim all liability for everything? In my legally uneducated mind that seems unlikely.

<comment><AUTHOR> In most jurisdictions, whenever you accept money for a product there are implied warranties of merchantability and fitness for a particular purpose. Since TM is widely touted by Apple as one of the features of OS X they cannot disclaim the warranty that it does what it is supposed to do, namely, back up your data. What Apple has done is analogous to selling a fire extinguisher that actually sets your house on fire. You can't disclaim liability for something like that.

<comment><USER12> Apple have always sold TM as a way of doing backups not archiving. The original article seems to want to have used TM as an archive, not a backup, and that's where the problems came in.

That said, maybe I've missed something in the marketing that Apple have done for the product - your comment seems to indicate that. Do you have any specific examples of false claims made in marketing TM by Apple?

<comment><USER13> TM would have deleted every backup they had if they didn't stop it in time. Wanting every backup you've ever made is one thing; wanting the last backup you made to be available doesn't sound like archiving.

<comment><USER14> No, it wouldn't. If it doesn't have enough space to keep at least one backup, it won't delete previous backup, it will tell you that you need more space.

<comment><AUTHOR> Happily, the logs record exactly what happened. The request was for 877.54 GB. The TM volume is 731.95 GB. It deleted 14 old monthly backups (so just over a year's worth) before I stopped it. But given that it had requested more space than was actually available there is no reason to believe it would have stopped before wiping them all, and in particular, before wiping all my Leopard backups, which would have made it impossible for me to revert.

<comment><USER14> It didn't delete the latest backup on my drive. It told me that there's not enough space, but backup is still there.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER15> It is not clear from the article whether this happened because the author upgraded to Snow Leopard, or because they switched their external hard drive to eSATA. If the former, the behavior might be caused by the fact that Snow Leopard is a new operating system, and many of the system files that were in OS 10.5 have changed.

Have any readers that have upgraded to OS X 10.6 experienced the same issue?

<comment><AUTHOR> It's both. I upgraded to Snow Leopard, _and_ I remounted external drives. In Leopard, there was an option to warn before deleting old backups. In SL that option has changed to "notify after deleting old backups." That plus deleting the external volumes from the exclude list was the nasty combination.

<comment><USER16> You also say that you switched from USB to eSATA. There is a very good chance that from the perspective of the OS these are entirely new dives, maybe even seen as internal drives. There may not be an issue in the Time Machine code, but rather this whole thing is unfortunate edge case of intended behavior.

<comment><AUTHOR> Yes, I think that might be right. That hadn't occurred to me before.

</comment></comment></comment></comment><comment><USER17> This behaviour sounds perfectly reasonable for the common case, for whom it was designed.

Furthermore it sounds like this guy is using Time Machine as an archiving solution, not a backup solution. You shouldn't be all that worried to lose backups a year old if you know newer ones are intact. You should never delete a document there's any chance of ever needing again, then rely on TM or similar to save it for you.

<comment><AUTHOR> I'm not worried about losing the old backups. But I would be worried about losing _all_ of my backups, which is what _would_ have happened if I had not noticed what was happening and stopped it manually.</comment></comment>


What to do after a patent is discovered? - <AUTHOR>

I began developing an adtech product about 6 months ago after checking the USPTO and finding nothing that I was infringing on. Recently I discovered a patent granted to Yahoo in the beginning of June that describes my product exactly. I am not sure what to do now. Any recommendations on a course of action would be appreciated. <comment><USER2> Ask at [http://patents.stackexchange.com/](http://patents.stackexchange.com/) ?

</comment><comment><USER3> Your smart patent laywer might say: delete this post asap.

If you KNOW you're infringing a patent, it's MUCH worse than if you're infringing without knowing. That's why it can be better to not do patent searches.

<comment><USER4> this. and do get a patent attorney asap so that you don't run the risk of having worked in vain.

</comment></comment><comment> Alex-Galapagos You need to get a pattern attorney and ask. Basically if you word it differently and describe it differently then it's a different product

</comment><comment><USER5> 3/16/2013 the rule "first inventor to file (FITF)" was put in place, you are out of luck and now infringing on Yahoo's patent. [http://en.wikipedia.org/wiki/First_to_file_and_first_to_inve...](http://en.wikipedia.org/wiki/First_to_file_and_first_to_invent)

<comment><USER6> IANAL, but I was under the impression that prior art can invalidate a patent even with FITF if it wasn't an issue of contesting the ownership of the patent, since an invention has to be novel and publicly disclosed prior art would mean that the patent was issued invalidly rather than issued to the wrong inventor.

</comment></comment><comment><USER7> Open your wallet?</comment>


The Curse of the Excluded Middle (2014) - <AUTHOR> http://queue.acm.org/detail.cfm?id=2611829 <comment><USER3> So the first example, in C#,

(1) uses logging in the middle of an otherwise pure function,

(2) is criticized for the unpredictability of a bounded/well-defined instance of laziness, namely iteration.

The author's proposed solution is Haskell, which

(1) makes it difficult to add logging _at all_ to pure code without refactoring to add monads, unless you subvert the type system by using functions like 'trace'...

(2) ...in which case behavior will be significantly _less_ predictable than C# due to Haskell's _pervasive_ laziness - something which also comes up in other situations, like reasoning about performance/memory use.

I find it hard to take this article seriously.

<comment><USER4> Yes, Haskell is a very flawed language, however:

(0) The fact you need to modify value-level code to put it in a monadic context is a defect of Haskell's design, not of monads per se. See: [http://www.cs.umd.edu/~mwh/papers/monadic.pdf](http://www.cs.umd.edu/~mwh/papers/monadic.pdf)

(1) Laziness has nothing to do with being purely functional.

<comment><USER5> On the second point, laziness and purity are very closely related.

See [http://haskell.cs.yale.edu/wp- content/uploads/2011/02/histor...](http://haskell.cs.yale.edu/wp- content/uploads/2011/02/history.pdf) (§3.2)

See [http://stackoverflow.com/questions/31477074/how-lazy- evaluat...](http://stackoverflow.com/questions/31477074/how-lazy-evaluation- forced-haskell-to-be-pure)

<comment><USER4> Maybe for social reasons. From a purely technical standpoint:

(0) You can have a pure strict language just fine. You can also have a lazy impure language just fine.

(1) Nontermination is an effect anyway, so Haskell isn't pure unless you wish nontermination out of existence. (Which the Haskell community is in the habit of doing.)

<comment><USER5> I think you may have glossed over that paper I linked.

No, you can't have a non-strict impure language just fine, the semantics would make it too difficult to reason about side effects. I am not aware of any such language. According to Wikipedia, Miranda and Clean are also non-strict but they are also purely functional.

[https://en.wikipedia.org/wiki/Strict_programming_language](https://en.wikipedia.org/wiki/Strict_programming_language)

Haskell allows you to create non-strict impure code using functions like unsafeInterleaveIO, this is generally regarded as a bad idea for technical reasons except in very special cases. See for example Data.ByteString.Lazy.readFile, which is more or less obsolete since the creation of iteratees.

[http://stackoverflow.com/questions/5892653/whats-so-bad- abou...](http://stackoverflow.com/questions/5892653/whats-so-bad-about-lazy- i-o)

On your second point, nontermination is treated semantically as a value in Haskell. It is an error to consider nontermination an "effect". See:

[https://en.wikibooks.org/wiki/Haskell/Denotational_semantics](https://en.wikibooks.org/wiki/Haskell/Denotational_semantics)

In short, a function which does not terminate is said to return ⊥ from a semantic perspective. The monotonicity of Haskell functions is what makes this work as you'd expect without violating purity, since ⊥ is the least element.

<comment><USER4> > I think you may have glossed over that paper I linked.

I've read it before, and not precisely yesterday or today.

> No, you can't have a non-strict impure language just fine, the semantics > would make it too difficult to reason about side effects.

Sure, such a language wouldn't be very usable, but it's very much definable.

> See for example Data.ByteString.Lazy.readFile, which is more or less > obsolete since the creation of iteratees.

You don't need to convince me that lazy I/O is a bad idea. If it were up to me, the type of a lazy stream would be:

        data Stream m a = Nil | Cons a (m (Stream m a))

Where m is Identity when you have a strict (and guaranteed finite!) list. Of course, laziness is a monadic effect just like any other.

> On your second point, nontermination is treated semantically as a value in > Haskell.

 _Nothing_ is a value in Haskell. (If you work out the operational semantics of a non-strict language, you'll see that you don't need to "carve out" a subset of terms that must be considered "values", unlike the case for a strict language.) All you have is computations, some terminating, some not.

<comment><USER5> These comments seem to be redefining words each time they are used. A parent comment refers to "value-level code" in Haskell and now this comment says that "nothing is a value in Haskell". This seems to be contradictory.

It's hard to come to any other conclusion here—other than the conclusion that the author of these comments wishes to argue rather than discuss.

There's a lot to learn from the Haskell community if you're willing. For example, there's probably about two decades of work that's been done past the definition of lazy streams which you provide. Conduit, iteratee, pipes, etc. Really cool stuff with stream fusion rules and bounded resource usage.

<comment><USER4> Errr, oops, yes, my bad. I should've said _term-level_ code.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> I like the article as a roundup of some ways in which programming is usually done in a non-pure way. However, this early quote is an example of why I disagree with the article:

"The slightest implicit imperative effect erases all the benefits of purity, just as a single bacterium can infect a sterile wound."

This is just not true. It's a fair rule of thumb that the more pure a function is, the easier it is to reason about its behaviour in a larger program.

If the article had stuck to a survey of common non-pure programming style and its pitfalls, I'd be much happier, but then it started on the "Informal Introduction to Monads", a topic which requires more ink and motivation than it was given here.

<comment><USER4> The type structure of call-by-value languages is inherently monadic. The monad in question is the identity functor iff the language is pure and terminating. Given the predominance of call-by-value languages, I'd expect every programmer to have a modicum of understanding of monads, if only to have the faintest idea how popular languages (yes, popular languages, not Haskell) work.

<comment><USER6> By the way, it's very early on the morning after Christmas Day for me, so I'm struggling to load the structures into my mind properly. What category is the functor going from and to? The Precise Monadicity Theorem states that a functor G is monadic iff it has a left adjoint and it creates coequalizers of G-split pairs; what is the left adjoint of this functor? What does a coequalizer look like in the category? (I suspect I can work these out myself given a good statement of what the category _is_.)

<comment><USER7> Holy crap. This.

To paraphrase: "English, mofo. Do you speak it?!?" :-)

I have been programming for over 30 years (which I suppose means nothing if it's just doing the same year over 30 times, but bear with me). Every other language (at the risk of being largely self selected, I know) I have ever worked on seems to incrementally add things or shift ideas around a little bit.

Some communities might try to helpfully explain things to newcomers. But the perception of the Haskell community is that they revel in being as cryptic as possible in their purity.

I'm not even sure if the above comment was serious (it was probably deadly serious), or absolute troll fodder nonsense.

I know (more or less???) what a functor is, and I know what IFF means. Unfortunately, most of what I remember from mathematics comes down to a function has a domain of inputs and a range of outputs, and being a function implies you can predict the output from the input; a "type" is a rule for describing a set, possibly infinite (not much use for computing), of values; values can be "scalar", or multidimensional such as a vector/matrix/tensor. I know that "category" means something very specific to you guys, but not what. You lost me.

<comment><USER8> A category is a set of objects and a set of arrows between those objects, and an operator that composes those arrows associatively.

For much, much more, this might be a good fit for you: [https://bartoszmilewski.com/2014/10/28/category-theory- for-p...](https://bartoszmilewski.com/2014/10/28/category-theory-for- programmers-the-preface/)

</comment></comment></comment></comment></comment><comment><USER9> The problem with making effects explicit is that effects pollute: a callee doing IO means that all is callers also acquire the IO effect. Which is as it should be, except that this makes code modifications tedious if effects have to be explicitly mentioned. Haskell makes it harder still by using completely different syntax for monadic and pure code which means that introducing an effect in a callee requires rewriting the caller.

<comment><USER5> This is not a problem in practice, having worked on decent size (10s of kloc, so not huge) Haskell projects. It just doesn't really come up. You end up with the outer bits of your program written in some monad, and then smaller, self- contained modules will be pure. So if you need to rewrite some pure code and have it use IO, it's usually called from a place that already uses IO to begin with.

You might compare it to checked exceptions in Java, which are _actually_ a total pain in the butt in practice.

Even if you did need to "rewrite" the caller—which is rare, as I mentioned—the change is usually pretty trivial. You might change the type and stick an fmap, <$> <*>, liftM2 or whatever in the caller, but it won't be a big deal.

<comment><USER10> Agreed. That's not where I felt pain...it's swerving in and out of pure/impure/different monads code when I really just want to speak my logic outright, without threading it through the type system with a pipe cleaner. I always loved pure Haskell code like I'd use if I was solving project Euler problems or other toy logic things...but when I had to go into a monad with the lifts and *M methods and do notation...felt like tramping around in a suit of armor. Not to mention hideous type signatures a mile long, that you sweep under the rug with type synonyms. I grinned and bore it for a while in the name of "purity", but it sucked.

I guess my point is just to add some counterweight to this myth that once you grok pure FP you never come back to the "dirty" old world...and also to give some color to the particular pain points that arose. Haskell, I think exposes both a lot of the potential in pure FP, and several pitfalls.

<comment><USER5> The "suit of armor" may be an apt metaphor, clumsy but safe.

I personally haven't had problems with long type signatures in my code. Typically, they'll disappear when you define the right monad for your application. But I don't have much experience solving "toy" problems in Haskell, I've leaned more towards production code, so my experiences are different.

That's one of the main problems with Haskell—so many people learn it to get better at programming rather than to use it in earnest, write blog posts about monads, and solve toy problems like Project Euler. Like other languages, it takes a different form when you're managing larger projects. Just my 2¢.

</comment></comment></comment></comment><comment><USER11> > Operations as ordinary as exceptions, threading, and I/O all cause as much > hardship as simple mutable state.

Not sure that threading is so ordinary considering that the whole argument is about how to make parallel programming safe.

There may be good reasons to keep as much code as possible purely functional. Even my impure imperative mind finds it comforting not to have to worry about ground shifting below me. Impurities affect the purity of the caller - <USER2> paper has a valid point but then it was never considered a good idea to manipulate state hidden deep down in a remote function. In fact I would argue that the place for manipulating state is exactly in the center middle where it can be seen and reasoned about clearly.

I find it at times surprising how far people can get with purely functional programming. It may be required to use black and white thinking when deciding what to allow in the functional space. But then for any program affecting the real world there is going to be some I/O. The physics of the real world has state as have ledgers. Going to the extremes likely is not a good communication strategy when reaching out to the imperative world.

FYI: Link to the author's Wikipedia page: [https://en.wikipedia.org/wiki/Erik_Meijer_(computer_scientis...](https://en.wikipedia.org/wiki/Erik_Meijer_\(computer_scientist\))

<comment><USER7> E.g. - "Erlang, you suck because you allow message queues. You might as well just make empty Java Beans and mutate them whenever you want". (OK, I embellished a bit).

Really??? The author sees no point having an explicit mechanism through which updated values are obtained, and considers it the same as PERFORMing an update on something in the DATA-DIVISION at any point in time???</comment></comment>


Cstore_fdw: A Columnar Store for PostgreSQL - <AUTHOR> https://github.com/citusdata/cstore_fdw

<comment><USER2> There's also the In Memory Columnar Store [0, 1] by Konstantin Knizhnik, if anyone's interested.

[0]: [http://www.pgcon.org/2014/schedule/events/643.en.html](http://www.pgcon.org/2014/schedule/events/643.en.html)

[1]: [http://www.garret.ru/imcs/user_guide.html](http://www.garret.ru/imcs/user_guide.html)

</comment><comment><USER3> Compared to Monetdb:

[https://www.monetdb.org/content/citusdb-postgresql-column- st...](https://www.monetdb.org/content/citusdb-postgresql-column-store-vs- monetdb-tpc-h-shootout)

</comment><comment><USER4> I briefly worked for a place that had been through cstore and greenplum before buying into Vertica, right before I started. They were simply hungry for analytical performance and had the cash. It was a really great buy for them in performance but also ease of use.

<comment><USER4> Hindsight: I was skeptical that they'd left all the metadata (configuration/application domain data) in PostgreSQL while moving the (100x larger) business data to Vertica. At this point I'd repeat the pattern just for the sheer delight of dealing with a complex application schema in PostgreSQL. The flexibility demanded of Vertica was decidedly less in that shop.

<comment><USER5> Did you guys perhaps benchmark kdb vs. Vertica?

<comment><USER4> Nope. Vertica provided the right performance and the right (SQL) interface for analysts.

</comment></comment></comment></comment><comment><USER6> This isn't the same cstore as in Stonebraker et al, right? [http://db.csail.mit.edu/projects/cstore/](http://db.csail.mit.edu/projects/cstore/)

<comment><AUTHOR> Nope, they're two different systems with a similar name.

</comment></comment><comment><USER7> This deserves a closer look. It appears to be making use of the ORC format used in Hive, and last I looked was implemented in Java and tied in tight with the Hive arch. So this must be a rewrite of an ORC reader/writer. Selfishly I wish they'd picked Parquet instead of ORC.

</comment><comment><USER8> This looks very promising

</comment><comment><USER9> Excited! Let's see how it performs.</comment>


Machine pareidolia - when computer vision "sees" faces - <AUTHOR> http://www.flickr.com/photos/unavoidablegrain/sets/72157628855014523/

<comment> daniel-cussen Some of these are in fact modeled after human faces, so there's a false negative rate here.</comment>


Quick start projects - <AUTHOR> http://blog.sayan.ee/quick-start/ <comment><USER2> This is super cool! I've often thought about doing exactly this but never got off my ass to start building it!

The only other functionality I would like out of my own setup which I might play around with is having each of the projects have some keyword / context based on the project. So for example all of my 'blog' browser windows are in one browser window with multiple tabs and I could be in another project and move a tab to my blog project easily.

The other thing I would like about that is sometimes I end up with 25+ tabs, all over the place relating to one or two of my projects. If when I did the equivalent of virtualenvs "deactivate", it would save all of the tabs in the active projects browser window and they would open later when I deactivated the project. This also would be useful for to leave a project all alone, as close to how I left off as possible to focus on something else, but without losing that state.

Like screen for your entire work setup.

Thanks for sharing your setup!

</comment><comment><USER3> In osx I just have a desktop per project and never close anything related to it. I code using vim though so I'm not sure if this works with an ide.

<comment><USER4> I have tried this setup but I found it particularly problematic that there isn't a separate 'dock' per desktop so I end out accidentally switching desktops for common apps such as a browser and shell. Have you found a work around for this issue?

<comment><USER5> How about having a separate user for projects, too radical? Automate user creation and initialization.

<comment><USER6> That's part of the reason you have tools like Vagrant and VMs.You don't have to create separate accounts for each project but emulate that in your VMs.

</comment></comment></comment></comment><comment><USER7> Interesting! At some point I have started creating small shell functions to switch between environments (setting the correct paths, etc.). This was when I was mainly doing C, C++, and Prolog work.

In my last two positions I primarily work on mavenized Java projects and find that IntelliJ projects provide most of the context switching.

The upside is that it is far less work to set up and customise projects than vim/emacs plus shell. The downside is that the customisation that you can do is far more limited.

I'd love to hear experiences from someone who works on Python or Ruby code, and tried both PyCharm/RubyMine vs. vim/emacs/tmux.

<comment><USER8> I am a diehard Emacs fan. However, I use IntelliJ for all projects with the Python, Ruby, HTML plugins.

I have a keybinding (C-c, C-e) to open up the current buffer in Emacs if I'm trying to use Emacs, but I mostly stay in IntelliJ for most of my time.

</comment></comment><comment><USER9> Wow, I just built nearly the same thing for my windows environment using autohoykey.

I might put together a post about it now that I see there is some interest

<comment><USER10> Since I end up using Windows more than I'd like, I'd like to see it. Thanks.

</comment></comment><comment><USER11> My setup is Emacs + workgroups.el + prodigy.el. No GUI apps to manage, everything configurable via elisp. It is grand.

</comment><comment><USER12> Congratulations, you just invented the IDE.

</comment><comment><USER13> I'd love to see someone build a similar script that can also work with desktop applications like Eclipse or creative tools like Photoshop so we can with one click switch between project without having to close / open files and applications. There's a good chance there's something out there like that, if so I'd live to know about it, and if not it might be a cool thing to have. Thanks!

<comment><USER14> One of the startup companies I used to work with was called Cupcloud ([http://cupcloud.com](http://cupcloud.com)). If you're OK with a cloud app, it can support Photoshop and Illustrator

</comment></comment><comment><USER15> Very interesting. I also find myself often doing the same repetitive workspace setup. I think I'll go ahead and try to apply this technique on my Linux environment.

<comment><USER15> Update: Easier than I thought. I just created a Bash script with the project's name and with a few lines I open my most used applications for working on that project.

</comment></comment><comment><USER16> The only problem I have with tmux is copy/paste texts from outside. Maybe, I am too noob. How do you guys do that ?

<comment><USER6> I am guessing you are aware of. [https://github.com/ChrisJohnsen/tmux-MacOSX- pasteboard](https://github.com/ChrisJohnsen/tmux-MacOSX-pasteboard) That solves a lot of issues I have with copy paste.</comment></comment>


"She Said Thank You" (Avoiding Procrastination) - <AUTHOR> http://avanoo.wordpress.com/2007/08/26/she-said-thank-you/

<comment><USER2> Why would she? So you can avoid procrastinating by not feeling the urge to procrastinate? Isn't that entirely useless advice?</comment>


One Weird Trick for 2020 - <AUTHOR> https://tinyletter.com/garybasin/letters/one-weird-trick-for-2020 <comment><USER2> Simplicity.

That's the answer to the click-bait title. It's simplicity.

<comment><USER3> My answer was to never click a link that says "One Weird Trick..."

Haven't read the article, so it seems to be working!

</comment></comment><comment><USER4> For me it was taking more time off to take care of my health, everything be damned.</comment>


How We Are Improving Performance - <AUTHOR> https://blog.imgix.com/2017/03/17/note-on-performance.html <comment><USER2> We were using imgix for a while and were generally happy, but things started go downhill and some point, or so it felt anyway. Their support was always a bit opaque. And the service itself didn't evolve (as far as rendering. E.g. Composing more than one of the same filter wasn't really possible... for example adding two watermarks). We've also had issues with CORS headers that weren't resolved and our end users couldn't get images some times...

We switched to host our own thumbor (open source), and couldn't be happier. We pay around a quarter or less than before (even with failover in place) as well.

We really wanted to use a hosted service. We're not keen on hosting stuff out of our core business. But in this case it just didn't work out.

EDIT: link to Thumbor [https://github.com/thumbor/thumbor](https://github.com/thumbor/thumbor)

<comment><USER3> Sorry to hear that your experience with imgix wasn't great. Performance, quality and correctness are obviously super important and non-negotiable aspects of our service. While we have always heavily focused on these areas, we're going to double down for the short term.

We do have some interesting solutions in the lab for more elaborate compositing amongst other things. Part of the challenge there is to stand up a new API interface, since the URL API gets pretty cumbersome for coordinates and tons of composites. We have some interesting ideas there, but it needs to bake a little while longer.

It's also a challenge to get that new stuff out while focusing heavily on our core areas, but we have to do both. You'll start to see more exciting new things ship from us over the next few months.

If you have a solution that works for you, that's great and best of luck. If you do ever feel the urge to check in on what imgix is up to, we would welcome it. Feel free to reach out to me directly any time (e-mail in profile) and I'll work with you to get what you need.

<comment><USER2> Thanks. Not sure why you're being downvoted. Recognising those shortcomings is important, and I understand that you might not have all the answers to all customers.

I think your support could have been more responsive though, and communication improved. You guys could probably detect that we weren't happy and about to leave. Reaching out at that point could have made a difference perhaps...

We did genuinely want to work with imgix, but those hurdles left us looking elsewhere. These things happen all the time. Competition is a good thing overall. I'd like to hope that our feedback helps in some way. Even if it's hard to hear.

<comment><USER3> Yes, your feedback is definitely appreciated, and I regret that we didn't act fast enough to retain your business. I will use it as a teaching exercise with the rest of the team for what we could do better on next time.

The unfortunate reality this time around is that we got overwhelmed and had to focus on doing work that had the most impact to the largest number of customers. That meant that more personalized approaches to customer retention kind of slipped through the cracks. This can't happen again, and I'm adamant that it won't.

I see image processing services as an important area for innovation and growth on the web over the next several years. It doesn't mean that imgix has to get the business of every site on the Internet, but fundamentally I believe that anyone who cares about how their content is presented should use a service like imgix. Doing image manipulations by hand, or serving non-optimal formats or sizes to browsers is going to be viewed as serving your site without using a CDN: it just isn't the state of the art anymore, and the benefits are worth it.

I welcome competition from other companies who run competing services, or build-it-yourself solutions like thumbor, because that means we really were on to something: the market is real and these services provide real value to customers. I want imgix to be at the front of the pack, and that means doing the hard work every day to make sure we're there.

</comment></comment></comment></comment><comment><USER4> There's two kinds of apologies, and they look very similar on the surface. There's "Sorry for this problem, but here's why it's not my fault" and there's "Sorry for this problem. Here's what happened". Am I taking the blame for the failure of others, or deflecting the blame _onto_ others?

This feels a lot more like the latter, and it's wonderful to see. I'm sure it's a lot harder to write, but it shows a heck of a lot more integrity.

<comment><USER5> I generally have issues when an engineering manager blames bad luck. Luck doesn't happen as often as people like to believe.

For instance, losing three backup generators in ten minutes (as happened to someone else a while back) might be bad luck, but I'd want to see the maintenance logs and talk to the vendor. Because what's much more likely is that they hadn't been maintained properly, or there was an manufacturing defect in that run and they all came from the same batch (see also, clustered hard drive failures).

If their upstreams have had intermittent problems before, it was only a matter of time until they both had problems at the same time. And capacity planning is one of those strategic decisions that gets bland on the engineering team, who never get what they need because they're a 'cost center', which is a very different and highly damaging way of thinking about the cost of doing business (intrinsic costs).

<comment><USER3> [I work at imgix and have helped lead the team on the production issues we face and gathering the details for this blog post]

> If their upstreams have had intermittent problems before

Sorry, this might not have been clear. We had never had major issues with any of our upstream providers before -- beyond scheduled maintenances there had never been a service interruption over a number of years.

There's almost always the risk of coordinated failure, no matter how many levels of redundancy you put into place. The cost gap between 99.99% and 100% is humongous for this reason.

From these failures, we learned some valuable lessons. One of our transit providers was acquired a year or so back, and it's apparent that things have dramatically changed in how they operate their service and respond to issues. We won't be continuing the customer relationship there for much longer unless we see positive steps taken. Another transit provider has been much more responsive and the conversations have been more productive -- we can work through the issue we observed and get back to a positive situation there (I think).

And then finally, we need more diversity. We turned up another circuit yesterday, and I'm still working on adding another 1-2 providers in addition to more peering or direct connect situations. These take a bit of time, but it's work we need to do before we actually require it, because by then it's too late. We're working as quickly as possible to get this part of our infrastructure on rock solid footing.

> capacity planning is one of those strategic decisions that gets bland on the > engineering team

Absolutely. We were doing capacity planning, but it wasn't anyone's idea of a good time.

We are correcting this with the resources we have today, and we're going to do a better job in the short term. In the long term, this is a non-negotiable core aspect of someone's job -- either already on the team, or a new role that we create. It can't be a secondary, rainy day kind of task you take on time permitting and it also can't be a task that we're not well equipped to handle on a regular basis.

</comment></comment></comment><comment><USER6> They put out a blogpost while their support keeps stonewalling and denying any such issues. Nor do they reply on twitter or even tweet the blogpost. This seems too little too late.

Disclaimer: I am currently a customer who had financial loss because of them.

<comment><USER3> [I work at imgix and have helped lead the team on the production issues we face and gathering the details for this blog post]

I'm sorry that you had this experience with imgix. We have had challenges in correctly addressing support issues during this period -- support has been swamped. We've expanded that team, and we're continuing to do so.

You're right, we didn't tweet this blog post, but we did e-mail it to all customers with activity since Jan 1 2017.

In this case, we didn't properly convey that the Shopify integration guide we provided wasn't an official integration with Shopify. It was meant as a best effort "here's how you can do it" sort of thing. I requested that we take it down because it wasn't working due to a change on the Shopify side which prevented us from purging images.

I'd love to talk with you directly to try to make this right for you. If you'd like, my e-mail is in my profile.

<comment><USER6> I understand your support might be swamped but this is what your support wrote to me;

"Support is on-call during regular business hours, Monday - Friday, and premium customers can request SLAs for support."

This, while other users complained to me via twitter/email that they were also not getting any valid responses. I would not send those words while my service is not working as expected.

I'm in the process of moving out of Imgix esp. after your support said they would be happy to cancel my account if thats what I wanted.

The pushback from your support made me setup a production level thumbor instance which will go live next week.

Thanks for your reply but I wish you had replied to [https://twitter.com/troysk704/status/841158793287278593](https://twitter.com/troysk704/status/841158793287278593) sooner.

<comment><USER3> OK, totally understand your concerns. I see a few tickets from you in our ticket tracking system, and in the interest of improving next time, we'll use this as a teaching example with our support team.

Best of luck with your new solution.

<comment><USER6> Glad to see that you have acknowledged the problem. I am sure you will fix them :)

</comment></comment></comment></comment></comment><comment><USER7> I wish this post gave more concrete details, so we could learn from it.

 _A critical piece of our network infrastructure failed after 3 years of correct operation in a way that proved difficult for our network engineering team to troubleshoot._

Which piece of infrastructure? How did it fail? Why was it difficult to troubleshoot? Can you do anything to prevent this type of issue in the future?

 _We observed new traffic patterns with significantly lower cache hit rates than our historical median, and it took us some time to determine whether the source was abusive in nature or a legitimate new customer use case._

What were the new traffic patterns, and why did this cause lower cache hit rates? Why did it take longer than expected to determine the nature of the traffic?

<comment><USER3> [I work at imgix and have helped lead the team on the production issues we face and gathering the details for this blog post]

> Which piece of infrastructure?

In this case it was a top-of-rack switch. That rack had a few of our external DNS resolvers on it. Normally this wouldn't be a problem, because we deploy services across multiple racks to prevent this kind of failure mode.

In this case it was a bit of a problem, because this was the first rack deployed at this site and it turned out to not conform to the standard configuration across the rest of the racks (because it was what we bootstrapped the site with).

So two things: it was tough to get into the device to troubleshoot it (we wound up using the serial port infrastructure deployed for this purpose), and it had a service impact even though it shouldn't have (we have since migrated DNS resolvers out of this rack and have scheduled a future maintenance to get this switch's configuration corrected and ugprade its OS).

> New traffic patterns

This is a bit tougher to dig into, but essentially it was a needle-in-a- haystack kind of situation. Customers can use any imgix URL API parameter they like -- some of these use cases get pretty complicated for the backend to handle. Think of the watermark parameter stacked with another URL that is also an imgix URL with another watermark parameter, five layers deep. These sorts of operations take a lot more rendering resources than a simple ?h=600&w=600 operation.

In this case, we observed an influx of these sorts of more difficult operations. We have various logs and metrics sprinkled throughout the system -- we use Prometheus, Kafka, heka, BigQuery, Grafana and a few other systems to collect and present the data we need to run the service. We also issue unique id's per request to track their path through the system. What we don't have -- and need to -- is one end-to-end view of a request's path through the system and the system's capacity and performance at each point in our stack.

It turned out that for some amount of the increased rendering traffic we saw, it wasn't that we suddenly got more requests. There were simply many new permutations for each original object. That lowers the CDN cache hit rate for a period of time.

The other thing that comes to mind (and I think I'm forgetting a third type here), is that some of our request parameters require normalization so that they can utilize the same cache object. Think of parameters like dpr, which is a floating point number but realistically is only useful up to a few decimal places. dpr=1.33 and dpr=1.3333333333 are actually the same image, but they would have different cache keys and require two renders that are effectively the same object.

We normalize the dpr parameter down to three significant digits. What we found is that this sort of normalization was necessary for another parameter as well.

<comment><USER8> Why isn't this in your blog post? Or at least a separate "technical details" blog post linked from the announcement?

When you have an incident, especially something that hurts your customers beyond just a service outage, you come clean, fast, and as completely transparent as possible.

<comment><USER3> [I work at imgix and have helped lead the team on the production issues we face and gathering the details for this blog post]

We posted the blog post yesterday (Friday) afternoon, after getting the copy ready Thursday evening. Friday was spent acquiring the right customer e-mail list (a little harder than it sounds and something to improve in the future) and taking one last pass over the post (among other tasks we're all doing to actually operate the service, communicate to individual customers, etc.).

It's super important to be open and honest and fast in your response, but it's also important to communicate correctly. The post in its current form is meant to be transparent and apologetic but also show that positive steps have been taken and results are being seen now. If we missed the mark on transparency, or honesty, or the hints of positivity, it wouldn't have been the right message for where we truly are at today.

I'm completely open to writing a more in-depth technical post, but that would have compromised our timing to get this post out on Friday. That's why we went with the post in its current form -- anything else would have taken longer to get to our customers.

It's something that I'll think about more this coming week and hopefully get out soon.

</comment></comment></comment></comment><comment><USER9> I wonder if their render service is still backed by racked Mac Pros ([http://photos.imgix.com/racking-mac-pros](http://photos.imgix.com/racking- mac-pros)). If so, considering the lack of updates around that machine for the last several years, I wonder if they are planning to remain with that solution.

<comment><USER10> Wow. I just read that.

> We operate our own hardware, run our own datacenters, and manage our own > network infrastructure.

This seems insane to me. Although I don't work with image processing beyond "saving for web" in Photoshop, so I could be wrong. Why would they not use AWS or any number of other cloud providers where capacity planning is handled for you?

<comment><USER11> I would guess because of costs. At one point I was going to do a video processing app and after doing some calculations the costs were insane, and the savings with actual hardware were tremendous. I imagine image processing is a lot less than video, but maybe the same deal. Then again... they did rack mac pros so who the heck knows what they're thinking.

<comment><USER3> [I work at imgix and have helped lead the team on the production issues we face and gathering the details for this blog post]

The main reasons are twofold:

A) cost. I haven't done the math yet in 2017, but my recollection of the difference circa 2015-2016 between GPU instances on AWS and our solution is a COGS that's about 3x higher on AWS. GPU instance prices have come down a bit since then, but they're still very expensive and somewhat supply constrained as well.

B) Technical frameworks. We'll reach an inflection point this year or next where we've added or changed so much of the rendering pipeline that we've basically written it from scratch, but until then we're still reaping the benefit of building on top of CoreImage. It really benefitted us early on in getting a MVP faster, and it's continued to pay (slightly reduced) dividends over the course of the company's life. At some point the dividends will stop, and then we'll move on.

I understand the skepticism around Mac Pros. There are challenges there -- it isn't my most favorite solution of all time. It is a practical solution for us though. I have no room for computer religion or deciding to do things because they're cool or shiny or new. Anyone that's ever spoken to me for a few minutes in person about technology stuff can attest to that.

The Macs don't have IPMI, for example. That sucks but we do have power outlet control and a network installer as a way to back into "out of band management" for them. They're largely stateless, so it's a tolerable solution.

They do run a Unix-like OS (thank god), and they do represent a good price / point ratio for image rendering hardware. We could do a little bit better with Linux servers and GPUs -- but the upside is only around 10-20%, and there's still engineering work to do there. It's on our roadmap to explore this more fully and maybe start taking more concrete steps in that direction. For now, OSX still gives us more than it takes in terms of cost.

</comment></comment></comment></comment><comment><USER12> Thanks for explaining and openness. We were bit frustrated as just finished migration to imgix and hit many issues with images delivery.

<comment><USER3> [I work at imgix and have helped lead the team on the production issues we face and gathering the details for this blog post]

Sorry that you experienced these issues. I appreciate your understanding.

Please feel free to reach out directly to me (e-mail in profile) if there's anything you'd like to talk through.

I'm optimistic that you will have a better experience going forward, but I can guarantee you that we are doing everything possible to provide the service to you and we will do a better job of communicating in future.

</comment></comment><comment><USER13> _Improved our GIF encoding pathway to increase throughput._

I'd be curious to hear exactly what they did around this. I recently worked on some gif encoding, and was surprised that there's actually quite a tradeoff to be made between performance and good color palette choices.

<comment><USER3> [I work at imgix and have helped lead the team on the production issues we face and gathering the details for this blog post]

I don't have the specific details handy, but I can confirm that it's a bit of a tough one. It's actually much easier to do work in pretty much any other format -- the very reason why people use GIFs (universal support) is also what makes them tough to work with (the spec is old and crusty and from 1989).

I'd like to do more of these sorts of posts -- talk about the challenges with GIF encoding and how imgix has solved them. Probably not this month or anything, but I'll try to get it on the content roadmap for April or May.

<comment><USER13> That'd be great. I've found [https://www.lcdf.org/gifsicle/](https://www.lcdf.org/gifsicle/) to be a good code resource for optimization, if a bit hard to follow.

</comment></comment></comment><comment><USER14> The company that literally racks Mac Pro trashcans sideways, is telling us that they have made technical decisions that allow them to offer a competitive price for features.

<comment><USER15> Ha, I thought that was a joke.

Apparently not: [https://blog.imgix.com/2015/05/08/racking-mac-pros- hardware-...](https://blog.imgix.com/2015/05/08/racking-mac-pros-hardware- design-for-web-scale.html)

<comment><USER16> Can you really not run image rendering code on Linux servers? Vs you know, spending thousands of dollars racking OSX servers? Because that would make "increasing capacity" about 3 clicks in AWS....

Perhaps I am missing the competitive advantage of using OSX to resize images, but it sure seems like the shortcomings are obvious :)

<comment><USER14> Well, basically they developed the thing on macbooks with Apple's image manipulation libraries (which are basically just implementations of published, standard algorithms). Then they didn't bother porting it to something they could scale. Instead, they opted to commit to putting genuine Apple hardware on racks, something that even Apple doesn't do.

It's an understandable direction if your business is at the "Well, I have a few of these mac pros sitting around, I wonder if I could make a living at this" stage.

<comment><USER16> Sure, great way to POC. Then don't you immediately get funding and go hire some devs to code it up for real? Vs hiring server guys to rack 100s of Macs? Seems like something your VC should suggest to you, not "heres a million bucks, go create rack mount macs.."

<comment><USER3> [I work at imgix and have helped lead the team on the production issues we face and gathering the details for this blog post]

I've touched on this elsewhere, but here's the crux of the problem:

Racking servers, even if they're cylindrical, is actually much easier than building an entire real-time image rendering pipeline from scratch.

The chassis that we co-designed and had made works out to a few hundred dollars per machine, since we aren't buying in giant quantity. When you talk about servers that run 24/7 and cost between $5k-25k per box, that's not the dominant expense by any measure. It's also been picked up by some other companies to use, which is always nice to see -- we aren't keeping it as a proprietary solution, anyone can buy it from our vendor.

Either way we decided to go, we would still be solving both the image rendering and machine racking / operation problems simultaneously to different degrees. As it sits now, we have about 4-5x the number of infrastructure and imaging engineers as we have datacenter managers / technicians on staff. As it should be, I think.

This design decision is actually part of our advantage, in that it allows us to deliver more functionality to our customers at a lower price per unit on the backend. There's always room for improvement, and we continue to iterate on it -- some day the Macs will probably be gone from the datacenter, but only when it's the right move to make.

<comment><USER16> > Racking servers, even if they're cylindrical, is actually much easier than > building an entire real-time image rendering pipeline from scratch.

I think this is still a penny wise, pound foolish decision.

I am sure it is less work to figure out how to buy and rack some trashcans, vs writing an entire image pipeline for 1.0. I get that.

I also don't know your full specs. There are tons of FOSS image libraries out there. What % of your workflow do they cover? Are you sure you are estimating taking something like Pillow and adding in missing pieces, vs writing an entire image library from scratch? Things like basic filters and image resizes are a solved linux problem for a long long time.

Hardware lock-in always has a price. People that build on AWS exclusive things like lambda do the same thing, to a smaller scale. It seems like you are paying this lock-in price, and will continue to pay it. Perhaps it is the right decision, perhaps not. But I cannot image locking my entire company into the whims of a consumer trendy product line, when there is no fundamental need. I would look really hard at hiring some devs to build out a proper rendering engine...

<comment><USER3> Without getting too into the weeds -- we do use OSS and third-party libraries where it makes sense. Our rendering pipeline incorporates things like mozjpeg, Intel IPP, jxrlib and other technologies. We're doing the integration and augmentation work to make this all play nicely together with the right quality and performance.

However, there is no off-the-shelf solution that solves the same core problem as the imgix pipeline. Resizing or format conversion, sure. That plus the types of graphical operations that Photoshop supports? Less common. Now do all of that in one pass in 30-50ms, streaming to/from the GPU, with the ability to easily add new workflows and operations? It's our core business, and we own that workflow completely inside of our stack.

We built on the foundation of CoreImage and that gave us a leg up -- we can take most of what we've built, replace the foundation and keep going on another platform (such as Linux). The main challenge is that there is no equivalent technology to CoreImage on any other platform. PIL (or Pillow) are adjacent technologies, but I can tell you from experience trying to operate a PIL-backed image thumbnailing service at scale it is not a large scale solution. Better than ImageMagick for sure, but it still has limitations. Even something like RenderMan is not tuned for this kind of workflow -- it's awesome at what it does, but latency and scale are not its central concerns.

There are risks to imgix's business, and this is one of them, but I'll be pretty surprised if it sinks us. We have some pretty sharp cookies working on our rendering platform, and while success on that front isn't guaranteed it's entirely within the realm of possibility.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER17> It says on their homepage that they use a 3rdparty CDN. I'm wondering how you can improve upon that if you are just a customer from the CDN's point of view.

<comment><USER3> [I work at imgix and have helped lead the team on the production issues we face and gathering the details for this blog post]

imgix does utilize a partner to handle some of our image delivery components. We're a customer of that CDN in that we pay them money for the services they provide, but we're actually more of a partner in that we have done considerable integration work and we work closely with them to build what we need to deliver our product.

We talk with them frequently, and there are pieces that aren't just off-the- shelf that we utilize to provide the integrated service we sell to our own customers.

To be clear though, the issues we faced that are discussed in this blog post are not related to the CDN or delivering already rendered images to end users. It's around rendering new images, which doesn't involve the CDN.

</comment></comment><comment><USER18> Font on [https://www.imgix.com/pricing](https://www.imgix.com/pricing) page is broken in Windows/Firefox (look how small "e" and "a" are displayed): [https://image.ibb.co/ei7m8v/windows_font_problem.png](https://image.ibb.co/ei7m8v/windows_font_problem.png)

This is without zooming (100%), 120 DPI, Windows 7 x64.

Please use standard fonts and don't overengineer, font hacks like this will never work as expected across all platforms.

<comment><USER3> Thanks for pointing this out. The pricing page is on an older version of our site design, and I believe this may already have been corrected in the design proofs we've been working on.

I'll prioritize rolling out the fix for this particular issue next week.

</comment></comment><comment><USER19> Imgix is a good example of burden of legacy code / infrastructure that they can't get away from.</comment>


Ask HN: Transitioning to Linux - <AUTHOR>

Last year I downloaded the Windows RC, and it is about to run out, but I've got a new laptop with Win7 coming in a few weeks, so thought I'd load linux onto the machine currently running Win7 RC.<p>Any advice about transitioning to Linux? What distro do you recommend?
I'm a php dev, I do my own design work, I'll need an email client, and the usual tools.
Any advice? <comment><USER2> When I got my new computer I tried Gnome, the desktop environment (DE) that is default in UBunutu, for a few weeks. I had been using KDE3 prior and KDE4 had just surfaced and was practically unusable.

I tried really hard to like Gnome but I just couldn't get anything done with it. I find it very easy to switch between XP/Vista and KDE4 (which I went to instead of Gnome). YMMV.

What app are you using for your PHP development? I've tried about a dozen different ones (Kate, Eclipse, jEdit, Notepad++, Dreamweaver, ...) and am happy at the moment with Quanta. Unfortunately it's now going through the transition to KDE4 and is under resourced with devs so I'm considering moving to KDevelop.

<comment><AUTHOR> I wasn't familiar with KDE4, I'll take a look. I like Eclipse for my dev, but I'll take a look at Quanta, I'm not familiar with that one either.

</comment></comment><comment><USER3> I'd recommend Ubuntu. Good software, good community.

<comment><USER4> +1

I'm using it for about half a year now. I was surprised how easy it was. Most software you can get in debian packages and it installs similar like in windows.

For php I use gEdit, very lightweight Gnome editor with plugins. If you want full IDE eclipse PDT is the way to go.

For images I like GIMP.

I've 1GB of ram and Flash seems to be really heavy on the laptop, fan is on max when viewing flash.

<comment><USER5> Have been using Ubuntu on HP notebook exclusively for the past 2.5 years. Programming with GCC + Python + several browsers and multiple web servers.

Apt manages all the upgrades, patches fine. Upgraded from Ibex to Jaunty to Karmic without having to re-install anything. Just follow the release information.

Although the HP has only 1.5GB RAM, single 1.8GHz Celeron. I run the notebook screen and an external 22" LCD without problems across 4 desktops and typically have 10+ apps running concurrently. Gimp & OpenOffice are fine for typical uses. With more up to date hardware you won't be disappointed.

</comment></comment></comment><comment><USER6> Just go with Ubuntu, don't even think about it. Most things work right out of the box, there's a ton of help out there and the package management is ace. Installing anything pretty much only takes: sudo apt-get install anything.

(Confession: As far as the desktop experience is concerned, I do miss XP. Ubuntu has come a long way but there are just these little things in the UI...)

<comment><USER7> > I do miss XP

Oh boy. I am an Ubuntu user and I have been using Windows for a year now (corporate policy - don't ask...) and in the first 6 months it drove me crazy. The first thing I did was to install Cygwin and Emacs in order to have a proper terminal and a text-editor, but, still, the seemingly random slowdowns, moments when even the mouse pointer wouldn't move... More memory reduced the pain, but, still, I miss select-copy and middle-click-paste.

If you can read Portuguese, I wrote about it on [http://www.dieblinkenlights.com/artigos_pt/vivendo-no- window...](http://www.dieblinkenlights.com/artigos_pt/vivendo-no-windows) and [http://www.dieblinkenlights.com/artigos_pt/o-windows-esta- pr...](http://www.dieblinkenlights.com/artigos_pt/o-windows-esta-pronto-para- o-usuario-experiente)

<comment><USER8> select-copy and middle-click-paste lacking are one of the biggest annoyances for me whenever I have to use windows or osx.

Linux/Ubuntu/Gnome have a lot of small things like that that dramatically improve usability. It annoys me to no end when people complain about usability on linux and then gloss over such things.

</comment></comment></comment><comment><USER9> For a first timer, probably Ubuntu or Linux Mint.

<comment><USER10> _"... Ubuntu or Linux Mint. ..."_

but not debian (hardest technically to install).

<comment><USER11> Linux From Scratch, and I put it on a Game Cube. Nothing like researching for 2 months and then realizing your TV and couch made for a lousy hacking terminal.

</comment></comment></comment><comment><USER12> You'll be fine.</comment>


Someone Invent This - The Death Clock - <AUTHOR> http://bennesvig.com/post/4525079119/death-clock

<comment><USER2> For me this would serve as a very positive reminder. Since both my parents (independently of each other) passed away at 55 years of age, I can't shake away the feeling that 55 years is my life expectancy as well.

I know in my head that it is completely irrational, but over and over I find myself thinking of 55 years as a proper estimate. The other day when a friend complained that the average age of her colleagues must be like 60, in my head I thought "Wait, how does that work? Oh, that's right, most people actually get to live beyond 55 years of age".

A daily reminder that my true statistical life expectancy actually goes quite far beyond 55 years would be something that gave me a slightly more positive outlook every day. Right now I'm 35, if anyone wants to put it into perspective.

</comment><comment><USER3> This was already invented 13 years ago: <http://www.deathclock.com/>

It's the first hit on google for death clock.

<comment><USER4> He wanted a physical clock. Not an app or website.

<comment><USER3> To be fair, his post never mentioned a physical clock until he edited.

<comment><USER5> "With this clock right on my desk..."

'Physical' is a fairly safe assumption from that phrase. Yeah, it's still an assumption and you are technically correct.

</comment></comment></comment></comment><comment><USER6> Find some life expectancy stats and build the clock with an Arduino board.

[http://www.google.com/publicdata?ds=wb- wdi&met=sp_dyn_le...](http://www.google.com/publicdata?ds=wb- wdi&met=sp_dyn_le00_in&idim=country:USA&dl=en&hl=en&q=life+expectancy+statistics)

</comment><comment><USER7> One aspect to make sure to take into account is that it has to count down at less than real-time speed, since conditional-with-age life expectancy increases as you age (e.g. if your life expectancy at age 60 is 12 years, your life expectancy at age 71 is not 1 year).

<comment><AUTHOR> True. You could always adjust it and it would still serve it's main purpose.

</comment></comment><comment><USER8> Already Invented for those who plan to die in the next 27 years or so: <http://www.brgprecision.com/products/timers/simpletimer.php>

</comment><comment><USER4> Trivial with an Arduino. He could even build it himself in a weekend.

</comment><comment><USER9> Professor Farnsworth has invented this at least twice. It's occasionally off by a few seconds, though, what with "free will" and all.

</comment><comment><USER10> This is all very well but if I unexpectedly get knocked down dead by a bus I will want my money back.

</comment><comment><USER11> Have you seen <http://lifepath.me>?

<comment><AUTHOR> No. What is it?

<comment><USER11> It allows you to create a page like this: <http://lifepath.me/kyle>

</comment></comment></comment><comment><USER12> knowing me, if I didn't think today may be my last day on earth, I might just end up procrastinating :-P So, death clock based on life expectancy would probably have a negative effect on others like myself :)</comment>


I Was Able to Buy an AR-15 in Five Minutes - <AUTHOR> http://thetab.com/us/2016/06/14/i-was-able-to-buy-an-ar-15-in-five-minutes-19833 <comment><USER2> "Orlando gunman used Sig Sauer MCX, not AR-15"

[http://www.mercurynews.com/crime- courts/ci_30016806/orlando-...](http://www.mercurynews.com/crime- courts/ci_30016806/orlando-gunman-used-sig-sauer-mcx-not-ar)

Just for the record.

</comment><comment><USER3> He failed to mention in his piece that he had not gone through the federal background check process, which is the 5 page document. The gun dealer still has to enter this into their computer system and wait for a response from NICS.

[https://www.fbi.gov/about-us/cjis/nics](https://www.fbi.gov/about- us/cjis/nics)

This could take a few minutes to a few hours, depending on how busy the system is.</comment>


Shaun Wylie  - <AUTHOR> http://www.telegraph.co.uk/news/6389667/Shaun-Wylie.html <comment><USER2> Another giant departs.

Some time ago I discovered that one of the advisors for my undergraduate degree was a good friend of Turing. I never knew, and now that I finally do know, it's too late to talk with him about it. It's too late to let him talk about his time and accomplishments with someone who would appreciate them.

Now I discover that my Ph.D. great-grandfather was personally invited by Turing to join the team at Bletchley, and was regarded by many as second only to Turing himself.

My past is disappearing, often without my even knowing it existed.

So what about you? Who are the giants in your background? Did you go to college/university? Were you lectured by someone apparently indistinguished? Are you sure they are complete unknowns?

Most people have a story to tell, and many, perhaps most, are worth listening to. But sometimes there are stories that need the right kind of audience.

Might that be you?

EDIT:

I've been down-mud for this comment, and I'm not really surprised. I was in two minds as to whether to submit it, since it's not really Hacker News. But then, neither is the original submission.

Think of this as a comment that you should eat right and exercise properly, because otherwise your body won't work well enough to support your brain. In a similar vein, get to know the giants that surround you, otherwise you lose the opportunity, and might fail to learn things you later value.

Just my $0.02.

<comment><AUTHOR> I studied under C J Rallis (thermodynamics). He was an extraordinary man and my only regrets was that I did not spend enough time with him. He was born in Mozambique and taught at the University of the Witwatersrand for many years. He was the only engineer that had a Thermodynamic Cycle named after him since the 19th century.

I could write a book about my experiences with him and how we finally produced an engine that had only one moving part!

>My past is disappearing, often without my even knowing it existed.

This reminds me of a friend that is fond of saying we are becoming ... yesterday's people! Luckily with the internet now we may leave something behind us that future generations may or may not find interesting:)

<comment><USER3> > I could write a book about my experiences with him and how we finally > produced an engine that had only one moving part!

That sounds like a fascinating story. Please consider writing about it.

<comment><AUTHOR> Thanks for the encouragement :)</comment></comment></comment></comment>


Paradoxes of Probability and Other Statistical Strangeness - <AUTHOR> http://quillette.com/2017/05/26/paradoxes-probability-statistical-strangeness/ <comment><USER3> For those who might be interested, and in a slightly different vein than the examples in the article, there's the "sleeping beauty" paradox: [https://en.wikipedia.org/wiki/Sleeping_Beauty_problem](https://en.wikipedia.org/wiki/Sleeping_Beauty_problem)

Basically, an agent is put to sleep and told they will be woken up once or twice, depending on the results of a fair coin flip, without the ability to remember other awakenings.

What probability does the agent assign to the event that the coin landed heads?

The intuitive response is 1/3, but this poses obvious epistemological problems. The agent has, ostensibly, no new information at all, and their prior is surely 1/2\. Hope someone else finds this as interesting as I do!

<comment><USER4> I mostly find it interesting in that people could think that the chance is 1/3 (and that it may even be obvious!). After reading the description I can understand what they are getting at, but I think the conditional probability is messed up.

Instead of P(Monday | Heads) = P(Monday | Tails) = P(Tuesday | Tails) it is really P(Monday | Heads&Awake) = P(Monday | Tails&Awake) = P(Tuesday| Tails&Awake) or something like that. But the interviewer isn't asking about that, they are asking for the probability of the coin. The 3 positions are only exhaustive given that you are awake to be interviewed about them, not exhaustive of possible states (it's missing P(Tuesday | Heads&Asleep)). Since you're always awakened at least once, I find the argument that being awake has 'given you information that it is not tuesday AND heads' is pretty weak. While true, both heads and tails expect to be awoken while it is not both tuesday AND heads.

<comment><USER5> The outcomes can be easily enumerated. If Sleeping Beauty always answers "Heads" she will be right 33% of the times she is asked. This is pretty close to the definition of a 33% chance.

She hasn't been given new information by waking up, she also knows as she goes into the experiment - "most of the outcomes where I am being interviewed involve the coin toss coming up tails".

</comment></comment></comment><comment><USER6> By far the most unintuitive paradox for me personally is the one presented here: [https://youtu.be/go3xtDdsNQM?t=3m27s](https://youtu.be/go3xtDdsNQM?t=3m27s)

"Mr. Jones has 2 children. What is the probability he has a girl if he has a boy born on Tuesday?" Somehow knowing the day of the week the boy was born changes the result. It's completely bizarre.

<comment><USER7> I'm an idiot, but I'm going to throw my hat in the ring here:

The video is wrong. The problem reads: Jones has 2 kids. What is P(he has a girl) given that he has a boy born on a Tuesday. Consider, for a moment, what information we're getting from "boy born on a Tuesday." This is no different than "boy with red hair," or "boy with 5 freckles." The fact that the BOY was born on a tuesday does not change P(day of the week girl was born). Imagine the "boy with 5 freckles" case - let 5 freckles be denoted by F5, six freckles by F6 and so on... would the appropriate calculation include enumerating P(boy F5, boy Fn) for all n? No.

The "born on Tuesday" is irrelevant. Thus you have the following scenarios: \- one kid is TuesdayBoy and the other is also a boy, born at any time \- one kid is TuesdayBoy and the other is a girl, born at any time

Out of these options P(Jones has a girl) is a flat out 50%. There is no need to bring in concepts of "which was born first" or enumerate all possible days of the week each child could have been born.

Ok... now all the real smartypants here can correct me :)

<comment><USER8> There are 2 * 7 * 2 * 7 ways to assign gender and birth-day-of-week to two children. By convention, all are considered equiprobable (this is the same as assuming kids' genders and birth day-of-weeks are independent of each other and of all facts about other kids, and that both genders are equally likely and all 7 days are equally likely for any given kid.)

Of these possibilities, 27 are situations where one kid is a Tuesday boy. [Do you dispute this count?]

Of those, 14 are situations where one kid is a girl. [Do you dispute this count?]

The answer to "What proportion of cases where there is at least one Tuesday boy also have a girl?" is thus 14/27.

You have stated by fiat that certain things are irrelevant to certain other things, that certain things have probability 50%, etc, but in doing so, you have not considered the count correctly. You are likely misled by phrasing such as "the boy", when there are families with two boys in which there is no proper referent of "the boy" and no particular answer to question like "Which day was 'the boy' born?".

<comment><USER7> ok ok... let me try to get this straight. Just as kind of a mental process for trying to understand whether or not something passes the smell test, I typically try to take the basic premise and turn it up to 11 and see if that still makes sense.

In this problem, as you've described it, we're enumerating "ways to assign gender and birth-day-of-week." We can do this because there are a _countable_ number of "days of the week" (so we can map to the integers: 1-6) AND there is also a surjective function of [child] -> [day of the week they were born]. Am I right so far?

Now let's replace the set [1-6] with another countable set that also maintains the surjective function. We could say "day in the lunar cycle" (so ~27 options), or better "day of the year" (366 options), for example. Do we now need to consider the 2 _366_ 2 _366 ways to assign gender and birth-day-of- the-year? Take it further with whatever you want: "birth weight in milligrams" or "number of freckles" (as I previously suggested). All countable things that meet the surjective requirement.

This is starting to smell funny, right? So let's take a look at the math.

You say there are 2_7 _2_ 7 ways to configure day+gender, assuming independence for kid 1 (k1) and kid 2 (k2). This represents: (k1 gender options * k1 day of week options) * (k2 gender options * k2 day of week options). Right? I'm with you so far. Then you say "Of these possibilities, 27 are situations where one kid is a Tuesday boy." Hold up.

We are given two pieces of information: that one of the kids is a boy, and that particular boy was born on a Tuesday. Let's say the boy is k1 (this is an assignment of enumeration, not of "who came first;" just like Sunday = 1 does not mean that any kid born on a Sunday was born before every kid born on Monday = 2). So now the k1 options are [1 _1] (boy, tuesday), and the total number of options are: [1_ 1] * [2 _7] = 14. Of those 14, 7 are girl options. And we 're back to a straight 50%.

So yes, I dispute the 27 number. It seems like it is arrived at by 2_1 _2_ 7, minus one for an apparent duplicate. But the 2 _1_ 2*7 represents maintaining gender non-specificity for Tuesday boy, which should be incorrect, no?

> You have stated by fiat that certain things are irrelevant to certain other > things...

Yes, but that's what "independent" means, right? You also stated that you're assuming these two things are independent, hence equiprobability. But independence is defined by P(A) = P(A|B). The probability of A is completely unaffected by B. Yet the outcome you arrive at is that P(A) IS affected by B, so the math presented is internally inconsistent.

What am I missing here? I'm fascinated by the uncertainty around this little problem.

<comment><USER9> Let's see if I can help you understand this a bit better. First, let's clarify the problem being asked. There are 2 different problems with different solutions and it helps to explicitly separate them.

problem 1) You go up to a person and ask them if they have exactly 2 children, at least one of which is a boy born on Tuesday. They say yes. What is the probability that they have a girl?

problem 2) You go up to a person and ask them if they have exactly 2 children, at least one of which is a boy. They say yes. You then ask them which day of the week a boy they have was born on. They say Tuesday. What is the probability that they have a girl?

The original problem that was posed is equivalent to problem 1, but not equivalent to problem 2. This could be what is confusing you, because in problem 2 the extra information plays no role in the selection process, while it does play a role in problem 1. In problem 2, the answer is the standard 2/3\. Why are the probabilities different between problem 1 and 2? Here's why:

Think about the set of people who could answer yes to the question in problem 2. The ratio of these groups is important. A parent with BB (two boys) is equally likely to answer yes to problem 2 (100% likely to be exact) as a parent with BG and GB (also 100% likely to answer yes), which leads to the correct solution of 2/3\. However, in problem 1 a parent with BB is NOT EQUALLY LIKELY to answer yes as a parent with BG. This is because we added an extra qualifier (must be born on Tuesday). The parent with BB has two chances to meet this qualifier because they have two boys, so the parent with BB is actually more likely to answer yes to the question than the parent with BG. As the qualifier becomes more and more rare (day of lunar cycle), the probability of the BB parent answer yes P(yes|BB) approaches twice the value of P(yes|BG). So now you're left with some subset of parents with BB, BG, and GB, but in this scenario you've sampled from BB approximately twice as much as you've sampled from each of the BG and GB groups, leaving you with approximately the same number of people from group BB as the combined amount from groups BG and GB. This is why the probability approaches 50%

I spend a while writing this, so hopefully it helps!

<comment><USER7> :) Thanks for taking the time. I've realized a few things, and found it helped to get a bit more formal.

Jones has 2 kids. Let A be "he has a girl" and B be "he has a boy born on tuesday." First thing I realized is A and B are NOT independent - this is key. P(A) includes the option of Jones having two girls. But if B is true, then the two girls option isn't on the table anymore, which affects P(A). Realizing this helped me start to better understand what kind of problem we're dealing with.

Second was realizing that P(A&B) is not at all the same thing as P(A|B) - <USER2> probability of A _given_ B - when A and B aren't independent. The problem is asking for P(A|B), and by the rule of conditional probability: P(A|B) = P(A&B)/P(B)

P(B) can be solved for without too much fuss: solve 1-P(!B). For each kid you have 2 genders and 7 days of the week, or 2 * 7 = 14 options. 13 of those are not "Boy & Tuesday." So you have P(!B) is (13/14) * (13/14) = 169/196\. P(B) = 1 - 169/196 = 27/196.

This leaves us trying to figure out P(A&B). I can't think of any other way to do it other than enumerating all options. We can take a shortcut and just look at all 27 possible scenarios where B is true. This seems to be the method of choice ;) As others have shown, we see that 14 of those satisfy A. So P(A&B) = 14/196.

Now, we can solve: P(A|B) = P(A&B)/P(B) = (14/196)/(27/196) = 14/27

So I'm now part of the "math checks out" club. Thanks for all the help people!

</comment></comment></comment></comment></comment></comment><comment><USER10> Another "paradox": even though it's possible to randomly pick a rational number from the reals, the probability of this happening is 0.

<comment><USER11> Please describe how it is possible to pick such a number. For example, I can readily imagine how to pick a random 32b float, but that it is an entirely problem with a nonzero probability.

<comment><USER10> Usually in math we assume the axiom of choice :) [https://en.wikipedia.org/wiki/Axiom_of_choice](https://en.wikipedia.org/wiki/Axiom_of_choice) I'm assuming this could somehow lead to such a "random" pick in the technical sense.

In terms of implementation, I'm not aware of an algorithm that can randomly pick a real number on an actual computer. Perhaps a mathematician could show how to pick one on some abstract machine with infinite resources, and not constrained by finite bit representations of numbers.

<comment><USER12> > In terms of implementation, I'm not aware of an algorithm that can randomly > pick a real number on an actual computer

An actual (finite in time and space) computer can't even _represent_ arbitrary real numbers, much less _randomly choose_ them.

<comment><USER13> What about PI? We can represent it in terms of "we know what we are talking about" and we can distinguish it from other numbers.

<comment><USER14> You can only have countable number of first or second order logic statements each defining a specific real number.

</comment></comment></comment></comment></comment></comment><comment><USER15> My favorite statistical/probability paradox has always been the birthday paradox.

<comment><USER16> I don't know if Monty hall problem counts as a paradox, but that is quite high on my favourite list of counterintuitive probability results.

<comment><USER17> In my experience the only reason the Monty Hall problem comes off as paradoxical is because it is usually poorly explained.

</comment></comment></comment><comment><USER18> [https://en.wikipedia.org/wiki/Category:Statistical_paradoxes](https://en.wikipedia.org/wiki/Category:Statistical_paradoxes)

</comment><comment><USER19> "Paradox" is a pretty strong term. The items presented are more in the category of common errors and counter-intuitiveness.

<comment><USER6> [https://en.wikipedia.org/wiki/Veridical_paradox](https://en.wikipedia.org/wiki/Veridical_paradox)

<comment><USER19> Fine, but I was left feeling blah by the "paradoxes" presented.</comment></comment></comment>