OpenSSL's squaring bug, and opportunistic formal verification - <AUTHOR> http://kryptoslogic.blogspot.com/2015/01/openssls-squaring-bug-and-opportunistic.html <comment><USER2> The proof appears to be incorrect.

Consider the case where c2,c1,c0 are all UINT_MAX.

In OpenSSL that can't be the case because the outside environment constrains the accumulator; but I see nothing on that page showing such a constraint there.

I'm not familiar with the z3 python interface, clearly something interesting implicit must be going on there. (As it does appear to be expressing that the values are finitely ranged, and doesn't appear to be expressing a constraint for ref)

In general I've had bad luck getting results out of SMT solvers on finite ranged problems-- often they just get stuck. If you can ensure there is no overflow then its easier, but then again it's also easer to reason about them absent the mechanical prover in that case too.

The other major hurdles is extracting the code in a way that can be usefully presented to solvers without a lot of potential for human error or extreme effort (e.g. OpenSSL is 400k lines of code, good luck with making progress if you're hand rewriting things for the prover). Frama-C ([http://frama-c.com/](http://frama-c.com/)) can help with this and I've used it to good effect on very small pieces of code.

Hopefully in the future we'll see more programming languages with better accordances for using formal methods-- ATS is pretty interesting in that respect--, and also bridging the usability gap enough to see them widely used in industry.

<comment><USER3> The proof is implicitly modulo 2^96, which I believed to be the intention of the original code, but it does miss a possible overflow when c is too large, which shouldn't happen in its intended use: c2 is already meant to be the overflow limb. I've added an addendum to the post describing, and fixing, this.

<comment><USER2> Thanks!

One of the things I find useful for avoiding overconfidence in proof results, beyond making sure to test the proof on multiple broken versions, is to spell out many additional implications of the function that I and/or the caller might expect of it and ask for proofs on them too, even if they seem somewhat redundant.

For example, cout = 2 _a_ b + cin; I would also prove that for all a,b,cin that cout >= cin; for a or b ==0 that cout == cin, etc.

</comment></comment></comment><comment><USER4> Beautiful. Formal verification does take more time over empirical testing methods such as unit testing, but when it comes to mission critical components such as cryptography or system code, this is time well spent.

<comment><USER5> edit: just to be clear, I'm responding to this statement:

> _Formal verification does take more time over empirical testing methods_

I don't even know if that is true, at least, I'm unconvinced about it being true in hardware verification. The amount of manpower and effort companies like Intel and AMD put into simulation-based verification is insane and a lot of that could be replaced (and has in some case been replaced) by a well- designed formal assertion suite.

The real bottleneck is people. It's a lot easier to find competent engineers who can hack around with tests and simulation suites than it is to find people who can hack around with abstractions for scalability in a model checker.

<comment><USER4> There is a pragmatic perspective and a theoretic perspective. Pragmatically speaking, empirical methods can get one pretty far. It's better to have full coverage of empirical tests across an entire system than limited coverage of formal tests, for instance. So, on that account, you may be correct from a pragmatic perspective.

From an epistemological perspective, a test suite to confirm a property using chosen examples is not nearly as good as a formal proof that guarantees a property over a given domain. For simple cases, such as branch coverage, the test suite may be good enough. As complexity increases, or when testing things like modular arithmetic over an elliptic curve, unit testing leads to false confidence.

But, just like in empirical testing, it's possible to build bad proofs. This is where the specialization comes in. There may be plenty of competent engineers who can hack at a test suite, but their results will not be nearly as comprehensive as someone trained in formal verification.

It all comes down to epistemology. What do we know, and how do we know it? What possible defects exist in our system, and how bad can they be? Can we ship yet?

<comment><USER2> Any kind of testing or review can lead to false confidence. This is why its important to use multiple approaches in the hope that their errors are not completely correlated.

<comment><USER4> I agree.</comment></comment></comment></comment></comment>


Morocco moves to ban plastic bags - <AUTHOR> http://www.huffingtonpost.com/entry/morocco-ban-plastic-bags_563934cde4b0307f2cab21a8 <comment><USER2> Based on the primary photo of refuse in the article the problem is not plastic bags but garbage in general. My city (Portland, Oregon) outlawed plastic bags years ago and messy parts of the city are still messy while clean parts of the city are still clean. The bag-ban made no difference. Visiting neighboring Beaverton, where plastic bags are still allowed, one is not overwhelmed by a mass of plastic bags in the streets...

It has always been my understanding that if a store is going to provide disposable bags plastic bags are far superior - functionally and environmentally - to the alternative (generally paper):

[http://www.appropedia.org/Paper_versus_plastic_bags](http://www.appropedia.org/Paper_versus_plastic_bags)

<comment><AUTHOR> Problems with plastic bags:

1\. clog up storm drains.

2\. clog up the screens that water treatment plants use to filter out large debris.

3\. degrade into micro particles that treatment plants can't easily remove.

4\. mistaken for floating jellyfish by seabirds and fish and they get eaten and block the digestive tract.

5\. degrade into microparticles which reenter the food chain.

6\. plastics bond with other nasty chemicals to form even more toxic compounds

</comment></comment><comment><USER3> I love Morocco but I've taken a bunch of sad photos there of huge masses of carrier bags in otherwise idyllic spots. I'm really happy to hear this news.

</comment><comment><USER4> Way to go, Morocco!!!

</comment><comment><USER5> Bravo!</comment>


How to Be Jason Bourne: Multiple Passports, Swiss Banking, and Crossing Borders - <AUTHOR> http://www.fourhourworkweek.com/blog/2009/03/03/how-to-be-jason-bourne-multiple-passports-swiss-banking-and-crossing-borders/ <comment><USER2> This book sounds like it's entertaining, but laughably out of date. He's saying you should hide your money with AIG? The company that's losing money faster than the US government can bail it out? Then move to Iceland to escape the depression? Isn't Iceland the country that's been hit hardest, with the government now completely bankrupt?

<comment><USER3> Yeah, those are definitely not the best ideas right now, but I would think that he uses AIG the same way we use kleenex to mean tissue. It's the concept there.

Same with Iceland, that may not be the country of choice anymore, but the concept remains the same.

Obviously not ideal, but not a reason to ditch the book. Strauss' writing is entertaining, so I'll probably pick it up for the enjoyment factor :)

<comment><USER4> I want to echo that. I've been a fan of Strauss' writings for a long time now. I wasn't a fan of Marilyn Manson, but reading about the two years Strauss spent on tour with him was awesome. I never intended to become a PTA, but learning about his movement in the PTA circles as "Style" was awesome. His writing delivers genuine enjoyment.

<comment><USER5> PTA being...?

<comment><USER6> I wanted to make a funny remark, but Urban Dictionary is way better:

<http://www.urbandictionary.com/define.php?term=pta>

BTW I think the original term is PUA = Pick Up Artist:

<http://en.wikipedia.org/wiki/Pickup_artist>

<comment><USER4> Yeah, sorry, I did mean PUA for "Pickup Artist".

But since you already corrected it for me, I'll leave it. And give you my upvote, sir.

</comment></comment></comment></comment></comment></comment><comment><USER3> Neil Strauss is a writer. His last book was about his descent and experiences once he got into the world of pick up artists, guys who follow a system to pick up women.

Ferriss sells hope about the better life, living a life that would normally be above your means. The secret agent life is part of that dream.

<comment><USER7> My impression was that "The Game" was credulous and a bit sleazy. Am I wrong? Is Strauss actually a good writer/reporter?

<comment><USER8> If you're asking about the quality of his writing, it's up to snuff. That said, having read his last book and excerpts from this one, it's probably not fair to try and compare him to a journalist.

Neil chases stories and ideas that are both a little crazy and secretly appealing to most men. Let's face it, all of us guys have wished at some point or other that they could seduce any women they wanted. In the same way, a lot of us have probably wondered what it would be like to live a spy lifestyle, or to have bank accounts in exclusive foreign countries. This is basically just the equivalent of a harlequin novel, but for guys.

<comment><USER9> Seriously, _harlequin novels_? That implies that books like these are both a dime-a-dozen and uncreative in addition to being fictional - nothing could be further from the truth. Strauss' writing is interesting precisely because it's about really crazy subjects that no one else writes about, or at least where the existing literature is aimed at very narrow audiences. I don't know about the "quality of writing"; this seems to be a quality you can define to mean just about anything. I agree that some of Strauss' plot devices are pretty lame and predictable - you'd probably not want to discuss his technique in a literature program. And like all supposedly "real" novels, there's probably a decent amount of distortions and exaggerations. But his books are engaging, and very much in the direction of "satisfy intellectual curiosity" that I love. No one is supposed to be manipulating social situations like the characters of "The Game", and yet it happens.

I found it eye-opening in a lot of ways, and it brings a host of new questions: what is "genuine" social interaction? How much of people's personality is learned and internalized behavior? How much of sexual attraction is? Should you try to change your personality for something that is supposedly better, and can it be done? In relation to dominance/submission in social situation (the implicit structure of command and leadership that automatically arises when a group of people meets), how much is genuine (or "natural" - what does "natural" even mean?) and to what degrees do natural leaders actively manipulate the people around them?

You shouldn't read books just for some supposed literary value, the real value comes from the questions you have when you're done.

</comment></comment></comment></comment><comment><AUTHOR> Is this what "jumping the shark" means?

While I can totally relate to the adolescent fantasy of having super-fugitive skills and money stashed all over the world, it seems wasteful if folks take it too seriously.

If this guy had put all this effort and cleverness into a startup rather than paranoia, he would probably have made off quite nicely. Who knows, maybe he will with his book and that was the whole point.

It's funny to see this kinda stuff coming from Tim Ferriss though. Even though his stuff can be pretty cheesy - I love the usual emphasis on simplicity, experimenting, and outsourcing. While a post like this drives traffic, it seems counter to his "message".

<comment><USER10> Ferriss basically says "This stuff is too extreme for me to actually do, but it's freaking awesome to read about."

</comment></comment><comment><USER11> This article certainly has interesting ideas, but I think the author is wildly overplaying the risks of the current economic situation, and is indulging in a bit too much fantasy. I mean, submarines? Really?

<comment><USER12> "I mean, submarines? Really?"

Yeah that's the point at which I would have declared the billionaire to be a complete fucking nutbar, and would decide to immediately stop trying to replicate his behavior.

</comment></comment><comment><USER13> Neither Strauss nor Ferriss impress me, and though Ferriss offers some humorous ideas in his book, I would say that most of it is entertainment and self-promotion.

</comment><comment><USER14> "When Tarasov was through explaining everything, I couldn't tell whether I was protecting myself from being scammed or actually being scammed myself."

Count me out!

<comment><USER15> Although from looking at the diagram he was referring to it looks pretty straight forward. Split up each type of asset into a separate LLC, all of them owned by a parent LLC that is controlled by a trust. (House is separate since it's more of a liability, the author still owed money on it and there is always a risk someone will trip and fall on your property). Nothing about this setup looks at all unusual or suspicious to me. So I'm either missing something obvious or I need to get in the business of hiding other people's assets.

</comment></comment><comment><USER16> Re: Money laundering

I once heard that some, if not many, of the Chinese all-you-can-eat-buffet restaurants are/were fronts for the Chinese mafia. Makes you wonder - as a _lot_ of them go out of business in 6 months - 1 year.

<comment><USER17> Urban legend. A lot of _all_ restaurants go out of business in six months to a year. It's a brutal industry.

<comment><USER18> I don't know much about Chinese Restaurants, but I do know a good bit about nightclubs, and virtually all of them (in major cities at least) launder organized crime money as a primary business.

<comment><USER16> On the subject of losing money, anecdotally, two of the coffee shops I frequent are owned by successful business owners (e.g. one of them owns multiple laundromats, other has a portfolio of rental apartments).

My hunch is that they use the _losses_ from the coffee shop to offset their profits at their other cash-flow businesses. Is this a common practice (if it is so)? I know Microsoft's Office and Operating System businesses effectively subsidize the rest of the company (Internet, etc.)

This has been posted here before but "My cofeehouse nightmare" is a good read <http://www.slate.com/id/2132576/>

<comment><USER19> I don't get it. Aren't you supposed to use profits to offset losses?

Wait...you mean cooking the books and making it seem like both businesses are doing just okay so they're taxed less...right?

<comment><USER20> Why would "cooking the books" be necessary? Structure the assets in holding companies properly and they should be able to do it legally.

</comment></comment></comment></comment></comment></comment><comment><USER21> I'd read it as a fictional novel, and for 10$, I feel like it is.

</comment><comment><USER22> Wait, so this isn't a joke?

<comment><USER23> it's not. outside of the U.S. there are numerous situations that would require you to have a second passport - the one I hear the most are people who travel into Middle East countries but require them also to to Israel. You can basically branch off from that situation to other "real" situations.

</comment></comment><comment><USER24> A little recession, it seems, is all it takes for do good hackers to start entertaining the dark side. A pity.

<comment><USER25> I'd much rather learn how it works so I can identify, avoid, or fight. Which are all preferable to me then ignoring it. Criminals are hackers in their own right (never mind criminal hackers, but society hackers, they make society do things it was never intended to.) and I have to say reading about a good scam is almost as entertaining as reading about a good startup. One just has more practical value.

<comment><USER26> > One just has more practical value. Hmm, you're probably right, but as I am > not (yet) thinking of creating a startup, it is debatable which one it is...

My own philosophy is that, if the feces doth impact the rotary blades, I'd rather minimize the damage as much as possible. Most might consider it paranoia; I consider it an ingrained distrust of about 90 percent of humanity. As they say, the best defense is a good offense.

</comment></comment></comment><comment><USER27> There's a sample chapter that you can read.

Link:<http://www.inquisitr.com/19245/neil-strauss-emergency/>

</comment><comment><USER28> If only it was that easy.

</comment><comment><USER29> What does this have to do with Hacker News? Why are people voting this up?

Come on guys.

<comment><USER30> Why does your concept of hacking begin and end at writing code? This book describes a number of real-life hacks: social engineering, exploiting legal vulnerabilities, circumventing security devices, etc.

<comment><USER29> Considering how popular your remark is; it is safe to say that people around here think "Hacker News" is about cracking. I had thought it was about building things, silly me.

<comment><USER30> Sometimes hacking and cracking coexist. Examples include the reverse engineering of CSS (DVD encryption), figuring out how to jailbreak an iPhone, and even more questionable acts like writing a virus to innoculate systems against other viruses. These cracks required technical proficiency, real thought and good, old-fashioned effort.

Sometimes cracking is just mindless. I'm guessing most people here don't respect that. Had the book been a series of "...and then I paid this dude $N and he handed me a new passport", I don't think it would have hit the front page.

<comment><USER31> I've always considered "hacking" relating to making a system do something it wasn't intended to do. You have to know the system well enough and be creative enough to go beyond its perceived limitations. I get a kick out of being creative; others get a kick out of having power over something. In either case, the real thrill is feeling alive and not feeling like a mindless robot.

It is funny we're having this conversation, considering Jason Calacanis's open letter, "Why I hired a felon" relating to one of his employees at Mahalo: <http://calacanis.com/2009/03/05/why-i-employed-a-felon/>

</comment></comment></comment></comment></comment><comment><USER32> This guy Ferriss is a crackpot.</comment>


Show HN: Unhackable Modular Home Automation Kit - <AUTHOR> http://www.ELFblocks.com

<comment><USER2> Why did you describe this as unhackable?

I'm not too into the video, especially since it doesn't even show the blocks.

That scrolling reveal interface might not be the best fit for efficiently describing the features of your blocks. This more details page, though, is much better.

[http://www.elfblocks.com/more#.VXtm3hNVhBc](http://www.elfblocks.com/more#.VXtm3hNVhBc)

Just use that instead. If your call to action is to sign up for a mailing list...I'd put it above the fold.

When is this going to be available? It would be nice to know more about the team behind this...maybe an about page?

<comment><USER3> > Why did you describe this as unhackable?

I think they mean 'uncrackable':

        Unhackable by design

        When it comes to our home, the security standard we're using on our computers aren't enough. The security has to be dramatically improved.
        That's why we created the world's most secure home automation kit.

        ELFblocks are designed to use read-only memory, which means they can't be hacked, since there is nothing that can be changed by the attacker.

(from [http://www.elfblocks.com/more#.VX52g_mqpBc](http://www.elfblocks.com/more#.VX52g_mqpBc))</comment></comment>


The Agile Labour Union - <AUTHOR> http://westspacejournal.org.au/article/summer-2013/the-agile-union/ <comment><USER5> I couldn't make heads or tails of it. There's an interesting point about Lean development coupled to Taylorism, but I saw no ties with how agile is tied to unionization.

The article says: [Agile] does this by limiting the number of working hours, demanding autonomy and self-management, rejecting the arbitrary dominating power of the boss, preventing their jobs from being shipped overseas, and ensuring that software engineers are involved in their workplace as decision makers.

1) "limited working hours" \- the essay quotes Beck on prohibiting more than 40 hours of work per week, and says of crunch time that "Agile methodologies rightly reject this as inhumane and unsustainable." However, waterfall methodology _also_ rejects this as unsustainable. The reference I have is for Jones 1991, which says that Voluntary Overtime is effective up to about 44 hours per week, and beyond that effectiveness is likely to get worse.

So, you're doing agile. Your boss wants mandatory overtime, with 50 hour weeks for the next 3 months, in order to beat some deadline promised to the board. In a union, you talk with your union rep (who may have been involved in your contract), and present a united front against the arbitrary and fundamentally poor decision.

In an agile shop you ... point to a book which says that 40 hours is enough? Then when your boss points out that the industry average is 48-50 hours, you say ... ? And you are effective because ... ?

2) "rejecting the arbitrary dominating power of the boss"

I didn't see where that was really developed, other than some specifically project related decisions. Can agile prevent the boss from firing an employee any more than waterfall can do so? Can agile fight towards getting the salary increase you were promised for working 60 hour weeks, but never got? Not that I'm aware of.

While unionization is supposed to offer that sort of protection, agile only rejects certain limited forms of boss power.

3) "preventing their jobs from being shipped overseas"

The point here is that outsourcing can't happen with a methodology which requires co-location with the customer. "The forth and fifth principles mandate co-location of the engineering team and daily interaction with business people" where from the Agile Manifesto: "4) The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. 5) Business people and developers must work together daily throughout the project."

Except that those principles don't require on-site work, only say that on-site is the "most efficient and effective method." There are any number of web page which talk about "distributed agile", for example.

For example, here's a page talking about how to build an agile off-shore [http://www.agilebuddha.com/agile/distributed-agile10-good- pr...](http://www.agilebuddha.com/agile/distributed-agile10-good-practices-of- successful-teams/) . It argues that frequent travel for in-person contacts is essential, and still requires daily meetings, only through videoconferences and other means.

While agile would say this not as effective, it doesn't reject this outright, so it cannot be anti-offshoring.

<comment><USER6> Loomio's comment gets the gist of it. The software creation process has been moving to Agile, which beats waterfall, but there's an emerging practice of UCD. If programmers cannot find some way to work with and within UCD, a division of labor between designing the product with the end user, and writing the code to implement it will develop. This will lead to declines in programmer autonomy, and will demand changes to Agile. The seed for this change is already planted within Agile by people advocating for Taylorism.

That's not my thinking - just my interpretation of the text.

<comment><USER5> Thanks for highlighting that. I at first found the UCD reference even less understandable than the starting thesis comparing agile to unionism.

I agree, btw, with the assessment that Agile is "naïve in assuming that a new system's customer ... can adequately represent the goals." Agile seems designed for the admittedly large number of in-house development efforts, where end-user and paying customer are very similar. It does not have anything like persona development from UCD. Indeed, I am annoyed with agile's papering over of the term "customer."

I agree too that software development methodologies have not, in general, been influenced much by UCD. I say from reading in 1990s the ideas promoted by Raskin, Cooper, Norman, Nielsen, and Tognazzini, and watching the jump and seemingly rapid decline in interaction design over the last 10 years. Based on that, I don't see UCD as having emerged any more now than it was 10 years ago.

But since I don't see Agile as a sort of unionization which separates programmers vs. non-programmers (with Waterfall the stand-in for Taylorism), then the rest of the comparison doesn't hold together. It feels more like someone who doesn't know what unionization does.

Focusing on the idea that early Agile is a non-Taylorism style, I recall that even early on, CRC cards were associated with an "IdealTime." People wanted to turn that into real time, and determined the load factor through XpVelocity. According to Fowler though, StoryPoints (or StoryCounting) are "easier to calculate, less likely to be abused, and every bit as accurate."

Quoting c2.com: "Because C3 measures estimated and used IdealProgrammingTime, as well as elapsed time, I believe that it does have something to do with how many minutes per day are spent developing on the task. While I may be in disagreement with Kent on that, I'm not in disagreement on the utility of what's below. --RonJeffries"; C3 being the project which heavily influenced early Agile.

Thus, I think time estimate and measurement has _always_ been a part of Agile.

The author made a reference to "Alternatives to Lean Production: Work Organization in the Swedish Auto Industry". I was curious about that, since I live in Sweden, and Saab's nearly empty car factories lie on the other side of town.

That book was written in 1992, which is just a few years before the start of Saab's gradual decline. Volvo no longer has a plant in Kalmar, nor in Uddevalla, though they are still strong elsewhere. Searching now, there was a 2012 symposium titled "What have we learnt from Volvo Uddevalla?" [http://ilpc2012stockholm.wordpress.com/2012/03/25/symposium-...](http://ilpc2012stockholm.wordpress.com/2012/03/25/symposium- what-have-we-learnt-from-volvo-uddevalla/) . It says:

> Reflective Production, implemented by Volvo in its Uddevalla plant ... [was] > renounced [in the 1990s]. But some aspects of reflective production were > adopted by other companies, especially in Japan.

> Do we know better today the reasons for the development and subsequent > abandonment of Reflective Production? ... Are they specific to the Swedish > context or is it possible to find them elsewhere?

It's interesting then that while the author proposes that non-lean methods are possible, and quotes from that source, that non-lean method was abandoned. Perhaps the alternative approach is actually worse than lean?

FWIW, the author of that 1992 book, in [http://www.akesandberg.se/wp- content/uploads/downloads/2010/...](http://www.akesandberg.se/wp- content/uploads/downloads/2010/05/Enriching-Production_Ake-SANDBERG- ed.-1995-2007.pdf) , "criticizes Volvo's official justification of the closure and demonstrates the weaknesses of its calculation of cost savings", by comparing the Kalmar plant to the lean approach in the Gothenburg plant.

Moreover, the quote from Alistair Cockburn takes place at the Scandinavia Developer Conference, which is in Gothenburg, the home of Volvo. You would think that if Reflective Production were still influential then people there would know about it.

(Hmm, I should ask about it next time I'm in Gothenburg to talk with software people.)

</comment></comment></comment><comment><USER7> This quote summed it up for me:

> Agile processes are optimised for constructing technical systems conceived > and built by programmers, but poorly suited to building socio-technical > systems that seek an integration between humans and technology—a model that > requires software engineers to embrace a wider range of disciplines and > skills as equals, and adopt a more comprehensive vision of what it means to > successfully build software. Failing to make this transition is very likely > a death sentence for the Agile movement.

<comment><USER8> Reminds me of:

"Instead, the technical talent goes to work on elaborate frameworks, trying to solve domain problems with technology. Learning about and modelling the domain is left to others. Complexity in the heart of software has to be tackled head on. To do otherwise is to risk irrelevance."

\- From Domain Driven Design by Eric Evans.

Also in the original article is this:

"Except for the most senior software engineers, most of the creativity and innovation in software will move to specialised design and research activities using techniques that exceed the reductionist thinking that dominates computer science departments."

I think that part of the problem is that we need very few real computer scientists (but they are needed). However most people who want to build software are trained as computer scientists. The people who train them naturally teach them to value things that are useful to computer science research but this can cause them to be limited when it comes to solving real world domain problems holistically.

Or as patio11 says "Don't call yourself a programmer" [http://www.kalzumeus.com/2011/10/28/dont-call-yourself-a- pro...](http://www.kalzumeus.com/2011/10/28/dont-call-yourself-a-programmer/)

</comment></comment><comment><USER9> I am not sure if this falls into Agile or not (proponents seem to differ), but I don't think you can build a house just driven by customer requirements and _their_ priorities.

Imagine, how it would look like:

\- "Well, my number one priority is to have a shelter from elements.." OK, so we build the roof first.

\- "OK, now I would also like to stand in there." Uh, then, lift up the roof and build some walls around it.

\- "How do I get in? The trapdoor in the roof is too high now, I know there is a ladder, but.." Let's make some door, then.

\- "Right, but the floor inside is a bit dirty. Can you do something about that?" Sure, let's put some concrete inside.

Etc.

The reason why you can do something like this in programming is a) it's abstract, there is no extra cost for changing things, b) you do as much as possible modularly (OOP, FP, TDD...) anyway. But I can't really imagine any other engineering project without (at least a bit of) waterfall.

<comment><USER10> [http://brainslink.com/2010/12/be-agile-build-a-house- iterati...](http://brainslink.com/2010/12/be-agile-build-a-house-iteratively/)

Not that I agree that the linked process would be a good process for building the average house - Agile is only more efficient if there is a high risk of requirements changing or of being unknown. I (think) many people have a fairly set idea of what they want from a house, and have a finite amount of funding for it, where a functioning roof, but missing walls would be a major problem :)

That said I think these fit the bill for agile buildings:

[http://en.wikipedia.org/wiki/Sutyagin_House](http://en.wikipedia.org/wiki/Sutyagin_House) [http://www.telegraph.co.uk/news/worldnews/1544827/Gangster-w...](http://www.telegraph.co.uk/news/worldnews/1544827/Gangster- who-built-worlds-tallest-log-cabin.html)

[http://gizmodo.com/10-incredible-diy-castles-built-by- just-o...](http://gizmodo.com/10-incredible-diy-castles-built-by-just-one- single-pers-1452608410)

</comment></comment><comment><USER11> This article is all over the place...

They cite the transition to a service-based economy a good match for Agile (Which is a problem in itself). Unfortunately we don't live in that utopian vision. For example, it would be quite unreasonable for a contractor to change the design of a home in iterative cycles.

Agile is not a union. It just optimizes management in a specific direction at the expense of others.

<comment><USER6> The article says that Agile was a way to get what amounts to a contract of work rules, without the labor unions.

</comment></comment><comment><USER12> Zooming out, how does UCD protect against the basic risk of paying too much attention to the customers/users? It's like the old story of Henry Ford saying if he had asked his customers what they wanted, they would have said faster horses. It's the programmers/technologists that often have the insight of knowing what the customers _would_ want if they knew they could have it.

<comment><USER13> I think this is a very good point, and I think it suggests a resolution to the "antagonism" between developers and UCD practitioners suggested in the OP.

In designing a system, the developers are the ones with the best grasp of what is _possible_ , in both the positive ("here's a new way of solving this old problem") and negative ("that solution would not work because of _X_ ") senses. If the UCD practitioner is a true specialist, they will by necessity be unaware of the implementation details and even system architecture, which might cause them to either fail to spot a potential improvement or to suggest features which have poor cost-benefit ratios (even if the benefit is positive).

UCD practitioners don't just ask the users "what do you want?" (I'm a developer but I have done some user research as part of UCD projects). They ask "what are your problems?" or "what are your needs?", and then create mock- up solutions which address those needs, which are then validated with the customer (think "lean startup", if it helps). It's not that different from traditional agile, except instead of "the customer" you now have "the customers" and you do a lot of your early-stage iterating in wireframes and mock-ups.

This does create risks, of course. The UCD practitioner might propose designs that are unnecessarily difficult to deliver - yes, "the customer is always right", but the customer doesn't always want to pay enough to get what they're asking for, and it's the developers who will evaluate how much the designs are going to cost to implement. Or the UCDer might over-specify things, mapping out several months worth of development in wireframes that might imply major back-end work, with all of the waterfall delivery problems that entails - supposing that the back-end work takes longer than expected, would the customer prefer to cut some features or deliver later? If so, which ones? If the design reaches the developers as a single, cohesive package that cannot be broken down, we're back in waterfall-land and that's no good for anyone.

To my mind, this can be resolved within the classic agile framework of cross- functional teams. User research should be a function of the team, and the decisions about what solutions to propose to the users should be team decisions. The designer should not be proposing work that will be carried out by the developers without consulting them first, and the developers should play an active role in suggesting features and solutions - offering their knowledge of what's technically possible to inform the designs. This keeps the design process grounded in technical reality but also allows for the designer to challenge the developers to come up with a way of achieving something that customers want.

</comment></comment><comment><USER14> Well I loved it. I thought it covered a lot of interesting ground using the Agile == Union analogy.

There's some points I disagree with (UCD designers are not managers, so allowing them to design a system is not reverting to management control, and it seems to me the easiest solution to the proposed quandary is to consider UCD as part of the engineering process and the designers as part of the engineering team). But on the whole it says interesting things about the function of Agile in our workplaces.

<comment><USER15> I'm uncertain to what extent I can agree that agile and labor unions are similar. They have some superficial features in common, but I think the fundamental structure is radically different:

1\. A unionized factory worker performs the same sequence of operations over and over to achieve a standardized type of output. An agile software developer is generally combining operations in new ways: if it's the same thing over and over again, your developer is not very good at figuring out how to reuse code.

2\. The nature of the employer relationship is different, at least in today's economy. A typical software engineer is reasonably hard to replace and can apply skills at a variety of alternative employers. The best case for a labor union is when the opposite is true.

3\. I think the causal relationship is reversed. The reason that agile is adopted is not to protect programmers from off-shoring or long hours (typical union pitches). It is adopted because the firm is willing to believe that the co-located team with decent hours is a better deal. The union/management relationship may be starkly adversarial; the agile-team/management relationship far less so.

So if there's a UCD fad going around, I don't see it as capable of disrupting the "agile" ecosystem's employees' lifestyle the same way that some new process or external shock could disrupt a unionized manufacturing ecosystem's employees -- not unless it really fundamentally changes how a coherent piece of software ends up finding itself structured, and a couple of wireframes and the like aren't going to do that.

<comment><USER14> 1\. I think you have a strange idea of what unions are and do.

2\. ditto. The 'best case' for a labor union is that their members are highly skilled and hard to replace, with lots of alternative employers.

3\. The article clearly states that Agile is not implemented in order to protect developers, but the net effect is the same. Agile team/management relationships cannot be adversarial because Agile.

The point the article makes is that UCD separates the design of the system (the bit that needs management and user contact) from the implementation, which then allows the implementation (as wildgift points out) to be considered a cost and Taylorised (squeezed) into 'efficient' permanent crunch time.

<comment><USER15> Ah. My ideas of labor unions presented here are in fact strange, because they have been narrowed (for the purposes of this discussion) to the perspective of "the union exists to protect your interests against rampant Taylorism imposed from the management, in the same way that Agile is posited in The Article to protect programmers from something analogous." While labor unions' members may in fact be best off under a scenario such as you describe in yourpost#2, that's not a grand example of a case where you can say "we need a labor union to protect our interests from management or our lives will be made miserable by exploitation" and seems not to relate to the issues raised by the article.

</comment></comment></comment></comment><comment><USER16> I need to create a autohotkey or something to insert this everywhere where I see a comparison between Agile and Waterfall:

\- This content contains a false dichotomy between Agile and Waterfall, used as an argument to justify Agile. Every healthy person can imagine that iterations are often needed and sometimes even inevitable. In the real world, there isn't a situation with a sharp distinction between projects that prohibit iterations and projects that have iterations in all phases. Iterations are not invented in 2001 by a couple of guys thinking up a manifesto.

The term Waterfall is an imaginary enemy (maybe once used with good intentions to make a point clear) and Agile isn't a solution for all (software project) world problems. In the years before the "Agile enlightenment" successful products are built with or without iterations.

</comment><comment><USER17> So, according to this, User-Centred Design (UCD) is going to decrease the status of software engineers and make their working conditions closer to that of blue-collar workers by completely separating conception from execution.

I guess designers will be able to do user research and rapid prototyping with tools that are provided to them and then once they have a good product it will be passed to an implementation team to get implemented? Obviously once that's done it's released to the customer and there will be no need to ever change the design because by this point it is infallible.

One of the issues I can see here is that the design prototype will need to be kept in sync with the implementation, and designers will have to rely on prototypes being an accurate prediction of user's behaviour.

I do think the "interesting/creative work" baseline will probably get worse for many software engineers but I hope this takes 50+ years. I guess creativity and innovation is a scary thing for people and it's probably only a matter of time before the bean counters try to optimise/regulate and ultimately squash it. (And I believe they will for the non-creative have very little clue on where ideas come from and will squeeze until the golden goose stops producing eggs.)

When/if this happens I don't plan to be working. It would be unbearable to do non-creative work. I'd exit somewhere that I could keep my dignity and write poetry or programming for people that I met without the meddling of inward- facing fools.

<comment><USER6> It's warning that UCD is going in that direction - and that management is going to use UCD to marginalize programmers. It doesn't have to be that way, though, if Agile is willing to get in and work with UCD to develop disciplines that work for both designers and programmers.

I think one place this might happen is the issue of speed. All new programs are pretty slow, but they get sped up with "optimization" (LOL - more like caching and clever laziness tricks.) Evaluating speed needs during the design process brings programming into the design process a lot sooner, opening channels of communication that can reach to the user.

</comment></comment><comment><USER18> Have you ever seen unions concerned in improving work results, quality of the product and productivity? Accepting Even if workers conditions that could be more demanding?

the contrast is much more apparent than the analogy.

I worked in waterfall shops long enough to know that those were like unionized jobs where you could fair better by producing more code lines by copy/paste than by thinking. And the worst programmers moved the ladder to better paid non-programming "analyst" or "architect" jobs faster, not wasting their time on code details.

<comment><USER6> There's that saying, you can have it done right, fast, and cheap... pick any two. Generally the construction unions go for right and fast, not cheap. Often, they won't even do fast, mainly because speed and safety are often at odds with each other.

Of course, the meaning of cheap here is different. Cheap in the general market is $10 an hour, and union is $15. Qualified is around $20 to $30 an hour for the more common things, and a lot more for exotic skills like building bridges.

The logic is simple, though. Because construction unions aren't involved in producing product, they aren't interested in increasing speed and productivity. What they can increase is skills and quality and safety. So construction unions own training centers - gigantic warehouses and lots with classrooms and vocational education facilities. They force their members to get training.

In industrial unions in factories, the dynamic is different. Everything is based on the assembly line, so it's a different situation, and the unions have different strategies centered around that.

Likewise, trucker unions and other unions in the supply chain are increasingly strategizing around the supply chain.

I think you mischaracterize the waterfall shops. What waterfall did wasn't create union-like conditions. Waterfall rationalized the software production process, and created a kind of "factory" for software. The union-like behavior you notice is really factory behavior; when people in a factory resist management, even individually, it'll end up looking like a factory union's resistance.

When a union in a factory decides to do a "slow down" or "work to rule", it's resisting by putting pressure on the factory's workflow.

Unions operating in other sectors have different strategies. In the service sector, they use public campaigns to gain sympathy in the community. Teacher unions tend to use the political process (being a public sector that is popular with the electorate).

Agile's strategy was to improve quality, and give the programmer more autonomy. A side effect of autonomy (and TDD) is to allow the programmers to leverage productivity gains of new libraries and even new languages.

We don't associate what agile did with unionization, but that's largely because the craft-guild style union is moribund. It hasn't really revived and adapted for the age of data and software. That said, there are things like FOSS, GNU/FSF, Creative Commons and other organizations that operate in the space. The PC revolution, and then desktop publishing, and the internet, and now 3d printing and open hardware have all created a kind of "labor movement" of individual workers. It just lacks a political voice and a labor strategy.

</comment></comment><comment><USER13> I found this article to be very interesting. Does anyone have any pointers to similar writing about organisational politics in relation to agile development, lean or UCD?

<comment><USER19> Author of the article here. On Lean, I'd recommend a book called "Notes from Toyota-Land: An American Engineer in Japan" that is in part about the dark side of Lean and its impact on workers. Or just read an interview with the author: [http://www.gembapantarei.com/2006/05/interview_with_darius_m...](http://www.gembapantarei.com/2006/05/interview_with_darius_mehri_author_of_notes_from_toyotaland.html)

I also recommend reading The Management Myth which is about management theory in general: [http://www.theatlantic.com/magazine/archive/2006/06/the- mana...](http://www.theatlantic.com/magazine/archive/2006/06/the-management- myth/304883/?single_page=true)

The wikipedia page on participatory design gives a nice overview: [http://en.wikipedia.org/wiki/Participatory_design](http://en.wikipedia.org/wiki/Participatory_design)

Unfortunately most of the work in these areas is highly academic and assumes a lot of background knowledge. I thought this article on participatory design was quite good, but again, it's an academic paper: [http://peterasaro.org/writing/Asaro%20PD.pdf](http://peterasaro.org/writing/Asaro%20PD.pdf)

</comment></comment><comment><USER20> Agile seems to drive longer working hours. Instead of a big release 4 times a year, its lots of short very intense sprints.

Also, light blue on white is basically unreadable.

<comment><USER10> The intense sprints should only include the work that the team believes they can complete (and commit to completing) within the sprint period.

So the team shouldn't end up having to work extended hours due to Agile - <USER2> they are then either they are estimating incorrectly, or are being forced/asked to estimate incorrectly.

</comment></comment><comment><USER21> The article seems quite confusing on a number of fronts.

Architects of buildings don't actually use waterfall for a start - <USER3> continually adapt and improvise (in response to things like variable materials supply, soil conditions, weather, unexpected archeological finds etc.)

The UCD practitioner as outlined here is a type of, or proxy for, a product manager, which would fit in very well with any agile process.

The insight about unionisation is interesting, but the fact that agile turns out to share some subset of traits with unions doesn't really say much about agile - as nobody designed agile as a form of unionisation by the backdoor. It could be entirely coincidental and irrelevant or it might reveal something about aspects of unions being an emergent property of labour. (We certainly see unions appear across many industries in most nations.)

The point about lean and agile seems misconceived to me. Agile is essentially about preserving your options for change by reducing batch sizes and thereby improving flow of business value delivered. Scrum is just one form of agile, which reduces the epic cycles of waterfall from 6+ months to 1-4 weeks. More lean-focussed approaches like Kanban just take that further to a granularity of individual features. In fact you will have much more measuring and chartism under Scrum. A major lean argument is that optimal program velocity is achieved by continuous flow, even where it means that some individual items move at a slower pace - much like traffic control. This is precisely not an inhumane 'whip the horses harder' PM approach. It is also easier to visualise without needing to constantly measure an refine - bottlenecks should be self- evident from a kanban board for example.

Design thinking isn't something I'd say was at odds with agile at all, but in my experience is considered part of the wider movement away from 'throw over the wall' approaches (along with devops and lean UX). It's fundamental to lean startup for example.

And Taylorism is misapplied by many people to software development. In one way you can comfortably separate conception from execution with software. You could call them design and production line. The commonly made mistake is to think of coding as a production line activity rather than one of design. In fact the process of software development is one of automation generally - <USER4> as a high level of standardisation allows you to have robots building cars, a high level of software design allows you to automate execution. Where you have shoddy design, you end up with an inefficient process with humans doing production line work - that's a failure of industrialisation and Taylorism, not an example of it. This is in fact why software projects that attempt to separate design from technical implementation always run into huge problems. The technical implementation isn't a separate thing but is the actual design (and UCD here is closer to discovering the goals than defining implementation).</comment>


Mathematician's anger over his unread 500-page proof - <AUTHOR> https://newscientist.com/article/dn26753?cmpid=NLC%7CNSNS%7C2015-0108-GLOBAL <comment><USER2> This and many comments seem to miss the facts, at least as presented on the ABC conjecture's wikipedia page [http://en.wikipedia.org/wiki/Abc_conjecture](http://en.wikipedia.org/wiki/Abc_conjecture):

\- An error was found in 2012 and was just recently (Nov 2014) corrected \- He's re-released his notes & lectures on the subject \- A workshop will be held on the topic in March

That doesn't seem to be unusual / isolationist behavior, though it's obviously a complicated topic and since Fermat's Last Theorem has been deemed 'solved' already, the fact that it makes a FLT proof easier isn't much appreciated by the press / community.

<comment><USER3> Hey, it's New Scientist - [http://www.wired.com/images_blogs/beyond_the_beyond/2009/07/...](http://www.wired.com/images_blogs/beyond_the_beyond/2009/07/020709.jpg)

The angry isolated maths geek sells better.

This happened to Scientific American though and I stopped buying it.

On a long enough timeline, it seems all popular science magazines may turn into Focus.

</comment></comment><comment><USER4> No doubt the guy's exceptionally brilliant, and perhaps one day a mathematician with enough motivation to do it will spend a very long time trying to understand and decipher what he constructed, but my take on it is that anyone that wants to communicate a message, no matter how brilliant, should also go to the trouble of making it as legible as possible to others.

<comment><USER5> This is actually a major gripe of mine when reading papers - not enough effort to communicate the math so it's readable. It's really frustrating to decipher a paper when there's a lot of hand waving. It's 10x worse when buzzwords or unimportant relationships are also included.

<comment><USER6> If you've gone through paper-publishing, editors ask for pretty large reductions. A 29 page paper can easily be a 50-60 pages paper with "decent details" or a 150 pages paper with clear explanations of all details.

<comment><USER7> Sure, publish the reduced paper in the journal, then have the "Scenic Guide" in your webpage or ArXiv, or wherever

Or, I dunno, just create a subreddit or a StackOverflow page for it

<comment><USER6> This assumes infinite time to add all minutiae to LaTeX. Not everyone works directly on that, almost everyone thinks on paper, develops ideas on paper and then TeXifies it. In my case, most of the "it is clear that" never made it into any version of the final file.

<comment><USER7> Well, you certainly don't need all the minutiae, but some hand holding, especially on specific papers that deal with some theorems that might not be so well know would be helpful.

</comment></comment></comment></comment></comment></comment><comment><USER8> He did the entire 500-page paper in isolation and refuses to lecture on it in any capacity anywhere other than his own university. He may be right, and he may not be, but his failure to get an answer from the community is his fault, not theirs.

<comment><USER9> Math is supposed to be about pursuit of truth, not how congenial you are.

<comment><USER10> Like it or not, but math these days is a kind of social activity.

</comment></comment></comment><comment><USER11> Don't feel bad. I can't get coworkers to read 2 page approach docs.

<comment><USER12> I not too sure about emails with ten lines.

</comment></comment><comment><USER13> > If nobody understands a mathematical proof, does it count?

Nope. That's the social aspect of mathematics: your proofs have to be comprehensible and considered to be correct by those in the mathematical community that read them (at least peer review) before they're accepted.

<comment><USER14> A professor once defined a proof as "That which is convincing", and we were free to say, "I am not convinced"

While the topic being proven may be technically correct, the act of proving is a social act. The formal notation just makes this more precise than human language.

<comment><USER15> Proof is "that which is convincing to a fair and rational mind." It should compel a rational actor into the verifiable mitigation of some risk.

This particular fuss is one of the side effects of mathematicians' failing to properly adopt computers into their process. One of these days we will not accept a proof unless it is computer verified, and there will be no social aspect to it.

It should not matter if your proof is 10,000 pages if a computer can follow it.

<comment><USER16> It is still better to have a clean 1-page proof. There is no direct value for proving a theorem. Mathematics is nice because it's useful for further mathematics, useful for real-world applications, and beautiful. A 10,000 page proof that only a computer can follow is none of these things.

<comment><USER15> It is the second thing. Anyway, the number of pages is arbitrary -- it is a function of your language and terminology. Not every proof can be simplified, so surely there exist useful proofs that require 10,000 pages minimum for a given finite set of terms.

So you are very wrong. You are essentially asserting that the only mathematics worth doing is easy or already known in some compact form.

<comment><USER13> > surely there exist useful proofs that require 10,000 pages minimum

The classification of finite simple groups "consists of tens of thousands of pages in several hundred journal articles written by about 100 authors, published mostly between 1955 and 2004."[1]

[1]: [http://en.wikipedia.org/wiki/Classification_of_finite_simple...](http://en.wikipedia.org/wiki/Classification_of_finite_simple_groups)

<comment><USER15> That is a long proof, yes, but it doesn't support my point. One could easily imagine a formalization that could compress all that work into a single sentence. What you can't do is compress _all_ proofs to single-sentence proofs using a finite number of unique symbols.

</comment></comment></comment></comment></comment></comment></comment><comment><USER17> _> Grigori Perelman, another mathematician who refused to engage with the mathematical community_

That's not quite it.

[http://www.newyorker.com/magazine/2006/08/28/manifold- destin...](http://www.newyorker.com/magazine/2006/08/28/manifold-destiny)

<comment><USER18> That article isn't quite it either: [http://en.wikipedia.org/wiki/Manifold_Destiny#Controversy](http://en.wikipedia.org/wiki/Manifold_Destiny#Controversy)

</comment></comment><comment><USER19> Hopefully someday, model checkers and proof assistants can help mathematicians to put these monster proofs into format that can be verified with computer.

<comment><USER20> Yes, assuming that there remain any proofs which the model checkers and proof assistants haven't already discovered!

<comment><USER21> The space of proofs is enormous. Model checkers and proof assistants will probably become user friendly much sooner than they'll get good enough heuristics to navigate such spaces.

</comment></comment></comment><comment><USER22> There is a really good article here :[http://projectwordsworth.com/the- paradox-of-the-proof/](http://projectwordsworth.com/the-paradox-of-the- proof/). That is from May 2013; the situation is basically unchanged since.

<comment><USER23> [https://news.ycombinator.com/item?id=5685166](https://news.ycombinator.com/item?id=5685166)

</comment></comment><comment><USER24> Often happens that after an hard proof there are less t'han 10 People that can understand it. Also happened for Fermat Conjecture proved in 1997, after 5k years! Also with shorter proofs can be tricky. I studied at University a proof of Stoker Theorem by Poincaré that fits in one page, but, trust me that it is really diabolic.

<comment><USER25> You probably mean 5C=D years but the logic is the same... :-)

<comment><USER26> What are C=D years?

<comment><USER27> C and D are the Roman numerals corresponding to 100 and 500 respectively. Hence, 5C = D.

But, the larger point was that took 500 (5C) rather than 5000 (5K) years to prove Fermat's Last Theorem.

</comment></comment></comment></comment><comment><USER14> _He has also criticised the rest of the community for not studying his work in detail, and says most other mathematicians are "simply not qualified" to issue a definitive statement on the proof unless they start from the very basics of his theory._

Math this advanced can be very hard to follow. There are more incentives to do original work than check someone else's, especially when they have drifted far off the mainstream. It takes a special arrogance to say, "You have to come to me" and then be angry when nobody follows.

<comment><USER28> > "simply not qualified" to issue a definitive statement on the proof unless > they start from the very basics of his theory

How is it arrogant to say people need to study the whole branch, from the beginning, rather than just try to cherry pick a few advanced parts of it, if he really did come up with a substantially new branch?

That sounds more like practicality than arrogance to me.

<comment><USER14> This requires years of in-depth study without guideposts.

If it's a brand new field, help people along the way. (Write books, explain the concepts globally, create projects for Phds, etc.)

<comment><USER28> As I recall, he's personally tutoring at least one other mathematician, doing lectures, etc -- that is, I'm pretty sure he's doing the last two of your suggestions. Writing books, of course, is a multi-year process in addition to his other work.

The problem is that they're disqualifying people who study under him in this manner from verifying his work, which seems like something of a catch-22.

<comment><USER14> It's tough, but that standard exists for a reason. They want it to be somewhat arms length. But this means that if you go too far off the beaten path, it's hard to follow.

<comment><USER28> I agree that the standard exists for a reason; my point was that it's unfair to characterize him as not trying.

The standard doesn't just make it hard to follow someone who sets out a new branch in depth - it makes it virtually impossible for an extended period of time, because it disqualifies anyone he guides. Such a standard requires that we essentially wait 5+ years (at best) until either high quality maps or guides trained by the initial guided group exist. (And even then, we'll probably wait longer.)

It's not exactly fair to paint that fact as a failing on his part to educate others -- it's a nasty corner case in a generally good academic standard.

</comment></comment></comment></comment></comment></comment><comment><USER29> In other news, engineers refuse to do a code review of a pull request with 1 million lines of code changed.

As has been mentioned elsewhere, the issue may be the "code bomb" nature of the way the paper was published, and if it was released in even smaller chunks, it may have garnered more analysis.

</comment><comment><USER30> _> This sense of stubbornness, dignity and pride is a part of what gives him the personality necessary to embark on a project like this_

It is always interesting to me, as I see it often in the software world, that people expect extraordinary people to behave like ordinary people.

<comment><USER26> Fuck you (I'm extraordinary)

</comment></comment><comment><USER31> I feel for the guy.

At the same time one can argue that if len ( Proof1 ) < len ( Proof2 ), where Proof1 and Proof2 are of the same theorem, then Proof1 imposes less cost, and hence is more valuable, since it will be easier to teach, use. etc.

<comment><USER18> This would be relevant if there was another proof of the ABC conjecture available.

<comment><USER31> just take len ( Proof2 ) = infinity

</comment></comment></comment><comment><USER32> He probably should try to use Coq proof assistant. If Coq proves his proof, then mostly like it's correct. Of course, it's easier to say than to do, not a trivial work, and he's the only one who can do it, since he's the only one who can understand the proof.

But if he indeed able to write the entire proof in Coq, then I'm sure at that point, his proof will be much cleaner as well.

Then again, this will never happen.

<comment><USER18> That is like a complete no-go. It took something like 6 years[0] to formalize the proof of Feit-Thomson, one of the first "long" proofs in group theory, and that was done by experts who understood both Coq and the proof. In the time it would take him to write the proof in Coq, dozens of mathematicians could learn the theory and independently verify it.

[0] [http://www.msr-inria.fr/news/feit-thomson-proved-in-coq/](http://www.msr- inria.fr/news/feit-thomson-proved-in-coq/)

<comment><USER32> "Oh my god, it's such a big project, nobody should do it alone". It's very obvious, isn't it? 500+ pages of abstract proofs need to be expressed into some kind of programming language. Not to mention, the worst case is that he doesn't even know what coq is, or have never used a programming language (probably just TeX, since he's mathematician) and will take him sometime to pick it up, and then learn how to express his idea using it, etc ...

Look at the linux kernel we have right now, let's go back to early 90s, and should we tell Linus not to do it because it's so big? And did it end up Linus doing it all alone himself?

Coq is certainly not a short term solution, but definitely a valid one if no one else wants to read his proof, and it's very important for him validate it.

The reason I bring Coq up is that in general, I do feel proving things in maths is very much like writing a program in a programming language. Maybe the 500+ pages of proof is like a spaghetti code, refactoring might make it more readable, or more clear, so people are more likely to study it. Or the 500+ pages of proof is very elegantly constructed, just need someone to appreciate it. Coq can certain help in both cases.

But then is it worth it for him to do it? That's really he's judgement call based on how important the proof is, what he see he can get out of this, etc.

Remember that line from the article you quoted: "Fun ~enormous!"

<comment><USER33> Writing a program is exactly the same thing as proving a mathematical statement. This is an extremely cool and deep result. Start with the Church- Turing thesis and go from there.

<comment><USER34> Except in Math we accept duck-typed functions to prove our results. Advanced math is not fully rigorous and often has mistakes.

</comment></comment></comment></comment></comment><comment><USER35> He will be doing a workshop:

RIMS Joint Research Workshop: On the verification and further development of inter-universal Teichmuller theory (in Japanese) [http://www.kurims.kyoto-u.ac.jp/~motizuki/2015-03%20IUTeich%...](http://www.kurims.kyoto-u.ac.jp/~motizuki/2015-03%20IUTeich%20Program%20\(English\).pdf)

March 9-20 2015

</comment><comment><USER36> I understand the trouble, but in this case it seems like the trouble might well be worth it. If the ABC conjecture is proven, the consequences are quite substantial:

[http://en.wikipedia.org/wiki/Abc_conjecture#Some_consequence...](http://en.wikipedia.org/wiki/Abc_conjecture#Some_consequences)

</comment><comment><USER37> On a side note: What is the Latex font name used in this document (Published by Shinichi Mochizuki)?

[http://www.kurims.kyoto-u.ac.jp/~motizuki/IUTeich%20Verifica...](http://www.kurims.kyoto-u.ac.jp/~motizuki/IUTeich%20Verification%20Report%202014-12.pdf)

Looks really nice to eye.

<comment> alex-g Computer Modern [http://en.wikipedia.org/wiki/Computer_Modern](http://en.wikipedia.org/wiki/Computer_Modern)

</comment></comment><comment><USER38> > If nobody understands a mathematical proof, does it count?

That depends. If it deals with integers, yes. If it deals with real numbers, no.

<comment><USER39> boo. hiss. (and actually a clever reply tottenhm)

</comment></comment><comment><USER40> s/anger/frustration/

</comment><comment><USER41> "An overview of Inter-universal Teichmüller Theory and [...] how we can begin to understand this theory":

[http://www.quora.com/Joseph-Heavner/Posts/An-overview-of- Int...](http://www.quora.com/Joseph-Heavner/Posts/An-overview-of-Inter- universal-Teichm%C3%BCller-Theory-and-Shinichi-Mochizukis-proof-of-the-ABC- Conjecture-along-with-th)

Also, from an anonymous post on 4chan @ [http://boards.4chan.org/sci/thread/6931488/are-you-ready- for...](http://boards.4chan.org/sci/thread/6931488/are-you-ready-for- interuniversal-mellin-transform#p6956712) :

"How to understand IUTeich from the bottom up:

\- Algebra (Israel Gelfand)

\- The Method of Coordinates (Israel Gelfand)

\- How to Prove It: A Structured Approach (Daniel Velleman)

\- Kiselev's Geometry - Planimetry and Stereometry

\- Trigonometry (Israel Gelfand)

\- What Is Mathematics? An Elementary Approach to Ideas and Methods (Richard Courant)

\- A Course of Pure Mathematics (G.H. Hardy)

\- Linear Algebra (Kenneth Hoffman and Ray Kunze)

\- Elementary Differential Equations (William Boyce and Richard DiPrima)

\- Topology (James Munkres)

\- Calculus On Manifolds (Michael Spivak)

\- Principles of Mathematical Analysis (Walter Rudin)

\- Real and Complex Analysis (Walter Rudin)

\- Functional Analysis (Walter Rudin)

\- Partial Differential Equations (Lawrence Evans)

\- Analysis On Manifolds (James Munkres)

\- Abstract Algebra (David Dummit and Richard Foote)

\- Algebraic Topology (Allen Hatcher)

\- Introduction to Smooth Manifolds (John Lee)

\- Foundations of Differentiable Manifolds and Lie Groups - (Frank Warner)

\- Galois Theory (Harold Edwards)

\- Linear Representations of Finite Groups (Jean-Pierre Serre and Leonhard Scott)

\- A Classical Introduction to Modern Number Theory (Kenneth Ireland and Michael Rosen)

\- Introduction to Analytic Number Theory (Tom Apostol)

\- Modular Functions and Dirichlet Series in Number Theory (Tom Apostol)

\- Riemannian geometry (Peter Petersen)

\- The Theory of the Riemann Zeta-Function (Edward Charles Titchmarsh)

\- An Introduction to Teichmüller Spaces (Yoichi Imayoshi and Masahiko Taniguchi)

\- A Course in p-adic Analysis (Alain Robert)

\- Foundations of p-Adic Teichmuller Theory (Shinichi Mochizuki)

\- Mochizuki's papers at [http://www.kurims.kyoto-u.ac.jp/~motizuki/papers- english.htm...](http://www.kurims.kyoto-u.ac.jp/~motizuki/papers-english.html) "

</comment><comment><USER42> Guys. This is my problem with reading white papers. People use notations and assumptions which I am not familiar with. Hence, I have to go decipher what people are trying to day. There needs to be a course for math notation. Is there a book or a online course I can take to familiarize myself with math as a language.

<comment><USER43> They use (slighly) different notions in different parts of mathematics. And usually, it's not just the notation that's difficult, but the concepts themselves.

In general, you are best off studying introductory texts to the topics you are interested. You will pick up the notation.

</comment></comment><comment><USER44> This man invented bitcoin.

<comment><USER45> I thought he looked familiar... I definitely remember his photo making the rounds as "Potential Satoshi"'s when that was the media craze of the week

<comment><USER46> yup, i recognized the photo as well [http://chartgirl.com/wordpress/wp- content/uploads/2013/05/SA...](http://chartgirl.com/wordpress/wp- content/uploads/2013/05/SATOSHI_large.jpg)

<comment><USER47> Do you people not understand most of that infographic is a joke?

</comment></comment></comment></comment><comment><USER48> What real-world purpose does this conjecture hold, and why is its proof of any importance?

<comment><USER46> see

[http://en.wikipedia.org/wiki/Fermat%27s_Last_Theorem](http://en.wikipedia.org/wiki/Fermat%27s_Last_Theorem)

[http://en.wikipedia.org/wiki/Fermat%27s_Last_Theorem#Monetar...](http://en.wikipedia.org/wiki/Fermat%27s_Last_Theorem#Monetary_prizes)

<comment><USER49> ? The Fermat prize money has already been collected. If the real world purpose of this conjecture is to collect prize money, it won't be that prize.

</comment></comment></comment><comment><USER50> I have a similar issue with a philosophy that I'm creating.

There are some barriers. Off the top of my head...

1\. The reader has to care enough to expend time & energy to understand the concepts

2\. The reader comes in with an Existential Context that may have to be suspended or expanded. This is tricky because Existence is a fractal of nuance and it's difficult to know when there's a misunderstanding. Conversations & isolating examples help.

3\. The reader has to be willing to adopt the new Existential Context while reading the works.

4\. The more ridged & complex the context, the more the reader has to deny self.

</comment><comment><USER51> I'm kinda amazed that no one has been able to write a proof solving AI yet. While a human AI is a very daunting task in order to capture "humaness" and large problem-set it would seem to me that an AI working solely as a domain expert within maths should be much more feasible.

Shouldn't you be able to build a database of all current proofs and have a computer be able to cross-reference and extrapolate from that? Sure it would be far from trivial but it seems doable. Mathematica/Wolfram for instance should be half way there.

Here's the golden chance for a programmer to win x amount of Nobel prizes! :)

<comment><USER52> There are computer generated proofs. They are also completely unreadable, much less understandable, by humans. In reality, most of mathematics is based on set theory, which makes it hard for computers to reason about proofs. There is currently an effort to rebase mathematics on type theory, which would allow computers to verify and create proofs much more easily. However, basically all of mathematics has to be reproved using this new foundation. To give an example, group theory as taught in undergraduate classes depends completely on sets. A group itself is a set with additional properties. Subgroups are defined in terms of set intersection and union. Type theory is still in its infancy, and this stuff is still being figured out. In terms of Mathematica, it doesn't know jack shit about complicated math theorems. Can it tell you if a group is solvable?

<comment><USER51> Yeah, classifying math aspects and creating relations between different proofs is probably one of the thornier issues I'd guess. It does feel however that mathematics is a domain where an AI should be able to train itself in some way</comment></comment></comment>


Probabilistic Scraping of Plain Text Tables - <AUTHOR> http://edinburghhacklab.com/2013/09/probabalistic-scraping-of-plain-text-tables/ <comment><USER2> Where is a HN mod when you need one? The original title of the OP is much more accurate and much more interesting:

"Probabalistic Scraping of Plain Text Tables"

vs the submitted title, "Accurate, efficient and maintainable scrapers"

As someone who's scraped a lot of plain text government data, I've given up on finding a flexible machine-learning way and just hand-coded for specific usecases (because government software changes so rarely). However, the OP presents an interesting strategy (with math that I haven't quite mastered, unfortunately)

<comment><AUTHOR> its not so hard,

if you understand how to force boolean algebra into intger form, like:

x + y >= 1, where x={0,1}

is a boolean OR between x, y, then it is easy to expressed in PuLP like:

x = variable("x", lowerbound=0,upperbound=1, cat='Integer') y = variable("y", lowerbound=0,upperbound=1, cat='Integer')

problem += x + y >= 1

tada its really easy to write (once the math is defined, the modelling bit is the hard part, but it gets easier over time)

</comment></comment><comment><USER3> [Statistical parsing]([http://nlp.stanford.edu/courses/lsa354/](http://nlp.stanford.edu/courses/lsa354/)) is often used in Natural Language Processing, where can't decide which how to arrange sentences into trees, so they pick one at random among acceptable choices.

I've gotten very frustrated writing rules and regexes and exceptions. This method seems really great when scraping data which is structured but a little bit noisy.

<comment><AUTHOR> I got sick of writing rules too, hence the diversion!

yeah so you could extend that tree decision processes this. You could add a cost function on the trees, which includes long range communication between nodes (e.g. the results section of a document includes at least one word commonly associated with results). So you could go to higher levels of abstractions.

However, I say limiting yourself to even a very abstract parse tree structure does kinda exclude the spatial layout of words on a page and the semantic meaning behind the layout (tables, web pages, code formatting!).

MIPs can fully express tree logic so you lose nothing.

<comment><USER3> Looks like you were trying to complete data found with Optical Character Recognition by maximum-entropy.

This is awesome. What are some other use cases for this?

With APIs much of the data I use has already been formatted in JSON or CSV already, so I don't get to scrape much anymore.

</comment></comment></comment><comment><USER4> On the subject of scraping data from OCR'd tables:

I heard from a colleague who moved into finance, that there's a mini arms race going on between some funds(?) who are subject to regulatory requirements to release financial performance metrics but for a variety of reasons would rather not (and certainly would rather not make the data machine readable), and other hedge funds who want to run automated trading strategies off said released figures.

They keep obfuscating the tables to make them harder and harder to parse algorithmically while still remaining theoretically human-readable.

Great fun no doubt for everyone involved.

<comment><USER5> any resources to back this up?

<comment><USER4> Entirely anecdotal.

</comment></comment></comment><comment><USER6> I have imminent use for this algorithm at the start-up I am interning at, but unfortunately the concepts are beyond my understanding. Could anyone please recommend resources that will help me understand and apply this method?

</comment><comment><USER7> I remember attending a thesis defense while I was attending Edinburgh Uni. Is that you?

<comment><AUTHOR> um, you are in the right geographic area, but there were no strangers in my thesis defence... so unless you are my supervisor, external or internal examiner its unlikely you were there (sidenote my PhD in robotics not this kind of stuff).</comment></comment>


MaxItUp, the new way to setup your newly formatted Mac devices - <AUTHOR> https://max-itup.github.io/mac/ <comment><USER2> An equivalent service for Windows: [https://ninite.com/](https://ninite.com/)</comment>


Apple: Open Source - <AUTHOR> https://developer.apple.com/opensource/ <comment><USER2> _" Apple, the first major computer company to make Open Source development a key part of its software strategy"_

Well, I guess that's arguable.

 _" WebKit — the open source rendering engine introduced by Apple"_

Yeah, _" forked from KHTML"_ wouldn't sound as nice as _" introduced by Apple"_.

 _" OS X combines a proven UNIX foundation with the easy-to-use Mac interface to bring multicore technology and 48-bit power to the mass market."_

Wait, what?

<comment><USER3> Most (all?) amd64 processors only actually use 48 bits of the 64-bit virtual address space. It's split into 00000000 00000000 - 00007fff ffffffff and ffff8000 00000000 - ffffffff ffffffff, so you can do the usual thing of userspace at the bottom and kernelspace at the top, but it continues to be backwards-compatible if we ever need more than 48 bits of virtual address space.

From the Intel programmer's manual:

> _In 64-bit mode, an address is considered to be in canonical form if address > bits 63 through to the most-significant implemented bit by the > microarchitecture are set to either all ones or all zeros._

> _Intel 64 architecture defines a 64-bit linear address. Implementations can > support less. The first implementation of IA-32 processors with Intel 64 > architecture supports a 48-bit linear address. This means a canonical > address must have bits 63 through 48 set to zeros or ones (depending on > whether bit 47 is a zero or one)._

This is a confusingly obscure and pedantic point to make on a marketing page, though. Most kernel hackers on 64-bit platforms barely think about this.

<comment><USER4> I'm pretty sure they're referring to PAE , and haven't updated since they dropped i686.

It would've made sense in the 32-bit era to market as a 48-bit operating system, since it's bigger than 32.

[https://en.wikipedia.org/wiki/Physical_Address_Extension](https://en.wikipedia.org/wiki/Physical_Address_Extension)

</comment></comment></comment><comment><USER5> "Apple, the first major computer company to make Open Source development a key part of its software strategy"

Wait what? Am I missing something here

<comment><USER6> I guess they're referring to projects like Darwin (released in 2000) and Webkit (open-sourced in 2005). They also invested heavily in LLVM starting in 2005.

Those are all major strategic investments into open-source, at a time where companies like Microsoft, Google and Amazon were not making any comparable investments that I can think of. For example, in 2005 Android was freshly acquired by Google, Chrome was still 3 years away. Go was introduced in 2009.

One exception I can think of is IBM which invested heavily in Linux around the same time Apple was open-sourcing Darwin.

Overall, their claim is arguable but certainly not without merit.

EDIT: remember when IBM covered the streets of San Francisco in Linux graffiti, and were threatened with a huge fine when the paint didn't wash off as planned? [http://usatoday30.usatoday.com/tech/news/2001-04-25-ibm- linu...](http://usatoday30.usatoday.com/tech/news/2001-04-25-ibm-linux- graffiti.htm)

<comment><USER7> WebKit was a fork of KDE's KHTML engine, and so had to be open source from the outset. IIRC, in the early days Apple was _not_ on good terms with KHTML upstream and weren't exactly model participants in the open source community.

<comment><USER8> They basically started the development of WebKit in secret based on KHTML despite having to release the source when they released Safari. The result was two codebases too different to merge, and the improvements couldn't go "back to the community" without discarding many other improvements, not to mention the work needed to integrate it into KDE.

</comment></comment></comment></comment><comment><USER9> Maybe my ignorance is showing, but I've felt like Apple has been the least involved major company of them all when it comes to community stuff. They feel so aloof to me. My impression is Microsoft used to be that way but is changing.

<comment><USER10> Well, Safari/Chrome/Android browser/etc is a result of an open source project Apple adopted (KHTML), expanded and developed way beyond the original code. And while Webkit couldn't merge back into KHTML easily, it became the basis for a full blown Open Source engine of its own with tons of adoption.

Also LLVM. Apple heavily funded work on it, and it's now used by everybody who's anybody in the industry, from Google to MS.

Apple just doesn't care to go into much community development by committee (bazaar) because they have their own (cathedral-like) agenda to follow.

</comment></comment><comment><USER11> > Apple, the first major computer company to make Open Source development a > key part of its software strategy, continues to use and release significant > quantities of open source software.

"first major _computer_ company"

Interesting phrasing.

<comment><USER12> Yes, especially since they specifically removed _computer_ from their company name almost 10 years ago.

<comment><USER13> They never stopped being a computer company, though. The word "Computer" doesn't have to be in your name to qualify.

</comment></comment></comment><comment><USER14> I recently interviewed at Apple. They told me you can not contribute to open source even on your own time if you get hired!

They said we use "tons of open source software" but we don't contribute back because it's going to "impact company's picture".

<comment><USER15> So which should I believe, a random poster on Hacker News that says that Apple forbids employees to contribute to open source, or the fact that Apple has many published open source projects?

<comment><USER14> I guess it depends on the team. Apple unlike Google hires for a specific team. Team that I was talking with would not allow open source contributions. This is true for most of Apple. I have a lot of friends working in Apple.

They sent project Titan people to a battery conference with a fake company name which they actually registered and made people believe it's a real company. It has a Crunchbase profile and all that. They published zero papers from their battery research. That's the Apple you should know.

</comment></comment></comment><comment><USER16> I went to a reactjs meetup at Apple a few days ago. There they showed off Revis, a react library for making charts and visualizations that looked really interesting, with the selling point being that it enables you to create graphs declaratively. They said they had plans of open sourcing it, although they didn't give a due date.

Another thing they indirectly showed off was a reactjs playground which was very similar to swift playgrounds. I say indirectly, because the person that was presenting was showing revis using it. There's some public playground components that are around (e.g. component-playground [0]), but it's the type of thing where I think there's still room to experiment and innovate. Bret Victor's Inventing on Principle video [1] comes to mind.

I don't know how big the web presence at Apple has been in the last few years, but I'm hopeful that this is a sign that it's growing and that it'll push more resources to the Safari team so it'll keep up better with other browser vendors. Although they have a history of open source, I don't think they've open sourced much JavaScript? From googling around all I could find was that they were one of the biggest contributors to SproutCore [2], which looks fairly dated by today's standards.

[0] [http://projects.formidablelabs.com/component- playground/](http://projects.formidablelabs.com/component-playground/)

[1] [https://vimeo.com/36579366](https://vimeo.com/36579366)

[2] [http://sproutcore.com/](http://sproutcore.com/)

<comment><USER4> I work at a web shop, and we're convinced that Apple is deliberately failing to make improvements to Safari (specifically on iOS). The theory is that they are conspiring to lock in people who depend on the platform, but also prevent them from delivering anything approaching the quality of native apps on iOS.

Though usually I try to avoid thinking of conspiracy; the more I work with their platform, the more reasonable this theory sounds.

I have been solicited to work on WebKit for Apple, but given how isolated Apple employees seem from the world, I don't think I could do it. I'm not willing to sign away my personality or existence for cash-on-hand. I suspect they're having trouble hiring people to work on WebKit, because although I think I'm a pretty good developer, I'm not the top of the barrel when it comes to WebKit developer candidates.

<comment><USER10> > _I work at a web shop, and we 're convinced that Apple is deliberately > failing to make improvements to Safari (specifically on iOS). The theory is > that they are conspiring to lock in people who depend on the platform, but > also prevent them from delivering anything approaching the quality of native > apps on iOS._

Classic cooky conspiracy theory.

In fact the Android browser is in many ways worse than Mobile Safari.

So there goes the whole theory...

<comment><USER4> Most Android devices these days run Chrome.

</comment></comment></comment></comment><comment><USER17> _> OS X combines a proven UNIX foundation with the easy-to-use Mac interface to bring multicore technology and 48-bit power to the mass market._

48-bit power?…

<comment><USER18> Perhaps that statement refers to the 48-bit x86-64 virtual memory address space.

<comment><USER4> Maybe it refers to PAE and hasn't been updated since they dropped i686?

</comment></comment></comment><comment><USER19> Do they seriously think they can pull the marketing lingo shroud over actual developers?

<comment><USER20> > Apple, the first major computer company to make Open Source development a > key part of its software strategy

It's actually pretty factual. They open sourced Darwin/WebKit/LLVM/Clang, and legitimized BSD and Unix as a consumer product via OS X... in 2001/2005/2005\. The only "big" company that I can think of that made a move that big was Mozilla, which arguably doesn't even compare to a company as huge as Apple. Plus Mozilla open sourced itself out of pure desparation from pressure via Internet Explorer (which, I might remind you, was one of the worst proprietary pieces of shit to scorch this earth, by none other than Microsoft).

It's only been a recent thing where companies like Google/Microsoft have been embracing open source. Companies like Gitlab/Docker weren't even concievable back then.

<comment><USER2> Well, they couldn't not open source WebKit, as at its core lies a fork of LGPL licensed KHTML and KJS (that's why WebCore and JavaScriptCore are LGPL while the rest of WebKit is BSD)

<comment><USER21> Does it really matter if they were forced to open source it or not? The point is they poured a ton of money into Webkit and that did a lot of good for the industry as a whole.

</comment></comment></comment></comment><comment><USER22> Waiting for Apple's TensorFlow alternative.

</comment><comment><USER23> Open source development might be arguable with Darwin (although I think IBM still has earlier major investments), but open source philosophy is certainly not the case.

</comment><comment><USER7> Update: they've changed the wording on [https://developer.apple.com/opensource/](https://developer.apple.com/opensource/)

"Open source software is at the heart of Apple platforms and developer tools, and Apple continues to contribute and release significant quantities of open source code."

</comment><comment><USER22> This is more impressive than Goldman Sachs's open sourcing effort. [https://github.com/goldmansachs](https://github.com/goldmansachs)

</comment><comment><USER24> > Apple, the first major computer company to make Open Source development a > key part of its software strategy

Sounds true assuming Apple thinks of itself as "the only major computer company"

</comment><comment><USER4> Webkit was introduced by Apple like OpenOffice was introduced by Apache.

<comment><USER25> WebKit was a fork of KHTML originally... but does that mean Apple didn't introduce it?

<comment><USER4> I don't know, all I said is that much like Apache OpenOffice is a fork of OpenOffice, WebKit is a fork of KHTML.

Not sure that's what most people mean when they say "introduce".

Like for example, if I were operating a public water supply: I could say that I introduced the practice of chlorinating water, but obviously the practice has been demonstrated with other water supplies. Have I introduced it, or just adopted it?

It would be more accurate to say that Apple adopted KHTML as WebKit, then made major contributions to it.</comment></comment></comment>