See That Billboard? It May See You, Too - <AUTHOR> http://www.nytimes.com/2016/02/29/business/media/see-that-billboard-it-may-see-you-too.html <comment><USER2> What do I need to do on my Android phone to make sure I can't be tracked in this way?

<comment><AUTHOR> A Faraday cage, maybe? So long as your phone is sending signal to a mobile network, it can be triangulated in some way.</comment></comment>


GPGMail 2 is finally here - <AUTHOR> https://gpgtools.org/news.html <comment><USER2> Email encryption is a good start!

Personally, I think, the user is best served with the "darknet" [0] approach.

It's unfortunate that term "darknet" leads a big chunk of the general public to believe it's something "dirty", "illegal" or otherwise undesirable or even dangerous, which doesn't help its cause. So help the Internet out and spread the word:

[0] [http://en.wikipedia.org/wiki/Darknet_%28file_sharing%29](http://en.wikipedia.org/wiki/Darknet_%28file_sharing%29)

RetroShare [1] is one of them and has the advantage of being the all-in-one encryption solution (VOIP, chat, messages, file sharing), while encrypting everything. It also eliminates the meta data problem which encrypted email has.

[1] [http://en.wikipedia.org/wiki/Retroshare](http://en.wikipedia.org/wiki/Retroshare)

EDIT: Direct link - [http://retroshare.sourceforge.net/](http://retroshare.sourceforge.net/)

<comment><USER3> The perfect is the enemy of the good.

<comment><USER2> Now I'm curious about how you're applying this to the current context... Care to explain?

<comment><USER4> He likely means that email encryption (the good) is incrementally achievable, while RetroShare and Darknet (the perfect) requires a much more disruptive change.

<comment><USER2> Ok.

I'd just like to add to that:

Very serious attacks on democratic principles have been carried out behind our backs, for a very long time, apparently.

That is why I think, in order to counter these threats to society appropriately, disruptive approaches should really be welcomed and preferred as often as possible.

And then, of course, it's not an either/or question. Both should be promoted alongside, as lots of different people have lots of different needs/requirements. (Personally, I use both RetroShare and encrypted email, and migrate as many people as possible over to _total encryption_ as soon as possible.)

<comment><USER4> Email encryption is going to be effective against 99% of relevant attacks, and if the NSA wants to know what you're doing, they'll put a bug in your laptop and, no, you won't notice.

Remember, even with lofty "serious attacks on democratic principles" going on, you're still infinity times more likely to have your bank account information stolen by drive-by script kiddies with a 0-day than be a target of government persecution on the back of illicitly obtained intelligence. Even then, it's highly unlikely that even the NSA has the capacity to decrypt internet traffic at scale.

<comment><USER2> I know, I know, I don't even consider myself a target, at all. I don't really do this to just protect myself.

It's just that we should all do our best in order to erect the collective hurdle that the mass surveillance efforts now require. There's nothing lofty about democracy being attacked, it's _very_ real. And don't forget that you can't really do encryption on your own, you depend on all your contacts doing it, too. Society needs to make that as normal as brushing their teeth.

</comment></comment></comment></comment></comment></comment></comment><comment><USER5> For those of us who have never heard of this software, it would be great if somewhere prominent on the webpage it said what GPGMail actually is. Instead the most prominent thing is how many bug fixes and sleepless nights this mystery software required.

<comment><AUTHOR> You got a point. Added that!

</comment></comment><comment><USER6> The last version of GPGTools I looked at had the irritating habit of always installing its own copy of GPG into /usr/local and not letting me use my own version (e.g., from Homebrew). Is this still the case in version 2?

It would be far cleaner if it was more self-contained (e.g., included GPG inside its installation bundle), and then let the user pick an alternative OpenPGP installation in the preferences.

<comment><AUTHOR> It's "self-contained" in /usr/local/MacGPG2. Only creates symlinks into /usr/local/bin but will avoid that if it recognizes another gpg already being linked there. Also, linking warnings when using homebrew's GPG are resolved.

<comment><USER6> I see. This isn't clean enough for my tastes. I run Homebrew from a custom directory, not /usr/local, and I want to make GPGMail refer to the gpg binary there. No Mac app installer should ever write anything to /usr/local.

PS: If you are a maintainer, thank you for all the hard work. I'm only criticizing the current installation system because I really want to use GPGMail, and it does not fit into the way I like to set up my machine.

</comment></comment></comment><comment><USER7> Observations after installing on 10.6.8:

* GPG2 seems to be up and running very nicely, and it was an easy switch to get Enigmail set up to recognize it. (My previous MacGPG installation evidently installed GPG1. That seems to be orphaned now, I guess? Any suggestions for an ideal way to clean out its old stuff? I haven't seen it mentioned on your site.)

* This is the first GPG distribution that I can remember using that didn't provide hashes and a detached signature to verify the integrity of the downloaded file.

* It looks like the provided man pages were not symlinked into /usr/local/share/man/man1 (to match the way that the binaries were symlinked into /usr/local/bin).

* For reasons I've yet to track down, the GPGPreferences preference pane hangs whenever I try to open it. (I'll file a bug and/or ask for help on your forums eventually; just mentioning it here as part of the experience.)

<comment><AUTHOR> SHA Hash and signature will be added tomorrow. In all the excitement we forgot about that. Please file the bug report for GPGPreferences on [https://support.gpgtools.org](https://support.gpgtools.org) Thanks!

</comment></comment><comment><USER8> I had some experience with GnuPG and Symantec PGP and Outlook a few years back. All non-public information like CAD files were supposed to be PGP encrypted. Yet, even the engineers would send most files in plain text. I remember many times having to logmein to a clients machine to try to figure out why they couldn't read our emails. This is why PGP never took off.

Until the tools take 5 min to setup, and encryption/decryption is automatically handled by the mail client, PGP will never take off. Things like the public key directory have to handled transparently to the user.

It's too bad Mozilla dropped support for Thunderbird. Tight integration with GnuPG plugin could have made mainstream PGP a reality. For OS X at least, it looks like GPGMail is nearly there.

<comment><USER9> I'd say ("first time") setup is pretty easy (and has been for a while). The tricky part (as always) is managing the keys (the private key, and the (optional) revocation key) -- and managing trust.

Key management is tricky because if you have a truly secure pass-phrase (that is, one that contains >= 128 bits "worth" of entropy (or even >= 65 bits which might be enough), a pass-phrase that can be considered at least as secure as the symmetric session keys) -- then that is going to be awkward to type in (and remember). And if you don't -- then you need to be (extra) careful about where you store your secret key ring, where it is backed up, etc (you should be careful about this anyway).

And it is still tricky to carefully manage which keys you trust, and bootstrapping trust is hard. The latter can be alleviated somewhat by having a few "designated CAs" in a company -- eg: have the IT department set up GPG, and make sure that they verify and sign people's keys along with setting up accounts etc.

</comment></comment><comment><USER10> My company's internal mail goes through gmail so I decided after recent news to setup GPGmail and s/mime.

I identified a couple of usability issues, which where fixed. I'd say all in all its very good.

Regardless if you believe or care about the NSA issues, simply the idea of routing clear text email through mail exchanges, and advertisers should give you enough reason to follow the few steps it requires to generate a key, and start encrypting and/or signing email. Except for post cards we don't do this with our regular mail, so why are you ok with it with you email (and your email is far more machine readable).

GPGMail is not quite Grandmother ready, and unlike s/mime it doesn't really have an incremental value[1], but it is far more secure, and very easy to use once setup. Plus the other tools in the toolkit are useful for general encryption.

s/mime is another option, here are some pros and cons:

s/mime pros

      integrated with many mail apps
      usually plays nice with mailing lists (adding a footer doesn't invalidate a sig)
      works on iOS devices (perhaps others?)
      has an incremental value even before all your contacts are using it[1]

s/mime cons

      based on a certificate authority model
      cost money depending on the cert you get
      requires a 3rd 'trusted' party
      does not seem to be secure in some respects:
        (web cert generation, no rules regarding sigh/encrypt/sign[2],
        does not make use of a certificate request so anyone who has
        even momentary access to your email can generate a cert to
        masquerade as you)
      your identity is associate with your email address not you
        (you will need certs for each email address)

\--

GPGmail/tools pros

      Based on web of trust instead of CA (web of trust is not required)
      You can revoke your key if it is compromised
      Based on you not your email, so you can use the same sig with any email address
      You can even associate your picture with your key
      Optional Anonymity
      Strong cryptography
      Use the same keys for non email encryption
      Free

GPGmail/tools cons

      Less widely integrated.
      Does not work on devices yet.
      May break email lists (adding footers may change the sig, I haven't tested though)
      Can't help much until your have other people to use it with.

[1] With s/mime you can sign email documents even if your friends don't have s/mime that can still see your signature is validate.

[2] See the answer by Adam Liss (not the accepted answer) for the security issues [http://stackoverflow.com/questions/13512026/how-to-check- if-...](http://stackoverflow.com/questions/13512026/how-to-check-if-encrypted- s-mime-message-is-also-signed-without-decrypting-it)

[Edit: formatting]

<comment><USER11> Just two nitpicks: You can sign email with GPG even if others aren't using it (of course it will be of little value to them until they, possibly at a later date, verified your key), and it is supported on Android by K-9, I believe.

<comment><USER10> Yes you can, but my point was that this is of no utility with GPG. Whereas with s/mime _anyone_ can confirm that your email was signed (with many email clients), so there is value to using s/mime prior to all your contacts also using it.

<comment><USER11> > anyone can confirm that your email was signed (with many email clients), so > there is value to using s/mime prior to all your contacts also using it.

Provided that they trust the people handing out these certificates – with PGP, they need a chain of trust to your key to verify that it is you, with S/MIME, they have to trust random third parties. Or do I miss something and you mean something else that is possible with S/MIME but not PGP?

<comment><USER10> Right now if I send my Grandmother a signed email with s/mime she will see a little notice in her email client that says the message is signed and valid. If I send her an email with a PGP signature it will not.

This is because just like her browser the CA is trusted by her OS.

</comment></comment></comment></comment></comment><comment><USER12> How does this work? I assume all encrypted emails require both parties use the software, right? So, all my friends, associates, coworkers have to have GPGMail to read my encrypted emails?

<comment><AUTHOR> GPGMail uses a very well known and white spread technology OpenPGP as its base. Everyone you want to use it with has to have a mail client which supports OpenPGP in one form or another, but there are many plugins out there, who add support for your favorite mail clients on windows and linux.

Enigmail for Thunderbird: [http://www.enigmail.net/home/index.php](http://www.enigmail.net/home/index.php)

GPG4Win provide plugins for Outlook: [http://gpg4win.org](http://gpg4win.org)

Evolution on Linux has OpenPGP support built in.

FYI: Another term used for OpenPGP is gnupg or GPG, should you want to google for other solutions on other operating systems.

</comment></comment><comment><USER13> Huh, has Google said anything about the naming similarity between GPGMail and Gmail?

Edit: Sorry HN for not knowing something that you know and asking a question about it.

<comment><USER14> GPGMail predates Gmail by three years.

<comment><USER15> Not (initially) in the online domain, but I know of significantly sized, well established U.S. institution that was using "Gmail" (yes, exactly that term) publicly and in promotional materials including particularly mailings... oh, probably a good decade before Google.

<comment><USER16> Nobody sane would think GPGMail is infringing on Gmail in any shape or form.

<comment><USER17> It's good that legal teams never do anything that looks insane then ;-)

</comment></comment></comment></comment></comment><comment><USER18> That was far and away the easiest and fastest walk through from download -> sending an encrypted and signed email that I've ever seen. Obviously, it only covers one platform, but it's a great start.

</comment><comment><USER19> Mavericks is not yet supported it seems - "Incompatible Plug-ins Disabled [..] Contact the makers of these plug-ins for versions that are compatible with Mail 7.0 and Message 7.0."

<comment><AUTHOR> There's a special hack version out for Mavericks:

[https://s3.amazonaws.com/gpgtools/GPGMail- Mavericks-P2-hack....](https://s3.amazonaws.com/gpgtools/GPGMail- Mavericks-P2-hack.dmg)

And you need this fix for the latest DP 4: [http://support.gpgtools.org/discussions/everything/9888-mave...](http://support.gpgtools.org/discussions/everything/9888-mavericks- dp-4)

<comment><USER20> They just shipped support for Mountain Lion, which was released one year ago. I hope there is not as big a delay for Mavericks. I understand that it's a volunteer project, but a kickstarter could surely help them muster the $100 for a Mac developer subscription and access to the developer previews of 10.9.

<comment><AUTHOR> 10.9 is being actively tested internally and we've already released two hacked together preview versions for it. Doing everything we can to be on time this time for real.

<comment><USER20> Great!

</comment></comment></comment></comment></comment><comment><USER21> I currently use Mail.app's built-in signing and encrypting capability with a free StartCom S/MIME cert. Does this offer something more/different?

<comment><USER22> signature process is almost the same, but this thing uses different approach to "trust" to the signatures.

In case of S/MIME email client will tell you that signature is good if it the key, which was used for creating it was issued by a known CA (Certificate Authority).

In case of OpenPGP(GPG,GnuPG,…) you explicitly decide if signature is good either by verifying the key (once) directly with the sender or by using web- of-trust (you trust keys of your friends, who trust keys of their friends, who trusts your correspondent)

[http://security.stackexchange.com/questions/7874/how-does- pg...](http://security.stackexchange.com/questions/7874/how-does-pgp-differ- from-s-mime)

</comment></comment><comment><USER23> I am a bit surprised by the use of the modified Apple Mail.app icon. Are they connected to Apple somehow?

edit: just curious, sorry :/

<comment><AUTHOR> Not connected to Apple, just a very old icon which was never really updated. Might happen in the future though, to avoid legal issues

</comment></comment><comment><USER24> I thought they just announced a new version recently? Is this a new version again, or just a new website?

<comment><AUTHOR> All versions before this were beta. This is the stable version of GPGMail 2 with 77 bugs fixed from the last beta, so you should update.

<comment><USER24> Thanks -- it turns out the the update checker on my version wasn't working (for whatever reason), so I mistakenly thought I had the latest version.

</comment></comment></comment><comment><USER25> This is just MacOS nonsense. It plays in the Valley and at the mall, and nowhere else. The only reason it is on the front page of Hacker News is that we can't see outside our own event horizon.

Poke me when a popular web email service implements GPG.

<comment><USER26> Hushmail implements GPG, though perhaps it does not pass your definition of 'popular'.

Any current implementation of GPG by _web email_ is probably insecure as it would rely on JavaScript cryptography. Perhaps when the W3C passes the browser cryptography draft, and browsers start adding that in, we might see this. But the economics aren't aligned, because popular web email services want to see what you read and write, so they're not particularly motivated to give you strong encryption.

<comment><USER2> I wouldn't really recommend Hushmail:

[http://www.cybercrimereview.com/2012/11/hushmail-provides- un...](http://www.cybercrimereview.com/2012/11/hushmail-provides-unencrypted- e-mails.html)</comment></comment></comment>


Cherrypeel.com is a reddit like music service - <AUTHOR> http://cherrypeel.com

<comment><USER2> This is a pretty cool take on what a music version of reddit would be like. I like how simple and straight forward it is compared to the busy looking music sites out there.</comment>


Immutability and loops - <AUTHOR> http://codingrehab.com/posts/2017-07-31-replacing-loops.html <comment><USER2> I was just reading over some examples on the F# website and now this pops up. I like why functional languages exist I don't like the readability of the languages. If I find a nice piece of code it's like how John carmack put it "these small things are just toys" but unfortunately you have to learn them to be able to read a large production application. the thing is I grew up with C and variations to scripting and JavaScript and I can jump into nearly any code and read it if my life depended on it but that's excluding functional programming although I'm trying at the moment.

Screw classes and hour long design talks? Cool.

make everything immutable? eh ok....

write pure reusable functions? Awesome I'm on board.

ok let's see the language ? Oh it's that car crash of code that I have to jump all over the place to read what's this variable ? Oh it's a function that takes a function that takes a function...ok read that 10 more times and I'll know how this fizz buzz works.

</comment><comment><USER3> > Collections in languages that enforce immutability are optimized for these > kind of operations so they are very efficient and there is no penalty for > immutability.

That's just good old fashioned wrong.

<comment><USER4> Can you elaborate?

</comment></comment><comment><USER5> This page made my iPhone 5S really sad, scrolling was really laggy - i'm guessing all the clojure code was using something like clojurescript? If so, it really didn't need to as it was all static but causes a seemingly really large amount of computation while scrolling.

<comment><USER6> Likely it is the JavaScript. It looks like many of the code blocks are live, using Klipse[0], which is a multi-language client-side code evaluator. It's not ClojureScript specific.

[0]: [https://github.com/viebel/klipse](https://github.com/viebel/klipse)

Edit to add: it _is_ written in ClojureScript, however.</comment></comment>


Steve Jobs Understands Team Building - <AUTHOR> http://taosecurity.blogspot.com/2010/12/steve-jobs-understands-team-building.html <comment><USER2> It's amazing how many successful business types like SJ or PG recognize that people are the most important thing, and still the majority of high level managers view people as completely interchangeable cogs.

<comment><USER3> _still the majority of high level managers view people as completely interchangeable cogs._

I'd say it's more common to make people _feel_ like cogs.

<comment><USER4> I always cringe when I hear the word "resource" instead of "team member."

</comment></comment></comment><comment><USER5> This is what made interviewing at Apple so exciting recently. I didn't get this particular job (and plan to apply again), but the adrenaline rush from realizing that all the folks interviewing me were extremely bright, talented, motivated, and into their jobs was instantly addictive. It's very easy to understand why Apple works and how it keeps its people motivated.

<comment><USER6> Nothing personal, but you are highly confused about the real world. You will not be employee #10 at Apple and they will not value contributions as wanting to make a difference.

It's nice to think they will, but I so long as I have direct line to someone who thinks it's "different," I feel the need to tell you it's not.

<comment><USER5> I've held about 10 jobs in Silicon Valley. I can tell you, the environment at Apple was rather different, and my friends who work there paint a picture of their employment there that is indeed quite different from many other stories I've heard, and jobs I've held.

<comment><USER7> Is it the fear, the secrecy or the interrogations that stand out the most?

[http://www.geek.com/articles/apple/steve-wozniak-spills- the-...](http://www.geek.com/articles/apple/steve-wozniak-spills-the-beans-on- apples-fear-culture-20100426/)

<comment><USER5> Your comments are extremely unimpressive.

Comparing Apple to the Gestapo because they have a legit business need to keep new products secret until they are released?

Do you even know what the Gestapo was?

Are you even serious, posting this? Ludicrous.

<comment><USER7> Gestapo? I take it you think I authored the Gizmodo features that I linked to?

</comment></comment></comment></comment></comment></comment><comment><USER8> Related John Sculley interview on Jobs from October. It's a lengthy read but incredibly insightful for fans of Apple and great minds in general:

[http://www.cultofmac.com/john-sculley-on-steve-jobs-the- full...](http://www.cultofmac.com/john-sculley-on-steve-jobs-the-full- interview-transcript/63295)

</comment><comment><USER9> Really wish Steve could expand beyond the soundbyte and go deeper into how they hire, manage, do product dev, etc. Very much looking forward to the biography he is supposedly working with Walter Isaacson on.

<comment><USER10> The obstacle is that it'd probably reveal too much about Apple's secret sauce, or at least there's a perception that it would.

People usually cite all kinds of general reasons why Apple is thriving - product design, emotional marketing, vertical integration - while ignoring the unsexy foundation that makes all of that possible: hiring the best people and setting them up in highly interdisciplinary teams. And any edge there is going to be kept under wraps, especially since competitors like Google are doing it wrong - I've heard of how siloed the designers are there, just like at Yahoo or Microsoft. That's the real reason Google's products look flat and overengineered - their designers hop from team to team like little bees, they're not a core part of the culture.

</comment></comment><comment><USER11> Most people who read this article will self-identify as an 'A+ player'.

<comment><USER12> I definitely consider myself somewhere in the B range, but with the potential to be an A player. Today I'm A only in some respects.

<comment><USER13> Most A+ players probably self-identify as B rangers. :-)

</comment></comment></comment><comment><USER14> It is largely true that the quality of the people (from HR to security to engineering to the trainers at the gym) that I dealt with at Apple were excellent, but one seldom noted problem is that in an intensely competitive environment, there is little more demotivating than running into a jobsworth, particularly one that you have to work around to get things done. It can really derail the train.

<comment><USER15> I had to look up 'jobsworth'. Glad I did. It's an excellent word.

</comment></comment><comment><USER16> reading this quote:

"When you're in a startup, the first ten people will determine whether the company succeeds or not. "

gave me some immediate deja vu. pg quoted jobs jobs in 'how to make wealth', one of the better essays that made it into his book.

"Steve Jobs once said that the success or failure of a startup depends on the first ten employees. I agree. If anything, it's more like the first five"

<comment><USER17> Jim Collins (a "management guru" if you're up for that kind of thing) says in _Good to Great_ :

> Leaders of companies that go from good to great start not with "where" but > with "who." They start by getting the right people on the bus, the wrong > people off the bus, and the right people in the right seats. And they stick > with that discipline---first the people, then the direction---no matter how > dire the circumstances.

He repeats the "first who, then what" numerous times as the only way to turn- around a struggling company.

</comment></comment><comment><USER18> Ed Catmull always says that his approach to hiring was to find someone smarter than himself and that meant he was the dumbest person left @ Pixar.

</comment><comment><USER19> I talked at one stage about starting a business with another IT guy. He had many fine attributes, such as being fiercely adamant that if any manager ever described a person as an asset or a resource, that they would get sacked.

People deserve respect, not to be objectified by describing them using the same language as things that are bought and sold and casually discarded.</comment>


Ask HN: What's the best self-publishing platform for a technical ebook? - <AUTHOR>

Looking to publish an ebook and evaluating my options now. Any ideas on a good platform to use? I prefer writing in markdown. <comment><USER2> I'm using IPython Notebook. It has its pluses and minuses. The pluses are all code is inline, and you don't have to worry about having to keep things in sync (change the code for this plot, make sure the output gets named correctly and copied to the right place in the book). LaTeX is supported. It is easy to do animations with either javascript or Python. It is a very hackable platform. nbviewer.ipython.org will serve a static version of your book (it accesses it from Github, it does not store it on its servers).

Downside is it doesn't have the concept of 'books'. No chapters, etc. PDF generation is clunky at best. There is currently a version 3 in development that is changing the format of the ipynb format. It only supports Python, Julia, and Ruby as languages right now (but it turning into the Jupyter project with plans to support many more). You are kind of on the bleeding edge.

Mostly I like it, but I do wish there was more support/concept of books vs notebooks.

edit: This is not a recommendation; I did not survey all of the options. For me the mix of executable code and text and LaTeX in one document was a killer feature. That may be irrelevant for your book.

</comment><comment><AUTHOR> I decided to go with Leanpub. I really like it's philosophy of making it easy to push updates, which I consider particularly important to a technical manual. And of course, being able to write everything in markdown and push to a private Github repo to start a new version suits me like a glove!</comment>


Show HN: Cheqbook - Super simple fast accounting - <AUTHOR> https://cheqbook.com

<comment><AUTHOR> I made Cheqbook with my accountant co-founder after he got sick of only being able to recommend Quickbooks Desktop for his clients. Most people still use accounting on desktops and still have to move the company files around by thumb drive or email.

We tried the other online solutions first but found they were not proper accounting tools. They didn't have necessary reports and double-entry books. In most cases, we had to fight with the UI to get anything done. Ultimately, we built this so accountants and bookkeepers could handle more clients and business owners could concentrate on their business.

I would love any feedback.

</comment><comment><USER2> As a dev: Not sure why the blog sidebar needs to be on every page, especially Pricing. Doesn't come off as "pro" and at first made fear that the entire app may have been built upon Wordpress (I see it's not though).

As a user: Love it, need it and will be trying it out myself!

<comment><AUTHOR> > Not sure why the blog sidebar needs to be on every page

Good point. I will remove it tonight!

> fear that the entire app may have been built upon Wordpress (I see it's not > though).

I know. We really wanted to build out our sales site in Rails, but that would have taken resources away from the app.

>Love it, need it and will be trying it out myself!

Thanks! Here is a link for 90 days free instead of 30. <http://bit.ly/111Wp5n>

There is also a free tier.</comment></comment>


This is How Wrong Kurzweil Is - Digit-Al http://asserttrue.blogspot.co.uk/2013/01/this-is-how-wrong-kurzweil-is.html

<comment><AUTHOR> > But to say that we will see, by 2029, the development of computers with true > consciousness, plus emotions and all the other things that make the human > brain human, is nonsense. We'll be lucky to see such a thing in less than > several hundred years—if ever.

2029 may be a bit early, but I think that actually to say it will take at least several hundred years is nonsense.

Look at the timescales of scientific and technological progress. Pretty much 99% of all knowledge and technologies we use are less than 200 years old. Most of it is less than 100 years old. We pretty much went from zero to space in a single life time. And the progress is not steady, nor is it slowing down, it's _accelerating_. One thing Kurzweil is definitely right about is that people don't understand exponential growth. Or any superlinear growth for that matter.

<comment> Digit-Al I agree with you to a point. However, one of the big problems as I see it is that we don't have any clue what creates consciousness, and can't even agree what defines consciousness. It is very difficult to create something when you're not sure what the goal is.

I think the writer of the article I linked to makes some good points, but I don't necessarily agree with everything he says.

I think the problem with trying to predict something like that is that it is not fully a technological problem. We can make predictions about when the technology will be potentially capable of housing an intelligence, but it is much more difficult to predict when we might understand intelligence and consciousness enough to duplicate it. It could suddenly happen with the next few decades, or it might never happen.

</comment></comment><comment><USER2> This is how wrong Kurzweil is:

[http://en.wikipedia.org/wiki/The_Singularity_Is_Near#Predict...](http://en.wikipedia.org/wiki/The_Singularity_Is_Near#Predictions)

<comment><USER3> I am going to assume you're being serious here, and that you believe the link you provided proves "how wrong" he is. The other option is that you're stating the opposite, but I can't tell.

That said, I don't see how that link proves him wrong at all. I don't think it's reasonable to expect anyone to be exactly right with their predictions - no matter how informed they are - so some inaccuracies in his predictions are completely expected.

Especially when you are looking at an exponential curve, the absolute values along the way can seem far off, but compared to a linear approach they are a much closer fit to reality. Obviously the "singularity" is a huge point of contention because of how ridiculous it sounds, but if we disagree, we should come up with our own explanation. If AIs start improving on themselves in a self-directed way with similar pattern-recognition systems as our brain, do we not expect the advancement of technology to explode? If not, then why not?

It's really easy for us to sit back and criticize his very public, specific predictions without making them ourselves. The questions we should be asking are not whether the exact things he is predicting happen (although he has been very close in the past), but whether his method of doing the predictions and lines of logic make sense. If so, then he may be off by a constant factor, not an exponential one.</comment></comment>


Why Oakland's a Tech Start-up Game Changer - <AUTHOR> http://oaklandlocal.com/2014/12/why-oaklands-a-tech-start-up-game-changer/ <comment><USER2> I don't live in the US, but it's weird: downtown San Francisco people tend to look down on Oakland, but every time I've been, San Francisco is unique in having human feces on the streets.

In Oakland: there's a bunch of parks, a hacker space called the sudo room, a bunch of JS related companies included npm Inc are headquartered there and the last JSFest was there.

<comment><USER3> Please do not try to understand this through such a narrow lens.

San Francisco has human feces because it is one of the only cities in the USA which has not completely criminalized homelesness, because there are a lot of services there, and because once people decide that San Francisco is home, they really, really, really do not want to leave.

First of all, there are plenty of human feces in Oakland, but again, let's try and crawl away from that lens. If you become homeless in Oakland, the best advice I can give you is to spend your last $5 on a BART ticket to San Francisco. Oakland and a lot of Alameda County's solution to these sort of problems are a humongous jail called Santa Rita which is comparable to the size of the nearby San Quentin prison. Most of the people in there are there for minor crimes related to homelessness or poverty, and their situations escalate once they enter what is a major portal to our for-profit prison system, which per-capita is the most heavily populated in the world.

<comment><USER2> I'm glad you acknowledge the problem, but I think you're understanding it through a narrow lens yourself: there are many cities worldwide with the the same levels of homeslessness as San Francisco (or higher), none are as filthy. Hell even in the US Portland has methheads everywhere: they don't shit on the street. SF has a unique problem.

Your claims about for profit prisons may be true, but they don't change the fact that San Francisco is uniquely filthy.

<comment><USER4> Agreed. San Francisco is the filthiest city I've ever worked in. That the residents appear A-OK with public activity that would at least warrant a visit by social services or the cops in other cities amuses and amazes.

<comment><USER5> > San Francisco is the filthiest city I've ever worked in.

Literally.

How many of you reading this are in San Francisco and live in an apartment building? Go and check the trash room.

You'll probably find that Recology emptied out the bins today but left garbage, even if tied up neatly in trash bags, piled up against the wall or door.

Disgusting and unsanitary. What happened to common sense? What happened to people doing their job? Seriously, is there some law which says that trashmen are not allowed to place trash bags into a wheelie bin for expensive compacting trucks to get to work?

San Francisco is one of the worst run cities I have ever seen.

<comment><USER6> BTW, Recology will not pick up trash that's not in the bin because the trash collection service is billed by volume. They'll usually ask the subscriber to upgrade to a larger bin.

[That's what happens when a private company is granted a monopoly to perform a necessary public service.]([http://www.trashrecology.com/stop-the-sf- monopoly.html](http://www.trashrecology.com/stop-the-sf-monopoly.html))

</comment></comment></comment></comment></comment></comment><comment><USER7> According to Wikipedia, "Oakland has the fifth largest cluster of 'elite zip codes' ranked by the number of households with the highest combination of income and education". That's a good sign.

On the other hand, Oakland is also persistently near the top of the list of most dangerous cities in the US. One recent report put it at #2 [1]. That's generally not a good sign.

Is there much interaction between the high income/high education areas (presumably where you'd want to put your start up), and the high crime areas?

BTW, in addition to having one of the highest crime cities in the nation with Oakland, the Silicon Valley area also has one of the lowest. That would be Sunnyvale, which was the 10th on this recent list of safest cities in the US [2].

[1] [http://247wallst.com/special-report/2014/11/11/the-most- dang...](http://247wallst.com/special-report/2014/11/11/the-most-dangerous- cities-in-america-4/)

[2] [http://247wallst.com/special-report/2014/11/12/the-safest- ci...](http://247wallst.com/special-report/2014/11/12/the-safest-cities-in- america/)

<comment><USER8> Being in danger in Oakland is related to being part of a dense network of dangerous people. If you're not in that network, you're not in danger. I don't know why those people make their home in Oakland, but basically if you're not a criminal or friends with criminals than you'll survive.

<comment><USER9> Is this really true though? I mean even if you are not in that crowd you could just happen to be a witness to a crime. In that case I imagine they might not be so happy about it.

Or, probably more commonly, if you have nice things you could be a victim of theft.

</comment></comment></comment><comment><USER10> _"Oakland has a unique opportunity. It has an opportunity that no other region has anywhere in the world, so if Oakland can't do it, no one else can," Selassie said._

I don't see any reason to believe that. I mean if Oakland pulls it off -- diversifying tech and becoming an important hub in that regard -- awesome. But I see no reason to believe that it has to be something in the SF Bay Area. Part of what made Silicon Valley a growth opportunity is that that area was relatively undeveloped and it was, thus, cheap. Historically, a lot of our biggest companies were founded during a recession or depression and a lot of stuff was founded someplace cheap. In order to go live someplace expensive, you need to already be making good money. By definition, brand spanking new businesses and industries are not doing that.

So I don't see any real reason to believe that any place in the SFBA is The Spot to bet on for becoming "the next Silicon Valley -- only for X thing or with X twist."

</comment><comment><USER11> It's only a "game changer" because SF is now too crowded and expensive, so startups are moving to Oakland because it's still close by and a little cheaper.

</comment><comment><USER12> Oakland is incredibly close to San Francisco. If you want all the benefits of being in Silicon Valley's startup network with only a fraction of the cost, then Oakland is your spot. The only downside is that many businesses already realize this and have begun driving up renting costs.

<comment><USER13> > have begun driving up renting costs

The two bedroom I rented near the 19th St BART in Oakland was $1,700 in 2009 and is now on the market for over $4,000 a month.

<comment><USER12> In Oakland? That seems about right, considering it's a 2 bedroom, a comparable spot in SF would be closer to $6 or $7k.

Out of curiosity do you know the folks living their now? I'm always curious to hear who these people that are making enough money to justify a $4k rent payment.

The Bay Area never ceases to amaze me. The rent prices are so crazy yet the pay increase is negligible.

<comment><USER14> 19th St Bart is _right_ by Pandora, which (per glassdoor) seems to have a number of jobs around $100k. Cutting the cost and time of a commute is worth paying for.

<comment><USER12> That still seems way low to me. A qualified software engineer or designer can command a six figure salary remotely, thus eliminating a commute and the need to live in an expensive city. If Pandora is doing $150k for mid level developers then I'd say it's worth it, but not a penny less.

<comment><USER14> Not all those jobs were development roles.

[http://www.glassdoor.com/Salary/Pandora- Salaries-E262334.htm](http://www.glassdoor.com/Salary/Pandora- Salaries-E262334.htm)

</comment></comment></comment></comment></comment></comment><comment><USER15> It makes me happy to see other markets besides Silicon Valley and SF (yes I do not consider SF part of the Silicon Valley) get some attention about what they're doing.

US News did a great piece about my hometown, Portland about how it's becoming a great place to start a company.

[http://www.usnews.com/opinion/economic- intelligence/2014/12/...](http://www.usnews.com/opinion/economic- intelligence/2014/12/23/portland-oregon-is-a-hub-for-entrepreneurship-and- startups)

I can attest to this. SF is a great place to start a company, but Portland is a different world. People are much more collaborative. If you start a company here, you'll be able to find help. It's incredible. I'm Bay Area born and raised, but I'll never go back. The underserved markets are less toxic, more collaborative, and foster an ecosystem in which people can focus on their product or services without a lot of underlying tension.

</comment><comment><USER16> Great article. The diversity in tech is definitely apparent in Oakland. Not just in employee demographics, but in how companies tend to think in a different manner.

The article mentions that start-ups in Oakland are driven by a social mission, and it's very true. I work at VSCO and we're based in Oakland. VSCO is not about giving likes, being popular, or generating money through ads. We are simply about empowering the community with the art of photography. Without trying to sound too self-righteous, I like to think that the tech companies coming to Oakland also share a sentiment of contributing to social causes bigger than themselves.

<comment><USER17> > I like to think that the tech companies coming to Oakland also share a > sentiment of contributing to social causes bigger than themselves.

I'm pretty skeptical. It's telling, too, that there was no mention of the big companies with their names on towers.

<comment><USER3> I interviewed with one of them, and they picked a fight with me about advertising because I felt my previous employer had taken the absolute lowest possible road. I continually tried to acquiesce, even noting that they had a paid option, and they basically said that they hate their paid customers, who they lose money on (why not charge us more?!?), and that they only ever did a paid option because they were forced to.

Pretty sure I had been using their fucking service longer than anyone I interviewed with there, but whatever. ;)

<comment><USER18> Fuck Pandora. They use to have a mission. Nowadays they've even given up on the vaunted Music Genome project in order to play more "hits" and shove in more advertising.

<comment><USER19> If the startup is venture backed, the mission is creating shareholder value. The rest is just lip service. You should know as much too.

</comment></comment></comment></comment></comment><comment><USER20> My personal experience is trying to run a food related startup in Oakland. It didn't go so hot, but then again I think we tried to do too much.

We tried to make the food, deliver the food, and provide the technology to order the food. (a menu that changes day to day - different menu items based on what was in season, etc) In 2010.

It may seem that you can be the 'local boy makes good' when in Oakland, rather than SF. You can garner a lot of support from various groups by bringing business to Oakland as well. Just realize that as long as you are tech focused most of your customer base will be in SF.

<comment><USER8> Maybe people in Oakland don't need food delivered because we don't have multi- hour lines out the door of every restaurant like SF has :)

</comment></comment><comment><USER21> I found the title very hard to parse. It would be better as: "Why Oakland Is a Tech Start-up Game Changer"

</comment><comment><USER22> Really only two takeaways from an overly long article:

1) the diversity thing has more than a whiff of wish fulfillment to it. Repetition in this regard just makes the dubious sound downright desperate.

2) if money is really flowing to the next Netflix for Latinos, what stronger "sell" signal do you need?

</comment><comment><USER23> 45 comments? God you fucking sub-par coding python hipsters only care about things unrelated to actually doing work. Go make some commits on useful public libraries and stop circle jerking to your shitty cities.</comment>


Twitter: Like a Headless Chicken - <AUTHOR> http://brooksreview.net/2011/05/chicken/ <comment><USER2> I interviewed there recently and went through a few rounds - it was very clear to me in the interview process that the internals of twitter were chaotic.

Much like Google is today as well.</comment>


Or How I Learned to Stop Worrying and Love Premium - <AUTHOR> http://www.techamo.com/2011/12/01/or-how-i-learned-to-stop-worrying-and-love-premium/

<comment><USER2> I think people generally just have a really hard time understanding the value of a non physical good.

The product I work on has this very problem. We have saved thousands of people's lives with the application we made. Seriously, we get an email every single day where someone explains how they lost 50 or more pounds using the product we make and it took them from being pre-diabetic with high blood pressure to normal insulin levels with normal blood pressure. They have full energy again and are living a wonderful life now. This does feel pretty good when we hear these types of things, however we are a startup and we have investors that want to see a return. If the user does mention money, they always say how happy they are that they would easily pay 99 cents, maybe even all the way to 5 dollars for the app now that they know how good it is. Thats not a good sign for us. We made something that saved their life (maybe forever, maybe they will gain the weight back over months time period) and they value it at the same price as value meal at McDonalds that has a short term (maybe 20 minutes of enjoyment) and has the reverse effect on their life. I don't think the users measure the value the same way because our application is not a physical good. It is assumed by people that apps, software, or websites are little worthless toys that should be free (I don't know how computers came along and all of sudden people didn't want to pay for anything, though I did fall in this trap until I got my degree and worked in the field). I know my mother has this problem as well, the only apps she has ever bought for her iPad is when I bought her a iTunes gift card for her birthday. She loves the apps she paid for more then free ones. Especially, Words with Friends for example, but she has a hard time rationalizing how it is worth ~4 dollars yet has entertained her at least 20 minutes everyday day for the last year, and she hated the advertisements in the free one!

<comment><AUTHOR> Brandon, I completely agree. Though as someone who dabbles in mobile development on the side I am encouraged by recent trends. As developers become less willing to give away high quality apps for free or ruin the experience with advertisements, I think we will see a trend where more apps will be pay only and users will be forced to acclimate themselves to paying for apps in order to use the latest and greatest technology. I especially think the gaming side of mobile will help to drive away this resistance to premium. As hardware specs increase and the quality of games converges with those on PC and console platforms, I think the price will do the same. It will take time.</comment></comment>


Show HN: Ruby static type checker – proof of concept - <AUTHOR> https://github.com/stereobooster/diamondback-ruby/tree/master/examples <comment><USER2> If it meets the guidelines, this might make a good 'Show HN'. Show HN guidelines: [https://news.ycombinator.com/showhn.html](https://news.ycombinator.com/showhn.html)</comment>


Defunctionalisation: An underappreciated tool for writing good software - <AUTHOR> https://www.gresearch.co.uk/article/defunctionalisation/ <comment><USER2> Great article, great technique and lot of on-point advice for intermediate level programmers, BUT, it has the same problem many functional programming advocacy posts suffer from: the code examples are given in a language everyday programmers will probably not recognize.

Don't get me wrong, I love functional programming. I use Haskell and OCaml with joy. I also read about Idris, F#, Elixir and such quite often but I can also remember when all of this was alien to me.

Useful advice requires accessible set of examples and the very first example in this article (basic calculator) already make use of sum types, pattern matching, higher order functions and recursion, in a programming language with a relatively low adoption rate.

I don't know a solution to this issue without giving extra burden to the author. They can use one pure functional language alongside with a widely used, strongly typed language (Typescript, C++ etc) in their examples but that's probably too much to ask for.

Maybe my understanding of the target audience is wrong and my whole criticism is obsolete. Please correct me if that is the case.

I have friends with 5+ years of industry experience with languages like C#, Javascript, Java, PHP and they tend to verify my claims about accessibility of these type of articles.

Does anyone agree/disagree?

<comment><AUTHOR> I (author) agree with you. The target audience of this post was originally my coworkers, if I even targeted it at all; but there is certainly room in the world for more basic explanations targeted at people who aren't used to the idioms of functional programming.

The trouble is really that defunctionalisation is much, much easier if you've got sum types, pattern-matching, and higher-order functions. (It's not clear to me that it has any use at all if you don't have higher-order functions.) Is it worth the time trying to implement this sort of pattern in C#? I don't know. Insofar as C# is nice to write, it's because the IDE writes so much of the boilerplate for you, and no IDE is set up to admit this kind of pattern.

<comment><USER3> C#'s Expression<T> is almost exactly what you use in your example, baked into the language - you specify a lambda but the compiler emits construction of an expression tree.

[https://docs.microsoft.com/en- us/dotnet/api/system.linq.expr...](https://docs.microsoft.com/en- us/dotnet/api/system.linq.expressions.expression-1?view=netframework-4.8)

[https://docs.microsoft.com/en- us/dotnet/csharp/programming-g...](https://docs.microsoft.com/en- us/dotnet/csharp/programming-guide/concepts/expression-trees/)

<comment><AUTHOR> Indeed, we have used `Expr` and the `ReflectedDefinition` attribute internally in F#. However, if the user is capable of giving you literally anything, you'll struggle to optimise it! We've found it important to artificially restrict what the user can give us, by explicitly modelling the domain.

</comment></comment></comment></comment><comment><USER4> I might explain this concept to OOP-minded programmers like this:

Sometimes, you can improve your code by having it return a description of what to do, rather than doing the thing directly. This is like in SQL where you might return a query plan, rather than executing a query. Once you have this description, or plan, you can analyze, transform and inspect it before passing it to some execution engine that actually does the work.

<comment><USER5> _> I might explain this concept to OOP-minded programmers like this:

Sometimes, you can improve your code by having it return a description of what to do, rather than doing the thing directly._

...which requires you to turn functions into data structures and then interpret those.

Meanwhile, in OOP land you have objects, which can be seen as self- interpreting data structures.

<comment><USER4> I don't claim that this technique is impossible in OOP; it's just not as natural without discriminated unions and match expressions.

<comment><USER5> You're missing my point. In OOP every object _is_ computation represented as data. It's not some kind of "unnatural" design pattern programmer should concoct in a special way. Erasing the distinction between data and procedures (or getting rid of both, if you look at it in another way) in one of the fundamental ideas that lead to creation of OOP in the first place. It sounds like Alan Kay was more interested in getting rid of data, but it does work both ways.

Thus, saying "let me explain this to OOP developers" is highly ironic.

If that's not clear, let's go back to your description:

 _> Sometimes, you can improve your code by having it return a description of what to do_

Every object is (or at least should be) "a description of what to do" in the exact sense you're using here. This is crucial to understand for properly using OOP.

<comment><USER6> I think that's a vision of OO that not many OO programs that I've seen actually follow. It's more common to see OO programs where object methods represent the computations, not the whole objects, while the bundled data is used to store resulting state of the computations, with much or all of the data hidden to preserve invariants for that state. Just acting directly on the data like that without an intermediate representation of the computation itself is just the sort of liberty that the video I mentioned in the sibling post, warns about that can turn into unwanted constraints in some cases.

<comment><USER5> _> I think that's a vision of OO that not many OO programs that I've seen actually follow._

Maybe not, but it absolutely was part of the original vision. This is why Smalltalk 80 implements if/else statements and loops as methods, rather than keywords. A boolean in smalltalk is not just a value. It's a latent algorithm for choosing between two blocks of code at some later point in time. It's _also_ a latent algorithm for operating on other booleans via binary logic methods. Until you start seeing objects this way, you will not be able to appreciate the elegance of object-oriented programming.

Here are some examples of how this can work in non-trivial scenarios:

[http://www.vpri.org/pdf/tr2007003_ometa.pdf](http://www.vpri.org/pdf/tr2007003_ometa.pdf)

[https://bracha.org/executableGrammars.pdf](https://bracha.org/executableGrammars.pdf)

</comment></comment></comment></comment></comment></comment><comment><USER7> There are, of course, tradeoffs to using defunctionalization (I've also heard this called a "data DSL"). I have seen these tradeoffs often ignored or argued past when discussing various solutions that take advantage of it.

The cons to defunctionalization that I have experienced are things like:

You now not only need to test your application, but also the runtime that turns this data representation into actual work. Bugs can now occur in the runtime, in the reification of the app logic, or in the integration between the two.

If your application is particularly concurrent or lazy, then ensuring that your DSL works well with your main language's concurrency and laziness machinery can get pretty hairy when you start executing your side effects.

It becomes harder to leverage your languages developer tools; breakpoints and debugging often end up in your DSL runtime's code, not your application code, often requiring special-purpose tools to be built to debug your defunctionalized DSL.

Performance can also be a double-edged sword. On the one hand, you can do some very clever things; use tricks like memoization, all the way up to writing a JIT compiler for your defunctionalized DSL to improve performance. However, you're taking on that work due to the fact that your main language's runtime can no longer do that work for you. Often these data DSLs end up allocating a lot of data structures that end up being parsed and thrown away later, and those allocations increase work in cleanup and the parsing itself.

I also heavily question the efficacy of testing these data DSLs. It is objectively easier to test pure functions, but on the other hand, how do you validate that they are correct? Often we don't care about the actual data representation, we care that it does the work it describes; properly testing then essentially becomes a re-implementation of the DSL runtime with mocks / etc.

For a concrete example of all of these tradeoffs, take a look at React in the webdev world. React is unequivocally a good idea, but it has required a massive investment from the React team and the ecosystem to make it correct, make it fast enough, to create developer tools for it, and to figure out how to effectively test applications that use it.

</comment><comment><USER8> This one might be a bit more approachable: [https://blog.sigplan.org/2019/12/30/defunctionalization- ever...](https://blog.sigplan.org/2019/12/30/defunctionalization-everybody- does-it-nobody-talks-about-it/) as originally shared by [https://news.ycombinator.com/item?id=21916774](https://news.ycombinator.com/item?id=21916774)

<comment><USER9> There appears to be a fuller version of this talk on the author's own site: [http://www.pathsensitive.com/2019/07/the-best-refactoring- yo...](http://www.pathsensitive.com/2019/07/the-best-refactoring-youve-never- heard.html)

Agreed it's quite good.

</comment></comment><comment><AUTHOR> Author here: I'm happy to elaborate on any of this, except where I've signed NDAs about specific projects.

Also an obligatory "we are hiring" on behalf of G-Research; feel free to get in touch at firstname.lastname@example.com or firstname.lastname@example.com if you're interested in quant finance research/development in central London.

<comment><USER10> G-Research should come with the health warning of just how litigous and paranoid around IP theft they are. My understanding is that you can't have a personal phone whilst at work, you're weighed on the way in and out, and they sent one of their former quants to jail for several years.

One might suspect that the periodic renames (De Putron, Glouster Research, G Research) are mostly a tactic to distance themselves from the negative image.

1\. [https://www.bloomberg.com/news/features/2018-11-19/the- tripl...](https://www.bloomberg.com/news/features/2018-11-19/the-triple- jeopardy-of-ke-xu-a-chinese-hedge-fund-quant)

<comment><USER11> Its quite common in the quantitative finance world. Unscrupulous people steal a model, try and peddle it to a competitor. Thing is most models only work if noone (or only a few others) are doing the same. Usually when approached, most firms stay above board and report it to the person's employer.

Citadel, for instance sued a former employee for steeling a model (he emailed the source for a model to his personal email. He was sued in federal at 8am on a Monday, fired at noon. Criminal charges came months later. Tried dispose g of the evidence by tossing hard drives in the Chicago River. Dive teams were involved to recover the drives. But, Citadel already had all they needed because they monitored all outgoing and internal communication, including MITMing SSL email services.

<comment><USER10> I'm fully aware of the sensitivites of the quant finance world, but there are plenty of high quality workplaces out there that don't have the extremely invasive approach to security that e.g. g research do.

</comment></comment></comment></comment><comment><USER12> Fantastic article. I didn't know about this FP terminology but this seems to be a common, general concept.

While reading it immediately reminded me of:

(1) re-frame (ClojureScript) and Redux (JavaScript), which are both web- frontend libraries for managing state and event handling. Browser events dispatch plain, serializable data instead of invoking behaviour directly, which are in practice tagged union types, kind of similar to the provided examples? (I'm not very familiar with the syntax in the article).

(2) it seems to be a common idiom in Rust to move branching logic into pattern matching and enumerations. So defunctionalisation is applicable here and is likely very common to refine Result types and defer execution.

(3) as user edflsafoiewq mentioned this is strongly related to the OO Command Pattern

</comment><comment><USER13> Prior Art: "The Best Refactoring You've Never Heard Of"

[http://www.pathsensitive.com/2019/07/the-best-refactoring- yo...](http://www.pathsensitive.com/2019/07/the-best-refactoring-youve-never- heard.html)

<comment><USER14> I think this is a much better introduction than TLA, thanks.

</comment></comment><comment><USER15> This seems to be advocating to turn program programming into compiler programming. Like it took "every sufficiently complicated program reimplements half of lisp" and said that was an admirable goal and here's how to do it well.

I only skimmed the initial algebra talk linked out, but it seems to mirror what any given programming language does with your code. It takes your text, does some processing to turn it into an AST, then turns that into a compiled program and runs it. AFAICT the initial algebra suggestion says to basically program directly with your AST wherever you can, and then write code to turn that into a program. Now you've written an AST and a compiler, but naively interpreting that AST will probably be slow, so maybe you work in some optimizations. And you don't always write correct code, so you have to add a debugger too.

Now you've just written a programming language and toolset. Why not just use an existing language and mature toolset? Is it because your DSL might be constrained to the original problem well enough to be simpler than anything general purpose?

Replacing e.g. `map` and a native function with `MyMap` and a few of my own functions seems to be throwing out all the good that comes with a mature language ecosystem.

<comment><AUTHOR> Yes, I think you've basically summed it up very well!

One of the reasons we use this pattern internally is because we have quite specific performance requirements which are not well-served by your standard language compiler/runtime. If we handle some of this compilation workload, we can make sure we emit constructs in the underlying runtime which have the right performance properties. We also want to make sure our users don't really need to think about this sort of low-level mucking around with performance; so we use the initial algebra to expose natural data-driven abstractions to them, which we then carefully manipulate into the right forms for the .NET runtime to have our desired performance properties.

This is an area where F# really shines. Its "computation expressions" make it really easy to construct DSLs embedded in F#. We offer our users this DSL, and we promise that anything written using this DSL will be appropriately fast and safe; but we do sometimes give them an escape hatch. By encouraging the user to stick to this heavily restricted DSL, we make it easier for them to write code that we guarantee will perform well. If the code doesn't perform well, that's a big problem, but crucially it's a problem for the _devs_ to solve, not for the quant researchers to waste hours digging into.

Entirely separately from the above, we can also offer certain safety guarantees by restricting to our DSLs. One project in particular has involved taking something that previously existed but required a lot of manual procedural bookkeeping on the part of the user, and extracting the "intent" of the library in a purely data-oriented DSL. As long as the user sticks to our DSL, they don't need to consider how their computations are sequenced; we'll do that for them, in the process of converting from the DSL into the underlying system.

<comment><USER15> Are you effectively interpreting it, or are you compiling it to something else? I'm really curious how you do this in a performant way.

<comment><AUTHOR> We have places where we ultimately emit IL (the bytecode of .NET). However, usually we just emit F#, restricted to certain constructs which are allocation-free and so forth.

We aren't producing F# in the sense that we have to invoke the F# compiler ourselves, though. In the calculator example, given the description `Add (Negate (Const 5)) (Const 3)`, we might ultimately end up having assembled an F# function in memory like `fun () -> -5 + 3`; when this function is invoked, the value `-2` will be calculated. Ultimately we usually put together the functions in the usual way you make a function: by composition of smaller functions. The very simplest expressions like `Const 5` have a simple template like `fun () -> 5` which we can just directly produce; more complex expressions have to be interpreted recursively into F# function objects.

I agree it's a bit hard to explain unless you're trying to solve a less trivial problem :(

</comment></comment></comment></comment><comment><USER16> Lisp hacker summary.

Defunctionalization is the elimination of function objects (closures) from run-time representations.

Functions that don't carry an environment (such as global functions) can be replaced by symbols. For instance, a simple calculator's binding for the negation button - can use a symbol such as Negate instead of the #<closure> for the negation function.

Functions that carry and environment can be replaced by objects which represent that information with explicit properties. The object can somehow later be used as if it were a function anyway. (Either a closure can be made which references the object in its environment, or else the object itself can be callable like a function.)

These representations are more readable than #<closure> when debugging, usefully susceptible to manipulation by code, and susceptible to validation.

</comment><comment><USER17> In the testing example, I honestly don't see the value of the suggested approach over passing in two continuations. In production, the actual I/O methods can be passed in, and in testing we can pass in functions that validate their arguments.

This "dependency injection" type approach would be... functionalisation (?) i.e. the opposite of the suggested approach. But it also leads to greater decoupling.

I often find refactoring in the spirit of functionalisation _more_ powerful, because then what I do becomes less about implementing a specific piece of logic, and more about creating a robust library of combinators that can be puzzled together to implement the desired business logic. In my experience, getting combinators right is easier but also more productive.

With the suggested approach, changes to the logic requires mirroring sets of changes across many locations (because of the expected protocol of discriminated cases), whereas the functionalised code requires changes only either in the continuations or in the combinator, but generally not in both.

<comment><USER18> > I often find refactoring in the spirit of functionalisation more powerful, > because then what I do becomes less about implementing a specific piece of > logic, and more about creating a robust library of combinators that can be > puzzled together to implement the desired business logic.

The problem here is that your state is almost always transient and non- inspectable. Almost all the complexity in any non-trivial program is in the state, whereas the transformations are usually trivial and things like combinators don't really matter one way or another.

</comment></comment><comment><USER19> What programming language is that?

I can't read it enough to understand the article.

<comment><USER20> Looks like Haskell to me.

<comment><USER21> It's similar, but Haskell uses :: for types, and : for building lists, while the other ML family languages (and typescript) uses : for types, and some use :: for lists.

Also Haskell doesn't have modules, so I thought it was ML. Though I think someone said it was F#? I don't know those well enough to tell them apart.

<comment><USER22> Haskell has modules.

<comment><USER21> uh, parametric modules? Haskell tends to use Typeclasses instead. (Yes, I know each file gets called a module, but that's not really the same as seeing a "module" keyword in the file). Unless backpack counts? I haven't been following it.

I know there's a thing in Coq/ML/OCAML which uses the keyword `module` and can do some of the same things Typeclasses can do but aren't exactly the same. I don't know them well enough to explain them, but I know Haskell doesn't have them.

[https://gitlab.haskell.org/ghc/ghc/wikis/backpack](https://gitlab.haskell.org/ghc/ghc/wikis/backpack)

</comment></comment></comment></comment></comment><comment><USER23> The article makes a lot of references to an 'initial algebra', and there's a link, but it goes to a PDF of a talk. I didn't get anything useful from just the PDF itself.

Is there a video of the talk?

<comment><AUTHOR> When Skills Matter went bust, the only public recording I know of that talk vanished from the Internet. They've found a buyer now, though, and hopefully soon the talk will return. I will suggest to the talk's author that he hold it again so that he can put the recording somewhere more permanent.

</comment></comment><comment><USER24> The article was good enough. There was a lack of mentioning the many drawbacks that this approach can have[1], but it was a well written description of it.

Many of the comments, however, are... Not quite as good; there was a bit of elitism and maybe even zealotry[2]. As usual, it soured an interest in FP (not that I've ever needed it) that an even slightly humbler approach might have fostered.

Now I'll dabble a bit in the self-congratulatory tone of some of the comments: I have never written a line of F# and the last time I dabbled with a primarily functional language was a couple of toy projects in Clojure (quite different) years ago and yet I was able to follow along the code and the explanations.

Functional approaches aren't all that different or revelatory for someone with enough development experience[3] and it's just a different approach, who would have thought it?

[1]: Not unexpected; functional programmers who've convinced themselves that FP has no drawbacks are unfortunately common. I'm not saying the author is one of those, though; he probably thought the article was long enough as is.

[2]: Also not unexpected in any discussion about paradigms, _specially_ functional programming.

[3]: I've seen the general idea discussed here in different more multiparadigm languages.

</comment><comment><USER25> I don't completely understand this technique, so maybe I'm off in my interpretation of it. But it seems like something, flexibility-wise, between hard coding things and embedding a scripting language.

Back in high school I liked to work on MUDs (multiplayer text roleplaying games). It feels like this is sort of thing we resorted to a lot. Content creators that couldn't code could use a text UI to build up spells, skills, etc that the code would use to apply the specified effects. We had basic things (such as "fire damage") and more generic modifiers (such as "make the next thing apply to everyone close to you rather than just 1 person").

And every step of the way other subsystems could intervene and modify the outcome, so someone could e.g. create a monster that nullified all fire damage someone tried to deal to anyone nearby.

For people that did know how to code, we had a basic scripting language for more flexibility. But these scripts still plugged into the above system.

<comment><USER26> Yeah, it's very similar conceptually to adding a scripting language. But the point isn't to add user extensibility to the system, it's to have an introspectable description of the program.

So in the scripting analogy, what you'd do is take the program you wrote in C++ or whatever, and write a little mini-language that allows you to express all of the functionality needed to write the game in. So for example, C++ features like template metaprogramming aren't necessarily something you need to write games, but "spawn an NPC" is a very common task, for example. You reduce this language to just whats needed to write the game with.

The difference from a scripting language is that instead of the game being written in C++, with a little interpreter for the scripting language, instead the C++ is just a compiler for the scripting language. You take the script description of the game, and compile it down to the final game.

That little level of indirection allows you to do things like optimization, but also lets you do things like easily swap out how the program is evaluated. For example, in a graphical game, you could skip rendering to the screen, or advance time in jumps instead of needing to wait for timers, etc.

</comment></comment><comment><USER27> So.... this is just a fancy name for the advice "give your functions a name"?

The example he gave is just replacing an anonynous lambda for addition with a named function call Add?

Maybe I'm too critical, but this is a thing I see in many fields:

1\. Take a simple well-known idea and give it a new obscure name. 2\. Write blogs, give lectures, organize summit and conventions. 3\. Profit!

<comment><USER28> Add is a piece of data, not a function. The equivalent in C would be something like:

        enum function { add, ... };

        // then inside your code
        switch(myfunction) {
        case add:
            return a + b;
        ...

Defunctionalisation is about removing higher-order functions, not about naming anonymous functions.

It's not a new trend, the first citation on wikipedia is to a paper in 1972.

<comment><USER27> You're focusing on my not knowing the syntax and specific of the languages. It's still just giving names to things that did not have a name. Whether the name is a variable name, a function name, a type name, doesn't matter.

It's still a fancy, obscure jargonish way of saying "name things".

<comment><USER12> There is naming but that is not the point. The point is that you turn functions into (serializable) data. Nothing fancy happening here.

</comment></comment></comment></comment><comment><USER29> A lot of this and a few other links in this thread end up looking like creating an AST limited to the operations you want to expose to the user, in a serializable way. I think that's even called out explicitly in the articles. But I'm left thinking "what is the serialized structure of an AST?" It's source code. So we've implemented a language that then we have to use from within our code that is not as ergonomic as the host language.

In a way, it's kind of like type-safe eval(), with the limited nature of the purpose-specific implementation creating a sandbox of types around the eval, i.e. the impl isn't complete enough to give us turing completeness, File IO, or other things that would let the data turn into executable code that could escape the sandbox. So I'm curious to see if some wrappers around Roslyn[0] could get us less verbosity than Roslyn itself, a more complete impl that doesn't have to be written for every application, and the sandboxing necessary to prevent it from being handing the user a fully-automatic submachine gun pointed at our database.

Maybe it's something like a white-list for Roslyn structures. You could take a C# file, parse it with Roslyn, filter through the white-list, and either error-out if the user is doing something nefarious, or compile the structure and continue as normal. And then the code that we pass around as data could just be C# code, rather than having to have our own AST structures that we serialize on our own.

Maybe you could even allow limited forms of looping by injecting infinite- loop-breaking code that does some simple pre-/post-condition checking. Like "while(true) loops are only allowed to execute 10K times", or "for loops need to be bounded correctly for the update expression, and the index must not be modified in the body".

[0] Side note: my office is in a neighborhood called "Rosslyn", so I'm constantly misspelling one or the other.

</comment><comment><USER30> These are very useful patterns!

I feel that the thing that I haven't found my way around about this style of programming is how to make the types scalable. Once the type `Expr` is set to include literals, addition, substraction, and custom functions, there's no clean way to extend it in a modular fashion with more _typed_ operations (eg say you wanted to add multiplication).

When I say modular I mean either in a separate file, or in any other way that doesn't turn every function of `Expr`, over time, into a giant, unreadable pattern-matching statement spanning multiple pages once there are multiple dozens of operators in the type. Any suggestions for that?

</comment><comment><USER31> This could benefit from a discussion of when it is and when it isn't useful or practical to apply the technique described.

It helps with serialization, and optimization in cases like deep learning frameworks. In other cases I think it smacks of overengineering. If you need a representation of a function along with its arguments to pass to other systems, then do it as needed. Otherwise, YAGNI.

</comment><comment><USER32> I have a budding interest in ML-languages and was seeing a couple of references to category theory in this article and the linked slides. Are there any good resources for learning category theory applied to ML programmers or is there a list of category theory concepts that programmers will often come across?

<comment><USER33> I think this is fairly popular, though I have not read it myself: [https://bartoszmilewski.com/2014/10/28/category-theory- for-p...](https://bartoszmilewski.com/2014/10/28/category-theory-for- programmers-the-preface/)

</comment></comment><comment><USER34> As an F# developer with just enough category theory to follow along, I think this is really great.

I find the workaround for existential types in .NET particularly interesting, but it seems so verbose. Is there no way to do it with plain functions in order to avoid all the explicit type signatures?

<comment><AUTHOR> Thanks!

I'm afraid I don't know of any way to make the existential types hack neater. If you find one, I'm all ears!

There is an issue open with the F# compiler ([https://github.com/fsharp/fslang- suggestions/issues/567](https://github.com/fsharp/fslang- suggestions/issues/567)) to allow higher-ranked types, but there are (well- founded) objections that this could be a pretty big jump in language complexity.

</comment></comment><comment><USER35> _The major thing that we found was that you had to look at the whole problem._ \- Joseph Henry Condon, Bell Labs

... via [https://github.com/globalcitizen/taoup](https://github.com/globalcitizen/taoup)

</comment><comment><USER36> Guessing one of the highest velocity teams on the planet

</comment><comment><USER37> nice! I haven't seen this expressed in functional programming, but it feels more natural in ML.

Is it possible to nest these computations?

<comment><AUTHOR> I'm not quite sure what you mean by "nest". You can certainly have a computation expressed in terms of a defunctionalised initial algebra, which contains computations expressed in terms of different defunctionalised initial algebras.

</comment></comment><comment><USER38> Correct use of functional programming:

[The Pure Function Pipeline Data Flow v3.0 with Warehouse / Workshop Model]([https://github.com/linpengcheng/PurefunctionPipelineDataflow](https://github.com/linpengcheng/PurefunctionPipelineDataflow))

1\. Perfectly defeat other messy and complex software engineering methodologies in a simple and unified way.

2\. Realize the unification of software and hardware on the logical model.

3\. Achieve a leap in software production theory from the era of manual workshops to the era of standardized production in large industries.

4\. The basics and the only way to `Software Design Automation (SDA)`, just like `Electronic Design Automation (EDA)`.Because [The Pure Function Pipeline Data Flow] systematically simulates integrated circuit systems.</comment>