Ask HN: Self Hosted vs. Gmail / Outlook? - <AUTHOR> I'm currently using Gmail and getting more and more worried about my privacy. I'd like to know my options if I decide to jump ship. Should I setup my own email server ? Or use another less invasive service ? What are your thoughts ? How do you concile email and privacy ? <comment><USER2> If you're worried about privacy, I hope all your contacts use GPG. Practically none of my contacts know how to use that. Practically all of my contacts use Gmail and Outlook.com. That means all of my emails will end up in Google and Microsofts hands anyway. I'm sure they will build shadow profiles on me (especially since I'm a former customer, using the same domain alias on my new host as I did with them).

That being said, I recently switched to [https://mailbox.org](https://mailbox.org) (they have a very good reputation). Mainly because I love the web UI it's an awesome service and I get CalDAV and CardDAV that works beautifully.

<comment><USER3> I found that mailcow is a bit too heavy on the requirements. Instead I've started using mailcow ([https://mailcow.email/](https://mailcow.email/)) which runs a mail server in docker.

It's a bit less of a hassle to actually start up and keep maintained.

</comment></comment><comment><USER4> Setting up your own email server will bring you into the wonderful world of big email corporations not delivering your emails until you subscribe to their whitelist with, for some of them, subscription fee.

Back in the time, I had this problem with sending emails from my private server to yahoo or microsoft (hotmail, live.com, etc...), both refusing to deliver my emails to their clients as I could be a potential evil spammer.

For the subscription fees, a few weeks ago I saw a price chart for <I don't remember which company, probably Microsoft> about how much you have to pay them depending on your situation and how many emails you plan to send to their servers. Unfortunately I didn't find this page again.

I think the best option is to go for a paid service with a good privacy policy. It will cost you a lot less in time and probably in money. Also, they will probably be more reactive than you in case of problem, and more aware about security.

<comment><USER5> I've heard this before, but I have my own vanity domain, a well run mail server (postfix), and I I've never had a problem. I do support DKIM, SPF, DNSSEC, and of course I don't send spam.

I have heard that newly registered domains do have a period before they are trusted.

<comment><USER6> I can confirm that. It took me a while to get a sane mailserver setup, but eventually I don't have any troubles with Gmail/MS/Yahoo anymore.

<comment><USER7> I used to run my own mail server for ~7 years and never had issues. I used DKIM, SPF, etc and still do although they're problematic with mailing lists...

Postfix before, openSMTPd the last few years.

</comment></comment></comment></comment><comment><USER8> Use a paid service. I use Fastmail. It baffles me every day, how HN is obsessed with Gmail as if there was no alternative and are willing to trade a minor improvement in comfort over having every email read, analyzed, indexed, profiled and put into the Ad machine. I'm also surprised that so many people use the web client and not a native client with IMAP.

<comment><USER9> What are you talking about? HN is full of people pushing Fastmail over Gmail.

<comment><USER8> I'm talking about the number of stories about Gmail and their respective upvotes: [https://hn.algolia.com/?query=gmail&sort=byPopularity&prefix...](https://hn.algolia.com/?query=gmail&sort=byPopularity&prefix&page=0&dateRange=pastYear&type=story)

Gmail is popular on HN, and probably more popular than all other alternatives.

</comment></comment></comment><comment><USER10> After years of self-hosting I finally switched to [https://protonmail.com](https://protonmail.com) and I'm much happier for it.

Self-hosting is still possible nowadays but email delivery is an uphill battle. You can expect to write several major email providers to remove you from their blacklists even if your address and domain reputation is good.

VPS privacy and security is questionable and dedicated servers are usually expensive. Hosting SMTP from home is virtually impossible without a VPN to a "proper" IP.

Are you willing to spend the time to update all parts of your infrastructure on a regular basis? Are you certain you will keep up to date on recommended ciphers and protocols?

How is your data going to be secured at rest? If it's encrypted, how are you going to provide the keys during unexpected reboots?

If you want push notifications, synchronized calendars, contacts and notes you will need to add another layer of complexity to your setup.

Critics of Protonmail and similar will point out that browser based encryption is a weakness, however that doesn't change the fact that it is a major step in the right direction. The battle for privacy is fought in depth, not absolutes.

Protonmail is hosted in a Swiss datacenter, run by a Swiss company under strict data protection laws. They offer a free tier and a paid one for your own domains.

If you still want to go the self hosted route iRedMail and Mailinabox both work well. Sovereign runs too many services - it should really be split into VMs or containers.

<comment><USER11> I can second this. I used to selfhost mail and it was a lot of work to keep up. Nowadays I am also quite happy with protonmail but since I got cloudron installed for other services, I just enabled mail there and will see how that goes over the next few months, so far so good. Hopefully I can go back to email selfhosting through that in the long run.

Also I agree, the privacy implications when using a VPS is still something worth taken into account.

</comment></comment><comment><USER12> Hi, we had similar concerns about privacy and have built [https://cloudron.io](https://cloudron.io) to solve not only the hassle of setting up email but also other services where applicable selfhostable options are available.

The mail server is fully built into the platform itself and automatically takes care of all the tiny details required to get over the often stated deliverability issues (SPF, dkim, PTR, ...). So far we have found that many of the issues described here are not actually a big issue as long as everything is setup the way those large providers want it to be. The occasional report from a user about getting blacklisted usually is a matter of submitting the required form on the providers unlisting site. They do act timely as well in my experience and the process is not very time consuming.

Overall I was pretty surprised how well it works in the end, given that there are so many reports about selfhosting email is too complex to deal with.

<comment><USER13> Cloudron is the best of its kind, I always recommend it to non-technical people that need self hosted services.

</comment></comment><comment><USER14> I have been setting up mailservers since the '90s when you still had to deal with sendmail's configuration format. I've used most email servers available on unix platforms. And I'm also someone who wants to do everything myself and not depend on anybody else if I don't have to. Still, and it hurts me to say this, it might simply not be worth your time. I use a paid service for my main mailbox now.

I have a mailserver handling some personal email, but I feel it's too risky (to take the responsibility) and too much effort to host email accounts for just a few other people. You can and probably will be every once in a while blacklisted by one of the big providers or have legitimate email bounce, even if you have SPF, DKIM, TLS and your own spam filters set up. You also have to keep an eye on your servers to see if no new filth gets through. And you'd probably want to keep a backup relay ready. You have to provide ways for the users to configure or fine-tune their individual spam settings and mark messages. You most likely want to install a web interface next to the IMAP and/or POP service, which opens another can of worms.

I feel I'm too old now – meaning I have so many other responsibilities – that I don't want to babysit something that is after all rather crucial and should "just work". If you have the energy and time, please go for it, otherwise just search for a reliable paid service.

Note that assuming privacy when talking about email, even though most protocol interactions might be encrypted these days, is in my opinion somewhat misguided. Don't use email if it's truly private. Or use end-to-end encryption, such as PGP.

There might be a hole in the market for a company that helps geeks host reliable email servers, for those that want more control than just an IMAP account with sieve support, but maybe the margins are too low and fighting spamming subscribers too hard.

</comment><comment><USER15> I'm the ex-CTO of Lavaboom, a German startup that did encrypted email. Right now I'm working on Oakmail, which will be even more radically open and easy to use. I reckon it will be 2-3 months before we launch an open beta (and of course you will be able to deploy it any time once it's usable).

[https://oakmail.io/](https://oakmail.io/)

</comment><comment><USER16> If you have concerns for privacy, find a paid service you trust.

Hosting an email server yourself is a great learning exercise but you'll be forever playing whack-a-mole with spam and wondering if your setup is actually properly secure and waiting for the day you get hacked.

I did this myself for a few years and at one point had very few deliverability problems, then one day out of the blue I ended up on a black list and started getting complaint emails. After that it was either rebuild on a new ip address and start again or choose a paid provider and move on, I did the latter and opted for Fastmail.

</comment><comment><USER17> If you're concerned about privacy, don't use a free service. Pay for it and the privacy concern usually goes away. If you specifically concerned with US laws go German: [https://posteo.de/](https://posteo.de/) is a good one to consider.

<comment><USER18> How exactly do privacy concerns "go away" when you start paying? Your email is still readable by a third party.

On the other hand, I can see trusting a paid provider more for reasons of stability, level of support (in case, say, I lose access to my account), and continued development (the Gmail webclient has been relatively stagnant for a while now).

<comment><USER19> It may be readable by them but their life (as a business) does not depend on the content of your conversations. That's a big difference.

</comment></comment></comment><comment><USER20> If you're worried about privacy check [https://protonmail.com](https://protonmail.com)

</comment><comment><USER21> [http://mailinabox.email](http://mailinabox.email)

Formerly, I'd say maintaining your own email server isn't easy. It was hell trying to set one up 10-15 years ago. This guy (and the contributors) have made it about as easy as it can get. I've hosted mail for one of my domains on a DO droplet, where I set up a mail server with that guide. Been running it for ~4 years. No issues. Highly recommended.

<comment><USER22> In a similar vein there is docker-mailserver [1].

[1]: [https://github.com/tomav/docker- mailserver](https://github.com/tomav/docker-mailserver)

</comment></comment><comment><USER23> Interesting, I am not the only one. Additionally for me, its also pretty expensive to host domains for all my startup-ideas on gmail. I know, it's just $5/user/mo, but if you run 20-30 "fun ideas" it adds up...

So I used a scaleway.com instance and installed [https://github.com/sovereign/sovereign/](https://github.com/sovereign/sovereign/)

I forked it and made it especially for my own usecase working for the scaleway VPN [https://github.com/tomw1808/sovereign](https://github.com/tomw1808/sovereign)

So far I am pretty pleased. I opted against mailinabox because I want to use the server for other things too and mailinabox strongly suggests against it...

</comment><comment><USER24> I can highly recommend Mail-in-a-Box [1], especially if you're looking for a solution that is secure, easy to install, and doesn't require any fiddling. You can host it on a cheap VPS for $5 a month and it'll happily chug along without any problems.

Deliverability will only be an issue if you land up on an IP address that was previously abused, so it may be worth checking out the IP address reputation on DNSBL [2] before setting up Mail-in-a-Box.

Make sure you configure an SPF record for the server's IP address, and then also set up DKIM and SPF. I have yet to see any deliverability issues using this setup.

[1] [https://mailinabox.email](https://mailinabox.email)

[2] [http://www.dnsbl.info](http://www.dnsbl.info)

<comment><USER25> I find this funny because your own email is on Google apps.

</comment></comment><comment><USER26> You may want to consider German-based Tutanota ([https://tutanota.com](https://tutanota.com)) who uses open-source cryptography, rather than some alternatives such as Swiss-based ProtonMail who use a combination of open-source and proprietary closed-source cryptography.

Failing that, head on over to [https://privacytoolsio.github.io/privacytools.io/](https://privacytoolsio.github.io/privacytools.io/) and check out alternatives and other related information.

</comment><comment><USER27> I use GoogleMail behind a custom domain ($50/year) and am quite pleased. I used to be a FastMail customer but a couple minor outages and weirdness around billing made me switch a few years ago. Functional "report spam" is a big win for GoogleMail, and as an apps (or is it "GSuite" now?) customer you don't have to worry about ads/privacy issues. Don't self-host, I self-hosted (Postfix/Dovecot) before using FastMail and it was a huge headache between reasonable spam filtering on the receiving end, and undelivered/spam-marked emails on the send side, although I learned a lot by self-hosting. Also, it was hard to pretend I was serious about privacy/security when I was self- hosting on a box that any Linode admin could shell into as root, especially after Linode's security dramas. This is not to say that self-hosting cannot be cheaper and more secure than alternatives, but if you're not a full-time sysadmin regularly setting up mail hosts, you probably will get something(s) wrong. I never could silence that voice in the back of my head saying "what if some really important email couldn't be delivered to/from me?", which was sometimes right. As others have said, you have to use GPG if you're serious about privacy, regardless of your email provider. Anyway, for me, $50/year is a great deal for reliable email with good spam filtering, and being able to use my personal address/domain for Google Hangouts and Docs is a decent win for collaboration.

</comment><comment><USER28> I run Postfix & Dovecot (with SPF, DKIM, DMARC, DNSSEC, TLS) from my home network with a remote backup just in case it goes down, as well as my own DNS servers.

I had to ask my ISP to disable some rules on their end and pay a fee to have a static IP address, but overall it was pretty painless. Though I can imagine some providers being much worse.

After the initial hurdle of setting everything up in my experience everything went mostly fine. I had to whitelist my domain on Microsoft's site, but Gmail and Yahoo worked fine from the start. I haven't had a problem since. My university teachers receive my email just fine, so did my co-workers before I was given a corporate email address.

Is it worth it? Maybe not. It was more of a learning experience for me, but I find it works just as well as any other provider I've used. At least for now.

As others have said there are lots of outdated guides. I found the Archlinux Wiki and the manpages to be the most useful resources. Also please stay up to date on the software.

</comment><comment><USER29> Setting up Dovecot (with master-master replication) and Postfix (+ spamassassin, dmarc, SPF) isn't too bad. There's a lot of dated guides out there though. Stick to the man pages as far as possible.

</comment><comment><USER30> Running your own mail server is more work than it might seem, especially when it comes to setting up security and spam filtering and such.

If you want to use another web mail service other than GMail then I can recommend;

[https://kolabnow.com/](https://kolabnow.com/) (the lite option just gives you webmail)

or

[https://posteo.de/en](https://posteo.de/en) (very green-energy and privacy focused

</comment><comment><USER31> If you care about privacy _and_ freedom, check out Kolab Now.

[https://kolabnow.com/](https://kolabnow.com/)

<comment><USER32> Can someone share their experience with Kolab Now? I'm almost convinced but still have some doubts. What about them versus Fastmail? Fastmail is USA based entity so has their own cons. What about customer service if something go wrong? Can we report bugs somewhere etc.

I'm not planning to hide from Mossad or NSA, but have some above average quality and privacy service, not funded by bulk selling my data.

<comment><USER33> Fastmail is an Australia based company. The do operate servers around the globe and have a significant presence in the USA.

[https://www.fastmail.com/about/company.html](https://www.fastmail.com/about/company.html)

I am not implying this is any better or worse for your privacy. I personally use and trust Fastmail with the security and privacy of my email. However everyone concerned about such matters should do their own research and decided based on their own needs.

</comment></comment></comment><comment><USER34> The premise here is thatservices such as gmail or outlook don't respect the privacy of their customers. Can someone point me to an actual case where gmail for business (using gsuite) or outlook haven't respected their privacy engagement ? Or a serious report on that matter ? thanks.

</comment><comment><USER35> I chose Google GSuite to avoid non-delivery of emails which was happening when I was still on self-hosting. Most emails I send used to be marked as spam and blocked. Not anymore. Fee I pay every month is very low compared to the time I used to spend managing my own servers.

</comment><comment><USER36> I'm using gmail for my daily use and tried maintaining my own server and it was too much effort.

I'm a little bit worried what happens when gmail is blocking my account for whatever reasons, but if, i would create a second own managed mail address only for accounts.

</comment><comment><USER37> I'm really happy with [https://www.migadu.com](https://www.migadu.com). i just converted to paid. It's nice to be able to add users and domains without a price change.

<comment><USER38> One of the small businesses I work with has used them for ~6 months. Way too many rejected emails and mail server failures for my comfort (about 1%). I'm probably going to switch soon if the problems aren't resolved. Too bad, the product and price is perfect for our needs otherwise.

</comment></comment><comment><USER39> ProtonMail? Fastmail?

<comment><USER40> I just set up a new domain on ProtonMail. I like it. No IMAP though.

<comment><USER41> So it's just browser based, but they do have mobile apps it seems.

Also looking in to it there's a "IMAP bridge" being developed to allow use of traditional MUA.

<comment><USER42> Currently in beta [https://protonmail.com/blog/bridge-beta- signup/](https://protonmail.com/blog/bridge-beta-signup/)

</comment></comment></comment></comment><comment><USER43> Pay for a good email service instead of selling your private data to the advertisement industry. I personally use Fastmail and they are awesome.

</comment><comment><USER44> Setting up your own mailserver and especially maintaining it is a lot of work.

Hosted I would look at fastmail, mailbox.org and proton mail.</comment>


What's New in WordPress 4.9 and My Impressions About It – WPCouple - <AUTHOR> https://wpcouple.com/whats-new-wordpress-4-9-impressions/ <comment><USER2> Thanks for let us know about WordPress 4.9</comment>


Show HN: My first startup, Cubical Drift video game studio - <AUTHOR> http://www.planets-cube.com/

<comment><USER2> Nice to see some frenchies doing something in that space. The site says that all the elements are here for a first PC release, do you plan to release an early demo? Or keep the game playable by contributors (release regular builds that add stuff)? Or go the traditional route and wait to have a complete product?

<comment><AUTHOR> Indeed, the early "demo" (we call it alpha) will be playable by contributors few months after our crowd funding campaign. "Non Contributors" will have to wait for the final product.

</comment></comment><comment><USER3> Just curious, is this a rendered demo? If you started in October 2013 and actually wrote an engine that could do this in 4 months, its time for me to to leave software and go into lawn mowing. It looks great, and I like the concept, but devil is in the details of actually making it work.

<comment><USER4> You can see some texts like "concept art" or "actual game footage" in the trailer. The trailer is a mix of rendered images and in game captures.

We launched the studio in october but we are on the project since much longer.

Just to ease you, the engine is far form being finished! Don't leave software!

<comment><USER3> Whew, I have been working on a next gen voxel engine for over a year now. Its amazing what you can do with it, but due to the level of interaction, complexity keeps raising.

</comment></comment></comment><comment><USER5> Being a RPG and a construction game at the same time, I hope players will have the possibility to interact with the dungeons' layouts to lay traps for the monsters to kill them smartly instead of simply shooting them. I love RPG but I always regretted the lack of interaction with the environment.

</comment><comment><USER6> Looks really awesome. Since it looks like a fighting game, Something like this could be implemented for multiplayer: each face of a cube is owned by a faction in the beginning. Then the factions fight to gain the different sides.

Just a thought.

BTW awesome team portfolio. Good luck

<comment><USER7> Nice idea, but I am wondering: will too much fight impedes space development? Space discovery would really be what I'am looking for when playing such a game. Saying that, I might not be in the set of targeted customer for the game...

<comment><AUTHOR> We see this game more like a rpg than a pvp game, the first gaming hours will be dedicated for discovery. When the storyline is finished then players will surely start "pvp".

</comment></comment></comment><comment><USER7> Extract from the website: "While on a planet's surface, the view out into the rest of the universe is nearly endless! This means even from the surface of your own planet, you'll be able to see moons and other planets in the distance. To ensure the game runs with a variety of graphics cards, we have implemented an algorithm of sub-resolution to be able to display a quantity of blocks a individual graphic cards can manage. " May you elaborate on that please?

<comment><AUTHOR> Not every actual cards can display the billions of blocks we want to display. So we needed to create specific algorithms to reduce the number of blocks to display without "cutting" the viewing distance. This means that from every point in the system solar you will see every planets, allowing you to know where to go.

</comment></comment><comment><USER8> I'm actually really excited about this! Reminds me a bit of minecraft, but different theme and far more polished.

</comment><comment><USER9> How much experience do you have in game designing and developing?

<comment><AUTHOR> The dev team has 9 years of developing experience. One of us has 4 years of experience in gaming development.

</comment></comment><comment><USER5> Looks impressive, can't wait to play it !

</comment><comment> josh-wrale Reminds me of Hexen and Heretic. Awesome!

</comment><comment><USER10> Stunning graphics. Very cool!

</comment><comment><USER11> This looks nice.

</comment><comment><USER7> When will you release the game?

</comment><comment><USER7> Cute!</comment>


Ask HN: What are your recommended django web hosting provider? - <AUTHOR> <comment><USER2> Divio Cloud offers very easy to use dev and deployment tools: www.divio.com</comment>


The plumber programmer - <AUTHOR> http://www.johndcook.com/blog/2011/11/15/plumber-programmers/ <comment><USER2> Most of the enterprise developers' job is connecting components together. I hate it. I hate modifying existing systems and hate connecting them together. It is usually hard, but the kind of hard which is not really fulfilling for me. It is about fighting with accidental complexity created by other people. It is not why I started programming when I was 12 years old. But this is what needs to be done: it is rational from business perspective. Of course at least in my side project I create something new from scratch (a component-based UI system painted on HTML5 Canvas).

<comment><USER3> It's not that difficult when you work with the right tools.

Most of these problems boil down to a couple operations.

      map
      reduce
      filter

Most of the data comes from a couple places:

      Some kind of SQL system
      Text files
      Webservices (of the JSON/XML/HTML variety)

Most of the data goes back to the same places that it came from:

      Some kind of SQL system
      Text files
      Webservices (of the JSON/XML/HTML variety)

All you need to make are four types:

      inputs: () -> Some<T>
      outputs: Seq<T> -> ()
      pipes: Seq<T> -> Seq<Q>
      tees: Seq<T> -> () -> Seq<T> -> Seq<Q> (once partially   applied the tee becomes a pipe)

Using the usual sequence operators pretty much anything is possible with those inputs, outputs, and transforms (pipes) and you avoid a lot of overhead making data pipelines.

<comment><USER4>
      > Using the usual sequence operators pretty much anything is possible

i would absolutely love if this is true. however, i don't know of anyone who has constructed a convincing argument that it is true. even if it were true, it is probably impractical to require all systems you interface with to be composed in these terms. is it cheaper to work with a ball of mud (which poisons everything it touches), or is it cheaper to rewrite the ball of mud in the style you describe?

" _People build BIG BALLS OF MUD because they work. In many domains, they are the only things that have been shown to work. Indeed, they work where loftier approaches have yet to demonstrate that they can compete.

"It is not our purpose to condemn BIG BALLS OF MUD. Casual architecture is natural during the early stages of a system's evolution. The reader must surely suspect, however, that our hope is that we can aspire to do better. By recognizing the forces and pressures that lead to architectural malaise, and how and when they might be confronted, we hope to set the stage for the emergence of truly durable artifacts that can put architects in dominant positions for years to come. The key is to ensure that the system, its programmers, and, indeed the entire organization, learn about the domain, and the architectural opportunities looming within it, as the system grows and matures._"[1]

Maybe, you are correct. Probably. We all hope. But, practically, in the 'bigger' problem domains, you can put a lot of smart and experienced people on a project, and it still comes out a ball of mud. Maybe, there just aren't yet enough experts to go around.

[1] <http://www.laputan.org/mud/> (conclusion)

BTW, any references you know that can articulate your assertion without hand- waving, I would love to read. I must read. I'm currently devouring everything I can about FP and there's a lot of concrete stuff, even more stuff with handwaving, and then some alarming articles about people who did their startup in haskell and wouldn't do it again, even if for out-of-band factors.

<comment><USER5> Well I think its only cheaper at first and even if people know better they are forced to use a stupid java framework (that abstracts away the network) even if the know that it will end up badly.

A other problem is that people often get tought that you need <input some bad framework>. You can't do distributed computing if you done use some kind of framework. At least the places I know they would never tell you something like "just send json from one node to the other if thats all you need".

At clojure conj there where some talks about this, but the videos are not out jet. See these presentation on Concurrent Stream Processing ([https://github.com/relevance/clojure- conj/blob/master/2011-s...](https://github.com/relevance/clojure- conj/blob/master/2011-slides/david-mcneil-concurrent-stream-processing.pdf)) or this on Logs as Data ([https://github.com/relevance/clojure- conj/blob/master/2011-s...](https://github.com/relevance/clojure- conj/blob/master/2011-slides/mark-mcgranaghan-logs-as-data.pdf))

For a other example that works in quite simular ways look at Storm (in use at twitter. Its all sequential abstractions. See this video by Nathan Marz (look all videos you can find): <http://www.infoq.com/presentations/Storm>

For a more philospical perspective look at the videos by rich hickey: <http://www.infoq.com/presentations/Simple-Made-Easy>

</comment></comment></comment></comment><comment><USER6> Definitely. An awful lot of programming is "Take data from place A (in format 1) and place B (in format 2) do magic thing J to it and put it in location Z (in format 3)".

Process J is usually the simple bit - the hard bit is taking it from A and B, converting (1) and (2) so that they can be used together, dealing with all the different eventualities that can cause things to go wrong when fetching the data, parsing the data into (3) and then writing to Z (while dealing with all the things that can go wrong when writing to it).

<comment><USER7> _"Yeah, meanwhile Z is temporarily unavailable and by the time it's available again, the data from A and B is no longer consistent, so you have to re-query again, ouch, your pointers to last read data on A are no longer valid anymore and yeah, we forgot to tell you this little funny thing about B... <the dramatic music in the background pauses here>"_ \- this is what you should be thinking when looking at a small arrow drawn on a whiteboard.

<comment><USER8> In my experience in Big Freaking Enterprise development, the issue is closer to "this little funny thing about B" x 100, plus "group Q doesn't want this project to succeed", "you have to be compliant with Regulation F which means, most relevantly, that all good technical solutions to this problem are legally forbidden", "your success with this will threaten the budget of the Z department -- be prepared for a knife fight with the VP", "group P will veto the project if it is delivered with property G and group R will veto the project if it is delivered without property G (bonus points awarded for values of P equal to R)" and "by the way, this doesn't advance an actual business goal."

The technical details are usually pretty boring. Data goes in, data comes out, if it weren't for all the bloody people involved it would be very civilized.

<comment><USER9> I spent over _two years_ trying to get the contents of a single dropbox on one form changed. Coding time, about 10 minutes. I guess if you want to look on the bright side you can think about how much political weight a single HTML element can carry.

</comment></comment></comment></comment><comment><USER10> Very true.

Funny anecdote, last time I heard about "plumber programmers", the meaning was very different and actually quite pejorative. The person I was talking to was referring to the type of programmers that can only write applications by assembling third party components such as ruby gems while lacking the algorithmic skills to solve problems that haven't been solved before.

<comment><USER11> I've talked about this too. Sadly, this is what programming has started to become: slapping third-party components together and calling yourself a developer.

It's one of the reasons I tend to stay away from things like Django and RoR. It makes it easy to develop applications quickly, but I many times have to create hacks to make any changes that don't fit into the one-size fits all libraries.

</comment></comment><comment><USER12> I think it depends on context.

A well-architected application (they _do_ exist) where a lot of work was put into designing the arrows and boxes will suffer from this effect less so than one where the arrows and boxes were added on haphazardly. In fact, I would argue that spending too much time worrying about the arrows is a sign that something's likely wrong with your architecture.

Of course, that's not to say that architecture can cure this altogether. In particular, enterprise software tends to focus on integration a lot, and mostly because enterprises have a lot of boxes that need to be put together.

Startups will tend to do a lot less of this, but it's still important.

</comment><comment><USER13> For the past 8 years I've been designing .NET enterprise integration solutions and I love it. I love the problem solving that is required and working on getting square pages to fit seamlessly into round holes. Recently I switched companies and was working with jQuery and PHP doing front end things like UI validation and was losing my mind. It's not the pencil and paper type of challenge as integration design.

<comment><USER14> Wow, could you elaborate more? This sounds like a very interesting viewpoint, and one that I've not seen before.

</comment></comment><comment><USER15> I always shudder when people say it is just a simple integration project. The arrows are the hardest part.

</comment><comment><USER16> I've used the "plumber" term to refer to my own deeds, if only because they don't resemble the shiny and showy stuff which is out there. I don't make games, and I don't generate graphics. My stuff tends to be the glue which makes other things happen in the first place.

I'm far more likely to write the server underneath that huge multiplayer game and deal with all of the systems stuff, in other words. The actual gameplay, graphics, sound, and user interface? That's someone else's realm.

</comment><comment><USER17> Reminded me of this section from the Rspec book - <http://cl.ly/2x231s0N2v3C2P1S1z04>, <http://cl.ly/2R1Z1D450Y1n1n3L170b> \- which applies to testing object-oriented code by mocking objects and putting assertions on the interaction between objects (vs vis versa).

"Object-oriented systems are all about interfaces and interactions. An object's internal state is an implementation detail and not part of its observable behavior. As such, it is more subject to change than the object's interface. We can therefore keep specs more flexible and less brittle by avoiding reference to the internal state of an object."

This 'Plumber Programmer" article is meant to be more general than object- oriented code, but I think the analogy applies. Building systems is about interaction and communication - system state is just an illusion that gives us a first approximation of how things talk to one another.

</comment><comment><USER18> I was hoping this was going to be about programmers that moonlight as plumbers on the side.

</comment><comment><USER19> At the telecom company I work for, this is called 'System Integration'. It takes most of our time.

</comment><comment><USER20> Tangentially related: the plumber program from UNIX R11^W^W Plan 9 makes programs work together by piping text around. <http://doc.cat-v.org/plan_9/4th_edition/papers/plumb>

Again, there is more value in the connections than in the individual programs themselves.

</comment><comment><USER21> Seymour Cray said that he was an overpaid plumber....</comment>


Is it smart to use DropBox for backups?  - <AUTHOR> http://beredundant.com/2010/is-it-smart-to-use-dropbox-for-backups/ <comment><USER2> Synopsis:

Yes. And use my url-shortener-disguised-referral-links to sign up, giving me a referral bonus of extra space.

</comment><comment><USER3> You could also encrypt with AxCrypt or similar beforehand to give defence in depth</comment>


Hacker Claims He Gave FBI Info That Led to Killing of ISIS Leader - <AUTHOR> http://www.buzzfeed.com/josephbernstein/hacker-claims-he-gave-fbi-info-that-led-to-killing-of-isis-l#.epKX3V6yPG <comment><USER2> There is so much propaganda and lies involved with this conflict that I don't trust anything anyone says anymore.</comment>


Let Me Think - <AUTHOR> https://letmethink.io <comment><AUTHOR> Let Me Think is a product based on a connected totem that you need to touch with your phone to launch a workspace on it. Through a scientific and design approach we figured out that a conscious use allows you to our behavior with our phones at work.</comment>


Free, 3 week course on product design - <AUTHOR> https://swd.substack.com/p/coming-soon <comment><AUTHOR> A couple of friends and I open sourced our 3 week course on product design. We're hoping this can serve as a resource to those looking to up-skill or find accessible resources to learn design. Just sign up for the newsletter and get the course straight to your inbox.

We have (combined) design experience from companies like Instagram, Dropbox, Figma, Spotify, Apple, and more. We also taught this course when we were students in college :)</comment>


Some 2020 MacBook Pro and MacBook Air Users Experiencing Issues with USB 2.0 - <AUTHOR> https://www.macrumors.com/2020/06/17/macbook-pro-usb-2-0-accessory-issues/ <comment><USER2> one of my only major gripes with my 2015 MBP is the usb performance, constantly dropping connection or being generally unreliable with harddrives etc. a shame to see this hasn't improved with the new models....</comment>


Yes, You Can Be Fired for Getting Covid-19 - <AUTHOR> https://www.theatlantic.com/health/archive/2020/06/sick-leave-covid-time-off/612361/ <comment><USER2> "in the USA".</comment>


What skills doth a good developer possess? - <AUTHOR> http://careers.directi.com/pages/viewpage.action?pageId=30638092

<comment><USER2> At the bottom of the table humility is marked as a "must have" for everyone. I disagree completely. I'm from the "laziness, impatience, and hubris" school of thinking popularised by Larry Wall: [http://en.wikipedia.org/wiki/Larry_Wall#Virtues_of_a_program...](http://en.wikipedia.org/wiki/Larry_Wall#Virtues_of_a_programmer)

I'm not trying to argue that people are justified in being rude, cruel, or impolite as a matter of course. It just takes a certain degree of arrogance to look at the state of the world and declare that you can do better. Yet innovation would never happen without people doing exactly that.</comment>


Disney CEO Bob Iger immediately steps down from CEO position - <AUTHOR> https://techcrunch.com/2020/02/25/disney-ceo-bob-iger-immediately-steps-down-from-ceo-position/ <comment><USER2> As a former Disney employee and someone who's kept up with internal politics, even though his contract was up at the end of next year this feels sudden and not in line with how Disney tends to do things. Disney has quite a number of disparate businesses, from theme parks to streaming services to movies. Chapek is a business guy that has done consumer products and parks (parts of the business that aren't huge growth areas), and hasn't even been president of parks for that long. He has no experience on the creative side. Usually the future CEO is appointed to a COO/President role in order to help them learn the parts of the business they don't have experience in. It also helps vet if they'll be capable of leading all those business heads, which is how they determined not to go with Tom Staggs after he was made COO. Not doing this is pretty irregular, especially with all the shakeup that occurred with Fox. They're still working through a lot of that.

Similarly, I don't buy the story that Iger is moving into a creative leadership role. It's true there's been a vacuum there since Lasseter left, but unless there's long term tenure planned, Iger won't be very influential. Anyone who's worked at a large company knows that no one listens to the guy whose only responsibility is to have creative ideas if they have no other real responsibility and will be out in a few months. Iger put himself into a meaningless, insignificant role, and no one has satisfactorily explained why.

I don't think there's a scandal. If there was, the narrative around his departure would be different and he wouldn't be sticking out to the end of his contract. He wouldn't still be chairman of the board. I've heard a rumor the board is upset about Star Wars, but Q1 2020 was pretty solid and Mandalorian was well received. I'm actually worried he's sick, though I hope not because I admire him a lot. Regardless, I do feel like there's another shoe. I guess we'll have to wait to find out what it is.

<comment><USER3> Why would the board be upset about star wars?

<comment><USER4> The first movie I remember ever seeing was the first star wars movie at a drive-in when I was 5 years old. I love the Star Wars movies. But, I haven't even bothered to see the one that came out this past December. I'm not even sure of it's name. I was severely underwhelmed by the previous ones in the most recent trilogy. They weren't as bad as the prequels, but bad enough that I honestly don't care whether I see the most recent one or not.

<comment><USER5> My exact sentiments. When Disney Plus came out, I actually attempted to watch the entire star wars lineup from start to finish, the prequels (episodes 4-6) were unwatchable, couldn't even get through those, but watching (more like sitting through) the last 2 reminded me of exactly why I wouldn't be seeing the latest one ...

<comment><USER6> The Prequels were episodes 1-3. The Original Trilogy was episodes 4-6.

</comment></comment></comment></comment></comment><comment><USER7> I read "ride of a lifetime" by Bob Iger last month. It was a pretty nice book, similar to ShoeDog by Phil Knight. I learnt a lot about leadership, about Bob himself and started to like him (even though I had no idea about him before the book).

A few things that stuck with me: 1\. Have three priorities or less. Otherwise people can't relate. Jim Collins also said: "If you have more than three priorities, you don't have any". 2\. He did everything to show that he cared about what his people were doing. He would take part in many aspects of the star wars movie (and be there). This sent a message, a good one. 3\. Earning trust is important. He did Pixar acquisition well. That earned the trust of Marvel CEO.

I recommend checking that book out.

<comment><USER8> I'm halfway into that audio book (which is self-narrated) and so far it's been great. Point #2 is definitely a theme. He seemed to truly care about what impact his actions would have on his employees and consumers.

</comment></comment><comment><USER9> I just finished his book last night (it's a really good read), and this feels weird to me. It was clear he was going to leave soon, but why now?

He talked so much about succession plans and outward appearances (like, that was most of the book, honestly...), and this feels so weird that out of nowhere Bob Chapek became CEO. He doesn't seem (at the surface) to represent what Iger does, and there was no clear succession plan or public narrative put forward. (Like, really, the whole book was about 50 years of succession plans and presenting things in a way that wouldn't cause issues externally... and this was the opposite of that.)

I also remember thinking how poor Eisner's legacy came off in his book, and was sure he'd optimize for a great legacy. He is ending his tenure on a relatively high note for the company, but something about the exact timing feels really off to me. Especially right after such a huge press tour for himself.

I don't think it's a scandal; he wouldn't stick around. The only thing I can think of is a political run (maybe not for president? Seems too late for that, and he seems to smart to run for something he can't win), or possibly family issues (sickness, etc).

<comment><USER10> He also has been on every TV show and podcast he could be recently.

Like it's as if he calendared retiring and he is executing on post retirement plans (book and publicity) without stepping down.

If he's checked out and not working...and the next guy is identified and in waiting....it's reasonable the board would boot him early.

Heck in his own book while Bob was waiting to be CEO he approached Goldman to do the Pixar deal. Which is super....well.....forward looking / potentially undermining to the current CEO. Maybe history repeats.

</comment></comment><comment><USER11> This is pretty shocking to me. I literally just got my shareholder information in the mail.

When I bought Disney stock it was a 20th century management style company. Now it's one of my best Blue Chips. Bob Iger's leadership has helped make Disney into the media behemoth we now pretty much can't escape whenever we turn on the tv or check the news.

I only hope that there are no scandals that are the cause of his move from CEO to Executive Chairman.

</comment><comment><USER12> Weird, techcrunch.com is serving an expired SSL certificate.

        $ openssl s_client -showcerts -servername techcrunch.com -connect techcrunch.com:443
        CONNECTED(00000003)
        depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
        verify return:1
        depth=1 C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA
        verify return:1
        depth=0 C = US, ST = Virginia, L = Sterling, O = Oath Inc., OU = SecOps, CN = techcrunch.com
        verify error:num=10:certificate has expired
        notAfter=Feb 26 12:00:00 2020 GMT

<comment><USER13> Same here : [https://crt.sh/?id=2500908236](https://crt.sh/?id=2500908236)

12h long certificate

</comment></comment><comment><USER14> "Speculation is still high for the exact reason behind Iger's departure, with many hoping for something benign (ish) like a presidential run versus a personal issue."

This country seems desperate for the hope of better leadership.

<comment><USER15> Wish they'd stop looking to CEOs then.

<comment><USER16> Unless leadership skills have any overlap in the private and public space.

<comment><USER17> They don't.

CEO's don't have to build consensus. They can, and do, purge their ranks of dissidents. Iger, in particular, has been aggressive about this over time.

You cannot simply "fire" the people who disagree with you when you are President. We have seen what happens when you try to do that.

<comment><USER18> Of course the President can fire the people who disagree with him. In fact he should do that if the people are preventing the implementation of the President's lawful policies. The Constitution is pretty darn clear on this point:

Article 2. Section 1: The executive Power shall be vested in a President of the United States of America.

Of course it is also possible for a President to make poor decisions, but the trend towards criminalizing policy differences is not a healthy one. Elections are the solution to poor executive decisions, which is _NOT_ the same thing as unlawful actions.

<comment><USER19> > In fact he should do that if the people are preventing the implementation of > the President's lawful policies.

I believe the unwritten assumption was that the president was firing people who disagreed with him when he made unlawful policies.

</comment></comment></comment></comment></comment></comment><comment><USER20> Probably just taking advantage of the massive market downswing yesterday/today. They know the Disney stock price is going to tank in the short term along with the rest of the market, might as well rip off the band- aid of Iger leaving while people are already selling.

</comment><comment><USER21> Is it possible that he believes that the markets will take a hit in the near future with the virus scares these last two days?

If I were CEO of a behemoth giant like Disney and held the credit for its rising in several industries, I'd bounce ship before it all went down to avoid any blame?

Maybe I'm naive, but this thought crossed my mind.

<comment><USER22> COVID-2019 is going to hit everybody more or less to the same degree. The only people who are going to be significantly damaged by it are those who bet everything on doing good in 2020 (which is still more likely to be seen as bad luck than poor skill), or those who manage to absolutely screw the handling up.

<comment><USER23> Disney will likely gain from it: what better time to be selling digital subscription services than when everyone is locked up in their houses and schools are closing?

<comment><USER22> Given that a lot of people who would be interested in Disney's subscription service probably already have it, I don't think there's likely to be a major increase in subscriptions due to coronavirus-induced shutdowns.

People cancelling their trips to Disney parks and cruises will likely hurt them a lot. If people start cancelling spring break en masse, or this lasts long enough to affect the summer season, the lost ticket sales will be painful.

<comment><USER24> Disney parks in China where they've invested billions are prolly already feeling the heat.

</comment></comment></comment></comment></comment><comment><USER25> Bob Iger has long voiced his desire to retire, it's been postponed several times. He wrote about it at length in his most recent book. Just pulled the trigger.

<comment><USER26> Surely you can see why "I have decided to retire right _now,_ effective _immediately,_ today, not even finishing out the week, going home this afternoon and never comin' back see y'all on the golf course" is going to raise a few eyebrows. It's... not the way this sort of thing is usually handled.

<comment><USER25> Yes, that would raise eyebrows, except that's not the case here at all.

<comment><USER26> Those weren't his literal words, but I trust you get my point: in most cases, this sort of transition is handled with something like "Joe Foobar will step down as FoobarCo's CEO at the end of next month, transitioning to [insert obviously ceremonial position]." This is an immediate transition to a position which isn't obviously ceremonial at all.

I'm not suggesting anything tragic like a fatal illness, a #MeToo event, or a presidential run, but only suggesting that this is an unusually-handled event. Look at his predecessor Michael Eisner, who was forced to resign after losing a boardroom battle -- Iger was announced as the next CEO in March 2005, but Eisner didn't actually _leave_ until the end of September 2005, over six months later.

<comment><USER27> It's hard to tell from outside of any place. I felt once that I was a victim of attempted "constructive discharge", but I was not (and never have been) immediately marched out after giving my two week notice. Conversely, I had a boss that I never heard anything bad about or a scandal, who _was_ escorted out with his stuff mailed to him after like a decade of service. Nobody admitted to knowing why, except for the usual generic reasons for a layoff.

</comment></comment></comment></comment></comment><comment><USER28> It was announced in April, so I'm not sure exactly why people are saying it's sudden. Am I missing something?

(Stupid headline, but it's what Wikipedia had cited for it prior to this month.)

[https://www.cbsnews.com/news/abigail-disney-tells-house- comm...](https://www.cbsnews.com/news/abigail-disney-tells-house-committee- ceo-bob-igers-65m-pay-is-a-moral-issue/)

<comment><USER29> The "walking away" in that statement had nothing to do with leaving the company, it was about his take-home.

<comment><USER28> The revision history of the article that cites that post is really interesting, I guess someone in the know let on more than they were supposed to?

[https://en.wikipedia.org/w/index.php?title=Bob_Iger&oldid=92...](https://en.wikipedia.org/w/index.php?title=Bob_Iger&oldid=928585142)

</comment></comment></comment><comment><USER30> Multiple CEOs stepped down today (02/25): Disney, MasterCard, Salesforce, Uber eats.

Could it be that their compensation is tied to the stock price and sticking around for a few more months could cost them millions as the coronavirus collapses the world economy?

<comment><USER31> COVID-19 is not going to collapse anything, although it may disrupt the economy.

The CFR is currently estimated at 2-3% of KNOWN cases with an enormous number of unreported cases due to often mild symptoms. While it has and will spread very wide, the vast majority of people will not even experience significant complications.

We ARE experiencing a pandemic with significant impact on global economy but it's not a collapse scenario.

</comment></comment><comment><USER32> For anyone who wishes to see the Bobs discuss this change, they had ~10 minute interview on CNBC at 5:30 EST earlier.

[https://www.cnbc.com/video/2020/02/25/disneys-bob-iger-i- wan...](https://www.cnbc.com/video/2020/02/25/disneys-bob-iger-i-want-to- spend-more-time-on-creative.html)

</comment><comment><USER33> Is this unexpected? All the headlines make it sound like there's some sort of scandal.

<comment><USER34> It's a little goofy.

<comment><USER35> It's an extremely goofy move.

</comment></comment></comment><comment><USER36> He was ready to retire anyway and I think he's getting out of dodge while the going is good. He successfully launched their streaming service, and if I were him, I'd also want to get as far away as possible from the hospitality industry given how the spread of covid-19 is going.

</comment><comment><USER37> Something doesn't add up here. Why would Disney promote the head of parks in the middle of the coronavirus outbreak? Really odd timing.

<comment><USER38> Parks will temporarily close. He'll have time/space to take on other extra responsibility. Just boring reallocation of human resources?

</comment></comment><comment><USER39> Very recently they released a new MasterClass with Bob Iger. It's very unlikely he would engage in that if his plan was to step down soon. Also, if any publicly-traded company CEO steps down so suddenly, there's always some hidden reason behind that...

</comment><comment><USER40> Board of Directors appoints budget conscious CEO, top line drops 2.53%.

Who's running the board?

<comment><USER41> NASDAQ dropped 2.77%, seems like Disney beat the curve!

</comment></comment><comment><USER42> I wonder if he's scared about coronavirus?

</comment><comment><USER43> He ruined Star Wars

</comment><comment><USER44> After some issues that he has faced this month, he finally gives up his title/position that must be tough for him.

</comment><comment><USER45> Don't forget another "accomplishment" \- the Mickey Mouse curve.

</comment><comment><USER46> Hey new emperor of mouse, how about put starwars charactet into marvel universe, we can make all these even worse.</comment>


From Night Owl to Productive Early Bird - <AUTHOR> http://skim.me/home/2012/08/09/from-night-owl-to-productive-early-bird/ Inspired by Ryan Carson from Treehouse! <comment><USER2> Whoever wrote this is brilliant. I'm very excited to see what these guys are doing. Skim.me looks like a complete game changer. Totally amazing. This is probably the most excited I've been for a startup launch, ever.

I look at a lot of sites throughout the day, and waste a lot of time checking them more often than I need to, typing in urls, navigating to content, scrolling past ads, and even just trying to remember what sites I want to look at.

With skim.me, I could just go to one spot each morning and evening, and scroll through everything I was going to look at that day in 15 just minutes, so I can save time and focus on other stuff for the rest of the day. It would change my life. I can't wait.

And I am definitely not saying because I'm on the skim.me team or anything like that. Nope. No affiliation whatsoever. I definitely don't know Clinton or Neil or go to school with them or co-found skim.me and work out of Varick St with these awesome guys every day.

</comment><comment><USER3> it's the damn weekend that screws up these good routines though (curse you leisure!). one thing that could help is to give up drinking, then you wouldn't have to sleep in as much.

<comment><AUTHOR> you're right. i've already given up hard alcohol. maybe beer's next. red wine probably never

</comment></comment><comment><USER4> Great Stuff

<comment><AUTHOR> Thanks teammate Neil</comment></comment>


Real Blogs Don't Buy Ads On Google - <AUTHOR> http://techcrunch.com/2011/01/11/real-blogs-dont-buy-ads-on-google/ <comment><USER2> _"One thing big blogs don't do is buy traffic."_

If it's true that there was an unwritten code amongst bloggers not to advertise, then that is a silly and outdated rule.

</comment><comment><USER3> Newspapers advertise. Magazines advertise. Radio stations advertise. TV channels advertise. Why shouldn't blogs advertise?</comment>


PRU tips: Understanding the BeagleBone's built-in microcontrollers - <AUTHOR> http://www.righto.com/2016/08/pru-tips-understanding-beaglebones.html <comment><USER2> The PRU is a fantastic bit of hardware. With two of these running at 200MHz, and direct register-mapped access to GPIO pins, there's a lot of cool stuff you can do with them.

For example, in a recent project, I used one of the PRUs to generate a precise 40MHz square wave clock signal with 40% duty cycle, and the other to read the signal pin of a camera module into shared RAM. It worked extremely well, allowing me to obtain camera data at hundreds of FPS, and freed up the main CPU to do some fairly heavy image processing - all without involving an expensive camera capture rig or an external PC.

<comment><USER3> Neat! Is the source available?

<comment><USER2> It's for an academic project which is ongoing (the camera control is just one part of the system). When it's done, I'd be happy to share source (and indeed we may simply open-source everything related to the project to accompany the publication).

<comment><USER4> Hmm, I wish HN had reddit's remindme bot! Sounds great.

</comment></comment></comment></comment><comment><USER5> BeagleBone's PRUs are an enabler of Bela, "the embedded platform for ultra-low latency audio and sensor processing" which was ~1000% funded on KickStarter earlier this year.

Bela is being used at the Augmented (Musical) Instruments Lab in London (and now the community) to make rich, responsive digital musical instruments. It has a Web IDE, supports C++, Pure Data, Faust, SuperCollider, etc., but again thanks to the BB's PRU's, it supports audio-rate sensor sampling!

Site: [http://bela.io/](http://bela.io/)

Code: [https://github.com/BelaPlatform/Bela](https://github.com/BelaPlatform/Bela)

Videos: [https://www.youtube.com/channel/UCgWd1Q2dcWdqCGNl5BijFsA](https://www.youtube.com/channel/UCgWd1Q2dcWdqCGNl5BijFsA)

Paper: "An environment for submillisecond-latency audio and sensor processing on BeagleBone Black" [http://www.eecs.qmul.ac.uk/~andrewm/mcpherson_aes2015.pdf](http://www.eecs.qmul.ac.uk/~andrewm/mcpherson_aes2015.pdf)

Lab: [http://www.eecs.qmul.ac.uk/~andrewm/](http://www.eecs.qmul.ac.uk/~andrewm/)

<comment><USER6> What a beautiful project. I started to hack around for a guitar amp emulator on the bb in 2014, but it quickly became too involved. Thanks for sharing.

</comment></comment><comment><USER7> The PRU is pretty cool. Just had to simulate a dozen eCAP devices and its just a few lines of code. An eCAP records the period (in ticks) of a signal on an input pin. It can be triggered on a rising or falling signal, and can record absolute ticks or the number of ticks since the last trigger (differential mode). Here's the main loop:

       while (1)
    	{
    		uint32_t sample = __R31;
    		uint32_t change = sample ^ pEcap->sample;

    		if (change)
    		{
    			// Calculate which bits have seen a desired edge (it changed, and we trigger on that change)
    			uint32_t edge = change & trigger;

    			if (edge)
    			{
    				uint32_t ts = ReadTimestamp();

    				int bit = 1;
    				int iBit = 0;
    				while (iBit < 30)
    				{
    					if (edge & bit)
    					{
    						// Store the ts (or the difference in ts) into the capture table for this bit
    						//  and increment the slot index (aICap)
    						pEcap->ECAP[iBit][pEcap->aICap[iBit]++] = (pEcap->differential & bit)? (ts - tsLast[iBit]) :ts;
    						pEcap->aICap[iBit] %= 4;

    						// Next time we calculate the difference from this time
    						tsLast[iBit] = ts;
    					}
    					iBit++;
    					bit <<= 1;
    				}

    				pEcap->edgeDetected |= edge;
    			}

    			pEcap->sample = sample;
    			trigger = sample ^ pEcap->edgeUpDown;
    		}
    	}

It detects an edge within a few dozen nanoseconds (low jitter). A while loop like this would kill a main processor thread; and it would have terrible latency when other threads were scheduled during a trigger event.

And it detects edges on 30 pins in parallel! I could work on the "which pin had a trigger" code to reduce the period calculation from X30 to log(30) but I have no need for that fine latency in my current application.

</comment><comment><USER8> I actually had no idea that the Beaglebone had these guys hiding onboard. A nice alternative to something like a RasPi with an Arduino hat or a USB- connected Arduino.

This is great for projects where you control the whole stack and don't plan to support anything else. I guess the only downside to going down this path is that you're locking in the Beaglebone Black as your sole hardware platform and losing some modularity.

For example it would be great to control a 3D printer by running something like OpenGB ([http://opengb.readthedocs.io/](http://opengb.readthedocs.io/)) on the main CPU and something like Marlin ([https://github.com/MarlinFirmware](https://github.com/MarlinFirmware)) on a PRU. But that would require a Beaglebone-centric approach which wouldn't work on other hardware combinations.

That's just an observation though - I'm really impressed that the PRUs are there!

<comment><USER9> So, structure your project that you can either use the internal PRU or an external motor controller… I don't see the big problem.

I'd say that most embedded related projects will have to deal with newer revisions of their hardware, maybe because an older part is no longer available or because people came up with more intelligent or less buggy circuits over time. So that's already a few (prob. very minor) variations you'll have to support.

And besides very trivial projects, you should always try to have at least one "dummy" implementation of everything, to facilitate automated testing of your code.

So, in your case, yout 3D printer controller could support the internal PRU of the Sitara, an external servo controller, or some dummy library that just logs positions to a file, for testing.

<comment><USER8> On reflection - you're absolutely right. In fact this is exactly the way OpenGB is designed!

OpenGB uses an abstract base class (called IPrinter) to describe a printer interface. At the moment there exists a Marlin implementation and a Dummy implementation (as you describe) of IPrinter.

Other comments mention BeagleG and MachineKit. It _should_ be pretty trivial to add IPrinter implementations of both of these.

Thanks for the inspiration! :)

</comment></comment></comment><comment><USER10> Modern SoCs are a circus of various CPU cores, and there's a huge amount of work going on at the companies that design them to make them work together. Even for those SoCs that run open-source software, many of those secondary cores aren't visible to people outside the company creating the SoC. They're hidden behind binary blob "firmware". One of the systems I worked on over 5 years ago had ARM Cortex-A7s running the main OS (could be Linux), talking to a proprietary DSP running some in-house RTOS, talking to an 6502... The systems I'm currently working on have way more and more modern cores than that. As a user, or even as a developper, you wouldn't know.

Kudos to Ken for lifting the curtain a bit on the Sitara's PRUs!

It's a shame there isn't more developer availability of these cores. I'm kind of shocked that the way to program these is still just through assembly, but really not that surprised.

<comment><USER11> > I'm kind of shocked that the way to program these is still just through > assembly, but really not that surprised.

This is not true anymore (was in the beginning though)- there is a C/C++ compiler available.

</comment></comment><comment><USER12> Wow, a lot more PRU interest than I expected. Thanks everyone for the comments! Looking online, I got the impression that hardly anyone was using the PRU. Are there any forums I should visit where PRU programmers hang out?

</comment><comment><USER13> I think the main thing to think about PRUs is that they are the way to do a class of things in software that you'd normally break out some custom logic or an fpga to implement. Sure they're hard to initially set up for but way less effort than breaking out the PCB design software

There is also a C compiler available, I haven't used it, but Tridge (of Samba fame) gave a demo at linuxconf a couple of years back in which he launched a plane (remotely) and ran the guidance software on a BB, with the PRUs doing all the servo/etc work coded in C ..... and compiled the linux kernel onboard at the same time ....

<comment><USER11> Two things from my experience:

1) Read the fine print when it comes to processor manufacturers telling how much time something takes. Although PRU subsystem is deterministic and __most__ instructions take 5ns, there are quite a few cases which take an order of magnitude more time[1]. Sure, the access times might be deterministic but that doesn't make it easy to know how much something will take.

2) Remote-controlled airborne vehicles need a lot less computation power than I expected. PX4 runs it's "main loop" at just 400Hz and I've seen PX4 or ArduPilot devs (might even be tridge) saying that 50Hz would be enough. Sure, you need accurate timing for PWM outputs (~1us resolution) and most important: low jitter.

The first point has bitten me personally- I found it non-trivial to get reliable < 100ns interrupt jitter on Cortex-M4. It really got down to what was happening on the bus between CPU/memory/peripherals at the point interrupt was supposed to fire (e.g. getting the documented latency of 10-odd cycles when CPU is idle but a lot more when there is a DMA transfer in progress).

1: [http://processors.wiki.ti.com/index.php/AM335x_PRU_Read_Late...](http://processors.wiki.ti.com/index.php/AM335x_PRU_Read_Latencies)

<comment><USER14> Tridge gives a good summary of the different timing requirements in a presentation he gave back in 2014 (page 8).

[http://uav.tridgell.net/LCA2014/AP_Linux.pdf](http://uav.tridgell.net/LCA2014/AP_Linux.pdf)

</comment></comment></comment><comment><USER15> This article makes repeated references like the following:

> If you want to perform real-time operations, the BeagleBone's ARM processor > won't work well since Linux isn't a real-time operating system.

I get that Linux is not a real time system, but the article seems to be making the implication that the ARM processor cannot support real time operations. I was under the impression that the real-ness of time was determined solely by the operating system and not by the hardware. Is this not the case, or is the article just making the assumption that no one is going to port a real time operating system to the BeagleBone?

<comment><USER10> Amusingly enough, performance enhancing features like cache or a flexible interrupt controller get in the way of a system's "real-timeness", because the timing of operations is no longer clearly deterministic. That's why ARM has cores like the Cortex-R series (R as in Realtime) that do promise real-time characteristics. The Sitara in the Beagleboard has a Cortex-A (A as in Application), which is not designed for realtime.

So yeah, it's more than just the OS.

<comment><USER16> Just simply having interrupts will mess with a hard real time processes. Total sketchtastic, some modern processors have background stuff that takes control every so often.

</comment></comment></comment><comment><USER17> The PRU is the unsung gem of the BeagleBone. I used it to prototype an optical control system that needed sub-100ns timing. As the article hints, it's not an easy beast to get started with, but it offers crazy realtime abilities.

<comment><USER18> That's pretty interesting, could you detail a little bit what was executed by the PRU and what was executed by the CPU ?

<comment><USER17> The CPU loaded a bytecode program into the PRU's memory and told it to run.

The PRU ran a bytecode execution engine that was handcoded in assembly. The core of the engine was a loop:

\- perform the routine for the current bytecode

\- when the routine finished, spin on the PRU's embedded timer (the IEP) until 256 ticks (1.28us) had passed since the start of the cycle

\- advanced to the next bytecode and repeated

External devices were connected either via I2C (those that didn't need super- tight timing) or to the pins controlled by the PRU. Some of the bytecodes controlled execution flow like loops while others were specific to the application, things like turn light on, move lens, trigger camera, etc.

There were a few more bells and whistles, like memory locations where flags were set by the engine to let the CPU know how progress was going.

It was a fun project. Unfortunately, the manufacturer of one of the core devices decided to stop making it, so that start-up had to go in a different direction.

</comment></comment></comment><comment><USER19> I am driving a RGB led matrix from a PRU unit: [http://heeen.de/proj/rgbmatrix/fablab.mp4](http://heeen.de/proj/rgbmatrix/fablab.mp4)

</comment><comment><USER20> Machinekit, originally derived from Linux CNC, uses the BeagleBone's PRU's to handle the real time I/O.

Code: [https://github.com/machinekit/](https://github.com/machinekit/) PRU links: [http://blog.machinekit.io/2013/06/beagle-bone-pru- links.html](http://blog.machinekit.io/2013/06/beagle-bone-pru-links.html)

</comment><comment><USER21> Can someone say how the PRU would compare to a "similar" microcontroller? And what the most similar core would be, e.g. M0 or M3.</comment>


Steve Jobs: Unseen Images by Norman Seeff 1984 - <AUTHOR> http://www.retronaut.co/2012/08/steve-jobs-unseen-images-by-norman-seeff-1984/

<comment><USER2> I've never seen him so "human"...</comment>


Visualising Chicago's Ageing Infrastructure - <AUTHOR> http://www.cmap.illinois.gov/mobility/explore

<comment><USER2> This is really cool: [http://www.cmap.illinois.gov/mobility/explore#/topic/roads/c...](http://www.cmap.illinois.gov/mobility/explore#/topic/roads/congestion)</comment>


Ask HN: what ways have you sourced a prototype for your physical product? - <AUTHOR> Howdy!
I am a programmer looking to create my first prototype for a simple product idea that I have. It is about the same complexity as a Pringles can, has no moving parts, and will be made of plastic.
I do not want to order a 3d printer for this, and I am looking for ways that any of y'all have sourced a physical prototype. I will only need a few runs of the product in less than ~5 quantity.
Any suggestions are greatly appreciated and I don't mind being pointed to a blog or being pointed somewhere where I have to do some heavy lifting (in terms of my own research)! <comment><USER2> Depending on the design, you could either have it 3d printed or have it machined. Online services are available for either option (e.g. shapeways.com and emachineshop.com, respectively), or you could find a local job shop to do either. The online services require you to provide them with prepared CAD files. A local job shop would be able to work with you on the CAD files if you don't have them already. Another option would be to try printing it at a local hackerspace (e.g. dallasmakerspace.org). Feel free to contact me if you want some more direction. I haven't prototyped consumer products, but I do spend a lot of time building one-off devices for physics research.

You're right not to want to order your own 3d printer for this. The hobbyist grade ones still aren't plug and play, so you'd burn too much time getting it working.

<comment><AUTHOR> I really appreciate the response!! Also, idk if you knew where I am from when you wrote this, but I absolutely love the shoutout to the local Dallas shop. I will for sure start there and then exhaust all of the other resources that you mentioned.

Thank you very much for your time!

</comment></comment><comment><USER3> Have you already modeled it? If you're comfortable leaving an email address so I could get in touch, I can send you a few prints of your design.

<comment><AUTHOR> I have created a rough proof of concept at this point and I am using google sketch up right now to try and model it in CAD format. I hope that answers your question!

I would love for any advice you could give so please do shoot me a line at bentechwild at gmail dot com! Thank you very much!!</comment></comment>


Show HN: Clout Report – A crowdsourced database of upcoming rappers - <AUTHOR> https://www.clout.report <comment><AUTHOR> Hey HN,

I compiled a list of over one hundred upcoming rappers and used APIs and scrapers to get over 3,000 total data points.

Sort, filter, and search by:

\- Gender, age, location, record label, and region

\- SoundCloud follower and track count

\- Instagram, twitter, and facebook follower data

\- YouTube subscriber data, plays, and video count

\- Spotify popularity, followers, track count

I think this is particularly useful/monetizable for:

\- record labels looking for new talent

\- concert/entertainment venues that want to gauge the size of the audience of an artists

\- artists looking to collaborate with other artists

Appreciate any feedback and ideas for new features that could be added.</comment>


Ask HN: Do non-tech folks read Hacker News? - <AUTHOR> Assuming most of the readers are in developer jobs, I'm interested in knowing if people in completely different professions e.g. law, medicine, plumbing, cooking etc read this site. <comment><USER2> I work in accounting (manufacturing, non-tech) and read HN daily. The discussions here are often balanced and insightful. HN feels quite mature. I learn a lot here - how to be a better mentor, how to be mentored, how to cope with depression, etc. I'm interested in tech (as a consumer), but not as a engineer.

</comment><comment><USER3> I work in purchasing (manufacturing) and I'm react hobbyist.

Due to the fact that my career turned in the automotive direction (too long story why that happens), the change of industry will probably never come. Saving money to transfer to IT job is not easy with a 2 + 2 family and a new home.

In general in the office is like: "Did you saw yesterday the game in TV? It was amazing" "Naah, I'm working now on a Reddit clone" and they just don't understand me.

IT or to be more precise programming is my passion or maybe a dream. That's why I return everyday on HN and read Dan Abramov tweets

</comment><comment><USER4> Sure, you can see professionals from all kinds of backgrounds chipping in, just read some threads?

Naturally, techies have more exposure to the platform, but the diversity of topics and desired quality of content is attractive to a much broader audience than just "devs".

Also: not everyone working in IT is a dev :-)

</comment><comment><USER5> I'm into sales copywriting (direct response marketing) and HN is my homepage. I love to read it daily because of intellectual discussions and also to check if my Internet is working.

</comment><comment><USER6> I am a pro gambler and I read HN almost daily. I'd like to change my career soon and I am hoping HN will help me decide what my next step should be (I have basic experience with MySQL and PHP).

</comment><comment><USER7> I'm in Sales/Marketing and I read HN every day. With that said, I work in the tech industry. I hardly ever chip in, though, because of the more technical focus within the comment threads.

</comment><comment><USER8> I used to be in the trades. I read hacker news when I started getting into programming and now I work full time as a software engineer. So.. yes non-tech folks read hacker news!

</comment><comment><USER9> I 'd say most readers seem to be in management or sales. Would be nice to have a poll

<comment><USER10> I used to see my former (non-technical) dev manager browsing HN occasionally.

</comment></comment><comment><USER11> I work in government.

I think the hacker ethos can be found alive and well in a variey of career paths.

</comment><comment><USER12> I am not a developer but I am interested in technology.

</comment><comment><USER13> Non developer and read everyday

</comment><comment><USER14> Your first mistake was assuming not everyone works for a tech company.</comment>


How far is TRAPPIST-1 – visualizing distance - <AUTHOR> https://sheep.horse/2017/3/how_far_is_trappist-1_-_visualizing_distance.html <comment><USER2> That's very cool, but upon first opening up, and having not read the introduction, I took the blue line markers to mean the extent of the Solar System (duh) so my first thought was "huh, that's actually fairly close!".

Might've been a good idea to write "Light year" in the same blue as the markers and have some smaller text, or whatever, to indicate the zoomed Voyager and Solar-system extent.

If the farther Voyager hit the solar bow shock, is that now a fair indicator of the extent of the solar system? Or is it fairer to says it stretches out as far as the Oort cloud (which of course it does, but in turn mangles any appreciable scale)?

<comment><USER2> Also, I'm amused to live in an age where [http://sheep.horse](http://sheep.horse) is a valid URL :)</comment></comment>


Ask HN: How do I convince a company to participate in Open Source? - <AUTHOR>

I have been asked to give a proposal to my friend's company for why they should participate in Open Source.  They use a lot of open source, but I think their current policies do not allow employees to contribute code or time.  I am looking for concrete reasons such as proof it is more secure, help from outside developers finding bugs, Return on Investment, etc. <comment><USER2> the only strong case i've heard for that is that contributing code will boost your reputation and thus improve sales. but i think that only works well if you sponsor an entire project or if you're a custom software shop looking for clients.

<comment><AUTHOR> Thank you. I guess it could also improve their reputation to be able to hire developers

<comment><USER2> Put another way "street cred".

: D</comment></comment></comment>


An Arguably Unreal Particle Powers All of Your Electronics (2013) - <AUTHOR> http://nautil.us/blog/an-arguably-unreal-particle-powers-all-of-your-electronics

<comment><USER2> Interesting, though I can't see how these properties have any bearing on the particle's status as 'real' versus 'imaginary.' The rules describing this entity's behavior are pretty incongruous with the behavior of other known entities—but we shouldn't forget that, just as when laying out a framework for a software system, we have chosen which our basic entities are going to be. If we were to select a different framework for describing physical phenomena, something less object-centric (something derived from Whitehead's process philosophy, for instance), what are the chances any notion resembling pseudoparticles would exist?

When I consider that a framework based on a taxonomy of spatial object types may only have seemed natural to us because we started with macroscopic phenomena (which our minds 'objectize') and brought our existing tools and heuristics with us, I can't help but feel like all of this 'incomprehensible' quantum weirdness is just our modern version of trying to model elliptical orbits with circles.

<comment><USER3> It is best to think of particles as clumps of properties that are roughly localized in space-time. They can smear out, and then they look less like particles.

The properties themselves are ephemeral and confusing. Is mass real? Is charge fundamental? We don't know, and we don't know how to know. We can measure it, and that's as real as anything ever gets. In fact, that's pretty much what 'real' means, since we can't usefully define 'real' in terms of anything else.

I think you are hitting the nail on the head here. Of course these things are real. Everything is real. The relevant question is: why do you care if they are or not? What are you gonna do if some particle is shown to be imaginary? How you can manifest such a distinction into the construction of a better world?

I've always felt that quantum mechanics was mostly intuitive. But people want to quote Feynman at me and say "if you think you understand it, then you don't." Well, guess what? I was born in the 1980's. There was never a day of my life where I went around knowing anything about physics without being told quantum physics was better. I learned Newtonian physics under the assumption that it was incorrect. Quantum physics isn't weird. Objects are weird. Perfectly smooth curves are weird. Hard categorical distinctions are weird. They are all obviously approximations that may simply be a result of the fact that all our words happen to have spaces between them, when this isn't necessary.

<comment><USER2> Interesting. I like this, "Hard categorical distinctions are weird. They are all obviously approximations that may simply be a result of the fact that all our words happen to have spaces between them, when this isn't necessary." Reminds me of something I've been thinking a lot about recently, which is how often even extraordinarily smart people fail to make the distinction between systems of reference and their referents (even when aware that it's a potential pitfall). It's a pervasive mistake in physics-derived philosophies, too: thinking properties of our system of description are properties of the system being described.

There's this question in philosophy of mathematics, whether math is discovery or invention. It's good that the issue is recognized there, but it seems very rarely acknowledged in science. Sure, it starts and ends empirically—but we can't say anything without inventing a way of stating it: we always choose our terms, a framework to express our discoveries in. When we find 'weird' properties cropping up in physics, why is it immediately assumed to be a property of reality itself rather than in our form of expression?

As an example, Arthur Eddington describes quantum uncertainty as a consequence of the fact that the language of physics switched from referring directly to physical phenomena, to describing our knowledge of these things; so when we only have partial knowledge, we frame our descriptions in terms of probabilities.

Anyway, I guess we may be products of our era since I'm from the 80's as well.

<comment><USER3> >As an example, Arthur Eddington describes quantum uncertainty as a consequence of the fact that the language of physics switched from referring directly to physical phenomena, to describing our knowledge of these things; so when we only have partial knowledge, we frame our descriptions in terms of probabilities.

I have never heard this before, but it is _exactly_ what I have said about the subject. This usually causes me to have to explain how I'm not saying we _could_ know that stuff: the information may not even exist. But we can't even say whether or not that is true.

>There's this question in philosophy of mathematics, whether math is discovery or invention.

I have also had many discussions about this. A lot of it stems from the fact that mathematicians tend to say "we are free to choose our axioms," when all my experiences say otherwise. We cannot choose contradictory axioms because useless math isn't math. (I.e., there is more to good math than truth: it must have possible utility.) And furthermore, if I were to drop two pennies on the table and consistently get you to agree there are three cents there, then you would be damned sure some mathematician would want to have a model for what is going on!

So mathematics is fundamentally tied to observation. That's why they spent so much effort trying to define things like 'continuity' and 'smooth curve': we need a model for reality, not just for our 'freely chosen' axioms.

So mathematics is discovery in the sense that we must observe its correctness, but invention in the sense that we must invent its usefulness.

In the end, all of our efforts are about developing language that allows us to describe our reality in a reliable and teachable manner -- Physicists and mathematicians are primarily trying to develop better language.

</comment></comment></comment></comment><comment> semi-extrinsic Actually a quite good, if quite brief, explanation of quasiparticles in various forms. Alas, the author did not give his audience sufficient benefit of the doubt as to embark upon an equally brief explanation of the pn junction as an example, which would have made this so much more interesting. Full disclosure: I am a physicist.

<comment><USER4> Agreed, I wish there was an example to keep people interested and educated rather than "amazed and entertained"... I actually don't like articles like this that make the simplifications and models we use (in this case for semiconductors) seem opaque, "unreal", and more difficult than they really are. It's true that there are abstractions (like bubbles aren't real?) that are convenient for calculation that otherwise seem strange, but showing how those are useful would be more interesting than saying... Look how amazing my magic is, you just couldn't understand it, because it makes no sense to anyone but us wizards.</comment></comment>


Steve Wozniak comes out in support of Mike Daisey, says 'his method succeeded' - <AUTHOR> http://mobile.theverge.com/2012/3/19/2885422/steve-wozniak-comes-out-in-support-of-mike-daisey-says-his-method <comment><USER2> something to keep in mind is that my bet is that due to the high profile scrutiny that workers at that plant are most likely treated above average when put next to other plants in china. so the plants making products that nobody has heard of most likely will have the worst conditions -- but sadly the media will never care about that</comment>


Life Expectancy Tied to Education - <AUTHOR> http://www.businessweek.com/lifestyle/content/healthday/613422.html

<comment><USER2> Did they do regressions to take income out of the picture?</comment>


The Long Tail of the English Language - <AUTHOR> http://blog.wordsapi.com/2015/01/the-long-tail-of-english-language.html

<comment><USER2> Now I understand somewhat better why I often end up ceasing midway through discourse, as my colloquists often end up interjecting requesting definition of whatever neologism has just emanated from lexicon into conversation.... a random sampling of words that I have just today been asked to define either do not feature in this list at all, or are 90,000+ in terms of use. Brobdingnagian, deleterious, autodidact, loquaciousness... are these really such strange vernacular?

I suppose this is what happens when you spend your formative years buried in literature - it probably doesn't help that I mispronounce all sorts, as books make poor elocutionists.

I do worry that as we further and further consolidate our vocabulary that we lose the breadth and depth of thought that nuanced words provide... so did Orwell...

<comment><USER3> Ah, another person who talks like a book. I also get that quite regularly. I have a bit of a personal rule, though, where I never use a more complicated word when a simpler one would work. After all, conversation is at heart a means of communication.

Although we might get a bit of a thrill from talking over people's heads, it doesn't really mean much of anything. And all it takes is one misuse -- for example, calling Brobdingnagian a neologism -- and we tear ourselves down more than we could hope to build ourselves up.

<comment><USER2> Brobdingnagian is a neologism though. Swift quoined it scarcely two centuries ago, which is a ... Lilliputian (sorry)... amount of time in linguistic terms. I suppose it also depends on what you consider to be contemporaneous. I'm my case, this era, which opened with the enlightenment...

Agreed - never use a complicated word when a simple one will do - but you either have to turn verbose or lose nuance, as we're not talking synonyms, but words which have meanings distinct from some of their more commonplace counterparts.

No thrill, I just hate taking the colour out of language, and I like to share words far and wide, so they don't drift into oblivion, along with the thought and meaning behind them.

<comment><USER3> Oh well. I guess if you want "neologism" to mean "any word invented in the past 300 years", that's your prerogative, but I take a very different meaning. One interesting side-effect of that interpretation is that "neologism" becomes autological, since it was itself coined in that period.

It's true that there are sometimes nuances lost in the conversion from large to small, but there is far more nuance lost once your partner doesn't know what the hell you are saying. For you, Brobdingnagian might have some special nuance (I'm picking on that word because it is ridiculous to me), but to your partner it has no nuance or flavor -- they have no associations with that word, nor any grasp of any subtleties of meaning -- in fact they don't grasp the meaning at all. So if you are trying to get something across, it's simply a bad way to do it.

I mean, don't get me wrong -- I don't want to deride big-word-users. If you want to do to because you like it -- that's obviously fine. Also, if someone does happen to have a thing with Brobdingnagian, I guess you might have just made their day a lot better. But at some point, you have to acknowledge that obscure vocabulary is more of a hindrance than a boon to communication.

</comment></comment></comment></comment><comment><USER4> I wonder how other languages compare to English. I know English is far from pure.

"The problem with defending the purity of the English language is that English is about as pure as a cribhouse whore. We don't just borrow words; on occasion, English has pursued other languages down alleyways to beat them unconscious and rifle their pockets for new vocabulary." \-- James Nicoll

<comment><USER5> What do you mean by "pure"? Lack of loanwoards? Lack of any linguistic changes (e.g., sound, meaning) for existing words? Lack of any innovation, i.e. no new words for new concepts/objects? You could try to find some of these things in languages spoken by rather isolated groups. Yet I don't think one should call such a language "pure", implying some kind or (moral) superiority.

<comment><USER6> Compared to Classical Latin for instance, I can sort of intuitively see how a "purity" comparison might make some sense. The grammar is more rich and precise, suffixes and prefixes attached to primitive roots are more regular in terms of meaning. Meaning is very decoupled from word order, which allows much richer possibilities for rhyme and prose. If the restored pronunciation is anything to go by, then the pronunciation is very regular. Then again I'm not a linguist.

I would say it's like arguing whether Python is more "pure" than Perl. They're both Turing complete, they express very similar concepts but I'd wager most people would concede that Python feels "purer".

<comment><USER7> Isn't the 'purity' of Latin due more to the fact that we just don't have much data on the languages that informed it? I'm sure Latin borrowed heaps of words and grammar from other languages, but those languages are mostly lost to history now, since the people who spoke them didn't conquer and hold the Mediterranean for one thousand years.

</comment></comment></comment></comment><comment><USER8> The source material for this frequency count comes from Open Subtitles [[http://www.opensubtitles.org/en/search](http://www.opensubtitles.org/en/search)]. Hence the frequencies here apply to spoken English, not written English. In written English the three most common words are "the", "of" and "and", whereas here they are "you", "I", and "the".

</comment><comment><USER9> If you have the right kind of friends you can play "Who knows the most obscure word?". Everyone picks a word from memory, check each word's position (We used to use google ngrams, but this would work well.), whoever gets the least common word wins that round, repeat until it's not fun anymore. That's how I learned defenestrate, obsequious, and a few other words.

<comment><USER10> I love "sesquipedalian", because it's so self-referential.

<comment><USER11> I love "non-self-referential" because if I use it enough times with all the correct stresses and pauses, I can build a computer.

</comment></comment></comment><comment><USER12> For a corpus much more complete on the long tail, try google books ngram viewer : [https://books.google.com/ngrams](https://books.google.com/ngrams). That uses full data from google's book-scanning endeavors, millions of books compared to millions of words in the originally linked article.

<comment><USER2> I love to play with this, and see how thought waxes and wanes - a good one is to stick in every soviet premier since Lenin, or 20th c. US presidents - really tells you rather a bit about the mindshare that these individuals had.

</comment></comment><comment><USER13> [http://en.wikipedia.org/wiki/Zipf%27s_law](http://en.wikipedia.org/wiki/Zipf%27s_law)

EDIT ADD: Zip's Law was not mentioned directly in the blog post but the reply clarified that the API returns a zipf score. However, a word's zipf ranking is _dependent on the corpus used_. The Wordapi "About" page[1] says most data came from Princeton WordNet but a sibling comment says it came from a subtitles compilation. If the project could clarify the data sources, it would be helpful.

[1][https://www.wordsapi.com/about](https://www.wordsapi.com/about)

<comment><AUTHOR> The "frequency" score returned by Words API is the a Zipf score for the word. Ranges from ~1.6 to ~7.6.

Regarding your update - I'll update the About page.

</comment></comment><comment><USER14> 'I' doesn't seem to work. It is very common. [https://books.google.com/ngrams/graph?content=I%2C+you&year_...](https://books.google.com/ngrams/graph?content=I%2C+you&year_start=1800&year_end=2000&corpus=15&smoothing=3&share=&direct_url=t1%3B%2CI%3B%2Cc0%3B.t1%3B%2Cyou%3B%2Cc0)

<comment><USER15> Try lower-casing it.

EDIT: This looks like a case of a common programming antipattern: you don't care about the casing for comparison purposes, so instead of implementing a case-insensitive compare, you downcase the strings and call it a day. But that's inherently a loss of data, and not having that data will eventually come back to bite you.

</comment></comment><comment><USER16> So I'm wondering, if you just learned the 200 most popular words, you might get pretty far in learning a new language, no?

<comment><USER17> 200? no. Try 2,000. That should be representable of a barely usable level of the language. If you are what might be described as fluent, you're probably at >20,000. Take this headline:

JACUZZIS FOUND EFFECTIVE IN TREATING PHLEBITIS

"EFFECTIVE" and "IN" are the only two words found in the first 2,000 words sorted by frequency, although "FOUND" is close. So with only 200 words you'd understand:

WNPHMMVF SBHAQ RSSRPGVIR IN GERNGVAT CUYROVGVF

2,000 words would give you:

WNPHMMVF FOUND EFFECTIVE IN GERNGVAT CUYROVGVF

And some grammar knowledge would tell you

WNPHMMVF(N) FOUND EFFECTIVE IN GERNGVAT(V) CUYROVGVF(N)

Which is enough to know that you only really need to look up "TREATING" in the dictionary to understand the gist of the sentence But it'd hardly pass as a fluent understanding...

<comment><USER18> "Some grammar knowledge" probably also tells you the structure of the verb, namely TREAT+ING(V,present progressive). "TREAT" is the 1,118th most common word, so that gives you:

WNPHMMVF FOUND EFFECTIVE IN TREATING CUYROVGVF

And then some contextual knowledge of what sort of things get treated tells you that CUYROVGVF is a disease or injury, and WNPHMMVF is a medicine or therapy. So knowing 2,000 words and some grammar isn't quite fluent, but it's around the level of a middle-school child who has to ask what lots of nouns are.

</comment></comment></comment><comment><USER19> Very nice product! As a sidenote: anyway knows a service that provides human associations? Eg What do you associate with "Hacker" as a person "Computer", "Night", "Internet"

<comment><USER20> I work on ConceptNet, which does this.

[http://conceptnet5.media.mit.edu](http://conceptnet5.media.mit.edu)

</comment></comment><comment><USER21> The most striking thing about Randall Munroe's "Up Goer Five" comic [1] was that the word "computer" is on the list, but "thousand" isn't.

        "(Explained using only the ten hundred words people use the most often)"

[1] [http://www.xkcd.com/1133/](http://www.xkcd.com/1133/)

<comment><USER22> Well to be fair, thousand is a number, so analysis of written text will find it a lot rarer than it is actually used/spoken.

</comment></comment><comment><USER23> I poked around in the neighborhood of "Tremulous" and saw a lot of obvious data errors: "ladiesand","manklnd", "confldentlal", "howdare", "monment". Other words don't seem particularly rare: "productively", "areolas", "combusts", "lazier".

</comment><comment><USER24> It seems like their long-tail data is full of misspellings. Try typing in "vacillate" and looking at the other words on the graph in relative frequency, for example.

<comment><USER25> No corpus is ever "clean". Depending on the type of corpus there might be many misspellings, so obviously they will occur in the graph alongside other low- frequency words.

</comment></comment><comment><USER26> TIL "fucking" is the 214th most popular word in the English language.

</comment><comment><USER27> I'm guessing "icing" is more popular in Canada.

<comment><USER28> Only if you're thinking of icing on roads (or in hockey), but not if you're thinking of icing on top of a cake.</comment></comment>