An Interactive Guide To The Fourier Transform - <AUTHOR> http://betterexplained.com/articles/an-interactive-guide-to-the-fourier-transform/

<comment><USER2> I have desperately wanted to understand the Fourier Transform and signal processing for a long time. In high school I read every tutorial or book I could get my hands on, hoping that one would finally "click." The very first thing I did when I got to college was approach the CS faculty and ask for recommendations for literature that would help me understand it. My college didn't have a signals and systems class, but a few years later I heard that Richard Lyons's book "Understanding Digital Signal Processing" was supposed to be the friendliest book on the subject so I bought and read it.

I say all of this just to impress upon you that I was _serious_ about wanting to understand this stuff.

Despite all the effort, the FFT and DFT were never more than opaque blobs of math to me, mysterious boxes that you put inputs into and could extract outputs from (albeit in a very obscure format that seemed to involve complex numbers for no apparent reason). I sort of gave up and forgot about it for several years.

Recently I came upon Stuart Riffle's article ([http://www.altdevblogaday.com/2011/05/17/understanding- the-f...](http://www.altdevblogaday.com/2011/05/17/understanding-the-fourier- transform/)) which explained it in terms of the circular approach that this article adopts. Rotate the waveform about the origin at the frequency of interest and average the samples, simple as that.

I am not exaggerating when I say that after 10 minutes of reading that article, I had an understanding that years of highly motivated knowledge- seeking had not given me.

I take a few things from this story. One is that, given a specific learner (ie. me), some ways of explaining things are infinitely better than others. I mean that literally, because this new, circle-based explanation gave me an understanding that I literally was not capable of achieving with all of the symbolic and algebraic explanations I had studied previously. Judging from other comments I've read, I'm not alone in this, which says to me that there could be exciting advances ahead of us in the way we learn. Just as the Khan Academy is making learning more accessible, I hope that it could also help discover and widely disseminate the _best_ explanations for things. Sal Khan is very good at explaining things but we can't expect him to think of everything. This fantastic DFT explanation was created by a random systems/graphics programmer; I hope it will percolate into DFT curricula until everyone who is learning about the subject is at least exposed to it.

My other takeaway, though, is that hard-core math people really think in a fundamentally different way than I do. I'm a highly intuitive thinker, and formulas are a sea of meaningless symbols to me without an intuitive understanding of what is going on. That someone could understand the DFT _without_ thinking in terms of the circular interpretation is amazing to me. I now know that I am indeed capable of understanding the concept, but only by thinking about it in a different way than most math people do. I suspect, however, that my intuitive way of thinking about it would be more difficult to formalize and make rigorous, so in the end I am dependent on the mathematicians and their way of thinking, even if I can't as easily understand things in their terms.

(One other example of this: I think calculus is simpler to understand in terms of infinitesimals rather than limits, but this is another example where the infinitesimals are more difficult to make rigorous).

<comment><USER3> (Author here)

I totally agree. There are certain explanations out there which have orders- of-magnitude differences in ease of understanding (seeing i as a rotation, seeing radians as the "mover's perspective", seeing integrals as "better multiplication"). My personal mission is finding these aha! moments which unravel years of confusing symbol manipulation. Calculus is definitely more easily understood with infinitesimals vs. limits (ask any physics major or engineer).

Personally, I'm looking forward to a world where the very best explanations / analogies can bubble to the top. It's ridiculous that 200+ years after Calculus was invented, we still teach it poorly, and nearly everyone struggles.

Rigor & Intuition have a delicate balance in math. I see it as language: children can speak fluently, even if they don't know the "rigorous" rules of grammar & spelling [which are left to linguists]. I suspect the reason most adults have trouble learning languages is because they try to start from rigor (vocab lists and grammar structure) vs. absorbing an intuitive notion of what's going on (and later refining with rigor, "me want food" => "I want food").

<comment><USER4> >Personally, I'm looking forward to a world where the very best explanations / analogies can bubble to the top.

Please do keep in mind that there isn't one best explanation per subject, just a "local maximum" explanation that is best to a class of people sharing a similar way of thinking. Personally, the normal explanation of "representing (approximating) a complex cyclic function as a linear combination of complex trigonometric functions" made the most sense, while your explanation reads like a convoluted mess of analogies. Therefore, try to have a few completely different explanations per topic, rather than just the one which makes most intuitive sense to you.

<comment><USER3> Definitely, appreciate the feedback here. It's easiest to share the explanations that come to me, but I love finding a few different ways to look at things, and as they emerge I like to include them.

As a simple example, here's the formula for adding the numbers 1...n:

[http://betterexplained.com/articles/techniques-for-adding- th...](http://betterexplained.com/articles/techniques-for-adding-the- numbers-1-to-100/)

The explanation I was originally given (pair the first and last items, and count the pairs) seems gnarly because you have even/odd issues, off-by-one errors, etc. There are others which click better.

</comment></comment></comment></comment><comment><USER5> Great article. I did Fourier Transforms in EE maths and could apply them but never properly understand them. Typical calculation versus understanding.

The greatest graphic in this entire article is if you ask me:

[http://betterexplained.com/wp- content/uploads/images/Derived...](http://betterexplained.com/wp- content/uploads/images/DerivedDFT.png)

If all mathematics was explained like this, many less eyeballs would have been gouged out of sockets.

<comment><USER3> Thanks. I love the conversion of equations into "math English" (that particular diagram is from Stuart Riffle).

</comment></comment><comment><USER6> Very good explanation, the smoothie-recipe decomposition approach is the one I used when I taught this to students (I used drinks, but well). However, this example really works only for teaching Fourier Series, which should be the first step to the FT anyway. If you want to understand this stuff, the order you should approach, I think is the Fourier Series, the FT (ignoring many mathematical difficulties), the discrete-time FT, which has its own quirks.

</comment><comment> mark-r This line towards the end has a wonderful meta cleverness about it:

"The analogy is flawed, and that's ok: it's a raft to use, and leave behind once we cross the river."

</comment><comment><USER7> Ended up playing with the animation for a while, this is probably nothing new but just thought I'd share a funny quirk I discovered. If you plug in the fibonacci sequence for the time, you get symmetrical strengths and phases (except for the first term):

For (0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610) you get:

99.74 77.77:51.4 53.46:82.8 39.97:104.3 32.41:121.8 27.98:137.3 25.39:152 24.03:166.2 23.62:180 24.03:-166.2 25.39:-152 27.98:-137.3 32.41:-121.8 39.97:-104.3 53.46:-82.8 77.77:-51.4

Formatting magic:

99.74

1 - 77.78 51.4

2 - 53.46 82.8

3 - 39.97 104.3

4 - 32.42 121.8

5 - 27.98 137.3

6 - 25.39 152

7 - 24.03 166.2

8 - 23.63 180

7 - 24.03 -166.2

6 - 25.39 -152

5 - 27.98 -137.3

4 - 32.42 -121.8

3 - 39.97 -104.3

2 - 53.46 -82.8

1 - 77.78 -51.4

It works for all sequence lengths I've tried, although the app starts rounding off when you start getting in the hundreds.. not that it matters in this case. Couldn't find any relationship between the first constant and the pairs, nor any relationship between the ratio of the pairs. Just something interesting.

<comment><USER8> Without wanting to quell any enthusiasm (or cause too much confusion), it's probably worth pointing out that _any_ real-valued time-series input will result in a similarly symmetric pattern. I can't explain why that's true in a way that makes sense with the analogy used here, but it is.

</comment></comment><comment><USER9> I studied frequency domain/time domain when I was in eight grade and high school learning the technical aspects of how radios work. An oscilloscope was a big boost for developing this intuition, particularly if hooked to a microphone where you could see how various sounds showed up, then how tones from the speaker of a radio would look. And using radios day-in, day-out, and understanding how the spectrum of an AM radio signal looked.

I am glad that this analogy works for many of you here, but I have always been deeply suspicious of using analogies to teach concepts. For one thing, there is always the part that "Ok, what I just told you and you learned is not true in the following ways.." And the famous analogy between water in a pipe and electricity being actually potentially dangerous to a new student. In fact, I am sure that Fred remembers me railing "all analogies are False". And I have had very little success teaching complex topics using analogies.

Incidentally, the AM radio spectrum is quite easy to understand. It wasn't until an advanced signals course that I saw the math for the spectrum of an FM signal. Much more complicated.

</comment><comment><USER10> This is indeed an awesome explanation. And I'm regularly shocked when people do not explain Fourier Transform as generalization of Fourier Series.

This makes it quite intuitive to me (well, at least as far as FT can be intuitive ...)

</comment><comment><USER11> This amuses me. My friend and I spent part of our time at a bar trying to explain Fourier transforms to her philosophy major boyfriend. My explanation of how optical trapping using Snell's law and light momentum went much more smoothly.

</comment><comment><USER12> > music recognition services compare recipes, not individual drops

What does this mean? I understand 'recipes' here, but what's 'drop' in this context? A continuation of the food metaphor?

<comment><USER3> I should probably change that. In this case, a "drop" would be something like a single second of audio.

Instead of saying "Does this single second of audio show up in other songs?" we should ask "Do the frequency components in this song show up in other songs?" (similar ratios of bass, treble, etc.)</comment></comment>


PostgreSQL needs a new load balancer - <AUTHOR> http://www.databasesoup.com/2012/03/postgresql-needs-new-load-balancer.html

<comment><USER2> The article is from 2012, but its contents is still very true. The only general purpose solution out there is pgpool, but from my own experience, unfortunately, it's less than stable at times and it has its own interesting quirks you have to live with (like having a SQL parser that lags behind PostgreSQL 1-2 releases).

Then there are bugs. Sure: As they are discovered, they get solved quite quickly, but if you are depending on certain functionality, it can still be very annoying.

It's also touted as a HA solution, but from my own experience last year, I greatly increased the overall stability of an installation by getting rid of pgpool - there's too much magic going on in there.

Finally, I was quite unhappy about them silently fixing a case where an unauthenicated client could request all the servers memory to be consumed. No security announcement - not even a release was made.

Unless you are very willing to spend a lot of time working around quirks and deeply familiarizing yourself with the source code, I would recommend against pgpool at this time, which leaves us, again, without any general-purpose load balancing solution. Other solutions have restrictions that make them quite situational (no prepared statements, non-deterministic cleanup after the client disconnects, etc).

<comment><USER3> The article is definitely still true. When using Postgres in larger systems it's always a problem when the client asks "well what about redundancy?" — there is no good answer to that, for reasons outlined in the article and the post above.

<comment><USER2> That's entirely not true. Postgres has built-in replication and you can even do synchronous replication if you want to pay for the latency - you can't have more redundancy than this.

Doing the failover is trivially done using something like keepalived or heartbeat as Postgres has built-in support for quickly promoting a slave to the master.

What we don't have, however is good load balancing. Also, because load balancing is f'ing hard to get right when you are in a master/slave replication configuration because it's really hard to know what queries you can safely send to the slave. After thinking quickly, here are some cases: use of nextval/currval, use of select after insert/update in a transaction, querying unlogged tables, and probably many more.

A good load balancer knows about all of this.

<comment><USER4> I think in spirit the post you were replying to meant load balancing?

</comment></comment></comment></comment><comment><USER5> We ditched PgPool for PgBouncer for connection pooling from our webservers. PgPool tries to do way to much, and the documentation is horrible. PGBouncer been rock solid compared to PgPool.

I'd love for PgBouncer to poll the servers it's connected to and automatically talk to the master, so in case of master failure I would only have to promote a new master (and STONITH) and PgBouncer would auto-failover to the new master.

<comment><USER6> Yes, PgBouncer is a little better but could still be improved upon in significant ways. E.g. Getting PgBouncer working with SSL connections- You have to do it through a specific version of stunnel. Then stunnel always has to be running in addition to PgBouncer. Not friendly or fun to setup.

<comment><USER7> Out of curiosity, are you using Postgres with SSL for an offsite or multi-DC replica? Would switching to a site-to-site VPN be more efficient here?

<comment><USER8> Wouldn't using site-to-site VPN be adding a SPOF just as well, namely the VPN server (and, in addition to this, make the VPN server a bottleneck for transfer speed) ?

<comment><USER9> You can "easily" enough set up two VPN connections on separate pairs of machines and route to a virtual IP on each end and use ucarp or keepalived to have one or the other take over. It's not pretty, but it works.

And the VPN server may very well become a bottleneck for transfer speed at some point, but most of us won't ever need to deal with a level of bandwidth where that's an issue.

</comment></comment></comment></comment></comment><comment><USER10> I think the elephant in the room for postgres is its lack of bidirectional log replication. Once that finally happens, I'm sure dead-simple balancing tools will start popping up everywhere.

<comment><USER11> Is there progress on this front?

<comment><USER12> Is that what [http://wiki.postgresql.org/wiki/BDR_Project](http://wiki.postgresql.org/wiki/BDR_Project) is about? They aim to "implement main BDR features into core Postgres" in versions 9.4. Looking at past history, that could be the third quarter of 2014 ([http://en.wikipedia.org/wiki/PostgreSQL#Major_releases](http://en.wikipedia.org/wiki/PostgreSQL#Major_releases)), but of course, that is just a guess.

</comment></comment></comment><comment><USER13> Clustering is very hard to do correct. Clustering while maintaining unique auto_increments, solving split brain, rolling back transactions, ... while maintaining speed: even harder. So the cluster is never going to be "simple", might be simple to install, but never simple.

Side node: if you want load balancing, no replication, yes then it's easy to do :)

</comment><comment><USER14> What about Postgres-XC for load balancing? I know it does several things, but isn't that one of the use cases?</comment>


Show HN: Introducing Magic – Decentralized Internet Service from the Future - <AUTHOR> https://medium.com/helloitsmagic/introducing-magic-decentralized-internet-service-from-the-future-80250cd037fa <comment><USER2> what is "proof of transport"? how it's implemented? What is your added value? (and how are you making money...)</comment>


How the Internet killed porn - <AUTHOR> http://www.guardian.co.uk/culture/2012/jun/05/how-internet-killed-porn?cat=culture&type=article&utm_source=dlvr.it&utm_medium=twitter

<comment><USER2> I like Theroux, he's really smart, and not afraid to ask hard questions. But I think he missed something about the US porn industry. It was sort of the reverse of the MPAA movie industry, in that lots of money went in the bottom, for VHS tapes, magazines, XXX theatres, etc, and very little came out the top. It appears that "actors", "producers" and "directors" were all paid a marginal amount. Were did the rest of the input money go? My guess is to the Mob or to various Mobs, but who really knows? Maybe it was all money-laundering in the first place.</comment>


NSA fallout hits U.S. business to the tune of $4 billion: Brazil ditches Boeing - <AUTHOR> http://falkvinge.net/2013/12/18/nsa-fallout-hits-american-business-to-the-tune-of-four-billion-dollars-brazil-ditches-boeing-buys-gripen/

<comment><USER2> No US tech company can be trusted anymore. I expect Cisco to be banned worldwide in particular.</comment>


Opera Mail – standalone mail client - <AUTHOR> http://my.opera.com/chooseopera/blog/2013/06/11/introducing-opera-mail <comment> mike-cardwell Just tested this with [https://emailprivacytester.com/](https://emailprivacytester.com/) and it _failed_

Even though it claims to not load remote content by default, and pops up a "Load Images" dialogue, it actually loads content from the html5 video and audio tags immediately.

So whilst this doesn't work unless the user clicks "Load Images":

      <img src="http://TRACKING_URL/">

This does:

      <video autoplay="true" src="http://TRACKING_URL/"></video>

<comment><USER2> Ouch! Looking at it. Have created a bug about it already, so no need to report.

<comment><AUTHOR> I like it a lot when someone involved with the project answers to "bug reports" or complaints like this. Gives me the feeling that they care. I see this a lot on hacker news and it always puts a smile on my face.

</comment></comment></comment><comment><USER3> Does it offer PGP integration?

Well-designed email clients with proper privacy seem to be very hard to come by, and new players in that niche would be welcome. Particularly in the current climate, I would argue.

<comment><USER4> Yeah, what happened there? I had Eudora in the mid-90s running PGP integrated really nicely. I was talking with a friend how had never used PGP before and I realised somehow I just stopped using it and don't remember when. Maybe it was just the romantic ideal back then that we were important enough to have our mail read, or maybe it was that we actually personally knew the ISP admins and really knew they could read our mail.

</comment></comment><comment><USER5> Well, just installed and I rather like it. Sleek design. Simple interface. RSS is a good addition. Sadly, though, no auto-feed discovery like many RSS clients do nowadays.

At first glance looks more appealing than Thunderbird. Certainly so in configuration, which is simpler and more intuitive. Thunderbird's single SMTP pane for all accounts annoys me each time I see it.

Lacking vertical positioning for the "List and Message below" layout. But I guess it's just a bug.

They launched it very timely, when the cloud is no more to be trusted. Behold! The return of the desktop app! :)

<comment><USER6> What does the way you access your data has anything to do with where it is stored? Just because you read your emails from a local copy, don't fool yourself into thinking that the drawbacks of the cloud no longer applies to you.

Now, if you were to host your own mail gateway, that is a different matter entirely.

<comment><USER7> Using a standalone desktop app allows you to use encryption (PGP/GPG) while continuing to use hosted services like Gmail/Hotmail. In other words, you can again start treating networks as dumb pipes.

<comment><USER8> They still get access to the "metadata" though

<comment><USER9> Fake it, then. Fake yor location, fake your mac address, fake your time and timezone. &c.

<comment> mike-cardwell You can't fake the IP address you connect from. You can't fake the email address you're sending to. You can't fake the account you're using to send. You can't fake the time that you connect.

<comment><USER9> Then use Tor.

Have a disposable email address.

You don't have to fake the account just make one just for this purpose. Well, time you cannot fake but with just known value of time and many other unknown variables the equation and possible solution would be useless, practically.

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> Look at all the comments under the announcement. Really sum's up how I feel as well. A selling point of Opera was the built in email client.

I liked how Opera used to be a silo for a whole bunch of features. When you closed it the rest closed as well. Was a great way to turn off work.

Now we have ANOTHER fairly generic stand alone email client. ANOTHER fairly generic webkit browser. If it wasn't branded with an Opera logo would anyone really give a shit?

<comment><USER11> > a silo for a whole bunch of features. When you closed it the rest closed as > well. Was a great way to turn off work.

Honestly, it sounds like you want an OS running in a virtual machine. Nobody is going to pursue your ideal "monolithic productivity suite that does everything" path; it's just a plain-bad idea.

<comment><USER12> > it's just a plain-bad idea.

That's a very nice opinion, but without actually qualifying why you think that way it has nothing to do with reality.

<comment><USER11> Because an operating system is a _good_ idea, and a monolithic do-everything productivity suite subverts the point of an operating system by creating a very limited, hard-coded, non-extensible OS _on top of_ the OS, reimplementing half of what the OS already does, but poorly, failing to interact with any components not distributed with the suite, and not allowing for any of the things modern OSes allow for (multi-display MDI support, for example, or package management.) There is no justification for having two redundant layers of OS; anything the suite does in terms of integration (for example, a clipboard implementation), could be converted to an OS feature of the parent system and used to integrate _anything_ with _anything_.

And if you really think the do-everything productivity suite is the one with the right ideas and the OS you're running is the one that's screwed-up--well, then, there's _still_ no reason to run two layers; instead, what should be done is to expand the productivity suite _into_ an OS. Like Chromium OS and Firefox OS are currently in the process of.

<comment><USER12> And now comes the part where you explain to me how i combine Firefox and Thunderbird into one app such that i can have various emails open in first- class-citizen tabs alongside web pages. Since apparently Firefox does such an amazing job of it.

</comment></comment></comment></comment></comment><comment><USER13> Looks interesting. I've been a Thunderbird user for a long time but the fact it appears to have been practically abandoned[0] by Mozilla is a shame.

The 'Import from Thunderbird' wizard appears to be a bit broken, though. It tried to import all my Thunderbird accounts but seems to have just munged them all into one account in Opera Mail. It would be nice if I could selectively import single Thunderbird accounts.

Also, as others have mentioned, it seems to be lacking compared to Thunderbird in auto configuration (in TB, I can enter a Google Apps email address and it'll figure out it's Google for me).

[0] [https://blog.lizardwrangler.com/2012/07/06/thunderbird- stabi...](https://blog.lizardwrangler.com/2012/07/06/thunderbird-stability- and-community-innovation/)

</comment><comment><USER14> I am not particularly happy with its integration with Gmail. Each email from facebook group is assigned separate mailing list. So now when I expand mailing list it is just cluttered with rubbish sublists like [0-9]*.groups.facebook... In general part which is responsible for detecting what is actually a mailing list isn't very clever. It would be nice if instead of creating folders for each label on Gmail it would create new label - so that it will be 1-1 integration.

Design seems really neat and if it will provide better Gmail integration I would love to use it. Of course I realise it is not going to be number one front-end for Gmail ported from web to standalone application, but I think that most popular mailboxes deserve it.

Is anyone here using fastmail and can tell how well it integrates in it? Is it more ingelligent and joins labels from fastmail with those in OperaMail or it just works the same way as with Gmail - new folder for each label?

<comment><USER2> Mailing lists are not the same as labels. Labels and Mailing lists are two different features (albeit very similar).

Mailing list only looks at the List-Id: email header. Facebook is sending that in its emails, and that's why it gets its own "Mailing list" automatic entry.

Not that beautiful, I agree.

<comment><USER14> As for labels, I meant that both OperaMail and Gmail have system of tags. However Gmail tags are treated as IMAP folders not using OperaMail tags. I realise that creating new tags in OperaMail may get out of sync with Gmail, but the the other way round - having labels from Gmail imported as labels to OperaMail should be possible to do.

</comment></comment></comment><comment><USER15> Good. But what I want to see, is Opera web browser _without_ an email client (and all the other stuff I never use.) Opera browser badly needs some slimming down.

Call me old fashioned, but I deliberately use a prehistoric email client _precisely_ because it does not know anything about active content. Barely even handles html. And I like it that way.

Now if Opera's email client had a big master switch, to totally disable all features beyond strictly dumb ASCII, and a means to import huge old mail archives from Eudora, then I'd be interested.

Edit: Correction: to disable anything beyond dumb ASCII plus UTF-8 foreign language support. But NOT html or any other active content.

<comment><AUTHOR> opera browser without e-mail client: [http://www.opera.com/developer/next](http://www.opera.com/developer/next) Not for linux yet, though.

<comment><USER15> Good news. I'm OK with Windows - I'd rather be enraged with Microsoft, than with the Million Monkey syndrome.

<comment><USER6> I think I could connect the Million Monkey syndrome to the Linux ecosystem, but what does it exactly mean?

</comment></comment></comment></comment><comment> mike-cardwell I just tested running the Windows version under Wine on Debian Wheezy and it works.

</comment><comment><USER16> The main thing I care about with a mail client is the exit strategy - if I don't like it, can I export all my mail losslessly to mbox or eml files?

Does anyone know how it fares at this?

I use gmail now, though in the past I've used The Bat.

<comment><USER17> If your server has IMAP, you shouldn't need that.

<comment> T-hawk You may also care about a server exit strategy. Your local storage format becomes relevant in the event you want to change email servers. (Maybe if, for example, your email server decides to stop supporting IMAP to get everybody to use the ad-serving web interface instead. Or your email server belongs to a large company suspected of leaking to the NSA.)

</comment></comment></comment><comment> mike-cardwell Always good to see a new desktop client. Hopefully they'll add Kerberos auth, PGP/MIME and read/write LDAP based address books. Until then I'll stick with Evolution for those features.

</comment><comment><USER18> The download page is so sleek it doesn't even have a link to documentation so I can see what I'm downloading before wasting my time.

</comment><comment><USER19> Great news! Thunderbird was horrific on my aging Windows machine last year, so always had Opera open just for email.

</comment><comment><USER20> Been using this since the release candidate a week or so ago and absolutely loving it. Has a few quirks here and there which need to be configured or ignored, but I've found it the best organising tool for mail on OSX in a long.

</comment><comment><USER21> I'm kind of disappointed by the first-launch experience: there's no configuration wizards for common email providers, not even for fastmail which is an Opera division AFAIK.

Edit : Actually there is, but it's not very discoverable.

<comment><USER22> There is, I only entered my fastmail address (not even using the fastmail.fm domain) and it configured automatically.

<comment><USER2> Yes, we have a shipped XML configuration file which finds and sets up many common domains automatically.

</comment></comment></comment><comment><USER23> Loaded up my gmail in it. Then went to attachments, documents. Lots of hits from my gmail spam-folder. This needs work...

edit: the basic mail function looks okay. looking forward to testing the linux version against exchange.

</comment><comment><USER24> Wow, that sure is an annoying product page. Where are the details about what mail providers they support? Does it do calendering? What are the system requirements (WinXP, OS X 10.5, CentOS 5, etc...)?

<comment><USER2> Supports IMAP and POP. Mail providers like gmail, yahoo and hotmail should work pretty well. Doesn't do calendaring. Runs on WinXP. Don't know about that old OS X version, but I think it probably does. Runs on old Linux (when we release the Linux version that is). You don't even need gtk or qt, though it'll use them if it can find them.

<comment><USER24> Excellent! Now put that information on the web page somewhere so that others don't have to wonder. It doesn't even have to be on the home page, just _somewhere_. :)

As a downside, I see that it doesn't support Exchange and that pretty much dooms it for me, at least on OS X. I hate Exchange, but I have to deal with it. Luckily, Apple's Mail.app seems to work fairly well with Exchange so at least I'm not forced to use Outlook.

<comment><USER25> [http://davmail.sourceforge.net/](http://davmail.sourceforge.net/)

</comment></comment></comment></comment><comment><USER26> Installed it but didn't use it. I don't want to go through the hassle of looking for the configuration info of my GMail (one of the most popular email services) account. So 2003.

<comment><AUTHOR> it should guess the details correctly after you entered your address.

</comment></comment><comment><USER27> Looks interesting, definitely will give it a go.

Can anyone suggest other mail clients worth trying out? I'm tired of Mail.app these days.

</comment><comment><USER28> I never understood tabs in an email client

<comment><USER6> I use it to put important mails such as tasklists on separate tabs so that I can come back to them easily and keep reading other emails.

</comment></comment><comment><USER29> Is there a way to connect it to the OSX Address Book or Gmail contacts? Doesn't look like it?

</comment><comment><USER30> Looks great, I just need to wait for the Linux version before I give it a go :)

</comment><comment><USER31> When is Mozilla going to make a Thunderbird addon for Firefox?

</comment><comment><USER32> An important feature for email: fast and flexible search over a large dataset. This is easier to implement in the cloud than on the client. So I don't see a strong future for desktop-based email clients.

<comment><AUTHOR> Funny you mention it. Search in opera mail is excellent and extremely fast. It searches through all of my 12k mails in the blink of an eye.

</comment></comment><comment><USER33> It is also a RSS reader. I don't really get it.

<comment><USER34> Can't you also read RSS using the OSX Mail.app?

<comment><USER35> Or any other mail client really. What I don't get is why they stripped it out from the browser (I didn't take up any disk space really).

edit: I really hope they don't take out RSS reader functionality out as well. I never got why other browser didn't support such basics when it came to browsing.

<comment><AUTHOR> They are building a completely new browser based on chromium. So they didn't really strip it out, they just didn't add it to the new browser.

<comment><USER36> Switching to Chromium was a big mistake IMO, seeing how inferior, unstable, and slow the new Android app is compared to the original. I get it, they had to cut costs, so they fired a bunch of people and gave up their proprietary edge. It just seems to have been the wrong decision because they took away one of the core features that used to make it great (at least with regards to the Android app): speed.

<comment><USER35> From what I've read the desktop version lost basically everything that was special to Opera. I know it's early builds but I think many features won't make it back.

</comment></comment></comment></comment></comment></comment><comment><USER37> Windows only?!?

<comment><AUTHOR> I expect Linux and Mac versions to follow. They have released windows version first on new products in the past.

<comment><USER2> I work on Linux version of Opera Mail. Windows and Mac is released today. A rc/test version of Linux will follow soon.

(What is soon, you ask. Well that depends, but I guess in this case over a month would /not/ be soon.)

<comment><USER37> Thanks for replying here. I couldn't find other links to the Mac version so I inferred that there was only Windows support. It would be nice to have "Linux coming soon" page instead of "Download for Windows" by default :)

</comment></comment></comment></comment><comment><USER38> So, how secure do we know this is?

</comment><comment><USER39> No Linux support boo!

</comment><comment><USER40> I see no mention is made of their secret access portal to NSA...

<comment><USER41> lol? This comment is a little confusing considering Opera is based on Norway. And as far as I know Opera is probably one of the better ones in regards to NSA said stuff.

</comment></comment><comment><USER42> I'd like to hear the full story on how Opera self-destructed as a company. Someone should write a book or something.

<comment><USER43> Why do you think Opera self-destructed? They announced their quarterly financial results a couple of weeks ago, and they set another revenue and profit record. They're basically growing on all fronts, have plenty of cash in the bank, and are growing their profits.

So again, what self-destruction are you referring to?

<comment><USER12> They just announced that Opera 12 will be discontinued, with Opera 15 being a Chrome fork in which they will attempt to reimplement features from Opera 12. They also announced that several features which people liked a lot will never make it into Opera 15. And to put the cherry on the top: The very first feature that they announced removed were "Bookmarks".

In other words: They're very busy alienating massive parts of their user base.

<comment><USER43> You don't seem to understand what's going on.

Most Opera users are on mobile, not desktop. Opera is struggling on desktop. Sites weren't working, and that was the #1 problem with the browser. Fixing that will let them grow. Opera's got nothing to lose on the desktop. They can only gain users by doing this, even if a handful of hardcore users drop them.

Most people don't need most features in Opera 12, but they're just getting started with the new Opera. Now they can create a proper foundation instead of this:

[http://www.favbrowser.com/opera-the-past-the-present-the- fut...](http://www.favbrowser.com/opera-the-past-the-present-the-future/)

Opera 15 is not a fork of Chrome either. It's using Chrome (Chromium) as-is, but with a new user interface on top of it.

In other words: They may alienate a small hardcore group, but fixing site compatibility will more than make up for it because other people can finally start using it.

<comment><USER12> Check their financial reports, desktop is 25% of their income: [http://business.opera.com/company/investors/finance/](http://business.opera.com/company/investors/finance/)

> Opera 15 is not a fork of Chrome either.

Are you thenwhat? Just on the off-chance that you're not him: What you described is exactly what a fork is. Please don't try to comment on software development until you've actually gained the experience that job entails.

<comment><USER43> Desktop is a smaller and smaller part of their income. It could disappear tomorrow, and the company would still be in great shape. It's growing all over the place. So again, claiming that Opera self-destructed is silly.

Thenwhat? Huh? A fork is taking the code in a new direction and making it differ from the original. Using the code as-is, is not a fork.</comment></comment></comment></comment></comment></comment>


Something wrong with iCal - <AUTHOR> http://imagebin.ca/view/wYqCsa.html

<comment><USER2> Mine is ok: <http://srtd.org/~yan/Picture%204.png>

<comment><USER3> Just checked. Mine is fine too...

</comment></comment><comment><AUTHOR> Another one: <http://imagebin.ca/view/UmJ_Midi.html>

</comment><comment><USER4> What are we looking for here?

<comment><USER5> Day 14 shows up twice (day 15 is missing).</comment></comment>


To Fork, or to PR; thats the question - <AUTHOR> Hello,
What i want to discuss here and maybe in the end can come out with a manual about "The Etiquette of Open Source" is how to approach various situation when doing open source software. For example, some of the questions i constantly face are:

  - What is the best practice to define if you should PR the original repo or fork from it ? What if the original repo has no activity anymore or is not maintained by the author (sometime this is even clearly stated by the author)
  - What is the Etiquette to maintain the original authors rights when forking ? How can you define his contribution ? is it based on the code yo exactly copied, the idea, the architecture ?
  - How to handle cases when there are no license information attached ?
  - How to handle PRs/forks when you work merges/combines more than one original repo ?
  - How to handle cases when conflicting licenses are used across repos that you have referenced ?
 I look forward for your comments and insights <comment><USER2> > What is the best practice to define if you should PR the original repo or > fork from it ? What if the original repo has no activity anymore or is not > maintained by the author (sometime this is even clearly stated by the > author)

Always send a PR unless the maintainer has made it clear they're not accepting PRs. If you can't manage to get your PR merged, then maintain a fork instead.

If the original repo is no longer maintained, check to see if someone else has already started a fork and see if you can contribute to that. Otherwise, start your own fork.

> What is the Etiquette to maintain the original authors rights when forking ?

Legally speaking, follow the license. Most licenses which require attribution include a copyright notice from the original author, which should be sufficient attribution in and of itself.

Ethically speaking, git keeps track of everyone's contributions automatically, meaning you don't have to worry about distinguishing what you wrote from what the original maintainer wrote; it's all in the history.

> How to handle cases when there are no license information attached ?

In that case the software is proprietary and you have no license to use, copy, or modify it without explicit permission from the copyright holder. (This isn't merely etiquette, it's the law.) Contact the author and see if you can get them to license the software to you, or release it under and open-source license.

> How to handle PRs/forks when you work merges/combines more than one original > repo ?

Send PRs to both? I'm not really sure what you're really asking here.

> How to handle cases when conflicting licenses are used across repos that you > have referenced ?

You mean if you're grabbing random bits of code from multiple different projects and using them in your own project? Again, follow the license. Some licenses may be incompatible with each other, meaning you legally _can 't_ mix code between them. Other licenses may allow it as long as you clearly attribute where each bit of code you're using originally came from. Read and follow the license.

<comment><AUTHOR> > How to handle PRs/forks when you work merges/combines more than one original > repo ?

What i meant here .. is that you might end up merging code from multiple repos together. To elaborate more, your work is based on two repos X,Y .. you might end up committing code from Y into X and code from X into Y with your edits in those, which will result in license conflicts that maybe the original repo owners do not want to adhere to .. so maybe a fork would be more suitable here as you will be the one responsible for that. I hope i was clear

<comment><USER2> In that case, see my answer to "How to handle cases when conflicting licenses are used across repos that you have referenced ?".</comment></comment></comment>


Go is Not Python - <AUTHOR> http://www.seanhelvey.com/go-is-not-python <comment><USER2> I'm not sure what the title has to do with the article at all. It's just pointing out a language gotcha/subtlety. The title might as well have been "Go is Not Javascript", "Go is not QBasic", etc.

<comment><USER3> This is the kind of "gotcha" that you learn pretty quickly when working with go. The problem is that Go has created a new variable called "student" that is assigned as its value each item in students as it ranges through. Go is not going to allocate a brand new "student" variable for each iteration of the loop. That would be a waste of time and memory both during the loop, and later during GC. So your "fredPtr" given this syntax, would be more accurately named "studentPtr."

Go won't hold your hands because it's more efficient if you are explicit. If you want a "reference" to the real original, your array needs to be an array of pointers.

It's pretty simple to understand when you have worked with pointers before and understand how the memory is allocated and eventually collected (and why you don't want to reallocate every loop iteration).

Finally, nil is the default value of anything with type (star)Student, and almost nobody uses arrays in Go ([2]Student is an array, []Student is a much- more-useful slice). In general, I don't think this author has written much Go code. It's probably a good idea to get a little more experience with whatever you're complaining about before writing a blog post about it, and likely misinforming your readers.

It's pretty straightforward code when you write what you explicitly want:

        package main

        import (
          "fmt"
        )

        type Student struct {
          name string
        }

        func main() {
          students := []*Student{{"fred"}, {"karen"}}
          var fredPtr *Student

          for _, student := range students {
            if student.name == "fred" {
              fredPtr = student
            }
          }

          fmt.Println(fredPtr.name)
        }

<comment><USER4> Ironically, your code looks much more like Python than the author's. "Simple is better than complex" at play, I suppose. If I were to sit down and start writing Go, the idea to use equality would come long before I bothered with pointer notation, especially for such a simple task.

</comment></comment></comment><comment><USER5> Here's a more idiomatic translation of the Python code to Go, and there's no surprise to how the code behaves:

[https://play.golang.org/p/rpqvVDR9v8](https://play.golang.org/p/rpqvVDR9v8)

<comment><USER6> Nope, that's not what the OP is trying to do.

eg:

[https://play.golang.org/p/kFjXSlf5fw](https://play.golang.org/p/kFjXSlf5fw)

</comment></comment><comment><USER7>
      fredPtr = nil // This line is superfluous.

In Go, declarations always initialise variables to their zero value, which for all pointers is nil.

(Perhaps worthy of a follow up article: "Go is Not C/C++"? ;)

</comment><comment><USER8> Alternative options of solving this issue include `break`ing when the right option is found (so the loop variable doesn't get overwritten again) or moving everything into a function:

      func findByName(students []*Student, name string) *Student {
        for _, s := range students {
          if s.name == name {
            return s
          }
        }
        return nil
      }

Both options have the added advantage of stopping iteration as soon as the result is found.

</comment><comment><USER9> I think you can put this more generally as "syntax is not semantics." Go takes some syntactic cues from python (but who doesn't these days?) but its semantics can be pretty different. I think it's pretty common for new programmers especially to get caught up in the syntax of languages without giving enough attention to their semantics, which causes the sort of problems the author highlights here.

<comment><USER10> > I think you can put this more generally as "syntax is not semantics." Go > takes some syntactic cues from python (but who doesn't these days?) but its > semantics can be pretty different.

Except the difference is not in similar syntax with different semantics; since it's the application of the & operator, which has no close parallel in Python, that is the source of the surprise.

</comment></comment><comment><USER11> I love Python. But honestly speaking, the point raised in this article is fairly trivial. I would not, not learn Go just because there are small differences in the way reference / pointers work. I do not see this as an inherent shortcoming of Go.

Both languages have their strengths. When you pick a language for an application, it is your responsibility to use it properly. Otherwise you will make mistakes and it won't be the fault of the language.

<comment><USER10> > But honestly speaking, the point raised in this article is fairly trivial. I > would not, not learn Go just because there are small differences in the way > reference / pointers work.

Nothing in the article suggests not learning Go because of this behavior; the article is warning about a behavior developers coming to Go from Python are, in the author's experience, often bitten by, so that other people taking the same path can avoid making the same mistake.

> When you pick a language for an application, it is your responsibility to > use it properly.

Yes, and this article aims to help people coming to Go from Python to do that by pointing out a common pitfall.

</comment></comment><comment><USER12> I don't know Python or Go, but both examples seem _wrong_.

Python: How is `Fred` available to be printed in the Python example, is not it out of scope?

Go: Why use `var fredPtr *Student`? What's the point? Can not use `var fredPtr Student` and the set `fredPtr = student` in the loop?

</comment><comment><USER13> been a while since i've done pointer stuff, isn't that how it works in c? go seems de-evolved to me.

<comment><USER14> C has no concept of the kind of for loop used in the article, so there is no "how it works" in C.

<comment><USER10> While C has no for-in loop, the natural translation of that to a C style for loop and the C address-of operator would work the same way; the key thing is that student is a variable defined outside the loop whose value is modified each iteration, not a fresh variable allocated with each loop iteration.

</comment></comment></comment><comment><USER15> This has nothing to do with Python. It's just a "gotcha" of Go.

<comment><USER16> I believe he's talking about what he should teach to students, and his point is that Python is more intuitive than Go for newcomers.

I personally don't really care how easy it is to write a "hello world" in Go, nor Python. My day job involves big projects, so I'm pretty fine with "hello world" taking 10k lines as long as it will scale slower with complexity than other language. The same argument about learning curve -- I would be happy with a language that requires couple of years of learning to even start, as long as it takes less time in the following 20 years I'm going to be using it.

<comment><USER17> _I believe he 's talking about what he should teach to students_

What gives you that impression?

<comment><USER16> Huh, I was reading it fast, and didn't realize that word "students" in the text meant a variable name, not his students. My bad :)

<comment><USER17> Heh yeah and I ended up re-reading it looking for some indication the writer was an instructor that I'd missed and didn't notice 'students' at all.

</comment></comment></comment></comment></comment><comment><USER18> for a no-goer: will this copy by value? ``` var fred Student for _, student := range students { if student.name == "fred" { fred = student } } ```

<comment><USER8> Yes. A copy into type `Student` will copy each field, whereas `*Student` only takes the address. (Which would not be a problem if the loop didn't reuse the memory slot for the loop variable.)

Also, indent with two spaces to get code formatting. HN formatting is (sadly) not Markdown.</comment></comment>


JavaScript Developer Survey 2011 Results - <AUTHOR> http://dailyjs.com/2011/12/15/javascript-survey-results/

<comment><USER2> I don't understand why more people aren't using Rhino. Or are they just not taking that survey? Doesn't anyone else see the beauty in scripting with js, with all the power of the jvm?</comment>


Back home with Debian - <AUTHOR> http://www.ebb.org/bkuhn/blog/2010/01/14/ubuntu-debian.html

<comment><USER2> Is it odd that I (somewhat) agree that the priority should be:

1\. Open Source 2\. Free Software</comment>


Microsoft's 'mixed reality' headsets are a bit of a mixed bag - <AUTHOR> https://techcrunch.com/2017/08/28/microsofts-mixed-reality-headsets-are-a-bit-of-a-mixed-bag/ <comment><USER2> I've built a few HoloLens prototypes for my present company, and a few Occulus ones for my former. I haven't stayed up to date with all of the internal hardware specs, but from my experience building for the HoloLens was easier and a lot more fun. I haven't seen these new headsets yet, but I believe they use the same tools and frameworks. There's some really cool stuff that some Xbox developers have put out that you can build on top of. And the Skype app works amazingly well.

I'm just curious if these headsets are aimed at gaming as opposed to the HoloLens which is geared towards productivity.

<comment><USER3> I'm interested in getting into Hololens development. What languages, frameworks and tools would you recommend? From what I've read it seems like I'd be using Unity and C#.

<comment><USER4> Unity for 3D applications. You can use UWP to make traditional 2D apps which you can also use in hololens

<comment><USER5> You can also use C++ and DirectX for 3D applications.

</comment></comment></comment></comment><comment><USER6> Why did they decide they could sacrifice frame-rate when the first generation of PC VR - after extensive study - decided it was the one thing you should never sacrifice?

Did somebody in sales overrule the engineers?

<comment><USER7> > first generation of PC VR

What do you mean by this?

I'm only asking for clarification, because when I hear "first generation of PC VR" \- I think "REND386".

I doubt, though, that many other people do, because for some reason, all of that old history and the lessons from it have seemingly had to be "re- discovered" with VR technology after the Rift came out.

<comment><USER6> Yeah - I did wonder about clarifying that as I typed it - there's obviously a long history to VR (I'm a child of the 80s so I'm not completely ignorant of the twists and turns of computing history)

<comment><USER8> VR was a big deal in 1995:

[https://books.google.com/books?id=79i1lfAqumUC&lpg=PP1&pg=PA...](https://books.google.com/books?id=79i1lfAqumUC&lpg=PP1&pg=PA168#v=onepage&q&f=true)

<comment><USER6> I remember it well on BBC Tomorrow's World.

I'm always tempted to watch "The Lawnmower Man" although I suspect it hasn't aged terribly well.

</comment></comment></comment></comment></comment><comment><USER9> I wasn't able to tell from the video or the article... what does mixed reality mean?

I was assuming it was something like augmented reality lite. This just looks like VR-lite.

<comment><USER10> It's a cross between AR and VR in which a different virtual world is inserted into the real world [1]. It predates the term, but the HoloChess scene in Star Wars is to me a good example. The contrast is with AR, in which the real world is annotated with virtual info, and VR, in which the real world is hidden. But the terms are a bit blurred.

[1] [https://en.wikipedia.org/wiki/Mixed_reality](https://en.wikipedia.org/wiki/Mixed_reality)

<comment><USER6> This isn't how Microsoft is using the term. For them it's essentially a hollow marketing differentiator and a way to lump Hololens and inside-out tracking under the same banner.

To all intents and purposes just substitute "VR" whenever Microsoft says "Mixed reality".

I really hope their usage doesn't catch on.

</comment></comment></comment><comment><USER11> I wish the article would have gone into more detail about the tracking on the controllers. My biggest concern with those is that they might not work well in games where you have to reach and grab something not currently in your field of view, like guns holstered on your back or at your hips.

<comment><USER12> This article takes a deep dive on the controllers:

[https://www.roadtovr.com/microsoft-mixed-reality-vr- motion-c...](https://www.roadtovr.com/microsoft-mixed-reality-vr-motion- controllers-hands-on/)

<comment><USER11> Thanks, that _is_ a bit more detailed. Unfortunately, it seems like my concerns were well founded:

> And for times when your hands will go out of the camera's field of view, > Microsoft is doing its best to compensate. When that happens, the system > relies purely on the controller's on-board IMU to estimate positional > movement until it reappears in the camera's view. This works well enough for > quick jumps in and out of the camera's view, but after a second or two, the > IMU-only tracking estimation is too unreliable, and it appears that the > system will eventually freeze the location of the controllers in the air and > only feed them the rotation data from the IMU

Not a lot of detail there on how well the IMU-only tracking works, but it doesn't sound like it's all that accurate. And I guess there's only one front- facing camera, so it'll be pretty easy to lose tracking if you ever need to interact with something outside your FOV.

<comment><USER13> This same thing happens with the Vive controllers. The shape of the controller and how you use it make it much more susceptible to occlusion issues than the headset. It also has an internal IMU to attempt to compensate, but it's only good for about 3 seconds before you start to notice the drift. There are a lot of sub-3 second occlusion events that you just never notice, otherwise. So who knows, I'd have to see the MS controller in action to get a better idea. I wouldn't be surprised if MS could come up with a decent mitigation.

<comment><USER11> The Vive has two base stations which are supposed to be positioned in opposite corners of the room. In order for the controller to lose tracking, it has to be hidden from the view of _both_ of those base stations at the same time. Still possible, but I suspect that will happen much less frequently than with this system, which as far as I can tell will lose tracking whenever your controller goes outside the view of a single, front-facing camera on the headset itself.

For example, if you reach behind your back it loses tracking. If you look in one direction and aim a gun in the other, it loses tracking, if you look up and reach for pistols holstered at your hips, it loses tracking, etc. Someone please correct me if I'm wrong, because as-is it sounds to me like this system isn't going to work very well for a lot of the more action-based games in VR.

<comment><USER13> There are definitely "shadow spaces" in even an ideal Vive setup where you're body could be occluding one base station and your hand holding the controller could be occluding the other.

It's really going to depend on the application. You don't focus your attention uniformly through the 3D space. if the application design has important interaction points that just so happen to intersect with those shadow spaces, you'll find yourself in a very frustrating situation of never being able to get good tracking. There are times when I find I can't finish a drawing in Tilt Brush or a machine in Fantastic Contraption very easily because the detail I want to work on is in a shadow space. And for many reasons, I think gun-games are quite far from the best uses of VR. I am sure a certain class of gamers will like it, but I don't think that class intersects well with current, 2D-display FPS gamers, and I don't think it will be anything close to the majority use of VR.

So yes, some UI designs will be ideal for one design and particularly buggy in another. The Vive is optimized for broad-stroke gesture control radiating out from the user in (roughly, minus the shadow spaces) equal precision 360 degrees around the user. The Windows MR is optimized for fine-detail control centered in front of the user. If you like playing wave shooters and only wave shooters, I suppose you'll want a Vive. Otherwise, I don't think it's a completely cut-and-dried situation.

<comment><USER11> > There are definitely "shadow spaces" in even an ideal Vive setup where > you're body could be occluding one base station and your hand holding the > controller could be occluding the other.

Huh, I've never really experienced that. Or if I have, it doesn't happen frequently enough for me to notice. It does happen in some spaces outside my play area, beyond the guardian system, but inside it tracking is near perfect. I have a Rift, not a Vive though, so maybe this is an issue specific to the Vive? My Rift uses 3 sensors, not 2, so perhaps the occlusion issue is lessened by that. Or perhaps your Lighthouses are just positioned a bit strangely?

> I think gun-games are quite far from the best uses of VR. I am sure a > certain class of gamers will like it, but I don't think that class > intersects well with current, 2D-display FPS gamers

Agreed. There are other games which would suffer from this type of occlusion issue though. Echo Arena, for example, often requires you to grab onto a surface with one hand while turning your head in a completely different direction to look at your surroundings. Any game that requires you to interact with something not in your field of view would have this problem.

For non-game applications like Tilt Brush, Google Blocks, or Google Earth though I can see this headset working just fine. Casual games which involve primarily working with your hands might also work well. (Job Simulator, Rick and Morty VR, etc. Not Fantastic Contraption though; that one requires you to reach behind your head to grab new parts, if I recall correctly.)

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> Isn't Microsoft jumbled anyways? In my opinion they're just wanting to get third hands in everything...

</comment><comment><USER15> I really feel like going through my post history on here and on reddit to point out all the people that said this was going to be a "game changer" that would effectively kill Oculus and Valve's VR hopes. I was skeptical back then and this only confirms that my skepticism was warranted. Everyone was saying that MS would be able to take a hit on the price and would effectively neuter the higher-price-point VR headsets. I kept making the point that _some_ compromise would have to be made to keep the price point down and several people fell back to Microsoft's financial status as proof of their ability to deliver a competitive product at a lower cost. Glad to see that I wasn't crazy then and that this is exactly what's happening here.</comment>


Four new NSA slides air on Brazilian TV - <AUTHOR> http://www.dailydot.com/news/nsa-fairview-slides-brazil-spying/

<comment><USER2> These slides don't seem nearly as alarming as the previous ones. The NSA intercepts data between Brazil and Pakistan. OK, that's its job. Maybe concerning if you're Brazilian or Pakistani, but no more so than espionage has always been.

</comment><comment><USER3> I'm sorry, but I feel like these could have easily been made up.

<comment><USER4> In which case, Greenwald will certainly make a public statement that they aren't real.

Until then, I'll believe that they're authentic.</comment></comment>


Asynchronous IO in Rust - <AUTHOR> https://medium.com/@paulcolomiets/asynchronous-io-in-rust-36b623e7b965 <comment><USER2> If you use threads, green or otherwise, you don't have to "implement" special code for composing things together, you get the full set of tools for composing code together, which includes, in passing, state machines, among all the other things it includes. This basically implements an Inner Platform Effect of an internal data-based language for concurrency that the language interprets, which will A: forever be weaker than the exterior language (such is the nature of the Inner Platform Effect) and B: require a lot of work that is essentially duplicating program control flow and all sorts of other things the exterior language already has.

There are some programming languages that have sufficiently impoverished semantics that this is their best effort that they can make towards concurrency.

But this is _Rust_. It's the language that fixes all the reasons to be afraid of threading in the first place. What's actually _wrong_ with threads here? This isn't Java. And having written network servers with pretty much every abstraction so much as mentioned in the article, green threads are a _dream_ for writing network servers. You can hardly believe how much accidental complexity you're fighting with every day in non-threaded solutions until you try something like Erlang or Go. Rust could be something that I mention in the same breath for network servers. But not with this approach.

There's plenty to debate in this post and I don't expect to go unchallenged. But I would remind repliers that we are _explicitly_ in a Rust context. We must talk about _Rust_ here, not 1998-C++. What's so wrong with _Rust_ threads, excepting perhaps them being heavyweight? (Far better to solve that problem directly.)

<comment><USER3> > What's so wrong with Rust threads, excepting perhaps them being heavyweight? > (Far better to solve that problem directly.)

Nothing. If threads work fine, use them! That's what most Rust network apps do, and they work fine and run fast. A modern Linux kernel is very good at making 1:1 threading fast these days.

> And having written network servers with pretty much every abstraction so > much as mentioned in the article, green threads are a dream for writing > network servers.

Green threads didn't provide performance benefits over native threads in Rust. That's why they were removed.

Based on my experience, goroutine-style green threads don't really provide an performance benefit over native threads. The overhead that really matters for threads is in stack management, not the syscalls, and M:N threading has a ton of practical disadvantages (which is why Java removed it, for example). It's worth going back to the debate around the time of NPTL and look at what the conclusions of the Linux community were around this time (which were, essentially, that 1:1 is the way to go and M:N is not worth it).

There _are_ benefits to be had with goroutine-style threads in stack management. For example, if you have a GC that can relocate pointers, then you can start with small stacks. But note that this is a property of the stack, not the scheduler, and you could theoretically do the same with 1:1 threading. It also doesn't get you to the same level of performance as something like nginx, which forgoes the stack entirely.

If you really want to eliminate the overhead and go as fast as possible, you need to get rid of not only the syscall overhead but also the stack overhead. Eliminating the stack means that there is no "simple" solution at the runtime level: your compiler has to be heavily involved. (Look at async/await in C# for an example of one way to do this.) This is the approach that I'm most keen on for Rust, given Rust's focus on achieving maximum performance. To that end, I really like the "bottom-up" approach that the Rust community is taking: let's get the infrastructure working first as libraries, and then we'll figure out how to make it as ergonomic as possible, possibly (probably?) with language extensions.

My overarching point is this: it's very tempting to just say "I/O is a solved problem, just use green threads". But it's not that simple. M:N is obviously a viable approach, but it leaves a lot of performance on the table by keeping around stacks and comes with a lot of disadvantages (slow FFI, for example, and fairness).

<comment><USER4> > Green threads didn't provide performance benefits over native threads in > Rust. That's why they were removed.

This seems wrong. This shouldn't be the point of green threads -- green threads aren't a "faster" alternative to native threads. How would that work? How could it be the case that virtual threads implemented in user space are faster than native threads provided by the OS? I don't think anyone expects that to be true of green threads in general.

We don't have green threads as a useful abstraction because they're fast, we have them because they're _flexible_ , and because they allow us to easily capture a lot of patterns. They let you easily and efficiently distribute work amongst multiple processors without having to think about the subtleties of the underlying threading model.

Of course, given Rust's other goals, shying away from green threads makes sense.

<comment><USER3> I'm confused. To me, green threads are just an implementation detail. The fact that green threading is being used shouldn't leak into the interface. To take Go for an example, you could perfectly well write a conforming implementation of Go that used 1:1 native threading: it would just have different performance characteristics and things like LockOSThread() would become a no-op.

Could you elaborate on what you consider the interface difference to be?

<comment><USER4> This is backwards. Green threading can be an implementation detail, but in languages where it's a key feature, green threads let you write code that would otherwise be incorrect. For example, in a native threading model, it is an awful idea to spawn a thread for every incoming connection on a server. It's the easy way to write it, but it's wrong. With a green threading model, though, that's easy and efficient.

<comment><USER5> I was curious what the actual state was of the "modern Linux kernel" pcwalton mentioned, so I tried running a test program to create a million threads on a VM - x86-64 with 8GB of RAM, Linux 4.0. For comparison, Go apparently uses about 4KB per goroutine, so it should be possible to create somewhat under 2 million goroutines. To be fair, I allocated the stacks manually in one large allocation; otherwise it dies quite quickly running out of VM mappings. I set the stack size to the pthread minimum of 16KB (actually, I tried cheating and making it smaller, but it crashed, so I gave up - not a good idea anyway). The threads waited for an indication from the main thread, sent after thread creation was done, to exit; in an attempt to avoid the overhead associated with pthread conditions, I just used futex directly:

        while (!ready)
            assert(!syscall(SYS_futex, &ready, FUTEX_WAIT, 0, NULL, NULL, 0));

The program caused the kernel to hit OOM (rather ungracefully!) somewhere around 270,000 threads. To see how long it took while ensuring all the threads actually ran, I reduced the thread count to 200,000, had it join all the threads at the end, and timed this whole process: after the first run it took about 4 seconds. (The first run was considerably slower, but that isn't a big deal for a server, which is the most important use case for having such a large number of goroutines/threads.) Therefore, the C version uses about 20 microseconds and 32 KB of memory per thread.

For completeness, I also tested a similar Go program on Go 1.4 (the version available from Debian on the VM); it actually got up to 3,150,000 before OOM, and took 9 seconds to do 2 million - 4.5 microseconds and 2.7KB per thread.

In other words, Linux is about an order of magnitude slower at managing a lrge number of threads. That looks pretty bad, but on the other hand, it's not that much in absolute terms! I'm pretty sure most server programs don't need more than 250,000 simultaneous connections (or can afford to spend more than 8GB of RAM on them) and don't mind spending an extra 20 microseconds to initiate a connection, so if operating systems other than Linux aren't a concern, they could be written to create a thread per connection without too much trouble. It's not going to give you the absolute maximum performance (meaning it's not appropriate for a decent class of program - then again, I suspect Go isn't either), but it's not terrible either.

I'd like to see it improve. I wouldn't be surprised if there is (still) some low hanging fruit; do kernel developers actually care about this use case?

(And yes, I know this doesn't really test performance of the scheduler during sustained operation. That's its own can of worms.)

<comment><USER4> > To be fair, I allocated the stacks manually in one large allocation; > otherwise it dies quite quickly running out of VM mappings.

Okay, so the test you did doesn't actually reflect the use case in practice. Can I expect to reach 200,000 threads if the threads are not all created at exactly the same moment? What if (God forbid) they're doing memory allocation? And if it does work out, will everything be handled efficiently?

<comment><USER6> Hope comex replies to your question. Typical green thread usage is spawn-em- as-you-need-em, so if in order to spawn lots of 1:1 threads I need to do it all up front, that could be very limiting or complicating.

<comment><USER5> Yeah, I just made a mistake - you can increase the maximum number of mappings using /proc/sys/vm/max_map_count; I tried doing that and switching back to normal stack allocation (but still specifying the minimum size of 16KB using pthread_attr_setstacksize) and it doesn't change the number of threads I was able to create.

...in fact, neither did removing the setstacksize call and having default 8MB stacks. I guess this makes sense: of course the extra VM space reserved for the stacks doesn't require actual RAM to back it; there is some page table overhead, but I guess it's not enough to make a significant difference at this number of allocations. Of course, on 32-bit architectures this would quickly exhaust the address space.

If increasing max_map_count hadn't worked, it would still be possible to allocate stacks on the fly - but you would get a bunch of them in one mmap() call, and therefore in one VM mapping, and dole them out in userland. However, in this case guard pages wouldn't separate different threads' stacks, you would have to generate code that manually checks the stack pointer to avoid security issues from stack overflows, rather than relying on crashing by hitting the guard page. Rust actually already does this, mostly unnecessarily; I'm not sure what Go is doing these days but I think it does too. Anyway, given that the above result I suspect this won't be an actual issue, at least until the number of threads goes up by an order of magnitude or something.

<comment><USER6> Thanks for your reply. Wonder how other platforms fare.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> It would be much nicer to use coroutines instead of state machines. This means that you could write code as if it were doing i/o in a "blocking" fashion, but behind the scenes it is actually asynchronous.

</comment><comment><USER8> I think a lot of folks think of Network IO when they say Asynchronous IO. That's only half the story, unless you're just building proxies and caches you have to deal with Disk IO at some point in time. And, async disk IO is horrible in every OS / language.

<comment><USER9> Well, apparently not on Windows. But this is a kernel interface issue not a language issue, and clearly needs fixing now we have SSDs where having thousands of outstanding requests is useful, if not required for performance, unless the hardware APIs are going to change (maybe if they get memory interfaces this does change).

<comment><USER8> The async story on windows is better, but not great. In many cases the async calls on windows silently block due to a number of special cases (that are not so special).

<comment><USER10> (Presuming you're referring to this: [https://support.microsoft.com/en- us/kb/156932](https://support.microsoft.com/en-us/kb/156932))

Those are all entirely reasonable things to block for if you understand the mechanics behind _why_ those calls may block. And none of it should be fast- path stuff; it's easy to architect a solution where you avoid those things except in corner cases.

The other thing you're missing is that when you architect around Windows completion ports and threadpool I/O facilities, it doesn't matter if one thread blocks every so often. Windows will detect this and schedule another one to run, such that there is one active thread scheduled on every core.

I/O completion ports facilitate thread-agnostic I/O completion; the thread that blocked because it was extending an encrypted file won't impede the latency of other clients because there is no thread-specific association.

I exploit all of this with PyParallel. [http://pyparallel.org](http://pyparallel.org)

For more details on asynchronous I/O on Windows and why it's fundamentally better than the UNIX I/O model in every possible way: [https://speakerdeck.com/trent/pyparallel-how-we-removed- the-...](https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and- exploited-all-cores?slide=47)

</comment></comment></comment></comment><comment><USER11> Really awesome stuff, great example of how well composability works with traits and the nice handling that tagged unions bring to state machines.

</comment><comment><USER12> This is identical to the Protocol abstraction in Python's asyncio, right?</comment>


Nearly Infinite - <AUTHOR> http://sethgodin.typepad.com/seths_blog/2008/04/nearly-infinite.html

<comment><AUTHOR> The sentence I liked most: "Just knowing what to search for is now as difficult as the search itself."</comment>