Eclipse committer on Android IDE switch to IntelliJ - <AUTHOR> https://twitter.com/guw/status/334793907625938944 <comment><USER3> The "workspaces" feature is at least partly broken, and there are bugs open which have been open for years, if not decades, suggesting or asking for improvements. But despite the weight of evidence that people want this improved, the Eclipse developers are generally more interested in adding the latest "gee whiz" features than dealing with "bread and butter" issues.

And, sadly, the UX aspect isn't even the biggest problem with Eclipse (all "IMO" of course). Performance, especially memory usage is a huge issue (especially on 64 bit systems) and the updater/plugin mechanism is too damn unreliable. If you use more than 2 or 3 plugins, it's almost inevitable that you will, at some point, try to update something and get the infamous "could not complete operation" message when adding a new plugin, or updating an existing one. Now, that's not really the problem... the problem is that eclipse just vomits a page and a half of error messages and opaque version numbers and strings at you, when this happens... instead of giving useful help on how to resolve the situation.

All of that said, I still use (and, in many ways _like_ ) Eclipse, but it is very frustrating to see these same problems drag on for years and years.

<comment><USER4> About updater/plugin mechanism in eclipse I just want to point out that is actually being fixed/improved greatly in upcoming Eclipse 4.3 (Kepler).

Eclipse update mechanism (called p2) actually at is core is very powerful and can actually find out/remedy the installation/update but it was just never exposed in the UI used by Eclipse IDE's.

Simply explained before Kepler p2 would just try and do exactly what you told it to "install these 2 plugins" and if its dependencies somehow conflicted it would give up ("could not complete operation").

In Kepler if this situation occurs p2 will try remedy the situation and try find out which plugins would need to be updated or maybe even downgraded to make it possible for you to install your plugins.

Thus we are some that actually care and do what we can to improve it - <USER2> feature got in with help from @prapicault

And for full disclosure i'm @maxandersen from the conversation OP linked to and lead of JBoss Developer Studio.

<comment><USER5> Kepler is also on track to fix the rest of the 4x perf bugs right? I dled M7 yesterday and noticed it still flashes full screen on Win764 when I move view groups around in a perspective.

I still love eclipse. I'm not ashamed to admit it. I've helped people sell businesses and go public with this damn platform creating millions in revenue along the way. When it's powering my livelihood and my employers happiness, I feel like something went right somewhere.

It's a shame the UX in 4.2 has been such a thing to focus on.

</comment></comment></comment><comment><USER6> I'm an eclipse user and love it. I don't understand why so many people hate it. However, I keep hearing, from very respectable developers, that intellij is a far better IDE. Even as I type this, I have an eclipse window open for main development and an intellij window open with some test code so I can get used to it, and see what the hype is about.

One thing is for sure, intellij (the dark theme) looks beautiful. Makes me wonder if eclipse should still be spending (wasting?) resources developing their own GUI toolkit.

<comment><USER7> My main problems with eclipse:

1\. Startup time. I just ran it twice on my machine. First time it took 20seconds before showing the splash, then 10 seconds after that (I'd never launched it before). Second time it took over 5 seconds to launch completely

2\. Cluttered UI. menu bar + status bar + project explorer + tasks + outline take up between 20 and 70% of the window depending on how big I have the window (god help me if i try to use it on my old laptop with its 1300x700 resolution)

3\. Difficulty of editing stand alone files. It's just a pain in the arse editing files that aren't in the project, and that you don't want to be in the project

For comparison, sublime text 2 launches in <1second on the same machine, and that includes restoring 100% of the previous working state, including unsaved changes to all files, even if the computer crashed and it wasn't closed down properly.

edit: I almost forgot

4\. It also takes 1-5seconds to __close __.

<comment><USER6> If you've used both, wouldn't you consider Eclipse and sublime text to be two very different tools? If you want to quickly edit some text, then Eclipse is overkil. If you are going to write programs for 8 hours a day, then Eclipse's benefits far out weigh its problems.

To me, it is the difference between a car and a bike. The two serve different purposes.

I don't mean to downplay problems with Eclipse. I remember that it was confusing when I first started using it. In the intervening years, I think the problem with eclipse is one of missed opportunities (or misplaced priorities).

<comment><USER8> This is a boat i bought into only in college. Once my professional career started, and i started viewing coding as an artisan craft, i realized that less tends to be more. I want an editor that i _never_ have to wait for. Every IDE i have tried has time where i pause waiting for it to try and do something.

Give me column editing, multiple cursors, and snippets and i will show you an editor i can code a system of any real size with.

<comment><USER9> "column editing, multiple cursors, and snippets"

It's more about smart refactoring, code completion, extensibility, project management, "open type hierarchy", "show usages", etc. You kind of need these things in big projects. Especially enterprise Java ones.

"Of any real size" - I dispute you've written one that counts.

<comment><USER8> Size in terms of software (and data) is always a perspective thing to be sure. I have worked on 2 separate millions of LoC codebases. Maybe its my ignorance, but i think there is more difference between a 10k Loc vs a 2mm Loc than 2mm vs 20mm.

I might be wrong, I've never worked in something an order of magnitude larger than a few million.

</comment></comment></comment></comment></comment></comment><comment><USER10> I really want to love Eclipse but I've never quite managed to. And the reason is quite simple: an IDE should enhance your productivity (otherwise we might as well just be editing files in vi / emacs). Eclipse doesn't always do that; in fact quite often it just gets in the way. It has so many silly nuances and bugs that I've now just given up on it completely (I now use LiteIDE for Go, plain text editors for C++ and the only reason I kept Eclipse on my workstation was for Android development but I'll be looking rather closely at IntelliJ for my next project).

It's a great pity because Eclipse -on paper- is the kind of IDE I should love. It's powerful, flexible and let's you tune it to behave the way you want it to behave. But getting it to bend to my will has on occasions in the past, taken more time than it would have taken just to bang the code out in a plain text editor and then compiling it myself via the command line.

</comment><comment><USER11> I don't "get" Eclipse personally.

Say I have some code in some directory. I want to edit that code. How do I do that?

The whole mess with workspaces is so confusing, even resetting all settings like your shortcuts! Why would anyone want their shortcuts reset when working on something new?

Also, ctrl+k for find next while every normal UI program uses F3 for that (and some gnome ones settled for ctrl+g): why? Seriously: why?

<comment><USER12> Is IDEA much different in this regard?

<comment><USER11> Honestly: yes. I find IDEA much easier to use. It has hundreds of shortcuts, but the basic ones match other editors.

<comment><USER13> That's hard to take, considering how long it took me to stop pressing ctrl-y for redo when I switched to intellij.

It's not just that it doesn't do what you expect, it's that what it does do is disasterous for someone who was expecting redo (because it destroys your forwards edit action tree).

<comment><USER14> IJ's mac settings are absurd. But you can change the settings to whatever you like. I've synched up my keystrokes from IntelliJ/RubyMine/AppCode and Sublime Text 2.

</comment></comment></comment></comment></comment><comment><USER15> Interesting and sad. IDEs are like editors, very personal things.

I wonder sometimes if it is an ego thing. How often is the whole direction of a particular featured changed completely because the author feels they have to be better than some other similar feature in a competitor environment? When someone says "Yeah, but even Visual Studio does X" does that send the developer into a flurry of "Oh yeah I can do X++?" or not?

I see similar things in the window systems, and find it sad that even in an all open source space there are such vitriolic attacks on KDE/Unity/Gnome from both inside and outside the various camps. Why is that? Its like sports franchises for nerds where there is some sort of built in comparison conflict, some need to be the "winner".

I completely agree that Eclipse could use a 'benevolent dictator' who not only was able to make decisions by fiat if necessary, but also was not caught up in defining their self worth by how Eclipse compared to other IDEs.

<comment><USER16>
      | vitriolic attacks on KDE/Unity/Gnome

I can understand the attacks on Unity. It was swapped to the default environment in Ubuntu before it was ready for primetime, and it represents Ubuntu's increasing distance from the rest of the Open Source (or at least Linux) community (see Mir vs. Wayland).

<comment><USER17> Ubuntu is not distancing themselves from anyone, least of all Open Source. Everything they are doing is open, and I'd argue in the spirit of the Linux community, as well (experiment, scratch your own it, prove your idea by building it, etc.)

I may not agree with all their technical & business decisions, but I do love that they are willing to experiment on the "big things" and provide a little competition in the space.

That said, their arguments against Wayland were way off. Which was a very big mistake. Though I tend to attribute that to not understanding Wayland well rather than malice.

<comment><USER16>
      | That said, their arguments against Wayland were
      | way off. Which was a very big mistake. Though I
      | tend to attribute that to not understanding Wayland
      | well rather than malice.

I don't necessarily think that it was malice (you can bad-mouth someone without malice), but it raises a lot of questions.

\- Why was Ubuntu so misinformed? Why didn't they discuss things with the Wayland devs?

\- Why haven't they admitted their mistake in their reasons[1]?

\- Why is Ubuntu still going forward with Mir? Why not just use Wayland if the 'shortcomings' of Wayland that drove them to create Mir don't exist?

[1] Maybe they have, but I haven't come across it.

<comment><USER18> > Why is Ubuntu still going forward with Mir? Why not just use Wayland if the > 'shortcomings' of Wayland that drove them to create Mir don't exist?

Maybe the switching cost of moving to Wayland outweighs the remaining cost of completing Mir.

Whatever the reason, I'm not complaining about the result, since I actually think it's healthy for multiple competitors to exist in the same space, especially for something as widely used as a windowing system:

(1) They can cross-pollinate ideas

(2) Competition puts pressure on sub-par projects to innovate.

Regarding (2), IMHO it's much easier for your project to innovate when a competitor who's on your radar pulls ahead in mindshare or market share gives you the kick in the pants you need when you're merely starting down the road to stagnation. If you're a near-monopoly on your market niche (post-X FOSS windowing systems in this case), you can get completely ossified and not realize it until everyone leaves as soon as your monopoly disappears due to new competitors (funded by the fraction of your own dissatisfied customers who have deep pockets) -- a position which is much more difficult to recover from.

</comment></comment></comment></comment></comment><comment><USER19> Eclipse user here. I use it on a daily basis and there are things I really like about it:

\- debugger (and everything about it, like hot code swapping, etc)

\- run configurations (setting multiple run configs for one project happens!)

\- views: I mean i use only "java" and "debug". they could be faster but i don't mind, you don't switch them often.

\- workspaces are ok too, when it comes to organizing your projects in groups. It's ok, really.

What i hate:

\- performance. for example the typical problem (i think it is a bug) when you switch between java and xml editor and you wait like 5 sec.

\- workspaces that resets everything. wokspaces should group projects, not the user preferences. Dear eclipse, it's still the same person opening them!

\- why the hell i have to spend 2h configuring eclipse after downloading a new version? does not svn/git/maven a default requirement for java development today?

\- search. i never remember which tab out of 6 and what setting will allow me to just do grep in the project folder.

\- the toolbar - please remove all this crap. I will only use those 3 run buttons and switch perspective anyway.

<comment><USER20> > \- search. i never remember which tab out of 6 and what setting will allow > me to just do grep in the project folder.

Just disable all the other tabs by pressing "Customize..." in the search dialog and only selecting "File Search".

Or in the upcoming Eclipse 4.3, there is now an option to always show the last used page, see here: [http://download.eclipse.org/eclipse/downloads/drops4/S-4.3M6...](http://download.eclipse.org/eclipse/downloads/drops4/S-4.3M6-201303141330/news/#use- last-page-option)

</comment></comment><comment><USER21> My gripe with Eclipse is: workspaces, perspectives, SCM plugins and the worst is Build Path.

Some plugins like the Maven one is something that I can't stand. It's confusing, lacks things like running a clean without needing to create a bogus "run" configuration.

EDIT: I forgot to say that I hate editing HTML code in Eclipse. It's slow, autocompletion is hit and miss and package explorer doesn't know how to group web content.

EDIT2: How could I forget about the pesky hidden directories full of stuff in it? Nightmarish!

</comment><comment><USER22> I think it's funny that they don't seem to think of Eclipse as an IDE even though that is probably where it is most used and most valuable, esp. when you consider things like Zend Studio, Flex Builder, and Aptana Studio (which powers Appcelerator now). Yet, the Eclipse team doesn't seem to see itself as an IDE but more of a toolkit. That is completely stupid. Eclipse deserves to have its core market slowly taken away by IntelliJ. Also JetBrain's various flavors of its IDE are far superior to Eclipse.

<comment><USER23> All the rebranded Eclipses remind me of those concert tours that rename themselves for every local market to make it look like the rinky dink local radio station organized a massive 3 stage music festival. "Oh wow, you guys wrote your own IDE??! That's gr... Oh."

</comment></comment><comment><USER24> I think it is great to see some reflection happening in the eclipse community. It's a hallmark of a vibrant open source community.

The next question is, what will come out of the reflection?

<comment><USER25> > The next question is, what will come out of the reflection?

Probably a better understanding of the class hierarchy.

<comment><USER24> Right. But such understanding will be slow at first, only performing better after years of optimization.

</comment></comment></comment><comment><USER26> I tried getting into IntelliJ and, though look and feel wise and it is superior to Eclipse in every way, there is just so much lacking that I am used to in Eclipse. Maybe I am just not used to how IntelliJ does things...

I use IntelliJ primarily for Play Framework development. I've installed plugins for almost all other languages (node.js, python, mustache etc) and use IntelliJ for that but for non-Play Java, I tend to go back to Eclipse.

Some of that may be the 10 years or so now that I've spent with Eclipse.

<comment><USER27> In terms of multilanguage editing, refactoring, and just basic Java handling, IntelliJ blows away Eclipse. I've been switching back and forth between the two for years because of internal Google bias towards Eclipse, but I find Eclipse to be very lacking. Just simple stuff like Find Usages or navigating type hierarchies is done far better in IJ.

<comment><USER26> What do you find that is different? I've never had a problem with type hierarchies or Find Usages in Java (not counting JSP - JSP support is terrible).

For JUST Java, IMO Eclipse wins for me.

<comment><USER27> Well, if there are two inner interfaces called "Callback" like

public class Foo { interface Callback {} }

public class Bar { interface Callback {} }

The last time I asked Eclipse to find all implementors of Bar.Callback, it actually returned ALL instances of any interface called "Callback" as if it is just doing a String search and not actually using type hierarchy information in the search.

Eclipse auto-assist often popups up tons of bullshit that is not relevent, like asking to import say, a class called 'Util', Eclipse would often show all kinds of entries from private com.sun.* classes in the JRE/JDK first. It's as irritating and choosing what Country you're form in a Web Form and it has Afghanistan first and United States at the end, even though it should know from locale or Geo-IP how to sort better.

Maybe this has been fixed recently, but it has always worked the way it should in IntelliJ. IntelliJ also handles Maven better. And it handles integration between Java and other DSL languages better. IntelliJ's PSI model of code is far richer than JDT.

</comment></comment></comment></comment><comment><USER28> Love that we're seeing more choices in terms of Android development. This can only be a good thing!

I was very frustrated the last time I tried going to IDEA and I wrote about it on my blog ([http://www.fisharefriends.us/wordpress/2013/01/27/intellij-i...](http://www.fisharefriends.us/wordpress/2013/01/27/intellij- idea-from-a-long-time-eclipse-user/)). My observations:

. It seems to take a long time to "warm up" before IDEA runs as smooth as Eclipse.

. Eclipse is stupid about detecting when you changed a file from outside of Eclipse; IDEA is not. Eclipse is very annoying when you use Emacs/vi/OxygenXML as an additional editor.

. "JetBrains :: IntelliJ IDEA" is a stupid, stupid name.

. Emacs key bindings in Eclipse is better, alt-w and ctrl-y "just works".

. Maven projects is poorly integrated into Eclipse's concept of "projects".

. I miss the native "choose file dialog" you'd get with Eclipse.

. I think the new version has incremental compile like Eclipse, but it doesn't seem to work by default.

. The Maven daemon in IDEA used around 300MB of memory and continually consumed CPU resources.

. The keyboard stopped working for some reason.

My experience was with the community edition, perhaps I should give the professional edition a try. But at $200 and having had such a poor experience with the community edition, it is a hard sell.

<comment><USER20> > . Eclipse is stupid about detecting when you changed a file from outside of > Eclipse; IDEA is not. Eclipse is very annoying when you use > Emacs/vi/OxygenXML as an additional editor.

Go to Preferences > General > Workspace. Enable the "Refresh" option(s) there.

</comment></comment><comment><USER29> The main problem I have with Eclipse and heavy IDE's in general, is their build systems rarely play well with the standard, standalone build systems such as maven or sbt, which are already necessary and sufficient with the latter even doing incremental building already. These standard build systems are a requirement because they track external dependencies and not everyone in the group can be assumed to use the same IDE anyway.

Sure there are plugins to "integrate" these build systems, but I've wasted too many hours futzing around with keeping the external build system in sync with their Eclipse plugins (many times having to recreate projects altogether to get past some sorts of corruption). Anybody who's had to pollute their poms with Eclipse specific config after a lot of research because of the "Plugin execution not covered by lifecycle configuration" problem will know what I mean. Don't get me started with sbt integration.

</comment><comment><USER30> Android did not switch to IntelliJ. Google has given you the option of also using IntelliJ. Many people who do not like eclipse requested IntelliJ support and Google listened.

This has been brought up a couple times in the sessions, they didn't do a great job of explaining in the keynote. If you want to hear the official story, watch a recording of the Android Fireside Chat session.

<comment><USER31> Actually I think you are wrong. Android Studio is not 100% complete yet, and probably won't be for some time - but when it is done, Eclipse will slowly go away - Google style. Meaning it will languish and its users will bitch and moan until eventually it is canceled.

</comment></comment><comment><USER32> They're right. The UX sucks.

This is big problem in most open source software projects :\

</comment><comment><USER33> I am probably a weird case: The major purposes of eclipse for me is purely CDT, and for this it does 2 things well

1\. Graphical interface for compilation errors

2\. Graphical interface for debugging under gdb.

I haven't seen anything like it (Visual Studio is nice, but only runs on MS). Would love to hear if anyone uses different IDE's for c++ devel that works cross platform.

<comment><USER34> CDT?

<comment><USER35> C/C++ Development Tooling <http://www.eclipse.org/cdt/>

</comment></comment></comment><comment><USER36> If any Eclipse developer(s) is here, please make the following feature happen without a plugin. (it doesn't make sense)

Q: In Eclipse, how to reveal current file in filesystem

[http://stackoverflow.com/questions/1161240/in-eclipse- reveal...](http://stackoverflow.com/questions/1161240/in-eclipse-reveal- current-file-in-filesystem)

</comment><comment><USER37> JetBrains tools are the only software I happily pay to use. Their support, though not nearly perfect, is by far better than any other software I paid for.

I hope Google and JB can work together to do a nice job and provide a better development experience for Android.

</comment><comment><USER36> I still remember some people joke around that the official Android coding style [1] requires "Non-public, non-static field names start with m." is because the core developers don't use IDE.

Anyway, Eclipse for Java development is definitely okay IMHO; but for Android development, there are always nasty bugs like this [2]

[1] <http://source.android.com/source/code-style.html>

[2] <https://code.google.com/p/android/issues/detail?id=27940>

<comment><USER38> I really wish hungarian notation would die already. Might be one of the worst trends to ever catch on.

</comment></comment><comment><USER39> I'm wondering how long until someone ports the Energia/Arduino IDE to work with Androids.

Energia is basically a port of the Arduino IDE to work with TI Launchpads, including the TI Stellaris, which has an ARM Cortex-M4F. Before Energia was ported, Eclipse was the IDE to use for TI Launchpads.

Working with this IDE has been pure joy. Adding libraries is as simple as copying the library into the folder and restarting the IDE. Simply by replacing Eclipse with Energia has made hacking with TI launchpad devices more fun.

</comment><comment><USER40> It's interesting that the Eclipse devs were not aware until now that users actually don't like using it.

</comment><comment><USER41> Those tweets were getting shorter and shorter, mainly because they kept adding more and more people.

</comment><comment><USER42> Have still to try studio, but my experience with IDE is that I'm going to regret it soon down the road. They always limit or screw me in a way that I regret not starting with vim and a set of tools instead of packaging it depending on some complex/closed codebase

</comment><comment><USER43> i'm actually sad and happy at the same time. eclim just got usable for me for android development. i don't think you can run a headless intellij.

but i still consider intellij the better standalone IDE.

</comment><comment><USER44> I love IntelliJ but I am so pissed that the only Vim plugin does not support key mapping. Which means to me it's nearly unusable.

<comment><USER45> Fix it! <https://github.com/JetBrains/ideavim> I'm sure they would happily take a pull request

<comment><USER44> I was actually thinking about it but I don't have much time (as everyone) and I really do not know where to start from... I will still try give it a look! (truth is I am afraid my code is not good enough)

</comment></comment></comment><comment><USER46> This might sound like small potatoes, but a big gripe of mine about Eclipse is the code completion. Almost every other decent IDE (and even advanced enough editors) doesn't wait for the dot to offer you code completion.

I never understood why that was never offered at least as an option.</comment>


Backbone everywhere - <AUTHOR> http://www.idiotz.nl/2011/08/28/backbone-everywhere/

<comment><USER2> Really waiting for a frameworks that makes sharing server/client models easy. Using backbone on the server feels like such a hack, as it was designed for the client. You end up being forced to use jsdom to create a server side dom so that the jquery heavy backbones views can work.

<comment><USER3> Very early days but I came across a project called Racer, which is aiming to do just that: [http://blog.nateps.com/announcing-racer-experimental- realtim...](http://blog.nateps.com/announcing-racer-experimental-realtime- model)</comment></comment>


Architecting a Machine Learning System for Risk - <AUTHOR> http://nerds.airbnb.com/architecting-machine-learning-system-risk/ <comment><USER2> I'm actually bit surprised at people using PMML and this architecture. Clearly, here the attempt is to isolate model generation and runtime prediction but doing this also confines you to least common denominator. This means you can't generate any model that Opescoring can't handle. If you think about it, there is no real need for Opescoring. You can wip up REST service very easily that wraps sk-learn predictor and I would bet it's actually much easier to do than writing PMML exporters. Then you can use all the goodness of top of the line models white your service interface still remains same. The architecture that enforces you to use lowest common denominator just for abstraction purposes is a poor design, IMO.

<comment><USER3> PMML allows you to use models generated by a variety of tools and systems, including external vendors. While you are right that making it core does create some constraints, it also allows them to easily replace portions of the model building pipeline with relative ease.

Yes, PMML isn't perfect (being kind), but it continues to be extended and is the one shared lingua franca we have across model creation systems, short of (sigh) SAS code and "recode the model in generic C", both of which I see too often.

I suspect in the future we'll see "standard" architecture with pipelines with multiple parallel feeds and runtime engines into ensembles, each of which allows various model types in "native" format (sklearn and other pythonics, R, java, etc.) which would be interesting, instead of having to cram all into PMML. Just a thought.

<comment><USER4> Agreed, this describes our reasoning pretty well. Additionally, as we mentioned towards the end of the post, the incremental benefit of a model not supported by PMML is unlikely to be more significant than the incremental improvements we see from investing in improving our features and ground truth. As such, our lowest common denominator isn't actually the model, but rather the features and data pipelines. The suggestion sytelus makes is a perfectly good way of doing it though, and we will likely change our approach when we find that our models do become the lowest common denominator, or have a higher relative ROI for the time we invest in improving them.

</comment></comment></comment><comment><USER5> I've just built a system very much like this for a large customer. Extremely interesting and I learned a lot while doing it. Funny to see companies operating at a similar scale running into similar problems and solving them in roughly similar ways.

</comment><comment><USER6> Looks like a cool project... but I hope the plan to open source their library to export Scikit-Learn classifiers to PMML! This would be a great way for them to give back to the open source community.

<comment><USER4> We'd love to do this. As you can imagine, however, it's a very time consuming task, and there are a lot of competing priorities (including other projects we've open sourced) and we therefore can't make any guarantees about if/when we'll be able to do it.

</comment></comment><comment><USER7> Nice writeup. It seems like a supervised learning approach to fraud detection. I have a question: Where does the is_fraud variable come? Is it done by humans?

<comment><USER8> (I don't work for Airbnb, but work in a similar space)

Yes, this variable is usually set after the fact. For instance, a given transaction may have led to a chargeback, or may be done by a known fraudster. These models are usually trained on historical data, so we can know with some certainty which transactions are fraud.

It could be that a transaction was fraudulent but has not yet led to a chargeback (maybe the real cardholder hasn't yet seen their statement?), so there's still some uncertainty, but hopefully that approaches a minimum after some time passes.

<comment><USER9> what software are you using?

<comment><USER8> I don't want to hijack this thread too much from the original post, but we use some of the same software as Airbnb (scikit-learn, randomforest models, etc.) as well as some stuff developed in-house. Credit card fraud has been one of our biggest issues, and we've developed some pretty robust systems to fight it. Contact me privately and I'd be happy to talk about it -- this stuff is what I do for a living.

</comment></comment></comment></comment><comment><USER10> Don't mean to hijack the comment thread, but can anyone recommend any good videos that introduce machine learning or courses? I studied computer science, but was not able to take any classes on the subject. I found the Stanford one, anyone have experience with it?

[http://online.stanford.edu/course/machine- learning](http://online.stanford.edu/course/machine-learning)

<comment><USER11> I've heard nothing but good reviews of the online version of the class. I took the class at Stanford (and actually worked on the system mentioned in the article) and I found its content to be useful. I believe that the online version contains less theory but this is not necessarily a bad thing if all you want is an introduction.

I've also found reading papers to be illuminating: often the first article about a given classifier is fairly well written and accessible if you have a strong background in math.

This is also a useful thing to keep in mind: [http://scikit- learn.org/stable/tutorial/machine_learning_map...](http://scikit- learn.org/stable/tutorial/machine_learning_map/)

<comment><USER12> This looks clipped in my browser, but you can see the original diagram here: [http://peekaboo-vision.blogspot.com/2013/01/machine- learning...](http://peekaboo-vision.blogspot.com/2013/01/machine-learning- cheat-sheet-for-scikit.html)

</comment></comment></comment><comment><USER13> I was thinking about the sorts of fraud categories AirBnB likely experiences. Most fraudsters want cash or cash equivalents, and the use of lodging on a particular night is nearly as illiquid as stolen fine art. So, those seeking stuff to resell will choose to defraud one of the zillion online marketers who ship stuff to doorsteps. A buyer who actually used the space he reserved could initiate a chargeback later claiming that the service promised via AirBnB wasn't provided -- couldn't access apartment, wasn't as described, etc. However, space providers likely will cooperate with AirBnB and provide evidence in their defense. Better to attempt a chargeback elsewhere if one is short on money. It seems that using AirBnB as a platform for crimes between buyer and space provider is possible, and there certainly has been at least one heavily publicized case, but we would hear a lot more about these events if they were happening much.

So, what's left? Collusion between buyer and space provider -- in all likelihood, they are one in the same, or identities have been stolen. For example, I list my condo on AirBnB for $100/night. Someone books it for the weekend, and then doesn't show up. AirBnB owes me $200 -- after all, I gave up other options to profit from its use. An honest buyer pays up. But, maybe the buyer is dishonest -- he used a stolen credit card, etc. In this case, AirBnB eats the loss and pays me as the space provider. Now, wouldn't it be convenient if I was also the buyer? Cash from stolen credit cards, funneled through AirBnB (much akin to the way online poker sites were used to transfer stolen money via bad heads-up play). This would work until AirBnB noticed that my listing seems to have a suspicious propensity to attract fraudulent buyers. Then, they'll shut me down. So, I'll pop-up elsewhere. After all, no need to actually have a space because no one I accept will ever show up!

I bet the usage patterns of the party/parties involved in this fraud are drastically different than those of legitimate market participants. Someone with a fraudulent listing could out himself by rejecting a bunch of legitimate AirBnB buyers, and this behavior would stand-out as it's the opposite of the behavior expected of an honest seller. So, he must protect against this risk by making his listing unappealing (high price, bad photos/description, unpopular location, etc.). The behavior of users browsing AirBnB when viewing this property could identify its relative undesirability (few clicks, etc.), and price outliers could be identified by comparing similar offerings by date/location/type. The click stream of the "buyer" likely is most revealing. Someone selecting an unappealing property without doing much comparison shopping likely isn't a legit buyer.

What other stuff might predict fraud? Vague descriptions might indicate a fraudulent listing. Most space providers love to tell buyers what's special about their offering. Could some scoring of a listing's prose prove a strong predictor? I've never listed with AirBnB. What do they do to verify listings? As a buyer, they verified my identity. Could this serve multiple purposes? Certainly, I'd feel better listing my guest room if I know that AirBnB will know the identity of the guy who rented the room and then stabbed me at 3AM. But, in addition, does identifying market participants in strong ways help keep fraudsters from repeating their crimes by setting up multiple accounts? Obviously, newer market participants are more risky than established ones, especially those who have interacted with known legit, long-time users. The social graph comes to the rescue here. Even astroturfing ought to show up as a small, disconnected graph unless legit users' identities are stolen.

Of course, this comment is all just conjecture. Obviously, AirBnB can't tell the public about specific fraud methods or how they identify suspicious activity. However, I like the concreteness of considering actual fraud scenarios, so I decided to put forth some ideas for discussion.

</comment><comment><USER14> I didn't quite understand the need for openscoring and pmml. If it's just a question of using a sklearn model to predict an outcome, why not just build it into a simple json-rpc with Tornado, Gevent or whatever the rage is, currently?

<comment><USER15> As I'm working on a very similar problem right now, the difficulty is that to save the fitted sklearn model you have to pickle it (pickled decent size random forest are several megabytes). Then, at classification time, you have to import pickle, sklearn (and numpy), depickle the object, run the example through the classifier and extract the output. Perhaps the Openscoring model is more efficient?

<comment><USER16> You can use `all_model_filenames = joblib.dump(model, filename)` after fit on your dev enviroment. joblib will store each numpy array in the model datastructure as an independent file and `all_model_filenames[0] == filename` refers to the file holding the main pickle structure.

Then on your prediction servers, ensure that you have a copy of `all_model_filenames` in the same folder. You can then load the model with `model = joblib.load(filenames[0], mmap_mode='r')`. This will make it possible to use shared memory (memory mapping) for the model parameters of a large random forest so that all the Gunicorn, Celery or Storm worker processes running on the same server will use the same memory pages, making it a very efficient way to deploy large models on RAM constrained servers.

You can even use docker to ship the model as part of a container and treat the model as binary software configuration.

</comment></comment></comment><comment><USER17> I was wondering if their solution is a home grown version of SiftScience?

<comment><USER18> Actually SiftScience lists AirBnb as one of their customers on their site, so I'd say it is very likely built on top of SiftScience.

<comment><USER17> Great catch. I had thought about doing the same.

</comment></comment></comment><comment><USER19> does anyone know if Java can port a gradient boosting model from R

</comment><comment><USER20> OMG I love that cartoon. "Machine learning" is such a funny phrase if you think about it.

</comment><comment><USER21> Pet peeve: the verb is _designing_.

Besides that, let me rephrase here.

 _At <west coast startup, essentially a copy of earlier successful European businesses such as HouseTrip, but with access to stupid amounts of US capital and therefore more profitable>, we <make superfluous, keyword-laden, unverifiable claim about ourselves in the future>. We <continue to integrate feel-good community pronouns>. We <here discuss something only tangential to our core business and assert that we have allocated at least two people to this area>. We <have nothing better to do than write it up, because quite frankly, there's nothing more pressing for us to work on in an already automated business of relative simplicity>_.

OK, so that's a bit harsh, but there's some points toward reality in there. Sorry, as someone who used to run a complex travel industry business (3200+ hotel contracts... all of them in Chinese, all business by digital fax (no convenience here!), constant rate changes, in 6 human languages and multiple currencies with a real time call center) and who co-pitched for VC with HouseTrip's management in London in 2009, I just have very little respect for AirBNB.

<comment><USER22> Care to do a write up of what you're doing as far as fraud prevention goes? However you feel about Airbnb, this was an interesting post. It's not ground breaking or earth shattering, but it shows the tech stack that a large company is using to solve a real problem, and that's useful. They even did a fairly good job of explaining the why.

<comment><USER21> _Care to do a write up of what you 're doing as far as fraud prevention goes?_

Sure, but only high level. I would hazard a guess that fraud prevention is a lot more complex for us at [https://www.kraken.com/](https://www.kraken.com/) ... dealing with many cryptographic currencies and conventional currencies spread across probably over a hundred legal jurisdictions is not easy. We likely have to consider far more factors than these guys. We have recently added two more quants from programs highly regarded in the conventional finance industry to our team, plus we have over seven figures of investment in legal and training programs in the area. We also use _R_.

Basically, it's inputs (behavior), processing (metric extraction, risk model), output (boolean choices, statistical cluster membership, etc.)... where a series of such outputs may feed in to a heirarchy of scores for different elements within a system. Some applications may be real time, others after- the-fact.

At a high level, which is mostly where my involvement is in hiring people, fraud prevention is not dissimilar to spam or intrusion detection: you can basically use a combined, constantly tweaked set of inputs to a Bayesian-style scoring algorithm. Inputs include both static rules and statistical anomaly detection.

[http://en.wikipedia.org/wiki/Bayesian_probability](http://en.wikipedia.org/wiki/Bayesian_probability)</comment></comment></comment>


Ask HN: Interview Saturdays instead of interviews during work hours? - <AUTHOR> As I know people and as I have myself interviewed in places, it becomes pretty clear that the process itself is pretty awful. You have to take off from work and then meet the prospective employers - there is the obvious aspect of lying to your current employer as to where you are really going. Then the whole process becomes even more difficult when you want to evaluate multiple opportunities. Most startups want you to meet the team, and when I say team I mean the actual team, the CEO, the office manager, the spouse, the resident dog and the delivery man. The whole thing becomes this massive time sink which, although completely necessary, starts intruding on the day to day operations of the current job. Some folks dont have the luxury of quitting before looking and I was wondering if the concept of Interview Saturdays make sense? The prospective employers and interviewers can get through their day jobs without constant interview interruptions. The prospective interviewee doesnt have to lie or get behind on work. Everyone wins! Except Saturday, but then if you are in a hot startup, then tough shit. The time spent interviewing folks during the week will carry over to your nights and weekend anyway. This Interview Saturday will help you figure out people who are willing to make the sacrifices (Saturday gone) and will lead to less drawn out interview loops - also more efficient tracking of time spent on interviews if you silo the time.<p>What do you think? <comment><USER2> _" there is the obvious aspect of lying to your current employer as to where you are really going"_

Why is it any of your employer's business where you're going? You presumably have some number of vacation days that you're entitled to as part of your compensation package, so use them. If your company insists on having an accounting of what you do during your free time, they're behaving like unreasonable jerks, so don't feel guilty about lying to them.

 _" Everyone wins!"_

I'm sure that not all companies have dysfunctional cultures that include routinely working on weekends. (Apparently the place where you work now doesn't, since you're free on weekends.) By demanding that they interview you on Saturday, you're asking all the employees who would be interviewing you (which, as you noted, might be the entire company) to make the sacrifice of coming in to the office on Saturday for your convenience. This makes you look like a high-maintenance prima donna, and also gives the company an incentive to ignore you and choose a candidate who is available during the week.

</comment><comment><USER3> Off hours interviews are not uncommon in other industries. In fact, when it comes to headhunters, the better ones I worked with when job hunting generally scheduled all their contact outside of office hours and initial interviews with their clients with the fact that I had a job in mind.

The full team followup on Fridays seems to also be a technique that's available. It lets candidates travel out on Saturday or Saturday afternoon if the window for meeting staff needs to be larger.

Scheduling interviews during normal work hours as policy suggests that there's a either just a need for a warm body and not careful selection driving the process or there's a not much respect for the idea that people have lives outside of working for them. Seriously, what kind of person thinks getting interviewed is how people want to use their vacation time? The clueless and the callous - of course.

</comment><comment><USER4> I think Interview "Off-Hours" might make more sense. That is, interviewing after 5pm. I've stayed longer to help interview a perspective candidate. But if I was asked to come in on a Saturday just to meet a candidate, that's a whole different beast.

</comment><comment><USER5> To distil - the new employer and their whole team must accommodate you as an individual so you do not piss off your old employer?

Doesn't sound like the interview will go too well if the whole team is expected to work on a Saturday just for you.

Some team members have children, family commitments or simply adhere to the idea that free time is beneficial for employees.

Personally, I would accommodate _one_ candidate interview on a Saturday if they asked for a specific reason; if it was suggested as a regular occurrence I would question his/her judgement.

</comment><comment><USER6> Like everything else, it's a question of supply and demand. If they want you badly enough, they'll interview you on whatever terms you demand. If you want them more than they want you, they'll set the terms and you'll have to figure it out.

FWIW, I think the hiring/recruiting interface is a pretty good indicator of broader attributes. When I'm recruiting, I definitely notice if a candidate is responsive/eager in the scheduling process and has his or her act together on the day (shows up on time, leaves enough time to park, etc). And same when I interview with companies. I've rarely found false signals either way.

</comment><comment><USER7> I like it... I've done Saturday interviews. As well as interviews on a private plane, in Aspen, at bar in palm springs, at a starbucks next to OAK, and in a founder's backyard. So Saturday isn't too bad...</comment>


Local maxima and the perils of data-driven design - <AUTHOR> http://www.90percentofeverything.com/2011/01/06/local-maxima-and-the-perils-of-data-driven-design/ <comment><USER2> There does not exist a single methodology that can always avoid local minima/maxima, short of somehow brute forcing the entire design space. (If evaluating the design space and locating the best design is somehow not an NP- hard problem, it is close enough for human purposes.) Accusing one methodology of producing local minima/maxima is therefore a noop. One must demonstrate that it is somehow more _prone_ to such a result.

Good luck with that. I would despair of even getting the definitions correct, let alone demonstrating a result with even minimal rigor.

My intuitive feel is that if you feel like testing is getting you stuck in a local optima, this is more likely to be because your testing is actually giving enough sense of how your design fares that you can actually feel you are in a local optima, rather than the usual situation where you are just clueless, happy about it, and not even in a local optima, let alone the global one. You can't blame the equipment that allows you to detect that your are in a local optimum for getting you stuck in it; that's textbook observation bias.

<comment><USER3> There doesn't exist a single _data-driven_ methodology that can avoid local maxima without brute force the entire design space. Conceptualizing things and being creative is what avoids local maxima.

<comment><USER2> Nope. Being a human is not the magic algorithm for finding global optima. We aren't anywhere near that smart. There's no methodology for finding the global maxima, period.

<comment><AUTHOR> No-one is talking about finding the global maxima. We are simply discussing ways to avoid local maxima. "Better" not "Best"...

</comment></comment></comment></comment><comment><USER4> My counter-point (written earlier): In defense of A/B testing [http://www.smashingmagazine.com/2010/08/26/in-defense-of- a-b...](http://www.smashingmagazine.com/2010/08/26/in-defense-of-a-b-testing/)

<comment><USER5> I don't think the facebook case studies in the article are _against_ AB testing per se, but more against the fact that AB testing is often used to focus on minutae. It seems to be a psychological temptation to use it to focus on minutiae, rather than something that's inherent in the research method itself.</comment></comment>


Skype protocol dumps - <AUTHOR> http://skype-open-source2.blogspot.com/2016/06/skype-protocol-dumps.html <comment><AUTHOR> Hello guys,

this is some thougts and result of my investigation into skype protocol and how it is worked.

First of all, what we need to reconstruct network protocol?

We need plain text data which transfered from host A to host B, right? Right. So, there it is. There is a text files with plain text data which sended from one skype client to another skype client. Yes, it have much crypto on lower levels. For now, just dont think about it. It was researched one time and after we just encapsulate it in function or class and dont care about it. Text files of skype send message sequence:

[http://dev.epycslib.ru/dumps/parse_proto.zip](http://dev.epycslib.ru/dumps/parse_proto.zip)

Okay, we got plain text data transfered. What we need next? We need understand, that all this data means. Where is destination username, where is time and where is message. And how protocol care it all for working good. How to do this? Best way, its rewrite it on paper in your own blocks and visual representation to understand internal structure of protocol. Make visual blocks and etc. Like in UML. So, there it is. My scans of hand written paper with skype protocol send message sequences:

[http://dev.epycslib.ru/dumps/skypeproto.zip](http://dev.epycslib.ru/dumps/skypeproto.zip)

More info at:

[http://skype-open-source2.blogspot.ru/2016/06/skype- protocol...](http://skype-open-source2.blogspot.ru/2016/06/skype-protocol- dumps.html)

<comment><AUTHOR> Working version of first Skype Network compatible client (Epycs) which creation was based on this protocol dumps will be released 1 september 2016.

<comment><USER2> Speaking as a layman here, but would it be trivial for actual-Skype to be tweaked in such a way as to break a compatible client that only currently works? Reminds me of all the third party clients or plugins for (insert service here) that work great...until something changes and it stops working.

<comment><USER3> Yes, it'd be trivial. In fact, this has pretty much happened with every single IM protocol at some point that was subject to reverse-engineering efforts. Here's some history from the early 2000s [1]. Libpurple changelogs are also enlightening.

[1] [https://en.wikipedia.org/wiki/Trillian_(software)#History](https://en.wikipedia.org/wiki/Trillian_\(software\)#History)

<comment><USER4> Trivial to break, yes, but also trivial to fix usually. Once the initial RE effort is done, minor obfuscations to the protocol are typically not major harms.

</comment></comment></comment></comment></comment><comment><USER5> The protocol of which version of skype client are you reversing? I guess it is NOT the same protocol web.skype.com uses?

BTW, ladies and gentlemen, I'd like to mention that I daily chat with skype contacts by means of XMPP clients and gateway software "spectrum" [http://spectrum.im/](http://spectrum.im/) (which works by web protocol of skype). All 1-to-1 chats work, recent groupchats work. File transfers from skype contacts don't work, and also calls. But day-to-day it is enough for me. I gain a lot of flexibility by this.

<comment><AUTHOR> Protocol from Skype Desktop version.</comment></comment>


Einstein's 1905 Paper on the Photoelectric Effect - <AUTHOR> https://medium.com/cantors-paradise/einsteins-1905-paper-on-the-photoelectric-effect-d258739ef8d1 <comment><USER2> Link to English translation of Einstein's actual paper: [http://www.physik.fu- berlin.de/~kleinert/files/eins_lq.pdf](http://www.physik.fu- berlin.de/~kleinert/files/eins_lq.pdf)</comment>


This to That (Glue Advice) - <AUTHOR> http://www.thistothat.com/

<comment><USER2> I like the idea here, and would love something like Wolfram Alpha with this kind of practical materials information. And questions about packaging and shipping. "How do I ship canned soda?".

<comment><AUTHOR> Do you often have questions about that? I figured, unless it was flammable or fire arms, it usually goes in a box?

<comment><USER2> I meant pricing comparisons -- is it worth shipping item as media mail, fedex ground, ups, usps parcel post or priority, etc. And packaging advice for different kinds of things (like, standard sized boxes).</comment></comment></comment>


Clientsy: A simple and beautiful area for you and your clients - <AUTHOR> http://clientsy.nl

<comment><USER2> Sweet. Clean design, and I like that its quite clear cut and focused in terms of what it does unlike, say, Basecamp.

Have you spoken to the clients of your potential clients, i.e the clients of the agencies and designers who it seems you're targeting, at least based on the opening text?

I can definitely see some designers and agencies buying into this, however I wonder how stubbornly set in their ways some of their own clients are.

</comment><comment><USER3> Looks nice</comment>


19.5% of HTTPS sites trigger browser warnings because of SHA-1 certificates - <AUTHOR> http://www.elie.net/blog/security/19.5-percent-of-https-sites-trigger-browser-warning-as-they-use-sha-1-signed-certificates

<comment><AUTHOR> On a related note, I submitted this article with an HTTPS URL and HN parsed it as a blank page (but http obviously worked). You can see that blank page here: [https://news.ycombinator.com/item?id=8982899](https://news.ycombinator.com/item?id=8982899)

My guess is that HN doesn't support the SNI extension to HTTPS, so it's unable to parse certain sites. Y'all should fix this.

I should note that (assuming the SNI theory is correct) that HN isn't alone. My understanding is that Twitter didn't support SNI until last month and some other major sites still don't.

<comment><USER2> "Y'all should fix this" \- there is a 'support' link in the footer to email the HN administrators.</comment></comment>


What's up with the number 163? - <AUTHOR> https://plus.google.com/101584889282878921052/posts/7u73y5FzEZY?pid=5963315119367216690&oid=101584889282878921052 <comment><USER2> I'll be honest -- I don't find this sort of thing terribly interesting. I feel when reading this as if I'm being pressured to ooh and ahh, but to me the magic is just not there.

It also strikes me as backwards that people are referring to these incredibly deep algebraic connections as explanations. They're not explanations -- they're just connections. I don't need any of that stuff to prove these identities. I just need a pocket calculator. The algebraic connections might be useful for discovering these identities in the first place, but are not necessary for proving anything here. For any given arithmetic identity I'm sure you can find some remarkably obtuse algebraic connection which illustrates the same identity. These connections are interesting only insofar as the _other stuff_ is interesting. The Monster group is interesting, but this identity itself is not particularly amazing to me.

Finally, is .999 really "extremely" close to an integer? In engineering, maybe. But in pure mathematics that seems "extremely" far from an integer.

<comment><USER3> > I don't need any of that stuff to prove these identities. I just need a > pocket calculator.

But numerical results don't establish identities, and identities don't depend on numerical results. After all, with only numerical results for guidance, all sorts of illusory outcomes appear to mean more (or less) than they do. For example, this integral is equal to Pi:

[http://i.imgur.com/tJS1ZaA.png](http://i.imgur.com/tJS1ZaA.png)

If one approximates the above integral by summing a bunch of numerical results on the interval between -1 and 1, one can approximate Pi, but the above integral equals Pi -- it's an identity. It has all sorts of advantages over a numerical result, not least of which is the fact that one cannot express Pi using a finite number of decimal places.

The article's examples weren't really identities, they were just coincidences (for the most part).

> Finally, is .999 really "extremely" close to an integer?

This is why mathematics has it all over words for expressing certain ideas.

<comment><USER4> _For example, this integral is equal to Pi:

[http://i.imgur.com/tJS1ZaA.png](http://i.imgur.com/tJS1ZaA.png) _

It's been years since I've done any calculus. Do I understand that integral correctly as finding the area of a circle of radius 1, which is π?

<comment><USER3> Yes -- well, 1/2 the area of a unit circle, which is why I multiply by 2. I used Sage ([http://www.sagemath.org/](http://www.sagemath.org/)) to create these quick results:

        f(x) = sqrt(1-x^2)
        2 * integrate(f(x),x,-1,1)

Sage replies: pi (Sage is smart enough to knows this result is equal to pi, so it doesn't provide a numerical approximation, but simply prints "pi").

        plot(f(x),x,-1,1,gridlines=True,aspect_ratio=1)

Sage replies: [http://i.imgur.com/GZdnT8z.png](http://i.imgur.com/GZdnT8z.png)

The function by itself produces a line equal to 1/2 the distance across a unit circle at the coordinate provided by x. Therefore a definite integral of the function on the interval -1 to 1 should produce an area equal to 1/2 that of a unit circle, or pi/2.

My Calculus tutorial: [http://www.arachnoid.com/calculus/](http://www.arachnoid.com/calculus/)

<comment><USER5> Or you can do it quick and dirty with a simple Google search: [https://www.google.com/search?q=y+%3D+sqrt%281-x^2%29](https://www.google.com/search?q=y+%3D+sqrt%281-x^2%29)

<comment><USER3> A nice, convenient way to get the graph. For the integral:

[http://www.wolframalpha.com/input/?i=2+integrate%28sqrt%281-...](http://www.wolframalpha.com/input/?i=2+integrate%28sqrt%281-x%5E2%29%2Cx%2C-1%2C1%29)

</comment></comment></comment></comment></comment></comment><comment><USER6> In Major League Baseball ([http://mlb.com](http://mlb.com)), the season lasts 162 games, from late March/early April to the very end of September. Game 163 is played, only if necessary, as a tiebreaker game ([https://en.wikipedia.org/wiki/List_of_Major_League_Baseball_...](https://en.wikipedia.org/wiki/List_of_Major_League_Baseball_tie- breakers)) that usually determines which team is heading to the postseason and which team is "going home."

Why 162? In 1961, the American League expanded from 8 teams to 10 teams, so instead of playing 154 games against 7 different opponents (22 each), a team now played each other 18 times. (This also works well because 18 is a factor of 3, and baseball often plays its games in a series of 3, held at the same ballpark.) The next year the National League followed suit.

If there are any Game 163's this season they'll happen Monday.

<comment><USER7> In 1997 interleague play screwed up that nice round robin, among other things.

<comment><USER6> Yeah, interleague play, as certainly now with the odd number of teams in each league has prevented a good, consistent number of games between teams — though more teams and divisions didn't help either.

</comment></comment></comment><comment><USER8> Math continues to fascinate me. Here's another one I love: [http://xkcd.com/217/](http://xkcd.com/217/)

I've spent hours in front of Google trying to find other random symmetries.. :]

<comment><USER9> [https://www.youtube.com/user/numberphile](https://www.youtube.com/user/numberphile)

<comment><USER10> Love that channel. Add to that list: [https://www.youtube.com/user/computerphile](https://www.youtube.com/user/computerphile)

</comment></comment></comment><comment><USER11> Since numbers are infinite and the ways we can manipulate them are infinite (or at least large) doesn't it mean that unlikely or strange things happen all the time?

I would be more surprised if they wouldn't happen.

<comment><USER12> > Since numbers are infinite and the ways we can manipulate them are infinite > (or at least large) doesn't it mean that unlikely or strange things happen > all the time?

They do. If you haven't seen it already I can recommend a movie about this topic. It can be enojyed for other reasons as well.

 _Pi: "A paranoid mathematician searches for a key number that will unlock the universal patterns found in nature."_ [http://www.imdb.com/title/tt0138704/](http://www.imdb.com/title/tt0138704/)

<comment><USER3> I like the movie, it deserves its fame, but there are some errors in the script that could have been easily avoided. For example, at one point a character says something to the effect of, "Surely they've printed out every 216-digit number?" No, not likely. :)

</comment></comment></comment><comment><USER13> Here's another directly related phenomenon.

Look at the function f(n) = n^2 + n + 41. Now start plugging in nonnegative integers. You get 41, 43, 47, 53, 61, 71, 83, 97, 113, 131, 151, 173, .....

 _All primes?!_ Well, not quite, plug in n = 41. But the fact that you get primes for so long is quite directly (but in a rather complicated way) explained by what ColinWright describes.

</comment><comment><USER14> From the article:

> e^{pi * sqrt(163)} ~= (5280 x 5280 x 5280) + 744

> e^{pi * sqrt(67)} ~= (640320 x 640320 x 640320) + 744

I really have to ask: What's up with 744?

<comment><USER15> It's the coefficient of the linear term in the q-expansion of the j-invariant. The j-invariant of i _sqrt(163) is an integer exactly, and e^{pi_ sqrt(163)} is the first term.

<comment><USER16> Now I want a t-shirt where the front reads "What's up with 744?" and the back is your response.

</comment></comment></comment><comment><USER17> Better yet, what's up with Google+? I get kicked back to Hacker News when I try to close the photo overlay. How to view the original page?

<comment><USER18> Yeah, that's a rather obnoxious behavior. Perhaps someone can change the link to the following?

[https://plus.google.com/101584889282878921052/posts/7enyPxZW...](https://plus.google.com/101584889282878921052/posts/7enyPxZW3RB)

</comment></comment><comment><USER19> Am I the only one directed to a blue tiled dome post? Weird, i really don't understand.

<comment><USER20> I saw the same blue tile post on my phone. I think some wires got crossed somewhere. The link gets redirected correctly on my desktop, but google says it encountered an error.

Here's the image: [http://imgur.com/sFDCt7K](http://imgur.com/sFDCt7K)

Here's the comment: [http://pastie.org/private/v5jg4cs014bojf1synjgq](http://pastie.org/private/v5jg4cs014bojf1synjgq)

<comment><USER19> Thanks. A comment below about the problem with closing a photo and directed back to hn also directs correctly.

A post technically explaining what is happening would be worth my upvote.

</comment></comment></comment><comment><AUTHOR> For those struggling with the link and with Google+ (why is it so crap?) here is the image:

[https://lh4.googleusercontent.com/-h0MYU3EVETo/UsHzgY2ZXjI/A...](https://lh4.googleusercontent.com/-h0MYU3EVETo/UsHzgY2ZXjI/AAAAAAAAVsQ/j4isMZBaR84/w1106-h654-no/163-f.png)

The text says:

What's up with 163?

The number 163 is involved in a number of mathematical formulae that, for some reason, produce answers that are extremely close to integers.

The first identity shows that 163 x (π - e) is extremely close to the integer 69. Here, π and e are famous constants, approximately equal to 3.141592653589793... and 2.718281828459045..., respectively.

The second identity shows that 163/ln(163) is even closer still to the integer 32. Here, "ln" denotes the natural logarithm; that is, the logarithm to base e.

The number in the third identity is almost freakishly close to the integer (640320 x 640320 x 640320) + 744. Actually this is not a coincidence, but the reason that it happens is very deep and has something to do with the Monster simple group. The Monster is perhaps best understood in terms of the symmetries of a 196884-dimensional algebraic object. Yes, that's as bad as it sounds.

Fortunately, there is a way to understand what is so remarkable about the number 163 in terms of unique factorization. What does that mean? Well, an important and very useful property of the integers is that every integer greater than 1 can be written as a product of primes in an essentially unique way. For example, the integer 21 can be written as 3 x 7, or as 7 x 3; we don't consider these factorizations to be essentially different because the only difference between them is that the factors appear in different orders.

This unique factorization property is still true if we consider negative integers and think of negative prime numbers as prime. We could then factorize -21 as (-3) x 7, 3 x (-7), 7 x (-3) or (-7) x 3. We consider these factorizations to be essentially the same, because they only differ in (a) the order of the factors and (b) multiplication of the factors by the (multiplicatively) invertible integers +1 and -1.

Unique factorization is a rare property for a number system to have.

What does this have to do with 163? Well, the number 163 is the largest of the nine Heegner numbers; the complete list of these is 1, 2, 3, 7, 11, 19, 43, 67, 163. If d is a Heegner number, then we can enlarge the rational number system Q by adding a square root of -d. This enhanced version of the rational numbers comes equipped with an enhanced system of integers, called the ring of integers. This ring of integers can be constructed from the usual integers by adjoining the square root of -d, sqrt(-d), unless -d-1 is a multiple of 4, in which case we adjoin the number (1 + sqrt(-d))/2 instead.

Heegner proved in 1952 that the ring of integers constructed above has the unique factorization property if and only if d is a Heegner number. The other large Heegner numbers also give identities like the one in the picture. For example, if we replace 163 by 67 in the third identity, the result is within 0.0000013 of the integer (5280 x 5280 x 5280) + 744.

<comment><USER21> If you replace 163 by 67 in the second identity, you get 67 / log(67) ≈ 15.93. I had heard the explanation for e^(pi sqrt(x)) in terms of the j-function, but I don't know where this x / log(x) thing is coming from...

</comment></comment><comment><USER22> First thing I thought of was 163.com aka NetEase, one of the most popular Chinese web portals (basically like the Yahoo! of China): [http://en.wikipedia.org/wiki/NetEase](http://en.wikipedia.org/wiki/NetEase)

<comment><USER23> Most Chinese sites use arabic numerals as their domain names - Hanzi domains are pretty new, and there's some cool overlap between number pronunciation and that of normal words in Mandarin (Mandarin is pretty much just monosyllable words, so there's a lot of overloaded meaning).

See also: Japanese number puns

[http://en.wikipedia.org/wiki/Japanese_wordplay](http://en.wikipedia.org/wiki/Japanese_wordplay)

<comment><USER24> I can still remember the McDonalds home delivery number in Shanghai and I never called it, purely from hearing it on TV a few times two years ago:

4008517517 si ling ling ba wu yao chi wu yao chi "我要吃我要吃"

<comment><USER25> McDonald's delivers in Shanghai?

<comment><USER22> On scooters / mopeds I believe.

</comment></comment></comment></comment></comment><comment> 0-o This is like the Ancient Aliens series for numerology. For episode 2 I suggest the number 2. As a premeditation I suggest we all start to cogitate on the preternatural property of the number 2: it being a "magic number".

</comment><comment><USER26> There is a question in QuizUp which says "Where does 163.com take you?"

</comment><comment><USER27> Another cautionary tail:

(pi^5+pi^4)^(1/(3+2+1)) = 2.7182818...

</comment><comment><USER28> Math: fuck yeah!

</comment><comment><USER29> So you can always ask your girlfriend "Do you want 163 * (pi - e)" :)</comment>


Using Rust to Scale Elixir for 11M Concurrent Users - <AUTHOR> https://blog.discordapp.com/using-rust-to-scale-elixir-for-11-million-concurrent-users-c6f19fc029d3 <comment><USER2> Thanks for sharing. I like reading posts that show a few trials and before arriving at the solution. Well and Rust and Elixir just seem to go together.

I had noticed that both a list and ordsets were tested. Ordsets are just sorted lists. The huge difference I guess is because full sorting and uniqueness checking happens after every member addition, as in add_member(M, L) -> lists:usort([M | L]), while ordsets actually traverses the list and finds the right place where to insert the element.

Also wonder how gb_sets [http://erlang.org/doc/man/gb_sets.html](http://erlang.org/doc/man/gb_sets.html) would behave. Those should provide better asymptotic behavior as they actually implement a balanced tree data structure. Rust would still be faster but it would be an interesting comparison.

<comment><USER3> Ordsets have O(log n) time insert, remove, and lookup operations. They are definitely not lists, they must be trees.

<comment><USER4> It appears you don't understand sorted lists if you think a tree structure is required for those asymptotics.

<comment><USER5> Erlang's ordsets() module has O(N) insertion. Since there's theoretical disagreement, here's experimental verification:

      11> f(), Ord_bench = fun(N) -> Random = [random:uniform(N) || _ <- lists:seq(1,N)], F = fun(Item, Set) -> ordsets:add_element(Item, Set) end, lists:foldl(F, ordsets:new(), Random) end.

#Fun<erl_eval.6.99386804> 12> timer:tc(fun() -> Ord_bench(10000) end). {337691, [1,2,3,4,5,7,8,10,12,13,14,15,16,17,18,19,21,23,24,26,27,28, 29,36,39,40,42|...]} 13> timer:tc(fun() -> Ord_bench(100000) end). {37167977, [1,2,3,5,7,9,10,11,12,15,16,17,18,19,20,21,23,24,25,26,27, 29,30,33,34,35,38|...]}

So, 337 ms turns into 37 seconds, roughly 100 times slower, when I increase the data size tenfold, which is what you'd expect from O(N) insertion.

Repeating for gb_sets, the results are 49 ms versus 484 ms.

Could the misunderstanding be that you don't realise that the ordsets module specifically guarantees that the representation is an ordinary sorted list?

(I am aware of e.g. skip lists. But that is not the same Erlang data structure as a sorted list.)

<comment><USER3> No, the misunderstanding is that I was thinking of Ordsets in Rust [1], which are implemented as trees.

[1] [https://docs.rs/im/10.0.0/im/#sets](https://docs.rs/im/10.0.0/im/#sets)

</comment></comment></comment></comment></comment><comment><USER6> Looks like you've implemented a square-root decomposition based solution resulting in O(sqrt(N)) insert, remove and O(sqrt(N)+M) slice operations where M is the number of elements within the requested slice [1].

Better approach, imo, would be an order statistic tree with O(logN) insert, remove and O(logN+M) slice operations [2]. This would be usually implemented as an implicit treap [3]. Furthermore, balanced binary search trees can be implemented efficiently in a functional language like Elixir that doesn't have mutable data structures. One such example is Erlang's gb_tree [4].

[1] [https://cp- algorithms.com/data_structures/sqrt_decomposition...](https://cp- algorithms.com/data_structures/sqrt_decomposition.html)

[2] [http://www.cs.yale.edu/homes/aspnes/pinewiki/OrderStatistics...](http://www.cs.yale.edu/homes/aspnes/pinewiki/OrderStatisticsTree.html)

[3] [https://cp-algorithms.com/data_structures/treap.html#toc- tgt...](https://cp-algorithms.com/data_structures/treap.html#toc-tgt-6)

[4] [http://erlang.org/doc/man/gb_trees.html](http://erlang.org/doc/man/gb_trees.html)

</comment><comment><USER7> > large sorted sets

Surprised to see no mention of trees which are basically the standard datastructure for this.

<comment><USER8> One thing that our blog-post didn't mention too well was the need to be able to access arbitrary slices of data within the sorted set, as well as being able to know the index at which an item was inserted as well as removed. This is necessary for our usage of sorted sets, as clients can subscribe to a given window of the sorted-set (e.g. the top of a member-list), and in order to compute the delta operations to keep the client-side list in sync with the one on the server, we need this information.

<comment><USER9> This reads as an almost textbook description of Cartesian tree.

<comment><USER8> Cartesian trees do not provide the ability to get items at arbitrary indices within the data structure in an efficient member from my understanding. In order to get the Nth item in the tree, a linear traversal is required. Furthermore, to get the index at which an item is inserted or removed requires the same traversal to accumulate the index.

<comment><USER9> The common solution is to hold size of subtree in the nodes in addition to the value a.k.a treap with implicit keys.

<comment><USER10> But at this point you're adding even more bookkeeping and implementation complexity just to use a data structure that's not ideal in the first place (for performance, sequential memory layout is better than chasing pointers).

</comment></comment></comment></comment></comment></comment><comment><USER11> I recently wrote a parser in Elixir for a templating language. Binary pattern matching is pretty neat for this purpose, but performance is nevertheless sub- optimal. Elixir was chosen over more suitable languages because our team is Elixir-only. OP shows that delegating performance critical computations in Rust is viable and fairly cleanly. That's good to know.

<comment><USER12> Have you taken a look at [https://rhye.org/post/erlang-binary-matching- performance/](https://rhye.org/post/erlang-binary-matching-performance/) ? I've seen lots of posts in the erlang mailing list about how unintuitive getting optimal performance from binary pattern matching can be.

<comment><USER13> It's also worth trying on OTP-22, which is reportedly much better at optimizing binary patterns:

[http://blog.erlang.org/OTP-22-Highlights/](http://blog.erlang.org/OTP-22-Highlights/)

</comment></comment></comment><comment><USER14> Well played. Kudos for creativity ([https://github.com/rusterlium/rustler](https://github.com/rusterlium/rustler)) Thanks for write up and benchmarks.

<comment><USER15> Rustler doesn't get the notice it deserves. I see people talk about using Rust to extend Ruby and Python and Node all the time, but none of those value concurrency-safety as much as Erlang does, and in the domains people tend to use Erlang for, the difference between using C and using Rust for NIFs matters even more than in typical back-end serverland.

<comment><USER16> I have a feeling there are plenty of excellent NIFs for most situations so most people using Elixir haven't been very dependent on Rust optimizations. Especially the web people who usually don't need that level of optimization, unless you're a giant like Discord, which is relatively unique.

Although I wouldn't be surprised to see more and more of Rustler as both Elixir and Rust gain in popularity. There's probably plenty of low-hanging performance fruit for any future rustler devs.

</comment></comment></comment><comment><USER17> Wow, very cool! Two questions:

1) One of the things that used to be the case with NIFs is that they can interfere with the BEAM's famed pre-emptive scheduling. Any chance that's what's going on here? ie: the NIF solution was fast because it's getting an unfair amount of CPU time, potentially leading to latency issues elsewhere in the app? I believe recently that was changed, though, and the BEAM can pre- empt NIFs?

2) Did you consider/benchmark a port solution before NIF? I think of those as a little safer, but with not as good of performance.

<comment><USER18> Their SortedMap operation times are quite a lot less than the 1ms recommended yield time for NIFs [http://erlang.org/doc/man/erl_nif.html#lengthy_work](http://erlang.org/doc/man/erl_nif.html#lengthy_work)

<comment><USER17> Ah, good point! Thanks.

</comment></comment></comment><comment><USER13> I'd be curious how ETS does on that problem. The Erlang efficiency guide says an ETS table takes:

> Initially 768 words + the size of each element (6 words + the size of Erlang > data). The table grows when necessary.

This might be too large if there's a ton of them, but perhaps a single table per node could be workable? In particular, ordered_set tables with write_concurrency have been improved a lot on Erlang/OTP-22:

       OTP-15128    Application(s): erts, stdlib

                   ETS option write_concurrency now also affects and
                   improves the scalability of ordered_set tables. The
                   implementation is based on a data structure called
                   contention adapting search tree, where the lock
                   granularity adapts to the actual amount of concurrency
                   exploited by the applications in runtime.

<comment><USER8> ETS has no facility to insert/remove an object and get the index at which the object was inserted to/removed from. This is needed in order to synchronize windows within the sorted set to our clients. We would have loved to use ETS, and use it in many places actually, and have open-sourced a few:

\- [https://github.com/discordapp/zen_monitor](https://github.com/discordapp/zen_monitor)

\- [https://github.com/discordapp/gen_registry](https://github.com/discordapp/gen_registry)

\- [https://github.com/discordapp/ex_hash_ring](https://github.com/discordapp/ex_hash_ring)

<comment><USER19> I don't know if it would be good enough,and there's certainly not a big reason for you to try, given your solution works great; but you could probably modify ets to give you the slot number where it inserted/removed and use that as a non-exact indicator of where in the sequence it was. (Depending on how accurate you need it to be, this might work?)

</comment></comment></comment><comment><USER20> Its great to see two interesting languages used together.

I wonder if an out of process solution was considered. For instance, Redis supports sorted sets: [https://redis.io/commands#sorted_set](https://redis.io/commands#sorted_set)

Using something like Redis saves development time and makes builds and packaging easier as no FFI is needed. However Redis comes at the cost of having a more complicated infrastructure.

<comment><USER8> On a given node, we maintain a few million sorted sets that see a query and insertion velocity in the orders of millions of operations per second. When we are meticulously measuring operation time in the microsecond range to meet a performance target, a networked solution is out of the question.

<comment><USER21> Also, rustler handles compiling the rust code, and configuring the nif dynamic library details. I'd wager it'd be a ton easier than Redis infrastructure even if redis worked for the use case. The sheer amount of flexibility in the BEAM is one reason I love working in Elixir.

Really it seems like a "write once and forget" type of nif. Have you had to make a lot of updates after the initial tuning with Rustler?

Though I was curious if you tried ETS or mnesia? They both offer sorted sets. Not sure if they just use ordsets behind the scene.

Edit: nvm, I see you answered in another thread.

<comment><USER8> This data structure has been operating in production for almost a year now, we have not touched it for any modification after writing and deploying it. We had 0 issues deploying it, and thanks to the comprehensive testing suite and benchmarks, we were confident it would meet the performance objectives. It was actually a drop-in replacement for the pure elixir data structure we had written. Deploying it was literally just a find + replace of OrderedSet to SortedSet in our code, running our test suite and then gradually deploying the new code to production.

<comment><USER21> Impressive work!

</comment></comment></comment></comment></comment><comment><USER22> This is really cool.

Meanwhile, it also shows Elixir itself is already really fast for the rest of us. Unless you're Discord or WhatsApp.

<comment><USER23> Elixir is pretty slow tbh, like 3-10x slower than Java / C# / Go. Immutability / message passing is very heavy.

<comment><USER24> It's all relative. Plenty people come from Node, Ruby, and Python.

Coming from the Go! world, I view Elixir is trade of speed for OTP out the box niceness. But comparing Elixir to Node, then you're just gaining speed and stability.

And it keeps going -- I very frequently see Go criticized for slowness by C, C++, and Rust developers. It is all about the usecase, which is why I find this topic so interesting. It's one of the few situations Elixir mid-level speed is actually a real world problem.

</comment></comment></comment><comment><USER25> Were any thoughts given to a process dictionary implementation? One (maybe supervised) process per instance of the pd_ordset type. The process would only ever have one client so no locking needed. Only potential issue would be the cost of sending messages. I have seen this kind of solution bring 100x speed improvements to write once <GB strings storage over ETS tables. This could be a gen_server as well. I'm curious because the PDict is often frowned upon (especially vocalized on the mailing list) yet cannot be avoided in some parts of OTP.

Also looking at the code leads me to believe the benchmarking of the rust code was done without taking the usage of a NIF's overhead, if there is any. Is this intended or am I missing something?

Last question: are the great tracing capabilities of the BEAM lost when using NIFs and would this be an issue at Discord scale?

Thanks for the interesting write up.

</comment><comment><USER26> It seems like they are using a really weird data structure (it's not even clear whether it's sub-quadratic).

This is a textbook problem normally solved using a balanced binary tree augmented by adding a number with the size of the subtree to every node (an "order statistic tree").

</comment><comment><USER27> I'm curious, how do they handle persistence? Is this data structure just used as a cache or?

<comment><USER28> As I understand it, Sorted Sets are an in-memory feature.

Discords architecture is one where an entire guild stays on one machine. And the sorted set is part of a guild as well.

So, there is no need to persist it, since it is ephemeral in nature.

Edit: perhaps to expand on this a bit more, keep in mind that discord is not a cloud native application. They have a few strong server nodes and a Cassandra cluster (at this point I am curious if they switched to scylladb) that handle all or at least most of their client facing features.

Since they do not need to scale services, the servers they have are stateful. Hence, if the guild server crashes, so will the data structure, since the data structure is of no use if the server crashes and would need to be rebuilt from scratch either way.

(Disclaimer: I am not an insider, just an Elixir fanboy)

</comment></comment><comment><USER29> I wonder on how they did the benchmarks. In Java/JVM land you would use JMH. Is there anything similar for Erlang/beam?

<comment><USER30> You can use some built-in modules: [http://erlang.org/doc/efficiency_guide/profiling.html](http://erlang.org/doc/efficiency_guide/profiling.html) or some external tools like recon: [https://ferd.github.io/recon/index.html](https://ferd.github.io/recon/index.html)

</comment></comment><comment><USER31> Thank you for sharing, and for going so deep on it!

</comment><comment><USER32> Did they just reinvent B trees?

</comment><comment><USER33> This title is the platonic ideal of an HN post.

<comment><USER34> The article was unusually well-written as well.

</comment></comment><comment><USER35> It would be nice if they had also went the same lengths with their clients in terms of efficiency...</comment>