Researchers propose a new interdisciplinary subfield called "Machine Behaviour" - <AUTHOR> https://www.nature.com/articles/s41586-019-1138-y <comment><USER2> My highly cynical take as someone researching in ML is that all these interdisciplinary AI studies, AI policy, etc are mostly grifters trying to get in on the gold-rush.

What has AI policy produced so far, really? Algorithm analysis and safety research is done on the algorithmic side, not on the policy side. Every time I have read an AI policy paper, it has been listings of desirable properties with absolutely no input on how to make any real steps towards them. In the meantime, realities of algorithm driven mass surveillance and censorship is moving ahead at a rapid pace.

Possible conclusion 1: We just need way more AI policy research.

Possible conclusion 2: AI policy is done by the wrong people and deep algorithmic understanding is a pre-requisite.

Possible conclusion 3: Current AI policy is just used as a fig-leaf by tech companies who hire a few policy essay-writers without substance. The lack of progress in that area is a feature, not bug.

Not mutually exclusive, not exhaustive. Feel free to point to valuable policy research.

<comment><USER3> a common pattern I see in papers written by people on the technical side is:

"The correct way of ensuring algorithmic fairness is X" where X is something they already happened to work on.

In various papers X could be robust optimization, differential privacy, causal inference, adversarial training, etc. The papers are mostly good, actually, but it's sort of putting the cart before the horse.

I think there might be legitimate value in having a bunch of lawyers, sociologists, and philosophers set a target more-or-less in ignorance, and let the people on the technical side try to hit it.

Of course even better would be interdisciplinary collaboration but that's hard, there aren't many incentives in its favor (where do you publish? any given venue is worthless to half of the authors). and it requires humility on the part of everyone involved.

<comment><USER4> _> The papers are mostly good, actually_

Not surprising. Computer scientists have a _bit_ of experience with computation, after all. It would actually be _more_ concerning if a bunch of new stuff was being invented whole cloth.

 _> but it's sort of putting the cart before the horse._

Not at all. ML algorithms are _just fucking algorithms_ and computer scientists have been thinking about what it means for an algorithm to be correct since... Turing.

And have been proving various theorems about _ML algorithms in particular_ since at least the 60s.

Complaints that "AI safety looks a lot like previous CS research" are basically equivalent to observations that "neural nets have been around for a lot longer than alexnet".

 _> I think there might be legitimate value in having a bunch of lawyers, sociologists, and philosophers set a target more-or-less in ignorance, and let the people on the technical side try to hit it._

I disagree. This is how you end up with endless navel gazing about trolley problems while actual vehicles kill people by accelerating without control because redundant parts are too expensive and engineers don't have enough voice. Philosophers are rarely interested in honest-to-god engineering ethics, which almost always boils down to "pay well enough to hire good people, and then listen to the good people you're paying good money to have around".

<comment><USER5> Ah yes those money grubbing philosophers.

<comment><USER4> I don't think it's money grubbing. I think it's a genuine attempt to connect with the zeitgeist and be relevant, paired with a fundamental misunderstanding about what's actually happening inside self-driving groups.

But the intention doesn't really matter.

What matters is the utility of the output!

In fact, somewhat ironically, I think a lot of the _good_ work on ethics for AI is coming out of engineering, business, statistics, and economics departments. And those academic departments to do be a bit more "money grubbing" relative to philosophy :-)

</comment></comment></comment></comment></comment><comment><USER6> It's not clear to me how this "field" of study is any different from existing academic analyses of automation or computing. AI doesn't change what computers do; it affects only _how_ it's done. The problems inherent in automation are as old as windmills, falling water, and certainly electricity.

Sure, a few modern fields are affected more than others, but improving voice recognition or visual object identification component by 20% doesn't fundamentally change the system using such code to a degree that it warrants a new academic discipline of study.

A new _legal_ field of study on automation makes more sense to me. Algorithmic bias, product or service liability, baseline accountability, the standards of due diligence and safety -- these are rising concerns exacerbated by the recent increase in automation, AI-based or not. But these problems are hardly unique to AI any more than the misdirection of elections was unique to manipulation of digital social media.

I'm also doubtful that meaningful solutions to byproducts of automation by corporations and nations can be meaningfully addressed by a bunch of academic computer scientists.

<comment><USER7> >AI doesn't change what computers do, if affects only how it's done.

Except AI is very much changing how things are done _as well as_ what computers do. Image recognition, natural language processing, self piloted vehicles, these are all novel applications for machines which carry significant real world risks to life and property.

Moreover, AI in it's current state is a complex black box with unpredictable outputs for given inputs, which may be chaotic - see, for example, adversarial attacks. If we want a better handle on, say, regularizing outputs in a predictable way, or at least a clearer window into the occluded complexity of neural network decision making, a whole new field may very well be warranted. There's a reason we are calling these program outputs _behaviors_ now; it's a new type of loosely -deterministic computing until we develop a deeper understanding which will not be trivial.

<comment><USER8> > Image recognition, natural language processing, self piloted vehicles

Obviously none of these are AI and I don't see how you're helping anything by insinuating they are.

<comment><USER9> These absolutely are AI as far as the term has been used for a significant amount of time. There is no value in trying to redefine it after, what, 70 years?

<comment><USER8> > There is no value in trying to redefine it after, what, 70 years?

That implies any meaningful definition to begin with. That is entirely an illusion and people are using this to mislead investors (not that I'll ever shed a tear for them).

<comment><USER9> People absolutely use it usefully, that it can be used as marketing fluff is a far more recent thing. If it's never been used usefully and isn't now, there's no point arguing about it because that war has been lost and if it has been used usefully redefining it makes no sense.

Arguing about definitions is easily one of the least relevant parts of any discussion so I'll leave it here. I just wish AI topics didn't always have _someone_ say it wasn't "real AI".

</comment></comment></comment></comment></comment></comment><comment><USER10> If you lead, or aspire to lead, major projects, I submit a reasonable definition of 'major' includes sitting at the table in national capitols with senior policy makers. They will expect you to be as invested in understanding their concerns, and the concerns of their constituents, as they are in trying to understand your project and your objectives. Reading articles like this, and I read and annotated the whole thing, is the homework you are going to have to do. Yes, they should all learn category theory and linear algebra and they haven't. That doesn't mean you suddenly have insight on the evolution of the problem spaces they've spent their entire, obviously successful careers navigating. So maybe take the time pay the penence for your genius, and invest the time to understand something about their efforts as they are fumbling around in the dark. At least they had the decency to put in writing for you to read, or ridicule as the case may be.

Also, feel free to look up some of the authors on this paper. They don't suck.

</comment><comment><USER11> I get worried when I see something like this.

I'm inclined to believe a substantial number of researchers are currently being deliberately fuzzy about what "AI" can and cannot do. Why not call it algorithms and statistics? I think lay-people have a very skewed understanding of what has already been achieved through AI. They may also not understand the word algorithms but at least it doesn't make them think of Skynet. For example, if you asked a person on the street whether there exists an "artificially intelligent supercomputer" somewhere that could help you plan _all aspects_ of a small but entertaining dinner party, they would probably say yes. They imagine that you could just ask IBM Watson to help out, and he'd tell you what to do. This is completely false. "AI" systems are very fragile, and yes, we could build something that plans dinner parties, but we'd have to start over if we needed to plan a kid's birthday party instead. It's very far from strong AI. Ten years ago when it was mostly IBM telling lies, the end result was a couple of billion dollars wasted by hapless healthcare conglomerates. That's already bad. But now we have people from MIT, Stanford, Harvard, Yale, and more embracing the term AI and relying on unfounded hype to push for funding. It would be much less sexy if we called it "Facebook/Google/etc enable unfair/discriminatory advertising by combining intensive data collection with logistic regression, sometimes in multiple layers, and with some graph algorithms thrown in". But it would be a much better starting point for a well-informed debate. I'm not trying to minimize the importance of algorithms in our world, but a healthy discussion should be based on a sound understanding of the facts on the ground, and AI hype is not helping with that. I strongly prefer the less hyperbolic terminology adopted by someone like Aaron Roth at UPenn, e.g. see the blurb for "The Ethical Algorithm: The Science of Socially Aware Algorithm Design".

Bonus AI rant: For the celebration of the new MIT Schwarzman College of Computing, which is a huge expansion of the arguably most important computer science department in the world, there was a discussion panel on AI consisting of MIT President Reif, Henry Kissinger (War criminal?, Theranos board member, wannabe AI expert), Tom Friedman (columnist of limited substance), and Stephen Schwarzman (business man, coined the "increased taxes on carried interest are like Hitler's invasion of Poland" analogy, brought the dough). How the heck is that the inaugural panel!?!

<comment><USER3> most actual researchers are entirely sick of the hype and are desperately trying to correct it, to no avail. i think laypeople just have a strong desire to get overexcited about AI and nothing is going to stop them.

<comment><USER12> Maybe researchers would like that, but PR is important even for academic researchers. Sexy and hyped topics get more media coverage, more funding and lead to more impressive CVs and careers.

</comment></comment></comment><comment><USER13> "Means and motive matter as much as ends. AIs don't operate in isolation. Somebody designs them, somebody gathers the data to train them, somebody decides how to use the answers they give. Those human-scale decisions are –or should be– documented and understandable, especially for AIs operating in larger domains for higher stakes. It's natural to want to ask a programmer how you can trust an AI. More revealing is to ask why _they_ do."

[https://www.ribbonfarm.com/2018/03/13/justifiable- ai/](https://www.ribbonfarm.com/2018/03/13/justifiable-ai/)

</comment><comment><USER14> Paging Dr. Susan Calvin.

<comment><USER15> "Robopsychology!" was my first thought haha.

</comment></comment><comment><USER16> Is there a rigorous definition of "machine behavior" as a research field?

I can see how a good-enough marriage of ML/NN to classic symbolic AI could yield a system capable of higher-order, intention-based "behavior" in complex circumstances. But we're still a long way from achieving that, as far as I know.

</comment><comment><USER17> There is no way to stop the hype. You can just enjoy watching the waves.

</comment><comment><USER18> Human-Computer Interaction?

<comment> dane-pgp Human-Cyborg Relations?

</comment></comment><comment><USER19> > _Machines powered by artificial intelligence increasingly mediate our > social, cultural, economic and political interactions._

What about non-AI powered machines that mediate our social, cultural, economic and political interactions? For example, science journal paywalls.

<comment><USER20> In my experience most researchers are more than happy to share with you the paper upon askance via e-mail.

<comment><USER21> If I asked for a paper and they looked at me askance, I'd wonder why I emailed them instead of just asking - since we're in viewing range obviously.

<comment><USER22> Damn autocarrot.</comment></comment></comment></comment>


Ask HN: Any good stories you have about writing? - <AUTHOR>

For my online book (http://howtogetfocused.com), I'm writing a chapter on... writing.<p>I wanted to hear any stories you guys have come across as it relates to "writing." That is, interesting writing routines you've heard of, what  writing does for the mind. Why/why not you should write, etc. If you have any ideas, would appreciate any links or stories. <comment><USER2> I used to work as a scientist for the Air Force and part of my job was writing reports and proposals. My community expected clear and concise technical writing. At the time I had a lot of trouble with passive voice and complex expressions. I was intimidated by our style guidelines and it really made it hard for me to start writing. One day I took a look at Style and Dict (GNU Tools for checking your writing) and wondered if I could do better.

I then set out to create After the Deadline (<http://www.afterthedeadline.com>) and it has changed my writing in two major ways.

1) Having a good tool in my corner gave me the confidence to focus on the content of my writing and not the mechanics. I knew I could run my writing through my tool later and get rid of the passive voice and complex expressions.

2) Having software beat me over the head with my bad habits forced me to notice them and what they were. I then stopped making some of these same mistakes. My writing is not perfect (and I'm in the position where some people find meaning in their life pointing this out), but it's passable. :)

So my recommendation for writers:

Know that not all spell and grammar checking software sucks. There are solutions beyond Microsoft Word. Find something that matches the style of writing you're going for and use it to help with the mechanical things.

I also recommend turning off the "as you type" spell and grammar check as it's a distraction.

<comment><AUTHOR> Wow, raffi. That's really neat stuff. I'll have to check it out.

Have you seen a new product called <http://scribeseo.com>

You could perhaps get a bundle deal going with those guys? However, the trick would be balancing seo and writing for the reader.

Anyways, great stuff. Thanks a lot.

</comment></comment><comment><USER3> <http://news.ycombinator.com/item?id=1120356>

<http://news.ycombinator.com/item?id=778108>

(P.S. thanks, searchyc!)

</comment><comment><USER4> George Orwell - Why I Write

<http://www.orwell.ru/library/essays/wiw/english/e_wiw>

</comment><comment><USER5> I wrote this a while back. Writing is Thinking:

[http://www.petermichaud.com/essays/the-secret-about- writing-...](http://www.petermichaud.com/essays/the-secret-about-writing-that- no-one-has-the-balls-to-tell-you/)

</comment><comment><USER6> Maybe you will be as surprised as I was to hear the name Stephen King mentioned in the context of writing advice, but 896 Amazon reviewers can't be wrong:

Stephen King - On Writing:

[https://www.amazon.com/dp/0743455967?tag=0596800673-20&c...](https://www.amazon.com/dp/0743455967?tag=0596800673-20&camp=213381&creative=390973&linkCode=as4&creativeASIN=0743455967&adid=1GHWX3BNJZNJW5ZCMWCB&);

This is what Roger Ebert said about the book:

 _"A lot of people were outraged that he [King] was honored at the National Book Awards, as if a popular writer could not be taken seriously. But after finding that his book On Writing had more useful and observant things to say about the craft than any book since Strunk and White's The Elements of Style, I have gotten over my own snobbery."_

<http://en.wikipedia.org/wiki/On_Writing></comment>


Deep Dive into PHP 8's JIT - <AUTHOR> https://thephp.website/en/issue/php-8-jit/ <comment><USER3> This article is actually now quite out of date as PHP has switched to a tracing JIT which appears to be heavily based on LuaJIT.

[https://github.com/php/php-src/pull/5874](https://github.com/php/php- src/pull/5874)

[https://github.com/php/php- src/commit/4bf2d09edeb14467ba7955...](https://github.com/php/php- src/commit/4bf2d09edeb14467ba79551a08d84efdff314899)

<comment><USER4> Any ideas why they've moved from a method-based JIT to tracing?

AFAIK tracing JITs are generally inferior to method-based ones, which is why none of the major JavaScript engines use tracing. Their only advantage seems to be (relative) simplicity, which is essential for the lightweight LuaJIT but not for PHP.

<comment><USER5> That doesn't sound right. How would a trace-based JIT be easier to implement? There was a research project which made HotSpot into a trace-based JIT. It reported both faster compilation times, and superior quality of generated code.

(I don't know if the HotSpot folks ever considered adopting the changes.)

[http://www.ssw.jku.at/Research/Papers/Haeubl11/](http://www.ssw.jku.at/Research/Papers/Haeubl11/)

<comment><USER4> > How would a trace-based JIT be easier to implement?

When compiling at method-granularity, with incomplete information about types, complex features are required e.g. On-Stack Replacement (if the current method becomes "hot", it needs to be replaced by a compiled version), Inline Caches, Hidden Classes and Deoptimization.

In a trace-based JIT, all these features can be replaced by "just compile another trace".

<comment><USER3> We need all those complex features with tracing JITs too. Getting into a trace is basically OSR. Exiting a trace and restoring the interpreter state is basically an OSR exit or deoptimization. It's really just different terminology for the same thing. Inline caches on method calls become guards. We can't just ignore them.

LuaJIT does without hidden classes or property caching because on-trace it's able to eliminate table access and allocation. Performance tanks on OO Lua once you fall off trace. You can work around it with "compile another trace" in a sense that works as long as you have infinite cache.

</comment></comment></comment></comment></comment><comment><USER6> Tiny nitpick: the article describes JavaScript as an interpreted language without JIT when most major JS engines do have JIT optimisation. It's difficult to make this call about any language with more than one implementation.

But anyway, kudos to the PHP folks for yet another improvement. It's been a long time since I used it but I'm continually impressed by how far it's come. And it's a useful test to how flexible a developer is willing to be to ask them to code some PHP, there are fewer and fewer reasons against it these days other than personal style preference.

<comment><USER7> The whole idea of "language X is interpreted/compiled" is a simplification I wish people were more careful with.

Cpython is interpreted, pypy has JIT, mypyc is compiled. It's the implementation, or even implementation's specific runtime options that decide about things like that, not the language.

<comment><USER8> The statement would be far more accurate as "language X is [almost always/sometimes/often/commonly/etc.] interpreted/compiled". Unfortunately a lot of people seem to like speaking in absolutes.

C can be interpreted too: [https://en.wikipedia.org/wiki/CINT](https://en.wikipedia.org/wiki/CINT)

<comment><USER9> Theoretically, any language can be interpreted.

Compilation is just _interpreting_ what a program does _and_ producing machine code (or other code in the case of transpilers) that computes the same thing.

Interpreters are just that, but instead of _producing_ code, they run code in themselves thats computes the results directly.

One could even consider machine code just "obfuscated" assembly code. In that sense, machine code is just another language that _also_ could be interpreted (intepretation-based emulators like Bochs) or compiled again (JIT-based emulators like DOSBox).

This brings up a slightly related question: is there a language that _can't_ be _compiled_?

<comment><USER10> _" question: is there a language that can't be compiled?"_

There is some question whether Perl could be, because parsing it without running it has some ambiguity. [https://www.perlmonks.org/?node_id=663393](https://www.perlmonks.org/?node_id=663393)

<comment><USER7> That's not really a barrier. It's "can't be parsed without ambiguity", not "can't be parsed". You compile cases like that by compiling a check for the relevant condition, then compile the possible versions in the if/else branches. Unless you can prove which case it will be from other code - <USER2> you can simplify anyway.

</comment></comment></comment></comment></comment></comment><comment><USER11> "PHP as fast as C" \- People who are in php language development may die laughing reading this statement.

Truth can be harsh, but people sometime overvalue to such an extent is hard to understand.

<comment><USER12> What I've noticed is that JITs often can reach C speed in workloads like this:

        sum = 0
        for i in xrange(n):
          for j in xrange(i):
            sum += A[i][j]

And when they reach that milestone, some people call it "as fast as C".

Never mind that that's not what people actually write in Python or PHP. It's a synthetic benchmark, not a real workload.

The workloads in those languages are generally oriented around strings, hash tables, and function/method calls.

And the JITs don't seem to do nearly as good a job there. I tested PyPy on Oil [1] a few years ago, and it made it slower, not faster. And it used more memory. (Though PyPy is an amazing project in many respects.)

[1] [https://www.oilshell.org](https://www.oilshell.org)

<comment><USER13> This is not what people write in Python or PHP, but this is what people write in C extensions for Python or PHP. Having your JIT be that fast allows you to forego those extensions and write the low-level hot loops in the same language, and that's a huge improvement.

You usually don't care how your matrix multiplication/regex matching/unicode normalization/JSON parsing is implemented, but people had to make those, and they are users of the language too.

Even though it might not change the bottom-line for your high-level app.

<comment><USER12> Well, the problem is that Python and PHP are actually bad languages for expressing code like that. For expressing C. For one, they're not statically typed.

Julia is a dynamic language that seems to do better because it was designed for this purpose.

But it doesn't seem to have panned out in practice in Python, or PHP as far as I know. Those languages have huge piles of C, and whenever you call into C, the JIT gets confused. People don't seem to rewrite their huge piles of C in Python or PHP. In Python, it's more likely Cython.

I'd like to see pointers to counterexamples -- where people actually wrote some C-like code in Python or PHP and let the JIT do its work. I haven't seen it, aside from the PyPy project itself, and maybe a few other examples. I think you would still take a significant performance hit.

The issue is that C compilers in 2020 are _even better_ at compiling the example I showed. They do amazing things with that kind of code that state-of- the-art JITs don't in practice.

<comment><USER14> It's not that the JIT gets confused, it's that the C APIs for these languages can do almost anything - even stuff that you can't normally do in the language. So you are faced with a giant optimization boundary.

However a call to a shared library that isn't linked against your language API is not very expensive as you have a much better handle on the values that are escaping and can make much better optimization choices.

In the Truffle project we are using an LLVM bitcode interpreter that allows us to JIT right through that language boundary and still link to native shared libraries. This means people shouldn't have to rewrite their C extensions and we can hopefully still run the combination of high level language and C extension faster.

<comment><USER3> That optimization boundary seems like it's much more of a problem for TruffleRuby than it is for language-specific native implementations? IIRC TruffleRuby relies a lot on being able to optimize away Ruby objects and frames and there's quite a performance cliff if you have to materialize full escaping objects?

JSC and LuaJIT have simpler ways to deal with calling native code which might do weird stuff.

</comment></comment></comment></comment></comment></comment><comment><USER15> Unless we are speaking about Java before Java 1.2, it is definitely not interpreted, there are plenty of JIT and AOT implementations without any kind of interpretation step.

Since 25 years, time to learn that implementations and languages are not the same.

<comment><AUTHOR> I might be wrong here, as I'm not so close to Java development. But a language implementing JIT, at least to me, is interpreted.

Could you please point an implementation detail where a JIT-capable engine doesn't include interpretation in its runtime?

In every case, thanks a lot for your feedback!

<comment><USER15> Well, from CS compiler theory point of view it is not.

For example in .NET, MSIL goes directly into a pipeline that produces native. You can easily validate that RyuJIT has no interpretation.

Or for example, watchOS applications packaged with bitcode, get JIT compiled at installation time.

</comment></comment></comment><comment><USER16> There is a small error in the text. When describing the opcache.jit setting it shows examples where the third flag is set to 5, eg 1255, but the table of possible values for the third flag, 'T - JIT trigger', goes from 0 to 4.

<comment><AUTHOR> Thanks a lot! I blindly fetched this from the reference mentioned there. I'll update this as soon as I have time (probably during this week) and also let the author of the referenced post know about it.

</comment></comment><comment><USER17> Didn't Facebook also develop is JIT for their PHP? Or have they moved on frmo PHP?

<comment><USER10> Hack/HHVM no longer has a goal to be PHP compatible, but it looks pretty much the same. It does have a JIT: [https://hhvm.com/blog/2027/faster-and-cheaper- the-evolution-...](https://hhvm.com/blog/2027/faster-and-cheaper-the- evolution-of-the-hhvm-jit)

<comment><USER18> Do we know anyone outside of Facebook uses Hack/HHVM in production?

<comment><USER19> Wikipedia runs on HHVM IIRC. [https://hhvm.com/blog/7205/wikipedia-on- hhvm](https://hhvm.com/blog/7205/wikipedia-on-hhvm)

<comment><USER20> Wikipedia was moved back to PHP:

\- [https://phabricator.wikimedia.org/T176370](https://phabricator.wikimedia.org/T176370)

\- [https://phabricator.wikimedia.org/T229792](https://phabricator.wikimedia.org/T229792)

\- [https://launchdarkly.com/blog/how-the-wikimedia- foundation-s...](https://launchdarkly.com/blog/how-the-wikimedia-foundation- successfully-migrated-to-php7/)

<comment><USER19> Didn't know that. Thanks for the info!

</comment></comment></comment></comment></comment></comment><comment><USER18> May be everyone on HN has Adblock/ PiHole on.

But this Blog, a single page has 5 Google Ads in it. I dont mind one or two, top and bottom. But 5, right in the middle of every section.

<comment><USER21> Blogs like these are the main reason to use ad blocks. And from their perspective: everyone using adblocks is the reason they need 5 ads on the site

<comment><AUTHOR> Hey man, author here. Thanks for the feedback. I recently added adsense there and I'm testing with the amount of ads in the page. Currently I'm letting AdSense decide how many ads it should place and where.

But if you believe the amount is so harmful for your experience, don't worry. I'll be more than glad to reduce this amount.

Cheers!

</comment></comment></comment><comment><AUTHOR> Thank you all for sharing your comments and thoughts. I'm unfortunately out of time today, but I'll try to address your comments asap :)

Cheers!

</comment><comment><USER22> Not a deep dive, a rather very shallow dive.

Deep dives were handled before: [https://wiki.php.net/rfc/jit](https://wiki.php.net/rfc/jit) And its discussion [https://externals.io/message/103903](https://externals.io/message/103903)

</comment><comment><USER23> I definitely misread this title as the P8 JIT that IBM was working on for a while.</comment>


TechCrunch Comment System Kills URLs - <AUTHOR> http://mikecanex.wordpress.com/2011/07/21/techcrunch-comment-system-kills-urls/

<comment><USER2> What a juvenile post/bunch of comments, and total non-story (if it only happens with URI's that aren't actually valid).

Besides, this is an issue with Facebook's commenting system, if anything.

<comment><USER3> I posted the same on the blog post.

I would question that it could be the user (mikecane02)'s browser as it looks to me like what would happen if you didn't support the character set. Could be some linkbaiting on the OP's part.

</comment></comment><comment><USER4> That screenshot is painful to read. Why did you use a JPG rather than a PNG?</comment>


Algorithmic fitting of Japanese candy - <AUTHOR> https://www.candyjapan.com/behind-the-scenes/algorithmic-fitting-of-japanese-candy <comment><USER2> The very first line is "Candy Japan ships candy to subscribers twice a month." I read that and had to pause. Wow. I don't think if I tried for months that I could have done a better job describing _any_ business in such a small number of words. And that line and the last line are the only "marketing" this article does. It's subtle too, because the first line is completely necessary to introduce why the author wanted to algorithmically fit candy. It's just ... beautifully done. I'm not being sarcastic when I say any of that. Candy Japan has written a number of content-marketing articles and they're always informative and interesting. Obviously others agree because they bubble to the top of HN frequently.

Candy Japan's articles should probably be mandatory reading for anybody wanting to start a small, online business, because this type of marketing is gold.

<comment><USER3> My only complaint is the "dark pattern" of not listing prices. There are prominent calls-to-action to enter your email address and sign up. I don't want to enter my email address unless I want to sign up, which is dependent on the price.

"firstname.lastname@example.com" now has an email from Candy Japan, and I obviously can't click on the verification link there.

After some digging, I found [https://www.candyjapan.com/gift](https://www.candyjapan.com/gift), which lists prices:

        24 shipments! ($348.00, sent over one year)
        Dozen shipments ($174.00, sent over six months)
        Eight shipments ($116.00, sent over four months)
        Four shipments ($58.00, sent over two months)
        Two shipments ($29.00, sent over one month)

Doing the math, it's $14.50 per box. Even though many levels are listed, there is no quantity discount, which is not obvious - I'm not good at multiplying 29 or 14.5 by powers of 2 in my head. If it was $20 or $30/month, that would be a little different.

Some more digging with a disposable email finally brought me to [https://www.candyjapan.com/subscribe](https://www.candyjapan.com/subscribe), which shows that it's still $29/month whether gifting or subscribing. Furthermore, anyone can go to that page and see the price. No, the tiny "subscribe" link on the main page under the comic does not link there - it's an anchor to the email entry point.

Perhaps A/B testing says differently, but I would like the business a lot more if there was a big label, above the fold, on the main page, that said "$14.50 per box" or "$29 per month".

<comment><USER4> The home page says "For $29/month (shipping included) we'll start hunting for interesting sweets for you here in Japan twice a month, then box and ship them directly to you by Japan Post."

It's easy to miss, which is unfortunate, but I would classify it as more of a marketing oversight than as a "dark pattern" which would imply intent.

<comment><USER3> Yup, missed it. Agree it's an oversight, but it's still annoyingly difficult to find if you miss that line.

</comment></comment></comment></comment><comment><USER5> You have run into a monster.

[https://en.wikipedia.org/wiki/Packing_problems](https://en.wikipedia.org/wiki/Packing_problems)

Packing of regular shapes in a confined space is hard enough... much less a general collection of cuboids in a 3D box. On the bright side, if you find a good way to do this you will have accomplished a lot more than packaging candy.

Here is an annealing approach that might lead somewhere (although I'm not sure... I am no expert on packing and really smart people have been thinking about this for hundreds of years)

Instead of thinking of sharp cubes as geometric objects, think about each one as a function from space to 0 or 1. (0 if you're outside the box, 1 if you're in it).

Blur each cube function with a Gaussian, then try to position these blurry functions in your box so that they are well-separated. This positioning can be done using ordinary numerical optimization as a 'feasibility search': your objective function can be the maximum of the sum of the functions throughout the box, searching over the space of positions that lead to the cuboids being within the box's boundaries. You have to scale the blurred functions so that they all individually have the same max. Declare a positioning infeasible if the max of the sum is above the individual max.

The positions you get from this will be a first approximation of where you should put your candies.

After you've found the positions, reform your cube functions, this time using a sharper/thinner Gaussian. Then run the optimization again using the previous step's result as your first guess.

The idea here is that hopefully the problem of shifting blurry functions around to be uniform is a lot easier than the strict packing problem which most people try to avoid solving.

Also it's pretty amazing to me that you have made headway in this using Javascript. Good luck!

<comment><USER6> The trunk capacity of cars in defined as the number of a certain kind of box you can fit into it. Car manufacturers naturally want to maximize this number for their cars. Traditionally there are expert packers (pretty nice job if you ask me) who do this, but there are now some algorithmic approaches:

[http://www.joachim-reichel.de/publications/diss.pdf](http://www.joachim- reichel.de/publications/diss.pdf)

</comment></comment><comment><USER7> As a student a long time ago I worked with a friend on a 2-d version of this problem for a client who had a framing store. He was trying to cut mirrors of certain sizes in large sheets while minimizing the waste.

The approach we used was to start with a greedy best-first approach (like what is done by the Python library the author is using), and then try to improve on it by exploring the space of neighboring solutions. A "neighbor" here can be created by a pairwise permutation of candies (or mirrors in our case) in the order in which you're trying to fit them in the box, or trying to fit the candy flat instead of upright, etc.

You can do a hill-climb, which means you keep going as long as you can find a better neighbor, or you can use simulated annealing, which accepts a degradation in the solution (to avoid getting stuck in a local optimum) with a certain probability. This probability decreases over time so that you can eventually converge to a solution. Simulated annealing will work better than hill-climb, but its stochastic nature means two runs of the program might give you two different solutions (our client thought that was a bug!).

<comment><USER8> Yeah, simulated annealing as to the order to place objects in is probably optimal for non-trivial values of n. Also, determinism can often be important and stochastic doesn't have to mean non-deterministic - seed the RNG.

<comment><USER9> > seed the RNG

This is something that can't be repeated often enough! Especially when running unit tests.

</comment></comment></comment><comment><USER10> A friend of a friend used to work at NASA on how to pack the Space Shuttle, which is essentially the same problem... except that things need to be taken out in the correct order, through a small hatch, with a robot arm and in zero gravity.

TL;DR: Packing candy is rocket science.

<comment><USER11> Did he try to write an algo for that?

This sounds like the kind of problem were I immediately decide that using some good ol' fashioned human rationale is the best way to proceed. Too many rules and conditions. Software is sometimes more difficult than its alternative.

</comment></comment><comment><USER12> As others have said, I would go with simulated annealing, or any non convex optimization algorithm you're more comfortable with.

But first I'd investigate what's really the gain of closely packing the candies. Would shipment costs increase with a bigger non completely filled box? wouldn't a non rigid bag be better? Do non closely packed candies get damaged with shipment? Do customers prefer closely packed candies?

I mean, let's say you get good enough local optimum configurations, then you have to exactly follow them packing the candies one by one... wouldn't it be easier to throw everything in a bigger bag and ship that?

<comment><USER5> because it's cool

<comment><USER13> Yup.

This is one of those things that 99% of customers will not notice. Some small percentage may prefer the closely-packed box over the bag but not be able to articular why.

And some very small percentage will see it, recognize the difficulty in continuously doing something like this over the "throw it in a bag and ship the bag" approach, and think it's amazing. Those are the fans that will repost your stuff, talk about you to others, etc.

This type of stuff is what creates the "1" in the 90-9-1 split.

</comment></comment></comment><comment><USER14> We were presented with this problem in a course in constraint programming where we used a library called Gecode [2] to implicitly express and effectively search the solution space in terms of variable relationships/constraints. I remember symmetry breaking being a central concept [3].

[1] [https://en.wikipedia.org/wiki/Constraint_programming](https://en.wikipedia.org/wiki/Constraint_programming) [2] [http://www.gecode.org/](http://www.gecode.org/) [3] [https://en.wikipedia.org/wiki/Symmetry- breaking_constraints](https://en.wikipedia.org/wiki/Symmetry- breaking_constraints)

<comment><USER15> Yes, but even better there is the global constraint geost [1] which is specifically for this purpose in k-dimensions; see also [2] for a more detailed presentation.

Unfortunately, Gecode does not provide it (and you would have to express it in form of other, more lower level constraints which gets quickly complicated), and neither does Google's OR-tools which have a nice Python interface.

There is JaCoP which provides the geost global constraint and makes it available via the declarative MiniZinc problem description language so one needs not to use the Java API.

[1] [http://sofdem.github.io/gccat/gccat/Cgeost.html](http://sofdem.github.io/gccat/gccat/Cgeost.html) [2] [https://www.sics.se/~matsc/docent20090209.pdf](https://www.sics.se/~matsc/docent20090209.pdf) [3] [http://jacop.osolpro.com/](http://jacop.osolpro.com/)

</comment></comment><comment><USER16> Sometimes hackers want to solve the hard problem, but not the most important problems.

Check out some of the comments.

This company might want redirect some focus on first principal issues like [1] consistently shipping 2 boxes a month to customers, [2] insuring the candy is of high quality, or [3] making sure basic user / subscription management works on their webapp.

It takes discipline to direct resources at what will really move the needle the most.

<comment><USER17> I agree. When I was re-analyzing one of CJ's experiments using Bayesian decision theory ( [http://www.gwern.net/Candy%20Japan](http://www.gwern.net/Candy%20Japan) \- speaking of wanting to solve hard but not important problems...) what I was struck most by was the incredibly high cancellation rates he was reporting. Clearly _something_ is going wrong there.

</comment></comment><comment><USER18> BTW, I really like the concept of Candy Japan. But twice a month and $29 is too much. Hopefully they can get monthly or bi-monthly subscription soon.

<comment><USER19> I suspect shipping is the real problem. Here in Japan, while there are several shippers who give great service at low prices _within_ the country, there are almost none that do the same for international shipping.

I live in rural Japan in a famous tea growing area. Many of my neighbours are tea growers. They are desperate for ways to export their tea cheaply. I briefly looked into helping them out, but shipping looked impossible. They will sell me 100g of first cut shincha for $3-5 (in vacuum packed foil packages even!). But shipping? It will raise the price 5-10x. I bet the vast majority of his price is shipping :-(

The way to ship internationally is by slow boat. It's super cheap. What you want to do is ship by the palette (or better by the container) and then have someone on the other side forward it on with local shipping. But that takes both volume and cooperation of another entity.

For a small 1-2 person operation, it's infeasible.

<comment><AUTHOR> Exactly, especially now with yen being so expensive. For the past 6 months I've spent $29000 on candy and $26000 to ship it. If I used airmail with tracking it would cost 3 times as much.

<comment><USER20> How come I can order something fairly heavy from China for $3 with free shipping? Of course they do WAY more volume than Japan, but it still makes no sense, it costs me more than that to ship it across town, and it's literally the same guy doing the local delivery.

<comment><USER19> It really depends on the company you were dealing with. They almost certainly have someone stateside who are dealing with customs, picking up containers and unpacking them. With enough volume, it's quite doable.

Basically, for shipping by sea, weight is barely important. Volume is the most important thing. I forget how much it costs to ship a container (like those big containers you see at ship yards or on trains), but essentially it is little enough that cross atlantic/pacific is essentially free. If you have enough to fill up a container and you have someone who can do the paperwork on the other end, it is totally feasible.

I shipped my bicycles from England to Japan on a partial palette. I can't even remember how much I paid... Maybe $100. And that's for a third party shipping company that came to my door, wrapped my bike boxes, put them on a palette and found space in an unfilled container. The hitch? I had to go to the docks in Japan, fill in the paperwork, do the customs inspection, etc. It took me a whole day. To have someone else do the work for me on that side would have cost $1000.

If you can't receive large shipments, then they have you over a barrel and they will suck the money out of you. Probably there is a business opportunity there...

<comment><USER21> _I forget how much it costs to ship a container_

Order of magnitude: thing "new iPhone" not "new MacBook." If that leaves anyone flabbergasted, yep, that is why containerization is one of the most important technologies in the world.

<comment><USER22> And the current price may be even more surprisingly lower because we live through a period of shipping capacity glut, see the bankruptcy of Hanjin Shipping.

</comment></comment></comment></comment></comment></comment></comment><comment><USER23> > From this it seems enough to only test combinations where two edges align.

That can't be correct in general. Imagine a 9x9x9 cube, and nine 3x3x1 squares. This can fit into a 9x9x10 box, with the squares all on one side of the big cube. However, four of the squares will share only one edge with the cube, and one of the squares will share no edges with the cube.

<comment><AUTHOR> Like this? [http://imgur.com/a/61EnF](http://imgur.com/a/61EnF)

I probably misunderstood, but it seems to work. "Align with edge" I meant aligning with any edge of any box which is already in play.

<comment><USER23> Hmm, as long as you don't always use the big cube as the baseline, that works.

<comment><USER24> There are still cases it won't work; in fact, sometimes the optimal packing is pretty strange. For example, check the fifth optimal square packing: [https://en.wikipedia.org/wiki/Square_packing_in_a_square](https://en.wikipedia.org/wiki/Square_packing_in_a_square)

<comment><AUTHOR> Wow, that's a more bizarre case than I would have imagined. I started with the assumption that rotating them other than multiples of 90 degrees would never help, but looks like I was wrong.

<comment><USER25> Well to be fair, packing a physical good in that type of arrangement would often cause breakage of the diagonal object. So establishing flush edges seems like a good assumption here.

<comment><USER24> There's also an upper bound on how bad an axis-aligned solution can be. And, on a less mathematical note, weird diagonal packings are counter-intuitive to everyone and you're not likely to get customers complaining that you packed inefficiently just because you missed an opportunity to use one.

</comment></comment></comment></comment></comment></comment></comment><comment><USER26> There was a Kaggle competition on this subject a couple of years ago, plenty of interesting solutions : Packing Santa's Sleigh - [https://www.kaggle.com/c/packing-santas- sleigh](https://www.kaggle.com/c/packing-santas-sleigh)

</comment><comment><USER27> Our company runs an SaaS shopping cart and we had to work in similar algorithms to estimate the size of the package when two or more products are packaged together, we would then send the sizes and weight of the total package to the different shipping carrier to get shipping estimates.

</comment><comment><USER28> Correct me if I'm wrong, but this seems like a convex optimization problem, assuming you know the sizes of the boxes and volume to pack them in. I think you could define a set of equality/inequality constraints and solve the problem efficiently. For further reading: [http://www.stanford.edu/class/ee364a/](http://www.stanford.edu/class/ee364a/)

<comment><USER29> It's not. The intersection constraints are not convex. You could use sequential convex programming to find a local optimum.

edit: to back up my claim: the feasible set of a convex optimization problem must be convex. suppose you have two squares of side length 1 at positions x1 and x2. The following two solutions are feasible:

        x1a = (-2, 0), x2a = ( 2, 0)
        x1b = ( 2, 0), x2b = (-2, 0)

but their midpoint is not:

        x1c = (x1a + x1b)/2 = (0, 0)
        x2c = (x2a + x2b)/2 = (0, 0)

therefore the feasible set is not convex.</comment></comment>


Stackoverflow – "Last seen" not accurate? - <AUTHOR> On stackoverflow the last seen appears to be inaccurate as shown in the linked images. The user, ( I chose random user), asked a question 11 mins ago but his last seen is 21 mins ago ??<p>Images: https://imgur.com/a/qP2KUiE <comment><USER2> I imagine the "last seen" times are shown mainly to indicate whether users are still active members of the site, and if they are likely to have seen recent replies. For these purposes, a few minutes' error is harmless.

It wouldn't surprise me to learn that Stack Overflow's developers—like those of many large websites—have decided, as a performance optimization, to allow some information to be cached, and updated lazily, to avoid the cost of providing a globally-consistent, up-to-the-minute view to every user.</comment>


Introducing a new data structure, streams, in Javascript - <AUTHOR> http://streamjs.org/

<comment><USER2> As someone who conjured this exact thing up in another language after reading SICP, I have to suggest that the first two commenters here missed the point: this is a very basic but powerful structure that implements a lazy stream and is nearly a monad (many standard options are written for you but could be generalized).

Want it to apply to data I/O? Stick your reader into the HEAD function, store the pointer to the next byte in TAIL, and implement functions for reading X number of bytes and converting them into their new types.

Yes, you can do math with it. You can also implement process/procedure into it, if you consider yourself popping lambdas off the stream, and which lambda is Next can be guided by choices in the current lambda.

Mostly, you can use this device to hold any collection of things and operate upon them in a streamy-syntax kind of way. For those who are familiar with jQuery, it is a bit of a reimplementation of a lot of jQuery's basic array- data handling with extra helpers like "range" tossed in and a stronger focus on the laziness. Where jQuery's prime target is work with the DOM, this stream package's target is the more theoretical version of some of jQuery's features, specifically those that iterate over the contained arrays. You could wrap jQuery in this and get an amazing hybrid.

Don't sell it short; it really is another useful and powerful way to approach data management. Check out how you construct your own stream with this package, and think beyond the basic numeric approaches.

<comment><USER3> It's hard to escape reimplementing streams after reading SICP, because _Lazy evaluation is the red pill_ :

[http://weblog.raganwald.com/2007/02/haskell-ruby-and- infinit...](http://weblog.raganwald.com/2007/02/haskell-ruby-and- infinity.html)

<comment><USER4> As for "appending infinite lists" I'd like to suggest Control.Monad.Omega: <http://hackage.haskell.org/package/control-monad-omega>

"A monad for enumerating sets: like the list monad, but impervious to infinite descent.

A depth-first search of a data structure can fail to give a full traversal if it has an infinitely deep path. Likewise, a breadth-first search of a data structure can fall short if it has an infinitely branching node. Omega addresses this problem by using a "diagonal" traversal that gracefully dissolves such data."

</comment></comment></comment><comment><USER5> This comment thread reminds me a bit of "The Emperor's new clothes". I think the code for sieve (even though I understood what it did) is way too unnecessarily confusing and I would not want to see it in a project I'm on.

However, I do appreciate the usefulness of lazy evaluation. Still, I think this documentation could benefit by not making outlandish claims ("it will change the way you think") and just saying the author has implemented lazy evaluation in javascript (and maybe link to the wikipedia article on lazy evaluation and this stack overflow to explain why it's useful: [http://stackoverflow.com/questions/2151226/what-are-the- adva...](http://stackoverflow.com/questions/2151226/what-are-the-advantages- of-lazy-evaluation))

<comment><USER6> If your project team has people comfortable with functional programming - there's absolutely nothing wrong with that code.

As far as changing the way you think, it certainly came as a shock for me when I encountered it years ago in SICP and I'm still enjoying the benefits of lazy sequences in Clojure and ClojureScript.

</comment></comment><comment><USER7> I was first introduced to the concept of 'streams' akin to this in SuperCollider, which uses them as building blocks for musical patterns.

[http://danielnouri.org/docs/SuperColliderHelp/Streams- Patter...](http://danielnouri.org/docs/SuperColliderHelp/Streams-Patterns- Events/Pbind.html)

[http://danielnouri.org/docs/SuperColliderHelp/Streams- Patter...](http://danielnouri.org/docs/SuperColliderHelp/Streams-Patterns- Events/Pattern.html)

SuperCollider is a fun language to study. You end up with compositional structures like the following (notice 'inf', which indicates an infinite loop)

        Pbind(
          \octave, 4,
          \degree, PstepNadd(
                Pseq([1, 2, 3]),
                Pseq([0, -2, [1, 3], -5]),
                Pshuf([1, 0, 3, 0], 2)
              ),
          \dur, PstepNadd(
                Pseq([1, 0, 0, 1], 2),
                Pshuf([1, 1, 2, 1], 2)
            ).loop * (1/8),
          \legato, Pn(Pshuf([0.2, 0.2, 0.2, 0.5, 0.5, 1.6, 1.4], 4), inf),
          \scale, #[0, 1, 3, 4, 5, 7, 8]
        ).play;

</comment><comment><USER8> Um, 'new data structure'? You have to be kidding. This is implemented in practically every single functional programming language. Also, this was already done in JavaScript: <https://github.com/Gozala/streamer> and looks surprisingly similar to your library. As you can see, streams in JavaScript are nothing new.

<comment><USER9> Thanks for your feedback. You're quite right. I added some clarifications in the 'Tribute' section :)

</comment></comment><comment><USER10> First of all this is very nice. (I'm going to get a little negative shortly; I want to be clear that I'm not putting down the writer's efforts at all.)

Several commenters have compared these streams to Python generators. Yes, the two are similar. But there are are a couple of important things Python generators have, that these don't.

(1) Python generators, along with lists, tuples, files, net connections, etc., all support the standard iterator protocol. This means that, for many purposes, you don't need to know or care which kind of data source you are reading from. The itertools functions can be applied to all of them, etc. In contrast, these JS streams are dealt with via specialized member functions. You can't just take array-based code, pop in a stream, and expect it to work.

(2) The iterator returned by a Python generator is a wrapper around a single running computation, which can spit out multiple values. Retrieving a new value is not, at heart, a function call, but rather the continued execution of a coroutine that has already started. This has many advantages; for example, we can write a Python generator that produces an infinite dataset, using a simple loop. I don't think that works here.

Now, as for (1) above, I think this is something we could do in JS. It would just be a matter of everyone agreeing to support a particular interface.

OTOH, it seems to me that JS does not have the functionality to support (2). Python does. So do Ruby, Haskell, and Go -- each in its own way. But JS does not, AFAIK. Am I wrong? And should we be building support for (2) into JS and other languages?

\-----

BTW, I find these streams to be pedagogically interesting. Unlike, say, Haskell lists, the thunks in the data structure can be stored explicitly. It's a nice exercise in the actual implementation of a lazy sequence.

<comment><USER9> Hi ggchappell. First of all, thanks for your feedback. You're quite right about (2); but there's a new Javascript version coming out soon that proposes generators and 'yield' similar to how Python uses them. When it comes out and is supported by most browsers, we can probably implement that (or just use native syntax and get rid of stream.js altogether).

As for (1), I'd be happy to hear a proposal. If you'd like to work on this, feel free to submit an issue or even a pull request on github — suggestions like this are very welcome. Thanks! :)

<comment><USER11> Here's a project that simulates the "yield" keyword with current version of JavaScript: <https://bitbucket.org/balpha/lyfe/wiki/Home>

<comment><USER12> Thanks for this link, I've been looking for something like that.

Generators like in Python allow for nice AI scripting with explicite time- sharing - each AI actor is generator function, that in main loop has yield statement.

That way it all works in one thread, and ai code can be written straightforward, like each actor had it's own thread.

</comment></comment></comment></comment><comment><USER13> am i the only one here who finds the whole "omfg, this is _so_ amazing!!1" tone of this article mildly offensive? i mean, it's not like he has a self- levitating hamster or anything.

<comment><USER14> In his defense, the first time I _really_ understood Sequences after reading The Joy of Clojure, I thought it was awesome too.

Now here's where I really blow your brain: a Sequence is _some stuff_ , _in a particular order_. What about the keyUp event? If I were to listen to that event, it might give me ['T','e','s','t'].

That _too_ , is some stuff, in order. This means, that Events are Sequences. Which means, that all the cool stuff that can be done to Sequences, can be done to an Event too, which is exactly what the Reactive Extensions for JS from Microsoft does (disclaimer: I'm writing a book on this).

Haskell of course has this too, it's called the Continuation Monad.

<comment><USER15> Also see the paper "A Concurrent Window System" by Pike (89) which talks about representing keyboard and mouse events as items on a non-finite potentially- blocking channel. And Newsqueak almost certainly isn't the first.

</comment></comment></comment><comment><USER16> Sounds like generators in python, but implemented in javascript. The head element and tail function was a interesting way to view them.

Generators a great way to do processing on a lot of data, in a very list-y way, without having to read all the data into memory first. It allows you to start seeing results straight away.

</comment><comment><USER17> The advantage of "streams" is that it can be used to elegantly define infinite sequences recursively.

For example, you can define powers of two as:

       var powersOf2 = new Stream(1, function() {
         return powersOf2.scale(2);
       });

(untested) which means that the sequence of powers of 2 begins with 1 follows with the the same sequence scaled by 2!

The implementation in the linked article however seems to be flawed, since it doesn't store the node's tail after computing it. This means that to get each element of the sequence you have to compute all the elements before it (even though you already did in order to get to that element).

This works not only for numbers but for other infinite sequences such as expressions in a language. I recommend the book Higher Order Perl if you're interested in this.

<comment><USER18> MJD's Stream module from HOP can be found here: <https://metacpan.org/module/HOP::Stream>

</comment></comment><comment><USER19> Check out node-lazy, it does something similar and more nicely: <https://github.com/pkrumins/node-lazy>

        var Lazy = require('lazy');

        Lazy.range('1..') // infinite list of integers (see readme)
            .filter(function (n) {
                return n % 5 == 0
            })
            .take(5)
            .join(function (result) {
                console.log(result)
            })
        ;

Output:

        [ 5, 10, 15, 20, 25 ]

<comment><USER9> Hi pkrumins. Thanks for the feedback. I added a link to node-lazy in the 'Tribute' section of the stream.js tutorial. I like its syntax, and I'm sure many people would prefer it :)

<comment><USER19> Awesome!

</comment></comment></comment><comment><USER20> It'd be great if the websites for JavaScript libraries like this one would actually include the library in the page, so that while I'm reading about it, I can open up the JavaScript debugging console and actually try out the library as I'm reading about it.

<comment><USER9> Thanks, that's a great idea! I made it load up the library and added a note in the tutorial that one can use their browser JS console to fiddle with it :)

<comment><USER20> Thank ye!

Here's another example you might want to include, if you're going for the whole Haskell angle. Not as elegant, but illustrates the concept:

      var fibs = new Stream(1, function() {
          return new Stream(1, function() {
              return fibs.zip(function(a, b) { return a + b; }, fibs.tail());
          });
      });

      fibs.take(10).print();

<comment><USER9> Great example, I've added it. Thank you! :)

</comment></comment></comment></comment><comment><USER11> You can create lazy enumerables with linq.js as well.

<http://neue.cc/reference.htm>

        Enumerable
          .Range(0, 1000000000000)
          .Take(10)
          .ToArray()

Output:

        0,1,2,3,4,5,6,7,8,9

<comment><USER21> There is also "LINQ for Javascript", which is fully integrated in with JQuery and VS as well. <http://linqjs.codeplex.com/>

Streams seems to be a proper subset of the things you can do with linqjs. Sometimes MSFT can do cool things too, it just doesn't always get picked up :)

<comment><USER11> Actually, we're talking about the same project. I linked to its reference page for the examples.

</comment></comment></comment><comment><USER22> This is nothing like a real byte stream, and is very misleading. The name implies that it's for data i/o, when in reality it's for mathematics.

<comment><USER23> Agree, i thought the name is awful as I expected to see something related to I/O and buffering of content. It looks like a handy list object and seeing "stream" everywhere was just blinding.

However, right at the bottom the etymology of the name is made clear:

> The name 'stream' is used in Scheme, a LISP dialect that supports these > features.

So the name is from the FP community. Given the cultural issues, a name change seems unlikely...

<comment><USER24> Here's a helpful two-part Berkeley lecture on the topic: <http://academicearth.org/lectures/streams-i>

<comment><USER25> I didn't know that series of lectures, so thanks a hundred times. (Just spend the evening watching the first two hours and bought Harvey's book as well.)

</comment></comment></comment></comment><comment><USER26> Maybe I don't get it, but what is the practical use-case for this? Infinite arrays of numbers - maybe useful for math or statistical applications, but are people using javascript for this over things like MATLAB, R, etc?

I can't think of anywhere to use this in the context of building applications and I certainly don't feel like it "completely changed the way I think about programming" at all.

Edit: So it looks like you can put more than just numbers into the streams -- still what does this do that Underscore.js doesn't?

<comment><USER27> Consider an indeterminate array holding all the input data or events your program will receive in the future.

</comment></comment><comment><USER28> Here's the Python equivalent, for what it's worth: <http://www.trinhhaianh.com/stream.py/>

I've been playing around with this in my current project, yet I'm not convinced this is always such a great model. It requires a lot of mental work to remember what types you are operating on at each state in the pipeline. That being said, I'm a big fan of LINQ, and I'd like to see this model applied more frequently to DB queries.

<comment><USER29> Doesn't "stream" already mean something like "unix pipe" or "queue"? What about calling them Sequences like Clojure does?

</comment></comment><comment><USER6> For a high-powered version of this written in CoffeeScript which works over arrays and lazy sequences (streams) look here [https://github.com/swannodette/fun.coffee/blob/master/src/fu...](https://github.com/swannodette/fun.coffee/blob/master/src/fun.coffee)

ClojureScript of course takes it even further.

</comment><comment><USER30> "a new data structure" ... really?

</comment><comment><USER31> For anyone who wasn't paying attention, according to Brendan's slides for TXJS[1][2], TC39 approved iterators and generators for ES, so soon _all_ modern browsers will support them. (Spidermonkey has quietly supported them since Firefox 3—that is, summer 2008.)

1\. <http://brendaneich.com/2011/08/my-txjs-talk-twitter-remix/>

2\. <http://news.ycombinator.com/item?id=2982256>

</comment><comment><USER32> I'm used to calling these "Generators", as in a "Generator Function". We could use Forth's defining words to build a pretty succinct version of this type of functionality. Generators will be called by executing their address ("xt" in Forth parlance), and will leave a flag on the stack indicating whether the generator had another element available. If this flag is true, the element produced will be underneath.

To start with, let's create a word called "count" which produces a generator which will count from 1 to infinity:

      : count ( -- 'gen )
      	noname create 0 , latestxt
      	does> ( -- value? flag )
      	dup @ 1+ dup >r swap ! r> true
      ;

Then we build another generator which slices the first N elements of another generator, so that we can do things without going into an infinite loop:

      : slice ( 'gen count -- 'gen )
      	noname create , , latestxt
      	does> ( -- value? flag )
      	dup @ 0= if drop false exit then
      	dup dup @ 1- swap !
      	cell+ @ execute
      ;

Finally, let's implement "map", so that we can apply a word to every element produced by the generator:

      : map ( 'gen 'func -- )
      	2>r
      	begin
      		i' execute
      		if   i execute
      		else 2rdrop exit
      		then
      	again
      ;

Now the Forth stack gives us a nice, fluent way of chaining these building blocks together. Let's start with that infinite counter, slice off the first 10 elements and print them out by mapping them to the number printing word ".":

      count 10 slice ' . map

And we get:

      1 2 3 4 5 6 7 8 9 10

Spiffy, hunh?

</comment><comment><USER33> tl;dr; = It's an array with a few utility methods, but can also hold 'infinite' things like "all the positive even numbers".

Not terribly useful imho and certainly doesn't live up to the hyped intro.

<comment><USER34> Streams and other lazy infinite data structures can be very convenient, but this isn't really obvious at first glance. I recommend you check out some of the FP literature. A good place to start reading about the virtues of streams and lazy evaluation is "Why functional programming matters" (available at <http://www.cs.utexas.edu/~shmat/courses/cs345/whyfp.pdf>).

<comment><USER35> It's a good concept and all, the issue i've had with it for my own use is that the native Array functions are so fast in JavaScript (and function invocation so comparatively slow) that you really need a very specific use case for them to be worthwhile over simply performing your operations on the array directly like Underscore.js.

</comment></comment></comment><comment><USER36> Hope this doesn't seem like threadjacking, but your README looked like a fun exercise, so I decided to avoid looking at the code and re-implement in Coffeescript. I wound up memoizing while I was at it.

<https://github.com/MichaelBlume/coffeestream>

<comment><USER9> Michael — that's very cool. I added a link to your library in the 'Tribute' section. Thanks for taking the time to port it :)

<comment><USER36> Awesome, thanks for the link, and for inspiring a happy afternoon's coding =)

</comment></comment></comment><comment><USER37> Just curious -- and be honest -- the snippet at the end they ask you to decipher (and they say "Most programmers find it hard to understand unless they have a functional programming background, so don't feel bad if you don't get it immediately") did you have trouble with it?

For me, I thought I had the answer after maybe 30 seconds of reading it, but decided to "check my work" by walking through it again, and only then, a moment later, did I realize what it was actually doing with that recursive call.

I'm just curious about their "most programmers without a functional background" remark. So: did ya get it?

<comment><USER38> Yes, but it would have been significantly more baffling if the word "sieve" didn't hint at its behavior.

Still, I'm not sure that a functional background would have helped, or maybe my background is more functional than I think. I know some Python, and I read a few articles about functional programming a couple of years ago, but I don't think I could give you a clear definition of what functional programming is.

</comment></comment><comment><USER39> Here are some key aspects of Streams:

\- Streams are an alternative to generators (yield)

\- Streams can be used to traverse recursive structures

\- Streams implementation only require lambda functions with proper closure

\- Node.js continuation passing style is a form of Stream

I've implemented streams using continuations. You can also see a bunch of related functions such as: print, map, filter, reduce, range, enumerate, comprehension, traverse, empty, cons, zip, xrange, generator2stream.

[http://blog.vjeux.com/2011/javascript/stream-lazy- iteration-...](http://blog.vjeux.com/2011/javascript/stream-lazy-iteration- with-continuation.html)

Tell me if you have any remark :)

</comment><comment><USER40> I enjoy using the wonderful lazy list implementation here: <https://github.com/pkrumins/node-lazy> IMHO, it has abetter interface than streamjs

</comment><comment><USER41> Embracing functional programming in Javascript is trending. All these dollars, underscores and streams will make for an interesting benchmark against ClojureScript as it matures.

In ClojureScript, streams or lazy lists are an integral language feature. Standalone javascript libraries can not provide a functional framework of the same calibre. You would naturally end up wanting to, e.g., run an underscore.js chain on a stream and that would require more work.

</comment><comment><USER42> The guy just finished reading that chapter of SICP? ^_^

<comment><USER9> Hi schiptsov. Not far from the truth, actually! I just finished watching the online video lectures on the MIT OpenCourseWare page. I already knew about infinite lists and streams from Haskell, but their lectures inspired me to make a Javascript port. Their examples are quite nice too. Your comment made me think I should add a link to their book, so that's now included in the 'Tribute' section. Thanks! :)

</comment></comment><comment><USER39> I've written an article that talks about the same beast: [http://blog.vjeux.com/2011/javascript/stream-lazy- iteration-...](http://blog.vjeux.com/2011/javascript/stream-lazy-iteration- with-continuation.html)

However, I use Javascript functions as base instead of a Stream constructor.

</comment><comment><USER43> I am sorry but I found the tone of the article a bit offensive. You are not coming out as playful or funny, but as arrogant. Drop the attitude mate, try the guitar if you want to be a rockstar. Apart from that, this is a nice example of how versatile the language can be.

<comment><USER9> Hi BlehBlehBleh. Thank you very much for the feedback. I'm sorry it came out as arrogant — not my intention at all. I tried to make the article enthusiastic in a way that captures the reader's attention to read a bit further and see the interesting things that are possible with this library. The language reflects my own enthusiasm I felt when I first learned about streams and lazy data structures.

Based on your suggestion, I've added a few clarifying comments, especially in the 'Tributes' section to show that this isn't my own brand new idea but is based on other people's work; maybe this will help. Thanks :)

</comment></comment><comment><USER44> I couldn't get past the 2nd paragraph. the guy's infomercial salesman writing style was hurting my head.

</comment><comment><USER45> On the surface this looks like a port of Scala's Streams object to Javascript. Any difference?

</comment><comment><USER46> Seems like they (whoever they is) should add this to Javascript by default.

<comment><USER9> Dilon, glad you liked stream.js. There's actually a plan to support Iterators and Generators in one of the upcoming versions of Javascript: [https://developer.mozilla.org/en/JavaScript/Guide/Iterators_...](https://developer.mozilla.org/en/JavaScript/Guide/Iterators_and_Generators)

</comment></comment><comment><USER47> So how is this different from an array?

<comment><USER48> It is infinite, so you can for example have an stream of _all the prime numbers_ (only those you actually use will be computed, of course; the point is some algorithms are easier to express using this abstraction).</comment></comment>


Why I replaced MIT with copyleft license for Nodemailer - <AUTHOR> https://blog.nodemailer.com/2017/02/02/nodemailer-v3-0-0/ <comment><USER2> Donations for open source projects have notoriously low participation rates - to the point where popular and critical infrastructure projects can't afford to maintain a fulltime employee or two - so don't take it personally.

If you want companies to pay you, then ask them to pay you. Keep it as a simple license (not one that has devs asking wtf it is), but offer a commercial bundle with email and dev support, LTS, backwards compat and perhaps authentication modules (for LDAP/AD, OAuth) and I think you'd be surprised by just how many companies make these purchases (I make them all the time).

Also, on price - $100-300 is "no-brainer i'll put it on my credit card and expense it at the end of the month" while €800 is approaching purchasing decision territory for a lot of companies.

<comment><AUTHOR> The high pricing is intentional. If no-one is buying then I can just bring the price down, while it is really difficult to move on the other direction if I start with a low value.

<comment><USER3> A good choice, imo. You can now offer "sales" and special discounts for people who ask for them.

Might want to also offer a premium release for 10x or even 20x, since that falls into the "No Brainer" category for larger corporations who would look at 800 as a sign of low quality.

</comment></comment></comment><comment><USER4> Smart guy. Copyleft for the open source crowd to use freely, commercial license for businesses who want to keep their own source closed.

I wonder where you get a good commercial license to use like this. Is it always a custom thing requiring expensive lawyers or is there some repo of commercial licenses one can use?

<comment><USER5> Open source software developers probably won't want to touch this with a bargepole. Oddball copyleft licenses are a pain in the ass to deal with an invariably incompatible with every other copyleft license including the GPL.

<comment><USER6> Came here to say much the same. GPL / non-MIT generally = avoid, because IANAL and don't want to figure out if I'm in the right or wrong.

</comment></comment></comment><comment><USER7> It's a great piece of software. I use it in two projects both commercial platforms though not particularly big money makers.

The problem for me is I remain somewhat uncertain about the implications. If I use this within a service that requires payment from customers, it seems to me there is no change. And most software using this is likely to be delivered as a service right? Maybe I misunderstand the license.

In a way if the approach was more aggressive it would be easier for me. I could just go to the people that write the cheques and say: we use this software. New version requires payment. Write a cheque.

As it is I'm not sure whether we have to pay or not. As for donations - well I could make one personally (in fact I will) but it is unlikely to be 780 euros...

<comment><USER8> It looks very LGPL-like in spirit. It's hard to know if you can use it in something commercial or not, though. The pessimistic interpretation is that if you use it your software becomes a "derived work" of the library, in the same way your Harry Potter fan fiction is a derived work of Harry Potter. You probably don't want your commercial software under this license because it commits you to (among other things) granting royalty-free licenses to all patents, and settling all court cases in the jurisdiction of the European Court of Justice.

The optimistic assumption is that no, that would never happen, your product will not be a derived work because using the library with it is "mere aggregation." But the license doens't clearly spell out what is a dervied work and what is not-- there is not even the discussion of mere aggregation that the GPL has.

The license also has the "death penalty" clause that the GPL does, which means that your license to use it can be permanently terminated if you are found to have violated any part of it.

So basically, I would say, stay away. And people in open source projects should stay away too, for the same reasons... you don't want this fighting it out with the existing open source license of your project.

<comment><USER9> At work, if I see a product that offers a commercial licence and an copyleft licence, I immediately buy it.

The licence costs are always trivial compared to the department's revenue and budget, and it completely avoids any hashing over "will we violate copyleft if we do X". Now the answer is always "we bought it, its ours".

At home, I just use the copyleft version. My personal blog engine is open source, and on github Anything else I make, I don't mind open sourcing if it comes to it.

700 Euro a year is trivial. We've spent more money _on drinks_ while talking about licensing our own product.

<comment><USER10>
      Now the answer is always "we bought it, its ours".

I didn't realise commercial licenses are always so generous and easy to use.

<comment><USER9> I'll clarify. I mean that software that's dual-licenced, usually grants unlimited usage within the organization. Essentially you bought it, so it's yours to do with as you wish---though you can't duplicate it and give it to others on its own, you have to bundle it with your product.

If the licence is more restrictive than that, then it is likely the business will fail because if we have to jump through hoops, we might as well (a) use another product, cost isn't a factor or (b) use the copyleft licence and hire lawyers.

</comment></comment></comment></comment></comment><comment><USER11> I understand the switch to copyleft, but the EU's own website explicitly lists GPL licenses as incompatible, in the matrix on this page:

[https://joinup.ec.europa.eu/software/page/eupl/eupl- compatib...](https://joinup.ec.europa.eu/software/page/eupl/eupl-compatible- open-source-licences)

Dual-licensing under the EUPL and GPL, with the option to choose either, would be helpful. Otherwise Nodemailer is incompatible with the majority of other copyleft code.

<comment><USER12> Looking closer, you can't incorporate GPL code in a EUPL project, but you can incorporate EUPL code in a GPL project. The EUPL explicitly is allowed to be used as part of GPLv2 project, and indirectly as part of a GPLv2 project via the CeCILL v2 licenses which has a similar provision.

</comment></comment><comment><USER13> Hmm. A wee bit off-topic, but is there a license of these sorts that do the "free until you make X revenue", similar to those used by Unreal and Unity?

</comment><comment><USER14> andris9, you mentioned that the EUPL was more European Union friendly, similar to how GPL is more USA friendly.

Do you mind elaborating what you mean by this?

<comment><AUTHOR> IANAL but as EU citizen doing stuff in EU I don't really care about the implications of US patent system. What I do care about is a license that is in a local language, designed specifically for the EU jurisdiction so every court would understand what its exactly about. In general though I don't think there's a huge difference as it's hard to believe that I would actually ever go to court based on some license terms.

</comment></comment><comment><USER15> Spoiler: Money.

<comment><USER16> So why not just close it and sell it then?

Using a license to extort money from commercial entities who he seems to say use it without paying a dime for it and operating under the guise of open source is... warped in my pov.

I don't begrudge him making phat loot. But if that's the goal open source is not the right avenue imo.

Let the flame war begin but I don't see the point in open sourcing code if you don't want people to use it however they want.

YMMV and probably does.

<comment><USER17> Under a copyleft license you CAN use it however you want. You just are obliged to open source any modification you make to it.

I'm not sure how the "derived works" clause applies to node.js projects using this project.

<comment><USER18> You're obligated to also distribute sources to changes _if_ you distribute.

You can implement single-image clustering on top of the Linux kernel all you want, and keep your changes - so long as you don't distribute the software. If you do, you must distribute under the gpl.</comment></comment></comment></comment>


HipChat Alternatives - <AUTHOR> https://github.com/cjbarber/hipchat-alternatives

<comment><USER2> I don't understand why even most of the open-source projects aren't using a modern standard like XMPP, and simply focusing on the UX.

Is there a technical reason for it? Is XMPP broken in some way? Or is the chat space simply full of pathological cases of Not-Built-Here Syndrome?

<comment><USER3> Does XMPP support inline photo and video? Last I checked, it didn't. So why deal with the external dependency when you can re-write it yourself with additional features?

<comment><USER2> It does according to Wikipedia, although "Binary data must be first base64 encoded before it can be transmitted in-band", making it inefficient[1].

XMPP's based on XML anyway, so implementing inline media and extra features sounds easier than rolling an entirely new protocol.

But perhaps some of the proprietary systems use XMPP internally anyway? After all, it's used for more than just chat clients nowadays.

[1]: [https://en.wikipedia.org/wiki/Xmpp#Weaknesses](https://en.wikipedia.org/wiki/Xmpp#Weaknesses)

</comment></comment></comment><comment><USER4> Slack is amazing. We dumped HipChat which had been in beta for 4 years! Slack had good iPhone integration (HipChat would bot give me notifications?!) Really useful integration with Trello.

</comment><comment><USER5> Flowdock has changed the way that we do things - it is so much more than chat.

</comment><comment><USER6> Haven't personally tried it but maybe check out Slack

<comment><USER7> Everyone has been raving about it. Just raised $42 million today too.

</comment></comment><comment><USER8> what about [http://gitter.im](http://gitter.im)?</comment>


The Cost of Waterloo Software Engineering - <AUTHOR> http://blog.petersobot.com/the-cost-of-waterloo-software-engineering <comment> why-el I can testify that this (systematically alternating between school and work throughout your undergraduate studies) makes you a better engineer. I learned about it when I moved to Germany, and it shows in some of my colleagues who chose to do it.

This is not an option in a lot of countries, including my own (Morocco). The closest thing to this system that I did was to convince my computer science department to let me take a long pause from school (January to September) and move to Brazil where I found a good tech company that paid me for my ridiculously long internship. I came back with a very clear idea of my strengths and weaknesses when working on large and critical software systems, and I designed my last semester around these new insights.

Granted, most people achieve similar goals through traditional internships routes, but it helps to be able to see you work live on beyond summer and be able to identify medium term regressions, follow on on other use cases, and account for the fact that other people will be taking up the torch soon.

</comment><comment><USER5> I find HN fetish with UoW to be interesting , which has been my data point to be very skeptical about believing everything I read here.

1) UoW has a really low requirement for entry, of my class many students went to study at UoW most of them had average to below average grade, compared to Oxbridge UoW has a joke of an entry requirement, I later learned they have a very high dropout rate after first year - which explains things, but I think is disingenuous to their students. School should be a place of learning not a lab for natural selection.

2) I drew a spreadsheet for Unis across the global all from singapore,UK,US and Canada. Any engineer graduating with an average GPA working full time for 1 year and having done internships in uni will have earned more money than what this links shows.

3) The strong internship requirement means students who cannot find one within four months of joining uni is likely to fail his semester even if he is academically gifted. My UoW friend failed a year due to this stupid requirement, in most other universities it will not be an end to your stay if you fail to get a job within the first 4 months.

4) UoW has really bad reputation in the academical world and ppl doing basic research ( look up UoW ranking )

5) Their fees are only competitive if you are from canada, they are much more expensive for international students compared to elite universities in the UK.

6) Universities historically should have weak ties with industries. The reason for this is independence. I know for a fact that UoW has strong ties with microsoft in the past and used to push .NET stuff on their students ! how do you expect a university to be intellectually honest if they are attached to a corporation ?

I could add more but I am running out of time.

<comment><USER6> > _" 3) The strong internship requirement means students who cannot find one > within four months of joining uni is likely to fail his semester even if he > is academically gifted."_

This is unequivocally false. You seem to be very confused about how UW co-op works.

Firstly, since you seem to be going at this from a CS academic angle - <USER2> students are not required to join co-op, and can get a traditional 4-year CS degree without finding a single internship, ever.

Only engineering disciplines are required to join co-op, and for good reason - traditional engineering (mechanical, electrical, chemical, etc) have academia and industry practically fused at the hips. An engineering education without industry experience frankly doesn't make sense.

More than that though, a UW co-op student has 6 internship semesters in their degree program, out of which the student must be employed for 5 of them. This means all students get a "freebie" if they don't have an internship - <USER3> voluntarily or involuntarily.

There is no scenario under which a student will fail "within four months of joining uni" \- absolute worst case scenario they spend their freebie term.

For CS students the absolutes worst case if you are unable to find internships repeatedly is leaving the co-op program, and losing nothing academically.

> _" My UoW friend failed a year due to this stupid requirement, in most other > universities it will not be an end to your stay if you fail to get a job > within the first 4 months."_

I suspect there's more to your friend's story than you know. At UW it is most certainly _not_ the end of your stay if you fail to get a job for your 1A term in Stream 4/4S (i.e., within the first 4 months).

<comment><USER5> My friend is studying mechatronics in waterloo. In the link and also my friend told me that they attach university credit for your co-op. My friend wasn't able to gain employment within 4 months of joining uni and also failed to gain one in the next semester ( even though he did relatively well on his exams ). So he didn't have enough credit to move on to the next year. The fact they attach such an arbitrary requirement to your academics is what causes me concern.

<comment><USER7> My experience hiring from Waterloo co-op is that it is both well run and supports even the most junior students well.

The systems design and mechatronics programs there are very strong. Given that your friend was in a competitive engineering program aimed at putting graduates into good industrial positions, the inability to find a position in two terms is probably a good sign they needed to re-evaluate if they were in the right program for them.

There is nothing arbitrary about this, and they communicate it quite well. It sounds like the system was working as designed, although I understand how this could have been a rough time for your friend. Then again, that's one of the things college is for, and it's hardly a unique situation.

<comment><USER6> Agree in general - though I'd stop short of questioning if the person needs to re-evaluate whether or not they should be in the program.

A _lot_ of high school students come into university with no job-finding skills, which are going to be critical to their future success. Job-finding, interview, and general industry skills are hard won through experience.

Like failing a course, failing to demonstrate rudimentary job skills should hold you back until you can gain these skills. For some people this takes longer than others (1A + 1B unemployment at Waterloo is somewhat rare), but repeatedly being unemployed is not necessarily a sign that someone doesn't belong entirely.

Knowing little about his friend, my first reaction is that maybe he should've stuck it out. Waterloo is a tremendous place to gain job skills - there is no more fertile ground to test resumes and interview skills at volume than it - <USER4> personally had over 70 interviews under my belt by the time I left that school.

<comment><USER5> I agree with you. It wasn't that we were lazy, there were literally no opportunities to gain employment in high school for us. For example in my case I landed an internship within one month of joining University even though I was far away from silicon valley, they only used my academics and I turned out to be really good for them. Same cannot be said for canada where its a lot more competitive. I learnt a lot without having to take any risks as my uni life was independent of my proffesional life.

<comment><USER6> > _" Same cannot be said for canada where its a lot more competitive. I learnt > a lot without having to take any risks as my uni life was independent of my > proffesional life."_

I disagree strongly. Literally thousands of 1A students at Waterloo get placed every single semester, and while 1A unemployment rates are higher than any other term, still only a minority of students cannot find a job that term.

And nearly everyone finds something by their 1B term, putting them nowhere near at-risk from being held back.

Finding someone willing to let you intern as a kid fresh out of high school and only a few months into university is _hard_ , but it is not insurmountably hard. This is an opportunity, not a burden - you gain _far_ stronger job- finding skills than you would if you already had an impressive background.

Waterloo taught me more about hustle, presentation, honesty, spin, negotiation, and many other skills I'd never have learned had I already had strong credentials. The experience has made me a much, much stronger interviewee than most of the population, and has helped my career in huge ways.

A friend of mine literally drove down highway 401 with a pile of resumes, stopping at local factories and engineering shops to personally hand it to a hiring manager. This is the sort of desperation - but also hustle - that most people are never faced with, especially in our field. It is also the type of hustle that keeps paying even after you have a long and impressive resume.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> It might be obvious to American readers, but I had not idea that University of Waterloo mentioned here was in Canada, and not in Belgium.

<comment><USER9> Yes, in Ontario Canada. It's home of BlackBerry (the smartphones company) and a hotbed for startups. Google has an office downtown in downtown Waterloo.

<comment><USER10> It is in downtown Kitchener; the city beside Waterloo. (That doesn't change the semantics, here.)

(Waterloo's downtown is branded "uptown". Urg.)

</comment></comment></comment><comment><USER11> Dang, I was honestly expecting a Napoleonic analogy for software-design / project-management

<comment><USER8> I was also expecting some death march story. Interesting nonetheless.

</comment></comment><comment><USER12>
        After 5 long and difficult years,
        I'm extremely proud to say that
        I'm a Waterloo grad

Every Waterloo grad I've ever met has gone through their coop program, which adds an extra year to their undergrad degree: [https://uwaterloo.ca/find-out- more/co-op/how-co-op-works](https://uwaterloo.ca/find-out-more/co-op/how-co- op-works)

<comment><USER7> If I recall correctly it is required for all engineering degrees, and quite popular in programs like CS as well.

From the hiring end, it works well. These graduates are typically very competent.

<comment><USER13> It makes me a little jealous as a Carleton student.

<comment><USER14> uOttawa grad here: is there any co-op option at Carleton? It wasn't very popular at uOttawa, to the point that my employers were getting Waterloo grads because they couldn't find anyone local. There's a lot of cool companies in the capital (not just Shopify!), I'd recommend trying to find a position while you're still a student.

<comment><USER15> While not a 1:1 replacement, you could consider applying to jobs for the summer on your own. I've met a lot of people in California from the University of Toronto who've done so.

</comment></comment></comment></comment></comment><comment><USER16> I went through an EE co-op in UofGuelph, another university in the area. It was harder to get positions without the local prestige of the Waterloo brand, but it was still a great experience that convinced me that I liked software more than hardware.

</comment><comment><USER17> The Case for and against U Waterloo

I did my undergrad as well as my master's in Mechatronics at UWaterloo. Thought I'd pitch in to shed some light on two of the key themes of this conversation, Co-op and Academic research.

1\. Co-op - The Waterloo 4 month alternating Co-op program has its pros and cons. On one hand you get the chance to go out and sample a bunch of different kinds of jobs (if you proactively choose to do so), travel, fund your own education without drowning in student loans and get a taste for what engineering outside the classroom might be like. Currently with a large number of start-ups in Waterloo a lot of Co-op students work with start-ups which, for this forum, should undoubtedly be a good thing.

On the other hand, the obsession with co-op chips away from the culture of learning that a University should harbor. I remember freaking out about jobs, interviews and my resume from the 3rd week of classes in first year. It seems to be a common theme here. You spend most of your actual time on campus, preparing to get off campus and it creates less of an impetus to pursue deeper learning and research. That being said, as with everything in life, you get out of it what you put into it.

2\. Academic Research -

Most schools, In my opinion, have good pockets and bad pockets. The fact that the model of scientific research worldwide is pretty broken is well known. Industrial partnerships drive research because that's where the money is. At least until we all become millionaires and donate money back to our Universities for no-strings-attached research :D It also depends on the professor. Some professors choose to fund themselves off government grants that give them less money and more freedom while others choose to work with large companies like GM on industrial research projects. You need to find the professor that aligns with your interests. I was lucky enough to work on robotic path planning and multi-robot cooperation without any industrial interests breathing down my neck but that doesn't always happen of course.

The good thing about a Waterloo master's is that it is guaranteed to be fully funded by the University and, based on merit, you can choose to do either a course-based master's or a thesis-based master's. This is unlike American universities because we drive our own research for the entirety of our program. Being just a master's student I've published as a first author in most of the major robotics conferences and journals (ICRA, IROS, JFR, TRO, WAFR) and not many master's students in other Universities have the opportunity to do that.

</comment><comment><USER18> Drexel University has a similar program requiring five years to complete. However the co-op's are for six months at a time which is infinitely more useful to both the student and employer then more frequent shorter co-op's.

<comment><USER19> How does that work given that standard school terms are 4 months long and the lcm of 6 and 4 is 12? You'd have to bunch up consecutive co-ops or else have dead time to resync with the rest of the school.

Edit: fail math x2

<comment><USER18> At Drexel the school terms are three months. So each co-op is two terms. The normal cycle is two three month school terms one six month co-op.</comment></comment></comment>