The Word Count Problem - <AUTHOR> http://blog.peterdonis.com/opinions/still-another-nerd-interlude.html <comment><USER3> I want to be charitable but I find the criticism here of Knuth of be quite childish. Was the point to solve the problem with as little work as possible? If that's the case, there's probably some plugin for your favorite text editor that already does this. So what? What was the point exactly? You could just have easily built a library function that does all of this and then write a Pascal program that calls the library in one line of code, state the library is not part of your solution, and say the solution is better than the one with UNIX utilities. So what? Is the point to illustrate that Knuth is unaware of elementary concepts such as code reuse, come on that is just sort of ridiculous.

If you had asked Knuth to solve this problem using any means in the whole world and do it within 10 minutes, I don't think he would have any trouble doing it.

And honestly, if the lesson is to teach us some really basic lesson of code reuse, not reinventing the wheel, or some such like that, I think this was a pretty convoluted way to deliver sort of a common sense lesson.

<comment><USER4> Some other points from the linked-to article that back up that the criticism here is a bit ridiculous:

1) This was in 1986. That's about 4 lifetimes ago in computer programming.

2) This was an article specifically to demonstrate literate programming. It wasn't supposed to be "the most efficient way" possible. It's approximately the same argument as "why should I write a program to sort data in an interview; I'm not going to get paid to write sorting functions".

<comment><USER5> 1) I'd like to see you say the same about differential calculus, which actually _was_ several lifetimes ago.

2) I have made the same argument, in interviews, many times. It's a good argument to make.

</comment></comment></comment><comment><USER6> Against all conventional wisdom I actually think McIlroy was wrong.

In my experience writing reusable code, unless you're writing a library, is usually a waste of time. Must code, even when written with intent to be reusable -- isn't. Maintainability and readability almost always trump reusability. Refactoring fast the point where reuse is needed typically is the better approach.

McIlroys approach is the less common approach of library/framework writers while Knuth is the approach of app devs.

<comment><USER7> I don't agree -- you're papering over an enormous distinction. Unix tools and the shell are the biggest success story in software reuse that we have. (I would argue that it is why nearly everything we use today is Unix -- servers, iOS and MacOS, Android, etc.)

I somewhat agree when people say "code reuse has failed". When people set out to write "reusable code", it ends up not being reusable. Usually because they think of reuse as writing a bunch of classes that they can "import".

But writing tools is another way to reuse software. For example, in Unix, you reuse ssh for git and for scp. With the web, you can reuse a huge amount of work in Varnish and nginx by chaining components.

So library reuse is not all it's cracked up to be, but that's not what McIlroy is doing. His solution to the word count problem is fantastically and obviously better. It's real reuse.

<comment><USER6> _I don't agree -- you're papering over an enormous distinction. Unix tools and the shell are the biggest success story in software reuse that we have._

To be clear, I think there's a distinction between writing a library for well known use cases. I wouldn't suggest stdio is a waste. But it's almost always a waste of time to emerge from a client project with a bunch of libraries for functionality that will likely never be used again.

And among good developers I think this is one of the biggest problems. I've encountered many times good programmers building elaborate frameworks that take as long as the main project itself, who then present the utility of this framework, which we never use again. They didn't understand the problem space well enough to know when reusability made sense and across what pivots.

 _So library reuse is not all it's cracked up to be, but that's not what McIlroy is doing. His solution to the word count problem is fantastically and obviously better. It's real reuse._

Knuth also reused. He reused a compiler. He reused input output capabilities. He reused subroutine abstraction capability.

IMO it's less what they used and more what they produced (as all developers reuse). Knuth didn't produce a bunch of reusable tools and I think that is justified and the right approach most of the time.

I went back and reread the paper just now and McIlroy even alludes to this:

"The utilities employed in this trivial solution are Unix staples. They grew up over the years as people noticed useful steps that tended to recur in real problems. Every one was written first for a particular need, but untangled from the specific application.

With time they accreted a few optional parameters to handle variant, but closely related, tasks. Sort, for example, did not at first admit reverse or numeric ordering, but these options were eventually identified as worth adding."

What he describes is the right approach -- and what Knuth wrote is the first step here. I'm fairly certain if Knuth was a systems developer and this came up over and over again you'd see refinement and tools/libraries that made certain aspects of this more reusable. But that's neither his domain nor the specific ask for this client.

McIlroy's rant here seems pretentious in light of this. A much better rant would be to show how Unix tools could in six lines replace LaTex for typesetting (without using anything provided by LaTex).

<comment><USER7> I agree with your first point and didn't dispute it. Library reuse is overrated.

But it isn't contradicting the point of what McIlroy does -- it actually _supports_ it. Unix tools are more reusable than libraries full of code.

I can't tell what the second part of your post is saying. It doesn't matter if Knuth "would have" done something; McIlroy _already_ did it. The problem is solved with 6 lines. End of story. No pontificating. That's what Unix lets you do -- get on with your day :)

</comment></comment></comment></comment><comment><USER8> This is fascinating, and should be a source of reflection for anyone writing code for a living or being involved with programming on any serious level.

Additionally, I can't help but think that this essay (and others in this style) and subsequent reflections would make for a perfect seminar targeted towards final year CS students, as it merges perfectly academic reflection and real world engineering considerations, which is what many CS programs across the world sorely need.

</comment><comment><USER5> I think I found a copy of the original Knuth/McIlroy paper itself: [http://people.mokk.bme.hu/~kornai/AzkellHaskell/bentley_1986...](http://people.mokk.bme.hu/~kornai/AzkellHaskell/bentley_1986.pdf)

</comment><comment><USER9> If the problem is simple enough to be solved by a shell script, then obviously literate programming is going to be overkill. Algorithmic problems such as data-structures are well suited for literate programming.

<comment><USER10> Having reusable tools in the first place, to make it a problem simple enough to solve with a shell script, is the accomplishment that advances the state of our art. It reduces a complex data structure to passing data through a pipe, which makes the algorithm more usable for more people.

I wrote in a literate programming style for years, to generate LaTeX documents and a world-scale build system for a multinational. But I'm past that, since it's intricate and unmaintainable by others. Instead, I'd rather generate documents in UTF-8 (an ASCII-compatible and increasingly universal format), and sets of small tools which call each other and expose their data in files for new tools to use. With this approach I can evolve the toolset in small, state-preserving steps, to minimize how much I have to code to meet current needs and implement new approaches.

<comment><USER9> The re-usable tools in this example are only applicable because the problem is simplistic. If you were required to extend the solution to cover say language specific definitions of word boundaries, the literate style would be a definite win. The shell solution looks elegant but really it's just a quick and dirty hack as far as generality is concerned.

<comment><USER10> I'm pretty sure that the trick to tackling most problems in practice is to see the basic problems to solve, and find existing tools to apply. Solve 80% of the problem in 20% of the time, and move on to something else you need to get done.

If the problem warrants it, you can revisit it later to do something more ornate. In the meantime, you've got what you need to move on.

</comment></comment></comment></comment><comment><USER11> "Program specifically designed to demonstrate literate programming turns out to be a good demonstration of literate programming, not a good demonstration of library creation"

... can someone please point out what the problem is? I don't see it :-|

<comment><USER12> The problem is that it isn't what they wanted it to be.

</comment></comment><comment><USER13> I haven't checked out the original paper, but the solution presented on the blog is a non-solution. It only works for ascii characters and words composed out of ascii. When you reuse someone elses code, you always get stuck with someone elses assumptions. In this case, that English is the only language and that the regular expression [A-Za-z] encompasses all possible letters in the world. The shell solution, built upon standard tools, can not be extended to work in an international context but a custom solution in Python (or even Pascal) quite concievably could.

<comment><AUTHOR> > _I haven't checked out the original paper, but the solution presented on the > blog is a non-solution. It only works for ascii characters and words > composed out of ascii._

As I understand it, that was the original spec, which both Knuth and McIlroy wrote to. I agree that it is limited as you say.

> _The shell solution, built upon standard tools, can not be extended to work > in an international context but a custom solution in Python (or even Pascal) > quite concievably could._

As bryanlarsen pointed out, the shell solution can easily be extended by using an internationalized version of tr. The Python equivalent would be to use the built-in Unicode support. (If Pascal had that, you could do the same in Pascal.)

However, it's worth noting that by specifying the problem that way you still have the issue of how the input stream (which is going to be bytes) is encoded. Essentially, the original spec declared by fiat that the encoding was ASCII.

Also, btw, you can express non-English languages in ASCII (though certainly not as wide a variety as in Unicode); the program as written does assume that words are composed only of the 26 standard ASCII letters, but it could easily be extended to include the ASCII special characters. Another exercise for the reader. :-) Though if you're going to do this kind of extension, it might be better just to go the whole way and handle Unicode.

<comment><USER13> But your assertion was that this was an easy problem solved with just a few lines of shell code. That assertion is only true in an extremely limited context where the only language possible is English and the only character encoding is ascii. That's where the Unix tools shines because they are great at handling problems easily expressable as regular expressions. But it is not a very realistic example, nor a fair comparison.

> As bryanlarsen pointed out, the shell solution can easily be extended by > using an internationalized version of tr.

You should try that and then blog about it. :) I've spent lots of time battling issues with ascii-centric libraries. My conclusion is that it is not easy at all which is not strange because tools like tr and sed were written decades before unicode support became a must have. I can't say that it is impossible to write a shell script to count words in a text written in Arabic script, but it doesn't seem easy.

<comment><AUTHOR> > _But your assertion was that this was an easy problem solved with just a few > lines of shell code._

Where did I assert that?

> _I can't say that it is impossible to write a shell script to count words in > a text written in Arabic script, but it doesn't seem easy._

Is there a definition of what counts as a "letter" in Arabic script? That is, which Unicode code points correspond to letters? And does Arabic share the convention that a "word" is a sequence of letters delimited by non-letters?

If the answers to those questions are "yes", then the extension of the algorithm already presented to handle Arabic is straightforward; it's just a matter of substituting the Arabic definition of "letters" for the ASCII definition. (With, as I said in my previous comment, the additional issue of determining the encoding of the input and output.)

If some of the answers to the above questions are "no", then you have an issue with the problem specification, not with the algorithm you're going to use to solve it.

</comment></comment></comment></comment><comment><USER14> Both the article's improved pipeline and Doug McIlroy's original have a bug. I've just written about it on +Ade Oshineye's post about the original article. [https://plus.google.com/105037104815911535953/posts/KuXczyiw...](https://plus.google.com/105037104815911535953/posts/KuXczyiwqep)

<comment><AUTHOR> Good catch! I've implemented your suggested fix of adding a sed '/^$/d' at the beginning of McIlroy's pipeline. The github code is updated, along with an updated test file and expected output.

(Btw, the Python version did not have the bug, since the split method of Python strings already ignores initial whitespace.)

<comment><USER14> Your bug fix is broken; it makes no sense to place it there. I suggest increasing the number of test inputs to first show the problem.

<comment><AUTHOR> Oops, you're right, the sed command should be on the second line. Fixed the test inputs and the fix.

</comment></comment></comment></comment><comment><USER15> Doesn't Pascal have libraries for sorting? I find that hard to believe. So there must have been another reason why Knuth chose to implement everything from scratch. Probably to demonstrate programming techniques. You don't learn that much about algorithms by stringing together a couple of shell commands.

<comment><USER16> Pascal makes code reuse hard. <http://www.lysator.liu.se/c/bwk-on-pascal.html> puts it better than I could. And when I used Pascal in the early 1990s - <USER2> this was Borland Pascal, one of the better ones - it even had strings! - things had not improved massively.

<comment><USER15> When did that comparison take place anyway? I don't think anybody is still using Pascal in this day and age?

<comment><USER16> 1980s, I think it says?

At any rate, my point, though it wasn't really clear, was that pascal makes code reuse hard to such an extent that I don't think it's actually possible to have a general-purpose sort routine in the library that would be actually useful. Certainly nothing like qsort, anyway. It just can't be expressed.

(I'm sure modern versions of Pascal have this problem licked.)

</comment></comment></comment></comment><comment><USER17> Maybe I am missing something here, but The python version is: O(N) sequential disk access O(NlogN) cpu O(N) RAM

The shell script is the same for disk and cpu, however, it is O(1) of RAM, and therefor can operate on input of size limited to disk, not RAM

<comment><AUTHOR> You're correct, the Python version is O(N) instead of O(1) in RAM usage for two reasons, one easily fixed and one not. Sorry if I'm belaboring the point, but I think it's worth going into a little more detail.

The easily fixable issue is that the shell pipeline buffers reads, whereas my Python version just uses sys.stdin.read() for simplicity. I could have buffered the reads by using Python's generators/iterators instead. However, that alone isn't enough to get O(1) RAM usage.

The not so easily fixable issue is that the Unix sort command uses temporary files in order to not have to have all of its data in memory at once. See, for example, here:

[http://vkundeti.blogspot.com/2008/03/tech-algorithmic- detail...](http://vkundeti.blogspot.com/2008/03/tech-algorithmic-details-of- unix-sort.html)

Python's sorted builtin doesn't work like this; it can take a generator as input, but it returns a list. This is one respect in which Python's built-in functionality lacks a feature that the Unix utilities have. I don't know if anyone has tried to re-implement the Python sorted function to use temporary files and return a generator to reduce memory usage.

[Edit: the Python version would also have to re-implement the uniq function to take a generator as input and return a generator, which would, I believe, also require temporary files.]

<comment><USER17> Unix sort is a merge sort, using fixed RAM uniq -c is line by line sed is line by line

A superior solution in every way

If you were going to do this in Python (or another similar language), you need to write a sort function that operates on a file, not a Python collection, as the collection is always bound by RAM, which would at best be re-implementing the sort command. Once you can sort a file, everything else is trivial, as counts can be done line by line, or more siply, via uniq -c.

Of course, if you only care about things that fit in memory, you can do it in Python, but it is still far easier to use command line for these type of problems.

<comment><AUTHOR> > _Of course, if you only care about things that fit in memory, you can do it > in Python, but it is still far easier to use command line for these type of > problems._

I agree; I was not trying to claim that my Python solution should be used in preference to the shell pipeline solution in any kind of "production" environment. As I noted in another comment, Knuth's Pascal solution appears to be open to the same criticism.

</comment></comment></comment></comment><comment><USER18> I like that article very much, so I wanted to monitor the blog. However, I found neither an Atom feed nor an RSS feed. That's a pretty good way to turn off potential regular readers.

<comment><AUTHOR> I do have feeds set up, but I just realized they aren't linked to or autodiscoverable. :redface: I'll fix that ASAP. Thanks for the feedback!

<comment><AUTHOR> Fixed--added links on the home page to RSS 2.0 and Atom feeds. Also feed autodiscovery should work now.

<comment><USER18> Great! I just subscribed to your Atom feed.

(BTW, who needs RSS anymore? Atom is so much cleaner designed and thus easiert to handle with. I'm even planing use Atom as main format and generating my website from that ...)

<comment><AUTHOR> Thanks for subscribing!

I generate everything from Markdown source using PyBlosxom's static rendering (which has some issues that may eventually drive me to switch to something else or roll my own). It auto-generates RSS and Atom feeds, so it costs me nothing to have both just in case someone prefers RSS or can't use Atom for some reason.

</comment></comment></comment></comment></comment><comment><AUTHOR> Inspired by reading this:

[http://www.leancrew.com/all-this/2011/12/more-shell-less- egg...](http://www.leancrew.com/all-this/2011/12/more-shell-less-egg/)

</comment><comment><USER19> Neither Knuth's nor McIlroy's solution handles Unicode, so they both suck.</comment>


Stinging report says NASA lost, lacks real sense of direction - <AUTHOR> http://www.msnbc.msn.com/id/50091903/ns/technology_and_science-science/

<comment><USER2> >> "More than two years after the president announced the interim goal of sending humans to an asteroid by 2025, there has been little effort to initiate such a mission," the report said.

It's kind of politically naive to think the West Wing would have drawn any attention to this during the election cycle, especially given the focus on the domestic economy and deficit.

This is relevant: [http://news.discovery.com/space/obama-win-nasa-direction- ast...](http://news.discovery.com/space/obama-win-nasa-direction- asteroids-121107.html)

In terms of the mission itself, it seems suitably dangerous, ambitious, and exciting :)</comment>


Trends in Cloud Computing: Who Uses AWS, Who Uses Azure - <AUTHOR> https://stackoverflow.blog/2017/07/21/trends-cloud-computing-uses-aws-uses-azure/ <comment><USER2> As a related side note, I sent a support request to Vultr twice in the last week and both times I got a response within 5 minutes. All follow up responses for both those tickets were also within a few minutes.

Their uptime has been rock solid and performance consistently great for the 2 years I've been with them.

I dare say most projects (even moderately big ones) would do just fine without the mega cloud.

<comment><USER3> That's alright if EC2 is all you were using. Past a certain point though, it usually makes sense to begin using managed services like S3, BigQuery, lambda, etc.

<comment><USER2> I also use S3. I don't see why using Vultr prevents that.

</comment></comment></comment><comment><USER4> What about GCP and other providers?

</comment><comment><USER5> when they're plotting %azure vs %aws, they should really be using the same range for both axes. then they could get rid of the dashed line that shows what the ratio is.</comment>


What is the correct percentage of single family zoning in Seattle? - <AUTHOR> https://medium.com/@15kwhm2a/what-is-the-correct-percentage-of-single-family-zoning-in-seattle-7db6af731c21 <comment><USER2> Zero. Don't do zoning, just don't regulate the market.</comment>


A one-man experiment in anti-aging - <AUTHOR> http://joshmitteldorf.scienceblog.com/2014/03/21/a-one-man-experiment-in-radical-anti-aging/ <comment><USER2> Could there possibly be two worse pictures to compare if you want to see if someone has aged (visually) or not?

<comment><USER3> Particularly because that's the only evidence he's collecting to show that his experiments -- lots & lots of concurrent experiments that he's continually changing for murky reasons -- are working.

He just... _says_ stuff like this:

> At first I expected to be able to set back my telomere clock by up to 8 or 9 > years per year, but subsequently revised this to about 5.1 years per year.

> The astragalus extract actually finally turned out to work rather better, in > fact, than TA-65.

...with no explanation at all. He's clearly learned some domain-specific vocabulary, but spending all of his money on supplements rather than _any_ kind of testing whatsoever, or even taking the time to work out a more scientific approach.

I don't see how his "experiments" have any scientific value beyond "here's a guy who ate (or rubbed into his scalp, or whatever) X quantity of Y every day for a few months or years, and it didn't seem to do him any serious harm".

 _Edit: saying "whatsoever" 3 times is overkill..._

</comment></comment><comment><USER4> It says "scienceblog". It says "experiment". And yet I didn't see any science. Did I miss it?

</comment><comment><USER5> His profile on Longecity forums: [http://www.longecity.org/forum/user/4099-jamesagreen/](http://www.longecity.org/forum/user/4099-jamesagreen/) (You need an account to view it) Here's one of his lasts post there: [http://www.longecity.org/forum/topic/52748-can-we-get- straig...](http://www.longecity.org/forum/topic/52748-can-we-get-straight- information-about-telomerase-inducers/page-2#entry582293)

<comment><USER6> looking at his avatar - I'm not sure he's sharing the truth. If you're running all of these cutting edge anti aging experiments surely can upload a non pixelated photo?

<comment><USER5> From the horrible looking site to his blurry/pixelated photos and the hundreds of links on each of his posts (Not necessarily bad this one), there's definitely something not right with this guy.

</comment></comment></comment><comment><USER7> Medicine has a long and illustrious history of self-experimentation. Barry Marshall (lives in Perth, my dad worked in the same lab!) won a Nobel prize for proving peptic ulcers are caused by bacteria (not stress etc) and can be cured with antibiotics. As part of his research he drank a culture containing bacteria from a patient's ulcer (yuk!), became horribly infected and cured himself with antibiotics.

Sometimes self-experimentation turned out to be fatal (see Carrion's disease for example).

Like the article says, I am very grateful to the guy for putting his own body on the line... and I hope he too wins a Nobel prize!

<comment><AUTHOR> Yes, self-experimentation and knowingly risking the most precious thing there is - your own life, to advance science and help others, is one of the most heroic things. I'm not sure if Jim is doing anything as heroic and with the same admirable motives, but I really hope he's and other self-experimenters contribute to medicine and science, because really the pace, cost, and the moral concerns of studies slow progress down tremendously.

</comment></comment><comment><AUTHOR> Jim's page on longevity (looks like it's from the 90s, but it's not): [http://www.greenray4ever.com/longevity.html](http://www.greenray4ever.com/longevity.html)

Jim's biography: [http://www.greenray4ever.com/index-88.html](http://www.greenray4ever.com/index-88.html)

Although his website looks like a mess, a lot of the stuff he's doing and talking about are solid. I posted this, because he's essentially a computer engineer... or use to be.

<comment><USER8> How old is he?

<comment><USER9> He's 65 according to the main page.

</comment></comment></comment><comment><USER10> The bad quality of pictures don't help the story. Both his pictures and the website itself look like they were made in the 90s.

The author of the "interview" also admits that he has not physically met Jim Green.

What also does not help is that almost every hyperlink in the story goes back to Jim's page as a source. This does not seem like very high quality journalism to me.

</comment><comment><USER11> Slightly frightening that he takes telomerase activators one week (to reduce effects of ageing) and telomerase inhibitors the next (to reduce likelihood of cancer from telomerase activation, presumably).

<comment><AUTHOR> I never really understood the overall hypothetical effect of this approach. What I find interesting that his regimen does not have any mitochondrial enhancers (outside of the standard Co-Q10), which is something most anti-aging programs today focus on, specifically, PQQ, which generates new mitochondria.

</comment></comment><comment><USER12> If he's 65-66 now, he looks way older than a man in his 60 in the photo from 2007 (strangely the only photo not blurred on his site), isn't he?</comment>


My Broadcast [The Unix rwall problem] (1987) - <AUTHOR> http://catless.ncl.ac.uk/Risks/4.73.html#subj10.1 <comment><AUTHOR> Jordan Hubbard, of FreeBSD/Apple Darwin fame sends a broadcast message to the entire internet/ARPAnet</comment>


Seriously, what's up with old media not crediting bloggers? - <AUTHOR> http://cdixon.org/2012/04/06/seriously-whats-up-with-old-media-not-crediting-bloggers/

<comment><USER2> I had a journalist rip off one of my posts without mentioning me as the source or even that I discovered the story. I found him on Twitter and sent him a message about it, and got no response. I dropped it because it happens so often.

I ended up following him on Twitter. He later published another story that was a slight follow up to my own story and series of articles. This article of his got a lot of attention. In an irony of all ironies, he then tweeted:

"This is why we need old media and journalists, bloggers would never have found or investigated this story" [1]

He had a bunch of other journalists replying in strong agreement.

He was talking about a story that was 90% built on posts that I and other bloggers wrote, and not only took the credit for it but claimed that there is no way non-professional journalists would be able to 'investigate' such a story. I pinged him again with 'oh the irony' and a link to my blog. No response again.

[1] Paraphrasing. He later deleted the tweet

<comment><USER3> You should have screen-shoted the tweet, and send it along with your original story to some rival online outlet.

Journalists like exposing other such frauds...

</comment></comment><comment><USER4> Josh Linker, author of the Fast Company piece, responded in the comments:

      "Hi Chris.  Josh Linkner here, the author of this piece @ Fast Company.
      I owe you a HUGE apology!!  A friend of mine sent me that
      excerpt and I had no idea it was yours or anyone else's so
      I didn't attribute it when I wrote my post.  As an author,
      VC, and entrepreneur I hold myself to the highest standards
      and I'm deeply sorry this happened.  Will correct and cite
      you ASAP. Again, honest mistake and I'm sorry it happened.
      Happy to chat further if you'd like - www.JoshLinkner.com"

<comment><USER5> If you didn't write it and don't cite a source, it's plagiarism. Doesn't matter if a friend sent it to you in email -- that just means you plagiarized a friend, which is kinda worse actually.

It's really hard for me to see this as an "honest mistake"; it just seems more like your average, run-of-the-mill, laziness-induced plagiarism.

<comment><USER6> And in the age of Google, where you can just put quotes around the first sentence and find the original source, it shows a complete lack of care.

<comment><USER7> Once Google indexes Borges' Library of Babel, though, you will find that any sentence is already used, and every article is already written.

On a more serious note: it would be interesting to add a check to CMS systems to automatically look up all sentences in an article before submitting and apply information theory to compute a "plagiarism" coefficient. No doubt these kind of systems already exist to detect student plagiarism.

<comment><USER8> Indeed, my university uses a service called Turnitin to rate the percentage of coursework that is plagiarised, along with giving where from etc.

It's pretty nice actually, can recognise when things are quoted and not passed off as your own.

<comment><USER9> My high school actually used this too - my teacher once showed my a sample report, and it was actually very good at detecting plagiarism even with words changed, sentence order mildly munged, etc. (with percentage probabilities). They seem to have a fairly good model of what kind of changes writers make when copying material.

</comment></comment></comment></comment></comment></comment><comment><USER10> "Not giving credit" is not strong enough. That's the term you use when you pull a quote or a fact without attribution.

This is plagiarism. Or it _was_. Now it's just lazy writing.

</comment><comment><USER11> Old media in general does not cite its sources adequately. Most articles that report on research fail to cite the publications they are reporting on and, in most cases, butchering quite badly. This is probably a hold-over from a time when such publications were beyond easy reach of the average reader, but that has changed. Even if the article is behind a journal's pay-wall, pre-prints are usually available through sites such as arXiv. New media does a far better job of linking to sources and I thank them for it.

</comment><comment><USER12> Obviously the Fast Company blogger (Josh Linker) should have cited (at the least!) Chris Dixon. And his explanation doesn't really hold water: "A friend of mine sent me that excerpt and I had no idea it was yours or anyone else's so I didn't attribute it when I wrote my post."

But, speaking as a former reporter, it's a fallacy to tar every medium (or even "old media") for the f-up of a Fast Company blogger. The responsibility belongs to that writer and publication; other outlets do not bear collective guilt for a competitor's screwup.

See <http://xkcd.com/385/> \-- the same argument properly applies to generalizations about newsgathering organizations (or in this case, blogs hosted by magazines started in the first dot-com era).

<comment><USER10> _or anyone else's_

Via what, immaculate typing?

His explanation really betrays the fact that he doesn't understand what plagiarism is.

</comment></comment><comment><USER13> I remember reading Chris Dixon's original post. I even popped the question to a friend... "Pop Quiz.. Name one game other than Angry Birds that Rovio had produced".

He named three... all of them being Angry Birds update releases. lol.

MG Siegler has had multiple rants about mainstream news outlets not attributing him for scoops and releases.

<http://tinyurl.com/mgsieglerrant>

"The Wallstreet Journal is Fucking Bullshit"

[http://parislemon.com/post/18182094905/the-wall-street- journ...](http://parislemon.com/post/18182094905/the-wall-street-journal-is- fucking-bullshit)

<comment><USER14> There was some interesting discussion about bloggers breaking news and attribution here: [http://gigaom.com/2012/02/25/is-linking-just-polite-or-is- it...](http://gigaom.com/2012/02/25/is-linking-just-polite-or-is-it-a-core- value-of-journalism/) (and in related twitter discussions)

I'm not entirely familiar with 'old media' practices, but I always thought if two news outlets independently confirmed a news story they didn't acknowledge whoever broke the story first. Or if another news outlet breaks a story, they independently confirm and don't give attribution, since technically, they confirmed the story, so it's 'their news.' I've only heard a news source attribute another source when news is breaking in real time (i.e. X is reporting 7 are dead at the scene). Breaking a story != confirming or actually being a journalist's source.

In some cases, attribution is very important - like in the original post. However, I think a lot of 'new media' beef with old media practices is just a shift in ideology . Bloggers are closer to the action and transparency is more important. Old media is much more rigorous but is less transparent. I personally think the best approach is something in between the two.

<comment><USER15> You basically have the 'old media' practices right, but there's some variances. So yes, let's say your competitor breaks a story -- if they're the only ones and you write it up based on them bringing the story to your attention, you should give them credit, whether or not that's your only verification that the story happened.

However, if you feel that you would have gotten the story anyhow (or, even more likely, you were both working on it at the same time, and they just happened to publish first -- this happens all the time), as long as you were independently able to verify and really didn't rely on them for story, you don't have to credit. And this is a good thing -- you don't want reporters to rush their reporting over something silly like credit. But it's a trust thing, just like sourcing overall is.

Another variance: Let's say, for example, that a clown is riding a horse in Time Square, and for some reason you want that story for your Washington state paper. You really shouldn't write about it until you can independently verify. But you can't because you don't have reporters or sources on the scene, and you don't want to credit just the NYT. So then the Boston Globe reports it, then Wash Post -- and all three say they've independently verified the story (vs. citing original NYT report). THEN you can pretty much assume it's fact, then you don't really need to source the papers -- you can say according to multiple reports (or say nothing at all).

It also falls back to you can't copyright facts, just the language the information is presented. As a journalist you're responsible for making sure the facts are correct (generally held at 3 sources, whatever they are), but once you've done that, you can go for it.

But that's why just not sourcing someone (bad practice) is nothing close to the linked example because plagiarism (exact language = violating copyright) is something you can get sued for.

Probably WAY more info than anyone wanted, but FYI in case it helps.

</comment></comment></comment><comment><USER16> As a guy who runs a news site about a band that gives me exclusive news, I've been whining about this for a dozen years now. I've been ripped off by MTV, Spin, Rolling Stone, NME, Kerrang!, you name it. A friend of mine was at one point almost wholly responsible for the Wikipedia article on Nine Inch Nails, and related tales of seeing his writing in publications worldwide.

In my experience, I've seen things improve - at least in the realm of music news. I assume it's because music-related news outlets are hiring people who got their start blogging, and who have a better appreciation for citing and linking to your sources. I think it's going to take a while for this trend to seep into more 'serious' 'journalistic' organizations, but it's definitely getting better.

<comment><USER17> Your friend (or you, not sure if you mean your friend is complaining) understands that once he writes content into a Wikipedia article he agrees to give away the contents of his work to the public right? That's the whole point of Wikipedia. Your friend doesn't deserve credit, Wikipedia does.

<comment><USER16> Of course he does. I think it would be quite a task to bring a Wikipedia Article up to "featured article" status and not understand such a basic fact about Wikipedia. I suppose my inclusion of that bit was more about the oddity of seeing things you wrote, copied verbatim into hundreds of magazines, newspapers, and online articles. I think that when we discussed this, his point was also that Wikipedia was not credited.

The web was built for referential text. It's one of the basic concepts of "Hypertext" - that it's 2012 and so-called journalists fail to use this simple process to give credit where it's due is kind of sad.

<comment><USER17> Want to apologize for that last sentence since I wrote it after feeling sick in the middle of the night. After I reread it when I woke up it sounded pretty rude.

Not that he should or shouldn't deserve credit for editing but in the end Wikipedia should be the one cited. I would suggest that Wikipedia acknowledge some editors for their work but that to me would probably result in more bureaucracy and infighting than any good it would do acknowledging contributing editors.

But I agree, very sad when a big site doesn't cite their sources even if it was, "an email from a friend"

</comment></comment></comment></comment><comment><USER18> It's not always the case that they don't credit, I'm running a twitter feed of the Falklands conflict on a 30 year delay, and so far,in the first week, I have been contacted by two tv companies who wanted to know who they should attribute them to...

</comment><comment><USER19> Hey now! It's hard enough work putting food on your family without having to promote the people who are doing it for you.

</comment><comment><USER20> This happened to me too. I posted an article on my blog and it got on HN's front page. It was about how people have to stop sending .doc files as attachments via email and to use something like .pdf instead. (my article is here: [http://maxmackie.github.com/2012/03/19/Stop- distributing-.do...](http://maxmackie.github.com/2012/03/19/Stop- distributing-.doc-files))

A couple days later, this guy wrote about the same exact thing, bringing up the same exact arguments as I did (his link: [http://blogs.computerworlduk.com/simon-says/2012/03/the- magi...](http://blogs.computerworlduk.com/simon-says/2012/03/the-magic-of- editable-pdfs/index.htm)).

He posted about a week after me. I didn't really do anything about it, but he could have at least mentioned where he got the idea/information. Incompetence.

<comment><USER21> I wrote the article you're complaining about, which is actually mainly about making hybrid PDFs. I did not see your article, let alone copy it; I would have linked to it if I had. I have been working on that piece for a few months off-and-on and pitched it as a "lightning talk" at FLOSS UK Spring 2012 in Edinburgh beforehand.

More than that, I was a manager of the team that created the hybrid PDF feature in OpenOffice at Sun, and have been advocating avoiding editable attachments for years - the earliest I can find on my blog is <http://www.webmink.net/2003/07/feature-creep.htm> but I am pretty sure I was advocating it before.

The web is a big place where there are often people working on the same ideas as you (which is why software patents are a travesty), and I recommend avoiding accusing people of incompetence without a little more research.

</comment></comment><comment><USER22> One of the most egregious "old media disrespects new media" stories I've seen was last year when The Daily Mirror lifted a complete blog entry by comedian Dave Gorman and printed it in the paper. They did give him credit, but never asked permission or paid him for his work. (He decided not to take action against them for it, but it seemed that was mostly because he thought it was pointless to go up against their legal team.)

<http://gormano.blogspot.com/2011/12/ctrl-c-ctrl-v.html>

</comment><comment><USER23> Plagarism is generally more sucessful when you don't list your sources.

</comment><comment><USER24> "Old Media"!?

It was launched in November 1995. To many thats old, but to most I would think thats not "Old Media" in the usual sense of the word.

<comment><USER25> The "Old" refers to the medium of distribution.

I have no idea what "Fast Company" is, but according to wikipedia it's a printed magazine.

<comment><USER26> In that case, "Traditional Media" might be a better term.

</comment></comment></comment><comment><USER27> Josh Linkner publicly admitted his mistake, offerend to talk it over with Chris, and even changed the text of the article, all within one hour.

Dixon is being a little harsh, I think.

[http://www.philco.me/philcosblog/2012/4/6/apologizing-to- chr...](http://www.philco.me/philcosblog/2012/4/6/apologizing-to-chris-dixon- this-is-how-its-done.html)

<comment><USER15> Nope, no way. Dixon wasn't harsh enough, because he thought it was just a lack of credit, whereas it was actually plagiarism. The reporter took writing that wasn't his that he played off as his own (under his byline, no quotes, no source). Completely unethical and irresponsible for a journalist to do that. The fact that he got in via an e-mail and thought he could just "use" it shows that this reporter has a lot to learn (if that even happened -- I've never met another journalist who would do that, ever).

<comment><USER28> Agreed. It was a public offense, and the theft of his IP. And really really lazy since a simple Internet search would have returned the article. A public callout is certainly not unwarranted. However I question what would have happened were he not Chris Dixon. He has a solid platform and the community respect to take on a writer for fast company. What about one of the rest of us?

</comment></comment></comment><comment><USER29> Why not create a website where bloggers that find out these ripoffs can point out those cases? I think that the public exposure would create enough disincentive to do so.

</comment><comment><USER30> On a related note: It always saddens me when news programs on TV use footage from youtube and don't mention the author but youtube as source.

</comment><comment><USER31> I can't even imagine why you'd blame old media when AOL, Gawker, Business Insider, Mashable, The Verge etc show them over and over and over and over and over and over and over and over and over again that the key to success in the internet age is SEO spam and cheap writing / rewriting.

<comment><USER32> At least The Verge et al. credit bloggers.

<comment><USER31> ... with a discrete link at the end of their rewrite of that article that conveniently includes the punchline and any images or videos they felt would enable them to better hijack your work.

</comment></comment></comment><comment><USER33> Seriously, what's up with bloggers wanting credit from old media?

</comment><comment><USER34> The original article has a reference to your blog at the top (at least it did when I checked).

<comment><AUTHOR> Just to be clear, I'm just the submitter, this isn't my blog.

Also when I looked at the article originally it did _not_ have that link in the first sentence. Google Cache won't load for me but here's a screenshot of the preview from Google: <http://cl.ly/1h2w0I413X063l3e0Z2k>

</comment></comment><comment><USER35> _Fear_

</comment><comment><USER36> Probably the same thing issue that bloggers have with copying old media articles verbatim and then adding a modicum of "context". In their respective business models, things are done a certain way and they follow that model even if it conflicts with the model used by their source.</comment>


Deep Read: Nassim Taleb (The Black Swan Guy) : Planet Money - <AUTHOR> http://www.npr.org/blogs/money/2010/08/18/129276793/deep-read-nassim-taleb-the-black-swan-guy <comment><USER2> After listening to this, I'm more convinced than ever that Nassim Taleb is the most overrated phenomenon in business/economic literature. His books are vague and trite, and while his arguments touch on currently popular memes (i.e. "debt is bad, because it exposes you to downside risk"), his thoughts go no deeper than the level of analysis you can glean by watching five minutes of CNBC.

It was still interesting to hear this interview, because it's clear that when he's pressed by a moderately intelligent interviewer, Taleb has nothing to fall back on other than repetition of sound bites. By the end of the interview, his logic degenerates into the repeated invocation of a weak metaphor comparing US spending with the Madoff ponzi scheme.

Hopefully this guy's five minutes will be up soon.

<comment><USER3> His style tends to be hyperbolic but he does have interesting things to say. For example how he can't predict the (financial) future with respect to what will happen or when, but he can predict to whom it will happen to.

He makes some reasonable points on debt and leverage. He didn't seem vague to me - reduce debt, prefer equity over debt, build robustness by spare parts or by functional redundancy. That all seems pretty reasonable, specific and not really a view that's advocated by the main stream financial press.

<comment><USER2> His comments may be reasonable (for some value of the word), but you and I have very different definitions of "specific". I can reduce the entire talk to a one sentence summary ("debt is risky; redundancy is less risky."), with little loss of information.

More importantly, nearly everything the man says about debt or deficit spending revolves around a cartoon charicature of statistical reasoning, and boils down to the thesis that debt is always riskier than surplus, because the downside is larger. That's fine, but it's basically just folk wisdom. Even the notion of the "black swan" is just a goofy way of saying what every third-rate insurance agent knows: given a long enough outlook, improbable events are _guaranteed_ to occur. But the important question (the _relevant_ question) is: do you actually care? That's a far more interesting discussion (one that plays a prominent role in everything from insurance contracts to option pricing), but one that eats away at the simplistic appeal of his thesis.

His remarks about "spare parts" and "functional redundancy" are so vague as to be meaningless...sometimes, "spare parts" means cash surplus, but other times, they mean having more than one marketable skill. Maybe he means both. When you speak in glittering generalities like Taleb does, it's easy for other people to project their thoughts on to your words. I suspect that's what makes him so popular.</comment></comment></comment>


Etcher: Kickstarter campaign to turn your iPad into an Etch-a-Sketch - <AUTHOR> http://www.kickstarter.com/projects/ari-krupnik/etcher-etch-a-sketch-for-ipad

<comment><USER2> Congrats to Ari and the team! You are Awesome!</comment>


How I Learned to Stop Staging and Love Releasing - <AUTHOR> https://tech.yplanapp.com/2016/06/21/stop-staging-love-releasing/ <comment><USER2> I'd use staging when the test environments are different (data and hardware wise) to the production systems. A good way to avoid staging, but also avoid big incidents is to push the new release to a limited number of users first, and monitor carefully for a time until it is pushed to all users.

<comment><AUTHOR> This is a good point ! We do feature-scoping and limited releases for a number of high-impact patches, but not everywhere.</comment></comment>


Before Clearview Became a Police Tool, It Was a Secret Plaything of the Rich - <AUTHOR> https://www.nytimes.com/2020/03/05/technology/clearview-investors.html <comment><USER2> Clearview now says the app is "available only for law enforcement agencies and select security professionals to use as an investigative tool."

I'm fairly convinced by David Brin's argument that this is exactly the wrong approach. In _The Transparent Society_ he argues that privacy is no longer an option; our choice is between a society where the police surveil us all, and one where we all surveil each other. He says only the latter is compatible with freedom. We have to be able to monitor the cops and the powerful, just like they monitor us.

Maybe we need our own Clearview, with open source face recognition and data on the darknet.

<comment><USER3> I'm convinced that Gen-Z is going to blow all of this out of the water. That the world will eventually hit a point where there isn't anyone without something online that can come back to haunt them. And I think companies are starting to see it too.

A little anecdotal but when I was applying for jobs last summer there was a video that was brought up twice during interviews with different organizations. Both times I was told after the fact that it was just to see how I would react, all I did was tell them what happened, not that I was sorry because I wasn't, I told them I stood behind what my actions were and there was nothing I could do about it now (it hit 100k views on Instagram, not a situation where I can contain it). Both companies offered me a job.

I rewrote this a few times and I'm still not sure I got my point across, but I agree with the argument in the The Transparent Society, and I already see it shifting towards that.

<comment><USER4> > A little anecdotal but when I was applying for jobs last summer there was a > video that was brought up twice during interviews with different > organizations.

I find it interesting that a company would do this instead of silently reject you…

<comment><USER3> The one I took, the interviewer ended up being my boss. He told me that the reason they overlooked it was because:

1\. If there were videos of everything stupid he did, there would be no way he would ever get hired.

2\. It wasn't anything illegal or violent, so he didn't see it as a character issue.

3\. I was honest about it.

<comment><USER5> My Machiavellian side says that they have dirt on you now and know they can use it at any time. If anything, more dirt is then better, as it creates more leverage for them in the long run.

Still, Hanlon's Razor applies, and you boss is more than likely just telling the truth.

<comment><USER6> Your Machiavellian side is completely missing the point then - this is(was) a _public_ event (as evidenced by GP's reference to 100k views on IG), thus there is no leverage.

You'd have leverage if there were a reasonable likelihood that you were the only hiring manager to have this information, but if everyone is on a level playing field, the leverage disappears.

<comment><USER7> Not only is an easily discovered public event poor leverage, it becomes much _worse_ leverage if it comes up in an interview.

When companies (or governments) try to manipulate employees, they frequently rely on some kind of willful ignorance. Wells Fargo is a great example: they set impossible performance targets and turned a blind eye to fraud, then fired and blacklisted whistleblowers - ostensibly for knowing about that same fraud!

If a shady employer wants leverage, even public events can suffice as long as they can claim ignorance. For example, most stock option grants are immediately lost if you're fired, but even at-will employment can't be terminated specifically to deprive someone of their options. So an employer might give a generous options package, then "discover" the IG video and use it for dismissal at just the right time to prevent a profitable exercise. But if that video comes up during hiring, it's no longer a plausible reason for later dismissal, at least without committing perjury regarding the interview.

I can't even work out a scenario where "lots of people know about this including us" is an effective way to manipulate someone.

<comment><USER8> In any real going business, cost of hiring and training someone of a type who is eligible for compensation with stock options and the subsequent morale dip if discovered doing such manipulation would by far outweigh the benefits of doing this. So this is largely a tin foil hat scenario

<comment><USER7> For any business with decent size, absolutely. There are a thousand ways to claw back options, and the reason they don't get used is that doing it even once would make hiring practically impossible.

For a small enough company? It falls in the same category as "diluting out of one guy's shares" \- bad morals _and_ bad business, but it still happens.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> >"People were stealing our Häagen-Dazs. It was a big problem," he said. He described Clearview as a "good system" that helped security personnel identify problem shoppers.

>BuzzFeed News has reported that two other entities, a labor union and a real estate firm, also ran trials with a surveillance system developed by Clearview to flag individuals they deemed risky. The publication also reported that Clearview's software has been used by Best Buy, Macy's, Kohl's, the National Basketball Association and numerous other organizations.

this seems just like another tool that will be used to put the non-elite at a disadvantage. until proven otherwise - I can only think of negative outcomes for ex-felons, low wage workers, people of color, and the likes coming from the usage of this app by the rich and large corporations

<comment><USER10> > ex-felons, low wage workers, people of color, and the likes

"people of color" are not "the likes" of ex-felons and low wage workers.

<comment><USER11> People who are generally oppressed by the larger system/those in authority? There's a charitable understanding of the statement here.

</comment></comment></comment><comment><USER12> We're now constantly under surveillance and facial recognition is being used on that footage. Apple stores. Malls. Bus stands. Grocery stores. Train stations. Traffic stops. Schools.

Privacy is dead. Anyone with money or any government can use one picture of you and get basically every piece of information about you.

Maybe this isn't true to the same extent for the HN crowd who might be more privacy conscious, but it is true for 99% of the rest of the population.

Schools and governments failed to educate about the privacy concerns. Maybe that's understandable. But they still don't. Teens post all sorts of stuff that will come back to bite them, that will never be forgotten.

<comment><USER13> This is true. Only legislation will stop this and that seems unlikely.

The worst part is that most people upload high quality, high fidelity tagged training data on a daily basis. Add in the fact that social networking automatically builds network circles, it is not wonder we are riding a landslide towards 1984 and Fahrenheit 451.

<comment><USER14> We are already there, but with a healthy dose of Huxley's soma, so most of us don't really see it. I, too, partake the soma in terms of current discourse - otherwise it's very hard to survive out there, and be a member of society almost impossible.

</comment></comment></comment><comment><USER15> And they rage when the little guy does it to them. It's obvious that employers are either paying people to post good reviews or strong arming current employees to write positive reviews on Glassdoor. The good thing is they're easy to spot since they're usually some vapid garbage like "Best place in the world to work!" with 5 stars interspersed with the real 1 star reviews. You can also tell because there are usually just enough fake reviews to push the real ones off the front page.

</comment><comment><USER16> Seems like they're building the hype train so that consumers want to pay for a version they can use to "take control" of their digital identity.

<comment><USER17> Just like the credit reference agencies. The scum gets your data from everywhere (sometimes wrong data), shares it with whoever asks, but makes it super difficult for _you_ to get it or rectify any wrong data.

<comment><USER18> >shares it with whoever asks

Legally speaking they're supposed to get your authorization, but yeah it's an honor system.

>but makes it super difficult for you to get it or rectify any wrong data.

Is it? I thought all you had to do was write them a letter?

<comment><USER19> > Is it? I thought all you had to do was write them a letter?

Yes, it is. I've been trying to get this data on myself for years now, and have yet to succeed. Letter-writing is not sufficient (at least in the US).

</comment></comment></comment></comment><comment><USER20> Anyone that wants their own "Clearview" like app can take almost any FR application and create a database from scraping the web. All that Clearview did was pre-scrape the web for you, but to do that yourself or as an open source project of sorts is not difficult.

<comment><USER21> I wonder how legislating open algorithms as illegal will work out? If you can run this reasonably on a phone building your own face/object databases how could they stop anyone from doing it?

<comment><USER19> You wouldn't legislate against the algorithm. You would legislate to prevent the abuse of the data used to feed the algorithm.

</comment></comment></comment><comment><USER22> If they are able to do this by creating the dataset just by scraping public social media web images, and having a facial recognition algorithm in place, whats stopping anyone else to do that ?

<comment><USER23> I'd imagine that there are many, more clandestine versions of this.

</comment></comment><comment><USER24> This skit from Amazon Women on the Moon turned out to be prescient. It's not so funny anymore.

[https://www.nytimes.com/2020/03/05/technology/clearview- inve...](https://www.nytimes.com/2020/03/05/technology/clearview- investors.html)

<comment><USER25> Which skit?

</comment></comment><comment><USER26> Has anyone here tested the app and can speak to their direct experience? (throwaway or not)

</comment><comment><USER27> Doesn't the title imply it's not a secret plaything now?

I guess it's not secret...

</comment><comment><USER28> I would _love_ to produce my own ClearView.

</comment><comment><USER29> I know Hoan. He's a great guy. Very talented. Super capable. Totally trust him to deploy this responsibly.

<comment><USER30> Despite the many examples in this article and elsewhere about him already deploying it irresponsibly? Or maybe I missed the implied "/s"

</comment></comment><comment><USER31> Facial recognition technology as a tool for protecting retail establishments... Interesting!

</comment><comment><USER23> The problem I see with this tech is with insurance. Your life, your health, your well being your family's well being, will be dependent on your ability to hold good insurance. If everything is measured, everything is tracked, everything quantified, your life is going to be very different.

</comment><comment><USER32> I have lived in {} since the mid 2000's. Stalking by strangers and acquaintances has gotten out of hand in (at least) the past five years. (Any such behavior against me has since calmed down in the past year, after reworking my digital devices, but the effects have had significant impact on me. I also dropped out and gave up on life this past year, which may make me a much less interesting target to harass.)

Such technologies are part of an ongoing increase in information and power asymmetries that can be abused to harass innocent competitors, as has happened to me. I've had strangers come up to me in public and discuss specifics of my private life, including non public details about my since failed startup, and personal/private comms. Concurrently, I was falsely accused of a serious crime and was put under the microscope and harassed on a regular basis by strangers regarding this. It became apparent that my life was completely owned at that point, digitally and publicly. It amounted to ongoing bullying which really pushed me beyond thresholds of learned helplessness already long since established.

There seems to be no recourse against this behavior. If you have a digital "kick me" sign attached to your back, there's little you can do to remove it, short of avoiding being in public. Or, as in my case, one can drop out of life, go homeless, give up all of your assets, and prepare for suicide. Strangers can verbally harass/own/gaslight others, maintain perfect plausible deniability, have perfect encryption to cover their tracks, and devastate people who aren't equipped to deal with this behavior.

Evolution of survival going forward is trending towards resilience to increasingly sophisticated psychological violence and harassment, as well as the ability to accept being an unwitting voyeur in all public places.

One of the most difficult aspects to this was reporting these incidents (admittedly, under duress in the heat of the moment), and being told that I must be delusional and mentally ill. To me, the delusion is genuinely believing that technology is not used to stalk or harass people in public. As a counterpoint, I will say that being stalked repeatedly does increase your paranoia, so you'll start to look over your shoulder at every turn. If you believe that all of your devices and accounts are hacked and being used to harass you, the complete lack of digital privacy can have a profound impact on sanity.

To this day, I'm utterly freaked out by the presence of personal cameras, to the point where I've nudged people in the community to be aware of the cultural impact of holding phones vertically in coffee shops or other public places. As most people are of course good natured, I've noticed a trend in the places that I frequent towards people being more prudent in this regard. I personally cover the public facing back camera on my phone with my index finger as a matter of habit by now, to avoid pointing it at strangers in public. Personally I believe responsibility amongst the tech elite would include immediate installation of physical shutters that open only when a camera is in use. Shutters can be colored blue or yellow, perhaps as a culturally standardized signal that the camera is "closed".

There's clear benefit to tech such as Clearview but the potential for abuse by irresponsible or immoral actors is tremendous. As someone pointed out, such tech can be rolled yourself. It seems that the problem is therefore out of control. Welcome to the age of unwitting voyeurism.

Edit: I did make a comment on the linked NYT article, including my real identity. In this comment, I called out at least one person involved in shenanigans against me. This person name dropped {} as someone who would recognize him, before he trashed my startup without seeing it, encouraged me to drop out of my continuing Computer Science studies at the local University (due to the bad rep I would receive for doing so as a middle age adult, so he said), and then threatened my career/reputation if I told the truth about specific stalking incidents, all in one conversation. Not long thereafter, I experienced a stalking incident in public by two men with walkie talkies who harassed me about said startup, mentioning non-public specifics about an engagement we were seeking. In retrospect, these men could have been using tech such as Clearview to more easily enable their stalking and harassment of me. The location of this incident was the playground of wealthy folks in my city's most affluent public area. My comment on the NYT article was not approved by the moderators, understandably.

</comment><comment><USER33> Meh. You can be recognized by humans, you can be recognized by machines. I don't get the outrage.

<comment><USER34> The top of the story features an example where human recognition was not sufficient:

> _One Tuesday night in October 2018, John Catsimatidis, the billionaire owner > of the Gristedes grocery store chain, was having dinner at Cipriani, an > upscale Italian restaurant in Manhattan's SoHo neighborhood, when his > daughter, Andrea, walked in. She was on a date with a man Mr. Catsimatidis > didn't recognize. After the couple sat down at another table, Mr. > Catsimatidis asked a waiter to go over and take a photo._

> _Mr. Catsimatidis then uploaded the picture to a facial recognition app, > Clearview AI, on his phone. The start-up behind the app has a database of > billions of photos, scraped from sites such as Facebook, Twitter and > LinkedIn. Within seconds, Mr. Catsimatidis was viewing a collection of > photos of the mystery man, along with the web addresses where they appeared: > His daughter's date was a venture capitalist from San Francisco._

> _"I wanted to make sure he wasn't a charlatan," said Mr. Catsimatidis, who > then texted the man's bio to his daughter._

<comment><USER35> So it's OK these days to randomly ask a waiter to go and snap a photo of some other customer in a restaurant? Maybe call me old-fashioned, but that strikes me as outrageous behavior.

<comment><AUTHOR> "Let me tell you about the very rich. They are different from you and me. They possess and enjoy early, and it does something to them, makes them soft where we are hard, and cynical where we are trustful, in a way that, unless you were born rich, it is very difficult to understand. They think, deep in their hearts, that they are better than we are because we had to discover the compensations and refuges of life for ourselves. Even when they enter deep into our world or sink below us, they still think that they are better than we are. They are different. "

F. Scott Fitzgerald</comment></comment></comment></comment>


Friday Deploy Freezes Are Like Murdering Puppies - <AUTHOR> https://charity.wtf/2019/05/01/friday-deploy-freezes-are-exactly-like-murdering-puppies/ <comment><USER2> Some days I look at the continuous deployment crowd and wonder what they're possibly building that it can't handle a 3 day long feature freeze.

I understand the need for a well oiled system to identify an emergency and patch it. And do it as quickly and with as little fuss as possible. But if pushing to production should be so smooth it's a "non-event", turning the valve (for non-emergencies) off and on should be "non-events" too.

Is the pre-deploy testing so rotten that hitting production is a necessary step in the development loop? Then they should throw away CI/CD infrastructure because they're not using it. It's how they lie to themselves that they're doing something smarter than sshing into production and editing the live source files by hand.

Drowning under the technical debt accumulated by 1 workday without pushing? Stop writing code and build a development process, it's obvious none exists.

</comment><comment><USER3> If you want to continuously deploy during high-impact periods (last time I saw her bring this up was Christmas!), that takes 100% path coverage of every system you care about, full-scale load tests, and _zero_ software failures demonstrated over a very long time. After decades, I haven't seen this even once. I wouldn't upgrade kernels on Friday while everyone's heading for the door, and Linus' team is better than mine or yours.

</comment><comment><USER4> How to get clicks: write something controversial, post to HN.

Friday deploys are a fast way to end up ruining multiple people's weekends, and if you value your developers you will build a solid CI/CD pipeline and accept that you are only human and your tests may never cover 100% of stuff that can go wrong, so you can wait till Monday to deploy when everyone is around to fix anything that does go wrong.

Is the author's team also deploying on Fridays?

</comment><comment><USER5> I have no trouble deploying early on Friday (say before noon.)

So often though, people deploy something on the way out the door on Friday and then something goes wrong, my pager goes off...</comment>


Perl in Python - <AUTHOR> https://wildfish.com/blog/2019/09/18/perl-in-python/ <comment><USER2> I did something somewhat like this about 10 years ago, at [http://www.dalkescientific.com/writings/diary/archive/2008/0...](http://www.dalkescientific.com/writings/diary/archive/2008/03/10/python4ply_tutorial_3.html) :

      for line in open("python_yacc.py"):
          if line =~ m/def (?P<name>\w+) *(?P<args>\(.*\)) *:/:
              print repr($1), repr($args)

I did not use import hooks, like this project does.

It got appreciative hisses when I did a lightning talk about it at a PyCon.</comment>


LG Releases WebOS Open Source Edition, Looks to Expand WebOS Usage - <AUTHOR> https://liliputing.com/2018/03/lg-releases-webos-open-source-edition-looks-to-expand-webos-usage.html <comment><USER2> Glad to see this happen, Web OS on my late 2016 LG TV is actually very, very good. It offers far more advanced video configuration than any other TV I've seen and the OS is very fast (nothing at all like using those dreadful TVs with android running on them).

One thing that does seem rather odd and somewhat worrying is that it only builds on Ubuntu 14.04(!) and:

> "We strongly recommend you NOT to use Linux virtual machine on Windows or > Mac OS for building webOS OSE, as it may cause unexpected issues."

</comment><comment><USER3> Possibly of interest to LuneOS developers, whose codebase started with an earlier source drop of webOS.</comment>


Show HN: The Remote Clan - <AUTHOR> https://remoteclan.com <comment><AUTHOR> We feel the current communities around remote work only focus on jobs in the form of job boards. They miss the basic human need - to interact.

So, we built The Remote Clan, a community for remote workers to build a strong career with the help of those who are pursuing the same goals.

We are in a very early-stage and would love to hear your feedback!</comment>


Accounts at Apple and Amazon were hacked (2012) - <AUTHOR> https://www.wired.com/2012/08/apple-amazon-mat-honan-hacking/ <comment><AUTHOR> This is why I always started recommending appending an +alias to your login email, such as

firstname.lastname@example.com

That way the hackers won't even be able to find and confirm your account, let alone convince someone working for Amazon, AT&T, Apple to let them "back in" to the account, even if they have your other info like security questions.

Speaking of security questions, you should probably be giving the same answer to all of them, which is not the actual answer.

<comment><USER2> Some companies normalize usernames: email@example.com is stored as email@example.com</comment></comment>


Web of Trust caught selling private browsing history of millions - <AUTHOR> http://www.pcmag.com/news/349328/web-of-trust-browser-extension-cannot-be-trusted <comment><USER2> It looks like WOT was capturing browsing activity and selling an 'anonymised' version to other vendors. NDR then purchased some of this data and was able to de-anonymise it by examining the URLs for personally identifying information such as phone numbers and email addresses.

Here's the original report in German: [http://www.ndr.de/nachrichten/netzwelt/Nackt-im-Netz- Million...](http://www.ndr.de/nachrichten/netzwelt/Nackt-im-Netz-Millionen- Nutzer-ausgespaeht,nacktimnetz100.html)

"Um an die Informationen zu gelangen, haben die NDR Reporter eine Schein-Firma gegründet, die vorgeblich im "Big Data"-Geschäft aktiv ist. Gleich mehrere Firmen zeigten sich bereit, die Web-Daten deutscher Internet-Nutzer zu verkaufen - ein Unternehmen bot die nun ausgewerteten Daten schließlich als kostenlose Probe an."

</comment><comment><USER3> Thanks for sharing this. This type of behavior is really bothersome and sadly many general internet users probably won't see this post

<comment><AUTHOR> There definitely needs to be more awareness about this. Just from a quick read of the WOT forum it seems most users are completely shocked that their data was being sold, even though this is outlined in the WOT terms of service.

Here is what the WOT terms of service say about sharing their users' browsing data:

      The information we collect is aggregated, non-personal non-identifiable information which may be made available or gathered via the users' use of the WOT Utilities ("Non-Personal Information"). We are not aware of the identity of the user from which the Non-Personal Information is collected. We may disclose or share this information with third parties as specified below and solely if applicable. We collect the following Non-Personal Information from you when you install or use the Product or use the WOT Platform:

         Your Internet Protocol Address;
         Your geographic location (e.g., France, Canada, etc.);
         The type of device, operating system and browsers you use;
         Date and time stamp;
         Browsing usage, including visited web pages, clickstream data or web address accessed;
         Browser identifier and user ID;

<comment><USER4> Calling it non-identifiable is a lie.

</comment></comment></comment><comment><USER5> Everything with extensions is about trust, we recently started developing an extension for our customers to improve their experience with our product. While in theory you could look at the source code from your browser, the only solution for proving that we do not track their data is to publish the extension as an open-source project.

By the way: while testing around with extensions, I happened to realize that Kasperskys safe-form-input thingy does not work at all with other extensions, in fact: everyone can track your passwords, no matter what you do.

</comment><comment><USER6> Am I the only one not surprised by this, or frankly not caring?

I thought the business model was obvious and honestly admirable. It's a non- trivial amount of data they are working with. This Non-Personal Information is what almost every website ever, collects at some granularity (even if it's just in apache logs). It's kind of the basis of advertising and one of the only ways to monetize data other than ads eating up the real estate.

</comment><comment><USER7> Some details in the Debian bug report:

[https://bugs.debian.org/842939](https://bugs.debian.org/842939)</comment>


What you pay attention to when buying a new phone? - <AUTHOR> With excellent new flagships from Apple, Google and Samsung all vying for your attention, plus dozens of high-quality options for those on a budget, choosing the right smartphone has never been more challenging. what features you pay attention to when buying a new phone? <comment><USER2> Must Haves:

1\. Battery Size -- phones should last atleast a whole day of use

2\. Smaller Screen Size (5") (I dont need a phablet) -- I dont need huge ass screen. I have laptop for that.. I just need to read a few emails, message and take phone calls. Small screen is easy to hold and less changes of dropping.

3\. NFC / Contactless payment enabled -- This getting to must haves as new applications like pairing over bluetooth headphones via nfc, paying for groceries is getting contact less.

4\. Compatible with International Standards -- I do travel and hence would need phone to be compatible with various wireless standards.

Nice to haves

1\. Wireless charging -- This is a great luxury to just place the phone on a mat and not worry about whether the wire is connected correctly (considering some companies keep switching wire formats -- I am taking about Apple as you must have guessed.)

2\. Headphone jacks -- There is an ongoing war on headphone jacks -- Want to charge your phone and listen to music? well, you are out of luck.. you cant if you dont have headphone jack. Bring back headphone jack.

3\. Expandable memory -- 32 GB vs 128 GB its ridiculous that there is a difference of $100 or more?.

4\. Ability to uninstall all or any apps.. I dont want another (MUST have Homescreen app) -- Choose Android One if possible.

5\. Okish Camera -- Not much of a photo snob.

6\. Fingerprint scanner for easy unlocks.

</comment><comment><USER3> With android phones, for how long will it get updates, which seems to be a game Nokia is winning lately from what I can tell</comment>


Udacity nanodegree programs in india - <AUTHOR> How much recognition do nanodegree programs from Udacity have in Indian tech sector?? will they help a c++ software developer with 3years exp into product development to switch into other tech domains<p>https://www.udacity.com/ai <comment><USER2> What is the "Indian Tech Sector?" Indian the country or Indian the race?

<comment><AUTHOR> "India" country in Asian subcontinent

p.s - my bad should have framed the question correctly</comment></comment>


Brent Simmons: NetNewsWire iPhone Code Surprise - <AUTHOR> http://blogs.newsgator.com/newsgator_widget_blog/2008/10/netnewswire-iph.html <comment><USER2> This isn't surprising at all - that it would be more likely to port code from the iPhone to the desktop OS than the other way around. There are more constraints on the iPhone - especially memory, so anything that runs on the iPhone should run fine on the desktop, but not vice-versa. Moreover, it may not make economic sense to tune a piece of desktop code to fit into a small bit of memory, but if you have code that was so tuned to run on the iPhone, why not replace the untuned desktop code with it?</comment>


Survey Shows Pre-Launch Demand for Apple's iWatch on Par With iPad - <AUTHOR> http://www.macrumors.com/2013/04/19/survey-shows-pre-launch-demand-for-apples-iwatch-on-par-with-ipad/

<comment><USER2> They don't even know _what it is_ yet. I wouldn't put that much faith in such surveys. When people say "yes" they might have a specific image of how that device would work in their heads. And if it doesn't work like that they won't buy it.</comment>


Show HN: A simple register-based virtual machine - <AUTHOR>

<comment><AUTHOR> The last time a small C-based virtual machine was posted it went down very well [1]. So I'd be curious if there is any interesting feedback here.

The code is pretty well organized, and when it was posted to Reddit there were a few comments mentioning the readable implementation.

This is a stack-based virtual machine which is simple to embed and extend with custom opcodes, and while the existing instruction-set isn't huge it is sufficient to write toy-programs and be an interesting learning-experience.

[1] - [https://news.ycombinator.com/item?id=8130413](https://news.ycombinator.com/item?id=8130413)</comment>


Ask HN: Where to find a potential co-founder? - <AUTHOR>

I know that this might seem like the wrong question to ask. You are supposed to know your co-founder very well by the time you start a company together. But most of the people whom I know from college are not willing to take the risk of starting a company on their own. It's hard to persuade a risk-averse person to become an entrepreneur, and even if I am able to persuade him, some months down the road when the going gets tough, he might be tempted to chicken out.<p>So what are some of the events and groups in the Bay Area that provide good opportunities for meeting potential co-founders? Thanks. <comment><USER2> Perhaps you can be more specific about why you need a co-founder? Can you build the product yourself? If you cannot what value do you bring to any venture?

I'd suggest you put yourself in a position of strength by executing and other people will want to work with you. If you cannot clearly articulate why you are seeking/need a co-founder, you should not be looking for one.

<comment><AUTHOR> I am looking for a start-up for the all the reasons that PG and other experienced entrepreneurs give for having at least one co-founder.

I am building the prototype on my own, but I would like to have a co-founder to assist with version 1.0, and with the non-technical aspects of a startup.

</comment></comment><comment><USER3> I was lucky enough to meet a co-founder at school who is a good complement to my skills, and we have good "chemistry".

That being said, it's really a matter of pounding the pavement. Meet lots of people, the types of people you'll make sure to follow up on.

Since moving to the bay area, I've met plenty of really great people. I'd be happy to give you some advice or make introductions...are you already working on a project, or is this more hypothetical?

Ping me - email@example.com

</comment><comment><USER4> Let me rephrase this question because I have the exact same problem:

Granted that I do not have enough time to start a start up myself, or seek out VCs for money, since I NEED my 9-5 job to feed myself and family (literally), I am naturally seeking for one or few people to start the company with so that we have enough time/brainpower between the X of us to actually make the startup work at least while we are actually paying in peanuts.

Now my unfortunate problem is that I don't have a friend I am willing to do this with. So is there a methodology people here use to find someone to start the startup with?

Now it's either that OR I am forced to wait until my wife graduates with her masters and gets a higher-paying job than me. In which case I guess I can quit work and focus on the startup. Is this the better alternative than looking for someone?</comment>


Ask HN: Tool(s) to render Git repository as static website - <AUTHOR> So this is hard to google, as almost everything points to hosting static websites on GitHub etc..
I am looking for a tool that can generate a static website from a git repository. E.g. generate a set of html files with some markup so I can present the contents of a git repository similar to the way GitHub/GitLab but as a static website for self-hosting. Nice to have all branches etc in there, but would already be nice to just have a single branch..
The goal would be to have a stand alone / static site for archival / offline use / inclusion in other static site (self-hosted). Being able to browse the code and download a zip file would be minimum. Viewing commits and perhaps generated diffs between certain commits would be a bonus.
Only thing I could find was stuff that converts just the wiki part of a GitHub repo. I am really looking for something that takes a .git repo and spits out some static website. Perhaps as a plugin for Jekyll/Hugo / ?? but stand alone would be just as good. Thx. <comment><USER2> Git comes with a `git instaweb` command, that allows to visualize a repo. Which seems... completely broken.

The web server does launch at localhost:1234, but accessing this URL shows a HTTP 500 error on my two machines, whatever the repo (even on the git repo itself).

<comment><AUTHOR> Haha, That is awesome! Seems the git learning curve never ends. Alas, it is a cgi file which dynamically generates the pages over lighttpd. Good to know though!!

</comment></comment><comment><USER3> Somebody posted [Stagit]([https://git.codemadness.org/stagit/](https://git.codemadness.org/stagit/)) to HN a while ago, which seems to fit your description.

<comment><AUTHOR> Perfect!! Thanks, I will check it out.

</comment></comment><comment><AUTHOR> Finally found a magic word that gets me somewhere in the search engines... git2html. [Hypercubed/git2html]([https://github.com/Hypercubed/git2html](https://github.com/Hypercubed/git2html)) is the first hit. reading as we speak. Any alternatives / experience greatly appreciated though.

Can't find anything else really, and most discussions are years old. Anyone?

<comment><USER4> you can try doxygen. It generates class list and file list.</comment></comment>


Combine your Wi-Fi, Ethernet and 3G connections to increase your download speed - <AUTHOR> http://github.com/Morhaus/dispatch-proxy

<comment><AUTHOR> Dispatch-proxy lets you boot a proxy on your machine that dispatches every incoming connection to a different network interface.

The possibilities are endless:

* download a file through threaded downloading, effectively combining multiple connections' speed in one single file download,

* create two proxies, assign to each its own interface, and run two apps simultaneously that use a different interface,

* create a hotspot proxy at home that connects through Ethernet and your 4G card for all your mobile devices,

* etc.

Here are some instructions to quickly get started with using the proxy:

1) INSTALLATION

* Install Node.JS ([http://nodejs.org/](http://nodejs.org/)).

* Launch a terminal.

* Run the command `npm install -g dispatch-proxy` as a power user and press Enter. It will install the `dispatch` command-line utility.

2) STARTING AN HTTP PROXY

* To quickly start an HTTP proxy, run `dispatch start --http`.

* The program will print the proxy address and port (by default __localhost:8080 __) as well as the list of local addresses the program will be dispatching requests to.

* Configure your system to use the combination address:port as an HTTP proxy.

3) LISTING ALL AVAILABLE NETWORK INTERFACES

* To list all available network interfaces, run `dispatch list`

* Each available network interface will be printed with a list of addresses.

* Right now, the dispatch utility only supports IPv4 addresses.

* If an address is marked to be "internal", then you won't be able to use it to connect to the internet.

I encourage you to read the docs [https://github.com/Morhaus/dispatch- proxy](https://github.com/Morhaus/dispatch-proxy).

<comment><USER2> You should really consider to put those use-cases in your readme too! :)

<comment><AUTHOR> Just did, thanks for the input!

</comment></comment></comment><comment><USER3> This is a neat idea. Does anyone know if it's possible to achieve a similar effect natively on Linux ? i.e. make the linux network stack load-balance all outgoing connections to all available interfaces ?

<comment><USER4> "Bonding" at the kernel level:

The Linux bonding driver provides a method for aggregating multiple network interfaces into a single logical "bonded" interface. The behavior of the bonded interfaces depends on the mode. Generally speaking, modes provide either hot standby or load balancing services. Additionally, link integrity monitoring may be performed.

[https://www.kernel.org/doc/Documentation/networking/bonding....](https://www.kernel.org/doc/Documentation/networking/bonding.txt)

<comment><USER5> People who are downvoting dgquintas, can you explain why? Or explain why this doesn't work? It seems to fulfill the problem?

<comment><USER6> Bonded Interfaces operate at a lower level. There are many different "bonding modes" in the Linux Kernel. None of them would allow you to present a single "bonded" interface across multiple independent network interfaces where each interface belongs to a separate network. Case in point: You cannot "bond" two interfaces where one is connected to a cable modem and the other a DSL modem. Typically (in most configurations) a bonded interface receives one IP.

<comment><USER7> You can, but you need to make two tunnels (GRE, IPIP or whatever you fancy) over those two uplinks, then create a bond over those - if that's technically possible (I believe interfaces must at least have equal MTUs, can't remember what other requirements are). Obviously, this requires configuration on both sides of the bond.

And, ahem, correct load balancing over bonded interfaces is not trivial as it may seem (although, usually it's more likely you'll be able to distribute approximately evenly than not)

</comment></comment></comment></comment></comment><comment><USER8> I have used dual wan routers that do this: While speed can go up, and bandwidth certainly does, the latency variance makes many games unplayable, and really throws some video and audio streaming apps for a loop.

I'd love to see a write up how well this does stuff that is sensitive to latency and changing network conditions, beyond simple web surfing.

<comment><AUTHOR> One of the advantages of using a proxy is that you can configure it per app.

For example, let's say I have 3 interfaces.

Interfaces (1) and (2) are low-latency and interface (3) is high latency.

I can start a proxy on localhost:1080 that redirects to interfaces (1), (2) and (3), and configure my download manager to use it.

I can then start another proxy on localhost:1081 that redirects to interfaces (1) and (2), and configure my game client to use it.

</comment></comment><comment><USER9> Nice. Interesting that you're doing it at the userland level and balancing across IP address and not interfaces. I may find a use for this. (On a network rate limited by IP address, not per physical connection or per MAC address, and allows me to use multiple IPs. Go figure.)

I haven't set this up myself, but I think the iptables method for doing this is `nexthop via`. Never found any great documentation, but this seems to describe it: [http://lartc.org/howto/lartc.rpdb.multiple- links.html](http://lartc.org/howto/lartc.rpdb.multiple-links.html)

I think the iptables method balances across interfaces, not IPs. But I want to balance across IP aliases (multiple IPs, same interface). And I'm usually on a Mac. So I may see if dispatch-proxy is a better choice.

<comment><USER9> If anyone is interested in my particularly strange use case, this seems to work:

        $ sudo ifconfig en0 alias 192.168.2.210 255.255.255.0
        $ sudo ifconfig en0 alias 192.168.2.211 255.255.255.0
        $ sudo ifconfig en0 alias 192.168.2.212 255.255.255.0
        $ dispatch start
        SOCKS5 server started on localhost:1080
        Dispatching to addresses 192.168.2.28@1, 192.168.2.210@1, 192.168.2.211@1, 192.168.2.213@1

Also the link I mentioned is different from what other people have mentioned in this thread. It's not interface bonding or TCP multipath. It is actually similar to dispatch-proxy, it balances TCP connections across interfaces as far as I can tell.

</comment></comment><comment><USER10> As far as I know this is a DIY solution for what companies sell as a "bonded" internet connection..?

I spent weeks trying to get a reliable setup for streaming concerts in HD from all varieties of spotty WiFi and 4G hotspots. The streaming server wouldn't accept packets that seemed to come from totally different IP addresses so there had to be an intermediate proxy server in the cloud that would re- assemble the data from my various connections.

I was able to get it working and it would have been great for normal Internet usage, but I never found a reliable system for streaming. That was about a year and a half ago. I would love to see this technology mature to the point where it worked well for streaming.

<comment><USER7> From a quick glance over the code, it just seem to proxy TCP streams over one channel or another, probabilistically. Most trivial form of load balancing possible.

This approach certainly won't work for typical media streaming, as it usually uses a single connection. Particularly, because streaming over multiple connections is tricky because as they don't guarantee delivery order, so you'll probably get your packets "scrambled" and will have to reorder them by yourself (implementing your own flow control).

</comment></comment><comment><USER11> So if I am in a residence where I can connect to 3 independant WiFi networks, and have 3 wireless network adapters, I can combine their bandwidth?! because that would be freaking awesome.

<comment><AUTHOR> Yup.

<comment><USER11> You should totally add this use case to your README. That's the very first things my fellow student friends ask when I show them this.

<comment><AUTHOR> I just did, thanks :)

</comment></comment></comment></comment><comment><USER12> Since uncapped 3G is pretty common where I live, I've always thought to build something similar but without never finding the time to do it. This will also be great for load balancing my network the next time I gather to play some online-games with my friends.

Any idea will the proxy still be available if one of the connections drops? Like if my broadband cuts out (like it does on a daily basis) will the cellular data automatically take over?

<comment><AUTHOR> For now, the proxy will still try to dispatch sockets to every network interfaces specified at launch. But such a feature could perfectly be implemented.

</comment></comment><comment><USER13> Can anyone explain how this works? From previous experience I always thought that this was not possible without some cooperation on the other end because the problem is you want to split the data being sent to you to different ips. If the other end sends all the data to the same ip, you cannot redirect/load balance to other ips before the data actually gets to you. What am I missing?

<comment><USER14> Probably each TCP connection only goes over one IP address, so the remote end doesn't get confused. All the data coming back on that connection will have to come in on the same path the connection started on.

<comment><USER13> If that were the case, the download would be no faster than using a single connection. I guess it would help if you download multiple things, but that is a different claim than what is claimed in the title.

<comment><USER15> You can use multiple connections for a single download (e.g. HTTP range requests).

<comment><USER13> Yes, but that is at the application layer, not the transport layer (i.e. at TCP). Meaning, you need to be aware of what is actually being sent.

<comment><USER15> Well, if you just balance TCP connections across physical connections, you could then combine this with a multi-connection HTTP download client and get the effect without any one piece having knowledge of it all (aside from the human). But yes, it's not going to be useful in general for everything.

</comment></comment></comment></comment></comment></comment><comment><USER16> God I've been wanting to do something like this for a long time, but just didn't have the networking chops to get started. Glad someone beat me to it!

I imagine this would work well, not just on networks you own, but with cooperative neighbors as well. What if an entire floor of an apartment building used the same proxy? Would that work?

</comment><comment><USER17> I think Justin.TV used a setup like this in the early days to stream 24/7 live video -- several mobile hotspots from different carriers all working on sending data to the remote server, ensuring that the data got there somehow. The difference was that they controlled the server side.

</comment><comment><USER18> I use Super Download on Android that does the same thing.. Always wondered how they pulled it off.

</comment><comment><USER19> Why not just use Multipath TCP?

<comment><USER20> Ideally, in the future, yeah. But for now, two reasons I can think of: first, Multipath TCP doesn't cover the same use cases as dispatch. For example, having one proxy only route to low-latency connections (e.g. for gaming) and another to all connections (e.g. for downloads). Second, Multipath TCP isn't widely implemented and would require application developers to use it, whereas this can give many of the same benefits without waiting for your applications to support it.

</comment></comment><comment><USER21> is this the same as connectify dispatch?

[http://www.connectify.me/dispatch/](http://www.connectify.me/dispatch/)

</comment><comment><USER22> I'm guessing this is Linux-only? Would love to use my tablet's connection aswell.

<comment><AUTHOR> Windows, Mac OS X and Linux.

Every platform which runs NodeJS in fact.

<comment><USER23> Is that in the readme?

<comment><AUTHOR> Adding it now.

</comment></comment></comment></comment><comment><USER24> Does this do bonding at the packet level, or at the TCP level?

Outcome being, if I download one particular file over HTTP will it run N times as fast, or are you just increasing the total speed available for many connections?

<comment><AUTHOR> At the TCP level.

</comment></comment><comment><USER25> OP,

Great job. I am not too technical so pardon my question. Can multiple 3G connections be combined or have a 'random selector'/failover.

So where I live (you can make a guess), mobile connectivity is not the most reliable in the world. So for instance, POSs are not reliable since one network will always fail. With this, could we use multiple SIMs thereby making them more reliable?

BTW, have you seen BRCK [http://www.brck.com/specifications/](http://www.brck.com/specifications/)? Seems to have the sort of the same concept although I think yours is combining speed vs. having a fail-over system for internet access.

<comment><USER26> Take a look at Horde:

[http://nms.csail.mit.edu/projects/horde/](http://nms.csail.mit.edu/projects/horde/) [http://nms.lcs.mit.edu/papers/horde- mobisys-2005.pdf](http://nms.lcs.mit.edu/papers/horde-mobisys-2005.pdf)

</comment></comment><comment><USER27> now i can use dispatch proxy on application bases like firefox or aria2c, simply by adding the proxy "localhost:8080" to the proxy section on these applications

but how to use it system wide ? so apps that don't support the use of proxies will be able to benefit from all this ?

edit: I'm using Mac

<comment><USER9> On a Mac set it in System Preferences > Network > Advanced... > Proxies > SOCKS Proxy.

Many Mac programs will automatically pick it up.

</comment></comment><comment><USER28> Would using this improve the connection for Skype/Hangout video calls?

<comment><USER29> No. Simultaneous (locally bonded) connections do not have great enough threading integrity to bridge the two streams together.

</comment></comment><comment><USER30> Open Garden already does this. Plug and chug. Pretty straight forward.

</comment><comment><USER31> Is it possible to use one for uploads and one for downloads?

</comment><comment><USER32> iPhone's Siri does something similar, but for a different reason: reliability. It multipaths each query.

<comment><USER19> Siri does this in a more systematical way. It uses Multipath TCP, which is in transport layer. This is an application layer hack. This one is actually less convenient and less efficient (tunnelling in proxy adds extra headers and reduces available data space in each packet).

EDIT: typos

</comment></comment><comment><USER33> moderns systems comes with TCP Multipath enabled.</comment>