Machine Learning 101 slidedeck: 2 years of headbanging, so you don't have to - <AUTHOR> https://docs.google.com/presentation/d/1kSuQyW5DTnkVaZEjGYCkfOxvzCqGEFzWBy4e9Uedd9k/preview?imm_mid=0f9b7e&cmp=em-data-na-na-newsltr_20171213&slide=id.g183f28bdc3_0_90 <comment><USER3> As someone who works with a lot of people new to machine learning, I appreciate guides like this. I especially like the early slides that help frame AI vs ML vs DL so that people can have a realistic understanding of what these technologies are for.

For my part, one of the biggest realization I had after many years of applying machine learning was that I got too caught up in the machine learning algorithms themselves. I was often way too eager to guess and check across different algorithms and parameters in search of higher accuracy. Fortunately, there are new automated tools today that can do that automatically.

However, the key piece of advice I'd give someone new to machine learning is not to get caught up in the different machine learning techniques (SVM vs random forrest vs neural network, etc). Instead (1) spend more time on translating your problem into terms a machine can understand (i.e how are you defining and generating your labels) and (2) how do you perform feature engineering so the the right variables are available for machine learning to use. Focusing on these two things helped me build more accurate models that were more likely to be deployed in the real world.

Feature engineering in particular has become a bit of a passion of mine since that realization. I currently work on an open source project called Featuretools ([https://github.com/featuretools/featuretools/](https://github.com/featuretools/featuretools/)) that aims help people apply feature engineering to transactional or relational datasets. We just put out a tutorial on building models to predict what product a customer will buy next, which is a good hands on example to learn from [https://github.com/featuretools/predict_next_purchase](https://github.com/featuretools/predict_next_purchase) for beginners.

<comment><USER4> Don't you think people are, sometimes, just applying ML to their problem "because of hype" ?

One example I have in mind, was a contest where participants were given a series of satellite pictures and asked to write a classifier to detect icebergs and cargo ships (the two are quite similar). As someone else pointed out, trying to use classical computer vision and machine learning on these images will always have some error rate during identification. However, if we were able to extract speed and trajectory of all objects in the picture and mixing them with AIS data, finding which ones are ships, which ones are giant pieces of ice, and which one are non-moving structures to be avoided, becomes easy.

So, you have to choose between a black box that will give you potential results with a given error-rate, and a predictable algorithm that anyone can audit. Seems like a no-brainer situation to me. For what other reason would you choose the first solution, except hype-related decisions ?

<comment><USER5> Your comparison seems like a false dichotomy, and I think you are agreeing with OP. OP says, spend less time worrying about the algorithm and more time worrying about what data you are feeding the algorithm. You are saying, what if you had to choose between dataset A with algorithm A and dataset B with algorithm B.

You claim, (probably correctly) that dataset B, which includes velocity and trajectory, is more correct for the problem at hand, and given dataset B, I would suggest that either algorithm A or B would probably do just fine.

You also claim that algorithm A has "some error rate during identification." But so will algorithm B, and so will either algorithm on dataset A and B!

The question you should ask is, how much do I care about "black box" vs. "white box", and is there are trade-off? If the black-box solution (algorithm A, the "ML" solution) gives you 10% higher accuracy, and that accuracy is going to save lives, you bet I'd choose it. Or maybe I decide that interpretability is really important due to external audit reasons, so I need the white-box solution. But maybe I'd choose both, the interpretable one, and use the uninterpretable one as a flag for "a human should look at this." Or maybe I'd combine the results of both algorithms to get even higher accuracy.

There are just so many ways to configure a solution to the problem you propose, and you are only distinguishing between two of them. In the end the appropriate choice depends on context.

</comment></comment></comment><comment><USER6> The presentation goes straight from from linear regression and classification to _computer vision and reinforcement learning_.

The practical value of ML/AI is what's _in between_ and is something that isn't often discussed between all the hype. ML/AI can be used to build models which work well with nontabular data (e.g. text and images), and can solve such regression/classification problems more cleanly. (and with tools like Keras, they're as easy to train and deploy as a normal model)

<comment><AUTHOR> I think slide 12 touches on this. Even in the case of an image we can process it pixel by pixel, but that would be lunacy!

For text great results have been achieved using automatons, but they only work for structured strings and break if you add only a little bit of noise.

I feel like ML should be considered whenever you feel like programming something requires you to deal with many different cases, you have a lot of example data available, and having some false positives / true negatives is not a big problem.

</comment></comment><comment><USER7> If I understand correctly those are slides from a Googler (Not sure if those slides have corporate approval), that probably have as a side goal to showcase that Google is a fun place to do ML.

Not that I am judging or anything but, the author's personal website [http://www.jasonmayes.com/](http://www.jasonmayes.com/) whose link is displayed multiple times is a giant ad to get hired elsewhere and show at least some desire for other career opportunities. Not sure if that reflects greatly on the company.

<comment><USER8> Checking his website, it reeks of narcissism. There are better ways to assert yourself than to do all the corny things he has done on his self promotion website.

<comment><USER9> Are you honestly slagging a guy off for talking about himself on his _resume_???

I mean yeah, we computer folk are supposed to be all self deprecating and all. But if there is one place we should stop mumbling and talking ourselves down for a second, that is it.

At some point if you want people to know what you do, you're going to have to tell them.

<comment><USER8> I found his approach tacky, loud and insincere.

Of course you should be talking about yourself on your resume but a couple of this that are different here:

\- Wtf is up with music \- 51%/49% thing. \- Publicly asking to be hired that reflects poorly on his current job at Google. \- Excessively loud self marketing

why not have a simple site with your accomplishments? Why all the excess bullshit?

<comment><USER10> I think that's definitely a cultural thing. It would be in poor taste in my country, but seems very American.

And many of the things he did do sound like he could be a good contributor - <USER2> guess you can't know about his personality without an interview.

</comment></comment></comment></comment></comment><comment><USER11> Slide 64: _A whole tonne of stuff going on in robotics right now. Just take a look at Boston Dynamics YT channel for some mind bloding research, most of which is driven by ML._.

I _highly_ doubt that BD is doing _any_ ML work right now ... Can the author link to specific research that they are doing using ML?

<comment><USER12> You mean public work perhaps? I imagine they are doing a lot with vision, gait learning, object manipulation, task planning, autonomy, multi-robot coordination, etc. all of which can be enabled by or at least helped along by machine learning, no? Your request for links is valid, I just am surprised anyone would doubt that they are doing ML research unless you are thinking of a strangely narrow definition of ML.

<comment><USER13> Most of robotics is about reinforcement learning.

EDIT: Oh, and expert systems/rules. Lots of em.

EDIT2: Well, an engineering, obviously... :-) Heck, just check Wikipedia on the topic...

<comment><USER12> >Most of robotics is about reinforcement learning.

is != always will be.

But forget that, let's check Wikipedia, as you suggest.

What are the first few words of the article on Reinforcement Learning, hmm. The very first few words at the very beginning of the article:

"Reinforcement learning (RL) is an area of machine learning..."

Read that and tell me what the last two words are. "Machine learning."

</comment></comment></comment></comment><comment><USER14> Good slides, got me back in to the fever of wanting to learn; although a lot of the credit goes to the linked 3Blue1Brown videos (whose Linear Calculus series is excellent) which were a lot more technical but no less approachable.

Question to those versed in ML: I want to work on an AI that plays a video game (aspirations of playing something like Rocket League, but I know I need to start smaller with something like an old NES game). I understand these are usually done with Recurrent Neural Networks, but I'm a little lost as to how to get data in to the RNN -- will I need to make another AI or CNN to read the screen and interpret (including the score?) My 30k ft view is that if I can define a 'score', give it a 'reset' button, and define 'inputs (decision targets)', then I just need to give it the screen and let it do its thing. But getting the 'score' is the part I can't figure out short of adding another layer to the classifier.

<comment><USER15> You should check out Berkeley's deep reinforcement learning course[1]. There's lecture videos, slides, and homework assignments, and it's all very up-to- date.

[1] [http://rll.berkeley.edu/deeprlcourse/](http://rll.berkeley.edu/deeprlcourse/)

</comment></comment><comment><USER16> The document is awesome, but the animated backgrounds are distracting.

<comment><USER17> exactly, I stopped at the second slide because of that. "I ask for your undivided attention for two hours" is what it says, the background animation seems not helping that goal, quite the opposite.

</comment></comment><comment><USER18> Reading normally, and skipping the videos, the whole deck takes about 15 minutes. The last 3rd of the slides are basically promotional material for the various Cloud ML services that are out there.

It's nice deck, but I'd hoped the blue slides went more technical without dropping out to various videos. If wanted videos, I'd go to youtube directly. Not everyone wants to learn through watching people talk. I learn best when I read, it's unfortunate that youngsters these days think that the written word is now a poor cousin to flashy video.

<rant>

In the same way that new clothes are no longer for me, and new music is no longer for me, and all good TV shows and films are full of people half my age, I also now feel that I'm being aged off the internet.

I was here first, you young whippersnappers! It's MY lawn.

</rant>

</comment><comment><USER19> Cool presentation ... but there's a million ones like this. We don't need yet another basic introduction to machine learning, we need detailed practical studies of real problems.

<comment><USER20> Easier to curse the darkness than to light a match.

</comment></comment><comment><USER21> Loading...

Google Slides is really slow. That's why this needs two hours.

Most of the real content is in linked videos.

</comment><comment><USER22> "There is no golden road to geometry"

If you really want to understand you would be much better off starting here:

[https://work.caltech.edu/telecourse.html](https://work.caltech.edu/telecourse.html)

</comment><comment><USER23> I was absolutely convinced by the title that this would be a link to a research blog post about an analysis of hair motion at metal shows.

<comment><USER24> Me too; After I read Perceptron I fast forward lol

</comment></comment><comment><USER25> This ends up not being much more than an advertisement for google. Wikipedia's articles on these subjects have more depth.

</comment><comment><USER26> Information is great, but it would be much more readable in simple text form or pdf. It's strange that senior creative engineer at Google doesn't know presentation making basics.

<comment><USER27> It's not surprising that a Google engineer would use Google docs. It's at least easily shareable and there are complementary embedded videos that aren't suitable for text/PDF anyway.

Though, the options to export as a PDF didn't work for me (either via download or as an export to Google Drive). I'm assuming the presentation is too big.

<comment><USER28> Can't download the PDF as well.

<comment><USER29> This might be intended behavior. If you try to print the deck, it shows the Goole Drive access request page.

</comment></comment></comment></comment><comment><USER30> strobe morphing advertisements that are trained to adapt to your unique reaction(s) until it detects the same facial response as the last time you made a purchase. botox, masks, camouflage tattoos, permanent smiling, IR obscurants. the arms race begins a new. benevolent dictatorship by self regulating machines is probably going to plague us for several thousand years. wouldn't that be interesting, if the last evolutionary bottleneck is getting smart enough to create an enclosed planetary system that fully satisfies your biological needs and suppresses all intangible ones so there is no reason to go any further up the chain of exploration.

stopping death means stopping life, every generation is more forgetful of the past than the last. the light ages will be far more destructive, what could possibly motivate you to stop a perpetual pleasure machine? how do you prevent the inevitable conflict between those who insist pain and suffering is an essential part of the human experiment and those who just force them to feel good and change their mind? what will happen to these toddlers in 10-15 years? they will have grown up interfacing with some electronic device for every single day of their young lives, a different type of consciousness shaped by destruction of self-confidence in their own knowledge and memories and a complete trust in the needle-finders of big hay.

this shift will be as important as pre-writing to post-writing, except the transformation won't take centuries and millennia to propagate itself across the planet. a post-memory world, with every human enslaved by their base sensations. the first US president who is an internet addict.

"[Writing] will create forgetfulness in the learners' souls, because they will not use their memories; they will trust to the external written characters and not remember of themselves. The specific which you have discovered is an aid not to memory, but to reminiscence, and you give your disciples not truth, but only the semblance of truth; they will be hearers of many things and will have learned nothing; they will appear to be omniscient and will generally know nothing; they will be tiresome company, having the show of wisdom without the reality."

the future is a fate worse than death.

<comment><USER31> That's a strongly pessimistic view you got here. Are you aware that there's people thinking the opposite of you? The thing is: the reality is always far more balanced that what the extremists are preaching us.

<comment><USER30> is being disagreeable sufficient claim to argument?[0] is there a profit motive in balance?[1] people with vast weapons of control, deception and war shape reality, is it balanced?[2]

[0][1][2] no

<comment><USER31> I'm sorry, I just don't see what we are argumenting about, to be honest.

<comment> purple-again Dude look at the way he expressed his thoughts. Either a troll or a loon. Best not to engage him either way.

</comment></comment></comment></comment></comment><comment><USER32> Nice introduction, but I really don't see how "2 years of headbanging, so you don't have to" applies.

<comment><USER33> I think the author meant "banging my head against the wall" while getting it all working but didn't realize the term has a very different meaning...

</comment></comment><comment><USER34> I created nearly the same presentation this week. It's good to see that I didn't miss much, though this one goes deeper, which I don't do on purpose. I'll probably send the attendee this presentation afterwards for those who want to go deeper, very cool!

</comment><comment><USER35> Pretty similar information to my semester-long Machine Learning class in undergrad (with less detail). Good information to get the basics down, but can't say I've gone to apply any of the information I have learned yet... Still a useful set of slides.

</comment><comment><USER36> Not bad but toward the end it basically just becomes a big pitch for Google's ML products. It links to 3Blue1Brown's videos which are great!

</comment><comment><USER37> I really appreciate you sharing this, as myself always try to create a learning path and make things clearer as much as possible on top of it, slides are good looking (design awesome). the problem sometimes comes is at its overwhelming when advanced things pop up in talk and its hard to know what should we listen to it or is it above our level of the current situation.

</comment><comment><USER38> I honestly don't really see the value in a slidedeck, without the accompanying talk. It's the same as when someone proclaim: "Slides from this talk is available online", yeah that's not really any good without video or audio.

<comment><USER39> Disagree. Slides, even out of context slides, can be an excellent source of information for people new to a field: they still give a sense of the structure of the talk (what is related to what), and they give you keywords to start searching for. And for experienced readers, sometimes they just contain nice ideas or tips you had not been aware of.

<comment><USER40> No they aren't. Slides are a skeleton. Just bullet points that 99% of the time offer no usable information

</comment></comment></comment><comment><USER41> How can I download this slide?

<comment><USER42> Going to the /export/pdf link shows access denied. My quick 3min workaround was to use FF dev tools, settings enable screenshot, click through and save each page.

<comment><USER43> 96 x 2 clicks too

such modern

</comment></comment></comment><comment><USER44> This make it look like all interesting ML projects are by Google.

</comment><comment><USER45> Probability distributions are approximate world models. There are mathematical tools to management them. Once you see the world this way, ML/DL becomes more intuitive.

</comment><comment><USER46> >Artificial neuron (aka perceptron)

These aren't synonymous, a perceptron is a type of artificial neuron.

Also confusingly, 'multi layer perceptrons' might not contain perceptrons at all.

</comment><comment><USER47> As someone who has recently taken an interest in machine learning and AI, this deck is much appreciated.

</comment><comment><USER48> Thank you for this. This is an excellent slide deck

</comment><comment><USER49> Why is the file protected.. how to access it

</comment><comment><USER50> What was he headbanging about in these last 2 years? Just a linkbait-y title?

</comment><comment><USER51> I really wish that they hadn't decided on having moving images behind the text you are supposed to be reading.

<comment><USER52> _Give us your undivided attention._

Does everything to distract us.

</comment></comment><comment><USER53> The slide titled "A note on dimensionality" reminded me of this xkcd: [https://www.xkcd.com/547/](https://www.xkcd.com/547/)

"That would be (very) bad."

</comment><comment><USER54> Gold!</comment>


Ask HN: Large Conference Call setup help - <AUTHOR>

In the coming weeks, I will be having 50 people in a room conversing with one another.
I am also going to have 10-20 people calling in from their phone who also need to be heard and be involved in this conversation.
Do you guys have any recommendations to accomplish this? <comment><USER2> [http://www.uberconference.com/](http://www.uberconference.com/) Amazing product</comment>


Bad Character: If Chinese Were Phonetic - <AUTHOR> http://www.newyorker.com/magazine/2016/05/16/if-chinese-were-phonetic <comment><USER2> Is it just me, or does this essay seem to be missing an introduction, like someone accidentally deleted the first few paragraphs? I initially assumed I had been linked to "page 2" and went looking for a link back to the beginning.

Edit: Apparently, this is part of a series where guest authors are invited to choose something to "uninvent" and explains why they think it has had a negative impact on the world. The essay's abrupt beginning makes much more sense in that context.

<comment><USER3> [http://www.newyorker.com/magazine/uninvent- this](http://www.newyorker.com/magazine/uninvent-this)

For more of the series.

<comment><USER4> This one is great: [http://www.newyorker.com/magazine/2016/05/16/ban- dancing](http://www.newyorker.com/magazine/2016/05/16/ban-dancing)

<comment><USER5> A great read, with awesome quotes, but I wish it had been "ban social pressure to do things you don't enjoy doing."

"Come on, just dance for a little bit, you'll like it!"

"Rollercoasters are fun, trust me, this one is great!"

"Why won't you come eat Ramen with us, we know a great Ramen place! Don't be such a killjoy."

<comment><USER6> Do people really pressure other people into eating? In my group(s), it's always "I'll come for the company, but I'm not hungry", and that's fine.

People don't have to be putting stuff in their face to keep you company.

</comment></comment></comment></comment></comment><comment><USER7> I've seen this sentiment a lot in Chinese-Americans that are not educated in linguistics, along with other self-loathing sentiments.

First, literacy isn't completely related to the writing system. Look at Spanish speaking countries, where the alphabet is more phonetic than the English alphabet.

Second, Chinese characters that are more complex, i.e. consists of more than one radical, are usually composed with a semantic component, giving indication to the character's meaning, and a phonetic component, which gives an indication to the sound of the character. Although this isn't a rule, it helps a lot, and it's not like English doesn't have crazy non-phonetic spellings as well (how tf is "through" supposed to be pronounced for a English learner?)

Last, the Chinese language consists of MANY homophones. This isn't necessarily a bad thing, and not one out of design, but something that is the result of being one of the oldest language families in the world. It allows for the concise expression of many things using only single syllables. You might say, but what about the crazy amount of ambiguity if the language has a lot of homophones? Well, ambiguity is a huge problem in all languages and our brains seem to manage. Now, even though homophones aren't a big problem in spoken language, because of intonation and prosody giving a clue to how to analyze sentences, written language is a different story, and it would be very hard to make an easier system to handle it. For all you engineers, the fact that Chinese has characters is essentially a performance trade off. More information density for more ambiguity.

<comment><USER8> It's actually a myth that most Chinese characters have a semantic component (indicating meaning).

And the phonetic component often doesn't correspond to anything any modern person would know.

The problem in both cases is the shift of language through China's long history, and its divergence from the original design of the written characters.

The problem with the semantic components is that the meaning is re-used and stretched over and over. E.g. they used to use a 'foot' radical to represent a journey towards a destination. Later that shifted to mean "in a straight line, not veering left or right". Later that took on the meaning of "straight and narrow" or "straight shooter" or "not-deviant". Then it becomes, "not deviating from the right path". So now, the old foot radical typically means "justice" or "correct".

And the image shifts over time. This is the old foot radical now: 正 Does it look like a foot to you?

The problem is with the phonetics is language shifts. In many cases, in ancient Chinese, the characters do have a phonetic component that hints at pronunciation. but, the pronunciation changed over the last 2000-3000 years, so the pronunciation hint that made perfect sense in the Han dynasty is now meaningless because you're speaking a different language.

The result is that in the end the characters end up being arbitrary phonetic symbols with some arbitrary meanings attached.

<comment><USER7> You're speaking out of your behind. Of course the signs are arbitrary with respect to their signifiers, that's any language, but there are still mappings between signs and their signifiers. English is no difference, am I supposed to know what a 'm' sounds like by looking at it? Look up the Rebus principle, it happens in every language.

And with respect to the phonetic shifts, from a linguistic perspective, most changes of the phonetic radical involve one change of the initial sound, i.e. bilabial to interdental, dental to alveolar, voiced to unvoiced. With respect to the phonetic inventory of the language, these shifts are only one feature shifts within an old SPE framework. These small featural differences, whether conscious or not, (usually unconscious because we acquire languages during our infancy) are picked up and used by the speaker of the language to categorize words.

<comment><USER9> _You 're speaking out of your behind._

Your detailed and substantive comment works just as well without this uncivil lead-in.

<comment><USER7> it's a much easier way to say, I significantly doubt your credentials as a person educated in both Mandarin Chinese and linguistics

<comment><USER9> HN strongly values substantive and civil discourse. If that requires a little more work on your part, please engage in it to help make HN a productive forum where discussions like this can take place. (That said, I don't see it necessary for the longer version either.)

</comment></comment></comment></comment></comment></comment><comment><USER10> Color me shocked that somehow the author found the alphabet of his native language to be superior.

This article takes as a given some assumptions that I don't understand at all. China being resistant to change? There are few cohesive societies that I can think of that have experienced more change than China since the end of WWII. And it would be an understatement to say that the rise of literacy rates are more strongly correlated with industrialization than adoption of phonetic alphabets.

I think it's telling that the author grew up in a Chinese-American community. Expat communities tend to lag their mother cultures in terms of social progress. An experience growing up in a community could explain this "steeped in tradition" characteristic that is attributed to Chinese culture here.

Disclaimer: I'm a white American. My wife is Chinese and emigrated herself from China as a young adult; I get a lot of my perspective from her. Would be interested to hear what Chinese members of HN think of the article.

<comment><USER11> Here is David Moser also arguing that the Chinese writing system is objectively inferior: [http://pinyin.info/readings/texts/moser.html](http://pinyin.info/readings/texts/moser.html)

<comment><USER10> Not sure if you're trolling or not, but I'll bite.

Did you even read the article you linked to? The author makes so such argument. The point of the article is that it's extremely difficult for non- native speakers to learn written Chinese.

<comment><USER12> "What I mean is that Chinese is not only hard for us (English speakers), but it's also hard in absolute terms. Which means that Chinese is also hard for them, for Chinese people."

Author is saying it's unnecessarily hard for everybody.

<comment><USER10> Point conceded, but it still doesn't take away from the broader point which is that the parent makes a false assertion:

> David Moser also arguing that the Chinese writing system is objectively > inferior

<comment><USER10> @waqf my point is that being harder to learn the written language does not mean that the language is _objectively_ inferior as a whole. There is no universal set of objective metrics to judge the value of a language, and such an assertion approaches cultural chauvinism.

</comment></comment></comment></comment></comment></comment><comment><USER13> For context, to forestall some knee jerk reactions I'm seeing:

1\. This essay is part of a series, where guest authors are explicitly asked to "uninvent" something. So there's a level of built-in hot take to this.

2\. The author is Ted Chiang, writer of Story of Your Life (and the work that the movie Arrival was based on). He's explored how language affects how we think before (Story of Your Life/Arrival is explicitly about this). So this kind of falls in line with that.

Anyway, I think this is a fascinating thought experiment to work through. What _would_ China be like with a phonetic system? What would change? How much of the culture is derived from the method of language and how much from other factors?

</comment><comment><USER14> _> I would never have to read or hear any more popular misconceptions about Chinese characters—that they're like little pictures, that they represent ideas directly, that the Chinese word for "crisis" is "danger" plus "opportunity." That, at least, would be a relief._

But... they are? They're ideograms and there is at least _some_ reasoning to many of them. Things written with a part of "fire" tend to have a relation to..well...fire and heat.

火, 烧 炊. You might see the relation 秋 shares with fire. But don't fall into the trap that it is 100% consistent because you'd be wrong about 約.

And the last statement isn't even (technically) _wrong_. I'm not sure if the Chinese word is the same [0] but for Japanese it is 危機. With individual readings of "danger" and "opportunity" although a more reasonable 1-to-1 equation would be "danger" and "occasion". A crisis _is_ a dangerous occasion. 機 happens to have several meanings and one of those is "opportunity". It is probably intentionally misleading but not incorrect to say it is danger + occasion.

You can't have your cake and eat it too. For most compound words the individual meanings of the hanzi/kanji are very relevant - maybe not _always_ relevant but more often than not.

[0] Google Translate tells me it is, so my confidence is >0% but not by much.

<comment><USER15> Most of the time (In Japanese, at least), the Kanji do not map to meaning per- se, but to roots. For example, in English, we have words like "lap-top", "uni- cycle", "roof-top", "blue-berry", that clearly have multiple semantic components within them. In these cases, in Japanese, the words would be written with the kanji for their roots. In more linguistic terms, the kanjis often refer to morphemes, not ideas.

There are some exceptions. For example, in Japanese, today (kyou) is written as "今日", even though it is a single morpheme. In cases such as this the kanji are used for their meanings. In cases such as this, is is not clear how much the kanji actually help, because the sementic information provided is precisly the information that you would get from knowing the roots, and you (or rather a native speaker) does not have to be taught roots (or at least those roots that still have semantic meaning).

<comment><USER14> I'm not sure what you mean by separating "meaning" from "roots". To myself they are one in the same (the root word or a meaning of the root "word", there is no difference in my opinion). Could you provide some examples of what you mean? Maybe I'm thinking too literally or you meant "root" (as in radicals) different from "root" (as in root word).

「自転車」, 「図書館」, 「作家」, 「日光」,「棋士」. You even have witty ones like 「親子丼」. Just have to be careful about ones like 「多少」.

The reason I would say not to rely on such literal meanings to define words is because some require some stretch of the imagination like「子宮」 and my rule of thumb is if it it requires a stretch of the imagination it's because it doesn't actually mean that. But literal translations can still be useful as mnemonics to learn new words/kanji. Like a pair of training wheels that eventually get removed.

</comment></comment></comment><comment><USER16> One of my favorite Ted Chiang short stories also explores how much is shaped by our language: [http://subterraneanpress.com/magazine/fall_2013/the_truth_of...](http://subterraneanpress.com/magazine/fall_2013/the_truth_of_fact_the_truth_of_feeling_by_ted_chiang)

</comment><comment><USER17> For those unaware, Ted Chiang wrote the short story that the 2016 film Arrival was based on.

</comment><comment><USER18> >Imagine a world in which written English had changed so little that works of "Beowulf"'s era remained continuously readable for the past twelve hundred years.

This works with an alphabetic system: in Iceland kids read the sagas in school.

<comment><USER11> And in fact, written English has changed very little in the past five hundred years or so: it's easy to read Shakespeare and not too hard to read Chaucer, although their pronunciation would have been considerably different. Before that period there was a much greater rate of change in the written language.

<comment><USER19> Aside from the spelling, Chaucer really is not too bad. I must have had to read a really, really abysmal treatment in school, because I did not remember the Canterbury Tales rhyming at all...

[http://www.librarius.com/canttran/gptrfs.htm](http://www.librarius.com/canttran/gptrfs.htm)

</comment></comment></comment><comment><USER20> It is the intense educational requirements of the Chinese character system that led to the development of Hangul-am, the Korean character system. You still end up with a similar block structure, but it is phonetic and has about the same number of components as there are letters in the English alphabet.

<comment><USER21> That might be true, but an (perhaps more) important reason is literally the first 8 (chinese) characters in Hunminjeongeum [1], translated as follows:

> Because the speech of this country is different from that of China, it [the > spoken language] does not match the [Chinese] letters.

[1]: [https://en.wikipedia.org/wiki/Hunminjeongeum](https://en.wikipedia.org/wiki/Hunminjeongeum)

<comment><USER20> Yes, nationalism is an important consideration. I did not consider that in my opinion. I will now freely admit that there are probably other good reasons I have not considered. Thank you for the reference.

</comment></comment></comment><comment><USER22> Some linguists (mostly Western) have advocated transitioning to a mixed system of characters and pinyin. I don't think that will ever happen because an uglier and less harmonious writing system could not possibly be devised. Chinese would be much better off with something syllabic and square, like Hangul. With some amount of fiddling I think it could be made to work for most of the different Chinese languages and potentially even elucidate cognates between them, retaining one of the main benefits of characters.

</comment><comment><USER23> FWIW, I like Jared Diamond's thesis better, which is that the geography of China (a big habitable landmass with no sizable peninsulas or isolated areas) made a single political entity inevitable. It probably explains why the Chinese culture has more emphasis on tradition (if that is true, I mean): you can more easily identify yourself with your ancestors from 500 BC, because they are all Chinese. A Spaniard would have a harder time identifying with Celts, Romans, or Moors.

<comment><USER24> I mostly agree: 500 BC might be a tad too remote in time, but while risking being pedantic, Spaniards do identify themselves with Moors [1] or Romans [2] often, in most traditions.

[1] [https://en.m.wikipedia.org/wiki/Moros_y_cristianos](https://en.m.wikipedia.org/wiki/Moros_y_cristianos) [2] [http://www.spain.info/en_US/que- quieres/agenda/fiestas/murci...](http://www.spain.info/en_US/que- quieres/agenda/fiestas/murcia/fiestas_de_carthagineses_y_romanos.html)

</comment></comment><comment><USER25> I think in a software startup mindset it could be easy to think that more efficient => better, but a point not mentioned in the essay is that written Chinese is simply beautiful compared to e.g., the simple 26 character alphabet, which is quicker to learn. This also applies to some of the other alphabets such as Hindi, Arabic, etc. But Chinese has so many characters, it really shines.

</comment><comment><USER26> As someone who knows a bunch of different scripts (Perso-Arabic/Urdu, Brahmi- based scripts like Devanagari, Kannada, Tamil, Bengali and obviously the Roman script), scripts do make a difference.

Yes, a lousy script isn't a fatal impediment. You can produce beautiful literature even with a poorly-designed script. And I suspect script complexity is only weakly correlated to literacy. Learning to read involves a lot more than recognizing characters and words. But a bad script definitely adds a layer of confusion that is completely unnecessary and ends up creating an elitist and cumbersome language.

The standard counterpoint to complaints about scripts is some form of whataboutism involving English. Yes, English is not remotely phonetic, but the modern form of the Roman script for English is actually pretty decent. It has a lot of visual differentiation between different letters. Compare this to Persian and Arabic where the placement of a dot or three makes all the difference between wildly different sounds.

The Roman script also has relatively few letters. This comes at the cost of ambiguity, and we need to make up sequences for certain sounds (th, sh, ch), but also means the learning curve is a lot less steep. Persian and Arabic have the huge mess involving initial, medial, final and standalone forms of a letter -- it serves no purpose really. Brahmi-based scripts are all abugidas which means vowels are folded into adjoining consonants and so you need to learn about 4X as many symbols as English.

You do have to guess what a letter sounds like in English (g,j,k,c). But this is also pretty common. Tamil doesn't distinguish between voiced and unvoiced consonants. This is an unnecessary annoyance. A much worse example is of Persian, Arabic and Urdu where the script allows one to drop a lot of vowels. This is especially bad for Urdu. For example, the Hindustani words for here (idhar) and there (udhar) are written identically in Urdu, so you need to "backtrack" to fill-in vowels appropriately based on the rest of the sentence.

Yes, English isn't phonetic, and is unstructured and unsystematic. But it is still a pretty decent script compared to some of the others out there. From an Indian perspective, I wish we could move to one script -- perhaps some variant of Devanagari for all our languages. There's really little purpose in differentiating between Gurmukhi and Devanagari. And we should just forget about Urdu all together. It made sense when the goal was to teach people one script that gave them access to both Hindustani and the official court languages of Persian/Arabic. But for today, Devanagari is vastly superior for writing any dialect of Hindustani. Similarly, Kannada has an almost bijective mapping to Devanagari, but each symbol in Kannada is just so much more elaborate and a pain to write. Life would be much better with one script.

Bring up changing scripts though and people -- and this appears to be a global phenomenon -- just go nuts! People need to realize that changing scripts isn't a terrible thing and has occurred many times for many languages. We're not going to forget our languages/culture just because we choose to change the symbols we use in writing.

<comment><USER27> Perhaps someone with a linguistics background can chime in but my (layperson) understanding is that many writing systems start out pictographic and evolved over time, become syllabaries, then eventually alphabets. The Chinese writing system represents an incomplete transition. Egyptian represents a mixed but mostly complete transition, as by the late third period their writing system was mostly based on the sounds of the symbols, not the literal concepts they originally meant (e.g.: using a snake symbol for the "s" sound). Middle period writings are often a mix of all three: some symbols used for their literal meaning, some symbols used to represent syllables, and some for their sound. Sometimes (but not always) there were markings next to the symbols to indicate what "mode" you should read them in.

All YMMV of course, I'm not an expert. I try to avoid being too euro-centric as it is easy for me to claim alphabets are superior because that's what I know, but it does seem like memorizing abstract rules and a small set of letters is easier than memorizing thousands of characters. It is certainly easier to deal with by machine (whether that be a typewriter, a terminal, or designing a modern OpenType font).

</comment></comment><comment><USER28> > smartphones are impossible to use if you're restricted to Chinese characters

This isn't actually true. Handwriting recognition is much more popular than any other means of text input in Hong Kong.</comment>


Zeitgeist: Dependency Management for DevOps - <AUTHOR> https://blog.florentdelannoy.com/blog/2020/introducing-zeitgeist/ <comment><USER2> The Terraform example doesn't seem particularly useful to me, as something that runs Terraform would be on the CI side. I don't need to worry about dependencies locally or anywhere else because it should never ever be ran anywhere but the CI server, which we would steward over.

The AMI part of it I believe is also typically done with a "data" block to grab a new one, so I'm not sure what in particular is added there?

Dependency management for devops sounds reasonable, but to me it seems more like trying to make an NPM/Cargo/pip that is for every language instead of a particular language.

<comment><USER3> >The Terraform example doesn't seem particularly useful to me, as something that runs Terraform would be on the CI side. I don't need to worry about dependencies locally or anywhere else because it should never ever be ran anywhere but the CI server, which we would steward over.

Running terraform fmt locally is useful to be able to do even if the apply runs in CI. And for the 0.11->0.12 migration being able to run terraform 0.12upgrade locally was useful.

>Dependency management for devops sounds reasonable, but to me it seems more like trying to make an NPM/Cargo/pip that is for every language instead of a particular language.

This seems to me more of a dependency checker than a traditional manager which is useful if you pin dependencies (and for reproducible builds and identical prod/test envs you really should). There's a number of companies which do this for security on live environments (scanning your running containers, registry, AMIs, etc.).</comment></comment>


Congress May Shift Climate Research Away from NASA - <AUTHOR> https://www.scientificamerican.com/article/congress-may-shift-climate-research-away-from-nasa/ <comment><USER2> _Beyer said it's clear that the GOP is targeting research because it doesn't like the information it imparts, and that he expects it will soon come after NASA's climate change work._

These folks don't believe in climate change and in Darwinian evolution. And they want to make "America great again"!</comment>


Ask HN: OCR framework for extracting formatted text? - <AUTHOR> I'm a serial information hoarder, and often use screenshots in order to store comments, passages and fragments of conversations I find useful or insightful. This works well if I want to reference something recent, but obviously doesn't scale well. I'd like to integrate these into my personal archive, but don't know any frameworks (preferably for Go, Node, or Python) which could automatically extract the text from the images while retaining its formatting. I'm not against doing some image preprocessing myself, but I don't feel comfortable passing the images to a 3rd party service, since a portion of the images contain private or sensitive information that I can't readily sort out of my collection. <comment><USER2> To extract text from photos and non OCR-ed PDFs Tesseract[1] with language specific model[2] never fails me.

I use my shell utility[3] to automate the workflow with ImageMagick and Tesseract, with intermediate step using monochrome TIFFs. Extracting each page into separate text file allows to ag/grep a phrase and then find it easily back in the original PDF.

Having greppable libraries of books on various domains and not having to crawl through the web search each time is very useful and time-saving.

[1] [https://tesseract-ocr.github.io/](https://tesseract-ocr.github.io/)

[2] [https://github.com/tesseract-ocr/tessdata](https://github.com/tesseract- ocr/tessdata)

[3] [https://github.com/undebuggable/pdf2txt](https://github.com/undebuggable/pdf2txt)

<comment><USER3> I struggled to get tesseract to OCR my image based PDFs directly, so resorted to using GhostScript to extract the pages to pngs which I then put through tesseract. As an added bonus though, I gained the ability to have a thumbnail png for the search front end.

<comment><USER2> > so resorted to using GhostScript to extract the pages to pngs which I then > put through tesseract

I understand you use this to extract text from non OCR-ed PDFs, especially consisting of low quality scans or photos (e.g. low resolution, JPEG artifacts).

Ocassionally passing higher resolution to ImageMagick when converting a page to TIFF helped, but this sounds like a reasonable fallback as well.

</comment></comment></comment><comment><USER4> I built an application for exactly this. It's called A Personal Search Engine, APSE for short.[0]

It OCRs screenshots and stores the text in a search index, so you can query by keyword, date, boolean operators, the whole shebang.

It's all local. It is _really_ useful for me - yesterday it saved me after Firefox wigged out and lost all my tabs. It's in a great place to try out, and I am actively developing it.

[0] [https://apse.io](https://apse.io)

<comment><USER5> That's cool. How can I make sure it does not send my data somewhere? :)

<comment><USER3> Ahh the paranoia of HN strikes again.

Maybe, stop and think before you ask this. Someone offers up an example of their hard work and you instantly accuse them of being a malware author that steals your data. Nice.

<comment><USER6> I would rather call it the new-found paranoia most legal departments now have, when IT dept. mentions rolling out new, unknown, non-auditable software in the company from a new vendor that wasn't grand-fathered in (= the legacy windows/cisco deployments). I'm happy about it. I'm just waiting for them to forbid closed-source firmwares/hardware.

</comment></comment></comment></comment><comment><USER7> ocrmypdf and friends.

I've built an archival system based around Tesseract and PostgreSQL. It takes Images/PDFs, either scanned or generated, and rebuilds them as searchable PDFs before being extracted and inserted into Postgres' full-text search. I keep all of the original media because disk is cheap.

Originally I used Tesseract directly. But I found that ocrmypdf did a better job than my home-grown pipeline, so I switched.

<comment><USER8> I also built a system that extracted structured and unstructured text from images/pdfs. For the generated pdfs, I found pdftotext could pull with 100% fidelity, and so that was 'option #1'. for scanned-images-saved-as-pdfs, then tesseract could sometimes extract with 90+% accuracy. But never 100%. Combining pdftotext (with the right flags set) with some of the other associated pdf-tools, we were able to achieve what we were after: Building a searchable DB and auto-informing corpus of information derived entirely from various pdf sources. All in-house. No sending off to 3rd parties.

<comment><USER2> > For the generated pdfs, I found pdftotext could pull with 100% fidelity, and > so that was 'option #1'. for scanned-images-saved-as-pdfs, then tesseract > could sometimes extract with 90+% accuracy.

Arrived to a similar conclusion although never have bothered with DB or any web interface running locally. Simply grepping the text files works flawlessly for me.

</comment></comment></comment><comment><USER9> Here's a blog post showing self hosted PyTesseract finding text in an image and preserving the format: [https://stackabuse.com/pytesseract-simple-python- optical-cha...](https://stackabuse.com/pytesseract-simple-python-optical- character-recognition/)

There's a reason why the external services are popular though...lots of training data and tweaks to make them much more accurate. Try the Google demo here, for example: [https://cloud.google.com/vision/docs/ocr](https://cloud.google.com/vision/docs/ocr)

</comment><comment><USER10> Although [https://www.willus.com/k2pdfopt/](https://www.willus.com/k2pdfopt/) is meant for reformatting PDFs to view on e-readers, it does do a reasonable job of extracting text via OCR and storing as a PDF layer. The underlying engine can be either [https://github.com/tesseract- ocr/tesseract](https://github.com/tesseract-ocr/tesseract) or [http://jocr.sourceforge.net/](http://jocr.sourceforge.net/)

</comment><comment><USER11> If you are (a) willing to take the word bounding boxes and convert them to paragraphs yourself, and (b) okay with a deep learning approach, you may want to give keras-ocr [0] a try.

Full disclosure: I'm the primary package developer. Shameless plug. :)

[0] [https://github.com/faustomorales/keras- ocr](https://github.com/faustomorales/keras-ocr)

</comment><comment><USER12> This doesn't meet most your requirements, (Go, Node, Python, and it's a manual process...) but... maybe this would be helpful?

On Mac I use a modified version of this Keyboard Maestro script, to OCR a user selected area of the screen.

This script will result in the OCR Text on the clipboard. I'm sure Keyboard Maestro could automagically append it to a text file or something. I'm kinda a noob with Keyboard Maestro, so I don't know all of it's functionality.

I have a couple variations of this script, one that will use the Mac's speak this command to read aloud the OCR text, as I am a slow reader, and an auditory learner.

My father had a bunch of newspaper clippings scanned into the family tree application and wanted the text. I used this method to get the text instead of typing it all out.

[https://forum.keyboardmaestro.com/t/ocr-user-selected- area-m...](https://forum.keyboardmaestro.com/t/ocr-user-selected-area- macro-v9-0-1/15054)

</comment><comment><USER13> OCR techniques are general purpose in trying to map any conceivable text- looking shapes into actual text. Accuracy can vary wildly but the good ones will match against plausible words to eliminate low quality guesses.

Is there an accuracy optimization to be found if I can pre-train the OCR engine to look for a limited set of words instead of the entire dictionary- and printable character space?

The use case I have is OCRing shipping labels for packages that arrive at an office. The set of plausible matches is incredibly small as it is the set of employee names that work in said office.

Further optimizations include reducing the problem space by only considering computer printed glyphs and not bothering with handwritten labels, and the insight that the distribution of packages follow a power law where a disproportionately small group of people receive the largest number of packages.

The end goal is to perform this entirely on device, with low latency and high accuracy.

<comment><USER14> Consider looking into language models such as KenLM. It is used by ASR models like wav2letter and DeepSpeech to correct speech-to-text transcripts

</comment></comment><comment><USER15> Try [https://screenotate.com/](https://screenotate.com/)

(no affiliation, just a user)

</comment><comment><USER16> One problem that I have with OCR is dealing with images of pages that are warped. I have some books that I would like to turn into electronic books, but not enough to justify setting up a book scanning rig (framework, two cameras, platen, etc). Setting up a document camera is fairly easy, but using it to take pictures of a book laying flat on the base produces images where the pages are warped and most OCR software seems to have problems with warped pages.

After a fair amount of searching I found ScanTailor: [https://github.com/4lex4/scantailor-advanced#scan-tailor- adv...](https://github.com/4lex4/scantailor-advanced#scan-tailor-advanced) which seems to have the capability of dealing with warped page images. I haven't actually gone through the complete workflow with it yet, but it seems to be a very capable OCR package.

<comment><USER17> I used this[0] in conjunction with Tesseract, and it worked pretty well.

[0] [https://github.com/mzucker/page_dewarp](https://github.com/mzucker/page_dewarp)

<comment><USER16> Thank you. This does look like it has an easier workflow than ScanTailor. I'll have to give it a try.

</comment></comment></comment><comment><USER18> I hate to post my own app but it does do part of what you ask and it does it locally. Nothing is sent to any server.

[https://www.dizzybits.com/Photoplex](https://www.dizzybits.com/Photoplex)

It does on-device text recognition on your photos, stores on local SQLite database and lets you full text search.

</comment><comment><USER19> Site: [https://openpaper.work/](https://openpaper.work/)

Repo: [https://gitlab.gnome.org/World/OpenPaperwork/paperwork](https://gitlab.gnome.org/World/OpenPaperwork/paperwork)

</comment><comment><USER20> [https://github.com/axa-group/Parsr](https://github.com/axa-group/Parsr)

</comment><comment><USER21> I'm interested in drawing bounding boxes around text that can be displayed to the end user. In this way I don't care about OCR accuracy but the ability detect text accurately and across different mediums of type. Thoughts for a framework for this that's low latency under 150 ms or so?

</comment><comment><USER22> You may set up your OCR service on AWS Lambda.

I wrote a guide how to do it here:

[https://typless.com/2020/05/21/tesseract-on-aws-lambda- ocr-a...](https://typless.com/2020/05/21/tesseract-on-aws-lambda-ocr-as-a- service/)

Hope it helps

</comment><comment><USER23> just search for "tesseract GUI". if you are more technical, you can write code around tesseract. for what you get for free, it's really impressive what Google has done with this in just a few years to make it something that the average person can really consider using for free.

ex. [https://github.com/tesseract4java/tesseract4java](https://github.com/tesseract4java/tesseract4java)

</comment><comment><USER24> I know you said you didnt want to upload stuff to third parties but Amazon Textract works great and supports HIPPA data

</comment><comment><AUTHOR> Plenty of fantastic suggestions in the comments, any one of which looks like it could do the trick. Not having any experience in the problem domain, I'm afraid I don't have much to contribute in response, but I look forward to evaluating each framework/service.

</comment><comment><USER25> Why not upload it to Google Photos. It will do the OCR and make the text on your photos / screenshots searchable with a sweet UI in the browser.

If you still want to grab the text yourself you make a copy to Google Keep and use the "grab text" function.

Works for me, I take full screenshots of interesting stuff so the url is still visible when I want to go back to the original.

Obviously I have a paid G Suite account at Google. That comes with a very good set of privacy protecting rules. Doesn't matter how you roll your stack eventually you are going to be dependent on a 3th party. Better use one that offers full encryption and 2FA to lockup your data.

[https://gsuite.google.com/learn-more/security/security- white...](https://gsuite.google.com/learn-more/security/security- whitepaper/page-6.html)

<comment><AUTHOR> I've a number of screenshots concerning conversations, documents, and pii I don't necessarily trust in the hands of third parties, as well as don't feel I'm at liberty to share with third parties.

Beyond that, as exceptional as Gsuite is, I've been making a conscious effort to excise Alphabet/Google services from my life - it's just not a company I trust.

<comment><USER25> Isn't that data already in the hands of 3th parties when it are screenshots of conversations and documents, or did you also build that communication stack from the ground up?

<comment><AUTHOR> I'd frame it like this:

With respect to online conversations - most of them are on the open-web, anyone can see them. I don't care if their content gets out. Private conversations should be kept between their participants, their host, and their host's infrastructure provider.

More saliently however, many of these screenshots contain incidental data which I wouldn't necessarily want to be centralized off of my own hardware. This ranges from the identities of multiple alt-accounts, who they follow on social media, to generic information about my social graph. They also include receipts of much of my online transaction history.

While I'm under no delusion that much of that data doesn't travel all over the universe via data brokers and information sharing agreements, I'm just not comfortable directly handing it all to any one company.

If I was working on a commercial project, I'd leap at the opportunity to outsource the task of content transcription - it would save me time, money, and quite probably give me better results.

But since I want to feed it all into my personal archive, which runs on my own hardware and is as much a learning project as it is a utility, and since I like to keep my personal life as personal as possible, I make a point of keeping everything self-hosted wherever possible.

I'll fully admit that it's paranoid, labor-intensive, likely ineffectual, and by most measures a bit excessive.

But there are few places where one is at liberty to draw a line in the sand anymore with how their data is distributed. This is simply where I've chosen to draw one of mine.

<comment><USER25> Look I fully agree with you if that is what you want, and you are fully aware of the trade-off you make.

When you pull this off you are a very talented skilled engineer. I hope you open source your solution so friction is removed for other people with a similar dilemma in the future.

Our time is the only currency we have and we can pursue activities we love or fear. The line between paranoia or choosing for personal freedom is thin and very personal.

I came to the conclusion for myself I have spend to much time on home grown solution for problems others have solved better and cheaper. Getting from it works 80% of the time to 99% and I can blindly trust my infra is the difference between a weekend and year fulltime work.

I choose for G Suite because at least Google offers me a paid option to exclude my account from their advertisement data monetizing branch.

I do really respect that you make a deliberate effort in this.</comment></comment></comment></comment></comment>


NPR Syndrome - <AUTHOR> http://www.whattofix.com/blog/archives/2011/10/npr-syndrome.php

<comment><USER2> _We're teaching people to put on the affectation of being smart, of having the right opinions, being upset at the right things, listening to the right radio stations, hanging out with the right crowd -- with actually being smart._

I think you meant _without_ rather than _with_ , and I believe your argumentation could be cleaned up a lot. Overall though, I agree that there has been a loss of content in liberal education. One can observe this in any English or Philosophy department at liberal universities, where it's de rigeur to assign a commentary on Plato rather than read his writings, or instead of becoming familiar with the political and religious views of Lady Jane Grey herself, we read a discourse by someone invested in enforcing gender tallying and political correctness. Not exactly a new observation, but one confirmed by books such as _The Closing of the American Mind._

</comment><comment><USER3> Care to include any examples? You wrote a long winded rant without any substance. I took a look around the <http://www.npr.org/sections/news/> and didn't find anything that came close to your straw man.

Meanwhile your characterization of <http://www.npr.org/music/> is a complete joke. Just glance at the categories of music NPR covers in the various colored tabs.

I suppose a kind reading of your writing would be that you misunderstand what you are hearing when your local classical station syndicates NPR content.

<comment><AUTHOR> The point is that we listen to NPR because it is a way of posturing, not that the music is only classical.

The argument is about cargo cult intellectualism. NPR is great for what it is -- but the discussion here is about the way it fits into society, not the types of music it offers.

I'm not sure exactly what your question is. Am I to provide examples of social posturing and signalling? Aren't they fairly well-understood phenomenon?

<comment><USER3> You made the claim NPR Syndrome exists yet provide no examples besides hearing classical music discussed and a made up straw-man news story about Global Warming and puppies.

Then tell me I should just assume NPR Syndrome exists because you need it to for the rest of your piece to hold up? I don't think so. Especially when finding examples hosted on NPR's own website should be trivial.

Not to mention your vomit inducing caricature from Good Will Hunting generalized to a liberal arts education; reminds me of people claiming Steve Jobs and Bill Gates represent great role models for college freshman to follow and go drop out to strike out on their own.

You basically set up the article just so that you can argue against what you believe is the liberal intellectual poser who only exists in your head because it fits your political world view.

</comment></comment></comment><comment><USER4> I like much of your writing Daniel, but I found this piece to be way off the mark.

You're conflating political liberalism and liberal arts ... with disastrous results.

<comment><AUTHOR> Thanks for the comment.

Can you elaborate? Like most mistakes, the person making it is always the last to see it.

My point was that the facade that a liberal arts education brings is many (not all) times associated with a set of behaviors: NPR, certain websites, ability to reason about economics, etc. My point is that to most people dropping 80K or more on an education, the only thing they're getting is a pre-canned set of attitudes and opinions, which is not supposed to be the goal of a liberal arts education at all. So yes, I am conflating them, but that's by design, not accident. Perhaps I should have stated my thesis clearer?

<comment><USER5> _Perhaps I should have stated my thesis clearer?_

You could get a good jump on this by excising all of the sentences where you talk about what you're _not_ doing.

<comment><AUTHOR> The problem here is that there is a larger political argument, which I acknowledge but really don't want to participate in, which goes something like _college only exists to indoctrinate young liberals._ As you can see from one of the other commenters on this article, people take it like that whether I mean it that way or not.

I find this statement to be true, but an oversimplification, and it's not my point at all. I think what we're really seeing is a sociological phenomenon: teaching people to appear as if they've learned something instead of actually teaching them anything. I find this deeper point to be much more important. You might find the same thing at right-leaning universities -- perhaps. (Although there is an interesting inverse relationship on the right. Perhaps at right-leaning universities you would find students who pose as tolerant and worldly yet have no idea why they do so? It bears study.)

Not an excuse for not doing a rewrite, but I wanted to point out that there is a fine line here, hence the "I'm not saying" text. (Still needs a rewrite. Your point is well made. Thank you.)

</comment></comment></comment></comment><comment><USER6> Not particularly relevant, but I've stopped listening to NPR in fear that Diane Rehm will come on and attempt to slowly murder me with her murder voice.

<comment><USER5> It's called Spasmodic Dysphonia.

<http://en.wikipedia.org/wiki/Spasmodic_dysphonia></comment></comment>


A New SubReddit for Buying/Selling/Trading Unwanted Domains - <AUTHOR> https://www.reddit.com/r/DomainsForSaleorTrade/

<comment><USER2> A new subreddit obviously does not qualify to be a Show HN. Please read the rules: [https://news.ycombinator.com/showhn.html](https://news.ycombinator.com/showhn.html).

<comment><AUTHOR> sry</comment></comment>


Microsoft .NET SDK is violating the GDPR, object now - <AUTHOR> https://dgl.cx/2020/08/dotnet-sdk-gdpr <comment><USER2> For a client we added support for some synonymous Azure services to the AWS ones we use and I'll be honest, the amount of garbage the Azure SDK created in our JVM codebase was quite annoying.

Now, I fully understand a Java SDK isn't going to get quite the same love as their bread and butter, but everything about their Java SDK feels like, just copy the .NET SDK and make it work in Java.

I assumed the telemetry traces (or attempted traces) I was seeing, even with every configurable telemetry config flag EXPLICITLY set to disabled was just more of their bad choices of log levels, leaky internal log messages, and just generally not doing things "the Java way".

Now I'm starting to wonder if there's just flat out no way to turn all the telemetry off. Originally we built with the AWS and Azure SDKs and the cloud providers were configurable at run time. I was getting so annoyed with the garbage that MSFT was leaving in our logs even when disabled and running in AWS that we eventually bifurcated our build and re-organized the project.

We spent about a week adding abstractions we previously didn't need, and another week testing, for what we were assuming was just an annoyance. Turns out maybe we were doing our other clients a solid and didn't even know it.

<comment><USER3> Could you elaborate on your first point? What kind of stuff was added?

</comment></comment><comment><USER4> The author doesn't understand GDPR.

The GDPR governs consent for processing of _personal data_. Generally, anonymized data is not personal.

The author attacks the anonymization method, but that is a thin branch to stand on. In particular, given how GDPR is very much about usage, Microsoft helps substantiate their anonymization simply by not attempting to deanonymize.

See eg Art 7,

> _Where processing is based on consent, the controller shall be able to > demonstrate that the data subject has consented to processing of his or her > personal data._

And Recital 26

> _The principles of data protection should therefore not apply to anonymous > information, namely information which does not relate to an identified or > identifiable natural person or to personal data rendered anonymous in such a > manner that the data subject is not or no longer identifiable._

<comment><USER5> It's personal data, they collect the MAC address and can search by MAC address.

That's literally an identifier to identity any computer on the planet. That's quite personal and traceable really.

<comment><USER6> No, read the description. To be personal data for the purposes of the GDPR, it must be related to an identifiable natural person (ie not a corporation). I don't see how Microsoft could relate a MAC address to a natural person.

<comment><USER5> The MAC address uniquely identifies a computer/laptop on the planet, that belongs to a person. It is as personal an identifier as it gets ;)

</comment></comment></comment></comment><comment><USER7> Who enforces GDPR? Can you file a formal complaint, or do you just have to complain about it on the internet and hope that the offender complies?

<comment><USER8> Countries within the EU have their own regulators. For instance, France has the National Data Protection Commission (CNIL)

<comment><USER9> There is nonetheless the European private data protection office as well</comment></comment></comment>


Attention Suckers: Please Send Us Your Money - <AUTHOR> http://www.bloombergview.com/articles/2014-03-26/attention-suckers-please-send-us-your-money <comment><USER2> I bought an early Oculus Rift developer kit as a part of the Kickstarter. I paid $300 for it and it was worth every penny. I had no expectation of equity from it, because the product was cool and something I wanted. It's the wrong thing to criticize Oculus Rift for the simple reason that hardware crowdfunding has unlocked a whole new world of things that could be built if only you could prove people want it.

The crowdfunding campaign Oculus Rift ran proved people wanted it. We see this over and over again — if you can create a new class of consumer behavior (Uber, Google, etc) then you can create a valuable company.

Articles like this are sensational in that they sometimes incite misguided lawmakers to regulate. But in this case, crowdfunding is creating new economic activity, and any new regulation would actively squelch that — and kill viable new products and companies in the process.

<comment><USER3> I get this "it was money in exchange for goods" point of view, and I certainly find this article stupid and misleading.

But I'm still a little uneasy about what happened with Oculus, and the trouble that I have is, if you look at the the pledge levels, a lot of that money was nothing near a "cash-for-goods" exchange. 7 people pledged more than $5000 to get to meet the Oculus team. 100 people pledged $75 for a poster and a t-shirt. Those aren't purchases of goods for market prices; those are donations for token rewards, no different than when you give to public radio. Oculus solicited private donations to continue their work, and if the Facebook acquisition results in a dramatic change of direction for Oculus I think that will be a betrayal of donor trust.

<comment><USER4> > Oculus solicited private donations to continue their work, and if the > Facebook acquisition results in a dramatic change of direction for Oculus I > think that will be a betrayal of donor trust.

That's what I think is the issue, not the equity. If Facebook's software is attached to the device, it's a different device (more like Google Glass) and I don't want to use it.

</comment></comment></comment><comment><USER5> ... Except $25 for a t-shirt and $300 for a developer kit are about market rate for such items. Not anymore of a scam than buying t-shirts to help a rock band who might later make it big. And those fans usually complain too about sell-outs.

</comment><comment><USER6> I think the point of this article is not that people think they got a bad deal.

The point is, if those 9500 people had pooled their money together beforehand and negotiated as a collective -- maybe you could say they _incorporated_ \-- they could have probably gotten a much _better_ deal. This is to say, hindsight is not very clear, because you don't know what the other road looked like.

If y'all had sent your $2.4m to professionals to actually negotiate the funding of Oculus, I'm pretty sure you'd have walked away with at least 500% return on investment. Instead, you're defending your decision by saying you got at least 100% return, maybe, even though the target price for the product is < $300.

And you're right -- this _is_ a Wall Street-like way of thinking. It also turns out to be pretty effective, overall.

<comment><USER7> Well the reality of that is that 1 in 1000 start-ups make money.

The odds are, they would lose money.

They traded a 1 in a 1000 shot at big bucks for a guaranteed 100% return, that is the small investors take on risk mitigation.

The wall street take on risk mitigation is to take and hold large chunks of equity, hoping for future growth which only occasionally materializes. They do this with other people's money. Those being people who can afford to take such risks.

The people that "bought" into O have a different risk profile and appetite for loss and acted appropriately.

</comment></comment><comment><USER8> That article is a very large troll. Any valid criticism seems to be run over by the bulldozer of BS. To extend that argument to _all_ of crowdsourcing from one event seems dubious considering he works for a company that was built on money from wall street markets and is very closely tied to traditional finance. Could wall street handle the same scrutiny?

People will get more savvy regarding crowdsourcing but was anyone really duped? No lies were told afaik.

</comment><comment><USER9> While I agree with the author that people need protection from themselves in some situations, Kickstartering ain't one of them.

I paid $300 for a dev kit and funded a company to do serious research on VR - the first serious research in oh, forever. They now employ John Carmack and Michael Abrash TOGETHER to work on this problem. This is AMAZING and the next few years are going to be AMAZING because of it.

Honestly, it's stated in every Kickstarter campaign that things might not turn out the way you want. You may fund a game that never comes to fruition. There's specifically a section on "risks" that campaign starters are supposed to fill out so that backers are properly informed. We know this going in, that things might not turn out the way we hope.

That doesn't mean the model is broken, and it doesn't mean we're all sheep.

I want to take a screenshot of this article and add the classic political cartoon caption: "Christ, what an asshole."

</comment><comment><USER10> I'm really not sure what this article is suggesting, but it makes me angry. At first glance it seems like it is simply trolling for clicks, and in that regard it seems to have succeeded. However, its stance on investor regulation and somehow relating it to crowdfunding is confusing. Are they suggesting that anyone thought they could make a profit by funding Oculus? Are they calling for tighter regulation that would outlaw crowd funding? Or perhaps they think there should be less regulation to allow the masses to purchase equity in startups? It just feels like they are purposely misrepresenting the facts, and have some sort of agenda.

</comment><comment><USER11> "In the case of Oculus, where you weren't even promised equity, you simply revealed yourself as clueless naifs."

This article has a fundamental misunderstanding of why people contribute to kickstarters. I think most people's motivation is closer to charity than it is to investing and I think people understand that difference.

Ironically, the author implies he knows what is best and needs to protect the "sheep" from this "scam", yet he obviously doesn't have an even basic understanding of why people contribute to kickstarter.

</comment><comment><USER12> A much less click-baity pice on the same site by a differentauthor: [http://www.bloombergview.com/articles/2014-03-28/crowdfundin...](http://www.bloombergview.com/articles/2014-03-28/crowdfunding- is-not-a-scam-it-s-market-research)

A valid point is that what a kickstarter is exactly can be a bit ambiguous and Kickstarter arguably makes it more so. It's not an equity investment. OK. It's also not a pre-order. OK. But (according to Kickstarter's terms of service) if you don't deliver on the promised reward you have to refund the money. OK? Well, doesn't that sound a lot like a pre-order. And where's the money coming from if it's already been invested in an idea that just didn't work out?

Now, in practice, things generally work out. Lots of projects fail and most rational people realize that sometimes stuff just doesn't play out like they'd wish. (OTOH, one can certainly find discussion boards filled with unreasonable people talking lawsuits etc. for failed projects.)

</comment><comment><USER13> This article is pretty misinformed. Title III of the JOBS act, which would allow crowdunding-for-equity, is still pending SEC rulemaking [1].

[1] [http://en.wikipedia.org/wiki/Jumpstart_Our_Business_Startups...](http://en.wikipedia.org/wiki/Jumpstart_Our_Business_Startups_Act#Provisions_of_the_bill)

</comment><comment><USER7> The author of this piece is pretty naive on what average people actually "Contributed" to Oculus. $25 - $5000 for actual material goods is akin to pre- orders, at market prices.

Everyone sees risk differently, the "contributors" for Oculus, saw paying $300 for a dev kit as a risk they were willing to take vs. $10K for a .01% of the company that may never produce a return.

The sale to facebook (of all places) was not reasonably foreseen and especially at the price paid.

If the author would have looked at contributions thru the lens of risk mitigation, he would see that the "contributors" actually made a very sound financial decision.

</comment><comment><USER14> Agreed, the current crowd funding model is broken.

Though interestring, if you backed Oculus with $300 and got a devkit for that, I think that's reasonable.

So the lesson should probably be only do crowd funding when it's essentially a pre order?

</comment><comment><USER15> The oddest thing about this article is that the author seems to think that the JOBS act is to blame for Kickstarter backers being fleeced, which is just weird.

Unless I'm missing something, JOBS doesn't have anything to say about non- equity crowdfunding:

[http://en.wikipedia.org/wiki/Jumpstart_Our_Business_Startups...](http://en.wikipedia.org/wiki/Jumpstart_Our_Business_Startups_Act#Provisions_of_the_bill)

</comment><comment><USER16> I don't get how this is a scam. Did the people get their thank you, their tshirt, or development kit? AND didn't the company get kickstarted? So the people got exactly what they paid for, and presumably exactly what they wanted. So where is the scam?</comment>


Facing net neutrality critics, Verizon lifts data caps on public safety workers - <AUTHOR> https://yro.slashdot.org/story/18/08/26/001242/facing-net-neutrality-criticism-verizon-suddenly-lifts-data-caps-on-all-public-safety-workers <comment><USER2> I never quite understood what the net neutrality debate was about, having even spent some years at ISP in NYC. So net neutrality is all about giving unlimited data access to pulic safety workers?

<comment><USER3> People who support net neutrality believe that if private corporations are allowed to bill packets based on destination/origin/content instead of treating them all equally, society would be worse off. Here is a prime example of a corporation being extremely petty in a emergency situation when possibly lives are at risk. To quote:

'Verizon's throttling has everything to do with net neutrality -- it shows that the ISPs will act in their economic interests, even at the expense of public safety," County Counsel James Williams said on behalf of the county and fire department.'

How then will a corporation act when the stakes are even lower? Act in public interest or profit seeking?

<comment><USER4> _Verizon 's throttling has everything to do with net neutrality_

And that's just wrong. Verizon throttled the fire department after they hit the specified bandwidth limit, without regard to the content of the traffic. The people who are mad at Verizon are claiming that they should have treated the traffic differently because it was more important, which is exactly the opposite of net neutrality.

<comment><USER5> There isn't (read: shouldn't be) a bandwidth limit on unlimited plans, which the fire department had

<comment><USER6> Sure, _but global bandwidth limits have nothing to do with net neutrality._

</comment></comment></comment></comment></comment><comment> ex-pralite-monk Right after the last two hurricanes hit Florida, Verizon temporarily granted unlimited data to those in the affected area, which I thought was very sporting of them.

Even if your DSL/cable connection was still alive, your router battery backup ran out of power within hours, so cell phone data was the only way to communicate and stay informed.

</comment><comment><USER7> Net neutrality means that every packet should be treated exactly the same, how does that marry up with giving preferential treatment for specific packets for safety workers?

This is the exact opposite of net "neutrality".

</comment><comment><USER8> This a fire dept wanting people outside of taxable area to pay for it's data usage and calling it "Net Neutrality" for news coverage.

Sighs. So much fail.</comment>