Steps to a clean dataset with Pandas - <AUTHOR> https://towardsdatascience.com/3-steps-to-a-clean-dataset-with-pandas-2b80ef0c81ae?source <comment><USER2> To add to the three points in the article:

Data quality [https://en.wikipedia.org/wiki/Data_quality](https://en.wikipedia.org/wiki/Data_quality)

Imputation [https://en.wikipedia.org/wiki/Imputation_(statistics)](https://en.wikipedia.org/wiki/Imputation_\(statistics\))

Feature selection [https://en.wikipedia.org/wiki/Feature_selection](https://en.wikipedia.org/wiki/Feature_selection)

datacleaner can drop NaNs, do imputation with " _the mode (for categorical variables) or median (for continuous variables) on a column-by-column basis_ ", and encode " _non-numerical variables (e.g., categorical variables with strings) with numerical equivalents_ " with Pandas DataFrames and scikit- learn. [https://github.com/rhiever/datacleaner](https://github.com/rhiever/datacleaner)

sklearn-pandas " _[maps] DataFrame columns to transformations, which are later recombined into features_ ", and provides " _A couple of special transformers that work well with pandas inputs: CategoricalImputer and FunctionTransformer_ " [https://github.com/scikit-learn-contrib/sklearn- pandas](https://github.com/scikit-learn-contrib/sklearn-pandas)

Featuretools [https://github.com/Featuretools/featuretools](https://github.com/Featuretools/featuretools)

> _Featuretools is a python library for automated feature engineering._ [using > DFS: Deep Feature Synthesis]

auto-sklearn does feature selection (with e.g. PCA) in a "preprocessing" step; as well as " _One-Hot encoding of categorical features, imputation of missing values and the normalization of features or samples_ " [https://automl.github.io/auto- sklearn/master/manual.html#tur...](https://automl.github.io/auto- sklearn/master/manual.html#turning-off-preprocessing)

auto_ml uses " _Deep Learning [with Keras and TensorFlow] to learn features for us, and Gradient Boosting [with XGBoost] to turn those features into accurate predictions_ " [https://auto- ml.readthedocs.io/en/latest/deep_learning.html#...](https://auto- ml.readthedocs.io/en/latest/deep_learning.html#feature-learning)</comment>


Scryptic iPhone game: words are your weapons and strategy your friend - <AUTHOR> http://www.lksgameworks.com/

<comment><AUTHOR> This was developed by Computer Science professors at the University of Cape Town.

Video for those on Facebook: [https://www.facebook.com/video/video.php?v=10150400020984867...](https://www.facebook.com/video/video.php?v=10150400020984867&oid=182353021829936)

Design docs: <http://www.lksgameworks.com/dev.html></comment>


Voting Turnout Statistics - <AUTHOR> http://www.statisticbrain.com/voting-statistics/ <comment><AUTHOR> So, I guess "stay at home" got almost double the votes of any one candidate... :-/

</comment><comment><USER2> Just FYI be careful some of the numbers are definitely mixed. This can't be right...

Registered Democrat 89 % 9 %

Registered Republican 90 % 7 %

<comment><AUTHOR> How so? That of those that are registered Democrat / republican a little under one in ten (of those that voted) voted for the other candidate? You think it's too high?</comment></comment>


Most British scientists feel Richard Dawkins misrepresents science - <AUTHOR> http://news.rice.edu/2016/10/31/most-british-scientists-cited-in-study-feel-richard-dawkins-work-misrepresents-science-2/ <comment><USER2> Religion, when it conflicts with other people in society, is politics. Dawkings tries to actively change people's minds. It's political crusade. Not scientific one.

Politically influencing people is not scientific presentation. There is tension between representing facts and effect. If Dawkins wants to have maximum political effect, he must optimize his fact/demagoque ratio for maximum impact. It's probably enough for Dawkins to be just little more accurate and convincing on his critique than opposition to turn the minds of people and win arguments.

I have always understood that when Dawkins talks about religion, he is not representing scientific approach. It's not even his field of research. It's unfortunate if he is conveying the false impression of solid science. He uses valid scientific arguments, but his writings as a whole are not coherent scientific argument against religion.

</comment><comment><USER3> What a terrible title. 137 interviews, of those 48 scientists mentioned Dawkins (unprompted) of those 80% felt he "in some way misrepresented science or the scientific process".

It's also unclear if they have a problem with the man's scientific work (he is/was an evolutionary biologist first) or is hardline stance against religion.

Nothing to see here, just click bait at work.

<comment><USER4> Yes. Really: most of a self-selected group of scientists who chose to bring up Dawkins as an example brought him up as a negative example.

</comment></comment><comment><USER5> Science is becoming too politicized

<comment><USER6> Science has always been too politicized</comment></comment>


Hidden Technical Debt in Machine Learning Systems (2015) [pdf] - <AUTHOR> http://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf <comment><AUTHOR> Between this and google's "Machine Learning: The High Interest Credit Card of Technical Debt", I think they're trying to send us messages from the future about what happens when you build a company out of ML and don't fully account for the repercussions of creating systems sensitive to hard-to-measure software, systems, data generating process changes. It's a message I plan on heeding :)

<comment><USER2> Are you talking about this paper: [http://research.google.com/pubs/pub43146.html](http://research.google.com/pubs/pub43146.html)? It appears to be the same as the linked PDF.</comment></comment>


What's happening in the A/B Testing Market? - <AUTHOR> https://blog.builtwith.com/2013/07/19/whats-happening-in-the-ab-testing-market/

<comment><USER2> Great data from BuiltWith (as usual), but a few key questions remain unanswered:

1) how much _usage_ are the sites measured getting out of the A/B testing solutions they're using? I rather suspect that there are order of magnitude differences in tool usage that this measurement doesn't capture.

2) IMO it's really the usage among the top 2-5K sites that matters, and even within that it's the top few hundred that drive 80%+ of ecommerce. What tools _they 're_ using are a very, very important measure of share of A/B testing budgets being spent.</comment>


Why I Banished My Ego from My Writing Process - <AUTHOR> http://passionatereason.com/2019/06/why-i-banished-my-ego-from-my-writing-process/ <comment><USER2> I would love to see other writers responding if this rings true for them.</comment>


Researcher puzzled by Seattle's poor startup activity ranking - <AUTHOR> http://www.techflash.com/seattle/2010/06/researcher_puzzled_by_seattles_poor_startup_ranking.html

<comment><USER2> This is not puzzling at all. It's just yet another instance of conflating startups with newly founded companies. In fact the former are a tiny subset of the latter. All the Kauffman Foundation study tells us is that people aren't starting as many barber shops and landscaping companies in Seattle as in the other cities.</comment>


How Cyberbullying Prevention Act Could Land You In Prison - <AUTHOR> http://startupearth.com/2009/05/23/how-cyberbullying-prevention-act-could-land-you-in-prison/

<comment><USER2> _"Whoever transmits in interstate or foreign commerce any communication, with the intent to coerce, intimidate, harass, or cause substantial emotional distress to a person, using electronic means to support severe, repeated, and hostile behavior, shall be fined under this title or imprisoned not more than two years, or both."_

Good! People who set out to _intentionally_ ("with the intent") intimidate, harass, or distress another individual ("a person") deserves punishment. The word "substantial" demonstrates it's for incidents far beyond just saying something like "You're a dick" or "You're wrong because X, Y and Z."

As is usual, though, the problem here isn't with the law as-written but the way it can be _misinterpreted_ and abused by unethical lawyers and judges. How do you prove _intent_ to cause distress? If I were to make a flippant comment about "Christians" believing in "fictional characters", let's say, it should be obvious I'm not intending to offend or cause emotional distress to "a person", but could the law be twisted to force me to prove my innocence in such a minor instance?

</comment><comment><USER3> IANAL, but _intent_ is explicitly required: "...with the intent to coerce, intimidate, harass, or cause substantial emotional distress to a person..."

To a lay person, it seems no small thing to _prove_ such an intent in writing critical material on web (or elsewhere).

</comment><comment><USER4> I doubt this law will pass constitutional muster.

But, if you don't want to be the first test case, I suggest using Tor for posting anything online. You might want to use an anonymous username, but every time cops have been sent to my house over a forum posting, the username gets lost and they only care about the IP that they got obtained from the site operator. (But also keep in mind that an IP address and a username don't prove that you did anything illegal, so laws like this are extremely hard to enforce.)

IANAL, of course, but I do have firsthand experience on this matter. (Also, Slashdot says they delete the IP address of comments, but they don't. :)

Anyway, let them harass the Tor endpoint in Egypt, or something, not you.</comment>


Honda Riding Assist [video] - <AUTHOR> https://www.youtube.com/watch?v=VH60-R8MOKo <comment><USER2> Must use a gyroscope, which requires energy. Impact to gas mileage?

<comment><AUTHOR> apparently from the press release it just uses a motor to alter the rake of the fork, so avoiding gyroscopes (which as you said would be bad)</comment></comment>


Tesla vs Alameda County [pdf] - <AUTHOR> https://wholemars.com/alameda-lawsuit.pdf <comment><USER2> Neighboring counties have already allowed work like construction to resume. Is a car factory really that different in terms of risk? Just based on the sheer size of the facilities, I can't imagine the density of people is that high, and due to at least some of the work going on, ventilation is probably good.

It definitely makes sense to keep businesses like bars and restaurants closed given their nature as a "hub" in the graph of human contacts, but we also need to allow what work we can to continue as long as it doesn't contribute to significant spread of the virus.

At the same time, last I saw, car sales had fallen off a cliff. Is there much point to aggressively reopening if those cars can't be sold? But that's a question for Tesla to answer, not Alameda County.

<comment><USER3> > ventilation is probably good

I'm not clear on what ventilation being "probably good" means in the context of reducing SARS-Cov-2 exposure to a safe level. What if the air is recirculated and the filters don't capture enough of the virus?

<comment><USER4> Nobody knows anything for certain. We do not even have good data on mask effectiveness. The basis of all the restrictions is already a gut feeling/whim of the governors, not any concrete scientific data about what works and what doesn't. Asking whether air conditioning system is good enough doesn't make sense if we don't even know what's good enough. Nobody is doing any studies in that direction anyway, so we'll never know.

I don't know if lockdowns and restrictions make any practical difference or not, but people affected are definitely justified in complaining about large degree of arbitrariness in all of it. For example, NYC is still running subway and never stopped through all of it. Sure, people are relying on it to get places in New York, but so is everyone else in this country relying on their job to make a living, and yet many of their jobs were banned just like that.

<comment><USER5> New infections in NY i heard are majority 76% from ppl staying indoors. Not sure how are they are getting it, from recirculated air in apt complexes?

<comment><USER6> Last measured antigen rate in NYC was north of 20%. My bet is almost all "essential" workers have been exposed and that part of the "herd" is immune.

Who does that leave but everyone else, who are still shopping, going for walks, exercising, doing laundry in communal facilities and all the other minutiae of life that cannot be paused.

I've seen this comment a few times and don't see the mystery. The virus has saturated the mobile and now is finding vectors into the less mobile.

This is a feature, not a bug, since the pickup rate is hopefully under the critical care capabilities.

<comment><USER5> > I've seen this comment a few times and don't see the mystery.

Well, its not clear to me what "people who are staying indoors" means. I am exclusively inside my apt for last 8 weeks and have groceries delivered and workout on my bike trainer/zwift indoors.

I feel like there should be some distinction between, 1. mostly staying indoors 2. exclusively staying indoors.

If second group of people are getting it, then yea its mysterious.

</comment></comment></comment></comment></comment></comment><comment><USER7> Sigh, it is kind of embarrassing for me that I was a fan of this guy and even own a Tesla. Elon has gone off the rocks quite and bit and now is a straight up loon. He is now full on spreading FUD, which is weird considering the amount of FUD That was spread against Tesla for a while

</comment><comment><USER8> Two threads on this already:

[https://news.ycombinator.com/item?id=23126517](https://news.ycombinator.com/item?id=23126517)

[https://news.ycombinator.com/item?id=23127552](https://news.ycombinator.com/item?id=23127552)

Not sure this is likely to go any better.

<comment><AUTHOR> It's the actual lawsuit though. Much more substantive.

<comment><USER8> Yes, and the hope is that it would support a better thread.

</comment></comment></comment><comment><USER9> Is this legal argument actually any good?

It seems like the major claim is that Alameda County's order "overrides an express order of the Governor of California," which doesn't make much sense to me. Newsom's order said that _individuals_ working in critical infrastructure sectors are exempt from staying at home. I don't see how Alameda County saying Tesla must stay closed contradicts Newsom saying Tesla employees could go to work, if their workplace was open. Seems sorta weaksauce to interpret an exemption for individuals from a stay at home order as a order than Tesla must remain open.

Anyway, seems like a mess, will be interesting to see what happens with this.

<comment><USER10> The exemption for individuals is so that they can go to work what good would it be if their work place was closed?

<comment><USER9> My thinking is that the exemption allows for counties to have the opportunity to allow those businesses in critical infrastructure sectors to stay open. The order says that employees "may continue their work," and Tesla's lawsuit seems to interpret that to mean that Tesla must stay open.

I'm obviously not a lawyer, I'm just interested in if the argument in the complaint is likely to get them anywhere.

</comment></comment></comment><comment><USER11> I am curious how Tesla employees feel about this, and more generally Musk's continuous and baffling insistence that the coronavirus is no big deal. (Heck, it seems like until and unless this wipes out > 10% of humanity it seems like he'll keep saying 'no big deal')

It's one thing if your company and its CEO acknowledge the seriousness of the situation and develop a plausible contingency plan, including masks and random asymptomatic testing. But he seems rather unconcerned about such pedestrian matters and has decided it's a nothingburger and all hype.

Has Tesla actually proposed a reasonable plan to do this while Alameda county is seeing a spike in new cases?

<comment><USER12> A decent CEO would've explained the precautions to local government officials, persuaded them it was the right thing to do and then kicked in a donation of 500,000 masks or something. Instead, Musk is tweeting conspiracy theories and going on Joe Rogan. It's just incompetence. Even if the Tesla factory does have real and effective measures to reduce the risks, no one is going to trust Elon Musk.

</comment></comment><comment><USER13> Off-topic: I haven't downloaded many legal PDFs lately, but whatever legal software was used to make this one seems to obfuscate the text? The PDF seems like a normal text PDF (as opposed to scanned images), and you can highlight lines of text, but copy-pasting it results in gibberish; find-text-in-document is likewise borked. Using `pdftotext' generates a mostly empty file, though Google Docs conversion manages to extract and preserve the text.

I bring this up because the document has several URLs which are non-clickable and non-copy-pastable. I know PDFs can be encoded in away in that the visual layout of the content is completely different to the underlying data, this was just the first time I've seen such obfuscation.

edit: here's a normally-behaving copy of the PDF from one of the SF local news stations:

[https://sanfrancisco.cbslocal.com/wp- content/uploads/sites/1...](https://sanfrancisco.cbslocal.com/wp- content/uploads/sites/15116056/2020/05/Tesla-Incorporated-v.-County-of- Alameda-5-9-20.pdf)

Assuming the TV station has uploaded the PDF straight from the source (Tesla's lawyers, or the county court system), then maybe the site wholemars.com re- processed its copy of the PDF? In any case, the text/data obfuscation is pretty interesting and I sure hope it doesn't become a standard practice!

(I don't blame the OP for posting the wholemars.com link; that was the first I saw of the lawsuit too, at least on Twitter)

<comment><USER14> I have to wonder if there's any rules that would be violated by crap like this in the courts. It's 100% to prevent other lawyers from copy-pasting things, and that's absurd since it's defeated by OCR. It's extremely not accessible to the blind as a result.

<comment><USER15> It also makes it invisible to search engines. Google "Tesla COVID-19 lawsuit" and this PDF will not show up...

<comment><USER14> The curious thing is not _everything_ is obscured. Namely, `Attorneys for Plaintiff Tesla, Inc. ` isn't.

Either is `Identifying Critical Infrastructure During Covid-19`

So eventually this is probably going to get indexed by Google.

EDIT: It is already!

[https://www.google.com/search?q=%22Identifying+Critical+Infr...](https://www.google.com/search?q=%22Identifying+Critical+Infrastructure+During+Covid-19%22+%22tesla%22&oq=%22Identifying+Critical+Infrastructure+During+Covid-19%22+%22tesla%22)

EDIT 2: Why the downvotes already? I was pointing out that not all text is obfuscated in the text, namely COVID and Tesla are not. So it _is_ going to get indexed to some extent, it seems. If you want to downvote me, at least give me the courtesy to reply to your reasoning for it!

EDIT 3: You can check it yourself. Open the PDF, ctrl+f tesla, the only sentence unobscured shows up. ctrl+f for Covid-19, only one unobfuscated occurrence..

<comment><USER13> What Google is indexing appears to be the original and normally behaving PDF (before processing by wholemars.com):[https://sanfrancisco.cbslocal.com/wp- content/uploads/sites/1...](https://sanfrancisco.cbslocal.com/wp- content/uploads/sites/15116056/2020/05/Tesla-Incorporated-v.-County-of- Alameda-5-9-20.pdf)

(the submitted link for this thread should probably be changed to the sanfrancisco.cbslocal.com hosted version of the PDF)

But yes, I do agree that not all the text in the wholemars.com version was obfuscated. Seems like the metatext and boilerplate stuff was left as is in this processed version.

</comment></comment></comment></comment></comment><comment><USER16> Edit: I was a little late in noticing a "show more" link at the bottom of the list that Google gave me for [16 critical infrastructure sectors] and found "Transportation Equipment Manufacturing" is also among the 16. So consider the below moot.

In spite of my sympathy for Tesla in general and maybe even their wish to resume operations in particular, their arguments, as I understand it, doesn't seem to be very convincing?

It appears to hinge on the categorisation, with Tesla's view being that they belong in the "Energy Sector", which is among the "16 critical infrastructure sectors" exempt from any requirements to shut down.

While I guess you can draw a connection of Tesla with the energy sector if you're doing so in a complete vacuum, it is entirely obvious that, when the "energy sector" was exempt from closure, it was based on a definition that under no circumstances would include Tesla. I. e.: power plants and gas stations obviously need to stay open for reasons that just don't apply to manufacturing cars.

<comment><USER15> Yeah, IANAL, but this document doesn't sound to have been written by someone punching arguments, it sounds more like a whiny teenager.

On top of page 2 it somehow claims a car factory is permitted to operate because this FAQ says so:

>My business installs distributed solar, storage, and/or electric vehicle charging systems – can it continue to operate?

> Yes, this is permissible construction activity and must comply with the > Construction Project Safety Protocols in Appendix B of the Order. Businesses > may also operate to manufacture distributed energy resource components, like > solar panels.

I guess they're going to go to court over the definition of "distributed energy resource components". Tesla cars are consumers (also storers) of (electrical) energy resources, but a gas-powered car does the same for energy stored in gasoline. If they let this in, then I could argue that my battery- powered phone is also an "energy resource component"...

<comment><USER10> That isn't their argument they are presenting that as an example of how the county isn't being reasonable by saying one thing then acting contrary to their publicly stated guidance. Their core argument begins on page three paragraph 8 and is more about the county claiming to superseded the authority of the State and Federal government when they don't have the legal basis to do so.

</comment></comment></comment><comment><USER17> For what it's worth, Fremont, the city where Tesla is located, appears more sympathetic to the company

[http://www.fremont.gov/CivicAlerts.aspx?AID=1768](http://www.fremont.gov/CivicAlerts.aspx?AID=1768)

</comment><comment><USER18> I don't know that this is that notable, is it? Surely there are dozens or hundreds of suits all over the country from businesses quibbling over their non/categorization as essential?

It's a pandemic. No one wins here. No policy can be completely fair. Hell, I have opinions about everything and I genuinely don't know that I can find one about this situation. Obviously Tesla (and every employer) wants to open. The county wants people to stay home. Some exceptions get made. Should Tesla be one? Meh.

<comment><USER19> You don't think a controversial, SV darling CEO _suing_ to allow his non essential business to open amidst a pandemic is notable?

What about his public attacks on the civil servants involved in determine the rules for public safety?

What about the fact he's decrying "unelected officials" and "fascism" in the US,while praising the CCP and expanding Chinese production?

It's unbelievable people continue to defend this man here.

<comment><USER10> If the factory is operating safely in China w/o employees getting sick why isn't it possible in the USA?

<comment><USER18> Objectively it's because the Chinese outbreak is well controlled at this point without lockdown measures. Alameda isn't there yet, though the Bay Area is closer than many areas in the US which are still growing.

<comment><USER10> The Chinese outbreak was much less controlled when they allowed the China factory to reopen.

</comment></comment></comment></comment></comment><comment><USER20> how is car manufacturing 'critical infrastructure'?

Is the case number, 'Case 4:20-cv-03186' significant?

<comment><USER10> The list of critical infrastructure industries is outlined in this federal government document...

[https://www.cisa.gov/publication/guidance-essential- critical...](https://www.cisa.gov/publication/guidance-essential-critical- infrastructure-workforce)

It lists among other things...

"Workers critical to the manufacturing, distribution, sales, rental, leasing, repair, and maintenance of vehicles and other transportation equipment (including electric vehicle charging stations) and the supply chains that enable these operations to facilitate continuity of travel-related operations for essential workers."

<comment><USER20> I don't think this really covers manufacture of general consumer passenger vehicles.

The intent is clearly to maintain transportation capability for genuinely critical workers (healthcare, food distribution and sales, police and emergency, etc). If their tesla breaks down, there is no need to manufacture a new one, they can just grab one from the sales lot.

<comment><USER10> It definitely does, you are reading way to much into it. Look at the full list. How do you think all the other essential workers are going to get to and from work?

<comment><USER20> Well i don't think they are going to buy a new car for the purpose. Even if they did, there are many already-manufactured waiting to be sold.

<comment><USER10> I feel like you are confusing your personal opinion and preferences with what the document says in writing.

<comment><USER20> I think you are reading the document too broadly.

</comment></comment></comment></comment></comment></comment></comment><comment><USER21> Suing the government for COVID closure only six days after tweeting "Minecraft has amazing legs"

</comment><comment><USER22> Tesla compared to their competition is actually doing very well even during the pandemic. So Musk would not be hurt even if its shut down. It's disappointing that a man of science, who understands data is throwing a fit like this.

<comment><USER15> "It is difficult to get a man to understand something, when his salary depends on his not understanding it."

― Upton Sinclair, a once failed candidate for the Governor of California

</comment></comment><comment><USER23> Just move then, dont threaten, don't sue. Just make the change, its cheaper and faster.

<comment><USER24> It's cheaper to sue and win.

<comment><USER23> Not with the government. If you win you now have soured relationships that will last terms and terms.</comment></comment></comment>


Show HN: Word2Bits – Quantized Word Vectors - <AUTHOR> https://github.com/agnusmaximus/Word2Bits <comment><USER2> This is great and echoes a recent fascination for me. One application of compact word embeddings is that if they're small enough you can ship a whole model to the user in a web app so that semantic computations can be done entirely client-side, which is useful for privacy. I did a naive 1-bit quantization a few months ago in order to fit a large-vocabulary word embedding into a smallish (< 3 MB) JSON file so that synonyms could be generated by the user without hitting a server. (See [https://docs.google.com/presentation/d/1SfbfnuNSlDRtUOpN1KNc...](https://docs.google.com/presentation/d/1SfbfnuNSlDRtUOpN1KNc2RMtw030defCDME6p_268fQ/edit#slide=id.g28366446a3_0_110) for a description and [https://prototypes.cortico.ai/search- history/](https://prototypes.cortico.ai/search-history/) for the application -- happy to send you more details/code if you want) I never got as far as formally evaluating the quality of these vectors, though.

<comment><USER3> Can you expand on the privacy use case? I don't understand how applying a word vector model on the client improves privacy.

<comment><USER4> You presumably do not have to ship the content of the user's query up to the server.

<comment><USER3> How does that improve privacy in this case? The sort of queries that can be made against a word vector model, and only a word vector model, are pretty trivial.

</comment></comment></comment></comment><comment><USER5> Interesting to see that "science" and "fiction" are so similar according to this metric. Not surprising, though, given how often they co-occur in text, but this clearly shows the limitations of the method. The unit of interest is not really character strings but lexical entries and there can be multiple lexical entries associated with one character string. For instance, the word "bank" can mean "financial institution" or "edge of a river" but both meanings would contribute to a common word vector. You could say that this only affects ambiguous words, but first ambiguous words are very common and second these words probably distort the whole vector space (by introducing "short circuits") and therefore also affect the vectors for non-ambiguous words. A long way left to go for these methods I suppose.

Edit: Have people tried to detect ambiguous words by measuring local conflict in word2vec space? E.g. "laboratory" is similar to "science" and "science" is similar to "fiction" but there is no evidence to suggest that "laboratory" should be similar to "fiction".

<comment><USER6> One method to partially rectify the problem you mention is to add additional preprocessing to the text to identify ngrams and add part of speech tags to words. For example [The, river, bank] might be restructed as [The, river bank (bigram)] where "river bank" is embedded as its own word vector.

This can also be used to disambiguate words like "hit" which can be used as a verb and a noun. You just replace "hit" with "hit|noun" and "hit|verb".

<comment><USER7> Yes! And, also, I swear I've seen this method done before. I can't remember the paper. Can anyone help?

Basically, instead of making neural word embeddings, much like you describe, the objects being embedded in a vector space were "hit|noun(1)" "hit|noun(2)" "hit|verb", and so on.

I believe they used WordNet or some ontology, or maybe a POS corpus like Penn Treebank....

<comment><USER8> I think you're talking about sense2vec

<comment><USER7> Yes!!! It was! Thanks so much!

</comment></comment></comment></comment></comment><comment><USER9> I thought I knew what a vector was, but I have no idea what any of this project means.

Would someone please explain in simple language what this is, and why it's cool?

<comment><USER10> Quick run down on word2vec and what happened here:

A big problem with NLP is understanding the semantic associations between words (or even lemmas. Lemmas in this context refer to different meanings of the same word, like a baseball _bat_ vs. a vampire _bat_ ). For example "run" and "sprint" are similar in meaning but convey different connotations; kings and queens are both high-level monarchs but we need to encode the difference in gender between them for a true semantic understanding. The problem is that the information in words themselves don't accurately convey all of this information through their string (or spoken) representations. Even dictionary definitions lack explanations of connotations or subtle differences in context, and furthermore aren't easily explained to a computer

Word2vec is an algorithm that maps words to vectors, which can be compared to one another to analyze semantic meaning. These are typically high-dimensional (e.g. several hundred dimensions). When words are used often together, or in similar contexts, they are embedded within the vector space closer to each other. The idea is that words like "computer" and "digital" are placed closer together than "inference" and "paddling".

Usually these vector mappings are represented as something like a python dictionary with each key in the dictionary corresponding to some token or word appearing at least one (maybe more) times in some set of training data. As you can imagine these can be quite large if the vocabulary of the training data is diverse, and due to being in such a high-dimensional space, the precision of a vector entry may not need to be as high as doubles or floats encode. Floats are 32 bits. The authors of this paper/repo figured out a way to quantize vector entries into representations with smaller numbers of bits, which can be used in storage to make saved word2vec models even smaller. This is really useful because a big problem with running word2vec instances is that they can take up space on the order of gigabytes. I haven't read it all yet but it seems the big innovation might have been figuring out a way to work with these quantized word vectors in-memory without losing much performance

Edit: seems it may not work in-memory

<comment><USER11> I wonder, shouldn't the vector space _ideally_ be much more than high- dimensional than hundreds? Like many thousands at least. (I mean, our brains likely can hold much more than hundreds of dimensions internally, I'd guess.) It's just that we don't have the computing power yet?

<comment><USER12> That's the curse of dimensionality in effect. In very high dimensions, the probability that two randomly chosen vectors are orthogonal increases, and conventional distance metrics become less useful since most vectors end up very far from each other.

In geometric terms, think of a circle embedded in a square. As the number of dimensions increases (e.g. a sphere embedded in a cube, a 4-dimensional ball embedded in a 4-dimensional cube, etc), most of the volume in the cube is outside the radius of the sphere. Any vectors you have effectively become sparse.

Basically this means that, while you need large dimensionality to model complex relations, high dimensionality makes it difficult to model these relationships (and need exponentially more data).

</comment></comment></comment></comment><comment><USER13> Only tangentially related, but we've recently tried to find an encoding of text that's "stable" with regards to it's characters (as opposed to stable wrt semantic meaning as here). That is, similar words (or fragments) such as "carg" and "cargo" should yield a simliar encoding.

To our surprise, we couldn't find any example or description of someone doing this before. Is this such an uncommon problem or did we just not search in the right places?

FWIW, our use case is a search tool with live result listing, so we're dealing with word fragments and would like the outcome to be somewhat stable as the user types along. We ended up rolling our own, but it has certain shortcomings, such as a hard character limit.

<comment><USER14> > To our surprise, we couldn't find any example or description of someone > doing this before. Is this such an uncommon problem or did we just not > search in the right places?

This is one of the defining differences between Word2Vec and Fasttext. But fasttext incorporates these character vectors as part of calculating the semantic vector, so you can't expect carg and cargo to end up being similar, but people have thought of it.

I don't think partial search is that uncommon, but I don't think it is usually solved by using vector representations similarly to word vectors. It seems like what you are looking for is usually accomplished by edit-distance / Levenshtein distance [1]

[1] [https://en.wikipedia.org/wiki/Levenshtein_distance](https://en.wikipedia.org/wiki/Levenshtein_distance)

<comment><USER13> Yeah, Levenstein distance is pretty close to our goal metric of "similarity". The thing is that we're feeding the search query into a neural network, hence we need some kind of vector represenation.

<comment><USER15> Need?

Or don't feed the search query directly into a neural network?

<comment><USER13> I don't understand; what do you mean?

<comment><USER16> Do what a search engine does: detect a more common form via edit distance and substitute that word.

</comment></comment></comment></comment></comment></comment><comment><USER17> Optimizations are always great :-)

This sounds similar to Maciej Kula's experiments in _" Binary Latent Representations for Efficient Ranking: Empirical Assessment"_, [https://arxiv.org/abs/1706.07479](https://arxiv.org/abs/1706.07479).

Maciej shared this [1]: "FWIW I ran similar experiments on recommendation tasks. My initial results were very encouraging (and similar to those reported here), but the effect disappeared entirely after adding regularization to the baseline. I would have more confidence in the results reported here if the authors ran a more principled hyperparameter search, and added regularization to their baseline."

[1] [https://twitter.com/Maciej_Kula/status/976028573487239168](https://twitter.com/Maciej_Kula/status/976028573487239168)

</comment><comment><USER18> This _could_ be useful with DNA/protein sequences - there's dna2vec and similar approaches to mine sequences for regulatory elements and more, the problem when compared with word2vec is that you have an order of magnitude more words. You don't really have words so you split the continuous DNA sequence into overlapping mers and treat that as words (afaik). That means that dna2vec takes forever (>24h with one chromosome) and builds a huge vector. I wonder whether this will speed up things...

</comment><comment><USER19> Very cool that this beats Word2Vec on SQuAD! I wonder if the current state of the art models are using the standard GloVe word vecs and might see improvement from this. Tbh I don't know too much about how those have been implemented though, haha.

I'm curious, how many values did you try for the quantization functions? Without thinking too much about it, that seems like one of the hyperparams that could have a pretty big impact on performance.

<comment><AUTHOR> You're definitely right, the quantization function and its values definitely have an impact on performance.

For 1 bit I think I tried something like -1/+1, -.5/+.5, -.25/+.25, -.333/+.333. and something like -10/+10 -- (and I think a few more). It seemed -.333/+.333 worked the best while +10/-10 did the worst on the google analogy task (getting like 0% right). All this was tuned on 100MB of Wikipedia data.

<comment><USER20> Have you considered doing gradient descent on the quantization steps? It looks to me like the model should be differentiable with respect to those values, so I'm not sure why you'd have to fix them to a constant.

<comment><AUTHOR> Hm what do you mean? I'm not quite seeing how to differentiate with respect to the quantization steps.

<comment><USER20> Say you have a function f(q(x)) where q quantizes x into one of s_1, ..., s_n. Then if q(x) = s_i for a certain x, df/ds_i = df/dq and df/ds_j = 0 for all j != i.

That breaks down for values of x precisely at the boundary between steps, so I should have qualified "differentiable" with "almost everywhere".

It also occurs to me that this might interact strangely with the approximation dq/dx = 1, but since the quantization steps are globally shared, I think it should be stable anyway.

If the evaluation suite for your code doesn't require too much manual interaction, I might try and see for myself.

<comment><AUTHOR> That's definitely an interesting idea -- it seems this would allow for boundaries that "change" along with the data (instead of having static boundaries as it is). Would be interested to know how that turns out!

</comment></comment></comment></comment></comment></comment><comment><USER7> Interesting.

So, this approach computes a "traditional" neural embedding, in say, R^50, and then "brutally" replaces each of the reals with an integer in Z_2,4,8...

I can't quite put my finger on it, but my hunch is that this naive method, while already delivering interesting results, can be __drastically __improved upon.

* don't use a fixed bit depth for all vector components

I guess it depends on what you're trying to optimize for -- what algebraic properties you wish to preserve for the end application:

If the end goal is: "linearity be damned, I want a stupidly fast but inaccurate way of doing approximate nearest neighbor search", then turning words into bitvectors, and using hamming distance, not(xor(a,b)), &c" works.

Either way, thanks for the ideas, OP. (was going to go on with some mathematical stuff which I suspect would improve upon it, but decided to either shutup, or put-up-and-credit-you.)

<comment><USER21> I would appreciate if you elaborated on your supposed improvements.

</comment></comment><comment><USER22> Very nice, thanks for creating this!

In addition to (possibly new) hardware that supports much larger memory, compression techniques like this might allow us to start operations with "phrase embedding" or eventually even whole "sentence embedding."

</comment><comment><USER23> It's interesting and slightly uncomfortable that the illustration for similar words uses the word "man" as an example, given the gender biases that result from learning word vectors solely from word distributions.

To explain, although I'm sure the author himself is familiar with the issue: For any word that is disproportionately associated with one gender in the corpus, the model will learn that gender difference as part of the representation of the word, pretty much baking it into all applications. [1] It's all fun and games when this helps you find the difference between "king" and "queen", but it becomes a problem when the same difference appears between "genius" and "beautiful".

I haven't evaluated these vectors for built-in biases, but I assume they would have similar problems to the pre-computed word2vec and GloVe embeddings. (If they don't -- if quantization is a natural way to counteract bias -- then that's an awesome result! But an unlikely one.)

To the author: I don't mean this to sound like an accusation that you haven't done this yet; I know that short papers can't tell two stories at the same time. But the next step is pretty clear, right? How do these vectors measure on the Word Embedding Association Test for implicit bias? Does Bolukbasi's de- biasing method, or an analogue of it, work on quantized embeddings?

[1] Bolukbasi et al., "Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings." [https://arxiv.org/abs/1607.06520](https://arxiv.org/abs/1607.06520)

<comment><USER24> In modern linguistics grammar is used descriptively rather than prescriptively, i.e. it's about describing how speakers actually use language rather than being about telling them how to use language 'properly'.

If there's gender bias in certain words it might be interesting to point that out but it's not the linguists' job to 'de-bias' the grammar and the underlying model.

There's an undeniable gender bias in certain words (men probably are less frequently referred to as 'beautiful' than women while on the other hand 'genius' likely is more often used when referring to men). Glossing over that by smoothing models not only misrepresents how speakers use language but probably doesn't really help the cause either.

If the corpus used displays a disproportionate association of certain words with one gender this could just mean that the corpus is insufficient for representing a language in general (as opposed to just a particular register or sociolect), which is a common problem in computational linguistics, not just when it comes to gender biases.

<comment><USER25> > it's not the linguists' job to 'de-bias' the grammar and the underlying > model.

Sure, but this is not a project of linguistic research. It's a tool that could see real-world usage. Not perpetuating the stereotypes that suffuse the data we feed into such models does seem like a worthy Endeavour, and I find it encouraging to see the creator taking these concerns seriously in this threat.

(As opposed to the community-at-large, which quickly send them to the bottom of the threat)

<comment><USER26> The issue is that the undesired language constructs one would like to remove are not universal. What is an improvement for one user group will be a regression for another & vice versa.

<comment><USER25> I'm a white, able-bodied, mostly heterosexual man, and I reject the theory that (non)discrimination is a zero-sum game.

<comment><USER20> Debiasing word vectors is not guaranteed to yield less discriminatory results in every application. In many languages, gender is expressed grammatically and there are different words for e.g. male and female scientists. If you debias the word vectors to remove associations with gender, it becomes impossible to distinguish those two. Now if you translate between two languages with that distinction, the model will have to fix itself to one of the two possibilities, and the translation error will be minimized by always picking the more common form. _" First year of all-female Nobel prize winners and machine translation gets it totally wrong!"_ By debiasing the word vectors, the model has become more biased and less accurate overall.

If you create a discriminatory machine learning model, it's unlikely to be because you didn't debias your input representation. More often, you're training it on a task whose real-world statistics are biased, and the model learns to accurately reflect that bias to solve your task. The solution to that is not to modify the input, but rather the output you expect the model to provide.

<comment><USER23> Ah yes. This approach to ML bias is popular in some large corporate research groups:

\- Say a lot of nice things about fairness

\- Talk about the importance of de-biasing and doing it as well as possible

\- Show (legitimately) that the farther downstream in your ML process you apply de-biasing, the more sound the results are

\- Assume that any successful ML model will be used downstream in something else

\- Therefore, never de-bias anything, except the final output of something where your company benefits from showing fairness: that is, a demonstration at a conference talk about fairness

\- The time to de-bias real applications is always "later" and everyone can say they are working on doing the right thing

People who release data artifacts have the ability to de-bias now instead of later. It's not perfect, but it is good. If you rely on the developer after you to do the de-biasing that you're not doing, you're ensuring that de- biasing won't happen, because they won't do it either.

<comment><USER20> My point is that the appropriate debiasing is task-specific. You can completely eradicate bias one one task, but as soon as someone builds on top of your results, the bias is going to creep right back in, and likely to be worse due to the information loss in previous layers. If you just hand a bias- conscious developer a bunch of debiased word vectors with the implication that it will make their model less biased, I don't think that's really helpful. Like security, bias in machine learning needs to be addressed on the level of complete systems, it's not something you can leave to some simple input sanitizer.

<comment><USER26> That's because what's called 'debiasing' is not actually removing bias but adding another opposite one at the superficial layers we can actually notice.

<comment><USER23> That's possibly a fair description of the kind of as-late-as-possible de- biasing that I find insufficient.

I think there is not nearly enough research into de-biasing at training time, perhaps because it would be discouraged by the reward structure of our field.

("You mean I can make my model more complex and my paper longer, and in return I lose the 0.15% accuracy gain that's the entire reason I could call the result 'state of the art'? Sure, I'll get right on that!")

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER27> Very cool! I like the visualizations a lot. Did you try to get an interpretation for what each quantized vector dimension means (have just skimmed, not read)?

Also, I am curious why you chose to go straight to publishing on Arxiv? I am actually also in CS224N right now and have a project me and my collaborator feel is publication worthy, but our plan is to go the normal route of submitting to a conference and only putting it on Arxiv after the review process (though our code is open source, so now that I think about it maybe that's not that useful a plan...).

<comment><AUTHOR> Definitely tried to figure out if the dimensions mean anything -- as far as I can tell they don't really mean much :(

<comment><USER28> If you want them to be meaningful without changing the model...

Couldn't you rotate the basis to minimize the distance between each basis vector and it's nearest neighbor?

<comment><AUTHOR> Haven't tried this but this is definitely a good idea for visualizing what's going on!

</comment></comment></comment></comment><comment><USER29> Is there an explanation for why almost all the words most distant from "man" are either uppercase or otherwise lexically anomalous? Is it largely driven by their being low frequency in the corpus?

</comment><comment><USER30> can this be done with fasttext as well?

Word2bits is definitely great for memory-constrained applications but for server use memory isn't as much a constraint (there's a direct word -> vector relationship so you can just put it in a database)

it would be amazing to combine this with fasttext's ability to generate vectors for out-of-vocab words.

<comment><USER31> Yeah you can - [https://github.com/facebookresearch/fastText/blob/master/qua...](https://github.com/facebookresearch/fastText/blob/master/quantization- example.sh)

</comment></comment><comment><USER32> The data engineer in me thanks you. Great work.

The paper would be a great weekend read.

</comment><comment><USER14> Can someone explain how to read the graph?

<comment><AUTHOR> Yeah, I should definitely put more detail in the writeup -- thanks for the feedback!

What's happening with figure 1a (epochs vs google accuracy) is that as you train for more epochs the full precision loss continues to decrease (dotted red line) but accuracy also starts decreasing (solid red line). This indicates overfitting (since you'd expect accuracy to increase if loss decreases). The blue lines (quantized training with 1 bit) do not show this which suggests that quantized training seems to act as a form of regularization.

Figure 1b is pretty similar, except on the x axis we have vector dimension. As you increase vector dimension, full precision loss decreases, yet after a certain point full precision accuracy decreases as well. I took this to mean that word2vec training was overfitting with respect to vector dimension.

<comment><AUTHOR> Oops, thought you meant the graphs (with the dotted/solid lines) in the writeup.

If you're referring to the image under "Visualizing Quantized Word Vectors" then each row is a word vector (and there are only two colors since each parameter is either -1/3 or +1/3).

<comment><USER14> Thanks for the reply. Yes, I did mean the image under "Visualizing Quantized Word Vectors".

I did get that the colors indicated values of dimensions, but I suppose what I really meant is, what is the take-away message? To me, it just looks like noise. Is there a pattern I should look for and go "a-ha, I see"?

<comment><AUTHOR> You can kind of see that words that are similar have similar looking vector values (that's why there are vertical stripes of yellow / black). But you're right in that most of it just looks like noise. I put the picture there mainly to show what the quantized vectors look like.

<comment><USER33> In Figure 2, have you noticed that every 25 words or so, there are off-pattern words? They are represented by visually differing lines (looks like a glitch, and it may be a glitch as this happens for all visualization charts).

What words correspond to these "glitches" for "mushroom" for example? In the case of "mushroom" there is a glitch line just below "earthstar".

Can you provide the full y-axis word vector for any of the visualization charts?

Nice work.

<comment><AUTHOR> Here's the one for man:

['man', 'woman', 'boy', 'handsome', 'stranger', 'gentleman', 'young', 'drunkard', 'devil', 'lonely', 'lady', 'lad', 'drunken', 'beggar', 'kid', 'effeminate', 'brave', 'bearded', 'himself', 'dressed', 'loner', 'meek', 'sees', 'hustler', 'girl', 'coward', 'thief', 'wicked', 'person', 'balding', 'dashing', 'deranged', 'tramp', 'mysterious', 'him', 'pretends', 'lecherous', 'friend', 'shepherd', 'portly', 'bespectacled', 'jolly', 'thug', 'gangster', 'dapper', 'genius', 'slob', 'beast', 'hero', 'hoodlum', 'policeman', 'elderly', 'drunk', 'manly', 'mustachioed', 'ruffian', 'cop', 'burly', 'beard', 'fool', 'terrified', 'scarecrow', 'scruffy', 'lover', 'peddler', 'remembers', 'supposedly', 'gambler', 'bloke', 'bastard', 'acquaintance', 'mighty', 'playboy', 'unshaven', 'prostitute', 'pimp', 'mans', 'skinny', 'carefree', 'scoundrel', 'crook', 'obsessed', 'surly', 'fancies', 'accosted', 'foolish', 'jovial', 'cocky', 'shifty', 'loves', 'narrator', 'butler', 'dying', 'casually', 'waiter', 'evil', 'frightened', 'gigolo', 'conman', 'cunning']

(Edit: Actually this isn't quite right as it doesn't match the image. Many of these vectors actually have the same distance to "man" and the dict doesn't keep a deterministic order. What you can do is modify [https://github.com/agnusmaximus/Word2Bits/blob/development/s...](https://github.com/agnusmaximus/Word2Bits/blob/development/src/misc/analysis/w2bvisualize.py) and run it on the 1 bit 400k vectors and see what it prints out. To run it do: `python w2bvisualize.py path_to_1bit800d400kvectors` then it should generate similar figures as in the writeup)

</comment></comment></comment></comment></comment></comment></comment><comment><USER34> interesting that 'artists' is a furthest neighbor of 'man' in the example

<comment><AUTHOR> This might be because "Artist" has an uppercase "A" \-- I trained all the word vectors to be case sensitive so "Artist" is not the same as "artist" (which should be closer to "man" than "Artist")

<comment><USER35> Why would you do that? If you look at something like LSA, they goal is to uniform those, rather than distinguish. artist and Artist should be (near)100% match; what are you trying to do here?

<comment><AUTHOR> Main reason I did it this way is because Facebook's DrQA (which I evaluate the vectors on for the SQuAD task) uses case sensitive vectors.

Was a tough decision between choosing whether to train case sensitive vectors / case insensitive vectors and a future task would be to train case- insensitive vectors.</comment></comment></comment></comment>


Show HN: Blog Reader – Listen to webpages via a personal podcast - <AUTHOR> https://www.blogreader.com.au/ <comment><AUTHOR> I've been working on this for the past few weeks and was surprised to see pretty polished competitor posted yesterday: [https://news.ycombinator.com/item?id=19454863](https://news.ycombinator.com/item?id=19454863)

Oh well, great minds think alike!

EDIT: Similarly, this uses Google's Text to Speech ([https://cloud.google.com/text-to-speech/](https://cloud.google.com/text-to- speech/)). If this gets some traction I would love to develop my own TTS model using WaveNet ([https://deepmind.com/blog/wavenet-generative-model-raw- audio...](https://deepmind.com/blog/wavenet-generative-model-raw-audio/)), since the TTS API is super expensive.

<comment><USER2> I thought about adding tts to my blog too. I would start from [https://github.com/mozilla/TTS](https://github.com/mozilla/TTS)

<comment><USER3> Thanks for sharing. Went through the sample audios they seem to be much better than Google TTS.

<comment><AUTHOR> Are you referring to the "basic" Google TTS or the WaveNet option? I believe the WaveNet voices are much, much better than those produced my any other TTS service or publicly available model, but I'd love to see some counter- examples.

</comment></comment></comment></comment><comment><USER4> I can use Edge browser for the same. Then why should I pay $12 per hour?

<comment><AUTHOR> \- The Edge browser feature requires you to select text and then listen to it while you have the webpage open, my service can be used asynchronously like any other podcast - when you are commuting, cooking etc.

\- The Edge browser text-to-speech feature sounds stilted and robotic, and I personally would not want to listen to it for an hour. The TTS converter than I am using sounds better, but costs money. Check out the demos at the bottom of the landing page for some examples.

\- My service costs $2 AUD per hour, not $12 per hour.</comment></comment>


The beginnings of the end of private life in the late nineteenth century - <AUTHOR> https://www.laphamsquarterly.org/roundtable/defining-privacy-and-then-getting-rid-it?page=all <comment><USER2> It's a good article, but the title is somewhat misleading. The article argues that "private life" historically meant the affairs of "gentlemen". And not privacy generally, as we think of it today. So strengthening of women's rights, for example, involved weakening "private life" protections.

But "private life" protections have also been strengthened, more generally. The right to use contraception. Gay and lesbian rights.

<comment><USER3> >But "private life" protections have also been strengthened, more generally. The right to use contraception. Gay and lesbian rights.

I think the article makes a sensible distinction here between 'personal privacy' and private space which the author calls 'aggregate privacy'.

The former is really a way for individuals to obtain autonomy and control, whereas private spaces can be used to dominate and escape from civil society.

The conception of the home as 'a man's personal castle' was probably the culmination of this. People who lost more and more control in an increasingly more open public space and expanding commons reserved their right to control their family without anybody watching.</comment></comment>


Ask HN: Strategy for handling user-managed 3rd party secrets for SaaS App? - <AUTHOR> I'm looking for current best practices/recommendations for how to properly store, access, and manage credentials (tokens, certificates, keys etc) when they are provided by an end-user to our app.
This is mainly in context of a multi-tenant SaaS app. I realize providing secrets to a 3rd party causes most people to shudder, but this could apply to a dedicated or on-premises version of our app, where users still have the ability to manage custom connections via the UI.
This is different than managing the secrets for our own internal infrastructure.
The scenario when this issue comes up is when you provide users the ability (via a frontend UI etc) to manage:
1. Custom API Connections
2. Custom Database connections
3. Managing custom hosting/server settings
Example use case:
User uses our app to view data from a database they own and manage.
1. An admin type user adds connection info/credentials via a settings page in our app.
2. We store those settings permanently.
3. The admin (using our UI) adds authorization rules for other users in our app (within their tenant/company).
4. These users access our app to view data from their db.
5. Our backend service receives the request, retrieves/caches their tenant db settings, connects to their db, retrieves data.
Although this case is for a database, it could be for a GraphQL/REST Api service also.
The Issue:
Everyone know's not to store passwords in plain text, but these are basically passwords. The difference is that our backend needs to be able to read and use them. They need to be managed and stored in a scalable way since they are managed by the end-user.
We have a few strategies in mind but are looking for the most current recommendations.
Most of our infrastructure is using Kubernetes and Compute Engine on Google Cloud.
edit: formatting <comment><USER2> [https://www.vaultproject.io/](https://www.vaultproject.io/)</comment>


How to Crawl the Web Politely with Scrapy - <AUTHOR> https://blog.scrapinghub.com/2016/08/25/how-to-crawl-the-web-politely-with-scrapy/ <comment><USER2> In the past we built and operated Greece's largest search engine(Trinity), and we would crawl/refresh all Greek pages fairly regularly.

If memory serves, the frequency was computed for clusters of pages from the same site, and it depended on how often they were updated(news sites front- pages were in practice different in successive updates, whereas e.g users homepage were not, they rarely were updated), and how resilient the sites were to aggressive indexing (if they 'd fail or timeout, or it 'd take longer than expected to download the page contents than what we expected based on site- wide aggregated metrics, we 'd adjust the frequency, etc).

The crawlers were all draining multiple queues, whereas URLs from the same site would always end up on the same queue(via consistent hashing, based on the hostname's hash), so a single crawler process was responsible for throttling requests and respecting robots.txt rules for any single site, without need for cross-crawler state synchronisation.

In practice this worked quite well. Also, this was before Google and its PageRank and social networks (we 'd probably have also considered pages popularity based on PageRank like metrics and social 'signals' in the frequency computation, among other variables).

<comment><USER3> In the current web, sites like Amazon are so large that you'll need many crawlers. On the plus side, it appears that almost all large sites don't have rate limits.

<comment><AUTHOR> Crawl-delay is not in the standard robots.txt protocol, and according to Wikipedia, some bots have different interpretations for this value. That's why maybe many websites don't even bother defining the rate limits in robots.txt.

<comment><USER3> I was referring to an actual rate limit, not crawl-delay. For example, YouTube is pretty strict about rate limits:

[http://www.bing.com/search?q=%22We+have+been+receiving+a+lar...](http://www.bing.com/search?q=%22We+have+been+receiving+a+large+volume+of+requests+from+your+network%22+site%3Ayoutube.com)

I agree that crawl-delay is rare, and often it's set too long so that it's impossible to fully crawl a site -- as if the webmaster set it up 10 years ago and never updated it as their site got faster and bigger.

</comment></comment></comment></comment><comment><USER4> In my experience the best way to crawl in a polite way is to never use an asynchronous crawler. The vast majority of small to medium sites out there have absolutely no kind of protection from an aggressive crawler. You make 50 to 100 requests per second chances are you're DDoS-ing the shit out of most sites.

As for robots.txt problem is most sites don't even have one. Especially e-commerce sites. They also don't have a sitemap.xml in case you don't want to hit every url just to find the structure of the site. Being polite in many cases takes a considerable effort.

<comment><USER3> Search engine crawlers use adaptive politeness: start being very polite, and ramp up parallel fetches if the site responds quickly and has a lot of pages.

<comment><AUTHOR> That's kind of what Scrapy's AUTO_THROTTLE middleware does.

</comment></comment></comment><comment><USER5> See also Tuesday's HN discussion on the ethics of data scraping ([https://news.ycombinator.com/item?id=12345952](https://news.ycombinator.com/item?id=12345952)), in which Hacker News is _completely split_ on whether data scraping is ethical even if the Terms of Service _explicitly forbids it_.

<comment><USER6> Are you trying to imply that's a ridiculous position? I don't see it as one.

<comment><USER7> I'll say it's a ridiculous position. How can it possibly be ethical? The owner of the server and the content has specifically told you to stop sending packets at it.

I honestly don't know how to construct an argument for this because it's so obvious to me.

<comment><USER8> It's ethical because it's a public internet. The same way you can't use the force of law to stop a homeless guy from asking you for change as you walk along a public street, you can't [shouldn't be able to] use force of law to stop a client from asking your server for data as it sits connected to a public network.

It's not unethical for a beggar to continue asking for change. It's up to the passerbys to choose whether or not they'll honor his request, but he is free to make it as long as he doesn't get out of control. Many people see the client-server relationship that exists online similarly. As the beggar can't receive anything that the giver doesn't willingly give, neither can the client receive anything the server doesn't willingly give.

It wouldn't make any sense if a guy could give the beggar change and then sue him and say that he shouldn't have gotten change because he actually wanted to use it for his lunch. The judge would say, "Well, why did you give it away? You can't just change your mind and then sue someone over it." This is also what judges should ask servers who dispense information to clients and then try to take it back.

tl;dr there's no harm in asking for data, even after someone has told you no, as long as you do so reasonably.

<comment><USER6> Strictly speaking there are a lot of places where panhandling is not legal.

</comment></comment></comment></comment></comment><comment><USER9> Reading the previous thread again, I suppose that many of those against scraping didn't realized they've already lost : with Ghost, Phantom, and now headless Chrome you're going to have a hard time to detect a well built scraper.

Instead of fighting against scrapers that don't want to harm you, maybe it's about time to invest in your robots.txt and cooperate.

You could say that scraping you're website is FORBIDDEN, but come on : if Airbnb can rent houses, I can scrap you site.

<comment><USER10> It depends on your definition of harm. When your product is what's published on the websites and you regularly find ripoffs of said website publishing your ripped off content, maybe you'd feel differently about it.

<comment><USER11> fair enough but I don't think that's the main purpose. There are many many cases where you would want to scrape something and often people would probably be encouraged in doing so in a "polite" way if websites didn't make it hard.

<comment><USER12> Yes, or if they just provided a csv with all the data most people wanted to scrape anyway with a plain English explanation about how it can be used.

</comment></comment></comment></comment><comment><USER13> I worked on a research project to develop a web-scale "google" for scientific data and we found very interesting things on robots.txt, from "don't crawl us" to "crawl 1 page every other day" or even better "don't crawl unless you're google".

Another thing we noticed is that google's crawler is kind of aggressive, I guess they are in a position to do it.

Our paper in case someone is interested: Optimizing Apache Nutch for domain specific crawling at large scale ([http://ieeexplore.ieee.org/document/7363976/?arnumber=736397...](http://ieeexplore.ieee.org/document/7363976/?arnumber=7363976))

<comment><USER14> This is why I think Google's position as the #1 search engine will never go away. Many sites will tell your bot to go away if you're not Google. They don't care if you're building a search engine that will compete with Google.

<comment><USER3> At blekko, we did not find this issue to be a significant one... almost everyone who banned our crawler was a crappy over-SEOed website.

<comment><USER14> [https://www.linkedin.com/robots.txt](https://www.linkedin.com/robots.txt)

[https://yelp.com/robots.txt](https://yelp.com/robots.txt)

There goes all Linkedin + Yelp content from your index.

<comment><USER13> What about [https://www.facebook.com/robots.txt](https://www.facebook.com/robots.txt)

..and medium-sized/small sites are even worse.

The irony of Facebook being a core part of all NSA surveillance programs and their terms of service including their "Automated Data Collection Terms" [https://www.facebook.com/apps/site_scraping_tos_terms.php](https://www.facebook.com/apps/site_scraping_tos_terms.php)

</comment></comment></comment></comment></comment><comment><USER15> The current protocols promote data exchange and since websites are primarily designed to be consumed, there is really no way to stop automated requests. Even companies like distilli[1] networks that parse inflight requests have trouble stopping any sufficiently motivated outfit.

I think data should be disseminated and free info exchange is great. If possible, devs should respect website owners as much as possible; although in my experience people seem to be more willing to rip off large "faceless" sites rather than mom&&pops. Both because that is where valuable data is, and it seems more justifiable even if morally gray.

Regardless, the thing I find most interesting is that Google is most often criticized for selling user data/out their users privacy. However, it is oft not mentioned that Googlebot & the army of chrome browsers are not only permitted, but encouraged to crawl all sites except a scant few that gave achieved escape velocity. Sites that wish to protect their data must disallow and forcibly stop most crawlers except google, otherwise they will be unranked. This creates an odd dichotomy where not only does google retain massive leverage, but another search engine or aggregator has more hurdles and less resources to compete.

[1] They protect crunchbase and many media companies.

</comment><comment><USER16> If you're worried about being a pain in the ass to administrators, with a web- scraper, they probably need to rethink the way they have their website set up.

</comment><comment><USER17> An alternative to Scrapinghub: PhantomJsCloud.com

It's a bit more "raw" than Scrapinghub but full featured and cheap.

Disclaimer: I'm the author!</comment>


Pharmaceutical giant 'plotted to destroy cancer drugs to drive prices up 4000%' - <AUTHOR> http://www.independent.co.uk/news/health/drug-giant-aspen-plot-destroy-cancer-medicine-big-pharma-times-investigation-a7683521.html <comment><USER2> Yet the names of those responsible are omitted, safely hidden behind 'the company'.

</comment><comment><USER3> It's exactly like the cyberpunk future I've read about in novels.

<comment><USER4> With the difference that in most novels usually the good guys win.</comment></comment>


Ask HN: Where to write and publish simple post? - <AUTHOR>

I just want to express some though about the startup world and write about my experiences but I don't want to set up a blog or something. http://www.medium.com Looked nice for that purpose but it is still in private beta for writers.<p>Any ideas where I could write my content? <comment><USER2> If you already have a github account: <http://gist.github.com>

Share it via <http://gist.io>

</comment><comment><USER3> These were on HN a couple of months ago:

<http://authpad.com>

<http://throwww.com>

<comment><AUTHOR> Throwww looks very nice! Thanks

</comment></comment><comment><USER4> Anything wrong with tumblr, posterous, wordpress.com, blogger, etc.

The world is not short of free/cheap blogging tools ;)

<comment><AUTHOR> All those platform makes you create a "blog". I don't want a blog that I have to keep updating otherwise it will look odds with only 1 post every 2 months.

<comment><USER5> Being highly dependent upon acceptance by others is probably not a desirable trait in when it comes to writing for public consumption.

Most blogs aren't outstanding. Half are below average. That doesn't change what you have to say.

I've got three blogs I've started. I've got about half a dozen journals. Sometimes the gaps between entries in any of them will be a couple of years. Think less of me as a human being if you wish.

</comment></comment></comment><comment><AUTHOR> I would like to add that Google+ Is not an option for me. Its a social network I don't and won't use.

</comment><comment><USER6> HN?</comment>


Ask HN: Should an early engineer leave a startup facing down round/weak exit? - <AUTHOR> Hired after series A. Quite a few vested options, although the spread isn't huge. Company is 7 years old and seemingly facing either a down round or an exit. There has been an up and down but current trajectory seems ok.<p>Any reason to wait for the exit or is this a likely loser? Any way to judge the probability of the outcome or red flags to look for? <comment><USER2> No way to know based on the given information.

Listed companies lose value and face challenges, too. Just like you wouldn't view the outcome of a GE catalyst as a binary situation where the stock either zeroes or rallies, you shouldn't view a semi mature startup that way either. Try to think of the possible outcomes, assign probabilities to them, and decide whether they're favorable.

Ultimately it's your life. Nobody else can take risks for you, nor allocate years of your career on your behalf.

If you're turning to unknown people with near-zero information to give you career advice, then what you probably need is to sort out your thoughts alone or with a trusted friend. What would the exit look like? What signs of failure do you think you have seen? What is the current trajectory?

</comment><comment><USER3> Startups have very high highs and very low lows. It's normal for things to go extremely bad. I used to keep to a saying: "If you can avoid dying long enough, you'll be rich."

However, another strong rule of thumb I keep to is 'no bullshit'. Bullshit means retaining people who should be fired. Bullshit means following down a long route in order to blame someone. If you're spending a lot of time covering your butt, setting up documents of proof, attending pointless meetings so people can't blame you for not attending meetings... then that's a sign things are going downhill.

</comment><comment><USER4> Assuming you're in the US.

Unfortunately one thing you need to consider is your options exercise window. Most startups require you to exercise them within 90 days of leaving the company or you lose them. When you exercise them, you need to pay AMT tax on the "spread X number of exercised shares." (Or is this no longer true under Trump tax law? How did that saga end?) Depending on the spread and number of shares, this could be a significant price. But you have no way to liquidate the shares you receive, so you need to front a huge tax bill for receiving shares that may never liquidate.

Depending on the age of the company and your share agreement, you _mignt_ be able to sell the shares on a secondary market. Or if you have a good relationship with the founders you may be able to arrange a buy back, or possibly an extension of the exercise window.

The worst case scenario is you leave, exercise your options, pay a huge tax bill, and the company never liquidates so you're out a bunch of money.

Or, some pessimistic optimists might say the worst case scenario is you leave, do NOT exercise your options within the window, and then the company DOES liquidate...

All said, it's not an easy decision and you have much more information to work with than us. My advice: don't gamble your tax bill on weakly valued options. Whether that means you should quit or not, idk... up for you to decide.

<comment><USER5> You can defer the tax for 5 years, but you owe on the value at the time of deferment. So, if the valuation drops after you leave your still on the hook for the original deferred amount.</comment></comment>


A Refugee in Puerto Rico, 1942: Claude Lévi-Strauss and the Burden of Archives - <AUTHOR> https://www.bunkhistory.org/resources/1089 <comment><USER2> I get the not wanting to be sentimental (I did get rid of nearly everything I owned once) but there's a modicum of things people may want to keep and may regret not keeping: things related to family ancestry, pivotal life events come to mind.

More generally: I still don't fathom the populist, knee-jerk, sweeping generalizations of castigating and demonizing immigrants and refugees whom the Western countries created, and that often they themselves were once also. If a country becomes predominantly xenophobic, like say China or Japan were long ago, they fall into backwards ways: socially, politically, economically and technologically. It shouldn't be a political or partisan animosity issue that should be argued over or not discussed at all via the isolation of separation political universes, but a strength of values, integrity, community and collaboration with those whom may build future gigabusinesses elsewhere. Self- interest would welcome intellectuals, engineers, entrepreneurs and others with open arms and facilitate their integration as fast as they would like. Immigration is a form of customer service that signals values and qualities of the country... is our or your OOBE good?

<comment><USER3> The recent Pew poll on partisan attitudes showed that both Democrats and Republicans had much better views of immigrants than they did in 1994 (they used to be equally skeptical, they diverged but both are well below the previous values).</comment></comment>


Unsplash releases biggest tagged image dataset - <AUTHOR> https://github.com/unsplash/datasets <comment><AUTHOR> We are excited to launch this dataset to help with innovations that are on- going with ML research. The lite dataset contains 25k nature-themed Unsplash photos, 25k keywords, and 1M searches. The full dataset contains 2M+ high- quality Unsplash photos, 5M keywords, and over 250M searches.

<comment><USER2> Very cool! Quick question‐when unsplash first started, the images used cc licenses. What happened to those images? Did they get switched to the new unsplash license, or are they still cc licensed? Is this information contained in the dataset?

</comment></comment><comment><USER3> Unsplash is an amazing service (my website uses many, with credit), but the tags are often wrong.

Have you thought about letting the creators and people with sufficient "karma" downvote incorrect tags, and suggest new tags (perhaps from an edited, limited ontology?)</comment>


Interviewee opens for private beta - <AUTHOR> http://blog.clutterme.com/2007/08/today-is-big-day.html We went ahead with it anyway.  And in all fairness to pg and for the benefit of knowledge of everyone in this forum, ClutterMe isn't the idea that we originally applied with.<p>We'd like to thank you for your patience in my spamming of this forum.  I will keep you updated on our progress.<p>~Mark <comment><USER2> I watched the demo video. Congrats on what looks to be an easy to use user interface. The kind of app you are making is the type where the ui pretty much makes or breaks the entire thing. You seem to have pulled off a fairly good one

</comment><comment><AUTHOR> We went ahead with it anyway. And in all fairness to pg and for the benefit of knowledge of everyone in this forum, ClutterMe isn't the idea that we originally applied with.

We'd like to thank you for your patience in my spamming of this forum. I will keep you updated on our progress.

~Mark</comment>


Ask HN: Does Google rummage through the Gmail account of job-seekers? - <AUTHOR>

If you apply for a job at Google, will they read your personal Gmail account to get more information about you as a candidate? <comment><USER2> No.

I imagine google has a tight audit of who accessed emails, and when, and that those people need a good reason to do so. Poking around to see what emails a job seeker has sent or received is not one of those things.

I guess Google needs to work on their image if you seriously think there is any chance that Google is reading your private email because you applied for a job there.

</comment><comment><USER3> Probably not. But only because they probably don't have to actually read your mail to know everything they want to about you.

source: baseless paranoia and cynicism.

</comment><comment><USER4> no

<comment><AUTHOR> Source?

</comment></comment><comment><USER5> yes

<comment><AUTHOR> Source/link?</comment></comment>


Ask HN: What's my best option for managing Macbooks in my organisation? - <AUTHOR>

I've currently been tasked with the management of a small number of Macbooks. I just need to be able to enforce policies and full disk encryption, and limit user rights. Does anyone have any suggestions of what could suit my needs? <comment><USER2> A Google employee gave an overview of the tools they use to manage their (1000s of) staff Macs. [https://www.usenix.org/conference/lisa13/managing- macs-googl...](https://www.usenix.org/conference/lisa13/managing-macs-google- scale) For example he points to [https://github.com/google/macops](https://github.com/google/macops)

<comment><AUTHOR> They have quite the suite that you can host on GCE. If I can eek out the time to deploy it, this looks like a winner, thanks.

</comment></comment><comment><USER3> Windows</sarcasm>

In all seriousness, I'm not sure that Apple supply anything remotely close to the functionality of windows policy manager, though I've heard some good mumblings about centrify and their offerings.

[http://www.centrify.com/mac/group-policy-for-mac-os-x- deskto...](http://www.centrify.com/mac/group-policy-for-mac-os-x-desktops.asp)

</comment><comment><USER4> Profile manager?

[https://help.apple.com/serverapp/mac/4.0/#/apd0E2214C6-50F0-...](https://help.apple.com/serverapp/mac/4.0/#/apd0E2214C6-50F0-48C9-A482-74CEA1D77A9F)

<comment><USER5> I've had pretty mixed-to-bad experiences with profile manager. It doesn't seem to be a product that Apple puts much care into. On the other hand, I've had very good luck with Casper from JAMF.

</comment></comment><comment><USER6> You can use AirWatch to manage Macs, although this may be too much for your needs.

[http://www.air-watch.com/solutions/macos](http://www.air- watch.com/solutions/macos)</comment>


Why the Future of Work is "Smart Work" - <AUTHOR> http://blog.evenues.com/post/2013/03/19/Smart-Work-Spaces.aspx

<comment><USER2> I really like the idea of this type of physical office layout. I noticed the Mozilla office picture at the bottom of this post the other day: ( <https://news.ycombinator.com/item?id=5393864> ), and it really stuck with me. I forgot what the article was about, but kept it open in a tab for a couple days to keep looking back at that picture.

It really seems perfect for everyone working on laptops, but I think most people are more effective with an additional screen. Ideally, I would want some sort of docking station or at least a big monitor on an arm at each booth/table.

I guess the other downside is that it promotes very poor ergonomics. It would be a great setup for a change of pace, but I think if I were looking at a couple years of working in that environment, I'd start seeing some health problems. I guess if you're 22, you can handle anything for a while, but ergonomic mistakes accumulate over the years and can really start to cause a lot of pain. I think I'd love it at first, but after a few months, I'd long for an Aeron chair in a cube again.

Does anyone work in one of these type of offices and can you speak to the short-term and long-term comfort?

</comment><comment><USER3> "Smart Work" will be predicated by "Smart Managers."

Between the time and expense of commuting and the amount of time wasted at the office and in meetings it makes sense to monitor productivity and not attendance.

</comment><comment><USER4> Not sure how commuting to a "smart office" is any better than commuting to a regular office. Wouldn't the truly smart thing be to work from home?

</comment><comment><USER5> I wish more companies like Cisco and Google would do this...

</comment><comment><USER6> Right side - notice hard furniture, glare, total colour-insensitivity. Designers: no. Hackers: no. Business people talking loudly: yes. Lame!</comment>


Interested in BSD ports or are we all wasting time here? - <AUTHOR> https://github.com/dart-lang/sdk/issues/10260#issuecomment-174287146 <comment><USER2> Arrogance on all sides. Dart isn't obliged to support any platforms they don't want to or don't have the staffpower for.</comment>


Trump Tweeted a Sensitive Photo. Internet Sleuths Decoded It - <AUTHOR> https://www.wired.com/story/trump-tweeted-a-sensitive-photo-internet-sleuths-decoded-it/ <comment><USER2> It's a real marvel to be able to take such detailed photographs from that kind of distance.

As unhappy as I am out of principle with the undisciplined sharing of the photo, I have to admit it's really neat to see this kind of technology, as an ordinary citizen.

<comment><USER3> "undisciplined sharing of the photo"...How do you know this was the case? Let's think about this for a second...This wasn't a picture of a picture. This was the image in the brief. This image would have needed to be shared onto a different network that has access to Twitter or a phone with twitter access. That image would have needed to be removed from the brief, shared onto that system and then tweeted. You think this wasn't planned? The President isn't going to go through so many steps just to tweet a photo himself. This was done with staff who knows the classification.

<comment><USER4> Probably not a "plan". As President one can usually legally "release" classified info on a whim, and if T is allowed to do something on a whim, he typically will.

</comment></comment></comment><comment><USER5> Meaning he's storing classified stuffs on his iphone, syncing it to icloud and granting all kinds of other apps access to it.

<comment><USER6> It doesn't mean that at all. The President can (and did, in this case) unilaterally declassify anything. It's not classified.

<comment><USER7> Ah yes, the dingbat vs 8D chess debate.

<comment><USER4> The President is allowed to do certain Checkers moves on a Chessboard just by being President. Doesn't necessarily mean it's a good move, only that he can.

</comment></comment></comment></comment><comment><USER8> [https://youtu.be/MzYmwvhVkw0?t=1230](https://youtu.be/MzYmwvhVkw0?t=1230)</comment>


Dpulls – a dependency manager for your pull requests - <AUTHOR> https://www.dpulls.com/ <comment><AUTHOR> Hey everyone, I wanted to share with you something that I've been working on for couple of weeks now. If you are familiar with the dependency problem among Pull Requests and are struggling with it (more context here [https://github.com/isaacs/github/issues/959](https://github.com/isaacs/github/issues/959)), this might be interesting for you. It's something that already exists for gitlab and bitbucket but unfortunately not with github, so I decided to create my own service. Basically, dpulls will allow you to express a required order of merging for your pull requests (plus other features).

So hope this could be helpful for you.</comment>


Apple Working on Smaller, Cheaper iPhone to Combat Android - <AUTHOR> http://www.nytimes.com/external/venturebeat/2011/02/11/11venturebeat-apple-working-on-smaller-cheaper-iphone-to-c-29590.html?ref=technology

<comment><AUTHOR> I guess this is going to be game changer - bottom of pyramid at BRIC and other similar markets is huge and targeting them is a smart move ... i wish apple replicates this for all their product lines ...</comment>


XLR: Extensible Language and Runtime - <AUTHOR> http://xlr.sourceforge.net/

<comment><USER2> It's really difficult at that site to get a description of what a concept _is_ in their language. You can get a lot of what it _isn't_ , but nailing down what it _is_ is quite a challenge. Even the "what is the difference between objects and concepts" page is very vague: <http://xlr.sourceforge.net/concept/objects.html>

It seems to be a Haskell typeclass, or a Java interface. Arguably this entire language is a reaction to the fallacious belief than an object _must_ be a concrete noun and all methods _must_ be concrete verbs. Contra to repeated assertions to the contrary, there are _numerous_ Object-Oriented languages that can express the exact same solution to the "max" problem that the site tends to go on and on about: <http://xlr.sourceforge.net/examples/maximum.html> , with varying levels of precision on the mapping. But I actually am having a hard time coming up with any current language that is actually incapable of expressing the putatively- hard generic "max" function, and a number of them are pretty much as clean as XLR.

If I am wrong about what a concept is, I'd suggest the correct solution is a page somewhere early in the sequence that lays out exactly what a concept is. No handwaving. Not something at the level of "An object is some data with associated methods", but a literal discussion of what a "concept" is with actual syntax, preferably running code. If that is there, I couldn't find it in ~10 minutes clicking around looking for it.</comment>


2008 Startup School April 19. Confirmed so far: Andreessen, Bezos, DHH - <AUTHOR> http://www.startupschool.org/?x

<comment><USER2> Can you automatically allow YCNews users above a karma threshold? (It will help me book flight tickets in advance)

<comment><USER3> I like that suggestion (assuming I'm above the threshold).

</comment></comment><comment><USER4> Adding event to my google calendar... :) (Note to self: bring lunch as buying lunch last year was really hard because of the huge lines)

<comment><USER5> This is a really great suggestion. I think I ended up eating a bag of pretzels and a diet coke for lunch that day.

</comment></comment><comment><USER6> No spot on the app for yc.news usernames? You know - in case you think our comments here might be proof enough of technical ability and worthiness :)

<comment><USER7> I don't want to seem like we're trying to force people to sign up for News.YC. I'll tell you what, though: if everyone uses the same email address when they apply that they have in their profile, I'll try to match them.

That means you can't say e.g. whatever [at] gmail in your profile email field. But you don't need to anyway; it isn't visible to crawlers.

<comment><USER8> Is the majority recent college grads? Any 30ish entrepreneurs?

<comment><USER3> I can tell you that last year there was a good amount of 30ish entrepreneurs although the majority of folks did seem to be college students/recent grads (myself included).

</comment></comment></comment></comment><comment><USER9> Small suggestion: Add geographic location on the name tags so those traveling from afar can network with local hackers. I was really wishing for that last year...

<comment><USER7> If people have other suggestions, tell us here. One thing we'd thought of doing was having different colored dots people could put on their nametags if they were looking for a startup or looking for more people for one...

<comment><USER10> justin.tv could have a live link for the rest of us? maybe?

<comment><USER11> They did last year.

</comment></comment></comment></comment><comment><USER11> Sweet! Jessica pulls through with another great lineup.

Though I admit I'm most looking forward to pg's always entertaining talk.

</comment><comment><USER12> Anyone want to webcast this? I'm near Denver, and it's quite a commute.

<comment><USER13> second. illinois would be quite a trek, i'd LOVE to go.

student = poor = no way can i fly and hotel stay unless its SUPER cheap.

<comment><USER14> checkout couchsurfing.com. i bet you can find somewhere to stay near the stanford campus. and because you are a student, it will be minimally creepy.

still, a $300 ticket is too much for me

</comment></comment></comment><comment><USER15> Major props, in advance, to whomever can manage a high-quality recording of the talks. The podcasts are a great service for those of us who can't make it in person!

</comment><comment><USER16> Does anyone have figures for acceptance rates from previous years? I'd like to have a rough idea what my chances of getting to go are.

</comment><comment><USER17> okay - okay - my application is in. was surprised not to see a spot for YC News username, guess I would be really bummed to miss this.

</comment><comment><USER18> _"... 2008 Startup School April 19 ..."_

Has anyone ever thought to invite "John McCarthy" to talk?

</comment><comment><USER19> Startup School is BY FAR the best startup event to attend. The line-up is top notch year after year.

</comment><comment><USER20> my dilemma: I'm on the east coast so if I attend I would have to fly out just for the weekend. Should I still go or should I fly out to CA to visit friends for my longer spring break in late March?

</comment><comment><USER21> wow I cant wait. It looks like way to many people are going to register for this, they prob wont accept most of them.

</comment><comment><USER22> Awesomeness. Looking forward to it...

</comment><comment><USER23> woah. Nice work Jessica!

</comment><comment><USER24> Bezos + Andreessen FTW

</comment><comment><USER25> Quite a list there.

</comment><comment><USER26> last year's was great

</comment><comment><USER27> Damn, California.

And here's me, on the east coast.

<comment><USER28> So am I, but I've flown to San Francisco for the last 2 years for this - contemplating applying for another year to go out and network. I have to admit, I partied a little to hard the night before SS last year with some fellow hackers and was very late for the sessions.</comment></comment>


Strong types and testing – in Haskell - <AUTHOR> http://bitemyapp.com/posts/2014-11-18-strong-types-and-testing.html

<comment><AUTHOR> Written by the same guy (Chris Allen, aka bitemyapp) who wrote the "How I Start - Haskell" article from the other day. ([http://howistart.org/posts/haskell/1](http://howistart.org/posts/haskell/1) HN discussion: [https://news.ycombinator.com/item?id=8618486](https://news.ycombinator.com/item?id=8618486)) I thought this was a great example of doing a real world task in Haskell and also taking advantage of static typing.

</comment><comment><USER2> A shame this one didn't get any discussion, especially from those who didn't see the benefit in the original version.</comment>