Gene discovery from a genome-wide association study of educational attainment - <AUTHOR> https://www.dropbox.com/s/2x277c71kuszwwt/2018-lee.pdf?dl=0 <comment><USER2> > "If you did a study like ours 100 years ago, the strongest genetic predictor > of education would be how many X chromosomes you had, because society was > set up in a way that it was much harder for women to get educated than men," > says Benjamin. Likewise, many of the genes that are associated with > education today are likely important "because of how today's educational > system is set up. It requires people to sit at desks for hours, and listen > to instructions from a teacher. People who get restless, or are less > obedient to authority, will fare less well in that environment."

I think the above snip (ha) from [https://www.theatlantic.com/science/archive/2018/07/staying-...](https://www.theatlantic.com/science/archive/2018/07/staying- in-school-genetics/565832/) is useful in interpreting the results.

<comment><USER3> I would much prefer direct IQ correlates rather than using education as a proxy for something better measured with IQ tests.

<comment><USER4> What is "better measured" with IQ tests?

<comment><USER5> Intelligence.

<comment><USER4> I see so much work before me. Just google it:

[https://futurism.com/intelligence-changing-what-you-think- yo...](https://futurism.com/intelligence-changing-what-you-think-you-know- part-one/)

<comment><USER6> Do you have more good info on the subject?

</comment></comment></comment></comment></comment></comment><comment><AUTHOR> Media: [https://www.theatlantic.com/science/archive/2018/07/staying-...](https://www.theatlantic.com/science/archive/2018/07/staying- in-school-genetics/565832/)

FAQ: [https://www.thessgac.org/faqs](https://www.thessgac.org/faqs)

</comment><comment><USER7> From the FAQ:

> The current study further confirmed the finding from our earlier work that > the effects of individual genetic variants on educational attainment are > extremely small. The average effect size across the 1,271 genetic variants > was just 1.8 weeks of schooling per allele; even the SNPs with the strongest > associations only predicted around 3 weeks of additional schooling per > allele. Taken together, these 1,271 SNPs accounted for just 3.9% of the > variation across individuals in years of education completed.  > Here is > another way to think about this result. Imagine that we used the results for > these 1,271 genetic variants (not the ~1 million SNPs across entire genome > we discuss in FAQ 2.3) to predict the educational attainment for a new group > of people (separate from our discovery sample). We could then compare each > individual's predicted educational attainment to their actual educational > attainment. If we did so, our results suggest that we would find that the > predictions and actual outcomes correlate only very modestly (at about r = > 0.20). That, in turn, means that if someone were predicted to complete an > above average number of years of schooling (i.e., to be in the top half of > educational attainment), that person would have about a 58% chance of > actually being in the top half of educational attainment. Fifty-eight > percent is better than chance (i.e., 50%), suggesting that a prediction > based on these 1,271 SNPs has more power to predict educational attainment > than a coin flip—but only a bit more power. By contrast, a prediction based > on a polygenic score that combines ~1 million SNPs that we studied (see FAQs > 1.5 & 2.3) has more predictive power: r = 0.33, corresponding to 11% of the > variation across individuals.  > The contrast between the 3.9% of the > variation predicted by the 1,271 SNPs and the 20% known to be explained by > common SNPs (see FAQ 1.7) implies that there are many other SNPs that have > not yet been identified. Even larger sample sizes will be needed to identify > them.

They only included people of European ancestry to eliminate sources of noise and cofounding factors. The effect size is very small, yet it does exist.

This seems to prove that intelligence influenced by genetics but is affected by many thousands of genetic variations. There is no "smart" gene, just lots and lots of randomness.

<comment><USER8> >This seems to prove that intelligence influenced by genetics

Only if the educational attainment difference predicted by these genes is because of "intelligence." It could just as easily have to do with "stick- with-it-ness" or "I-have-parents-that-insist-I-go-to-college" or "I-do-what- is-expected-of-me."

</comment></comment><comment><USER9> I absolutely love that this study was accompanied by a FAQ ([https://www.thessgac.org/faqs](https://www.thessgac.org/faqs)).

My interpretation of this FAQ is that it provides a great deal of context that would be apparent to people in the field (and therefore not included in the paper itself) but is critical background for anyone else who is trying to make sense of it.

It therefore provides a bridge between the highly technical paper itself and journalism which too often misunderstands the true meaning of research results.

I think this is so valuable. I hope that this kind of thing will catch on for more research that attracts public interest.

<comment><USER10> I think the FAQ has a strong self-defence purpose. See the last question covered, which is probably the first one people care about:

> 3.7. Could this kind of research lead to discrimination against, or > stigmatization of, people with the relevant genetic variants? If so, why > conduct this research?

</comment></comment><comment><USER11> Has anyone made a tool to run your own 23andme genome data through this and get a score yet?

<comment><USER12> check out promethease.com

</comment></comment><comment><USER13> The 207 page Supplementary Material is here: [https://static- content.springer.com/esm/art%3A10.1038%2Fs415...](https://static- content.springer.com/esm/art%3A10.1038%2Fs41588-018-0147-3/MediaObjects/41588_2018_147_MOESM1_ESM.pdf) (probably pay walled).

Interesting discussion of genes and pathways involved around page 84. "Results: Causal Genes and Enriched Gene Sets"

I'll just type in the pathways. ( Supplementary Figure 22.) "Brain Specific Expression of Significantly Enriched Gene Sets across Development. (page 196)": Chromatic modification, protein binding transcription factor activity, npBAF complex, Central nervous system neuron differentiation, forebrain development, partial posttnatal lethality, abnormal cerebral cortext morphology, endometrial cancer , GAB1 signalosome, regulation of nervous system development, telencephaol cell migration, protein tyrosine kinase activity , neuron recognition, axon guidance, small cerebellum protein phosphatase regulator activity, signaling by eGFR, dendrite morphogenesis, signaling by NGF, regulation of neuron projection development, behavioral fear response, DAG and IP signaling, associative learning , voltage-gate calcium channel activity , ataxia, extracellular-glutamte-gate ion channel activity, post NMDA receptor activation events, regulation of neurotransmitter levels, regulation of synaptic transmission, synapse part ,gate channel activity.

Single Nucleotide Polymorphisms (snps) discussed in supplement are:rs10189857 rs1167898 rs11678980 rs18235539 rs182355396 rs186456786 rs186456786 rs186456786 rs5951458 rs61734410

Links to dbsnp for snps discussed in supplement: [https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=101...](https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=10189857) [https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=116...](https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=1167898) [https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=116...](https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=11678980) [https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=182...](https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=18235539) [https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=182...](https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=182355396) [https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=186...](https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=186456786) [https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=595...](https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=5951458) [https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=617...](https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=61734410) [https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=624...](https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=62422687)

<comment><USER14> It is worth noting that these SNPs aren't causal, they are just associated.

The resolution of GWAS isn't at the level yet where we can pick out individual changes that cause a phenotype. Instead we can detect that a chunk of the genome is associated with a phenotype. These snps are like a street name, they identify the region associated.

We need whole-genome sequence on a million-person scale to actually find the causal variant. That will come with time.

</comment></comment><comment><USER15> Please don't post Dropbox links, the PDF downloads are unusable on Android.

<comment><USER12> Android is confused by PDFs ?

</comment></comment><comment><USER6> I'll have to read this later since I'm at work but is there a layman's explanation for how these studies distinguish environmental and genetic influence? I don't see how you could accurately separate the two.

<comment><AUTHOR> Not entirely sure what you mean: genes cause environments, and interact with environments.

If you mean something like whether it's just picking up ancestry markers for families which happen to be rich etc, there are a lot of methodological components which try to minimize the more obvious ways of population stratification affecting results.

The most iron-clad part of showing direct gene->trait causation is the 4 sibling-comparison studies, where you see whether you can predict the difference between 2 siblings; as they have identical ancestries, families, locations etc, their environments are near-identical in the first place, and since meiosis at conception _randomizes_ genes between siblings, success in sibling comparison shows the genes must be causal. IQ GWASes have always passed the within-family test since Rietveld et al 2013, and this is no exception.

<comment><USER6> >If you mean something like whether it's just picking up ancestry markers for families which happen to be rich etc, there are a lot of methodological components which try to minimize the more obvious ways of population stratification affecting results.

Yeah this is actually what I'm interested in.

>The most iron-clad part of showing direct gene->trait causation is the 4 sibling-comparison studies, where you see whether you can predict the difference between 2 siblings; as they have identical ancestries, families, locations etc, their environments are near-identical in the first place, and since meiosis at conception randomizes genes between siblings, success in sibling comparison shows the genes must be causal. IQ GWASes have always passed the within-family test since Rietveld et al 2013, and this is no exception.

Couldn't this still lead to an overstating of the effect? So two brothers John and Jim have different builds due to genetics, John is a bit shorter and stockier and Jim is a bit taller and leaner. When they play with each other as children John usually wins when they wrestle and Jim usually wins when they play basketball. They both like winning so in highschool John joins the wrestling team and Jim joins the basketball team. Now if I do some gene testing wizardry on them as adults and try to determine the difference in basketball-skill I'll predict a large gap due to height genes and I'll get one. But part of that skill gap is due to environment (4 years of basketball practice) yet still getting attributed to genes.

<comment><AUTHOR> > ...When they play with each other as children John usually wins when they > wrestle and Jim usually wins when they play basketball...

No, this is still genetic causation (and something like this story is usually what is invoked for why the heritability of intelligence increases with age: more intelligent people seeking out niches which then encourage intelligence further rather than preferring to veg out in front of the boobtube, leading shared-environment to disappear while the permanent ongoing influence from genetics leads to heritability increases).

What's the definition of causation? It's a 'difference which makes a difference', or in Pearlean terms, if you reach in and do surgery on one node in a causal network, the counterfactual difference.

So in your basketball example, what if we reached in with CRISPR, say, and edited just the genes for builds in John to make him taller/leaner? What is the counterfactual of our intervention on the node of genes? As he grows up and becomes taller/leaner, now he still likes winning and playing with his brother is an even challenge, so he joins the basketball team along with Jim and they do well and keep playing and so on and so forth and your basketball PGS predicts less or no difference between them (since we caused them to have the same relevant genes) and indeed there is less or no difference; and if we had instead edited the other way, it'd go the other way etc. Thus, causation. That the causal pathway of the genes goes 'outside the skin', as some people put it, makes no difference that makes a difference for this particular question.

(A similar point applies to the 'nature of nurture' findings mentioned in Lee et al 2018 and the two earlier papers. Are they genetic causation? Yes, because if you counterfactually changed the genes and nothing but the genes, you would get different outcomes. They are differences which make a difference. They are, however, causal on things you didn't expect - eg edits in children would cause gains in grandchildren rather than the children.)

<comment><USER6> If what we're interested in is genetic causation then the relevant comparison is to John if he was still shorter but joined the basketball team anyways. In that case the variance between their skills would be less and yet no genes were changed. That illustrates that the original difference wasn't due to genetic but rather environmental factors.

>That the causal pathway of the genes goes 'outside the skin', as some people put it, makes no difference that makes a difference for this particular question.

When genes cause changes 'outside the skin' the way those outside changes affect your skill are environmental changes by definition. For example you're saying that in our current society, where you can choose your extracurricular activity, the difference in their basketball-skill that's attributable to genetics is _x_. But in a different society where high school basketball practice was state mandated the difference in their basketball-skill that's attributable to genetics would be _y_ where _y_ < _x_. The difference between _x_ and _y_ can't be caused by genetics because we haven't changed any genes so it shows that we were just misestimating genetic causation to begin with.

<comment><USER16> You are using a definition of genetic causation that is not used by anyone in the field of genetics. It sounds like your definition is "Is there any hypothetical environment where the phenotypic expression of a gene could be changed?" The answer is almost always yes.

Growth hormone influences height, imagine a world where everyone's level of growth hormone is regulated. Suddenly all of these growth hormone influencing genes no longer have any effect on height. By your definition growth hormone does not genetically cause height because of this hypothetical. You could see how using a definition is both incredibly restrictive as well as almost impossible to measure, and therefore not a very useful definition.

<comment><USER6> >You are using a definition of genetic causation that is not used by anyone in the field of genetics.

That's worrying.

>It sounds like your definition is "Is there any hypothetical environment where the phenotypic expression of a gene could be changed?" The answer is almost always yes.

Sure but we don't even need a hypothetical environment where high school basketball practice is state-mandated, we just need any hypothetical environment where John and Jim both go to basketball practice and their genes stay the same to see that what is labeled as genetic causation is actually environmental. I mean are you telling me the entire field has no way to distinguish a skill->environment->skill feedback effect from a genes->skill effect?

I'd say that genes seem essentialist, they're something that defines you and can't be changed but environment isn't that way, it's malleable and can be changed. So take this example, we're gonna say at least 11% of IQ is genetic, right? But if we pass some laws so that we have better teachers and smaller class sizes and better remedial programs then what IQ is only 8% genetic? Or even in the case that the laws help the top more and IQ expands to 14% genetic. In that case it doesn't seem that our measurements of genetics effects were accurately measuring the genetics at all, just environmental effects that are hard to account for.

>You could see how using a definition is both incredibly restrictive as well as almost impossible to measure, and therefore not a very useful definition.

Yes, I agree that what I'm saying would make genetic causation very difficult to measure. But it's also the truth about what genetic causation is. Instead we're just opting for what is easy to measure and labeling it as the thing that is hard to measure. That's not right.

<comment><USER16> Maybe if you gave some examples that would help.

Can you describe how you would even hypothetically measure your definition of "genetic causation"? Could you give an example of a "gene causing" the expression of an interesting trait in a way that is immune to environmental changes?

You're looking for a gene that causes an outcome completely independently of all environments. I don't think this exists for any interesting traits. Not to mention how much effect a change has on a given trait is dependent on the genetic make up of the population.

> But it's also the truth about what genetic causation is. No it's not, > everyone who talks about genetic causation seriously use a different > definition. I don't know why your own personal definition is better than the > definition used by every single person who studies genetics.

<comment><USER6> I would ditch the idea of widespread genetic correlations on complex traits, switch to a method that describes what genes (and their interactions) actually do to the human body. For example, this specific gene does x to a neurotransmitter which can interact with y or z to cause changes like this or that. (And I know it's actually more complicated than just a straight mapping of gene X -> trait Y.)

With my method you can still say that no matter the environment gene x does this thing which can interact with gene y or gene z or environmental factors a, b, c in certain ways.

I think that proving intelligence is XX% genetic is basically meaningless since we can't actually remove environmental effects and if I'm really honest I think it's popularity in the culture is usually driven by spurious motives.

>No it's not, everyone who talks about genetic causation seriously use a different definition. I don't know why your own personal definition is better than the definition used by every single person who studies genetics.

Why do you think this is at the top of Hacker News? Because people's common understanding of a title like:

 _New study shows that IQ is 11% genetic_

Is that at least 11% of their intelligence comes straight from their DNA, not that actually we could change 11% to 5% or 8% or 17% or most any number we want just by changing some laws or fortifying food or any number of environmental changes. The popularity of genetics research in popular media requires this bait-and-switch about what is actually being (mis) measured.

<comment><USER16> > I would ditch the idea of widespread genetic correlations on complex traits, > switch to a method that describes what genes (and their interactions) > actually do to the human body. For example, this specific gene does x to a > neurotransmitter which can interact with y or z to cause changes like this > or that. (And I know it's actually more complicated than just a straight > mapping of gene X -> trait Y.)

> With my method you can still say that no matter the environment gene x does > this thing which can interact with gene y or gene z or environmental factors > a, b, c in certain ways.

If you arguing that it is better to understand all of the casual pathways a gene influences a trait than to not understand that, then I strongly agree. If you are arguing that understanding which genes casually influence which complex traits is useless until we know all of the casual pathways that gene affect a trait then I'll have to disagree just as strongly.

> I think that proving intelligence is XX% genetic is basically meaningless > since we can't actually remove environmental effects and if I'm really > honest I think it's popularity in the culture is usually driven by spurious > motives.

First proving what % of intelligence is heritable is not the purpose of GWAS's. Twin studies are used for this, and they've found that intelligence is highly heritable.(far higher than 11%)

Ways that GWAS's are useful without removing environment effects.

\- Embryo selection

\- Tests to figure out risk for certain diseases for preventative purposes

\- Used to figure out what are important pathways in complex traits/disease. For instance if everyone with a certain defective enzyme has a 50% reduction in heart attacks this could lead to a drug candidate

> New study shows that IQ is 11% genetic > Is that at least 11% of their > intelligence comes straight from their DNA, not that actually we could > change 11% to 5% or 8% or 17% or most any number we want just by changing > some laws or fortifying food or any number of environmental changes. The > popularity of genetics research in popular media requires this bait-and- > switch about what is actually going on.

If you are arguing that some people(maybe even a lot) misunderstand the definition of genetic causation in a similar way that you previously did, then that might be true. But when any experts, or any study talks about genetic causation they are using the definition the rest of us are using on the this thread.

<comment><USER6> >First proving what % of intelligence is heritable is not the purpose of GWAS's.

And yet here we are on a thread about using a GWAS to prove what % of intelligence is heritable.

>If you are arguing that some people(maybe even a lot) misunderstand the definition of genetic causation in a similar way that you previously did, then that might be true.

My understanding isn't past tense. Here, allow me to illustrate why this makes no sense to me. Let's take what Gwern said above:

 _Are they genetic causation? Yes, because if you counterfactually changed the genes and nothing but the genes, you would get different outcomes. They are differences which make a difference._

But in my basketball example I could also say this:

 _Are they environmental causation? Yes, because if you counterfactually changed the environment and nothing but the environment, you would get different outcomes. They are differences which make a difference._

So we are going to take factors where both of those statements are true and _still_ assert that they are genetically causal? That's incorrect.

>But when any experts, or any study talks about genetic causation they are using the definition the rest of us are using on the this thread.

The 'rest of us' in this thread are definitely not using the same language you use. Let's take a look at just the post author's comments:

\---

>The most iron-clad part of showing direct gene->trait causation is the 4 sibling-comparison studies

except it's not iron-clad at all and it doesn't show direct gene->trait causation.

>where you see whether you can predict the difference between 2 siblings; as they have identical ancestries, families, locations etc, their environments are near-identical in the first place

Their environments are not near-identical.

>No, it doesn't. It merely tells us that with the data up to now, we've reached 11% (up from, incidentally, ~0.3% just 5 years ago). It provides a very loose lower bound on genes.

Except you're telling me that not only is a GWAS not useful for this purpose but that even this number shouldn't be treated as a 'lower bound' but as a potential identifier to further study the genes identified. It would seem that this

>> Observational studies of this type cannot show causality >They can: sibling comparisons.

Again, this quote appears to conflict with what you believe GWASs are capable of.

\---

Again, why are these falsehoods being spread around? Because of an (mis)understanding that 'genetic causation' as measured in the paper is not manipulatable by the environment. Except that it actually is. And a misunderstanding of what GWASs are good for.

>\- Embryo selection

No it's not. GWASs don't understand what they're measuring and label the environmental as genetic biasing their measurements. Embryo selection is done the old-fashioned way looking for specific genetic variants that we understand through the use of other types of studies.

>\- Tests to figure out risk for certain diseases for preventative purposes

Look at the use of GWASs in identifying schizophrenia. They typically fail to replicate. GWASs find genetic connections where none exist.

Take schizophrenia. It could be that we run a GWAS on schizophrenia and find some genetic similarities. Then we look at these similarities and find one of them is a cuddle gene. A cuddle gene what the heck? Well the cuddle gene makes you want to cuddle which makes their mothers more likely to own pets to cuddle which makes them more likely to own cats which increases their risk of exposure to toxoplasmosis which increases their of schizophrenia. Is that a genetic causation? Because the way genetic causation is _actually being used_ in this thread really doesn't lead to that conclusion. It leads to the idea that the identified genes make you a little more unstable in a certain way, not that we could totally eliminate that element of genetic causation by just educating mothers.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER17> In what sense is this a "discovery"?

Observational studies of this type cannot show causality, and the results have not been confirmed by independent methods.

<comment><AUTHOR> > Observational studies of this type cannot show causality

They can: sibling comparisons.

> the results have not been confirmed by independent methods.

They're tested out of sample many times (4 for sibling comparisons alone!), and you can find many other papers successfully using the IQ GWASes. Belsky et al 2018 just came out a few weeks ago, for a particularly good example testing OP's PGS. (The PGS has been out since like March, so several papers have used it.)

<comment><USER17> >> Observational studies of this type cannot show causality

>They can: sibling comparisons.

But then they get much lower concordance so they come up with various ad-hoc hypotheses to explain it, instead of acknowledging the negative result.

This is an hypothesis generating paper, using a highly suspect methods. Not a discovery.

</comment></comment></comment><comment><USER18> taking all of the associations into account and making a polygenic predictor score "does a lousy job predicting the outcome for any specific individual, but it can explain 11 percent of the population-wide variation in years of schooling."

only 11 percent. this tells us that the remainder is more likely to be attributable to environmental factors.

in other words, even with a view of the genes, we would need to encourage education more strenuously if we wanted to improve educational outcomes for society.

<comment><AUTHOR> > only 11 percent. this tells us that the remainder is more likely to be > attributable to environmental factors.

No, it doesn't. It merely tells us that with the data up to now, we've reached 11% (up from, incidentally, ~0.3% just 5 years ago). It provides a very loose lower bound on genes. Saying that is like saying, 'our software testing up to now has only revealed bugs in 11 places, therefore, every other line of code must be bug-free!' Even if you knew nothing about the actual heritability estimates, one would think that finding so much so quickly would make one suspect that there's a lot more to find...

<comment><USER18> there's always more to find, but consider that this model is leaving out the environment entirely and its predictive value is even less than you might think at first glance.

we know the environment impacts gene transcription, and we know that genetic determinism (as far as outcome phenotypes) is not a thing. likewise, there may be epigenetic factors at play which wouldn't be captured in a GWAS yet still might (or might not) have an impact on outcomes.

it isn't a "the other lines of code are bug free" situation so much as a "we expect there to be an abundance of bugs with undefined consequences outside of the subset which we have examined which most likely outweigh the stuff we found"

</comment></comment></comment><comment><USER19> From the discussion:

> However, theoretical projections that failed to consider heterogeneity of > effect sizes were optimistic Our and others' findings suggest that imperfect > genetic correlation across cohorts will be the norm for phenotypes, such as > educational attainment, that are environmentally contingent.

So it is somewhat worrying that genetics predict the clearly environmentally contingent like educational attainment better than "cognitive performance."

> Our results also highlight two caveats to the use of the polygenic scores in > research. First, our within-family analyses suggest that GWAS estimates may > overstate the causal effect sizes: if educational attainment-increasing > genotypes are associated with parental educational attainment-increasing > genotypes, which are in turn associated with rearing environments that > promote educational attainment, then failure to control for rearing > environment will bias GWAS estimates.

That is a very complicated way of saying that some genes express themselves by black skin, and these genes have a strong effect on funds available to the school.

<comment><AUTHOR> > So it is somewhat worrying that genetics predict the clearly environmentally > contingent like educational attainment better than "cognitive performance."

I don't agree that schools are any more 'clearly environmentally contingent' than intelligence, especially with the mandatory public schooling applicable to all of the cohorts used.

> That is a very complicated way of saying that some genes express themselves > by black skin

That's a remarkable assertion, considering that this dataset is made of 100% white people, works on white siblings like Swedish twins, and the PGS is still predictive when applied within a separate African-American dataset.

<comment><USER20> It would be remarkable if there was not some predictive value when applied to AA datasets given the level of European ancestory in the AA population is around 30%.

<comment><AUTHOR> (Closer to 20%, and the attenuation of 85% is about what you would expect; we already know that that's the usual level of decrease when you test PGSes in AAs, and it goes to near-zero in African populations. It looks like it's not an issue of entirely different causal variants or different environments, but mostly LD tagging decay.)

<comment><USER20> The estimated range is 16.7% to 29% according to the fount of all wisdom wikipedia [0].

0\. [https://en.wikipedia.org/wiki/African_Americans](https://en.wikipedia.org/wiki/African_Americans)</comment></comment></comment></comment></comment>


Ask HN: What is your favorite work music? - <AUTHOR>

hey guys,
I'm going on a 72 hour coding binge tomorrow, and I'm just preparing some music for the kickass journey. Can you guys share some of your favorite work songs with me?
To be fair, I'll share this remix with you. I personally think it's kickass, and I'll give you my first born child if you think this song sucks =p  [Taio Cruz - Break Your Heart Remix ] (http://fasho.me/music/49) <comment><USER2> Anything I've listened to enough that the words no longer register.

I owe my master's degree to Pink Floyd.

</comment><comment><USER3> 72 hours? Sheesh… I miss being young(er). If I pulled that today my playlist would be mostly funeral music, but I wouldn't be the one listening!

My favorite work music is anything that I've heard enough that my brain is resonating with it instead of listening for details. My musical staples tend to be Dead Can Dance, the later albums by Savatage, Penguin Cafe Orchestra, Tom Petty, just about everything techno, and lots of Bob Marley. Here lately I've been spinning parts of Trent Reznor's "The Social Network" soundtrack a bit more than anything else.

I can listen and work to just about anything, so sometimes I just let my entire collection go on random.

<comment><USER4> > My favorite work music is anything that I've heard enough that my brain is > resonating with it instead of listening for details.

Same for me. I have a Grooveshark "work" playlist that I've been adding songs to for the last 2 years or so. Here it is: <http://grooveshark.com/#/playlist/Work/57992001>

I find music that has no lyrics and has a slower tempo to be the least distracting when working. This includes genres such as post-rock, trip-hop, launge, ambiental.

</comment></comment><comment><USER5> >72 hour coding binge

Just listen to some prog epics. Genesis, King Crimson, Rush... you'll get through three, maybe four songs in that timespan.

</comment><comment><USER6> Daft Punk, Joe Satriani, Dead Mau5, Lange's Intercity podcast, Thievery Corporation

<comment><USER7> Thank you for deadmou5!

<comment><USER6> Glad to share! Cosmic Gate is also great.

</comment></comment></comment><comment><USER7> I often listen to Sneaker Pimps, Four tet, Massive Attack, Thievery Corporation, Röyksopp and Air... I like to mix it up as well (aka. shuffle)!

</comment><comment><USER8> Fuck Buttons (Tarot Sport album) Bone Thugs-N-Harmony (All albums)

</comment><comment><USER9> Uniko by Kronos Quartet, Kimmo Pohjonen & Samuli Kosminen

</comment><comment><USER10> Flying Lotus, Daft Punk, Mr Oizo, Com Truise.

</comment><comment><USER11> Try Steve Reich's "Music for 18 Musicians."</comment>


NPR Crowdsources ID of lobbyists at congressional health care hearing - <AUTHOR> http://www.npr.org/news/specials/2009/hearing-pano/

<comment><USER2> This is a great idea. I bet more stuff like this would have a significant impact-- like the amateur tail-number watchers at airports who helped uncover the secret "rendition" flights.

<comment><USER3> That sounds pretty cool. Got a link?

<comment><USER4> <http://www.guardian.co.uk/uk/2005/dec/10/usa.terrorism1>

</comment></comment></comment><comment><USER5> I heard about that on NPR this morning, I'm all for it. My only concern, do they have any privacy rights against this? Or does the gov't building imply same protections as being in public?

<comment><USER6> They are at a public meeting, in a government building, exercising their 1st amendment rights to petition their government. They have absolutely no privacy rights here. They are not compelled to identify themselves but they have no protection against others identifying them nor are they likely to receive such protection given the legitimate concerns the public has over corruption (the Lobbying Disclosure Act [2 USC 1605] does permit lobbyists to perform routine information discovery, such as attending this event, without requiring disclosure but the fact that they are not required to report their attendance would probably not be enough for a court to find any expectation of privacy for anyone attending such an event.)

<comment><USER7> That's true only until they lobby congress and get a national lobbyist protection act passed.

</comment></comment></comment><comment><USER8> They should have an image map over the faces and you can click on the face and then it goes to a form where you can add details about the person.

<comment><USER9> I love this idea; how does the sponsoring site ensure the quality of the user supplied data?

<comment><USER8> NPR will have to use other sources to verify the information people on the internet submit. They can build it so that the submitted information isn't publicly available until it's verified.

Perhaps if they get multiple people suggesting that a particular person is _this_ person, then that may be a good indicator.

</comment></comment></comment><comment><USER10> Could someone put into proper context why this is crowdsourcing? It's an interesting application but I don't think it has anything to do with crowdsourcing.

<comment><USER11> They ask people to email them if they know anyone in the pictures.</comment></comment>


I Got More Data, My Model Is More Refined, but My Estimator Is Getting Worse [pdf] - <AUTHOR> https://statmodeling.stat.columbia.edu/wp-content/uploads/2012/07/timeseries06272012.pdf <comment><AUTHOR> I am the OP. For what it's worth, I barely understand the paper, way too technical for me, so let me give some context as to why I posted this.

It was refered to in a course I've just started self learning, Berkeley's Principles and Techniques of Data Science (aka Data 100), and the lecturer wanted to highlight the crucial importance of data acquisition in the Data Science lifecycle, sampling in particular, given that mistakes at this stage will propagate to the later stages. My understanding of his example (~1h10 into [https://youtu.be/JtwBwogRZkI](https://youtu.be/JtwBwogRZkI)) is that a 400 sized probability sample (a small random sample from the whole population) is often better than a millions sized administrative sample (of the kind you can download from gov sites). The reason is that an arbitrary sample (as opposed to a random one) is very likely to be biased, and, if large enough, a confidence interval (which actually doesn't really make sense except for probability samples) will be so narrow that, because of the bias, it will actually rarely, if ever, include the true value we are trying to estimate. On the other hand, the small, random sample will be very likely to include the true value in its (wider) confidence interval.

The paper seems to elaborate on this, which I find interesting, and I was hoping for someone to explain it [EDIT: the paper, that is, I think I understand the lecture].

<comment><USER3> The paper is talking about a closely related, but rather different topic than the video (relying on your summary, I did not watch it).

The paper is mostly concerned with _model misspecification_ , in the sense that your model is not actually capable of describing reality (no matter what parameters you set it to). The first example uses the OLS model, e.g.

Y = βX + ε

where the implicit assumption is that your observations (Y) are purely a linear function of your input(s) (X) plus a _independent, equal variance, normally distributed_ error.

Let's say your data really does fit a straight line, but the later data points you get are really high variance. Intuitively, including the later data points will make your estimated parameters more noisy than if we had simply thrown them out.

So in this sense, 'more data is worse', because your model assumes all the data points are the same, but the later points you collected are actually worse (noisier).

There are other more obvious ways this happen - for example, if you assume that wealth is normally distributed, then try to estimate the distribution's parameters, then any small sample + any billionaire will get you nonsense — your model will be very intolerant of sampling noise compared to a better model.

The video's observation only deals with data: you might prefer quality over quantity.

E.g. compare a wider cone centered at zero (the true average) vs. a narrow cone that's centered away from zero - depending on what we care about, we may prefer some (hopefully uncorrelated) noise in our model rather than learning some systematic untruth.

So these are two different ways more data can be counterproductive: either your model is _biased_ in a way that more data can be hurtful, or your sampling procedure for larger sets is increasingly biased.

Note that older stats/econometric techniques are more vulnerable to model misspecification because they have many fewer parameters than, say, a DNN; conversely, more modern ML techniques are able to closely replicate input data. This helps explain the difference in focus.

<comment><USER4> > Let's say your data really does fit a straight line, but the later data > points you get are really high variance. Intuitively, including the later > data points will make your estimated parameters more noisy than if we had > simply thrown them out.

> So in this sense, 'more data is worse', because your model assumes all the > data points are the same, but the later points you collected are actually > worse (noisier).

The first example in the paper makes it less-than-obvious by writing in formula form that the error noise variance increases with time, but what's described is essentially this picture: [https://en.wikipedia.org/wiki/Heteroscedasticity#/media/File...](https://en.wikipedia.org/wiki/Heteroscedasticity#/media/File:Hsked_residual_compare.svg) So it's intuitively easy to grok that if the more recent variances are bad enough it's probably better not to use them at all (or to properly downweight them like with MLE or GLS).

The described result is a bit more subtle though. The classical results are that with heteroskedasticity, OLS is not an _efficient_ estimator [1] (you could do better with for example GLS/WLS), but it's still _unbiased_ [2] (the estimated beta's are centered around the true beta's) and _consistent_ [3] (the probability limit of the OLS beta is the true beta by the Law of Large Numbers). These are what the paper calls "robustness" of OLS in the first example. The consistency of OLS even with heteroskedastic errors is what makes it surprising that adding more (true!) observations can make your estimates worse. Intuitively most people expect consistent estimators to get better with more observations.

[1] [https://en.wikipedia.org/wiki/Efficient_estimator](https://en.wikipedia.org/wiki/Efficient_estimator)

[2] [https://en.wikipedia.org/wiki/Bias_of_an_estimator](https://en.wikipedia.org/wiki/Bias_of_an_estimator)

[3] [https://en.wikipedia.org/wiki/Consistent_estimator](https://en.wikipedia.org/wiki/Consistent_estimator)

</comment></comment></comment><comment><USER5> It is well known among practitioners that not all models have equal learning capacity and that additional data is wasted on those with lower capacity - <USER2> you plot a performance metric vs dataset size, you can see it just saturates and stops increasing. The point where it saturates is a proxy for the model's learning capacity.

One of the pros of very big models like as deep networks was that they never seem to saturate in this manner - more data always gave better results, because there are ways to increase the model's learning capacity.

We've recently learned that deep nets work in deep overfitting mode, beyond the usual U-shaped regime of bias-variance tradeoff that can be seen in the empirical risk curve. See this post for a good overview:

[https://lilianweng.github.io/lil-log/2019/03/14/are-deep- neu...](https://lilianweng.github.io/lil-log/2019/03/14/are-deep-neural- networks-dramatically-overfitted.html#modern-risk-curve-for-deep-learning)

It was a surprise to me that adding more data could lower a performance metric tough - that's counter-intuitive.

<comment><USER6> > It is well known among practitioners that not all models have equal learning > capacity and that additional data is wasted on those with lower capacity

Not just practitioners, IIRC there's a whole bunch of theory under the handle of Vapnik-Chervonenkis dimension.

</comment></comment><comment><USER7> (2013) - edit: yeah, not critizing OP, it's just useful metadata

<comment><AUTHOR> I would indeed have liked to append the date to the title, but Too Long

<comment><USER8> The link is a preprint. Here is the published version: [https://www.tandfonline.com/doi/full/10.1080/07474938.2013.8...](https://www.tandfonline.com/doi/full/10.1080/07474938.2013.808567).

</comment></comment></comment><comment><USER9> Huh, the bit about just plugging in the real values for variables being a bad idea is very interesting. Pretty sure I've done that in the past, without really thinking too much about it.

Does anyone know of books that cover stuff like that? There's plenty of books explaining Simpson's paradox to laymen, but I want something that explains common statistical pitfalls for people with already some education in the area.

</comment><comment><USER10> If adding data AND variables to your model is making it less accurate, this is probably good evidence that your first model was cheating.</comment>


Nick Carr: Is Office the new Netscape? - <AUTHOR> http://www.roughtype.com/archives/2008/05/one_of_the_corn.php <comment><USER2> I don't think Office will become the next Netscape at least not in the revenue side of things. Microsoft can still build its own online version of office, they have the name and customer base, I personally don't think Google can do them any harm there.

I will make a point here since this article refers to one great tool death.

Google should get busy building product they can sell and expand their revenue base. The rule of all trends is that, time and evolution always takes over. Microsoft remains powerful for all these years because the revenue source is extremely diversified. Once Microsoft takes over Yahoo they will have the strength to give Google a real hard time on their turf; mo matter how cool Google is, they make money because people LIKE them. What happen when people stop liking them? Remember all great Tools, Musicians, Politicians fade over time. It's just a matter of time before people like another search engine.

I think Google should look at apple with iTunes, iPod, iPhone a range of products that actually brings in money.</comment>


How Ford Avoided the Meltdown that Hit GM, Chrysler - <AUTHOR> http://www.cnbc.com/id/30134908 <comment><USER2> Summary: They got lucky.

They were doing so bad in late-2006 (in an economic boom) that they were forced to leverage the entire company to raise capital for restructuring. So they had $23 billion in cash when the market for automobiles collapsed 18 months later. The only reason they didn't get government bailout money was because they'd just had a private bailout.

If you recall, Ford went to Congress for a bailout just like the others. They didn't let the fact that they didn't need the money stop them from lining up at the trough alongside GM and Chrysler. One can only assume their sole reason for dropping out was realizing the billions on their balance sheet would disqualify them from receiving funds.

<comment><USER3> I don't know if it's that simple. Ford didn't ask for the bailout after all, probably because they realized how much leverage it would give the government over their operations. You know how startup founders are wary about giving VC's too much control over their business? I'd imagine automakers (especially Ford, which is still largely a family business) are just as wary about giving the government too much control over their business. And, unlike GM and Chrysler, Ford has the good fortune to be a good ways away from bankruptcy (though from a business standpoint, bankruptcy would still be better than nationalization).

</comment></comment><comment><USER4> Clearly the Mr. Mullaly is a 9th degree Mason and had advance notice that the economy was being taken into recession by the Pentaverate.

EDIT: So humor isn't a valuable contribution to the discussion? Now I'm sad. :(</comment>


The Startup CTO's Guide to Ops (1 of 3): Guiding Principles - <AUTHOR> https://medium.com/@cgroom/the-startup-ctos-guide-to-ops-1-of-3-guiding-principles-2607e21d9f89 <comment><USER2> > While I love reading Hacker News posts about the amazing infrastructure at > successful companies, I worry that these discussions may encourage an over- > emphasis on perfection and scale.

God yes! I read all these blog posts about people's amazing automated deployment systems that use 17 different technologies where they have F on top of E on top of D on top of C on top of B on top of A on top of Kubernetes on top of Docker on top of AWS, and it's like, A.) When did you actually do any, you know, "work" \- the stuff that you can charge clients for and make money? And B.) What's going to happen when some company or someone releases a patch or security update for any of the 7829271 applications in the stack that breaks everything?

You know what we run? Linux. On AWS. Using chef - although every community cookbook I ever look at makes me less sure that that was the right call (most community cookbooks make me think suicide might be the right call).</comment>


4th State of Water - <AUTHOR> http://news.discovery.com/earth/new-water-super-cool-below-zero-111111.html#mkcpgn=fbdsc6 <comment><USER2> Just to clarify, this isn't a fourth state, this just seems to be a different liquid phase. Water has a hugely complicated phase diagram -- there's fifteen phases of water ice known experimentally and a few more predicted in theory.

I haven't been able to find the original paper (damn you discovery.com) so let me know if you find it.

<comment><USER3> Couldn't find it either, but after looking around, it appears to me that the research discussed in the posted article is not so ground-breaking as the writer suggests (of course, it hardly ever is, y'know). Papers about liquid- liquid phase transitions in water go back a number of years.

For example, here is a paper from 1997 that speculates about such a phase transition.

<http://iopscience.iop.org/0295-5075/42/2/161>

And here is a recent paper about such a phase transition found in a computer model (which seems to be what the posted article is talking about). This paper is not by Kumar & Stanley, but it cites a number of papers of theirs.

<http://arxiv.org/abs/1105.5295>

</comment></comment><comment><USER4> Is this just an addition to the already-known 15 types of ice? <http://en.wikipedia.org/wiki/Ice#Phases>

<comment><USER2> Yes, except it's a new liquid phase rather than a new solid one.

</comment></comment><comment><USER5> As someone who did computational modeling of water, this strikes me as kind of odd. The "hello world" of computational molecular dynamics is simulating water in different phases, phase changes, etc. (including the different ices). I'm interested to hear more, but incredulous.

<comment><USER6> Can this kind of molecular dynamics simulation by done on a simple desktop machine? What system and software do you recommend if I want to experiment with simulation, starting with this "hello world" project? Do you recommend starting from scratch or using some specialized framework? (For context, I'm a professional programmer and mathematician by training.)

</comment></comment><comment><USER7> Why is water thought to have only 3 states? Is it impossible to transform water into a plasma state?

<comment><USER8> Water is a molecule, not an atom. There are no molecules in plasma since the electrons have all wandered off. So if you heat water gas enough, it becomes a plasma of hydrogen and oxygen.

<comment><USER7> You are almost 100% right. There are almost never any molecules in plasma: <http://physics.aps.org/story/v22/st17>

</comment></comment></comment><comment><USER9> Celsius please, or at least have both.This is supposed to be science not a localised weather report.

<comment><USER10> Better yet, Kelvin. I cringe when science-y things don't use that.

<comment><USER11> Perhaps, but given that we are discussing water, using Celsius seems a particularly appropriate choice.

</comment></comment></comment><comment><USER12> Looks like people have known about many phases of water for ages: <http://www.lsbu.ac.uk/water/phase.html>

Oddly, it appears that Ice XIII and Ice X exist, but not Ice IX.

<comment><USER2> Look closer, Ice IX is on that phase diagram. Temperatures below 200K, and a very narrow pressure range just above I and XI.

<comment><USER13> > very narrow pressure range

On a logarithmic scale.

</comment></comment></comment><comment><USER14> This article is EMPTY of details, and for someone actually interested in this new phase of water, I am severly annoyed.

<comment><USER15> I genuinely hope that "someone actually interested in this new phase of water" is on your business card, resume, or at least your Twitter profile.

</comment></comment><comment><USER16> 4th state of water is a slushi... good observation scientists. good observation.

</comment><comment><USER17> From the looks of the article, it seems like it's meant for people who are interested in popular science, but then the author uses degrees Fahrenheit everywhere.</comment>


Middle Earth Map Style - <AUTHOR> https://adventuresinmapping.com/2018/09/10/middle-earth-map-style/ <comment><USER2> Just as an FYI: if anyone here is near to or planning a trip to Oxford (UK), there is currently a free Tolkien exhibition that includes many original maps (and working drafts) at the Weston Library.

<comment><USER3> I saw it last month -- it's quite small (one room), but exquisitely designed, with numerous original maps, drawings and manuscripts, and interactive displays (the relief map is most beautiful).

</comment></comment><comment><USER4> These are beautiful I cannot help but nitpick the mountains in California. Those mountains bordering NV and CA are majestic, and their tallest peaks are not at the delta near Sacramento, but south of there, closer to Visalia and Fresno. The Blueridge and Appalchian mountains appear majestic in these maps, but these do not really compare in height (e.g. tallest peak 4,000 ft compared to 14,505 in Sierra Nevada), although they do have a greater length. [https://www.ducksters.com/geography/us_states/us_mountain_ra...](https://www.ducksters.com/geography/us_states/us_mountain_ranges.php)

<comment><USER5> I cannot help but nitpick either. You're off by over 50% on your eastern mountains:

Mount Mitchell, NC: 6684 feet [1]

Mount Washington, NH: 6288 feet [2]

These eastern Mountains are indeed majestic, towering overhead when you look to ascend them. Mt Washington would stand in perfectly for Caradhras.

[1] [https://en.wikipedia.org/wiki/Mount_Mitchell](https://en.wikipedia.org/wiki/Mount_Mitchell)

[2] [https://en.wikipedia.org/wiki/Mount_Washington_(New_Hampshir...](https://en.wikipedia.org/wiki/Mount_Washington_\(New_Hampshire\))

<comment><USER4> good catch, but add those two peaks together and they are still shorter than Mount Whitney.

[https://en.wikipedia.org/wiki/Mount_Whitney](https://en.wikipedia.org/wiki/Mount_Whitney)

[https://en.wikipedia.org/wiki/Sierra_Nevada_(U.S.)](https://en.wikipedia.org/wiki/Sierra_Nevada_\(U.S.\))

</comment></comment></comment><comment><USER6> Print them out poster-sized and sell them on Etsy. I'd buy one in a heartbeat.

<comment><USER7> I have an old HP pen plotter, I may give it a go.

</comment></comment><comment><USER8> Anyone want to stand up an OSM tileserver for this?

<comment><USER9> That was my first thought too. If I get some spare time this weekend I'll take a swing at it. I'll comment back here if I do it, or you can email me.

<comment><USER10> Would be interested in helping. If you need any. Otherwise I may do one on my own too.

</comment></comment></comment><comment><USER11> If you like cool (live) maps, I've always enjoyed the watercolor setting on Stamen Maps[1]. They have others, too.

[1] [http://maps.stamen.com/#toner/12/37.7706/-122.3782](http://maps.stamen.com/#toner/12/37.7706/-122.3782)

</comment><comment><USER12> Cracks me up to see the mostly-treeless Columbia River basin rendered as though it were Mirkwood, but I guess that's the best fit for the style.

<comment><USER13> Same for southern Utah, Arizona, and New Mexico

</comment></comment><comment><USER14> Wow! I've seen artists who hand draw in Tolkien's style ([https://www.middleearthsmaps.com/](https://www.middleearthsmaps.com/)) but I never would have thought it could be done so well automatically!

<comment><USER15> Those are a poor copy of the style. They're way more simplistic and less detailed.

</comment></comment><comment><USER16> Nitpick: the map he shows was drawn by Christopher Tolkien, not Pauline Baynes.

</comment><comment><USER17> This looks super interesting. I was expecting a Github link with a Mapbox Stlyesheet, if anyone has one please send it my way.

<comment><USER3> From the article: "If you are interested in trying out making digital Middle Earths, here is an ArcGIS Pro style file with all the doodads you'll need. If you don't run that, then here is a zip file with all of the textures and graphics that you can use to symbolize your layers." Links are included there.

I suppose Mapbox could be created from that?

</comment></comment><comment><USER18> here's a link to some background and resource files: [https://www.esri.com/arcgis-blog/products/arcgis- pro/mapping...](https://www.esri.com/arcgis-blog/products/arcgis- pro/mapping/we-wants-it/)

</comment><comment><USER19> fantastic. great writing style on this page also, not at all like too few bits spread over too much whitespace</comment>


Millions of stolen Last.fm passwords have been decrypted. These are the top 50 - <AUTHOR> http://qz.com/772977/millions-of-stolen-last-fm-passwords-have-been-decrypted-these-are-the-top-50/ <comment><USER2> I wonder how much passwords like "lastfm", "music" and "abcdefg123" are an indication of people just being bad with security, as opposed to people just not giving a damn... Amid the overflow of trivial sites requiring more and more login details, is there really a value in having strong passwords? Personally, I mostly don't care; I have strong passwords for important sites (banks, online stores that have my credit card details), but for the rest of logins, I use pretty trivial passwords - if someone "steals" my account, I'll just create another one!

<comment><USER3> But what if they post snarky comments as you on HN?

<comment><USER4> That would be disastrous.

<comment><USER5> na, no problem. 'tomp2' is still available

</comment></comment></comment></comment><comment><USER6> I feel like this is misleading without the % share of the total. These top 50 represent 2.7% of the total. OK, that sounds bad, but I'm not really sure how to interpret it.

Even if we get better at this, the top passwords will still be these, they'll just be lower as a % of the total. So until we have 100% unique passwords, this story will always be there, which just seems lazy.

</comment><comment><USER7> If you follow the link in the article to leakedsource.com, it says:

        Passwords were stored using unsalted MD5 hashing.

It's 2016... why is this still happening?

<comment><USER8> Presumably because last.fm was created in 2002 and they never changed it.

</comment></comment><comment><USER9> This has been my go-to for passwords for awhile. Top 100, 500, 1000, etc all the way to millions. [https://github.com/danielmiessler/SecLists/tree/master/Passw...](https://github.com/danielmiessler/SecLists/tree/master/Passwords)

</comment><comment><USER10> I do hobby research into password guessing entropy and I'd love to get ahold of these lists of in-the-wild passwords. Is there any good source for such things? I don't need usernames, just "real" passwords (with frequency data if I can get it).

<comment><USER8> [http://academictorrents.com/details/fd62cc1d79f595cbe1de6356...](http://academictorrents.com/details/fd62cc1d79f595cbe1de6356fb13c2165994e469)

<comment><USER10> Whoa! This is a great link and thank you but thank you even more for making me aware of academictorrents.com.

I may never see the sunlight again.</comment></comment></comment>


Mapping the World of Amiga Samples - <AUTHOR> https://chipflip.wordpress.com/2019/10/28/mapping-the-world-of-amiga-samples/ <comment><USER2> Mandatory warning to developers thinking about playimng Amiga music or emulating the Amiga's audio.

[https://bel.fi/alankila/modguide/interpolate.txt](https://bel.fi/alankila/modguide/interpolate.txt)

<comment><USER3> That is an amazing document.

</comment></comment><comment><USER4> I hope that some day the 90s demo scene and its music gets a documentary film made about it before all the pioneers pass away. It's such a special cultural moment. The early steps into the distributed artistic movements we take for granted today.

<comment><USER5> There's an initiative by the "European Federation of Game Archives, Museums and Preservation Projects" to get the Demo Scene on the "UNESCO Cultural Heritage of Humanity" list:

[https://efgamp.eu/2019/05/20/efgamp-launches-campaign-to- pro...](https://efgamp.eu/2019/05/20/efgamp-launches-campaign-to-promote-the- demoscene-as-unesco-cultural-heritage-of-humanity/)

</comment></comment><comment><USER6> I feel the nostalgia kicks in. I found many of my old samples and mods here, many forgotten. And had no idea that the samples was used by so many. Guess floppy swapping was more efficient than it appeared to be :)

</comment><comment><USER7> gdi another wave of nostalgia :o) The Amiga showed what amazing things can happen when you combine a community with great tech and basically get all the lawyers to piss off =)

</comment><comment><USER8> I wish the site had a way for us to listen to the samples.

<comment><USER9> It's mentioned in the "Future" section.

</comment></comment><comment><USER10> I can't wait to actually use all of these samples in Renoise.</comment>


Trump says he will look into a pardon for Edward Snowden - <AUTHOR> https://www.washingtonpost.com/elections/2020/08/15/trump-biden-live-updates/ <comment><USER2> I don't care who pardons him, I really hope that it will happen. Edward Snowden deserves to be treated as an American hero, not as a pariah.

</comment><comment><USER3> >Trump...did not definitively say whether he thinks Snowden is a traitor.

He has repeatedly called for his execution as a spy and a traitor. If I were Snowden I would be extremely wary of such a pardon; it could simply be a ruse to get him on a plane home. It's not as if the security services are a stranger to extra-judicial action.

More likely it's part of some larger piece of realpolitik involving Russia and/or the security services. I think one thing we can be reasonably sure of is that Trump hasn't suddenly had a change of heart over the constitutionality of whistleblowing.

As an aside - I consider "opinion on Edward Snowden" to be the gold standard litmus test on the integrity of a politician. It's surprisingly bi-partisan.</comment>


An interview with Barbara Liskov - <AUTHOR> https://www.quantamagazine.org/barbara-liskov-is-the-architect-of-modern-algorithms-20191120/ <comment><USER2> > In my version of computational thinking, I imagine an abstract machine with > just the data types and operations that I want. If this machine existed, > then I could write the program I want. But it doesn't. Instead I have > introduced a bunch of subproblems — the data types and operations — and I > need to figure out how to implement them. I do this over and over until I'm > working with a real machine or a real programming language. That's the art > of design.

Love the way she put this. Good software design is a process of coming up with layers of vocabularies that most expressively describe what needs to be described, until you hit the bedrock of something that already exists that you can hand the rest off to.

<comment><USER3> Which is why I prefer working with a strongly typed functional language. Domain Driven Development (well, my version of it) is the first step. Get all the types right. The necessary operations should be obvious by this point. Code them up! Write unit tests as you go along. You're likely to do some refactoring as your operations materialize, and unit tests are going to help you stay safe.

</comment></comment><comment><USER4> If you're interested in computer history, here's a transcript of a long, in- depth interview with Barbara Liskov in 2016 about how she got into computing, the CLU project, and many other aspects of her career. Quite inspiring!

[https://amturing.acm.org/pdf/LiskovTuringTranscript.pdf](https://amturing.acm.org/pdf/LiskovTuringTranscript.pdf)

</comment><comment><USER5> Such a waste of a good title. Should have been:

"An interview with Barbara Liskov. Accept no substitutes."

<comment><USER6> It would have been a great title.

But the pedants like me would have crept up to say "but Liskov tells us we _can_ accept substitutes".

</comment></comment><comment><USER7> You know, I've heard about the Liskov substitution principle but have never really equated it with anyone (especially living). Its nice to see that some pioneers in our field is still alive and well.

</comment><comment><USER8> "Barbara Liskov invented the architecture that underlies modern programs."

Basic explanation of this claim: she invented CLU, the first language without goto statements.

<comment><USER9> I don't think your statement is intended to be dismissive, but it is a bit... sparse.

Related to CLU: It introduced a lot of concepts together in one language. "Key contributions include abstract data types,[6] call-by-sharing, iterators, multiple return values (a form of parallel assignment), type-safe parameterized types, and type-safe variant types." [1]

Liskov also has other contributions[2]. Per the article, she says herself in the early days there were a lot of big problems to solve, so it was somewhat easier to make a "big" contribution.

It's interesting to look at where many modern concepts came from. It would be interesting to dig in to these a little more. Time is the one resource that's hard to come by though.

[1] [https://en.wikipedia.org/wiki/CLU_(programming_language)](https://en.wikipedia.org/wiki/CLU_\(programming_language\)) [2] [https://en.wikipedia.org/wiki/Barbara_Liskov](https://en.wikipedia.org/wiki/Barbara_Liskov)

<comment><USER10> Reading about the history of so many major design decisions coming from one person also makes me wonder how different modern languages would be without the contributions of that person. Would we be more or less where we are, but a couple years behind? Would we have gone a different route with some of these decisions?

</comment></comment></comment><comment><USER11> The article doesn't seem to mention [https://en.wikipedia.org/wiki/Liskov_substitution_principle](https://en.wikipedia.org/wiki/Liskov_substitution_principle) at all, which I think is her greatest accomplishment, formulating it.

</comment><comment><USER12> pg (amongst others) talked a lot about this. Middle layer language design, eDSL, language oriented programming ala racket.

Fun to see liskov mention that when so few, if at all, of the oo World thought this way.

<comment><USER13> Liskov is one of the few programmers who truly understood OO-programming waaaaayyyy back in the 80s.

[https://dl.acm.org/citation.cfm?doid=62139.62141](https://dl.acm.org/citation.cfm?doid=62139.62141)

And later developed into the Liskov Substitution Principle (`94): [https://www.cs.cmu.edu/~wing/publications/LiskovWing94.pdf](https://www.cs.cmu.edu/~wing/publications/LiskovWing94.pdf)

Over the next 20 years, people would be writing about bad examples like "Class Car extends class transportation", but the fundamentals of good OO-programming were laid out by Liskov pretty early on...

<comment><USER12> The thing with pioneers is that most of the followers in the decades after have no clue about what they meant.

</comment></comment></comment><comment><USER14> Back in the day, I treasured my copy of Liskov and Guttag's "Program Development in Java". I enjoyed its discussion of a disciplined programming style centered around maintaining invariants in your data structures. It was also the only book I've seen which talked about modeling the state of your design using a graphical notation inspired from Alloy. I wanted more exposure to this material so I even purchased a copy of their earlier work (which is also excellent). Point being, I have a lot of respect for her.

Like some of the other comments, I loved how she expresses her thoughts about computing in this article, e.g: "In my version of computational thinking, I imagine an abstract machine with just the data types and operations that I want..." (reminds me of Norvig/Graham talking about Lisp). But I don't like this article overall because I think it takes cheap shots at men in computing.

"I'm worried about the divorced couple in which the husband publishes slander about the wife, including information about where she lives." When I first encountered this statement, I thought it was being unnecessarily specific but didn't think anything else of it. After finishing the rest of the article, I think it was a purposeful choice, and in the current environment, I don't think it's a fair statement to make. A male making the opposite statement would likely be called out, so I'm calling her out here.

"At Berkeley, I was one of one or two women in classes of 100. No one ever said, "Gee, you're doing well, why don't you work with me?" I didn't know such things went on. I went to graduate school at Stanford. When I graduated, nobody talked to me about jobs. I did notice that male colleagues, like Raj Reddy, who was a friend of mine, were recruited for academic positions. Nobody recruited me." She didn't say this explicitly, but my interpretation is that she's using this as evidence of a gender bias in recruiting. Clearly, gender bias is one possibility, but what about giving people the benefit of the doubt? There are other more charitable explanations (somebody who understands conditional probability better than me could comment on this situation from a probabilistic perspective).

"Back then, advisers placed graduates through deals with departments around the country.

Yes, but nobody made deals for me. In the '90s, I went back to Stanford for a department celebration. A panel of the old professors, without knowing what they were doing, described the old boy network. They said, "Oh, my friend over there told me that I've got this nice young guy you should hire." It was just how it was. They were clueless. They talked about a young woman who did so well because she married a professor! Clueless. Another colleague had a pinup in his office. I asked him, "What's that pinup in your office?" Clueless." I love her directness here, but on the other hand her choice of words suggests to me that she has a chip on her shoulder, something I wasn't expecting from somebody so accomplished!

This reminds me of how greatly our beliefs affect our perception of the world, like in the beginning of a relationship when everything seems perfect, but after a fight, suddenly even the smallest things about your partner start to annoy you. The quirkyness of their laugh, or the way they chew their food – how did I miss how annoying that was before? But in reality they didn't change at all – only my perceptions did, because I no longer liked them.

"Even so, is it correct that there were approximately 1,000 faculty members when you started at MIT, only 10 of whom were women?

That was my recollection.

So there was progress, but …" I don't like the use of the word "progress" here, but I realize that is potentially another highly flammatory discussion. Here, it's just more evidence that this discussion is politically charged.

"My sense is that all scientific fields have failed to recognize some foundational contributions by women.

In the 10 years before I was head of computer science at MIT, the department identified only one woman worth hiring. When I was the head [from 2001 to 2004], I hired seven women. We didn't scrape the bottom of the barrel. All three junior women I hired are outstanding. There was a long period of time where women were not considered at all." Is that last sentence meant to be taken literally? I would be surprised if it were true. I'm ignorant of the goings-on at these levels, but I have faith that the majority of people involved in these decisions (around the time that she's referring to) were not sexist (the mere fact that she became the department head is evidence of this). I would also be interested in more details about why she was successful in hiring women compared to the previous head.

The big tech companies are making an effort to hire more women. One way they do this is by recruiting heavily at events where there are a preponderance of women. Somehow, this is seen as not being discriminatory, but I disagree. It's like there's two separate lines of applicants, and by choosing to focus on one, the people in the other line are hurt (and to be clear, my position is that I'd rather gender not be part of the equation at all, at least for tech jobs).

I could go on about the article, but I think I've made my basic point.

I also realize that I'm subject to the same biases that I allude to above. I too have a chip on my shoulder because I believe I was unfairly accused of being a sexist at work. As a result, I've become highly sensitive around this topic (e.g. I followed the James Damore events very closely) and I too view things through an (even more) distorted lens.

<comment><USER15> you are in fact the one who sounds like you have a chip on your shoulder.

<comment><USER16> OP says as much, which I respect, even if I don't share his position. It's just a rough subject these days.

</comment></comment></comment><comment><USER17> Liskov's work on programming languages is (of course) well known. She then switched research topic to distributed systems and among other things developed the "Practical Byzantine Fault Tolerance" (PBFT) algorithm.

I thought this was interesting, because like 10 years ago I had never really heard of her distributed systems work, and kindof thought of Liskov as a one- hit-wonder. But now in the last few years PBFT is suddenly pretty hyped, because it's a key part of a bunch of new blockchains, so maybe she was far ahead of the curve...

<comment><USER18> She's done both for a long time. She came up with the replication protocol that Lamport termed Paxos a year before Lamport did (she called it Viewstamped Replication). That was 1988.

</comment></comment><comment><USER19> Meta: Until I clicked this link, I had no idea who Barbara Liskov is.

Perhaps re-title this to "An interview with Barbara Liskov, The Architect of Modern Algorithms"

<comment><USER20> Not every title should explain itself. On HN, it's good for readers to work a little. [https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...](https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=by%3Adang%20%22work%20a%20little%22&sort=byDate&type=comment)

Among other reasons, it interrupts the reflexive circuits and gives time for the reflective ones to kick in. [https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...](https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20reflexive%20reflect&sort=byDate&type=comment)

</comment></comment><comment><USER21> Poor journalism and writing ruined an otherwise thoughtful piece of influential computer scientist.

The root cause probably is that the author does not understand the ideas well enough, so he/she is urged to overblown the implications...

<comment><USER2> Yeah, "You came of age professionally during the development of artificial intelligence" in the opening question told me right off the bat that something was off with the interviewer. Some version of AI research has been around nearly as long as computers themselves, but the late 60s weren't exactly a focal point like, say, the 80s were. Plus, her primary contribution to computer science has little to do with AI directly. It felt like misunderstanding and/or a wide reach for a hot topic.

<comment><USER22> Perceptron, which preceded both the minsky induced AI winter and the current hype around deep learning was in fact invented in 1958.

</comment></comment></comment><comment><USER23> Hmmm...

"The language, CLU (short for "cluster"), relied on an approach she invented — data abstraction — that organized code into modules."

Let's ignore how preposterous it is to claim you invented data abstraction and move to the module claim... that a simple search for "who discovered programming modules" shows:

"The Modula programming language ... by Niklaus Wirth, the same person who designed Pascal. The main innovation of Modula over Pascal is a module system, "

And it seems others were thinking the same way: 'After you won the Turing Award, a comment appeared online saying, "Why did she get this award? She didn't do anything that we didn't already know."'

Not sure what to think here but I never heard CLU was the first to add modules. I'll just leave it at that.

<comment><USER24> _And it seems others were thinking the same way: 'After you won the Turing Award, a comment appeared online saying, "Why did she get this award? She didn't do anything that we didn't already know."'_

This is not only distasteful but false.

First, modules in Modula were primarily about _scope_ not abstraction.

Liskov pioneered a core aspect of _abstraction_ which, among other things, allows modules to rely on abstract interfaces rather than their implementations (or a type or class rather than its subtypes or subclasses).

Abstraction only works reliably if any provably true property of a type is also provably true about its subtype.

Want to learn more? You'll find it referenced in the literature as the _Liskov_ Substitution Principle.

[https://dl.acm.org/citation.cfm?doid=197320.197383](https://dl.acm.org/citation.cfm?doid=197320.197383)

<comment><USER25> When did ML gain its module feature (which does enable the equivalent to data abstraction)? From a _very_ casual search, the earliest references I can find to it are from the early 1980s, so well after CLU - but the very first versions of ML itself were being worked on around the same time.

<comment><USER24> That's a good question, I don't know the history well enough to comment on that.

Much more important than CLU itself — groundbreaking as it was — was Liskov's formalization of substitutability as a rigorous principle one can use to reason about the quality or correctness of an arbitrary abstraction (or model thereof).</comment></comment></comment></comment>


Marvin Minsky's "The Society of Mind" now CC licensed - <AUTHOR>

http://aurellem.org/society-of-mind/ <comment><USER2> This is great! The only thing missing is a .pdf version.

Here's what Minsky himself says [0] about the book:

"My first book about artificial intelligence was a book called the Society of Mind and it had one page chapters.

And this book was very influential because it had the feature that if you didn't understand one of these chapters you could just skim it and it wouldn't matter very much. And the result was that great numbers of high school students understood most of the theories and knew more than their professors did when they got to college."

[0] [http://www.youtube.com/watch?v=3PdxQbOvAlI#t=16m7s](http://www.youtube.com/watch?v=3PdxQbOvAlI#t=16m7s)

</comment><comment><USER3> ah - inspirational memories:)</comment>


Properly Ground your Circuits - <AUTHOR> http://www.edn.com/design/pc-board/4443239/Properly-ground-your-circuits <comment><AUTHOR> Poor grounding is a mistake that comes up a lot when people are learning electronics. This article has an excellent discussion about the various types of 'ground' references and how they interact.

<comment><USER2> I found this article too basic to be actionable.

Here is a thorough reference on the subject from Artech: [https://www.amazon.com/Essentials-Microwave-Grounding- Artech...](https://www.amazon.com/Essentials-Microwave-Grounding-Artech- Library/dp/1580539416/)

A related classic is: High Speed Digital Design: A Handbook of Black Magic, [https://www.amazon.com/gp/product/0133957241](https://www.amazon.com/gp/product/0133957241)</comment></comment>


The 5 Best Geeky Kickstarter Products Released In 2013  - <AUTHOR> http://crowddistrict.com/five-best-geeky-kickstarter-products-2013/

<comment><USER2> This is a great list. Spot On

<comment><USER3> Yeah. I was expecting Pebble to be on the list.</comment></comment>


Sheep Darknet Marketplace Owner Arrested While Trying to Buy Luxury Home - <AUTHOR> http://247cryptonews.com/sheep-darknet-marketplace-owner-arrested-while-trying-to-buy-luxury-home/ <comment><USER2> What an idiot.</comment>


How Mailinator compresses its email stream by 90% - <AUTHOR> http://mailinator.blogspot.com/2012/02/how-mailinator-compresses-email-by-90.html <comment><USER2> Mailinator is a great product. My favorite part about it is how whenever I register for something, the clever form validation software always rejects the mailinator.com email address. Then I visit mailinator, see their alternate domain name du jour in image form (so bots can't harvest it, hah!), and then that works perfectly. It makes me giggle with joy every time I do it.

It's also nice not receiving ads in the mail every hour of every day just because I wanted to try some new YC-startup's product.

<comment><USER3> I worry we're going to need disposable social ids now too like email. That's something facebook/twitter probably won't appreciate. I would LOVE to try a lot of services that require one of those logins but am not comfortable giving them access to my info.

<comment><USER4> Create an "alter ego" account on Facebook. <http://www.fakenamegenerator.com/gen-random-us-us.php> if you need fake name and address ideas, or just use whatever you wish your name was. If you get too funny with the name though, FB's filter won't believe your name.

Then, keep a separate Chrome profile that's logged into the fake facebook, and your identity is safe. Probably best to lock down all privacy settings on the fake user to absolute maximum lockdown, to avoid information leaking onto Facebook's public layer that may tie you to it.

That was the strategy I used when I worked for a company that did business with Facebook games. Those things were so spammy, I would never want my actual friends to be spammed with all the viral content. To this day I keep my fake account for any and all untrusted FB-related things.

<comment><USER5> Facebook now requires your phone number in order to register. Have a way to use fake phone too?

<comment><USER6> I'm fairly certain this is untrue. Looking at the signup form, all I can see is name, email, password, birthdate and gender. They may ask for it later, but that almost certainly is optional.

<comment><USER7> This is sort of true. If you want to use the account as a developer account you have to register your phone number. Also probably to advertise.

I recently wanted to create an "Admin" account for a client I was developing a web-app for as I didn't want to link my personal facebook account to my work. But with the phone number requirement the account got flagged and rejected from the system.

Frustrating!

</comment></comment></comment></comment></comment></comment><comment><USER8> I run a similar (though waaaay less popular) site. My mail is stored on disk in a mysql db so I don't have quite the same memory constraints as this.

I had originally created this site naively stashing the uncompressed source straight into the db. For the ~100,000 mails I'd typically retain this would take up anywhere from 800mb to slightly over a gig.

At a recent rails camp, I was in need of a mini project so decided that some sort of compression was in order. Not being quite so clever I just used the readily available Zlib library in ruby.

This took about 30 minutes to implement and a couple of hours to test and debug. An obvious bug (very large emails were causing me to exceed the BLOB size limit and truncating the compressed source) was the main problem there...

I didn't quite reach 90% compression, but my database is now typically around 200-350mb. So about 70-80% compression. So, I didn't reach 90% compression, but I did manage to implement it in about 6 lines of code =)

<comment><USER9> Can you tell us what the site is? I've once or twice found sites that blocked even mailinator's alternate domains. Having a less well known alternative to mailinator would be nice.

<comment><USER10> 10MinuteMail.com is another option.

(biased creator here)

<comment><USER11> There's also 20minutemail.com - random address, allows reply, forward and download of message

yopmail.com - any address (like Mailanator) and allows download of attachments directly

There are a ton of these available, just depends on your specific needs. I regularly use them for signing up to those sites that need sign-up before access to a file/article/image (e.g. forums).

I've also used them in testing web apps, using Selenium to grab the email address and registering a user and then going back to check the welcome email had arrived.

</comment></comment></comment></comment><comment><USER12> Both blogs (mailinator and paultyma) are awesome. I need more stuff like this than a typical Web 2.0, how we use NoSQL, and Cache-Everything (without a clue how to do cache properly, but 37Signals cache solution is by far in line with mailinator techniques: smart and elegant).

<comment><USER13> It's awesome, yes - but it's worth keeping in mind that this trick works because he has an intensely write-heavy application (he probably does orders of magnitude more writes than reads). Very few applications have such a data access pattern.

</comment></comment><comment><USER14> Great read. I wish there were more articles like these.

<comment><USER15> Then rest of the mailinator blog and his blog (<http://paultyma.blogspot.com/>) are filed with gems like this.

There are a few of entries on multi-threaded synchronous io vs asynch io; he's a big proponent of the former.

</comment></comment><comment><USER16> Redis works great as an LRU cache and is much more space-efficient than an in- process LinkedHashMap, especially when the keys and values are small. Plus, an LRU wreaks havoc with the the Java generational garbage collector as soon as it fills up (every entry you put in is about guaranteed to last until the oldest generation, then likely be removed).

<comment><USER17> Redis would blow latency budget though, right?

<comment><USER18> Maybe I'm missing something here, but I was under the impression Redis is one of the fastest data stores out there. What do you mean it would blow the latency budget? I'm curious because I've switched my startup's backend to node+redis.

Thanks in advance!

<comment><USER19> He means it is being compared to CPU cache (from the article), so there is tens of orders of magnitude difference. From this summary on stack overflow: [http://stackoverflow.com/questions/433105/exactly-how- fast-a...](http://stackoverflow.com/questions/433105/exactly-how-fast-are- modern-cpus)

CPU registers (8-32 registers) – immediate access (0-1 clock cycles) L1 CPU caches (32 KiB to 128 KiB) – fast access (3 clock cycles) L2 CPU caches (128 KiB to 12 MiB) – slightly slower access (10 clock cycles) Main physical memory (RAM) (256 MiB to 4 GiB) – slow access (100 clock cycles) Disk (file system) (1 GiB to 1 TiB) – very slow (10,000,000 clock cycles) Remote Memory (such as other computers or the Internet) (Practically unlimited) – speed varies

<comment><USER16> But if you have a Redis cache on the same box (he says he only has one box anyway) it's still in the same category: "Main physical memory", with maybe some communication overheard.

<comment><USER17> "maybe some communication overheard" is orders of magnitude slower than L1/L2.

Hmm, there is something very wrong here. I'll try and explain in a blog post.

<comment><USER16> But we're not talking register caches. 800 megs stashed in a giant Java hash are not going to be in L1 or L2 cache.

<comment><USER17> [http://williamedwardscoder.tumblr.com/post/18065079081/cogs-...](http://williamedwardscoder.tumblr.com/post/18065079081/cogs- bad#comment-446733732) might be interesting

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER20> Reading about another algo (Locality Sensitive Hashing) as referenced in the first comment.

<http://www.stanford.edu/class/cs345a/slides/05-LSH.pdf>

<comment><USER21> See also this library for locality sensitive hashing:

<http://lshkit.sourceforge.net/>

</comment></comment><comment><USER22> Great write up. One of the more interesting things I've read on here in a while. Thanks for sharing.

</comment><comment><USER23> In an aside he mentions you should use bubblesort instead of quicksort for small arrays, due to cache locality, etc. I'd recommend using insertion sort instead of bubblesort - it does much better in both cache locality and branch performance (one branch prediction miss per key).

<comment><USER24> That seems a strange assertion - I was under the impression that insertion sort beats bubble sort on small arrays, pretty much always, and even in 'nearly sorted' arrays, which is often touted as a strength of bubble sort.

<http://www.sorting-algorithms.com/> gives a decent overview with n equal to 20, 30, 40, or 50, but nothing smaller. Now I'm curious.

</comment></comment><comment><USER25> 1\. I think the author calls DEFLATE an LZW algorithm. It isn't.

2\. Has the author looked at Google Snappy? It does 500MB/sec. [http://code.google.com/p/snappy/source/browse/trunk/format_d...](http://code.google.com/p/snappy/source/browse/trunk/format_description.txt)

There is a pure-C implementation that might be easier to port: <https://github.com/zeevt/csnappy>

<comment><USER26> There is already an excellent native port of Snappy to Java: <https://github.com/dain/snappy>

</comment></comment><comment><USER27> Just when I thought I knew everything there is to know about compression algorithms, along came Pauli, and Voila, mind now blown.

<comment><USER28> Check out what RainStor does to compress relational data by a factor of 20-40 or so (that's 95%-97.5%): [http://www.youtube.com/watch?v=rIsVcaMaAgg&feature=playe...](http://www.youtube.com/watch?v=rIsVcaMaAgg&feature=player_embedded)

</comment></comment><comment><USER29> _Voila. 90%. (Two notes: 1: that's a reasonable average at least... sometimes better, sometimes worse and 2: I realize I'm not exactly sure what "Voila" means, looking that up now)._

If mailinator wasn't already awesome, his writing about it sure is.

</comment><comment><USER30> Something else that might work is content-dependent deduplication, with variable chunk boundaries determined by a sliding Rabin-Karp-style (or XOR) fingerprint on the content and a second cryptographic hash calculated for chunks where the cheap fingerprint has a match. It's naive and can find matches across headers, body and attachment parts.

</comment><comment><USER31> I recently worked on a project where, to cut down on space, I built a custom "compressor" for lists of tree-like records. You might think of a Person record with N past addresses although this was not the actual domain. No records were cross-linked (at least not formally in the code) and the records were trees, not more general DAGs. The data contained a lot of enumerated data types (address type, gender, etc.). I didn't really care about the space usage for 1K records, but I cared about 1M. I used variable length encoding (a hacked-up subset of the Apache Avro project) for integers to take advantage of most of them being small in the datasets. Lots of lookup tables for enumerated values and commonly-repeated in practice string values. Implicit "keying" based on a record schema to avoid specifying key identifiers all over (our data was not very sparse, so this beat out a serialized pattern of KVKVKV etc.). I thought about taking advantage of most strings having very limited character sets and doing Huffman encoding per data element type, but the results were good enough before I got there. A co-worker also noted that, because parts of these records were de-normalized data, subtree pattern recognition could provide huge gains. I added some record length prefixes to allow individual records to be read one-at-a-time so that the entire dataset would not have to be read into memory at once. IIRC, compression speed was 2-3x gzip(9), and large record sets were 1/10th the size of using Java serialization plus gzip. [Yes, Java serialization is likely the worst way to serialize this sort of data]

Was all of this worth it? It solved the problem of not burning through network and memory, but it was a local optima. The root problem was that this data came from another system which did not provide repeatable reads, and providing them would have been a massive effort. However, our users wanted to meander through a consistent data set over the course of an hour or so. To provide this ability to browse, we throw these records into a somewhat transient embedded H2 DB instance. The serialized format is required primarily to provide high availability via a clustered cache. In retrospect, I would have pushed for using a MongoDB-esque cluster which could have replaced both H2 (query-ability) and the need for the the serialized format (HA).

It surprised me that there were no open source projects (at least Java- friendly ones) which provided compression schemes taking advantage of the combination of well-defined record schema and redundant-in-practice data. Kyro (<http://code.google.com/p/kryo>) comes closest as a space-efficient serializer, but it treats each record individually. Protobufs, Thrift, Avro, etc. are designed for RPC/Messaging wire formats and, as an explicit design decision (at least in the protobufs case) optimize on speed and the size of an individual record vs. the size of many records. The binary standards for JSON and XML beat the hell out of their textual equivalents, but they don't have any tricks which optimize on patterns in the repeated record structures.

Is this just an odd use case? Does anyone else have a similar need?

<comment><USER32> Most people are probably happy relying on gzip for compression. It's "Good Enough" for most use cases. It seems like it made sense for you though.

</comment></comment><comment><USER33> Further efficiencies can be gained by removing extraneous apostrophes from possessive "its".

<comment><USER21> While that would result in decreased entropy that a decent general-purpose compression algorithm like LZ77 could take advantage of, this preprocessing step is lossy and probably not worth the bytes it would save you.

</comment></comment><comment><USER34> The line based thing smells like doing LCS but with string elements of length N rather than 1...

</comment><comment><USER35> Mailinator, even considering any praise it has ever gotten, is still one of the most underrated tools on the internet. I love it and use it all the time.

</comment><comment><USER36> Reading this article makes me giggly inside.</comment>


Google Assistant interpreter mode real-time translation on iOS andAndroid phones - <AUTHOR> https://www.theverge.com/2019/12/12/21010451/google-assistant-interpreter-mode-android-ios-apps <comment><AUTHOR> >Once you're in interpreter mode, the Assistant will translate your speech and read it out loud. On phone screens, the Assistant is able to offer up Smart Replies that can speed up the conversation by letting you respond without having to speak.</comment>


When to use STDERR instead of STDOUT - <AUTHOR> http://jstorimer.com/2011/12/29/the-difference-between-stdout-and-stderr.html

<comment><USER2> Interesting, but that font looks terrible in Opera.</comment>


If the car were invented today... - <AUTHOR>

How would you design it? <comment><USER3> It would be a hybrid tram/car, they would have to follow a track but that track wouldn't be electrified fully it would be a guide for where they can go, just at stops and parking is where the car could charge. the cars would have an urban and rural mode. rural would work without those tracks.

</comment><comment><USER4> I don't know a lot about car to suggest changes to wheels and engines - <USER2> line of thing.

Automating cars... well, that's a relatively hard problem to do wholesale.

I suspect the first thing I'd look into would be simplifying.

Lane changing is stressful, people tailgating is stressful, parking can be difficult.

Lane keeping isn't as hard a problem as automating the entire drive. And it doesn't seem like it should be hard to link that up to something that says move one lane over. And from there to check if a space in the next lane is clear - iterate that until it is and then move over.

Replace the steering wheel with a clicky joystick. Set it up so that when someone's near a parking space and they move the joystick towards it the car parks in that space.

Keeping distance isn't hard, but a lot of people don't do it. It's not hard in a mathematical sense either. Replace the gearstick and so on with a slider with a desired speed setting. Approximate that speed as closely as possible while maintaining a decent distance from the car in front.

And, incase the software went squify, a big red button that would stop the car via some non-software set-up.

You can probably break it down into many more sub-points. This is like a five minute 'if I', after all. But, I suspect that sort of thing would remove a lot of the day to day aggravation of driving. You'd be indicating your _intent_ to the car, doing the planning, but the car would be doing the stressful implementation details.

And as computer vision, general automation, got better you could start to do things like automatically moving out around junctions to maintain a smooth flow of traffic, executing better zipper merges - and so on. So that there wouldn't be that hard line between the driver being in control of the car and the computer being responsible. Each iteration would just require the person to touch the controls less, until they didn't have to touch them at all.</comment>