A plan to change how Harvard teaches economics - <AUTHOR> https://www.vox.com/the-highlight/2019/5/14/18520783/harvard-economics-chetty <comment><USER2> This isn't an economics class. It's a public policy class. Here are the course topics:[1]

\- Part I: Equality of Opportunity

\- Part II: Education

\- Part III: Racial Disparities

\- Part IV: Health

\- Part V: Criminal Justice

\- Part VI: Climate Change

\- Part VII: Tax Policy

\- Part VIII: Economic Development and Institutional Change

This really belongs in Harvard's "JFK School of Government", not economics.

Possible topics for a modern economics intro class:

\- Instability and equilibrium, or why markets oscillate.

\- From zero to one, the tendency to and effects of monopoly and near- monopoly.

\- Externalities, their uses and discontents.

\- Debt vs. equity vs. what tax policy rewards

\- Scarce resources that don't map to money - attention and time.

\- Finance as a system decoupled from productive activity

[1] [https://opportunityinsights.org/wp- content/uploads/2019/05/E...](https://opportunityinsights.org/wp- content/uploads/2019/05/Econ1152_syllabus_spring19_forweb-1.pdf)

<comment><USER3> Is there a single book that covers all that? That would be awesome.

<comment><USER4> Schools like Harvard don't teach from a book.

<comment> barry-cotter Wrong side of the Atlantic. I didn't have to buy a single textbook my entire undergrad because we had libraries and lecture notes. Required textbooks are far more a US thing than most other countries.

Harvard absolutely has courses taught from a book. I'm sure Mankiw requires his textbook for Harvard's intro econ course. He wrote it, he thinks it's good.

<comment><USER5> And he's made $42 _million_ in royalties on the book. It's almost endearing how economists claim they are somehow immune to incentives, until you pause to consider they are primarily employed as apologists for the continuation of rent-seeking policies that entrench the rich and mighty.

<comment><USER6> > _apologists for the continuation of rent-seeking policies that entrench the > rich and mighty._

This.

"THE IMF CONFIRMS THAT 'TRICKLE-DOWN' ECONOMICS IS, INDEED, A JOKE" [https://psmag.com/economics/trickle-down-economics-is- indeed...](https://psmag.com/economics/trickle-down-economics-is-indeed-a- joke)

> _INCREASING THE INCOME SHARE TO THE BOTTOM 20 PERCENT OF CITIZENS BY A MERE > ONE PERCENT RESULTS IN A 0.38 PERCENTAGE POINT JUMP IN GDP GROWTH._

> _The IMF report, authored by five economists, presents a scathing rejection > of the trickle-down approach, arguing that the monetary philosophy has been > used as a justification for growing income inequality over the past several > decades. "Income distribution matters for growth," they write. > "Specifically, if the income share of the top 20 percent increases, then GDP > growth actually declined over the medium term, suggesting that the benefits > do not trickle down."_

"Causes and Consequences of Income Inequality: A Global Perspective" (2015) [https://scholar.google.com/scholar?hl=en&as_sdt=0%2C43&q=%22...](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C43&q=%22Causes+and+Consequences+of+Income+Inequality%3A+A+Global+Perspective%22)

I'll add that we tend to overlook the level of government spending during periods trickle-down economics and confound. Change in government spending (somewhat unfortunately regardless of revenues) is a relevant factor.

Let's make this economy great again? How about you identify the decade(s) you're referring to and I'll show you the tax revenue (on income and now capital gains), federal debt per capital, and the growth in GDP.

</comment></comment></comment></comment></comment></comment><comment><USER7> You can look through any mainstream micro textbook, graduate or undergrad, and in 1,000 pages won't see a single citation to support any model empirically. Compare that to any decent physics textbook, which will link models to the experiments that back them up. Economics for the real world won't be simple and pure like physics, it'll be more like geology or biology, with a lot more facts and a lot fewer theories of everything.

<comment><USER8> This is not true. I've been through an economics program and a lot of theories have empirical data to support them. Of course it's much harder to prove there is causation and not only correlation (and all that that entails), but that's part of being a topic which to a large extent deal with social phenomena (people).

Can't dig up anything now but a lot of phenomena in papers (though of course far from all /majority) do or reference some sort of quantitative research).

<comment><USER9> Meh. The standard of "support" is pretty low. I spent a lot of time looking through the literature for empirical support for 1/ the Philips curve, and 2/ gravity models of trade. I looked for the first out of personal interest, and for the second because economists often claim that gravity models are well- supported.

Where empirical studies were actually available, they usually had large numbers of parameters, they did not test on diverse datasets (e.g. Philips curves across different economies), and they never considered an alternative hypothesis or simple baseline.

<comment><USER10> Indeed. Even within a single economy, plotting a scatter plot for enough data points (e.g. 25 years) makes Philips curves just broken.

It was a gong moment for me when I came across a short passage discussing its foundedness.

* Freedman, Pisani and Purves Statistics, 4th edition (p153)

</comment></comment></comment></comment><comment><USER11> This initiative is very close in spirit to the CORE economics curriculum ([https://www.core-econ.org/](https://www.core-econ.org/)).

It focuses on observable social and economics consequences (failures of equality, opportunity and sustainability in particular), when introducing new economic concepts.

The goal is to avoid students being blinded by appealing, but simplistic, economics models.

Having been seduced by many an economic theory before, I for one welcome this perspective.

</comment><comment><USER12> Cool. One thing I hated about my Econ minor was that so much of undergraduate economics education assumes conditions which are never true. It's like studying aerospace engineering and only introducing air resistance in graduate school.

<comment><USER13> Good comparison. This is how I felt when I took ECON 101. Though I was fascinated with the concepts, they felt so far from reality as to be effectively useless.

<comment><USER14> That's what ECON 101 is, you need to get deeper into the curriculum. It's like complaining intro to algebra didn't teach you differential equations

<comment><USER13> The problem was that this stance was used to justify claims like "the free market allocates resources in the most efficient way" and "all taxes incur a deadweight loss" which are big statements to make based on such basic coursework. If the class is going to make claims like these, it needs to start with realistic assumptions.

</comment></comment></comment></comment><comment><USER15> The course is EC1152 and titled "Big Data to Solve Economic and Social Problems". Lecture notes are at [https://opportunityinsights.org/wp- content/uploads/2019/05/B...](https://opportunityinsights.org/wp- content/uploads/2019/05/Big-Data-Course-Slides-in-PDF-Format-1.zip)

</comment><comment><USER16> If I read this right, Chetty's new EC course tries not only to put the focus on empirical data and its analysis, but also to disentangle ethical value judgments from economics.

This is sorely needed. I've often thought that economists (on either side of the political divide) love to make pompous pronouncements about what's mathematically right and wrong and somehow translate that into what's ethically right and wrong, and then say that their opponents are stupid and evil.

<comment><USER17> Chetty's own inequality/mobility data gets interpreted differently by the left and the right. There is no free lunch here, when it comes to dealing with people's vested interests (biases).

Policy/decision makers know anytime things get political the issue is never about the data. If someone somewhere has to give up something, try doing it just showing them your data and analysis.

You can run the experiment on HN. Try getting something in the architecture to change based on data, and pitch it to the HN community. See what happens...

</comment></comment><comment><USER18> Suggested parts, considering what Harvard econ has done for the human race:

IX: "How we looted the former Soviet Union, blew up their economy, got away with it, and blamed it on the Russians" (team seminar by Andrei Shleifer and Larry Summers)

X: "Selling your country to foreigners, indebting the masses for fun and profit, then telling the fools the GDP got bigger" (Greg Mankiw)

XI: "Linear regression, with ideology" (everyone else)

<comment><USER19> Can you explain how the US looted the former Soviet Union? Never heard that one before.

<comment><USER20> It's considered among knowledgeable circles that the creation of the Russia we know of today was largely the work of the "Chicago Boys" type group of economists who were sent to liberalize the Russian economy through "shock therapy".

There are lots of well researched stories that show they did stupid badly thought of things such as, famously, handing over stocks in major companies to its workers, instantly making them shareholders.

That sounds fine in theory. But with poverty at their throat workers simply sold their sto luck to the first person who asked. Often these were people with "connections" or some money stashed away or criminally-funded "entrepreneurs", who with some barebones organisation and planning could quickly scoop up billions of dollars worth of stock and take a controlling position in major companies by buying them cents for a dollar.

Thus the oligarchs were born which quickly led to the kleptocratic Russia we know of today.

Liberal minded and educated Russians realised western economists were annihilating their country. One well known Russian independent magazine called the newly rebuilt Russia a "neoliberal dystopia". Which more and more I think of as quite accurate.

<comment><USER21> Which knowledgeable circles?

This seems demonstrably untrue: "Thus the oligarchs were born which quickly led to the kleptocratic Russia we know of today"

Oligarchs have been a mainstay of Russian society since at least the time of Peter the Great when one of his main policy positions as Tsar was reducing the influence of the oligarchs (then called Boyars).

Plus, Russia under Soviet rule was unbelievably kleptocratic and corrupt.

<comment><USER22> Most Russian oligarchs today trace their wealth back to the acquisition of shares in 1991, often with the help of loans from banks run by friends.

Larry Summers, when he was "advising", didn't make a secret of the fact that he didn't really care who "owned" most Russian assets provided it was in private hands.

<comment><USER23> > Larry Summers, when he was "advising", didn't make a secret of the fact that > he didn't really care who "owned" most Russian assets provided it was in > private hands.

Ideology over practicality. Lovely.

</comment></comment></comment></comment></comment></comment><comment><USER24> I wonder how long it will take before the Ivys preference of social justice over teaching core curricula will affect their market value.

<comment><USER25> They probably don't see this false dichotomy of social justice vs. core curricula, which would explain why they are fine with it.

</comment></comment><comment><AUTHOR> The course is called Economics 1152. The video lectures and course slides are available here:

[https://opportunityinsights.org/course/](https://opportunityinsights.org/course/)

</comment><comment><USER26> The difference big data/data science/empirical study and 'classical' economics (by which I'd include any system of economics that seeks to explain human behavior via an underlying metatheory) is that a primarily empirical approach obscures the necessary underlying theory present in any experiment where you're trying to fit data to a curve.

For example, when you run a science experiment and you plot the data, you may find that you're looking at a line. While this is an interesting finding, it has zero predictive value for anything other than the exact situation you've collected data for. In order to formulate scientific law, you first must (a) believe that such a thing exists and (b) have some theory as to what shape the curve ought to fit. For example, a naive look at physics using an 'empirical' approach might incorrectly conclude that force is mass times acceleration. While moderately useful for many problems, this offers little predictive power in the general case. In order to actually formulate a law that can be of predictive value, you have to first consider various other laws and axioms (such as the constant speed of light for force), at which point -- by deduction, without any need of empirism -- you determine that this is wrong, and you need another kind of equation to fit your data to.

I don't know if the simplistic demand curves drawn in the original text book are correct or not. However, at least those are based on a particular set of assumptions that can be validated or not. The kind of empiricism put forth by Mr Chetty does not offer this at all.

All this is to say that, while data is useful for validation, it is not useful for prediction. The last thing we need is a black-box machine learning model to make major economic decisions off of. What we do need is proper models that are then validated, which don't necessarily need 'big data.'

<comment><USER6> > _All this is to say that, while data is useful for validation, it is not > useful for prediction. The last thing we need is a black-box machine > learning model to make major economic decisions off of. What we do need is > proper models that are then validated, which don 't necessarily need 'big > data.'_

Hand-wavy theory - predicated upon physical-world models of equillibrium which are themselves classical and incomplete - without validation is preferable to empirical models? Please.

Estimating the predictive power of some LaTeX equations is a different task than measuring error of a trained model.

If the model does not fit all of the _big_ data, the error term is higher; regardless of whether the model was pulled out of a hat in front of a captive audience or deduced though inference from actual data fed through an unbiased analysis pipeline.

If the 'black-box predictive model' has lower error for all available data, the task is then to reverse the model! Not to argue for unvalidated theory.

Here are a few discussions regarding validating economic models, some excellent open econometric lectures (as notebooks that are unfortunately not in an easily-testable programmatic form), the lack of responsible validation, and some tools and datasets that may be useful for validating hand-wavy _classical_ economic theories:

"When does the concept of equilibrium work in economics?" [https://news.ycombinator.com/item?id=19214650](https://news.ycombinator.com/item?id=19214650)

> "Lectures in Quantitative Economics as Python and Julia Notebooks" > [https://news.ycombinator.com/item?id=19083479](https://news.ycombinator.com/item?id=19083479) > (data sources (pandas-datareader, pandaSDMX), tools, _latex2sympy_ )

That's just an equation in a PDF.

(edit) Here's another useful thread: "Ask HN: Data analysis workflow?" [https://news.ycombinator.com/item?id=18798244](https://news.ycombinator.com/item?id=18798244)

<comment><USER27> Most of the interesting economic questions are inference problems, not prediction problems The question is not "what is the best guess of y[i] given these values of x[i]'s", but what would y[i] have been for this very individual i (or country in macro-economics) if we could have wound back the clock and change the values of x[i]'s for this individual. The methods that economists know and use may not be the best, but the standard ML prediction methods do not address the same questions, and data scientists without a social / economic / medical background are often not even aware of the distinction.

Economists and social scientists try to do non-experimental causal inference. Maybe they're not good at it, maybe the very problem is unsolvable, but it's not because they don't know how Random Forests or RNNs work. Economists already _know_ that students from single parent families do worse at school than from married families. If the problem is just to predict individual student results, number of parents in the household is certainly a good predictor. The problem facing economists is, would encouraging marriage or discouraging diveroce improve student results? Nothing in PyTorch or Tensorflow will help with the answer..

<comment><USER6> Backtesting algorithmic trading algorithms is fairly simple: what actions would the model have taken given the available data at that time, and how would those trading decisions have affected the single objective dependent variable. Backtesting, paper trading, live trading.

Medicine (and also social sciences) is indeed more complex; but classification and prediction are still the basis for making treatment recommendations, for example.

Still, the task really is the same. A NN (like those that Torch, Theano, TensorFlow, and PyTorch produce; now with the ONNX standard for neural network model interchange) learns complex relations and really doesn't care about causality: minimize the error term. Recent progress in reducing the size of NN models e.g. for offline natural language classification on mobile devices has centered around identifying redundant neuronal connections ("from 100GB to just 0.5GB"). Reversing a NN into a far less complex symbolic model (with variable names) is not a new objective. NNs are being applied for feature selection, XGBoost wins many Kaggle competitions, and combinations thereof appear to be promising.

Actually testing second-order effects of evidence-based economic policy recommendations is certainly a complex highly-multivariate task (with unfortunate ideological digression that presumes a higher-order understanding based upon seeming truisms that are not at all validated given, in many instances, _any_ data). A causal model may not be necessary or even reasonably explainable; and what objective dependent variables should we optimize for? Short term growth or long-term prosperity with environmental sustainability?

... "Please highly weight _voluntary_ sustainability reporting metrics along with fundamentals" when making investments and policy decisions?

Were/are the World3 models causal? Many of their predictions have subsequently been validated. Are those policy recommendations (e.g. in "The Limits to Growth") even more applicable today, or do we need to add more labeled data and "Restart and Run All"?

...

From [https://research.stlouisfed.org/useraccount/fredcast/faq/](https://research.stlouisfed.org/useraccount/fredcast/faq/) :

> _FREDcast™ is an interactive forecasting game in which players make > forecasts for four economic releases: GDP, inflation, employment, and > unemployment. All forecasts are for the current month—or current quarter in > the case of GDP. Forecasts must be submitted by the 20th of the current > month. For real GDP growth, players submit a forecast for current-quarter > GDP each month during the current quarter. Forecasts for each of the four > variables are scored for accuracy, and a total monthly score is obtained > from these scores. Scores for each monthly forecast are based on the > magnitude of the forecast error. These monthly scores are weighted over time > and accumulated to give an overall performance._

> _Higher scores reflect greater accuracy over time. Past months ' > performances are downweighted so that more-recent performance plays a larger > part in the scoring._

The #GobalGoals Targets and Indicators may be our best set of variables to optimize for from 2015 through 2030; I suppose all of them are economic.

<comment><USER28> Using predictive models for policy is not new, in fact it was the standard approach long before more inferential models, and the famed Lucas critique precisely targets a primitive approach similar to what you are proposing.

The issue is the following: In economics, one is interested in an underlying parameter of a complex equilibrium system (or, if you wish, a non-equilibrium complex system of multi-agentic behavior). This may be, for example, some pricing parameter for a given firm - say - how your sold units react to setting a price.

Economics faces two basic issues:

First, any predictive model (like a NN or simple regression) that takes price as an input, will not correctly estimate the sensitivity of revenue to price. It is actually usually the case, that the inference is reversed.

A model where price is input, and sold units or revenue is output (or vice- versa) will predict (you can check that using pretty much any dataset of prices and outputs) that higher prices lead to higher outputs, because that is the association in the data. Of course we know that in truth, prices and outputs are co-determined. They are simultaneous phenomena, and regressing one on the other is not sufficient to "causally identify" the correct effect.

This is independent of how sophisticated your model is otherwise. Fitting a better non-linear representation does not help.

The solution is of course to reduce down these "endogenous" phenomena to their basic ingredients. Say you have cost data, and some demand parameters. Then, using a regression model (or NN) to predict the vector of endogenous outcome variables will work, and roughly give you the right inference.

Then, as a firm, you are able to use these (more) exogenous predictive variables to find your correct pricing.

This is not new, pops up everywhere in social science, is the basis of a gigantic literature called econometrics, and really has nothing to do with how you do the prediction.

The only thing that NN add are better predictions (better fitting) and the ability to deal with more data. As this inferential problem shows, using more (and more fine-grained) data is indeed crucial to predicting what a firm should do.

BUT, it is crucial to understand and reason about the underlying causality FIRST, because otherwise even the most sophisticated statistical approach will simply give you wrong results.

Secondly, the counterfactual data for economic issues is usually very scarce. The approach taken by machine learning is problematic, not only because of potentially wrong inference, but also because two points in time may simply not be based on comparable data-generating processes.

In fact, this is exactly the blindness that led to people missing the financial crisis. Of course, with enough data, and long enough samples, one should expect to be become pretty good at predicting economic outcomes. But experience has shown that in economics, these data are simply too scarce. The unobserved variation between two quarters, two years, two countries, two firms (etc.) is simply very large and has fat tails. This leads to spontaneous breakdowns of such predicitive models.

Taking these two issues together, we see that better non-linear function approximation is not the solution to our problems. Instead, it is a methodological improvement that must be used in conjunction with what we have learned about causality.

Indeed the literature moves into a different direction. Good economic science nowadays means to identify effects via natural experiments and other exogenous shifts that can plausibly show causality.

Of course such experiments are more rare, and more difficult, the larger the scale becomes. Which is why Macroeconomics is arguably the "worst science" in economics, while things like auctions and microstructure of markets are actually surprisingly good science (nowadays).

Doors are wide open for ML techniques, but really only to the point that they are useful in operationalizing more and better data.

Anyone trying to understand economic phenomena needs to be keenly aware of how inference can be done, which requires an understanding (or an approach to) - that is, a theory - of the underlying mechanisms.

<comment><USER6> Yes, some combination of variables/features grouped and connected with operators that correlate to an optima (some of which are parameters we can specify) that occurs immediately or after a period of lag during which other variables of the given complex system are dangerously assumed to remain constant.

> _In fact, this is exactly the blindness that led to people missing the > financial crisis_

ML was not necessary to recognize the yield curve inversion as a strongly predictive signal correlating to subsequent contraction.

An NN can certainly learn to predict according to the presence or magnitude of a yield curve inversion and which combinations of other features.

\- [ ] Exercise: Learning this and other predictive signals by cherry-picking data and hand-optimizing features may be an extremely appropriate exercise.

"This field is different because it's nonlinear, very complex, there are unquantified and/or uncollected human factors, and temporal"

Maybe we're not in agreement about whether AI and ML can do causal inference just as well if not better than humans manipulating symbols with human cognition and physical world intuition. The time is nigh!

In general, while skepticism and caution are appropriate, many fields suffer from a degree of hubris which prevents them from truly embracing stronger AI in their problem domain. (A human person cannot mutate symbol trees and validate with shuffled and split test data all night long)

> _Anyone trying to understand economic phenomena needs to be keenly aware of > how inference can be done, which requires an understanding (or an approach > to) - that is, a theory - of the underlying mechanisms._

I read this as "must be biased by the literature and willing to disregard an unacceptable error term"; but also caution against rationalizing blind findings which can easily be rationalized as logical due to any number of cognitive biases.

Compared to AI, we're not too rigorous about inductive or deductive inference; we simply store generalizations about human behavior and predict according to syntheses of activations in our human NNs.

If you're suggesting that the information theory that underlies AI and ML is insufficient to learn what we humans have learned in a few hundred years of observing and attempting to optimize, I must disagree (regardless of the hardness or softness of the given complex field). Beyond a few combinations/scenarios, our puny little brains are no match for our department's new willing AI scientist.

<comment><USER28> > ML was not necessary to recognize the yield curve inversion as a strongly > predictive signal correlating to subsequent contraction.

> An NN can certainly learn to predict according to the presence or magnitude > of a yield curve inversion and which combinations of other features.

> \- [ ] Exercise: Learning this and other predictive signals by cherry- > picking data and hand-optimizing features may be an extremely appropriate > exercise.

If the financial crisis has not yet occurred, how will the NN learn a relationship that does not exist in the data?

The exercise of cherry picking data and hand-optimizing is equivalent to applying theory to your statistical problem. It is what is required if you lack data points - using ML or otherwise. Nevertheless, we (as in humans) are bad at it. Speaking of the financial crisis. It was not AI's that picked up on it, it was some guys applying sophisticated and deep understanding of causal relationships. And that so few people did this, shows how bad we humans are at doing this implicitly and automatically by just looking at data!

> Maybe we're not in agreement about whether AI and ML can do causal inference > just as well if not better than humans manipulating symbols with human > cognition and physical world intuition. The time is nigh! In general, while > skepticism and caution are appropriate, many fields suffer from a degree of > hubris which prevents them from truly embracing stronger AI in their problem > domain. (A human person cannot mutate symbol trees and validate with > shuffled and split test data all night long)

ML and AI certainly can do causal inference. But then you have to do causal inference. Again, prediction on historical data is not equivalent to causal analysis, and neither is backtesting or validation. At the end of the day, AI and ML improves on predictions, but the distinction of causal analysis is a qualitative one.

> I read this as "must be biased by the literature and willing to disregard an > unacceptable error term"; but also caution against rationalizing blind > findings which can easily be rationalized as logical due to any number of > cognitive biases.

No. My point is that for causal analysis, you have to leverage assumptions that are beyond your data set. Where these come from is besides the point. You will always employ a theory, implicitly or explicitly.

The major issue is not the we use theories, but rather that we might do it implicitly, hiding the assumptions about the DGP that allows causal inference. This is where humans are bad. Theories are just theories. With precise assumptions giving us causal identification, we are in a good position to argue where we stand.

If we just run algorithms without really understand what is going on, we are just repeating the mistakes from the last forty years!

> If you're suggesting that the information theory that underlies AI and ML is > insufficient to learn what we humans have learned in a few hundred years of > observing and attempting to optimize, I must disagree (regardless of the > hardness or softness of the given complex field). Beyond a few > combinations/scenarios, our puny little brains are no match for our > department's new willing AI scientist.

All the information theory I have seen in any of the Machine Learning textbooks I have picked up is methodologically equivalent to statistics. In particular, the standard textbooks (Elements, Murhpy etc.) treatment of information theory would only allow causal identification under the exact same conditions that the statistics literature treats.

I fail to see the difference, or what AI in particular adds. The issue of causal inference is a "hot topic" in many fields, including AI, but the underlying philosophical issues are not exactly new. This includes information theory.

You seem to think that ML has somehow solved this problem. From my reading of these books, I certainly disagree. Causal inference is certainly POSSIBLE - just as in statistics, but ML doesn't give it to you for free!

In particular, note the following issue: To show causal identification, you need to make assumptions on your DGP (exogenous variation, timing, graphical causal relations ... whatever). Even if these assumptions are very implicit, they do exist. Just by looking at data, and running a model, you do not get causal inference. It can not be done "within" the system/model.

If you bake these things into your AI, then it, too makes these assumptions. There really is no difference. For example, I could imagine an AI that can identify likely exogenous variations in the data and use them to predict counterfactuals. That's probably not too far off, if it doesn't exist already. But, this is still based on the assumption that these variations are, indeed exogenous, which can never be proven within the DGP.

In contrast, I find that most "AI scientists" care very much about prediction, and very little about causal inference. I don't mean this subfield doesn't exist. But it is a subfield. In contrast, for many non-AI scientists, causal inference IS the fundamental question, and prediction is only an afterthought. ML in practice involved doing correct experiments (AB testing), at best. It will sooner or later also adopt all other causal inference techniques. But, my point stands, I have yet to see what ML adds. Enlighten me!

AI, ML and stats will merge, if they haven't already. The distinction will disappear. I believe the issues will not. I employ a lot of AI/ML techniques in my scientific work. Never have they solved the underlying issue of causal inference for me!

<comment><USER6> A causal model is a predictive model. We must validate the error of a causal model.

Why are theoretic models hand-wavy? "That's just because noise, the model is correct." No, such a model is insufficient to predict changes in dependent variables when in the presence of noise; which is always the case. How does validating a causal model differ from validating a predictive model with historical and future data?

Yield-curve inversion as a signal can be learned by human and artificial NNs. Period. There are a few false positives in historical data: indeed, describe the variance due to "noise" by searching for additional causal and correlative relations in additional datasets.

I searched for "python causal inference" and found a few resources on the first page of search results: [https://www.google.com/search?q=python+causal+inference](https://www.google.com/search?q=python+causal+inference)

CausalInference: [https://pypi.org/project/CausalInference/](https://pypi.org/project/CausalInference/)

DoWhy: [https://github.com/microsoft/dowhy](https://github.com/microsoft/dowhy)

CausalImpact (Python port of the R package): [https://github.com/dafiti/causalimpact](https://github.com/dafiti/causalimpact)

"What is the best Python package for causal inference?" [https://www.quora.com/What-is-the-best-Python-package-for- ca...](https://www.quora.com/What-is-the-best-Python-package-for-causal- inference)

Search: graphical model "information theory" [causal] [https://www.google.com/search?q=graphical+model+%22informati...](https://www.google.com/search?q=graphical+model+%22information+theory%22)

Search: opencog causal inference [https://www.google.com/search?q=opencog+causal+inference](https://www.google.com/search?q=opencog+causal+inference) (MOSES, PLN,)

If you were to write a pseudocode algorithm for an econometric researcher's process of causal inference (and also their cognitive processes (as executed in a NN with a topology)), how would that read?

(Edit) Something about the sufficiency of RL (Reinforcement Learning) for controlling cybernetic systems. [https://en.wikipedia.org/wiki/Cybernetics](https://en.wikipedia.org/wiki/Cybernetics)

<comment><USER27> What's the point of dumping a bunch of Google results here? At least half the results are about implementations of pretty traditional etatistical / econometric inference techniques. The Rudin causal inference framework requires either randomized controlled trials or for propensity score models an essentially unverifiable separate model step.

Google's CausalImpact model, despite having been featured on Google's AI blog, is a statistical/econometric model (essentially the same as [https://www.jstor.org/stable/2981553](https://www.jstor.org/stable/2981553)). It leaves it _up to the user_ to find and designate a set of control variables, which has to be _designated by the user_ to be unaffected by the treatment. This is not done algorithmically, and has very little to do with RNNs, Random Forests or regression regularization.

> If you were to write a pseudocode algorithm for an econometric researcher's > process of causal inference (and also their cognitive processes (as executed > in a NN with a topology)), how would that read?

[1] Set up a proper RCT, that is randomly assign the treatment to different subjects [2] Calculate the outcome diffences between the treated and untreated

For A/B testing your website, the work division between [1] and [2] might be 50-50, or at least at similar order of magnitudes.

For the questions that academic economists wrstle with, say, estimate the effect of increasing school funding / decreasing class size, the effect of shifts between tax deductions vs tax credits vs changing tax rates or bands, or of the different outcome on GDP growth and unemployment of monetary vs fiscal expansion [1] would be 99.9999% of the work, or completely impossible.

Faced with the impracticallity/impossiblility of proper experiments, academic micro-economists have typically resorted to Instrumental Variable regressions. AFAICT finding (or rather, convincing the audience that you have) a proper instrument is not very amendable to automation or data mining.

In academic macro-economics (and hence at Serious Institutions such as central banks and the IMF), the most popular approaches to building causal models in the last 3 or 4 decades have probably been 1) making a bunch of unrealistic assumpsions of the behaviour individual agents (microfoundations/DSGE models) 2) making a bunch of uninterpretable and unverifyable technical assumptions on the parameters in a generic dynamic stochastic vector process fitted to macro- aggregates (Structural VAR with "identifying restrictions") 3) manually grouping different events in different countries from different periods in history as "similar enough" to support your pet theory: lowering interest rates can lead to a) high inflation, high unemployment (USA 1970s), b) high inflation, low unemployment (Japan 1970s), b) low inflation, high unemployment (EU 2010s) c) low inflation, low unemployment (USA, Japan past 2010s)

I really don't see how a RL would help with any of this. Care to come up with something concrete?

<comment><USER6> > _What 's the point of dumping a bunch of Google results here? At least half > the results are about implementations of pretty traditional etatistical / > econometric inference techniques._

Here are some tools for causal inference (and a process for finding projects to contribute to instead of arguing about insufficiency of AI/ML for our very special problem domain here). At least one AGI implementation doesn't need to do causal inference in order to predict the outcomes of actions in a noisy field.

Weather forecasting models don't / don't need to do causal inference.

> _A /B testing_

Is multi-armed bandit feasible for the domain? Or, in practice, are there too many concurrent changes in variables to have any sort of a controlled experiment. Then, aren't you trying to do causal inference with mostly observational data.

> _I really don 't see how a RL would help with any of this. Care to come up > with something concrete?_

The practice of developing models and continuing on with them when they seem to fit and citations or impact reinforce is very much entirely an exercise in RL. This is a control system with a feedback loop. A "Cybernetic system". It's not unique. It's not too hard for symbolic or neural AI/ML. Stronger AI _can_ or _could_ do [causal] inference.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER19> "They were teaching their students big ideas. But they were ideas about what causes what — not about supply and demand."

Because supply and demand doesn't cause things? I think there are some citations needed for that claim.

Reading between the lines, I think the VOX article makes the "new approach" sound more stupid than it actually is. It is of course a good idea to test economic ideas with rigorous methods.

Article makes it sound almost as if the "new economics" was just "grievance studies" \- creating statistics about how disadvantaged some people are (as if society and economics have been previously unaware that such people exist). That would be stupid, because it doesn't teach you anything about what you could possibly do about it. But between the lines, the professor seems to conduct experiments to determine actual outcomes of economic measures. That makes sense. But you still need "normal economics" to come up with measures that have a shot at improving things.

</comment><comment><USER29> It's about time! The equilibrium theory of supply and demand has set economics back for decades. There is literally a century of criticism of the theory from postkeynesians, not only the lack of dynamism, but also all the strange assumptions.

<comment><USER30> People keep saying supply and demand is a broken theory, but I'm not sure how to interpret that.

When oil prices fall some producers seem to shut off their pumps. When restaurants raise their prices the people I know tend to cook at home more.

Do you think those are weird outliers, and increasing prices should spur increasing purchases as a general rule? Do you think consumer and producer behaviors have no relationship with price whatsoever?

I have a hard time imagining what those worlds would look like, so I expect your critique is far more nuanced.

But if it is a nuanced critique, then it seems we all agree on the general principles in broad strokes, and we don't need to throw out supply and demand after all.

<comment><USER31> The biggest problem is with the supply curve, really. Economics tends to pretend that more supply requires a higher price. In reality, more supply very often leads to a _lower_ price due to economies of scale.

This is a fundamental problem because it means that even if you assume fixed supply and demand curves (which is very dubious), there can be more than one equilibrium. That in itself pretty much invalidates most of the standard subsequent analysis.

The other big problem is that it assumes too much that all economic actors are price takers. In reality, prices are largely administered via cost-plus pricing, and advertisement and related tricks are used extensively to subvert the basic principles of supply and demand.

Another problem specific to macroeconomics is that it largely ignored the effect of demand. Increased demand very often leads to an increase in production rather than an increase of prices, a fact that was largely ignored in decades of supply-centred thinking. This has led to bad policies in response to the global financial crisis, for example.

I agree though that it makes no sense to throw out supply and demand as concepts entirely.

<comment> barry-cotter Newaccount456 has already covered your confusion over the difference between a movement of and a shift along the supply curve adequately. That entirely covers the more than one equilibrium point because even in Econ 101 they tell you the curves shift. Of course there are multiple equilibria. Increasing, decreasing and constant economies of scale will be in every intro textbook.

Re: Price taking or perfect competition, that's just one of the basic models of price determination. Monopoly, monopsony, oligopoly and monopolistic competition are all treated in introductory courses too and if you got as far as an intermediate course they'd cover the Bertrand, Cournot and Stackleberg models of oligopolistic competition and how they contrast with perfect competition and monopoly. Even introductory microeconomics will cover the three basic methods of price determination verbally.

Your penultimate paragraph also relies on not being able to distinguish between a movement of a demand curve and a shift along an existing one.

Everything you wrote is covered in intro micro.

<comment><USER29> > Newaccount456 has already covered your confusion over the difference between > a movement of and a shift along the supply curve adequately.

I think you're wrong, nobody is confused about shift of the curves. The problem really is, there can be multiple equilibriums even if the curves stay the same.

It is kind of difficult to see and understand why, because the traditional supply/demand theory and diagrams obscure this heavily. But if you look at the problem differently (see my other reference to Blatt), and start taking into account several products at the same time, you will see why.

<comment><USER31> This is exactly it. The single equilibrium hinges on the assumption about the monotonicity and slopes of supply and demand. But these assumptions are simply wrong, and so multiple equilibria are possible even with a single market. Multiple markets of course make the problem worse.

</comment></comment></comment></comment></comment></comment><comment><USER32> > Mankiw's textbook covers the abstract theory that underpins economics as it > has been understood for decades. It is about supply and demand, about how > prices can be used to match production of a good to its consumption, and > about the power of markets as a tool for allocating scarce resources.

Few things are as abstract as supply and demand.

</comment><comment><USER33> I've worked with lots of Harvard graduates over the years. They all seemed to be cloned with some kind of "group-think" about Economics. Essentially, they were all convinced that markets solved big problems, and anything vaguely Keynesian or Socialist or Marxist was doomed to failure.

So I guess I view Harvard Economics as a tool of those with power, and probably something which has contributed to creating a worse world. (Hey, I believe markets have benefits too, but not to the point of dismissing moderating policies.)

</comment><comment><AUTHOR> Vox article:

[https://www.vox.com/the- highlight/2019/5/14/18520783/harvard...](https://www.vox.com/the- highlight/2019/5/14/18520783/harvard-economics-chetty)

<comment><USER34> That gives more background, so we've changed the url to it from [https://opportunityinsights.org/course/](https://opportunityinsights.org/course/). Thanks!

</comment></comment><comment><USER35> It's hard not to be cynical when I see a link to vox.com, especially when my cynical assumptions end up being correct.

"Please don't be about revolutionizing economics via identitarianism... <taps link> ... sigh..."

<comment><USER36> Did we read the same article? The one I read is about changing how economics is taught by treating it more like a data-driven empirical science.

<comment><USER37> Keep reading. It goes so far as to hold Card up as some master of this new data-oriented movement - the guy massages it better than the girl offering you a happy ending. And it goes on to highlight other orthodox turned radical anti-capitalists.

Anybody skilled enough can take a sample of data and play with it enough - fill in holes with other data - exclude some points - until they get the story they want. To some extent there needs to be a mechanism (beside "people are stupid") to tie things together. And this has been the general MO of left-wing economists. They lost ideas and theory so they moved to numbers. The numbers tend to not be predictive because they are so overly played with, so obviously everybody is just being irrational - the world is wrong not their theory because they proved it right with the data.

It is Vox after all - a self proclaimed socialist website that has given page space to the the Jacobins on how Democratic Socialism isn't social democracy - it is only a starting point for pure socialism.

<comment><USER38> Card doesn't rely on irrational firms. He showed with data that labor demand is inelastic at minimum wage in the region studied. When this is true, the benefits of increasing minimum wage outweigh the costs.

<comment><USER37> On that infamous study he used phone calls that had very strange results like doubling number of employees or going to entirely fulltime. When the study was redone with payroll reports, the results reversed - back to the start minimum wage effect.

So then they republished and it was like "well if we only look at this group and we take out these and adjust this number" they were able to recover their results but the data hacking was pretty terrible.

If you ate going to write a study like that, relying on phone data screams bias (and these were two economists that couldn't claim they didn't know better).

<comment><USER38> Citation, please? Card's matching analysis has been repeated many times (e.g., Dube, et al), and each time, they have found little demand elasticity for labor at the extremely low end.

</comment></comment></comment></comment></comment></comment><comment><USER39> >That shift could change economics itself, by attracting a new breed of students who are intrigued by the field's new empiricism, not put off by its mathiness and high theory. It could make economics departments more diverse, and more open to new perspectives from women and students of color.

>He also gives Mankiw credit for moving the curve of Ec 10 to match the curves of other large Harvard classes, based on research showing that unnecessarily tough grading of economics classes disproportionately discourages women from taking them.

I do not understand how attracting members of underrepresented classes benefits them when standards need to be lowered to do so. Sure, the practice opens up opportunities to a more diverse sampling of individuals in the short term, but in the long term you devalue the credentials and, as bad or worse, you risk churning out graduates who are more qualified on paper than in reality.

I feel like this is a dangerous, growing trend in modern Western society and I do not understand how nobody seems to see it as a problem.

<comment><USER40> That's not what that quote implies. It implies that underrepresented demographics were less likely to enroll in classes that were known as having harsher grading curves than other classes. It doesn't imply that the standards for passing the class were lowered or the performance of underrepresented demographics was lower than non-underrepresented demographics once they enrolled in the course.

<comment><USER39> >It doesn't imply that the standards for passing the class were lowered

That's literally what it implies. The curve was relaxed. It became easier to meet the standard of a minimum pass grade for a given level of performance.

> or the performance of underrepresented demographics was lower than non- > underrepresented demographics once they enrolled in the course

Nor do I. But lower standards (easier to pass/earn a high grade) implies poorer qualification/ability.

Curves by definition bring up the grades of those who perform worse, without changing how they perform. In this example, if we peel away the euphemism, women were deterred from enrolling because the class was too hard to pass, so it was made easier to pass without changing the content. That does not necessarily mean that the women now taking the course are poor performers, but it does mean that on average the course will be graduating more students who would not have met previous standards.

<comment><USER40> >That's literally what it implies. The curve was relaxed. It became easier to meet the standard of a minimum pass grade for a given level of performance.

A curve can be shifted without changing the standards for passing the class. Eg the original curve allowed only for 5% of the students to get A's while the new curve allows 10%. You can get that additional 5% by shifting B's, C's, and D's upward which doesn't change the cutoff for failing with an F.

>Curves by definition bring up the grades of those who perform worse

Curves can also drag down students who are performing at a high level. If 10% of the class has performance deserving of A's but the curve says only 5% of the class can have an A, then some group is getting B's instead.</comment></comment></comment></comment>


Transforming NYC's Payphones into a 'Personalized Propaganda Engine' - <AUTHOR> http://www.villagevoice.com/news/google-is-transforming-nycs-payphones-into-a-personalized-propaganda-engine-8822938 <comment><USER2> Alphabet's NYC kiosks also have two cameras which are recording passers-by, [https://cryptome.org/2016/06/linknyc-spy-kiosks- installation...](https://cryptome.org/2016/06/linknyc-spy-kiosks-installation- videos.htm)

A dystopian "smart city" vision can be seen in the "Hyper Reality" video, [http://hyper-reality.co](http://hyper-reality.co) that is inspired by Magic Leap's AR, in which Google is an investor.

<comment><USER3> Or the Syndicate Wars intro sequence:

[https://www.youtube.com/watch?v=kGvIrf86g4Y](https://www.youtube.com/watch?v=kGvIrf86g4Y)

<comment><USER4> Or the _Self-Aware Colony_ Secret Project video from SMAC.

</comment></comment></comment><comment><USER5> I was walking down 3rd ave at 2am a week ago. I saw a homeless man with headphones listening to music on YouTube on a link. He had a little chair and was rocking out. It was beautiful.

We might give up a lot, but maybe those with nothing to give have plenty to gain.

<comment><USER6> I came here to say something similar. I've seen many people using them all hours of the day & night. People are watching music videos, charging their phone, getting directions, and surfing the internet. The people using them are often those that don't have the luxury (necessity?) of walking around with a nice smart phone and a big data plan.

Do they have ads on them? Of course. So do the bus shelters which are funded using a similar public/private partnership.

If you are worried about the data collection, don't use them or the wifi provided by them. But you are basically saying you trust Verizon/AT&T/T-Mobile/Sprint with your data more than you trust Sidewalk Labs/Google. Is that an informed tradeoff?

<comment><USER2> Google's business model is based on selling user-generated data to advertisers, unlike telcos who sell network data _to you_.

See Harvard professor Shoshana Zuboff's video on Surveillance Capitalism, [https://vimeo.com/110222526](https://vimeo.com/110222526)

NYC has the leverage to negotiate better financial terms for capturing the value of city residents data. The money generated _from_ harvesting city resident behavior can then be invested into city infrastructure and services for residents.

Data and networks are here to stay. What is open for negotiation is the split of revenue between city residents and vendors. For example, kiosks could be funded as neutral infrastructure that could be shared by competing vendors. A city itself is "shared infrastructure".

<comment><USER7> > Google's business model is based on selling user-generated data to > advertisers, unlike telcos who sell network data to you.

Really? Where?

I have worked on some large marketing projects. We'd _love_ to be able to buy user data from Google.

This facile argument is really getting old. When you get a free newspaper, does that mean you're getting "sold?"

<comment><USER8> Google generates most of its revenue from selling targeted ads to advertisers. A targeted ad is selling data about users.

Just because you aren't looking directly at the data doesn't mean you are getting information as a result of it.

<comment><USER7> > A targeted ad is selling data about users.

That's a lazy and incorrect causal link.

Google goes to great lengths _not_ to sell user data. It's far more valuable if they can hold on to it and use it to continue selling advertising.

It's important to understand the difference. This isn't a great metaphor, but imagine a top university. The professors at said university have accumulated many Nobel prizes. The university uses those Nobel prizes to sell classes to students. Does that mean the university is selling Nobel prizes? Obviously not.

Google uses data to target advertising. That doesn't mean Google's business is selling data.

<comment><USER8> Targeted ads are worthless if they aren't targeted based on correct data. Google would lose massive amounts of advertising if they didn't heavily base it on user data.

As an advertiser, Google's ability to target users based on user data is very important to me, so your analogy with Nobel prizes at University misses the mark by a mile.

If Google doesn't let me target users in a region with a specific interest, then I will just go to Facebook.

<comment><USER7> > Targeted ads are worthless if they aren't targeted based on correct data.

A university without well-respected and celebrated faculty is similarly "worthless." In fact, the gap in price between community colleges and prestigious universities might even be _greater_ than the gap between targeted ads and non-targeted ads.

> As an advertiser, Google's ability to target users based on user data is > very important to me, so your analogy with Nobel prizes at University misses > the mark by a mile.

As a student, the prestige of an institution and the prizes it's faculty receive is an important component of choosing to purchase education there.

There's a reason that college admissions brochures love to tout the number of Nobel prizes their faculty have received.

<comment><USER8> >A university without well-respected and celebrated faculty is similarly "worthless".

Not even close. As an undergrad you are getting ripped off if you choose a research university. As a student the product you are getting is an education, which has little to do with research quality.

Targeted ads depend entirely on accurate user data, so the analogy is a bit brain dead.

>There's a reason that college admissions brochures love to tout the number of Nobel prizes

Yes, to attract chumps. Anyone who does the minimum amount of research or thinking will quickly discover that nobel prize winners are approximately useless to undergraduate educations. Nobel prizes are the gold plating to the hdmi cables of education.

<comment><USER7> Your argument has devolved to insults. Goodbye.

Calling consumers "chumps" doesn't change the fact of their choices.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> _As he spoke, the screens shifted and a new message appeared: "If You See Something, Say Something. Be Suspicious of Anything Unattended."_

Okay, that's kind of creepy.

 _[Google] actually went so far last year as to shit-can its own "Don't be evil" admonition to its employees in favor of "Obey the law"_

No, they didn't[1]. Alphabet has a slightly more formal code of conduct[2], and it's not simply "Obey the law", it's "follow the law, act honorably, and treat each other with respect." Sloppy. Reporting using another news article as a source rather than just looking it up.

1: [https://abc.xyz/investor/other/google-code-of- conduct.html](https://abc.xyz/investor/other/google-code-of-conduct.html)

2: [https://abc.xyz/investor/other/code-of- conduct.html](https://abc.xyz/investor/other/code-of-conduct.html)

<comment><USER10> _" If you see something, say something"_ (spanish _" Si ves algo, di algo"_) is NYC's campaign in the aftermath of terrorism, and has been going on for over a decade. Its origins are not creepy and at this point most of the city's population is used to it. You can see it especially on any MTA properties such as subways, where they worry unattended luggage may contain a bomb, so they ask people to report such.

<comment><USER11> Yes the crazy thing is I remember going to London back in 90, and at that time in the US, most major stores would request that you checked backpacks in before going into the store, to prevent theft. I walked in to Tower records in Piccadilly Circus and handed my bag, expecting they would give me a claim check, and they were freaking out. As they had been sending out the same message at that time due to the IRA bombings. So they gave me the full instruction about unattended bags and how they were more worried over terrorism than theft. Crazy how it is common here 20 years later.

</comment></comment></comment><comment><USER12> > They're telling you that they can use your IP address, which is essentially > you, and all of the pages you've viewed or searched for, including how long > you visited them, to serve ads.

You don't bring your own IP when you use public WiFi, so this feels like a very confused argument.

<comment><USER13> You do however bring (and broadcast) your MAC address, which was mentioned as permissible to collect.

<comment><USER14> Which is pretty spoofable.

<comment><USER13> True, although that's hardly common and completely impossible on many embedded devices - devices which are only becoming more common.

<comment><USER15> don't iPhones do that by default? Not sure if it is common in Android-land by now, my old-ish phone required a custom ROM for that.

<comment><USER16> Sort of [1]. It also sounds like it's just for scanning networks. If you ever actually connect it still uses your real MAC.

[1] An older article about the feature that may be out of date: [http://9to5mac.com/2014/09/26/more-details-on-how- ios-8s-mac...](http://9to5mac.com/2014/09/26/more-details-on-how-ios-8s-mac- address-randomization-feature-works-and-when-it-doesnt/)

</comment></comment></comment></comment></comment></comment><comment><USER17> While I certainly understand the paranoia here (hell, I uninstalled PokemonGO due to the invasive permissions), I see the positive here far outweighing the negative. The Internet is becoming akin to fresh air or clean water—a right for all citizens, not just those with means. I don't know the percentage, but there are a huge number of New Yorkers who can't afford Internet and are missing out on the digital world we're all engaged in on a daily basis.

As a New Yorker, I heartily welcome a city blanketed in free wifi. For the 300Mbs I get off of one of those things, I'll use a VPN just like I do with all public wifi networks.

<comment><USER18> > I uninstalled PokemonGO due to the invasive permissions

They are fixing that.

[https://support.pokemongo.nianticlabs.com/hc/en- us/articles/...](https://support.pokemongo.nianticlabs.com/hc/en- us/articles/222648408-Permissions-update)

<comment><USER19> I think that went out yesterday.

</comment></comment></comment><comment><USER7> Funny, I don't see the Village Voice writing screeds about how terrible it is that NYC has a subway system or busses, even though advertising (ie. "propoganda") helps to underwrite their costs as well.

<comment><USER20> even more funny: the village voice is ad supported.

</comment></comment><comment><USER21> I really do hope that by now, everyone knows not to do anything they wouldn't want the public to know about on public WiFi, especially these systems. Using a VPN when you are on public WiFi such as this helps solve a lot of the privacy problems that you face using said service.

<comment><USER22> >I really do hope that by now, everyone knows not to do anything they wouldn't want the public to know about on public WiFi

What's your definition of "everyone"? Those of us in the tech bubble? Sure. Your everyday smartphone user? The internet's just the internet to them, and where it comes from is not a concern.

</comment></comment><comment><USER23> I'd really like to hear from a domain expert whether tor or something like it could a) drastically mitigate the impact of this type of surveillance and b) how likely it is that it (tor) could become mainstream (sort of like WhatsApp made end to end encryption mainstream)?

<comment><USER24> When you take into account the full range of tracking methods cited in the article, I think the impact of Tor would be limited. For example: you probably already have an app on your phone that has an SDK from Gimbal (beacon company in the article) or a similar firm that reads beacons and tracks your location passively. So while Tor would disguise the source of your web requests, the SDK still has free reign to send your location and your Ad ID back to the mothership.

<comment><USER25> We have at least five different kinds of problems with mobile device privacy here:

\- Apps that intentionally identify you and your location to _somebody_ (like apps that have permission to use location services, supposedly for some user benefit, and tell the app developer that location)

\- Apps that intentionally reveal your presence in a physical location to other devices nearby

\- Apps that unintentionally identify you to a network operator or wiretapper (because of unencrypted unique identifiers like cookies)

\- Aggregate device profiling because of a unique combination of observable behaviors (e.g., this person has this OS version and this combination of apps)

\- Observability of hardware and subscriber identifiers in the RF protocols

All of these are bad for privacy, but the way of fixing them is different.

Tor helps a lot with the third one. To deal with the others, apart from somehow getting that software not to exist or not to be installed on people's devices, it could be sufficient to make some of the software _not communicate on the network_ at certain times or in certain ways, like if the OS could say "maintain radio silence, except for Tor-aware apps".

For the device RF identifiers, we need the ability to change those identifiers, which is kind of sort of there for wifi on some devices (except there was just a paper showing it often doesn't succeed in protecting you), and not there at all for mobile network interfaces.

</comment></comment></comment><comment><USER26> This sounds eerily like the plot of Watch Dogs

[https://en.m.wikipedia.org/wiki/Watch_Dogs](https://en.m.wikipedia.org/wiki/Watch_Dogs)

<comment><USER27> Google has repetitively made their interest in running cities public, Larry Page talks about it often. While this article is primarily focused on the Wi- Fi efforts in New York City, Google has been working with other cities to replace management of public transit and parking enforcement.

<comment><USER28> Then they've chosen city services that definitely could stand for some drastic improvement.

</comment></comment></comment><comment><USER29> 400 mbs/200 mbs data rates in Manhattan on the Mac laptop where I've tried it. The iPhone 6s+ with the security cert. is about 185 mbs/195 mbs.

Not bad for free.

<comment><USER30> I pay 120 AUD a month for 35Mb/s down and a 1GB cap here in Brisbane, Australia :c

<comment><USER29> Well, may the (Google) force be with you in your future.

</comment></comment></comment><comment><USER31> For the alternate perspective: [https://www.link.nyc](https://www.link.nyc)

</comment><comment><USER32> I'd love to see a thoughtful article on what it took to get these kiosks installed, or what awesome tech is inside, or what the traffic looks like, or what lessons we can learn. How is tin foil hat-ism making the world better for those who are most benefiting from being able to access these kiosks?

</comment><comment><USER33> 'That LinkNYC is, ultimately, underwritten by Google should tell you a lot about why New York got so very lucky as to receive an unprecedentedly fast network of citywide public Wi-Fi — for "free."'

Except that none of this is free. Google gets all of the data collected by these kiosks and the ad revenue. Why was this allowed without any public say in the matter? The very public whose data was bought will also be sold by Google. What's the ad revenue worth on these? There are ads on each sides of these kiosks. These are effectively billboards on some of the most trafficked street on the planet. Does our society really need more advertising?

So did New York get lucky or did Google get lucky?

<comment><USER33> There is a really disturbing trend on HN whereby expressing anything negative about Google, gets down voted.

I've actually spent a fair amount of time around these kiosks as theres one a few blocks from me. I've actually observed it its environment, as part of the urban scenery and I've interacted with it a bunch. I think my comment has merit.

</comment></comment><comment><USER34> My new startup is building a system like this out for retailers. It has the added benefit of helping customer service personalize a shoppers experience as well as connecting digital ad spend (ads and landing page vidits) with foot traffic in both real-time and aggreated. Prototype is built and we are currently seeking investors and developers that would like to help us flesh this out. Pm me for details or questions.

<comment><USER34> FYI. Our system can also do "minority report" based ad retargeting. Also, we have a few large retail/auto dealers ready to deploy sooner then later. We are getting there but things could be moving much quicker with a little capital for additional developers, etc.

</comment></comment><comment><USER35> The village voice talking about another company being a propaganda engine is laughable. Also VV requires logging into Facebook to leave a comment on that story too.

</comment><comment><USER36> I don't know if I should be impressed or scared. Everything google^1 does in the last few years feels like they really know what they are doing, autonomous cars, everything they do about machine learning, now they are integrating that with fiber. It feels like Microsoft, if they would have nailed the browser and activeX in 95.

^1 I will use alphabet and google interchangeably, since I am not trying to do math here. If you are looking for a mathematical theory of world domination, just ask google's famously responsive customer service.

</comment><comment><USER37> A ++good piece. I particularly liked this astute observation by one of the article's sources:-

> "They are working hard to get you to behave true to your statistical > profile," Rushkoff says, "and in doing so they reduce your spontaneity, your > anomalous behavior, your human agency, as they try to get you to conform to > the most marketable probable outcome. When we're doing that en masse, to an > entire city - that kind of long-term manipulation is just astounding."

</comment><comment><USER38> If they're serving ads based on personal information, some people might have a little embarrassment heading their way.

Jokes aside, we all knew this was coming, right? It's not like your telecom isn't serving you up to the NSA already, and with this you know you're being watched. It's just driving VPN sales.

</comment><comment><USER39> Welcome To City 17

</comment><comment><USER40> The irony: "This website (villagevoice.com) wants to know your location"...

<comment><USER41> It would only be comparable if they didn't bother asking.

<comment><USER42> I don't think this has anything to do with the website asking instead of just collecting. The "wants to know" is based on browser settings to not just share your location without approval.

</comment></comment></comment><comment><USER3> I counted 21 cases of mostly targeted ads in that single article.

<comment><USER43> And what makes you think the author of that article has any control over the business model of the publication they're writing for? Are we not allowed to criticize ad tech online in ad supported venues? Because all of the most popular venues are ad supported. If you want a voice, you have to make compromises.

<comment><USER44> A digital ocean VPS is $5 a month. Is village voice.com really that big a draw?

<comment><USER45> Presumably villagevoice actually pay their writers?

<comment><USER7> Presumably installing free wifi kiosks actually costs money.

<comment><USER43> Yeah but is this something anybody asked for? The signal is always so great in NYC I never even want WiFi. This just seems like a largely pointless intrusion in service of selling more and more targeted ads.

<comment><USER7> Not everyone can afford (large) data plans. Look at some of the other comments in this thread for anecdotes of people enjoying the free access.

Also, your experience with service definitely doesn't match my own. There might always be a signal, but it's not always very fast...

<comment><USER43> Well this is in comparison to my home in rural Vermont where the signal is always very spotty. I realize people enjoy it but ultimately this was not voted on and thus saying "you asked for it therefore you must deal with the compromise" is a bit shitty. I think I'd enjoy it too if I didn't know the tradeoff, and even then gigabit WiFi would be pretty cool. However Google are not doing this out of the goodness of their hearts, if they were they would not put ads on it. Of course then the taxpayers would have to foot the bill, which would be severely unjust considering the undemocratic nature of this product's implementation in NYC.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER46> This is very hit-piece and I'm someone generally wary of Google.

Titan has owned and operated all of the city's remaining phonebooths for near 20 years. And I believe the MTA bus stop and subway advertising too. This is something they've already been doing.

Control Group developed those info kiosks you may have seen on subway platforms. They're good people that have got some cool tech. They even post in Who's Hiring here frequently (Hi, Ben!).

If this is something that the city wants, I don't see that there's another way that they would get it. The real problem that I have with LinkNYC though, especially in midtown where I live, is that all the terminals have become 24/7 campgrounds for the city's homeless. They charge their phones and browse the web there. There is no way that I would use one of these things now.

<comment><USER47> >They charge their phones and browse the web there.

What exactly is the problem with that?

<comment><USER46> With charging phones? Nothing.

But the reality is that one person is sat camped out with a chair in front of the thing for the entire day. Nobody else can use it. I've seen folks get in fights over these things already.

I spent part of my life homeless. I still see a major problem with how these things are being used right now. We shouldn't be encouraging people to be out on the street all day -- it doesn't solve anything.

<comment><USER48> What do you propose as an alternative? Charge them 25c for an hour of use?

I find it hard to blame the homeless (as you are) for taking advantage of a free perk, when it should be the cities and companies being held accountable for returning some portion of the profit back to those whom they are profiting from. Alphabet wants the information -- do you think they would have turned down a request to say that some nominal percentage of the revenue goes back to city social service programs?

<comment><USER46> Alphabet is paying the city money here.

The city should be kicking some of this money into infrastructure/services to help the homeless. Instead right now they're doing massive cuts and leaving people to the streets.

<comment><USER47> Aren't these kiosks an infrastructure/service to help anyone, including homeless?

Imagine some homeless spends a day at the kiosk and gets a remote job that requires an hour or so a day of online presence. And he/she can _actually_ perform this job thanks to this infrastructure. A couple of months later he/she spends another day at the kiosk and finds an affordable apartment (most likely not in NY). And Voila, we have -1 homeless.

<comment><USER46> Imagine all the homeless that can't be helped because the one angry violent guy you know that is HIV+ and likes to fist-fight is camped out there all day.

This is a real concern when you're homeless. [http://www.nydailynews.com/new- york/nyc-homeless-prefer-stre...](http://www.nydailynews.com/new-york/nyc- homeless-prefer-streets-violent-shelters-article-1.2564370)

Also, I'm talking about a real guy. He's frequently camped out at the LinkNYC terminals on my corners. (55th & 8th and 56th & 8th).

Of course this is HN where everyone is a self-made success and it's easier to pretend we care about the poor than actually care about the poor. I've been down this road here before.</comment></comment></comment></comment></comment></comment></comment>


Ask HN: What is the state of ESB? - <AUTHOR> Traditionally ESB is the centerpiece of an architecture. How relevant is it these  days. Do people still start grand projects to unite all systems and what are the tools and OS projects taking its place? Kafka is the 'cool' thing to build stuff with.I assume people can get a lot of mileage with pub-sub and use grpc where they can. What mordern OS tools are filling a similar gap? <comment><USER2> As a huge proponent of all things ESB related, I can confidently say that ESB is dead.

This is very unfortunate because I think we still need a way to do cross service mediation, process orchestration, etc.

That said, you will not see any new deployments utilizing an ESB. The fact that all open-source ESB projects are essentially abandon ware tells you all you need to know.

<comment><AUTHOR> Yea, its sad cause you have this cool middleware that can do so many things with the whole EIP theory and all the premade plugins.

What is eating this guys lunch really? and I think there is a void there somewhere thats not nicely filled currently. Some pain points that are exposed because of its demise. Thoughts?</comment></comment>


Pdf2htmlEX – Convert PDF to HTML without losing text or format - <AUTHOR> https://github.com/coolwanglu/pdf2htmlEX <comment><USER2> I've actually been using this to convert large PDF files to HTML to be displayed in-browser. It's for my work, so I don't feel comfortable posting a link to the demo instance here.

It is definitely the best solution I've found so far. The outputted HTML / CSS / images look almost identical to the source PDF. That being said, there are a few issues still:

* One Gigantic (600kb) CSS file from a single PDF

* Hundreds of individual fonts

* HTML semantics are non-existent

These are all relatively easy to fix, I believe. I have found my own solutions to most of the issues in post-processing.

Kudos to you, coolwanglu. Also, I'd like to get in touch with you about lending a hand to fix some of the issues I've encountered.

Thanks for a cool piece of software!

<comment><AUTHOR> Hey thanks for the info!

2nd & 3rd are in the future plan, as I'm still working on accuracy and speed. And #115(<https://github.com/coolwanglu/pdf2htmlEX/issues/115>) is about the 2nd issue.

About the first one, I've not got an elegant solution yet, maybe a CSS file per page?

Please file new issues at GitHub if you think it's necessary :)

<comment><USER3> I love this! Kudos for this awesome app.

</comment></comment></comment><comment><USER4> Can anyone recommend an equally good opposite (HTML to PDF)?

wkhtmltopdf [0] is probably the most popular, but it's also ridiculously buggy.

0: <https://code.google.com/p/wkhtmltopdf/>

<comment><USER5> <http://phantomjs.org/> is the best so far in my experience since it handles all the client side javascript properly.

The PDF's it outputs are full vector not just rasters, it the same engine used in Chrome to view PDF's and print web pages from my understanding.

<comment><USER6> We've tried everything, including PrinceXML, and PhantomJS has been the best for us so far.

</comment></comment></comment><comment><USER7> This works and displays correctly, but is unbearably slow on iPad 2 whereas the PDF loads instantly. What is the point then or does it work a lot better in desktop browsers?

<comment><AUTHOR> I heard that with careful optimization on the server side and a clever JS may solve this. So far the default UI just demostrates the ability of reading- while-downloading.

The idea is that now the document becomes more controllable and accessible, say you can put Google Analytics in your resume written in LaTeX; or maybe an social reading service, where you can comment, annotate and share.

Unlike PDF viewers, web browers are never optimized for this kind of messy inputs. The next version of pdf2htmlEX will be focused on optimizations, e.g. smaller size of background images, hopefully that would help.

<comment><USER8> > social reading service

I truly wish there was at least one ground that hadn't been touched by "social" crap.

<comment><USER9> Porn?

<comment><USER8> Nope.

</comment></comment></comment></comment></comment><comment><USER10> Interesting. So it converts all vector graphics to a background image per page, but keeps all text as browser-rendered on top of it.

I guess I don't really see much practical purpose for it -- most browsers these days seem perfectly fine opening PDF files natively, after all. But it's a very cool technological demonstration.

Maybe this could be some kind of bridge tool for generating sites with fancy typographical layout? You could use Adobe Illustrator etc. to do fancy column work, drop caps, hyphenation, all that jazz -- and then "render" into HTML. It would certainly be as anti-"responsive" as you can get, but it would certainly have the ability to generate more advanced typography much faster than you can produce with HTML/CSS by hand.

<comment><USER11> As a practical purpose, how about being able to edit a PDF document? I understand that it can be done through some other tools, but this is one more - and would be free and easy.

Convert to HTML -> Edit -> Print back to PDF (if needed)

<comment><USER12> I'm not sure the html will be clean enough to edit, sadly...

</comment></comment></comment><comment><USER13> I do this almost daily. I use a PDF converter driver found on the internet . Install it and it becomes a selectable converter option.Then you can convert PDFs to many forms in any program at all, including Adobe Acrobat . Just open a PDF, select convert, and choice a form you want, the task will be finished in several seconds. if you haven't found a good choice , you can have a try. best wishes. [http://www.rasteredge.com/how-to/csharp-imaging/pdf- convert-...](http://www.rasteredge.com/how-to/csharp-imaging/pdf-convert- html/)

</comment><comment><USER14> Question, does your public folder periodically delete files? I accidentally uploaded something confidential and it seems to be gone. I was wondering if this was a manual deletion or just expired since I still see files that were uploaded around the same time still there.

</comment><comment><USER15> Can't Mozilla's pdf.js be used to get the same result? Great results anyway!

<comment><AUTHOR> You don't want to rely on the computing power at the client side, do you? :)

<comment><USER16> I guess one possible setup would be pdf.js running on server-side and having its output captured. One advantage of this, from what I can see, is that there would probably be fewer external dependencies than this setup.

<comment><AUTHOR> Yes, actually they had this kind of plan, but I am not sure how it has been going.

It would be definitely interesting in that way, but in that case it may not be worth it to rewrite everything in JS.

</comment></comment></comment></comment><comment><USER17> Promising start. Hopefully performance improves with each release.

<comment><AUTHOR> Right, that is in the schedule, just heard enough complains, in a good way.

</comment></comment><comment><USER18> I didn't see any mention of tables in the doc. Does this means it's outside of the "good enough" range? Table extraction would be a great feature.

<comment><AUTHOR> It's still a startup, so currently it's focused on accurate rendering, and fast speed(which is not achieved yet so far).

Features about recognition would be planned in the future, usually PDF viewers do not recognize too many things, do they? :)

</comment></comment><comment><USER19> How did you manage to get Mediafire to host your demo?

<comment><AUTHOR> MF uses pdf2htmlEX :) And it also provides public folder and public dropbox <\- I really like that.

This means that you can create one of your own.

</comment></comment><comment><USER20> 路过拜大牛

<comment> v-yadli 路过拜大牛 +1</comment></comment>


Best Consulting Gig Ever... - <AUTHOR> http://joedantoni.wordpress.com/2012/11/20/the-best-consulting-gig-ever/

<comment><USER2> So they want someone to work on performance problems for free for a week? And then pay $24/hr afterwards for someone who knows all major RDBMSs? I think they might just get what they pay for.

</comment><comment><USER3> In southern europe that'd actually be a fair offer.</comment>


IPhone 4 no longer in the top 5 selling smartphones in UK - <AUTHOR> http://www.unthinkable.biz/home/article/2331/apple-thrown-out-of-top-five-barrel-in-uk <comment><USER2> This is the top ten list of what? Popularity? What do they mean by that? Where are the numbers? Where did uswitch.com get the data from? Manufacturers? Distributors? Network operators? A web poll? Some other methodology?

<comment><USER3> It's a perfectly valid concern, but the way you write it, it sounds like a panicked frenzy...

[http://www.uswitch.com/news/communications/apple-drops- out-o...](http://www.uswitch.com/news/communications/apple-drops-out-of-top- five-in-best-mobile-phone-chart-340182/) is the original press release, I believe. There's precious little mention of methodology on the entire uswitch.com website, apart from "live searches and sales". Could it simply be the outgoing links from the uswitch website?

<comment><USER4> Notice that the iphone is not linked on the front page of <http://www.uswitch.com/mobiles/>. This would indicate it's not a target market for people using uSwitch.

If you put in any search criteria the iPhone scores lower as it is more expensive (any search for <£30 p/m it doesn't feature, anything for more it is consistently 8th or worse).

This 'survey' is a total farce.

The linked article claims top 5 _selling_ which is totally and utterly wrong. It seems the author can't read. And also doesn't even link to the source article from uswitch.

</comment></comment></comment><comment><USER5> How large is the timeslice on this? Apple releases the iPhone in the summer like clockwork. Lots of people buy during the initial push and follow-on buyers peter out in the period just before the next release. I think Apple even acknowledged this in their last earnings call, but I could be wrong. Unless you absolutely needed an iPhone right now, you'd want to hold out from February - May.

Apple's annual obsolescence for their consumer products isn't a secret to the mainstream, either. It's almost a running gag.

</comment><comment><USER6> Is Galaxy S one smartphone in the UK, or a family of phones like here in the US? If it's a family, it should be broken out as their constituent phones. In which case the iPhone will likely be #5.

<comment><USER7> It is only in the US and (IIRC) a couple of far eastern markets where the galaxy S is mucked about with by the networks. In the UK and elsewhere in europe we just get the stock galaxy S as samsung intended it.

</comment></comment><comment><USER8> I'm an Android fanboy, but I can't stand these types of posts.

</comment><comment><USER9> Interesting error message: "Could not connect to localhost"</comment>


Show HN: The Orbital Index Reaches Issue #40 - <AUTHOR> Hey HN,
I started Orbital Index 40 weeks ago based on feedback that I received on HN (https://news.ycombinator.com/item?id=19227434) and I've been sending it out weekly ever since. The response has been great, and readership just hit 2500 people. I just wanted to say "thank you" to HN for the motivation to start this, I've learned a ton.
Here's issue #40! https://orbitalindex.com/archive/2019-11-26-Issue-40/
-Andrew <comment><AUTHOR> Here's issue #40: [https://orbitalindex.com/archive/2019-11-26-Issue-40/](https://orbitalindex.com/archive/2019-11-26-Issue-40/)</comment>


Why having an MBA is a negative hiring signal (for startups) - <AUTHOR> http://daltoncaldwell.com/why-having-an-mba-is-a-negative-hiring-signal

<comment><USER2> Can't have this without quoting Kawasaki's Valuation Formula for startups: Value = (500K * Engineers) - (250K * MBAs)

<comment><AUTHOR> I would have put that in the post, but it seemed too snarky.

I also considered linking to this: <http://whartoniteseekscodemonkey.tumblr.com/>

:)

</comment></comment><comment><USER3> Even as an MBA I agree with his thesis that the degree is a negative signal... however that's only true for the early stage tech-based startups. And while that subsection might be 99% of the HN crowd, as a whole for the US it's a pretty small chunk (by count).

I love the notion that MBA's get the degree to get rich. In reality most folks at the top tier schools do the analysis of comparing opportunity costs and realize they may not recoup the investment for 10/15 years! The reason most people get the degree is to switch careers, learn new skills, and build their network.

I'm a bit biased here at UC Berkeley, but I'm actually inspired to see all the startups come out of our class!

</comment><comment><USER4> I just feel like we should caution against stigmatizing large swaths of anyone, be them technical v non-technical, MBAs, Berkly CS v Standford CS, Vegetarians v meat eaters, or whatever the case may be. If there's one thing that's been proven by startups it's that it's all about finding exceptional individuals, in whatever shape or form they come in. I know there are qualifications at the beginning of the post, but I still feel like point number 2, "Entitlement" can be roughly translated to "MBAs are entitled, overvalued egotists". While I agree this happens to be the case sometimes, I do not have an MBA but did do business undergrad so I know the type, the best people will always rise to the top. So while I totally agree that hiring an MBA just to have an MBA is a terrible idea, there will be MBAs who provide startups with INVALUABLE help because of who they are as individuals. Just like hiring a Stanford CS major doesn't automatically mean you hired the best possible programmer, hiring the best possible programmer does that. Hiring based upon intelligence, perseverance, creativity, know how, how someone fits into the team, etc. etc. will put any startup in the best position to be successful. And those people come from everywhere. They may not have even finished college, maybe they worked in a completely unrelated field, maybe they traveled for a while, or decided to give professional archery a try. If they're smart, they're smart. If they fit in, they fit in. No need to limit where startups should be looking for people, its already tough enough to find amazing talent.

<comment><AUTHOR> um did you read the first paragraph of the post?

</comment></comment><comment><USER5> Pretty straightforward, reasonable argument. Under opportunity cost (or another heading), I'd include the cash outlay for business school which can be $50-100k, plenty of money to fund 18 months of starting something up.

</comment><comment><USER6> Being an MBA and having spent a lot of time with a large cohort of the same, the "Entitlement," "Misguided Thinking,"and "Selection Bias" critiques sound a lot more like the caricature that's made of MBAs by non-MBAs (or self-loathing MBAs trying to impress non-MBAs) than the vast majority of real people who have the degree. If this represents the majority of the MBAs you've met, then I fear you have an unrepresentative sample. We had those folks, for sure. But they were the dorks and the minority.

<comment><USER7> In my anecdotal experience (albeit typically in larger corporate environments), many of those possessing or actively pursuing MBA's are or present themselves as fairly sincere. Nonetheless, they often simply have a _perspective_ and approach that is not very useful for the problems at hand. Or they go about solving the _wrong_ problems. Things get all (re)organized for a year or two, and perhaps the activity looks progressive. Perhaps it somewhat masks underlying circumstances.

But... moving dollars and org around does _not_ , of itself, motivate nor solve production problems.

And a lot of those same people seem to embrace typical "HR best practices" in a rather claustrophobia-inducing fashion.

Mine is just a rather narrowly informed anecdotal experience. But many of the MBA types around me have not/hardly inspired me.

</comment></comment><comment><USER8> I hope I can be the outlier. I'm doing an MBA but I'm also working and whenever I do have free time, I try to get some coding done for some project. I disagree that strategy > implementation - I believe they both go hand in hand but slightly more to implementation. I don't think I'm entitled to anything but I do think that I have learned some things of value in school.

However, I do agree that there is an oversupply of MBAs who do not actually add value. I won't be one of them though.

</comment><comment><USER9> Maybe run an A/B test?

Suppose 1) you had 1,000 resumes of near identical startup-ready backgrounds (top tier CS, design, entrepreneurship experience etc.), and 2) of those resumes 10 also had top tier MBAs. Which resumes would you prioritize reading?

If you believe an MBA is a negative hiring signal, then you'd read 990 other resumes first--which doesn't make sense to me. I may be missing something...

</comment><comment><USER10> Is it ethical to leave the MBA off of the resume? For some of us, it was simply a box to check in order not to stunt our future growth.

<comment><USER11> Then you'd have to explain the 2 year gap...

</comment></comment><comment><USER12> Interesting observations around strategy and implementation. I agree with the reasoning behind the idea that at a startup, implementation > strategy... but it does still keep these notions separate. Is it going to far to say that implementation = strategy = idea? Do successful startups actually draw a sharp distinction between these things?

</comment><comment><USER13> IME, this isn't only applicable to start-ups. The same considerations need to be taken into account when filling a senior professional or mid-level management role within a large enterprise. Doers verus Thinkers.

</comment><comment><USER7> Making > managing.

And/or, making precedes managing.

I've seen, as a small minority of my experience, some very good management. But, that good management always _understood and respected its context._

</comment><comment><USER14> I am reminded of an anecdote I read years ago in a book by an entrepreneur. He related a conversation he had with another entrepreneur who did happen to have an MBA. Mr. MBA had a lot of fancy numbers and analysis written up but no customers nor any real idea how to get them. The author had customers and revenue. While he envied the MBA his education and wished he had similar analytical ability/tools, he also took it as a "count your blessings" moment and did not expect the MBA's business to make it.

I am also reminded of all the business books I have read which did not result in a business being born. I have had a business bank account, registered business names, written business plans, etc. It was all style and no substance. It was all window dressing. I am still trying to figure out what spark turns into flame in this regard.</comment>


Linuxfoundation.org also hacked - <AUTHOR> http://www.linuxfoundation.org/

<comment><USER2> My bet is we are going to see more and more of this, as fallout from the kernel.org crack.

Kernel.org said "Don't worry about linux, the source tree is in git and tamper-proof. All they messed with was SSH. It was amateur, really." (Some paraphrasing.)

Well, a modified SSH could easily log interesting details that pass through it. So if you used [ed: gpg] private key forwarding, the crackers have your private key. [ed: See [http://www.unixwiz.net/techtips/ssh-agent- forwarding.html#se...](http://www.unixwiz.net/techtips/ssh-agent- forwarding.html#sec) for the SSH vulnerabilities.]

The only question is how fast the attackers have moved. Blitz all the servers at once, or try to carefully lay something individually tailored and undetectable. It's been long enough for either.

edit: Erroneous simplification, sorry. The attacker could imitate you on the remote system. This is not the same as having your ssh private key (my bad) but the result is the same. The third party server you connected to through kernel.org is compromised.

edit edit: But check your gpg keys! Gpg signing does require the full private key on the remote system. If you signed any files on kernel.org with forwarding, they could have your gpg private key. (Though this might need modification to gpg, which was not mentioned by kernel.org.)

<comment><USER3> Is that true? I didn't think private key forwarding was vulnerable to a MITM like that.

<comment><USER2> Sort of. See [http://www.unixwiz.net/techtips/ssh-agent- forwarding.html#se...](http://www.unixwiz.net/techtips/ssh-agent- forwarding.html#sec) for more details.

I left out a few steps, but at the very least the system you connected to through kernel.org could be compromised.

</comment></comment></comment><comment><USER4> Who stands to gain from an attack on linuxfoundation.org and kernel.org?

<comment><USER3> My guess is the same people who stand to gain from knocking over mailboxes.

<comment><USER5> Driver's ed students?

<comment><USER6> He's probably referring to people who purposely whack mail boxes with baseball bats out car windows.

So presumably drunk highschool football players.

</comment></comment></comment></comment><comment><USER7> kernel.org has been down now for three or four days now. Seems there is more to this hack than we've been told. For awhile kernel.org just didn't respond to requests, then they put up this maintenance page:

<http://kernel.org/>

</comment><comment><USER8> Quote: "...you should consider the passwords and SSH keys that you have used on these sites compromised. If you have reused these passwords on other sites, please change them immediately"

Did they really store clear text passwords? Or perhaps I'm misinterpreting the announcement?

<comment><USER9> It's unlikely they stored plaintext passwords. That doesn't mean that the attacker couldn't crack the hash.

More importantly, if the hacker modified the ssh binary then they could make it retransmit passwords before hashing them. Once the attacker has such a high level of access to the system it doesn't matter if the system has otherwise sound security; that security can simply be removed or altered.

I think the announcement is made as it is because it's far safer to assume the worst than the most likely or best. Even if your keys and passwords aren't compromised it doesn't hurt to change them.

</comment></comment><comment><USER10> Linux.com is also down with the same message.</comment>