Using AI to design board games - <AUTHOR> http://boardgamegeek.com/blogpost/6931/ai-game-design-the-shibumi-challenge <comment><USER2> From the article: "Should the mantle of 'creator' lie with the program or the programmer?"

Strangely enough, this was the exact dilemma that Clarissa faced in in Episode 29 of Clarissa Explains it All, "Poetic Justice," in which she developed a program that generated poetry. When her program's poetry won the school's contest, she decided to publicly relinquish her award to her computer. [http://en.wikipedia.org/wiki/List_of_Clarissa_Explains_It_Al...](http://en.wikipedia.org/wiki/List_of_Clarissa_Explains_It_All_episodes#Season_3:_1992.E2.80.931993)

<comment><USER3> What an obscure reference to use. That show aired from 1991 to 1994, when I was a mere toddler.

<comment><USER2> As far as pop-culture goes, I think Clarissa still holds the cake for female programmer characters. Acid burn (from Hackers) is a close second...

<comment><USER4> My winner is Veronica Mars's Mac whose scheme to get a new car was downright evil genius (scalable too, but she chose to limit herself). Of course, Burn was right, RISC changed everything, though not yet in desktop PCs.

</comment></comment></comment></comment><comment><USER5> Interesting application of genetic algorithms. The weird thing about GA though, and any optimiser, is that you need an "error function" that you will us to determine when one solution is better than another. Presumably he designed an error function for "fun", and his GA system finds local minima within it. But this is what I mean by "weird thing": you have to come up with a quantification of what makes a game fun, and _this_ is ultimately what will determine what kinds of solutions you come up with. Regardless of how the optimiser works, the error function is what the programmer is designing, and therefore the programmer is designing the space in which solutions will exist, so I would argue that any specific games are attributed to the programmer (or whoever designed the error function), and any specific game just happens to be one choice within the available search space.

Unless you want to manually try every iteration and rate it from 1 to 10, this implies that it's possible to come up with a "funness" model of human game playing. That in itself is sort of a interesting thing, falling somewhere between philosophy and psychology, rather than computer science.

<comment><USER6> Given enough money, you can outsource the error function to mechanical turk. Or to go more meta (and possibly cheaper): Make designing a good error function part of a meta GA-run, and let those functions be judged by comparing their judgements with human judgements.

<comment><USER7> Given the effort the play a game and the inter-subject noise in a numerical judgement, I think a better use of Mechanical Turk would be to ask the player to do some blame assignment. That is, instead of rating a game as 4-stars, they could perhaps give a thumbs up/down to particular rules or sets of rules in abstracted representation of the game.

This kind of feedback extracts several more bits of information from the player than a single rating (making better use of them). However, it breaks the applicability of an evolutionary algorithm which treats both artifacts and fitness evaluation as black boxes. If you use a search algorithm that was aware of how the game was built from components, I'm guessing that component- level feedback (being both more objective and more specific) could provide more informative pressure to drive the algorithm than the standard interactive genetic algorithm setup gets.

<comment><USER6> Yes, I also thought of giving multi-dimensional feedback, instead of just a single number, when writing the comment.

</comment></comment></comment></comment><comment><USER8> This paper, "Automatic Design of Balanced Board Games" [2007] might be of interest for similar reasons: <http://www.aaai.org/Papers/AIIDE/2007/AIIDE07-005.pdf>

<comment><USER9> Thanks!

I've been kicking around the idea of using evolutionary algorithms to test gameplay balance, and it's cool to see that someone else is already working on it.

<comment><USER7> Before you jump to evolutionary algorithms, note "The Problem with Evolution" section in the OP. This was written by a guy who knows what he's doing with evolution, and it's interesting to see him turning to MCTS given his previous success with evolution.

The gist is that games are fragile with respect to mutations (motivating the need for a search paradigm not based on gradualism). This fragility may not exhibit itself as strongly for the parameter tweaking involved in the balancing problem, but balancing quickly shades into general game design when you start to consider non-trivial tweaks.

<comment><USER9> Thanks for pointing that out.

The project I was thinking of was going to be a tower defense game, and I wasn't planning on trying to develop any rules automatically. Rather, I was thinking in terms of using an evolutionary system to pick weapon and enemy strengths that lead to balanced gameplay.

So, I would probably get away with it, but I'm wide open to different ideas.

</comment></comment></comment></comment><comment><USER10> A section of the article says this: _Raf describes two requirements for serendipity to occur: 1\. Active searching: The designer should not simply wait for inspiration to strike, but should immerse himself in ideas and look for harmonies between them..._

It reminds me of one of my favorite quotes from Picasso:

"That inspiration comes, does not depend on me. The only thing I can do is make sure it catches me working." ― Pablo Picasso

</comment><comment><USER11> The principle of subset contradiction (as in Yavalath, where you win by making four in a row but lose if you make three first) is extremely interesting, and it also seems like it can be applied in a fairly simple, atomic way to various types of rulesets. I wonder if a Monte Carlo ruleset search system could integrate higher-level rule change "principles" such as subset contradiction, and apply them as operations to existing rulesets...

</comment><comment><USER12> Not unrelated is Kevan Davis' _Ludemetic Game Generator_ (2003), which randomly combines categories and mechanics from those at BoardGameGeek to create new (and largely useless) game ideas, with arbitrarily appropriate titles:

<http://kevan.org/ludeme>

Some of the random games sound more fun than others. ;)

      Game: "Indkub"
      Categories: Industry / Manufacturing, Comic Book.
      Mechanics: Set Collection, Hand Management.

      Game: "Ugplay"
      Categories: Prehistoric, Trains.
      Mechanics: Acting, Trading.

</comment><comment><USER13> Great thoughts and very well explained also - and neat to see BoardGameGeek get linked on HN - been a BGGer for many years.

</comment><comment><USER14> A friend of mine implemented the GA-generated game Yavalath for Android. You can find it on Android Market at <https://market.android.com/details?id=boardgamer.yavalath>.

Yavalath has surprising depth, but it is no BattleField 3.

It is nice to see BoardGameGeek listed on HN.

</comment><comment><USER15> Hmm i recall Trillion Credit Squadron back in the 80s where the use of computers was almost required to play and being at the bleeding edge of Ai research would be useful

<comment><USER16> Thanks for providing the name, I've been trying to remember that story for a while now but my google-fu was weak and I couldn't recall any name involved (person, program or game).

I think you're refering to Dr. Douglas Lenat's Eurisko program. He applied it to the game Trillion Credit Squadron and won two years in a row with fleets the other participants scoffed at. The first year he/Eurisko won by creating a largely stationary fleet that could take enough damage to survive long enough to destroy its opponent. It exploited a damage rule in the competition. The damaged party got to select which component/subsystem was damaged. Eurisko constructed a fleet with many small, useless components which could be individually destroyed without impacting the effectiveness of the crafts themselves.

Edit: Reading more about it apparently the primary advantage was the shear number of craft. Essentially a fleet of kamikaze craft that could overwhelm the enemy.

<http://en.wikipedia.org/wiki/Douglas_Lenat>

<http://en.wikipedia.org/wiki/Eurisko>

<comment><USER7> I'm in grad school for AI right now as a result of reading about Eurisko as a kid _. Though I haven't followed up on the greater vision, my thesis proposal a few years ago pitched the project of building a Eurisko-style discovery system for game design:[http://users.soe.ucsc.edu/~amsmith/proposal/amsmith- proposal...](http://users.soe.ucsc.edu/~amsmith/proposal/amsmith-proposal- talk.pdf)

_ Also, the fact that I had a good experience playing an "academic" class character in Traveller during college was also an influence in deciding to go to grad school. I haven't seen this class in any other RPG.</comment></comment></comment>


Social Strand Turns Your Facebook Profile Into an Infographic - <AUTHOR> http://mashable.com/2012/03/06/my-social-strand/

<comment><AUTHOR> _"... Once you connect the Facebook app and answer a few questions, your personalized infographic will feature information such as how your social sharing behaviors compare to your Facebook friends', how your interests compare to other respondents, how long you've been alive, the most popular song the day you were born ..."_

It might also uncover things you might not want others to know. There must be a better way to dissect someone than a bunch of stats & comparisons?</comment>


Real world Clojure - <AUTHOR> http://tech.puredanger.com/2011/10/20/real-world-clojure/ <comment><USER2> I saw he mentioned switching to Emacs. Is the emacs - clojure workflow really as good as it's made out to be, as in good enough to both switching from Vim? I was planning on getting into Clojure this weekend, and if it's actually worth doing I suppose I could learn Emacs at the same time, but I've always been turned off by the weird key chords.

<comment><USER3> The last time I used it, SLIME integration for Clojure was very slick. It was a while back so there were a fair number of moving parts to align, but I'd guess that has gotten better.

It is worth doing, especially if you've never used another integrated Lisp environment, but I wouldn't recommend laying it on top of learning Clojure. I'd say learn Clojure first, then try out emacs and the emacs integration.

<comment><USER4> Slime and Clojure have improved, mainly through clojure-jack-in and ritz. Of course the Common Lisp Slime experience is still better but the essentials are definitely there with Clojure.

There are several heavyweights in the Clojure community that use VIM and have developed good plugins, and Rich never seems to be using SLIME during presentations, so you can definitely get by fine without it.

<comment><AUTHOR> Agreed. There are many ways to fly. I'd say Emacs is well-developed and probably the most common, but certainly not the only choice.

</comment></comment></comment></comment><comment><USER5> It would be interesting to see an explicit example of

"In the area of query planning and optimization, I found that at some point I hit a wall with what I could do in Java. There was latent abstraction that I understood but could not express in the code."

<comment><AUTHOR> Unfortunately, this is one of things that I have found very challenging to convey due to the scope of the code base.

</comment></comment><comment><USER6> "I don't find that using Clojure making the overall process of writing new code faster. My thinking/typing ratio is much higher though. I think the reason is that I have a huge confidence in Java refactoring tools and my abilities to morph the code towards where it should go. In Clojure, if you start writing code there is no ceremony and within two minutes you realize that your first idea was dumb and your data structure should be totally different and it drives you back into the thinking phase."

I'm not sure that convinces me to try Clojure. To me less typing is great, but only if it still makes sense and I don't have to come at it a few ways before I get it right. That is probably why I never took to Perl. It just looks like garbage and I couldn't get past that.

<comment><USER7> > I don't find that using Clojure making the overall process > of writing new > code faster

He's talking specifically about writing _new_ code. I think the implication here is that Clojure encourages more consideration up-front, resulting in designs that are easier to modify down the road.

Given how much time is spent maintaining code vs writing new code this seems like a reasonable trade-off.

<comment><USER8> I find that with a REPL in hand, I tend to play with functions while letting my mind almost drop into the problem space.

So I do less up-front design, but wind-up with somewhat better code at the end.

The other weird effect is that I find that this helps me think better about coding in something like C as well. A little experimentation at the REPL helps me understand pieces of the solution better.</comment></comment></comment>


Ask HN:Xcode on new mac air with SSD? - <AUTHOR>

I was considering getting the new Macbook Air with SSD.  Does anyone have one running Xcode? If so how is the performance?  Any Pros/Cons? <comment><USER2> I use a MBP with an SSD for iPhone dev. The only difference is the screen size; 15" vs whatever size you have for the Air.

Xcode runs much faster now. Its still going to be the slowest (and largest) application you'll load (besides iTunes). But, it doesn't feel as heavy anymore.

Builds are _much_ faster for me; one project used to take > 2m30s to build and now builds in just over 1m (between getting an SSD and switching from gcc-4.2 to clang).

<comment><AUTHOR> Thats great info? BTW have you tried Windows in a VM on it?

<comment><USER2> _Thats great info?_

I wasn't quite sure what you were asking, so I went with a generalized response. If you want more info (Xbench, etc), please ask (here, or in email if you prefer). I figured since you were asking about Xcode and the SSD, rather than the screen size specifically, I might be able to answer your questions.

 _BTW have you tried Windows in a VM on it?_

Only briefly. I spend more time running a VM of Slackware. I don't notice any slowdown in using it and it does seem faster than before. But, I haven't done any benchmarking (since I don't have a "before" to compare it to).

</comment></comment></comment><comment><USER3> I use a 2.4Ghz MB for iPhone dev. I'd love to have a smaller laptop with an SSD, but my games are very, very shader heavy. They run fine on the iPhone4, but the software gpu emulation in the simulator chugs on my machine. I don't expect that 1.6Ghz will be usable for my situation...

For everyone else, the SSD should more than compensate for the old CPU. My Win7 box at home has an Intel SSD and it is like butter.

</comment><comment><USER4> Just got mine today. Very interested in this.</comment>


Data Science: Reality Doesn't Meet Expectations - <AUTHOR> https://dfrieds.com/articles/data-science-reality-vs-expectations.html <comment><USER2> > Moreover, you may quickly realize much of this work is repetitive and while > time-consuming, is "easy". In fact, most analyses involve a great deal of > time to understand the data, clean it and organize it. You may spend a > minimal amount of time doing the "fun" parts that data scientists think of: > complex statistics, machine learning and experimentation with tangible > results.

This. Universities and online challenges provide clean labeled data, and score on model performance. The real world will provide you... "real data" and score you (hopefully) by impact. Real data work requires much more than modeling. Understanding the data, the business and value you create are important.

As per #6, better data and model infrastructure is crucial in keeping the time spent on these activities manageable, but I do think they're important parts of the job.

I've seen data science teams at other companies working for years on topics that never see production because they only saw modeling as their responsibility. Even the best data and infrastructure in the world won't help if data scientists do not feel co-responsible for the realization of measurable value for their business.

Training integrative data professionals could be a great opportunity for bootcamps. Universities will (understandably) focus on the academically interesting topic of models, while companies will increasingly realize they need people with skills across the data value chain. I know I would be interested in such profiles. :)

<comment><USER3> I took a data visualisation class in uni that handled this really cleverly. The second assignment sounded very easy. The teacher provided links to the sources where we could find data.

Most people figured that with such a simple assignment (not significantly harder than the first one, which was also easy-ish) they could put off doing it until the last moment.

Most people failed.

This real world data needed hours upon hours of cleaning before it was in any way useable. Of course, the teacher knew this, gave bonus points to the ones who did start in time, and then extended the deadline as he had expected to from the start.

Never again will I underestimate the dirtiness of real world data. One of the best teachers I had.

<comment><USER4> This is universal to STEM degrees I think. In mechanical engineering classes you analyze a beam, in real life you analyze an assembly with 50 components that have undergone 100 revisions with 20 different materials and loading from 4 directions that vary with time. Oh, and you have 4 sensors to give you information to analyze critical stresses. But one of them is broken, and Bob who can fix it is on PTO until next Monday, so...

Internships are supposed to fill this gap but it'd be nice if all students could get a taste of real world systems and data. For tech, maybe if they could partner with the IT department at the school to get them exposed to real, messy data. Maybe there are some teaching datasets with over a billion rows that people could play around with.

<comment><USER5> > This is universal to STEM degrees I think. In mechanical engineering classes > you analyze a beam, in real life you ...

Hard to believe this. Don't these degrees require rigorous laboratory assignments where the student learns to differentiate best case scenario with real world uncertainties? STEM is not just some IT certification

<comment><USER6> As a mechanical engineer : No, my education didn't.

The problem is that most real world problems take too much time to really solve to fit in any modern ciriculum.

<comment><USER5> Hmmm. We had a whole course on measurement systems that get to the heart of understanding that source of your data and inevitable bias/error is more important than just crunching the data as given. For example, from a typical four year degree.

</comment></comment></comment></comment></comment></comment><comment><USER7> As a research-oriented data scientist at one of the larger tech companies, I can confirm that even here, a lot of people are unsure about what exactly data scientists are supposed to do. My most frequent request is "tell us why metric X dropped", to which the answer is often a subtle combination of many different factors (often random fluctuation) that doesn't lead to a pleasing actionable result in the sense of "here's why it dropped; go do this to fix it".

The really interesting research type work (Bayesian modeling, convolutional neural networks, etc.) takes a _long_ time to implement and may produce no useful results, which is a really bad outcome at a company that measures performance in six month units of work and highly values scheduled deliverables and concrete impact. Many of the data scientists I work with tend to stick to methods that are actually quite simple (e.g., logistic regression, ARIMA) because these at least deliver _something_ quickly, despite the fact that many of my coworkers come from research-heavy backgrounds.

In my org, there's nothing stopping anyone from pursuing advanced machine learning; for the most part we set our own agenda (in fact, determining priorities is part of the job role). And some people do in fact go after state-of-the-art ML, with some really cool results to show for it. But in terms of career progression and job safety, the risk is just way too high, at least for me personally. I save the highly mathematical stuff for a hobby.

Edit: while this may sound a bit negative, I will add that my description of data science isn't a complaint _per se_ ; I am mainly trying to inform those who are seeking a career in data science of what to expect compared to what is often promised. The work that is most valuable to a business is not exciting all of the time, but I don't think there is another job in the tech industry that I would find more enjoyable than my current one at the moment.

<comment><USER8> >But in terms of career progression and job safety, the risk is just way too high, at least for me personally. I save the highly mathematical stuff for a hobby.

I think the sad truth is that this is the reality of work no matter if you are a Data Scientist or not. What you thought you would be doing to show your worth and climb the ladder gets blurred in with KPIs you didn't set, politics you didn't create, goals and deadlines you had no input into, etc. One of the unique challenges you can face as a Data Scientist is that you may interface with people in many different groups, all of which have different goals which may be in conflict with each other. Compare this to other roles where you ultimately only follow the goals of the organization you report into.

<comment><USER9> Sounds more like it simply doesn't work very well, rather than any of the reasons you listed.

It's often the case, I remember when that stupid Amazon infographic was going around about decreased load times meaning big upswings in conversions.

A client paid for a significant project to reduce load times, which we succeeded in to a huge degree with most of the pages going from 1.5-3 seconds secs down to 250-500 ms. Absolutely no meaningful swing in conversions at all. I've done this a few times since, but never seen conversion move at all when I've done performance improvements.

Nada, zilch. I honestly think it's absolute bullshit. I've always suspected since that it was someone massaging figures in Amazon to justify their job.

<comment><USER10> Load times might not effect conversion linearly. People deal with 3 second loads until one day a competitor does .3 second loads and gives a better experience, then in a matter of months you lose your customer base.

<comment><USER11> I think this is where causal inference and experimental design are important.

</comment></comment></comment></comment></comment><comment><USER12> This article is pretty spot on. As someone who has worked in data science/analytics for over 6 years I have found that the field is filled with hype, managers who are not sure what data science actually is, and an absurdly wide amount of skills jobs expect you to be able to do well.

Apply for and interviewing for data science jobs is a total nightmare. You are competing against 100s or even 1000s of applicants for every job posting because someone said it was one of the sexiest careers of the 21st century. Further exacerbating this, Everyone believes that data is the new oil, and large profit multipliers are just waiting to be discovered in this virgin data that companies are sitting on. All that is missing is someone to run some neural network, or deep learning algo on it to discover the insights that nobody else can see.

The reality is that there is an army of people who know how to run these algos. MOOC's, blogs, youtube, etc have been teaching everyone how to use these python/R packages for years. The lucky few who get that coveted data science job can't wait to apply these libraries to the virgin data only to find that they have to do all kinda of data manipulating to make the algos even work, which takes days and weeks of mundane work. Finally they find out the data is so lacking that their deep learning model does very little in providing actual business value. It is overly complicated, computationally expensive, and in the back of your mind know you can get the same results using some simple logic.

Managers who don't understand data science fundamentals learn from the news and have their data scientist implement those buzz words so they can look good in front of their bosses.

I think there is a place for data scientists who understand the fundamentals of the models out there, and know when you should not use them. Data science is also increasingly a subset of software engineering and a good data science in a tech company should be able to code well. I also think that there is not some huge unmet demand for data scientists. Just a huge amount of hype and managers wanting to look good by saying they managed a data science team.

<comment><USER13> Any work is dull and depressing when done under the supervision of idiots. Some companies, although probably less than claimed, are genuinely data driven rather than HiPPO driven, though. This might be particularly important to look for theses to do interesting stuff in the fields of data science.

</comment></comment><comment><USER14> Data science is correctly valued when you realize how relatively unimportant it is. It is a small cog in a larger machinery (or at least it ought to be).

You see, decision-making involves (1) getting data, (2) summarizing and predicting, and (3) taking action. Continuous decision-making -- the kind that leads to impact -- involves doing this repeatedly in a principled fashion, which means creating a system around the decision process.

For systems thinkers, this is analogous to a feedback control loop which includes sensor measurements + filters, controllers and actuators.

(1) involves _programmers /data engineers_ who have to create/manage/monitor data pipelines (that often break). This the sensor + filters part, which is ~40% of the system.

(2) involves _data scientists_ creating a model that guides the decision- making process. This is the model of the controller (not even the controller itself!), which is ~20% of the system. Having the right model is great, but as most control engineers will tell you, even having the wrong model is not as terrible as most people think because the feedback loop is self-correcting. A good-enough model is all you need.

(3) involves _business /front-line people_ who actually implement decisions in real-life. This is where impact is delivered. ~40% of the system. This is the controller + actuator part, which makes the decisions and carries them out.

Most data scientists think their value is in creating the most accurate model possible in Jupyter. This is nice, but in real-life not really that critical because the feedback-loop inherently moderates the error when deployed in a complex, stochastic environment. The right level of optimization would be to optimize the entire decision-making control feedback loop instead of just the small part that is "data science".

p.s. data scientists who have particularly low-impact are those who focus on producing once-off reports (like consultant reports). Reports are rarely read, and often forgotten. Real impact comes from continuous decision-making and implementing actions with feedback.

Source: practicing data scientist

<comment><USER15> Had to make an account to upvote this. Absolutely dead-on. I think you can generalize this comment to almost any specialist skill. "No Silver Bullet" should be a business doctrine as well as a technical one. You need to do a lot of things well to succeed in business. Specialists just provide you a capability. You have to implement and use those capabilities as part of a larger system if you want to create a machine that generates profit.

</comment></comment><comment><USER16> > I attended a 12-week data science bootcamp in mid-2016. ...

Yeah, well there's your problem, my dude. I've been doing what might be described as "data science" since I quit physics in 2004. Aka before the term existed. It's a great area to work in for intelligent people who want to use their brains to impact the real world; vastly better than what people get paid to do in physics. If customers don't know what the tools can do, it's because _you_ as the data scientist have failed to explain it to the customer. If your work product isn't in front of the decision makers, you've also failed: they can tell the bottom line impact and will reward you accordingly. Sometimes there is no data in their data; they should know that up front.

As for whining about poor data quality: n00b. What do you think they're paying you for? Nobody gives a shit what people do in Kaggle competitions.

<comment><USER17> I don't think the op would care much for your delivery but you make some great points.

> If your work product isn't in front of the decision makers, you've also > failed: they can tell the bottom line impact and will reward you > accordingly.

This one in particular stood out. There is an aspect of salesmanship (or navigating corporate hierarchies) to the role. Things will not be obvious to the decision makers. Perhaps the data scientist has to take some responsibility in bringing their work to the fore.

</comment></comment><comment><USER18> I stood up a data science operation at my company over the last few years, and have noticed a key difference in data-science projects that have been successful and those that have failed. It hits on a number of points brought up in the article, namely where does data science "fit" in an organization delivering software and how is the value realized by the business.

The worst cases I have seen is when executives take a problem and ask data scientists to "do some of that data science" on the problem, looking for trends, patterns, automating workflows, making recommendations, etc. This is high-level pie in the sky stuff that works well in pitch meetings and client meetings, but when it comes down to brass tacks this leaves very little vision of what is trying to be achieved and even less on a viable execution path.

More successful deployments have had a few items in common

1\. A reasonably solid understanding of what the data could and couldn't do. What can we actually expect our data to achieve? What does it do well? What does it do poorly? Will we need to add other data sets? Propagate new data? How will we get or generate that data?

2\. The business case or user problem was understood up front. In our most successful project, we saw users continuously miscategorized items on input and built a model to make recommendations. It greatly improved the efficacy of our ingested user data.

3\. Break it into small chunks and wins. Promising a mega-model that will do all the things is never a good way to deliver aspirational data goals. Little model wins were celebrated regularly and we found homes and utility for those wins in our codebase along the way.

4\. Make is accessible to other members of the company. We always ensure our models have an API that can be accessed by any other services in our ecosystem, so other feature teams can tap into data science work. There's a big difference between "I can run this model on my computer, let me output the results" and "this model can be called anywhere at any time."

While not exhaustive, a few solid fundamentals like the above I think align data science capabilities to business objectives and let the organization get "smarter" as time goes on as to what is possible and not possible.

<comment><USER19> As a person doing data science / ML in the last 4 years, I mostly agree with your points. Especially about the hype driven demand for DS/ML. One thing that is often neglected though is the exploration part it. There really is a lot of data out/in there that your company knows anything about, but can probably benefit from knowing. E.g. even a simple crawl of a popular jobs/ads/... site done diligently for e.g. 6 months can reveal many interesting insights about market structure and trends. Google and its mission to organize all data in the world exist for a reason. This however is in stark contrast with the approach that most executives take. Instead of managing it as a well thought strategic/long term investment, they want to time-box it, to get immediate value and to show off to senior management or customers. I've seen this tendency in both big corporations (mid-level management) and startups, which makes me think that the confounding variable is the fund/incentive management process. In both big corps and startups, there is a limited time&budget to show meaningful results and people optimize for that, which often involves taking shortcuts, neglecting strategy and outright lying. In contrast to that, I've seen projects driven by wealthy individuals, who don't look for immediate value, but are scratching an itch (e.g. curiosity). These usually fare better than the former as long as budgets don't get out of hand (to exhaust the cash cow). I would argue that these are most successful, because of better alignment of motivation (person paying the bill) and execution (person driving the process).

</comment></comment><comment><USER20> Teams being small, data being crummy, infra being hard, and yet expectations being high aren't so much complaints as the they are the job description.

The point of data scientists and the related roles listed in the article are not to just churn out the fun stuff, but to wade through the institutional and technical muck and mire it takes to bring the fun stuff to bear on a relevant business problem and to communicate the results in a way that people of all walks can understand.

<comment><USER21> Yeah this guy seems to think Data Science work should be like doing a problem set for CS class. Sorry that you have to deal with messy data, fragile infra, and limited resources - I know it's not "fun", but frankly that's what the money is for.

<comment><USER22> That's the whole point of the article. Expectation (in this case, his own coming out of the bootcamp) vs. reality (what data science is actually like).

</comment></comment></comment><comment><USER23> I'm generally confused by the hype around ML and 'data science'. it seems like CS has somehow regressed to the behavourism era of psychology or economics before the Lucas critique.

The problem with all this data talk isn't just about implementation or bad structure, the limitations of putting all your bets on inductive reasoning are systemic.

The insights that economists had in the 70s and 80s was that reasoning from aggregated quantities is extremely limited. Without understanding at a structural level the generators of your data, trying to create policy based on outputs is like trying to reason about inhabitants of a city by looking at light pollution from the sky.

My guess why data science so rarely delivers what it promises is because you can't get any value from historical data if your circumstances change to the point where past data is irrelevant. Which in the world of business happens pretty quickly. To have a competitive advantage, one needs to figure out _what has not been seen yet_.

And trying to exploit signals suffers from the issue laid out above. There was a funny case of an AI hiring startup trying to predict good applicants, and the result was people putting "Oxford" in their application in a font matching the background color

<comment><USER24> > CS has somehow regressed to the behavourism era of psychology or economics > before the Lucas critique?

Can you please elaborate on this please?

<comment><USER25> See: [https://en.m.wikipedia.org/wiki/Lucas_critique](https://en.m.wikipedia.org/wiki/Lucas_critique)

At a high level, it argued that basing predictions on historical data is problematic. The details of the argument are somewhat specific to economics, but the principle is more general. That's also why people recommending stocks say "past performance is no guarantee of future results."

One of the key issues is that circumstances change, and information about such changes will often be external to a data set.

In the Lucas critique, policy changes are an example of this. You can't predict future economic performance based on past economic performance if relevant policies have changed. But any complex situation has such factors that are external to the data that one can easily collect about it.

</comment></comment></comment><comment><USER26> As a scientist, I've worked with data for decades. There's always been a prevailing belief that scientists and engineers with specialized domain knowledge are mostly fumbling in the dark and can be replaced with a general purpose technique.

This was certainly the vibe that I got from "design of experiments" when it was the statistical method _du jour_. Then from "Bayesian everything" and now "data science." I remember "design of experiments" studies being conducted with great fanfare and success theater, while producing zero results.

The long term theme is that science is hard for reasons that managers don't understand, can't manage, and are reluctant to reward.

</comment><comment><USER27> I've seen a few similar articles now. Does this represent the general view of folks working in data science? "Data Science" is such as meaningless catch all term. The reality is in many organizations it's simply advanced business intelligence or advanced business analytics. There are some industries that lend themselves well to this whole practice, and they tend to be industries that have been borne out of the internet age (e.g. social media, internet advertising, etc.)

Some other industries have been doing "data science" for ages. Credit Risk Modelling, insurance and so on.

Every time I read one of these articles, I feel it's just an individual who entered a kind of crummy situation and they're learning what it means to work in a corporate environment. Some are better than others. Some are more motivated than others. Some have better cultures than others. Some are more willing to make technology a key part of their business strategy. Some are more data driven than others.

My recommendation is to always ask the fundamental question before joining: what are you trying to achieve with data science, and is it actually achievable?

<comment><USER28> I always thought the non-specificity of the term Data Science was a strange criticism for those in the tech industry to make. How many types of SWE are there? Front-end, back-end, full-stack, devops, security, QA...

I agree wholeheartedly with your recommendation. Like any other job, each company has different needs and expectations and if you want something else out of the role you'd best avoid that company.

<comment><USER27> Frankly I have the same criticism of those who use the term software engineer. Engineering is a pretty established profession with a set of standards, ethics and practices. Most of us who work in software are not engineers. We are developers. Similarly, a scientist is one who follows the scientific method to do research. So by that logic a data scientist should be a person who uses the scientific method to do research on data. Does that make any sense? And let's be serious, is that what most data scientists are being hired to do?

<comment><USER29> > the scientific method to do research on data

Exploratory data analysis is often overlooked and underrated.

<comment><USER30> Ppphhh we don't need to do exploratory data analysis or prepare the days, don't you know that neural networks will do all that themselves!

Doesn't yield the right results? Clearly not enough data.

Still doesn't work? Change to whatever the latest model google or fb is using and try again.

/s

<comment><USER31> And the model will train itself right? That means that you'll have all that empty time to do more data science!

/s

</comment></comment></comment></comment></comment></comment><comment><USER32> As a data dude in public/nonprofit healthcare-landia I agree with all this, plus:

\- It's essential to have/develop domain expertise in your industry.

\- Beware plausible, but incorrect (or poorly interpreted) data that supports yours (or others') assumptions/biases.

\- Add on to #4 - at least as bad as this is having well-intentioned people on your team who "know enough (a bit of SQL or low/no-code data tool") to be dangerous. Um, why are you joining unnecessary tables, or using a different alias for the same columns/tables in different queries, with no comments or standard formatting?

\- Hold your nose, but anything you do in SQL/R/Python/even fancier programming tool/language is going to pass through MS Excel at least once sooner or later which can irreversibly bastardize CSVs (even just opening without saving!), truncate precision to 15 digits, change data types, etc.

\- So glad for the callout in #7 - there are clearly devs/data folks out there who are happy to take on an "interesting programming project at a great paying job" \- that isn't serving the best interests of humanity.

</comment><comment><USER33> This rings very true to me. I'm working on moving over to an SWE role in the next few years for many of these reasons.

I'll just add one: the business absolutely doesn't care how you get your answer, only if they're reliable enough (hand grenade close is better than most companies have today).

While this seems obvious enough to anyone with a few years under their belt, to the new DS grad who has their time series analysis canned in favor of slapping a simple moving average in place and shipping it can be rather disillusioning.

<comment><USER34> Nothing reliably consistently beats ARIMA models in time series forecasting _to this day_

That's pretty sad when you think about it, but it's painfully true.

<comment><USER14> > Nothing reliably consistently beats ARIMA models in time series forecasting > to this day

Not sure this is true in practice. In some situations, Holt-Winters (ie. algorithms in the ETS family) may do better, and it's often a good idea to try both.

There's a claim that Holt-Winters is a special case of ARIMA (the claim is ARIMA is more general), but this is actually not the case. There is equivalence in only a subset of cases. [1]

I've fitted Holt-Winters models that beat ARIMA models. ARIMA models can have trouble generalizing from training data with long horizons because they tend to overfit to the distant past. Holt-Winters on the other hand has a natural "forgetting factor" built-in which moderates this.

As well, my experience is that stacked models with well-chosen exogeneous variables (if you have causal variables) tend to outperform pure time-series methods because they are anchored on more independent variables than just t. Pure time-series models bank on the assumption that patterns have a repeatable time-dependence, and most of the time this is just not true, so have to be augmented with other variables.

[1] [https://otexts.com/fpp2/arima-ets.html](https://otexts.com/fpp2/arima- ets.html)

</comment></comment></comment><comment><USER35> I've been doing an MS in Data Science very slowly due to work and 2 new kids. Finishing the degree this year in year 4. I was very excited about the prospect of doing something different. A few things have changed for me.

1). I am hearing about Data Science Teams being furloughed during these times. That isn't happening in my function (Corporate Finance). I am glad to be secure even though I enjoy much of the data sci work.

2) I'm able to apply Data Science concepts in my current role, and it's adding a lot of job security and providing me with exposure. I am much less interested now in moving to straight Data Science and instead am applying my learnings in my current role as a sort of in-house Data Science guy. But I have a lot to learn to be honest.

3). There seem to be a lot of "thought leaders" acting like they are big experts in the area and really don't know anything many of us amateur scientists don't know. They pull perfect clean datasets and show these magic transformations they just copy from others to get YouTube hits or Twitter followers. That just never happens in real life, and many leaders are seeing this and losing interest in this function in the returns they are getting from sole data science folks.

<comment><USER36> This isn't unique to data science. I personally know people in finance that are poor coders and even worse quants, yet they go around lecturing at universities.

</comment></comment><comment><USER37> I work as a data scientist. Some of the author's points are workplace- specific: lack of leadership, being the only data person, ethical concerns. The others are just aspects of the job - communicating about your job and impact, dealing with vague specs or managing low-quality datasets.

Neither of those quite match the articles title, perhaps it just refers to the author's personal expectations. Neither of them seem that specific to data science, or without parallels in other software jobs. And neither of the points read like a slight towards data science to me, like some of the other commenters here suggest.

</comment><comment><USER38> One issue might be that organizations subconsciously resist the data scientist, or more generally, the nerd in his/her attempt to take over decisions. If these decisions are invariably tied to the goals and careers of managers, how can the data scientist have a "seat at the table" in all but the most enlightened and technical companies? The disorganized state of data and infrastructure suits the ambitious manager well, who can just put in enough effort to find data to have their project greenlightened or to answer one specific question.

Progress may only come slowly, ideally through products bought from 3rd parties whose results are understood and controlled by management.

</comment><comment><USER39> I did "data science" for about a decade, consulting with plaintiffs firms and state AGs on antitrust and fraud cases. For each case, the work flow was roughly this:

\-- write discovery requests

\-- review production, and check out data and documentation

\-- write supplementary discovery requests

\-- review production, and check out data and documentation

[repeat as needed]

\-- analyze data, and write deposition questions

\-- help attorneys wring answers from deponents

[repeat as needed]

\-- analyze data, and produce required output

\-- write parts of briefs and expert reports

I generally did that in consultation with testimonial experts and their data analysts. Sometimes that didn't happen until we'd documented the case enough to know that it was worth it. And occasionally small cases settled with just me as the "expert".

It's a small industry, and not easy to get into, unless you know key players at key firms. But the money's pretty good, and the work can be exciting. I loved being that guy in depositions whispering questions to the attorneys :)

This all involved pretty simple calculation of damages, through comparing what actually happened vs what would have happened but for the illegal behavior. But-for models were typically based on benchmarks.

After data cleanup in UltraEdit, I did most of the analysis in SQL Server. I used Excel for charting and final calculations.

<comment><USER40> I would expect "data science" is doing some form of numerical analysis. Otherwise it's just record keeping... with computers.

<comment><USER31> Record keeping is 90% of data projects.

The second 90% is basic math at high speeds.

<comment><USER39> Right, record keeping. But when it's not your data, things get complicated. Imagine trying to understand how another firm's data systems work. You can talk with managers, who know how the business uses data. But they have no clue how the data are stored or managed. And you can talk with IT people, who know how data are stored or managed. But they have no clue how the data are used.

And yes, speed. Aggregating hundreds of gigabytes was nontrivial to do quickly. I started with Access, and then learned to manage and use SQL Server. And eventually a multi-Xeon server with lots of RAM and SAS-attached storage.

</comment></comment></comment></comment><comment><USER41> This reads like Indiana Jones teaching Archeology. Yes, as a data-scientist you actually have to work, most of the work is digging in dirt, and mostly you won't find anything of interest.

</comment><comment><USER42> It works well when subject matter experts exist in the org and collaborate/supervise/drive data folk, to solve some issue the sme's have spent enough of their own time thinking about.

If its just data folk by themselves getting dumped with org data and told to find pirate gold...then its a crap shoot.

</comment><comment><USER43> The real issue with data science, from the perspective of ML pipelines/using ml in products, is most people are straight up not smart enough for it. The second the problem falls outside the bounds of a commonly used model, 90% of data scientists are ill equipped to come up with a profitable solution. So they stumble around in the dark, producing nothing of real value. People underestimate the degree to which extreme mathematical maturity and skill can bend the results of commonly used ml models.

</comment><comment><USER44> This reads as a series of bad job experiences and I think is explained by a wide variety of job functions that all can have "Data Scientist" as a title. Someone else's experience could be totally different. You have to know what to look for and what to avoid. If you're trying to find a DS job, one of your top priorities is finding out what the actual job consists of. For instance, a Data Scientist at Facebook might be called a Data Analyst at many other places --no modeling required.

I know this because I've been on that journey. But there's no reason to expect some department head that's never been exposed to DS to know this. They just copy/paste some other company's job req. If you're more junior, here are my tips:

\- If it's a "new DS team" that supports a variety of teams: beware. Bolt-on DS doesn't work well, as it's really hard to build a meaningful solution that's not deeply integrated.

\- If it's an old company or in a conservative industry: beware. There are likely to be data silos and difficult ownership models that make it nearly impossible to get and join the data you need.

\- If it's a small company: beware. You're likely going to need a broad set of knowledge that's won with several years of experience to be able to build end- to-end solutions that are integrated into the rest of the tech stack.

\- If it's not an engineering-driven culture: beware. DS will often be used to provide evidence to someone else whose already made up their mind and pretend they're being data-driven, and you'll be the disrespected nerd that's expected to do what it takes to deliver the answer they want. Most companies claim to be "data-driven", few are, and even fewer understand data-driven isn't always possible or desirable.

Industry is still trying to figure out how to use ML and are still learning that it's not as easy as hiring someone that knows about all the algorithms, but rather it takes deep technological changes to data infrastructure to enable the datasets that can then be used by the ML experts. But you don't have to be the person that helps them figure this out the hard way (i.e. by being paid to not accomplish much due to problems outside of your control). Better to find a place with a healthy data science team that can help you learn and contribute. They exist.

<comment><USER45> Agree with your points on "old company/conservative industry" and "non- engineering culture"

I'm at a place that is both, and both are huge pains.

On the engineering side, it's a bit different though: technical roles are looked down on, and there is no engineering culture, eg, for data. Data is just a bunch of flat files everywhere, across many silos. No leadership to put it together into logical buckets for easy access and interoperability

</comment></comment><comment><USER46> Great read. A lot of those problems are real, and some of those I've experienced myself. But I think at least some of them are related to the immaturity of the field. We're only at the beginning of creating the tools and platforms to facilitate DS, making it more reproducible and easier to measure.

For example, I'm working on the tool to make data management easier and convert datasets into a structured representation. If you have experienced that you spend a lot of time on preparing and analyzing data, and it is tedious, please reach out to me michael at heartex.net, would love to get your feedback on the product we have built so far.

<comment><USER47> > But I think at least some of them are related to the immaturity of the > field.

I agree. More so, I sometimes feel that in the end the field will break up once things start settling down. Some roles will migrate more towards engineering, some will go back towards data analysis.

The expectation that a data scientist is a funnel that can turn anything into magical insights and tools can't last forever.

</comment></comment><comment><USER48> A really easy way that I try to explain things to people is like this:

You can't compress information until you have it in a format that is appropriate for compression.

That is:

You can't compress (apply/create algorithms) information (data) until you have it (instrumented data collection) in a format (schema) that is appropriate for efficient compression (structured logging/cleaning).

99% of that is Data Engineering and building good engineering practices which have good data practices as a priority.

For any organization that has more than a handful of employees and more than one product, that is a non trivial task and gets more difficult the larger the organization gets.

<comment><USER45> Totally agree. Non-tech companies that think they need "data science" should instead put same effort into (data) engineering.

It's not quite 99% of the effort but close enough ;)

Search "data science hierarchy of needs"

</comment></comment><comment><USER49> I wrote a blog post along similar lines in 2018 ([https://minimaxir.com/2018/10/data-science- protips/](https://minimaxir.com/2018/10/data-science-protips/) ); unfortunately, the industry hasn't changed much since then.

As noted in the submission, there's a lot of flexibility in what a "data scientist" is. Normally that's good and healthy for the industry. However, it contradicts a lot of optimistic bootcamps/Medium/YouTube videos, and many won't be prepared for the difference.

<comment><USER50> My industry (information security) is the same way. Far too broad of a category, leading a lot of people to get confused and frustrated. I see a lot of analysts (defensive security) who thought they were going to be pen testers (offensive security) and didn't realize those jobs were in two completely separate career paths.

</comment></comment><comment><USER51> I've been a data person for the past year and a half and I'm very disappointed with the bewildering array of titles out there and the rather vague meanings behind them (Data Analyst, Data Scientist, Data Engineer, ML Engineer).

It's overall hurting my ability to build my personal brand and seek roles that are a fit for my existing skillset and aspirations.

What exactly does 'ML Engineer' communicate to employers in terms of baseline skills? Is the role closer to that of a data engineer or an analyst?

<comment><USER52> I've been working in data roles for 10 years and hold a masters in ML. I've hired and managed each of the roles you mentioned. I think of the responsibilities of each of those roles as:

-ML Engineers as building software infrastructure to scale machine learning inference and training.

-Data engineers focusing on data infrastructure and pipelining into either model inference, training, or other business intelligence platforms

-Analysts consume the product of the data engineer in the BI platform or excel, where the results would be consumed as a report in some form.

-And ML Researchers would be those inventing novel machine learning algorithms to deploy in the ML Infrastructure managed by the ML Engineers

-And data scientists to deploy well-known ML algorithms or statistical inference on varying datasets on the ML Infrascturue or as a slide deck.

<comment><USER31> How hard could it be to find one person who can do all that?

<comment><USER53> Depends on the amount of data, reports, pipelines... If the company is small you might not have any of these problems. Every Mom&Pop store has some sort of data to run the business but they don't need a "data" person.

Once you have 10s of datastores + pipelines, 100s of reports and a "data lake" in the TBs you'll likely be needing specialized people.

<comment><USER51> So far I've spent my career in small teams / startups and it's starting to become apparent that a lot of what's assumed in these titles only applies in larger corporations where resources are abundant and it makes business sense to have a specialist focused on a single aspect.

Unfortunately I'm at a point where I have 'jack of all trades master of none' syndrome and it's causing me to fall in between the cracks professionally. I'd like to move to a larger company where I can develop deep expertise in a narrow topic.

<comment><USER54> ymmv, but as a data scientist at young startups, I often am the one giving new tasks to the software engineers, and facilitate teaching and training if they need help.

Most of those roles a software engineer can do.

</comment></comment></comment></comment></comment></comment><comment><USER55> From my perspective as a data person, everything on this list is true. I would add on to #4 to say "You're likely the only data person" to say "You're likely the only data person and expected to do everything you need to do your job yourself" (from sourcing the data to deploying your model).

High Data Scientist salaries and expectations combined with a shortage of qualified people often mean you're expected to be a one-person band, which I find to be miserable.

</comment><comment><USER56> Point 5, " Your impact is tough to measure" is also shared by Quality Engineering and SRE, and not unique to Data Science. The point about being a support role holds true for them and it is thoroughly frustrating when a front-end dev makes a small change to a visual element is praised to the roof while complex automation projects by the quality team, ingenious recovery and reliability projects by SRE, and massive and fascinating inferences by data science are undervalued by leadership. The truth is most leaders just can't connect the dots. I've worked as a full stack engineer btw do not taking a dig at front-end work, but it's clearly easier to measure impact. I've worked in quality too and when you're only called in to ask why one bug got out and never asked about the thousands you've stopped it's demoralizing. It's part of the reason I started Tesults ([https://www.tesults.com](https://www.tesults.com)), if you're in one of these support roles, measure, measure and measure and throw those reports into the faces of leadership. It shouldn't have to be done but without it, the point the author is making here will take place.

<comment><USER30> > and it is thoroughly frustrating when a front-end dev makes a small change > to a visual element is praised to the roof while complex automation projects > by the quality team, ingenious recovery and reliability projects by SRE, and > massive and fascinating inferences by data science are undervalued by > leadership.

I feel this in my bones lol.

The frustration when the results of weeks/months of hard work are glossed over with a "oh that's nice" in favour of endless praise for the front end team putting a picture backdrop on the search page or something.

Didn't matter how many times we sold them on the benefits, or explained the work that went into it (at both executive summary level and detail) or did all those things you're supposed to do, if it was more than one step away from directly causing it, or slightly more abstract than "we moved the button" it was wasted on leadership/management.

Spent a couple of weeks fixing data pipelines and ETL/database infrastructure and processes and now everything runs faster, and runs on a smaller and cheaper cluster and as a result managed to put together some analysis and modelling on customer behaviour that shows if you do xyz you'd expect to see uptick in this thing. Doesn't matter, Bob changed where the button sits and we saw 20% more sales, good job Bob, everyone: be more like Bob.

</comment></comment><comment><USER57> Coase: "If you torture the data long enough, it will confess to anything."

via [https://www.reddit.com/r/QuotesPorn/comments/b76ujr/if_you_t...](https://www.reddit.com/r/QuotesPorn/comments/b76ujr/if_you_torture_the_data_long_enough_it_will/)

</comment><comment><USER58> I guess I'm in the minority in these threads..? I've been doing machine learning / model-building / pushing models to prod and maintaining for about 6 years now. It's still 50/50 understanding the data and building/tweaking/training/testing models. But it sounds like most people with this title are analysts? At least that's what posts and threads lead me to believe. I've also met a lot of people with titles like "ML Engineer" or "Data Scientist" who don't do machine learning. They are analysts, engineers, or maintaining data pipelines.

<comment><USER54> Pushing models to prod is often MLE work depending on the organization, though MLE is often slang like how dev is slang. MLE an be a job title just as Developer can be a job title, but more common than not the job title is Machine Learning Software Engineer, or just Software Engineer for short.

I suspect a lot of people want the sexy Data Science job title, which is why there has been such a push for it, and why most new "data scientists" take the title but do Data Engineer / Infrastructure Software Engineer work or MLE work instead.

I think MLE is more sexy in a lot of ways, and it often pays better than DS work, so it's odd that many haven't flocked to that job title, but maybe the whole software engineer part turns people off for some sort of reason.

Me, I'm more a classic data scientist / research engineer, which involves a lot of digging through data and research and generalized learning, then presenting my findings. I'm not using any ML on the job right now, but often I have in the past. It's just a tool, not an end.

</comment></comment><comment><USER30> > You're likely the only "data person... Because people don't know what data > science does, you may have to support yourself with work in devops, software > engineering, data engineering, etc.

Nothing has summed up my entire working experience more than this, it's almost painfully accurate.

On one hand it's an exciting challenge, you learn a lot and you get good at adapting to these situations.

On the downside I have practically no senior data science people to turn to for help when I do need it, which is frustrating.

</comment><comment><USER59> I am sorry to sound like I am being obstinate, but my opinion about this is that as a society, since the early 90s, we have put way to much focus on "tech" than we have put on plain old mathematics or foundational science.

I don't mean manufacturing (which is doing really well), but companies like Microsoft, Google, Facebook (and even Apple) and others do encourage you to try to compete against their founders (or maybe society does that) rather than focusing on being solid mathematically. Yes, Google pays people well with those skills, but movies portray mostly their founders, emphasising how rich they are, while mathematicians are generally portrayed as weird. Society as a whole puts more emphasis on Bill Gates than on fundamental researchers.

In fact, if you really want to have a rich representative, you can pick the Simons guy. (See, I don't even know his name.) His Medallion hedge fund was built on mathematics. Ironically, Bill Gates is these days one of the biggest financial supporters of people with science skills that he doesn't have.

It is a fad to be a techie. Mathematics is not a fad, although it does have internal fads.

</comment><comment><USER60> I do not understand. Have never understood. "Data Science" is, surly, newspeak. The appropriate term, surly, is "statistics".

<comment><USER61> A new title means a new opportunity to ask for more money and influence. See also, "devops".

<comment><USER31> Going from IT to devops is a great way to double your salary.

</comment></comment></comment><comment><USER62> I worked as a data scientist for 4 months at a VC firm. I have a PhD and thought the work might be legit when I was hired. After the 4 months I quit when it became apparent that my credentials were being used for managerial intrigue and the work was essentially a joke, with no rigor at all. This article hits the nail on the head, unfortunately, these positions are not often real jobs.

</comment><comment><USER63> There is a philosophical principle which says that any model superimposed on reality could be seen as reality itself, while it is merely a superimposed interpretation, in principle.

Korzybski formulated these principles, among other things.

Most of data science models are as wrong as astrology and numerology. They have no connection to reality, or rather inadequate.

This principle explains abysmal failures of all Model-based "sciences", stating from financial markets and up to virus spreading models.

Simulations of non-discrete, non-fully-observable (AI terminology) system has exactly the same relationships with underlying reality as a Disney cartoon to a real world.

This is why expectations will never be meet, except for natural (non- inaginary) pattern recognition.

A drop of proper philosophy worth years of virtue signalling.

</comment><comment><USER64> > internally, you can make inroads supporting stakeholders with evidence for > their decisions!

The problem is that this can all too easily become motivated reasoning: one provides a stakeholder with support for the decision he already made. From his point of view, this is a valuable service, but it does the _organisation_ a disservice: decisions should be made after considering the data, rather than consider only those data which support a decision.

Also, while ethical issues certainly arise, I think that Greyball is not a good example. Uber evading police enforcing the taxi monopoly is no more unethical than the Underground Railroad evading fugitive-slave agents. The taxi monopoly is itself unethical, and evading it increases the common good.

</comment><comment><USER65> Disclaimer: I use the term Data Scientist throughout this post; however, popular titles such as Data Analyst, Data Engineers and BI analyst are randomly applied by people who know nothing, and these people share none of the responsibilities of a Data Scientist.

I have never had hopes about the potential impact of being a Data Scientist. I felt every company should be a "data company", but everything I knew told me that companies are political institutions bounded by the pressures of late stage capitalism. Anyone who things different is dim, anyone who blogs about it is a moron.

My expectations did meet reality.

Where did my expectations come from?

I attended a four year Computer Science degree, followed by four and a half years of earning a Ph.D. I then spent 20 years in industry. 19 of the 20 weeks' focus were not on machine learning (ML) and artificial intelligence (AI).

I figured I'd spend most of my time buried in code and data, I was right, I had to find shit buried in it, and dig it out with my teeth. Executives hated me because I was a threat, but they needed me so I continued to get paid. I continue to be able to create insight and predictions that almost no one else can, and until this stops I will get a 200k a year salary, benefits and a Tesla.

All of this happened, I can't be bothered to waste my time commenting on this moronic blog post.

<comment><USER66> How important is Ph.D for data science?

<comment><USER65> I think it's quite important - or an equivalent.

From about 2012 to 2018 I went round a lot of universities, conferences and companies doing presentations and I used to often ask the audience for a definition of data science (in the hope of getting a good one). The best one I heard came at the University of Bath where someone (I know who, but he didn't say it to back it with his reputation so it's not fair to name him - it wasn't me though) said "Just drop the data, it's science".

I totally think that - Data Science is about doing Science with found and evolving data sources, we aren't often able to construct our experiments from scratch, but we often get to augment them, but we always start from the data we are given - which is why it's a sub-field.

In any case - the Ph.D's I have employed have almost all known how to do Science, and it has really helped. Some people without a Ph.D. learn to do it. Experimental Ph.D's are best.

Maths and theoretical Physics Ph.D's are generally not able to do this!

<comment><USER66> Will you employ data scientist that have articles and years of expirience as data analyst in University but not Ph.D? How much role will be lack of Ph.D in this case?

<comment><USER65> As I said - if the person is capable of independent scientific investigation then I think they'd be good. I think that a Ph.D is formal training for that - but not the only way to learn.

</comment></comment></comment></comment></comment><comment><USER67> Work reality rarely meets expectations. I am sure a lot of "UX" people also got a little disappointed that reality is much more mundane than the fancy title suggests. Or the people whose last algorithm work was the interview.

</comment><comment><USER68> It really depends on the niche or the industry considered, though: I can happily say that I can do materials informatics from my basement at home now and much faster and better than as a cog in any lab anywhere in the world. Same for a great number of STEM applications, if you ask or follow high-level practitioners through conferences, journals and social media. The elephant in the room is the Intellectual Property generated through STEM applied data science, which is hot and even dangerous as you can see from superstars like OpenAI, DeepMind or politically-motivated aggregations.

</comment><comment><USER69> The most common complaint I've heard from the data science team is that there isn't enough data to work with.

I'm not fully convinced that data science with ML and more modern techniques are applicable across domains out of the box. I think there is value to be added if data scientists can specialise in domains.

If we take humans as an analogy, even with the kind of general intelligence we have, we need domain expertise to be able to have advanced intuitions and make predictions about the future. I believe this is true for data science as well.

</comment><comment><USER70> I was a data scientist for one year. Experienced many of the adverse situations explained in the article, plus I thought it isn't for me. I joined my next job as a software engineer (after an extensive interview prep). Couldn't be happier. Still doing plenty of data science. But my product is actually a product, not the analysis (as is often the case with DS). I feel "more central" to the project, to the company. I'm still building ML models, features etc for a living.

</comment><comment><USER71> As the lead data scientist at a small-ish fintech, I can confirm many of the frustrations and disappointments in the OP. But my trajectory was slightly different - from being the only "data science guy" in 2016, to now leading an autonomous team of four, with quarterly meetings with the CEO, and monthly meetings with our tech leadership. I decide tech stack, workflow, and hiring. Execs decide priorities. Sure, some of it was dumb luck, some of it was actually having a CEO that cares about data strategy, but I like to think at least some of it was me.

So here's what I think I did right:

1\. Provide indisputable, obvious business value every month. You should consider yourself an in-house consultant to whichever cost center your salary is drawn from. If you're product development, prove value to them. If you're operations, or sales, or marketing, prove value to them. After about two months, you should be able to justify your existence in two sentences. Just remember, most of your company probably thinks of you as a optional add-on.

Your first few projects should attack high-impact pain points with the simplest solutions possible. My first projects were basically ETL into some basic regression into a dashboard. No machine learning required. But it was better then what they had (which was often nothing), and it was STABLE and RELIABLE. And that leads to the next point...

2\. Build trust. With my dead-simple models, nothing ever blew up, there were no nonsensical answers, and there wasn't much brittleness when new categorical features or more cardinality was added. It mostly just worked. And that built my reputation for me. They didn't have to understand what was going on in the model, but they knew, from experience, that they could trust the result. Once I had the credibility, I could start building more complex, more elaborate models, and asked them to trust those as well. If they don't trust your models, then no business value has been created, and your job is worthless.

3\. Recognize that data science is being done everywhere in the organization, and respect it. Every department has someone who has built a monster spreadsheet that contains more embedded domain knowledge then you could hope to learn in a month. As data scientists, we like to think that we're helping the organization by building critical metrics to improve performance. But here's the catch. If the metric was truly critical, someone has built it already. It might be ad-hoc, use poor-methodology, and be somewhat wrong, but it works and is good enough. You have to find that person, learn from them, and improve on it.

4\. Be as self-contained as possible. Ideally, your critical path should not depend on other teams doing things for you (except for IT setting up data access). You should be able to do it all. From front-end dashboards, to ETL, to DevOps. Remember, you're an in-house consultancy. You should be able to take problems and just handle them, rather then be a perpetual bother and distraction to other teams.

There's more, but if you do these four things, I think you can build the reputation in your company for creating useful, accurate data tools that help other people do their jobs better. After that's achieved, people will breaking down your door to get your help. That's where my team is now - we've got a backlog for at least 18 months, with our work priorities often being set directly by the CEO.

</comment><comment><USER72> My feeling is that a lot of companies think: "We need a data scientist because all the big players also have one!"

In fact, they actually don't need a data scientist. At best they need someone who cleans data, creates pie charts or even worse, they relabel the database admin job as "Data scientist".

</comment><comment><USER73> Can definitely relate to this. Work for big consulting firm (F500) as a data scientist, end up in this weird software engineer/ml engineer hybrid role.

I personally love it but am doing more pure software engineering now as the infrastructure is not there and I need to build it myself.

</comment><comment><USER74> To point #5 in the article, in my experience, ascending order of potential to generate value for business:

An astonishingly large fraction of Data Science output goes to die in pretty presentations.

From what's left, a large fraction ends up in Spreadsheets.

A disappointingly small fraction ends up in live services.

</comment><comment><USER75> Completely agree. I made a post couple of weeks ago trying to find some solutions for this: [https://news.ycombinator.com/item?id=22673236](https://news.ycombinator.com/item?id=22673236)

</comment><comment><USER76> it's a problem with tech in general. some things come over-hyped. and in the process people forget what's the actual problem to be solved because they fell in love with tools | tech. maybe the solution could easily be done in excel but then that's not sexy. I personally prefer to handle most parts in Python because of automation. writing functions in python is easier than writing functions in SQL or Excel(macros)

</comment><comment><USER77> Maybe I have a narrow set of experience, but in my mind a "data engineer" is not a substitute for a "data scientist".

</comment><comment><USER78> The situation sounds similar to ones years ago for statistics, operations research, optimization, and management science.

I view all of such work as applied math.

My experience is that applied math, from the fields I mentioned and some more recent ones, and more, with emphasis on the more, can be valuable and result in attention, usage, and maybe money.

I've had such good results and have seen more by others.

Some examples:

(1) Airline fleet scheduling and crew scheduling long were important, taken seriously, pursued heavily, with results visible and wanted all the way up to the C-suite.

(2) Similarly for optimization for operating oil refineries: So, here is the inventory of the crude oil inputs and the prices of the possible outputs. Now what outputs to make? The first cut, decades ago, was linear programming, and IBM sold some big blue boxes for that. More recently the work has been nonlinear programming.

(3) The rumors are, and I believe some of them, that linear programming is just accepted, used everyday, in mixing animal feed.

No surprise and common enough, IMHO what really talks is money. If can save significant bucks and clearly demonstrate that, then can be taken seriously.

But from 50,000 feet up, tough to get rich saving money for others. If they have a $100 million project and you save them $10 million, then maybe you will get a raise.

What's better, quite generally in US careers, is to start, own, and run a successful business. If that business is to supply the results of some applied math, and the results pass the KFC test, "finger lick'n good", then charge what the work is worth.

Maybe now Internet ad targeting is an example.

I'm doing a startup, a Web site. The crucial enabling core of what I'm doing has some advanced pure math and some applied math I derived. Users won't be aware of anything mathematical. But if users really like the site, then it will be mostly because of the math. So, it's some math -- not really statistics, operations research, optimization, machine learning, artificial intelligence, or management science -- it's just some math. The research libraries have rows and rows of racks of math; I'm using some of it and have derived some more.

Generally I found that the best customer for math is US national security, especially near DC. E.g., now some people are building models to predict the growth of COVID-19. Likely the core of that work is continuous time, discrete state space Markov processes, maybe subordinated to Poisson processes. Okay: One of the military projects I did was to evaluate the survivability of the US SSBN (ballistic missile firing submarines) under a special scenario of global nuclear war limited to sea -- a continuous time, discrete state space Markov process subordinated to a Poisson process. Another project was to measure the power spectra of ocean waves and, then, generate sample paths with that power spectrum -- for some submarines. There was some more applied math in nonlinear game theory of nuclear war.

Here's some applied math, curiously also related to the COVID-19 pandemic: Predict revenue for FedEx. So, for time t, let y(t) be the revenue per day at time t. Let b be the total market. Assume growth via _virality_ , i.e., word of mouth advertising from current customers communicating with remaining target customers. So, ..., get the simple first order differential equation, for some k,

y'(t) = k y(t) (b - y(t))

where the solution is the logistic curve which can also be applied to make predictions for epidemics. This little puppy pleased the FedEx BoD and saved the company. Now, what was that, data science, AI, ML, OR, MS, optimization? Nope -- just some applied math.

I have high hopes for the importance, relevance, power, fortunes from applied math, but can't pick good applications like apples from a three.</comment>


Learning Modern 3D Graphics Programming with OpenGL 3 - <AUTHOR> http://www.arcsynthesis.org/gltut/ <comment><USER2> Hmm, the book is incomplete (the chapter about texturing is empty as are all subsequent chapters). Is there a blog following the development of the book? A twitter account? Anything?

<comment><USER3> There's a bitbucket account with the source of the book: <https://bitbucket.org/alfonse/gltut/wiki/Home>

</comment></comment><comment><USER4> I discovered these pages while looking for an OpenGL tutorial that actually used the new shader-driven style and not the deprecated fixed-function API. They're very good (although as others note, still a work in progress).

</comment><comment><USER5> Oh, boy! I was actually looking for a good book aimed at a beginner. And this one looks to be EXACTLY what I need. Many thanks!

<comment><USER6> Same here. In depth, but very understandable. A very good book!

</comment></comment><comment><USER7> How much of this would be applicable to OpenGL ES programming?

<comment><USER8> I've only glanced at it so far but if you're using ES 2.0 then it would be a big help as ES 2.0 has no fixed function pipeline so everything is done in shaders. The only real difference IIRC between OGL 3.0 and ES 2.0 is the unavailability of many extensions OGL users are familiar with.

</comment></comment><comment><USER9> Anybody aware of anything remotely similar for OpenCL?

<comment><USER10> OpenCL looks like a generalization of the OpenGL shader language GLSL. The logistics of passing buffers and controlling execution should be very similar as well.

I think if you learn OpenGL 3.3 (as in this tutorial), you'll get enough of the common stuff down that you can fill in the missing spots from OpenCL reference material directly.

<comment><USER11> I'm only just getting into OpenCL (I already know OpenGL very well) and I'd say OpenCL's programming language is closer to C than to GLSL. In any case, I'd say the hard part is parallelising your algorithms, understanding the memory model (private vs local vs constant vs global pointers) and using it and the the task dispatch mechanism effectively.

Being experienced with OpenGL (or Direct3D, or game console) shaders will probably help for performance tweaking as the mindset is similar (memory access is evil, irregular memory access more so), but other than that I doubt it will help.

</comment></comment></comment><comment><USER12> Here is another similar book in the works: <http://openglbook.com/>

</comment><comment><USER13> Looks like a good book. I'm looking forward to going through it and seeing how it goes.

</comment><comment><USER14> what ide can this language be programmed in.can i use eclipse

<comment><USER15> OpenGL is not a language - it is an API. The book is using it with the C++ language which you can program in Eclipse.</comment></comment>


Why don't we use weights to store energy? - <AUTHOR> https://physics.stackexchange.com/questions/305563/why-dont-we-use-weights-to-store-energy <comment><USER2> We do, of course.

[https://www.youtube.com/watch?v=a8g-pg50bf4](https://www.youtube.com/watch?v=a8g-pg50bf4)

But it needs constant attention, that's why we tend to use more convenient things like batteries these days.</comment>


JQuery Core 1.9 Upgrade Guide - <AUTHOR> http://jquery.com/upgrade-guide/1.9/

<comment><USER2> "Prior to 1.9, an ajax call that expected a return data type of JSON or JSONP would consider a return value of an empty string to be a success case, but return a null to the success handler or promise. As of 1.9, an empty string returned for JSON data is considered to be malformed JSON (because it is); this will now throw an error."

That one is gonna cause some headaches, I've abused this behavior for years, as recently as yesterday.

<comment><USER3> So your server returns an HTTP 200 with an empty response, and you process the `null` in your success handler? Do you really want a `null` returned? Is this a server you control, or is it some third-party service? Just curious about the use case and how widespread it might be.

<comment><USER2> I was probably in control of all of them at some level but I either didn't think about it or didn't care enough. Sometimes I suck. Today I can start sucking a little less.

</comment></comment></comment><comment><USER4> Since so very very many things changed in backwards incompatible ways, why didn't they just go with calling it 2.0 rather than 1.9? I mean, the changes are huge and most versioning schemes increase the major version number on backwards incompatibility.

<comment><USER3> > so very very many things changed in backwards incompatible ways

jQuery has always made feature changes/additions on .x releases. We're saving the 2.0 moniker for removing oldIE because that is a much more significant change than these API cleanups.

At this point the jQuery installed base is so big that any change, however innocuous, is a breaking change for someone. Even fixing bugs, conforming to W3C standards, or making API return values consistent can cause trouble because someone depends on the old way of doing things. We try to be respectful of people's existing work but also want to make forward progress.

Like the intro at the top of the upgrade guide page says, this list seems a lot more scary than it really is. We're just trying to call out every change that we think may cause issues, so people can assess and address them in advance.

The jQuery Migrate plugin is designed to make it easier to find any compatibility problems, and can be included with jQuery versions as old as 1.6. Add the plugin to your current code, run it, and view the console to see if there are warnings.

</comment></comment><comment><USER5> If you're not targeting old browsers there's no reason to use JQuery. Plain JavaScript offers you the same features.

<comment><USER6> I'm certainly not a jQuery fanboy, but jQuery's DOM-manipulation functions are certainly more _concise_ and abstract away some of the funky internals (NodeLists, etc.).

That said, I've always found jQuery's function naming to be particularly ambiguous. There's no indication that "find" searches an element's descendants for a matching CSS selector based on its name. On the other hand, vanilla JS's element.getElementsByTagName is so much more verbose. Back in the day I found PrototypeJS [0] to be much better organized and offer the same feature set, but it's largely fallen out of use since Rails dropped it. mootools is nice but the docs, while comprehensive, are arranged in no logical order. Every time I try to pick up mootools I end up running around the docs in circles.

tl;dr jQuery is overblown but I think there's room for improvement over vanilla JS.

Plus there's always the (literally) hundreds of plugins for every possible application of jQuery.

[0] <http://prototypejs.org>

<comment><USER7> I actually like long verbose names. Unfortunately, plain JS names are often long, verbose _and_ unclear, so that's not a win. (e.g. document.querySelectorAll - that should have made some reference to css; and "all" should be implicit as it is the norm in CSS).

I'd love to see a JS library that doesn't try to add a lot of functionality, but only decruftify the browser API.

The right aim would be to end up with a complete API that might actually replace the normal JS api (DOM+otherwise) in actual browsers.

<comment><USER8> What do you think of Zepto?</comment></comment></comment></comment>