Desperate for Data Scientists - <AUTHOR> https://spectrum.ieee.org/view-from-the-valley/at-work/tech-careers/desperate-for-data-scientists <comment><USER2> Two remarks from an old AI guy here.

(1) If you're going to hop on the 'hot' AI/ML/DS bandwagon, make sure that you actually have 'work' for the highly coveted person you are trying to hire. 'Everyone else is doing it', 'all our clients have data, so selling them data- sciency projects shoudn't be a problem', 'Hire a first DS person, to have them start a brand new DS division' are things that probably are doomed to fail. If you weren't selling classic Data Analyses/Intelligence services before, chances are you are lacking the foundation in your sales and support staff, and maybe even your customer type, to be able to embed this into your company and into a new line of business. The risk your prize new employee will walk to greener horizons to work with peers on exiting real projects is high.

(2) Know where you are heading. If you just want to embellish your 'normal' projects with some gimmicks from the online 'Cognitive Services' api's offered by the various vendors, then yes, sending some of your regular staff on a quick 'boot-camp' style course will probably work. You don't need a 'data- scientist' for that. And if you hire a real one for that type of work he'll probably walk within 2 weeks.

<comment><USER3> >(1) If you're going to hop on the 'hot' AI/ML/DS bandwagon, make sure that you actually have 'work' for the highly coveted person you are trying to hire. 'Everyone else is doing it', 'all our clients have data, so selling them data- sciency projects shoudn't be a problem', 'Hire a first DS person, to have them start a brand new DS division' are things that probably are doomed to fail.

I'd add that if you have all these conditions and still want some hope of getting value from your Data Scientists, give your data scientist(s) a period of time (6 - 12 months) where they can discover how they can provide business value. Accept that during that period they may not provide the value you are hoping for. Support them if they recommend changes (e.g. different ways of collecting or processing data)

Lots of middle/upper managers want to hire a team of Data Scientists and want them them to deliver high business value from day one. They want a magic black box to solve their business problems. This results in exactly what you describe - constant fires, games, gimmicks (that look good on PowerPoint), and good people eventually quitting or getting fired.

On the other hand, if you give them 6-12 months to interact with different teams and identify relevant business problems, they'll find ways to provide value. Or, worst case scenario, they'll tell you (maybe by all quitting) that your problems aren't data science problems and the money you are spending on them is best invested elsewhere.

<comment><USER4>
        > if you give them 6-12 months to interact with different teams and identify relevant business problems...

I fully agree with your points BUT 6-12 months of self-directed, open-ended discovery just ain't gonna happen. This isn't the era of Bell-Labs, this is the era of PMP-certified project managers. In many places the "data scientist" will be dropped into a project and expected to start producing "deliverables", outcomes will be a mixed bag.

A more organic approach would be for teams to start picking up tools and skills for basic data analysis. We used to call it "stats". Doesn't have to be cutting-edge ML stuff to provide value and new opportunities for the practitioners. Of course whatever they start doing, HR and management will pump it up: mining a few hundred megs of csv files all of sudden will get called "big data".

Something similar happened ~10 years ago with the advent of "BI" (Business Intelligence). It was nothing more than folks realizing that their corporate databases have useful stuff in them and they could query them in an ad-hoc fashion _outside_ of the applications that typically use them to provide "business value".

<comment><USER3> >A more organic approach would be for teams to start picking up tools and skills for basic data analysis. We used to call it "stats". Doesn't have to be cutting-edge ML stuff to provide value and new opportunities for the practitioners. Of course whatever they start doing, HR and management will pump it up: mining a few hundred megs of csv files all of sudden will get called "big data".

Here's my one caveat to this based on personal experience. This happens to be why an entire data science team in one division of a large company quit over the course of a year.

If middle/upper management is purely focused on "show value on day one", it's very easy to get stuck in a position where basic stats and building Dashboards for simple data mining becomes the gold standard for what a Data Science team is for. Management develops an addiction to fast data mining projects that look good in PowerPoint, but don't actually deliver much business value because nobody is actually measuring business impact outside of how impressive it sounds. When the Data Science team tries to focus on projects that require more work that deliver more value, they're shot down because they don't fit into the "monthly management update" mentality.

You need a good management structure that realizes that quick data mining exercises are a stepping stone to delivering longer term value. They are not the ultimate goal.

</comment></comment></comment></comment><comment><USER5> As someone who has been employed in this space recently: No there is not.

There is however a lot of middle managers who have heard the term and want to pay crud vba developer wages for people to make pretty graphs in excel.

The credential inflation is also getting ridiculous. I have an MSc and I don't see how me being able to do quantum chromodynamics at one point in my mid 20s adds value to a business ten years later.

<comment><USER6> That's what I am thinking. Any half decent software engineer could probably produce the results they need from a few SQL queries. But "data science" is hot at the moment.

<comment> i-am-charmander Comments like this reinforce the idea that data science is too broad a term. __Everything __data-related is thrown into the "data science" bucket.

You describe what I'd call business intelligence or even advanced analytics. SQL jockeys (not a pejorative) really slicing into the data and presenting it in meaningful ways. There's usually exists some science behind these exercises.

Then there's machine learning, which also gets clubbed into the data science camp. Try capturing a convolutional neural network in SQL - you might go bald from pulling your hair out. It's totally __not __the tool for the job.

While perhaps any half-decent software engineer could maneuver his/her way around the great ML libraries out there to produce a CNN (or something of the like), I would hesitate to give them a project that involves _true_ science (bring out your pitchforks). Experimental design, hypothesis testing, etc. While being a SQL expert could help turnaround time and enable you to test more interesting hypotheses, I wouldn't leave most DBAs to develop and carry out an experiment.

So I guess my point is "data science" is too broad and overused a term to be meaningful.

<comment><USER7> A lot of "machine learning" is the more traditional statistics-based data science, not deep learning using a CNN or something...

<comment> i-am-charmander I understand that, but my point is good luck doing a lot of that "statistics- based data science" using just SQL.

</comment></comment></comment></comment></comment><comment><USER8> It would be nice if the story touched on compensation levels.

If all of those companies offered $400k/year, with guaranteed employment for 7 years, they would suddenly find no shortage of data scientists.

Although other industries might suddenly wonder what happened to all their statisticians / physicists / etc.

And two years later there would be a glut of data scientists who had just completed their master's degree.

<comment><USER9> Not every labor shortage problem is solved by adjusting compensation levels upwards.

The compensation has to match the value created.

<comment><USER10> Why is that, economically speaking? If there's a labor shortage, doesn't that mean it's undervalued in the first place? Forgive me if that sounds naive, I'm no economist.

<comment><USER9> Because it makes the assumption that the system has no constraints, which is often untrue.

First of all, you have to ask yourself _why_ there is a labor shortage. If the underlying reason is insufficient pay, then sure, increasing pay will fix it. This is often the case for fungible talent -- incentivize something enough, and people will shift resources to it. However in many situations, certain types of work are not fungible.

There are natural barriers to entry and qualification issues. Surgeons, for instance. You can incentivize and compensate all you want, but the fact is, not everyone is cut out to be a surgeon, so you have a funneling effect. It's not even a matter of pay.

Then there's desirability issues. Deep sea welding is a highly specialized (and dangerous) trade that pays handsomely, but not everyone wants to do it.

Some physically demanding jobs also have natural attrition issues in spite of compensation. When the inflow of talent is smaller than outflow over a long period of time, a shortage results.

There's also a training and timing issues. Let's say it's 2011 you want someone who can build the kind of infrastructure that powers Netflix, in 6-12 months. The talent and experience pipeline would still have been brewing at the time, so there's going to be a temporary shortage of talent and experience until folks gain experience and mature in the field. In that instance, you can still hire and grow personnel over time, but there would still be a shortage of existing talent.

Talent pipelines take time to build, are highly dependent on talent pools available and the ecosystems around it. Just throwing money at the problem doesn't always work. Compensation is just one end of it (the opportunity end); there are also significant long-term investments needed on the other end (the cradle end, which includes education, development, etc.). Texas Instruments did this -- they funded what eventually became UT Dallas... but it took many many years and the outcome was uncertain.

There's geographic issues. To use an extremely unlikely example, let's say you wanted someone at the level of Jeff Dean or Sanjay Ghemawat, but you would need them to relocate to Podunk, Iowa. Very few people at that level would want to relocate for any amount of pay, hence a shortage. I realize this is a pretty extreme example, but I worked at a company in an undesirable part of the country and it was difficult to get truly talented folks to move out there even with significant premiums on compensation. Now you can keep increasing compensation until you get someone who's willing to move, but they're usually not the kind of talent you were looking for in the first place. Also there's a break-even point at which the compensation doesn't make sense for the value the position is likely to generate, so companies will just not hire.

My point is there are all kinds of _real_ and _complex_ reasons why increasing compensation alone will not solve all shortage problems.

<comment><USER8> Thanks for the great rundown of some factors! I would like to take issue with just one part:

> Very few people at that level would want to relocate for any amount of pay, > hence a shortage.

I'm not sure if you're using hyperbole or not. But I'm guessing that if you paid $1 million USD / year, there would be no practical shortage of data scientists willing to live in Podunk, Iowa.

<comment><USER9> There is a cost-benefit proposition to every hire.

Unfortunately, the issue there is that the majority of data scientists don't generate $1 mil of value (exceptions exist of course), and so it's difficult to justify that level of compensation to management, which means the position may never get created in the first place.

Absent an existential threat to their bottom lines, companies will just muddle on without hiring.

This may be different in high growth companies, but in most traditional companies that aren't sloshing around in VC cash, to increase headcount you need to provide a value justification vis-a-vis salary (unless it's for cost center positions, but even then..)

There's also an underlying assumption that talent flocks to the highest bidder.... but most HR folks will tell you that it's more complex than that. There are many quality of life issues that come into play, like weather, peer- group, spousal happiness, etc. Money doesn't buy everything, and humans aren't optimizers but satificers.

I'm a sample of one, but for me, I would absolutely not move to Podunk, IA for a $1mil salary. I'm happy taking a lower salary living in a city and intellectual milieu that feeds me. The kind of person who would take the $1mil may likely not be the kind of talent you want.

</comment></comment></comment></comment></comment></comment><comment><USER11> The market for serious data scientists is growing, but is not as dire as these articles make it seem.

The private sector has wrapped 'senior data analyst','BI developer','data base administrator', and 'data engineer' into 'data scientist', along with the ML/AI/stats roles you would expect.

So sure, when you put all the roles that touch data into one basket, and we're in a world that demands increasing familiarity with data systems and data analysis techniques... a gap is going to appear. Color me shocked

Typical data scientist role I see posted is looking for: \- SQL \- AWS/Azure \- Tableau/Power BI/Other BI tools \- Python/R \- Increasingly Hadoop/Spark

With Java/Scala/C#/C++ listed as 'nice to haves' along with 'machine learning' and 'big data'. These are not research roles, and while they often want a masters degree or more they really don't need them.

If you fit that bill you can probably get a role paying $95-120k just about anywhere in the US. You'll wrangle data, make pipelines, build dashboards and occasionally deploy a model or two when the planets align.

If you want to/are capable of doing more ground breaking/cutting edge work, these roles will crush your soul in 8-12 months.

<comment><USER5> >If you fit that bill you can probably get a role paying $95-120k just about anywhere in the US. You'll wrangle data, make pipelines, build dashboards and occasionally deploy a model or two when the planets align.

If you expect to pay that little to anyone who knows the above to any degree you will have a bad time.

A number of projects I have joined absolutely look like someone treated them as a way to learn the technology before moving onto better paying jobs, leaving the business with a terrible mess that's largely unfixable without a rewrite. You're better off hiring one developer that knows what they are doing at above market rates than three who don't for the same price.

<comment><USER12> >If you expect to pay that little to anyone who knows the above to any degree you will have a bad time.

I believe you misunderstood his point. The $95-120K figure is for people who do _not_ do ground breaking research. Anecdotally, I know people who get hired for data science who are doing very little data sciencey stuff (a bit of data manipulation/slicing and lots of plots - almost no actual ML). These people should not get paid more than a typical SW developer, and decent SW developers should get paid more.

<comment><USER5> > \- SQL - AWS/Azure - Tableau/Power BI/Other BI tools - Python/R - > Increasingly Hadoop/Spark

Is above and beyond a typical SW developer. That's three languages and two dsls.

If you expect that on a 120k salary, you will be what you pay for, e.g. a code monkey living on peanuts.

<comment><USER12> First, I don't think he meant they expect you to know _all_ of these - just a subset.

Second, most people I know who know R or Pandas or other Python numerical stuff (NumPy/SciPy) are _horrible_ programmers. They're good at numerical work - but that doesn't translate to good coding skills. The bar is not that high to know it.

It's somewhat similar with AWS/Hadoop. Most people I know have taken learn these (online training, in person classes, etc) with very basic SW skills.

I don't doubt expert programmers exist who know these technologies - I'm just pointing out that it's not that hard to learn some of these. Plenty of people learn these without knowing much SW. To be a good SW developer takes a lot more.

>If you expect that on a 120k salary, you will be what you pay for, e.g. a code monkey living on peanuts.

You must be in the Bay Area. A 120K salary is above the median for SW developers for most of the country, and grants them a great lifestyle: Big house, two cars, vacation travel, etc. Hardly "living on peanuts".

The GP was clear he was not talking about the Bay area:

>If you fit that bill you can probably get a role paying $95-120k just about anywhere in the US.

Finally, the point is: If you studied actual Machine Learning, he is pointing out that these jobs are mostly not that, and you will just do a boring job.

<comment><USER5> >You must be in the Bay Area. A 120K salary is above the median for SW developers for most of the country, and grants them a great lifestyle: Big house, two cars, vacation travel, etc. Hardly "living on peanuts".

I'm not in the US currently. I have worked with teams in New York in finance and the wages expected were multiples of 120k. Likewise in London and Honk Kong.

<comment><USER12> Most of the US is not California or the NE.

When someone says "most", they mean the outliers are not included. When one says "most Americans can't point out Egypt on the map", it's rather pointless to give examples of PhD's who can.

<comment><USER13> That made me wonder, though, in this particular context, are these really outliers?

Even just counting population, the Bay Area's CSA is 8.8M, NYC is 23.9M, Boston is 8.2M and LA is 18.9M. That's already over 18% of the population, just in the major urban centers of California and the Northeast.

It's not much of a stretch for the percentage of tech jobs to be in those areas, and less of a stretch to imagine that the vast majority would be in some kind of high-cost area.

I wasn't able to come up with suitable web search terms to find any numbers, but perhaps someone else has.

<comment><USER5> Throw in Seattle and there are no tech jobs outside those areas to a first approximation.

I don't know where the people above are getting hired, but the only places I've ever seen outside those major cities are incidental programming jobs to babysit VBA code that runs some business that has nothing to do with computers.

<comment><USER12> Denver, Austin, Houston, North Carolina, Chicago, Atlanta, Salt Lake City, Boise to name a few. Granted, some of these have recently become pricey, but still nowhere near Seattle levels.

Look at all big companies (and not just the recent Internet ones of the last 20 years), and look at where they have sites. Jobs in most of these places.

<comment><USER13> > Look at all big companies (and not just the recent Internet ones of the last > 20 years), and look at where they have sites. Jobs in most of these places.

But how many of those jobs are technical/programming (in those places)?

Even by population alone (Chicago, Houston, and Atlanta making up the bulk), that's under 10.7%, less than CA+NE (and only half of CA+NE+Seattle?). That persuades me that these lower-cost areas are the exception, not the rule.

<comment><USER14> Not sure why you'd compare Houston to all of California?

[https://en.m.wikipedia.org/wiki/List_of_metropolitan_statist...](https://en.m.wikipedia.org/wiki/List_of_metropolitan_statistical_areas?wprov=sfla1)

Just looking at top 50, you've got a real geographic spread - about half of the people are in CA/NE, the rest are split pretty evenly between the four quadrants...

<comment><USER13> > Not sure why you'd compare Houston to all of California?

I believe everyone in this sub-thread has used "CA" to mean the SF Bay Area plus greater Los Angeles (and certainly I have). Similary, "NE" was just shorthand for the big cities therein, not the whole region.

Ultimately, population is just a _very_ loose proxy for number of tech jobs.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER15> I feel like I hear just as many people getting degrees in "AI" nowdays as plain CS degrees. A much higher percentage than five years ago. It feels like everyone and their grandma is minoring or majoring in AI or machine learning. And when I talk with companies about it, they're lukewarm at best.

I really wouldn't bet my personal career at this point on going into this area. It'll be filled to the brim sooner than you expect. It's a much better, and just as intellectually satisfying, bet to just go into plain traditional CS in my opinion.

(A counter argument is that I felt about the same sentiment around three years ago, but my intuitions doesn't appear to have manifested, so maybe I have poor intuitions, or I'm missing some piece of the puzzle.)

</comment><comment><USER16> I've heard it said, "A data scientist is a better statistician than a typical programmer and a better programmer than a typical statistician."

I 'graduated' from the Data Analyst Nanodegree @ Udacity in my spare time, and have experience with both. I lack a background in math beyond university Calculus, though, so that seems the major barrier for me to get a foothold in the industry.

<comment><USER17> The other way to phrase that is "A data scientist is a worse programmer than a typical programmer and a worse statistician than a typical statistician." If you need a programmer, hire a programmer. If you need a statistician, hire a statistician. I wouldn't ever hire someone who took a bootcamp that is "a better plumber than a typical doctor and a better doctor than a typical plumber."

</comment></comment><comment><USER18> I'd have a hard time finding a job in ML for $300K but I'd have an easy time finding a job in PHP contracting for $350K. This is despite having 10 years experience as a ML practitioner.

I quit my last job after bringing in over $100M in profit and only getting a $30K bonus.

Now it would take ~$500K and an 'eat what you kill model' for me to consider going back to work.

These days I make my money from licensing ML products which gives me a lot of time to study - which is what I enjoy doing.

<comment><USER19> where do you find php contracting for that which lasts long enough to make 350 a year?

<comment><USER18> Initially one of those elite consulting marketplace things. And then my own pool of referrals. I hate PHP so I set my rate higher on those projects and people still pay it.

<comment><USER20> What is an elite consulting marketplace? Where do I find one?

</comment></comment></comment></comment><comment><USER21> Can anyone with industry hiring knowledge comment on this gap?

Maybe I've misunderstood but from HN / other online discussions it seems that employers in Data Science have been increasing education requirements to MSc, PhD, and so forth for many jobs that more appropriately require a strong statistical understanding combined with rigorous data processing skills and some imagination. Is this a real "skill gap" or an attempt to create artificial scarcity of hot jobs?

<comment><USER22> IMO, as companies' data science capabilities mature, people with actual skills are needed.

5 years ago DS was hot and new. As a middle manager you would call up a consulting meat farm and ask for data science consultants: presto, you have a data science team. These people would, likely, be random BSc's with a SAS base certificate.

In 2018 data science, and data driven decision making, is still hot, but hiring managers know they need people with a formal education. So, education reqs go up and scarcity increases.

<comment><USER23> +1 because mooc style learning alone just isn't enough in most cases.

<comment><USER24> What would you say is missing from a mooc-based training?

<comment><USER23> Having looked through the contents of a couple of data science moocs they often try to give intuition without math.

When trying to put it into practice, if we go the slightest distance off piste we are lost without the math. So it's a fragile form of knowledge.

If we take someone who's already strong at math from some other training and teach them more stats/ML concepts and intuition, then great. But the value add of the mooc learning is quite low in that scenario.

The best alternatives (barring a formal course) imo, are text books that have exercises like Bayesian Data Analysis by Gelman.

</comment></comment></comment></comment></comment><comment><USER25> I've had a few friends express interest in pivoting their domain knowledge + small programming knowledge into a Data Science job. Are there many success stories or guides out there for those who make such a transition? I went the formal route of getting the CS degree, so I never really thought about how you turn a non-CS degree into a CSy type job.

<comment><USER26> Most data scientists I know don't have CS degrees but instead a background in physics, biology, math, economics or statistics.

For a data science role focused around exploratory analysis, model building and idea generation (as opposed to a more engineering-focused data scientist) I'd say that a basic understanding of programming concepts and familiarity with the most essential tools (version control, Linux, Python/R) is probably enough to be accepted into a data science role. Data science degree programs are still quite novel so most companies don't expect to recruit people with such a degree, hence there are good opportunities for people with a STEM education I'd say.

<comment><USER27> I've actually seen many many job postings looking for people with a PhD in Data Science which is truly mindblowing because there are only 2-3 such programs in the Country and most of them just started so none of their students are even on the job market yet.

<comment><USER28> ummm you are probably wrong. those job postings probably want people with PhDs in Computer Science/Statistics/Applied Math that did hard things with data.

</comment></comment></comment></comment><comment><USER29> Any time I see businesses complaining about shortage of skilled labor, I read it to mean they aren't willing to pay enough to attract the skilled labor they want.

The formula is simple. Businesses are in business to return as much money to investors as possible. IE: they're all about the money. Employees, more and more, are saying, "So am I. You want yours, so I want mine." Pay more and you'll attract more talent. Simple.

</comment><comment><USER30> In my experience, the title 'Data Scientist' has become to mean data analyst at most companies, meaning working with Excel, Tableau, and SQL. Maybe R if you're lucky.

Companies doing ML/AI will usually have a small team of Research Scientists who mostly hold PhDs, and a team of supporting ML Engineers.

<comment><USER31> I'd be more inclined to think that at a startup, because of the lack of infrastructure, a data scientist will do a lot more than research.

<comment><USER32> It's even worse at established companies because data scientists still only do the data plumbing and simplistic analytics tasks, but not due to anything reasonable, like the "many hats" needs of a startup, but instead because of IT dysfunctiom and bureaucracy.

<comment><USER31> Think it depends but it does happen. We've been able to carve ourselves out of IT with our own infrastructure and our data scientists specifically focus on research and analysis. The only issue is when we try to get anything to prod and we hit IT.

<comment><USER32> > "The only issue is when we try to get anything to prod and we hit IT."

Which inevitably means the one person on the data science team who is good with linux and docker suddenly becomes the IT wizard, and their time gets sucked up by having to find ways to go around utterly stupid barriers and tactics used by IT to avoid doing work to help you.

Source: I am currently this person for my team.

<comment><USER33> As someone who is about to be tasked with building this sort of infrastructure for a hospital system what would be your dream architecture? I hate IT hurdles and I am hoping I can avoid building them into our infrastructure.

<comment><USER32> The number one thing is to make containers a first class deployment and provisioning artifact. As long as dev teams can control their own containers, they will be able to do what they need no matter how arbitrary or assumption- breaking.

Do not ever require dev teams to go through IT to get their chosen tools deployed to the right places or with the right resources provisioned. Never.

This is the root of all evil with infra teams: if they see themselves or their mandate as being gatekeepers of provisioned resources, then dev teams have lost and you as a data scientist / ML engineer, you'll never get your work done.

As an ML engineer, I want to define the entire runtime and development environments of any analytics artifacts or web services that I create, and to change these environments as needed, as indicated by what's required to get the job done.

Let me define containers, hook them up in whatever CI tools are used, push and pull them from some internal container repository, and describe the configuration for the resources they need. Offer that as the contract to dev teams and then infra's job is to maintain the underlying data center that physically supports running the containers and occasional hand holding for special exceptions, networking, secrets management, and cost tracking.

<comment><USER31> Very well said. I can't stress the container environment enough not to mention the ability to control the toolset.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER34> As someone who has been producing value in a data science/machine learning role for multiple years, it's disheartening to see comments that I may be blacklisted from positions due to "only" having a bachelor's degree.

Somewhat non-humbly, I was valedictorian at my high school, I triple-majored at a respectable Big 10 school, I actively use all 3 majors on a daily basis, in a foreign country, and sometimes in a language that is not my mother tongue (as an American).

I can't justify spending time and money on a master's degree (millennial wealth problems) where many courses would just be putting a formal, academic spin on ideas that I'm familiar with from a practical business-value-producing point of view.

Any advice on how I can effectively jump off the black-lists?

<comment><USER35> Maybe a bit (a lot?) out there: why not go for a business degree instead, e.g. EMBA?

You mention "foreign country" so I'm guessing you can have access to good curriculums without paying the US premium on education (or just go for an online course).

Pros: the time and money is not "wasted" as you actually pick up new skills; the MBA card should be enough to trump any education requirement; and you become that most desirable of hybrids: the tech/data guy who can talk business (or vice versa).

Cons: significant time (and money) investment; doesn't help you get expert DS jobs (you'd be aiming for team/program manager, consultant, etc)

My own experience: completed an EMBA in 2017. Ranked in FT's top 10, the program cost was around 50 k€ (it's increased a bit since) and I was able to get 25 k€ of outside funding. The program I followed lasts 2.5 years, meaning I was able to do it while keeping my job (and having a kid) without losing my sanity or my wife. Landed my dream job just before completing the curriculum for a nice 40% pay increase (not saying the EMBA alone had that effect, far from it —but it definitely helped).

<comment><USER34> Huh. This is actually a route I hadn't considered. Thanks for pointing it out! I will definitely consider it as I continue researching my next opportunities...

<comment><USER35> Glad that helped ;)

If you want more feedback on my personal experience, feel free to reach out (will update my profile with an email address).

<comment><USER36> Is there any difference in terms of curriculum between the EMBA and the MBA? As in would people consider the EMBA a "lesser" MBA so to speak?

I'm an engineer looking at MBAs right now too, but it seems like a huge investment.

<comment><USER35> EMBAs are usually part-time over 18 to 24 months, MBAs are full time over 12 to 24 months.

So an MBA naturally has a bigger (as in more in-depth) curriculum than an EMBA. It is also a significantly higher investment in terms of time and opportunity cost, since you're not getting paid during the program.

EMBAs compensate with 1) more experienced participants (so in theory you don't need the introductory classes) and 2) a lot of pre- or post-readings (e.g. my corporate law module was 12 hours in the classroom, but you were expected to have read the 400-page book, and the numerous case studies).

But the bottom line is that you don't go into as much detail as you would during a full-time program. OTOH, since EMBAs are attended by "senior" employees (managers / VPs / directors / etc), and because they're part-time, what you learn is usually directly relevant and applicable in everyday work - and you usually get to work on real-life problems (yours or your teammates) during classes.

I'm not the best placed to say whether an EMBA is considered a "lesser" MBA. They don't really fill the same niche. An EMBA is a career booster if you're say a technical manager and want to move into business or senior management. An MBA is when you haven't started working (or are still junior) and are looking for a fast track to C-level, or to work in a specific area (e.g. consulting, finance, etc.). So basically MBA vs EMBA is mainly a function of your current experience level.

</comment></comment></comment></comment></comment></comment><comment><USER37> I view modern AI as a combination of things that are too early and too late - data science hasn't changed much since the 1990s, but people have a misconception that there is still a lot more to do in data science because of the undeserved excitement around deep learning.

<comment><USER38> Errm - I am pretty sure that we didn't have MCMC or causality analysis in the 90's.

Or R-studio.

<comment><USER39> Perhaps I've misunderstood your desire to be sarcastic, but I think your timeline is off. To give two concrete examples, Geman and Geman's "Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images" paper is from 1984 and Holland's "Statistics and Causal Inference" paper is from 1986.

</comment></comment></comment><comment><USER40> You don't need to have a CS degree to be in "Data Scientist". In fact, I would argue that you need a statistics degree. There is a difference between a data scientist and an engineer.

<comment><USER41> The problem with stats degrees is twofold:

1) They often lack experience with messy (i.e. real) data 2) They sometimes lack experience with programming.

Data science, as currently practiced requires some CS, some stats (with a focus on experimental design/observational analysis) and communication skills.

These skills are rarely taught together in undergrad, but lots of PhD's acquire them by accident, hence the propensity of PhD's to end up in these roles.

</comment></comment><comment><USER42> Who are you kidding? They aren't hungry. There is lots of propaganda flying around, but they aren't really even looking hard.

HR games the system.

</comment><comment> another-cuppa I was hired as a data scientist for a large company that thought they needed data scientists. They didn't and I added no value to the company. I left after about a year. If there are many people like me then that might explain the shortage.

</comment><comment><USER43> How do you see the chances of someone without a degree getting an entry-level, low-paid, remote job in this space? What would be things that could help out? Certifications? Nano-degrees like Udacitys? A portfolio of public work?

<comment><USER44> It's highly unlikely. There is a lot of competition for entry-level jobs and not many remote positions available. The barriers to entry are exactly academic qualifications over certifications and portfolios.

</comment></comment><comment><USER45> I think that the challenge to hire a good data scientist is that it requires skills in both:

\- hard science: strong statistics or ML to analyze data, some CS to organize the data.

\- soft skills: strong business sense to understand what is important, some communication skills to explain it.

A data scientist needs very good skills and experience in both, but they are somehow anti-correlated in my experience.

<comment><USER31> That's why we usually have a team that complements each others' skill set. The data scientist while great to have everything does not necessarily need to have everything. I tend to focus on their core strengths and depending on what their growth/career aspirations are, help to develop the skills. If it's soft skills, happy to help them develop that edge.

</comment></comment><comment><USER46> I'm teaching stats and scientific computing at university. I thought that data science might be a good exit plan in case I get feed up with academia. However, even though data scientists seem to be highly sought after in Germany (judging from what I see on linkedin), the salaries are fairly disappointing.

</comment><comment><USER47> Ah yes. A shortage. Someone didn't do their planning. Bubble bubble tool and trouble. Or is this about more visas? And paying people less.

Seriously though, what exactly is a data scientist? What is the requisite skill set? Is this masters level entry or would my feline require a Pointy Headed Degree upgrade to play this game?

<comment><USER47> Masters and PhD are required for particular employers. Others not so much but there are caveats. High probability of being the "data" guy who is then made to be the database admin.

Better summary at: [https://blog.insightdatascience.com/from-phd-to-data- scienti...](https://blog.insightdatascience.com/from-phd-to-data- scientist-17209c66a9d9)

</comment></comment><comment><USER48> Maybe they're all desperate for good data scientists? The truly good data scientists that I've found have actually been scientists.

Literally all "data scientists" I've found in a corporate environment have been so useless it is depressing.

<comment><USER49> My experience is that most valuable data scientists are those who are good engineers.

</comment></comment><comment><USER50> I'm available for the right data science gig. Somewhat unconventional CV available on request. I'll try to bring my competent but intellectually foggy business analyst colleague along for the ride if you like.

</comment><comment><USER51> I smell buzzword HR bingo

<comment><USER52> Yeah.

I find "data science" a cringey expression and I'm not looking for a job currently, so my LinkedIn emphasizes my strengths according to my ego/self- appraisal, not a "desperate" reaching out for fashionable job titles.

</comment></comment><comment><USER53> Maybe oral communication skills are overrated. Who knows? People currently talk a lot also with their hands.</comment>


Y Combinator Numbers - <AUTHOR> http://ycombinator.com/nums.html <comment><USER3> Back of napkin math:

Funding of $25,000 per startup (funding plus costs) x 210 = $5,250,000 in funding costs. To factor other costs and be conservative, double that to $10.5 million.

YC usually takes 6% equity. Assume that's diluted in half, conservatively again, to get 3%.

Total current valuation of 4.7 billion: $4,700,000,000 * .03 = $141,000,000

So... using all conservative numbers, YC's return has been .... $141,000,000 / $10,500,000 = 13.4

Whoa... did I do that math right?

A return of 1340%

That's... pretty astounding. Talk about a win/win/win situation for everyone involved... if the order of magnitudes here are even close, well, massive congratulations to everyone involved. Wow. Wow.

<comment><USER4> My problem with pg's numbers is that they are almost entirely theoretical. The valuations don't represent anything close to liquidity, and the chances of those companes generating anywhere near that return in real terms are slim at best. The portfolio might be "worth" 4.7 billion, but what are the chances they'll generate a real return anywhere close to that?

I'm not arguing against pg's methodology, as I realize it's fairly consistent with standard practices, but the ycombinator investors are nowhere near as wealthy in real terms as these numbers would suggest. If airbnb were to fail that would destroy a huge amount of that wealth, with no ability to extract any value before then.

None of this is meant to rain on YC's parade. I have a great deal of respect and admiration for all involved, and congratulate them heartily on the very real successes they have had and continue to have. I'd simply be heasitant to describe YC as being "worth" 5BB, given a reasonable expected cash return.

<comment><USER5> This is obviously a valid criticism.

But it would be far more valid to criticize any alternative valuation that fixed the problem you're addressing. For example, it would be conservative to value all of YC's startups in terms of their cash and receivables, plus the liquidation value of their furniture and servers. But then it would be trivial to say "Hey! AirBNB isn't an asset play. It's valued based on growth. The asset value is a worst-case scenario, but most of the other scenarios involve a lot of revenue growth and free cash flow."

Basically, is there any valuation method you have that would make you indifferent to a bet on the portfolio's valuation going up or down? (i.e. anything you think would produce a "fair" market price?)

Actually, more directly: how much of your net worth would you invest in shorting YC's portfolio at this inflated $4.7bn valuation?

<comment><USER4> >Basically, is there any valuation method you have that would make you indifferent to a bet on the portfolio's valuation going up or down?

No, there isn't. Angel investing is an inherently speculative game. What I would say is this: a YC investor could reasonably claim to have seen a return on any shares he sold, or any shares he holds of a company that has gone public. That's it. Beyond that, all investments are good only in theory. Were I to buy a house, I would decide what was affordable based on the former, not the latter. The same is true for my contributions to my children's college fund or my retirement fund. In short, all of the things that actually matter in life would be based off of what I had sold and what shares I held of public companies. To make any long term financial decisions based on the projected worth of companies that had, at best, gone through a successful series A or B would be foolish. They're nice numbers to think about, but they don't mean a damned thing until they're putting food in my kids' mouths.

Let's illustrate my point like this: Suppose you are given the chance to pay 1MM for some good that has a 20% chance of being worth 1BB in 5 years. If you can spare the million, then this is a fantastic investment by any measure. Having bought into this investment, however, how would you describe your net worth? Your investment is valued at 1BB. The fact that it will likely never be worth that much doesn't change its value. Heading off the foreseen criticism, let's say that savvy investors would only say it is worth 200MM, the ultimate value times the expectation of realizing it. Still, that 200MM is fairy money. It doesn't exist, and you can't do a damned thing with it until you somehow manage to convince someone to give it to you. To claim that you're "worth" 200MM is silly, as you can't spend a dime of it.

This is the problem I have with pg's numbers: They sound great until you ask what the cash-out is. Saying "I've invested in companies that are worth 4.7BB" is great, but it doesn't say anything useful about how well your money has been used. Remember, prior to investment it was real money that could buy apples and tuitions. Now it's theoretical money that could buy a whole lot of apples, but only under a specific set of favorable circumstances.

A final reduction: There is a claimed ROI of ~13:1. Being generous, I would ask this: how many people who invested 1MM into YC have pulled 5MM out?

<comment><USER5> _The fact that it will likely never be worth that much doesn't change its value. Heading off the foreseen criticism, let's say that savvy investors would only say it is worth 200MM, the ultimate value times the expectation of realizing it._

A market price is a price at which someone with good information is indifferent between going long and short. If you're applying some discount to the $200mm based on the possibility of a $0 outcome, you'd have to put your money where your mouth is and sell short at that level.

If you'd made $200mm through some other means, and bought that 20% chance for a billion dollars at the market rate, would you treat that as a loss? At what market value would buying that constitute a good deal?

The whole reason we have valuation is to answer questions like that. The question of "What is this worth?" is very different from the question "How much cash can I extract from this now? "What is this worth?" means "What is the net present value of all the cash I can extract from this asset, assuming I extract at the optimal time?" It's the difference between the value of a couple scraps of gold in a creek and the value of a massive gold mine.

 _A final reduction: There is a claimed ROI of ~13:1. Being generous, I would ask this: how many people who invested 1MM into YC have pulled 5MM out?_

That's not the point, though. Startups are illiquid, so it doesn't make sense to sell. Plus, the YC people just don't seem to need that much money. Some people get rich in order to buy jets and vacation homes--the impression I get is that the YCers got rich enough to spend their time working on cool stuff, e.g. YC.

</comment></comment></comment></comment></comment><comment><USER6> Long term I think the big question is how much of the success of YC companies is directly related to the external benefits of being YC companies. There's no question they tend to get far more attention on every front (VC/valuations/press/bizdev/hiring, etc) than they otherwise would.

In a few years when there are 1000 YC companies it will start to mean a lot less to all those people who have been helping YC companies succeed to date. It probably already means a lot less to a VC to hear you're a YC company, because they've met 300 of them.

If it's the YC halo effect that's driving high returns it will be a big problem if it starts to wear off.

<comment><USER7> >In a few years when there are 1000 YC companies it will start to mean a lot less to all those people who have been helping YC companies succeed to date.

Harvard, Yale, MIT, Stanford, etc. seem to be doing rather well, as far as higher education goes.

And I would imagine those that have partnered with them are doing okay too.

Quality matters.

>It probably already means a lot less to a VC to hear you're a YC company, because they've met 300 of them.

I doubt that. A VC is just trying to pick the next Google. Or the next Google founder(s).

<comment><USER8> Right on. I think as long as the quality of YC companies remains high, the presence of imitators will of course increase the value of the original, just as in higher ed.

Startup incubators may in some ways be easier to replace than universities (Harvard, Brown, etc. are still going strong after 200+ years), but they are subject to similar dynamics, insofar as there is team loyalty and networking benefits (as there obviously are). Personally, I would accept 10% dilution of my company simply to be an YCombinator alumnus since I think the benefits are and will remain that high.

</comment></comment></comment><comment><USER8> Far better than I would have expected, given the various spreadsheets, etc. that I had seen floating around. I was a bit skeptical about YCombinator's growth and the ability to mentor new startups, but with numbers like these (esp. the 34/36 number) I've a lot more confidence that they will be able to do just fine with more startups in each class.

In many ways, it also makes me comfortable saying that the "Harvard of startups" description is appropriate. Sprinkle in a bit of elitism based on merit, give people some general framework, a bit of advice, and a lot of free reign, and brilliant and hardworking people will make things happen. Much better to be a bit hands-off and cultivate excellence then micro-managing the actions of the less focused or ambitious.

This, even more than recent acquisitions, is the best news I've seen out of YC in awhile, and truly excellent. Congratulations!

btw, to pg or anyone else, would be a little bit interesting to see more about the power law distribution as it applies to investing. I'm assuming this is true across the field, but don't know for sure (do all vcs get the same distribution, or is it simply true of software startups, or is it simply true of software startups that manage to include big hits? Moreover, do you simply get big hits by funding enough decent sluggers, or are there other strategies to make sure you have a decent shot at the next Dropbox, Twitter, Google, etc.)

</comment><comment><USER9> It seems safe to say that Y Combinator's expectancy* as an investor is currently unsurpassed - no private investor, mutual fund, hedge fund, VC, etc. I've ever heard of comes close. Congratulations!

* expectancy = (probability of win * average win) – (probability of loss * average loss)

<comment><USER10> I guess if Warren Buffet were to start again with only a few million USD, like Y Combinator did, he would do really well in terms of percentage return, too.

</comment></comment><comment><USER11> A question for PG - What's been the most surprising numbers to emerge, in your opinion, up to this point?

You mention the concern that would be inherent if funding was at 100% prior to yuri with regards to an overly conservative stance. With, as you have stated, the real goal being finding the next google or facebook isn't a 90+% level almost as worrying ??? Wouldn't something closer to 70 make a better balance in terms of maximizing coverage to increase the chance of landing the outliers ?

<comment><AUTHOR> The most surprising thing was the number of companies with valuations between 30 and 60 million.

</comment></comment><comment><USER12> Please enlighten me if I'm missing something obvious here, but wouldn't a pretty interesting number be the percentage of funded companies that are _profitable_ today? I mean, maybe not as a measure of YC's success monetarily, but certainly as a success rate for choosing viable products/teams, no?

So to be clear. I'm most interested in how many of the 208 considered as part of the $22.4 million calculation are self-sustaining and making a profit today.

<comment><USER13> That would be the wrong metric. A sandwich shop could become profitable much faster than a Google, but I'd much rather invest in the next Google.

<comment><USER12> Well, it still has to be a good sandwich shop to succeed, and I'd rather invest in a successful sandwich shop than one which will flop. So I'm just curious to know the overall track record of success had by YC funded companies - from a perspective other than 'got more funding' or 'is valued at X'.

<comment><USER6> Investors would rather invest in a company that has a 1% shot at being Google than 100% shot at being a normally successful sandwich shop.

<comment><USER14> But can we make the assumption that YC companies are chosen because they have the potential at becoming the next Google, therefore if they are profitable they're on their way towards that goal?

It's a small assumption, but I think one that could be made.

It makes more of a success/failure comparison than a return percentage calculation, but it would still be really interesting to know.

<comment><USER12> Yes exactly. I definitely didn't mean to say I wanted the numbers for profitable _as opposed to_ the '1% chance of being Google'.

</comment></comment></comment></comment></comment></comment><comment><USER15> Terminology question: In footnote 1 it says

      "We're looking for companies with high beta. E.g. we're
      ok funding groups for whom the likelihood of failure is
      high but for whom success, if it happens, will be big."

I'm not sure what beta means here. My understanding had been that beta for an investment refers to correlation with a benchmark index (see <http://en.wikipedia.org/wiki/Beta_(finance)> ). So high beta for YC would mean startups that perform similarly to the average, which is probably not what YC seeks. Alpha ( <http://en.wikipedia.org/wiki/Alpha_(finance)> ), or risk-adjusted return, is what any investor seeks, but still doesn't seem to capture the idea of seeking very high-risk, very high-return investments.

Am I using the wrong definition of beta? Is there another term that better expresses this characteristic of investments?

<comment><USER5> Beta can also refer to relative volatility. A stock with a beta of 2 has twice the fluctuation of the index.

But in terms of asset allocation, you're right: PG is actually looking for alpha. You can get all the beta you want with leverage: just buy short-term out-of-the-money calls if you want extremely high beta.

(There's actually a lot to think about, here. Startups are pure "alpha" since the beta component is an expected return of zero.)

<comment><USER16> Beta does not measure volatility.

PG means alpha here.

<comment><USER5> <http://www.investopedia.com/terms/b/beta.asp>

"Beta" has two popular definitions. One is "market beta," i.e. the return you've gotten from the market you're in versus the particular trades you make in that market. I have only heard professional investors refer to this. The other kind is the one defined by investopedia. I only hear this mentioned by retail brokers.

If PG said he wanted high alpha, it wouldn't make sense to increase the probability of failure. All else being equal, higher alpha means a lower chance of failure (at any given volatility level, the higher-alpha portfolio has a lower chance of going to zero, since e.g. if beta is -99% and the lower- alpha portfolio generates -1% alpha, that takes it to zero. Right?

<comment><USER16> Return of a security = return due to correlation to the market + return due not due to correlation to the market.

Rs = BRm + A

It doesn't speak to general volatility at all. The definition linked is inane; Beta of 2 only twice as volatile as the market if the security is highly correlated to the market.

Beta is literally cov(Rs, Rm) / var(Rm).

In this particular case, if he wanted returns highly correlated to beta he would just invest in VCs. To get a beta of 2 he'd leverage the funds somehow and invest that.

He thinks he has an edge, which is contained in alpha (return not captured by market performance.)

<comment><USER5> I don't know. Still sounds to me like he's using the linked definition. _Which I agree is not a useful concept_.

People do cite betas. e.g. here's a quote page for Yahoo, on a big personal finance site, that cites the "beta" as defined earlier: <http://www.dailyfinance.com/quotes/yahoo-inc/yhoo/nas> It's entirely possible for a typical investor to only encounter that version.

Colloquially, if someone says "I like high-beta stocks," they mean "I like really volatile stocks," not "I like stocks that have some combination of high volatility and high correlation to some index." So the statement makes sense colloquially.

A more accurate version would be "YC looks for a moderately negative beta and expects a very high alpha." Since that describes companies that fundamentally alter their target industries--e.g. by harming all the established players, but still making money.

<comment><USER16> The only time I've heard people talking about high beta has been in the context of leveraged securities that track the indexes.

But what do I know, I only spent a decade on wall street.

</comment></comment></comment></comment></comment></comment></comment><comment><USER17> These numbers are astonishing. By my calculation YC has a cash-on-cash return of 67X!

Avg. value (22.4M) * Avg. equity (6%) = Value of average stake (1.34M)

Compare this to an investment of $20k, wow.

I am sure there is some sort of dilution that occurs, but anywhere in this order of magnitude is remarkable.

<comment><USER18> 6% is undiluted, you have to assume YC has been diluted quite a bit during fundraising. But even assuming a more realistic 3%, those are great returns.

<comment><USER19> Also, remember this is a power law distribution. So talking about an average amount of dilution is probably risky. The dilution on the single largest exit is going to move the needle the most.

</comment></comment></comment><comment><USER20> One way to measure is just on exists so far, which is a strict lower bound, but doesn't change with the market. I bet YC is in the black just from exits, and this can be measured explicitly. What is the sum of YC's returns from exits (and cash value for stock) over the sum of money spent on investments?

Measuring the value of companies that have grown dramatically since the last round if pretty arbitrary. If this estimate were done on the same companies in the same states but in 2009, wouldn't it be off by 50%?

</comment><comment><USER21> > When startups are acquired, acquirers usually give the founders incentives > to stay on. Sometimes to make the founders feel better these incentives are > included in the number the acquirer quotes as the acquisition price. In that > situation there are effectively two prices, the quoted price, and a lower > price investors see.

Wow - I didn't know this. Does anyone have any more detailed examples of this happening they can share?

<comment><USER22> What, of acquisitions with earn-outs? Search NEWS.GOOGLE.COM for "earnout".

</comment></comment><comment><USER23> I'm curious what the top 5 or 10 YC companies by valuation are, and what their valuations are. AirBnB? Heroku? Dropbox? What else is in the $1B range?

Most of this seems like it should be public information, since fundings that large are usually announced by press release. Has anyone compiled a list?

<comment><USER24> yclist.com

<comment><USER23> Interesting, but it only has valuations for exits. I'm kinda curious what valuations for going companies are, based on recent rounds of funding. The totals on YCList are nowhere near the $4.7B that PG quotes.

Actually, I'm surprised how many of the acquisitions were talent acquisitions for ~$2M/founder. Decent sized chunk of change, but you can get that much by going to work for Google, kicking ass, and then getting a FaceBook offer.

</comment></comment></comment><comment><USER25> "1 didn't bother because they were already so profitable" -> anyone an idea which one that was?

<comment><USER26> My guess would be Wufoo.

<comment><USER26> Never mind - that statement referred to a startup in a recent batch while Wufoo was part of a much earlier batch.

Interesting to note that there are probably a few Wufoos in the total YC universe that reached meaningful profitability without having to raise more money.

Those are the ones worth emulating even though they don't get nearly the amount of attention as ones raising mega vc rounds.

</comment></comment></comment><comment><USER27> I'd be interested in the valuations based on traditional things such as revenue, assets, cash on hand etc... not that these things necessarily matter to Y Combinator as a company making $0 that is bought for $1 billion is still significant, I'm just curious what percentage actually is profitable.

</comment><comment><USER28> This reminds me of my family's business. We do grant writing for nonprofit and public agencies, and people always want to know if they're going to get funded. We always say that we have no idea, for reasons discussed in this post: [http://blog.seliger.com/2008/12/21/the-worse-it-is-the- bette...](http://blog.seliger.com/2008/12/21/the-worse-it-is-the-better-it-is- your-grant-story-needs-to-get-the-money) :

"All this also helps explain why a "batting average" or "track record" figure is useless regarding general purpose grant writers, as we describe in this FAQ question: <http://seliger.com/faq.html#anchor5> . We don't know if our clients are going to come from Beverly Hills or from places where most residents haven't graduated from high school. From a grant writing perspective, a client from the latter place might be more likely to be funded than someone from Beverly Hills."

</comment><comment><USER29> Here's a question to PG:

I can imagine many reasons, but I'd be interested to hear from you why you chose to share these numbers at this time?

<comment><USER30> Possibly prompted by PG's interview with Charlie Rose last week on stage at TechCrunch Disrupt.

I was present, and a two things were clear from the interview:

1\. Charlie Rose had a few wildly incorrect assumptions about YC. (e.g. "so, to date, you've invested, what, $100M?")

2\. Paul Graham had not given a lot of thought to aggregate numbers. When asked, he about total the total value of the YC portfolio, he could only guess.

Edit: interview link <http://www.ustream.tv/recorded/14928956>

<comment><AUTHOR> Yes, the interview I mention in the first sentence of the post was the Charlie Rose interview. When he asked how we were doing, I quoted some numbers based on a back of the envelope calculation that turned out to be way off (an underestimate, fortunately).

<comment><USER30> BTW, I found that interview very enlightening.

I hadn't realized how _nice_ you are. Your real-life persona completely contradicts the caricature I had developed in my mind after years of reading your essays.

I was expecting Vulcan (hyper-smart, non-sentimental), but instead got Camp Counselor (humble, encouraging).

You should do more on-stage stuff.

<comment><AUTHOR> Seeming Vulcan is probably an artifact of conciseness.

</comment></comment></comment></comment></comment><comment><USER31> _$4.7 billion / 210 = $22.4 million, so the average value of startups we've funded is about $22.4 million.

Of course that could be cut in half tomorrow. Startup valuations are even more volatile than ordinary equities._

Couldn't one argue that startup valuations are sticky in a downward direction? The 21 largest companies are highly valued because they're generating tons of revenue (or even profits). Nothing short of a market collapse would hault that revenue generation, and it's hard to imagine such a collapse in the market of, say, peer-to-peer vacation rentals or cloud-based multi-platform storage.

It's entirely possible, though, that by "cut in half tomorrow" PG meant that Winter 2012 will include 210 companies. :P

<comment><USER32> _The 21 largest companies are highly valued because they're generating tons of revenue (or even profits), and nothing short of a collapse in their market could halt that income generation._

I don't think valuation of startups are strongly based on revenue and profit. At least not in a way where their revenue/profits force a certain valuation, rather than simply suggest that the potential and growth is there. (if that makes sense).

<comment><USER31> Yeah, absolutely makes sense. Here's a proof by cases on my point:

Companies are highly valued b/c they (a) generate income or (b) have the potential to generate income. (a) companies are trivial cases, since they already make money. Let's move on.

Looking at (b) companies: they're highly valued because market conditions + company fundamentals suggest they CAN generate income soon. Let's assume company fundamentals (leadership, labor) are constant and solid (trivial, since they're being advised by PG and other VCs).

Therefore, highly-valued companies that do no yet generate income can only be negatively disrupted by market forces. Based on the initial assumption about (b) companies' potential, I argue that (b) have been selected such that they are not particularly vulnerable to negative shocks.

Therefore, their valuations have a very, very high floor and are sticky in the downward direction.

<comment><USER32> What about AirBnB? You don't think a $500M valuation would be considered reasonable by many? AirBnB seems like a classic example. $25M in revenue and $1B valuation (40x). The market moving slightly to a more hardcore earnings based valuation could easily see their valuation cut in half (20x).

No less of a company, but a slight change (yet reasonable change) could drastically effect their value (as it would with most startups at a similar phase).

</comment></comment></comment></comment><comment><USER33> Congratulations, I'm sure you've already felt it, but I just wanted to say you've built something truly amazing that helps so many other people realize their dreams. And as an added bonus it validates your assertions on what a startup needs to do.

Visionary indeed!

</comment><comment><USER34> Interesting analysis but the statistic I find most disturbing (and selective) is the "average" value of startups funded. What would be more informative is the mean value. I'm guessing the mean value would be extremely lower since most of the total value is made by a few outliers on the top end.

By taking the average value in this case, it's likely giving an inaccurate picture of what the average YC funded company is actually worth.

Just a thought. I could be wrong assuming most of the total value does not come from outliers but my sense is that I'm right based on the news I've seen around YC companies.

<comment><USER35> He did show the mean. It looks like you're interested in the median. From a potential startup founder's perspective, the median is pretty important. From YC's perspective, mean is all that really matters (except to the extent that median affects the input to their program).

<comment><USER34> Yes, thank you. I meant to say median.

</comment></comment></comment><comment><USER36> >If it were 100%, I'd worry we were being too conservative in >who we funded."

Note to self: if you're succeeding in everything you're doing, you're probably failing to attempt awesome things in which you could succeed.

</comment><comment><USER19> >> That number is about as high as I'd want it to be. If it were 100%, I'd worry we were being too conservative in who we funded.

There is market feedback in this thought (and in the footnote where you cite 'beta'). Really what you are saying is that you are aiming for a level of conservatism/risk relative to the market.

I'm curious what the funding rates were in 2006 and how they changed. How much of the increase on funding percentages were due to being better at doing YC and how much of the increase was due to their being much more risk capital in the market?

</comment><comment><USER37> With the numbers of startups growing with every batch. Do you think it dilutes the value Y Combinator provides?

I wonder if pg and YC partners would be able to give enough time and attention to 62 startups.

YC brand and alumni network is stronger than ever. YC is great at scouting great entrepreneurs at early stage. At this point YC may be acting as a filter for investors.

The big question is, Will YC still the advisory role to create great products? or Is it now a funding school that provides a brand and access to Angels and VCs?

Would love pg, YC partners or alumni to chime in on this.

<comment><USER38> Currently the request queue for office hours is empty, meaning everyone who wants to meet with us has been able to. We have capacity for more office hours this week and will be adding more. PG added office hours for this afternoon, with more slots than there were requests in the queue. Things seem to be working so far.

</comment></comment><comment><USER39> "...so far at least the great majority of VC investments in YC-funded startups have been by the better firms. I'm not sure exactly why, since bad VCs are more numerous than good ones; perhaps the bad VCs assume the companies at an event as public as Demo Day will already have been picked over by the time they get a shot at them."

Or perhaps the lower tier VCs just can't compete with the "better" firms.

I find it interesting that you so readily equate "lower-tier" VCs with "lower quality" VCs. Is this a reliable rule?

<comment><USER40> _I find it interesting that you so readily equate "lower-tier" VCs with "lower quality" VCs. Is this a reliable rule?_

If you were a low-tier good-quality VC, surely you'd become known as a top- tier VC pretty soon?

</comment></comment><comment><USER20> Fun math: if

\- the average YC investment is $20K for 5% (400K valuation),

\- YC's share get's diluted in each company by a round giving up 30% of the company

\- the average YC company is worth $22.4M

...then that translates to a 43X return for YC. Woah.

</comment><comment><USER41> If YC holds common shares, then you need to adjust the expected value down quite a bit. On any exit that isn't a huge win, non-employee common shareholders are by far the most likely to get short end. The expected value is probably 10-50% of the fully diluted headline value of a VC deal.

But YC is clearly going to do spectacularly well. They deserve big congrats for what they've accomplished.

<comment><USER42> This isn't generally the case anymore--funding rounds these days tend to assign a 1X liquidation preference to both common and preferred shares, so assuming that a company gets acquired for more than its valuation, you shouldn't need to adjust EV down.

<comment><USER43> True, but there's still a very good chance of being acquired for less - <USER2> means you have to adjust the EV down.

<comment><USER44> Except all the maths here has only accounted for the top 10%. Those top 10% are fairly unlikely to be sold off at scrap value.

<comment><USER41> You'd be surprised. The ones that raise the most money also have the hardest time getting over the preferred total. The purchaser will take care of the employees it wants, but other common shareholders often end up out of luck.

</comment></comment></comment></comment></comment><comment><USER45> This investment strategy probably doesn't have a ton of capacity though. I could make a thousandfold annualized return on a dollar by reselling snacks outside my apartment for ten days.

</comment><comment><USER46> Are there any numbers on how many YC companies are now defunct? Would be interesting to see how this compares to the anecdotal 9/10 startups fail in the first year.

<comment><USER40> There was a thread about this at some point. The answer was "a bunch", though the numbers were complicated. Some had merged, others experienced "exits" which were really just talent acquisitions and shut down all operations on their product. Very few really "failed".

The 9/10 startups thing was always a myth, though, especially if you're talking about startups which get to the stage of getting _any_ funding at all. It's more like 1/2.

</comment></comment><comment><USER47> "like any portfolio of startups ours has a pretty steep power law distribution. If we can produce accurate estimates of the values of the top 10%, we'll have a sufficiently accurate estimate of the total value. "

No... this is incorrect. Power laws looks like this (imagine the "top 10%" in green): <http://en.wikipedia.org/wiki/Power_law> so the median value, or even showing a histogram of all values, would be WAY more informative. My hunch is that the "22.4m" average valuation is way higher than most YC grads.

<comment><USER48> You've misunderstood the math. The $22.4M figure was reached by assuming the bottom 90% of companies are worth $0. Therefore, it's strictly a _lower_ bound on the mean valuation.

<comment><USER47> You're right.

</comment></comment></comment><comment><USER49> Does it make sense to generalize about 200 startups based on 5 exits?

</comment><comment><USER50> Thank you for posting about this I find it fascinating.

</comment><comment><USER51> Impressive.

What about the control group though? You know, the 50% that were deemed fit for YC after the interview but lost at the coin toss backstage. How did those compare?

<comment><AUTHOR> I don't understand. Do you mean startups we fund that don't do well, or startups we don't fund?

<comment><USER52> My understanding is that the person who asked that question wants to measure the value YC is adding, separate from YC's ability to select promising teams. Obviously there's no way to separate those things, so s/he suggested an absurd experimental setup in which half of the accepted startups, chosen at random, would be excluded from YC.

<comment><USER10> Don't just exclude them. Replace them at random with normally rejected companies.</comment></comment></comment></comment>