Why switching jobs is almost always a good idea - <AUTHOR> http://alexpotato.com/blog/why-switching-jobs-is-almost-always-a-good-idea/ <comment><USER3> If you're not happy in your current role, make your employer aware of that fact. If you believe you are underpaid, say it. If you're overworked, say it. I'm not saying you'll necessarily get results but I think people all too often look for the door when what is making them unhappy can probably be resolved where they are. Of course if you're miserable and need a change that's a different story. But remember that if you're good, losing you is going to hit your employer hard. The cost of replacing good people is so high, any good employer will try to resolve any issues you may have in order to keep you.

<comment><USER4> I'm curious to hear your advice on the following: At once point I was hired by a bigCo, had been working at a university as a research sysadmin. They made a mistake, and thought that because they hired me _at_ a university I was a college hire, apparently ignoring the years of industry experience on the resume, and placed me at the lowest salary level. After 6 months, this mistake came to light, and after agreeing with my manager that the position was basically insulting given the background, I was told simply "well we can't do anything about it until the next promotion cycle, and even then we can't really 'catch you up'". This still stings in terms of time that I've now entirely lost pushing my career upwards. I decided to stay and wait it out rather than fight the decision or leave, I'm just curious about whether that would have been your decision in the situation.

(this turned out a bit long and ranty; my takeaway is just, I've run into too many situations where telling my employer I was unhappy/underpaid resulted in basically a "nothing we can do", and my switching to a higher paying base job)

<comment><USER5> Your manager was wrong to not fight to correct this for you.

Even if his/her hands were tied higher up, I would have considered leaving on this basis alone: it shows nobody at the company is willing to look after your interests.

However, you must have agreed your salary at some point?

As illustrated in the article, the start of the job is the most opportune time to negotiate. Once you have started on a low salary, employer can safely assume you don't need a high salary and has little reason to pay you more. (You confirmed that this was the correct decision for them by deciding to wait it out)

My conclusion is that you'll now likely always be behind on your salary for as long as you stay with this company.

<comment><USER4> To his credit, my manager was extremely forthright in fighting for me, but his hands were tied in being limited to the typical periods.

Yes, I did agree to the salary, but as that I was coming from an academic position (and was honestly/still am relatively new to shopping around for jobs) the salary seemed like a vast improvement and I didn't think to dig deep into actual level.

Thanks (to other child posters as well) for your thoughts.

(to answer some side posts as well; I didn't mean to come across as having excessively many years experience, but that I was just not a college hire and hope I didn't mislead. This introspection also doesn't preclude looking for other options simultaneously, and I've certainly learned my lesson about asking more of some sorts of questions during my offer.)

</comment></comment></comment></comment><comment><USER6> > The first six months of a new job is taken up primarily by learning new > systems, procedures, who to talk to etc. <…> in the beginning, you will > probably feel a lot less stressed out.

Weird, for me it's the opposite—the most stressful time is when I don't know how things work. Battling lacking or missing onboarding processes instead of working on challenges I thought I was hired to solve can be demotivating.

<comment><USER7> Yes, the first 6 months are by far the most stressful. You are far more replaceable in the first 6 months than you are after a few years.

You will earn more money though. Switching jobs is the fastest way to get a pay raise.

</comment></comment><comment><USER8> The post really makes the OP sounds like a headhunter ("don't worry, the grass is always greener somewhere else..."), in which case I would be very wary of his advice.

<comment><USER9> "I'm a career coach and can help you master every part of the job search process."

Right on the money.

</comment></comment><comment><USER10> My Dad always said to me that if you ever wake up and realise that you're not enjoying work anymore (or even worse, dreading it) then that's the day you start finding something or somewhere else.

<comment><USER11> He's right, burning out in your job and staying in that job makes people age quickly. Life is too short to be stuck in a miserable job, if you can do anything about it.

</comment></comment><comment><USER12> In my current role, I like the company, the product, and my co-workers, but I'm almost positive I could be making 20,000 more a year in the same area. My pay is (I feel) relatively low because I'm a junior software engineer, so I'm torn on whether I should just stick it out and ask for a large raise/promotion in a few months, or put my feelers out.

<comment><USER13> Is it the lack of making enough money that bothers you? If you dig deeper, what would you say? If you were to ask yourself "What it is that would keep me motivated?" \-- what answer emerges?

In his book, "Drive", author Dan Pink argues mostly successfully that it's not the salary or the stock options or other perks that keeps effective individuals motivated. He summarizes that it is the pursuit of that illusive trio -- "Mastery, Autonomy, and Purpose" that keeps people motivated in their work and in their life.

In my experience, if remaining in a job or switching jobs cannot be traced to one of these three, then the decision one takes (remain in a job or switching jobs) is not likely to turn out to be a good decision.

<comment><USER12> It's not the money that motivates me. I get the most satisfaction from writing good code and pretty/usable UIs, and that is what ultimately drives me.

My feeling is that I could get the satisfaction I describe above at many workplaces, but all things equal, I'd like the option that pays more.

</comment></comment></comment><comment><USER14> This post shows the best of hacker news: comments are often way more better than the original post!

I'd agree with one of the comment got downvoted: the post itself is really 'manipulating' and like an ad of nonsense: the only good thing is its title.

But the comments here, which are people's real experiences and lessons, are so valuable!

</comment><comment><USER15> If every "five years of experience Java Swing developer" starts looking around for a better job position, isn't that a bit similar to a sector wide union asking for a raise?

<comment><USER16> You say that as if it were a bad thing...

</comment></comment><comment><USER17> The problem with this mentality is that if you move around TOO much, then people won't want to hire you because they will think, rightfully so, that you won't stick around. I routinely reject resumes where the person has 3 or more jobs of 2 years or less on their resume.

<comment><USER18> You're missing out on a lot of talented people with that kind of prejudice against short term job terms. You have no idea the background on why people chose to move jobs so simply dismissing them on that one fact seems premature.

<comment><USER17> I disagree, and I very much disagree with the assertion that I would be missing out on "a lot" of talented people. "Talented" and "excellent contributor" are not the same thing. I would much rather put in the effort to recruit someone who is very good that will stick around for 2-3+ years and contribute versus a genius that sticks around for 12-18 months. This comes from experience.

<comment><USER19> Very short sighted mentality. It assumes that people are cogs and the measurable effect of working with someone is the amount of tangible output. Working with extremely talented developers can fundamentally change your perspective about how you should be building your system / approaching a problem that can have huge future benefits. I would rather work with a genius for 12 months than a cog for 3 years.

<comment><USER20> Are you kidding? The notion that people (even talented people) can be swapped in and out every 12 months is a far greater assumption that people are cogs.

From a co-worker perspective, yes, I'd rather work short-term with someone talented. From a managerial perspective, I'd first choose the talented employee who has demonstrated an ability to commit, then I'd choose a less talented (but with potential to improve) employee who I believe has an ability to commit, then it's roughly a tie between someone mediocre and someone talented who will likely abandon their project in the lurch. Those last two are both pretty lame options.

</comment></comment></comment></comment></comment><comment><USER21> Can't think of a better thread then this for a question that's been on my mind. I started at a new company last May. In the beginning I was on one team, I had a pretty rough time because the work was so different from my previous company (very boring). I was about ready to move on, but then my wife found she was pregnant, and so I decided to stay anyways because the fact that the work was easy meant I never worked over 40 hours.

A few months later my boss moved me to a new team because I had completed literally the next 5 months of planned work in about a month so there was little left to do, and things did a complete 180. On this new team I had a bigger role, but I was able to work on more interesting things. Basically I built a framework that around 8 developers worked full time on. It's now at the point where just one overseas guy can do all the work.

Last week we started talking about new projects I could attack at the company. Then on Friday he announced that he's leaving. When I went into his office to congratulate him on the new position, he hinted that he wanted me to join him at the new company in a position that would be kind of a bump for me.

I'm inclined to follow him because the current company is a company where engineers are lead by non-engineers, and he's one of the few gems in the place. However i'm not sure if it's a good move (especially with the baby coming in 3 months!).

I should also mention that last week he told me they were planning on letting go of all but 2 of the developers that my framework replaced :( I felt safe with him around because I built it, but i'm not sure if they hired an MBA to replace him he would do the same math.

<comment><USER22> If you have a great boss and he leaves for greener pastures, and gives you the opportunity to follow him, I think it's usually a good idea to accept. Having a boss who you like and who likes and trusts you enough to take you with him to the next company, is priceless. Staying can be risky, as the replacement boss is an unknown factor. In your case, it sounds like a no-brainer.

<comment><USER23> My company made employees sign agreements that if they did leave, they were not allowed to actively recruit people from the company. One guy did it and they sent him a cease and desist. Your manager should be careful incase he signed something similar and doesn't remember.

<comment><USER24> That's why his boss _hinted_ about job opportunity at the next company, but did not directly offered a job.

</comment></comment></comment></comment><comment><USER25> This is why all the big tech companies give substantial raises in RSUs. It's that carrot stick along with the promise of a promotion always just within reach that makes job hopping difficult. At least plan to stay with a company for 2 years. It never looks good to have resumes with pages of jobs.

</comment><comment><USER26> I have been in the same job for the past 9 years. I am happy here, but I feel a need to change. Team is great and friendly, pay is above average, but I am not concerned with it much as it's enough for me and my family to live happily.

Another reason I feel is that I have become comfortable here. Things are being gotten-done pretty easily. I don't have to exert myself too much. Team has faith in me. I love the ppl and do as much as possible for them (I am an Engineering Manager). But the urge to change is heavy and to some extent inexplicable to me. I already have 2 offers from other companies. I am wanting to talk to my boss, but fearing a bit. Sigh...

<comment><USER27> If you're happy why change? Yes change is almost always good. But it's similar to saying that you should leave your girlfriend simply because you've been with her for too many years. Think of a job as a relationship. If it's good both ways, no need to change.

<comment><USER26> That's another way of looking at it, and that's where the confusion is stemming from. I am still undecided.

<comment><USER27> If you're undecided you're probably a bit unhappy in your current position. Therefore change! :)

<comment><USER26> Nice revelation :) Thanks. I resigned.

<comment><USER27> Wow really? Well I hope you thought it through and didn't just rely on a "stranger's comment" for this decision. In any case, I wish you the best and trust you'll have the skills to find something you really love - <USER2> should aspire to do what they love.

<comment><USER26> Oh no, I thought it through. There's some risk involved and some leap of faith, but I will own anything that happens. Good or bad, it's going to be my shit.

</comment></comment></comment></comment></comment></comment></comment><comment><USER28> There is one point that was missed in this post: promotion. Are you more likely to land a promotion in your current job, or are you more likely to find a another job that is also a promotion to a position that you have no experience in?

<comment><USER29> My guess would be that it doesn't matter. Plenty of managers and senior people are hired from outside of companies these days. And in any case, if you don't make more money, it doesn't even really matter: [http://www.forbes.com/sites/cameronkeng/2014/06/22/employees...](http://www.forbes.com/sites/cameronkeng/2014/06/22/employees- that-stay-in-companies-longer-than-2-years-get-paid-50-less/)

<comment><USER30> The set of candidates that leave to get another job is different from those that stay. They are likely more skilled/motivated, and that's a large part of the job-switching premium. If everyone were to try to switch jobs many people would not be as effective.

</comment></comment></comment><comment><USER31> I thought this was gonna be a riff on the Monty Hall problem. [http://en.wikipedia.org/wiki/Monty_Hall_problem](http://en.wikipedia.org/wiki/Monty_Hall_problem)

<comment><USER32> You wanted a more fulfilling job, but what you got was a goat. Bummer!

<comment><USER31> #winner [http://xkcd.com/1282/](http://xkcd.com/1282/)

</comment></comment></comment><comment><USER33> This should be titled, "Why switching jobs is usually a good idea for overworked, underpaid people who have been working for at least a few years at their current job."

But that wouldn't fit on HN.

</comment><comment><USER34> This is bad advice. The right answer is often to change jobs, but the explanation given here is pretty terrible.

If you have a pattern of working significantly less in your first 6 months on a job than later on, then you're making a mistake. That's the time to establish a reputation, figure out what is worth working on, and get a mental map of the organization so that you can actually get things done. It's the hardest time in a job, if you do it right.

If you let the "honeymoon period" blind you and slack in your first 6 months, you start getting grunt work thrown at you and that's how you end up overwhelmed and struggling at the 3-year mark. If you do the first 6 months _right_ and gain the credibility, alliances, and reputation that'll put you on a good vector, you (a) have a much higher chance of getting on a fast-track, which means better work rather than more of the same, and (b) can get away with slacking and recharging (as you seek external promotion) if you are passed-over for some reason.

Also, for higher-level positions, switching jobs often means changing locations and doing that every year is pretty miserable... especially if you have kids. At some point, you're specialized enough that unless you live in New York or SF-- which are pretty much out once you have kids, unless you're in a hedge fund or a VC firm-- you're going to have to stick with a job for a few years just because there aren't many jobs in your specialty and location.

It's worth changing jobs for a genuine promotion, but a high frequency of lateral movement looks really bad. Given also that it can be hard to tell if a new job is a genuine promotion, it's better to stay where you are if you have something good and you're continuing to advance.

</comment><comment><USER35> If somebody ever asks me "what does mansplaining mean?" I'm just going to say, "Well you see, it's quite easy. Just read this blog post"

In all seriousness, there was no need for the point in this post to be explained through such exemplary mansplaining.

<comment><USER36> Oh come on. I'm not even sure "mansplaining" is a thing, but even it is, in no way does "mansplaining" == "socratic method".

Also, did you even know Alex Potato is a man†? I know two Alex's, one male, and one female. I've never spoken to a potato.

Or is it automatically "mansplaining" because the topic is about a specific woman being underpaid/overworked, and any helpful suggestions to that specific women (that reframe the situation and empower her to improve things herself) are automatically sexist? I mean, I'm happy to condemn employment conditions that systemically underpay women, but that won't actually help Sam, whereas this advice might.

In conclusion, whilst obviously not everything men write or say that is intended to be supportive of feminism is actually helpful and supportive, it really worries me that if men who are trying to help are shot down for missing the mark, then fewer will try in future, and the only ones left commenting will be the trolls who are looking for such a reaction.

† I mean, we could assume they were because they appear to work in software or the like, and play paintball, but we shouldn't do that because it would be sexist, right‡?

‡ Did I just "mansplain" there too? Ooops.

<comment><USER37> He's a man: [http://alexpotato.com/career/about.pl](http://alexpotato.com/career/about.pl)

<comment><USER38> I don't see anything on that page that indicates Alex has stated a gender identity, so that's an awfully presumptuous thing to say. Maybe you should stop mansplaining?</comment></comment></comment></comment>


Startup Metrics: Anecdotes vs. Analytics - <AUTHOR> http://blog.wepay.com/2010/06/startup-metrics-anecdotes-vs-analytics/

<comment><USER2> I am a big fan of data, but there is no substitute for talking with your customers. I had to choose just one, I'd take customer knowledge over data because it will tell you what's next, and why…data will just tell you what's better without explaining why, so it won't show a path to the next breakthrough, or lead you to product market fit.

So…start working those support lines!

</comment><comment><USER3> This article talks about anecdotes vs analytics, but in the end, it advocates anecdotes _and_ analytics. It feels like anecdotes give you a gut feeling of where the opportunities are and that analytics give the gut feeling some substantive backing. Companies need to understand and use both types.

</comment><comment><USER4> I am a firm believer of What Get Measured, Get Done. The same is for website, when you measure and understand what works and what dun. You are definitely able to come up with more ways to do things better.

</comment><comment><USER5> I've been looking at this qualitative vs quantitative tradeoff for a recent analytics project.

We've adopted the general framework in Getting to Plan B - using 'Leap of Faith' questions to hone various hypotheses using qualitative data. Eventually when you characterize enough assumptions from qualitative inputs, you are left with a set of concrete measures you can use quantitative analytics to measure.

Would be curious what approaches others are using to balance this relationship.</comment>


Manifest: An instant grid-based pinboard for note taking in the browser - <AUTHOR> https://www.manifest.app/ <comment><USER2> Hi all, I'm Jonathon, the maintainer of the Manifest repo. Thanks to whoever linked it here on Hacker News, I was wondering why it was get more attention that I expected...Appreciate the support and feedback thus far!

</comment><comment><USER3> One minor suggestion: _just in case you haven 't considered it_, is to `favor undo over confirmation`. So when the user deletes a note, instead of asking if they are sure with an alert dialog, just delete the note but also have an undo buffer (or trash can) so they can restore it easily.

<comment><USER2> Someone suggested this recently. My personal thought on it is not to try to be too smart and over-engineer things. Relying on a browser's native dialog system simply just works and doesn't really need to be maintained. Doesn't mean it will always remain this way, that's just my current attitude regarding it.

<comment><USER4> Delete with confirm gives you a far worse UX than undo does though.

<comment><USER5> Also having the confirm there, kind of disrupts the flow. At least for me, your mileage may vary. Cool website though.

</comment></comment></comment></comment><comment><USER6> This is a great little tool!

The subtle grid reminds me, oddly perhaps, of OneNote. It was my favorite note taking app in college, and there was an implicit/invisible grid at the base of every page with similar click and drag boxes for text and other media. Ever since 2018 when Microsoft began removing features (i.e. local notebook support) and stopped updating the full desktop version, I've been searching for a viable replacement and debating just building one myself. The continued free availability of the depreciated desktop app has stopped me from taking the plunge, but OneNote is no longer something I expect to count on forever.

There's definitely a crowd of people who like visual note taking apps with just a little bit of organizing framework. And there's definitely demand for that kind of application.

<comment><USER7> Hi - I'm on the OneNote team. We're bringing the full desktop version back into active support and development (announced at Ignite in Nov). We made this decision after hearing strong and consistent feedback from customers like you. Hope to keep you as a fan!

</comment></comment><comment><USER8> I did something similar a few years ago, I used markdown inside my notes so I could put links / images and format the text easily.

The notes were also stored on a mongodb, so they can be accessible from anywhere.

The only thing I was a bit stuck on, is when you define a grid size, it is quite hard to make it work anywhere, especially if you have big differences in resolution between your devices.

</comment><comment><USER9> Nice work! Safari is kind of limited with it, it takes a few clicks to create a note, sometimes(?)

Also a no-select might be good for certain part of the page, or for the ones not being edited presently, as text selection seem to grab most of the page text at any click and drag.

<comment><USER2> As the original developer on this I was building everything on Windows so didn't have Safari immediately available to test on. I'll take a look though and make sure improvements get made if needed!

</comment></comment><comment><USER10> Feels great, but is there an option to switch out of the dark theme or is it the only one

<comment><USER11> Yes, press Alt+T :)

[https://github.com/jonathontoon/manifest/blob/c51828086d972f...](https://github.com/jonathontoon/manifest/blob/c51828086d972f4aa1332e7a545b334f11dc2345/src/js/index.js#L441)

<comment><USER9> If you're going to have obscure shortcuts, may I suggest that you document them part of a note somewhere maybe accessible via a '(?)' icon at the bottom of the page?

Also, your dark mode is way too dark and contrasted. Consider something like this:

[https://imgur.com/a/8FLdGb5](https://imgur.com/a/8FLdGb5)

HTH

<comment><USER2> Dark mode was just added over the weekend, so there's probably a better way to address how to find out about features, shortcuts etc as they get built.

As for the contrast I think it's just a case of visual style preference. I think the current contrast works for now but it's only been out for a little while, so happy to see more responses down the road.

</comment></comment></comment></comment><comment><USER12> We changed the URL from [https://github.com/jonathontoon/manifest](https://github.com/jonathontoon/manifest) to the project page.</comment>


How To Hire A Data Scientist - <AUTHOR> http://blog.bright.com/2012/11/13/how-to-hire-a-data-scientist/ <comment><USER2> For recruiters who aren't sure where to find these mythical post-docs who do science (with "Data"):

1\. Go to a scientific journal that involves serious computational work.

2\. Look at the last author on an article. This is usually the lab boss.

3\. Google "$LASTNAME lab webpage".

4\. Look for graduate student and post-doc profiles.

5\. Email them and offer more than the 38k they make for working 60+ hrs/wk.

6\. Now you have a "Data Scientist"

Alternatively, just Google "Argonne National Labs".

There is no shortage of Data Scientists with years of experience (every scientist I've talked to has guffawed heartily and asked "what other kind of scientist is there?"). Yes, I get that the term is just a very unfortunate choice of words (on the order of General Linear Model vs General _ized_ Linear Model), but the point is that no one knows to brand themselves this way. Recruiters are probably just frustrated by the lack of linked-in-ness, and the fact that most people competent at "Data Science" don't know what that means.

Here are things virtually every PhD in a computational discipline will have done:

1\. Written code. It might be just Matlab, Python, R, etc.

2\. Written up and communicated the results with compelling visualizations, both orally and written

3\. Published a paper at some point demonstrating they can do this.

4\. Dealt with failure and hunches that didn't work out after weeks of work.

On the hiring part, after you've found these mythical creatures:

As someone who does science (with Data, even!), here is the best question I can think of to ascertain my competence: "Diagram an approach to answer a particular question, with emphasis on ruling out competing explanations and demonstrating whether a result is true. Sketch some hypothetical visualizations you'd use that show what your result is, how large the effect is, and how sure you are that you're right."

I should be able to do that. On the flip side, if an employer can reason with a post-doc about that process in an intelligent fashion, they would be excited about leaving academia. You might need to teach them to use a cluster, EC2, whatever, but you will not have to teach them to ask and answer questions.

<comment><USER3> Erm... isn't a 'Data Scientist' just a shitty name for a Statistician? Maybe I am biased, but it seems like somebody just came up with an ass-hat new name for my profession.

<comment><USER4> There are enough people who come from top notch statistical backgrounds who don't necessarily have a firm grasp in whatever fancy machine learning heuristic is in flavor today (which is not a deal killer), or don't know how to code (which probably is depending on the company)

</comment></comment></comment><comment><USER5> I think I'm in the minority, but I really hate the term, "data scientist." It seems usually to mean, "senior statistician, but with training and credentials expected of an RA" (to clarify, that isn't meant as a comment on the original article). I would be especially skeptical about hiring someone who _self- identifies_ as a "data scientist," people are trained as Statisticians, Biostatiticians, computer scientists, various subspecialties that end in "-metrician" (e.g. Econometrician, Psychometrician, Cliometrician), etc; no one is trained as a "data scientist." Unless you're hiring someone really junior, you want the "data scientist" to have a specialty -- anyone good will have one.

But the best way to find a good "data scientist" is probably the best way to find a good programmer -- be one yourself; tap your professional network; and hire people as consultants/freelancers on non-critical projects before making a real commitment. Identifying someone with a deep skill that one doesn't possess oneself is pretty much impossible. And on the flip side, I have trouble imagining that someone who really knows what he or she is doing would want to work for some unknown.

If you want someone to scrape and clean data with Perl and generate some scatter plots and histograms, look for undergrads with good grades who worked as Research Assistants, or recent grads working as RAs at consulting firms, research centers, governmental agencies, or think tanks. They'll do great (by and large), they've have had some informal training from a more senior researcher to help put everything in context, and faculty often steer their best students into those sorts of jobs, so there's a pretty strong quality screen. I'm sure there are other places to find people too.

<comment><USER6> >but I really hate the term, "data scientist."

I think most people do, but I've never heard a good term for the job. It's like "we want someone who can take large amounts of data and do something awesome with it". What do you call that?

>Unless you're hiring someone really junior, you want the "data scientist" to have a specialty -- anyone good will have one.

Not sure I agree with this, I want people who are well-rounded. I think it's great to find someone who specialized at something, but I'd want that person to be able to grow the rest of his abilities up to par. Example: let's say you specialized in machine learning. If you don't understand building scalable systems, you can't take a holistic view of a project; how will you know if your algorithms can scale to a production environment? Or, if you can't program well, you can't write code to actually get your algorithms into place. Or if you can't understand the business side of things, you won't be able to build trust with the rest of the company, and hence you won't be able to contribute.

<comment><USER7> >"we want someone who can take large amounts of data and do something awesome with it". What do you call that?

Analyst

<comment><USER6> That's close to what I used to be called. It's a very overused and vague term, I'd argue far worse than the already bad 'data scientist'. Search job postings for 'analyst' and see the wide variety of jobs that turn up.

</comment></comment></comment></comment><comment><USER8> "1. Tell me about some peer reviewed papers that you published as first author?"

It could be that the author's first criterion is an important predictor. But it seems to me that unless somebody is actually _in_ academia, publishing papers is more akin to a hobby than a professional qualification, especially given the inherent bias against unaffiliated authors.

Edit: On re-reading, the author (hardtke) writes the above when talking _specifically about weeding through post-doc applicants_. So my quote is out of context and my criticism unfair.

<comment><USER9> Yeah, I fail that criterion hard (I've had five rejections, does that count?). Nonetheless, while the author is probably showing his (or her) biases, there's definitely a nugget of truth there. For a more coding focused candidate, an equivalent question would be questions about software you have designed, built and promoted (as that's essentially what the question is asking).

<comment><USER10> Basically, I want to see if people can finish stuff. For software engineers, tasks tend to be easily defined and of shorter duration. At least at Bright, we have Data Scientists working on multiple simultaneous projects that take a few days to several weeks to complete. Knowing when you are done with a complicated, possibly open-ended, project is very difficult.

<comment><USER11> _For software engineers, tasks tend to be easily defined and of shorter duration._

For crappy ones doing commodity work, this is true. For good software engineers, not so much.

I'm a data scientist by pedigree (before it was called that) who's spent the past few years in "regular old" software engineering (and probably heading back in the DS direction). Trust me that software engineering done right is as subtle and talent-intensive as DS.

The problem is that SWE's are terrible at marketing themselves as a group and generally get too little respect and autonomy to have architectural successes.

</comment></comment></comment></comment><comment><USER12> Data Scientist simply means a developer who also knows statistics. Here's how to sniff them out:

1\. What is you favorite programming language and why?

2\. Which is your least favorite programming language and why?

3\. Explain how Bayesian spam filtering works.

4\. How do you determine if a given data sample is statistically significant?

5\. Suppose we ran a brand advertising campaign on radio and on television. Neither ad campaign uses special tracking codes or custom landing pages, both ads simply mention the web site address. What tools and methodology would you use to measure the response rate from radio ads vs. tv ads, and predict the total response that will generated from each ad?

<comment><USER13> >4\. How do you determine if a given data sample is statistically significant?

That's a weird question - what am I to prove? Are there differences in the dataset? How does the data look like? Is there a second dataset against which to compare? What kind of data is it, ordinal or nominal? Is the dataset normally distributed?

I can't just say "that data sample is statistically significant" without a background against which to compare against!

>3\. Explain how Bayesian spam filtering works. I would say the majority of non-web developer scientists don't know how this works, why should we?

<comment><USER14> If you've had any exposure to Bayesian statistics, it's very easy to explain the approach. You're basically tokenizing the email by word, and then predicting the probability that P(email is spam|token).

I would expect any data science candidate to have (at least) a basic understanding of Bayesian statistics.

<comment><USER13> It's just that an e-mail filter is the last thing most scientists working in biology, chemistry, physics, medicine are actually working with, so it's (in my opinion) a rather unfitting example, you might be selecting against these people who fit the bill of "data scientist" probably better than the average "I once did something in Excel"-web guy.

</comment></comment></comment></comment><comment><AUTHOR> "If you want to find a Data Scientist, find yourself a disgruntled postdoc toiling away on brilliant scientific research, but failing to land a professorship because … all the professor jobs are taken!"

That quote is more than just humorous, it points out one compelling answer to the question in the title of the post -- Perhaps all scientists are data scientists, you simply have to lure them into a new domain of study.

<comment><USER9> Possibly, i have certainly made that point before (and it remains my most upvoted post, so certainly other phd students and postdocs agree with me).

I do think that people need to stop expecting to get a physicist, statistician, economist or applied mathematician (every graduate analysis job I've seen had that wonderful qualification) as most of those people already have really well paying jobs in finance (or satisfying careers in academia), and open their eyes to the fact that for many data science roles, social scientists are probably a better fit.

If you're dealing with numbers generated by people's interaction with a website, a handy background to have is in some form of quantitative social science. (I am of course horribly biased, by being a quantitatively trained social scientist).

In any case, I expect data science to go through a dot.com like boom and then a horrible crash, so it may be a good idea to get the skills (and possibly qualifications) now, while the sun still shines and people are still hypnotised by the promise of big data, rather than the tedium and slog of extracting value from it.

<comment><USER11> _most of those people already have really well paying jobs in finance (or satisfying careers in academia)_

I've always taken the data scientist distinction to be the startup world's answer to the "quant" designation. Quant jobs are a lot better than "just a" programmer jobs, even at hot startups, so I see the "data scientist" title as an answer to that. It's a startup quant.

<comment><USER6> Yeah, I've actually heard people use the terms interchangeably. Although 'quant' is almost exclusively used in the finance world. I interviewed for a couple of those jobs and got caught up on stuff I've never heard of before, like pricing bonds and combining interest rates.

</comment></comment></comment></comment><comment><USER6> Your article seems to be written from a phd-centric, low-industry-experience perspective. I'd reconsider this bias if it's real, finding good data scientists is hard and you may be eliminating good potential candidates.

I've been what you could call a data scientist for over five years now, and worked with dozens of people you could also call data scientists with different degrees and varying experience. From my sample, I don't think PhDs add much, if any, value over masters degrees after a year or two of experience (I'm biased here, I don't have a PhD). I think industry experience can add a tremendous amount of value you can't get from a degree, but it comes at a cost premium. Not related to your article: I've also found the best people have physics or computer science + applied math backgrounds.

</comment><comment><USER15> One of the best data scientists I know doesn't have anything near a PhD. Just a plain old Undergrad.

<http://www.youtube.com/watch?v=-3dw09N5_Aw>

I think there are too many ways to miss the mark when it comes to hiring data scientists. Looking for PhDs only is just as dumb as looking for people with 10 years of experience with Hadoop. There are some important things that they need to know, sure, but where they come from is next to meaningless.

</comment><comment><USER16> One problem with hiring academics is that they can be far too focused on their subdiscipline as this is necessary to break new ground. An astro physicist might be amazing at identifying peaks in tremendous amounts of data but have no idea how to do basic analysis of a graph.

When we interview someone I like to start talking/asking about matrix decomposition (eigenvectors, svd ect) and see how excited they get.

I consider knowing about things like MDS and pagerank a bare minimum and if someone can bring up a more recent or esoteric application (locally linear embedding, graph partitioning) they stand out.

Asking about the nuances of estimating probability densities from data (bin/histogram vs kde etc) is another good one and something that stops a lot of the cooler theoretical statistics and information theory from getting used (or used well) in the real world.

Both of these questions get more at "do you understand the basic building blocks that come up over and over again" more then "is your research groundbreaking and new." Asking about techniques to do the above at scale also ups the difficulty of the interview.

</comment><comment><USER17> "Data Scientist" is such a weird title for somebody who is basically a statistician with good IT skills. I think in some market verticals they're simply called "Quantitative Analysts" and make _very_ good money.

In a previous job I did lots of analysis on very large data (at the time) data volumes, millions of structured or unstructured records, homo and heterogeneous datasets. Lots of aggregation, sifting, sorting, simplifying, deduping, summarizing, etc. All in support of similar kinds of things that "Data Scientist" positions seem to be intended to support. But the output was not a statistical model, or a machine learning exercise or some other similar. It was the distillation of gigabytes of data into a handful of slides and a report. Usually with a virtuous cycle of feedback directly into software development to improve and expand the next go-around.

But almost no statistics. Very very little, and what I did was very basic stuff.

What is that kind of job called? In my day we called it a "Data Analyst" but I don't see that around much.

I'm going to make a prediction, "Data Scientist" as "Senior Statistician" is going to be short-lived. I don't think they're going to provide the value companies think they will in most cases. "Data Analyst" is much more general purpose and useful cross-domains, except most Data Analyst don't have proper statistical training.

A Data Analyst with statistical training would be a much more useful tool to an organization seeking to make sense out of large volumes of data than a Senior Statistician as they'll have a much wider variety of tools at their disposal than just looking at the world through the statistics lens.

Bonus, jobs advertising "Data Analyst" can _demand_ things like machine learning AND entity extraction AND automatic summarization AND data sanitation AND automatic correlation analysis AND automatic colocation analysis etc.

Most of the jobs I've seen looking for Data Scientists are for companies that are probably going to try and end up using them as high-priced Data Analysts, except the job reqs are all wrong and the candidates that get hired are _way_ over qualified.

But this role is still evolving I suppose, IBM [1] views it as an evolution from the business/data analyst. So they definitely seem to be on the side of _not so much statistics_ and more _analysis_.

1 - [http://www-01.ibm.com/software/data/infosphere/data- scientis...](http://www-01.ibm.com/software/data/infosphere/data-scientist/)

<comment><USER3> I'm going to throw it out there that a Statistician with poor IT skills nowadays is like a carpenter with poor measuring, cutting and hammering skills.

</comment></comment><comment><USER18> I'm actually on the other side: currently writing my bachelor's thesis in statistical physics, having a lot to do with probability stuff, statistics and data. I'd like to take a year off and work in a company to get some real life experience before I start my Master's degree, because I don't want to stay in academia after that. But I have no idea where I can find companies which could need my abilities and where I could work for 0.5-1 year. Any ideas? I live in Germany.

<comment><AUTHOR> Most shops are willing to let you work remotely. Anyone that's down with stat. mech. has a solid base for quantitatively attacking most problems. Luckily the 'domain' knowledge required is general human consumer behavior, which you'll know quite a bit about already. And a surprising amount of that can be reasonably modeled by a microcanonical ensemble.

</comment></comment><comment><USER19> Are people hiring freelance/part time for data science work? I'm a physics professor at a small liberal arts college, and would like to move my career in the direction of data science. Picking up a client or two for smaller/short term projects would really help, I think.

As an aside, it isn't just the postdocs who are disgruntled. I was awarded tenure last year, and while there are aspects of the job I love, there are others that push me toward making a change.

</comment><comment><USER20> JavaScript Ninja, Scrum master and now data scientists?

By their definition i'm a scientist because I've built a few products! I certainly don't see myself as one.</comment>


Digital natives can handle the truth. Trouble is, they can't find it - <AUTHOR> https://www.theguardian.com/commentisfree/2016/dec/11/digital-natives-cant-find-the-truth-on-the-internet <comment><USER2> Could this be partly atteibuted to the shuttering and lack of resiucres directed to libraries? This is where I learned how to research and discern legitimate sources.</comment>


The Good Times for Airlines Are Over - <AUTHOR> https://www.bloomberg.com/opinion/articles/2020-03-17/the-good-times-for-airlines-are-over <comment><USER2> The core problem is that the airlines fixated on returns and their stock price with little focus on disaster planning. They cut costs by slashing their frequent flyer programs, buying more fuel efficient planes, and for American, they cut some really badly performing routes.

For any of the majors to make a real comeback after Covid, we'll need to see how much of their bread and butter (corporate contracts) comes back and how quickly. If there is a severe lag in that money flowing into the airlines, consolidation isn't far away.</comment>


DuckDuckGo is blowing up - <AUTHOR> http://duckduckgo.com/traffic.html <comment><USER3> Thank you all! (I'm the founder.)

In response to a lot of the comments here, please know that two major things we're working on are better programming queries (<https://duckduckgo.com/tech.html> \-- one of my new favorites <https://duckduckgo.com/?q=alternative+to+picasa>) and speed.

For speed, just this week we upgraded our whole caching system, which should significantly speed up a lot of queries. I'd be interested to know if anyone has noticed any difference over the past day or so. This change should equalize a lot of the location differences, which is the main issue. In some parts of the world we were way slower.

Also, for anyone wanting to get involved we've been open sourcing more and more (<https://github.com/duckduckgo>). We're working on better entry points, but one could start here now: <https://github.com/duckduckgo/duckduckgo/wiki>. For programming documentation in particular, this is the repo: <https://github.com/duckduckgo/zeroclickinfo-fathead>. That will answer queries like <https://duckduckgo.com/?q=perl+split>

On the back-end we could also use some sysadmin help :). Here's our hiring info: <http://help.duckduckgo.com/customer/portal/articles/216387>

Of course, we're also always looking for feedback, the more detailed/specific the better: <https://duckduckgo.com/feedback.html>

<comment><USER4> Way cool. Going to try using this as my default search engine for a while.

I really really like the "alternative to" feature. What could I do to help you get better at alternatives which are not as popular? (eg, "alternative to cheetahmail" or "alternative to grubhub" or "alternative to xkcd"?)

Also, what kind of feedback would you need to help improve the tech-related searches? Like "ebcdic table" or "1010 in hex"?

<comment><USER5> The alternatives are pulled from <http://alternativeto.net> so adding to their site will add to the zero-click info.

You can contribute to tech-related searches by contributing to the open source zero-click info (see this tutorial [https://github.com/duckduckgo/duckduckgo/wiki/Open-Source- Pl...](https://github.com/duckduckgo/duckduckgo/wiki/Open-Source-Plugin- Tutorial)).

You can also send in ideas here: <https://duckduckgo.uservoice.com/forums/5168-plugins>.

<comment><USER6> Obligatory meta-comment:

<http://alternativeto.net/software/alternativeto/>

</comment></comment></comment></comment><comment><USER7> Wow, thats a great curve. Just wanted to throw in my personal experience with DuckDuckGo, and it doesn't necessarily reflect on DuckDuckGo specifically, however:

After seeing them pop up here once or twice quite recently I thought I'd give it a go, I opened up preferences in Chrome and switched my default search over to DDG. It felt good, made a nice change but was certainly a bit "odd". I found what I wanted, most of the time albeit it with a slight drop in quality vs Google.

But my biggest gripe with them was confidence. I'd just started out at a new job as a programmer and as you can imagine, I was searching for a good few things, usually things I was struggling with and I just didn't feel sure that I was getting the best results I possibly could be. You might think thats totally crazy, but when your battling an issue as many of you know, you want more than anything to work out whats going on. I didn't have any margin for error, I wanted the best results right there that second.

Specifics aside, knowing that Google is far superior in their results makes it real difficult to use another search engine really, extra features (!so etc) or no extra features. I personally search because I need to "find" something and I usually don't know where that something is, opening up Stack Overflow isn't that much of a chore for me. Its the _other_ bit I need help with.

Anyway, I commend their mission and hope they succeed in taking a fair slice of the market. I think he's a great entrepreneur and I wish him the best of luck. I can't imagine what it must be like to be head on with... Google.

<comment><USER8> I find that I am slowly losing confidence in Google. What I often do is input searches in the form: "general term", "specific term", then click on a link and immediately do a find for the specific term I searched for. In the last few months, the specific term I searched for often does not exist on the linked site at all, which I find quite annoying.

I tried bing a couple of times, but the results are even worse.

Maybe Google verbatim mode will fix the problem? I've set up a quick search recently, but have yet to start using it.

<comment><USER9> Would love to see some example queries to debug if you have them handy. (You can check your search history <https://www.google.com/history/> to jog your memory if you have it turned on.)

<comment><USER8> Here's one: python list popleft

Third result is: <http://docs.python.org/library/stdtypes.html>

6th result is: [http://stackoverflow.com/questions/4426663/how-do-i- remove-t...](http://stackoverflow.com/questions/4426663/how-do-i-remove-the- first-item-from-a-python-list)

Neither one of those contains "popleft".

<comment><USER10> Try putting quotes around popleft in your query. That works in this case to get rid of the results that don't specifically include the word.

<comment><USER11> In this case it does, but very often doesn't. It seems to me that this happens mostly with rapidly changing websites, as if the cache showed a somewhat later state than when it was indexed.

And this has been an annoyance since years.

</comment></comment></comment></comment></comment></comment><comment><USER12> DuckDuckGo is definitely pulling users from Google. By mimicking Facebook, Google has lost its mojo. Stepped right into the "social" honeypot. No news. The more interesting thing is the fantastic feeling more and more people have: that Facebook and Google will take eachother down the spiral.

As for the Google part, right now it's probably just an alarmist prediction, but it's going to be fun to watch un/fold. Especially if DuckDuckGo and other engines continue to get better while practicing their principles instead of a crude mixture of addictive search quality, addictedness to bucks and PR along the lines of "Don't be evil".

<comment><USER13> [1998]

Google is definitely pulling users from Yahoo. By mimicking Alta Vista, Yahoo has lost its mojo. Stepped right into the "portal" honeypot. No news. The more interesting thing is the fantastic feeling more and more people have: that Alta Vista and Yahoo will take eachother down the spiral.

As for the Yahoo part, right now it's probably just an alarmist prediction, but it's going to be fun to watch un/fold. Especially if Google and other engines continue to get better while practicing their principles instead of a crude mixture of addictive search quality, addictedness to bucks and PR along the lines of "Yahhooooo!".

<comment><USER14> One difference in this story is....

Google was vertically integrated from the start: They sent out their own bots to scan the internet, they had their own reverse index, and created their own algorithm.

I'm not sure about DDGs backend---but if people use the g! tag, its almost as if DDG is simple chrome for Google.

Maybe that'll be the future: Google will become the maker of engines, and let everyone else make the car.

<comment><USER15> As far as I know, DDG uses both data from other search engines (Google, Bing) as well as its own crawling bots. If they only used their own crawlers, with the current size of the internet it would take a long time for them to get a good amount of data.

</comment></comment></comment></comment><comment><USER16> !g is the google killer. I switched to DDG, but kept drifting back to see if Google was doing better. I find myself using !g less and less, as I get used to the result format.

At the moment, DDG reminds me of Firefox (or was it WaterSquirrel then?) back- in-the-day. Its something that "feels" better than what is the standard - <USER2> it's something that I go on and on about to other nerds, when I'm drunk ;)

<comment><USER17> I hate this (!g) though.

If I want Google results, it's a heck of a lot faster, easier, and natural to just Google it instead of adding some kind of search operator BS to my query. I used DDG for a while (a long time ago, admittedly) and basically had to !g every query I ran in order to get acceptable results. In the end, I gave up because it was just adding typing and thinking time to search.

I suppose one could argue that the interface lacks some of the annoyances of Google, but I feel like quality and speed are the most desirable traits of a search engine (in that order). If we're just using somebody else's results anyway then why should we accept a slowdown?

<comment><USER18> I like the !bang operators because I can get results from Google for specialty searches like maps (!m) and images (!i) right from the address bar in Chrome, but still keep DDG as my primary search engine. Google's results _are_ slightly better, but it hasn't been a big deal for me. I still end up finding what I am looking for.

</comment></comment></comment><comment><USER19> People care about privacy and not being tracked. I predict Google's arms race with Facebook to profit more from user's data will augment this growth. And/or dampen that arms race.

That's why I search with DuckDuckGo by default (previously Scroogle).

<comment><USER20> With everyone's concern with privacy, I'm surprised Ask Eraser isn't mentioned more frequently (<http://sp.ask.com/en/docs/about/askeraser.shtml>).

<comment><USER21> I used to use Ask until they made an announcement that they were leaving the search business and staying in the toolbar sector. I'm surprised they're still up.

</comment></comment></comment><comment><USER22> I certainly felt some difficulty first in setting Duck Duck Go as my default search engine. Google trains you repeatedly, implicitly, over the years on how to optimise your query for Google. I often still find myself adding "!g" to my DDG query to perform the search on Google, but at least Google isn't my first point of contact any more.

</comment><comment><USER23> I've gotta say, Gabriel's tenacity is both admirable and inspirational. I remember long ago in its early days I tried it and shrugged it off as just another search engine, destined for failure after a few months of obscurity and being largely unknown.

It's now my default browser, and has been as of about 2 weeks ago. Gabriel has been very keen on picking up where other search site are slacking.

</comment><comment><USER24> DDG: [http://duckduckgo.com/?q=the+movie+where+evil+is+in+a+toaste...](http://duckduckgo.com/?q=the+movie+where+evil+is+in+a+toaster+oven)

Time Bandits is #8

vs [https://www.google.com/#hl=en&output=search&q=the+mo...](https://www.google.com/#hl=en&output=search&q=the+movie+where+evil+is+in+a+toaster+oven)

First hit.

I realize it is in fashion to make bold claims that somehow Google destroyed its search in a monomaniacal pursuit of social, but a hyperbolic if not patently untrue story. Yes, the user interface has changed and gotten fancier and more cluttered with the introduction of toolbar and sidebar (that's been happening over years, prior to G+), but the actual organic results have not changed much at all except to improve on average, even if you take into account Search Plus Your World (which you can disable)

Shocking, but most people at Google do not work on G+, it is not the sole focus of the company, and the narrative that Google is simply dropping everything else to mindlessly chase Facebook is blogosphere fiction. There are over 30,000 employees and I would be surprised if even 2% of them worked on G+. Teams at Google are typically small and at any given time, people are working on a large number of projects. It's simply not Google's culture to mindlessly focus on a single unproven product to the detriment of others. If anything, Google is frequently criticized for doing too many new things at once, and that's what you get when you have a company run with engineering culture.

Google makes most of their money from Search and Ads. You can bet that they watch data from search quality and marketshare like a hawk, and if social was causing Google search to suck more and lose customers, you can bet they'd turn it off in a heartbeat. It's been said by some analysts that every 1% loss in search marketshare is $1 billion in revenue. Facebook isn't making that much money, and so from a cost-benefit analysis, sacrificing billions in search share revenue to try and gain social-search revenue doesn't sound rational.

It would thus be reasonable to assume, that any changes Google actually makes to core search are a) conservative b) subjected to a battery of scientific tests and c) as low risk as they can make them to their core business.

But the way you read things in the Blogosphere, Search has been radically altered. The evidence just isn't there.

<comment><USER25> It's very common these days to take many amazing products for granted. Google search is an amazing tool that suffers a lot from very vocal complaints and not enough praise. The only areas it is lacking in are those that are heavily targeted by spammers.

</comment></comment><comment><USER26> One of National Public Radio's (NPR) shows -- "On the Media", perhaps -- had a several minute segment on Google's privacy changes and someone who decided to "divorce" themselves from Google. They mentioned DDG as the search engine they'd switched to and commented favorably on the experience.

That's at least the second time in recent weeks that I've heard DDG mentioned on national or state public radio.

It might sound... "incidental", but those kinds of exposures really get "the masses" to go and have a look.

EDIT: Here's a reference to the "divorce" segment:

<http://www.onthemedia.org/2012/mar/23/divorcing-google/>

</comment><comment><USER27> I don't much like DuckDuckGo's results. For example a search for "Minecraft" via DuckDuckGo produces:

#1 minecraft.net (official site) #2 facebook.com/minecraft #3 joystiq.com/game/minecraft (news blog) #4 minecraftportal.com (minecraft blog) #5 minecraft.en.softonic.com (a site distributing a copy of the phased out minecraft trial version) #6 minecraftwiki.net (official Minecraft wiki) #7 kotaku.com/minecraft/ (blog)

The #1 result makes sense and is great, but why are the rest of the top results all blogs (besides the wiki)? It seems to heavily favour blogs. The official Minecraft forum isn't even in the top 20...

<comment><USER28> I'm not exactly sure what your talking about. The results for Google's minecraft search are pretty crummy too.

#1 minecraft.net (can't tell from the search results if it's even the games real website, since there are ~.com, ~.net and ~.org sites all run by others) #2 minecraftforfree.com (a site distributing the full version for free?!) #3 minecraftwiki #4 wikipedia's article #5 pocket edition on the android app store, not even the actual website and no links to the iOS version? #6-10 youtube videos of people playing or reviewing minecraft #11 3 news links about minecraft #12-13 both link to googlemapsmania articles on minecraft maps.

Oh and a nice link at the bottom about DMCA requested removal of a link (sent by Mojang at least). I imagine due to copyright/IP violations like link #2.

YMMV though due to Google's search personalisation...

The minecraft forum isn't listed until the second link on page 2. I'm not sure if you missed it but the forum is link #10 on the DDG search...

<comment><USER27> It was definitely ~#20 when I checked, it was below the page (I had to scroll then ctrl + f to make sure I found it) but now I can't even find it any more, the only thing I can find is "minecraftnews.net" which is around #30 and is a rip off of our site.

Here is a screenshot of the results I see: <http://i.imgur.com/qIc1J.png>

I've visited google from a VM that has never used google before and I see:

#1 minecraft.net #2 minecraftwiki.net #3 wikipedia.org/wiki/minecraft #4 Minecraft fan made trailer (video with ~8m views) #5 minecraft.org (fansite) #6 minecraftforum.net #7 Minecraft android app (official version of the game for android) #8 Minecraft on Facebook #9 IGN Minecraft review #10 Minecraft.com (a site capitalising on the domain but offering no Minecraft content)

Google is certainly not perfect but they are a mile better than DuckDuckGo, which does not even include the official Minecraft forum which is linked from the wiki, from the official Minecraft site, from almost every Minecraft Youtube video...

Wait I just did more "investigation", DuckDuckGo cares about capitalisation. What the hell? "Minecraft" and "minecraft" have different results and even the pages differ. Sometimes I can find minecraftforum.net listed for "Minecraft" (but not minecraft) and sometimes it just doesn't show at all. That's on typing in my search and refreshing; seconds apart and the results change.

</comment></comment></comment><comment><USER29> I use duckduckgo as my default search engine now. If i need some more info i just use g!. Really like the bang syntax!

And its doesnt track all my stuff, and this stupid google+ (failure) is not integrated. I just want to search stuff. Google did it right in the past, now duck duck go is my uberlord.

</comment><comment><USER30> That's a fantastic looking curve, but keep in mind the absolute numbers shown. They are at 1.6 million direct queries per day. Still a ways to go to be truly competitive (3 billion per day for Google). Still, if they keep up with current growth rates, they could do some real damage.

<comment><USER31> Just one at a time is all it takes. :) Replace starfish with searches..and bend your imagination a bit.

"A man was walking along a deserted beach at sunset. As he walked he could see a young boy in the distance, as he drew nearer he noticed that the boy kept bending down, picking something up and throwing it into the water. Time and again he kept hurling things into the ocean.

As the man approached even closer, he was able to see that the boy was picking up starfish that had been washed up on the beach and, one at a time he was throwing them back into the water.

The man asked the boy what he was doing, the boy replied,"I am throwing these washed up starfish back into the ocean, or else they will die through lack of oxygen. "But", said the man, "You can't possibly save them all, there are thousands on this beach, and this must be happening on hundreds of beaches along the coast. You can't possibly make a difference." The boy looked down, frowning for a moment; then bent down to pick up another starfish, smiling as he threw it back into the sea. He replied,

"I made a huge difference to that one!""

</comment></comment><comment><USER32> I swapped to DDG on all of my browsers about a month ago and I've been really pleased. Like others, I find myself occasionally using !g, particularly for programming queries. Things like !so, !mdn and !msdn actually help out a lot in some of these cases, which makes me think that as I learn the bangs I'll rely on Google less and less.

</comment><comment><USER33> pg's analysis was spot on, then [1]. I recently realized DuckDuckGo would be a hit when a hacker friend who does not read HN started using DDG as his primary search engine.

To me, this means it is out there. I switched too, at least for a while, and now I use both google and DDG, constantly having the impression that DDG's results are getting better and better.

[1] <http://paulgraham.com/ambitious.html>

<comment><USER34> I remember specifically thinking of DDG when PG mentioned search engines in that keynote. I've used DDG since 2010 and I would never switch back.

<comment><USER35> Although I don't use DDG, I too thought of them specifically when PG mentioned it.

Let's see if PG turns out to be right on this one.

</comment></comment></comment><comment><USER36> That's consistent with our Google Analytics. Visits from DDG are up to 58 last week, up from 15 the first week in January with a steady climb. There was a dip last week over week for some reason.

Compared to Google's 38K visits for last week, they have a long way to go to make a dent. We perform better in DDG results, as well.

</comment><comment><USER37> I've been using DuckDuckGo more and more lately since google added a redirect[1] for any link clicked in their search results a couple months ago.

I'm in China and most links I click on google don't load. Perhaps because China is messing with DNS queries to google.com and trying to limit the amount of data they collect on their citizens.

[1] The google link for duckduckgo search result first hits this link, and this is where the pageload errors occur: [http://www.google.com/url?sa=t&rct=j&q=&esrc=s&#...</a>

</comment><comment><USER38> I've been using DDG for quite a few months now and in general it works well. However I still find myself adding that "!g" at times when they find no results, and seeing what I was looking for at Google. I am also a bit disappointed with the "I'm feeling ducky"-results which are rarely what a person is likely to be looking for as soon as you get more complex than one word.

But that's the negative parts, and otherwise the positive outweighs those by far. I love the bang-syntax and I found myself using their information-box directly on the search page very often, mostly for converting currency.

I think better results will come with time and increasing usage, and I wish them good luck in the future.

</comment><comment><USER39> if i get that right, people are doing a lot more API calls than direct searches... Which begs the questions of (i) who they are, (ii) what they're doing, and (iii) how this can be monetized?

<comment><USER40> The graph is for direct searches though. You can see API calls stable at around 9M for quite a while, and direct searches have sky-rocketed.

You're right that there are a lot more API calls still (~x6). But the gap is closing quickly.

</comment></comment><comment><USER41> I just wish DDG could implement autocomplete/search suggestions. It's probably silly, but I've become as accustomed to typing a few letters into Google and it knowing exactly what I'm looking for as I have to tab complete in a unix terminal.

I've also really become accustomed to "instant" search. And I'll be the first to admit I thought was a total gimmick when it was announced.

Edit: After reading what I wrote, I realized it came off as if I didn't like DDG or didn't use it. Neither of those are true. I really like DDG and use it fairly often.

</comment><comment><USER42> Nice. If nothing else this proves that there is unmet demand in search.

<comment><USER43> It was mentioned before, but I think it's more a kind of "different" and unpersonalized search. Like proposed by Paul Graham: <http://paulgraham.com/ambitious.html>

<comment><USER42> I agree. My point is simply that this is strong evidence that Google's 'one size fits all' approach leaves room for other entrants.

</comment></comment></comment><comment><USER44> It's great to hear that DDG is doing well. The problem is that I will never switch over to it due to its shortcomings compared to Google and even Bing. I tried to give it a go for a day or so, but quickly realized the results being returned were not good enough.

1) It's not as accurate as Google. Sometimes DDG might return a better result (such as <https://duckduckgo.com/?q=alternative+to+picasa>), but 9/10 times Google is much more accurate.

2) DDG is playing catchup to a search engine that has hundreds of employees and a 10+ year head start. The types of queries DDG is trying to solve now were likely solved years ago and have been finely tuned numerous times.

3) I think this good video [http://insidesearch.blogspot.com/2012/03/video- search-qualit...](http://insidesearch.blogspot.com/2012/03/video-search- quality-meeting-uncut.html) about sums up the type of problems they are solving now and how advanced their platform is. Google is tweaking an edge- case scenario that is only .1% of their traffic. That is more traffic than all of DDG.

4) If DDG is all about respecting privacy and not tracking users, then use Google or Bing in incognito mode. Problem solved.

I'm not trying to be negative, but all signs point to a losing battle.

<comment><USER45> As a counterexample for number one, I've found DDG to be as good or better than Google on almost everything. The only place Google really beats it is in mailing list archives. DDG doesn't seem to crawl sites like gmane.

Numbers two and three don't make any sense to me as reasons not to use DDG. Using those criteria nobody would have switched to Google because they were just some college guys, while AltaVista and Yahoo have a big head start and huge departments working on search...

Number four is simply too inconvenient. I could go through the trouble of switching to incognito mode every time I needed to search, or I could just use DDG.

<comment><USER44> The difference now is that Google is constantly trying to improve now more than ever, whereas before the search engines thought it was "good enough". AltaVista and Yahoo were stuck in the technology of the day. The guys at Google had a crazy idea to implement "pagerank" and turn everything upside down.

I switched to Google because it was more accurate, and for no reason other than that. That's my #1 priority.

Fast forward to today and DDG is not trying to solve an accuracy or crawling problem. I'm not really sure what problem they are trying to solve other than to be an alternative search. There is nothing wrong with that, but I just don't think it fits me.

<comment><AUTHOR> They are trying to solve the same problems, just coming at it from different angles.

Google has given up on providing accurate search results to the anonymous user. They now believe that customization is the way forward (Global Warming results depending on your political leaning, etc.).

One aspect where Google's size is really killing it: _they should be manipulating results manually_. For example, a JavaScript search should return MDN in the top 5 results and W3Schools not at all. That doesn't happen, I believe because Google can't manually do stuff like this because of the fact that they compete in many different markets. That's the advantage of only being a search engine; a search engine's job is to be biased.

</comment></comment></comment></comment><comment><USER46> I use DDG a lot.

The reality is, it's not as tuned a search engine as Google is. However, I find it to be a more serendipitous search engine. It's better in breadth than G, IMO. That is to say, the top hit on Google is more precise if I want a specific thing, but the breadth of results on DDG is much more useful than Google in general.

As a matter of fact, it reminds me of the '90s, where a search could teach you more than just the one thing you were looking for.

I like it... most of the time. :-)

</comment><comment><USER47> Great to see DDG gaining traction.

However, I switched my Chrome search to DDG last year and the one thing that I realised I couldn't live without was maps.

I didn't realise how often I must search for an address and to have location appear in search makes life so much easier.

If you guys integrated with Openstreetmap, I'd switch back in an instant!

<comment><USER48> I'm a heavy Google Maps user as well.

OSM integration would be nice, but I've found that DDG still has slightly inferior results to Google.

I do like the approach to privacy and occasionaly use it :)

</comment></comment><comment><USER49> Last week I was searching for a: Sabian 18" HHX Chinese cymbal. I was considering buying one for my kit and wanted to see some reviews, video samples, prices etc.

DuckDuckGo: The top there results were American Ebay search results (I live in Canada), a drum shop in Memphis (15 hour drive), and various other online stores (discountdrumequipment.info).

Google: The first page of results had links to the official Sabian product pages, Youtube videos of people playing it, and the most impressive thing was a 3 day old Kijiji ad (again, I live in Canada) with the exact cymbal I wanted for a good price in a city just 45 minutes from me.

I responded to the ad, drove the 45 minutes, and purchased it. DuckDuckGo's stance on user privacy is admirable, but Google still gets my vote... it's accuracy still impresses me.

</comment><comment><USER50> I recently switched to using ddg for all of my searches.

So far, I have been happy. I normally find what I am looking for within the first few results, and I find myself using !g less frequently.

ddg certainly feels faster and less irritating than google. I have been very satisfied.

</comment><comment><USER51> Hi, I've only been using DuckDuckGo lately and I love it. Search quality is so impressive and the respect for privacy is very welcome. I hope it is able to maintain this independent feel and not sell out like others did.

</comment><comment><USER52> Awesome. I've heard of duckduckgo from a coworker and use it regularly.

I'm creating an art site and have a search feature. I made sure to include duckduckgo along with google, and bing as search providers. It only makes sense, right?

</comment><comment><USER53> The only thing that's missing from DDG taking over the world is good i18n. Really, it's the only thing that ever made it impossible for me to recommend it. A lot of especially older people outside the US use Google mainly because they don't or can't care for the rest of the world and simply don't have the required English skills.

This isn't just about the interface language and search results, but also about !bangs. To be useable !ebay has to use the the already available region setting.

Other than that I have been an advocate from the early days, because DDG is what the world needs.

</comment><comment><USER54> I don't really understand why so many people think that the results on DDG are worse than Google. Maybe I just don't search for things that other people do, or I search differently, but for me, I usually prefer the DDG results. Most of it has to do with the zero click info and the disambiguation. Especially the disambiguation. I get better results because most things I search for could be disambiguated, and I take full advantage of that to quickly give me just the results related to what I'm looking for.

</comment><comment><USER55> Incidentally, I switched my default search to ddg 2 days ago. Use !g a lot but also ddg native frequently. Starting to get a feel for what type of search will benefit more from DDGs extras.

</comment><comment><USER56> Related discussion on the previous thread here, about a month ago: <http://news.ycombinator.com/item?id=3661345>

</comment><comment><USER57> It's interesting to see how many requests are coming from the API, offhand I can't think of any uses that I've come into contact with. Does anyone know who is using the API?

<comment><USER58> Fastestfox, Cyberspace and others use our API: <http://duckduckgo.com/api.html>

</comment></comment><comment><USER59> It's very exciting to see usage of DDG increasing and the service getting more and more recognition. I've been using DDG for almost two years now and love the service. The privacy is what drew me in, but the goodies are what made me a user for life. It's the default for my browser and I use the iPhone and iPad apps regularly. Keep up the good work DDG! I'd be happy to lend my sysadmin skills to help :)

</comment><comment><USER60> I've been using blekko.com and found that I prefer it to ddg. Like others have, I too go back to Google for programming related searches at times.

</comment><comment><USER61> I tried to switch to DDG recently; after a couple of weeks using it as my default search engine in Chrome, I had to switch back to Google.

Speed was a pretty big factor, but more than that, the search results just didn't compare. Still, I'm rooting for them, and hope that they're able to close the gap, both in speed and results. I love their style, and all of the site-specific search shortcuts.

</comment><comment><USER62> Did you know Duck.com redirects to Google.com? That's pretty smart of Google. In other thoughts, congratulations on the exploding growth!

</comment><comment><USER63> I just want to say I use duckduckgo, but the reason is because I find Google's recent policies to be unsupportable and I no longer want to use them. So far duckduckgo has been inferior in just about every way to google, most notably search speed and search quality. Also I have seen some extremely dubious scammy ads in ddg, which make me uncomfortable for sure.

</comment><comment><USER64> You are #1 for programming related queries. I am curious about seeing you build up the similar precision/recall for other areas.

</comment><comment><USER65> The thing that worries me about DDG is their reliance on others' search engines.

If there was ever a true defection from one of the big guys' services to DDG (i.e., if DDG was _actually_ blowing up), it seems to me they'd just replicate many of DDG's features for their userbase and DDG would wither, or even just pull DDG's access to their search engines.

</comment><comment><USER66> Nice curve.

March 1st caused me to go from DDG dabbler to browser default.

Still use !g, but I find that I recommend DDG more often than I use !g.

</comment><comment><USER67> Have been using DDG for the last 2 months but have to admit I'm feeling withdrawal symptoms, mainly due to speed. I hope that the increase in traffic is mainly responsible for things being a bit slow - the sluggishness (NOT results) is the primary thing making me yearn for Google at the moment.

</comment><comment><USER68> Pretty damn good. I love how the search engine itself helps you find a better context to frame your search into. It's like having a conversation with the website (instead of it assuming I want something because I searched for it before and it's "most likely" I was looking for this)

</comment><comment><USER69> Does anyone know how much of that comes from Linux Mint?

I dabbled with DDG a few times, but only when I changed to Mint did I start using it frequently. I've since wondered how much of a proportion of the traffic of DDG is because it's now the default search engine for Firefox in Mint.

</comment><comment><USER70> I've been using this search engine for a week and I really like it. The only thing I've noticed is that Google has conditioned me to never look at the top section because that's where sponsored ad's are so I have to force myself to click on the top link.

Brainwashed by GOOG :)

</comment><comment><USER71> I love DuckDuckGo. My only real complaint is when their search falls back on Google (for maps, and things like that). It would be great if DDG were able to start providing their own alternative image/map searches. I would probably switch to them fulltime.

</comment><comment><USER72> As with most new search startups, people will give it a chance. Hence the increase in traffic. I tried it a couple of times but, as most people do, I then checked Google just to make sure I'd really covered what is considered the full spectrum of search.

<comment><USER58> !g "search query" on DDG will do the search on google.

Entire list here: <http://duckduckgo.com/bang.html>

</comment></comment><comment><USER73> Give me a quick (or configurable, or special URL) that includes a "search on Google Scholar" link and I will switch from Google. Ever since Google's visual refresh that removed the menu item for search on google scholar, I've lost one of my timesavers.

</comment><comment><USER74> For everyone finding DDG to slow, my last recommendation on this one.. startpage.com :)

<https://news.ycombinator.com/item?id=3662475>

(just to make people aware of alternatives, before returning to google/bing/etc)

</comment><comment><USER75> I feel part of it's sudden huge growth has to do with udacity search engine course.

</comment><comment><USER76> Hooray I like DDG and switched awhile ago making it my default search. Congrats.

</comment><comment><USER77> I've tried hacking the Safari Binary using the online instructions to make DDG my default search (GLIMS sucks), but I can't get it working at all. I can't find the lines that I'm meant to be replacing.

Has anyone else had any luck?

</comment><comment><USER78> _Wed 28-Mar-2012 1,518,581 direct 9,580,029 api 225,161 bot_

Am I reading this right? DDG does ~6 times as much traffic in API requests compared to search requests?

That makes a pretty significant difference to the monetisation strategy!

<comment><USER79> Note though that the growth of direct requests is much faster than the growth of API requests.

</comment></comment><comment><USER80> That's awesome! Just goes to show that search is still an area for growth.

</comment><comment><USER81> Yes, I've noticed an increase in speed over the last couple of days.

Keep up the good work!

</comment><comment><USER82> I really wish I could use DDG on iOS then I could fully replace google.

</comment><comment><USER83> as a happy DDG user I have to say congrats and I absolutely love the zero click box, a feature that often provides superior search quality than Google.

</comment><comment><USER84> I love using DDG through the Tor network. <http://3g2upl4pq6kufc4m.onion>

</comment><comment><USER85> I started to use it about a couple of weeks ago

</comment><comment><USER86> Do they publish revenue? I wonder what the infrastructure costs are and how they scale in comparison to the number of request.

<comment><USER87> I think that most of their results come from Bing if I remember correctly. So the scaling doesn't have to be as heavy as it might seem. But I may be wrong.

</comment></comment><comment><USER88> Careful with the headline, I'm sure it caused some terrorist alert bells to ring in som US three-letter agency. :)

</comment><comment><USER89> My kids like the cute picture of the duck.

</comment><comment><USER90> Nice to see such an important and modern project written in Perl, which people around here shit on all the time.

</comment><comment><USER91> Nice! You guys deserve it

</comment><comment><USER92> It would be nice to see those graphs with a logarithmic y-axis.

</comment><comment><USER93> Thanks for this! I was finally fed up with goCIAoglNSAe.cFBIom

</comment><comment><USER94> Tip: Use the short URL ddg.gg for less typing.

</comment><comment><USER95> congrats. i stopped using google after their last policy change. DDG mostly fits the bill, apart from some special cases.

</comment><comment><USER96> Chrome autocomplete support would be awesome.

</comment><comment><USER97> Glad to hear. Been using it more of late.

</comment><comment><USER98> So how is DuckDuckGo going to make money?

<comment><USER99> It's already monetizing via ads specifically, sponsored links which are highlighted in Yellow and are syndicated through Microsoft adCenter. They also take affiliate commission on sites like Amazon and eBay.

</comment></comment><comment><USER100> Long live the Hockey Stick!

</comment><comment><USER101> hurry up and get some servers in India.

</comment><comment><USER102> Congrats!

</comment><comment><USER103> I like the concept of DDG but unfortunately the results just aren't there yet...

Look at the search results for "Google Verbatim Mode" on DDG vs Google.

On Google, all results are actually about the "Verbatim" search mode on Google. We even get results like "How to set Verbatim mode as default on Google Chrome" with actual instructions.

On DDG, we instead get a really crappy blog article as the first result, and some unrelated junk about Google Earth DirectX and Windows 8. There's one result that asks the question most people searching for that query would want answered, "I would like to be able to use verbatim mode as standard". However... it's a link to a newsgroup posting with no answers, unlike the Google results which actually provided links to blog postings with the answer.

So, I'm going to use Verbatim mode on Google for now, now that I know how to activate it by default... Let me know if you guys work these issues out, I'd love to use DDG, but it's just not returning valuable results reliably yet.

</comment><comment><USER104> Treading carefully here ... I love DDG and find it yields far more useful results than anything else.

Besides, I like not getting tracked all the time and I can switch off ads without needing any extensions. That in my view is a very civilised set of options and the rest of the Internet should follow by example.

</comment><comment><USER105> I'm using DuckDuckGo since two days now as default _everything_ (since I've read this thread) and my surprise is that I've not switch back to google yet. Hey that's awesome since I'm not specially an early adopter statistically. Btw, I particularly like the insight box.

</comment><comment><USER106> Google has been annoying me lately. I'm willing to give this a shot. Good luck DDG. Resetting Chrome default search engine now.

</comment><comment><USER107> <http://www.mbtforshoes.com/> good shoes

</comment><comment><USER108> I <3 DDG.</comment>


Ask HN: Would spellchecker, dictionary definition and thesaurus APIs be useful? - <AUTHOR>

As a by-product of a larger web project we're working on, we ended up doing a fair amount of database and coding work on English dictionary / definitions / synonyms / thesaurus recently.
Thinking about this in retrospect leads me to believe that if spell-checker, word definition lookup and synonym lookup functionalities are made available in the form of web APIs, these could be useful to (many) developers. (I'm not really sure if such APIs are already available.)
One mass-use application that I can think of is to spell-check/correct user generated/submitted content.
What do you think? Do you think this is something you and/or others will find useful? If so, in what ways?
If we find enough interest (>20/25 ayes?) here, we'll develop and publish these (free) APIs in the next couple of weeks or so and announce the availability here.
What do HNers think? <comment><USER2> It would definitely be useful, but I'm wondering what licence you would/could release under? Where did the initial datasets themselves come from?

<comment><AUTHOR> Thanks for the feedback.

> Where did the initial datasets themselves come from?

Public domain. :-)

Edit: You can also check dict.org for sources etc.

</comment></comment><comment><AUTHOR> I forgot to mention earlier, the main dictionary database (GCIDE) will contain ~120,000 headwords. An example implementation can be seen here:

<http://www.TheEnglishDictionary.org></comment>