I Took a Pay Cut for a More Meaningful Job - <AUTHOR> https://www.fastcompany.com/90308995/i-took-a-huge-paycut-for-a-more-meaningful-job <comment><USER2> The problem with trying to find meaningful work is that 1. the pay is generally terrible, and hours will likely be long 2. if someone is paying you, it's still a "job", with all the implications, responsibilities, and power dynamics that go with that. You still take orders from a boss all day. And if it were so damn fun, then they probably wouldn't be paying you for it.

The most fun internships/jobs I've ever had were all unpaid, because I could come and go as I please, and literally work on whatever I wanted. The second you're getting paid, your personal interests are thrown out the window, and your job is now to take orders from your boss.

Of course there are exceptions. I'd imagine that being a tenured professor would be pretty sweet (though getting a job like that these days is extremely difficult and competitive), or if you're a doctor then helping save lives in Africa or something would probably be extremely fulfilling. Being a politician seems like it would feel meaningful, and they get paid solid six figure salaries.

But overall, the search for a meaningful job that also pays decently and doesn't require crazy hours is like searching for a unicorn, so for most people with the talent/luck/opportunity of working high paying jobs, achieving financial independence first before seeking meaningful work is probably the smartest move.

<comment><USER3> I don't think meaningful work has to be necessarily something with crap pay and the workload of a donkey in the first world war.

The key is the word "meaningful". I know a lot of people (and I used to be one), who would shift from job to job and leave when things weren't right any more. But this typically seems to involve taking what you are as a person as given, preferring instead to simply try and change one's circumstances to match that.

It's much easier to find meaning in your work if you're prepared to allow what you are as a person be up for negotiation somewhat, in addition to wanting to exercise a degree of control over your working circumstances.

A reflection of this is pretty easy to see in the way some people frame their decision-making when deciding whether to get a new job or not. They ask themselves "is this really what I'll want?", as opposed to "is this what the person I'll be after a few years of this will want?" or "will this turn me into a person I want?". By unconsciously taking one's self-image as fixed in that way, it becomes harder to find meaning in work and as a bonus also makes it somewhat more likely that you'll make the wrong decision because your potential future mental state isn't accounted for.

Not saying you do this at all, but that there are a lot of shades of grey between being an intern and having to do what your bosses say slavishly. And a lot of them are only really visible when you look inward at yourself rather than outward at your environment.

I've got a job at the moment that's great because of this - it's not perfect, but being able to work on what I am at the same time as shaping what this job is makes the job of finding a happy means of existence drastically easier.

<comment><USER4> If the meaningful work you want to do is in education or environment... There is basically no well paying jobs.

There's no business profit in clean water or air, so capitalism doesn't cover the need. The fact there might be no customers in an area in 50 years is outside the scope of economic patterns. In 40-45 years all the current business leaders will have their pockets lined & be retired... and new leaders will beg the government to fix "the problem no one saw coming"... Shifting businesses profits today for taxpayer-funded fixes later for "economic revitalization" or "environmental repairs"

On top of these issues... if you have bills, you often can't take a pay cut to pull this off. I'd be willing to take a pay cut for meaningful work, but it would still need to cover child care and rent. My wife looked at jobs with the EPA, they offered $50k for masters degree and 8 years of experience. A single family home on 1/5th acre is $500k+. If we both did meaningful work we could not live there, let alone have kids ($1250/month child care).

We decided we just couldn't do meaningful work in the DC area and left. This stinks because we really need to be worried about this stuff for our children's future. But as long as private industry pays 50-75% more, the rising costs of everything to accommodate those well paying jobs chokes out teachers, environmental workers, and low-wage retail jobs (aside from kids still living at home)

<comment> barry-cotter > If the meaningful work you want to do is in education or environment... > There is basically no well paying jobs.

Lambda School is an existence proof that you can do good and make money in education. Minerva University looks pretty good too. Udacity and Coursera are helping and have helped a ton of people as well. When I was a kindergarten teacher I used Starfall.com all the time. Mathletics and ixl.com come highly recommended by friends who teach primary school. Outschool.com are working on a teacher student marketplace for classes. Baselang.com is the best place to learn Spanish on the internet, hands down. Italki.com is a pretty good marketplace for language lessons too.

You can't do much with schools really because schools are run for the benefit of teachers not students but if you want to help students there's a lot to be done in education, and plenty of people doing it.

</comment></comment></comment></comment><comment><USER5> I did this in 2013. Went from working in a lucrative, stressful high frequency trading systems development environment, into developing systems for humanitarian organizations. Loved the work, the next two years were the happiest and the most chilled out in my life. The negatives: burnt through a lot of my savings. I'm back in a different industry now, and the return was a bit painful, but looking back, it was one of the best decisions I've ever made.

Advice:

1\. Have at least two years worth of savings, or an alternative source of income that can cover all your basic expenses, or don't do it.

2\. Don't take a job that requires you to work harder than you did in your old job. In the beginning, the change will be invigorating, but soon it'll start to wear you down. During my "sabbatical", I worked only six hours a day at most, saving plenty of time for family and personal pursuits.

<comment><USER6> But in your case there is also a moral dimension, not just self-fulfillment: Financial speculation is parasitic and socially harmful without actually providing anything of actual real value. So, IMHO that's a switch worth making for essentially any decent job.

Also, about working harder - that depends on how hard your old job was; and some of that hardship is sometimes due to working on something you would rather not be doing.

<comment><USER7> _Financial speculation is parasitic and socially harmful without actually providing anything of actual real value._

Is it? Farmers hedging against crop failure are getting a useful service, it's allowing them to invest in a crop without the risk of bankruptcy. The same goes for airlines who hedge the price of fuel, not many people would run a business that risked collapse if the price of oil goes up by 10%.

<comment><USER8> The futures market ran fine before the massive high frequency trading vampires running around. Liquidity isn't a moral good.

<comment><USER7> _Liquidity isn't a moral good_

It does make markets more price efficient though, as a small investor I prefer this over being dependent on the whims of someone like Warren Buffet.

 _massive high frequency trading vampires_

It's a relatively small industry, profits in the US peaked at approx $5B in 2009 and have declined since. [1]

[1]: [https://en.wikipedia.org/wiki/High- frequency_trading](https://en.wikipedia.org/wiki/High-frequency_trading)

</comment></comment></comment></comment></comment><comment><USER9> I will chime in here as I did something along these lines. It's not worth it. If you/anyone wants to do meaningful work. Make lots of money, use the extra money for things that matter to you. It's a hard learned lesson. But everything improves when you are getting paid more. Your colleagues, the opportunities, literally everything

<comment><USER10> "everything improves when you are getting paid more"

This is demonstrably not true, as shown by a slew of studies on happiness. Past a certain point, more money does not meaningfully impact your life and make things better. That point is different for different people in different locations, but signing up for golden handcuffs is a fantastic way to make yourself miserable.

If you've got a choice between 10% more pay to work with people who make everything suck, and 10% less to work at a place with interesting projects and people you like, take the pay cut. From personal experience, no amount of hobby funding will make up for you hating your job.

<comment><USER11> Often, the pay at the more meaningful job is under that happiness threshold.

<comment><USER10> Outside of the likely salaries for people on HN I would agree with you. However, I'm fairly confident tech worker salaries start at or above the threshold in most locations.

<comment><USER12> Also, tech workers often have the luxury of working remotely, so there may be the option of relocating to a place with lower cost of living (if the meaningful workplace allows remote work).

</comment></comment></comment></comment></comment><comment><USER13> This is what the FIRE movement (Financial Independence and Retire Early) is all about. I would highly recommend the book "Your money or your life" by Vicki Robin. The best approach to switching to more meaningful work is to start by getting your financial life in order. I speak with personal experience: I realized earlier this year that I did not need to feel stuck in a dead-end job and this gave me the courage to quit.

<comment><USER14> The Fire movement is pretty harmful imho. I think it benefits the authors and podcasters much more than it helps actual people. It probably works for very specific personality types. And really shouldn't be sold to anyone and everyone.

I have friends in their 40s who retired early. And most of them are loosing their minds. It's sad and frustrating to watch. They have nothing much to focus on daily, and after a few years of it, you can see the cognitive decline compared to the rest of the peer group. Their people skills keep degenerating, from lack of constant challenges that the middle aged adult deals with at work or home. And that mounts over time, making lot of things harder and harder.

I get all the frustrations people (rightly) have with institutions, govt and corporate robot wonderland, but I have done more interesting and personally satisfying things being within those structures, than I have done anything outside them.

Financial independence is not just an emergency eject button when faced with difficult choices. It a big asset in the corporate world while competing with a cohort that is deep in debt (alimony/kids college/health expense/mortgages/golf club membership etc etc). Sooner or later everyone in any org knows who can be bought and who can't. Both kind of people have value to the org and impact on the org. People who misunderstand that, end up pushing the eject button prematurely.

<comment><USER15> Cognitive declines in their 40s because of not working professionally?

That's the first time I've ever read this, so I'm very skeptical.

But if it were me (and I'm working in that age range right now), I'd be jumping on personal projects I've always desired. Writing, in particular. And probably a little tech project stuff in areas that interest me.

May I offer that many of these wealthy people you know aren't intellectually- minded? That's certainly true of most financially successful leaders I've come to interact with.

<comment><USER16> This article and post is timely, because I "retired" last Friday (I'm 37). It doesn't ring true _at all_ to me that retiring would necessarily be bad for your cognitive skills. Maybe it depends on the type of person, but I'm engrossed in personal cognitive pursuits, such as teaching myself machine learning, putting together an app of my own for the first time, indulging my writing hobby, and of course spending more time with my daughter. It is these very pursuits which led to me retiring; I want to throw myself completely into them instead of dabbling with them on the side. I don't see how someone would "lose their mind" from retiring, unless retiring aggravated other vices such as drinking or watching excessive television.

I've no doubt one's personality slowly changes when no longer chained to authorities saying what to do and when to do it, and when no longer exposed to corporate "culture". That's quite different than one's brain turning to goop. After only a week, my brain feels clearer than ever as the cobwebs of work stress slowly clear away (it still hasn't landed that I don't _have_ to work ever again). Maybe his friends who are "losing their minds" are just seeing the world a bit more clearly? Or perhaps they were uniquely unsuited to retirement, lacking any intellectual pursuits?

I'll probably write an article on my experience in a year or two and let you know if my brain has turned to goop, assuming I can still read/write when that time comes!

Edit: as long as we're throwing out competing anecdotes, the only person I know personally who retired so early is wildly successful. Far more than he was at work.

<comment><USER17> > the only person I know personally who retired so early is wildly successful. > Far more than he was at work.

If he's wildly successful, did he really retire?

<comment><USER16> Well, that's the rub, right? Anyway, what he did was get super into his hobbies, one of which was "urban gymnastics" (or whatever it's called), and eventually ended up on the Indonesian version of American Ninja Warrior, in his 40s (!!), which I found pretty inspirational.

Not a lot of money in that, at least for most people, so I think it's fair to call it retirement.

</comment></comment></comment></comment></comment></comment><comment><USER18> > _Most Americans say they would give up a more lucrative job for a more > meaningful job that pays less in a heartbeat._

{citation-needed}

Obviously, someone struggling to make ends meet and having little retirement savings is hardy in that position.

Oh wait, we're only talking about what people _say_! Well, sure, people _say_ such things --- in an idle moment when thinking ramifications through is temporarily off the table.

"Gee, I wish I had a more meaningful job, even if it paid less (just as long as I could somehow maintain my current lifestyle). Also, I won't get into numbers about _how much less_. Maybe just a few hundred dollars a year less, that type of thing."

The key metric would be, how many people _act_ on this impulse. How many Americans are actually switching to lower paid jobs that are more meaningful? And are actually meaningful, that is, and where that is a choice: not being forced to switch to a lower-paying job and then rationalizing it afterward as being more meaningful.

I suspect plenty of people would also take a _less_ meaningful job for more pay. What could be less meaningful then getting money for doing nothing? Yet, that's what a lot of people want, such as anyone who buys a lottery ticket.

<comment><USER19> > Also, I won't get into numbers about how much less.

They get into numbers in the second paragraph; up to 23% of lifetime earnings.

> The key metric would be, how many people act on this impulse.

We can get a rough tally of how many are acting on it from labor stats[1]. And I'm assuming all of these are what people view as "meaningful" work; I work in finance and find it plenty meaningful but I imagine that's a minority view.

Quoth the BLS:

Total employment: 144,733,270

Community and social service: 2,171,820

Arts, Design, Entertainment, Sports, and Media Occupations: 1,951,170

Healthcare Practitioners and Technical Occupations: 8,646,730

Healthcare Support Occupations: 4,117,450

Education, Training, and Library Occupations: 8,779,780

So if those broad areas hit the "meaningful" buttons, you're looking at about 17% of the workforce.

> What could be less meaningful then getting money for doing nothing?

Totally had that doing government contracting. The trouble is you realize it's a trap: if you ever need a real job your lack of doing anything means you're fucked.

> Yet, that's what a lot of people want, such as anyone who buys a lottery > ticket.

More realistically, anyone who is investing passively. But that's not a job or your profession, and when people talk about "meaning" they're asking "who am I?" If you're a carpenter, your identity is wrapped up in the houses you build, for instance.

What the article is missing is the classic dichotomy of work to live vs. live to work.

If your profession is who you are, meaningful work is more important in shaping your identity.

But if it's not, the "work to live" paradigm, you're making money to support something else. If you're working to take care of your family, you're a father or a mother first. If you're working to support your art or projects, you're an artist or an engineer regardless of your paid job title.

[1]: [https://www.bls.gov/oes/current/oes_nat.htm](https://www.bls.gov/oes/current/oes_nat.htm)

<comment><USER16> 23% is remarkably little considering compound interest and the marginal utility of money. It sounds a little bit like the old nugget, "how much money do you need to retire?" and the answer is always "a little bit more than I have".

</comment></comment></comment><comment><USER20> Personally I disagree with both the article and most of the comments. I don't particularly care about meaningful work, I enjoy solving problems that matter and I'm good at it so I want to do that. Retirement, however, doesn't interest me since most interesting impact-full problems are in industry. I actually like having a boss who does all the boring things regarding running a company so I don't have to. At the same time I will trade money for a less stressful job (ie: less politics, better boss, less boring work, etc.).

<comment><USER21> Can you say more about the difference between impact and meaning? What would be an example of meaningless impact that has value to you?

<comment><USER20> To me, meaningful would mean something like helping the poor or working for a decent non-profit.

On the other hand, working for an ad-tech company and increasing revenue by 40% through some machine learning would have impact (on the company, clients, users, etc.) but not be particularly meaningful.

<comment><USER22> I don't see "meaningful" as referring to something like helping the poor, I see it as referring to feeling fulfilled. If that involves helping the poor, more power to you. Me, I took a $20k paycut to switch from writing node js all day to pen testing at another firm and I could not be happier. It's been two years since switching and I still look forward to waking up and going to work each day instead of groaning and dying a little on the inside. Paying $20k/year to be happy and feel fulfilled is a complete steal in my book.

Just my own anecdata though.

</comment></comment></comment></comment><comment><USER23> Nothing creates more meaning for you than to create the opportunities based on knowing what you love. Since we're all different, different things create meaning for us differently. Rarely will a job someone else creates push our buttons best.

I created my course at NYU and wrote the book _Initiative_ [https://www.amazon.com/Initiative-Proven-Method-Bring- Passio...](https://www.amazon.com/Initiative-Proven-Method-Bring- Passions/dp/1733039902) to create a step-by-step process to unearth each person's passions. I find that reflection alone or taking online quizzes doesn't reveal passions like action does -- that is, taking initiative (thoughtfully, with a process that works).

Recently, one person started blogging his results doing the exercises. As of August 25, here's his latest: [https://anthemoftheadventurer.com/2019/08/23/exercise-6-10-p...](https://anthemoftheadventurer.com/2019/08/23/exercise-6-10-people- closer-to-my-field).

<comment><USER24> Thanks. I hope I can find ways to extract the essence of your method and subtly incorporate it into CS education.

<comment><USER23> Let me know if I can help. I'm easy to find online.

</comment></comment></comment><comment><USER25> I can add some perspective. One and a half years back I took a 80 percent pay cut to join a 10 member startup. My salary now has improved to almost 60 percent of my previous company. I don't really regret that decision though and here are few things I clearly thought about before making the decision.

1\. Learning and growth opportunities at the new place

2\. Autonomy around my work

3\. People

I think people derive meaning when they see that their ideas are appreciated and users care about the product. I liked the people at my previous job, however the first two items were somehow missing.

It also helped that one of the co-founders at the startup was a friend from undergrad.

</comment><comment><USER26> Looks like some of the people interviewed in the article are unhappy with their job change, and some are satisfied.

The bottom line is it's a big risk to take, and such a choice may mean having a more fulfilling career but also having a lot less money in retirement. It would be interesting to see how many of the happy people in the article will still be happy when they are retired.

<comment><USER27> ... and how many of them died before retirement and never got the enjoy the fruits of their hard work.

... and how many of them find themselves with lots of money in retirement, but discover that they have missed their window to spend quality time with the people they love. Spouse has died, children have grown and moved away, etc.

So there are risks in the other direction as well.

<comment><USER16> My father retired "early" (at age 59) and was diagnosed with a brain tumor the same year and passed away. It's so true that "waiting to retire" is a huge risk if you're banking on indulging your hobbies then.

</comment></comment></comment><comment><USER28> Life is short. You can't take money with you. Do what you want to be doing now, not later. Scrimp and live cheaply > hating a stressful life of luxury/debt/peonage/agony.

</comment><comment><USER29> > _"I was eight months pregnant," she says. "They escorted me to my desk with > a box to collect my personal belongings, and that was it."_

In many a European country the company would be illegal in doing this.

Employers only recently got the ability to fire pregnant/breastfeeding women in the EU, and only as part of mass lay-offs.

<comment><USER30> It's illegal in the US to fire someone _for_ being pregnant (FMLA).

<comment><USER31> unless you work at will

<comment><USER30> AFAIK FMLA covers at-will employment (which is the vast majority of employment in the US)

</comment></comment></comment></comment><comment><USER32> This confirms my belief that one should find a job they're good at and retire early instead of attempting to improve work-life-balance by trying out "feel- good" jobs. Success at what you're cut out to do makes you happier than chasing the current mainstream moral agenda, saving the world and so on.

<comment> marble-drink Absolutely. There's this popular sentiment today that says anyone can do/be anything. It's certainly popular in tech, but I have limited knowledge of other areas. Talent is becoming a bad word.

But I'm convinced that one of the keys to happiness is to live a life that plays to your strengths. If you have a natural aptitude to problem solving etc then you could become a really good engineer, or you could become a mediocre violinist. Which is going to lead to a more fulfilling life?

People are born with natural talents. We should be encouraging people to do what they're good at, not telling them they can do whatever they want to.

<comment><USER33> I think this comes from the fact that we have automated/formalized most of the meaningful work. If you spend 10% of the time solving problems and 90% playing petty office politics, it doesn't really matter how good of a problem-solver you are. Especially if most of the problem-solving part comes down to following clearly defined procedures.

The flip side is that the economic contribution of a single employee becomes vanishingly small, affecting the salaries, or rather what you can buy from them.

I agree with you that building your life on top of your natural strengths is the most fulfilling way of spending it (having done so myself), but I think these days it is extremely hard and is only possible if you manage to start your own business and grow it to the point where it can pay your bills.

</comment></comment></comment><comment><USER34> I think the reality is, if we're talking about tech, that many companies market their "mission-driven" culture to make it seem like what they do somehow has more meaning than working at Google or Amazon but it's mostly just a ploy to get people to do exactly this e.g. take a pay cut and feel OK about it when, in reality, what they do is probably not any more "good" for the world than what a big co is doing.

I recently went to a talk on "effective altruism" and came away feeling like that was a better model for impact e.g. make a lot of money and give a lot of it away to organizations and causes where the measurable ROI is the highest.

</comment><comment><USER35> Can't see how encouraging people to take more "meaningful" jobs isn't just another form a vicious ladder kicking disguised as feel-good advice that will stunt their financial growth for several years, and possibly make them unable to ever catch up to their peers.

People who already have a fair amount of wealth independence are the ones who can afford to take "meaningful" jobs. If that's not you, there's no shame in being in service to a corporation and getting paid good money for it. Our values of what is meaningful changes over time, and it's not worth taking paycuts to chase such mercurial ideals.

<comment><USER36> I agree, sometimes you have wait and work for a guy you do not like in a job you do not particularly enjoy. If you can stand it and it pays your bills it's ok. I guess, the real problem is when people have the opportunity to change their jobs for the one of their dreams but they are simply too scared to get out of their comfort zone.

</comment></comment><comment><USER37> I took a $40k pay cut to work in social good, and I can't recommend it enough. I have an advantage in that I don't have a family or dependents though. Just putting this out there in case any one's thinking about it but thinks it's too crazy: it's a lot more crazy to spend the most productive hours of your life on something you at best don't care about, and at worst actively despise.

</comment><comment><USER29> > _Katy Rey was running a large call center in Florida and working long hours > when she heard about opportunities at Tesla's solar panel division. Rey > jumped at the chance to work for a company where she felt she could do > something more meaningful, even though her salary would be slashed in half. > "Everyone was taking pay cuts in the room," she says of an early training in > Vegas. "And the people they recruited were outstanding." They could deal > with the pay cut, Rey says, because they felt like they were "saving the > world."_

Well, Musk himself did not take any pay cut, so that should have been a pointer...

> _Rey felt like the company found intelligent people and convinced them to > push their limits. (...) Six months into the job, Rey was abruptly laid off > over the phone_

When "changing the world", companies should start with how the treat their own employees...

</comment><comment><USER38> I think the question around performing meaningful work is largely rhetorical; the challenge is where do you find meaningful work?

We tend to love the romantic notion of throwing away the high paying meaningless (to us) job to follow your dream, but in reality it tends to be more of a dream than the way things work.

I wish more people would put some deep thought into (1) defining what meaningful work in their context would look like, and (2) building a strategy for moving from their current state to the desire one that doesn't involve essentially throwing away what they have to start over.

As far as trading money for meaning, I see this as an alternative view on the well known limits of extrinsic motivators; you'll trade any money beyond your personal level for deeper purpose. The key then is figuring out what this amount is, then evolving into meaningfulness.

</comment><comment><USER39> I took a pay cut to go to my last company. Every employee of the original company was getting abused so badly by the founder that I really would've taken anything. It just so happened that the company I was taking a pay cut to go to had tremendous opportunity for growth, and what started as a medium sized paycut turned into a huge pay increase. I'm glad it worked out that way, but I'm certain that pattern is in no way common.

Now, I would probably take a small cut to do something I was excited about. I've been working with my therapist to try and figure out specifically what that would be, but I haven't had a ton of luck.

</comment><comment><USER40> I'm also now kind of doing the same thing. I left a high-paying job in NYC/SF and am now applying for remote jobs. The compensation for remote work looks like it'll technically be a paycut from my NYC/SF salary, but I think the freedom of being able to live where I want, work my own hours, and not have to deal with office bullshit will made it worth it - not to mention the cost savings from not giving away an extra #1-2k/month of your paycheck to NYC/SF landlords.

</comment><comment><USER41> I feel this on some level but at the same time, I've personally never had an issue finding money to go along with meaningful work in our industry. Maybe I'm just lucky though! A few of my friends work in public sectors, love the work, and don't make much but they are definitely happy. Just can't see myself not making good money over something I'm passionate and reasonably good at. Especially for the level of effort and impact I'm making.

</comment><comment><USER42> I would be happy to take a pay cut to help people again, but I'm not willing to take a big enough pay cut to be a realistic candidate in the non profit world. The conflict between my real and ideal values tears me apart daily

</comment><comment><USER43> Any discussion about a meaningful job will benefit from a look at 80,000 Hours -- [https://80000hours.org](https://80000hours.org)

Making a positive difference in the world is a great thing to do.

</comment><comment><USER44> Meaning is just a story that people tell themselves about their job. Its as much about personality and framing as it is about the actual tasks and duties performed.

</comment><comment><USER45> I did this once. Great life experience. I left two years later, but that was after traveling the world on the company dime.

</comment><comment><USER46> as a developer, and software builder this is relevant. would you make more writing in a code / framework you don't like. or take a small paycut but be remote and write code in a language / framework you like ?

</comment><comment><USER47> Honestly, there's so many variables you can't assess about a new job, that unless you are 100% sure about it (or driven mad by your current job) you shouldn't make the jump for less money. You wanna be happier? Then make the jump for a job that pays less because it requires you to WORK less. Try working 4 days instead of five or 6 hours instead of 8. Now that's a change.

<comment><USER48> You make a very good point. I think the biggest factors that can determine happiness at work are the immediate boss and the immediate teammates, two things that are very hard to assess until you have already joined the new place. But the money is easy to assess and guaranteed because it is written on paper and it is legally binding. A boss with a good attitude, however, isn't guaranteed, no matter how rosy things look during the interviews. So it makes sense to never take a pay cut unless there are other compensating factors that are also guaranteed. Working 4 days instead of 5, or 6 instead of 8, seem like those things that can be guaranteed by the contract and would be worth taking a pay cut for.

This reminds me of a personal story when I interviewed for a very popular startup in the valley. They boasted of working 12 hours every day. I had a competing offer from a non-startup which required me to work about 7 hours every day. The CTO of the startup agreed to pay me 20% more than the non- startup. I politely explained the CTO that I could work for 12 hours every day, however, I would need a 70% more pay than the non-startup's offer, so that I feel I am being compensated commensurate to the time and effort I spend on the company. The CTO, now visibly upset, said, "If you like working 7 hours every day, please do join that other company!" while completing missing the point that I am okay to work for 7 hours or 12 hours as long as the pay is commensurate. Needless to say, I joined the other company for lesser pay and lesser working hours, and I am quite happy about the decision because it leaves with me a lot of time in the evenings to pursue my hobbies and also pick up new fun technologies to learn.

<comment><USER49> I think it was pretty nice of you to use a linear scale. If a company wanted that much more of my time I would have quoted them a premium (just as an hourly employee might make overtime in such a situation).

<comment><USER50> Exactly. The utility of income is not linear. The first 10,000 dollars are much more useful than going from 190,000 to 200,000 a year. That's why the tax system has a progressive structure. If you want me to work 10% more then you need to pay me more than 10% more.

And that does not even consider that my free time is very limited. If I have say 2h of completely free time everyday (24 hours minus work, commute, chores, gym, household crap, etc) and I need to work just 1 hour per day longer, then that cuts my free time per day in half. That is HUGE. You better pay me a lot for that.

<comment><USER51> What the tax system doesn't respect is that my life _might_ be better and more productive if I spent a year earning 200,000 and then took a year off, then if I worked a 100,000/year job the entire time.

<comment><USER52> You can get around this by taking the year off from July through June (I've done September through March it worked well for me and my taxes) :)

<comment><USER51> I thought someone would say that :), that's a combinatorial optimization that happens to work in the exact scenario I described, and assuming no further constraints.

But you lose out if you instead want to do 2 years on 2 years off, or if your one-year employer doesn't want you to work July to June, or if your year-off plans don't work July to June, or any number of more complicated scenarios. My point stands that the system isn't _designed_ to support this kind of irregular high-income work.

</comment></comment></comment></comment></comment></comment></comment><comment><USER53> What part of the employment agreement guarantees you "meaningfulness"? I know the parts that guarantee my pay & benefits.</comment>


How multi-disk failures happen - <AUTHOR> http://sysadmin1138.net/mt/blog/2012/12/how-multi-disk-failures-happen.shtml <comment><USER2> My favorite raid 5 failure mode is when a old hardware card fails and you have no stock spare and the maint contract was not renewed years ago and the cost of a new card is too much to be expensed and buying as a capital replacement will take a week or two minimum to be approved. And/or the card has been discontinued so you have to buy a used one from a shady foreign surplus dealer. I've seen too much of this type of thing... I can tolerate the (minimal) cost of software raid but I can't survive the possible downtime of hardware raid, so its been software raid for me for pretty much the last decade.

Another fun one was the quad redundant power supply with all four plugs going into the same power strip.

And there was the power supply that blew out every drive in the box simultaneously. I suppose its not any worse than a lightning strike, other than the onsite tech assumed it was a cooling failure, so he replaced the dusty fans and all the drives, thus destroying an entire set of drives upon powerup (and fans, I would guess).

The poison drive tray where every slot you jammed it into, it bent the backplane pins. That turned into a huge expensive disaster.

<comment><USER3> You don't need to replace old/discontinued RAID card with the used one (assuming you mean same make and model). The new card from same manufacturer will work. The RAID card writes DCBs on disks that are read by the new RAID card upon replacement.

<comment><USER4> "The RAID card writes DCBs on disks that are read by the new RAID card upon replacement."

Which also makes for very nice RAID failures, like this one that has happened to me on an HP controller:

A drive fails because of some SCSI electronics problem and when you replace it, the controller gives it a different SCSI ID. Now, the controller maps RAID arrays to drives, and it is now impossible to add the replacement drive to the degraded array because SCSI IDs in these controllers aren't user defined and the controller doesn't allow the degraded array to be modified.

And since the controller has now happily written it's configuration onto the drives, it doesn't matter that you shuffle the drives around to try to force the controller into giving up it's internal configuration.

Oh, and the controller is an onboard controller, so you can't just replace it with another one (which would also read the configuration on disk and put himself in the same stupid state, I suppose).

</comment></comment></comment><comment><USER5> Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is _obsolete_ , period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and all will be well.

I'd add that current "enhanced format" drives are tremendously better than most older drives. If your drive is a 1 or 2 TB old (512B sectors) drive, use it only for backup or whatever menial use for unimportant data.

<comment><USER6> Yes. The decision to use RAID5 instead of RAIDs 1, 10 or 0+1 shows a decision to cut costs. With mirroring, the mirror drives would take over when a disk failed.

Also, only one hot spare is in the set. Another cost saving.

Yet another decision - RAID5 across 7 disks plus a hot spare. Instead of say across 6 disks or 5 disks. You have two more chances for a disk to go bust and will have to be rebuilt from parity.

What if the disks are OK but the server host adapter card gets fried? Or the cable between the server and array? Some disk arrays allow for redundant access to the array, and some OS's can handle that failover.

Before I read the article, I thought it might discuss heat. Excessive heat is usually the cause when disk arrays start melting down one after another. Usually the meltdown happens in an on-site server closet/room which was never properly set up for having servers running 24/7. Usually the straw which breaks the back is a combination of more equipment added, and hot summer days. Then portable ACs are purchased to temporarily mitigate things, but if their condensation water deposits are not regularly dumped, they stop helping. This situation occurs more than you would imagine, luckily I have not been the one who had to deal with this for every time I have seen it (although sometimes I have). Usually the servers are non-production ones which don't make the cut to go into the data center.

The heat problem happens in data centers as well, believe it or not. A cheap thermometer is worth buying if you sense too much heat around your servers. Usually the heat problem is less bad, but the general data center temperature is a few degrees higher than what it should be, and this leads to more equipment failure.

<comment><USER7> Hard drives are pretty resilient to high temperatures. Google did a reliability analysis of thousands of hard drives and found:

"Overall our experiments can confirm previously reported temperature effects only for the high end of our temperature range and especially for older drives. In the lower and middle temperature ranges, higher temperatures are not associated with higher failure rates. This is a fairly surprising result, which could indicate that datacenter or server designers have more freedom than previously thought when setting operating temperatures for equipment that contains disk drives. We can conclude that at moderate temperature ranges it is likely that there are other effects which affect failure rates much more strongly than temperatures do."

<http://research.google.com/archive/disk_failures.pdf>

<comment><USER8> I'd love to see the same research for SSDs.

Also, parent post does talk about higher end temps, not middle range temps.

</comment></comment></comment></comment><comment><USER9> I'm going to be blunt for a moment. If you are not using ZFS, you deserve what you get.

As the author realizes, hardware RAID, or naive software RAID, is becoming more and more useless given the size of volumes and the bit densities (and thus error rates) of those drives.

The only solution to this is a proper file system and volume manager that can proactively discover bit rot and give you time to do something about it. At the moment, the only real solution is ZFS.

<comment><USER10> ZFS is beautiful and wonderful, but my perhaps outdated understanding is that the Linux port isn't as stable or mature as it is elsewhere. Is that not true? Or is it that the advantages of ZFS are so great that people should switch to illumos or some BSD?

<comment><USER11> > _the Linux port isn't as stable or mature as it is elsewhere._

Which one: the FUSE one, or the native ZFS on Linux under CDDL?

<comment><USER12> I haven't used the FUSE one extensively but the native one kept leaking memory until the system was exhausted. (Specifically it did this whenever I messed with a large number of files, I'm sure it works fine for media storage as StavrosK says.)

</comment></comment></comment></comment><comment><USER13> If you follow the advice in this paper[1] you will be measuring media errors in your drives. That means re-reading all data every N days, even archived data. Without periodically re-reading and validating (checksumming) the data you can't tell if it has rotted in place. Since the distribution of errors over drives is very exponential you should then pro-actively remove the worst drives in your system. That will avoid an accumulation of errors and sudden multiple drive failure as described here.

Durability is like a diamond: it is forever.

1\. <http://research.google.com/pubs/pub32774.html>

<comment><USER14> Diamonds burn just as well as any other carbon does, once you get them hot enough.

<comment><USER15> De Beers probably like this usage of diamonds.

</comment></comment></comment><comment><USER16> I have had a multi-disk failure occur with a RAID-1 setup. Server was pre- built from a large vendor and worked fine until both disks failed at the exact same time (within minutes).

Took the disks out to find that they had sequential serial numbers.

Called vendor for replacement only to have them tell me that they had issues with that batch, yet did not make any attempt to inform me.

Spent the day restoring from tape backup.

TLDR: If you buy a pre-built server check that the disks aren't all from the same batch.

<comment><USER17> This is a problem even if you don't buy pre-built. You're going to be buying similarly specced drives at similar times and you're probably buying from vendors from the same rough geographical area so chances are you're buying drives from the same batch anyway.

It used to be worse: all the drives in a RAID setup had to have the _exact_ same specifications or the thing wouldn't work, which pretty much guaranteed near simultaneous failure of multiple drives, but even today, with somewhat more flexible software raid setups, it's still a problem.

At a place I used to work we used to joke that a drive failure warning from a RAID controller was nothing more than a signal to get out the backup tapes and start building a new server.

</comment></comment><comment><USER4> One thing I've learnt early on my career as a sysadmin is that disk quality is _very_ important, and so is the quality of the RAID controller or software RAID subsystem. After you have a multiple drive failure on a supposedly safe RAID-1, and get forced into stitching it back into operation with a combination of "badblocks" and "dd", you'll quickly understand why...

A good RAID controller won't let a drive with bad sectors continue to operate silently in an array. Once an unreadable sector is detected, the drive is failed immediately, period.

The problem is in the detection, but good RAID controllers "scrub" the whole array periodically. If they don't, or if you are paranoid like me, the same can be accomplished by having "smartd" initiate a long "SMART" self-test on the drives every week.

Good controllers will even fail drives when a transient error happens, one which triggers a bad block reallocation by the drive, for example. This is what makes some people fix failed drives by taking them out and putting them back in. After a rebuild the drive will operate normally without any errors, but you are putting yourself at a serious risk of it failing during a rebuild if another drive happens to fail, so DON'T do this.

Some others will react differently to these transient errors. EMC arrays, for instance, will copy a suspicious drive to the hot-spare and call home for a replacement. This is much faster than a full rebuild, but also much safer because it doesn't increase the risk of a second drive failing while doing it.

Oh, and did I mention that cheap drives lie?

Avoid using desktop drives on production servers for important data, even in a RAID configuration, if you don't have some kind of replicated storage layer above your RAID layer (meaning you can afford recovering one node from backup for speed and resync with the master to make it current).

<comment><USER18> Your advice is ok for someone who is willing to take no risks and to spend the money on that. It is not strictly correct for all situations. In fact storage arrays are not likely to drop a disk on the first medium error since medium errors are a fact of life and do not necessarily indicate a bad disk. Ofcourse, given that there is a medium error it warrants a long term inspection to make sure that the medium errors are not consistent and come too often on a specific drive, that is a cause of concern but a single medium error is of no real significance.

I also found that higher-end drives lie, I used SAS Nearline drives that failed easy and often and I used standard SATA drives that were more resilient. It depends on the vendor and make. May also depend on the batch but I never found a proof for that in my work.

<comment><USER4> Maybe I was wrong in using the term "transient error"...

A bad block reallocation can be seen as a transient error from the controller's perspective, but it isn't silent provided the drive doesn't lie about it (and one would expect that a particular storage system vendor doesn't choose - and brand - drives that lie to their own controllers).

The storage system may ignore medium errors that force a repeated read (below a certain threshold), but they shouldn't ignore a medium error where the bad sector reallocation count increases afterwards (which is just another medium error threshold being hit, this time by the drive itself).

I'm not saying that higher-end drives are more reliable or not. Given that most standard SATA errors go undetected for longer, one could even argue that higher-end drives seem to fail much more frequently... I've had more FC drives replaced in a single EMC storage array than in the rest of the servers (which have a mix of internal 2.5in SAS and older 3.5in SCSI320 drives), and we certainly replace more drives in servers than desktops.

But that's another topic entirely.

</comment></comment></comment><comment><USER19> Having worked at NetApp for a few years at the turn of the century this is "old" news :-) But it is always important to internalize this stuff. M3 (or 3x mirrors) is still computationally the most efficient (no compute just I/O to three drives), R6 is the most space efficient at the expense of some CPU. Erasure codes are great for read only data sets (they can be relatively cheap and achieve good bandwidth) but they suffer a fairly high I/O burden during write (n data + m code blocks for one read-modify-write).

Bottom line is that reliably storing data is more complicated than just writing it on to a disk.

</comment><comment><USER20> The author is wrong that "enterprise quality disks just plain last longer". CMU did a study on this topic on a population of 100 thousand drives, and found that enterprise-grade drives do not seem to be more reliable than consumer-grade drives. See the conclusion in: <http://www.cs.cmu.edu/~bianca/fast07.pdf> This legend must die.

The author is also wrong when saying "a non-recoverable read error [is] a function of disk electronics not a bad block". An NRE can happen for different reasons, one of them is when (data and error-correction) bits in the block _get corrupted_ in a way that prevent the error-correction logic from detecting this error. So the block is technically bad, just not bad enough to cause the drive logic to declare it as a read failure.

<comment><USER3> The CMU study is most probably flawed as it looked at hardware replacement records and didn't take into consideration the different usage and threshold for replacement between enterprise and consumer drives. Most enterprise drives are used in enterprise servers and storage systems that monitor drive errors closely using SMART. The threshold for drive errors is much lower with such systems and drives are replaced quickly. My company (storage system vendor) replaces disk even when the SMART alerts impending failure and doesn't wait for actual failure. This will come across in hardware replacement log as more frequent replacement of enterprise disks. The consumer disks are used in consumer systems. The consumers don't proactively replace disks, they wait until disk actually fails. This will show up in hardware replacement log as less frequent replacement of consumer disk.

I have actually used 'defective' enterprise disks in consumer systems for years after they were labeled defective by storage system vendors. About a decade ago, I used to buy such defective enterprise disks in bulk at auction from server and storage manufacturers and sold them as refurbished disks to consumers after testing.

<comment><USER20> I fail to see your point about the threshold of replacement. Assuming that enterprise-class drives get replaced sooner because sysadmins monitor SMART, it is still widely acknowledged that SMART errors are a strong indicated that the drive _will fail_ soon. For example the Google study on drive reliability showed this correlation on consumer-class drives [1] There is no reason to believe this correlation doesn't exist with enterprise-class drives (or else, what would be the point of SMART?). Therefore the replacement threshold is mostly irrelevant as the enterprise drive replaced due to SMART would have failed soon anyway.

I really don't understand this skewed perception of consumer- vs enterprise- grade harddrives. Do you believe that enterprise CPUs are more reliable than consumer CPUs? How about enterprise NICs vs consumer NICs?

Consumer-grade drives are sold in volumes so much larger than enterprise-grade drives, that vendors have strong incentives to make them as reliable as possible. I would even say they have incentives to make them more reliable than enterprise-grade drives. Because a single percentage point improvement in their reliability will drastically reduce the costs associated to warranty claims and repairs.

My own experience confirms the CMU study. I have worked at 2 companies selling each about 2-5 thousand drives as part of appliances, to customers across the world. One company was using SCSI drives, the other IDE/SATA. And the replacement rates were similar.

I can see your point about the usage being different which could invalidate the CMU findings about consumer vs enterprise drive reliability. But I don't personally believe it explains it. The CMU study + my annecdotal evidence one 2-5 thousand drives + the fact that no study has ever _showed data_ suggesting enterprise drives are more reliable, makes me think that they are not.

[1] [http://static.googleusercontent.com/external_content/untrust...](http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/disk_failures.pdf)

</comment></comment></comment><comment><USER21> I have been doing raid data recovery for many years. A very common scenario is "Two Drives Failed at Once". This is usually not the case. What usually happens is 1 drive fails. The raid then goes into degraded mode and continues to function. Nobody notices the warnings. Some time later... months or years even, a second drive fails, the raid goes down - now they notice. They call in the techs who declare 'two drives failed'. This is when your data is most at risk. People start swapping boards and drives, repower the system, rebuild drives, rebuild parity, force stale drives online etc. I have seen alot of raids that would have been recoverable had they done the proper steps. Then they hand it over and say they found it this way... didn't touch a thing... www.alandata.com

<comment><USER22> Would have upvoted if not for the web link at the end of the comment.

</comment></comment><comment><USER23> Let us not forget the little considered other cause which usually takes out your tape infrastructure as well: fire.

One outfit I worked decided to stick a brand new APC UPS in the bottom of the rack as it was in their office. It promptly caught fire and burned the entire rack out. The fire protection system did fuck all as well other than scare the shit out of the staff. Scraping molten cat5 cables off with a paint scraper was not fun.

Fortunately it was all covered by the DR procedure. Tip: write one and test it. That's more important than anything.

<comment><USER24> Do you know what caused the fire? Improper installation? Overheating? Defective equipment?

<comment><USER23> Defective APC charge regulator apparently. To be honest they were good and their insurance paid out very quickly to both us and the site owners.

</comment></comment></comment><comment><USER25> And this is why, everywhere I've ever worked, I've had to say RAID is _NOT_ backup. There are varying degrees of receptiveness to this, because actual backups are a giant pain in the ass, and have a very annoying lifecycle.

<comment><USER8> There are better reasons for RAID not being back than this obscure rare fault condition.

</comment></comment><comment><USER26> Another way to think about this problem is to turn it on its head and assume that hard drives _will_ fail and make it a strength and not a weakness. Something like OpenStack Storage[1] is built around the idea that hard drives are transient and replacing them should be painless. In fact, the more drives you have the less problems you have.

Basically, you keep multiple copies of the same data across different clusters of hardware. If a drive or two (or ten) go bad, just replace them, there is no rebuild time. Sure, it costs some disk space to keep n copies of data, but drives are just getting cheaper and there are de-duplication schemes being developed to help with this. Its not like RAID-6 is super efficient either.

Just my two cents...

[1] <http://www.openstack.org/software/openstack-storage/>

</comment><comment><USER27> isn't this what raid scrubbing is for? [http://en.gentoo- wiki.com/wiki/Software_RAID_Install#Data_Sc...](http://en.gentoo- wiki.com/wiki/Software_RAID_Install#Data_Scrubbing)

        for raid in /sys/block/md*/md/sync_action; do
            echo "check" >> ${raid}

does that fix the issue? i run that once a week. i thought i was ok. am i not? if i am, isn't this old news?

<comment><USER28> yes, you should be fine. AFAIK most distros set up cronjobs for this automatically (for example, Debian/Ubuntu runs that once a month).

</comment></comment><comment><USER29> I am going to ask a honestly stupid question. What is going to happen to ZFS? Sorry if this is Slightly off topic. Although the comments has already started discussing on it.

The OpenSource version of it, or the BSD version of it is only up to v28. And it seems after that Oracle is no longer putting out update as open source and what will happen after that? Disparity between Oracle version and BSD version? And are features still being developed? Most of limitation listed in Wiki hasn't change at all for the past years and are still listed as under development.

</comment><comment><USER30> R.I.N.A.B

Raid Is Not A Backup!

Oh, and your raid controller should monitor for smart errors and you should seek to replace disks when you start seeing sector rewrites.

</comment><comment><USER31> Not very efficient but I try to avoid placing all volumes on a single raid set.

</comment><comment><USER32> I'm not a Un _x sysadmin at all and don't know much about hard drive: I'm just a software dev.

But from the beginning of TFA, after reading this:

"Bad blocks. Two of them. However, as the blocks aren't anywhere near the active volumes they go largely undetected."

The _FIRST* thing that came to my mind was: "What!? Isn't that a long-solved problem!? Aren't disks / controllers / RAID setups much better now at detecting such problem right away".

I've got a huge issue with the "largely undetected". I may, at one point, need storage for a gig I'm working on. And I certainly don't want stuff problems like that to go "largely undetected".

So quickly skipping most of the article and going to the comments:

"It's worth pointing out that many hardware RAID controller support a periodic "scrubbing" operation ("Patrol Read" on Dell PERC controllers, "background scrub" on HP Smart Array controllers), and some software RAID implementations can do something similar (the "check" functionality in Linux md-style software RAID, for example). Running these kinds of tasks periodically will help maintain the "health" of your RAID arrays by forcing disks to perform block- level relocations for media defects and to accurately report uncorrectable errors up to the RAID controller or software RAID in a timely fashion."

To which the author of TFA himself replies:

"Yes, that is something I should have made clearer. This is the very reason that RAID systems have background processes that scan all the blocks."

Which leaves me all a bit confused about TFA, despite all the shiny graphs.

Basically, I don't really understand the premises of "bad blocks going largely undetected" in 2013...

<comment><USER33> I have a home server with three disks and ZFS, for my photos and things, so I'm not an expert. However, Ubuntu's md-raid includes scrubbing once a week by default, and I added scrubbing to my ZFS setup via crontab, again once a week (I'm not sure if ZFS does it automatically, but I don't think it does. I would appreciate a correction, if someone knows for sure).

The article assumes no scrubbing, which is a stupid thing to run without, as detailed from the article. So it's basically "why pointing a gun at your foot and pulling the trigger is bad", "because you're going to shoot yourself in the foot".

<comment><USER34> You may add something like this to /etc/periodic.conf:

daily_status_zfs_enable="YES" daily_scrub_zfs_enable="YES" daily_scrub_zfs_default_threshold="6" # in days

and it will scrub the pools every 6 days (and send you a report in the daily run output).

<comment><USER33> Very nice, thank you, I will try that. I am rather dismayed, however, by learning in this thread that my disks have 4K sector sizes and ZFS autodetected 512 bytes, which means I'll have to destroy the pool and recreate it...

<comment><USER34> It happens all the time...

If you run "camcontrol identify ada0" (or whatever your device is) you can find out before it is too late:

sector size logical 512, physical 512, offset 0

This is from a lucky drive of course :)

<comment><USER33> Hmm, there's no such command in Ubuntu, maybe it's from BSD?

<comment><USER34> camcontrol is from FreeBSD.

I don't have a Linux box available right now but maybe "hdparm -I" does something similar: "request identification info directly from the drive".

<comment><USER33> Yep, that works:

        Logical  Sector size:                   512 bytes
        Physical Sector size:                  4096 bytes

I'm guessing that's not very nice. My ZFS pool was created with ashift 9 (this is 2^9=512 bytes), when it should be 12 (2^12=4096). I will have to copy everything off and back on again.

For everyone who wants to check, and because I couldn't find info on it, run:

        zdb | grep ashift

And see if it's 9 or 12.</comment></comment></comment></comment></comment></comment></comment></comment>


Show HN: How to make your Python code more idiomatic – 25 tips and tricks - jerry-hn https://github.com/jerry-git/learn-python3#idiomatic-python <comment><AUTHOR> Good tips, but all of these really seem to be for Python beginners (even the ones labelled "intermediate"). Not sure too many Python programmers who use HN would learn anything new here.

</comment><comment><USER2> This is missing an extremely difficult to grasp thing - namespaces and binding. [https://docs.python.org/3/reference/executionmodel.html#nami...](https://docs.python.org/3/reference/executionmodel.html#naming- and-binding)

      #example:
      scalers = {}
      for k in range(1,55):
        scalers[k] = lambda x: x*k
      print( scaler[2](3))
      #Above does not work, or worse, sometimes it may work..
      #eo-example

And without it, it's all just another procedural indents-instead-of-brackets thingie

<comment><USER3> Just doublechecking to make sure I get it, the solution there is to replace the lambda with something like this right?:

        lambda x, k=k: x*k</comment></comment>


Everything You Know About Lawyer Selection Is Wrong - <AUTHOR> http://premonition.ai/everything-you-know-about-lawyer-selection-is-wrong-big-data-analyzes-litigation/ <comment><USER2> This website's formatting is absolutely horrid (seems to be a company blog for Premonition), but the story is surprisingly interesting.

However, isn't this how people have been selecting lawyers for years? See what their win/loss ratio is, and see if there is any reason to discount a favorable ratio.

Doesn't really seem like a big data problem to me.</comment>


Ask HN: Recommended journey on becoming a distributed systems practitioner? - <AUTHOR> I've been programming professionally for ~3 years now and I started out knowing absolutely nothing. It's been a fun journey, and I've learned a great deal about straight-forward application development (NodeJS/Spring). Lately though, I've been really fascinated by distributed systems architectures, why they're built in a particular way and what they're hoping to achieve. I find myself hopping hopelessly between blogposts by notable practitioners[1][2][3], early academic literature[4] and conference videos[5].<p>Let me be honest: I'm drowning in a deluge of information, but I'm convinced that these are the kinds of interesting problems I want to work on moving forward. I'm caught between trying too hard to focus on the theory/literature (which is a time-consuming endeavor) vs duct-taping systems that I may not fully understand the tradeoffs of (which feels haphazard). Is there a middle ground? Is there a sane way for an app-developer-by-day like myself to work out a solid understanding of these concepts so that I can build these systems with a degree of confidence? A colleague recently recommended that building out my own simple distributed key value store would force me to learn a lot of things in the process (consistent hashing, leader election, MQs, Merkel Trees, Vector clocks). Does HN have any such recommendations and motivating projects to kick things off? It could advice to build something particular or some repo that I should study intently to glean insights from.<p>[1] https://aphyr.com/posts/291-call-me-maybe-zookeeper [2] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying [3] http://book.mixu.net/distsys/single-page.html [4] http://research.microsoft.com/en-us/um/people/lamport/pubs/pubs.html [5] https://www.youtube.com/watch?v=BKqgGpAOv1w <comment><USER2> Get a job at a big tech company (Google, Facebook, Microsoft, Yahoo, DropBox, Uber, etc.). Beyond a certain scale, every problem is distributed, and you'll be dealing with distributed systems and their pitfalls as a routine part of your job.

It's very difficult to tell what is of practical significance vs. what is just theoretically interesting without directly working with a real distributed system storing real data for real users. For example, master election is something you rarely face in the day-to-day: you build a system once (or just use Zookeeper) and then re-use it for all your systems. Merkel trees and vector clocks are fairly special-purpose - once in a while you'll find a system where they're critical, but a lot of the time you don't need them. Versioning problems and choices of ID schemas, however, come up _all the time_. Bloom filters are far more useful than the amount of theoretical or blogosphere attention given to them would indicate, sharding functions are important to know, and concurrency & buffering are critical skills.

<comment><AUTHOR> My personal experience at at large tech company is that there are usually designated teams thinking about these infrastructural problems so that the rest of the engineers don't have to worry about them. So as someone managing a "service" in a company that has a SOA model, sure, I'm technically a part of a distributed system but the deliberate abstractions minimize the "necessary" exposure to dealing with distributed systems.

I'm in complete agreement with your thoughts on stuff with practical significance vs theoretically interesting topics (I'm far more interested in building some expertise around practically significant things _first_.) Are there any particular projects you'd recommend I follow to grow my knowledge in this space? Alternatively, would you have any recommendations on specific side-projects that could really motivate a good understanding of these concepts?</comment></comment>