Fast Way to Load Data into PostgreSQL Using Python - <AUTHOR> https://hakibenita.com/fast-load-data-python-postgresql <comment><USER2> As a former DBA and closet python super fan I found this post to follow all the things I would have done. Kudos to the op.

Loading to a temp non-logged table is the right way to go and using built in bulk loading options is smart, too. The type annotations looked really good and kudos for using modern python here (although for the examples I'm not sure they were needed, but being that they could have been used in a larger context that this article was just a microcosm if I could be wrong). Very good article. I've bookmarked the blog to consume other such articles.

<comment><USER3> What about translating data on the fly to the one big copy statement ?

<comment><USER2> I could see that working. Just depends on your use case and what you want to accomplish.

The approach the author takes has the advantage that when data is in the load table it can then be used to load into a real table via some transform in batches and be done in the background.

</comment></comment></comment><comment><USER4> I haven't done any benchmarking, but for around 200,000 rows of data, which I am handling daily at work, the following line of python code takes around 10 seconds.

pandas.read_json(json_obj).to_sql()

Any transformation of the data before writing into DB is split second with vectorized pandas operation.

Pandas is still my first choice when it comes to tasks like this.

</comment><comment><USER5> Not sure what the python code translates to, so maybe this is covered, but in my experience, if you can't use COPY, you should:

1 - Generate a static SQL to insert N rows (say 10 to 50): insert into blah (id, name) values ($1, $2), ($3, $4),

2 - Loop through the data in batch size

3 - Generate a dynamic sql for M rows where M is whatever's left over

Wrap it in a transaction. You can have an SQL builder where you give it the list of columns and how many rows you want to generate and use that for both steps 1 and 3. You can prepare statement 1 (and keep reusing it), but I'mnot sure how much that'll buy you.

If you need to deal with conflicts. Inserting the above into a temp table (on commit drop) and then doing an insert + update from the team into the real table works well.

Then there's a handful of postgresql configs to tweak, but if you're willing to risk a bit of data loss (but not corruption), asynchronous_commit=off.

</comment><comment><USER6> I am looking for ways to quickly load data into a PG database for local testing. An initial seed run is ok, but I want test suites to run on identical data sets and since PG doesn't (yet) support nested transactions, I a currently use my seed database as a TEMPLATE for suite-specific databases.

It works reasonably well but creating the database with a template takes several seconds, even though my test data is fairly simple. I imagine it should be faster than anything I can do from my programming language (nodejs in my case), or is creating with templates slow for some reason that I don't know of?

<comment><USER7> Not an expert, but I'm curious - what use case is not sufficiently solved by savepoints?

<comment><USER6> They might well be faster, I haven't tried. I just remember looking into them and thinking that it was more involved than what I had now so I decided against them.

</comment></comment></comment><comment><USER8> I concur - we load a fair amount of data into Postgres using Python and we've learned these lessons the hard way. We ended up feeding generators into execute_batch.

We don't use execute_values, because we want to generate dictionaries (since there are usually multiple generators along the way) and we don't use copy_from, because the cognitive overhead of data prep is just way too high for the benefit (for us! your mileage may vary).

Great stuff, I've already circulated it at work.

<comment><USER9> For old school CSV and Excel that requires any processing before inserting I Have found Perl with the Relevant Modules from CPAN for CSV and Excel is the best solution

</comment></comment><comment><USER10> Seems like a pretty good overview, but I'm not surprised at all that in the end it's COPY TO/FROM that won out by a good margin.

Listing all the columns in order seems a bit superfluous and error prone, though. I'd recommend creating the list of columns to be inserted (or copied; copy_from has a 'columns' arg) and slicing the input hash with the same list (dict comprehension in Py).

</comment><comment><USER11> Very nice article! Explores all the tricks I know of, and a few more. Never heard of the execute_values before, only a bit (3x in the example) slower than using copy but much simpler code.

</comment><comment><USER12> Why do you drop and recreate an unlogged table? Why not create a temporary table, which is also not WAL-logged and automatically gets cleaned up? It looks like you're not closing the connection. The only advantage is that the staging table remains in place until the next execution, but presumably so does the data file.

Alternately, if you have an unlogged table already created, why not simply truncate the table if it exists instead of recreating it on every execution? Drop and truncate should take about the same amount of time, but creating a table is going to be slightly more expensive. If you're worried about making sure the db table schema matches the file format, I don't think I would be. File formats change, you're already going to be manually working on it. You can drop the table so the updated script recreates it.

I suppose it doesn't really matter, but I'm wondering if your choice was one of intent or one of convenience.

</comment><comment><USER13> I followed a similar path with TEMPORARY tables and using pgcopy [1] to insert data quickly - the copy operation was extremely quick and didn't require only passing strings.

[1] [https://github.com/altaurog/pgcopy](https://github.com/altaurog/pgcopy)

</comment><comment><USER14> You can go faster than this. (Mind you, the following suggestions will only matter if the DB is still your real bottleneck. Once you're using COPY, you start to have to optimize the rest of your ETL pipeline as well.)

I've done all of the following (except the last bit about pg_bulkload) in my own ETL stack. They work!

——

1\. COPY supports a binary format in place of CSV. It's proprietary (it's not even exposed in libpq, only statically linked into the postmaster, pgdump and pgrestore binaries!) but also pretty easy to implement. Here's a python package implementing it: [https://pypi.org/project/pgcopy/](https://pypi.org/project/pgcopy/). As long as you've got the rows in memory at some point, just generate that instead of CSV. The rest of the architecture is the same.

2\. Run concurrent SQL COPY commands on separate connections for each [staging] table you're populating. This allows PG to populate the tables in parallel.

3\. Create (or TRUNCATE) the staging tables in the same transaction that contains the COPY. This way, the COPY won't need to operate on a differential MVCC state, but rather can operate on the "raw" table. (And, if this is the "base" data load for a given table, rather than an incremental one against an already-populated table, then you don't need a staging table; the target table itself is now your staging table. Just ensure that any CREATE INDEX or CREATE PRIMARY KEY or ADD CONSTRAINT happens _after_ you populate the table with data. No need for a TEMPORARY/unlogged intermediate; the performance would actually be worse if you added one.)

4\. Pre-bake your .sql, embedding the `FORMAT binary` data, _then_ stream it once you're done. I can't tell you how many times the problem with an ETL pipeline wasn't PG's ingestion speed, but rather the speed at which the ETL transformer process was writing to the socket. The rest of your pipeline can be asynchronous, but actually streaming into your PG instance takes up valuable shared resources on the instance. Assuming that the instance being ETLed to isn't just a replication master, and actually receives read traffic, you'll want writes to that instance to complete as quickly as possible—even at the expense of making the rest of your ETL pipeline more complicated. So pre- bake your .sql files, and then stream them. (One advantage, once you do this: you don't need a runtime with an SQL client library any more. You can just spawn instances of `psql foo.sql`. These processes don't even have to coincide with your transform stage any more! Push your `.sql` files to object storage, and then run `psql` on a dedicated loader VM—or from an object-storage- lifecycle-event-triggered Lambda function, if you like.)

Fun fact: when you `pgdump --format=directory`, pgrestore(1) will do techniques #1 - #4 for you automatically.

Key insight: the pgdump(1) and pgrestore(1) pair of binaries are optimized to heck and back. If pgdump(1) or pgrestore(1) has some special way of doing something, it's probably for performance; look into doing that thing in your pipeline. (Even if there's no library for your runtime to allow you to do that thing. Write one!)

——

You can go faster than even pgdump+pgrestore will "naively" do, while still being compatible with "standard Postgres", if you're willing to mangle your data architecture a bit. Specifically, you can take advantage of Postgres's table partitioning support. (Yes, now you have to deal with partitions. But hey, you probably already had to at the data sizes we're talking about.)

5\. Turn your incremental loads into table foo (via fresh staging tables) into base loads of fresh partitions of table foo. Each load creates its own partition. Techniques #1 and #2 will accelerate and parallelize the loading of sibling partitions of a parent table, just as well as they'll accelerate the loading of separate tables.

Once you do this, you don't need _any_ staging tables, because every load is a "base" load of a table. (Yes, you'll have a lot of tables. You'll need some agent sitting around to do async rollups using `CREATE TABLE AS ...` to consolidate these. If you have access to the TimescaleDB extension, you might be able to trick it into doing this for you automatically.)

#6 (or #3b). Include your CREATE INDEX statements in the same transaction that CREATEs and COPYies into the partitions (after the data is populated, though.) One of the worst thing for the production performance of a data warehouse is contention between application reads and an index rebuild. If you always create new partitions, and only ever index them when you're creating them, then your indices will be created against tables that aren't visible to new clients yet, and so clients and the indexer will never get in one-another's way.

——

If, after all this, you're looking for the absolute fastest way to load data into Postgres, _and_ you have control over your PG instance (i.e. it's not a hosted DBaaS), you could try [https://pgxn.org/dist/pg_bulkload/](https://pgxn.org/dist/pg_bulkload/). It's a PG extension that serves as a faster replacement for the COPY command, by bypassing most of the buffers and locks related to PG's individual request- handling. It shouldn't cause any problems with MVCC, but only because it assumes/requires that you're also doing all of the above, so you're never trying to load into a table users can "see." (I haven't tried it personally, but if PG's ingest speed becomes my bottleneck again, I just might.)

</comment><comment><USER15> Somewhat related: how does one insert a Pandas DataFrame into a Postgres table, in which most fields are merely integers being foreign keys of other tables? The DataFrame itself contains the real values (e.g. Cities, Countries etc.).

<comment><USER16> i use write_sql but in this case you would have to resolve the object values to their integer primary keys.

</comment></comment><comment><USER17> Very nice and educational read ! one note though, it's usually a good practice in performance testing to repeat your measurements more than once, it will mitigate local problems or at least point to problems in the measurement.

</comment><comment><USER18> Would also like this with asyncpg. psycopg2 is kind of obsolete today.

<comment><USER19> Bold claim, since psycopg2 is used overwhelmingly more than any other library. Though asyncpg (or something like it) is likely the future, it's a long way to go before calling psycopg obsolete.

<comment><USER20> Asyncpg is not future, it is present. We use it in production and quite happy with it.

<comment><USER12> That still doesn't really challenge parent's claims.

</comment></comment></comment></comment><comment><USER21> There's also copy_expert in psycopg2. I believe csv with copy_expert should be the fastest but have not tested in as much detail as op. Great work on the article.

<comment><USER10> IIRC copy_expert just lets you enter the postgres COPY command that's used to start the transfer manually, which by itself doesn't change anything.

It does allow you to use the BINARY transfer format, though, and that's quite often the fastest version, at the cost of a more complicated data transformation step.

<comment><USER21> Plus freeze for use cases like the example in the post.</comment></comment></comment>


Waymo obtains California's first fully driverless car permit - <AUTHOR> https://venturebeat.com/2018/10/30/waymo-becomes-the-first-company-to-obtain-a-fully-driverless-car-permit-in-california/ <comment><USER2> This is great news. Waymo chose to expand to Mountain View for their second location because their HQ is in that location then I presume?</comment>


Sucks to be an old engineer - <AUTHOR> http://lionoftheblogosphere.wordpress.com/2013/05/25/sucks-to-be-an-old-engineer/ <comment><USER3> It doesn't suck to be an old engineer. It sucks to be an engineer with outdated skills.

I learned FORTRAN in 1968. I learned Scala in 2012.

I'll be 61 this week...and I just started a new job developing in Scala.

<comment><USER4> One of the sharpest developers I have ever worked with -- was 63. He did only mobile development and only would agree to work 30 hours a week. When I pressed him on it, he explained that he liked to constantly be working on both Android and iOS projects, 30 hours a week each, and that kept his backlog of waiting clients happy.

EDIT: He completed the iOS application under budget with wonderful tests.

<comment><USER5> I really wonder at what rates one might work at these ages.

<comment><USER6> At that age the rates are irrelevant compared to the health insurance compensation. Most people at that age would be willing to take a pay cut in exchange for generous health benefits and flexible hours.

<comment><USER7> I must have missed the part where MetaCosm stated his developer was buying health insurance in an age-discriminating market.

<comment><USER4> He was in the US (Arizona if I remember correctly), and had been incorporated for a number of years... but I have no idea about his health care situation... the joy of contractors, I don't have to care. I give him $180 an hour for X hours, he gives me a wonderful product, everyone goes home happy.

<comment><USER8> > the joy of contractors, I don't have to care.

Not really. If the contractor gets sick, it's still your problem.

I care for my contractors and always make sure they are paid enough to afford good medical care and a stable financial situation. It makes sense, even from a deeply selfish point of view - the less time they spend worrying about such things, the more they spend working.

<comment><USER4> First of all, I don't think it is my responsibility to ensure contractors are paid X, that is their responsibility, they set the rate. I decide if I think their skills match their rate. I can't decide what is "good" medical care for them, nor can I decide how much it costs (maybe he has a medical condition and needs 20k a month just for insurance). I can't decide what a stable financial situation is for someone either. The arrogance of someone who thinks they can baffles me. I might think 200k is plenty, but I have no idea about what contractor X consider "plenty" and to assume is simply projecting.

As for him being my problem. If a contractor can't work, you replace them, depending on how the contract was setup, you might collect some insurance to help get over the ugly transition.

<comment><USER8> Even if it's not your responsibility, it's your problem. Replacing a contractor is not like replacing a cog in a machine.

As for deciding what's enough for a stable financial condition, I can't. What you I do, as a manager is to pay attention for warning signs.

Years ago, I had a programmer, young and very bright, who kept entering lousy freelance projects on the side. He was under great pressure because he had financial problems. I didn't want to lose him, but burnout was already on the horizon. When performance review time came, I enrolled him on a personal finance course. Within six months, the freelance jobs went away, he started coming in earlier and the already good quality of his work improved further. He had more time to study, more time for his family and more attention for his work.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> This post is touching on two issues that are somewhat related but different:

1\. Finding ANY job when you're 60 is hard;

2\. Engineers (who are still engineers) at 40+ will often be passed over in favour of twentysomethings.

Engineering is such a young industry that I'm not sure we've really faced (1) yet (since the number of people who started engineering in the 70s is but a drop in the ocean compared to the number in the workforce today). It'll be interesting to see how the industry has evolved in 30 years.

(1) is a pretty strong driver for everyone having a long term plan to control your own fate, which means working for yourself. That is, if you're not in industry with inbuilt protections (eg teaching).

(2) is harder to pin down.

On the one hand, there is a certain (un)survivor bias in that many who started as engineers in their 20s are managers or the like by their 40s. So are those who remain in engineering a good sample?

Also, the older you get in general the more non-work priorities you have. Family, for example. This can reduce the amount of time you spend on self- improvement. This industry almost requires constant learning, reinvention and skill acquisition. Is it an example of ageism if someone who is 45 can't find a job when what they know is Ada, Cobol and Forth?

It's often said that a good technical foundation and education means you can pickup any language. This is true but picking up new skills is both a habit and a skill, one that atrophies if left untapped (IMHO). It's also a question of degree. Perl to Python? Not a big gap. Ada to Ruby? Well....

That all being said, there is age discrimination in this industry. I've personally witnessed someone say "I prefer new grads so I can mold them". That's fine and all but if you say that and don't hire a 50 year old programmer that's a good basis for an age discrimination suit because, well, it is age discrimination.

So yes, you either need to find a job that doesn't change (eg a plumber or a teacher) or you need to constantly battle to maintain relevancy in a fast- changing industry or you need to become master of your own fate.

This is known.

<comment><USER10> > Is it an example of ageism if someone who is 45 can't find a job when what > they know is Ada, Cobol and Forth?

Um, I'm 54 and the languages of my 20's were C, C++, etc.

I also fairly regularly get calls from recruiters.

> So yes, you either need to find a job that doesn't change (eg a plumber or a > teacher) or you need to constantly battle to maintain relevancy in a fast- > changing industry or you need to become master of your own fate.

Or, you can be a driver of this change yourself by developing a modern, advanced programming language.

<comment><USER11> Java is nearly 20 years old.

Do people really think if you're 45 you worked in Cobol?

<comment><USER12> I work in finance. We are teaching plenty of fresh-faced graduate recruit devs COBOL.

<comment><USER8> For someone into retrocomputing (who even made a 327x-ish terminal font), being paid to learn COBOL is weirdly interesting.

<comment><USER12> Most of the org's original COBOL programmers are retirement age. So there's 60yo greybeards teaching COBOL to 20yos.

It's a weird time to be a mainframer. If you are an experienced specialist you can charge _insane_ hourly rates. At the same time, some bigger orgs have started transitioning off the old stack because all the experts keep dying or retiring.

At some point mainframe specialist skills will go from insanely valuable to completely worthless.

</comment></comment></comment></comment></comment></comment><comment><USER13> I think that this article, though terse and generalistic, addresses a valid issue. However, I agree with many posters here in saying that it's not exclusively age that hampers older engineers, but an ossified skillset. Anyone who becomes entrenched in a nichey vocation then strikes out at 50+ to find new employment is going to suffer. The tech world worships youth and forgives hubris and foolishness, so it's especially unfertile ground.

That being said, as a serial startup founder, on a few occasions I've had developers working for me in their late fifties and even sixties. I was, in fact, in my mid to late twenties at the time, and I had great relationships with these guys. They had kept their skills up and their experience enabled them to sometimes do the work of ten people.

In my first company, I had a sixteen year old whiz kid who was possibly the most gifted hacker I've ever worked with, and we later hired a portly, bearded, late-fifties nudist Unix guy who balanced the youngster out quite nicely.

I'm now working in a startup where of the eleven of us, nine of us are in our forties, including three 40-something engineers. I've never worked with a better crew. And our stack is about as cutting-edge as would be responsible in a serious startup. Our VP of engineering literally helped invent Java at Sun, but moved on to new things later on instead of getting stuck.

<comment><USER14> >portly, bearded, late-fifties nudist Unix guy

You had me until "nudist." Dare I ask?

<comment><USER10> Hopefully he works remotely from home.

<comment><USER15> The other guy is sixteen - so he better work from home!

</comment></comment></comment></comment><comment><USER16> This isn't really new, nor is it specific to software engineering. "Line engineers" have always had a very hard time with employment as they age (not that it doesn't happen). The usual ways to avoid this are (in order of risk):

1) Get your MBA and go into management; 2) Start or join a consultancy; 3) Start a company.

My buddies in more traditional engineering fields (aerospace, chemical, etc) are all in the process of setting themselves up with exit options (we're all just around 30).

<comment><USER17> Much of my family are in chemical engineering, petroleum engineering, and similar engineering fields, and from what I can tell just about the opposite is true. The companies would _much_ rather hire someone back out of retirement as a consultant at 3x their previous salary, than train up someone younger. They'll hire someone younger, but seemingly as a last resort. This seems _most_ true in a few specific fields (e.g. metallurgy specialists), but also true for more common things like process engineers.

<comment><USER18> From my interactions with the O&G industry, primarily upstream, there is growing concern that the knowledge base is too old and that they are not getting enough young professionals into the pipeline to cover for the engineers who will, in the next 10 - 15 years, simply be too old to work. It is one of the big goals of the Society of Petroleum Engineers right now, and has quite a bit of support from Halliburton and Chevron.

</comment></comment></comment><comment><USER19> So here's what the article says:

"I know two or more engineers around the age of 60 that graduated from MIT or Stanford 35 years ago. Of those two or more, at least one has lost his job, and at least one is afraid of losing his job."

Here's what the article doesn't say: what field these engineers are in, whether or not they're good at their job, what company they work(ed) for, what they work on, how up-to-date their knowledge is, etc.

Maybe there is age discrimination in the engineering field, but you certainly can't infer that from this anecdote.

</comment><comment><USER4> """When not even a Stanford or MIT engineering degree is good enough to keep an engineer employed at 60, there is genuinely no market for engineers that age. Plan accordingly."""

Wait, what I did 40 years ago won't guarantee my employment now?!? What is this insanity, I thought once I got my degree in 1973 I was done with all this "learning" nonsense. How can I be expected to just keep updating my skills?

<comment><USER20> Why does Google still care about people's undergraduate GPAs from 25 years ago? When my friend interviewed at Google (they did hire him), the recruiter wanted to know how many hours per week he worked (to pay his tuition) at university 25 years ago to account for his "low" undergraduate GPA (3.0 - 3.5).

<comment><USER21> What would happen if you didn't remember your gpa? It is shocking that they would ask something like that though.

<comment><USER22> GPA seems to be an American thing. I graduated Uni in 2007 (in Scotland) and I've never been asked mine.

<comment><USER23> No, it's a Google thing. No American company has ever asked for my GPA, but the Google HR person next door said that if I wanted to interview for a job at Google, HR would demand my GPA (from decades ago!) and would not grant an interview unless it was above some (secret) threshold. If their recruiters reached out to me, HR would still require the GPA, but she said that the interviewers might not care about it.

<comment><USER18> What stage of interview? I have had two phone screens with Google recruiters, and one actual phone interview. Nobody ever asked for my GPA, and my degree is from a university most people here have probably never heard of (and is EE, to boot).

</comment></comment></comment></comment></comment></comment><comment><USER24> The first book I co-authored on Android programming is already so obsolete it is in O'Reilly's repository of free downloads - the back list of the back list.

SICP, which, when it was published, summed up a lot of what I had learned as an undergrad. Nobody reads it to write a mobile app. Since I read it, I had jobs or book contracts that depended on knowledge of LISP machines, bit slice, the original Macintosh OS, Windows telephony API, edge routers, core routers trying to be edge routers, J2ME, Linux on ARM, and Android. Android will be around another 10 years, maybe a bit more or less, and it too will be a museum piece.

My shortest trajectory from zero to Noted Authority and back to zero was when I had a book contract to write about Visual J++. Thank you, Sun's lawyers, for that short trip.

All that said, if you don't know anything about project management tools, estimating market size, or how to write, or critique, a cash flow and P&L projection, your knowledge is going to be hard to package in a form that has high value.

</comment><comment><USER25> I've never understood why for many professions age/experience is coveted but engineers are assumed to have such a short shelf life. Nobody prefers a 30 year old lawyer or doctor over a more experienced one but an engineer of the same age needs to start planning a career transition...where is the logic in this?

<comment><USER26> Doctors and lawyers gain skill as they get older because the problems and solutions don't change quickly, and the fields have such depth that expertise takes decades to acquire. We work in an industry where the state-of-the-art is perpetually accessible to 20-somethings who haven't done anything, because the technology gets blown up once a decade (or so). Even if you stay up to date, why would anyone pay you a salary commensurate with your experience, when a total noob can do the same work for half as much?

For example, it's pretty embarrassing that we're re-inventing databases because today's generation of "experts" can't be bothered to understand SQL. And don't get me started on how an entire generation of coders will be able to work professionally without knowing how to manage memory. It isn't difficult to find examples of youth and exuberance trumping wisdom in this field -- I've already encountered working developers who are happy to tell me how technologies invented in the early part of this decade are "totally outdated".

<comment><USER4> I think we are reaching the golden age of the older developer. Companies are learning how much more great developers are worth than poor ones. Retention bonuses have gone crazy (getting a 3 year / 250k bonus package isn't even that interesting).

The evergreen underpinnings, the basic knowledge of how to build systems floats through languages and technologies.

<comment><USER26> Agree with you about the value of older devs, but I wish I could say I saw "the industry" paying for experience. The examples of ridiculous grants I know about have all gone to people who are in their 20s and early 30s.

Granted, there's some selection bias going on: I don't see many 40-something programmers in SOMA. That said: _I don't see many 40-something programmers in SOMA._

<comment><USER4> Rampant speculation, but while drinking me and some developer friends came up with a theory on this. There was a sweet spot, a golden age if you will where a lot of things came together for kids / young adults to drive them to be developers. They are 25 to 40 now.

Computers became cheap enough to have at home. BBSs and then the Internet took off. Computers were doing stuff -- but not common place yet. It was an odd window during a technologies birth that I think got people hooked in a way that is hard to emulate. Too early, you are on punch-cards and computers are rare. Too late, and you have app stores and iphones and stuff just generally works so it can be treated as a dumb tool. Sheer luck.

</comment></comment></comment></comment></comment><comment><USER27> And engineer in his 60 will be worth his weight in gold just for knowing all the ways a project can fail and all the unknown unknowns that can bite you in the ass.

And with every 5 years adding adding another level of abstraction he can be invaluable for hard to debug things that need knowledge what goes where under the hood.

That was this guy - now in his late 50s who mentored me as a young whelp - <USER2> have seen him open a dump with a hex editor (not dissembler) while tracking a bug, saying - aha a pointer is not initializing correctly and fixing it. Will hire him in a second given the chance.

</comment><comment><USER28> Whenever I read an article drawing conclusions about the world based on interviews with those attending a school reunion, I consider the inherent selection bias. If I'm an unemployed 60-something, I might consider a class reunion to be a great networking opportunity. And, of course, my time has a low opportunity cost. Conversely, am I willing to fly across the country for the weekend if I'm working a lot and need to be in the office on Monday morning? Also, those whose lives at age 60 aren't all that fulfilling professionally or otherwise might be more likely to attend a reunion where the could be reminded of better times. [I say this as someone excited to attend my 20 year high school reunion this summer.] A survey of Stanford engineering graduates done by random selection would be much more interesting.

</comment><comment><USER29> I've witnessed the exact opposite. My father stayed an engineer when most of his peers became managers; many of them became unhappy over time, and quite a few died from heart attacks before reaching 60.

OTOH my father, now in retirement, still occasionally works as a freelancer, because he still enjoys the job, and his skill is in demand.

<comment><USER30> _My father stayed an engineer when most of his peers became managers; many of them became unhappy over time, and quite a few died from heart attacks before reaching 60._

Yeah, I realized that I disliked management (the concept) when I realized that managing is, in most cases, just as bad as being managed.

<comment><USER31> And unless you're the CEO, being a manager is _both_ managing _and_ being managed at the same time.

<comment><USER30> Right. The corporate system seems to be designed to make almost everyone miserable so the people at the bottom want the jobs above them. The problem is that it leads to pernicious and destructive internal competition, not excellence.

My philosophy: mentoring, coordination, and very occasional police work (against internal harassment and bad behavior) are the only valuable contributions of management. The rest of it is useless and should go away. It's outdated.

<comment><USER32> You spend a lot of time -- "a lot" is a pretty big understatement based on your online activity, blog etc. -- thinking and writing about how miserable being employed at companies X and Y made you. I'm really curious why. I've been miserable in a series of jobs before and the last thing I want to do is go back and wallow in it.

<comment><USER30> The problem with the career/work game is that when you get robbed (a) it's considered to be your fault, and (b) you never get back the time lost.

If there were a way to pop back into the state one should be at, at one's age, were it not for the robberies, then forgiveness would be possible. You'd just recover and forget. But this is a game where if you get screwed, your competition is out there getting better experience so you keep getting screwed in the future, and soon enough you're 40 years old and still a nobody. Fuck that.

Also, these problems that I'm attacking are huge problems. If I could, for one example, start a process that made it socially unacceptable for a company to run closed allocation and call itself a technology company, that'd easily add $10+ trillion in value to the world economy. That's just a massive effect. Obviously, one person can't cause that kind of change, but there's no reason not to try to start the process.

<comment><USER32> I don't really buy it, based on my personal experiences, which play out exactly opposite of what you're saying here. I think you're wrong. It's been very strange watching your evolution as an online persona.

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> What have some of you more career-advanced HNers done to ensure yourselves employment as you guys age? I'm looking for tips to take into consideration through the years.

<comment><USER33> I've grown my leadership skills. As much as I'd love to just code and keep my head down the simple reality is they expect me to step up and guide others. So I study management and process more than I study the latest techniques. Part of it bums me out but part of it is very exciting. I realize now at the latter end of two decades in this career that the true "big" problems in software are usually people or process related. If a project fails or succeeds depends much more on who does the work and how it gets planned and executed than what particular language or techniques you use.

So to sum it all up - I grow my leadership, management, and strategy skills.

<comment><USER34> Do you have any advice/pointers for someone who is looking to get started on this path?

</comment></comment></comment><comment><USER35> This article is incredibly terse and uninformative. Why are older engineers having such a hard time? Are they keeping up with new technologies to remain competitive? What kind of engineering are we even talking about?

<comment><USER36> The problem of ageism is pretty well documented in engineering, especially software engineering.

I'm in my 40s, and I've been programming my entire career. I'm trying to get into management so as to extend my career into my 50s and 60s. The realities are that I likely won't be able to compete against kids 1/2 to 1/3 my age in the next 15 years, so I need to use my experience to my advantage.

It's too bad because I would rather just program.

<comment><USER37> why you think you will not be able to compete? I am similar age and really don't see age as issue if you can do the job. Even money is not something you need to compromise

</comment></comment></comment><comment><USER38> The hard thing is "keeping your skills current".

Once you have been in a job for a while it can be easy to stagnate because you end up spending most of your work time maintaining something that was written in ASP.Net or PHP4 or whatever and it doesn't make business sense to do a rewrite in something newer.

The alternatives are to job hop or to work somewhere that is constantly using or evaluating new tech. Or to spend your own time learning new skills , which you may have less of when you are older.

There's also the issue of deciding what to learn, since you can't realistically learn everything.

I've been burned in the past by spending time looking into various technologies that have never really gone anywhere. It's hard to make a bet like "Will I be more employable in 10 years time by learning node.js or go?"

</comment><comment><USER39> One of the best engineers I have worked with is on his 60's. When I worked with him, the rest of the engineering team had an average age of 23 or so (me being 32), but this guy could give everyone a run for their money. This is not about age, but about staying current and sharp. Keep learning, and you won't have this issue. Sure, a guy fresh out of school has a ton of energy and works crazy hours, but as an Engineering Manager and Product Manager I knew that if I needed something to be really done on time I could always trust the older (and quite frankly way smarter) guy.

<comment><USER40> _This is not about age, but about staying current and sharp._

Yep. I don't know why these types of posts get so many comments. It seems really obvious.

</comment></comment><comment><USER41> This might be irrelevant to the overall discussion, but reading comments to some other posts on this blog almost made my jaw drop.

"The reason why California is flooded with thriving Asians is because the Golden State is a beta state, and all California cities are beta towns with a beta White majority. Smart, talented, and hard driven good looking – alpha Whites from different parts of the country leave for places in the Northeast, especially NYC. "

lolwut

</comment><comment><USER42> TL;DR if you spend 35 years as an engineer, you'll still need to work for a living, but if you spend 35 years as an entrepreneur or VC, you'll likely be retired long before that. In other words, it's better to be at the top of the pyramid than working for someone at the top.

<comment><USER43> Yeah, that's not true at all. Lots of people find success as an entrepreneur but they're the exceptions, not the rule. I'm not saying that it's something that people shouldn't attempt, and it certainly has a higher success rate than attempting to be, say, a rock star or a professional athlete, but it is by nature a high risk endeavor.

</comment></comment><comment><USER44> Maybe its because I'm not old (30), but the sentiment of this post strikes me as bullshit.

At _any_ age, if you've got something unique to offer and can contribute- you can get hired. If you've let your skills lapse, or have become inflexible then that's a different problem.

New immigrants I don't view as a problem at all, but rather an opportunity for us to increase the size of the market overall. There's work enough for all, as long as you've got something to contribute.

Maybe at one point 35 years ago simply having a degree from somewhere great was enough. I don't see that today as being a thing that makes someone immediately awesome.

Invest daily in your education. It doesn't stop when you're 22. Today, instead of doing consulting- I'm learning about Google's Polymer framework/platform.

<comment><USER45> I'm 40, but I'm not about to tell a 55-year-old what it's like to try and find employment as a 55-year-old. Just saying 'keep yourself current' is missing a lot of the picture.

<comment><USER44> I guess I'm looking at it from a hiring perspective. I don't care how old someone is honestly. I just care what they can do and what they contribute to the team. Having a bit of grey hair is probably an asset, since hopefully they've been to this rodeo once or twice and have some good experiences that we don't.

</comment></comment></comment><comment><USER46> One datapoint from Chicago: Got laid off at age 56 last year and had two offers within two weeks. But I interviewed at several other places without receiving an offer whereas in the past I almost always got an offer if I got an interview (Google excepted). Here is some speculation about why my batting average dropped. I do feel that age worked against me but I don't think it was because of the companies having institutional biases against hiring older people. Rather, I think it was just the mindsets of individual 20-something/30-something interviewers making individual decisions based on what felt comfortable, based on the vibe feeling ok. Nobody here is doing anything wrong.

</comment><comment><USER47> There is active discrimination against older engineers. I should know, I'm on many interview/screening rounds.

</comment><comment><USER48> My stepfather is in this situation, just turned 66 and has told me on several occasions, shit, the writing's on the wall, can't compete with the young guns.

Of course, he's been in the business since the punch card days and has a huge amount of experience to draw on; as such, he's miraculously (to me) been able to land guaranteed 60 hour per week gigs (even though he likely only works 30 hours) at $125+ per hour.

K-rist, that is a skill in itself, landing high paying telecommute gigs and putting in half the hours you're paid for, the f-er ;-)

</comment><comment><USER49> Newton's First Law of Motion. Keep moving, stay moving. Keep eating healthy and exercising, stay healthy and fit. Keep learning, stay sharp and knowledgeable.

There are more life frictions as we age that dissuade us from continually learning. If you realize this and work against it, there is no way someone significantly younger than you can realistically be a better fit for a job. They may have knowledge, but you'll have that and decades more experience and wisdom.

</comment><comment><USER50> On the internet, nobody has to know how old you are. Or at least this was the case, before people started posting every personal detail of themselves online.

</comment><comment><USER51> The blog's author, later in the comments section, doing a great impersonation of my mother (sans the anti-immigrant bit):

 _All of the smartest Americans are going into more lucrative fields like finance, medicine, and law. As much as you think the creme-of-the-crop engineers are getting paid, they aren't making as much as investment bankers, and even the best computer programmers don't make as much money as an average surgeon.

The market is flooded with mediocre Indians willing to work cheap, and that's what every IT department wants, cheap labor._

It amazes me that people can still say things like this in post-crisis America. As if finance, which is still reeling, isn't already flooded with Indians and Asians with tech backgrounds, as if they aren't in law (horrifying debt, anemic job market) and medicine (even more horrifying debt, massive commitment) as well.

Hell, I wonder if the quote here, supposedly from the WSJ comments section, is even real, or if he's just looking to rile people up.

Ageism in engineering, especially software engineering, is a real issue worthy of vigorous debate, but I'd like to think there are better pieces we can argue around than this dubious anecdote.

</comment><comment><USER52> "Engineer" is a fairly vague term. Once you are 60, you have probably "graduated" to a higher-level engineering job that involves more consulting/management/planning and less gruntwork. If you still enjoy the gruntwork aspect (I do, at times), more power to you. Either way, anyone involved in a programming-related job should always keep themselves up to date by getting their hands dirty once in a while.

</comment><comment><USER53> A 40 year old degree in _anything_ alone isn't good enough to stay employed in any field. Except maybe teaching.

<comment><USER54> You make it sound like these people haven't been working in the field since they got the degree

<comment><USER55> They may have been working, however there's zero guarantee that they have been learning anything new or even spending hands-on time doing engineering work (as opposed to shuttling between meetings).

I interview 2-3 candidates a week, including many people almost twice my age. While these older engineers have impressive-sounding resumes (IBM, Sun/Oracle) most struggle to code anything more complex than fizzubzz. In my case I make it as easy as possible for the candidates by allowing them to use any IDE they can install on my Macbook and encouraging them to search Google/StackOverflow if they get stuck because I want them to be comfortable and actually complete the task. Majority still end up with trouble implementing a simple JUnit/Selenium automation test for a login page.

<comment><USER56>
      > Majority still end up with trouble implementing a simple
      > JUnit/Selenium automation test for a login page.

Somehow I doubt the result is different for younger ones.

</comment></comment></comment></comment><comment><USER57> Age discrimination is sometimes even publicly announced. For example European Space Agency has age limit for recruitment 55 years. This applies even to desk based jobs such as administrative or software developement.

Extract from job description:

\- The European Space Agency is an equal opportunity employer

\- Contracts Officer in the Procurement Department, Directorate of Procurement, Financial Operations and Legal Affairs

\- preparing requests for offers/invitations to tender;

\- dealing with administrative, contractual, legal and industrial problems arising from the procurements concerned.

\- Under ESA Regulations, the age limit for recruitment is 55.

More at: <http://www.esa.int/hr/PDF/ESA-VN-ESTEC-2012-091,REV.1.pdf>

<comment><USER38> That seems odd, I can't find anything explaining why such an age limit is in place. I am assuming that there is some reason for it.

I know that the military often has an upper age limit due to fitness requirements.

</comment></comment><comment><USER58> This article starts from a personal experience (with a really small social circle) and generalize to everyone. Worse, it generalizes to "engineering" which has wild and quite different fields.

If you have worked on Cobol or Fortran when you were 25 year-old, these things don't exist anymore (or enough to employ these old developers).

Otherwise, just put yourself in the place of the recruiter. I guess, except for special fields like Medicine, Law...; in most of the fields this ageism is the same.

If you want to have a builder, would you prefer a 65, 20 or 30? Certainly, you'll pick 30. He has more experience than 20 and way more energy than 65.

</comment><comment><USER59> No one's really mentioned that keep up to date or learn a new skill is great advice...but it sucks when someone's tells you that you only have six months with MongoDB and Node.Js, so we only want to pay you for a Junior position. This is where the ageism comes into play, where a young 22 year old who doesn't have the experience because of a new technology buzzword looks attractive to the hiring manager. Seen this move a lot.

"But what about all my experience with I/O Completion Ports and multi-threaded server development mastering mutexes and slinging semaphores!?

</comment><comment><USER60> ___SLAPs OP in the face_ __come to your senses man! Old is wise harness your skills to develop new ones. The dodo that can not fly shall go extinct no matter what the profession.

</comment><comment><USER43> I live in SV and am in my early 40s and can only comment on what I've seen firsthand or heard about secondhand. Based on that, I don't really see ageism. I work with people in their 20s, and I work with people in their 60s. It can be harder to find a new job at comparable pay as you get older, but it has a lot more to do with the following:

(1) Salary expectations. Somebody works at the same company for 20 years, gets an annual raise that's above the rate of inflation, and has an expectation that if they go someplace else, they're going to get as much or more money.

This might happen, but it's not guaranteed. I've worked at companies in other industries where it was common for people to work at those companies for 30+ years, then retire. These companies have different notions as far as annual raises and how generous the perqs and bonuses should be, because they expect people to be around for a long time.

Tech companies tend to be a lot more generous, because there are much higher turnover rates and they have to work harder to retain talent. The result is that after 10 or 20 years at a tech company, you may become accustomed to getting paid above your market value (I hope to have that problem someday :). This isn't always the case, but it does happen.

(2) Lack of relevant technical minutiae. Everybody loves to say things like, "it doesn't matter what you know, it's your capacity to learn!" Or, "my knowledge of abstract CS concepts means I can pick up the technical minutiae associated with a learning a new language/library/platform/etc. in no time!" All of these things are true to an extent, and you're not going to get very far without them. However, none of them have anything to do with actually accomplishing anything- they're all just modifiers to your ability to learn and adapt, prior to accomplishing anything.

Unfortunately, accomplishing anything requires technical minutiae. Some of it is more widely applicable stuff like programming in a particular language or using certain tools, but the stuff that makes you really really valuable to your current employer is more likely to be the niche that you carve out for yourself over the years. This means, knowledge of specific codebases, and domain-specific knowledge, some of which might be from other disciplines like math or finance or electrical engineering. When you go someplace new, your niche knowledge is either going to be invaluable (yay!) or meaningless (bummer :( ). Most of the time, it's the latter. So, again, this is a case where through no fault of their own, someone has ended up getting paid a lot of money for doing a great job, by acquiring lots of valuable skills that don't mean much outside of that company and position.

(3) Burnout. Sometimes people just get sick of spending so much time around computers after 20 or 30 years (or 5-10!). They lose their enthusiasm and aren't as motivated to keep up to date. If you're burned out on your profession, you will have a harder time finding new work, for any number of reasons. It's unfortunate but it happens to everyone at some point or another, maybe many times throughout their career. Some people find ways to renew their interest and get out of this mindset, some don't. Older people are more susceptible to this because they have more important things in their lives (like family).

TLDR: don't get frustrated, try to see things from a prospective employer's point of view in addition to your own POV, and keep your skills sharp and up to date. Don't coast. Stay curious and make sure you're still having fun doing what you do. If you're no longer curious or no longer enjoy it, see if you can come at it from a different angle and rejuvenate your interest. If not, you'd probably be happier taking a left turn in your career anyway, doing something completely different.

</comment><comment><USER61> Any blog starting with a immigrates take our jobs implication is not really something to note IMO.

But anyway if you're an engineer and can't retire by 60 then you've done something wrong.

Just don't blow your money on stupid stuff.

And by retire at 60, that can mean anything, work for free for a charity, do a simple start up, work part time.

<comment><USER62> >"if you're an engineer and can't retire by 60 then you've >done something wrong."

I used to think that too....when I was 30&40 something. Life has a funny way of throwing a few curve balls at you that can drain your finances in short order.

</comment></comment><comment><USER63> Here's his follow up post that responds to comments in this thread: [http://lionoftheblogosphere.wordpress.com/2013/05/28/follow-...](http://lionoftheblogosphere.wordpress.com/2013/05/28/follow- up-on-old-engineers/)

</comment><comment><USER64> I've seen plenty of developers in their 30's unable to work because they were a one-language pony, and that language was no longer needed in the marketplace.

</comment><comment><USER65> "You know what they do with engineers when they turn 40? They take them out and shoot them."

</comment><comment><USER66> In what field of engineering is this person trying to find employment?

</comment><comment><USER67> So what's a good backup plan as you age?

<comment><USER68> Read Mr Money Moustache's blog. Live like a grad student even when you are earning six figures.

</comment></comment><comment><USER30> I see our age discrimination problem as the fundamental sign of us being a defeated tribe. Ageism is one of those horrible traits of management culture (i.e. if you have a younger boss, you're a loser) that we wouldn't face if we had kept our tribal integrity (because "bosses" would be coordinators and mentors, not hierarchical superiors, making age irrelevant.)

 _Good_ programmers aren't ageist. We look up to the gray-haired badasses. (Ok, old bad programmers are annoying, but so are young bad programmers.) Ageism is a managerland prejudice. What the fuck is that shit doing in our industry?

Most of us are intellectual whores: massively overqualified for the work we're assigned to do, but hired because it makes the boss feel important to be able to say things like, "I have four Ivy Leaguers working for me" even when the work could be done by one script-kiddie. Meanwhile, the work that could actually use our talents is hard to find, because our society is in such a depraved state that social media gets VC-megabucks but cancer research gets peanuts.

It's perverted. When you're young, it's hard as hell to get a good job but getting _a_ job is easy. But that's exactly when you care about getting the good jobs that will build your career. When you're old and have actual use for the crappy jobs (you need to sustain an income) you can't get them; and if you were sidetracked by taking too many crappy jobs (and that's not hard to have happen) you're not qualified for the good ones either.

<comment><USER18> > Meanwhile, the work that could actually use our talents is hard to find, > because our society is in such a depraved state that social media gets VC- > megabucks but cancer research gets peanuts.

Not only that, but social media will hire the college dropout with an impressive Github portfolio while the cancer research institute will shitcan any resume that doesn't have an MS or PhD on it.

<comment><USER30> It's really disturbing to realize what has happened to the sciences. The death of the scientific/academic job market is why us smart people hate "business douches" so much; the fuckers shut down all the R&D. Those assholes struck first.

(To be fair, plenty of businesspeople do more good for the world than bad and aren't douches. It's the shitty ones that have no better ideas than to cut costs recklessly. The problem is that no one shuts them down.)

In the 1970s, there were Bachelor-level science jobs. There was an expectation, of course, that most people would go back for graduate degrees; but it gave people a chance to spend a few years in the field and figure out what they wanted to do with their lives before committing to a graduate program.

Those Bachelor-level science jobs still exist. The problem is that they're all filled by PhDs.

Even most of this Silicon Valley "innovation" in social media isn't half as interesting as the stuff marketers did in the _Mad Men_ era. M&A replacing R&D is one of the worst social developments around.

Who's going to bring back Real Technology? Someone will see the value in doing so, I hope.

</comment></comment></comment><comment><USER69> I'm not buying it. I'm about to turn 50 - doing better work than ever and more in demand. The tools today are so good compared to 5-10 years ago. Now I have these awesome tools, and understand sales/marketing/management/finance.

ps - could I get a job as a developer? probably not. but there are other (better) alternatives.

<comment><USER70> I turn 41 next month and have been a developer for the two decades that I have been in IT. A 50 year old up to date on the latest tools seems to be the rare exception. The rule at my current employer seems to be after hitting 50-55 that they really have their eye set on retirement. Learning a new skillset has such low dividends that they don't believe it is worthwhile. They show up to work right on time at 8 am, they spend an hour getting their coffee, perhaps breakfast from the cafeteria, chatting with coworkers, etc. Then they stare at a mainframe screen full of Cobol code for an hour or two, take a 90 minute lunch, attend a few meetings in between staring at mainframe Cobol code and then leave at 5 on the dot.

They spend two weeks on changes like, "Modify program P90182 to set value 82 instead of value 81 in field CONVEXITY", dutifully put in their change request documents and then move on to the next change honestly believing they're earning their inflated salary.

<comment><USER45> Your stories of your 50-year-old colleagues match those of 18-19-year olds I've worked with in several different industries, except for the 'show up on time' thing. The only people I've ever seen expend more active effort to avoid doing work than the work itself actually required were people in this age group.

</comment></comment></comment><comment><USER71> If you think being an old engineer is tough, try being a new one.

<comment><USER72> Can you elaborate on this? I am a fairly new developer and haven't found it particularly difficult.

<comment><USER71> It's easy if you have 4 years of school to soak up knowledge. But if your family can't afford to pay your school fees, you end up working somewhere full-time. So you have to study nights. Except you fall in love and get married to a wonderful woman, but then she wants to spend those nights with you. So then you end up not sleeping and begging her to get a night-job so that you can learn how to install arch linux on a virtual machine. Becoming a software engineer when you have life commitments is tough, and you end up reading programming language books on your iPhone every time you go to take a crap. Believe me, it's hard.</comment></comment></comment>


Can you find a winning strategy for this iterated prisoner's dilemma? - <AUTHOR> https://math.stackexchange.com/questions/3255563/can-you-find-a-winning-strategy-for-this-iterated-prisoners-dilemma <comment><USER2> \- Create multiple bots.

\- If necessary (it isn't clear to me whether the API allows you to choose your bot's name) have those bots inform themselves of each other's identity by gossiping on themselves, specifying a secret number.

\- Only play against your own robots.

\- Either make them all cooperate all the time, or designate one winner that betrays all the time.

\- Play a lot of games.</comment>


Intel is giving up on its smart glasses - <AUTHOR> https://www.theverge.com/2018/4/18/17255354/intel-vaunt-shut-down <comment><AUTHOR> > if the company wasn't able to successfully find a similar partner to handle > Vaunt, it's not exactly a surprise that it chose not to try to bring them to > market directly.

That's too bad. It seemed like a solid idea, responses on HN last time it came up were fairly positive [0]

[0]: [https://news.ycombinator.com/item?id=16308522](https://news.ycombinator.com/item?id=16308522)

</comment><comment><USER2> Damn... this really blows! Someone else should do this then! I know the technology seems pretty insane but someone should :(</comment>


Bank of America: No More Free Checking for Customers with Low Balances - <AUTHOR> https://www.wsj.com/articles/bankofamerica-no-more-free-checking-for-customers-with-low-balances-1516625715?mod=ITP_businessandfinance_0&tesla=y <comment><USER2> Re-institute postal banking. Now. [http://www.slate.com/articles/news_and_politics/history/2014...](http://www.slate.com/articles/news_and_politics/history/2014/08/postal_banking_already_worked_in_the_usa_and_it_will_work_again.html)</comment>


5 Gmail Labs Features I can't live without - how are you living without them? - <AUTHOR> http://gettingmoreawesome.blogspot.com/2009/11/5-gmail-labs-features-i-cant-live.html

<comment><USER2> Sender time zone looks great. I wish Facebook would integrate that into their IM.

</comment><comment><USER3> Where is the monospace font feature? It _can_ be done with GreaseMonkey: <http://userscripts.org/scripts/show/43538> but I want it always on, from wherever I use gmail.</comment>


Sensors detect rise in nuclear particles on Baltic Sea - <AUTHOR> https://mobile.reuters.com/article/idUSKBN23X2TN <comment><USER2> Now, this is probably nothing. The sensors are doing their thing and from time to time there is a spike. On the other hand, this is how Chernobyl first was reported...

For more radiation horror pease read about the "Goiânia accident": [https://en.m.wikipedia.org/wiki/Goiânia_accident](https://en.m.wikipedia.org/wiki/Goiânia_accident) (Learned about it recently and wow. What a story.)

<comment><USER3> Or this: [https://en.wikipedia.org/wiki/Kramatorsk_radiological_accide...](https://en.wikipedia.org/wiki/Kramatorsk_radiological_accident)

In 1989, a small capsule containing highly radioactive caesium-137 was found inside the concrete wall of an apartment building, with a surface gamma radiation exposure dose rate of 1800 R/year.

The source, originally a part of a radiation level gauge, was lost in the Karansky quarry. The gravel from the quarry was used in construction. The cesium capsule ended up in the concrete panel of Apartment 85 of Building 7 on Gvardeytsev Kantemirovtsev Street, between apartments 85 and 52.

Over 9 years, two families lived in Apartment 85. A child's bed was located directly next to the wall containing the capsule. By the time the capsule was discovered, 4 residents of the building had died from leukemia and 17 more had received varying doses of radiation. The accident was detected only after the residents requested that the level of radiation be measured in the apartment by a health physicist. Part of the wall was removed and sent to the Institute for Nuclear Research (NASU), where the cesium capsule was removed and disposed of.

The total number of deaths is alternately reported as 2, 4 (3 children and one adult), or 6 (4 children and 2 adults).

<comment><USER4> It looks like China(and potentially other places) are willing to accept incredibly dangerous(and expensive to recycle) material and melt it into their main steel supply. After all, cobalt diluted into a supply of steel is harmless, right?

[https://www.nbcnewyork.com/news/local/radioactive-metal- tiss...](https://www.nbcnewyork.com/news/local/radioactive-metal-tissue-box- holder-bed-bath-beyond-pulled- cobalt-60/1977257/#:~:text=Cobalt%2D60%2C%20a%20man%2D,ray%2C%20said%20state%20health%20officials).

I would really like to know how much of a problem is it, how much of global steel supply is tainted. Can we be buying cars and other items made with steel that's contaminated?

<comment><USER5> Back when the US was still interested in nuclear non-proliferation, they sponsored (together with the EU) the installation of radiation detection equipment at borders and other chokepoints, mostly in poorer countries.

These are fixed installations, monitoring mostly truck traffic 24/7, so their effectiveness doesn't depend on available manpower or current interest. Eastern Europe from Poland to the -stans has a lot of them, because former soviet republics had a lot of radioactive material lying around that was suddenly less important and even less guarded after the Cold War.

(Scientists were also suddenly out of work, and feared to be receptive to corruption. Fun fact: George Soros spent dozens of millions of his own money to empty a bunch of nuclear physicists in the 90s).

The point being: if it were a widespread problem, it would definitely be noticed.

<comment><USER6> How, exactly, does one "empty" a person?

<comment><USER7> Psychologically abuse them for months or years, place impossible to meet expectations in front of them and tell them that any attempt to meet them is wrong before they've even started to make an effort. Let them be open and vulnerable with you and punish them for it, then later demand they be open and vulnerable and only randomly punish them for doing so. Oh and the entire time you are putting them through this, trivialize their emotions and show them that they're wrong for having them.

That should be a good start.

</comment></comment></comment></comment></comment></comment><comment><USER8> Do you think it might be related to this event? (linked below _) I 'm going to make a guess, that there might have been a nuclear detonation. It probably wasn't a test. This seems to be an accident. Not all weapons are accounted for, e.g. there a American hydrogen bombs lost in Spain (I know... it sounds crazy but look it up.). Could there have been a bomb laying around someone in a place everyone forgot existed maybe some forgotten relic from the Soviet day?

_ A GLOBAL MAGNETIC ANOMALY: On June 23rd, Earth's quiet magnetic field was unexpectedly disturbed by a wave of magnetism that rippled around much of the globe. There was no solar storm or geomagnetic storm to cause the disturbance. So what was it?

[https://wattsupwiththat.com/2020/06/25/weird-out-of- nowhere-...](https://wattsupwiththat.com/2020/06/25/weird-out-of-nowhere- something-just-rocked-earths-magnetic-field/)

<comment><USER9> Almost certainly not. A nuclear detonation would also produce noticeable seismic effects as well as release iodine-131, which was not detected.

<comment><USER10> [https://thebarentsobserver.com/en/ecology/2020/06/various- re...](https://thebarentsobserver.com/en/ecology/2020/06/various-reactor- related-isotopes-measured-over-scandinavia-and-svalbard) reports that Iodine 131 was measured in Norway during the week of June 2-8.

<comment><USER11> Ok, but that's two weeks off from this detection.

</comment></comment></comment></comment><comment><USER12> Since everyone is saying Russia, I just want to include a map of reactor locations. It _may_ be Russia, but there are quite a few reactors in the area.

[https://upload.wikimedia.org/wikipedia/commons/e/e9/%D0%90%D...](https://upload.wikimedia.org/wikipedia/commons/e/e9/%D0%90%D0%AD%D0%A1_%D0%95%D0%B2%D1%80%D0%BE%D0%BF%D1%8B.png)

<comment><USER13> An unreported release of radionuclides from a Western European reactor would be much more surprising.

<comment><USER14> In Germany they tried to use Chernobyl to hide a reactor accident [https://en.wikipedia.org/wiki/THTR-300](https://en.wikipedia.org/wiki/THTR-300)

<comment><USER15> Thanks for the link, it's fascinating:

> some radioactive dust was released to the environment. This happened just a > couple of days after the Chernobyl disaster. The operators played down the > incident, which caused a loss of confidence in the regulatory authorities.

Radiactive dust is not a small problem, it's fatal if inhaled. Skin can stop a lot of radiation, but dust in the lungs is a worst-case scenario.

> THTR-300 was only in full service for 423 days

It was dismantled after only 4(!) operating calendar years because of rising costs and lack of faith in the regulation process. Of course, the government is on the hook for the overruns. What a disaster.

Oh, exactly the same things the HN nuclear apologists gloss over.

The challenge in the US is that most reactors have their operating permits rubber-stamped to extend 20 additional years. It means they're active much longer than designed, inviting corrosion and brittleness problems, and the operator can avoid paying for the decommissioning "until later," when it's somebody else's problem, usually the tax payer.

Hence why nuclear reactors need to be stopped before construction even begins.

</comment></comment></comment></comment><comment><USER16> On the SE edge of the marked area Belarus is building a power plant. It is scheduled to go online July 2020 [1]. Can it be that fuel loading took place and somehow it got out and wind blew the particles towards east? (similar direction as in Chernobyl's case)

Pure speculation, but I'm Lithuanian and I'm concerned about a foreign country building a power plant 45km from my capital. CTBTO not disclosing their guess about the source makes me think it's a sensitive geopolitical matter. I just hope nothing serious has happened, but you never know with 2020.

[1] [https://en.wikipedia.org/wiki/Astravets_Nuclear_Power_Plant](https://en.wikipedia.org/wiki/Astravets_Nuclear_Power_Plant)

</comment><comment><USER17> I understand the political reason for not naming the reactors/nation-states most likely responsible for the release but I don't understand the lack of context. Why wasn't short-lived iodine-131 detected along with the caesium-134, caesium-137 and ruthenium-103? How do the measured levels of the long-lived caesium/ruthenium isotopes compare to known events like Fukushima? How do these measured levels compare to known nuclear weapon tests?

I'm worried about the geopolitical implications, not the global health implications.

<comment><USER9> The release likely happened during fuel reprocessing. Since fuel is left to cool for at least several months before reprocessing, there's essentially no I-131 left at that point.

<comment><USER17> How does the spent fuel get aerosolized and into the atmosphere? Some of the more recent comments link to a site that detected iodine-131 in Norway. Vented hydrogen/steam from partially uncovered fuel rods (e.g. Three Mile Island, Fukushima) in an active boiling water reactor is historically the most likely source (I think) but iodine-131 should be abundant. The same principle should apply to any nuclear powered submarine or ship. Other commenters point out that Russia has nuclear powered missiles and torpedoes that emit radiation. I expect the CTBTO to be able to categorize the nature of each radiation source but maybe that type of analysis takes time.

<comment><USER9> >How does the spent fuel get aerosolized and into the atmosphere?

I'm not an expert, but fuel reprocessing is generally done is an aqueous state. The widely used PUREX process, for example, starts with dissolving the spent fuel in nitric acid. Splashing or boiling the liquids involved will cause the formation of aerosols. In particular, concentrated nitric acid tends to react with things quite vigorously, generating a lot of heat. The process also uses flammable liquids like kerosene and hydrazine.

Some radioisotopes also have compounds that are volatile. Ruthenium, in particular can form Ruthenium Tetroxide which boils at only 40C.

</comment></comment></comment></comment><comment><USER18> To those saying it's nothing: Lookup the historical data and compare. Chernobyl caused 0.1 mSv/h in radiation in Sweden, this incident caused 0.18 mSv/h in Helsinki. Only two short spikes over one week, Chernobyl was over two weeks, but certainly not harmless. These are deadly poisons. [https://www.stralsakerhetsmyndigheten.se/contentassets/66f4f...](https://www.stralsakerhetsmyndigheten.se/contentassets/66f4f029351d4baea989543f24414795/198612-chernobyl ---its-impact-on-sweden)

It's either one of the old Chernobyl-like pressure water reactors in Tallin or St Petersburg. Really interesting is also why the control commission in Vienna is not allowed to call them out.

<comment><USER19> > old Chernobyl-like pressure water reactors in Tallin or St Petersburg

You must be confusing Tallinn with some other city - Estonia does not and has not had any nuclear power plants.

The map covers Sweden, Finland, Estonia, Latvia and Russia. Estonia and Latvia have no nuclear power or weapons programs, and the first four are EU countries where staying silent about an incident would be very unlikely. That leaves Russia.

Interestingly enough the possible source area includes the Arkhangelsk naval base, where the Russian nuclear submarine fleet is stationed. An incident there may explain the lack of information.

<comment><USER20> I think Arkhangelsk being involved seems likely. The Russian ship "Akademik Aleksandrov" aka _Project 20183_ was apparently in that area recently[1], and seems obviously related to Poseidon[2]. If the source of the radiation were a Poseidon, that would certainly explain the secrecy.

As for claims of the fission products coming from a "civil source", I don't think enough is known about Poseidon's reactors to rule that out. I could be wrong, but as far as I've read the details of that reactor are up in the air (besides being small, obviously.)

[1] [https://twitter.com/FrankBottema/status/1276930424170844162](https://twitter.com/FrankBottema/status/1276930424170844162)

[2] [http://www.hisutton.com/Akademik- Aleksandrov.html](http://www.hisutton.com/Akademik-Aleksandrov.html)

</comment></comment></comment><comment><USER21> Russia, damaged fuel rod.

[https://www.rivm.nl/nieuws/radioactieve-stoffen- gedetecteerd...](https://www.rivm.nl/nieuws/radioactieve-stoffen-gedetecteerd- in-lucht-boven-noord- europa?_ga=2.21439288.204026084.1593171260-1689041124.1593171260)

21st century and the current top comment on HN is it's a random spike. Even though the article says it's not. This shouldn't be hard people.

<comment><USER11> This doesn't say that it was a damaged fuel rod, only that that is a possibility and that it has been discovered to be the cause of similar readings previously.

<comment><USER21> I'm not sure what more you want exactly? What are you suggesting it could be?

You have the data from multiple agencies, what fits?

I'm happy for alternatives, but multiple agencies say it's civilian and Russia. I want to leave this crazy, it's weapons and it's machine reading malfunctions threads.

So given the data what are you saying it could be?

This is why the comment section exists, to work towards to better understanding what is going on. Unless you are from Russia trying to sow doubt, you have to give a plausible alternative that fits the data. Saying it's possible it's something else is just sowing disinformation.

<comment><USER11> > I'm not sure what more you want exactly? What are you suggesting it could > be?

Nothing. I'm pointing out the agency did not phrase it as strongly as you did for people who can't or don't want to read the article.

> This is why the comment section exists, to work towards to better > understanding what is going on. Unless you are from Russia trying to sow > doubt, you have to give a plausible alternative that fits the data. Saying > it's possible it's something else is just sowing disinformation.

I'm no expert and not capable of judging whether people commenting know what they're talking about or not. All I see from official sources is that experts are saying that a leak from Western Russia is a likely possibility. I can wait until more concrete information comes out to make a judgement.

</comment></comment></comment></comment><comment> Smooth-Weather Something weird is that from 21:50 the 24 of June, the weather captors in the area reported a temperature of 60°C.

[https://www.meteociel.fr/observations- meteo/temperatures.php...](https://www.meteociel.fr/observations- meteo/temperatures.php?archive=1&heure=21&jour=24&mois=06&annee=2020&region=eur2)

<comment><USER22> It might be due to a local wildfire.

[https://www.bbc.com/weather/features/53014423](https://www.bbc.com/weather/features/53014423)

<comment><USER23> That seems much more likely

</comment></comment></comment><comment><USER24> Could this be from the test of the nuclear powered hypersonic missile that Russia is developing [0]? Those things aren't designed to 'land' no pun intended

[0] [https://en.m.wikipedia.org/wiki/9M730_Burevestnik](https://en.m.wikipedia.org/wiki/9M730_Burevestnik)

<comment><USER20> My first guess is their nuclear powered torpedo, 'Poseidon', aka STATUS-6, or Kanyon:

[https://en.wikipedia.org/wiki/Status-6_Oceanic_Multipurpose_...](https://en.wikipedia.org/wiki/Status-6_Oceanic_Multipurpose_System)

I'd sooner expect it in the Arctic Ocean, but who knows.

Edit: Some supporting evidence from twitter: [https://twitter.com/FrankBottema/status/1276930424170844162](https://twitter.com/FrankBottema/status/1276930424170844162)

</comment></comment><comment><USER25> "The composition of the nuclides may indicate damage to a fuel element in a nuclear power plant. RIVM has made calculations to find out what may have been the origin of the detected radionuclides. These calculations show that the radionuclides come from the direction of Western Russia."

[https://translate.googleusercontent.com/translate_c?depth=1&...](https://translate.googleusercontent.com/translate_c?depth=1&nv=1&pto=aue&rurl=translate.google.com&sl=auto&sp=nmt4&tl=en&u=https://www.rivm.nl/nieuws/radioactieve- stoffen-gedetecteerd-in-lucht-boven-noord- europa%3F_ga%3D2.21439288.204026084.1593171260-1689041124.1593171260&usg=ALkJrhh- TIF9k2nJoLZNi32MU-I5FaFOPg)

</comment><comment><USER26> What's the base rate for this type of detection? Another commenter mentioned this is how Chernobyl was first reported, but I can't find anything on how often such things happen randomly without a catastrophic root cause.

<comment><USER27> I am quite disturbed to find that this is not a zero level base rate.

I remember the "CIA Whole Earth" from Snow Crash. I always thought that Google Earth would be something like that - some cross between a weather station and wikipedia.

<comment><USER28> > I am quite disturbed to find that this is not a zero level base rate.

speaking of disturbing factoids about radiation.. This is one of those nerdy cocktail topics I spout out about when i'm bored at a bar.[0]

[0]:[https://en.wikipedia.org/wiki/Low- background_steel](https://en.wikipedia.org/wiki/Low-background_steel)

<comment><USER29> Relevant xkcd: [https://xkcd.com/2321/](https://xkcd.com/2321/)

</comment></comment></comment></comment><comment><USER30> Nuclear energy is very safe, but to be safe it relies on the systems that support it. One of those systems is the economic system. I hope nothing bad has happened.

</comment><comment><USER31> Probable sources would be: [https://en.wikipedia.org/wiki/Kola_Nuclear_Power_Plant](https://en.wikipedia.org/wiki/Kola_Nuclear_Power_Plant) or [https://en.wikipedia.org/wiki/Leningrad_Nuclear_Power_Plant](https://en.wikipedia.org/wiki/Leningrad_Nuclear_Power_Plant)

Notably the second uses RBMK-1000 reactors, the same as used in Chernobyl.

<comment><USER11> TASS says the operators of those claim that they are fine [https://tass.com/world/1172279](https://tass.com/world/1172279)

</comment></comment><comment><USER32> EDIT: apologies; I commented under the parent when I meant to do under this comment about the Goiânia accident: [https://news.ycombinator.com/item?id=23662601](https://news.ycombinator.com/item?id=23662601)

My aunt still lives next door to the "Vigilância Sanitária" place where the material was quarantined.

At the time I was spending school holidays with my cousins and, after a day out, we came home to find the whole street blocked. Naturally we drove around the block and managed to get to their condo - a unit out of a set of three buildings.

We didn't see any neighbours and it was only when we turned on the TV that we found out why. I guess the authorities forgot to block the entrance to the car park. So out we went again to get tested for radiation.

Going back to school in Rio was fun. My friends were giving me all sorts of ribbing and some kids refused to sit next to me for a while.

</comment><comment><USER33> I saw this tweet a short while back. It does not add anything to the article but is an official source

[https://twitter.com/SSM_Nyheter/status/1275416038755139584](https://twitter.com/SSM_Nyheter/status/1275416038755139584)

<comment><USER11> Tack. Do they have a live map like Finland does [1]?

[1]. [https://www.stuk.fi/aiheet/sateily- ymparistossa/sateilytilan...](https://www.stuk.fi/aiheet/sateily- ymparistossa/sateilytilanne-tanaan)

<comment><USER33> Not that I am aware of. Kind of disappointed actually.

<comment><USER11> Yeah, same. I found a couple [1][2] of live maps but I'm not really sure how seriously to take them. Especially since they don't appear to show this spike.

[1]. [http://radioactiveathome.org/map/](http://radioactiveathome.org/map/)

[2]. [https://radmon.org/index.php](https://radmon.org/index.php)

</comment></comment></comment></comment><comment><USER34> This is probably just leakage to due to mismanagement. Russian government may prioritize well being of population but nuclear accident would drop land price economy would suffer and people who have power would suffer from, so I hope self-interest and The lesson of all lessons make me hopeful that the rain outside my window not radioactive (in Lithuania not Russia but still in the path)

</comment><comment><USER35> Wouldn't be surprised if nuclear fission waste is being dumped

<comment><USER20> I would be surprised if they were doing that in the Baltic Ocean. Traditionally Russia has dumped their nuclear waste near Novaya Zemlya, into the Arctic Ocean.

</comment></comment><comment><USER36> Has this been verified / confirmed by a second or third sensors location? Wouldn't multiple locationns help pin point the possible source?

</comment><comment><USER37> I wonder where this is from. First thought was Fukushima. As they say it's from a civil nature. It's very far from the Baltic though. Wouldn't really be possible.

Or one of the many nuclear subs that have sank perhaps? I believe most ship reactors are quite similar in design to civil power plants, just much smaller.

</comment><comment><USER38> Quote: "We are able to indicate the likely region of the source..."

Then tell us who. And since they didn't said it in the first place, I bet is mother Russia again.

<comment><USER39> They do indicate the region, and it does include sweden, finland, large parts of baltics and certain areas in russia: [https://mobile.twitter.com/SinaZerbo/status/1276559857731153...](https://mobile.twitter.com/SinaZerbo/status/1276559857731153921/photo/1)

Had you included the full quote you would have known why they did not point out the exact location: "We are able to indicate the likely region of the source, but it's outside the CTBTO's mandate to identify the exact origin"

If I interpret it correctly it means that they do not know the exact location and other organisations will now have to investigate it further to get the exact origin.

<comment><USER40> _If I interpret it correctly_

Sounds more like they are pretty sure where but aren't allowed to say.

<comment><USER41> I think that's the point. They are only "pretty" sure. There is always room for error, and not pointing out the likely culprit protects them from retaliation.

<comment><USER42> Or they _are_ sure, but relations with the guilty party are not conducive to revealing it.

</comment></comment></comment></comment></comment><comment><USER43> Can't even guess, who's riding the horse with the naked torso could be responsible.

<comment><USER44> that's speculation. i could be a horse, it could be a bear, a moose, you name it

<comment><USER43> That's common sense. The only authoritarian state in the region refusing to share information for common security. In 2019 they already tried to hide radioactive leak on the nuclear test site [0]

After the accident, people with the radioactive syndrome were admitted to a general hospital, alongside with children and pregnant women. Doctors were threatened by FSB not to disclose information about accident. Chernobyl did not teach them anything.

[0] [https://foreignpolicy.com/2019/08/12/security-brief- russia-w...](https://foreignpolicy.com/2019/08/12/security-brief-russia- weapon-nuclear-test-radiation-leaks-bolton-visits-london-yemen-saudi- coalition/)

</comment></comment></comment><comment><USER18> So they know where the emission originated from, but they are not telling it?? Please. Are we still in the 80ies?

<comment><USER45> [https://pbs.twimg.com/media/EbdAa9qXQAYji-z?format=jpg&name=...](https://pbs.twimg.com/media/EbdAa9qXQAYji-z?format=jpg&name=medium)

Spoiler: 99% chance it's from Russia.

<comment><USER18> 50/50 Either Tallin or St Petersburg. One of them is not in Russia, but both are Russian-style (ie Chernobyl) water pressure reactors. One of them leaked in a massive scale, but thanksfully only twice.

<comment><USER43> Estonia would have immediately shared the information about the accident with the allies. Russia always (without exceptions) tries to hide it from the world

[https://foreignpolicy.com/2019/08/12/security-brief- russia-w...](https://foreignpolicy.com/2019/08/12/security-brief-russia- weapon-nuclear-test-radiation-leaks-bolton-visits-london-yemen-saudi- coalition/)

<comment><USER18> Yes, I was wrong. The 2nd nearby is Murmansk, and that's to far away for that measurements. So it's clear that it was from the stone aged St Petersburg plant, a design before Chernobyl 3+4 (which was iter 2). Their broken Unit 1 is shut down, but the old 2-4 are still online, plus 4 modern ones.

[https://en.wikipedia.org/wiki/Leningrad_Nuclear_Power_Plant](https://en.wikipedia.org/wiki/Leningrad_Nuclear_Power_Plant)

Or that military ship Akademik Aleksandrov which was nearby at that time. [https://news.ycombinator.com/item?id=23664056](https://news.ycombinator.com/item?id=23664056)

</comment></comment></comment></comment></comment><comment><USER46> The original data seems to be available at [^1] but I dont see any spike around the 22-23 of June ... Let's wait to know more before saying anything stupid (looking at you Greenpeace)

[^1] [https://www.stuk.fi/aiheet/sateily- ymparistossa/sateilytilan...](https://www.stuk.fi/aiheet/sateily- ymparistossa/sateilytilanne-tanaan)

<comment><USER47> That system you linked only measures radiation. While the sensor in Sweden actually sample the air to detect certain isotopes. STUK does have this type of sensors also and did see these same type of isotopes in the air[1]. [1] Announcement in Finnish [https://www.stuk.fi/-/helsingin-ilmassa-pienia- maaria-keinot...](https://www.stuk.fi/-/helsingin-ilmassa-pienia-maaria- keinotekoisia-radioaktiivisia-aineita-viime-viikolla)

</comment></comment><comment><USER48> This couldn't be the leakage from that nuclear powered cruise missile 'test' lost by the Russians 2017-2018[1]? That is, is Barents sea too far from the Baltic for a couple of years of tides?

[1] [https://www.theverge.com/2018/8/21/17766426/russia- nuclear-p...](https://www.theverge.com/2018/8/21/17766426/russia-nuclear- powered-cruise-missile-lost-barents-sea)

<comment><USER49> If it was transmitted through the sea, then more or less (depending on where in the Barents) the whole Norwegian coast and then some is between the Barents sea and the Baltic.

And the Norwegian coast is long. Really long.

That said this seems to be airborne.</comment></comment>


Do alpha particle condensates exist in oxygen nuclei? - <AUTHOR> https://journals.aps.org/prc/abstract/10.1103/PhysRevC.98.044601 <comment><USER2> A little too technical. This link may help a little to understand the previous work: [https://en.wikipedia.org/wiki/Carbon-12#Hoyle_state](https://en.wikipedia.org/wiki/Carbon-12#Hoyle_state)</comment>


Ask HN: Based on your experience is the series Silicon Valley close to reality? - <AUTHOR>

Now i know that this is only a comedy series and merely meant for entertainment. But as someone who really doesn't know how the business side of development work, i'd like to know if it'is close to the real thing. And if so, i have to say, it's pretty terrifying <comment><USER2> It's about as accurate as Office Space. The inaccuracy is in the exaggeration, which is what makes it funny. The technical details and terminology are nearly flawless. It does notshow the thousands of quiet hours of work required in the real world.

</comment><comment><USER3> [http://www.nytimes.com/2015/05/05/opinion/joe-nocera- secrets...](http://www.nytimes.com/2015/05/05/opinion/joe-nocera-secrets- doomed-script.html)

</comment><comment><USER4> The business side in my opinion is much more harsh. If only it were a fun comedy. Many companies are much bigger and don't have comedic actors working there. Somewhere between SV and Breaking Bad is real life. Sometimes though I have felt like Jesse Pinkman chained to a desk when someone robots me. Robotting someone is a term I just made up to describe people that treat you like total shit, twisting every screw, but at the end of the day still robotically desire you to develop a product for them. You walk out of the deal or job and their mind breaks because it doesn't compute that their bad behavior is not liked.

</comment><comment><USER5> Obviously being a comedy TV show, there are certain elements of fiction to make it entertaining. However, the show in many ways actually does highlight the state of Silicon Valley or any startup tech mecha area like San Francisco. It is a perfect parody of the current state of startup culture and tech investment. After a recent stint with a large startup in Seattle, I can honestly say there are a heap of elements from the show that mimic real life (from incubators to work until you pass out culture).

</comment><comment><USER6> Spoiler Warnings ahead.

The TV show is (paradoxically?) more vulgar but also more good-natured (being a comedy) than the real thing.

For example, no one would call a VC a "choad-gargling fuck toilet" in a pitch-- VCs share notes more than is appropriate or legal, and have black- balled people for less-- and I doubt that a group of typical startup engineers could pull off a flawless 3-minute dick joke like the one in Season 1's finale. So, the TV version of the Valley is more overtly vulgar. The real thing tends to be crass and exclusionary, but you're not going to hear words like "shit-riddled anal wasteland" or nicknames like "Cunty" very often. The show focuses on superficial abuses that are funny rather than the long-lived, subtle, and often subversive ones that characterize the actual Valley.

Another discrepancy: when Richard achieves a record-breaking "Weissman score", the entirety of TechCrunch disrupt is in awe. In the real Valley, 95% of them wouldn't know what a Weissman Score is. The technical literacy of the people in the show is actually much higher than in the real Silicon Valley. At one point, there's a reference to a "DFT" (Discrete Fourier Transform, I presume) and I'd bet that only 1% of investors and 5% of SV founders know what a Fourier Transform even is.

The show gets far more right than I would have expected, and more right than it gets wrong. One of the things that is true to life is the purely political promotion of Bighead (an incompetent engineer who gets a great job because the CEO of Hooli wants to fuck over Pied Piper). That happens a lot and I'd love to see his career grow and for him to become a VP, a founder, or even get promoted into the investor ranks. The part that isn't true to life (or, at least, very common) is his humility and self-awareness. He knows that he's mediocre; whereas most people who benefit from political or aggressively aspirational promotions tend to believe they actually deserved it, and get quite arrogant.

Another thing that's true to life is Russ Hanneman's unethical use of information, when he poaches an engineer that Pied Piper wants for another company. That may be the core ethical flaw of Silicon Valley: a complete lack of respect for the idea that there are things that one should not do with others' information. That sort of ethical looseness is common and spot-on. Likewise, the herd mentality of the VCs around Pied Piper hits reality exactly.

Finally, you have Erlich Bachmann and Russ Hanneman (inspired by Sean Parker and, some have said, Mark Cuban). Bachmann is a jerk, superficially, but still a bit too good-natured to be a real player, as we've seen in the recent episodes. Russ Hanneman hits the archetype of the Silicon Valley billionaire ("three comma") on the nose. That said, even he's more self aware than most real SV billionaires.

<comment><USER7> > Another discrepancy: when Richard achieves a record-breaking "Weissman > score", the entirety of TechCrunch disrupt is in awe. In the real Valley, > 95% of them wouldn't know what a Weissman Score is. The technical literacy > of the people in the show is actually much higher than in the real Silicon > Valley.

I was also surprised by that when watching the show, but I believe it was Gavin Belson who hyped this metric while presenting Nucleus at Tech Crunch. Most non technical people would have understood the Weissman score as "something that measures how good a compression is, the higher the better, and since G.B says 2.98 [or something] is record breaking, it must be the case.", so when Richard achieved a Weissman score of 5+, the same non technical people concluded that an algorithm with a score of 5+ must be totally revolutionary.</comment></comment>


Any suggestions on what one should think about when scaling a site? - <AUTHOR>

Hi All,
Does anyone know a good resource (website, book, etc) on things to think about when scaling a website? For instance, I am looking for materials on:
1) How a scalable website is organized in terms of infrastructure (having a database server, having a web server, etc)?
2) Best practices for designing the database for a large website to prevent single-points-of-failure
3) What web services out there allow for easy scaling. I have heard of slicehost, Mosso, EC2, Google App Engine (any others? We use PHP)? Do you have to configure your own load balancer etc for these services?
4) Other useful things to think about when designing a scalable web application.
Thanks all <comment><USER2> <http://highscalability.com/> has some fantastic articles.

The best tool to learn how to use is memcached - it has clients in most major languages (And even some obscure ones) and when used properly can remove a lot of the typical problems for you. Part of what I found with memcached is that it forces you to think in terms of encapsulated data. Instead of thinking about your database all the time, the database becomes a tool. What's more important however is consumable bite sized chunks of data that your application is using, and finding ways to get to them quickly.

Stay away from apache: it's far to slow and unwieldy for most scaling. I'm a huge fan of LigHTTPD (<http://lighttpd.net>) but a lot of people also swear by Nginx (<http://nginx.net>).

Depending on your language... find ways to tune away any of the runtime slowdowns such as translation to byte code. With PHP find a byte code cacher, learn it tune and and rely upon it. The LigHTTPD guys make a great cacher called XCache (<http://xcache.lighttpd.net>) which I've used on a few big sites. It plays nice with Lighttpd and is actively maintained.

<comment><AUTHOR> Great. These should help me get started.</comment></comment>