VC from Asia is skyrocketing - <AUTHOR> https://www.wsj.com/articles/silicon-valley-long-dominated-startup-fundingnow-it-has-a-challenger-1523544804 <comment><USER2> I remember The Economist doing a piece on this fairly recently[0] that talked exclusively about how China's tech VCs and established corporations (referring to the likes of Alibaba/Tencent) are challenging (and to a certain extent dethroning) Silicon Valley "monopoly" when it comes to investing in innovation/R&D in super critical areas like A.I., for example.

One huge advantage Chinese corporations and startups have is China's super lax (non-existent, in fact) privacy and data protection laws -- enabling their companies to harvest ENORMOUS amounts of data to hone their A.I. algorithms on -- making them super attractive for investors.

Add to this China's culture of scientific inquiry, rampant IP theft from American and European corporations and startups alike, a stable administration that has made technological advancement it's razer sharp focus -- you have the perfect recipe for VCs' fund to start flowing on to the next big thing with relatively fewer barriers.

[0] [https://www.economist.com/news/business/21737075-silicon- val...](https://www.economist.com/news/business/21737075-silicon-valley-may- not-hold-its-global-superiority-much-longer-how-does-chinese-tech)

<comment><USER3> This may sound naive, but how does IP Theft even work?

I've worked as a software engineer for over 12 years at over half a dozen companies, with codebases numbering in the millions of lines of code and I have yet to see any software or code I could imagine anyone would want to steal. And if you really needed some code, I'm sure it's available somewhere as open source. And wouldn't the open source code be much more preferable? rather than ripping out some piece of "stolen" code that was never meant to serve other purposes.

<comment><USER4> IP that's worth stealing is not limited to one specific implementation, it's about using someone else's _idea_. (So "theft" is a misnomer, it's really about imitation.)

For example, if you buy a competitor's product to take it apart, replicate its components and assemble lots of knock-offs, you have some validation that the product is going to work, without having to do any costly planning and design yourself. All of that works without literally breaking into the competitor's factory to steal their equipment or their code.

For a lot of really valuable IP you won't find an open source implementation, or if you do, you won't be able to legally use it, at least as long as software patents are a thing.

<comment><USER5> It is utterly ridiculous that to take an =idea= is considered theft. In that case, pretty much every American corporation can be accused of theft. Google: Stolen from alta vista, Microsoft: Stolen from CP/M, Apple: Stolen from Xerox, The american space program: stolen from the Nazis! If that's the basis for accusing China of stealing IP, it's a pretty weak argument.

</comment></comment></comment></comment><comment><USER6> I think it's always strange that the media frames this as "threatening" to shift power. On the one hand I agree that I would like to keep America investing in R&D to create the next great innovation that revolutionizes the world, on the other hand I do not fear other countries of the world overtaking us. I frankly think it's awesome to see other countries of the world investing and leapfrogging off of American innovation and some day I would only hope that we can do the same.

I think instead of using attacking vocabulary like "threatening" it would be more worthwhile for the media to say that the US is not keeping up with it's peers.

<comment><USER7> > I think it's always strange that the media frames this as "threatening" to > shift power.

Why is it strange? That's the purpose of a nation's propaganda organizations. The chinese media does the same thing. Every nation's media does it.

<comment><USER8> You're not supposed to call our media propaganda. That's a word that can only be used for other nations' media.

<comment><USER9> Our media isn't propaganda, because it doesn't fit the definition. Propaganda is slanted views that are approved or promulgated by the reigning regime in a country. The news in the US simply isn't like that: just look at how anti- Trump most of it is (not saying this is a bad thing). Propagandized media is where the news media is really nothing more than a mouthpiece for the current regime, as you see in China and Russia, and doesn't air views contradictory to that regime.

This isn't to say that our media is completely neutral. No media really is. But it doesn't represent our current government, it represents the interests of its owners, who have different interests than our current leaders, and are frequently at odds with them. And of course, we have multiple media outlets, who frequently disagree with each other, a good example being Fox News.

<comment><USER10> >slanted views that are approved or promulgated by the reigning regime in a country

[https://www.youtube.com/watch?v=_fHfgU8oMSo](https://www.youtube.com/watch?v=_fHfgU8oMSo)

<comment><USER11> If you dont want to watch: Parent is posting a video of a bunch of pro- government news reports by Sinclair, owner of many local tv stations in the US, which is known to be particularly right-wing (and, if you ask me, bigoted), much like the current administration.

To my understanding, parent's hidden suggestion is that somehow these buffoons are representative of the average American media company.

<comment><USER10> >which is known to be particularly right-wing

This is a baseless claim with zero evidence to support it

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> Everyone is ignoring the fact that China is a _massive_ closed market. Most western services do not care about China. China culture is different and it doesn't seem that the "world" is willing to translate and adapt its services to it.

This means a huge opportunity for local ventures.

<comment><USER13> I'd argue China is actually one of the easiest markets to adapt for, given its size. We're talking 1.3 billion people with the same national language, 92% of which share the same ethnicity (Han). Most Chinese cities are quite consistent in terms of culture and regulations. It's a very homogeneous country with a large middle class

Indian or South-East Asia markets are much more difficult to adapt for. You've got a smaller population with less buying power divided up into more than dozens of major languages, cultures, ethnicities, laws, etc.

Protectionism is the real barrier in China, not Western ignorance.

This is what I experienced at a Chinese startup, where a large majority of the strategy/ops efforts were dedicated to localizing products for SEA, compared to the domestic market.

<comment><USER14> You need connections in China to do anything as a unfair advantages or you will never succeed.

</comment></comment></comment><comment><USER15> I think its prudent here to try to separate out innovation vs state backed and protected interests.

Although I do agree there are some eastern sentiments that are driving some interesting innovation.

Wired documentary on Shenzens rise: [https://m.youtube.com/watch?v=SGJ5cZnoodY](https://m.youtube.com/watch?v=SGJ5cZnoodY)

<comment><AUTHOR> There's one amazing tidbit in the documentary about how China's systems work. Starts around here:

[https://youtu.be/SGJ5cZnoodY?t=3585](https://youtu.be/SGJ5cZnoodY?t=3585)

The system seems so controlled which I don't think is feasible for any other country.

</comment></comment><comment><USER16> everyone already forgot the top story here on HN a few weeks ago?

[https://www.marketwatch.com/story/finance-documentary-the- ch...](https://www.marketwatch.com/story/finance-documentary-the-china-hustle- revisits-chinese-reverse-mergers-and-activist-short-sellers-2018-03-30)

</comment><comment><USER17> Good if true. Competition is always welcome.

</comment><comment><USER18> [http://archive.is/94TdJ](http://archive.is/94TdJ)

</comment><comment><USER19> VC is China is complex, where many VC investment firms are just arms of the Chinese government. I don't think it's fair to compare our VC system to theirs. While the US does some VC investment (In-Q-Tel) they are far more separated.

<comment><USER20> _many VC investment firms are just arms of the Chinese government_

As a serial entrepreneur in China I am really interested in your source here. Care to share?

IMHO based on experience to date I would say there is usually a _very_ clear distinction between government and VC firms.

</comment></comment><comment><USER21> paywall-less: [http://badblue.bitnamiapp.com/sources.php?to=https://t.co/2A...](http://badblue.bitnamiapp.com/sources.php?to=https://t.co/2AF5zdtloU?sid=11393785)

<comment><USER22> Hi, thanks! What does this script do?

</comment></comment><comment><USER23> > _VC from Asia is skyrocketing, threatening to shift power over innovation_

"Threatening" to whom? Not to Asians.

This is an offending us/euro-centric title in a global age...

<comment><USER24> You're assuming the wrong definition of threatening.

It's meant as "to announce as intended or possible".

That's true whether you're Asian or not.

<comment><USER23> > _It 's meant as "to announce as intended or possible"._

In a negative way. See my response below, and the lemmas for 3 major dictionaries...

<comment><USER24> Not necessarily. One could say that a rider in a bike race is threatening to overtake another.

That's not necessarily bad. One could say it's used to say it's unexpected, as I assumed for the quote we're discussing.

</comment></comment></comment></comment><comment><USER25> I got a random idea: what if sites like HN, Reddit, RSS readers, etc. would have netflix-like subscription for articles?

I don't care about WSJ or other paywall publishers. Especially not enough to subscribe. I however do care about occasional story that ends on HN.

<comment><USER9> That's not a bad idea, but it'd mean you'd need to subscribe to HN, Reddit, etc., because WSJ and other paywall publishers are going to want to get their money from somewhere. But I guess it's possible that sites like HN could institute an optional subscription system where you can pay a monthly subscription fee and then that's used to subscribe to these news sites through a special deal for linked articles.

</comment></comment><comment><USER26> "Paywall"

</comment><comment><USER27> freaking hate this paywall crap content on hacker news.

<comment><USER28> [https://news.ycombinator.com/item?id=10178989](https://news.ycombinator.com/item?id=10178989)

[https://news.ycombinator.com/newsfaq.html](https://news.ycombinator.com/newsfaq.html)

Users usually post workarounds in the threads.

</comment></comment><comment><USER29> China?? I heard they were bad though?

<comment><USER28> Would you please not post unsubstantive comments to Hacker News?</comment></comment>


We found only one-third of published psychology research is reliable – now what? - <AUTHOR> https://theconversation.com/we-found-only-one-third-of-published-psychology-research-is-reliable-now-what-46596 <comment><USER2> Sneaking suspicion that this will readily extend to other soft sciences, particularly sociology.

A good chunk of our social and educational policy is informed by psychology and sociology.

Since we have all grown up within various systems and institutions that have been largely informed by bad research, it's unlikely much will happen, since we see the claims made by these charlatans to be normal or common sense.

Nonetheless, it will be amusing watching the charlatans try to defend themselves in the coming years.

<comment><USER3> This seems to me to be a clear cut case of poor incentives in the academic research sector.

No graduate student is incentivized to re perform existing experiments to verify them, no they find much more success in creating a unique experiment which tests a previously untested hypothesis.

This is modern academic science. If you aren't doing something newsworthy, you're not getting any funding.

<comment><USER4> >No graduate student is incentivized to re perform existing experiments to verify them

If you look at how this thing with psychology started ("replication bullies") or at what Brockman said about the LaCour scandal earlier this year (he didn't pursue publication initially because he thought he'd risk his career), it's not that they aren't incentivized, it's that they are actively discouraged.

</comment></comment></comment><comment><USER5> One small suggestion: Eliminate "published and outa here," the policy where students have to publish their MS and Phd work in order to receive those degrees. All that does is to fill the journals with crap.

<comment><USER6> I agree in principle but oftentimes the research done by grad-students is part of a project that is spearheaded by a career researcher. I question the notion that MS and PhD work is filling journals with crap.

Instead, I think there are three problems:

1\. The inherent problem of inference. When you set p < .05, then 5% of studies will yield false positives. This can be mitigated by a variety of statistical approaches, but it's an irreducible problem on a fundamental level.

2\. Certain (sub)disciplines of science aren't very scientific. In particular, they suffer from a high incidence of unfalsifiable claims and hand-wavy definitions. A good example, I think, is the shame vs guilt literature.

3\. Certain (sub)disciplines of science are highly politicized. I point my finger at anything involving "identity" (in the sense of "identity politics"), "diversity" (sex, age, ethnicity, sexual orientation, etc) or "bias".

Regarding point 3, take a look at the faculty and students of any university you'd like and take stock of how many Black people are working on racism, women on sexism, LGBT on sexual orientation, etc. As a cognitive neuroscientist, I don't want to claim that people can't study themselves, but in light of the serious reproducibility issues that social psychology is facing, I think this correlation is anything but trivial.

I strongly suspect that the "hard psych vs social psych" dichotomy can be further broken down, and that the sub-disciplines listed in point 3 account for a disproportionate quantity of false-positive results, on accounts of flawed methodology. I suspect that much of this research is actually an attempt to validate political opinion.

I may be wrong about all of this, but at least the core claim testable with a handful of linear regressions.

<comment><USER5> All good stuff. Thanks. I'm just saying, don't make publication a _requirement_ for graduation.

I'm not a statistician, but as I understand it, a problem with the p value is that the standard calculation hinges on assumptions that are not necessarily met by the data, and if those assumptions were correctly accounted for, p would be higher.

If p < 0.05, and in practice 50% of published results are bunk, then regardless of the underlying math, a practical rule of thumb seems to be: Multiply p by 10.

<comment><USER6> The practical rule of thumb for a p-value is that it's nothing more than the rate of expected false-positives. p=.05 means you can expect 5% of false positives.

If more than 5% of your studies are failing to replicate, then something else is involved, but it's not necessarily fraudulent. Post-hoc hypotheses are usually the biggest culprit, but it's insidious to the point that researchers often don't realize they're doing it.

</comment></comment></comment></comment><comment><USER7> Let's be clear on the differencen between _reproducible_ and _replication_. Reproducible is when an experiment's data and methods are published and other researchers can reproduce the same statistical results. An experiment that has been replicated is a _new_ experiment that applies the same methods to a new set of subjects and arrives at statistically equivalent results.

The distinction is important. Reproducible research is a lot easier but there are still famous examples of attempts to reproduce published results that revealed serious flaws. Cases where the data are kept secret to thwart reproducibility are thoroughly suspect. Replication is a harder problem, both in being able to perform a potentially costly and difficult experiment, and in the statistical analysis to show equivalent results.

<comment><USER8> >"Let's be clear on the difference between reproducible and replication."

I did not find this clear. What is the distinction you are trying to make?

<comment><USER7> Reproduce: take the published data and methods, rerun the analysis, see if you get the same results. No new experiment.

Replicate: perform a new experiment with new subjects, apply the same methods, see if you get statistically equivalent results.

<comment><USER8> Thanks, I had not heard that distinction in terms before. I suppose it would be good to have, but I would not assume people are using them in that way.

<comment><USER7> Some people are making a lot of noise about making the distinction, like the guys teaching data science at John Hopkins. It's because both are important but they are distinctly different activities.

</comment></comment></comment></comment></comment><comment><USER9> A friend of mine is a social psychology researcher, who's published work destroying the statistics in other papers. He says this claim of unreliability is a lot more dubious than the media portrays.

In many cases, they say a finding was not reproduced simply because the new study didn't quite meet the cutoff for "significance." However, if you do a meta-analysis of both studies, you find that the likelihood of the effect being real is higher, rather than lower.

In other cases, the replications were done poorly. The first step of many studies is calibration. You do tests to find out what sort of things your subjects are familiar with, and calibrate questions accordingly. A common mistake in the replications was to skip the calibration step, and simply reuse the questions on subjects with different backgrounds.

<comment><USER10> So we need the original authors to come back with their own re-replication. Instead of throwing their published paper over the wall and moving on, they are the people in the best position to actually defend it and address problems with replications.

This publish-and-forget behavior annoys me about researchers. Sometimes I find results that look like they'll useful for my (non-academic) work, but the paper containing them is the final say on the matter. There's no version 2, there's no other work improving on it. It's just a dead end.

</comment></comment><comment><USER6> I actually have an answer for this:

Stop talking about "psychology" like it's a monolithic thing. Start by separating "hard" psychology (perception, attention, psychophysics, etc) from social psychology.

Papers about saccadic adaptation are, I suspect, much more reproducible than those about implicit association.

<comment><USER11> One thing I noticed is that the more focused the subset of psychology, the better the results. For example psychology that looks at a group of neurons and how they function is vastly better than psychology looking at how some early life event X leads to later problems in life.

<comment><USER12> Isn't that really neuroscience?

<comment><USER6> When neuroscience involves predicting behavior and/or mental state, it usually falls under the umbrella of "cognitive neuroscience", which in turn is a way of saying "psychology that's not soft as all hell".

The boundaries are fuzzy, and you'll find people with psych, medical, neuro, math, CS and philosophy degrees in experimental cogsci labs.

</comment></comment></comment></comment><comment><USER8> Now what? You need to attempt independent replications of every published claim going forward. It is clear a single published result is unreliable. This is well known but apparently needed to be rediscovered by those who misunderstand the meaning of a p-value.

>"The first is a p-value, which estimates the probability that the result was arrived at purely by chance and is a false positive. (Technically, the p-value is the chance that the result, or a stronger result, would have occurred even when there was no real effect.) Generally, if a statistical test shows that the p-value is lower than 5%, the study's results are considered "significant" – most likely due to actual effects."

No, there is endless literature on this. Just to start:

[http://andrewgelman.com/2015/07/21/a-bad-definition-of- stati...](http://andrewgelman.com/2015/07/21/a-bad-definition-of-statistical- significance-from-the-u-s-department-of-health-and-human-services-effective- health-care-program/)

<comment><USER7> More to the point is how "replication" is defined. This quote covers it: "Note that the 36% figure comes from a definition of replication that mimics the definition used by regulatory agencies: results are considered replicated if a p-value < 0.05 was reached in both the original study and the replicated one."

If you read the post where that quote comes from [1] they make a number of points about how a better and more rigorous definition of replicated is needed, because p-value alone doesn't tell the whole story.

[1] [http://simplystatistics.org/2015/10/20/we-need-a- statistical...](http://simplystatistics.org/2015/10/20/we-need-a- statistically-rigorous-and-scientifically-meaningful-definition-of- replication/)

<comment><USER9> So say one study has a p-value of .04, meaning there's a .04 probability that there's no effect and the results occurred by chance. A replication comes along and gets a p-value of .06, so it gets counted as a failed replication. And yet, the probability that both results happened by chance is only .0024.

<comment><USER8> >"there's a .04 probability that there's no effect and the results occurred by chance."

No, the p value is the probability of observing a result at least as extreme as your own given the null hypothesis is true. There are two errors here

1) Transposing the conditional: P(A|B) != P(B|A) [http://rationalwiki.org/wiki/Confusion_of_the_inverse](http://rationalwiki.org/wiki/Confusion_of_the_inverse)

2) Deviations from the null hypothesis can occur even the absence of a treatment effect, ie one of your model assumptions is wrong, baseline differences, etc.

Anyway, that is why instead of statistical significance researchers need to estimate the size of the effect. If you estimate the effect is in the range -1 to +2 then you publish that result. Others also estimate the range and see if these are consistent with each other.

<comment><USER9> > probability of observing a result at least as extreme as your own given the > null hypothesis is true.

My wording was poor but that's what I meant.

<comment><USER8> They are two completely different things. Check out the link. Would you accidentally say "It's cloudy therefore there is a high probability of rain" when you meant "It's raining so there is a high probability of clouds"?

</comment></comment></comment></comment></comment></comment><comment><USER13> Reach for an extra-large grain of salt whenever the nightly news trumpets some new breakthrough in the psychological field.

Continue avoiding psychologists whenever possible.

</comment><comment><USER14> Redistribute most of the $5 billion dollars of psychology and behavioral- related NIH research funding to historically high ROI areas like infectious disease research.

[http://report.nih.gov/categorical_spending.aspx](http://report.nih.gov/categorical_spending.aspx)

</comment><comment><USER15> From 10 years of working as a sysadmin in a Psych department, one constant in life I've learned is that Psychologists are the most incompetent, fanciful egoists in all the sciences. They have no idea what is going on basically. Or, put as a friend of mine said, "People who go into Psychology are precisely those that do not intuitively understand it already." Implying that most of us have a decent, commonsense appraisal of how things work, but psych people are the most clueless of the clueless, that's why they go into a field (first falling for its false promises) that professes to explain life's mysteries to them. None of this is surprising, except perhaps that they scored so far away from the median.

</comment><comment><USER16> This is quite frightening considering how people try to set government policies based on them.

</comment><comment><USER17> The reliability of Psychology throughout the 20th century is covered in depth in the book The Culture of Critique, where the exact persons responsible for fostering in Psychology an anti-Science environment are discussed.

</comment><comment><USER18> now what ... Accurately label psychology and the social sciences as philosophy not science.

</comment><comment><USER19> Far more worrying is that only a third of economics findings can be replicated without the author's help, and yet these are the supposedly scientific findings our politicians insist on basing policy off of. Even if policy seems counterintuitive and clearly harmful to the general public, we're given the explanation "because economics says so." It's clear now from the Federal Reserve replicability study that we're being duped.

[http://www.federalreserve.gov/econresdata/feds/2015/files/20...](http://www.federalreserve.gov/econresdata/feds/2015/files/2015083pap.pdf)

<comment><USER20> Economists are the modern equivalent to the oracles at Delphi. They speak in gibberish and the priesthood (politicians) make the interpretations which coincidentally happen to favour themselves and their friends.

<comment><USER21> I feel there's truth to that, but most at the edges. At core there is stuff economists tend to very widely agree on. There was a Planet Money episode outlining some pretty drastic policy changes that a panel of economists across the political spectrum all stood behind. Here's a write up: [http://www.npr.org/sections/money/2012/07/19/157047211/six-p...](http://www.npr.org/sections/money/2012/07/19/157047211/six- policies-economists-love-and-politicians-hate)

</comment></comment></comment><comment><USER22> It's not just psychological research that is deeply flawed. Only a quarter of scientific drug research is successfully reproduced as well.[1]

Carl Jung claimed one of the chief factors responsible for mass brainwashing is scientific rationality.[2] Society worships the Goddess of Reason while frowning down on "irrational" and non-verifiable religious testimony.

Now that science is proven to be systemically corrupt, what will "rational" people base their understanding on?

Aside: I designed a personality test / psychoanalytical tool that was inspired by Carl Jung. It's called Critical Stimulus and it can be found at [https://www.thegamecrafter.com/games/critical- stimulus](https://www.thegamecrafter.com/games/critical-stimulus). The printable version can be downloaded at gumroad: [https://gumroad.com/l/criticalstimulus](https://gumroad.com/l/criticalstimulus)

.

[1] [http://www.economist.com/news/briefing/21588057-scientists-t...](http://www.economist.com/news/briefing/21588057-scientists- think-science-self-correcting-alarming-degree-it-not-trouble)

[2] [https://en.wikipedia.org/wiki/Self_in_Jungian_psychology](https://en.wikipedia.org/wiki/Self_in_Jungian_psychology)

<comment><USER6> >Now that science is proven to be systemically corrupt, what will "rational" people base their understanding on?

This is not a new notion in the least. The "rational" people will continue doing what they've always done: revising their conclusions.

Science is a process, and while we can question the notion of "scientific progress" in philosophical terms, this is neither a new idea nor evidence that science doesn't work. It's certainly not evidence that science is no better than irrational thinking.

>Carl Jung claimed one of the chief factors responsible for mass brainwashing is scientific rationality.

Few people take psychoanalysts seriously these days, in large part because of their long record of absurd claims and shoddy clinical work. Jungian theory has it's place in a conversation about literary theory, but not in a conversation about science.

<comment><USER12> I don't know about psychoanalysis as defined by Freud and Jung, but therapy in general is an intensely personal experience.

I'm sure there are scientific aspects that can be brought to bear on a situation, but for a lot of people the "literary theory" part is just as helpful. Especially when you stop to think how much neurosis is fueled by pop culture (i.e. status envy).

<comment><USER6> >but therapy in general is an intensely personal experience.

And empirically speaking, psychoanalytic therapy has a piss-poor record in dealing with mental illness.

If you're looking for spiritual guidance, then _maybe_ a psychoanalyst can help. If you're looking for clinical efficacy, they demonstrably don't.

I'll leave it as an exercise to the reader to make a relevant Google Scholar query, but be careful not to confuse _psychoanalysis_ with _clinical psychology_.

> for a lot of people the "literary theory" part is just as helpful.

Fine, but this is a conversation about psychology as a science.</comment></comment></comment></comment>


Sysadmin mistakes start-ups make - <AUTHOR> https://www.cloudkick.com/blog/2009/oct/20/three-start-up-sysadmin-mistakes/ <comment><USER2> Here's one we made recently: Purchasing an array of hard drives (for storage servers) and not making sure that not all of them are from the same batch. Since they were made in the same batch, they had the same defects and when they failed, they failed one after each other in a very short interval. Since all of them failed, RAID didn't help, we had to restore the day-old offline backup.

<comment><USER3> That isn't something I'd ever thought about until now.

Would it make sense to use drives from more than one company as they would very different failure characteristics?

<comment><USER4> It's a terrible idea to mix and match drive models, because they'll have drastically different _performance_ characteristics, and if you're lucky you'll only get a little worse than the lowest common denominator on all axes.

What quality OEMs do is make sure they never ship you drives from the same manufacturing batch in one enclosure.

<comment><USER5> How do you make sure to buy from a quality OEM?

</comment></comment></comment></comment><comment><USER6> Fork is actually a very fast system call. It never blocks, and (on Linux), only involves copying a very small amount of bookkeeping information. If you exec right after the fork, there is basically no overhead.

However, forking a new shell to parse "mv foo bar" is more expensive than just using the rename system call. And it's easier to check for errors, and so on.

SQLite is also not as slow as people think it is; you can easily handle 10s of millions of requests per day with it. If your application's semantics require table locks, MySQL and Postgres are not going to magically eliminate competition for locks. It's just that they both pick very weak locking levels by default. (They run fast, but make it easy to corrupt your data. Incidentally, I think they do this not for speed, but so that transactions never abort. Apparently that scares people, even though it's the whole point of transactions. </rant>.)

Most of my production apps are SQLite or BerekelyDB, and they perform great. I am not Google, however.

<comment><USER7> > SQLite is also not as slow as people think it is; you can easily handle 10s > of millions of requests per day with it

Scaling relational databases to the point of 10s of millions of requests is extremely non-trivial. Unless you can show me personally or can show me evidence otherwise, don't make this claim. You're doing a disservice to the people that have worked countless hours to eke every last millisecond of performance out of MySQL and Postgres.

<comment><USER6> 10 million per day is about 100 per second. SQLite performs about this quickly. I wrote a test script and it did 125 (unindexed) lookups per second. Then I ran two of these tests at the same time, and the rate stayed about the same. I have 8 cores, so I made 8 processes, and it was the same. 125 requests/second * 8 * 86400 seconds/day = 86_400_000 requests per day.

I added another thread writing as quickly as possible to the mix (7 readers, 1 writer), and this brought the read rate down to about 45/reads per second per thread. Still more than 10 million per day, so technically I am right.

Also, I don't doubt that MySQL and Postgres (and BDB) are both significantly faster than this. It's just that SQLite is not going to guarantee "instant failure" of your project, as the article implies.

(One thing to note -- every time you type a character in Firefox's address bar, you are doing an SQLite query. It is Fast Enough for many, many applications.)

<comment><USER8> That -is- fast, but I still have trouble reconciling that deep down in my computer, a human readable SQL query gets built, and then another process parses that SQL. Seems so wasteful building and then parsing a human readable string for something that's happening on the same machine.

I know nothing of SQLites's internals, but wouldnt it make more sense to parse the query once and then store a compiled version of the query for subsequent lookups? Like you might do with a regexp?

<comment><USER9> Yes, This is known as a prepared statement. You compile a parametrized statement once, then execute it as many times as you like with different arguments.

Also, SQLite, unlike most other databases, is an embedded database which does everything in-process rather than invoking multiple processes.

</comment></comment></comment></comment></comment><comment><USER10> The memory use is not accurate unless you take shared pages into account. Copy-on-write will make it look like each apache child is using 40MB, when really it's only 10MB private RSS. Use a RSS-calculating script (<http://psydev.syw4e.info/new/misc/meminfo.pl>) to determine the close-to- real memory use. If you don't calculate your maximum memory use correctly you will run into swap with traffic peaks. Also keep in mind that swap is a _good_ thing. Is your app constantly cycling children? This isn't going to allow it to move unused/shared memory into swap. Don't ignore memory leaks by reducing your max requests per child.

The forking thing is more of the same. Copy-on-write means it's not going to balloon your memory unless some function turns that shared rss into private. It isn't something that you want to do a lot of, though.

<comment><USER11> This stood out to me as well. I like the script to actually calculate real usage. Modern operating systems are smarter than I am when it comes to memory management.

What you don't want is to have anything you use more than once a minute in swap, and preferably only the stuff you don't plan on using for an hour (i.e. not any time soon). That probably means you want your main application and web server in memory all the time. If there are pieces of it that are unused and you're hitting a resource cap then you have something mis-configured.

RAM is also dirt cheap right now, making it often easier to add RAM than to optimize slightly sloppy code.

</comment></comment><comment><USER12> One of the most common problems we see is DNS misconfiguration. It seems most folks just haven't read the grasshopper book. If you're doing anything on the Internet, you _need_ a basic understanding of DNS.

Once you grasp the fundamentals, most DNS problems become completely transparent, but I've seen people spend _weeks_ trying to solve DNS problems due to lack of understanding.

<comment><USER13> Could you name that book? Do you have know of any other books people should read for sys-adminning?

<comment><USER14> DNS and Bind (now in its 3rd edition) by Liu, Albitz, and Loukides. It's an O'Reilly book.

<http://www.amazon.com/DNS-BIND-Cricket-Liu/dp/1565925122>

<comment><USER15> Amazon shows it's actually in its 5th edition now:

<http://www.amazon.com/DNS-BIND-5th-Cricket-Liu/dp/0596100574>

<comment><USER14> Good call :) Too late to edit :(

</comment></comment></comment></comment></comment><comment><USER16> In my experience of the most common mistakes is the failure to realise that on pretty much all Linux distros, services like Apache and MySQL come conservatively tuned. This is deliberate; it means a DoS or out-of-control process within one of those domains is unlikely to take out the entire server, because there's a hard limit on consumption of memory, CPU, child processes, threads, etc.

However, this default configuration needs to be tuned to allow you to take advantage of the hardware - if you have generous hardware. Otherwise, you will wonder why your web sites are extremely unresponsive, yet the server load stands at something relatively unimpressive.

I found this out the first time a blog post on one of my servers got digg'd.

</comment><comment><USER3> I'd guess the real number one mistake is insufficient paranoia about backups.

I know lots of companies doing TDD but that have never done a full test restore from their backups.

<comment><USER17> The easiest way to do this is to make your backups the mechanism by which you refresh your Dev/QA environment from Production. It means your Ops team are very nearly doing a DR exercise every week.

<comment><USER18> I'd never heard that advice before - sounds like a great idea.

</comment></comment></comment><comment><USER19> This was a great article, but I ended it wondering whether they either (a) knew what a system call was (until the end, I thought maybe they meant a system() shell-out) or (b) realize how many system calls a vanilla request/response cycle incurs.

</comment><comment><USER20> I disagree with 1.3. "Serving static content is the easiest possible task for any web server." Yes, but keeping connections open for slow clients (esp with KeepAlive on) is not a good use of your 500MB Mongrel process' time. On the other hand, KeepAlive is a handy thing to have.

Using a proxy like nginx or varnish to serve static files (and even dynamic data) if you have the proper KeepAlive and Nagle bits flipped can save you a _lot_ of server resources at the application layer.

<comment><USER21> It's almost always a bad idea to use anything other than a non-blocking/async server to handle static content.

I think it's simpler/easier (maybe faster) to serve content from a separate sub-domain (static.site.com or whatever). Using a reverse proxy works too, but unless you're caching dynamic content it's probably no benefit and it's less efficient.

<comment><USER22> A good reverse proxy will buffer client and server side so that your heavy app can be available to serve the next request whilst the light proxy feeds the page back to a slow client.

Under certain circumstances serving static files from separate hostnames can be beneficial as HTTP clients are supposed to limit the number of simultaneous connections per hostname.

</comment></comment></comment><comment><USER23> Yep, #1 happened to me the other day. We hit our Apache server limit of 256 and the site slowed to a crawl. I'm not really sure what was causing the load to be like 50-90, but requests were quite delayed waiting for an open process (keepalive was at 5 secs).

Indeed, my first idea was indeed to install nginx for images really quick. However, I have no experience with nginx. Thankfully, we had a spare server and I offloaded the images to there for now... Throwing more hardware at the problem usually works.

</comment><comment><USER24> FTA:

 _However, sqlite should never be used in production. It is important to remember that sqlite is single flat file, which means any operation requires a global lock_

I don't know jack about sqlite's locking architecture or scalability, but this statement is just silly. There are a conceptually infinite number of ways to make fine-grained locking work on a single file, both within a single process, a single host, or across a network. Maybe the author is thinking fcntl() locking is somehow the only option.

I guess the corrolary to this article has to be "Don't let your startup's sysadmins diagnose development-side issues."

<comment><USER25> SQLite locking: <http://www.sqlite.org/lockingv3.html>

""" An EXCLUSIVE lock is needed in order to write to the database file. Only one EXCLUSIVE lock is allowed on the file and no other locks of any kind are allowed to coexist with an EXCLUSIVE lock. In order to maximize concurrency, SQLite works to minimize the amount of time that EXCLUSIVE locks are held. """

But compared to something like MySQL w/ InnoDB (or postgres, or Cassandra, or BerkeleyDB), which all have something closer to Row Level or Page Level locking, SQLite's concurrency for server side applications is a serious deficiency.

Yes, there are lots of ways to have fine grained locking, SQLite just doesn't do them.

<comment><USER26> Like many of SQLite's other quirks, this is because SQLite is designed to accommodate embedded usage.

</comment></comment></comment><comment><USER27> How are the last two system administration problems?

</comment><comment><USER28> This seems like an odd section of sysadmin mistakes - I would have thought there are some other ones being made more often.

<comment><USER29> Especially since the third one is a developer mistake that, as a sys admin and developer, I've had to point out to developers not to do -- but for security reasons, not because fork is oh-so-super expensive (even though it can be).

Also, there is no "system" system call. "system" is a library call that forks and execs a shell to evaluate and execute a string. Having a sys admin that doesn't know the difference may be the biggest sys admin mistake you could make. There are a lot of library wrappers for system calls, but these are documented in section 2 of the man pages as system calls.

</comment></comment><comment><USER30> I'd say their biggest mistake is usually not hiring a sysadmin who also has development experience (or developers without sysadmin experience). I've found that my knowledge in both realms has been invaluable in determining how to design the infrastructure and how to write the code.

</comment><comment><USER31> _If you fork inside an app server, such as mod_python, you will fork the entire parent process (apache!). This could happen by calling something like os.system("mv foo bar") from a python application._

I nominate this post as the most distressingly important bit of information I've ever received at 2:43 AM in the morning.

Now the question: what can I do in Ruby to avoid the four calls a second or so I'm currently making to system(big_command_to_invoke_imagemagick) ?

<comment><AUTHOR> The solution to use an image processing library such as RMagick, <http://rmagick.rubyforge.org/>

<comment><USER19> Calling into RMagick/ImageMagick from inside the request/response cycle is probably even worse than shelling out, because ImageMagick does grievous damage to your runtime.

<comment><AUTHOR> I guess it all depends how you design it and what you are doing. I would have to agree with others, the out of request cycle image processing solutions are definitely the right way to go overall.

</comment></comment></comment></comment><comment><USER32> hmm, I have a hard time understanding why anyone would try to use sqlite in production unless they explicitly wanted to?

<comment><USER33> Simpler to use (no external program to start and monitor) and to backup (just copy the sqlite file)

</comment></comment><comment><USER34> Personally I don't think we would ever run into those issues. A. we don't have other servers to switch over to B. We are using MySQL for testing and development and C. we don't like what happens when we make system calls for within a web app. forget about forking.

</comment><comment><USER35> Take #1 and generalize it to the mistake of trying to fix a problem without really understanding what the problem is. This has to be the most common mistake I've seen in the sysadmin world.

</comment><comment><USER36> Is this a example of a knowledge level of modern sysadmin? If so, we're in trouble. =)

Sysadmin should be able to think in terms of data flows, which means memory management, data partitioning, and network stack usage, able to put different types of data into different kinds of storage, and understand the role of cache and how data should be access.

Packages are just a tools.</comment>


Elitism of start-ups - <AUTHOR> http://blog.varunarora.com/2013/elitism-of-start-ups/

<comment><USER2> I believe startups arise from a need - and that can be cool and funky, or it can be something solving real problems. What matters is that humans will keep innovating no matter what the field is

</comment><comment><USER3> As shallow and horrible as it is, the author is going to lose some people simply for using "start-ups" instead of "startups."</comment>


Spain builds submarine 70 tons too heavy due to wrong decimal (2013) - <AUTHOR> https://o.canada.com/news/spain-builds-submarine-70-tons-too-heavy <comment><USER2> I'm terrified of making one of these 'big picture' mistakes. I've long been weary of the fact that there is a surprising amount of engineering going on without scrutiny or peer review. And over reliance on computer modelling and increasing complexity makes it less and less easy to grasp the big picture.

Just yesterday I made a trivial error crunching some numbers for when a construction site could turn off their dewatering. Quite the opposite of this submarine, my structure would be 1000s of kips too buoyant and could potentially float up out of its excavation due to hydrostatic forces. A mistake like going unnoticed could easily be a career-ender.

<comment><USER3> _And over reliance on computer modelling and increasing complexity makes it less and less easy to grasp the big picture._

More importantly, this also diminishes order-of-magnitude/approximation thinking, which is a very important source of "common sense". I've graded physics/electronics exam questions, and it's astonishing/amusing just how far- off some of the wrong answers are. Some memorable examples:

\- Asked to calculate the current-limiting resistor for an LED in a low- voltage circuit, some students gave answers in the megaohms, others in the miliohms (correct answer was in the low kiloohms.)

\- A question about the engine power of a dragster resulted in wrong answers of over a GW (roughly a million horsepower!) and under 1kW. (Correct answer was around 1MW.)

<comment><USER4> I find that students routinely divide instead of multiplying, especially if they are asked to solve for something and then calculate the result. For example, $f=ma$ gets transformed to $a=fm$ instead of $f/m$, yielding a large error unless $m$ happens to be near 1kg. Checking units is good way to check for incorrect formulae, but I've found that students who mix up division and multiplication are also uncomfortable with units. The use of calculators cannot explain lack of understanding of units, so I think the problem is more basic.

I've known school teachers, and am somewhat familiar with curriculum. Both are basically fine, and neither explains the problems I see with students in university.

It seems to me that the root problem is that students are never told, in serious terms, that getting the right answers is important. The way we signal importance is with grades. Students who have received passing grades for incorrect work through many years of school can hardly be expected to show attention to detail in university.

America (et al.) may be throwing away a generation of talent, by pursuing what seems to be a policy of unconditional approval. Luckily, there is some hope: my foreign-trained students hand in correct work, with appropriate digits and units. Quite often, they also show how they have checked the work. I don't think it's because their first language was French, or German, or Mandarin; I think it is because they went to schools where learning was still the focus, as opposed to the congratulation for simply _being_ , not _accomplishing_.

<comment><USER5> It's interesting that it's been common to see this exact same opinion (a lack of conscientiousness on the part of students) expressed in Germany about German education over the past 10+ years yet you cite German students as a good example.

In Germany it's commonly stated that things were better previously and that there's a ongoing decline.

My understanding in the German case is that it's not supported in any measures of student attainment - grades, earnings after 10 years etc.

<comment><USER6> It is not necessarily a contradiction, it is possible that German education is now clearly worse than 10 or 20 years ago, but still not as bad as the current situation in grandparent's country.

<comment><USER7> I think it's regression to the mean. Parents with lots of education see their kids struggling and don't understand that they themselves were positive outliers back then.

<comment><USER8> Also that rich kids think riches come for free. Boondocks to boardroom in 3 generations, and three generations back. Rags to riches to rags. Shirtsleeves to shirtsleeves. Clogs to clogs. Rice fields to Rice fields.

</comment></comment></comment></comment></comment></comment></comment><comment><USER3> _Apparently somebody in the calculations made a mistake in the very beginning and nobody paid attention to review the calculations._

This is common in projects of such scale, where everyone is so specialised that no one has a general idea of the whole project and could thus have raised a "something feels wrong" feeling.

<comment><USER9> Isn't that what systems engineering is for?

<comment><USER10> Only if you hire them.

</comment></comment></comment><comment><USER11> Any good tricks for catching these sorts of errors?

I'm a theoretical fluid dynamicist. When doing math I'll typically check the dimensions, order of magnitude, bounds, functional dependencies (anything missing or something there that shouldn't be?), overall trends of an equation, and limiting cases. For solutions to equations it's useful to also plug the apparent solution back into the equation. Another check ideally comes after waiting a while so that my mind is fresh. Sometimes what I did seems dumb by then.

Coding up the math is useful too. I've found that I can never add too many assertions. I might have caught the error described in the article as I often use assertions to check bounds.

Peer review isn't as reliable as I'd like. When it comes to math, most people seem to "rubber stamp" it. I can recall reviewing a paper that had math problems that were fairly obvious to me. For some reason I got the reports from the other reviewers, and I was the only one to notice...

<comment><USER12> Intuitively, only store numbers in their natural dimension. The same thing regularly happens in finance where interest rates (1.5% ie 0.015) is often stored as percent (1.5) in databases. And of course you never know what conventions people take, particularly in these days of super low interest rates. Then you have spreads which can either be stored in basis points (10bps), percent (0.1%) or natural (0.001). And currency of amounts, etc.

All table columns should also have unit in names, whether in charts, tables, database or variable names.

<comment><USER13> > _All table columns should also have unit in names, whether in charts, > tables, database or variable names._

Oh yes. That would mitigate a lot of stupidity. Imagine if math in programming and Excel had to have both a) always visible/known units, b) unit propagation and c) unit checks. So if A is seconds and B is kilograms, the compiler/spreadsheet wouldn't let you write C = A + B.

<comment><USER14> Blockpad ([http://www.blockpad.net](http://www.blockpad.net)) is the answer to your prayers.

</comment></comment></comment></comment><comment><USER15> The real joke is that Navantia which is building this submarine actually stole the intellectual property from their French counterpart during a previous joint venture [0] and then managed to fail building a submarine with it.

Having worked in the field, the idea that someone can just misplace a decimal point is laughable. It highlights a complete lack of quality control and engineering know-how from Naventia.

The issue here is not the calculation. Mistakes happen. The issue is procedural. You can't hope to build submarines if you have no process in place to systematically catch this kind of error. That's true at every level from conception to manufacturing.

That they ended up having to go see the Americans because the French wouldn't help them say a lot about European defense integration however.

[0] [https://globalarbitrationreview.com/article/1029783/submarin...](https://globalarbitrationreview.com/article/1029783/submarine- dispute-runs-aground)

<comment><USER16> IDK why this keeps coming again and again. I'm sadly in my phone but this is actually false. The client asked for changes mid construction phase so they had to redesign it. S81 and following ones already include this changes.

I'm not able to read your link but it's probably part of this FUD that comes over and over. Navantia and DCNS had a joint venture, but the requirements of the French and Spanish clients diverged so Navantia decided to go its own way. What you can find is that DCNS tried to sue Navantia back in 09, but AFAIK nothing came out of it (at least yet).

Navantia didn't have experience designing submarines from scratch but they have plenty with civil and military ships. The fact that such "laughable" mistake was apparently just made in this project is interesting to say the least.

Not to mention that it's not the first time that Navantia faces accusations without proof.

IMO this looks like DCNS trying to block their competitor, that not only got its new founded expertise in submarines from you but also knows your commercial channels, relationships etc.

<comment><USER15> > French and Spanish clients diverged

There are no French or Spanish clients. The Scorpene submarines were built strictly to be exported by the joint venture. Spain was expected to buy some, France wasn't.

As far as I know, France only uses nuclear propulsion for its submarine and doesn't export the submarines used by its navy.

> AFAIK nothing came out of it (at least yet)

It was settled. Navantia entirely gave up the export rights of the Scorpene. For a submarine built to be exported, that seems pretty damning to me.

> Navantia didn't have experience designing submarines from scratch but they > have plenty with civil and military ships.

Submarines are a lot more complicated to build.

Most of the Scorpene engineering came from France. Navantia clearly wasn't ready to be left on its own.

<comment><USER16> > There are no French or Spanish clients. The Scorpene submarines were built > strictly to be exported by the joint venture. Spain was expected to buy > some, France wasn't.

Not french and spanish clients for Scorpene, but a Spanish client for Navantia is the Spanish Navy for example, and DCNS had their own.

> It was settled. Navantia entirely gave up the export rights of the Scorpene. > For a submarine built to be exported, that seems pretty damning to me.

Hmm, do you have any source, I've tried but didn't find anything.

> Submarines are a lot more complicated to build.

> Most of the Scorpene engineering came from France. Navantia clearly wasn't > ready to be left on its own.

That's true, although my point is that they are not some bunch of amateurs that misplaced a decimal. They've been designing ships for a long time, and building submarines at least from the 60s, so the decimal displacing is probably FUD, specially when the client changed requirements mid-construction phase, and the source of such mistake is Rafael Bardaji, who seems that has never been a Spanish Official but conservative and later far-right advisor, and honestly, is quite a fishy source.

<comment><USER17> > Hmm, do you have any source, I've tried but didn't find anything. In french > you can find it: > [https://www.google.com/amp/s/www.lesechos.fr/amp/444277](https://www.google.com/amp/s/www.lesechos.fr/amp/444277)

<comment><USER18> So the spanish company was building a submarine for their own clients, they allied with the americans Lockheed Martin for electronics, and the french company parted ways avec le coeur brisé.

This things happen all the time, and nous aimons nous voisins du Nord plus qu'ils le croient. We enjoy all those silly jokes about spaniards in any case.

</comment></comment></comment></comment></comment></comment><comment><USER19> _The Isaac Peral, the first in a new class of diesel-electric submarines, was nearly completed when engineers discovered the problem. A U.S. Navy contractor in Connecticut, Electric Boat, has signed a deal to help the Spanish Defence Ministry find ways to slim down the 2,200-ton submarine._

In case you were wondering (I was) - the entire sub is 2,200 tons.

<comment><USER20> Slightly off topic, but my dad worked on submarine engineering at that E.B. facility, and in related subcontractor / government roles, in the 1970s - 1990s.

He had lots of stories about major oversights making it WAY farther into the production process than you'd expect.

</comment></comment><comment><USER21> It's like that essay about top-down vs bottom-up. It was written about Challenger but same principles apply.

[https://science.ksc.nasa.gov/shuttle/missions/51-l/docs/roge...](https://science.ksc.nasa.gov/shuttle/missions/51-l/docs/rogers- commission/Appendix-F.txt)

<comment><USER22> I get an ERR_CONNECTION_REFUSED accessing that URL. Do you have another link?

</comment></comment><comment><USER23> For scale, the submarine is 2,200 tons total, so it's a little over 3% overweight.

<comment><USER24> Which is more than enough to put it out of balance, depending on where it is.

<comment><USER25> Apparently the fear is much buoyancy issues than balance, the extra 70t means even with empty ballasts the sub might not be positively buoyant.

That's why this could be solved by lengthening the hull, increasing displacement and thus compensating for the extra weight.

</comment></comment></comment><comment><USER26> lol, does the Navantia logo look like a wavy equals sign, ie 'approximately equal'? I think it does. It's honest anyway.

</comment><comment><USER27> runs great in extremely dense water

<comment><USER28> Dead Sea Squadron will be equipped with Spanish S-80 submarines.

</comment></comment><comment><USER29> Was this kind of error common in the slide rule era? I never heard that it was.

<comment><USER30> Yes. What the slide rule era didn't have was the internet.

EDIT: Here's a 1992 GAO acquisition study[1] on the AN/BSY-1 combat system whose development started in 1980 and was an absolute clusterfuck for the Navy in the decade that followed, as history would have it. I originally learned of this disaster from a former sailor (and now good friend) who participated in sea trials for one of the submarines that this system was installed in. By his account, the submarine had had its hull recut just to get the system installed and routed properly, and it never really worked as intended. The study vaguely alludes to this:

> _As a result, the shipbuilder was eventually faced with rip-outs, recutting > of hulls, and rerouting of cables._

Electric Boat was the prime and responsible for system integration. They've had their fair share of glorious fuckups to the tune of $82.4MM in late 80s dollars, so it's funny to read the hero part on how they've "helped other countries with their submarine programs".

In the right professional circles, stories like these quite literally abound. Now what's the probability of learning about them without the internet?

[1] [https://www.gao.gov/assets/nsiad-92-50.pdf](https://www.gao.gov/assets/nsiad-92-50.pdf)

<comment><USER29> Thanks! Probably we only hear of the successes.

</comment></comment></comment><comment><USER31> The later development is even more of ... issues:

[https://www.popularmechanics.com/military/navy- ships/a224737...](https://www.popularmechanics.com/military/navy- ships/a22473758/spains-newest-submarine-is-too-big-for-port/)

"Spain's Newest Submarine is Too Big For Port" (July 2018)

"In 2013, ten years after the first boat was ordered, authorities detected a critical flaw in the design: The submarines were 75 to 100 tons heavier than anticipated. The submarines could dive but there was some question as to whether they could reliably surface again."

"The shipbuilder, Navantia, immediately suspended construction of the subs and called in U.S. submarine builder Electric Boat to solve the buoyancy issue. EB recommended the shipyards lengthen the S-80s by thirty feet, stretching it from 232 feet to 265 feet..."

"Now, fifteen years after the submarines were ordered there is a new problem: The first submarine, Isaac Peral, is too large to fit in its port at Cartagena. Authorities say the port will have to be enlarged and dredged to fit the updated design. It's odd that Spanish authorities are suddenly discovering the problem considering the submarines were actually built in Cartagena and everyone involved has had five years to ponder the ramifications of enlarging the sub's design."

</comment><comment><USER32> Boy, if that isn't an argument for peer review, I don't know what is...

<comment><USER33> Would you be able to spot the mistake in a system that complex?

<comment><USER34> shouldnt something be tracking the BOM and thus calculating the weight of everything being added in. It seems like catching this could even be automated.

<comment><USER33> What it something was omitted from the spreadsheet by accident?

That's how these mistakes happen.

</comment></comment></comment></comment><comment><USER35> How does this not get caught before now? The review/QA process for a program this expensive and sensitive should be very strong.

<comment><USER18> Unburying this old fail just right now is really interesting. In the same week when the minister of defense Margarita Robles is about to close several big contracts including S80 submarines, F110 frigates and MGCS chariots with several companies.

I would not discard that the decimal event happened either only in the mind of the journalist or as a scapegoat history after a major change of specifications in the middle of the project. Honestly, smells a lot like FUD

</comment></comment><comment><USER36> On they plus side, they won't have nearly as much trouble as they otherwise would have had to make it go down.

On a boat like this 70 tons isn't all that much, but it is outside the margin of error in terms of what you would correct with the ballast tanks. This will be expensive and tricky to fix without impacting other systems.

</comment><comment><USER37> In slide-rule world this did not happen. Slide rule gave you 3 numbers, but you were not always sure of the tailing zeros, so you checked the sanity of the scale all the time.

<comment><USER25> > In slide-rule world this did not happen.

Of course it did, you just didn't hear about it because either it wasn't publicised at all, or there wasn't enough tracking to find out where the error had originally been introduced, or nobody shifted through literal tons of papers to find out, or if they did there was no internet and it'd only appeared as a note on page 7 in the "journal of overweight submarines", circulation 500 copies.

This sort of errors is pretty much why double-entry bookkeeping was invented.

</comment></comment><comment><USER18> The scheduled date for floating is October 2020 and it looks pretty fine. If there was a problem it seems perfectly fixed.

[https://www.navantia.es/en/news/press-releases/navantia- clos...](https://www.navantia.es/en/news/press-releases/navantia-closes-the- resistant-hull-of-s81-isaac-peral-submarine/)

</comment><comment><USER38> As of January 2018, the intended delivery date of the first submarine is September 2022. [0]

Can anyone comment what the current status and solution is?

[0] [https://abcblogs.abc.es/tierra-mar-aire/industria-de- defensa...](https://abcblogs.abc.es/tierra-mar-aire/industria-de- defensa/defensa-submarino-s80.html)

<comment><USER39> From what I understand, they just lengthened the submarine to increase its displacement enough to counter the extra weight.

It also looks like the hull of the first submarine is now complete, [1] so there's some possibility they'll meet their new timeline.

[1] [https://www.globalsecurity.org/military/world/europe/s-80.ht...](https://www.globalsecurity.org/military/world/europe/s-80.htm)

</comment></comment><comment><USER40> 2013

[https://news.ycombinator.com/item?id=5842281](https://news.ycombinator.com/item?id=5842281)

</comment><comment><USER41> Yet another achievement for pacifism :-)

Seriously, in the olden days someone might have been put in jail for being a saboteur.

<comment><USER42> Sabotage should never be discounted. Great point!

</comment></comment><comment><USER43> _The Defence Ministry said technical problems are normal for projects of this scale._

</comment><comment><USER44> No one expects the Spanish Exponentiation!

</comment><comment><USER45> Ooops! there are bugs and bugs.

</comment><comment><USER46> Tie some balloons to it?

</comment><comment><USER47> Why does Spain need subs? Seems like a waste of resources.

<comment><USER48> Think about the Strait of Gibraltar, Ceuta, Melilla, and Canary Islands:

\- Gibraltar is a potential choke point for the mediterranean if something happens in the Suez Canal.

\- 3 nations (UK, Morocco, and Spain) have territorial waters on the strait. All three are NATO members, but the UK left the EU last month and cannot be considered as an ally for it gibraltar politics anymore...

\- Marocco is claiming more territry waters both in the ocean side near canary islands, and in small islands Ceuta and Melilla. See Perejil Island incident ([https://en.wikipedia.org/wiki/Perejil_Island](https://en.wikipedia.org/wiki/Perejil_Island)).

Also, Spain sells a lot of weapons. Submarines are a good addition to the catalog.

<comment><USER49> Spain is so hypocritical with Gibraltar and it's attitudes to Ceuta, Melilla and it's other African colonies, as well as the canaries.

<comment><USER50> I don't think this comparison is fair:

\- The nation of Morocco did not exist when Ceuta and Melilla became Portuguese and Spanish respectively, so I assume the only claim Morocco could have would be based on territorial integrity. They were also never part of the Spanish protectorate in Morocco, and are not recognised by the United Nations as colonies.

\- On the other hand, Gibraltar has been classified by the United Nations as a Non-Self-Governing Territory subject to decolonisation. Furthermore, it considers Gibraltarians mere "settlers" and as such does not recognise their right to self-determination, and urges the governments of Spain and the UK to negotiate the decolonisation process between themselves.

Whether the UN's position is correct or not is whole different matter, my point is that given the UN's point of view, Spain's position on Gibraltar and Ceuta/Melilla is not contradictory.

Regarding the Canary Islands and "other African colonies", I'm not sure what you are referring to as the only inhabited territories Spain has in Africa are Ceuta, Melilla and the Canary Islands, and as far as I know nobody is claiming the Canary Islands.

</comment></comment></comment></comment><comment><USER51> A Spanish shipyard also botched up bulkhead design on Helge Ingstad frigate. It sunk after collision in 2018 and was a total loss.

The collision itself was all on inept Norwegian crew, but the water sealing have not held. There are 4 others of her class still in service.

[https://en.wikipedia.org/wiki/HNoMS_Helge_Ingstad_(F313)#Col...](https://en.wikipedia.org/wiki/HNoMS_Helge_Ingstad_\(F313\)#Collision_with_oil_tanker)

<comment><USER52> This is not true. Norwegian investigations concluded that there were no technical defects involved in this incident, discharging Navantia.

Quoting your source: _The watertight condition of the ship was guaranteed by the 13 watertight bulkheads. Seven compartments were damaged as a result of the collision but initially the ship remained afloat. No one intervened to break the chain of errors._

See also: [https://www.aibn.no/About-us/News-archive/Status-of-the- inve...](https://www.aibn.no/About-us/News-archive/Status-of-the- investigation-into-the-collision-between-the-frigate-KNM-Helge-Ingstad-and- the-oil-tanker-Sola-TS-27-February-2019)

<comment><USER51> Yes it was guaranteed by the bulkheads, and the guarantee has failed. The ship should not have sunk with this amount of damage.

 _The vessel grounded and continued to take on water, through the propeller shaft and stuffing boxes.[9]_

The whole point of stuffing boxes was to contain water in damage like this.

The link you posted is in no way absolves the shipbuilder. Merely states that

 _The vessel 's Spanish designer Navantia received a notification of a critical safety issue in relation to the frigate's watertight compartments_

…and the investigation is ongoing.

<comment><USER53> > The whole point of stuffing boxes was to contain water in damage like this.

That is not the point of stuffing boxes. They keep water out when the shaft is not turning and lubricate the the shaft with sea water when it is. A grounding can bend the shaft easily letting lots of water in.

</comment></comment></comment></comment><comment><USER54> I get the whole "home grown industry" and sovereign production of defense and all that, but just like when Australia tried to build its own submarines, really you're better off just buying them off the shelf from your allies who've already been ploughing billions into design and development of the things for decades.

<comment><USER17> Spain has an active defense shipbuilding industry (e. Eg scorpene submarine), why would they buy them from someone else ? And these kind of program secure a lot of jobs, when you buy from outside you secure those jobs abroad.

By the way, how is the f35 going ?

<comment><USER55> I guess, they are still at the 'ploughing billions' stage.</comment></comment></comment>


For Young Traders, a Market Where Air Jordans Are Blue Chips - <AUTHOR> http://www.nytimes.com/2014/04/17/business/a-thriving-market-where-air-jordans-are-blue-chips.html

<comment><USER2> Several years ago, a boss of mine assigned me to an account to write sales copy for one of these sneaker trading services.

And then kicked off the account on the same day.

People will make markets for relics of the saints that they venerate. In this case, the saints are sports heroes, and the relics are just replicas of the holy artifacts worn by the saints when they performed their miracles.

This'll probably follow a similar trajectory to the bubbles in comic books and trading cards, which some of the interview objects even make themselves. 'Till then, there will probably be some opportunities here, so long as you don't expose yourself to the inventory and instead help to facilitate the transactions.

I doubt this has a lot more room to run, because NBA Finals TV ratings are in significant decline year by year, and college ball players do not play long enough to develop a strong enough cult of personality.

<comment><USER3> > I doubt this has a lot more room to run, because NBA Finals TV ratings are > in significant decline year by year, and college ball players do not play > long enough to develop a strong enough cult of personality.

I'm not so sure about this. Rap/hip-hop culture is fairly popular around the world and they are popularizing sneakers just as much as basketball is. I don't have numbers on this, but basketball is going global, too. American players play in pro leagues all across Europe and Asia. Sneaker culture is a representation of this lifestyle.

<comment><USER2> Could be, could be. It'd have to shift to more of the rap cult of personality, though, over the sports personalities.</comment></comment></comment>


15 Incredibly Stupid Ways People Made Their Millions - <AUTHOR> http://www.mathfinance.cn/15-stupid-ways-people-made-their-millions/

<comment><USER2> This seems to be a wee bit of "ideas which I didn't think of and don't appeal to me, which I'm bitter are successful."

Or, to put it another way, people busy making don't have time for complaining.

</comment><comment><USER3> Some of these businesses are definitely not stupid; they provide extremely useful services:

\- Bio-hazard cleanup (#14) -- most people would gladly pay someone else to clean up a murder scene rather than doing it themselves.

\- Public toilet finder (#2) -- not as glamorous as foursquare, but probably one of the most useful location-based services ever offered on a cell phone.</comment>


Plagiarized news sites are using Cyrillic characters to avoid detection - <AUTHOR> https://hoax-alert.leadstories.com/3469009-forget-russian-bots-fake-native-americans-are-using-russian-characters-to-avoid-fake-news-and-plagia.html <comment><USER2> I doubt websites hosted in Eastern Europe care about copyright legal threats. Even if they contact the hosting provider directly I doubt any action will be taken. Eastern Europe has plenty of cheap, shady hosting providers where you can host pretty much anything that you want. Unless the website is making a lot of money, nobody is going to spend significant resources to take those websites down.

Let me speculate of why they might be doing it. Google will de-rank pages that have content that's identical to others (e.g. identical paragraphs of text). Maybe Facebook is doing something similar?

Let's say one of your friends shares an article from BuzzFeed.com, then another friend shares an exact clone of this article from FakeBuzzFeed.com. Now, Facebook might not want to show two articles with the same title from two different websites on your timeline. And considering that BuzzFeed.com is a website with a higher ranking that FakeBuzzFeed.com, it will probably choose to display only the fist one. If you do the Cyrillic trick to the article in FakeBuzzFeed.com, Facebook will think it's something completely different and present it to you, thus getting you a higher reach.

The same applies to the advertising part: if you're constantly submitting page ads with exactly the same titles as the one's that real users are sharing, it might get you banned.

<comment><USER3> I think you're right. Also news stories tend to have a currency. By the time a copyright owner complains, the news is old news. They don't want some automatic process to mark their content as "spammy" and not feature their links or content if someone pushes it.

</comment></comment><comment><USER4> A bit of extrapolation here. In short, a few websites dedicated to make easy money on Facebook by copying articles have started to use unicode to obfuscate the title. They automatically replace latin characters with similar letters.

This makes their title harder to be detected by either Facebook, fact checking websites, or DMCA/copyright bots. Nothing related to Russia here.

<comment><USER5> Or Native Americans.

</comment></comment><comment><USER6> Here is a project that maintains a list of homoglyphs and has some Java and Javascript code for detecting them.

[https://github.com/codebox/homoglyph](https://github.com/codebox/homoglyph)

The list itself in sorted text format, each line a list of similar glyphs:

[https://github.com/codebox/homoglyph/blob/master/raw_data/ch...](https://github.com/codebox/homoglyph/blob/master/raw_data/chars.txt)

</comment><comment><USER7> This reminds me of a project back in 2014, where a school-mate and I created an "uncopyable" font using the same idea.

I put the site back online at [http://nopy.progresso- ict.nl/](http://nopy.progresso-ict.nl/) ($10 PayPal money has already been given away years ago)

</comment><comment><USER8> I think at some point, a sort of visual-normalization that converts similar looking unicode to a single unique string sequence (ex: convert certain letters from Cyrillic and other language sets that are also present in Latin to just Latin) is just going to be necessary as a security precaution.

Given the whole "fake news" thing over the past couple of years, I expect that the first step will be taken by one of Google/Twitter/Facebook/etc, but I hope that they (or someone else) releases a library (or worst case, an online API) that allows this sort of normalization for security verification. I get that having it open would make it easier for people to find loopholes by brute- force testing, but these sorts of loopholes could also be patched rather quickly as they came up, providing benefit to everyone (especially from a security perspective).

EDIT: Perhaps this could start out as a series of matches generated using ML classification? I don't know much about ML - does anyone who does think this is a realistic starting point?

<comment><USER9> Unicode normalization/equivalence is half of what you want and UCAPI is probably the other half.

[https://en.wikipedia.org/wiki/Unicode_equivalence](https://en.wikipedia.org/wiki/Unicode_equivalence) [https://www.casaba.com/products/UCAPI/](https://www.casaba.com/products/UCAPI/)

<comment><USER8> Those are interesting, especially the second one. I've read a little here and there about Unicode normalization before, but UCAPI does look like what I really wanted. However, seeing as UCAPI isn't free or even "listed price plans", I get the strong feeling that this will not see much pickup (at least until someone makes a free one).

<comment><USER10> There are plenty of free libraries that allow you to detect and compare strings with confusables

</comment></comment></comment></comment><comment><USER11> Should be easy enough for networks to detect and remove these, by identifying content where character ranges in words routinely fall outside the charsets of languages.

That, or some sort of fuzzy CV hashing, which is cool, but more intensive. That would also mitigate null length and invisible modifiers.

</comment><comment><USER12> This is an old trick, successfully used for a while in domain names (does gооglе.com look suspicious to you? what if it had a valid SSL certificate?) but hopefully all browsers and registrars have smarted up by now.

Another version of this trick has been popular in Russia with corrupt government workers: by law, a lot of government purchase/service contracts should be subject to public calls for bids, usually placed in a website which you can search. However, if you write what you need replacing some of Cyrillic characters with Latin ones, a honest supplier that is looking for a government contract will never find your entry. However a corrupt one that you have arranged with beforehand would, and will be the sole bidder on this contract, with a price that you have arranged before (which of course includes a juicy cut for the corrupt government official) and nobody is the wiser, all requirements of the law are fulfilled, who could be blamed that there's only one bidder?

</comment><comment><USER13> In Sweden (and probably other places), a service called URKUND[0] ("deed" in Swedish) is used for automatic detection of plagiarism for school work.

I have always wondered to what extent they identify stuff like this, and other potential trickery with UTF-8 or removing text layers from PDF files.

0: [http://www.urkund.com/en/](http://www.urkund.com/en/)

<comment><USER14> A neat trick would be to render the PDF into an image file, and then OCR that image file, and do the detection using the text file generated by the OCR process.

Just like VW's defeat device, the cheater would then need to create software that outputs something different when they see they are being rendered not to display, but to an image file...

<comment><USER15> So, one needs to use a font that OCR finds hard to avoid detection. Would rubbish keming do the trick, you could abide by imposed don't requirements but change kerning/leading?

<comment><USER16> Presumably if you have the original file, you could reset the formatting to known-readable state before the OCR step.

Edit: You could also just use the text, but red-flag submissions where X% of the words used fall outside of the submitted language.

</comment></comment></comment></comment><comment><USER17> Maarten Schenk (resident expert on fake news) create click-bait titles

<comment><USER18> It's not even fake news and even plagiarism is a weak description. It's blogspam / content farming for cheap advertising dollars by eastern europeans. This has been going on for ages, much like SEO gaming, it's just using another hacky trick, and using Cyrillic characters is hardly new trick either.

</comment></comment><comment><USER19> substitution of characters o/a/e (that are similar in Latin and Cyrillic alphabets) has been used for years to pass automatic plagiarism detectors.

</comment><comment><USER5> >The site is part of a growing list of fake Native American pages run out of places like Macedonia, Kosovo or Vietnam.

So the headline is a little misleading... It's just that there are a growing number of websites that simply plagiarize content to get views / ad revenue. Because their titles are obfuscated to prevent detection of the plagiarism, they have to target specific niche groups to drive views. So it's not some weird "fake Native American" scheme / scam / ploy... it's just that _this_ site in particular seems to focus on "Native American topics".

So it's not "Fake Native Americans Are Using Russian Characters to Avoid Plagiarism Detectors", it's "Fake News Sites Plagiarize Articles by Using Cyrillic Character Replacement to Avoid Detection", subtitle: "One such site targets Native Americans!"

<comment><USER18> This isn't even fake news. It's just blogspam for advertising dollars. Nothing fake about it... the articles they copy are real content.

This is just jumping on trendy words like Russians/bots/fake news for click bait.

</comment></comment><comment><USER20> Let me nitpick a bit: these characters are not Russian, they are Cyrillic. There are some Cyrillic characters that are distinctly Russian (i.e. used only in the Russian language), but these characters can't impersonate Latin letters because they are too different from them.

[https://en.wikipedia.org/wiki/Cyrillic_script](https://en.wikipedia.org/wiki/Cyrillic_script)

<comment><USER21> Yeah that bothered me as well.

Would you write: "Site uses American characters to publish fake news." ?

<comment><USER22> you gotta loop the Russians in to drive page views</comment></comment></comment>


The Two Unicorns of Technology: Full-Stack Engineers and General Data Scientists - <AUTHOR> http://www.wise.io/tech/two-unicorns-data-science-software-engineering-full-stack <comment><USER2> And that's why you'd want one of them to be a woman, otherwise, unicorns' race will go extinct.</comment>


Ask HN: What to do if I can't finish university? - <AUTHOR> I am currently trying to finish the second year of my university program for Computer Science despite actually it being my third year as I couldn't pass any subjects last year for various reasons.
One of these reasons is that I simply cannot get my brain to work in an exam setting and have it memorize a bunch of things to then try and remember it on the exam. As well as all the material in my university (in Netherlands) being presented in a horrible manner where is is nearly always best to actually learn it yourself online which is what I did.
It also doesn't help that I have 0 friends inside the university nor in the city I live in.
This constant depression and feeling like I am in in Groundhog Day movie where I repeat the same subjects again and again is making me ever more depressed. It also doesn't help that I cannot find a job of any kind despite actually knowing how to build things.
Trying to solve my own frustrations in life and education I actually started a project to try and solve this as I realized that most learning already happens online, it's just most don't know what they can learn about even and how everything connects. Google just provides a very powerful empty query box and you need to have a question first.
The project is a website and can be found here: https://learn-anything.xyz
I even launched it some time ago on HN and got some positive comments.
All that is to say that is it normal to always feel this depressed about education? Am I a weak person who can't handle the stress of it all so I just give up and try and find the 'easy' way out?
What can I do at this point? I really really hate university and I cannot wait to actually work on something I love and get paid to do it but it seems that to do that you need to get that CS degree paper. And the project I started is open source and has just enough money to stay afloat through donations.
I have no idea what to do at this point. <comment><USER2> No. Depression is as likely a _symptom_ of underlying learning issues.

Try a professional screening for the usual suspects this summer break to see if there are some easy starting points. Pop into some subreddits and ask there.

ADHD types, ASD, BPD, SCT, GAD

</comment><comment><USER3> Memory/learning difficulty is a symptom of depression. Taking your words at face value, maybe you go see a doctor.

<comment><AUTHOR> I agree with you. Depression certainly hinders nearly everything I try to do.

However going to a doctor is actually expensive in Netherlands. Even more so in US. And from my experience with doctors in NL, they don't actually help you in any practical way. Also I don't want to go and take any kind of anti- depressants as I think it's not a solution that actually works long term. Your body will just learn to rely on the medications.

The solution to depression comes from within. It's just that I can't really find the will in me to continue doing the thing I hate so much and feel 'forced' to do it.

<comment><USER4> Anti-depressants can let you feel normal for a while, it's a way to find your will again.

If you refuse all treatment options, you are going to have to find a way to continue on through your projects. You should continue on through the degree, but if you decide to leave, at least put things in order so that you can return and finish your degree once you have a few years of perspective.

Best of luck to you.

</comment></comment></comment><comment><USER5> C's get degrees. Suffer through it and get the degree - you'll be happy you did it.</comment>