Of Two Minds – A neuroscientist balances science and faith - <AUTHOR> http://stanmed.stanford.edu/2006summer/newsome.html

<comment><USER2> That's nice. In a sense he's rationalizing away the need to apply the same rigorous requirements to the existence of a god (and therefore whether or not obedience is necessary, followed by a source from which to gather his sense of morality) that one would apply to any other fantastical claim. So what? Nothing wrong with that. So long as he's not applying that same mode of thinking to the actual science he does (which he apparently does not), doesn't negatively impact others with that morality which he derives from his beliefs - or the source of those beliefs, and doesn't demand that others respect or accept those beliefs. Doesn't appear from the article that he's guilty of any of those actions. Then again, I'm unfamiliar with his name or his work.

*Edit: rephrased "...or acceptance of..." => "...or accept..."

</comment><comment><USER3> This is a great explanation of how faith and reason operate together. I have degrees in computer science and theology, so I don't have the scientific cred of a neuroscientist, but I can say this piece does a good job of articulating my own experience.

<comment><USER4> It captures what happens but I find the conclusions wrong. Yes, we are often unable to act with complete scientific rigor because the systems we have to deal with are way to complex and so we use our intuition and the like to deal with the situations, but this does not at all mean that this is a good or useful way to understand our world - after all our intuition and common sense is exactly what gives us a wrong picture of our world. Only scientific rigor was able to teach us that the world is very different from what we usually see, from space and time over energy and matter to the dynamics of life, the earth and the universe as whole.

<comment><USER3> I am not saying, and neither is the author, that we should reject scientific rigor in the cases where it can be applied. Rather, we should not limit ourselves to only asking questions that can be answered by science. Indeed, it is impossible to live that way. Therefore, the rejection of religious claims, such as the existence of God, on the basis that they cannot be scientifically proven is not reasonable. If all questions of life were subjected to the same standard of proof that some people demand of religious claims, we would not be able to function.

I used to be an atheist until I realized that my rejection of religion was intellectually dishonest. I therefore decided to investigate religion with an open mind, and that eventually led to my belief in God.

</comment></comment></comment><comment><USER5> Admitted I didn't think he would be sensible when he said he is a minority being "serious Christian" in a community of scientists, he turned out not to be sensible.

</comment><comment><USER6> "It hinges on a gut-level judgment about what sort of universe we inhabit."

The thing is, I don't think a gut-level judgment is the best course of action here. Similarly, I don't make gut-level judgments about whether I believe in gravity, or whether I believe in evolution. In my mind, the nature of the universe falls squarely under "science."

<comment><USER7> TLDR: Science is inconsistent, and so is faith. "Gut-level judgements" are all we have, both in science and in faith. Can people please stop ignorantly claiming otherwise ?

The laws of mathematics actually prohibit anything other than a "gut-level" judgement. Let's take one problem (but go ahead and take 5 books about "constructivism" out of the library and you'll get to 50 problems soon). Godel's theorem, because it's the most generic, but there's plenty of them. Assuming you believe in any science that uses natural numbers, rational ones, or real ones (ie. all of them), then there's 2 possibilities :

\- you're not convinced that there are no contradictions. In this case, what do you have that faith doesn't ?

\- you're convinced that there are no contradictions. In this case you can be sure : you're wrong. Either because you don't realize there are contradictions, or because you've got a wrong theory.

So how exactly is science better than faith ? It's not.

Here's how this argument evolved :

1) Faith does not follow reason ! It kind of does, or rather we can't find good examples of obvious inconsistencies. (the bible was written by people who had studied under Greek philosophers, so it really is quite consistent. It's not perfect, but compared to say, the quran or the vedas, it's bloody hard to find conclusive contradictions. Whereas the quran simply says that it has inconsistencies and you shouldn't care about that, for example, and points out a specific inconsistency in itself. But this was not a real argument at the time)

2) oh-oh the vatican claims they do follow reason. They have some good arguments on that too. But they don't follow reason entirely !

3) oops ... science doesn't actually follow reason either. We don't actually have proofs of consistency, and worse : we don't even construct most arguments. Let's fix that.

4) (~100 years later) we give up. This is not possible. Let's be brutally dishonest and at least look for a proof that we could write a constructivist science and do that. (this is brutally dishonest because it would be a non- constructivist proof that constructivism could work without actually making it work)

5) (~10 years later) okay, that can't be done either. We can't have a science without pulling new concepts out of thin air without any real reason to assume they work other than that we haven't found any mistakes !

6) (~20 years later) This idiot "Godel" comes on stage and claims if you have a proof that a scientific theory is consistent you can rewrite that into a proof that that theory is wrong ! Prepostrous ! Let's pelt him with tomatoes !

7) (1 day later) "Say guys ... I've been looking through that proof of Mr. Godel and I can't seem to find a mistake" ... followed by 1 month of doing nothing but looking for errors in that proof.

8) (1 month later) Success ! Godel made a mistake.

9) (2 days later) Godel writes an article fixing the mistake

10) (~2-20 years later, depending on who you ask) Okay, we give up. Science is not consistent and can never be consistent. "We cannot pull ourselves out of the mud".

Given this history, can we please drop the argument that science is consistent and faith is not ? (Christian faith, because it is the only one that claims and tries to be consistent, not saying you couldn't create others, but no-one did)

Science is inconsistent, and so is faith. Can people stop claiming otherwise ?

<comment><USER4> I have to disagree, science is not inconsistent. Science, more precisely natural science, is in the business of building models of our world and if the model is any good it is consistent with our world. Some models are precise to 10 digits, some models only describe the behavior of a school of fish on average. If a theory is found to disagree with reality it will get modified or maybe get abandoned and this is no problem because natural science never claims to provide any truth but only a good model of our world that may have to change in the future.

Religion is similar in the regard that it provides a model of our world but a very poor model under scientific standards like verifiability, predictability and falsifiability. While science constantly challenges its own results and even methods religion is far, far more in the eternal truth corner.

Finally mathematics is a very different thing and comparing it with physics or religion may be misleading. Like religion but unlike natural science mathematics is in the eternal truth business but unlike religion and natural science it is not in the describing the world business. Mathematics is used to describe the world but does not so on its own. And things like Gödel's incompleteness theorems are not a weakness of mathematics but a strength - knowing in principal limits is something that seems completely out of reach for everything but mathematics.

<comment><USER7> There is a mathematical definition of consistency that I was referring to. It basically means that you can prove that a model can't be used to create a contradiction. In maths, any model where you can't prove that is disqualified for most work.

Problem is, the numbers we use don't actually satisfy that standard. There is at least a chance that it isn't possible to create a correct model at all using what we generally call "numbers". If someone were to stand up and "complete" Godel's work, so to speak, that would invalidate all of science (Godel proved it isn't possible to prove science correct, so completing his project would be to find an actual deep, unsolveable problem with numbers. A number of good candidates have crept up over the years, famously the axiom of choice debacle, and a lot of interesting paradoxes, like "does the collection that contains every collection that doesn't contain itself contain itself ?" (last 4 words - not a typo))

There's actually 3 different consistency standards for models (and number 4 is that it's not consistent at all):

1) consistent <\- we can prove it !

2) inconsistent <\- we don't know, but we may find out at some point in the future

3) inconsistent <\- we don't know, and we can prove we don't know, that we will never know

4) inconsistent <\- we have an actual counterexample, and thus a proof that it's inconsistent

Science falls in the third category. So does (Christian) religion [1]. There is no real difference in consistency between science and faith by this standard. Problem is that other standards have similar problems. Again a distinction should be made between different religions, as there's only one where I've ever seen any serious discussion on the side of the religion.

a) verifiability : faith is not verifiable. A Christian would say that's part of the point. Of course, neither are most sciences. Certain aspects of mathematics are not verifiable (the other godel theorem), either not at all, not by any finite procedure, or not by any reasonable amount of effort. For physics there is actually only a relatively small portion that is directly verifiable, and once you go down the ladder ... verifiability becomes a distant memory by the time you get to things like history.

Btw: a misconception you seem to have is that theories get modified because of inconsistencies. Well, I'm not saying it never happens, but in practice theories get modified because there's a need for it. Architecture, for example, works with pre-Newtonian physics (buildings architects calculate on stand on a disc suspended in space, with a constant gravity vector straight down, in a barely viscous liquid. Has been that way since before Newton was born, and there's no good reason to change it, so nobody did (in fact there's good reasons not to change it, complexity for one)). Yes this is the utilitarian argument, but it's true. It may not control all academic activity but it controls 99.99% or more. If anything, it's been getting worse.

An illustration of that fact can be found in physics for example. Relativity was discovered because of the black body problem. You see, Newtonian physics predict that if you heat up any object, the universe should explode. Obviously it doesn't, and this was an eyesore. Now keep in mind that Einstein was a patent office clerk with a weird hobby, and this is no coincidence : there wasn't all that much attention going to that problem. Every theory has eyesores. Currently there are 2 huge established theories, and one that everybody believes in but no convincing predictions have been made using it. Here's the current eyesores : relativity theory only has gravity, doesn't have particles or any force (gravity's not actually a force in that theory), has a continuous infinite universe that is either flat or just ever so slightly bent (like a millimeter per lightyear or so). The standard model doesn't have gravity, yet things fall, it has a discontiuous universe that is bent twisted and shifting beyond recognition at small scales, it fails to predict large scale objects in any reasonable manner (e.g. large scale universe is obviously mostly flat ... why ? How does the shifting particle soup that is the standard model's playground become the flat expanse we call space where a photon can fly undisturbed for millions of lightyears ? This, to me, is not a small detail you can just gloss over), yet we can design things like cell phone radios and cpus using it. Then there's string theory ... string theory however is a theory with so many open ends that you can do pretty much anything what you want with it. It answers the question "hows does the universe work", by giving you a brainfuck compiler and get cracking on a simulation, stating that the answer is in there (and of course, it is, in the sense that you can probably write a correct simulation in it). Those are current eyesores. I sure hope some people at the patent office are about to fix them, because I doubt significant heyway will be made any time soon.

b) predictability. I would argue that religion makes predictions, and I would argue that, averaged, they're pretty good. E.g. islam says camel piss is good for you [2]. This sounds ridiculous of course, but taking 2 facts into account it makes sense, somewhat. First urine is the one bodily liquid that is sterile. If you care about disease spreading, you can't eat or touch the same food as someone else, you can't drink the same water, you of course can't have sex or other things that exchange bodily fluids, certainly not with animals, but urine is an exception, it's fine, it's sterile, it's safe. If you don't have a water source you trust, camel piss may be preferable. Knowing that poisoning water supplies was standard Roman army practice, it may make sense if you're waging a war on them. Eventually of course, it'll kill you, but only after months of drinking it. But it's not a stupid, random piece of gibberish either. And religion certainly makes predictions about what will happen, most of which you'll find yourself in agreement with.

A lot of sciences don't care about predictability. Maths would be one example. We study models, because of their implications, and because we like the idea of, shall we say, "classifying" logic. This is a fool's errand, but it's fun. "This does not predict a thing" is a perfectly valid criticism of all but one project I'm involved in, and that's because it's beside the point.

Obviously predictive ability is one aspect of the utilitarian view of science, so it can't be ignored completely. But most physics theories that are being actually worked on (in other words : string theory and nothing else), do not have any demonstrated predictive ability that I am aware of.

And how does e.g. history fit in with predictability at all ? I would argue that religion definitely scores higher on predictive ability than the humanities, somewhat lower than most exact sciences.

c) falsifiability. Again most sciences aren't falsifiable. Mathematical models work with axioms, and falsifying correctly constructed models is impossible (with of course, the glaring exception of the natural numbers (and thus rational numbers, and real numbers, complex numbers and anything based on them), which we know may be falsifiable. This is considered very, very bad)

But string theory for example isn't falsifiable. And again, how does that work for the humanities ? Any statistical work effectively isn't falsifiable. Take climate theory for example. It hangs together from approximated differential equations. The predictions made using them by the IPCC in 1990's ... well, they were wrong. I mean I'm not saying the earth isn't warming, but the IPCC's numbers from the 1990s are bullshit, but they correctly point out that that doesn't actually falsifies their models' claims (although, frankly, it does prove their error margins calculation is ridiculous). They only claim that to a best approximation, their model is what should happen to the variables they're monitoring over time. Medicine is the same. For most medicines we have a double blind study. Now you should think, really hard, about what the exact claim is that is made by those studies and how it's verified. You'll see the truck-sized hole in the thinking.

d) ...

The greater point here is that these are all standards of correctness, trustworthiness, "truth". They're all different, they all have their uses, and they are followed in various ways by various sciences. Within individual sciences, there tends to be a whole spectrum of theories that rank differently on all of these standards. Religion is no different. It simply has a place on the spectrum, just like any science does. Various components of religion very on where they fall on all these standards.

But it is certainly not the case that sciences score higher than religion on all of these standards, it is simply the case that there is utilitarian need for both, and for criticism of both. Science has it's uses, and so does religion. Where things fall on standards of consistency, falsifiability, predictive ability, ... is simply a property of whatever fills that particular niche best.

[1] [https://www.uwgb.edu/dutchs/PSEUDOSC/RammReconcile.HTM](https://www.uwgb.edu/dutchs/PSEUDOSC/RammReconcile.HTM) [2] [https://au.answers.yahoo.com/question/index?qid=201204210704...](https://au.answers.yahoo.com/question/index?qid=20120421070459AAEj4y6)

<comment><USER2> I'm definitely out of my league here, but it seems like you're overcomplicating the question.

I won't pretend to be qualified to say whether you are right or wrong, but I'd like to make a few points.

I get the impression that you are trying to address whether science is as [in]consistent with itself as religion is [in]consistent with itself in their attempts to land upon some truth or as another commenter put it, "model of our world".

But the question posed by the article is whether science is consistent with faith. For the author, it is. At least a few commenters are simply stating that the reasons given by the author for this consistency don't resonate with them.

It's also worth noting that while you're using a "mathematical definition of consistency", the commenters you are refuting are probably just meaning that whatever conclusions one reaches via the scientific method are often in disagreement with those arrived at via faith. Now, I don't know what the Christian faith tells us about String Theory - heck, I don't even know enough about what science says about string theory to speak of it with any clarity. But I do know that with the help of science we can figure that the earth we live on is spheroid and revolves around the sun, and I know that the Bible hints slightly at a vastly different conclusion, or at least it was interpreted to do so at one time. On one side of the coin, we can look at evidence, or at the very least hold a theory as probable - and if it's falsified - well it's sad, but the theory changes to better suit the findings. On the other side of the coin, we must take somebody's unchanging word for it, and maybe revisit our interpretations of the same idea only if we're forced to - and otherwise, any objections must be false. I just don't see how anyone can say there's no difference there.

<comment><USER7> It's not just about consistency. The post mentioned 5 different standards of "truth". In reality the method we use to judge something to be true or false is not uniform : there are lots of variations, even within single sciences.

My point is that for all the methods the parent poster pointed out, science does not actually fare better than (Christian) faith, barring a few exceptions.

He cherry-picks little pieces of specific theories that satisfy a high standard of truth, I merely point out that, first that does not mean science as a whole satisfies that standard, second you can do the same for the bible. And, when it comes to historical opinions that were held up using less-than- gentle means, science also has loads of black spots. E.g. [http://en.wikipedia.org/wiki/Lysenkoism](http://en.wikipedia.org/wiki/Lysenkoism), so again that is not a valid criticism of faith, or at least isn't black and white.

As an alternative I defend utilitarianism : scientific theories are not true or false, or better or worse than the bible, but they are more useful in some cases. Nobody will design a cell phone using the bible. Nobody should base their moral decisions on biology.

The big problem here seems to be that Christianity uses internally the same standards of truth to some extent, and so it can be attacked on those points. It's actually pretty consistent so I think the attacks against it are less than convincing, because there is no better alternative, there is just a completely unjustified faith in science that makes no sense whatsoever : logic, the branch that actually analyses whether science could be wrong and how and why trust in science should exist, has a simple conclusion since the 1950s or so : science fails it's own standards. Somehow this part is always missing from these arguments : there is nothing to back them.

</comment></comment></comment></comment></comment></comment><comment><USER8> This (short) book gives a sense of the "faith" required for scientists - [https://www.goodreads.com/book/show/13574594-ignorance](https://www.goodreads.com/book/show/13574594-ignorance)

</comment><comment><USER9> I think "faith" is the operative word here. I don't believe our "religions" are congruent with science. I have faith in science!

</comment><comment><USER10> "religious insights" he says ...

"religious delusion" I'd say.

IMHO, thinking that science and religion would not be in conflict is just an even worse level of delusion.</comment>


Ask HN: Review My Webapp: Sneffel - <AUTHOR>

I wrote a "virtual whiteboard" application several months ago in adobe flex.  I thought I'd re-write in HTML5 with websockets and canvas (apparently, I write really messy javascript.)
After talking with some of the few people who use it, I thought I'd experiment with monetizing it by making whiteboards expire and having users pay to re-activate.
Anyways.  Sneffel is a collaborative drawing pad, built using html5 (so no IE support.)  You can draw with friends, replay your drawings in an animation, customize the colors of your whiteboards, and embed them into your own websites.
Http://www.sneffel.com/about (demo of the replay) and http://www.sneffel.com (should dump you into a brand new whiteboard.)
I would very much appreciate your criticism.  Thanks. <comment><USER2> Not much criticism, but I like it so far. Does this look like something you'd try to monetize and keep proprietary, or open up the source to (or both perhaps?)</comment>


Ask HN: What useful but unknown chrome extensions do you use? - <AUTHOR> <comment><USER2> Vimium - Adds vim-like controls

My favorite parts: j and k to scroll, J and K to switch tabs, F to display 1 to 3 character sequences next to all links where typing one opens it in a new tab.</comment>


The Capital Egypt – The smart city of tomorrow - <AUTHOR> http://acud.eg/ <comment><USER2> Super interesting and a big improvement compared to the current Cairo. I've been exploring opportunities to move there from Sweden but it's hard to gauge the Software Engineering job market there and it's evidently hard to find someone over there who works in tech to talk to</comment>


Why Tool AIs Want to Be Agent AIs - <AUTHOR> https://www.gwern.net/Tool-AI <comment><USER2> As long as an agent acts in the best interest of the principals we should be fine.

the problem is if it learns to subvert for it's own self interest that things become problematic.

just like any relationship with an "agent":

1\. politicians

2\. lawyers

3\. trustees

4\. brokers

5\. CEOs

6\. cleaning person that steals you jewelry</comment>


Spotify Just Got Real Creepy with the Data It Collects on You - <AUTHOR> http://www.forbes.com/sites/thomasbrewster/2015/08/20/spotify-creepy-privacy-policy/

<comment><USER2> I just wanted to note that forbes uses at least 16 trackers in this article (and does not tell you about it).

I have no problem with journalists pointing out flaws, but especially forbes likes to know a lot about its readers.

<comment><USER3> FYI, I opened up the privacy policy from my Spotify install (I updated yesterday, I think?) and none of this is mentioned in it.

</comment></comment><comment><USER4> "Upon opening the Spotify app up this morning..."

"FORBES contacted Spotify for a response and is awaiting comment."

This tells you exactly how much time they waited for Spotify to respond. This is 'iterative journalism' at its worst: publish first, investigate later! Completely disgusting.

<comment><USER5> Why should Forbes wait for comment, when Spotify didn't wait for their users?

It's not as if this is something that took Spotify by surprise, like a data hack, that would require time to formulate a legally-sound response.

They will have been planning and implementing this change for months. They could have issued a press release two weeks ago advising their customers of the upcoming changes, but didn't.

Spotify seem to like surprises, so Forbes played one on them.

<comment><USER4> Spotify does wait for the user's consent. That's the whole point of agreeing to the privacy policy. Apart from that, Spotify doesn't have anything to do with basic principles of journalism, because they are not journalists.

<comment><USER5> Any other substantive change to the terms and conditions require prior written notice. For example, slipstreaming in an app update that said 'from right now your monthly fee will be €20' would not be considered sufficient notice.

So why should privacy be Any different? In fact in this Cloud Era, privacy terms are probably of more concern to users than another dollar on their subscription.

The only reason Spotify didn't give prior notice was to put users in a bind on the spot.

Even Google manages to do it properly!

</comment></comment></comment></comment><comment><USER6> Is this for real? I pay £10 a month to avoid this crap and now I'm lumped in as well?

Screw this, I'm off to Apple Music.

<comment><USER7> Why? Apple built your _telephone_ ; they already know about your "contacts, photos, or media files." They know your GPS location, too.

This reminds me of the quip about conservative folks threatening to move to Canada because they're unhappy about all the corrupting liberal values creeping into American politics.

<comment><USER6> I am under no illusions that Apple keeps this information. I've opted out of iAds so I know that my data is not being used against me, and Apple do not sell my data to third-parties.

To be clear, Spotify using my habits and data to improve their service I have no problem with; it makes a better service for me! Same with Apple.

Maybe I should have explained clearer in my original comment (though I thought it was obvious) but I have no problem dealing with companies that I deal with; it's companies I DONT have an arrangement with who I have trouble trusting they wont use my data against me.

And now it appears that Spotify is one of those companies. And all I want to do is stream my playlists on the way to work. Now I have to consider that they may be selling my contacts lists, movements, etc;

It's a plain cash grab at my expense. And I wont do business with businesses that don't respect me or my data.

<comment><USER8> "It's a plain cash grab at my expense."

It is absolutely not.

<comment><USER9> Explain please how it is ABSOLUTELY not. It must be at least a bit, because you can tick off a few from the list of the definitions:

Noun

cash grab (plural cash grabs)

        1. (derogatory) product designed without love or care, with the sole intent of generating profits
        2. (politics) Legislation that serves primarily the purpose of generating revenue.
        3. An activity engaged in with the intention of making money quickly.
        4. The money generated by a cash grab.
        5. A game in which players attempt to grab as much money as possible.

Source: [https://en.wiktionary.org/wiki/cash_grab](https://en.wiktionary.org/wiki/cash_grab)

<comment><USER8> You're doing exactly what Forbes wants you to do: fear mongering. Privacy policies are legal liability waivers. They are not statements of intent. There is absolutely no indication whatsoever that Spotify is harvesting your data and selling it to third parties to make some money on the side. That is a ridiculous assumption, and you need to provide damn good evidence if you expect me to even begin to consider that.

What any REASONABLE person would assume is that Spotify is simply doing what everyone else does, which is track your usage of their product in order to better target ads.

<comment><USER10> And we all know that if everyone else is doing it it's alright!

The next "reasonable" thing to do is to let them use your phone's camera 24/7 to film your life so that they can get an actually good grasp of your consuming habits.

I'm hyperbolising here of course but this trend of using niche features like syncing your playlist's BPM to your pace as an excuse to track even more data and destroy your user's privacy must stop.

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> I don't use spotify and I usually don't like social anything (except HN, of course). However...

Surely there are reasonable explanations. Let's break this down.

GPS and motion sensors. Why could they want this? Maybe they will switch up your play list when you go for that run. Seems plausible.

Contacts. Ok, everything is social, so maybe they want to send your current track to other spotify users in your contacts. Seems plausible.

Photos. Not sure about this one.

Sharing your information with partners. Does spotify play ads? wouldn't this be a standard part of sharing data with Apple and Google's ad network?

I'm not defending them, but this is the social world we live in. You either find alternatives or deal with apps that pack social features in.

<comment><USER12> > GPS and motion sensors. Why could they want this? Maybe they will switch up > your play list when you go for that run. Seems plausible.

They introduced a running feature a few months ago that does, indeed, attempt to match the bpm of a playlist to your running cadence.

> Contacts. Ok, everything is social, so maybe they want to send your current > track to other spotify users in your contacts. Seems plausible.

Share a playlist with a contact. Share a track with a contact. Etc etc. I'm surprised this wasn't already in there if it's really a new addition.

> Photos. Not sure about this one.

Might be related to running. Strava, fr'instance, lets you add photos taken on your run. Or might just be to let you adorn a playlist with your chosen photo.

> Sharing your information with partners.

Doesn't everyone? Isn't this standard boilerplate? Spotify are attempting brand tie-ups in many areas so I can't see this as something out of the ordinary.

<comment><USER13> The photo one stumped me as well, but I figured it was something innocuous. I remember similarly misguided outrage over the Facebook app's 'new' permissions just a few months (I think) ago.

Side note: these features make me think I should probably switch from Rdio, though I guess some of those use cases are satisfied by Songza...

</comment></comment></comment><comment><USER14> Companies need to take a new approach to the permissions they request - especially on Android where these are explicitly shown to the user. Often there are innocuous purposes behind them, but it's rarely immediately clear and results in people jumping to conclusions. Companies should be more explicit as to the reasons they request permissions.

<comment><USER15> The good news is that from Android 6.0, permissions will be more dynamic: [https://developer.android.com/preview/features/runtime- permi...](https://developer.android.com/preview/features/runtime- permissions.html)

Apps can ask for permissions only if/when they are actually needed for a niche feature, and the user can reject or revoke each permission individually.

</comment></comment><comment><USER16> > Local law may require that you seek the consent of your contacts to provide > their personal information to Spotify

Uhm so how does that work?

I'm not a Spotify user, but quite a few people who have me in their contact list definitely are.

Let's say I don't consent. Which I don't.

Will they have to cancel their Spotify subscription?

Doesn't that responsibility fall with Spotify just as much?

Because in that case, I'll gladly provide _my_ contact list, for Spotify's one time exclusive usage with which they can cancel all the subscriptions of everybody on it.

</comment><comment><USER17> in general, i really don't like still being a product although i'm paying for subscription.

</comment><comment><USER18> Sensor data could make sense to pick appropriate music based on activity. Perhaps the idea with the GPS is the same, although personally I'd still prefer not to share that.

<comment><USER14> I believe this is the case - they have a mode (so I'm told) that will choose music based on your current running pace.

</comment></comment><comment><USER19> I'm wondering how this would work with Apples _privacy features_?

Before any App can access your location, contacts, picture and so on, you need to give it permission.

<comment><USER20> Exactly, it specifically states: With your permission, we may...

Much about nothing...

</comment></comment><comment><USER21> I would guess that the new sensor usage and GPS data is for the Spotify Running feature which was announced a while back. From what I remember, it changes your songs based on your running speed. Makes sense to require this additional data.

</comment><comment><USER22> When he talks about mobile app, does he talks about the iOS one?

i have opened the app on Android, gone to the privacy policies and I don't see any of those things he talks about. I have privacy policy in Spanish

<comment><USER21> I had the new privacy policy appear when I opened the app this morning and had to accept it to continue. Probably just rolling out over time.

</comment></comment><comment><USER23> ok lets say this is true. If I want to continue using Spotify, how do I block them from looking at my pictures and contacts on a stock nexus 5 running a stock android 4.4.3 ?

</comment><comment><USER24> Apple are just getting dirty in the music business now. They're telling their favorite journalists about Spotify's new privacy policy - like a journalist would ever investigate such a thing!

<comment><USER25> _" Upon opening the Spotify app up this morning, your reporter was greeted with a request to agree to the new conditions."_

So Apple hacked the Spotify app to pop up a message announcing the new privacy policy? On iOS, Android, or both? Either way, that's quite an extraordinary claim.

I find the idea that Spotify made the change itself and that tech journalists routinely check for changes in privacy policies searching for something to write about much more plausible.

So, please give some evidence that this is what happened.

<comment><USER26> > So Apple hacked...

Think you misunderstood that aspect of jamesblonde's comment. He's suggesting the journalist is pushing Apple's PR for them. It does happen, journalists often use content PR firms send them. I have no idea if it happened in this case.</comment></comment></comment>


Nearly eight in ten of all suicides are male – a figure that has been rising - <AUTHOR> https://mosaicscience.com/story/male-suicide/ <comment><USER2> Probably related to studies claiming the Y-Chromosome (what makes males, males) is disappearing: [http://theconversation.com/the-y-chromosome-is- disappearing-...](http://theconversation.com/the-y-chromosome-is-disappearing- so-what-will-happen-to-men-90125)

~ Khayri R.R. Woulfe</comment>


Obama Acts to Aid Small Businesses - <AUTHOR> http://www.nytimes.com/2009/03/17/business/smallbusiness/17sbiz.html?_r=1&ref=politics

<comment><USER2> Speaking as the owner of a small business, I wish he'd help me by cutting spending, stop talking down the economy, allowing insolvent firms to fold, etc.

Micromanaging billions of "knobs" on the economy is a losing proposition and it creates "regime uncertainty", where folks are afraid that the rules (on taxes, regulation, inflation, contracts, etc.) can change out from under them at any time.

<comment><USER3> Cutting spending in a deflationary recession is never a good idea. You want to execute spending cuts when the economy is vibrant and growing, not when your GDP is shrinking and your unemployment numbers are kissing double-digits. If anything, Obama hasn't been doing enough spending on projects that will be putting people to work in the next 1-2 years.

<comment><USER4> Deficit spending is based on the fallacy of increasing aggregate demand. taking money from people and then spending it on government projects doesn't increase aggregate demand because you've just decreased demand by the same amount by taxing people in the first place. How is this any different from normal redistribution? You're redistributing from the private sector to public works contractors. Prove to me that the public works contractors will produce more jobs than the private sector would have with the same capital.

Money is a commodity. When the government prints more it reduces the value of all other dollars in circulation. This inflation is a hidden tax that we will be paying off for years.

<comment><USER5> > Prove to me that the public works contractors will produce more jobs than > the private sector would have with the same capital.

You are absolutely correct.

Dams, Bridges, Interstate Freeways, Air Traffic Control and Nuclear Power Plants all work so much better when they are built entirely by private enterprise. In cases where they were built using public obligations, they provide very little benefit and do not enable other industries.

<comment><USER6> You're changing the subject a bit from micromanagement of smaller more specific interests to infrastructure, which I would expect more people to agree should be regulated, and especially in the case of dams and freeways, should be government backed.

<comment><USER5> Obviously I was not sarcastic enough.

Look: public infrastructure spending stimulates the economy on multiple levels; directly, through spending on contractors and suppliers (who in turn buy from their contractors and their suppliers) and indirectly by creating public benefits that create new opportunities and enable new industries; and through the fact that it engenders stability (a dam has a lifespan measured in centuries if it's maintained, a functional transportation link is not likely to vanish overnight, and so forth). The thing is that these benefits are diffuse enough (though they are measurable) that no single commercial entity can afford the risk that they might not be able to capture those benefits.

The blinkered view that all public spending is automatically evil prevents clear discussion of the possible alternatives. Not all public spending is equal, some projects deliver more overall benefit than others.

I guess my take is that those who are strident in their distaste for any public spending are patsies in the hands of the special interests who are trying to capture as much of the benefits of public spending as possible.

<comment><USER4> you're completely ignoring my original point. government money comes from somewhere. I'd recommend a basic course on economics before going off on details. <http://www.fee.org/pdf/books/Economics_in_one_lesson.pdf>

<comment><USER5> That is patronizing and wrong. I did address your point fairly directly modulo the sarcasm.

Public infrastructure provides economic benefits throughout the economy, enabling more economic activity; which more than makes up for the money withdrawn from circulation by taxation.

If you cannot wrap your head around this concept. You have no business lecturing others about basic economics.

Governments are not exempt from economic forces, in fact a very good argument could be made that they are created and sustained by economic necessity. To put that argument in the simplest terms; governments save us money.

My other point is that blind negative rage toward all public spending makes you an easily manipulated member of a mob. A mob that others are adept at using to shelter their favored projects.

<comment><USER5> Well, I see that none of the "libertarian" downmod tribe wants to actually argue the point. It's so much easier to engage in groupthink and hit the little down arrow than to actually defend your assertions against someone who aggressively questions them.

<comment><USER6> FWIW, I didn't downvote you. And, upon review, it looks like Nazgul brought up infrustructure, sorry.

As I said in my earlier post infrustructure is fine depending on the situation. I believe government works are a necessary evil where we expect private interests to create monopolies for expoitation. Dam's are such a case. A good highway system is probably such a case. But these things take time and I expect them to be taken advantage of. They have to be maintained even if they aren't useful. You say. "Public infrastructure provides economic benefits throughout the economy, enabling more economic activity; which more than makes up for the money withdrawn from circulation by taxation."

We just disagree on the last sentence. I believe it only makes up for it if the end result of the infrastructure encourages more economic activity.

Of course, this tbread started complaining about giving small business owners stimulous. Well, I agree with the original sentiment. They could also try relaxing some laws that get in small business owners ways.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> To correct for the problems sins of loaning money to people who probably couldn't pay it back, he's loaning money to small businesses that probably can't pay it back?

I guess it's a relatively small amount.. so I'll still continue hating his policy mostly for the decision to tax my charitable donations.

<comment><USER8> I'm guessing you're referring to home loans, which had the lovely property of inflating prices without any correspondence to a real increase in value. There was more money in the system to bid on a fixed set of assets, so prices went up.

Business loans are fundamentally different, aren't they? Sure, a lot of these businesses will fail, but at least the whole point of a business is to create value, and there will be a lot of winners, too. Money has to move around, that's how the economy works. I'd much rather see it moving around trying to drive people to create value, than to consume concrete assets.

<comment><USER7> Yeah.. really though I think I was bitching more than trying to make a real point.

A couple of points though. First, there were a lot of "concrete assets" created in recent years. As simple evidence google "condo glut." Also, you're right - these business loans are fundamentally different. Unless I'm missing something, they're fundamentally riskier - there's no collateral (overvalued collateral is better than none), and the entity you loaned money to is FAR more likely to simply disappear.

As to whether they'll go to better uses - hey - I hope you're right. I hope it helps create more value / helps the economy, etc. However, plenty of people took money "out of equity" in loans and spent it on stuff that created value in a similar manner. For instance, there used to be quite a burgeoning business in selling solar panels to homeowners.. many would argue it's good for the environment, adds value to the home, etc. The interesting part here though is that the salesman weren't really just selling solar panels. Their job was to walk the homeowners through how to get the money in a mortgage "out of equity" on the always-rising value of their home and collect any tax credits available.

<comment><USER8> Condos built at breakneck pace to satisfy demand driven largely by spec rather than need, it's kind of hard to think of those as assets right now (figuratively speaking, of course). The biggest problem is that 'always-rising value' part - that was a complete myth, at least at the pace that was being seen and assumed would continue.

Businesses probably are fundamentally riskier, but there's probably a power distribution in there somewhere, where a few pay off like mad and offset everything else. I'm not advocating irresponsibly handing out business loans or anything like that, but if a bank has the choice between investing in a risky mortgage or a risky business, I guess I would hope they'd pick the business. Then again, I'm not really interested in buying a house anytime soon.

</comment></comment></comment></comment><comment><USER9> So he must have put all of this in the "stimulus" bill since he cares so much.</comment>


Gitfront – Track the most popular front-end repositories on GitHub - <AUTHOR> http://www.schkn.io <comment><AUTHOR> Hello fellow front-end developers, Gitfront is a brand new website that allows us to get the most popular front-end frameworks and libraries on Github. We have introduced in our v2 the ecosystems. Sometimes you want to see the repositories for a special library (ex: Redux), and this feature has been designed exactly for that. But Gitfront needs your help. We have yet to decide the libraries we want to implement for Angular, Vue and Ember. If you suggest a library of those frameworks and your suggestion gets selected, you will be marked as a backer on our repo!

Any feedback on the app so far? Thank you for your time! Gitfront Team</comment>


Let's See What We Know About Russia's 5th Generation Stealth Jet - <AUTHOR> https://theaviationist.com/2019/08/01/sukhoi-begins-production-of-the-su-57-meanwhile-lets-see-what-we-know-about-russias-5th-generation-stealth-jet/ <comment><USER2> Like the F-35s this airplane looks like it has oversized (for a 5th gen) radar and IR rear aspect cross sections. I suppose the idea is with supercruise you're fence out by the time those are concerns. Will it have Lightning II's near science fiction LPI flying LAN? Better search platform integration than America's been able to pull off during Link 16's dotage? The US may need to restart the F22, vary it, and get everyone off of Link 16 to maintain American air supremacy.</comment>


The 25 highest-paid hedge fund managers made a combined $21 billion in 2013 - <AUTHOR> http://www.nytimes.com/2014/05/09/opinion/krugman-now-thats-rich.html

<comment><USER2> This is highly misleading. This money is mostly not from fees, but simply the return made on their own wealth invested in their funds.

See [http://www.bloombergview.com/articles/2014-05-06/billionaire...](http://www.bloombergview.com/articles/2014-05-06/billionaire- hedge-funders-made-money-the-old-fashioned-way)

<comment><USER3> You have a point but that article is also a little misleading. It assumes every hedge fund manager has 100% of his wealth in the funds. To be fair the writer admits that he is just guessing but that is a high guess.

Also, the S&P had its best year in 16 years last year, so you would expect people to do well on their investments this year. The thing that Warren Buffer is very critical of is that hedge funds get a lot of fees even in years that they do really poorly. That sets them up to win big when the market does well.

There is an illusion, cultivated by instutional investors and the people ripping them off, that instutional investors are very savvy. It is not always true.

</comment></comment><comment><USER4> so none of em made as much as the whatsapp guys? bummer wall street, you gotta keep up.

<comment><USER5> I know you're joking, but you're comparing the one-off sale of an entire business to (very possibly repeatable) annual income.

<comment><USER4> Oh sure but they GENERALLY do an amazing job of what they sell("give us lots of money and we'll get you more back") in order to pull down that compensation. The people who hire them(i.e. deposit in their fund products) seem to be cool with the fees(meaning it is "worth it" to the high net worth crowd vs. vanguard or w/e...whether that's rational and based on returns or just prestige of "I have my money with Cerberus"). If it weren't "worth it" then either people would use the alternative lower fee investment vehicles or someone would "disrupt" the space with a cheaper hedge fund offering better returns.

Of course it can all fall apart(LTCM lol)

Plus the huge factor of their personal capital generating a sizeable chunk of this return at the same rate as their fund investor's capital.</comment></comment></comment>


How to learn Deep Learning in 6 months - <AUTHOR> https://towardsdatascience.com/how-to-learn-deep-learning-in-6-months-e45e40ef7d48 <comment><USER2> Here's Peter Norvig arguing that you shouldn't:

 _Teach Yourself Programming in Ten Years_

[https://www.norvig.com/21-days.html](https://www.norvig.com/21-days.html)

The article is about programming, specifically about books etc advising you on how to learn programming in ten days etc, but it totally applies to Deep Learnign also.

Deep Learning is an enormous discipline. In the current point in the hype cycle, there are literally thousands of papers published every month. Personally, I doubt anyone, evern the luminaries of the field are keeping up with it. 6 months is about enough to dip your toes into the preliminaries, and that's if you're willing to skip the relative background [1].

Of course, that's all true if you want to _learn_. If you just want to train yourself in the use of a few tools, then 6 months is plenty. But don't expect to be able to solve any new problems that way- someone else will always have to do all the hard work for you.

________

[1] A bit of background will help with questions such as: Why was Deep Learning necessary in the first place? (hint: search for "constant error carousel"); did Deep Learning just spring fully-formed from the brow of Hinton/ Schmidhuber? (hint: read wikipedia on "machine learning" and "Artificial Intelligence"); etc. If you're not asking those questions then you're holding it wrong.

<comment><USER3> I totally agree with you.

I started Deep Learning Specialization in Coursera: [https://www.coursera.org/specializations/deep- learning](https://www.coursera.org/specializations/deep-learning) last month and almost finished it, but I realized this field requires a lot of expertise, not something you can learn in a month. What I learned in the courses was just a basic topics in Deep Learning and how to use Numpy, TensorFlow and Keras.

I was considering diving into a Data Science job and started that specialization as a starting point, but I just realized how foolish I am. Chances are I'll find a job, but it definitely takes another 10 years/10,000 hours to master this discipline.

Anyway, the specialization is wonderful and Dr.Ng explains complicated Deep learning topics in a way that is understandable for everyone. So if just learning is what you want, you should take it, but I don't think you are prepared for a real world Data Science job after finishing it.

</comment></comment><comment><USER4> How to make background to a picture that is geeky for geeky subjects in 10 minutes:

1\. find javascript file that has been compressed on your system. 2\. open in color coded editor 3\. take screenshot

there you go, perfect 'code' background for your deep learning article!

</comment><comment><USER5> I'm trying to do something similar by starting the other way around (first why it works, then how it works) and so far it's working for me pretty well.[1]

For reference, I'm in high school, so I don't really have the necessary math background to understand everything first try-but I think I'll be able to push through and understand it.

The top down (fast.ai) approach didn't really work for me - as I felt like I didn't really understand how/why everything works. Learning the math behind it forces you to understand.

[1] [https://nikhiljha.com/2018/02/06/Neural-Networks-First- Steps...](https://nikhiljha.com/2018/02/06/Neural-Networks-First-Steps/)

</comment><comment><USER6> Sounds like a project manager. Step 1) Prepare for project. 1-2 weeks. Step 2) Do programming 3-4 weeks. Step 3) Unit tests and QA 2 weeks.

</comment><comment><USER7> My impression is that a lot of programmers have already followed a similar path, would like to find a job doing ML, but haven't found a way to make it happen. Does that sound correct? I'm not saying the effort isn't worth it anyway.

</comment><comment><USER8> How to learn in 3 months:

Complete said material in 3 months.

<comment><USER9> With double the effort too ie instead of 10-20 budget for 20-40 hours each week.

<comment><USER8> For some that might be triple the effort. I guess it depends on _a lot of things_.</comment></comment></comment>


Apache Pulsar is an open-source distributed pub-sub messaging system - <AUTHOR> https://pulsar.apache.org/ <comment><USER3> I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS).

It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to get auth tokens for an AWS IAM role.

Overall, we are very pleased and the rest of the engineering org is very excited about it and planning to migrate most of our SQS and Kinesis apps.

Ask me anything in thread and will try and answer questions. At some point we will do a blog post on our experience.

<comment><USER4> Isn't using Kubernetes kind of an anti-pattern due to failover and rebalancing logic clashing? If Kubernetes is killing and re-starting nodes and the cluster's brokers are detecting dead brokers and rebalancing partitions as a result, it seems counterproductive.

<comment><USER3> This is one of the main benefits of Pulsar is that because state is split between brokers and bookkeeper and bookkeeper doesn't need re-balanced (due to it's segment based architecture where you choose new bookies with each new segment), we really don't have to worry about re-balancing (in general, not just in case of failover) of storage. It is true that topics map to a single broker, but generally, Pulsar has _really_ good limits on memory so we don't see nodes getting killed by limits and we only really see re-scheduling for real issues.

While there certainly is some aspects you need to be aware of, generally, Pulsar is much more "cloud native" and maps quiet well to k8s primitives.

</comment></comment></comment><comment><USER5> We tried to adopt this but found the documentation very lacking and a severe lack of quality client libraries for our language of choice (go).the "official" one had race conditions in the code as well as "todo" for key pieces littered throughout. There is another from comcast which is abandoned. We had a serious discussion about picking up ownership of the library or writing our own but as a small start up we didnt feel we could do it and still develop the product. I'll continue to keep an eye on pulsar but for now Kafka is the clear go to imo. It's well documented, great SAS offerings (confluent) and tons of books and training courses for it.

<comment><USER6> We provide a SaaS offering of Apache Pulsar in AWS, Azure, and GCP: [https://kafkaesque.io/](https://kafkaesque.io/)

<comment><USER7> Cool name. That's one of those company names that almost seems like someone thought it would make a good company name first and thought it was so fitting, they should build a company around it.

<comment><USER6> Thanks!

</comment></comment></comment></comment><comment><USER8> Sidenote question :

Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack.

It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit too high. Is that a valid concern ?

Also, i've seen that kafka is working on removing the dependency to zookeeper, is pulsar going to take the same road ?

<comment><USER9> It's an interesting observation.

I think that the modern approach to distributed systems is moving towards golang style microservices and lightweight / simple system design with RPC communication, reconcile type loops for state reconciliation, and backing CP databases. I think this is the influence of k8s (and maybe google's approach to distributed systems).

I will almost certainly get downvoted for this (as I always seem to when I criticize the JVM), but Apache/JVM style architecture feels REALLY long in the tooth to me. I think you are committing to an outdated and very expensive approach to building software if you use anything running on the JVM, especially Apache based anything. Cassandra is a great example of this - <USER2> of the box it's a terribly performing database that is extremely expensive to run and tune. Throw enough resources and time at it and you can get it to acceptable scalability - but running on the JVM which is a huge memory hog will always make it expensive to run (and even then, you will always get terrible latency distributions with the JVM's awful GC).

If I was building a business I would run far far away from any JVM based solution. The only thing it has going for it is momentum. If you need to hire 100s of engineers off the street for a large project, then a JVM based stack is about your only option unfortunately.

<comment> james-mcelwain > the JVM's awful GC

This just makes it seem like you are trolling. JVM devs have done more to advance state of art in this area than any other language. The problem is that most JVM apps just produce too much garbage, not necessarily that the algo itself is awful.

Either way, there's no such thing as an optimal GC algorithm, just different trade-offs depending on your use case. Not everyone cares about latency.

<comment><USER10> > The problem is that most JVM apps just produce too much garbage,

This is strange way of saying that Java the language and most frameworks around it force apps to generate this much garbage.

<comment> james-mcelwain No, it's a way of saying that Java != the JVM. The JVM's GC is not "awful" precisely because so many Java frameworks _are_ awful.

</comment></comment></comment></comment></comment><comment><USER11> I keep seeing new message queue solutions pop up over the years and it's just been my impression at least that this is one area where silicon valley really is way behind the trading industry.

Reliable pub/sub that supports message rates over 100k/sec (even up to the millions) has been available for a while now and with a great deal of efficiency (eg; the Aeron project). The incredible amount of effort to support complex partitions, extreme fault tolerance (instead of more clever recovery logic), etc. add a lot of overhead. To the point of talking about "low latency" overhead in the order of 5ms instead of microseconds or even nanoseconds as is expected in trading.

Worse, many startups try to adopt these technologies where their message rates are miniscule. To give you some context, even two beefy machines with an older message queue solution like ZeroMQ can tolerate throughput in excess of what most companies produce.

This is not to discredit the authors of Pulsar or Kafka at all... but it's just a concerning trend where easy to use horizontally scalable message queues are being deployed everywhere. Similar to how everyone was running hadoop a few years back even when the data fit in memory.

<comment><USER12> ZeroMQ is not a message queue, it's a networking library.

<comment><USER13> I'll bet he is aware. The problem with the trading industry is that they have hundreds of users with bespoke solutions catering to extreme performance criteria rather than hundreds of thousands like NATS. They will keep reinventing the wheel every time a nanosecond can be saved by the latest hardware stack.

</comment></comment></comment><comment><USER14> This looks promising. Is there such thing as a generalized SQL query engine that runs over any key-value store that provides certain minimal core operations?

For example, say you have a KV Store with basic mathematical Set operations like GET, SET, UNION, INTERSECT, EXCEPT, etc. The Engine would parse the SQL and then call the low-level KV Store Set operations, returning the result or updating KV pairs. This explains how Join relates to Set operations:

[https://blog.jooq.org/2015/10/06/you-probably-dont-use- sql-i...](https://blog.jooq.org/2015/10/06/you-probably-dont-use-sql- intersect-or-except-often-enough/)

Another thing I'd like is if KV stores exposed a general purpose functional programming language (maybe a LISP or a minimal stack-based language like PostScript) for running the same SQL Set operations without ugly syntax. I don't know the exact name for this. But if we had that, then we could build our own distributed databases, similar to Firebase but with a SQL interface as well, from KV stores like Pulsar. I'm thinking something similar to RethinkDB but with a more distinct/open separation of layers.

The hard part would be around transactions and row locking. A slightly related question is if anyone has ever made a lock-free KV store with Set operations using something like atomic compare-and-swap (CAS) operations. There might be a way to leave requests "open" until the CAS has been fully committed. Not sure if this applies to ledger/log based databases since the transaction might already be deterministic as long as the servers have exact copies of the same query log.

Edit: I wrote this thinking of something like Redis, but maybe Pulsar is only the message component and not a store. So the layering might look like: [Pulsar][KV Store (like Redis)][minimal Set operations][SQL query engine].

<comment><USER15> One of the challenges with layering SQL on top of a KV store is query performance.

The most obvious way to model a secondary index on top of a pure KV store is to map indexed values to keys. For example, given the (rowID, name) tuples (123, "Bob"), (345, "Jane"), (234, "Zack"), you can store these as keys:

      name:Bob:123
      name:Jane:345
      name:Zack:234

At this point you don't need or even want values, so this is effectively a sorted set.

Now you can easily find the rowID of Jane by doing a key scan for "name:Jane:", which should be efficient in a KV store that supports key range scans. You can do prefix searches this way ("name:Jane" finds all keys starting with "Jane"), as well as ordinal constraints ("age > 32", which requires that the age index is encoded to something like:

      age:Bob:\x00\x00\x00\x20:123

To perform an union ("name = 'Bob' OR name = 'Jane'"), you simply do multiple range scans, performing a merge sort-ish union operation as you go. To perform an intersection ("name = 'Bob' AND age > 10"), you find the starting point for all the terms and use that as the key range, then do the merge sort.

This is what TiDB and FoundationDB's record layers do, which both have a strict separation between the stateless database layer and the stateful KV layer.

The performance bottleneck will be the network layer. Your range scan operations will be streaming a lot of data from the KV store to the SQL layer, and potentially you'll be reading a lot of data that is discarded by higher- level query layers. This is why TiKV has "co-processor" logic in the KV store that knows how to do things like filter; when TiDB plays your query, it pushes some query operators down to TiKV itself for performance.

Unfortunately, this is not possible with FoundationDB. This is why FoundationDB's authors recommend you co-locate FDB with your application on the same machine. But since FDB key ranges are distributed, there's no way to actually bring the query code close to the data (as far as I know!).

I'm sure you could do something similiar with Redis and Lua scripting, i.e. building query operators as Lua scripts that worked on sorted sets. I wouldn't trust Redis as a primary data store, but it can be a fast secondary index.

<comment><USER16> FoundationDB's client bindings have a locality API which allows you to query the client's metadata cache of which key ranges are on which storage processes. This would allow you to build that feature of routing a query to the data.

<comment><USER15> I didn't know that. Very cool, thanks!

</comment></comment></comment></comment><comment><USER17> For a moment I thought Bajaj and TVS came together.

<comment><USER18> Captain here.

TVS and Bajaj are major motorbike manufacturers in India, and TVS had a model named "Apache" and Bajaj had a model named "Pulsar".

Flies away

<comment><USER19> Thank you for the explanation.

</comment></comment></comment><comment><USER20> This might be entirely off topic, but I'm having issues using RabbitMQ whereby durability suffers because messages are sent to remote hosts thus exposing them to both the network and remote host availability. On a previous platform I used an MSMQ based system which didn't have this problem since it uses a local store and forward service. So all sends are to localhost and are not affected by the network or the receiver availability. The MSMQ system was my first and only experience with messaging up to now, so I was surprised that any system would not work that way. How is this dealt with in other systems? Is it just a feature that exists or not and you just decide if it's important? And maybe just to shoe horn it to be on topic, does Pulsar use a local service?

<comment><USER21> That's an inherent issue with distributed solutions and is impossible to solve. The only way to deal with it is using various techniques like acknowledgements, retries, local storage, idempotency, etc. MSMQ handles that stuff behind the scenes but the problem itself will always exist if there's a network boundary.

These other systems are designed to be remote with a network interface. You can use the client drivers to handle acknowledgements/retries/local-buffering in your own app or use something like Logstash [1], FluentD [2], or Vector [3] for message forwarding if you want a local agent to send to. You might have to wire up several connectors since none of them forward directly to Pulsar today.

Also RabbitMQ is absolute crap. There are better options for every scenario so I advise using something else like Redis, NATS, Kafka, or Pulsar.

1\. [https://www.elastic.co/products/logstash](https://www.elastic.co/products/logstash)

2\. [https://www.fluentd.org/](https://www.fluentd.org/)

3\. [https://vector.dev/](https://vector.dev/)

<comment><USER22> > Also RabbitMQ is absolute crap. There are better options for every scenario > so I advise using something else like Redis, NATS, Kafka, or Pulsar

Yikes, that's a bit harsh! I've been using RabbitMQ on multiple projects for several years, and I think it's a great pub/sub system. It also has a large userbase and community around it, as well as lots of plugins available.

I've never heard of using Redis for pub/sub - were you suggesting rolling your own on top of Redis? I'm not familiar with NATS (but it's been mentioned several times here, so I will definitely learn more!), but Kafka is a streaming log/event system, not a pub/sub system like RabbitMQ (different use cases).

I will say though that if something is wrong in your RabbitMQ config, the stack traces that Erlang produces when it crashes are a _nightmare_ to decipher!

<comment><USER21> Just because it's widely used doesn't mean it's good (see PHP). It's slow with single-threaded topics that usually max out around 25k msgs/sec, fragile with dropped connections, stalled queues, corrupted data, terrible clustering that breaks often and doesn't support sharding, has a silly HIPE mode where you can choose to compile the Erlang code for more performance which turns startup time into minutes, etc.

It's poor architecture and implementation. One of the worst products built with Erlang. It's hard to work on (for new contributors) while not really using any of the natural advantages of the language.

Redis supports pub/sub channels. It's very fast and if you already use it then it saves running another system.

Streaming log vs pub/sub is a fuzzy distinction and basically has no difference at this point. Publishers send to a topic and consumers listen. You can do the ephemeral pub/sub in Kafka by reducing retention to seconds, using a single partition per pub/sub topic, and having consumers listen from latest without a consumer group. Or use Pulsar which does both much more naturally.

Also I didn't mention but there are great commercial messaging products like AMPS [1] or Solace [2] if you need much more advanced features and support.

1\. [https://www.crankuptheamps.com/](https://www.crankuptheamps.com/)

2\. [https://solace.com/](https://solace.com/)

<comment><USER22> I didn't k ow about Redis pub/sub, I'll check it out! With a quick glance, I'm not sure it supports message durability (persisting messages to storage)?

IME, _any_ Erlang system is difficult for noobs to contribute to; Erlang's syntax alone is... frightening :)

I really haven't hit any of the issues you mentioned with rabbit. Several clusters in production for years have been rock solid, connections are stable (and when they do drop, the client libraries, of which there are a multitude, can handle reconnection), throughput is excellent, and I've never once seen data corruption (and combined we must have processed many billions of messages).

The whole HIPE debacle I can at least agree on though! (I seem to recall it's deprecated now, but I might have dreamed that)

<comment><USER21> Redis pub/sub is ephemeral. Send to a topic and any active listeners will receive.

If you need persistence then Redis v5 has a new data type called Streams which is similar to Kafka/Pulsar. Push messages to a stream and read with multiple consumers (using consumer groups) that can ack each msg to track read state.

</comment></comment></comment></comment></comment></comment><comment><USER23> I really enjoyed the talk given by Quentin Adam & Steven Le Roux at Devoxx 2019. It gave a great overview of Apache Pulsar. Hope someone else can find it useful too!

[0] [https://www.youtube.com/watch?v=De6avNyQUMw](https://www.youtube.com/watch?v=De6avNyQUMw)

</comment><comment><USER24> Splunk just acquired streamlio and most of the core devs got sucked up. While pulsar is a great product - are you not concerned that these guys are getting paid $$ bank to do something else now?

<comment><USER25> spoiler: we're still working on Pulsar

<comment><USER24> That's good news! I guess it would be very helpful to formally address this concern. Is there something that has been written / published to that effect?

</comment></comment></comment><comment><USER26> How does this compare to Redis Pub-Sub or RabbitMQ?

<comment><USER27> Very different. Pulsar is primarily a Kafka competitor.

\- it is much more performant than RabbitMQ \- it's a commit log as well, not just a pub-sub system, ie. it is a good candidate as the storage backend for event sourcing \- it supports geodistributed and tiered storage (eg. some data on NVMe drives, some on a coldline storage) \- it's persistent, not in-memory (primarily)

.. and so on.

<comment><USER28> What about ZeroMQ?

Why use RabbitMQ and Kafka if you can use ZeroMQ? Meaning, isn't it far more performant and distributed?

Maybe I am missing something here.

<comment><USER27> > Why use RabbitMQ and Kafka if you can use ZeroMQ?

They are totally different, you're comparing apples with oranges.

ZeroMQ gives you basic, very fast tooling to communicate between distributed processes. ZeroMQ does not provide tooling for e.g. maintaining a strictly ordered, multi-terabyte event log. And so on.

<comment><USER28> Yes but isn't this a bit like comparing git / bitkeeper vs subversion / perforce?

Basically, one is decentralized and you can set up a massively parallel architecture, with eg each topic or subthread having its own pubsub.

The other is a monolithic centralized pubsub architecture.

You could argue that git in large institutional projects converges to a monolithic repo so at that point it's less efficient even than svn.

But for most use cases, ZeroMQ would allow far more flexible distributed systems topologies and solutions. No?

Edit: HN and Google are both awesome: [https://news.ycombinator.com/item?id=9634925](https://news.ycombinator.com/item?id=9634925)

<comment><USER29> Zeromq is just a bit of sugar on top of tcp sockets. It isn't a message queue or anything close. You would be wasting a ton of time reimplementing a lot of basic features like retries, persistence, service discovery, dead letter queues, priority, and a ton of other stuff.

</comment></comment></comment></comment></comment></comment><comment><USER30> I'm still on the fence with these distributed log/queue hybrids. From a theoretical perspective it seems these are excellent. I just have this nagging suspicion that there is some even-worse problem architectures based on these systems will harbor. This kind of ambivalence is something I find myself having to battle more and more in my career as I age. Most of the time the hype around new design/development patterns leads to a worse situation. Very rarely it leads to a significant improvement. I dislike that my first impression looking at a system like this is risk aversion.

<comment><USER31> Your risk aversion seems justified. It seems reasonable to estimate that very few teams are in the position of needing the kind of scale/scalability that something like Apache Pulsar offers. They are much more likely to be in either a state where they will not put Pulsar through its paces or where they already have a solution in place that serves their scale/scalability needs.

When a team you are on starts discussing switching over to a technology like Pulsar because of its amazing benefits, unless your pants are on fire, it is much more likely than not that you do not stand to gain much from the benefits that such software brings but you are accepting the maintenance burden that it represents.

</comment></comment><comment><USER32> A lot is said or referenced in this conversation about why people chose Pulsar over Kafka. I'm not an expert in this area but are there use cases where Kafka is still better?

<comment><USER33> As someone with a few years of Kafka and the ecosystem under my belt, but no experience of using Pulsar in anger, the areas I can see where Pulsar is behind are mainly ancillary, and will likely be caught up by the community given a year or two.

Kafka Streaming - Pulsar functions don't intend to (by the looks of it) provide all of the functionality available in Kafka Streaming. E.g., joining streams, treating a stream as a table (a log with a key can be treated as a table), aggregating over a window. They seem to be more focused on do X or Y to a given record. That said, you don't need Kafka Streaming for that, other streaming products like Spark Streaming can do it also (although last I checked, Spark Structured Streaming still had some limitations compared to Kafka Streaming - can't do multiple aggregations on a stream etc.) A use case I have and love Kafka Streaming's KTables for is enriching/denormalising transaction records on their way through the pipeline.

Kafka Connect - Pulsar IO will get there with time, but currently KC has a lot more connectors - for example, Pulsar IO's Debezium source is nice, (Debezium was built to use Kafka Connect, but can run embedded), but you may not want to publish an entire change event stream onto a topic, you might just want a view of a given database table available - so KC's JDBC connector is a lot more flexible in that regard, and Pulsar IO currently doesn't have a JDBC source. It also looks like its JDBC sink only supports MySQL and SQLite (according to the docs) - KC's JDBC connector as a sink has a wider range of support for DBs, and can take advantage of functionality like Postgres 9.5+s UPSERT. Likewise, there's no S3 sources or sinks - the tiered storage Pulsar offers is really nice, but you may only want to persist a particular topic to S3.

KSQL - KSQL lets your BAs etc. write SQL queries for streams. That said, I do like Pulsar SQL's ability to query your stored log data. When I've needed to do this with Kafka, I've had to consume the data with a Spark job, which adds overhead for troubleshooting.

So yeah, that's the main areas I can see, but it's really a function of time until Pulsar or community code develops similar features.

The only other major difference I can see is that at the current time, it's comprised of three distributed systems (Pulsar brokers, Zookeeper, Bookkeeper) which is one more distributed system to maintain with all the fun that entails.

That said, I'll be keeping my eye on this, and trialling it when I get some spare time, as I've found that people will inevitably use Kafka like a messaging queue, and that is a bit clunky. Plus I'm a little over having people ask me how many partitions they need :D

<comment><USER34> Do they usually provide similar throughput on the same hardware?

<comment><USER33> I couldn't say tbh, I'm keen in running a trial with Pulsar alongside Kafka in production, might write it up when I've done so.

</comment></comment></comment></comment><comment><USER35> From reading their documents, I really like the design of Pulsar. However, Kafka has been working so well for us and has much better integration with other components of our stack (Flink, Spark, NiFi, etc) that there's no compelling reason to switch.

I think Pulse should really focus on the integration with the rest of the Apache stack if they want to gain traction.

</comment><comment><USER36> Being able to scale the durable-storage layer independently has a lot of advantages. More thoughts here: [https://twitter.com/breckcs/status/1203736751681896449](https://twitter.com/breckcs/status/1203736751681896449).

</comment><comment><USER37> Most of the comments are just pro-Pulsar but what's the architectural trade- off? (Non-architectural trade-off is that Pulsar is a new system to learn for folks familiar with maintaining and using Kafka.)

<comment><USER21> Pulsar is better designed than Kafka in every way with the main trade-off being more moving pieces. That's why the recommended deployment is Kubernetes which can manage all that complexity for you.

Pulsar also lacks in the size of the community and ecosystem where Kafka has much more available.

</comment></comment><comment><USER38> How does this compare with NATS?

<comment><USER39> NATS is a simpler PUB/SUB system that delivers in the UNIX spirit of small composable parts. Apache Pulsar or Apache Kafka deliver the banana, the ape holding it and the rest of the jungle.

<comment><USER12> Check out Liftbridge ([https://liftbridge.io](https://liftbridge.io)) as a way to add these capabilities to NATS.

<comment><USER40> your FAQ still says it's not production ready? Is this still the case, I've been keeping my eye on this project

<comment><USER12> It's getting very close. I had wanted to make a production-ready 1.0 release before the end of the year, but we're in the process of switching from protobuf to flatbuffers. Once that is complete, a stable release will be made.

</comment></comment></comment></comment></comment><comment><USER41> How is it compared to kafka?

<comment><USER42> Most of the flaws of Kafka are carefully studied and fixed in Apache pulsar. I have written a blog about why we went ahead with pulsar [https://medium.com/@yuvarajl/why-nutanix-beam-went-ahead- wit...](https://medium.com/@yuvarajl/why-nutanix-beam-went-ahead-with-apache- pulsar-instead-of-apache-kafka-1415f592dbbb)

<comment><USER43> > when consumers are lagging behind, producer throughput falls off a cliff > because lagging consumers introduce random reads

I am confused by this. The format of Kafka's log files is designed to allow reading and sending to clients directly using sendfile, in sequential reads of batches of messages. [http://kafka.apache.org/documentation/#maximizingefficiency](http://kafka.apache.org/documentation/#maximizingefficiency)

<comment><USER21> Kafka brokers handle connections to consumers and data storage. This creates contention as the primaries for each partition have to service the traffic and handle IO. Consumers that aren't tailing the stream will cause slowdowns because Kafka has to seek to that offset from files which aren't cached in RAM.

Pulsar separates storage into a different layer (powered by Apache Bookkeeper) which allows consumers to read directly from multiple nodes. There's much more IO throughput available to handle consumers picking up anywhere in the stream.

</comment></comment></comment></comment><comment><USER21> This is one of the best overviews on Pulsar with comparisons to Kafka: [https://jack-vanlightly.com/blog/2018/10/2/understanding- how...](https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache- pulsar-works)

</comment><comment><USER44> Is there a reason you went with Pulsar over Kafka? How is the pulsar community? Where are you turning when you have support issues?

</comment><comment><USER45> This is great !! What would be the easiest way to run a 3 node cluster ?

<comment><USER46> The standalone mode will let you get started as a developer. You grab tar.gz, uncompress, run standalone.sh.

There are helm charts for running an actual cluster.

</comment></comment><comment><USER47> Would Pulsar be suitable for IoT messaging? An alternative to mqtt?

</comment><comment><USER48> "high-level APIs for Java, C++, Python and GO", no love for Node.js? :(

<comment><USER49> It exists: [https://pulsar.apache.org/docs/en/next/client-libraries- node...](https://pulsar.apache.org/docs/en/next/client-libraries-node/)

<comment><USER50> However it currently lacks the ability to listen for messages and run an event handler when one comes in: [https://github.com/apache/pulsar-client- node/pull/56](https://github.com/apache/pulsar-client-node/pull/56)

You have to manually call ".receive()" to attempt to receive a message.

<comment><USER51> Using `.receive()` will occupy a worker thread from node until it returns. Having multiple consumers waiting on receive will clog up the worker threadpool, preventing anything that uses it from running. If you want to use the consumer right now, I would suggest always using a timeout on the receive call, and waiting between timed-out calls to receive. This is extremely important if you have multiple consumers.

</comment></comment></comment></comment><comment><USER52> I'm sure Pulsar is worth it if you use most of what they're offering, but the Java client library is crusty, throws exceptions for control flow. I'm looking at a persistence mechanism built on top of NATS to replace it. The NATS layer would make it simpler to decouple the gateways from the persistence layer, and support our bulk computing needs.

</comment><comment><USER53> Why is apache developing all those servers that are only useful to a handful of companies that are rich enough to build them themselves? How about building something that individuals can use, like, i dunno, apache server itself?

<comment><USER54> If you did bother to read the linked page, you would have understood that it's a yahoo project handed over to Apache for management like many of Apache's projects.

<comment><USER53> yeah i m talking more generally about their full list here: [https://www.apache.org/](https://www.apache.org/)

<comment><USER55> There are a lot of projects that were handed to Apache to manage. Kafka for example was initially created by LinkedIn. So yeah, you are right, big corps are actually creating those tools, and in addition to this, giving it away as open source to public.

<comment><USER56> Best part is that the tools are put into production before being open-sourced. In other words, they actually work.</comment></comment></comment></comment></comment>


Einstein Couldn't Have Imagined Technology to Directly Detect Gravity Waves - <AUTHOR> http://www.scientificamerican.com/article/even-einstein-could-not-have-imagined-technology-used-to-directly-detect-gravitational-waves/

<comment><USER2> As is so typical of Scientific American these days, there's no information whatsoever about the technology itself, other than that it includes lasers and various precision optics.

[http://www.cfa.harvard.edu/CMB/bicep2/instrument.html](http://www.cfa.harvard.edu/CMB/bicep2/instrument.html) which isn't included in the article either.</comment>


Ask HN: Will tech be saturated in 10 years? - <AUTHOR> Will better tooling (think WebFlow or Google's AI as a service) and more accessible training lead to:<p>- more devs -> lower wages for employees and;<p>- more startups -> higher funding requirements<p>for new entrepreneurs? <comment><USER2> Technology has been and and is increasingly so following a bimodal income distribution. There are two distinct groups.

1.) The people who work for FAANG corporations or high profile startups, a few places in finance. In depth knowledge of computer science required to get these jobs.

2.) People who are writing UIs, data flows, python apis, etc. Run of the mill software development.

The income for these two groups looks like this:

The top 5% (maybe less) in technology are making really good money, 250-1M. This is the group 1, FAANG group.

The second group:

The lower 95% are generally stuck between 75k and 175k. They are seeing increasing competition from Europe, China, and boot camps here in the USA.

So to answer your question, if you take writing software very seriously, and can out compete most engineers, its an amazing career and will only get better. If you are a middling programmer, youre better off doing something else.

<comment><USER3> _If you are a middling programmer, youre better off doing something else._

There are a massive amount of people that would absolutely _love_ to be "stuck between 75k and 175k."

<comment><USER4> There is an implicit belief by many people that you're a failure if you're not trying to get super rich in life.

</comment></comment></comment><comment><USER5> As time goes on and more people enter the industry, I wonder if tech as a whole will come to mirror the video game industry more closely. In video games, it does seem like the oversupply of programmers & their desire to work in the industry no matter what causes lower wages and longer hours for employees relative to the rest of the tech world. And in the independent video game development scene (which might be more analogous to startups), the number of financially unsuccessful games is growing massively as more and more people try to get involved. It seems to be at the point where you need to go into that space assuming it will be unprofitable.

I guess it depends on the talent distribution of the influx of new aspiring engineers -- after all, in some contexts it doesn't matter if 2-3x more people are lining up to willingly do your job if they don't have the skillset to actually replace you. But it could be unfortunate for the jobs that don't have the luxury of being protected by high barriers to entry.

</comment><comment><USER6> It's already saturated IMHO. Hackernews is an obvious example of it. There is more tooling than you can shake a stick at, but the real question is who is going to leverage all these tools and curate them so that they pick the right ones, and most importantly, leverage them at _the right time_? I say the right time, because tools come and go, depending on how many people are using it. Some tools simply stay because they have a loyal userbase/customer-base, others vanish because they are under utilized. I doubt there will be higher funding requirements since these days we have nearly infinite leverage, and leverage without permission. It's now permissionless to invest, use tooling, train people, etc

<comment><USER7> > It's already saturated

Given that basically every company I've seen the hiring process of has a lot of trouble attracting programmers of even average competence I have trouble believing this is true.

</comment></comment><comment><USER8> I'll cyte my brother who is a Phd currently teaching in top 5 Technical University in my country:

\- "Listen, your year was pretty stupid, but you did manage"

\- "Now, students are so bad they cannot even do a simple diff.."

I'm not afraid about the future. We are lacking here 50 000 software developers and this number will only grow because less and less finish studies or represent a decent skill level either way.

<comment><USER9> I'm one I'm those

</comment></comment><comment><USER10> Don't underestimate the compounding effect of innovation.

Uber would have been a giant flop just one decade previous.

Everyone having a internet-connected smartphone was a requirement for that business to succeed.

</comment><comment><USER11> In 10 years? I guess, not. Maybe 20 or 30? The world will have a different set of problems 10 years from now. A different set of verticals, regulations to work with. A different amount of platforms or hardware (quantum computers, more accessible VR/AR).

Take for example the current situation with agri-tech (vertical), it's pretty boring because it doesn't pay as much as putting different colored boxes in web pages for advertisers. With that vertical, you also have to take in localization for a region. Agri-tech vertical is nowhere near "tech-saturated" in other parts of the world. Random thought: it wouldn't be far-fetched to say agri-tech would pay a ton more if global warming went crazier in 10-20 years.

In 10 years, we can get better tooling, but if regional regulations/laws hamper new tech, we'll see tech-startups develop for that specific region. Things like data privacy laws in other countries would lessen usage of public clouds and more in-house data centers. Trade-wars also spawn events like Github/Google blocking developers based in other countries, etc. So you'll see more startups taking advantage of that by developing localized tooling leveraging regulations and laws.

Take for example, Alibaba. 10 years ago (~2009), you wouldn't know about that company unless you're from China. Now they're the Amazon/Google for a closed- economy of more than a billion people, more than twice the population of US. Grab, Paytm, Line are just getting started. So we'll probably saturate in 20-30 years? We might see more or less of these companies though, depending on the different set of problems/laws we'll see in 10 years.

</comment><comment><USER12> It's hard to imagine that the market for devs who have fluency, understanding, wisdom, and depth, who can quickly and reliably deliver value against challenging problems, could ever be saturated.

People calling themselves software engineers, absolutely. People who can pass today's interviews, doubtful but maybe. People who are actually good, who can get mental hooks into what is going on and use their tools fluently to address it, probably not.

<comment><USER13> > People who can pass today's interviews, doubtful but maybe. People who are > actually good, who can get mental hooks into what is going on and use their > tools fluently to address it, probably not.

Wildly amusing that these two things are mutually exclusive. There's a clear difference between an expert engineer/architect and an expert computer scientist.

</comment></comment><comment><USER14> assuming it's a fixed pie, which clearly it isn't. If you simply walk around you'll notice basic things that could and should be automated in some way that shows it's yet to reach the limit.

</comment><comment><USER15> There are more rich clubs than great football players. There are more new apartments than great electricians. There are more great projects than great programmers.

If you want to be payed well you need to be great as competition is stiffer than ever. Times when it was possible to create a site in Dreamweaver and sell it for $50k are long gone.

The speed of progress is not slowing down. Everybody is talking about the next recession yet somehow Softbank is about to pump another 100B into the system. India outsourcing their dev work to Philippines, Africa on the rise. AI/VR redefining marketing and entertainment industry. New 2.5T climate change market. We haven't even started to use tech to solve the really difficult problems of the civilization yet.

Don't want to participate in this chaos? Choose a quiet IT sector like security or software maintenance.

<comment><USER16> I would't say that security is a quite sector. Just google IT security innovation and you will see a lot of movements.

</comment></comment><comment><USER17> Your question sort of misses the bigger problem: if you're a developer over a certain age (35? 40?) you're completely unemployable in many areas. If you expect to hang on for 10 years you are also expected to move up to management, otherwise employers assume there is something wrong with you and you will never get interviews.

Then you get the elites and the children of the elites who assume that everyone who didn't make a billion dollars by the time they turned 30 has some defect with them. These people will never hire old developers either.

It's a horrible attitude, but very pervasive in tech. Expect to move on from development by some point, figure out what else you want to do after that and start reading more books.

<comment><USER18> I'm trying to come to terms with this inconvenient truth as a pretty good developer in my 30s. At some point you just can't compete with the 20 year old recent grad drinking Jolt cola and working till midnight. You have experience on them, but also a lot of other things that hold you down: family, mortgage, etc. The future looks very confusing.

</comment></comment><comment><USER19> I'm navigating my career as if the answer will be 'Yes'. It's hard to predict on a 10 year horizon. If a recession hits it might feel similar too: fewer opportunities but the same number of applicants.

<comment><USER20> What career choices are you making based on that assumption?

<comment><USER19> Some standard stuff: high savings rate, aggressive career progression to stay ahead, no specialization (which might get oversaturated e.g. all these new ML programs that colleges spin up).

<comment><USER20> I asked because none of the things you've mentioned are "standard". Except for the savings rate maybe.

Aggressive career progression - in a recession, it's often harder for senior people to find a job. Especially for managers (how many companies are looking for senior technical managers in your area?). _Not_ advancing might be a valid preparation strategy.

No specialization - Again, there's something to be said for specializing in _core_ IT areas which aren't booming. For instance storage technologies, networking, distributed systems, databases and data engineering. Heck, even being a top expert on something like JVM garbage collection. These areas are more mature and have a lower risk of oversaturation. Yet both big and small companies always need people with _expertise_ in these fields.

<comment><USER19> Re: career advancement, there's definitely a sweet spot, "senior" (google L5) makes 3x starting, beyond that it's diminishing returns and gets further from code. As long as I write code I can phrase my resume to get a programming job.

Re: specialization, seems risky to me, either way full stack suits my interests.

</comment></comment></comment></comment></comment><comment><USER21> I don't think tech will ever be saturated. Now, if we are asking if the tech of today will be saturated in 10 years, I would say it's possible, maybe even likely. We used to have fax machines and type writers, which got saturated and gone. A more interesting question might be, what technology will be relevant in 10 years?

To add some color, I rewatched Richie Rich today which shows that 20 years ago, we thought that having a digital assistant that can locate in real-time someone was something that only a fictional billionaire could have -- we have that in our pockets today.

</comment><comment><USER22> Part of me wonders if software engineering becomes more and more like manufacturing jobs. The jobs move overseas to whatever country has the cheapest and most talented workforce. Be it India, China, Philippines, Eastern Europe, Africa, etc.

There are people willing to do software development for $3-$10 an hour all over the world.

<comment><USER23> Guess what, over 70% of software project fail. The cycle of 'saving money' via outsourcing and offshoring never ends. "This time is different" continues.</comment></comment>