Tools you'd miss if you left a company - <AUTHOR> http://rachelbythebay.com/w/2020/02/08/miss/ <comment><USER2> Not everyone finds BigCorp problems to be the interesting problems. Not everyone wants to live in the Bay Area. There's a lot of jobs in the world that aren't at FAANG, and it's okay if the right job for someone is at a place whose entire dev shop is smaller than the team at Google that maintains Blaze. If you find yourself in a job like that, it is absolutely the right call to invest in the best commodity tools you can (where best is a local, not a global, maxima) and focus on the things you can do that add value to your org.

<comment><USER3> There seems to be an elitism when it comes to working for faang. there are a ton of people that have zero interest in building advertising engines or living in the Bay area. Quite frankly my min salary for having to live there sacrifice working on something interesting with be mid to high 6 figures so I could save and move the hell away after a few years. I'd consider it a life sacrifice to endure the bay area and culture.

<comment><USER4> I agree with your points but just want to add my anecdotal experience here from India that not all companies with great internal tooling are unicorns. They generally tend to be the slow but steadily moving companies who take their time and think about sustainability rather than 10x growth.

<comment><USER5> Yes. And I might suggest that the unicorns with great tooling just lucked out and has engineers who built great stuff irrespective of management

</comment></comment></comment></comment><comment><USER6> I have to strongly disagree. When doing business you are solving a business problems. Facebook, Google, Microsoft as business has grown so much they have problems of scale no open tools are able to handle. On the other hand, any kind of smaller company doesn't have those business problems, but a cost of having and maintaining in-house solutions of that scale will just effectively topple and sink them.

This is why I'm always frowning during job interviews with engineers telling stories about they migrations from monolith to microservices and putting it all into k8s, while their headcount in the firs tens and their DAU in about 10000 users.

Disclaimer: I worked in Yandex, virtually Russian Google. Loads and operations scale there are comparable with second rank international tech giants. Yandex has and have lots of internal tools to handle that load and scale, but I've almost never really _needed_ them in smaller companies. Maybe "nice to have" couple of times.

<comment><USER7> It's also a lot about the marketability of your skills.

The only reason the OP can boast about the experience with these proprietary tools is because they're used by FAANG. No other reason — it's just an indirect and inconspicuous boasting about having worked at FAANG.

If it was yet another proprietary non-OSS language used by one of the investment companies that has no direct analogues at any of their competitors — good luck boasting on your resume that you have skills in programming languages that noone cares about.

Proprietary software sucks. Just because it's written and used internally by FAANG doesn't make it suck any less. When you're evaluating whether or not to join a company, presence of such tools should be more of a red flag, if anything. Same goes with software-as-a-service. Tools should be OSS, and easily portable between the workplaces.

Another case in point — it kind of works both ways — many companies start around with Erlang and ejabberd, because it's a really great tool, but then, after having issues finding developers who can understand it properly, they eventually, one way or another, convert their stack back to the more common languages; I recall this has happened with Facebook Messenger and Heroku, and also at a random startup I was interviewing with during the course of my interview (they just hired a manager who knew Java — and, what do you know, he decides to set a direction to eventually replace all Erlang with Java).

</comment></comment><comment><USER8> At one of the places I worked, a relatively large university-hospital, there was an entire Java-UI framework build based on C#s XAML. But it could also experimentally be compiled to html/css/js.

Writing any type of UI in Swing is just painful - but this system made it so much nicer. It also had a WYSIWYG editor that you could interop with actually writing the code. Plus you could write either the 'markup language' or the "Java" flavour directly.

Making complex responsive UIs was a breeze. It's a shame it did not get open sourced - but it was tied into a great amount of internal tooling. (The entire codebase was 28+ million lines of code when I left).

It's one of the few non-open source libraries I've worked with that really impressed me.

<comment><USER9> I am very curious what the university-hospital was, if you can share it.

<comment><USER8> A Belgian one (Leuven) ;-)

</comment></comment></comment><comment><USER10> Doesn't it take months to train new developers in these mystical internal tools? I remember hearing specifically that Google expects 5+ months of training until a developer is productive. How many companies can afford to spend ~50k+/hire on on-boarding ((100k/yr * 5 months) + benefits)? What is the opportunity cost of holding so many people in training mode for this time period?

Different companies have different problems. Big companies need big expensive tools to help with their problems. Little companies usually get by with smaller more portable tools. Let's not use this to demean people at small companies with some false sense of superiority please...

<comment><USER11> Given the background of the blog's author, I wouldn't hold my breath

<comment><USER12> What, a successful engineer with tenure at two of the most successful business operating at scale?

</comment></comment></comment><comment><USER13> I'd miss the engineering machine shop. Those facilities only get developed by accident, for instance if a company gets rid of its professional machinist but keeps the equipment, or something like that. The machines and collections of tooling take a long time to curate. Any modern manager would refuse to allow that much stuff to be purchased if they were asked, and believe in the magic of outsourcing everything.

<comment><USER14> At Apple Inc in the Sculley days (long ago) there was a lab in .. Monroe? .. the guys there were super orderly.. sports dads with good jobs.. that sort of thing.. I asked one day, what do you do here? They had so much precision equipment in one place.. the guy said that the did compliance testing for standards? and, some performance/spec testing? He looked slightly proud, and said "they had won Industry Awards" .. a year later they were gone. All the equipment, and I suspect most of the guys. The VP name was Moon? seemed like outsourcing and unwise to me..

</comment></comment><comment><USER15> This whole premise falls completely flat if you've only ever worked at small and/or young companies:

> but which didn't exist on the outside world.

> You're not allowed to include things you or your friends built

If you work for a service provider, you probably have zero to not much internal tooling, because you're usually solving the customer's problems, not your company's. When I did, we open sourced everything cool we built - sometimes tried and true defaults, or sensible Vagrantboxes or something like that. (Today that would be Dockerfiles, I guess) - and I consider myself lucky to have worked at a smaller company that was close to the FOSS scene, and not something like Accenture. Companies like ThoughtWorks might be an outlier here.

You're unlikely to build something so immensely cool in your first 1-2 years in a startup that it will change your live (as a developer) completely. (This is not about your company's product, but internal developer tooling, so if your startup does developer tools as a product, you're the 1% where this would apply).

And finally, if your startup grows.. you might probably still not be the next Google or Facebook where the internal team providing nice things to your fleet of developers is bigger than some companies. Internal tooling will often be seen more of a cost factor than a tool for productivity enhancement. And often it's actually the case. In my current company we just have so many software development teams creating parts of our products.. so many languages (with good reason) - the only internal things I could think about that would even make sense for all of them would be some code review tool, or anything targeting source control or maybe CI. Anything else would have to be really workflow-centric and language-agnostic. Anything regarding the mentioned borg/k8s/BigTable would be completely useless, for example.

That said, at my last company, I think we created some pretty cool things that made a lot of things a lot easier for the team. But we were two small teams, so point to "we built it ourselves, for ourselves" applies. Also it probably wasn't so revolutionary in the greater sense, but everything made sense for that company and I'm often still sad if I have to solve some similar problem these days and don't have my old setup (more than a specific tool).

<comment><USER16> Counterpoint: Rackspace. The line internally was "service, not servers". They developed CORE which was their customer tracking and computer provisioning system. When RightNowWeb stopped working for them, they built the CATS ticketing system on top of that.

Those were nice tools which worked quite well most of the time. They worked in any browser on any OS, unlike some of the ActiveX-only stuff they tried to replace it with.

This is not the Rackspace you probably know. This is the teeny little one that nobody had heard of outside of a few advertisements in Linux Journal (and the occasional rant about spammers and malware hosting). This is them back before the whole cloud thing - over fifteen years ago.

<comment><USER15> Point taken. Rackspace at least seems to be technical enough to be able to put engineers first. :)

I've not had so many jobs at product companies, but I've never seen this even when I was doing consulting-type work and came around a bit, and many companies (at least here in Germany) being decades old are simply not digital- first. Maybe the mechanical engineers building the (non-computer) hardware have enough clout, but software is really often not more than a cost factor. I guess the bigger and older the company the more likely that everyone hates internal tooling, and you were focusing on the exceptions, whereas my life was the opposite. Waiting for an internal server just to deploy your shiny new thing could take months.

</comment></comment></comment><comment><USER17> This really resonates with me. I have seen the "buy before you build" attitude turn into "buy because we can't build". I know correlation is not causation and there are always plenty of other things going on... but it sure seems like the pride in our work dropped off and we were on the wrong side of every tipping point.

<comment><USER18> I've seen this at a few places now. Learned helplessness is definitely a thing and a lot of it comes down to an inability or unwillingness to hire broadly. When you're hiring Bolt Turner #3, they might know a little bit about turning bolts (but might not, either) but expect everything that isn't a turnable bolt to be handed to them. But the screw turners and the pipe welders you've also hired don't think a working ecosystem of tooling is their responsibility either and they're not really equipped to build it either, so the only option becomes to buy. And at that point you lack the breadth and the depth of knowledge to effectively choose the right thing to buy, and the outcome is as if by dart board.

There's a reason to hire specialists in this industry, but we conflate one- track ponies with specialists and it's to our detriment.

</comment></comment><comment><USER19> Couldn't it be argued that all software is built with more "glue" than non- glue? That's just the nature of the beast. If a user wants to, say, view a record from a remote database in her web browser, how many separate layers of code does that involve? (Several.) How many of them aren't "glue"? (I dunno, like two?)

Perhaps it's true that most glue is poorly written, but doesn't that imply that writing good glue is, in fact, difficult, and therefore it's as hard — or maybe harder - than building "real" things?

</comment><comment> yoz-y I don't know. Having an "80% solution" but not needing to support or fix it is not such a bad price to pay. Also this seems to be a bit skewed towards large networked solutions.

</comment><comment><USER20> Wanted to see what Facebook Scuba was, and found this comment thread that really highlights the point of this blog. [https://news.ycombinator.com/item?id=13463016](https://news.ycombinator.com/item?id=13463016)

<comment><USER21> I think it's somewhat like Apache Druid.

<comment><USER22> Honeycomb.io is a good place to look. It has a strong Scuba-like smell (unsurprising since founders are from Facebook (by way of Parse).

Basically: structured log events with dozens or hundreds of columns, no write time aggregation, and arbitrary folding and slicing of data for investigation and debugging.

</comment></comment></comment><comment><USER23> One problem that I've seen in one such large company, and heard of from some people in others (FAANGs), is that there's no incentive to maintain these internal tools. Sure, they often start out as sort of amazing and there's a ship email and for 1-3 years a strong team, but you are unlikely to be promoted a lot polishing UI, fixing bugs and adding features or even support for new backend systems... so, eventually there's a skeleton crew maintaining an old tool, everybody complains how it's slow and sucks and behind the times, and then a brand new replacement is built that is going to fix everything. There's a ship email, people gradually switch to it, rinse, repeat.

</comment><comment><USER24> ime, for many companies, standard, off-the-shelf tooling that's a mix of OSS and paid services works great. And one benefit is that when you leave, you can bring whatever tools you loved into the next company.

The reason Facebook and Google built their own is that they're at such a huge scale both in terms of usage and number of engineers, that none of the off- the-shelf things work for them. For most companies that's not the case.

</comment><comment><USER25> Nothing wrong with writing glue, if that's what the business / company / market needs.

Having said that, I do miss the data that's available at FAANG companies (at least ones I've been at). I could dig up all sorts of system and business data, and really get good insight into both what my application and the users are doing. Tends to lead to a lot better and more clear decisions and thus helps to build the right thing.

</comment><comment><USER26> I don't agree without reservations but like how against the grain this currently is. Outsourcing everything does seem like a sign of immediate hyperoptimization and missing building a more lasting engineering value. Do as little as possible to extract _some_ margin. There is business wisdom in this but indeed such companies would seem more brittle. Sometimes it's like there's an ecosystem of plumbing companies selling to other plumbing companies without much visible external cash intake.

That being said, I think most of the things she mentions are not customer facing but still pretty differentiating to the businesses.

Another perspective is that the first crop of truly successful Internet companies were in the position to solve the big backend problems with running big Internet companies in-house. Nowadays getting to the scale while building some fundamental in-house stuff makes less economic sense (you'd lose to someone who doesn't). It's probably better to have some unique technology but more closely related to the specific business.

</comment><comment><USER27> If the tool is valuable to you, make sure you know exactly how it works. Then you can recreate it, for yourself at least, whenever you go.

<comment><USER28> Also, if you _wrote_ it (as often happens in my case), put some effort into getting it open-sourced if possible, so that you can just use it elsewhere later.

These days, I very rarely enjoy using tools that aren't open source, or at least generic and very widely available. I simply don't want to spend my limited brain cells learning a piece of software that I won't be able to use in the future.

And in any case, large, internally developed tools are usually awful. Unless I wrote them. ;-)

</comment></comment><comment><USER29> For 99% of engineering organizations, internal engineering tooling is not your source of competitive advantage. Your advantage in the market comes from understanding the needs of your target user better than anyone else, and building a product that meets those needs.

For every piece of infrastructure or developer tooling you need, you have to make a buy vs. build decision based on cost/benefit analysis. Your cost to "buy" needs to include the upfront cost of developer time to configure the software for your needs, and the ongoing support cost to make sure it continues to work. When you include this, even open source software isn't "free", although it may be cheap. Your cost to "build" includes the upfront engineering time to design & code, ongoing support time to keep it working, _and_ opportunity costs incurred because until the system is actually ready, no one can use it, and they have to either meet their needs in some other way or simply block until the tool is complete.

Very large engineering organizations like Facebook, Netflix, etc. can amortize the cost of tool and infrastructure development over thousands of developers and an enormous infrastructure budget. If a tool would make most of Facebook's developer's 3% more efficient, than it's worth staffing a team to build that tool. If better caching infrastructure could reduce Netflix's network usage by 3%, then it's worth staffing a team to build that infrastructure.

If you're a small- or medium-sized business, then that math doesn't pencil out. A 3% (or even 10%) developer efficiency improvement doesn't even pay for the opportunity cost of one developer. A 10% decrease in a single dimension of your AWS bill probably doesn't pay for the developer time either.

If you're at a mega-corp with a huge engineering staff: Yes, absolutely invest in tools that make your engineers more productive, especially if you've identified truly unique needs that can't be met by off-the-shelf software.

If you're at a smaller business: Focus your developer energy on your user- facing problems, and almost always prefer to "buy" instead of "build" (where "buy" includes using open source software).

<comment><USER30> I do not think it's that simple. I understand the arguments for buying vs build. However, if you're a technical heavy organization, relying on buying things also has its costs. You're constrained on your product offerings because of the thing you bought. The decision is not that simple and requires a lot of work and foresight.

As such, my guiding principle has been to consider the layer at which the thing you're trying to build/buy fits. For most companies it doesn't make much sense to build data centers and any cloud vendor will do just fine. Infrastructure is therefore somewhat easier... almost always buy. But if your product requires some kind of integration that is critical for customers, I would think hard about it before buying.

The costs of integrating a bought tool are also overlooked in most of these conversations. External tools are marketed well but don't always deliver on their promises of being easy to use or setup.

Fundamentally I genuinely believe that if your product is tech heavy you absolutely need a CTO or a founding engineer who has a solid experience in building things. There is no replacement or magic pill to replace that just yet. And still, many fall into this trap.

<comment><USER31> > The costs of integrating a bought tool are also overlooked in most of these > conversations. External tools are marketed well but don't always deliver on > their promises of being easy to use or setup.

Exactly, for a lot of problem domains, the cost of integrating a third party solution is nearly equal to the cost of writing from scratch, because the hard part isn't the problem space but the environment-specific configuration.

<comment><USER32> It's also difficult to grow and maintain a high performing engineering team centered around integrating 3rd party products. You can't spend 2 years on vendor negotiations and integrations and then expect to build the next great thing for your business.

</comment></comment></comment></comment><comment><USER33> _If you can 't come up with anything, it's possible you're just beyond repair and too snarky to think reasonably_

Or perhaps that we've mostly worked in crap dev jobs/companies? There are easily enough of them around to encompass a career.

<comment><USER34> She explains that in the second half of the paragraph you quoted.

<comment><USER33> You're right. The internet has done nothing for my reading comprehension.

</comment></comment></comment><comment><USER35> This article makes little sense. "Engineering" by definition is some sort of "glue" work that reuses existing libraries/wheels and builds solutions out of them to address specific problems. I as an engineer am proud of this glue work that I do, just like the carpenters, plumbers, cathedral builders, doctors etc. If you really want to be _unique_ and invent something, you should be in the academia and are probably in the wrong place to begin with.

I have a friend who worked in the TensorFlow team and still quit to pursue a PhD. In his words "all engineering jobs are boring" from his perspective. I don't think being in a BigCorp or not makes much difference at all.

</comment><comment><USER36> I've recently branched out on my own. Never had great tooling, was in government or a F50 where chains of approvals for the slightest changes were cursed by all, including those doing the approving. But, there are some organisational things I miss:

Having someone/a team to ensure network uptime is fantastic and that have the scale and experience to anticipate and mitigate potential trouble before it comes up. Likewise all infra in general. Having a legal department to ask "Can I do this?" where often the answer may be "We don't know" but that's still an answer. A security force, property, accounting, providing peace of mind. Someone making sure the projector's setup, tiny things that add up quickly.

There are downsides to this, such 'organisation' often grinds things to a glacial pace, hence why I'm breaking out by myself now, but... not tooling, I can take care of that myself and I'm incredibly thankful for the SaaS ecosystem that would have made this impossible even a decade ago, or at least very inconvenient with 3rd party consultants providing such services for high fees and little more. There are a lot of things an organisation provides that we can take for granted, and having that experience provides a great bonus in expectations of robustness not only in technology but in all processes of a business. Tooling is, in the end, in the mind.

</comment><comment><USER37> I'm always so disappointed that we're all so closed when it comes to tooling. It's fortunate that some tools get open-sourced, but it's also astounding the amount of time and money that is wasted building the same tools over and over at each company.

Sure, sometimes paying for a hosted solution for your CI, deployments, container platform, data pipeline, etc. isn't the most ideal route, depending on your business, but I'd much rather focus on building actual product over reinventing the deployment system wheel for the Nth time.

> _People from Google probably miss Borg (and lament the whole Kubernetes > thing) or BigTable._

I've never worked at Google, and thus have no experience with Borg, but Kubernetes is _exactly_ what I want to see companies open-sourcing, or, if they must, selling. I don't see what there is to lament. Being able to start a new company and not have to reinvent and rebuild your own deployment/container platform is a _huge_ gift.

For the most part I find this blog to be insightful and fun to read, but this post IMO completely misses the mark. Given that I think the author works a lot on the operations side of the house, I guess I'm not surprised. Internal platform tools are there to serve the needs of the product teams; they don't exist just to make someone feel good about their ability to build something instead of buying instead, which she unfairly looks down on so disdainfully, as if it's not _real_ engineering. Part of doing engineering is putting your ego aside when it's more cost-effective and robust to use someone else's solution instead of rolling your own.

<comment><USER38> I feel like the author's point and your point would be applicable a lot more say 5 to 8 years ago. There has been so many SaaS startups in the past 8 years or so where many things that used to require intense internal tooling are now available as a SaaS.

</comment></comment><comment><USER39> The only thing I'd miss from a larger company is the travel budget. The 'devil may care' attitude to drop a couple grand on a short window flight... compared to travel with the small company where that meeting could break the company. Double goes for the in-house travel agent who sorted everything while in route - Jane... you were missed.

</comment><comment><USER40> I just joined apple and I miss all the Linux tools

<comment><USER41> Yeah but you have Radar. Didn't realize how much I liked it until after I left

<comment><USER42> I suspect only people at Apple like Radar ;)

</comment></comment></comment><comment><USER43> I honestly miss the internal tooling at AWS. The build system and CI/CD pipelines used internally are truely amazing.

<comment><USER44> Brazil? isn't it mostly there in work-alike saas?

</comment></comment><comment><USER45> I've been at places where they should of bought vs build, because they don't put enough staff on it

</comment><comment><USER46> I prefer Kubernetes and Prometheus to Borg and Borgmon. Better interoperability, portability, and documentation. Does any other Xoogler have thoughts on this? The author didn't mention it, but I also prefer Capn to Proto3.

<comment><USER22> Kubernetes has a lot going for it, but it isn't (yet) anywhere close to Borg in terms of scalability. 5000 maximum hosts in a cluster is a drop in the bucket for a Borg cluster. Google was publicly talking about 10,000 machines as the median size almost 5 years ago.

As for Prometheus, I like the rules engine VERY much (unsurprising, since they basically reimplemented borgmon, there). I'd take Viceroy any day over Grafana, though.

<comment><USER47> I have deep experience with operating Kubernetes and no experience with Google's Borg but I imagine Borg is significantly more challenging to operate? Kubernetes is designed to be a workload scheduler that a three or four person team can operate.

<comment><USER22> I imagine if you're Borg SRE, you'd say "yes, it's significantly more challenging to operate", but for the tens of thousands of other engineers? Not really, although there was a LOT of copypasta in terms of building new configs for services.

</comment></comment></comment></comment><comment><USER48> Every time I make a neat tool I open source it using the unlicense so I can use it elsewhere with no attribution. I'm happy to work at a company where this is possible. This doesn't apply to business critical code a competitor might want, just libraries and tools for making generic 'stuff'. Typically, I make the tools at home first and decide I want to use them at work, but I eventually end up patching them at work for company purposes.

</comment><comment><USER49> I solved this problem 10 years ago by building my own open-source stack and only working with that stack: [https://github.com/tinspin](https://github.com/tinspin)

You should build your own tools, that you bring with you, that is what all artisans have done for all time.

No reason to change that just because you can use the newest buss-word every week.

</comment><comment><USER50> The only thing I miss are teammates and some of the office moments we had together, but those luckily survive employers and we get to meet in other contexts.

I never been attached to work tools, or technology fads, things are ephemeral, best practices of today are the legacy codebase that no one wants to work on tomorrow.

</comment><comment><USER51> This article is dangerous in romanticizing the "not invented here" culture at many big tech companies and seems rooted more in the 90s than present day.

The world of open source tooling and easily re-usable SAAS offerings means everyone has access to the best tools, whether you're a small startup or a big company.

Anyone who longs for internal, corporate tooling baffles me when they can use things that actually have polish, user experience and likely better implementations under the hood.

Companies should spend their time/energy building things unique to their problem domain, not weak also-ran corporate tooling.

<comment><USER52> The internal build tools I had access to at Amazon are unrivaled in the public domain. I DO miss them, at every job.

<comment><USER53> I'm asking this sincerely: What is/was so great about them?

<comment><USER52> It was probably some combination of how well-integrated and how opinionated they were.

The build tool was multi-language. It allowed depending on arbitrary packages which had been imported to the Amazon package repository. It allowed package owners to annotate the packages with guidance - experimental, deprecated, forbidden (in the case of security issues), etc. You could also declare conflicts, which would notify consumers at compiletime and force them to resolve in some way. The tool deferred to a number of standard build tools in whatever language(s) you were using; it was just about getting and packaging the dependencies.

When you committed code, a build was submitted to a distributed build system. It would run your build, and then it would run the builds of every package which declared a dependency on your version. If those builds failed, your build failed (so, bump your version or make your change backwards compatible). On completion, it imported an immutable bundle of your artifact + dependencies to the deployment infrastructure.

This is the part I miss - the build stuff was great but I largely find that open source and paid options aren't so bad here. What I miss is how easy it was to manage the journey of a built package to your machines. They had a tool called Pipelines that had a visualization of this progression. Each stage, with associated machines, was linked here. The tool knew how to add stages of environments, each with their own set of deployment configurations. You could set up approval workflows: integration tests, manual approvals, etc. You could feed one pipeline into another. For each artifact, you could configure autobuilds into your pipeline, so that new versions flowed as long as tests/approvals allowed. There was support for sanity checks: if those failed, the tool would automatically rollback. In fact, if any stage fails, your pipeline would block, and you'd be notified. In some cases, a newer build that was fully functional could unblock your pipeline.

Pipelines was a pleasure to use - it really just got the hell out of my way, and nothing I've used since is as simple to integrate with.

And tying this all together, there was a tool that would allow you to initialize the end to end infrastructural pieces needed. You go to a wizard, tell it your language and purpose (webapp, CLI tool, service) and it initializes a repository, package, and pipeline with environments. Get a coffee, come back, and Do Your Job.

I imagine similar things exist at places like Google, but man everywhere else I've worked, so much developer productivity has been lost not even approximating the level of "let me do the interesting stuff" that Amazon provided.

<comment><USER53> Awesome, thanks for the informative reply!

</comment></comment></comment></comment></comment><comment><USER54> I'm very disappointed if I move from a company that uses macs, slack, Confluence to PCs, MS Teams (horrible) and SharePoint (worse).

<comment><USER55> Why is MS Teams horrible? We have an engineer who is trying to convince our team to move from Slack to MS Teams. My only gripe is the requirement for a M$ login account (we are a Google Apps house).

</comment></comment><comment><USER42> If you're not building complicated new systems and just gluing things together, you're not doing engineering? 1) definitely gate keeping 2) good engineering is about what not to build as much as it is what to build.

<comment><USER56> Yes, exactly.

The view that such work just constitutes writing "glue" is super reductive. So much time is also spent empathizing with others: talking to stakeholders, understanding existing processes, determining requirements, and simplifying. So if we can reduce the solution to glue instead of needing to write a complicated system, that's hands down a win.

<comment><USER42> Any engineer who comes up with a solution that requires less new code being written is a more valuable engineer IMO. Less code is less time spent on building then maintaining code for all eternity.

If you go through the build/buy/partner decision, and decide to buy or partner with development comprising of glue code - then that's probably the optimal solution for your org. And you made the right engineering choice.

<comment><USER57> > Any engineer who comes up with a solution that requires less new code being > written is a more valuable engineer IMO

Of course less code is good, all things being equal.

But taking that idea literally, basically no tooling code gets ever written and people develop some sort of learned helplessness around dysfunctional workflows that are just about feasible but fantastically wasteful of engineering time. This tends to happen a lot, also because tooling is often perceived by management as an unnecessary luxury.

<comment><USER42> Agree, In my own experience it's only orgs with a min 50 to 100 developers who can afford to have a single developer (it was me!) out working on productivity tools.

</comment></comment></comment></comment></comment><comment><USER58> This topic is one of the primary reasons I enjoy working at MS these days.

The tech stack is full of OSS or industry standard tools, not a bunch of internal NIH things.

My current stack is: React, Node, Kubernetes, Linux (for dev and deploy), Git for sc, GitHub, vs code.

I can't even think of some internal tool that I depend on at the moment.

<comment><USER59> Github is open source?

<comment><USER7> Like it or, GitHub is ubiquitous enough that noone really cares. If you do care, there's plenty of OSS alternatives that aren't much different in their UI than GitHub itself.

You can easily put GitHub on your resume, and everyone would know what you're talking about. Of course, by itself, it's not much of an experience, but, surprisingly, it is often listed as a requirement and/or nice-to-have for certain SWE job posts.</comment></comment></comment>


Create a CRUD Restful Service API Using Flask and MySQL [in 7 Minutes ] - <AUTHOR> https://www.nintyzeros.com/2019/11/flask-mysql-crud-restful-api.html <comment><USER2> There is bug of missing id in both put and delete method .I could sense that are due to parsing?? Since I could find it <> anywhere?? Please fix it</comment>


Likelihood of discontinuous progress around the development of AGI - <AUTHOR> https://aiimpacts.org/likelihood-of-discontinuous-progress-around-the-development-of-agi/ <comment><USER3> About AlphaZero particularly, a few things must be kept in mind.

First, AlphaZero still makes use of a Monte Carlo Tree Search algorithm to search for good moves. MCTS is a powerful algorithm with a very limited scope: zero-sum, perfect information games. So for instance, it would be very difficult to see how to use MCTS-based AlphaZero in, e.g., training self- driving cars.

Second, the AlphaZero architecture is precisely mapped onto a checkerboard and will not learn anything about games that don't use a checkerboard, or any situation that is not possible to model as a game played on a checkerboard.

Third, the AlphaZero architecture is also precisely mapped onto the range of moves of pieces in chess, shoggi and go. Again, AlphaZero would be useless in any game that used pieces with different moves (e.g. a piece with a zig-zag move, or a piece allowed to move in spirals etc).

All of the above of course can be mitigated with different architectural choices, but to make those choices, implement them and validate them will take a great deal of time.

So, AlphaZero doesn't mean we're closer to _general_ AI. Quite the contrary: it's a very specialised form of AI that will be very difficult to use in any different task than chess, shoggi or go.

<comment><USER4> _So, AlphaZero doesn 't mean we're closer to _general_ AI. Quite the contrary: it's a very specialised form of AI that will be very difficult to use in any different task than chess, shoggi or go._

This is a very true statement and one that I think a lot of people who aren't in ML/DL, but are "worried" about AGI, miss.

There is however a common thread with everyone in AI, that they tend to think of AGI as "One algorithm to rule them all."

As a practitioner and AGI researcher however I think that AGI is more of a system of specialized or narrow AI tasks that can together solve all tasks. At the risk of oversimplifying and anthropomorphizing, this type of problem solving is functionally how we do it as humans.

So having a corpus of solved narrow systems (discrete known rule space in the sense of AlphaGo etc...) that is "activated" by an executive function which can recognize the problem set and then pass subsets of a larger problem to the narrow solutions. Those solutions are then "backpropagated" and synthesized into the general problem solution.

In that sense, I would argue that narrow solutions like AlphaGo etc... do get us closer to General AI because they grow the corpus of solution paths for the general problems.

<comment><USER5> I disagree. The naming convention [Artificial Intelligence] still is a large shoe that these purpose built applied engineering solutions have yet to fill. Meanwhile, for profit/notoriety/marketing people want to trample on yet another name space? What you just described is essentially the architecture of a self-driving car. It's yet another applied engineering solution of Artificial Intelligence. It is not Artificial General Intelligence. Scaling/Distributing the computational space of an applied Artificial Intelligence solution is not Artificial General Intelligence. This is the same thing that lead to optimization algorithms being called Artificial Intelligence. If you aren't able to maintain foundational distinguishment, you lose track of what you're searching for and trying to achieve. Outwardly, you capture more money and attention. Inwardly, you become unraveled and lose your capability to solve the elusive problem. Eventually after much fame, wealth, and feigned 'success', one asks themselves : Was it worth it? Depends on what your original aim was.

<comment><USER4> I'm not sure what you're arguing but it seems like my key point wasn't communicated well.

 _What you just described is essentially the architecture of a self-driving car._

Yes, every narrow AI is a system of systems to an extent. So expand on that concept but outside of a single firm/system. Such that the self driving car system is one single solution path solving "transportation" which would comprise automated flight/rail etc... and is a node in a larger general system - like hub and spoke.

 _The naming convention [Artificial Intelligence] still is a large shoe that these purpose built applied engineering solutions have yet to fill._

Nobody is questioning that. The size of the narrow AI market is arguably infinite.

You seem to be arguing that a single entity will fail if it attempts to take a narrow AI system and make it generalizable. Of which I am in agreement with.

If however there were 10,000 or 100,000 or 1,000,000 narrow AI companies/systems (like a self driving car system or alphago etc...) those could fill the corpus of solutions which an executive function system could utilize depending on the application and together they would be what we call AGI.

<comment><USER5> > I'm not sure what you're arguing but it seems like my key point wasn't > communicated well.

It was and quite well. Were speaking the same language. We just have different conclusions.

> Yes, every narrow AI is a system of systems to an extent. So expand on that > concept but outside of a single firm/system. Such that the self driving car > system is one single solution path solving "transportation" which would > comprise automated flight/rail etc... and is a node in a larger general > system - like hub and spoke.

And you still have nothing more than a hub and spoke system of systems authored for specific problems spaces and you're spokes will increase with every new problem space until you overwhelm your hub. A horrible architectural approach that if not caught in the initial stages will result in catastrophe down the road... Weak AI is weak AI no matter how you scale it.

> You seem to be arguing that a single entity will fail if it attempts to take > a narrow AI system and make it generalizable. Of which I am in agreement > with.

This is a start in the right direction...

> If however there were 10,000 or 100,000 or 1,000,000 narrow AI > companies/systems (like a self driving car system or alphago etc...) those > could fill the corpus of solutions which an executive function system could > utilize depending on the application and together they would be what we call > AGI.

No, its strung together weak AI. It will require significantly and unreasonable amounts of resources. Its capability will increasingly reach diminishing returns and you'll end up with a frankenstein monster code base that no one can manage or understand.. Sounds a lot like the path Weak AI is already heading down.. At such a point, it's best to just scrap it and start all over. Something that Hinton and other prominent figures are finally admitting. Something I concluded year ago which lead me down a different path. Now, you're more than welcome to state : Well hey man that's your opinion and you're wrong and I'll wish the 10s,100s, million of narrow AI companies the best just as was conveyed to me a umber of years ago. Weak AI is Weak AI. It is a class of optimization algorithms. You can jerry rig this all you want.. You still have nothing more than a system of systems of optimization algos. If you think this is what intelligence is, I'm not sure what to say.

<comment><USER4> _You still have nothing more than a system of systems of optimization algos. If you think this is what intelligence is, I 'm not sure what to say._

Until someone comes up with a better definition of intelligence that's what I'm sticking with. I think you're looking for an elegant solution right out of the box - the "one algorithm to rule them all" and I don't think that is feasible from an engineering perspective if for no other reason than no singular system has anything near the data collection nodes needed for specificity on the range of tasks that would suffice any definition of "General."

Having raised three other humans and observing them while building DL systems myself for a living, I feel more strongly everyday that human intelligence is a hodgepodge of "weak AI" systems glued together with an exceptionally efficient executive function. AGI is as much a community building and humanity wide input collection challenge as it is a math problem. We need to think about it that way.

<comment><USER5> > Until someone comes up with a better definition of intelligence that's what > I'm sticking with.

You'll get a capability demo instead. It wont fail to impress. Definitions and designs are for another day.

> I think you're looking for an elegant solution right out of the box - <USER2> > "one algorithm to rule them all" and I don't think that is feasible from an > engineering perspective if for no other reason than no singular system has > anything near the data collection nodes needed for specificity on the range > of tasks that would suffice any definition of "General."

What else is one looking for who claims they're trying to solve the Intelligence problem? Marketing an optimization algorithm as the next coming might make you rich in the short term but it doesn't bring you closer to the truth. It does in fact take your further away. So, 'the elegant solution'/'the hard problem' was the only thing I set out to tackle some years ago. Otherwise, i'd have been wasting my time/not being truthful with myself. It's feasible from a research and engineering perspective. Few commit themselves to the TRUE task and the likelihood of failure. I was ok with that it and stuck with it. I self-funded my work. It mainly centered on research. Thus, there were no exits. I either saw it through and achieve it or I didn't.

As far as :

> no singular system has anything near the data collection nodes needed for > specificity on the range of tasks that would suffice any definition of > "General."

Sure it does. Look in the mirror and log onto the web. I've let the misses play online for a bit now ;).

> Having raised three other humans and observing them while building DL > systems myself for a living, I feel more strongly everyday that human > intelligence is a hodgepodge of "weak AI" systems glued together with an > exceptionally efficient executive function. AGI is as much a community > building and humanity wide input collection challenge as it is a math > problem. We need to think about it that way.

My graduate work centered on the underpinnings of DL (Distributed Optimization). After years of industry experience, I searched for a new challenge. After some open ended research in physics/photonics, I came to Artificial Intelligence. I scratched my head for 3-4 months as to why (Distributed Optimization) was being called Artificial Intelligence. I took the broad lot of it and threw it in the trash as prominent figures are only now stating : [https://www.axios.com/artificial-intelligence-pioneer- says-w...](https://www.axios.com/artificial-intelligence-pioneer-says-we-need- to-start-over-1513305524-f619efbd-9db0-4947-a9b2-7a4c310a28fe.html)

You're thinking about AGI as if its a chain of DL systems because that's what's made you money and where your work has centered on over the years. I took the broad majority and trashed it as Hinton now indicated others should do and started from scratch. I have no such bias. However, as my graduate work centered on the fundamental underpinnings of statistical optimization / distributed optimization, I know exactly what its limits are.

The human race is far more than a hodgepodge of optimization algos w/ an executive function (whatever that might be given the clearly varied forms of it).

<comment><USER6> Pardon my question: what kind of education did you have?

I deeply regret not studying Computer Science but yours seems to be deeper than that. I'd be very interested in the courses you took.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> We're all already part of a society-scale, distributed hive mind--have been since the invention of language. Birds flock together, they eat together, they _think_ together. Families, friend networks, cities, global societies, they all form communication topologies that have analogs in the brain. Thoughts bounce from person to person, memes spread amongst the computational fabric of groups of people. It's nothing new.

We as a society have formed networks and systems to solve problems like finding energy, producing food, and organizing economic output. We live in a distributed intelligent system that disseminates knowledge and programs our preferences, responses. We're all utilized as work units, and economics does that. I think it's a mistake to think of AGI as something separate from us, something that doesn't already exist. We're more like a cybernetic superorganism. We've put so much computational power in charge of the choices that we make, and we rely so often on recommendations from computational systems, that if you zoom out far enough, it becomes clear that we are part of a huge, cybernetic Overmind.

The Overmind is just moving more computation away from humans because hey, they're slow. It doesn't really speak to us. Do you speak to _your_ neurons? Nevertheless, it has its goals, its resources, its needs, its preferences. People carry out its wishes, statistically. It turns out that its wishes align very well with economics: More computers! More network! More screens! Connect all the stuff! All the companies doing this are making huge dollars. The mind or minds are just _centralizing_ now, and economics drives that. Computers already fully run the stock market. They run shipping and logistics. They are used to optimize all kinds of economic outputs. And they are used (by humans) to design better computers.

At the broadest scale, we are already that self-improving intelligent system, it just doesn't look like it from meatspace just yet.

It's kind of irrelevant if it could utter the words "I think for I am." Who would it tell that anyway?

<comment><USER8> The "hivemind" argument seems to predict that as society scales up (either through massive population growth or through faster and better interconnectedness, such as through the internet) that as a result we should be seeing much faster gains in technological progress especially in the last few decades or so. However there are quite a few observations that a lot of this progress has sort of slowed down compared to the early 20th century (see the arguments for "technological stagnation"). At the very least, technological progress hasn't increased linearly with population growth and better communication. In other words, the rate of technological progress looks more discontinuous and not obviously a function of societal coherence.

<comment><USER4> I don't think your interpretation of that prediction is accurate. Rather it would be that there are "bursts" of technological progress followed by slower or no gains while the world "catches up." That seems to more accurately follow the history of technological progress.

I think a better interpretation would be that those "bursts" happen at tighter intervals, and if you look at the course of history that seems to be the case.

For example, the period between the introduction of horses/plows widely adopted in agriculture in the 1700s and the wide adoption of internal combustion in the 1940s was ~300 years. From Internal Combustion to wide adoption of transistors (1970s) was about 30 years from transistors, transistors to internet about 20 years, internet to ? (Deep Learning 2012) looks like about 15 years.

Not sure if that's a perfect fit but I think it represents a pretty compelling case.

<comment><USER8> I think that bursts of technological progress follows more from the model of individualized intelligence, whereas continuous progress follows from the distributed, networked model of intelligence.

A promoter of the distributed model of intelligence might argue that Einstein was only able to produce the general theory of relativity because of the knowledge already contained within society, such as the mathematics and physics that had already been built up to that time. All the stuff from Euclid to Newton to Gauss to Poincare and Minkowski that Einstein's work relied upon.

Does that imply that Einstein wasn't really smart? If you narrow your focus to just the innovation Einstein made, where did that come from? Did it come from the "hivemind" or was Einstein himself doing something special that allowed him to develop the insight?

More individualized intelligence would predict that we would see smaller intervals between bursts as society increases in size and connectedness (more chances for Einsteins to appear, more likelihood that they can work together). But if intelligence is somehow an emergent process from the network of all humans itself, then as society grows we shouldn't see many bursts at all, just a fairly continuous increase in knowledge as little bits and pieces get absorbed and distributed.

<comment><USER9> I think you're making too many assumptions about the inner workings of "the brain".

If we look at actual brains - including Einstein's - are they not bursty? Don't people have periods of greater intellectual output with lulls in between? Seems to match pretty well.

</comment></comment></comment></comment></comment><comment><USER10> We know that for most problems, a group of people tends to be better at problem solving than an individual [1]. Even if AI technology only reaches human-level and does not exceed it, continually increasing efficiency would make AI smarter than any small groups of humans and immense bandwidth relative to human communications would make it more effective than any large human organizations.

[1] [https://aiimpacts.org/coordinated-human-action-example- super...](https://aiimpacts.org/coordinated-human-action-example-superhuman- intelligence/)

In addition, if AI posesses sufficient computing resources, which will certainly become available in the next few decades if not already, it will have inherent strong advantages like serial computation speed and memory size that exceed any human brains. So the real barrier for AGI is software and not hardware.

If AGI software is developed before we have sufficient hardware to run it at the human brain speed, then it will become more capable at the rate of hardware we can put into use, which is likely exponential given how parallelized the human brain appears to be.

The major counterargument I find most convincing regarding outsized impact of exploding intelligence is that many problems are exponentially hard (or harder than that) and thus exponential intelligence can only make linear or sublinear progress on them.

However, linear or even sublinear progress may still lead to quite drastic changes in the world. If an organization can marginally predict stock price movements better than the rest of the world, in a few decades it will accumulate great resources and power. The same is true for many other important domains.

<comment><USER11> This also assumes intelligence is the limiting factor for solving many problems. I suspect information and computation is probably the larger factor for most major issues. The smartest player possible would still lose poker to someone that can read their hand.

<comment><USER10> I agree that information is crucial to achieve a 'win' for many goals. Given today's amount of information on the Internet as well as electronic money and access to most officials, barring some sort of inviolable built-in moral core, an AGI would be able to use any methods, overt and covert, direct and cunning, technical and social, to achieve its information goals. [1]

Since an AGI can copy itself and be available at a multitude of access points at once and those copies can often communicate via extremely fast channels, it is human organizations that would be at an information disadvantage.

[1] This also assumes that the AGI does not have the will nor the capability to change its own moral core. I think an AGI will possibly be _capable_ of changing its own core, so a much more reliable safeguard is to make sure that it does not _want_ to change it.

<comment><USER5> [AGI Developer]

A controller/overseer can easily limit/block this sufficiently and securely. Were talking about hardware/software. There are systems/standardized approaches to solving this problem. The 'Control/Safety' problem for AI are lauded as theoretical and new. However, they are not. They are solved by industry standard approaches day in and out. Any seasoned/experienced engineer in this field could solve this with known approaches.

> Since an AGI can copy itself and be available at a multitude of access > points at once

Same comment above applies. This can only occur if done by a controller/overseer. Real-life isn't a sci-fi movie... There's engineering involved.

> AGI changing x,y,z

Not possible unless it is given access. Solved easily in industry standard ways.

<comment><USER10> Has the industry always been able to prevent smart, persistent actors from breaking access locks?

Why should we assume that an AGI which can accumulate experience over time, gain more knowledge, and make connections with others, including human actors, will not ever be able to break the locks?

<comment><USER5> > Has the industry always been able to prevent smart, persistent actors from > breaking the access locks? Why should we assume that an AGI which can > accumulate experience over time, gain more knowledge, and make connections > with others, including human actors, will not ever be able to break the > locks?

Yes, the industry has persistently been able to do this. It's why the whole world isn't falling apart as we speak. What limits the locks most often is cost not capability. As such, you are possibly mistaking one's business decision not to use a more costly lock for the lack of capability to create a capable lock. Furthermore, you mistakingly attributing the actor in this case. The actor in the case of AGI is in a carefully controlled/monitored box. Actors in the real-world are not. As such, please tell me how an absolutely monitored/restricted actor has the ability to go playing with locks that aren't within its reach? I have a more fundamental question even : Have you been able pick 'your locks' yet? Do you even know what they are? Where they are? Those capable of 'creation' hold certain things close to their chest .. The act of creation necessitates it and is [built in].

> Why should we assume that an AGI which can accumulate experience over time, > gain more knowledge, and make connections with others, including human > actors, will not ever be able to break the locks?

Show me how you're able to break your 'locks' and you'll have an argument for how AGI can break its locks. I don't think you're grasping the level of 'locks' that I'm speaking about. Humans have been around for how long and still don't even know what their [locks] even are... or where they are. It's quite easy to example at a certain level of visibility how your scenario is unwarranted. I can draw direct parallels to eons of human history.

<comment><USER10> Humanity as a whole is starting to be able to break our 'locks' with gene editing. It took a long time partly because biology is very complex and fragile. Its complexity is shaped over eons and we still do not really understand it that well, but we finally found some 'hacks'.

There is no reason to presume that a software system built by a team of humans will be nearly as complex, unless the AGI itself is not too bright or cannot self-improve to be smart enough to understand itself, or sufficiently clever to find a way to social engineer toward eventually getting access to its source code or to reverse engineer itself to an extent that even humans can.

<comment><USER5> > Gene editing

Those aren't the locks I'm talking about and you should take note that its possible because you have environmental access to them.

> is very complex and fragile

Indeed. Terminal error could result in a particular case. Game over man !

> There is no reason to presume that a software system built by a team of > humans will be nearly as complex, unless the AGI itself is not too bright or > cannot self-improve to be smart enough to understand itself, or sufficiently > clever to find a way to social engineer toward eventually getting access to > its source code or to reverse engineer itself to an extent that even humans > can.

You guys really don't want to let go of this sci-fi fantasy do you? LOL. How long did it take human beings to discover how to edit their genetic code? You were babbling in caves some time ago. You think a 10 year old knows how to modify themselves without self destructing in the initial trials?

> self-improve to be smart enough to understand itself

Many people don't have an even basic understanding of themselves much less how to even psychologically re-order their own behavior. In the scenario that someone becomes sufficiently capable of engineering an equivalent... What level of understanding do you think such an individual would have to be able to engineer AGI? What intelligence level would you attribute to that person? And you think they wont understand potential ways this can occur and prevent it? Also, you again talk about access... It's a running binary. A compiler is needed. There's a power plug. Its operations are monitored as is its output. It's literally a box with a tremendous number of locks it doesn't have the capability to pick... Just like (you) .. even as you go hacking about your genetic code ^_-

</comment></comment></comment></comment></comment></comment></comment></comment><comment> dane-pgp It's an interesting read, but perhaps it only succeeds in pointing out the gaps in our knowledge. When I read their counterargument to the possibility of intelligence explosion:

"Positive feedback loops are common in the world, and very rarely move fast enough and far enough to become a dominant dynamic in the world."

the idea that immediately comes to mind is the Harmless Supernova Fallacy, described on this (obnoxiously JavaScript-dependent) site:

[https://arbital.com/p/harmless_supernova/](https://arbital.com/p/harmless_supernova/)

Knowledge of this fallacy is a mental tool I have found quite useful, as it seems to be a type of fallacy that is easy to make by accident. To be fair, the reasoning in the article may not quite reach the level of a fallacy, but the intelligence explosion section ends saying effectively this:

"we think the intelligence explosion argument could be strong if strong reason is found to expect an unusually fast and persistent feedback loop [i.e. an intelligence explosion]"

which sounds like a classic case of Begging the Question:

[https://en.wikipedia.org/wiki/Begging_the_question](https://en.wikipedia.org/wiki/Begging_the_question)

<comment><USER12> I think the reasoning in the arguments is left a little loose on purpose simply because it is so difficult make strong arguments about something we know so little about.

<comment><USER5> It is difficult to make arguments in something that has no grounding. I'd expect someone who claims to have a valid argument for an [intelligence explosion] event to have formal education and industry experience designing computational systems such that they could clearly define how exactly it could occur. I have yet to see such an individual w/ such a viewpoint. Instead, I see a ridiculous argument being forwarded by people who are least informed/experienced so as to push fear, uncertainty, doubt either for profit/attention or because it fulfills some sci-fi oriented religious prophecy. Internally, companies require extensive and well reasoned documentation before funding an initiative. Externally, someone w/ no expertise/proof throws their hands up in the air speaking about armageddon and they are able to secure considerable attention and money.

<comment><USER13> What is your argument? Why a group of engineers who can think two times faster than average cannot perform the task of designing three times faster engineer quicker than a group of average engineers?

They will hit economical and physical limits eventually, sure. But what will stop them at the beginning?

<comment><USER14> I guess the argument is, that so far no one showed good examples of how would that lead to a total extinction. How would it start, etc.

So it's kind of a (fallacious) argument against "black box"-ing the whole intelligence explosion problem. If you can't define it, you can't analyze it sort of thing.

<comment><USER5> What I stated was that there isn't even a sound/reasoned example as to how a it would escape its bounds unless purposely authored to and even then you run into the scenario of human limits (its creator). Also, having no understanding on how AGI is structured, it is quite foolish to talk about bound leaping in the traditional sense of tech which is mainly associated with computer viruses that are purposely written for that sole purpose. There's nothing to even suggest you can purposely author AGI in such a fashion. So...

> How would it start, etc.

There has been no credible framing on how it would even start. As such, it's a null point of discussion.

> If you can't define it, you can't analyze it sort of thing.

With no definition, you can 'attempt' to analyze it and you'll likely be horribly off the mark, wasting tons of resources, and likely produce something that has no bearing on the real thing. Instead of admitting this, it's like people put full faith in these efforts being sound when the reality is the exact opposite. Why engage in this, unless your aim is profit/notoriety, when you could be working on the actual problem? Define intelligence [first]. Time to be honest with oneself. Time to stop projecting ones shortcomings on others. Time to stop using fear/uncertainty/doubt to obscure one's true intent. Time to stop pushing the disinformation cloud. Time for TRUE intelligence.

</comment></comment></comment></comment></comment></comment><comment><AUTHOR> See also: OpenAI's Paul Christiano wrote on the same topic — [https://www.lesserwrong.com/posts/AfGmsjGPXN97kNp57/argument...](https://www.lesserwrong.com/posts/AfGmsjGPXN97kNp57/arguments- about-fast-takeoff)

(cf. [https://intelligence.org/files/IEM.pdf](https://intelligence.org/files/IEM.pdf) for some of the arguments being argued against)

</comment><comment><USER15> > This argument seems weak to us currently, but further research could resolve > these questions in directions that would make it compelling:

> Are individual humans radically superior to apes on particular measures of > cognitive ability? What are those measures, and how plausible is it that > evolution was (perhaps indirectly) optimizing for them?

Yes, clearly. All the measures defined in this article are arbitrary (e.g. vehicular land speed), and so let's propose an arbitrary one for intelligence: Ability to prove mathematical theorems. Humans have proven many. Apes have proven zero. That is an extreme discontinuity. We can propose many others, of course: complex language development, building skyscrapers, landing on the moon, etc..

> How likely is improvement in individual cognitive ability to account for > humans' radical success over apes? (For instance, compared to new ability to > share innovations across the population)

How does human cognitive ability being over-rated counter the discontinuity in intelligence development? Is the argument that our success is due to some other characteristic than intelligence, and so our intelligence is not really that much greater than the apes? If so, that's just a restatement of point 1.

<comment><USER16> > let's propose an arbitrary one for intelligence: Ability to prove > mathematical theorems

The number of mathematical theorems proven by humankind has progressed in a continuous manner (as much as a quantity measured by integer numbers can) during history.

<comment><USER17> That only says something about maths, not about humans. For comparison, you could say that the number of literary masterpieces has been increased at what looks to be random intervals for the last 2 or 3 millennia, with no continuous progress in sight (i.e. the Spanish language has not had a new Cervantes for 400 years, the same applies to Skakespeare and the English language or to Dante and Italian). But this being an website mostly addressed to people who focus on technical stuff I expect a reply like "literature doesn't count, it's just words".

</comment></comment></comment><comment><USER18> AGI, for those wondering like myself what it is:

[https://en.wikipedia.org/wiki/Artificial_general_intelligenc...](https://en.wikipedia.org/wiki/Artificial_general_intelligence)

<comment><USER19> There are very few things in the world worse than an article that uses an acronym without first identifying what it is.

</comment></comment><comment><USER20> The counterargument on the last one (Human-competition threshold) seems weak (and simply wrong) to me. In related link argument for wide range of human abilities rests basically on comatose humans being unable to do anything at all and mutation-adds-random piece-to-machine metaphor.

Mentally-impaired human intelligence level is enough to replace some of the workforce, combined with some actuators and perception abilies of average human.

</comment><comment><USER21> "low base rate [of change] for all technologies" \- measured over centuries. Meanwhile all technologies (nearly) experience discontinuous advances, often near their start. See, steam engines and Watt, etc, etc, etc.

The rest of the argument seems to be grounded in an assumption that electronic neurons or sims of them won't ever be faster than meat. Really? Today's crude neural nets are already very useful here and there precisely because their speed means they scale and can repeat a task very, very frequently in a small amount of time.

</comment><comment><USER22> That feeling an article will help more confusing people than enlightening them, just reading it's title...</comment>


Cleaning Radar Images using Neural Nets and Computer Vision - <AUTHOR> http://blog.forecast.io/cleaning-radar-images-using-neural-nets-computer-vision/

<comment><USER2> This is neat, though there are far better methods for doing this type of thing. Now that the Nexrads are dual-pol, you can look at the correlation between the horizontal and the vertical polarizations to filter out the vast majority of the noise.

The noise you are filtering out is primarily insect returns. If you watch it, it will get worse at some times of the day rather than others. If inversions happen, you'll also see some beam bending that can cause you to pick up powerlines/roads, etc. You can verify it is mostly bugs by looking at the differential reflectivity. The differential reflectivity(difference of horizontal and vertical returned powers) will be somewhat random for ground clutter, and higher values for bugs.

With this approach, without knowing exactly how you do it, I'd be worried that it would have the tendency to filter out initial formations of stratiform clouds, and just leave convective formations. I can point you to some theory on how a lot of the filtering is done for research in atmos if you'd like, feel free to message me(I'm working on my Ph.D. in weather radar).

<comment><USER3> Dual pol wasn't deployed when we first started Dark Sky. Now that it's becoming available, it's definitely going to make our job a lot easier!

Not knowing much about it, though, it'll take some time for us to become comfortable with the data.

<comment><USER2> Well it's very impressive that you got that performance without using any dual-pol parameters. Abstract submission for the AMS radar conference is next week, and I'm sure they'd be interested in this. <http://www.ametsoc.org/MEET/fainst/201336radar.html>

<comment><USER3> Also, correct me if I'm wrong (and I very well could be), but the most egregious noise -- the giant Bagel Blobs -- aren't insect reflections, but rather night-time temperature inversions. Can dual-pol correlation help identify those?

<comment><USER2> It's likely a combination of both. A quick way to discern would be to look at velocity, zdr, and rhohv. For ground clutter and returns caused by the inversion, you will expect to see very low velocities(close to 0) relative to the background precipitation. Additionally Zdr will end up looking like a roughly random field. If it is insects, then the velocity will roughly match the background precipitation, but you will have a high zdr(as insects look like very very oblate bags of water). In both cases you should get a drop in rho_hv, the correlation coefficient between the channels which will help to differentiate it from actual precipitation. Also it looks like you're primarily concerned with pulling out the rainfall in several cases. For this, I'd look at the specific differential phase(K_DP) as it is a much better estimator of rainfall than reflectivity. In general, it is linearly proportional(exponential, with an exponent close to 1) to the rainfall rate. I'm not sure how good the nexrad estimators are for kdp though.

</comment></comment></comment></comment></comment><comment><USER3> Seeing this on HN, I realize I didn't go into a lot of technical detail in the blog post (we still aren't quite sure who our audience is with the Forecast blog). So if anyone has any specific questions, ask away.

<comment><USER4> I thought it was great.

Could you comment on why all the radar noise in the sample image seems to be almost perfectly confined to the eastern half of the US? Is that just an artifact of the time of day (perhaps the sun was setting the in middle of the US at the time)?

I'd love to know more about the neural network too, but I have no experience with them so I'm not even sure what to ask.

<comment><USER3> Exactly. Those enormous Bagel Blobs tend to occur after the sun goes down (and only in the warmer months. Two months ago, they weren't there).

The image was generated around 10:30 PM EST, so they hadn't had time to propagate further west.

Regarding neural nets, I have to admit that I didn't have much experience with them either before working on Dark Sky. They do, however, seem to be a little more forgiving than other kinds of classifiers (Support Vector Machines, and the like).

Check out the examples for the FANN C library: <http://leenissen.dk/fann/wp/>

Or in ruby: <https://github.com/tangledpath/ruby-fann>

<comment><USER2> You might find [http://www.chill.colostate.edu/w/Non- precipitation_echoes:_i...](http://www.chill.colostate.edu/w/Non- precipitation_echoes:_insects_and_smoke_particles) to be interesting. It shows some of the dual-pol stuff on insects.

</comment></comment></comment></comment><comment><USER5> See also Making Clouds Go Away on MapBox Satellite [http://mapbox.com/blog/improving-mapbox-satellite-by- making-...](http://mapbox.com/blog/improving-mapbox-satellite-by-making- clouds-disappear/) <https://news.ycombinator.com/item?id=5475571>

</comment><comment><USER6> Hey there. I've done a GPGPU implementation of a physical weather prediction model[1]. I'm interested in how you get your weather data. Do you use WSM for modeling? Also, have you considered licensing out Dark Sky for data assimilation purposes to weather agencies? What I hear from the weather researchers this is one of the hardest aspects of this field. I think that once you have a product that's able to integrate an automated feedback loop into existing weather models for all kinds of weather data, you've basically won the weather game.

[1]<https://github.com/muellermichel/Hybrid-Fortran>

<comment><USER7> Regarding data, have you looked at the API[1] they provide, also used to power Forecast.io and Dark Sky?

[1] <https://developer.forecast.io/>

</comment></comment><comment><USER8> Meteorologist turned HN junkie here: this is really cool and I love the commenter suggesting using dual pol data to do the heavy lifting for you.

My question about your radar data is this; it's beautiful but I absolutely hate the color palette you are using, from a weather perspective - why not just stick with the 'standard' green-to-red colors we all know and love? The purples are different, for sure - but really hard to make sense of!

<comment><USER3> The color palette of the images in the blog post was adjusted to highlight areas of precipitation at the expense of intensity discrimination. Dark Sky itself uses a wider range, but it's still a similar blue to purple to salmon to yellow color table.

The reason we went with that over the standard green-to-red color table is because I personally find the standard colors to be jarring and more confusing to the casual user because of the increased contrast between intensity bands (at least, it certainly was to me back when I was a casual weather app user!)

The tradeoff is that there isn't as fine an intensity discrimination at lower intensities. In theory I can see how that can be a problem, but I haven't noticed it in practice.

</comment></comment><comment><USER9> Interesting... as part of a little site I threw together, I have a ton of radar data for the region of Italy where I live, and have always thought it'd be fun to do something with it:

<http://www.meteo-veneto.net/></comment>


HN, We Need To Talk - <AUTHOR>

There are currently 8 stories about Apple's broken maps on the front page. It seems like people are really getting a lot of joy and satisfaction out of a bad component launch, similar to when someone goes through a bad breakup, then hopes the ex has bad fortune. It's really immature.<p>From an outsider's perspective it gets really tired. Can we stop with the broken maps submissions for a little while? Just give the rest of us a break, who aren't overly concerned with one app on one product. There is a whole world of stuff happening, vote it up. <comment><USER2> Some people hate success, or more accurately, they hate the success of others. There are people on HN who want Apple to fail. There are people who want Google to fail. There are people who want Microsoft to fail. There are people who want Amazon to fail. There are people who want Facebook to fail. And so on and so forth.

When any of the giants stumble with a mistake or show a sign of weakness, the people who want them to fail pounce with extreme agitation.

The important thing to remember is they didn't come to HN to contribute, instead, they came here to push their agenda and argue.

<comment><USER3> Also known as Tall Poppy Syndrome: <https://en.wikipedia.org/wiki/Tall_poppy_syndrome>

</comment></comment><comment><USER4> Yeah, let's just assume Apple was brave enough to launch an iterative product for once, forgetting to label it 'beta' and accidentally shutting out competition.

</comment><comment><USER5> Hint: flagging stuff tends to be more effective than posting to "Ask HN".

</comment><comment><USER6> I get tired of the redundancy. However, as specifically regards the maps story.

\- Apple heavily markets its "refinement" and "excellence". They just shipped a product that blows.

\- It's been about a year since Jobs passed. There was and continues to be much speculation whether Jobs was essential to Apple's performance (technical as well as financial). Well...

\- Apple vs Google. The saga continues...

\- Apple as monopolist. They may eventually get maps right. But in the meantime, they can more or less screw their users over in order to force this change. Remember, it took Microsoft "three revisions" to get many of its now well-regarded products to a performance point that induced large scale consumer acceptance. Is Apple playing that game, now?

\- Lots of people apparently depend on maps, daily and heavily. They are rightly upset.

So... there's a lot going on with and underneath this story. In addition to the Schadenfreude.

</comment><comment> saiko-chriskun nothx</comment>


Show HN: Mlfinlab Python Package Released - <AUTHOR> http://www.quantsportal.com/mlfinlab-on-pypi-index/ <comment><AUTHOR> mlfinlab is a "living and breathing" project in the sense that it is continually enhanced with new code from the chapters in the Advances Financial Machine Learning book. We have built this on lean principles with the goal of providing the greatest value to the quantitative community.</comment>


Self-Organizations - <AUTHOR> https://shoniko.com/posts/self-organizations/ <comment><USER3> Bees, ants and termites are considered super organisms. As discussed, they don't really act as a great example of emergent behavior of organisms - <USER2> analogous to the organization of, say, cells within a mammalian body. Drawing the lines around species properly is helpful to understand the implications of natural selection, and the fitness functions surrounding emergent behavior.

Without doubt, eusocial insects represent some of the most successful species on the planet from a biomass perspective. At the same time, if you study the evolution of these species, you won't be motivated to "be a bee".

One way to describe the very complex multi-adaptation evolution in brief: wasp egg dumpers, over generations had the tables turned; host wasps started charging rent, then enslaving younger generations; some of the wasps started cloning their slaves, even breeding these clones for specialized roles.

Its a really interesting topic to dive into. [https://en.wikipedia.org/wiki/Evolution_of_eusociality](https://en.wikipedia.org/wiki/Evolution_of_eusociality)

</comment><comment><USER4> Picking a nit, the article claims that a free market works only in equilibrium. This is a spherical cow in a vacuum view of economics.

Markets, in general, can be seen as moving toward an incalculable equilibrium (at least not calculable in less time than it takes to reach). There is not a single equilibrium, though. As time goes on, people are born and die, trends and preferences change. If you could calculate some optimal state as of the end of 2019 and name it as "equilibrium", that would be different than the optimal state as of the end of 2020 (and different (and likely without some continuous transition function) in the full duration between these points).

<comment><USER5> Agreed. Equilibrium is defined in terms of the market, not the other way around. The system does what it does.

I think a better way of phrasing the article's point is that markets are metastable; pushed hard enough, they'll stop falling back to the current equilibrium and instead find a new one. That transition process often involves massive amounts of death and suffering, so that's why we care about not disturbing the market.

<comment><USER4> I'd push back on your use of "current equilibrium", based on how I interpret this phrase. If my interpretation is off, then I apologize.

It reads to me as if you're presenting the world as having a single equilibrium toward which the market moves. There is no meaningful static idea of equilibrium. However you want to define and measure some optimal state that we can call equilibrium, this target moves discontinuously _at least_ with each birth and death that occurs. Each of these events either introduces or removes a set of preferences (a terrible reduction of humanity). Accounting for these will shift our "equilibrium" in any meaningful calculation of the same. There are 1.8 deaths per second and 4.3 births per second.[0][1]

Again, these birth and death rates represent the minimum rate of change of equilibrium. As a follow-on effect, a birth is very likely to alter the parents' preferences in a way that will shift the equilibrium. Deaths likely shift families' and friends' preferences in a way that will shift the equilibrium. Both these second-order effects are not instantaneous (information does not spread instantly). Invention and innovation represent changes. Discovery of new natural resources represent changes. And people are not static sets of preferences. People change.

Perhaps we may be able to define some equilibrium for an instant. And we can expect with near-certain probability that one instant's equilibrium is very close to neighboring instants' equilibria. But the shifts are discontinuous, and likely impossible to model with a function.

[0] [https://www.medindia.net/patients/calculators/world-death- cl...](https://www.medindia.net/patients/calculators/world-death-clock.asp)

[1] [https://www.theworldcounts.com/stories/How-Many-Babies- Are-B...](https://www.theworldcounts.com/stories/How-Many-Babies-Are-Born- Each-Day)

<comment><USER5> I meant something else entirely:

> _It reads to me as if you 're presenting the world as having a single > equilibrium toward which the market moves._

The opposite. I hinted at that by saying "the system does what it does", though I realize it may be an obscure reference[0]. The markets are. They are a dynamic system that naturally reaches _some_ sort of (meta)stable behavior, and this is what we call equilibrium. It's not defined, it's _observed_.

> _this target moves discontinuously at least with each birth and death that > occurs_

Disagree. To a good approximation, markets moves continuously, because it takes time for individual preferences to manifest, and preferences aren't binary switches - they're communicated through many individual decisions over time.

WRT. the "death and suffering" part, I meant that large disruptions to markets usually involve events that kill and impoverish lots of people, and markets rebalancing themselves also cause that. I'm thinking of events like COVID-19, 2008, 1929, and quite often, discovery of new deposits of highly demanded resources.

\--

[0] - [https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha...](https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does)

<comment><USER4> Thanks for the clarification. I did not know that reference, and it seems we're much more in alignment than I thought.

I still hold that the target moves discontinuously, though not universally so. Death is the strongest argument here. A set of preferences exists, and then it doesn't. Some deaths are expected. Some are slow. Some are unexpected. Some are abrupt. Death is a binary off-switch for all preferences the deceased individual may have had. Without this collection of preferences, equilibrium abruptly jumps to some other value. Of course, in a system as large as the market, these death-jumps are going to be so small as to be effectively immeasurable.

Note: expectation of death here means at a specific time. All death is expected on a long enough time scale.

> To a good approximation, markets moves continuously

I agree here. My discussion of discontinuity, though, is that of movement of whatever ideal equilibrium the market is moving toward (but does not achieve). The equilibrium is the target toward which a market moves. This equilibrium is incalculable or at least not calculable in less time than it would take to reach. The target may move discontinuously (though I would argue the discontinuities are small for the most part), and the process moving toward the target may be continuous, both at the same time.

Ninja edit: An equilibrium in this discussion is more akin to a platonic ideal than anything concrete. The market is the system which can be observed. The equilibrium is an abstract and incalculable target.

</comment></comment></comment></comment></comment><comment><USER6> In addition to bitcoion, mentioned in the OP, several crypto projects in the DAO space come to mind. What if you don't need managers, but strong contracts between individuals? I think that's one of the theses floating around in that space.

[https://colony.io/](https://colony.io/)

[https://medium.com/@simondlr/the-moloch-dao-collapsing- the-f...](https://medium.com/@simondlr/the-moloch-dao-collapsing-the- firm-2a800b3aa2e7)

</comment><comment><USER7> I'll be that guy to point out that bees have a queen.

But more seriously, imo, it is an error to conflate permanent organizational perches of top-down systems ("leader") with command and control networks and 'critical nodes'.

In context of considering bottom-up, self-organizing system, it is important to note that order increases, and 'organization' is a neutral term. Critical nodes may (temporarily) emerge.

There is also nothing fundamental in the bottom-up organization process that inherently precludes the emergence of a hierarchy.

<comment><USER8> > bees have a queen.

The "queens" are gonad not brain.

<comment><USER7> I've always viewed the entire colony as the "thinker".

It is true that the queen is not the "brain" but then neither is the individual worker bee. Considered as a cellular automata, the queen 'cell' is a super-cell. The queen regulates flow of pheromones, and communication is communication, whether it uses electric signals or gradient diffusion of chemicals. That 'cell' is sitting on top of a hierarchy, and whether she is just a glorified "gland" as you would have it or not, little worker bees will NOT disregard her input.

<comment><USER8> FWIW it's not like we disregard the, uh, _input_ of our gonads, eh?

Anyway, I basically agree with you, except that I doubt the queen is doing much more, uh, thinking than any particular worker. I don't think she's more of a person (independent mind) than a worker.

<comment><USER7> It's very simple: there is a _hierarchy_ in a hive. And yes, on occasion I surprise myself, to say nothing of my demanding gonad, and veto hormonal urges.

The topic at hand, my original remark above, is that self-organization does not speak to the necessity of leaders, and hierarchical organizational order.

</comment></comment></comment></comment></comment><comment><USER9> Natural phenomena amazing as they are aside...

The "free market" doesn't exist because the economy is governed by rules established by governments and rules between governments in extension.

Cryptocurrencies right now are a speculative trading tool and store of value for others (very volatile at that), but I think that the cryptocurrencies that will really take off and seep into the mainstream are the government digital currencies currently being worked on (China is set to start testing the digital Yuan soon if not already).

My point is that nothing is ever truly free because nothing is ever by itself in a vacuum. You have to interact with outside forces and follow certain rules - natural or otherwise.

<comment><USER10> The cryptocurrencies that will really take off (in the long run) will be the ones that are most programmable and win the most developer hearts and minds. Digital nation-state sponsored fiat won't be able to compete with the complexity that is made possible by "finance legos". I think Ethereum and maybe what's coming together at [https://www.avalabs.org/](https://www.avalabs.org/) are good examples.

</comment></comment><comment><USER11> Ashby's work on self-organisation [0] is a must read. I think it was he that defined the concept.

[http://csis.pace.edu/~marchese/CS396x/Computing/Ashby.pdf](http://csis.pace.edu/~marchese/CS396x/Computing/Ashby.pdf)

</comment><comment><USER12> Even nails can self-organize with a little help : [https://www.reddit.com/r/nextfuckinglevel/comments/fc7mmh/ho...](https://www.reddit.com/r/nextfuckinglevel/comments/fc7mmh/how_to_organise_nails_the_right_way/)</comment>


The Man Who Deleted All His Tweets - <AUTHOR> http://inessential.com/2015/05/03/the_man_who_deleted_all_his_tweets

<comment><USER2> Something similar here: [http://fusion.net/story/50322/meet-the-tweet- deleters-people...](http://fusion.net/story/50322/meet-the-tweet-deleters- people-who-are-making-their-twitter-histories-self-destruct/)</comment>