Consider Yourself a Developer? You Should Solve the Project Euler Problems - <AUTHOR> https://blog.usejournal.com/consider-yourself-a-developer-you-should-solve-the-project-euler-problems-ed8d13397c9c <comment><USER2> I used Project Euler as a framework of problems when I was learning Go. The experience disenchanted me from the value of Project Euler for that purpose.

Project Euler is more about mathematical problems that you need to apply some generic algorithmic concepts but more importantly mathematical thinking to. It didn't drive me to exercise anything particular feature of the language I was using. I used it with Go, but I just as well could've used C, or Python. Which is part of Euler's point, really: not to be linked to a particular language.

I didn't complete 50 exercises however. I think I reached in the low 20s, stopping at a problem that involved Pascal's Triangle [1]. By that point, all problems had exercised mathematical ability, and none of my programming ability, which is why I stopped.

Maybe further exercises provide more of a programming challenge?

[1] [https://en.wikipedia.org/wiki/Pascal's_triangle](https://en.wikipedia.org/wiki/Pascal's_triangle)

<comment><USER3> The nice thing about math problems is that they're reasonably self-contained. If they didn't focus on toy math problems they would have to do other stuff like endless, increasingly complex form validation, filesystem manipulation, or things like graphics that involve external libraries-- it would just become hard to manage for code-golf scenarios, I think.

What would be a non math-heavy alternative to project euler be like?

<comment><USER4> Similar to Project Euler but about bioinformatics I would say [http://rosalind.info/problems/locations/](http://rosalind.info/problems/locations/)

But for learning a specific programming language and confront with other developers, not mentioned yet, there is [http://exercism.io/](http://exercism.io/)

Edit: also [http://www.puzzlenode.com/](http://www.puzzlenode.com/)

</comment></comment></comment><comment><USER5> Consider yourself a goal-post mover? You should post an article detailing arbitrary gatekeeping exercises that separate yourself from the chaff.

</comment><comment><USER6> I found the Euler problems to quickly become too difficult to suss out if someone doesn't have a strong math background. While I did enjoy the first couple dozen, it quickly left me in the dust just because I didn't understand all the notation and concepts at play.

<comment><USER7> Well, even if you get all the notation in the problem statement the background math knowledge required to solve the problem is ridiculous.

I remember a problem with Fibonacci numbers that required me to know some really weird Fibonacci identity.

</comment></comment><comment><USER8> Counter to some of the other posters, I got a lot of value out of doing 50 Project Euler problems when I was learning python. (I don't have a strong math background-- I took Calc in college and majored in biochemistry.)

I agree that PE problems aren't the best are getting familiar with the whole language, but you'll own the math, array, loop, and recursive nature of whatever language you do the first 50 problems in.

My math-major buddy solved the first 20 or so PE problems using _excel_. His "loop" was just him dragging in excel to duplicate a recursive excel equation.

I think the problems are very do-able by noobs looking to learn some aspects of a language. And you get a sweet badge when you do it: [http://projecteuler.net/profile/civilian.png](http://projecteuler.net/profile/civilian.png)

If you're getting bogged down in the algorithms aspect of PE, then I highly recommend taking the Stanford Algorithms course: [https://lagunita.stanford.edu/courses/course-v1:Engineering+...](https://lagunita.stanford.edu/courses/course-v1:Engineering+Algorithms1+SelfPaced/about)

</comment><comment><USER9> I've completed 120 of them or so. It's definitely true that most of then are more challenging in terms of mathematics than programming, but there are plenty of exceptions. That said, I don't think people without a math background should get discouraged. The math required is mostly of the type that few people are taught in school (number theory in particular) which means most people working through these problems are probably going to be encountering many concepts for the first time. Playing with the problem, getting a feel for it, and discovering neat facts through exploration, that's the real beauty of Project Euler. It really is a pretty amazing experience.

</comment><comment><USER10> Modern cycling evangelism differentiates between a "cyclist" (someone who defines themselves by their mode of transport) and a "person on bike" (someone who just uses a bike as the most practical way to get places). The implication is that we'll build better towns by providing infrastructure that works for people on bikes, not just for self-identified cyclists.

So, "consider yourself a developer?". No, not really. I'm just a person who codes.

<comment><USER11> Would you consider yourself a person who codes, then?

There's a word for that, you know.

<comment><USER12> Maybe just a person who solves problems by writing/creating software.

</comment></comment></comment><comment><USER13> A lot of sentiment here seems to be that Project Euler is more about math than programming. I feel like that depends on ones definition of the term programming. Computer science of course has nothing at all to do with computers as it only asks the question "What can be computed?" and Project Euler is very much in this spirit just as Knuths "The Art of Computer Programming" is. For example many of the problems require you to use specific algorithms to find an answer in polynomial time.

If you define programming as software engineering then of course Project Euler is mostly irrelevant. But computer science is math, not software engineering.

</comment><comment><USER14> I have limited exposure to the problems but the few times I took a stab at them I found the problems to be more centered around math theory and less computer science. Sure there might be some basic optimizations to be made but at some point it was about understanding the math. I don't think its necessarily helpful as a developer.

EDIT: I find it much more useful to run through building basic data structures and writing base algorithms (like sorting) in a new language.

</comment><comment><USER15> I don't think solving project euler problems will make you any better at being an effective programmer for pretty much anything except maybe game programming or other really mathy puzzly programming

</comment><comment><USER16> Project Euler is great for teaching yourself all the parts of programming that you will never use in the real world.

On the flip side, it's a fantastic exercise for learning how important time complexity can be, but again, 99% of real world programming is CRUD.

</comment><comment><USER17> Create your own "problem" and post it to Reddit. Its fun seeing people strugle and to see creative solutions. Here is mine

[http://go-here.nl/the-rabbit-problem](http://go-here.nl/the-rabbit-problem)

</comment><comment><USER12> I've been a professional software developer for over 25 years and in that time I have seen very few "computer science" related problems that needed solving.

Sure, it depends what type of work you do. E.g. if you work on limited power embedded devices for the medical industry your work as a developer will probably lean more towards the comp-sci end of the spectrum whereas if you're building snazzy browser based UI's with React.

How well you perform with Project Euler problems has absolutely nothing to do with how great a developer you are or whether or not you could consider yourself a developer because of it.

</comment><comment><USER18> Reading the comments as a mathematician... yup they're tough and require enough knowledge that I know they'll be challenging the next time around.

Reading this as a computer scientist: Knuth is where the education is at. And like project Euler, don't feel bad if you miss a few of the hard ones.

I was trained in computational number theory, and when I used them to teach myself ruby, and I stopped when I got comfortable enough with the language to find the problems tiring

</comment><comment><USER19> Project Euler is cool, but sometimes I worry about the proliferation of these "No True Scotsman" statements.

</comment><comment><USER20> Have a look at the Online Judge [1]. I've worked through a handful of these problems and they are good for forcing you to understand the problem and not over solve things.

[1] [https://uva.onlinejudge.org](https://uva.onlinejudge.org)

</comment><comment><USER21> "Lombardi's ruthless focus on the fundamentals made him the winningest coach in the history of football, with more championships (5) than any other coach in history."

Much as I hate everything he stands for, there is a certain hoodie-wearing punk that disproves this.

</comment><comment><USER22> > Lombardi's ruthless focus on the fundamentals made him the winningest coach > in the history of football, with more championships (5) than any other coach > in history.

Needs and update

</comment><comment><USER23> Obligatory recommendation for the Structure and Interpretation of Computer Programs (SICP). It not only gets deep into programming fast, it really shows you what a program actually is, or can be.

[https://en.wikipedia.org/wiki/Structure_and_Interpretation_o...](https://en.wikipedia.org/wiki/Structure_and_Interpretation_of_Computer_Programs)

Edit: Also +1 for Advent of Code</comment>


Europe's recession will be even deeper than expected - <AUTHOR> https://lite.cnn.com/en/article/h_cf9744f2a6542445f3e4adfb1dce24c5 <comment><USER2> I'm not really that interested in the article, but thank you OP for linking to the lite version of CNN. It loaded so fast and didn't have any autoplaying videos.

But related to the article- We have multiple continents that are significantly curtailing their output due to the pandemic, and it will last for a while yet. I'm a little surprised to see that they think Europe will contract by only 8.3%. That might make sense if it were just Europe being affected by the pandemic but all of their trading partners are affected too. I don't think they've accounted for how the entire world going down at once is going to effect the EU as a multiplicative effect, not an additive effect.

<comment><USER3> It's a reasonable concern, but a look at the report indicates they seem to have attempted to take into account the effects on the EU's main global trading partners. You can get the report here.

[https://ec.europa.eu/info/business-economy-euro/economic- per...](https://ec.europa.eu/info/business-economy-euro/economic-performance- and-forecasts/economic-forecasts/summer-2020-economic-forecast-deeper- recession-wider-divergences_en)

<comment><USER2> The report claims that their economy will take a 17% hit in the first half of 2020 (it has already taken this hit, but it takes a while for the accounting to shake out). And they're claiming that the yearly hit will only be 8.3%. I understand that they are trying to take a lot of things into account, but this forecast is too optimistic. Europe won't bounce back that much in H2, because their recovery will only happen at the pace of everyone else's recovery.

<comment><USER3> I agree it's optimistic, it assumes no second wave which is actually already happening and is sure to happen elsewhere in the world too.

<comment><USER4> Nothing proves that a "real" second wave will come with a similar lockdown to the first one. The first lockdown was catastrophic for the economy of some countries and didn't really help anything at all in regard to the health (best example is France/Spain vs. Germany). It seems (some) countries have learnt the lessons and now apply successfully limited restrictions (masks, social distancing, limited gathering, etc.) and this shall be the sweet spot for limiting contamination while not impeding further the economy. Arguably one shall not, again, underestimate governments for self-sabotaging themselves, but we can hope to see more reasonable measures being undertaken.

<comment><USER5> > "Nothing proves that a "real" second wave will come with a similar lockdown > to the first one."

I dont know about similar lockdowns, but its reasonable to be concerned about a 2nd winter wave. We seen what happens once winter time comes in Brazil

<comment><USER6> Let's hope Covid took Bolsonaro by then and his replacement isn't somehow worse...

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> I'm german and I remember people being pessimistic naysayers in the banking crash 2008, the "refugee crisis" in 2015, the "climate crisis" last year and many times before that. Apparantly a year without a near end of the world is a wasted one.

The single most appropriate quote in these times (it comes from 4chan of all damn places) is, and always will be: "Your impending sense of doom is nothing but emotional residue - there is nothing wrong. Even when something is wrong, nothing is wrong. You deal with problems as they arise, no big deal."

<comment><USER8> There's one crisis in this set that's real and we're doing way too little about it (climate crisis). Probably the biggest harm from short term panickng about these smaller things is taking away attention from it.

<comment><USER7> Exactly! Short-term panicking is the worst thing to do. I definitely think the climate change is man-made. I also think it is impossible to realy predict its environmental, ecological and social consequences on a global scale.

By listening to those, who try to nudge us into panic mode, we neglect calm long-term strategies that would improve our lives over time.

Imho, this applies to every "immediate threats" that are regulary propagated by (social) media.

</comment></comment></comment><comment><USER9> I don't even want to know how hard Spain will be hit. The country was kind of recovering from the 2008 crisis which was not pretty. Unemployment was still high before Covid, around 14% down from 26% which was not a good number but things were going forward.

Now this new crisis will be 5 times worse, at least in terms of GDP. In 2008 the GDP went down around 2,5% now it'll go down more than 12%. Madness. Around 50.000 small businesses have already shut down and thousands more are barely surviving. And now the government is going to increase some taxes. Taxes which were supposed to affect the rich but guess what, they won't.

I'm pretty concerned about several people I know with small/family businesses.

</comment><comment><USER10> COVID is the massive elephant in the room that this article doesn't fully address. Everyone is very focused on GDP, but this is merely a single metric. What about new cases, patients hospitalized, patients dying from COVID? These have dramatic repercussions for economic policy, and it seems the countries doing the best right now economically are those who handle COVID the best.

<comment><USER11> Global warming is the same issue. We don't seem to realize that the economy kinda sorta depends entirely on the environment, just like the economy kinda sorta depends entirely on public health.

</comment></comment><comment><USER12> This pandemic will be alot worse in economic terms than predicted. So far the only bright spot will be deaths seem to have really dropped, but death is a low bar compared to having severe chronic symptoms after infection. Alot of people have really shut down spending(my family reduced it by 50%) Expect international travel/tourism to be completely shutdown as well as a majority of restaurants and bars until prb early 2021. Sports and concerts also. That is a considerable chunk of the economy. Will a majority of these restaurants even exist next year? A few assumptions that have been proven wrong with covid-19:

1\. The virus will subside in the summer like the flu does. Based on cases in the US and other areas (sans Europe) skyrocketing right now this is absolutely not true, forget a 2nd wave we have a growing mountain of a 1st wave hitting the US/Americas now.

2\. Children are not really affected by covid. Israel had to shut its schools down after cases there were jumping, in addition kawasaki disease seems to be prevalent in a higher percentage of infected children than normally seen.

3\. Antibodies will offer protection against covid-19 after being infected. This too has been disproved although the jury is not out yet. Studies have seen neutralizing antibodies stay in a persons system for about 2-3 months. There are other ways the body knows how to fight an infection so this is not definitive yet.[3]

[1] - [https://www.washingtonpost.com/nation/2020/07/07/coronavirus...](https://www.washingtonpost.com/nation/2020/07/07/coronavirus- live-updates-us/)

[2] - [https://www.npr.org/sections/coronavirus-live- updates/2020/0...](https://www.npr.org/sections/coronavirus-live- updates/2020/06/03/868507524/israel-orders-schools-to-close-when- covid-19-cases-are-discovered)

[3] - [https://www.cidrap.umn.edu/news- perspective/2020/06/chinese-...](https://www.cidrap.umn.edu/news- perspective/2020/06/chinese-study-antibodies-covid-19-patients-fade-quickly)

<comment><USER13> > as well as a majority of restaurants and bars until prb early 2021.

The majority of bars and restaurants are back open in Germany.

<comment><USER14> Yeah, they are open in Finland as well ([https://valtioneuvosto.fi/en/information-on- coronavirus/curr...](https://valtioneuvosto.fi/en/information-on- coronavirus/current-restrictions)).

Though I think the parent was talking about worldwide numbers which I have no idea about.

</comment></comment></comment><comment><USER15> Growth expected to return by 2022 then? European oil providers supplies were supposed to be contracting by 2028 according to data from Rystad, before the pandemic, mostly because of below ground issues (and with very, VERY optimistic previsions concerning above ground issues imo). So anyway, Covid or not, European industrial production will peak this decade.

Times will be harder and harder for a lot of people.

<comment><USER16> A bad economy is good for the climate, which is good for the economy.

<comment><USER17> Bad economy might lead to wars and conflicts. But if it kills a lot of people, I guess that's good for the climate in the long game. Wars are also great for the economy assuming you survive.

Wow, the outlook for the next decade is fantastic.

<comment><USER16> I personally can hardly handle thinking about the future without taking the possibility of conflicts caused by the climate mess into account.

<comment><USER6> I don't know your age, but I've been rather worried about the fact that almost all my < 30 years old friends seem to be some degree of 'doomer'. Whether it's climate, the economy (and the prospects of retirement), or even just expectations if what their next few decades will be like.

Obviously I don't just worry that they feel this way, but also that they probably have a good reason for feeling this way, but either way it's worrying.

</comment></comment></comment></comment></comment><comment><USER18> A thing I've thought is really interesting on how the news is covering Coronavirus here in the US is they like to talk about all the economic harm going on in the US and then they blame it on the government response rather than the pandemic itself.

I've heard more than one news piece where they imply that Europe and Asia will be somehow spared from the economic fallout of putting their countries on lockdown for months on end. Then in the next piece they'll talk about how the US unemployment rate is off the charts and find a way to blame the Trump administration for it.

The economy is suffering because we slammed on the brakes in what (at least at the time) seemed like a prudent measure to avoid a worse fate. Now we have to pay the price for that. Choices have consequences.

<comment><USER19> Well, a country will have less economic fallout if it manages to keep SARS- Cov-II at bay. Trump managed to evoke the collateral damage of a lockdown without the full benefits. And that is on the government.

<comment><USER18> > Trump managed to evoke the collateral damage of a lockdown without the full > benefits.

Besides the fact that the Federal government doesn't actually have the authority to invoke a lockdown, and it was done on a state by state basis, yes the Federal government did advocate for it.

As to the benefits of the lockdowns as performed in various countries, I think it really remains to be seen how badly the US will be affected vs the rest of the world.

As of right now, yes, the US looks particularly bad in terms of the number of infections / deaths. But it's also an area the size of Europe with the vast majority of the deaths restricted to New England.

I wonder how things would look if you compared individual states to European countries. For example, here in Iowa, we're at I think 700 deaths and 30,000 confirmed cases. That doesn't seem too catastrophic to me.

<comment><USER20> You can make your comparison with these two pages:

\- [https://www.worldometers.info/coronavirus/](https://www.worldometers.info/coronavirus/) ("Europe" is all Europe, not just the European Union. That's 747 million people.)

\- [https://www.worldometers.info/coronavirus/country/us/](https://www.worldometers.info/coronavirus/country/us/)

In that case, Iowa is doing very badly for cases/1M. I don't know if the comparison is fair, it might depend on who is tested.

For deaths/1M, it's somewhere around the European average.

Some European countries are very densely populated, like Belgium, the Netherlands, others less. Though much of Europe is as dense as New England.

<comment><USER18> > In that case, Iowa is doing very badly for cases/1M.

This is interesting to me. I don't personally know anyone who has caught Coronavirus, much less been hospitalized or died from it.

If this is average or worse than average, it really makes me wonder why we've gotten so worked up about the re-opening. I assumed based on the way it's covered on the news that people were dropping like flies in other parts of the world.

<comment><USER20> I'm really not sure if the cases/1M is a reasonable figure to compare.

At least between European countries, there have been quite different approaches to testing -- who gets tested varies depending on symptoms, perceived risk to themselves, risk to others, and availability of testing.

I don't personally know anyone here in Denmark that has tested positive. Two people I know know someone who has tested positive here. Many hotspots seem to be limited to quite small communities -- a single building, a single care home for the elderly, or a single food processing factory might have tens of cases (even deaths for the care home), so it's not so surprising that I'm unaware of anyone infected.

</comment></comment></comment></comment></comment></comment><comment><USER21> Looking to many people that I know that I quite obvious.

Plenty of people are still not able to work, and it is not like everyone can work from home.

</comment><comment><USER22> Fantastic news for the environment and ourselves if people overall consume less

<comment><USER23> Are you seeing that people are taking that message to heart? Those around me don't seem to be changing their attitudes. So, I don't see any reason why things will change long term.

<comment><USER24> > Are you seeing that people are taking that message to heart? Those around me > don't seem to be changing their attitudes.

Lots of aircraft still parked (~15k).

</comment></comment></comment><comment><USER25> 8.3% cause everything is fine and Covid will be gone next week... right?

<comment><USER6> that comment is even more useless than this one.</comment></comment>


California solar power output sometimes exceeds predictions - <AUTHOR> http://www.latimes.com/projects/la-fi-electricity-solar/ <comment><USER3> This isn't abnormal, other examples.

Paying the chemistry industry for shutting down energy intensive operations during low electricity periods

Transfers between countries in Europe and fines when your electricity prediction is off ( it costs a lot)

Useless spending power on hot days for not blasting the net because of solar power

Lower costs at night for using electricity

It's not as easy as it looks, solar power really puts an additional stress on our net, definitely in the beginning of the 'electricity usage behavior shift'. Read more about the reason why electricity is unstable in some countries : [http://infoboxdaily.com/3-african-countries-with-erratic- pow...](http://infoboxdaily.com/3-african-countries-with-erratic-power-supply- apart-from-ghana/) \- didn't find anything better

Source: done a project for 'Electricity usage prediction' in Belgium with one of my previous employers.

</comment><comment><USER4> And yet our rates keep going up, which encourages less electricity usage and higher adoption of personal solar installations. Seems like this is going to be a pretty rough cycle for the power companies to navigate if they aren't allowed to change their business models...

</comment><comment><USER5> The sad part is that many people will read the title and use the article to say "solar power is ruining everything and is a waste of taxpayer dollars!" But, when you actually read the article you realize that the utilities are the issue because they keep building natural gas plants and transmission lines rather than just investing in storage, demand response, or other technologies that add more flexibility to the grid.

</comment><comment><USER6> If the consumer electricity rates were adjusted dynamically, I bet that would soak up the extra power. For example, one could charge one's electric car when the power cost is very low.

</comment><comment><USER7> This seems to be a temporary problem until green electricity has the capacity and is cheap enough to be used year-round.

Natural gas is so much cheaper on a $/BTU basis, but perhaps with enough solar capacity (or other green sources) that wont be the case much longer. It would be great if I could keep cool and warm using electricity alone.

</comment><comment><USER8> Ill tell you what. Every single bulb is LED, tv plasma-> led, all appliances and fans upgraded to energy efficient, all HDDs->SSD, all lights on sensors, no heating used, no AC used, and my bills are bigger here in CA than ever. So if there is a glut of power I can also say that datacenters pay less than people in homes and its not cheap to charge electric cars, in fact, they are charging more per kwh and changed a lot of free charging stations to pay- for/metered. As a little guy in a brutal tax bracket its too bad all this energy progress hasnt helped out with the bills.

</comment><comment><USER9> California should use the power to mine bitcoin!! :)

<comment><USER10> this is the best answer. we should contact Jerry Brown.

</comment></comment><comment><USER11> Maybe the could make our PG&E bills not be (very quickly) .30/kwh and instead follow the .12/kwh average of the USA during these peak times.

</comment><comment><USER12> Utility companies usually have load shedding agreements with industrial consumers that can tolerate power supply temporarily going down.

Is the inverse - consuming above than average and building up heat, chemical stock or whatever they usually do that lets them ride out supply slumps - <USER2> difficult?

It seems extremely wasteful to pay another state just to have them power down their solar farms.

</comment><comment><USER13> Is this just a function of how many households in CA get 100% of their power from solar currently? I got an email recently from PGE where I can opt into and decide the % of my power that comes from solar.

Does the state generate enough power to move over every household to solar?

</comment><comment><USER14> What a difference 16 years makes:

[https://en.wikipedia.org/wiki/California_electricity_crisis](https://en.wikipedia.org/wiki/California_electricity_crisis)

<comment><USER15> Wasn't that crises entirely manufactured though?

</comment></comment><comment><USER16> Happened in Texas too

</comment><comment><USER17> I'm sure there's some business opportunities that can be taken to alleviate this surplus. There's a generation of people that have been groomed to use their electrical appliances during the night time wether to do laundries or wash dishes because they would be charged less from energy providers. In recent years, people are doing their charging at home in the middle of the night.

But perhaps this can flip around. Imagine laundromats that charge cheaper for drying or washing during the day vs during the night. Cheaper electric car charging during the day. Business opportunities that benefit everyone simply by changing some habits.

<comment><USER18> > I'm sure there's some business opportunities that can be taken to alleviate > this surplus.

You need to be able to message the energy consumers (most likely smart versions of large appliances) within minutes notice.

I think you would have to have appliances ready when you leave home in the morning and set the power price at which you are willing to run it. Right now, I don't think most large appliances support such a system. Perhaps home assistants (Amazon Alexa, Google Home, Apple HomePod, etc) would be the right place to integrate these settings, but you would still need the assistant to be able to interact with your dishwasher / clotheswasher / Tesla Powerwall.

At the moment, we are adding solar to the grid faster than we are adding smart appliances that are capable of deciding when to use power, based on the current market price of power.

<comment><USER19> This strikes me as a viable usefor IoT-style appliances. Or maybe IoT outlets.

Also, I bet a swimming pool could be cooled down when power is cheap, and the chilled water pumped through a heat exchanger as air conditioning.

</comment></comment></comment><comment><USER20> Seems like some local industry would be able to use this? Or maybe turn on some air conditioning or hot water heaters.

<comment><USER21> Making ice is good way to bank cheap power at a company-scale because they can use it to offset AC later.

</comment></comment><comment><USER22> Solar panels can also decentralize energy generation, letting communities be more resilient and prevent global blackouts.

</comment><comment><USER23> Alternative title: Solar power output exceeds most optimistic predictions.

It seems like a non-story for a burst-y power source to occasionally produce more power than we need.

<comment><USER24> Yeah, the more interesting aspect was that new traditional power plants are still being built, despite the glut of solar.

<comment><USER25> I work at The Times on the team behind these stories and FYI we did a whole previous investigation focused on just that

[http://www.latimes.com/projects/la-fi-electricity- capacity/](http://www.latimes.com/projects/la-fi-electricity-capacity/)

</comment></comment></comment><comment><USER26> I wondered whether TFA would ever get to the cause of this waste, and eventually it did!

 _Once state regulators approve new plants or transmission lines, the cost is now built into the amount that the utility can charge electricity users — no matter how much or how little it is used._

Oh, so _that 's_ why they kept building natural gas plants that don't get used most of the time! If you've ever been to a PUC meeting, you've seen corruption at its most brazen. "Citizens' groups" entirely funded by utilities and telcos wait their turn to dutifully read the prepared statements, the commissioners dutifully thank them, and then they go to work with the rubber stamp. It doesn't actually matter that power plants get built in a timely fashion or actually operate more than a small percentage of the time, the rate-payers are already on the hook. The only way a utility executive gets a bonus is by saddling the public with CWIP charges, so that's what they scheme to do every working day.

No coal, gas, or nuclear plant should ever be built by forcing the public to do the job of investors. If investors can't be found to support a plant that supposedly will generate for decades, maybe we should listen to the market?

[EDIT:] Great article, though, because it tells the whole story, and that seems rare anymore.

<comment><USER27> I don't understand so when a natural gas plant is built they adjust rates upward? Isn't that the opposite of supply and demand?

<comment><USER28> No. In theory, that new gas plant is "planned" and replacing aging capacity, while meeting expected demand.

Rates are separate.

Many power utilities are granted a monopoly, especially the transmission (high voltage, long distance) and distribution ("last mile" / neighborhood wires) horizontals as they're true natural monopolies.

In exchange for that concession, their profits are capped. The cap is a function of their "rate base" \-- i.e. qualifying capex -- and an approved rate. These rates are adjusted periodically, requiring public utility commission (State) or FERC (federal) rate setting sessions.

For example, "Total Revenue Requirement = Rate Base [assets] × Allowed Rate of Return + Expenses" [0]

Without having looked into this, it's easy to over-estimate how easy (hard) life as a public utility is.

[0] [http://blog.aee.net/how-do-electric-utilities-make- money](http://blog.aee.net/how-do-electric-utilities-make-money)

<comment><USER29> This kind of regulatory response to natural monopolies can (and does) result in 'gold-plating'. It incentivises profit-maximising monopolists to over- invest in qualifying capital: even though their ROI remains the same, their absolute profit grows larger.

However, there's no easy solution here: natural monopolies are a genuinely tricky public policy issue. Alternative solutions that come to mind are more direct price controls or public ownership. I'm not suggesting these are superior alternatives; just throwing them in there to provide more grist for the conversational mill.

<comment><USER30> California is an expert at this.

[https://en.wikipedia.org/wiki/California_electricity_crisis](https://en.wikipedia.org/wiki/California_electricity_crisis)

</comment></comment></comment></comment></comment><comment><USER31> Could excess power from CA be exported to Mexico or Central America?

<comment><USER32> I've heard that desalination of seawater is not a practical solution to drought because it's too energy-intensive. Does this change when energy temporarily costs less than $0/unit?

<comment><USER33> It's not true in the first place. Arab states, and Israel, make heavy use of it; desalinated water is easily affordable for US residents.

What makes US desalination impractical is not the cost of desalinating, but either or both of:

\- the cost of building a plant in the first place

\- the availability of cheaper fresh water

<comment><USER19> Southern California, for example, dumps most of it's rain water into the ocean. I'd bet that capturing some of that would be much cheaper in the long run.

it's not hard to come up with some creative solutions... How about a giant bag or huge tanks out in the ocean for holding the water, instead of building dams. Or injection wells to replenish the water tables during the rainy season, pumping it out during the rest of the year.

It just seems weird that millions of gallons of fresh water gets dumped into the ocean, and nothing is done to capture even some of that water.

</comment></comment></comment></comment><comment><USER34> If Tesla can really put up grid storage modules as fast as they claim, shouldn't this just be a big opportunity to print money? Get paid to charge the batteries on surplus surges, get paid to supply power at night discharging them...

<comment><USER35> Yes. The "obvious" question when reading an article like this is "Why pay someone to take the power, why not just turn off a gas or coal fired plant?" There are some obvious restrictions around that (like the plant would need to be in the general area of the grid where power was coming from of course) but the issue according to the PUC is that "cycle time" which is the amount of time it takes to turn off a power plant and then turn it on again, is longer than the "event time" where you get an over power event transient.

Grid storage nicely addresses both of these issues because it can absorb the extra energy (charging) and when the grid storage units are charged you can safely turn off a power plant because even if the power suddenly dips for what ever reason the grid storage can cover that loss while the power plant is restarting.

If you reach the point where you have enough grid storage to completely cover the variance between day time and night time power demand, then things get even more efficient as you switch to that storage a 'primary' and then cycle on plants as needed to top them off.

<comment><USER36> But so much of the demand is flexible. For example, I get zero benefit from heating my water tank at a particular time. Building grid storage so I can run my heat pump when it's dark and the air is still is inefficient and silly.

<comment><USER35> I'm not sure what you are trying to say here. At the individual user level energy usage may appear random and chaotic but at the community level that is not the case. To use your example, at any given instant in time _someone_ in your city is heating water (whether intentionally by consuming water from a hot water storage facility, or passively as that facility activates automatically to keep the water at a given temperature). All of those usages level out to an aggregate demand on energy to heat water over unit time.

If you think about it as a capacitor that "filters" high frequency noise on the electrical grid it might be clearer.

<comment><USER37> They level out to some extent, but they also enable far larger spikes to form when external events cause normally relatively "random" things to synchronise into narrower intervals across a large number of households.

E.g. the UK has what is called "TV pickup": Shortly after East Enders ends (daily evening TV soap opera), you'll get the sudden surge of 1.5m+ electric kettles turned on at once.

We see that type of situation where "random" events coalesce into very narrow time bands in all kinds of areas. I'd imagine we'll see this type of thing also happen for automated processes. E.g. air-con units ramping up based on thermostats in very lose proximity in the same areas etc.

I feel this is one of the areas where we need a smarter grid and smart home tech and home based batteries to work in conjunction to time-shift load and smooth out the demand more.

E.g. a hot water storage system can afford to turn off for 5 minutes (or whatever) and let the water cool slightly to take advantage of a discount 5 minutes down the line, or an air con unit can reduce speed for the same reason. Or a deep freezer that takes advantage of whatever the safe variation in freezer temperature would be.

Or, hey, I'd like a button on my washing machine and dishwashers that say "I want it to be done by time X. Find the lowest guaranteed rate (e.g. nightly rates) that will let you complete by X. If between now and then you get an offer for a lower rate, start then instead". Most of the time we just set "random" time delays to run them during the night, but most of the time I don't care if it _actually_ runs then as long as it's done by the morning or by the time I'm done working etc.

You already have that happening to an extent at industrial scale. E.g. with industrial processes that require lots of energy that can be slowed or sped up according to prices, but there's a lot of discretionary demand in home as well that could be leveraged to smooth out the demand curve a _lot_ more, and with a battery tied in it could smooth it even more.

</comment></comment></comment></comment></comment><comment><USER38> Reading these articles is like listening to five year olds arguing. "These guys says it's enough!" "These other guys say it's not!" "But what if it's cloudy?" You can picture the journalist holding a toy microphone and nodding intently at each source, the argument going entirely over their head.

The entire argument revolves around the distribution of power usage and of power generation throughout the day and throughout the year. So why not explain that for your readers? Show us charts with that information? Tell us how much energy is produced by solar on the cloudiest days? Show us a chart of power usage over solar power production over the last year, and tell us the minimum and maximum percent?

At least explain the difference between minimum, maximum and average production, and when each of those matters!

But no, none of that. We get a graph showing the increasing share of solar generation, and another with absolute figures of the curtailed solar capacity. And a bunch of pictures and animations, of course.

This is what innumeracy looks like.

<comment><USER39> > This is what innumeracy looks like.

I don't know whether it's innumeracy so much as lack of time, and also the specific expertise (not just ability to understand numbers, but expertise in this specific area) to be able to understand the information. For a general reporter (not an 'energy correspondent' or something similar) the article you're proposing would take at least a week of study and research, probably more, this 'he said, she said' stuff can be knocked off in a few hours.

<comment><USER40> It's innumeracy. I don't need to know anything about the subject matter to understand why the concepts, numbers and relationships mentioned above are important. In fact, I am absolutely not a solar expert -- I have no expertise in this specific area -- and I know that I am entirely qualified to understand the basics of load variation over time.

</comment></comment></comment><comment><USER41> They should pump water uphill during these surges. California hydropower is constrained by rainfall, so the dams and generators already exist. Pumps cost less than batteries and this process has about an 80% roundtrip efficiency

<comment><USER42> They do. Many of the reservoirs of the California Water Project also do power storage. They usually pump up at night and generate in the daytime. Sometimes they can't store more energy because they're full of water, as happened at Oroville Dam this year. Sometimes they don't have enough water. A year ago, some hydro plants in California were down due to lack of water.

There's one big pure pumped storage plant in California - Helms, near Fresno.[1] The geography has to be right for pumped storage. You need a high reservoir and a low reservoir near to each other, and since the water levels will change drastically each day, they can't be used for boating or recreation much. Helms has about 1500 meters of elevation difference between the reservoirs. Sites like that are hard to find.

Excess solar power isn't a big problem in areas that use air conditioning. Peak A/C load and peak solar output tend to coincide. The big excess power headache is wind, where output varies 4:1 over a day over wide areas. Excess wind generation late at night is common.

The best way to understand the power business is to read PJM 101, the introductory training material for the US's largest power grid. Unfortunately, PJM recently put that behind a login.

[1] [https://en.wikipedia.org/wiki/Helms_Pumped_Storage_Plant](https://en.wikipedia.org/wiki/Helms_Pumped_Storage_Plant)

<comment><USER41> Peak power usage is around 7-8PM, this discrepancy is the source of the duck curve. More storage could allow tons more solar to be installed.

[https://cleantechnica.com/2014/07/21/utilities-cry-fowl- over...](https://cleantechnica.com/2014/07/21/utilities-cry-fowl-over-duck- chart-and-distributed-solar-powercrying-fowl-or-crying-wolf-open-season-on- the-utilitys-solar-duck-chart/)

(deepsun - the "no sun" critical event happens every night)

<comment><USER43> 7-8pm according to that link is the _net_ peak, after solar production has canceled out the original afternoon peak. The problem is that even though solar supply does roughly coincide with the peak of demand, solar production has increased enough that, since about 2013, it's gone past the point of offsetting peak afternoon A/C usage, to now producing an actual net trough in the afternoon hours (you can see the afternoon extremum flips from a local maximum to a local minimum from 2012 to 2014, and just keeps going further down every year since, as peak afternoon solar grows faster than peak afternoon A/C).

The new net peak around 7-8pm is because demand has a long tail into the early evening, while solar drops off sharply around 5-7pm. But yeah the end result is that it does cause a problem for grid management if you add more incremental solar, since the original afternoon peak it helped smooth out has long since been smoothed and then some.

<comment><USER42> Wind is really variable, and often way out of sync with load. I just looked at PJM's dashboard, and peak wind in the last 24 hours was 4 gigawatts at 4 AM. Wind bottomed out at 0.65 GW at 8 PM. That's total output from wind farms from Illinois to New England. Averaging over a large area isn't that helpful.

Piping energy in from the wind corridor that runs up from the Texas panhandle to Canada has promise. The wind is steadier there. It's quite possible to run megavolt DC power lines now; China has 16,000Km of them. California has had a half-megavolt DC link from Oregon to Southern California since the 1970s.</comment></comment></comment></comment></comment>


Ask HN: Fear of getting on planes - <AUTHOR> Hi HN,
I'm looking for some advice.
I got a new job. It is my dream job at a company I've wanted to work at for a long time now. It's perfect, except for just one thing.
I'm going to have to do some long-haul flights (10-12 hours). And, I know it sounds crazy, but there is no place I'd rather be less than on a plane: trapped inside a small, cramped, flying metal box with 400 people for 10-12 hours. I get claustrophobic and panicked.
I don't know what to do, and I fear I might have to quit before I've even started and go somewhere else that doesn't require travel. I'm going to start seeking expert help for this phobia, but I don't think I'll be able "fix it" any time soon, let alone before I would have to fly.
Looking for your collective advice,
Thanks! <comment><USER2> It doesn't sound crazy at all -- lots of people have a fear of flying/planes.

I'm working on a program that uses VR to help people overcome their fear of flying. Would you mind filling out this short fear of flying survey, so I can get in touch with you when it's ready?

[https://goo.gl/forms/RA8RfTUebvZUM3Un2](https://goo.gl/forms/RA8RfTUebvZUM3Un2)</comment>


Boston Dynamics' "Spot" bots pulling a truck [video] - <AUTHOR> https://www.youtube.com/watch?v=OnWolLQSZic <comment><USER2> That's incredible! How is each robot getting so much grip to pull that much weight with those tiny legs hitting the ground?</comment>


Notice of Retraction due to a programming error - <AUTHOR> https://jamanetwork.com/journals/jama/fullarticle/2752474 <comment><USER2> _To reduce the occurrence of future similar programming errors, the Johns Hopkins Biostatistics Center has instituted a new standard operating procedure for checking randomization assignment to be followed in all trial analyses. To ensure that the group assignment used in any of the trial analyses is correct, a verification process will be included at the beginning and end of each analysis program. This process is intended to confirm that the group assignment separately provided by the trial team matches the group assignment used in the analysis program. The matching confirmation is reviewed by a second biostatistician /analyst before its use in the results._

I don't know what software quality control is already in place at this organization, but this corrective measure seems on its face wholly inadequate to me: they're just preventing a recurrence of _the same exact problem_ , rather than the much broader _class of problems_ due to programming errors. Do they have a code review process in place?

This speaks to a larger issue: if you write software for manipulating data as part of the production of a scientific paper, then the source code should be available for review as an attachment to that paper, and review of said code should be part of the peer review process in any reputable journal. Professional software engineers write bugs all the time that invalidate the correctness of their programs, never mind individuals whose primary job is research, not software.

<comment><USER3> >This speaks to a larger issue: if you write software for manipulating data as part of the production of a scientific paper, then the source code should be available for review as an attachment to that paper, and review of said code should be part of the peer review process in any reputable journal. Professional software engineers write bugs all the time that invalidate the correctness of their programs, never mind individuals whose primary job is research, not software.

Completely agreed, the source should be open (ideally FOSS) - but also, the software development should be conducted properly too. On a practical level, using version control and a code review mechanism (e.g. GitHub PRs) _within the research group_ equivalent in rigorousness to what you'd see at a good practice software development shop in industry.

Clinical decisions are made off the back evidence published in peer-reviewed, respected journals. It would seem to me that serious software errors in this domain have the capability to contribute to grave patient consequences. Much more serious consequences than if _I_ introduce a bug into a client project.

<comment><USER2> Right, that's what I meant by code review in the first paragraph: internal review within the research group.

Agreed also that they should use version control and other standard practices, etc.

<comment><USER3> > Right, that's what I meant by code review in the first paragraph: internal > review within the research group.

Ah, I took it as code review during the peer review process (i.e. from reviewers). Unsure how realistic this would be.

<comment><USER4> > Code review during the peer review process

Unfortunately, this would make the already laborious process of peer review even longer and require more work. Given the reality of academia today (publish vs perish), not to mention the added work of preparing even well structured, version controlled code (which is not, in my experience common) for publishing, most researchers would not opt-in (or support) something which would make publishing more difficult.

<comment><USER2> Maybe this sounds harsh, but:

Who cares? If researchers are producing crap to get published, why should we mind if they stop producing that crap when journals raise the standards of publication?

</comment></comment></comment></comment></comment></comment><comment><USER5> I previously made a big list of papers that were retracted due to software bugs. It was intended to go in a manuscript but I had to cut it out because the conference limited the number of references for the camera-ready version. If anyone is interested I can try to dig up the list again!

<comment><USER6> Sounds great for the arxiv!

In fact, I would treat the retracted papers more like a data set than things to be cited. Then you get a nice paper with counts and statements about common themes, and references on those themes. Then post the dataset as supplementary material available on the arxiv.

<comment><USER7> Good point, and by not citing the flawed papers you are not inflating their academic pagerank.

</comment></comment></comment><comment><USER8> This isn't surprising, and I'm sure has happened many times. If you get the result you expect, you are much less likely to check for a mistake. The authors deserve a lot of credit for owning up to it.

They did the analysis with Stata.

</comment><comment><USER9> This may be _the_ best way to get a negative result published: as a retraction of the published reversed-positive findings!

</comment><comment><USER10> > Given the corrected finding of a paradoxical increase in acute care use in > the intervention group

Now I'm curious why long term intervention/support increased the number of acute cases. Maybe people were more likely to find themselves sick when provided with additional monitoring after they leave the hospital? Some sort of psychological connection or being overly careful?

Plenty of doctors will simply blame your past diagnosis for any broad new symptoms, without doing much critical thinking or investigating. I've seen this personally many times in the years following a colitis diagnosis. The symptoms are quite broad and easily mistaken.

Anyone know if the new article is available yet?

<comment><USER11> You can find it here [0]. Easy to miss the reference since it is only indicated in the text by a superscript. Personally I prefer using the text as link or a reference put inside brackets inline.

0\. [https://jamanetwork.com/journals/jama/fullarticle/2752467](https://jamanetwork.com/journals/jama/fullarticle/2752467)

</comment></comment><comment><USER12> I feel like a "best retraction of the year" award with a real monetary prize would be a good incentive to keep these coming.

</comment><comment><USER13> How does this stand up as a post-mortem? Sufficient? Is there a standard for post-mortems?

</comment><comment><USER14> Curious what is the incentive for an author to retract a study?

Shoulnd't they just leave it out? Continuing to accrue more publications and quotes, or whatever the metrics are in research.

<comment><USER15> Well other than academic integrity and general honesty if people can't reproduce your research you might not keep your job.

</comment></comment><comment><USER11> >Over the course of this reanalysis, we detected an error in imputing missing values for the SGRQ, whereby the worst possible score (100) was incorrectly imputed for missing values of participants who had died beyond the 6-month study period. The correct approach would have been to classify those values as missing because those participants had not died by the 6 months after discharge study end point.

The reassignment error is possibly forgivable, but I think this second error should have been easier to catch and is much less easy to forgive. A simple filter check between possible score and some other status variable in the dataset would of caught this mistake. I am doing a Masters in Biostatistics and this kind of checking is being taught to us early on, I hope there is more focus on it later to help avoid mistakes like this.

<comment><USER16> Trivial errors can slip through the cracks easily.

For example, people sometimes misspell "would've" as "would of" even if they actually know that the latter spelling is actually incorrect.

Pointing fingers is easy after the fact, but spotting every possible error all of the time – no one is able to do that.

I've even made that very same spelling mistake you did a time or two myself even though I try really hard to be correct in spelling and in all aspects of grammar, and even though I am _well_ aware that "would of" is just plain wrong. We all slip up, and sometimes we do so in embarrassing ways. Especially when we are lacking sleep or when we are otherwise exhausted.

<comment><USER17> Exactly.

The notion of "failing forward" is somewhat related here -- don't focus on the blame game when addressing honest mistakes. (Fraud is a different matter). Focus on rectifying then moving forward.</comment></comment></comment>


Knuth's 2018 Christmas Lecture: Dancing Links - <AUTHOR> https://www.youtube.com/watch?v=t9OcDYfHqOk <comment><USER2> In Fall of 2017 I was still in school and had to write a sudoku solver in one of my CS classes. I ended up using dancing links to solve my sudoku problems. My implementation was in JavaScript and it could find solutions for sudoku problems very quickly. The paper I read on dancing links ended up being my favorite academic paper I read in 2017 (honestly probably one of the only ones I read that year.) I wrote about that in this thread [1].

[1]: [https://news.ycombinator.com/item?id=16036588](https://news.ycombinator.com/item?id=16036588)</comment>


E. chromi Designer Bacteria Will Color Your Poo According to Your Illness - <AUTHOR> http://inhabitat.com/e-chromi-designer-bacteria-will-color-your-poop-according-to-what-ails-you/ <comment><USER2> Warning: you can never un-see the images....</comment>


Ask HN: Are there bootcamps for developers to become founders? - <AUTHOR> There are bootcamps for people to become developers in 3-6 months of time but are there bootcamps that help you become a founder?<p>I feel there is a mental resistance in me (and may be others) to feel ready before they can start a company. There are a lot of skills involved in starting a company. I am looking for something like a Toastmaster but for Startups where you can learn skills and have a group that support you in starting a startup or small business. <comment><USER2> I cannot find the program I tried to Google for, but found this along the way, from a well known source of advice afair ...

[https://amp.reddit.com/r/Entrepreneur/comments/7taz53/is_amy...](https://amp.reddit.com/r/Entrepreneur/comments/7taz53/is_amy_hoys_30x500_course_worth_it/)</comment>


Researchers: Are we on the cusp of an 'AI winter'? - m-i-l https://www.bbc.co.uk/news/technology-51064369 <comment><USER2> I think there's an interesting disconnect right now between research and practice. Cutting-edge research does feel like it's reaching a plateau - across most AI fields even "major" breakthroughs are only gaining a couple percentage points and we're probably starting to hit the limits of what current approaches can achieve. When the state-of-the-art is 97% on a task, there's only so much room for improvement. Yoav Goldberg posted a tweet about Facebook's RoBERTa model that summed it up pretty well: "oh wow seems like this boring public hyperparameter search is going to take a while" [1]. There's a vague feeling of "What's next?" now that all the benchmarks are fairly well-solved but AI in general clearly doesn't feel solved.

However, state-of-the-art models aren't really used in production yet. I think the trend of "use AI/ML to solve X" has only started to pick up in the past 2 years, and it'll continue well into the 2020s. The process of taking research models and putting them into production is not standardized yet, and many models don't even really work in production - if your model takes a second to do an inference step that's fine for research but maybe not for a real product.

I think in the next decade, on the research side, benchmarks will be beaten less often, and instead there will be more focus on trying out radically new things, understanding weaknesses in current techniques, and finding new measurements that assess those weaknesses. On the industry side, there will still be lots of cool and exciting new achievements as already-known techniques are applied to old problems that haven't been addressed by AI yet.

As an aside, this was the first time in my life that I read the phrase "10s" referring to the 2010-2019. Kind of an odd-feeling moment!

[1] [https://twitter.com/yoavgo/status/1151977499259219968](https://twitter.com/yoavgo/status/1151977499259219968)

<comment><USER3> > Cutting-edge research does feel like it's reaching a plateau

It's really not. The second half of last year alone had MuZero and Megatron- LM, to name just a couple that most scream to me that we are actually progressing towards AGI.

You say 'When the state-of-the-art is 97% on a task', but solved tasks are the least interesting tasks.

<comment><USER4> Do we have any other indicators that it's actually progressing somewhere, besides screaming? Even such a triviality as "how do we recognize that we got there"? The research is still in very early phases, IMNSHO: impressive practical applications appear, but they're side effects of what appears as random flailing: "build it bigger, see if it helps. Build it sideways, see of it helps. Build it at full moon, see if it helps".

That suggests that the applications are the low-hanging fruit, with far more interesting results still to be discovered.

<comment><USER3> MuZero is in some sense the proto-holy grail, in that it implements learning and planning into unstructured tasks over purely internal models. While there is an obvious chasm between it and the end point, this is still something that has only recently become more than an abstract goal, at least to any effect.

Being able to perform planning over 'simple' domains like Atari games and Go (and not even in the same trained model!) might not seem very comparable to the real thing, but evolutionary history spent the bulk of its time building up the basics—most animals fail most cognitive tasks—so I don't think this is indicative of the progress being misguided, especially given networks-on-GPUs is literally a 10 year old field.

I think MuZero is a clear example of building by principles over random flailing. I get why there does also seem to be the latter, but it's certainly not the whole of it, and anyhow it worked for evolution ;P.

<comment><USER4> Sure, that looks promising. I'm not holding my breath for The Holy Grail: in an environment where it's been just-around-the-corner for as long as the field exists, there's always another unexpected corner;)

</comment></comment></comment></comment></comment><comment><USER5> > While AGI isn't going to be created any time soon, machines have learned how > to master complex tasks like:

> translating text into practically every language

Note: they said they have mastered these tasks.

Yeah... I'm not sure a lot of native speakers would agree. Here's a great example of using Google Translate to automatically translate a video game.

[https://www.youtube.com/watch?v=_uNkubEHfQU](https://www.youtube.com/watch?v=_uNkubEHfQU)

> Driving cars

I'm not so sure about that one either.

I think we're in the valley that AI can do a lot of things, but are hitting limits in accuracy where humans are still better at some of these things sometimes. That is, the AI isn't always better than humans, even at a specific, non-general task.

Now don't get me wrong, we've made a lot of progress, but I wonder if we can get these things to a place better than humans before the next economic recession. I think the biggest risk to AI is having the money dry up. Right now the hype is strong and the money is (nearly) free. If one of those changes, we could put this back on the shelf for another decade. If we go into a recession, labor will be cheap, so why bother automating with AI?

For example, we had self driving freeway cars back in the 90's.[0][1] Here's one of the lessons learned:

> In 1987, some UK Universities expressed concern that the industrial focus on > the project neglected import traffic safety issues such as pedestrian > protection.

And who doesn't remember the brilliant Dr. Sbaitso, my childhood therapist. [2]

[0] [https://en.wikipedia.org/wiki/Eureka_Prometheus_Project](https://en.wikipedia.org/wiki/Eureka_Prometheus_Project) [1] [https://www.youtube.com/watch?v=I39sxwYKlEE](https://www.youtube.com/watch?v=I39sxwYKlEE) [2] [https://en.wikipedia.org/wiki/Dr._Sbaitso](https://en.wikipedia.org/wiki/Dr._Sbaitso)

<comment><USER6> > Now don't get me wrong, we've made a lot of progress, but I wonder if we can > get these things to a place better than humans before the next economic > recession.

I don't think it's necessary to completely solve superhuman performance to achieve automation of great economic value. Some of the most famous AI achievements leverage a fairly modest intelligence improvement with massive amounts of classic automation. E.g., the AlphaGo policy/value networks, combined with MCTS.

It may also be possible for automation to reliably determine when it's encountering a situation it's going to fare badly in, and then hand off to human telepresence control. It wouldn't surprise me if the first self-driving systems worked that way.

<comment><USER7> > and then hand off to human telepresence control. It wouldn't surprise me if > the first self-driving systems worked that way.

The issue is that humans fare badly in situations where they don't have to pay attention until they suddenly have a short time to react before disaster, and that's exactly the problem that "supervised" self-driving systems have been suffering from when they crash.

And if you have to pay constant attention to the road anyway, I'd rather be driving myself. What's going to win won't be self-driving tech, it will be driver assist, like advanced anti-lock brakes, or self-adjusting cruise control.

</comment></comment></comment><comment><USER8> No. Maybe an "AI Fall", but I doubt there will ever be another true "AI Winter". The AI we have today is too good, and creates too much value... at this point, there is no longer any question as to whether or not there is value in continuing to research and invest in AI.

What will happen, almost without doubt, is that particular niches within the overall rubric of "AI" will go in and out of vogue, and investment in particular segments will fluctuate. For example, the steam will run out of the "deep learning revolution" at some point, as people realize that DL alone is not enough to make the leap to systems that employ common sense reasoning, have a grasp of intuitive physics, have an intuitive metaphysics, and have other such attributes that will be needed to come close to approximating human intelligence.

Disclaimer: credit for the observation about "intuitive physics" and "intuitive metaphysics" goes to Melanie Mitchell, via her recent AI Podcast interview with Lex Fridman.

One other observation... while we still don't know how far away AGI is (much less ASI), or even if it's possible, the important thing is that we don't need AGI to do many amazing and valuable things. I also doubt many people are actually all that disillusioned that we aren't yet living in The Matrix (or are we???).

<comment><USER9> We could still have the bottom fall out of the term "AI", since there's a big gap between the present reality - no matter how useful - and the aspirational nature of the phrase "Artificial Intelligence". Take any business that brands itself as an "AI startup", any quote from Mark Zuckerberg about solving Facebook's content problem with AI, etc., and replace "AI" with "statistical algorithms" and it just doesn't have nearly the same ring to it. That alone means we're due for some kind of big correction.

</comment></comment><comment><USER10> Reminder: Most of the seminal accomplishments of this era's AI wave were actually developed in the 70s-90s. Yes, even GANs and RL. This industry has been riding on the NVIDIA welfare program for the past 10 yrs. How long until the hardware gets maxed out?

[http://people.idsia.ch/~juergen/deep-learning-miraculous- yea...](http://people.idsia.ch/~juergen/deep-learning-miraculous- year-1990-1991.html)

<comment><USER11> We need yet another set of big breakthroughs because just adding more computing capacity is not going to carry the boom.

Maybe the correct way to measure advance in AI is Turing Awards.

<comment><USER10> Yeah, my opinion, neural networks alone aren't going to cut it. We need a better primitive.

</comment></comment></comment><comment><USER12> I've been waiting for the hype and marketing to collapse for a few years.

Probably the fastest way to tarnish public perception of AI would be to keep pushing "AI-enhanced" products in front of the consumer as has been done. These things tend to demo well and have a nice cool factor for the first fifteen minutes or so, but after any kind of prolonged usage the limitations and rough-edges come up quick.

<comment><USER13> This is brand new technology. It's going to take a few years to reliably productionize - and most of the applied solutions will look nothing like the research. Many real world problems are going to combine multiple neural nets into systems with specific applications and there's a lot of detail to work out.

The hype may collapse in the short term, but that's only because many of the first movers are stereotypical tech startups who overpromise without truly understanding the problem or solution spaces and therefore underdeliver.

But, speaking from personal experience, some of the tech has already been proven - one example is massively accelerated modeling as an alternative to slow finite difference/finite element simulation with 99% accuracy, which will in the next 6-12 months totally change the approach to a wide range of modeling problems, and enable a totally new form of work where instead of setting up a model and waiting days or weeks, one may iterate effectively in real time. There are emerging solutions to knowledge management and "intelligent" data harvesting, where ML outputs are being manipulated in a rudimentary form of reasoning. Think specialized industries like petroleum, mechanical engineering, EM engineering - plenty of "layman" related features like recommendation engines are going to flop, but the cat is out of the bag for heavy industrial knowledge work. Just give it some time - we are on the cusp of a monumental leap in R&D across the spectrum of human endeavor. Very exciting times.

<comment><USER14> I used to do electromagnetic modeling using finite element methods (though now a product manager for AI software infra) and it would to take me on the order of hours to days or weeks to model wave interaction with real-world objects.

A machine learning model trained to understand Maxwell's Equations can in principle be used perform said simulations, resulting in probably an order or more of magnitude increase in simulation speed. Getting this to work well will reduce the time (and cost) it takes to design optical sensors, radar for autonomous vehicles, smartphone antennas, MRI machines, and more.

Having said that, it would require a lot of heaving lifting to pull this off to achieve near-physical accuracy for real-world physics problems.

A cursory search on Google for "arxiv deep learning electromagnetics" returns results of proofs of concept in this direction.

<comment><USER15> Were would the speedup come from? I don't understand.

If I understand your comment correctly, essentially you have a hand-crafted simulator for some physical process and then you train a neural net model to approximate the simulator. Why would the approximated simulator have "an order or more of magnitude increase in simulation speed"? Unless the approximation has massive losses in accuracy, of course.

Honestly asking and really interested to know what you mean.

<comment><USER13> It's all about precision heuristics, derived from joint probabilities of inputs and outputs. That, by and large, is how I am increasingly coming to understand the power of neural networks.

Imagine you are given a picture of a candle, overlaid with a grid, and asked to fill in, with colored pencils, colors for the air surrounding the candle representing relative temperature. Of course a human utilizes intuition to rapidly assign high temperature to the flame and decreasing temperature with increasing distance.

A "dumb" finite method would need, even for such a relatively simple problem (for a human), to perform calculations for a series of time steps in each grid until some steady state condition to arrive at a much more precise but still overall similar coloring of the grid cells. You can do the same task much more quickly because you have developed intuition of the physics, which is to say you have learned heuristics which capture the general trends of the problem (air is hot close to a flame and cold far away).

Neural nets take the best of both worlds - by effectively learning probability relationships between input and output pixels, they internalize heuristic approaches to produce outputs approaching finite method accuracies at a fraction of the computation. There's a lot of waste that can be optimized out of finite computation by hardcoding rules (heuristics), but doing so for real problems is impractical. Neural nets learn these rules through training - <AUTHOR> far simpler task is organizing the data to teach the net the right trends; much like designing lessons for a child to teach a predictive ability.

<comment><USER15> I'm skeptical of the claim that it's easier to train a neural net than to hand-code a set of heuristics _when the heuristics are already known_. For the time being, optimal results with neural nets need more data and more computing power ("more" because it's never enough) and are primarily useful when a hand- coded solution is not possible.

I also don't understand how it is possible for a neural net (or any approximator, really) to approximate a "precision heuristic" faster than a hand-coded heuristic and without a gross loss of well, precision in the order that would make the results unusable for engineering or scientific tasks. Could you elaborate?

<comment><USER16> I'm also skeptical, but after reading the explanation above, I am intrigued.

Say I have a cube with 100 x 100 x 100 mesh cells inside, and ports on opposing faces. Given enough time, I can literally run through every possible combination of PEC and air for every cell and solve the FD form of maxwells equations, then save the results. Now, a user can ask my solver for any of those cases, and I simply pull the presolved result, and give the user the answer with orders of magnitude reduction in time.

Obviously, the presolving approach doesn't scale. More materials, more mesh cells, eventually it is impractical to presolve every case. But the beauty of neural networks is that they can be very good at generalizing from a partial sample of the problem space. In effect, they can give results close enough to the presolve solution with drastically reduced numbers of computations.

<comment><USER15> >> But the beauty of neural networks is that they can be very good at generalizing from a partial sample of the problem space.

That is really not the case. Neural nets generalise very poorly, hence the need for ever larger amounts of data: to overcome their lack of generalisation by attempting to cover as many "cases" as possible.

Edit: when this subject comes up I cite the following article, by François Chollet, maintainer of Keras:

 _The limitations of deep learning_

[https://blog.keras.io/the-limitations-of-deep- learning.html](https://blog.keras.io/the-limitations-of-deep-learning.html)

I quote from the article:

 _This stands in sharp contrast with what deep nets do, which I would call "local generalization": the mapping from inputs to outputs performed by deep nets quickly stops making sense if new inputs differ even slightly from what they saw at training time. Consider, for instance, the problem of learning the appropriate launch parameters to get a rocket to land on the moon. If you were to use a deep net for this task, whether training using supervised learning or reinforcement learning, you would need to feed it with thousands or even millions of launch trials, i.e. you would need to expose it to a dense sampling of the input space, in order to learn a reliable mapping from input space to output space._

<comment><USER13> Well...I think that take is a little overly cynical, and I disagree particularly with this:

>the mapping from inputs to outputs performed by deep nets quickly stops making sense if new inputs differ even slightly from what they saw at training time

In my experience that isn't really true, if you have an appropriately designed net, training data which appropriately samples the problem space, and the net is not overtrained (overfit).

You can think of training data as representing points in high dimensional space. Like any interpolation problem, if you sample the space with the right density, you can get accurate interpolation results - and neural nets have another huge advantage, in that they learn highly nonlinear interpolation in these high d spaces. So the net may be unlikely to generalize to points outside of the sampled space - although now that I think of it I'm not sure of how nets handle extrapolation - but when you're dealing with space with thousands of dimensions (like each pixel in an image) you can still derive a ton of utility from the interpolation which effectively replaces hardcoded rules about the problem you're solving.

<comment><USER15> I may be jumping the gun a little because I was thinking about this in the context of another thread, but a practical problem with machine learning in general is that, for a learned model to generalise well to unseen data, the training dataset (all the data that you have available, regardless of how you partition it to training, testing and validation) must be drawn from the same distribution as the "real world" data.

The actual problem is that this is very difficult, if not impossible, to know before training begins. Most of the time, the best that can be achieved is to train a model on whatever data you have and then painstakingly test it at length and at some cost, on the real-world inputs the trained model has to operate on.

Basically, it's very hard to know your sampling error.

Regarding interpolation and dense sampling etc, the larger the dimensionality of the problem the harder it gets to ensure your data is "dense", let alone that it covers an adequate region of the instance space. For example, the pixels in one image are a tiny, tiny subset of all pixels in all possible images- which is what you really want to represent. Come to that, the pixels in many hundred thousands of images are still a tiny, tiny subset of all pixels in all possible images. I find Chollet's criticism not cynical, but pragmatic and very useful. It's important to understand the limitations of whatever tool you're using.

>> although now that I think of it I'm not sure of how nets handle extrapolation

They don't. It's the gradient optimisation. Gets stuck to local minima, always has, always will. Maybe a new training method will come along at some point. Until then don't expect exrapolation.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER17> In the sense of AGI, it's _all_ been hype. We are in an ML summer and have been for the past few years.

But "deep learning" is nothing more than that, nothing to do with AGI, we're not approaching an AGI winter except for people who were daft enough to fall for the hype.

There have been no advances in AGI in decades, it's already winter, and we've _long_ been in it.

<comment><USER10> > it's already winter, and we've long been in it

In terms of research and innovation, yes, but in economic terms, it has not even begun. There is still huuuge VC and government money being pumped into anything with AI on it. The last AI winter started when the financiers discovered the disconnect between the money they put in and delivery on promises.

AI went from an obscure hard CS field that only a few graybeards at MIT knew anything about, to this worldwide meme. Before the default thing your grandmother would tell you to study in college was business. Now your grandmother would tell you to study AI. I'm seeing a lot of people enter this space with the vague goal of getting rich quick. This is the same cohort that jumped into tech in the late 90s and the real estate market in the mid-2000s. It's not the AI of the Norvig and Marvin Minsky days.

I couldn't be more bearish about AI. I still love it though. I won't stop studying it when it becomes not cool anymore.

<comment><USER17> Agree, me too. It's one of the most fascinating problems in the world, and I'll be delighted when "they" decide it's dead and the spotlight moves on.

</comment></comment></comment><comment><USER18> I have worked in the field since 1982, so I have experienced "the need to work on other things for a while" to earn a living.

My prediction is that we are going to see a small revolution in cost reduction: hardware for deep learning will get cheaper; great educational materials like fast.ai and Andrew Ng's lessons will increase the hiring pool of people who know enough to be useful; the large AI companies will continue to share technology and trained models to help their hiring funnel and general PR; programmer less modeling will really start to be a real thing.

<comment><USER19> Alot of the cost in AI projects now are in training or education, but instead in problem solving and plumbing. AI/ML free projects using things like Kafka and Flink are not cheap.

Coding up a CNN or MLP is not a big deal, but it never really was - it was work to build a c back propagation implementation but if I did it in 1995 then anyone could. The question and real differentiator is in answering three problems :

\- what's the problem? \- how can we get the data to the system? \- how do we frame the data and output in terms of (any) AI technology?

All of these steps are closely coupled and require expertise.

On the programmer less modelling; I still have not seen a tool that is better than code for expressing a model precisely and testably, and my experience is that until we have some running code we don't really know that we understand the system.

</comment></comment><comment><USER20> The AI summer/winter cycle feels to me like something similar to a search algorithm, we have a phase of exploration, which seem to have slower, if any, progress and no one is very sure what's the next big thing so they perhaps start trying many things (the winter "skepticism") and eventually someone finds some breakthrough and gets everyone to an exploitation phase in which everyone knows where to invest and comparatively small effort is required to create progress (the summer "hype"). And eventually all low-hanging fruits are over and the search seems to converge to a local maximum and again larger exploration is required.

So maybe the winter is just as important as the summer. Each winter lead to a summer with different focus points (specialist systems and logic followed by neural networks, bayesian models and SVMs and finally deep learning). And after each cycle we have more and more tools, each more useful than the last. And also maybe the key to avoid this strict cycle would be to encourage more exploration during the exploitation phase, giving full support to both incremental ideals that improve on the state of the art and (potentially) revolutionary ideas that give poor immediate results but create new venues to investigate.

Of course that's a simplification and there are many aspects to it, including data availability, hardware and tooling that can easily prevent brilliant ideas that were had too soon.

</comment><comment><USER21> Can't comment on every industry, but in medicine - especially the 'pattern- recognition-specialties' such as formost pathology and radiology - the actual implementation/usefulness/impact of "AI" (ML/DL) has not yet taken a foothold.

Yes it's hyped, but the match between even the current state of DL and what is needed and possible in these specialties is so close to being perfect, and the gain is so close. What is holding us back is regulatory issues and technical implementation issues that have nothing to with the state of DL, just basic IT problems, lack of standards.

Investments may fall back and companies may stop adversing it as "AI", but the impact of ML/DL in medicine will not fall back.

The "AI" we see today is already effective, just not applied at scale.

<comment><USER22> There is a business opportunity here.

<comment><USER21> Oh yes - and it's seeing massive investments. Last RSNA - largest Radiology conference - saw >100 AI startups showcasing their products. (half of which will probably be gone by December this year, bankrupt, taken over or merged)

</comment></comment></comment><comment><USER16> Why would there be an AI winter? Was there a car winter after cars became a growing product? Was there a processor winter after microprocessors became a growing product? ERP software?

Didn't the previous AI winter happen because the hardware wasn't advanced enough to make the technology useful to most people? Since that is no longer the case, why this consistent belief that there will be another winter?

<comment><USER23> 2001 wasn't exactly great for the semiconductor industry...

<comment><USER16> Sure, there will be ups and downs like any industry. But a "winter" implies a decade at least without substantial industry or research, not just a bad year.

</comment></comment></comment><comment><USER24> Research progress might slow down for a bit in some areas of machine learning but the commercialization of existing technology will keep us busy for the next 10 years.

Unlike the past two winters, deep learning is actually enabling a ton of applications that wouldn't have been possible otherwise and we now live in a world with a lot more data and computers to apply it to.

<comment><USER21> Exactly this

</comment></comment><comment><USER25> We don't want AI, we want systems that work better autonomously. We have lots of autonomous systems, mostly run by people (a shop keeper for a shop owner, for instance). Now that we have reached certain limits of pure digital systems, more innovations (ie, changes leading to better system outcomes) will happen due to human involvement in the data understanding and automation. It's just going to look more like people going to work.

The idea that AI (ML models) would be designed once is silly. The tuning and application always involves human judgment over time. We just hide the human contributions to AI/ML systems because it gets too complicated. But really, all good/practicable/in-the-wild AI systems involve a lot of people-in-the- loop!

</comment><comment><USER26> IMO, no. Unlike the last time, things actually work this time. Perceptual things especially. People in the thread seem to be dismissive of those "single digit percentage point" gains that are being made nearly every month in some important tasks, but those last few percentage points often decide whether the system is garbage or useful. Compare e.g. Siri and Google Assistant, for example. Likely relatively small difference on metrics which results in a _huge_ difference in usability.

Another mistake people make is they look at model performance on academic dataset and make unsubstantiated conclusions about usefulness of models. Guess what, practical tasks _do not_ involve academic datasets. Some of academic datasets are _stupid hard_ on purpose (e.g. ImageNet, which forces your net to recognize _dog breeds_ that few humans can recognize). If your problem is more constrained, and the dataset is large enough and clean enough, you can often get very good results on practical problems, even with models that do not do all that well in published research.

</comment><comment><USER27> I actually think that basic deep learning is well on its way into the plateau of productivity. It's not going to be used strictly as AI though, just a more robust type of model fitting than traditional ML which required cleaner data and better extracted features.

</comment><comment><USER28> While the expectation vs. reality dichotomy is very real, the cost vs. return is just as vital and, ultimately, more easily solvable in the years to come. Curbing the expectations of the money-givers in regards to what they might get out of these ventures is always tough but using tech is going to be cheaper because, well, the price trends for tech have been downward for a while.

Personally, hoping to see more shifts toward trying new things rather than attempting to perfect the already existing models. This would, well, not solve but circumvent the need to try and improve something when the tools are not there yet. This way, a broader groundwork will be laid.

</comment><comment><USER29> I heard that the top tech research labs are already experiencing cutbacks and hiring freezes. Can anyone confirm?

</comment><comment><USER30> I think the total economic impact of AI will be greatest for tasks that output high-dimensional data, such as GANs. For the simple reason that it can replace a lot more human labor. A great many jobs could be augmented with such tech.

Furthermore, I think the results from GPT-2 and similar language models show that researchers have found a scalable technique for sequence understanding. They are likely to just work better and better as you throw more data and training time at them. Imagine what GPT-2 could do if trained on 1000x more data and had 1000x more parameters. It would probably show deep understanding in a great variety of ideas and if prompted properly would probably pass a lot of Turing tests. There is evidence that this type of model learns somewhat generally, that is, structures it learns in one domain do help it learn faster in other domains. I am not sure exactly what would be possible with such a model, but I suspect it would be extremely impressive and meaningful economically.

I think we are likely to see that type of progress in the next year or two, and for there to be no AI winter.

<comment><USER31> Does GPT-2 really "understand" anything? I feel like this is pretty quickly going to devolve into a semantic argument, but having interacted with some trained GPT-2 models, it seems to produce only what Orwell would have called duckspeak[0]. There's very clearly no mind behind the words, so it's hard for me to credit it with understanding.

[0] [http://www.orwelltoday.com/duckspeak.shtml](http://www.orwelltoday.com/duckspeak.shtml)

<comment><USER30> I think the only time a system can be truly be said to understand something is when its answers are derived from logic (such as old school symbolic AI). No matter how good current statistical approaches get, they won't meet that bar.

However, I do believe we see evidence of approximate logical reasoning in these models, as well as the concept of abstraction.

Furthermore we can take statements generated with statistical techniques and validate them mechanically with older techniques. This is basically what recent work in automated theorem proving using deep learning is about.

Generating logical statements using heuristics and then validating them mechanically also sounds like a reasonable approximation of what a human often does, speaking as a human.

<comment><USER31> > Generating logical statements using heuristics and then validating them > mechanically also sounds like a reasonable approximation of what a human > often does, speaking as a human.

I think I agree with that, but I might add that humans who understand a topic well can also make novel connections and uncover further implications that might seem illogical at first glance. This process of "insight" seems poorly understood by everyone, but I think it goes beyond validating heuristic intuition.

</comment></comment></comment></comment><comment><USER32> AI winter came because the best in-market was Clippy and Naive Bayes spam detectors.

</comment><comment><USER33> Perhaps Artificial Intelligence needs to be rebranded?

Maybe call it:

Cybernetic Research (CR)

Computational Cognition (CC)

Statistical Reasoning (SR)

Computational Reasoning (CR)

</comment><comment><USER34> There might be another A(G)I winter, but there won't be an ML winter...

</comment><comment><USER13> I'm not sure about the pace of progress in research, but as an ML engineer at a startup who has been following developments, even if AI research stalls out completely, we've been given a huge set of amazing tools to apply to all kinds of technical problems for years to come.

I also think that even in the absence of massive breakthroughs, there's still plenty of work to be done during a "winter" in filling in the gaps of understanding between various SOTA advances. I think it may be the nature of science to advance in a sort of unbalanced tree of breakthroughs, where we drill down on certain popular and lucrative branches for a while before coming back to fill out and balance the width of the tree, if that analogy makes sense.

Just between transformers/autoencoders, GANs, classic classifiers, and combinations thereof I think we are already poised to see neural networks change society in the next ten or so years in a way similar to the influence of the internet. Especially if hardware and cloud computing continues to scale.

<comment><USER35> Can you give some examples of applications that you think will have big impacts? I see places where current AI techniques can make incremental improvements but I just don't see any applications that really seem game changing. The ones that come closest tend to be dystopian unfortunately, like most applications of facial recognition.

<comment><USER36> My startup is using AI in many forms in order to build an accurate digital twin of the world cheaply, and extract valuable insights from it.

In a few years we will have an accurate digital twin of the world, almost indistinguishable from the real world. this would have been impossible or way too expensive without massive automation with AI

<comment><USER37> > In a few years we will have an accurate digital twin of the world

No, you will not, unless you redefine what "accurate" means.

<comment><USER36> Google has 3D models of cities nowadays. 20years ago we only had 2D maps. Why do you think this trend will not continue?

<comment><USER37> Google Street View is full of artifacts, it's not even close to being accurate. The same goes for satellite imagery of rugged mountains. I'm not even mentioning the vegetation, snow cover, river levels, etc.

<comment><USER36> Accuracy is relative to the need. By accurate I mean accurate enough that we can extract actionable information from it.

For power plant critical structures we want 0.5mm, updated every 6months. For forest management we want 5m, updated every 2 years.

But this increase in spatial and temporal accuracy will keep on going. At some point in the future a small swarm of insect-sized drones will be able to capture a whole forest in a day for a super low cost. And a few people walking with basic smartphones for will be enough to map a whole city.

In 1980 you would have said that google maps 3D and google street view would never exist...

<comment><USER37> Well, you just redefined accurate to "accurate enough that we can extract actionable information from it". Your statement is true now.

BTW, I never say never.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER38> never use the phrase "AI Winter" again. never.</comment>


Insurance industry steps up to help airBnB renters - <AUTHOR> http://www.airbnbinsurance.com

<comment><USER2> This isn't the "insurance industry stepping up". This is an entrepreneur doing some market testing.

Yesterday, someone registered this domain name anonymously, and is now collecting email addresses to gauge market interest. fishcakes is an account that is a few weeks old, and submitting this article was hir only participation on this site so far.

I'm all in favor of lean startup metholodogies. Just call a spade a space, and please edit the title.

</comment><comment><USER3> I don't think this is going to work very well via a third party. This is something Airbnb has to offer themselves.

</comment><comment><USER4> I wouldn't trust a company to cover my insurance needs if I can't trust them to properly license the name of the brand they're leveraging.</comment>


Mailin – Receive inbound emails in your web app with Node.js - <AUTHOR> http://mailin.io/ <comment><USER2> This project looks like a wrapper over andris9/simplesmtp, and should be advertised as such instead of a smtp implementation. Check out [https://github.com/baudehlo/Haraka](https://github.com/baudehlo/Haraka) for a mature smtp project instead of a wrapper.

<comment><USER3> I'll second the vote for Haraka. Good support via IRC, and a very strong, accessible plugin architecture. I like the basic mechanism Mailin is providing (convert e-mail to a semi-structured doc) - I just don't know their smtp wrapper's reliability. It's tempting to just build a Haraka plugin to provide a similar output (and feed it to Elasticsearch for logging/searching later on).

Reading through the source it looks like it spawns a child_process per-message to for checking both SPF and DKIM via a python script... That feels inefficient, but I don't know if there's a better node-based solution.

<comment><USER4> The main objective of mailin is providing the service email-to-webhook with a very easy setup. You're right, for the spawned child processes, the correct way to do it while still using python would be to have some running daemons to connect to.

</comment></comment></comment><comment><USER5> This is really great. Plus I love the attention to detail in the documentation, and how the demo worked right away. We currently use Mailgun for this but I would love to switch over to using Mailin and host it ourself.

</comment><comment><USER6> "language": "hawaiian" ?

<comment><USER7> Language detection isn't something I would rely on. Especially since a lot of emails don't contain much text. (Likely even less so with your test message.) Looks like they're using the languagedetect[1] module which also returns a confidence score. This would be a helpful thing to include in the results.

[1] [https://github.com/FGRibreau/node-language- detect](https://github.com/FGRibreau/node-language-detect)</comment></comment>