McClatchy files bankruptcy - <AUTHOR> https://www.mcclatchydc.com/news/nation-world/national/article240139933.html <comment><USER3> I discovered the concept of pensions managed by corporations when Sears Canada went under and a ton of people lost their pensions. I guess it's a pretty common thing. But it seems absolutely insane to me.

Who in their right mind, given any choice on the matter, would want to lock their retirement into the existence and health of a corporation?

Until I saw the Sears debacle occur, I thought everyone basically puts together RRSP (Canadian retirement tax havens, I think 401k?) accounts and rely on their required payments into the Canada Pension Plan. It's much less likely that the government or a bank will fail, and the bank, unlike Sears, is CDIC backed.

I guess my question is: why do we allow companies to manage retirement funds?

<comment><USER4> The one thing that made them "make sense" to most people was that the average person is/was terrible at personal finance and retirement planning. The company running the pension was a way for the average person to be "guaranteed" a retirement income without having to worry about anything.

Remember, corporate pensions were a big thing when the US was at it's greatest economic strength (post-WW2) and the internet with all of its modern tools to make investing much simpler for the average person didn't exist. You had to pay brokers for every single trade or someone else to advise you or you had to just buy and hold a small set of stocks for 30 years and hope those companies would still be around (a fair bet at that time).

Back then companies didn't really go under like this. If anything, they got bought by a larger company and merged, but the frequency of that was nothing like today.

Today, with health care costs what they are, having pensions and defined benefit plans are insane to me. It's much better for the company to say "we will match x% in a 401k". They are defining their contribution and it has a cap. If anything happens to the company, the employee's funds are safe as the money is already transferred. For the employee, it does require them to manage this money but that's far easier and cheaper today and that money doesn't require the company to be around forever. If things tank, the employee is protected.

<comment><USER3> Thanks. That's all really quite educational. Does the U.S. have any social program like Canada Pension Plan where:

1\. your paycheque is forcibly deducted an amount based on a formula

2\. that amount goes into CPP investment

3\. when you retire you can get a monthly pension based on the total amount you paid in

To me that seems like a sane way to protect people from themselves.

<comment><USER4> As others have mentioned, this is referred to as Social Security. I'm not sure how it works in Canada once it's is forcibly deducted, but in the US, it's just a tax like any other tax. The idea is that the government will honor the promises made about the schedule of payments based on a complex formula - <USER2> basically it goes by the year you were born and the amount of money you have made over time. People who make less get a larger % of their earnings back from Social Security. The higher earners make more, but there is a cap. Similar to income taxes, there are steps in which you get back varying %'s, which is why those with lower income get more as a portion of their earnings.

In this way, the government simply collects money from current earners and then distributes that to the existing retirees. It's not like there is a trust set up and your money is locked away with your name on it. Canada's system may or may not be structured the same way. The idea is that if the government defaults on this, then there are much larger issues anyway. But because it's just a _tax_ they can technically do whatever they want with the money and being the scumbags that they are, they periodically do steal from that money, which is why Social Security is underfunded.

Personally, I think the payouts are even pretty respectable, despite what others might think. The average payout is currently around $1,500 per person per month. That's not a lot, but if you have a spouse that's $36,000 a year. Also, one thing that most forget is that even if your spouse doesn't work their entire life (i.e. stay at home parent), a spouse can claim an _additional_ 50% of their spouses as their own. Every person who is married has this option - they can claim their own or the 50% of their spouses (which does _not_ reduce their spouses, it's just on top). For anyone working, it usually makes sense to claim their own unless there is a massive difference in pay. So if one person is scheduled for 2k a month, the other is guaranteed at least 1k or their own if it's higher.

Honestly, it's a nice idea and has been working reasonably well for decades. But the issue is that when you look at it with a technical eye, you'll see it's all basically the same structure as a ponzi scheme. The current people taxed don't pay for themselves, they pay for others who came before them. Over time the number of people required to pay for a single retiree has gone up. This indicates a growth requirement in the tax base. Without this constant growth which seems to go up more and more over time, the system fails. People ignore this because they think it's impossible because it's run by government. I'm not so sure that's true.

<comment><USER3> Thanks so much for sharing this detail.

With regards to "it's like a ponzi scheme". I completely see what you're saying. But shouldn't it look more like a population pyramid? Where, yes, the word pyramid is in the name, but generally a mature nation won't be top-heavy (except for the baby Boomer issue but that's kind of transient, no?)

My sense is that it should work perpetually so long as a nation doesn't see massive population decline and employment remains reasonably consistent. And in years/decades when there are imbalances, the government can step in and prop it up through other means rather than watching a private corporation file bankruptcy.

I feel like maybe we're judging the long-term efficacy of such a system too early. Right now we're dealing with a one-two punch: baby boomers are retiring and people are living longer. But neither of those are going to increase perpetually.

<comment><USER5> Civilization is a ponzi scheme if you step back far enough, but it's not a useful way to model it. For a more accurate understanding of the way Social Security is structured, I would check these links. The second link is to an interview where it's explained in simple english.

[https://www.ssa.gov/policy/docs/chartbooks/fast_facts/2019/f...](https://www.ssa.gov/policy/docs/chartbooks/fast_facts/2019/fast_facts19.html#page35)

[https://us14.campaign- archive.com/?u=bd0d1b66f832083794c33c9...](https://us14.campaign- archive.com/?u=bd0d1b66f832083794c33c979&id=d540b66d82)

I also have not found a source to substantiate the claim that the US government periodically steals money from Social Security.

<comment><USER4> See my other comment(s) in this thread. I have acknowledged that was not in fact true after checking further once called out on it.

<comment><USER5> Yes, I hadn't seen that yet. Thanks for acknowledging! I learned the same way when I was researching it too.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> > While we tried hard to avoid this step, there's no question that the scale > of our 75-year-old pension plan – with 10 pensioners for every single active > employee

Holy crap. I remember a person from a newspaper told me it was 3-1 for them, and I thought it was hyperbole.

<comment><USER7> It's insane to me that private businesses are strapped with the burden of running the country's pension and healthcare systems. How on earth the US has managed to remain economically competitive under those circumstances is a miracle.

These functions really need to be divorced from one's employer, and either run independently via private industry, or via government. Especially in modern times when people job and career hop frequently.

<comment><USER8> > How on earth the US has managed to remain economically competitive

Well after you're done working, you don't work anymore. So if a country spends zero on providing pensions, they are actually more economically competitive.

Whether one would want to live there is another matter, but the lower income people have little money to move and build a life elsewhere and the higher income people can arrange a good future for themselves.

<comment><USER9> Losing their income has a greater economic impact than just their own wellbeing. Pensioners might be retired but they are still consumers with debts of their own. Homelessness alone is very costly to the U.S.

<comment><USER10> Pensioners are window-breakers, in today's society. They consume, but never produce. (This used to be less true, with grandparents providing childcare etc) This doesn't mean we should kill them all off! But the consumption of a pensioner (or retiree in general) is more akin to digging a ditch and filling it back in than it is to anything productive.

You can't build an economy on consumption alone.

</comment></comment></comment></comment></comment><comment><USER11> This has been a long time coming. In the mid aughts I worked at the Sacramento Bee, which is also the HQ for McClatchy Corporate. Every financial quarter the CEO would go into the newsroom and break the news of how print ad revenue was falling, but digital ads were looking promising, with double digit percentage increases, and that someday soon would make up the shortfall (most everyone in the industry knew, without knowing the denominators, how unlikely that was).

What really screwed McClatchy over was that in 2006, it bought Knight-Ridder – then the 2nd biggest chain – for $4.5 billion [0]. Historically speaking, it wasn't the worst deal on the face of it. But in retrospect, McClatchy bought when the price was highest, though it managed to quickly sell off a dozen papers shortly after for not terrible prices [1].

So along with the declining ad revenue, the CEO would also talk about the massive debt he had just acquired, and the amount of interest we were paying each quarter, but how, once this "secular downturn" got stabilized, the new acquisitions would start paying for themselves. I left the Bee before the first big layoffs hit, but I remember people being mostly "whatever" at the time and putting it out of sight and out of mind. As a regular rank-and-file person, not much you can do or fuss about when the CEO is making billion dollar deals.

[0] [https://www.npr.org/templates/story/story.php?storyId=526041...](https://www.npr.org/templates/story/story.php?storyId=5260417)

[1] [https://www.nytimes.com/2006/06/08/business/media/mcclatchy-...](https://www.nytimes.com/2006/06/08/business/media/mcclatchy- sells-5-papers-bought-from-knight-ridder.html)

</comment><comment><USER12> Dig deep into the article to understand that, while this was caused by burdensome pension obligations, politics was heavily at play (and federal legislation [Secure Act] spared other journalistic publications with similar pension obligations).

<comment><USER13> Pensions are going to cause enormous issues in the coming decades. Almost certainly taxpayers won't stomach paying for them, as happened in Detroit and smaller jurisdictions. Get ready for the implosion!

<comment> jagged-chisel How common are pensions these days? None of my employers have offered pensions, just a 401k. Current employer is over a century old; seems they got savvy to the pension liability long ago.

<comment><USER14> > None of my employers have offered pensions, just a 401k.

A 401k is a pension isn't it? Just a defined-contribution pension instead of a defined-benefit pension.

I get a pension at my large young tech company.

<comment><USER15> No, a 401k is not a pension. A pension guarantees a certain amount at specified dates while a 401k is a way of purchasing securities for you to manage your own retirement and is subject to the returns of your investment choices. In other words, a pension is a guarantee of future income in which the pension offerer is on the hook to ensure they have the funds to deliver the pension while the 401k pushes all that responsibility on to you.

My wording is probably off on some of that so an article like this one is probably helpful to flesh out the details:

[https://www.fool.com/retirement/2018/06/13/whats-the- differe...](https://www.fool.com/retirement/2018/06/13/whats-the-difference- between-a-pension-and-a-401k.aspx)

<comment><USER14> > A pension guarantees a certain amount at specified dates

That's specifically a _defined benefit_ pension - just one type of pension. There are many other types of pension. My pension is basically just a big saving account my employer and I both pay into - it's a _defined contribution_ pension.

</comment></comment></comment></comment></comment></comment><comment><USER16> There are financial costs to not having good local news:

[https://www.citylab.com/equity/2018/05/study-when-local- news...](https://www.citylab.com/equity/2018/05/study-when-local-newspaper- close-city-bond-finances-suffer/561422/)

</comment><comment><USER17> Love how they write an article about their own company's bankruptcy, complaining about pensioners and discusses getting out from under pension obligations as a purely good thing, and never once mentions the economic impact on those pensioners.

</comment><comment><USER18> In 2017, Forman's take-home pay from McClatchy was $1.7 million, excluding restricted stock. His newest contract with the company, dated January 25, 2019, includes a base pay of $1 million, a bonus of $1 million, and an additional $35,000 monthly stipend. According to Segal, this stipend will be used to pay for Forman's travel, housing, office, and security expenses. This monthly stipend alone, which is up from $5,000 in his previous contract, could fund several reporters' salaries every year.

"The CEO's compensation is set by the board of directors using public consultants and comparable," Forman tells CJR over the phone.

</comment><comment><USER19> This is a surprisingly informative article, given that it was published by McClatchy itself. Nice to see that journalistic standards stay high, even in the middle of what must be a harrowing time for everyone there.

</comment><comment><USER20> Wait, is this good or bad, I would hope this means we end up with more LOCAL news, buy who knows

<comment><USER21> Local news is being hurt most of all. Statehouse reporters are down across the country according to PEW.

==Those papers lost a total of 164 full-time statehouse reporters—a decline of 35%—between 2003 and 2014.==

[https://www.journalism.org/2014/07/10/americas-shifting- stat...](https://www.journalism.org/2014/07/10/americas-shifting-statehouse- press/)

</comment></comment><comment><USER22> Once again, we see why an interest and inflation based economy is unsustainable. It's scary to think about what is going to happen in the future when it all implodes.

</comment><comment><USER23> I'm glad I dumped my shares before they wind up being canceled. They only depreciated to being worth a sum total of USD$5.00, alas. Better than getting nothing.

</comment><comment><USER24> Stanford Public Pension tracker

[https://www.pensiontracker.org/](https://www.pensiontracker.org/)

</comment><comment><USER25> They bought Knight Ridder and a lot of that culture permeated particularly in the DC bureau. I hope they come-out of this okay.

</comment><comment><USER26> TL;DR Owner of major newspapers in Miami, Kansas City, Fort Worth, and Sacramento files bankruptcy and prepares for takeover by $4B hedge fund.

</comment><comment><USER27> It feels gross to see pensions listed among liabilities when a company is facing bankruptcy or ruin. Why aren't they protected and held separate from the company's finances?

My (maybe naïve) view is that whatever pension benefits are accrued by an employee, should be paid into an annuity or similar, separate fund that is firewalled off from the assets of the employer.

The choice whether to honor pension obligations or pay back debt holders shouldn't exist. The pension was already funded and is held somewhere else beyond the reach of creditors.

(I'm not saying that the _full_ dollar amount needs to be set aside today. Just an agreed-upon NPV of the future payouts for some standard actuarial retirement length.)

<comment><USER28> A version of what you suggest has been the law since ERISA was passed in 1974.

The problem is what to do if investment performance doesn't meet expectations, lifespan increases, or future assumed yield decreases. Generally companies had to pay enough to be back to even within seven years. Newspapers sought the right to have thirty years to fully fund. McClatchy was larger than Congress was comfortable with and found itself unable to pay the required fraction of the difference between the NPV and finding amount. Hence, it declared bankruptcy.

<comment><USER5> There's huge conflicts of interest with defined benefit pensions. It wasn't until PPA of 2006 that standards really tightened up. But imagine being an employer in 1970s. Instead of paying people higher wages, you could create a pension plan on paper, pay the plan however much you wanted (there was wide discretion on what assumptions could be used to calculate the cost of benefits), and get the recipients of the benefits to work for you today for an unknown benefit tomorrow.

If you're one of the decision makers, you're likely to be on the older side. So you're likely to start receiving the benefits soon anyway, and so any underfunding wouldn't affect you, since there would be a couple decades of money available before it started to run out. So the decision makers can easily choose to shortchange those 20 to 40+ years in the future in exchange for enriching themselves in the now (1970s, 80s, etc).

Seems like society should've seen it coming. Of course, if everyone kept having 4 kids, maybe those fantasy numbers could have been met, but who has the ability to predict numbers decades in the future?

</comment></comment></comment><comment><USER29> Regarding pensions, I never understood why an employer or a government would have an obligation to pay for retirement. It would seem that The onus of saving for retirement would be on the person. Crazy talk, I know.

<comment><USER5> It is on the employee. The employee chooses to accept an employer's promise to pay for retirement, in lieu of cash now (via choosing to work for that employer).

Government on the other hand taxes income in order to pay for retirement and people don't have the ability to opt out, so of course they have an obligation to pay it.</comment></comment>


Python Internals: PyObject - <AUTHOR> http://www.gahcep.com/python-internals-pyobject/ <comment><USER2> Although it's been posted here (only twice, though!) before, Philip Guo put up a 10-hour series of lectures on CPython internals[0]. I am nearly done with them, and have found them very useful and well-paced. Posting here because the author of this article seems to have stopped the series, and anyone interested in this stuff should definitely check out the Guo lectures.

[0]: [http://pgbovine.net/cpython-internals.htm](http://pgbovine.net/cpython- internals.htm)

<comment><USER3> +1 Guo series is excellent. I completed them twice & very confident about my python skills more than ever before :)

<comment><USER4> Thanks everyone! Recording lectures has been great since it doesn't take much extra work than not recording; just takes some time to render the final videos and upload. Ideally I'd have time to split the videos up into smaller chunks, but alas free time is diminishing nowadays.

<comment><USER2> Would you have any interest in providing someone (I would volunteer) the videos so that they can package them in smaller, topic-specific pieces for easier consumption? I don't know anything about editing videos, but if your writings and lectures have taught me anything, it's that I'm capable of learning things and using that knowledge to do interesting and useful work.

As an aside, you are one of my main sources of inspiration. I was a high school dropout with several terms of failing out of community college courses, but after reading your PhD Grind series/book, I'm now a (nearly) straight A student and am very determined to go as far with my education as I can. I would be honored to help you in any small way I can to express my gratitude!

Thanks again for unknowingly improving my life.

</comment></comment></comment></comment><comment><USER5> Contrary to what the article says, PyVarObject is still PyObject at the same time and PyVarObjects have nothing to do with mutability (certainly, it's trivialy obvious that not all PyVarObjects are mutable as immutable tuples are PyVarObjects, while mutable lists are not)

The thing that is handled by PyVarObject are types whose instances can differ in their size, with tuple and bytes objects being probably most common cases when it is used.

</comment><comment><USER6> If you try to embed or extend Python in C you will be exposed to PyObject very quickly. The official Python documentation can walk you through that process (visit the embedding or extending parts). The PyObject structure mentioned in the article also comes with many functions to manage them programatically, cast them to other types, etc.

<comment><USER7> I'm curious if you know much about jobs doing this kind of thing. I've worked a ton with Python, and know a lot about using Cython to write extension modules, wrap C++ code, etc.

I really want a job where I find performance bottlenecks in research or business analytics or financial analytics code, migrate parts to robust Cython implementations, and then build and maintain the broader Python libraries and APIs surrounding that.

But what I'm unfortunately finding is that basically nobody uses this stuff anywhere in practice. I've interviewed at a ton of places doing everything in C++ entirely, and I just see their code and I think, my god, why would anyone choose this? You can write the critical stuff in C++ just as you are, and then so easily wrap it for Python, then write the other 99% of the code in super easy, dynamic Python, even with various kinds of validating systems, or using typing and type annotations to avoid certain kinds of bugs, and just everyone's life is easier.

But they are just so entrenched in the old way of doing it that nobody is willing to try.

I've been consistently amazed at how few jobs in the monthly Who Is Hiring thread are tagged even just with NumPy. I think I've seen maybe one job in 6 months that mentions Cython, and then I found two others outside of HN and the interviews with those places didn't work out.

How to find a job doing this stuff is like my main vexing work problem at the moment. It's so useful, but it's sort of locked outside of some kind of energy barrier that quant teams seem unwilling to cross.

<comment><USER8> Suggest you look for an opening on JP Morgan's Athena project, or Bank of America Merrill Lynch Quartz. Both those platforms are C++ on the inside with Python APIs. The originators have gone independent here: [http://www.wsq.io/about-us/](http://www.wsq.io/about-us/) The site says they're hiring.

<comment><USER9> A few trading companies do this kind of work also.

Since you really want to write C and Python... Do you care if you have to write trading code to help billionaires make more money?

<comment><USER7> I don't think I would be a good match for bank culture or HFT, but if the work you describe is in a small-to-medium sized asset manager or hedge fund, then I would not mind, and would find it enjoyable. I worked at such a place a number of years ago and it was one of the better jobs I've had.

The trouble I find is that a lot of small-to-medium sized asset managers and hedge funds don't have very good technology practices in general, and are pretty far from sophisticated use of low-level Python. There are often a lot of political reasons why they cling to Excel/VBA, R, or MATLAB, and there's almost always some political battle happening between the people that want to rationalize the system to a proper software design, and people who just want to keep cranking on the hodge podge of existing tools.

I'd probably be fine with some trade-off regarding all of that, if the pay was acceptable and there was a strong commitment to a healthy work/life balance. But among finance firms this is extremely hard to find.

So even though I am not at all opposed to doing this work in finance, I still find the volume of acceptable job openings in that field to be too small to make it realistic.

</comment></comment></comment></comment></comment><comment><USER10> Here is blog post about python list implementation, one of 2 main data structures used internally: [http://www.laurentluce.com/posts/python-list- implementation/](http://www.laurentluce.com/posts/python-list-implementation/)

</comment><comment><USER10> Cython can be used to auto-generate optimized Python C-API code that gets source-mapped line by line to original Cython code in the form of HTML annotations. This stuff is very hard to write by hand.</comment>


A Good Man Is Getting Even Harder to Find - <AUTHOR> https://www.wsj.com/articles/a-good-man-is-getting-even-harder-to-find-11570200829?mod=rsswn <comment><USER2> From this article looks like a good man is a man with higher status than the woman, a bit sad.</comment>


My First Year in the App Store - <AUTHOR> http://www.trevormckendrick.com/my-first-year-in-the-app-store/ <comment><USER2> I was actually really excited by the "...you can get the next 9 posts delivered to your email by signing up here:" at the bottom. I like it. I like email, because I can read it anywhere and filter it any way I want. I dislike unsubscribing from email updates about as much as I dislike getting updates I'll never read. By making it easy to get emailed only about a story I've already demonstrated interest in (by reading all the way through), you've made sure I'll come back for your next 9 posts.

I'd like to see more people use something like this.

Or, if I've misunderstood how this works...well, the concept was exciting.

<comment><AUTHOR> Very kind words. Glad you liked it. Indeed, fewer people sign up, but you know they're the ones who value the content the most.

<comment><USER3> Given that you're using wordpress, is that a plugin for the "sign up by email"?

Thanks,

<comment><AUTHOR> It's just HTML that Mailchimp gave me that I stuck in the post.

</comment></comment></comment></comment><comment><USER4> Whenever anyone complains to me that its hard to make money on software, and/or the App Store is rigged, I always tell them I think anyone can make a living by making a bible app. You don't even have to be the best one, just pretty good, and iterate - it's the canonical product that you know has a big audience, willing buyers, and poor competition on the low end of the market.

Case proven by this guy! The next time you think to yourself that there is anything needed to make money on software besides persistence, thoughtfulness, and picking something you know people want, you're in a trap of your own creation.

People who don't make money on their apps typically fail because they try to be too novel, try to make the app too good, or they don't stick in there long enough.

<comment><USER5> Patio11 has argued forcefully that developers systematically neglect women and underserve that large market; I wonder if we can add to that 'and developers also ignore religious markets too'?

<comment><USER6> Really, developers ignore every demographic that they aren't a part of. There's a reason "suits" are often at the head of large creative focused corporations and compromise artistic integrity to make things more palatable to their target markets. Market research and knowledge is a very real thing that product focused developers often completely ignore or rely on instincts of unknown quality.

<comment><USER5> > Really, developers ignore every demographic that they aren't a part of.

Unfortunately, that is not really a meaningful thing to say as a guide to behavior: it's like suggesting that someone sell non-apples - meaningful, not wrong, and useless advice. You need to know what demographics developers _aren't_ before that can be useful; you need to know, say, that developers tend to be male, white/Asian, First Worlders, before you can say 'ah, females may be underserved!'

And you need to think of the categories in the first place, like this religion example shows. If someone asks you if developers might tend to not be fans of religion than the general population, it might seem obvious, but 'being atheists' is not the most salient feature of developers.

</comment></comment></comment></comment><comment><USER7> I'd be interested to hear how you marketed the app. Just building an app seems to be only half (or less) of the battle these days, as far as App Store success goes.

I've got seven apps on the App Store, and combined they usually don't even bring in $200/mo. Discouraging.

<comment><AUTHOR> I'll talk about that in one of the upcoming posts, but basically:

I started with an app niche that I knew people were already searching for. And I knew _that_ because there were crappy Spanish Bible apps already making money in the App Store.

Since then I have started collecting users' email addresses which helps with promotions and new app launches, but initially all my traction was through search.

<comment><USER8> Sounds like any successful business I've ever heard of. Find a niche with an unsatisfied need, then grow organically. Good for you.

<comment><AUTHOR> Thanks!

</comment></comment></comment></comment><comment><USER9> Thanks for writing this. It's very insightful.

One question - do you have any interest in spanish and/or the bible, or is this purely a business exercise?

</comment><comment><USER10> It's a good reminder to me, as a developer, that people simply buy (or use) what they want, not what _I_ think they want.

<comment><AUTHOR> Amen! (no pun intended :) )

</comment></comment><comment><USER11> I'm curious what translation you used (I signed up for the rest of the posts but haven't got through them).

Most modern English translations of the Bible are copyrighted and often you can only quote/print a limited number of verses without paying licensing. (I think the best public domain version is probably the ASV from 1901).

Did you license a spanish version of the Bible? Pay someone to translate a modern one? If so, were there legal hurdles to translating a copyrighted version?

<comment><AUTHOR> You're right that most well known Bible translations are copyrighted. I found one that is in the public domain so it was game on.

</comment></comment><comment><USER12> I thought the App Store had a rule that you couldn't submit an app that was essentially a 'book or a film' how did you get around that?

<comment><AUTHOR> Great point. I've actually run into that problem. On some updates they would reject me and I'd have to add a few features to be approved.

That's part of the reason I'm redoing the entire app from scratch with tons of extra functionality, etc.

</comment></comment><comment><USER13> Fascinating example of finding and successfully filling an under-served niche. I would have never thought of creating this kind of app.

It makes me contemplate - how can you break out of your filter bubble to come up with and test product ideas for markets you can't even imagine exist because they are so far from your experience?

<comment><AUTHOR> My two cents: I scrolled through tons and tons of top ranking apps in each category. That was where the initial brainstorm began: profitable apps that didn't look very good.

</comment></comment><comment><USER14> The coolest part of this post, IMO, is that by finding that niche you freed yourself up to work on whatever you want to work on. Legitimate residual income.

<comment><AUTHOR> Thanks, and yes and no. When I realized I'd found a decent sized niche I had to make a decision: do I let it sit and coast/build something else, or do I invest more here?

I've decided with the latter, which will play out for at least the next year. Jury still out whether it was the right decision.

<comment><USER15> If you completely abandoned the project as it currently exists, do you think your income would drop off sharply?

In other words, how much of an ongoing effort is it to ensure your income stays high? Does search favor apps that are being updated constantly? Or is there any reason not to eventually just stop working on the app?

<comment><AUTHOR> It wouldn't drop off. In fact there's many apps in the App Store that haven't been touched in years that continue to bring in money.

My thoughts are that I have so much to improve upon, and there's a lot of low hanging fruit left for me, so it's likely worth the additional investment.

I compared investing more time/money in this niche, vs in other unvalidated ideas I have and it made sense to stick with this for now.

</comment></comment></comment></comment><comment><USER16> Religious products are big sellers. Specially anything that has to do with the bible or prayers. I used to sell bibles as a teenager. Made more money than drug dealers. It was funny. But then I went ad turned atheist. I could no longer sell itwith a clean conscience.

</comment><comment><USER17> Is there a link to the app on the app store? Or did I miss it?

</comment><comment><AUTHOR> My server is struggling, here's a cached version for the interested:

[http://webcache.googleusercontent.com/search?q=cache:http://...](http://webcache.googleusercontent.com/search?q=cache:http://www.trevormckendrick.com/my- first-year-in-the-app-store/)

</comment><comment><USER18> This sounds very interesting and I'm looking forward to reading more.

How much time did you spend researching potential app categories?

Was your category decision driven by any hard data, other than anecdotal knowledge that Spanish bibles were a category that was selling and had search volume?

<comment><AUTHOR> Great question.

No hard data. I saw that Spanish Bibles were ranking in the Reference and Book categories, like you said.

Also important (but not "hard data") is that when I searched "la biblia" relatively few apps appeared in the search results compared to other search phrases.

</comment></comment><comment><USER19> Great idea, did it cost much to record the Bible as an audiobook?

<comment><AUTHOR> Yes. The most expensive part of the app by far.

<comment><USER20> So the $73k is net revenue after Apple but before costs, correct?

<comment><AUTHOR> That is correct.

</comment></comment></comment></comment><comment><USER21> trevmckendrick - good to see the app do well! If you remember I (Sleepyhead) helped with the email forms. We pleasantly surprised to see the post here :)

<comment><AUTHOR> Hello! You guys did great work; those forms have done perfectly... over 30,000 signups so far.

Thanks for saying hi!

</comment></comment><comment><USER22> Nice work for sure.

One thing that rang true, $73k isn't the sort of money that gets VCs going but that's full time salary for a lot of people. Making money on the app store is definitely possible, but we're no longer seeing the half a million in sales in one month figures we did at the start. Does that mean you can't make living from the App Store? No, it just means being realistic.

</comment><comment><USER23> Very surprised by that type of revenue. What kind of marketing did you put into it?

Without marketing, my first app on the store went as follows:

> The next morning (and literally every day since) I woke up and first thing > checked my email for that magical message from AppFigures. My total day one > net sales? $0.70. Admittedly not very much.

<comment><AUTHOR> As far as I know, all downloads are based on search.

</comment></comment><comment><USER24> Congrats on the success! Can you share a link to your app?

Also, do you think the only way to get discovered (for indie devs) is to pick app ideas that will match nicely to keywords that users search for? If you don't have a marketing budget, is there any other way to get discovered?

<comment><USER25> I'm going to be doing a similar writeup as my app launches in 2 weeks. I've got a small budget. All the design+dev was me, but I'm spending about $300-$400 on marketing. We'll see how that goes.

If anything I believe you should be spending more on marketing nowadays. There's no real way, at least to my knowledge, to search for a niche hole with keywords. I do, however, like to think of an app idea and then spend an hour search various keywords and seeing the results/competition. <https://appstorerankings.net/> is a good website to find the keywords of an app. I combined common keywords in the app that I'm releasing with common keywords that are missing among competitors that yield few results.

Again, I haven't launched this app (my 7th) yet, but I'll be doing some analysis of its success using the marketing budget and keyword research.

<comment><AUTHOR> I agree it's hard to do really detailed research similar to, say, the Google Keyword Tool. It's mostly a gut check. "Does this have a lot of competition? Does this keyword appear to get a lot of searches?"

What's your app name?

<comment><USER25> Poetreat. Aiming for May 7. Week of polish then submission. <http://poetreatapp.com/>

<comment><AUTHOR> Nice landing page for sure. Maybe this is obvious, but they key is finding out a solid upgrade that makes IAP's worth it to customers.

<comment><USER25> Agreed. Doing a freemium model where you can save only save 1 poem. Unlocking adds theming, Facebook+Twitter connection, and unlimited poems. Worth $.99? That's subjective, but this entire app is more of an experiment than a business. Just going to iterate and improve on the next one.

</comment></comment></comment></comment></comment></comment><comment><USER26> Is that a scroll bar down the right side? Seems poorly placed if you expect people to touch on the right side to advance a page.

<comment><AUTHOR> Great point. The navigation between chapters is still being decided.

</comment></comment><comment><USER27> Nicely done! I'd love to hear more about the marketing efforts that went into promoting the app.

<comment><AUTHOR> Thanks for the comment Nathan. Initially it was all via search.

Since then I've started collecting email addresses the very 1st time the app launches. I'm up to over 30,000 emails now so every new product launch starts relatively well.

<comment><USER28> Impressive...How do you collect the email email addresses? Do you just ask people for their addresses?

<comment><AUTHOR> Yep. When the app launches the very first time I ask for their 1st name and email if they want to hear about deals/offers, new apps, etc.

They can cancel out easily and never see the form again. But when you have a free app that quite a few people download, email addresses start accumulating pretty quickly.

<comment><USER29> I was under the impression that this was a paid app. How does it generate so much revenue while still being free?

I signed up for future posts ( _extremely_ well-placed sign-up form by the way!), but I didn't see a description of in-app purchases or other revenue model as a topic for a future post, which is why I ask. I am also using Android.

Thank you for this post by the way. This is amazing and I'm excited to read your future posts!

<comment><AUTHOR> Great questions. Revenue is where it's at!

Initially it was just a paid download for the text of the Bible.

With more research in the App Store though I realized audio Bibles were selling much more than I was.

So I outsourced the audio (mentioned in the post I think) and now the model is that users download the original app (with the text) for free, and then can buy the audio as an IAP.

This works great on multiple fronts: I collect more email address, I rank better in search because I get more downloads, and the audio sells better as an IAP than as a stand alone app.

<comment><USER29> Wow, you are incredibly ingenious! Thank you for responding!

<comment><AUTHOR> :) You bet

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER30> I'm sure you'll make it clear in later posts, but are you a developer?

<comment><AUTHOR> I'm not a dev or a designer. I'm tech savvy enough that I'm still the computer repair guy for my entire extended family, but I'm def not fluent in Objective C.

<comment><USER31> If so, and only the later posts are about hiring, what, I assume, means you were hiring only after you've found about initial success, how come the first version ever came out?

Beside that, great story, great spirit, congratulations, I admire everyone actually getting things done :)

<comment><AUTHOR> I'm not sure I understand the question, but I think you're asking how I made the app at all if I'm not a dev/designer?

I outsourced 95% of it. I created the initial mockups by hand and did some of the debugging, but the core app development was done by contractors. And yes, for $500.

The point is it wasn't great, but it was good enough to test the market.

</comment></comment></comment></comment><comment><USER32> Great first article in the series. Can't wait for the rest.

</comment><comment><USER33> Nice! Even the most frugal of people will shell out some money when it comes to religion.

<comment><AUTHOR> True about the copycats. Although there are a few things you have to do right in the app that most people don't know. Most versions of the Bible are copyrights for example.

And I also speak Spanish. I can't imagine doing this app in a different language. The text has to be perfect and it's (obviously?) hard to tell what's wrong when you can't read it.</comment></comment>


Show HN: Get Live Coronavirus Stats in Slack with Nimbella Commander - <AUTHOR> https://medium.com/@bhageena/get-live-coronavirus-stats-in-slack-with-nimbella-commander-a8cf6f2b5566 <comment><USER2> looks good! although media gave me enough depression already, and slack is for work & headache :)</comment>


What would Dijkstra do? Proving the associativity of min - <AUTHOR> https://byorgey.wordpress.com/2020/02/23/what-would-dijkstra-do-proving-the-associativity-of-min/ <comment><USER4> Here's the solution I thought of, which is pretty similar.

Start with the case definition of min.

Notice that min(a,b) <= a, min(a,b) <= b and min(a,b) in {a,b}

Let d = min(a,min(b,c)) and e = min(min(a,b),c)

Then d <= a, d <= b, d <= c and d in {a,b,c}

Same goes for e.

Since e is in {a,b,c} and d <= a, d <= b, d <= c, it must follow that d <= e. And by symmetry e <= d. So d=e.

<comment><USER5> Similar? In my opinion this is much faster, easier, and less burdened by notation.

<comment><USER6> It's certainly a nice solution, but OP's solution has the nice property of working directly on a universality definition, which makes the argument more general. For example, the universality argument also works for the infimum and gcd operations.

<comment><USER7> > For example, the universality argument also works for the infimum and gcd > operations.

I've been thinking about this and I don't see how the argument can possibly work for the infimum operation. The problem is that infimum is necessarily a unary operation -- it doesn't make any sense to consider it to have multiple arguments.

Consider the open interval (1,2). The infimum is 1. But 1 is not contained in the interval. And because 1 is not contained within the interval, _if_ infimum were a binary operation, we could not apply it to the interval and get 1. Binary infimum is the same operation as binary min, and while the infimum of (1,2) is 1, the minimum value of (1,2) does not exist.

We can actually use this to do a much more direct proof of the associativity of binary min: binary min is the same thing as unary min (min as applied to a single set of arguments rather than two scalar arguments), and unary min cannot fail to be associative because it has only one argument. Repeated binary mins do nothing more than apply unary min to the set of individual arguments.

</comment></comment></comment></comment><comment><USER8> The Dedekind cut formulation of the real numbers is that a real number is defined to be a nonempty "downwards closed" subset of rational numbers. Then the min operation corresponds to taking intersections, and taking intersections is associative.

It's cool they figured something like this out from first principles.

edit: oops, somehow didn't notice arnarbi already mentioned this!

<comment><USER7> > The Dedekind cut formulation of the real numbers is that a real number is > defined to be a nonempty "downwards closed" subset of rational numbers. Then > the min operation corresponds to taking intersections

They didn't figure that out exactly. They figured out something much less significant: that a real number is defined by a nonempty upwards closed subset of _real_ numbers:

>> My second epiphany was that equality of real numbers can also be characterized by having the same "downsets", i.e. two real numbers are equal if and only if the sets of real numbers less than or equal to them are the same.

In other words, we choose to represent the real number r as the closed interval (-∞, r].

The point of Dedekind cuts is that you can define real numbers without using real numbers in the definition.

They do then set up min as set intersection as suggested by the interval definition.

Writing out the choice of real number representation makes it clear that we could also represent a real number as the interval [r, ∞), in which case minimum would be set union rather than set intersection. It doesn't really matter whether we use a closed or open interval.

</comment></comment><comment><USER9> Similarly you can show that gcd and lcm are associative, once you know their universal properties (for gcd, being the one common divisor that every common divisor divides; for lcm, the corresponding dual property). Similarly the associativity (up to unique diagram-fitting isomorphism) of products (or even fibered products) in any category (when they exist).

The really high horse here is called "Yoneda embedding", but there is not much gained from going that high.

<comment><USER10> You don't really need to bring the Yoneda embedding into it. They're all just partial orders (for the gcd and lcm use the order defined by divisability), which can easily be shown to be categories with at most a single function between objects. After that it's just expanding definitions.

</comment></comment><comment><USER11> I think a shorter way to write that is to first note the downset representation of reals. Then your definition of min is just the set intersection and there's nothing more to show.

<comment><USER12> To be fair, reals have nothing to do with this - the 'min' makes sense - <USER2> is associative - in any ordered set (even partially ordered, if you accept that it may be undefined for some pairs).

<comment><USER11> Absolutely. Just going off the language used in the post which is maybe tailored to Brent's student audience.

</comment></comment></comment><comment><USER13> _This is the definition that leads to the ugly proof by cases._

Right there is the problem. There's nothing wrong with proof by cases. People suck at it, but machines do it just fine. I ran into this years ago in the early days of program proving. The kind of proofs that mathematicians like are not the kind you really want to use to get work done by machine.

<comment><USER14> "Programmers are not to be measured by their ingenuity and their logic but by the completeness of their case analysis." ~ Perlis

I personally pride myself on my ability to analyze situations by cases, and to grind through normally-opaque walls to find proofs, but we should always keep in mind that it is the hardest way. It's like fishing for algorithms.

<comment><USER15> Presumably, ingenuity and logic can lead one to a simplified case analysis (more cases covered by fewer separate conditions), which allows one to do a complete case analysis with less effort.

</comment></comment></comment><comment><USER16> > The book expected them to do a proof by cases, with some sort of case split > on the order of a , b , and c . What they turned in was mostly pretty good, > actually, but while grading it I became disgusted with the whole thing and > thought there has to be a better way.

This is how it's done in the Idris standard library[1]:

      total minimumAssociative : (l,c,r : Nat) -> minimum l (minimum c r) = minimum (minimum l c) r
      minimumAssociative Z c r = Refl
      minimumAssociative (S k) Z r = Refl
      minimumAssociative (S k) (S j) Z = Refl
      minimumAssociative (S k) (S j) (S i) = rewrite minimumAssociative k j i in Refl

[1] Or prelude, or whatever it's called. [https://github.com/idris-lang/Idris- dev/blob/master/libs/pre...](https://github.com/idris-lang/Idris- dev/blob/master/libs/prelude/Prelude/Nat.idr#L772)

<comment><USER17> That's doing induction its arguments, which relies on them being naturals - it's not an approach that would generalise to reals.

<comment><USER18> I think that's necessary, since the order on the full set of reals isn't constructively decideable, so min isn't even a total function there.

<comment><USER17> Almost all reals are not computable, but as far as I know min is total - <USER3> two reals (represented as e.g. lazy infinite decimal sequences), computing their min is trivial.

<comment><USER19> Unless when they happen to be equal. Then it never halts.

<comment><USER20> Actually, computing the minimum should work: If they are equal you can still lazily produce the digits of the decimal expansion. What you can't do in this case is to tell whether this minimum is equal to the first or the second number.

</comment></comment></comment></comment></comment></comment><comment><USER21> I find proofs like this weird.

I can never quite understand what it means to say anything beyond:

"whichever of {a,b,c} is the smallest is the answer to both sides".

 _Who is this audience_ for which all this extra ink carries extra meaning? _What additional insight_ can you have possibly communicated to them?

<comment><USER22> One way of looking at it is training, you practise punching a bag and later you go box, you prove trivial results 5 ways then when you have a problem that isn't trivial you have 5 approaches in your locker.</comment></comment>


Even in Go, concurrency is still not easy - <AUTHOR> https://utcc.utoronto.ca/~cks/space/blog/programming/GoConcurrencyStillNotEasy <comment><USER2> That's a classic deadlock bug - two producer/consumer relationships in opposite directions.

Have another goroutine receive and output the "found" results, rather than doing it in the first loop. Now you have a proper pipeline.

The big win with goroutines is that they're green threads. They can block, and you can have lots of them. With regular threads, too many will use up resources. With "async", anything that blocks or just takes too long locks up the whole system. Go has the best of both worlds, which is very useful for the case of a server holding open connections to a huge number of human-paced clients.

<comment><USER3> I've never worked with a green threads system in any real sort of environment where I was actually solving real problems at scale, but I always wonder, does the underlying runtime ever push them into actual real threads that can actually run on another core, or are they always the imaginary concurrency of using the gaps where another green thread needs to wait the comparative eternity for some I/O?

<comment><USER4> By default, a go program runs computations on as many threads as there are CPUs in your system (this can be adjusted via gomaxprocs). The execution of all goroutines is distributed amongst these threads. For maximum efficiency you want to avoid spawning much more threads than there are CPUs, as scheduling has an overhead and each thread consumes valuable memory for the stack space. The Go model of running many goroutines distributed across threads is a very efficient solution and also very flexible, as the program writer does not have to make any assumptions about the CPU count of the machine the program runs on.

</comment></comment></comment><comment><USER5> Hmm, I don't know. I had this limited concurrency issue on the first Go program I ever wrote, and I'm very new to concurrent stuff, certainly if you only count the past ten years or so.

I figured this out in Go about 30 minutes after I realized I needed to limit concurrency, and I'm a dumbass, basically.

So, I'm not sure that "still not easy" is accurate, especially given that I have a lot of trouble conceptualizing async and await, as used in C#. In fact, I don't think I've ever successfully used that paradigm in C#. I've been told that C# async and await are "easy" and this article says Go concurrency is "still not easy" and I've had the exact opposite experience in both cases.

In fact I really don't understand why async and await is even a thing anymore, given the paradigm that Go uses and the example it sets. It is almost supernatural in its ease, for me.

<comment><USER6> > I really don't understand why async and await is even a thing anymore

Programming with threads is hard because they can context-switch at arbitrary points. Goroutines are nothing more than (lightweight) threads.

Stackful coroutines (as in Lua) aren't much better, because they can context- switch at any function call.

Code that uses async-await is different - it can only context-switch at points explicitly marked with `await`.

<comment><USER7> async/await in the general sense doesn't have anything to do with the preemption policy of a particular language implementation. For instance, C# implement async/await as keywords on top of its existing concurrency model.

That's really what it comes down to: cooperative multi-tasking vs preemptive multi-tasking.

Windows 3.1 had cooperative multi-tasking. Everything went great until one program didn't yield control, and then the whole system ground to a halt. That should sound familiar to any Node.js developers with their event loop.

Cooperative multi-tasking also extremely complicates code that actually uses the CPU. If you look inside libuv, it even has to jump through hoops with its encryption functions by effectively calling yield() in between rounds of calculation.

<comment><USER8> >If you look inside libuv, it even has to jump through hoops with its encryption functions by effectively calling yield() in between rounds of calculation.

I would have assumed the Yield was there to help with protect against timing pokes...

</comment></comment></comment></comment><comment><USER9> Concurrency is hard and we have very poor support for testing correctness of concurrent and distributed systems. Language abstractions help but they aren't nearly enough (as evidenced by this post). My team at Microsoft leverages Coyote to check the safety of our services against such subtle race conditions. We blogged about using it to reliably reproduce and fix a very subtle bug in a bounded buffer implementation over at [https://cloudblogs.microsoft.com/opensource/2020/07/14/extre...](https://cloudblogs.microsoft.com/opensource/2020/07/14/extreme- programming-meets-systematic-testing-using-coyote/)

If you're using .NET in your projects, you can start taking advantage of such tools _today_. I would like for such tools and testing techniques to become more and more common place in the industry as concurrent and distributed systems are _hard_ and we should use all the help we can get.

<comment><USER10> Go comes with thread sanitizer, which you can enable with go test -race ... If your unit test exercises a race condition, this will blow up your test with stack traces of the data race.

It sounds a bit like Coyote, which also looks very useful for C# applications.

<comment><USER9> Neat to learn about thread sanitizer. It sounds similar to another tool from Microsoft Research called Torch ([https://www.microsoft.com/en- us/research/project/torch/](https://www.microsoft.com/en- us/research/project/torch/)) which automatically instruments binaries to detect data races. Coyote is similar in some ways but different in others. Coyote serializes the execution of the entire program (running one task at a time), exploring one set of interleavings and then rewinding, and then exploring another set of interleavings, hoping to hit hard-to-find safety and liveness bugs. In addition to finding concurrency bugs in one isolated process, we use it to find bugs in our distributed system by effectively running our entire distributed system in one process and having Coyote explore the various states our system can be in. It sounded mind-boggingly cool when I first came across this way of testing distributed systems through Foundation DB ([https://www.youtube.com/watch?v=4fFDFbi3toc);](https://www.youtube.com/watch?v=4fFDFbi3toc\);) we're emulating this kind of testing in our distributed system through Coyote. And unlike Foundation DB which had to develop their own variant of C++ to be able to do this kind of testing (kudos to them for doing it), Coyote allows us to do it on regular C# programs written using async/await asynchrony and benefit from decades of Microsoft Research in exploring large state spaces effectively.

</comment></comment></comment><comment><USER11> Did Go ever make concurrency easy?

Any modern language allows you to create threads (semantically equivalent to goroutines) and pass messages between them. Most of them even have high-level concurrency support such as parallel map, structured concurrency, reactive extensions, type-safe generic collections...

We could say that Golang makes concurrency using thread semantics scale to a large number of threads. When this paradigm is appropriate, then indeed Go is nice. The select statement is nice. But this is not always the right approach.

Yes, it is true that go needs slightly fewer characters to launch a naked thread than in C# or Java. In that sense, it is easier. Anyone who considers that a major selling point is a walking race condition.

As a tongue in cheek analogy, what if we consider my new language G, in which G is an alias for goto. G is going to finally make control flow easy. I've done away with complicated control flow constructs such as for or while - users wouldn't understand them anyway. Users are encouraged to use the G keyword liberally. After all, it's only 1 character. Indeed, by that metric, it's even simpler than go's go keyword. What could go wrong?

<comment><USER12> ESL here. What do you mean by " walking race condition"?

<comment><USER13> A "walking X" is a person who is or is going to create an X. For example, an unpleasant boss might be a "walking turnover problem".

<comment><USER12> Thank you for your detailed explanation. I really appreciate it.

</comment></comment></comment></comment><comment><USER14> My personal style here is to use a number of goroutines equal to the desired concurrency. I know goroutines are cheap and there is no particular reason to conserve them most of the time, but this feels more natural to me than using a semaphore--although it is less flexible.

Simplified:

        var wg sync.WaitGroup
        wg.Add(jobs)
        ch := make(chan workItem, jobs)
        for i := 0; i < jobs; i++ {
            go func() {
                defer wg.Done()
                for item := range ch {
                    processItem(item)
                }
            }()
        }
        for _, item := range items {
            ch <- item
        }
        close(ch)
        wg.Wait()

<comment><USER15> Here's how I like to do it:

[https://play.golang.com/p/AB6exUIkNRg](https://play.golang.com/p/AB6exUIkNRg)

Contents pasted below.

      package main

      import (
       "fmt"
       "sync"

      )

      func main() {
       wg := &sync.WaitGroup{}
       e := NewParallelExecutor(uint64(10))

      for i := 0; i < 100; i++ {
        wg.Add(1)
        func(_i int) {
         //make closure to capture i correctly
         e.Submit(func() {
          fmt.Println(_i)
          wg.Done()
         })
        }(i)

      }

      wg.Wait()
       fmt.Println("All done.")
      }

      type Executor interface {
       Submit(func())
      }

      type parallelExecutor struct {
       queue []func()
       inCh chan func()
       doneCh chan struct{}
      }

      func NewParallelExecutor(limit uint64) Executor {
       pe := &parallelExecutor{
        queue: make([]func(), 0, limit),
        inCh: make(chan func()),
        doneCh: make(chan struct{}),
       }

      go func() {
        running := uint64(0)
        for {
         select {
         case <- pe.doneCh:
          running -= 1
         case f := <- pe.inCh:
          pe.queue = append(pe.queue, f)
         }
         if len(pe.queue) > 0 && running < limit {
          running += 1
          fmt.Println(fmt.Sprintf("Goroutines in flight, waiting: %d | %d", running, len(pe.queue)))
          f := pe.queue[0]
          pe.queue = pe.queue[1:]
          go func(_f func()) {
           _f()
           pe.doneCh <- struct{}{}
          }(f)
         }
        }
       }()

      return pe
      }

      func (p *parallelExecutor) Submit(f func()) {
       p.inCh <- f
    }

</comment></comment><comment><USER16> Go just sells itself well.

Languages that _actually_ make concurrency easy, such as Haskell, Scala, OCAML or distributed concurreny easy (Erlang/Elixir) are way ahead of Go but are not as hyped and are not marketing themselves as well.

<comment><USER17> Concurrency isn't easy in any language.

<comment><USER18> Erlang comes pretty close to achieving this. When/if people get past the "weird" syntax it's a very straightforward model that minimizes a lot of the hazards, though locks are definitely still possible.

<comment><USER19> The actor model struggles when you run up against something that is naturally sequential and the order matters.

If you want a 1000 households to visit a set of shops in a repeatable Random order until the shops runs out of stock, that's a List sort with a seed and a sequential loop

Make the 1000 households concurrent and sending messages and it becomes a major exercise in clocking, scaling and scheduling.

<comment><USER20> Erlang processes (Actors) solve this kind of problem naturally, with code that is concise, asynchronous and parallelized. For example, look at solutions for _' Sleeping Barber'._ It's possible to write a full solution in <40 SLOC.

Starting 1000 processes is trivial and fast in Erlang. See the first few pages of Joe's presentation from 20 years ago: process creation time ~10us (up to 30k processes); message time ~1us [1]. The code is in his book and the email thread [2]:

[1] [https://www.rabbitmq.com/resources/armstrong.pdf](https://www.rabbitmq.com/resources/armstrong.pdf)

[2] [https://erlang.org/pipermail/erlang- questions/2007-July/0280...](https://erlang.org/pipermail/erlang- questions/2007-July/028068.html)

<comment><USER19> It's rather more tricky than the sleeping barber. It's more that their are 100 hairdressers, each household has a set of seven of them and each hairdresser only has a fixed amount of hair colour so not everybody will be served, and you need to be able to repeat the visits deterministically based on a random seed so the only whole process is verifiable.

</comment></comment></comment></comment></comment></comment><comment><USER4> I do think that Go has a great set of tools to write concurrent programs. But I think it is a fallacy to believe that as a consequence "concurrency is easy" in a general way. These tools still require the programmer to be aware of the challenges of concurrency. They do make it more easy to deal with them.

It is the same story as with garbage collection. Garbage collection prevents some kinds of errors and in general makes it way more easy to deal with dynamic memory allocations. However, garbage collections does not mean you don't have to think about allocation patters and object lifetime for example.

Go does make concurrency much easier by the tools it provides. Most importantly are goroutines, which are very lightweight, so within reasonable limits, you don't have to be concerned about the number of goroutines you spawen - but as the example shows, you shouldn't try to spawn more goroutines as there are file handles available to your process, if every goroutine allocates a file handle. Not only are goroutines very lightweight with small growable stacks, but as they are part of the language specification, the compiler can generate code which helps with scheduling, in most cases you have efficient cooperative sheduling, so thread-based preemption is less frequent.

On top of that, channels provide a very easy to use abstraction for communication between goroutines. There are a lot of use cases, where goroutines and channels give you very easy and safe concurrency. It does not save you from understanding concurrency issues and especially deadlocks. Which in a general sense is impossible to achieve, because to prove that a program is deadlock-free would require to prove that all goroutines return and that is equivalent to the halting problem, which is unsolveable.

It should also not be underestimated, that due to goroutines and channels being part of the language spec, these are very commonly used features. Concurrency is present in most Go programs. As a consequence, when one reads a lot of Go code, there will be plenty examples of their usage. And any library needs to be thread-safe as the likelyhood is very high that it does get called from goroutines.

</comment><comment><USER21> The problem is defined, but all of the proposed solutions don't seem fitting. When using a semaphore, it's usually for a reason. I think the solution with the least overhead and perhaps most idiomatic is to spin the found reading loop in a goroutine BEFORE you start any at all, and leaving the rest of the code as is. This prevents having x number of idling goroutines, where x can be billions(but not in this case). Though again, in this case, using a semaphore and perhaps goroutines at all seems spurious.

</comment><comment><USER22> I've been wondering for like 15 years when a language is actually going to show up where I can just write my code procedurally and the compiler automatically figures out what it can parallelize.

Like if I have 3 definitions in a row that set variables to the result of methods that don't share memory, that seems pretty obviously parallelizable. Why isn't anyone doing these sorts of optimizations?

I would have thought it would have been solved by now, but I was wrong.

<comment><USER23> That sound 'cheap' but if you think of the clock rate of a CPU, doing those tasks in parallel will take time to setup, process and teardown. That example doesn't really work well.

Fitting problems into grids of processing like SIMD or GPU's is tricky because not a lot of problems fit those well like images. Most of what you're asking for is about understanding business flow which a language will never do.

</comment></comment><comment><USER24> The following is my solution:

See the playground: [https://play.golang.org/p/pXJaGQ0efe8](https://play.golang.org/p/pXJaGQ0efe8)

Source Code is available on GitHub: [https://github.com/go- training/training/blob/2ddb95d08c654a6...](https://github.com/go- training/training/blob/2ddb95d08c654a6410c2cfa70b0f506d8deceda9/example38-concurrency- is-still-not-easy/answer02/main.go#L1)

</comment><comment><USER25> Maybe I'm dumb but I don't understand the bug. I understand why the main code blocks if no tokens are available in the found channel but why would writing to `found` block the code (and why only in the case when there's a lot of go processes)?

<comment><USER26> It deadlocks if len(pss) > concurrencyProcesses.

<comment><USER25> Even in that case, wouldn't the main loop block for a short time but then unblock after the goroutine returns (and the defer function is executed)?

I thought the whole point of limitCh is to support the case where len(pss) > concurrencyProcesses but in this case you're telling me that it's breaking things?

<comment><USER26> The inner goroutine cannot return because it blocks at found<-P, because found is not buffered and there is no reader.

This would work if the consumer was started in a goroutine first.

<comment><USER25> Thanks. The part about starting the consumer in a goroutine actually made more sense as a solution than either of the given solutions.

I guess I completely forgot how channels work and didn't realize that until there's a reader, the channel will block (which makes perfect sense in hindsight).

</comment></comment></comment></comment></comment><comment><USER27> A more common way to write limited concurrency is to use a worker pool: [https://blog.golang.org/pipelines](https://blog.golang.org/pipelines)

</comment><comment><USER28> The protoactor library for golang: [https://github.com/AsynkronIT/protoactor- go](https://github.com/AsynkronIT/protoactor-go).

I can't recommend it high enough. It makes working with highly concurrent go so much easier and is a blessing for anybody with prior experience in Erlang or Akka.

</comment><comment><USER29> When I first came to Go, I'd heard a lot of people speak profusely about CSP and how Go made concurrency easy. I was disappointed at just how _not_ easy it was.

For example, Go has no generic support for atomic primitives, arrays, slices, or maps. It has sync/atomic, but it's not generic. I always reach for Uber's atomic library, which has typesafe atomic wrappers such as atomic.Bool. They're so common, you'd think a CSP-aware language would provide keywords declare things atomic:

      var b atomic[bool]
      if cas(b, false, true) {
        ...
      }

"Classical" Go didn't have errgroups, contexts or cancellation. To build something truly robust, "modern" Go ends up involving all three (plus atomics, of course). That's because almost all complex situations need to build what are effectively nested trees of goroutines that all need to quickly abort and unwind on errors or panics.

But as you use more and more of these primitives, your code (or at least my code) becomes more and more obscured by the layers of error-handling, cancel- detection, retrying, and so on. I really wish some of this was built into the language, especially cancellation. For example, channels don't support contexts. So if you have a loop like this:

      for evt := range taskCh {
        handle(evt)
      }

...then you have to rewrite it to support cancellation. So you have a few options. One is to select on both the channel and context:

      for {
        select {
          case evt := <-taskCh:
            handle(evt)
          case <-ctx.Done():
            return ctx.Err()
        }
      }

Because of this, you can no longer use a for loop. Immediately the code got more bloated and less readable.

Another option is to spawn a goroutine whose only job is to abort the channel:

      go func() {
        <-ctx.Done()
        close(ch)
      }()
      for evt := range taskCh {
        handle(evt)
      }
      return ctx.Err()

That's a bit better (and you could move the for loop into its own function for clarify, without changing anything else). But it's still worse than a simple for loop!

As a real-world example, here [1] is some code I've been working on lately. A controller starts N workers that need to process a task queue. If the controller is stopped, all the workers need to stop. So I use the pattern above. But I can't use a for loop! That's because I use a task queue abstraction that needs to support features that Go channels don't.

I can't use a for loop or select block to block on the task queue, because only channels support that. So that's another problem with Go: Often, you can use raw channels as your data processing primitive if things are simple enough, but in practice, channels are best for coordination, and you have to build your own primitives — and yet, by building those primitives, you lose language expressiveness.

To address the concrete example, I'd love to do this instead:

      // Range over task queue (custom object with blocking
      // semantics), automatically cancel if ctx is cancelled
      for task := range taskQueue with ctx {
        handle(task)
      }

[1] [https://gist.github.com/atombender/6bcff2c2d8fec32bc80ce1f57...](https://gist.github.com/atombender/6bcff2c2d8fec32bc80ce1f57a8e61f6)

</comment><comment><USER30> What would a test for this look like, having a lot of processes?

<comment><USER9> That's a great question. Stress testing, which is what you are suggesting helps, but is not super effective and often misses bugs. You need tools which can precisely control the task/go-routine scheduling during testing and systematically explore the various interleavings which can happen in the system. We generally don't have good tool support for such testing. There are promising tools emerging however; here is a case study of one such tool and how it was used to reliably reproduce and fix a subtle concurrency bug: [https://cloudblogs.microsoft.com/opensource/2020/07/14/extre...](https://cloudblogs.microsoft.com/opensource/2020/07/14/extreme- programming-meets-systematic-testing-using-coyote/)

</comment></comment><comment><USER31> try erlang!

</comment><comment><USER32> But with Kotlin flow it is!

</comment><comment><USER33> Using goroutines in golang is not dissimilar from threads in other languages (except that you can spawn much more of them), with all the downfalls and gotchas. As a matter of fact, it's even worse than other languages with proper concurrency libraries and data structures (e.g. it has nothing remotely close to Java's `java.util.concurrent` package).

What we see with golang is a phenomenon where people just parrot what some well known figures said at some point in time, without proper evidence or any basis (and in some cases, even when the evidence is counter to those claims).

I'm looking forward to Java's green thread implementation (project Loom), as it has proper ways to manage cancellation and deadlines and hierarchies, all of which have are quite verbose and error prone, or not supported at all in golang.</comment>


Surprise: 1,000 Google Reps Handling Over 10,000 Calls A Week From 60 Countries - <AUTHOR> http://searchengineland.com/google-customer-support-suprise-1000-reps-handling-10000-calls-a-week-from-60-countries-91616

<comment><USER2> 10,000 is only 2 calls per person per day. Which means they're either very inefficient or (more likely) the number is such an underestimate that it's meaningless.

<comment><USER3> Or people have no idea that this group exists, much less how to contact it.</comment></comment>


Ego depletion, the idea of willpower as depletable resource, fails again - <AUTHOR> http://journal.frontiersin.org/article/10.3389/fpsyg.2016.00621/full <comment><USER2> Just needs a better experiment, on a sample body larger than the current one.

If only there was an environment where we could test the behaviours of people faced with repetitive, "will draining" tasks who would also be willing to participate in long term studies.

Something like, say, an MMO. Especially one where the scientific community was already involved and protocols for interaction had already been established.</comment>


A new web app for finding new entertainment. Personalized. - <AUTHOR> http://www.whatsnextup.com/

<comment><USER2> Does this extract information from my Facebook likes?

<comment><AUTHOR> It just extracts the list of your likes.

<comment><USER3> It extracts likes, finds similar items and recommends entertainment events for you across categories.</comment></comment></comment>


British woman held after being seen reading book about Syria on plane - <AUTHOR> https://www.theguardian.com/books/2016/aug/04/british-woman-held-after-being-seen-reading-book-about-syria-on-plane <comment><USER2> > However, like all airlines, our crew are trained to report any concerns they > may have as a precaution

And their superiors are trained to call the police in any case, just as a precaution. And the police are trained to jail anyone who is reported, just a as precaution. The only one who can stop this slippery slope is the person in charge with fighting terrorism, but he is too busy to intervene. /s

This kind of freedom - to read Syria related books in a plane - has long died. We're just now aware of it. Maybe 20 years ago it was OK, but now we got other priorities.

<comment><USER3> From the article

"Police officers questioned Shaheen for 15 minutes under Schedule 7 of the Terrorism Act, under which the police can detain individuals without grounds for suspicion of involvement in criminal activities, including terrorism."

So...someone over reacted/misunderstood, the police had a chat, and she went home. Detained is not arrested, it's not on record it just means you aren't allowed to walk off (IANAL but that's how I understand it).

Someone in the cabin crew was a bit dumb, but it was over within half an hour of it hitting an actual authority figure.

Not a pleasant experience, but the only people at fault here are cabin creq

<comment><USER4> We are now in a place where you are racist if you report something and you are wrong, and you are an idiot if you don't report something and something bad happens. Most people would rather be labeled an idiot, and we have yet another terror attack in Europe.

<comment><USER5> We are now in a place where you are a racist if you report something stupid based on someone's race and are wrong.

Are you trying to make the case that reporting a Muslim for reading a book is somehow _not_ racist?

<comment><USER4> It's safe to assume there were other muslims reading books on a plane going to Turkey, and they didn't get reported.

This woman wasn't reported "for reading a book," but because of what book she was reading. So it was not racist, though her appearance was definitely part of the decision making process.

<comment><USER6> >It's safe to assume there were other muslims reading books on a plane going to Turkey, and they didn't get reported.

If any of these books were about Syrian culture, you would have a point. But the issue is whether or not it's reasonable to suspect, detain and interrogate someone for reading a book about Syrian culture, regardless of their appearance.

> though her appearance was definitely part of the decision making process.

I think a more obviously European-looking person reading the same book on the same plane would not have seemed suspicious, and would never have been detained. Her appearance even being part of the decision making process is what makes it racist. That such an absurd thing was taken seriously makes it racist.

<comment><USER4> There is a lot of context not in the article. A white European man with a long beard and clean shaven lip reading the book might have seemed suspicious. Since what authorities are looking for is people going to fight for teh Islamic State, then how Islamic you looks is a valid factor in determining suspicion - that isn't racist.

<comment><USER6> "looking Islamic" while minding your own business isn't suspicious behavior.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><AUTHOR> Similar incident - a man doing math: [https://stactivist.com/2016/05/12/equations-on-an- airplane/](https://stactivist.com/2016/05/12/equations-on-an-airplane/)

<comment><USER7> by that logic - algebra is a word from arabic, al jabr, if you follow it to its ultimate conclusion of somebody with a racist myopic worldview, arabic = desert and camels and guys with swords = terrorism = ISIS

<comment><USER8> Keep going... Lock up everyone who uses those radical Arabic numerals then?

<comment><USER9> Also chemistry/alchemy comes (sortof) from the Arabic "al-kīmiyā" \- or further back, from the Egyptian name for Egypt.

So any chemist reading books on "Free Radicals" should be questioned immediately!

</comment></comment></comment></comment><comment><USER10> This reminds me a lot of the people questioned after the Boston bombing for "suspicious" Google searches. [0]

One person was questioned after searching for pressure cooker bombs at work, and the other was questioned because she was searching for pressure cookers, and her partner was searching for backpacks.

[0]: [http://nypost.com/2013/08/02/li-man-questioned-for- computer-...](http://nypost.com/2013/08/02/li-man-questioned-for-computer- history-after-searching-for-pressure-cooker-bombs/)

<comment><USER7> whatever you do, don't search for 55 gallon steel drums, the recipe for ANFO, and ryder rental trucks

</comment></comment><comment><USER11> When you get amateurs to do your security, you get amateur security.

</comment><comment><USER12> Sadly, if I were a Muslim (or just brown) I would be afraid to write code on my laptop while flying. I can hear it now...

"He's trying to hack the plane!"

<comment><USER13> This brings a lot of light when once I started programming in the plane and a guy suddenly started to ask me many questions about my job, nationality, etc.

</comment></comment><comment><USER14> I wonder, if she had a different name and appearance, if she wouldn't have been suspected.

</comment><comment><USER15> I guess she'll be switching to ebooks now...

<comment><USER16> As a practical consideration, one might be able to wrap a cover around a book. If one is peering over your shoulder, an ebook might be advised with a privacy screen.

<comment><USER17> Until the airline crew conclude that hiding the title of the book one is reading is equally suspicious. After all, innocent people have nothing to hide, right?

<comment><USER16> Use a cover depicting flowers or palm trees, nobody will suspect anything--or as the poster suggested via YouTube video, a cover with the title "Ass Eating Made Simple" /s

But yes, I could see that too...

</comment></comment></comment></comment><comment><USER18> Positive side of this story is that the community we are living in is actually not with their heads in the sand and report people that look suspicious to them.

Maybe in Turkey they will detain a turkish woman, coming from a honeymoon in UK, for reading an article about Fethullah Gülen. That's actually not funny anymore.

</comment><comment><USER19> > _on her return from her honeymoon in Turkey_

Many people left UK to join ISIS or to be kidnapped... Syria is a warzone, and if you are travelling in/from that direction it will raise a questions.

> _Police officers questioned Shaheen for 15 minutes_

This article is a joke.

<comment><USER20> You have no idea how many people are traveling from Europe to the general direction of Syria or back. A vanishingly small percentage of these people pose a security threat. Harassing people who read books about Syria is not going to make anyone safer.

<comment><USER19> It is not just about terrorism. It is a normal procedure to question people who travel from that region.

Most people traveling from Syria ask for asylum or have no valid travel documents, do not speak English and need some form of help. She could be someone who escaped ISIS and needed help.

Some other example: 20 years ago UK border force would question everyone who was traveling from Czechia. Because 0.1% of travelers from there would ask for asylum.

<comment><USER5> It is? It looks like there are maybe two dozen direct flights per day between London and Istanbul, not even looking at other UK or Turkish cities. That's probably thousands of passengers per day. Do they all get questioned when they enter the UK?

</comment></comment></comment></comment><comment><USER21> I know what I'm taking as reading material on my next flight...

</comment><comment><USER22> Big f __*n deal.

</comment><comment><USER23> > I do question if … it would be different if it was someone who wasn't > Muslim.

Doubtful. How would they know you're Muslim until after they questioned you?</comment>