Is productivity the victim of its own success? - <AUTHOR> https://growthecon.com/blog/Baumol/ <comment><USER2> The problem is not productivity growth it's resource allocation.

Markets only optimize for good market performance. Since we do nothing to stop all the non-market game playing (mainly regulatory capture, monopolies & advertising), our markets have been optimizing for firms best able to corrupt the process.

Why make an ISP that delivers good speeds at a low cost when you can bribe local government to make you the only supplier?

Why make good smartphone hardware when you can bundle a package of hardware, software and BRANDING! And charge vastly more?

Why sell access to information people want when you can sell access to marketing information on peoples searches helping pervert the information people use for their decisions?

The people in charge are too busy enriching themselves inside the existing system and as a result some huge fraction of the population is working to directly harm the interests of everyone else.

<comment><USER3> When used to allocate production, markets maximize efficiency. But when used to allocate consumption, markets often _minimize_ happiness. For example, if you sell a glass of water to the highest bidder, then Poor Alice who's dying of thirst will be outbid by Rich Bob who wants to water his potted plant. That's not just a theoretical example, you can name ten such situations in the real world without stopping for breath. In other words, diminishing marginal utility of money means that markets are only half of the solution, not the whole solution. It's a pretty simple idea and I'm not sure why it isn't well known.

<comment><USER4> If your anecdote were accurate, rich people would not have most of the things; they would have all the things. Your model is so incomplete that it is misleading you.

Mind you, I'm not saying you'll love everything about the market when you understand it, but that anecdote is not a useful way of thinking about the world, because the model that it implies clearly fails to accurately model the world. Which is probably why it is not "well known".

<comment><USER5> I don't follow. The rich don't have everything because they don't bid for everything, which is still consistent with your parent comment.

<comment><USER4> "The rich don't have everything because they don't bid for everything,"

No, they _can 't_ bid for everything. It's a critical distinction to understanding how the economy works. It is impossible to create a single entity that buys everything, or a single entity that has all the money, permanently. It's literally nonsensical; the result is a form of "money" that may have the name on it, but no longer has any of the properties we associate with money.

You appear to be proposing a model where the rich are _voluntarily_ not buying everything up for some reason. Which is probably just an element of your model that you haven't examined, because my guess is that as soon as you consciously examine that, you'll experience some cognitive dissonance because I bet you also model rich people as intrinsically greedy, and thus having no reason to voluntarily not grab resources.

People are _really_ bad at following through the implications of models. They make a model, figure that the model touches reality in one or two places, then decide the model must be correct. But reality is pretty darned big, and it's really, really hard to fit "reality" into a simple model. Correct models must match reality _everywhere_ , not just a few points. Useful models must match reality in more than just a couple of places.

Follow the model's implications through.... if Rich Bob really can outbid Poor Alice for a glass of water, then why isn't the world actually full of dehydrated poor people? ( _Not_ that some people do occasionally dehydrate to death, often for reasons only loosely related to wealth issues... _all_ the poor people should be dehydrated because the rich people are, after all, constantly outbidding them for water, apparently.)

The very act of answering that question must, with mathematical force, also demonstrate why the model that produced the question in the first place is irredeemably broken. If the model is that irredeemably broken, then the question is also broken, and what it is trying to imply is also false.

(Which, again, is why this result is not "well known". It's, well, not true. The fact that markets are not perfect does not mean that all criticisms of the market are automatically true.)

<comment><USER5> I think you misunderstood my comment. Yes, the rich don't bid for everything, and yes it's because they can't. But that's still consistent with what cousin_it said.

The main flaw with your argument is that "if Rich Bob really can outbid Poor Alice for a glass of water, then why isn't the world actually full of dehydrated poor people?" is 1) a non-sequitur, and 2) doesn't make any sense because the if condition is false.

<comment><USER6> > doesn't make any sense because the if condition is false.

Why is it false that Rich Bob can outbid Poor Alice for a glass of water?

<comment><USER5> Where I live, water is free.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> Not one word about demand.

Most of the US population is out of buying power. The savings rate is near zero. More than half the population couldn't come up with $400 in an emergency. The US economy is demand-limited, not production-limited

<comment><USER8> Not enough demand shouldn't be a problem: the central bank can just print money.

(I write "shouldn't" not "isn't" for good reason.)

<comment><USER9> The central bank is printing tons of money and continuously pumping it into the economy. Americans don't feel the effects of that as everyone uses American dollars. So it's almost as if the central bank is creating gold, which is suppose to be a finite quantity. It does nothing.

At the end of the day corporations must borrow that money and corporations are sitting on tons of their own money, they don't want more money to hire employees. They don't know what to do with the money they have. They can make goods, but no one will buy them at the levels of profit they need. So they just sit on the money.

When the fact is the economy is efficient and they should be making goods even if they are not profitable, i.e. at cost... just to create employment, give people money, with which they can buy things. But that's not how the profit motive works. The management consultants and management theory have pushed this idea in their heads, and they are sticking to it, that they need to create profit always. Shareholders demand it! But profit is hard to come buy these days. So they sit around and do nothing instead. Jobs don't get created, people are starved for money and that vicious cycle ensues and consumption goes down.

<comment><USER8> The Central Banks are simply not creating enough inflation, if demand is a problem.

In the most extreme case, the central bank can either just buy any assets under the sun, or start dropping cash from helicopters (slightly more realistically: give every citizen a bunch of money).

In any case, have you heard of nominal income level targeting?

Ie the central bank instead of targeting inflation, targets what's more important: the aggregate income of all workers. (Targeting Gross Domestic Income instead of just the wage share is also possible.)

They set a target of eg 5% nominal growth a year, and announce it. Then do everything necessary to make it happen.

<comment><USER10> If the post-2008 world isn't proof enough that printing money does not automatically increase aggregate demand, I don't know what would count as proof in Econ.

By the way, I'm not saying it can increase AD, just that it does not automatically does so, for some reason that is widespread now.

<comment><USER8> The central bank can always inflate. If printing lots of money isn't enough, they just need to print some more money.

And, if they manage to buy all of government debt without creating any inflation, that's awesome on some level as well: they just managed to monetize all the public debt without any adverse side effects. (And for sovereign debt, monetization is about as good as eliminating it.)

</comment></comment></comment></comment></comment></comment><comment><USER11> This is tangential, but it's an interesting choice to pick a URL which parses as "Grow The Con" just as well as "Growth Econ".

</comment><comment><USER12> So basically you have money, you build a plant and you an produce goods with a marginal cost of 0 and just get rich because you had access to capital mostly by birth.

Hug, it will not help most citizens to access the production sphere.

It will not help fair competition, hence innovations. It is just about who has the most capital. Are we back in XVIIth century were lords owns the lands and the production?

Does it strikes no one that most of western so called democracy were built to fight exactly this situation and the social inequalities raising from this case?

Especially when all countries are trying to make it impossible for citizens to produce on their own in their private habitation.

Regulations are creating an economy where the poorest cannot neither sell their workforce by lack of demande, nor produce by themselves, nor offer services.

<comment><USER13> > Does it strikes no one that most of western so called democracy were built > to fight exactly this situation and the social inequalities raising from > this case?

The problem has been identified _many_ times. What is left is to find a solution ...

<comment><USER14> We have a solution [0]. It just has its own issues and has to be redone every 100 years or so.

[0][http://youtube.com/watch?v=exnaY0l4XsM](http://youtube.com/watch?v=exnaY0l4XsM)

</comment></comment></comment><comment><USER15> > New services cannibalize time from old services. If I come up with a new way > to use worker's time, by necessity it must subtract from the time they spend > doing something else

This is not universally true. It is possible for technology to allow one service worker to service more people simultaneously. Consider the increased productivity of tech. support workers due to the introduction of asynchronous communication like chat as opposed to voice.

<comment><USER16> The model assumes that time spent on services is perfectly efficient, or rolls this into the service quality measure.

In reality, a potential service period can be divided into utilized and unutilized segments. The model of a CPU's run queue and time slots may be useful here.

If you're under-provisioning services, you'll stack up customers (processes) in the run queue. If yo're overprovisioning services, you'll have high idle times. Most likely you'll see a bit of both. If you have a perfect allocation system, the queue length is always one (the job running) and there's no idle time.

Technology allows you to approach that ideal. Waiting customers can be paired with idle servers. But it's not perfect, especially with services that have a physical component and travel. And you're left with the hard limit of there being only one hour per hour.

Scheduling and dispatch systems are also already aggressively persued. See Lyft.

<comment><USER8> Yes, though Lyft and Uber are on the verge of replacing their labour inputs with capital. Ie self-driving cars are around the corner.

<comment><USER16> Fair point.

What their _present_ model shows, though, is the increased efficiency of high- bandwidth signalling between customer (rider) and provider (driver).

Taxis had previously relied on radio dispatch, which was better than on-street hailing, but essentially had a single blocking communications stream between drivers and central dispatch, with high wait times for channel space, relatively high error rates in transmission, and (compared to digital dispatch) low total bandwidth and precision.

The Uber/Lyft model is famous for bypassing medallions (though I suspect those will re-emerge). But it also replaced an obsolete dispatch channel which incumbents had failed to innovate on (reasons for which might be worth investigating).

<comment><USER8> I don't know whether the dispatch was the bottleneck. How would we figure that out?

(But having to talk to a human dispatcher over the phone would eg put me off a bit. More than checking how much the uber is on my smartphone. And once I'm in the app to compare prices to my other options, ordering is only a tap away.)

</comment></comment></comment></comment></comment><comment><USER8> There's lots of little and big reforms possible to help the productivity of the economy. Of course, most of them are politically infeasible in most countries.

I wonder about the long term effect of these reforms. I am afraid there will be mostly one-off linear factors, and not contributing much to higher future growth.

(My argument rests on the observation that eg the poster child Singapore is rich, but not pulling ahead more-and-more.)

</comment><comment><USER17> There are so many potential places to find more productivity in our society, automation, AI, self-driving cars, and that's just the big, obvious stuff.

<comment><USER8> Yes, so the question is: why doesn't it show up in the statistics? (Please read the article..)

<comment><USER17> It will when AI is invented.

<comment><USER8> Perhaps. But that's not an answer to the "why don't we see all the technology that has already happened recently in the productivity stats?" question.

</comment></comment></comment></comment><comment><USER16> Reading this post is like walking into the middle of an old family dispute and thinking you've heard all of it. This is just one wrinkle of a _much_ larger and older discussion that's been ongoing in various forms for at least the past 240 years, going back to Adam Smith, though the modern concern with economic growth dates largely to the Great Depression and post-World War II period.

The model here is looking at the _question_ of economic growth (which is distinct from the question of distribution, though not unrelated), and of a simplified economy in which _only_ mutually exclusive goods and services can be offered.

The conclusion (under artificially but not un-usefully simplified terms) is that even with continued growth in efficiency of _goods_ provision, _services_ are ultimately constrained by the requirement of _time_ as the ultimate input.

It's possible to quibble in various ways about the model, but it's essentially correct. A services-based economy is limited by the element of time _in both_ production _and_ consumption. Media already has the problem of there being only so many consumption-hours per day, _and people are already consuming more than 24 hours of media per day_ through the mode of multitasking.

A key problem, of course, being that multi-tasking reduces the total capability to incorporate any given media stream.

Going more broadly:

There's the question of what accounts for economic growth. The traditional economic answer is "technology", though that answer itself is flawed and raises the question of what technology _is_ , and more specifically, how different types of technology _operate_ and _act_ , as well as what kinds of production or services they enable, and how.

Robert J. Gordon's book, _The Rise and Fall of American Growth_ looks at this, in detail, though imperfectly.

One of the book's better points is that the innovations of 1870 - 1950, generally, addressed very fundamental human needs: food, housing, clothing, transportation, and safety. With tremendous impacts on these: food and diet (both quantity and quality), health, work and productivity, communications, and transport.

The innovations of 1950-2015 have been far less fundamental. They've been concentrated in information and communications, _which yes, have changed and improved tremendously_ , but they _don 't_ address needs and capabilities at the base of human wants. A term not used anywhere in Gordon's book is Maslow's Hierarchy of Needs, among several disappointing deficiencies. The 1870-1950 innovations largely addressed the _base_ of the pyramid. The 1950-2015 innovations have largely addressed the higher elements. This isn't entirely useless, but it's _not_ foundational. And Maslow's Hierarchy cannot be built from the top down.

The discussion's been going on for a long time. Adam Smith, Thomas Malthus, John Stuart Mill, Thorstein Veblen, Joseph Shumpeter, John Maynard Keynes, John Kenneth Galbraith, Robert Solow, and more. My sense is that mainstream economics has rather lost the thread, though it does some work of interest. Gordon's book stumbles over the truth then picks itself up and walks on. It _does_ however point to several factors, including education, which _don 't_ account for growth. In that, it's useful.</comment>


Will You Be Able to Read This Article in 1,000 Years? - <AUTHOR> http://nautil.us/blog/will-you-be-able-to-read-this-article-in-1000-years <comment><USER2> "Will You Be Able to Read This Article in 1,000 Years?"

Barring some significant advancement in Sid-Meier's-Alpha-Centauri-style technologies like longevity vaccines or brain uploading, probably not.

Even if I do survive, will I still be speaking this dialect of English? Will I even be speaking English at all?

But back to the main point of the article, saving the article as a PDF is a _very_ wrong approach; binary formats are likely going to be the first to die off. I'd be much more confident in the article being stored more-or-less as it is now: as plain-text markup, style information, and some code executable by just about any modern Javascript implementation, accompanied by the texts of the relevant standards (for HTML, CSS, ECMAScript/Javascript, etc.), so that any individual could then (theoretically) piece everything together and display it as it was intended to be displayed.

Lots of very old software from the beginning of computing history has been preservable in source code form for this very reason of the robustness of plaintext; even with encoding differences, it's been possible to convert these documents (even if doing so has been a manual process). I'd expect that, due to this robustness of plaintext, free-and-open-source software has significant preservation advantages relative to proprietary software (of the closed-source variety, at least), since there's a greater chance of copies surviving due to how widespread said copies are.

</comment><comment><USER3> Flip this around and look backwards: how much exists from 1000BC–1000AD (using the "1,000 years to 3,000 years" figure cited by Cerf)? In popular knowledge, not much: Sun Tzu and Confucius, the Bible, the Kama Sutra, Homer, Virgil, Beowulf, Aristotle, Plato, Egyptian hieroglyphics. Machiavelli, Chaucer, and their peers around the time of Gutenberg are all "too late" in the 14th century.

These works survived because they were (presumably) important. Cultural understanding is important to anthropologists and shows up as well (e.g., graffiti at Pompeii). But how much is Crash Bandicoot going to _matter_ in 3000AD? How wide will the cultural divide be? (What's a "computer"? What's a "button"?) What language will they speak?

Thus, the concern to me is less _how_ we'll save everything, but more _what_ are the important things to save? After all, the Iliad doesn't stand out among zetabytes of video games, cat pictures, and Kardashian re-runs.

Interesting aside: the Department of Energy is facing a similar, simpler challenge with the Yucca Mountain nuclear waste site. Namely, how do you create signage that will work (survive, be understandable) for 10,000 years? ([http://www.salon.com/2002/05/10/yucca_mountain/](http://www.salon.com/2002/05/10/yucca_mountain/))

<comment><USER4> Well, it's also about the durability of the information medium. Clay tablets have survived remarkably well and they give us a better view into ancient Mesopotamia (particularly the ancient Ur-III dinasty) than later centuries' records kept as papyrus or strips of wood.

[https://en.wikipedia.org/wiki/Third_Dynasty_of_Ur](https://en.wikipedia.org/wiki/Third_Dynasty_of_Ur)

<comment><USER3> Very true. And the medium (among other factors, like literacy) has a self- selecting component: you're not going to take the time (money, etc.) to make clay tablets out of everything. There wasn't a clay-tablet-Twitter back in the Neo-Sumerian Empire.

As modern medium becomes a) cheap, b) ubiquitous, and c) durable, there's no reason to "edit" ourselves. Maybe that's a good thing: it would give future anthropologists a treasure trove of information. But again, my concern is that we'll end up with so much _quantity_ that the _quality_ data/recordings/thinking get subsumed, and future humans think we worship cats. (All those photos! And captions! And videos! I for one bow to our feline overlords.)

</comment></comment></comment><comment><USER5> The only reason we have anything from 1000yrs ago is because people deliberately conserved it - copying, translating and actively passing things down the generations - for example:

Media:

Oral history/stories -> early written versions on tablets/papyrus -> vellum/scrolls/hand copied books -> printed books -> digital.

Language:

Aramaic/Ancient Greek -> Latin/Greek -> Modern English

If the original source media is very long-lived, you can skip some steps, but you still need to actively conserve and translate things - not just between old human languages, but from old media to new.

This process is obviously very lossy - we have only a tiny fraction of the literature from 1000 yrs ago - we have only the intersection of the stuff that people cared enough about to transcribe many times over and the transcribed pieces that physically survived the ravages of time and history.

Expecting things to just survive over the long term without anyone doing this - without anyone caring enough to do this - is probably both idealistic and unrealistic.

</comment><comment><USER6> No, I don't think I'll be able to read anything in 1000 years, not given my life expectancy.

More seriously, on one hand, we have pretty good redundancy, since it's so easy to copy things.

On the other hand, the durability of the mediums we use to store information is probably not as good as writing stuff in stone or even on paper since in the worst case scenario, depending on the format used, minor corruption can make the data completely worthless.

Add to that the fact that future archaeologists will have to reverse-engineer the whole stack from hardware to software if they want to have a chance to make something of the (probably corrupt) data they have on hand. It probably puts the bar even higher than understanding Egyptian hieroglyphs.

Oh, and also add to that the signal vs. noise ratio of the data we produce today and the amount of data we produce. It would probably be interesting to see what picture future historians paint of our times with what they have on hand.

</comment><comment><USER7> People have been talking about this problem for decades. The only information from this era likely to survive is what's stored in plain text or other easy to interpret format. Do you really think people 500 years from now will be able to watch the Blu-ray of Paul Blart Mall Cop 2? It's more likely that they'll be able to create a C compiler and build some of the free software of today.

</comment><comment><USER8> One of the explicit goals of the Document Foundation is to be able to answer this question "yes". LibreOffice already does better on ancient Word documents than Word 2013, and the Document Liberation Project is _all about_ the old formats.

</comment><comment><USER9> Hmm, taken literally, this is a perfect example of Betteridge's Law, as I won't be around in 1,000 years to read anything. Rather than just the usual discussion of bit rot, I personally would wonder if humans will still exist in 1,000 years, and if we do, if we will still read.</comment>


Show HN: Customised meal plan to boost yourself - <AUTHOR>

Hey HN folks, what do you think of http://signup.mealthy.com/. The principle is simple. You choose what you'd like to do (exam preparation, gain muscles, marathon preparation, become a vege and so on) and we calculate a meal plan for you based on scientific studies which prove some nutrients have a real effects to help you achieve this goal. Meal plans are customised to your lifestyle, age, weight and, obviously, to your objectives. Let me know what you think about it and signup if you want to know more. <comment><USER2> As a European I am curious whether measures of meals, ingredients and weight will be in imperial or metric units, or both.

</comment><comment><USER3> How do you plan to compute that based on studies? If you could implement a nice way to do that based on actual study results it would be super interesting. As in not a bunch of hard-corded rules. Perhaps it could somehow be crowd-sourced. Not sure if there'll be enough people for that tho.

Also there should be a way to input what you actually it so it can self-adjust based on what you're actually doing rather than what you'd ideally be doing. As in "okay we can see that you don't really eat X so let's try Y". But I guess then you'd loose 95% of potential users cause that's a pain to do.

There was this mental health company that would suggest the right medication based on statistics. Supposedly it was much more accurate than regular diagnoses that you'd get from a doctor. I can't find it thought ;C

<comment><AUTHOR> We read scientific studies and extract a profile from them.

About adapting your profile, after now 40+ interviews it seems it would only work for really specific segment (like fitness addict) as it implies to much daily inputs.

<comment><USER3> Yep. Maybe you could try some kind of a weekly checklist kind of thing? Like your weight and a bunch of yes no questions. So it takes like 1 minute. I think you need to find a way to make it provide value on an on-going basis. Otherwise the utility of that app is pretty low. Maybe you guys could find some way to bridge what "regular people" and "maniacs" like. Basically to somehow gather input from people that are wiling to do that without scaring regulars away. The data you gather could be the most valuable thing your company has.

<comment><AUTHOR> We are looking for a way to do that but we haven't found any quick but accurate enough way to do that for the moment. I am confident we would find one day :)

</comment></comment></comment></comment><comment><USER4> Interested in what you guys have. I signed up!

One note that you have probably already thought of is make sure to have multiple options. In the working world, it seems to me that the hardest thing to do is find time for cooking good meals. So having a few options for quick, medium prep, and longer prep would be insanely helpful. Looking forward to it!

</comment><comment><USER5> People don't usually have the choices of what to eat to strictly follow a "meal plan", and everyone has their own dietary preferences.

</comment><comment> nhayes-roth This just seems to be a signup page.

<comment><AUTHOR> We are trying to validate the concept before jumping on the dev as suggested by lean startup methodology. Unfortunately, that means doing the "marketing" and boring stuffs before the interesting technical bit. Therefore, that's only a kind of "show your interest" page, I agree.</comment></comment>


Come and learn how to pitch to investors - <AUTHOR> http://pitchaccelerator0712-eorg.eventbrite.com/

<comment><AUTHOR> Startups that want to practice their pitch to investors with Startup Monthly mentors should send information to firstname.lastname@example.com</comment>


Software is narrative - <AUTHOR> http://infiniteundo.com/post/158336863413/software-as-narrative-1n <comment><USER2> I think I'll open with a dubious metaphor I just came up with and a demand that the reader immediately and wholeheartedly accept that metaphor as literal truth. That's how you grab an audience!

<comment><USER3> Yes, I write intentionally click-baity blog posts. You got me.

<comment><USER4> ajarmst's point goes beyond click-baitiness -- indeed it is about the little quote-box on the first para rather than the title. You say: _" Any software system begins as a shared narrative about a problem and the people who come together around solving that problem."_

I can, in fact, bend my view of facts to fit this metaphor. And if am also dubious about doing so, that's OK as it makes it all the more likely that I will learn something new from your post.

So why spoil that with some the arrogant demand that: _" If you don't accept the above proposition completely then nothing I have to say about software is going to work for you."_?

<comment><USER5> Just to advocate for the devil here, I think some advice is predicated on a shared model, between the giver and the taker, of how things work. If you don't buy the model, then you probably won't deeply understand or agree with the arguments that would get you from the model to the thesis.

<comment><USER6> I definitely appreciated this early warning to get off the ride.

<comment><USER3> While I appreciate the spirit of what you are saying, the average time on page of 3 minutes 45 seconds (across 2k uniques) indicates most visitors are continuing past the paragraph you claim would be problematic.

<comment><USER7> Devils advocate: Do you know what the average time is to read the entire thing is? If it's 7.5 minutes, then one possible explanation is that your audience is split (50/50). Median time would be a much better indicator.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> The contrast here is great.

Finding the problem and creating a solution along with focusing on the process of creating that solution in order to reduce risk of building the wrong solution. Contrasted to building a solution along with a sane dev and production environment for that solution to fit in, to increase possible iteration velocity by reducing deployment inertia.

</comment><comment><USER9> I think the OP is on a path that will eventually lead them to discover or re- invent Conway's Law[0]

[0][https://en.wikipedia.org/wiki/Conway's_law](https://en.wikipedia.org/wiki/Conway's_law)

<comment><USER10> Nice one! Thanks for sharing the insight

</comment></comment><comment><USER11> Just wondering if I'm the only one, but I like my logs to be the narrative of what's happening in the system.

<comment><USER12> Agreed. I think many people do a good job of logging when something goes wrong, and maybe they are good about logging inputs/outputs of a system, but I think that logging important decisions often falls off the table. Ideally I would like to take a request ID, grep the logs, and get an entire story of what happened to that request. In reality, this rarely happens!

<comment><USER13> _> take a request ID, grep the logs, and get an entire story of what happened to that request. In reality, this rarely happens!_

If that's the situation you find yourself in, I cannot praise centralized logging with a good frontend highly enough because I frequently find myself trying to figure out what happened to a request, and it's like night and day.

Needing to ssh anywhere and run grep against log files is functional if there's only one or a handful of VMs, but it gets complicated with a handful of machines, and even just SCP-ing the logs off becomes time consuming if there are a lot of machines. Then once the logs are off, 'grep' quickly becomes inadequate. (And I should know, I've built some truly horrible regexps to try and grep for dates because I didn't know any better.)

All that friction means that answering the original question; figuring out a detailed internal reason for why my customer received a 500 http status response error, is just too toilsome for all but the most (as you noted) doesn't happen in .

With centralized logging, I'm able to search for a request ID and see the logs, and this is a reality as often as I need, in order to debug complex multi-system issues.

<comment><USER14> Word of advice: the 'jq' tool for handling JSON files (couple with a glob like '*.log' or something fancier with xargs or parallel) will absolutely save your bacon in those situations. It's way more powerful than it appears on the surface.

We had a series of Docker json-file driver log files. It's done as a raw list (no array around it) of JSON objects -- which is a bit annoying to sort and filter based on properties of the objects.

'jq '[inputs]' (asterisk).log > combined.json' was my favourite command today; it combines all the files inputs and wraps them in an array correctly. No awk needed!

Combine that with its cute:

jq '.someProp as $var | test("some search"; "gi") as $r | if $r then ($var + $__loc__) else null end' (asterisk).log | grep -v "^null$" > filtered.json

And you're away to the races. Can then load the file directly in and group_by(.somePath) and it will all magically work!

Edit: had to remove the actual asterix symbols as they screw with formatting but are used for globbing the file names. Replace with the real character

</comment></comment></comment></comment><comment> d--b Sorry I just don't understand the article. Can anyone rephrase?

<comment><USER15> The main point of the article appears to be:

> You always have to design both "the product" your customers want and "the > environment" in which your product will run in production. Thus any software > product begins with two obvious categories of "work to be done." People > ignore this because it seems [counterintuitive.][going_there]

<comment><USER3> Hey you saved everyone a click!

I am the author of the post and I approve this summary :)

<comment><USER16> Your blog archive (where the title links to) seems to be broken.

<comment><USER3> Weird it just worked fine for me. [http://infiniteundo.com/archive/filter- by/text](http://infiniteundo.com/archive/filter-by/text)

<comment><USER16> Oh, don't mind. My network connection was the one that had a problem.

</comment></comment></comment></comment></comment></comment><comment><USER10> This is a good metanarrative for software authors. The more the product wants to be art, the more important this concept is. Humans are wired to spread stories so if a product doesn't communicate plenty of good ones, then there you have the anwer to why marketing budget isn't fixing the business.

</comment><comment><USER17> Okay, I'll take it at face value. Can you name a few competent programmers who are also competent book / story writers ?

<comment><USER18> I create video training courses on tech topics and one of the metrics I use to determine "is this performing well?" is the completion rate of the courses and individual videos.

I started taking communication and story telling pretty seriously about 9 months ago and the data on my 2 latest courses (where I tried to create a cohesive narrative of the content) shows the completion rate being quite a bit higher than my older courses.

Are there other things at play? Maybe, but I'm 100% convinced it takes more than "raw programming talent" to be a good programmer in general.

When it comes to designing library code or public APIs, I would rather use code written by someone with 5 years of writing / story telling experience and 2 years of coding experience vs someone with 10 years of coding experience but can't communicate well at all.

Chances are the 2nd person wouldn't be able to see the "big picture" stuff when it comes to designing APIs, and when it comes to API design, the big picture is the most important thing. The implementation details of each function is the easy stuff (since you're a Google search away from solving most technical problems).

There's also many other factors that matter besides your code. For instance, if I can't even find your project, or your documentation is lack luster then it doesn't matter if you wrote the most elegant code in the universe, I'm not going to use it.</comment></comment>


High-Dimensional Probability: An Intro with Applications in Data Science [pdf] - <AUTHOR> https://www.math.uci.edu/~rvershyn/papers/HDP-book/HDP-book.pdf <comment><AUTHOR> This is a textbook in probability in high dimensions with a view toward applications in data sciences. It is intended for doctoral and advanced masters students and beginning researchers in mathematics, statistics, electrical engineering, computational biology and related areas, who are looking to expand their knowledge of theoretical methods used in modern research in data sciences.

A typical graduate probability course is no longer sufficient to acquire the level of mathematical sophistication that is expected from a beginning researcher in data sciences today. The proposed book intends to partially cover this gap. It presents some of the key probabilistic methods and results that may form an essential toolbox for a mathematical data scientist. This book can be used as a textbook for a basic second course in probability with a view toward data science applications. It is also suitable for self-study.</comment>


Keep showering, California. Just lay off the burgers and nuts - <AUTHOR> https://medium.com/@lesliejz/don-t-stop-showering-california-lay-off-the-burgers-and-nuts-and-pick-up-some-crickets-eb63bddf0277

<comment><USER2> Interesting, informative article. But suggesting people should eat insects was an awful idea by the writer.

<comment><USER3> Curious: why?

Lobster is popular in the US and they're like giant sea insects. Or is it something else besides the "ew" factor that make you say it's an awful idea?

</comment></comment><comment><USER4> I only eat almond fed crickets.</comment>


Lisp as Blub - <AUTHOR> http://damienkatz.net/2008/04/lisp_as_blub.html <comment><USER2> I don't get it. The "blub" quality refers to expressiveness, not the robustness of the runtime system or suitability for particular kinds of application development.

To say that Arc is not concerned with brevity or expressiveness is simply ridiculous.

For the record, I think the way Erlang supports functional programming isn't particularly elegant and I think having Prolog-like syntax without Prolog-like semantics is silly.

<comment><USER3> _having Prolog-like syntax without Prolog-like semantics is silly._

Yes. Yes! I'm not the only one thinking it then!

I think the Erlang process model is brilliant but it really took on the ugliness of the Prolog syntax without attaining the full benefits of the Prolog logical semantics.

<comment><USER4> I think Erlang is brilliant because it keeps Prolog's unification (matching) but discards Prolog's backtracking in favor of procedural execution.

Unification can be very useful in solving some problems. Backtracking is also useful but sometimes gets in the way. Procedural code can be used to do anything that any language can do and students are more familiar with it. So I think Erlang is an interesting addition to languages. And that's without considering it's more practical aspects, e.g., multiprocessing, failover capability, etc.

Nor is Prolog syntax particularly ugly. A problem is that it is too simple, so simple that novices can become confused. Brna, Boulay and Pain discuss why Prolog is difficult in the book:

"Learning to Build and Comprehend Complex Information Structures: Prolog as a Case Study" [http://www.amazon.com/Learning-Comprehend-Complex- Informatio...](http://www.amazon.com/Learning-Comprehend-Complex-Information- Structures/dp/1567504345/ref=sr_1_2/102-1192735-0366500?ie=UTF8&s=books&qid=1208217565&sr=1-2)

See the review there by Randall Heltzerman titled "The Most Unlikely Book Ever to be Published".

</comment></comment></comment><comment><USER5> He's reading too much into this. At loads this low, it can't be a language design issue. There's probably just a bug in Mzscheme.

Hard to say whether this post is deliberate linkbait or just overeagerness to see this as evidence confirming a pre-existing bias. Probably a little of both.

<comment><USER6> Confirming bias? Maybe. Link bait? Definitely.

But you say it likes it's a bad thing. Do you think I don't have a valid point? Are you not debugging a memory leak in a production Lisp app?

With Lisp you can put together a site in a few hours. But how many days must later be spent debugging issues like this? Will this time spent debugging be counted against your productivity gains you attribute to Lisp?

This is exactly the sort of problem Erlang has nailed down. And it requires a whole new thinking about program structure. Is it possible you don't see the problems with Lisp because you can't imagine there is a fundamentally better way?

<comment><USER7> > Are you not debugging a memory leak in a production Lisp app?

But, you're confusing the actual application with the Lisp implementation. Arc (and Mzscheme for that matter) are not as mature as Erlang's 20 years (or whatever). If this was running on a more mature Lisp system, I don't think we'd be discussing the same thing.

<comment><USER8> You don't get to pick and choose implementations to compare. Mzscheme was probably chosen because it's a pretty well known scheme implementation and seems popular... likely because it's well done in many ways. But you have to take the bad with the good.

<comment><USER9> Maybe the problem is Arc's lack of tensile strength.

</comment></comment></comment></comment></comment><comment><USER10> Interesting point about having a process-oriented architecture where each process has its own independent GC, but boy-o-boy is the title a troll-tastic piece of link-bait.

</comment><comment><USER11> "Why current event X means we should all grind my favorite axe."

<comment><USER12> It's a pretty robust axe

</comment></comment><comment><USER13> This guy sounds like a suit in a boardroom . . . just spouting words devoid of validity . . .

</comment><comment><USER14> If you want to prove something seriously, please find some more examples. (Really!)</comment>


Node.js Interview: 4 Questions with Creator Ryan Dahl   - <AUTHOR> http://bostinnovation.com/2011/01/31/node-js-interview-4-questions-with-creator-ryan-dahl/?ref=hn

<comment><USER2> > _No matter what I did, someone was going to load an existing blocking Lua > library in and ruin it._

I don't think this is the case, the most popular library for networking in Lua has been COPAS, which does nonblocking by using coroutines. And even the canonical guide to Lua, PiL <http://www.lua.org/pil/>, also has a chapter on using coroutines with LuaSocket to download multiple files asynchronously at the same time. Most complaints of Lua are about having _too_ few libraries, so it's interesting to see an argument claiming it has too many.

If anyone is interested in a nonblocking networking library that uses _callbacks_ (similar to node.JS) instead of coroutines for Lua and LuaJIT, I've started such a project here: <https://github.com/davidhollander/ox>

</comment><comment><USER3> Previously: <http://news.ycombinator.com/item?id=2163234>

<comment><USER4> conversation on that is much better than this repost

</comment></comment><comment><USER5> FYI from January 31, 2011.</comment>


Ask HN: Routers that are secure out of the box? - <AUTHOR>

My apple router just broke due to a power surge. I've searched HN and apparently Linksys, Netgear, Asus and TP-LINK all have had decently-bad vulnerabilities/back doors, at least in the past. There's also discussion of open firmware but I don't want to have to maintain that.<p>Is there a safe router I can get that works out of the box or should I just stick with apple? <comment><USER2> To be honest I would go for the Ubiquiti Edgerouter series, super good bang for your buck, and advanced features if you ever want to play around with firewall/routing features.

[https://www.ubnt.com/edgemax/edgerouter- lite/](https://www.ubnt.com/edgemax/edgerouter-lite/)

<comment><USER3> That router doesn't include WiFi, so would you have to connect it to a second router that has it?

<comment><USER4> Correct. Instead of a second router, you would buy a dedicated wireless access point. Ubiquiti sells those as well.

</comment></comment></comment><comment><USER4> If you are familiar with Apple routers and have liked them in the past, I'd say stick with them and buy another.

They perform well, as easy to setup, and regularly receive updates from Apple.

If you decide you want something more powerful, with more features, or generally want to learn more about networking and security, I'd recommend buying an official pfsense device. See: [https://www.pfsense.org/](https://www.pfsense.org/)

</comment><comment><USER5> GateProtect UTM. Pricey but secure [made in Germany, regularly audited by other security companies, guaranteed no NSA back doors].

[http://www.gateprotect.com/en/Products/UTM- Solutions](http://www.gateprotect.com/en/Products/UTM-Solutions)

<comment><USER6> That would be a juicy target for NSA, with a rep like that.

<comment><USER5> I guess, they are very open about it.

They are selling an Intel based appliance running a hardened Linux [I think] distro that is customized to provide the service/protection they want. I guess it is only as secure as the packages they bundle.

Their site details their TeleTrust Cert comitment as:

The company headquarters is in Germany

The company offers trustworthy IT security solutions

No solutions that contain non-declared backdoors are offered

All the company's IT security research and development takes place in Germany

The company complies with the German data protection law.

[[http://www.gateprotect.com/en/why-you-should-consider- techno...](http://www.gateprotect.com/en/why-you-should-consider-technology- gateprotect)]

</comment></comment></comment><comment><USER7> You can always find one you can put DD-WRT on by reflashing.

<comment><USER8> The original poster mentioned that they did not want to have to deal with maintaining self-installed open source firmware.

Even if an open source firmware was the preferred solution, OpenWRT or another libre firmware would be a better choice. DD-WRT is harder to keep updated, often relies on closed source wireless drivers, and has had several remotely exploitable vulnerabilities in the past.</comment></comment>


Uber and Lyft Are Adding Jobs, Not Just Stealing Them - <AUTHOR> http://www.bloomberg.com/view/articles/2016-06-03/uber-and-lyft-are-adding-jobs-not-just-stealing-them <comment><USER2> "Stealing" jobs, is that the narrative we're suppose to subscribe to? Or are they breaking up a government backed monopoly and democratizing it. I personally don't care if there aren't any net new jobs if they've created thousands of small entrepreneurs and provided much needed competition.

Side note: I find it interesting how media outlets decide what the debate is, create a false dichotomy, then us simpletons are just suppose to pick a side.

What do you think chuck? Is it ok that they're stealing jobs?

<comment><USER3> Uber hasn't created "thousands of small entrepreneurs." They've created a special class of labor that has all the burdens of both employee and contractor status, with none of the privileges of either.

<comment><USER4> > They've created a special class of labor that has all the burdens of both > employee and contractor status, with none of the privileges of either.

I'm not even sure that's really the case; taxi drivers were already largely theoretically independent operators contracting with the cab company which provides dispatch, provides the vehicle under a lease arrangement, etc.

</comment></comment></comment><comment><USER5> Good to know quantity supplied/demanded is still related to price

</comment><comment><USER6> This is intentionally misleading (but in a typical, not uncommon way.) If you want to see if employment has expanded, you count hours, not employees. But, again, this is commonly done when a low unemployment/high employment rate is needed for press purposes - it's the reason why U-3 is used in government press releases instead of U-6, which also includes part-time workers who want full time employment. This is not a universal standard - in Germany, for example, the distributed unemployment rates include do that category (and others.) Even worse, US media (NYT, WaPo) frequently compare nominal US and German rates as if they were measured in the same way.

</comment><comment><USER7> Before Uber in SF, there were 2,000 taxis.

There are now 20,000 Ubers in SF.

Bad for the environment, good for jobs.

<comment><USER8> Those extra Ubers have surely reduced some cars (which are worse). Not saying the reduced cars outweigh the added cars, but could be worth considering when thinking about the environmental impact.

</comment></comment><comment><USER9> Yeah until they lay everyone off in favor of self-driving cars in 2 years...

<comment><USER10> What seems to be the issue here? If you can get a machine to do something that a human can do then you give that human the opportunity to place his focus, time and attention elsewhere.

<comment><USER11> The issue is that the only jobs that seem to be vanishing through such "opportunities" (what a weasel word) are all unskilled labor (easiest to automate). Sure, doing work in the car instead of driving home in California freeway traffic for an hour a day is an opportunity, except that's not the problem here...

Most people that work unskilled labor jobs today aren't qualified to work alternative skilled jobs.

A large class of unemployed (and unemployable) workers is a nightmare situation and is detrimental in any economy.

<comment><USER12> eh, if people are so much against basic income, get those unskilled hands fixing the country infrastructure.

I guess any country has plenty of fixing to do anyway.

most importantly, stop producing unskilled workers. automation isn't going to go away.

<comment><USER11> "most importantly, stop producing unskilled workers. automation isn't going to go away."

Now, we're talking about improving the education system, which has its own fair share of problems.

There are solutions, but none of them are easy to implement.

Or maybe this (fact that some humans cannot write code, etc. for a job) is a consequence of the bell curve or the human condition itself?

</comment></comment></comment></comment></comment><comment><USER13> They are adding jobs.

However, we now having more people making less money for essentially the same job. It's a win for consumers, because it's more convenient and in some cases, cheaper.

But it's a lose for the people that actually made a decent living driving who now are competing with college kids that are doing the same thing in their spare time without having any of the same startup costs.

Uber and Lyft removed the barrier to entry by breaking the law and the monopoly the Taxi unions had over the entire industry. But the result is lower wages and benefits.

<comment><USER14> Why should the incumbents have security in their position, when as you pointed out college kids can do the same work at lower cost? It is unfair to the unemployed and to consumers to provide special protection to incumbents based solely on their position and earlier entry to the market. That is a classic case of rent seeking with governmental protection.

Good for Uber and Lyft for breaking the law and pushing us out of that local minima. Sometimes you have to harm a few privileged rent seekers to remove coercive monopolies and reduce regulatory capture.

<comment><USER15> It isn't just regulatory capture though - permits for taxis carry all sorts of protections for consumers. Those protections are guaranteed by law and I see no replacement for those protections from either Uber or Lyft.

We used to have the free for all we're building towards and we regulated to stop it. Why won't history repeat itself?

<comment><USER16> It isn't clear that the permits systems provided better protection than UBER's system, and there's at least theoretical reasons to think UBER's data rich system could become better than permits, especially if combined with stuff like background checks,etc.

<comment><USER15> Agreed, but Uber has no reason to support background checks, etc. Any move towards regulation would make their business model fall apart.</comment></comment></comment></comment></comment>


Think Julia: How to Think Like a Computer Scientist - <AUTHOR> https://benlauwens.github.io/ThinkJulia.jl/latest/book.html <comment><USER2> PSA: The goal of the Julia 1.0 release was to stabilize language constructs for library authors to build upon. It will take time for them to update their libraries to be compatible with v1.0 -- so if you want to get a feel for the language, and things are breaking, stick to v0.7 for the near future.

<comment><USER3> Sadly not even 0.7 is usable right now, see [https://docs.julialang.org/en/v1.0.0/](https://docs.julialang.org/en/v1.0.0/) __* The only difference between 0.7 and 1.0 is the removal of deprecation warnings. __* I am currently using version 0.6.4. I feel it is still a great piece of software, even if is a few years old.

</comment></comment><comment><USER4> If I don't care about parallelism nor speed, is there a reason to learn Julia?

<comment><USER5> As somebody who appreciates Julia, my opinion is - probably not. That is, unless you want the opportunity to create a killer library that shows how the language features map well to other domains.

But, I think a _lot_ of people in scientific computing are tired of the typeless mess that Python/Numpy/Scipy code-bases evolve to be. And for those people, I think it has a _lot_ of merit.

At the end of the day, the language was designed to fill one major gap. A lot of time and effort in R&D is spent either; architecting sane C++ memory models, or reverse engineering existing Python code. Alternative well performing and safe languages like Java simply are not fast enough - to get the features of the modern CPU, you need to be native. And a side-note, MATLAB cannot usually be ran in a production environment.

<comment><USER6> Having spent years working with numpy and Cython, then switching to Scala for years as well, I much prefer dynamic typing.

Strong type safety is mostly just a waste of time.

<comment><USER7> As a long time Python/Cython user I can say that I have sorely missed static types in many occasions, especially for long running tasks. In fact I would sometimes use Cython not for performance but as a type checker.

I can describe a recent example. I had to ensure that an integer is always an int64 as the logic passes through different python modules and libraries. It was an absolute hell to track down all the places where things were dropping down to int32. With static types this would have been a no-brainer. This is not to say that I do not enjoy its dynamic typing where it is appropriate.

Hopefully Python 3 will make things better with optional types. But its still not statically typed, just a pass through a powerful linter.

<comment><USER6> While I certainly concede that dynamic typing will have painpoints like this, I just think on balance they create far fewer problems than the maintenance and inflexibility of type system enforcement patterns.

That said, I find your particular example with int64 extremely hard to believe. I assume you're using numpy or ctypes to get a fixed precision integer, in which case it should be extremely easy to guarantee no precision changes, and e.g. almost all operations between np.int64 and np.int32 or a Python infinite precision int will preserve the most restrictive type (highest fixed precision) in the operation.

I work in numerical linear algebra and data analytics and have used Python and Cython for years, often caring about precision issues— and have literally never encountered a situation where it was hard to verify what happens with precision.

Unless you're using some non-numpy custom int64 type that has bizarre lossy semantics, it is quite hard to trigger loss of precision. And even then, a solution using numpy precision-maintaining conventions will be better and easier than some heavy type enforcement.

<comment><USER7> I will agree about the 'on the balance' in the context of speed of prototyping and interactive sessions.

When rubber is about to hit the road, i.e. near deployment with money at stake, I would have love an option to freeze the types, at least in many places. Cython comes in handy, but its clunky and its syntax and semantics is not super obvious to a beginner (I am no longer one, but I remember my days of confusion regarding cyimporting std headers, python headers, how do you use python arrays (not numpy arrays) etc etc).

I am curious, have you put money at stake supported only by dynamic types ?

Regarding int32 vs int64, its not a precision issue its about sparse matrices with more than 1<<31 nonzeros. I am equally surprised that you have not run into this given your practical experience with matrices.

My case involves more than just numpy. There's hdf5, scipy.sparse, some memory mapped arrays and of course numpy.

Given the amount of time I spent to debug this, I would have killed for static type checks.

<comment><USER6> I happen to use scipy sparse csc and csr matrices for huge sparse tfidf data at work, but never encountered this (we have a numba utility function for operations we do directly on the data, indices, and indptr internal arrays, including counting).

But I do see that counting nnz boils down to a call to np.count_nonzero, which treats bools as np.intp, which is either going to be int32 or int64 (very weird that it chooses signed types), then calls np.sum.

The best solution would be to use np.seterr to warn exactly at call sites with int32 overflow, but amazingly, there seems to be an open numpy issue saying that seterr is not guaranteed for sum.

I do think seterr + logging would be better for this than roping in static typing everywhere just to get a one off benefit like this.

<comment><USER7> But thats just Numpy. As I mentioned the logic flows through other components too. I am guessing your nnzs are medium sized and hasnt hit 2 billion yet.

Quick question, when you create a scipy.csr how do you _ensure_ the subsequent multiplication operator falls back to C code that uses int64 to index the internals and not int32. I thought if indices array was a int64 array it would do the job. I was wrong. Anyway, even if that had worked it would still have fallen short of _ensuring_. If it worked, it just happened to work -- thats an anecdote.

If one had static typechecks one would not have to read through all the layers to be sure. Compile error, if any, would have told me.

We also cant directly use scipy.sparse because we dont have that much RAM on these machines. We do use scipy.sparse but they operate internally with memory mapped arrays. Now, depending on the platform memory mapped arrays can be limited to an index of 1<<31\. So we have to be extra careful what type is used for indexing in the native libraries that these layers are a wrappers over.

BTW its far from a one off benefit. This was just one of the examples fresh in my memory. It directly affects real money. There you dont want to ship code that could have bugs that can cost you. Static types help rule out these cases once for all. With run time checks it is very hard to be sure that you have caught all of the code paths that can have these mismatches.

I agree that in grad school its different :) One can play fast and loose. Even more, if research is not expected to be reproducible -- that would be pure science.

<comment><USER6> Our nnz is certainly far greater than 2 billion. The matrix size is around 150 million rows by around 1.7 million columns. We just accumulate the count with a python integer.

I don't know what you mean by "that's just numpy" though — since even if this flows through other systems, tracking it at the source in numpy would be obvious.

"Static types help rule out these cases.." — I just disagree. That is what's advertised, but it's just not true. Years of working in Scala for very heavy enterprise production systems has made me realize it's a very false promise. There are actually remarkably few classes of these errors that are removed by static type enforcement, and perfectly good patterns to deal with it in dynamic type situations.

If static typing was free, then sure, why not. But instead it's hugely costly and kills a lot of productivity, rather than the promise that it improves productivity over time by accumulating compounding type safety benefits.

I think a good rule of thumb is that anything that causes you to need to write more code will be worse in the long run. There's no guarantee you'll actually face fewer future bugs with static typing and visibility noise in the code, but you can guarantee it adds more to your maintenance costs, compile times, and complexity of refactoring.

I guess Python's gradual typing is a good compromise, since you don't have to choose between zero type safety or speculative all-in type safety where the maintenance overhead almost always outweighs the benefits (rendering it a huge and unreconcilable form of premature optimization).

You can only add it in those few, rare places where there is demonstrated evidence that the static typing optimization actually has a payoff.

<comment><USER7> > since even if this flows through other systems, tracking it at the source in > numpy would be obvious.

You cant possibly be saying that ! even if one assumes that source is numpy.

Regarding the rest, lets say my experience with Ocaml has been more gratifying than yours with Scala.

> We just accumulate the count with a python integer.

That wont help when you are using scipy.sparse for sparse on sparse multiplication, because the multiplications fall back to C code. You have to ensure that it falls back to C code that uses Int64 for indexing the arrays. I am sure you are not saying that you do sparse multiplications of this size in pure python.

Our differences in tastes aside, you seem to work on interesting stuff. Would love exchanging notes in case we run into each other one day. Should be fun.

<comment><USER6> > "You have to ensure that it falls back to C code that uses Int64 for > indexing the arrays. I am sure you are not saying that you do sparse > multiplications of this size in pure python."

For csc and csr matrices at least, these operations typically iterate the underlying indices, indptr and data arrays, and csc `nonzero` uses len(indices), which both relies on (eventually) the C-level call to malloc that defined `indices` (and so uses the systems address space precision, and would never report number of elements in a lower precision int than what the platform supports for memory addressing), and returns this as an infinite precision Python int. Afterwards it only uses arrays of indices, not integers holding sizes.

Long story short is that at least for csc matrices, the issue you describe wouldn't be possible internally to scipy's C operations, as you'd always be dealing with an integer type large enough for any possible contiguous array length that can be requested on that platform (and the nonzero items are stored in contiguous arrays under the hood).

On my team we are not doing pure Python ops on the sparse matrices, rather we needed customized weighted operations (for a sparse search engine representation that weights bigrams, trigrams, trending elements, etc., in customized ways) and some set operations to filter rows out of sparse matrices.

So we basically rip the internal representation (data, indices, and indptr) out of csc matrices and pass them into a toolkit of numba functions that we have spent time optimizing.

<comment><USER7> Lets not weasel with 'typically'.

The code that will get called for a multiply is this [https://github.com/scipy/scipy/blob/master/scipy/sparse/spar...](https://github.com/scipy/scipy/blob/master/scipy/sparse/sparsetools/csr.h#L562) and [https://github.com/scipy/scipy/blob/master/scipy/sparse/spar...](https://github.com/scipy/scipy/blob/master/scipy/sparse/sparsetools/csr.h#L609)

It's important that decisions at the python level trickles down to the correct choice when it comes down to this level.

On a 64 bit architecture one would expect that using 64bit int arrays for indices and indptr would ensure that. But thats not the way it works. We regularly encountered cases where it would call the code corresponding to int32. I know why and have special checks and jump hoops to prevent this.

 _Thats besides the point, with static types I wouldn 't need to do this, the compiler would take care of it._

I appreciate your effort to dig through the logic. You have spent time speaking at length in the comment above but unfortunately said little. Malloc has nothing to do with it. Your third paragraph is manifestly false. Why do I say so ? Because I deal with this everyday and have counterexamples.

I didnt mean to ask you to find out. Apologies if I wasted your time. I already know why the type mismatch happens. My point was to demonstrate that a lot of manual wading is needed to ensure that it finally bottoms out by calling native code with correct type.

<comment><USER6> The code you linked actually seems to refute your claim of this precision error, at least for multiply, because it is using npy_intp for nnz, which will be int64 on a 64 bit platform, and there is even an overflow check below!

Can you post a gist or link some other concrete example to show how it can overflow the intp type based on large nnz? Reading the code, it looks like this could not happen.

(Note that the entire second step function wouldn't have this problem, because it's accessing indices inside the other arrays, after nnz has already been computed, and is not looping over a variable that would overflow, apart from nnz from the first function, which I pointed out above seems not to overflow unless you're compiling things in a non-standard way that affects npy_intp).

I don't know what your comments about malloc having nothing to do with it are though. That is how numpy arrays possess their post-allocation result for __len__, such as for indices, indptr and data in csr. So __len__ could not overflow an int type (since it requires the platform address space's int type to allocate underlying contiguous arrays and returns a Python integer).

<comment><USER7> Can't help but say this, you are seriously confused. Not necessarily your fault, as obviously you dont have the full code.

I have mentioned earlier that it is not about precision but about index space. I don't think it's going to be productive use of my time to continue this thread.

One specific reason you are getting confused is because you are looking at function calls in isolation not the entire chain of calls through the different Python ecosystem libraries. The problem is the indptr and indices arrays that begin their life as int64 arrays get transformed into int32 arrays in specific code paths.

By stating that malloc is not relevant I mean its not relevant in this particular instance. By the time control reaches malloc the type mismatch damage has laready taken place.

Getting a runtime error is far from the end of the matter even if in certain cases we do get runtime errors. What static types saves the user from is the hunting needed to find out where in the chain of functions are we losing the type invariant we need.

Stopping such bugs is a no-brainer with static types. You claimed at one point up-streams [0] that type systems cannot rule out such errors. If you believe that, this discussion is a waste of time. That's one of the lowest forms of errors a type-system prevents. Your comments like these make me doubt your grasp over these things.

BTW, not sure if you believe large rows and cols imply large nnz [1]. That's not how sparse works.

Given your handle I would have expected you to be familiar, this is bread and butter stuff in day to day ML. On the other hand if your background was stats I would expect less of computational nitty gritties. Nothing wrong with that they focus on different but important aspects.

If you really care I would encourage you to track the flow of code from csr creation in scipy. sparse using memory mapped arrays of indptr, indices and nnz to the C code that will get invoked on two such objects, carefully. The key word here is _carefully_. There is no nonstandard compilation because there is no compilation. Its about dispatch to the correct C function.

You seem to believe that on a 64 bit platform such indexing error will not happen. That's patently false because it happened many times.

In other words, you are saying your ill conceived and incompletely considered notion of correctness are more correct and than test cases that fail.

This exactly where a static type system would have helped. Those ill conceived incomplete understanding would have been replaced by a proof that proper types have been maintained over the possible flows of control. In this case it would have saved me a lot of time tracking cases where int64 is dropping down to int32.

At this point I would stop engaging in this conversation because it has become an exercise in pointless dogma.

If you refuse to accept that runtime errors detected or undetected dont have a cost, or that static types can mitigate such costs, -- whatever rocks your boat. What I am claiming is that several times in my Python/Cython use I hit instances where static types would have saved a lot of trouble and time and money.

Another common type related problem happens when you need to ensure things remain float32 and do not get promoted to float64. I work both on the large and in the small, so I encounter these.

[0] [https://news.ycombinator.com/item?id=17789837](https://news.ycombinator.com/item?id=17789837)

[1] [https://news.ycombinator.com/item?id=17789837](https://news.ycombinator.com/item?id=17789837)

<comment><USER6> > "I have mentioned earlier that it is not about precision but about index > space."

Right, and if you read in my comments it shows I have also been only talking about the index space as well, where int64 vs int32 is a matter of int precision for representing large amounts of indices, but where npy_intp will be of the higher precision (to match the platform's address space) and will not be able to suffer the overflow issue you described unless it's a custom compilation of numpy defining npy_intp as int32 even on a 64-bit system (which you seem confused about by repeatedly saying compilation isn't a part of it, as if anyone is suggesting your personal workflow involves compiling anything, when I'm talking about how the numpy you have installed was compiled. If it's standard numpy compiled for a 64-bit system, then the evidence suggests your claim is just wrong.)

You claim that indptr and indices arrays are silently converted from int64 to int32 on 64-bit platforms but you offer no evidence. You just keep saying that it happened to you, despite the actual code you linked indicating that it couldn't happen. And I do actually work with indptr and indices arrays with tens of billions of nonzero elements in an Alexa top 300 website search engine every day, and have never encountered any such silent type conversion.

Given that this example of index int precision actually seems unfounded in the code, it just doesn't seem relevant to any sort of static vs dynamic typing debate. There's no such issue here that static typing would help with, because it's clearly not causing the problem you think it's causing in the dynamic typing code.

<comment><USER7> All I can say is you will learn a little bit more if your "know it all" persona is in proportion with your actual extensive knowledge :)

There was a recent thread on HN on Ousterhout on exactly that [https://news.ycombinator.com/item?id=17779953](https://news.ycombinator.com/item?id=17779953)

Re evidence, you cant possibly expect me to replicate the entire software library stack here on HN or elsewhere to show the loss of type information.

Hint you are still looking at functions in isolation and repeating loss of type info cannot happen and I have dealt with hundreds of counter examples. You can look up the conversation, I did not say the pointed Github code is to blame. I pointed the code saying we have to make sure that piece of code is eventually called with the right type. We have to ensure that at the time of creation of the sparse matrices. With the dynamic type handling this gets lost. Int64 gets dropped to Int32.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> You might contrast the approach here with say an Engineering textbook. This manual on a particular tool (Julia) seems to imply that it is the one way to engage with an entire discipline. An Engineering textbook might mention various tools for a particular job and even endorse one over the others but in general it will start with the problem and not the solution.

That said:

$ aurman -S julia

(rolls up sleeves)

<comment><USER9> Can you explain that last part?

<comment><USER10> `aurman` is a (unofficial?) package manager for Arch Linux. The standard package manager is called `pacman`. Arch User Repository(AUR) is (IIRC) a repository of uncurated packages compatible with Arch.

To put it simply: he is implying that he will check out the book.

<comment><USER8> Yes, I should have put pacman. The book is a great resource and despite my criticism has introduced me to julia.

</comment></comment></comment></comment><comment><USER11> Never wrapped my head around Julia. I like it, and I've used it for a couple things, but I've never had a use case compelling enough to keep at it.

<comment><USER12> It's elegant and powerful; there are very few coding constructs that are widely used that aren't in Julia, and those that are (like Classes) aren't there because the authors of the language don't think that they are useful, as opposed to "it's hard to implement". But YMMV, the downside is that the ecosystem is evolving, and it's just hit 1.0 so expect things to be smooth in 6mths to a year. The upside is that I find that the Julia code I write appears from the keyboard easily, quickly and in a form that I can understand a few weeks or a few months later.

<comment><USER13> > _those that [aren 't in the language ] (like Classes) aren't there because > the authors of the language don't think that they are useful_

From what I understand it's more that the combination of other features in the Julia language (like multiple dispatch) makes classes redundant.

</comment></comment></comment><comment><USER14> > In Julia indices start from 1.

Why? I programmed in Lua which also made such choice, and I find it rather inconvenient. Makes you always think if you got your indices right, since in all other programming languages indices start from 0.

</comment><comment><USER15> im trying to figure out how to apply this to coding challenges.</comment>


A Field Guide to Yelp's Unhappy, Unhelpful Eaters - <AUTHOR> http://www.theatlantic.com/life/archive/2011/03/a-field-guide-to-yelps-unhappy-unhelpful-eaters/72248/

<comment><USER2> As Merlin Mann so brilliantly put it, "Yelp.com: Explore Where Local Illiterates Have Recently Stopped Eating.™"

<http://twitter.com/#!/hotdogsladies/status/1453581139></comment>


Hearing brainwaves - epilepsy EEG sonified - <AUTHOR> http://sydney.edu.au/news/84.html?newsstoryid=11954 <comment><USER2> Why not apply ML techniques and get rid of the listener all together?

It's great work, but it just seems like an intermediary step.

p.s. I have no experience in analyzing EEGs, I am sure they are much more nuanced than I know, but could anyone explain the reasons preventing such a thing?</comment>


Tantalizing but preliminary evidence of a "brain microbiome" - <AUTHOR> https://www.sciencemag.org/news/2018/11/do-gut-bacteria-make-second-home-our-brains <comment><USER2> This is all very preliminary, a couple of quotes from the article stood out to me:

> Roberts wondered whether bacteria from the gut could have leaked from blood > vessels into the brain in the hours between a person's death and the brain's > removal.

> Roberts acknowledges that her team still needs to rule out contamination. > For example, could microbes from the air or from surgical instruments make > it into the tissue during brain extraction?

<comment><USER3> But you left off supporting evidence from mouse models: "So she looked at healthy mouse brains, which were preserved immediately after the mice were killed. More bacteria. Then she looked at the brains of germ-free mice, which are carefully raised to be devoid of microbial life. They were uniformly clean."

<comment><USER4> That does nothing to rule out contamination from a different tissue.

<comment><USER5> How could one possibly rule out contamination by this logic?

<comment><USER4> By using some procedure that controls for contamination from a different tissue, instead of having no relation to it.

</comment></comment></comment></comment></comment><comment><USER6> Would be fascinating if it turns out to be half as important as gut bacteria. Maybe certain human breakthroughs like the rise of civilization could one day be attributed to a bacterial infection. And then the inverse, changing diets and the brain biome might lead to their collapse.

<comment><USER7> I've thought the same of viruses as well. If the Zika virus can cause brain damage, perhaps there was another virus (and/or microbe) at some point that made us smarter.

</comment></comment><comment><USER8> Maybe its bacteria from periodontal disease

"DNA from the periodontal bacteria was also found in the brain tissue of mice in the study group, and a bacterial protein was observed inside their neurons."

[https://medicalxpress.com/news/2018-10-periodontal- disease-b...](https://medicalxpress.com/news/2018-10-periodontal-disease- bacteria-kick-start-alzheimer.html)

<comment><USER9> I know it is just pain radiating from the nerves of the tooth, but a couple times when I had a tooth go abscessed the pain was so intense and in my head I kept thinking my upper mouth is so close to my brain surely this infection can make it from my tooth into my brain. I know about the blood brain barrier but I kept thinking surely it is just a defense and not perfect and a bad infection could push its way in.

<comment><USER10> Infections in the facial area can indeed be quite serious:

[https://skeptics.stackexchange.com/questions/10674/can- poppi...](https://skeptics.stackexchange.com/questions/10674/can-popping-a- pimple-in-the-facial-danger-area-cause-brain-infection-or-death)

<comment><USER11> On that note, a warning:

Do NOT pluck your eyelashes or nose hairs, ever!! These hair roots can cause the trigeminal nerve to become infected which can then lead to meningitis/brain infection and ultimately retardism/coma/death.

Trim if you must -- and do not use a powered trimmer either! The so called "no-pull" nose hair trimmers still cut too short or pull hairs anyways, leading to infection.

Solution: just use mini cat nail scissors. They are actually perfect shape and size to cleave nose hairs at a good length without risk of infection.

If you appreciate quality Japanese craftsmanship that will last a lifetime, I recommend Cattyman. My cats and my nose swear by them. And it's the only brand we carry at the shelter that I volunteer at. [1]

[1] [https://amzn.to/2FkyiMv](https://amzn.to/2FkyiMv)

Source: I am a med student and graduated with honors in Biomedical Sciences

<comment><USER12> > These hair roots can cause the trigeminal nerve to become infected which can > then lead to meningitis/brain infection and ultimately retardism/coma/death.

Huh, that's alarming. Do you have any citations handy about the total relevant risk? Thanks.

<comment><USER13> Hm... I've seen this claim and the related claim that popping pimples in that area can cause fatal infections several times.

My first note is that I've never seen someone claim trigeminal nerve infection is the mechanism, and indeed trigeminal nerve infections appear to be exceedingly rare and not directly life-threatening. The relevant disorder would be trigeminal neuralgia, which causes severe and lasting pain, but is almost always a product of compression, inflammation, or demyelination of the nerve. I can't find any record of infection as a cause, only some hypothesizing that atypical forms of the disease might be based on infection. The only major connection I can find between hairplucking and that nerve is the claim that it's responsible for the sneezing reflex many people have when nose hairs are pulled.

As for "can plucking nose hairs kill you?" That's a bit more credible.

The mechanism here isn't nerve infection, it's infection of the veins in the area, which have unusually direct connections to the cavernous sinus and brain. This can lead to meningitis, brain abscesses, and disabling/fatal bloodclots, so I suspect it's the same thing described above. (It's worth noting that normal "sinus infections" can also have the same effect, but the direct bloodflow connection here threatens much faster escalation than normal sinusitis.)

On to rates, then. Meningitis, brain abscess, and cavernous sinus thrombosis (a blood clot in the cavernous sinus) are all exceedingly dangerous, so let's just worry about the rate of getting there.

The base rate of cavernous venous thrombosis is ~1 per 100,000 per year, mostly occurring alongside other major facial infections. Without strong reason to expect that you have heightened risk, it's largely irrelevant.

A 1937 study found that 61% of cases of cavernous sinus thrombosis (which is extremely dangerous) were a product of upper-face boils. Modern rates appear to be closer to 50% of cases, probably down to a mixture of dental surgery causing new cases and antibiotics preventing nasal-infection cases. (A venous infection progressing straight to brain abscess or meningitis appears to be possible, but unlikely without an intermediate infected-sinus stage.) Boils around the middle of the face are viewed as deserving prompt intervention to avoid this outcome.

Unfortunately, there's no clear data on how many of those cases were the result of hair-plucking. Staph infection can lead to boils at any irritated skin site, and are indeed the infectious agent in 70% of thrombosis cases. But severe boils are vastly more common in patients with specific risk factors (family history, immunosuppression, certain disease). Indeed most people who pluck hair never seem to develop boils. (N.b. boils are not the same as 'clean' inflammations from trapped hairs.) At-risk patients are also in far more danger from boils, since they face risks like immunosuppression and frequent treatment leading to drug resistance. One-time cases in healthy patients can be treated simply and effectively with antibiotics, steroids, and lancing.

So: plucking can lead to boils, and boils can lead to thrombosis, and thrombosis can lead to death. But any major infection from plucking appears to come after a noticeable period of surface infection at the site (i.e. a boil). Simple one-off boils are extremely treatable in the presence of modern antibiotics, so the risk can be arrested almost completely at that point as long as you seek prompt medical treatment. (Go to the doctor if you have a boil anywhere _anyway_ , because they rarely heal without treatment.)

As far as I can tell, this is one of those risks which is discussed because it's interesting and logical more than because it actually hurts anyone. "Don't pluck nose hairs" does not seem to be an important precaution unless you have other major risk factors like HIV or a drug-resistant staph infection.

[https://en.wikipedia.org/wiki/Danger_triangle_of_the_face](https://en.wikipedia.org/wiki/Danger_triangle_of_the_face)

[https://www.ncbi.nlm.nih.gov/books/NBK448177/](https://www.ncbi.nlm.nih.gov/books/NBK448177/)

[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1390530/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1390530/)

<comment><USER12> Well, this is pretty great analysis, thank you. My only suggestion is to put your last paragraph, the summary, right at the top so people are more likely to stop being afraid of this stuff. (:

<comment><USER13> Cheers, that's a good point. Too late to edit, but I tossed the summary in a followup comment.

Not sure how far I can go with Wikipedia, since I think even "I multiplied numbers from three sources together" is original research. Adding the annual frequency of the disease ought to be fair game though, and that's probably the most important part.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER14> I think we have all kinds of latent bacteria and viruses in our body (including the brain). It's however a interesting though that there might be _good_ brain bacteria, which might affect feelings, emotions, and alertness.

<comment><USER15> I already see my local dairy industry make money out of this: "Drink our new Brainimel, with the _best_ microbiome for your brain, so you can stay asleep, obey and consume..."

<comment><USER16> Where do you live that you have such an aggressive dairy industry?

<comment><USER15> You haven't heard of Hyperbole Town, have you? :)

<comment><USER16> I'm a very literal person.

</comment></comment></comment></comment></comment><comment><USER17> Seems to me that the more we sharpen our mental and methodological tools (in all areas of human inquiry), the weirder and more interesting things become.

Shit's messy, yo!

</comment><comment><USER18> Can't wait to order some "cerebral-probiotics 2000"

<comment><AUTHOR> We'll all laugh now but come the incessant infomercials (I'm guessing 2-3 years) and interstitials it won't be so funny.

</comment></comment><comment><USER19> It would be interesting to find out that a lot of modern psychological diseases might be caused by excessive antibiotic use disrupting the brain microbiome.

<comment><USER20> Can antibiotics pass the blood brain barrier?

<comment><USER21> Some do, including penicillin:

"Antibiotics that cross the BBB tend to have certain physical characteristics including: small molecular size, being lipophilic, binding to fatty molecules on cell membranes, low plasma protein binding and other specialized characteristics"

[http://lymemd.blogspot.com/2013/03/blood-brain- barrier.html?...](http://lymemd.blogspot.com/2013/03/blood-brain- barrier.html?m=1)

</comment></comment></comment><comment><USER22> A lot of people who started eating exclusively meat reported their depression, anxiety and (mental) fatigue going away, so this is should not be surprising.

[http://meatheals.com/](http://meatheals.com/)

[https://www.youtube.com/watch?v=jrF4aJvdU-A](https://www.youtube.com/watch?v=jrF4aJvdU-A)

</comment><comment><USER23> I can't believe you people haven't brought up the obvious zombie-gut microbiome connection yet...

</comment><comment><USER24> How is it possible we didn't notice anything like this before?

<comment><USER25> reminds me of [https://en.wikipedia.org/wiki/Archaea](https://en.wikipedia.org/wiki/Archaea)

Although these archaea can be present in extremely high numbers (up to 40% of the microbial biomass), almost none of these species have been isolated and studied in pure culture.

</comment></comment><comment><USER26> Wait a second, how did bacteria get through the blood-brain barrier?

<comment><USER27> Bacteria associated with meningitis get through with predictable results so what else might?

<comment><USER28> I do not know a lot about bacterial meningitis, but I don't believe that it actually passes through the barrier. Meningitis is caused by the inflammation of a membrane that the brain and spinal chord are encased within [1]. So it might be causing the membrane to become enflamed from the outward facing surface.

[1] [https://www.healthline.com/health/bacterial-meningitis- cause...](https://www.healthline.com/health/bacterial-meningitis-causes-and- how-they-re-spread#pictures)

</comment></comment></comment><comment><USER29> Marketers lickicking their chops for Yogurt for brain campaigns

</comment><comment><USER30> Incredible.

</comment><comment><USER31> What a great idea.

<comment><USER31> FWIW, that wasn't sarcasm.</comment></comment>


Reasons to Quit Twitter and Facebook and Switch to Mastodon - <AUTHOR> https://finertech.com/2019/03/21/reasons-to-quit-twitter-and-facebook-and-switch-to-mastodon/ <comment><USER3> >They are quickly banned from most instances (think of them like interconnected communities, or servers). The few instances that allow them are blocked by all the others. Toxic actors are effectively ostracized from the greater community.

Is banning and ostracizing for wrong think really the way we want to go? It works as long as you are on the politically correct side.

<comment><USER4> Nazis. We're talking about Nazis. And white supremacists. And "provacateours" or trolls or whatever they're called this week.

There was a time when the world didn't just ostracize people like this, the entire planet went to war to stop them. So, yeah. Silencing them on a social media site seems like a decent first step.

<comment><USER5> Nazis don't exist. They used to, before most of us were born, but they don't exist any more.

Of course those who want to censor will call their targets Nazis, and try to scare people like you by saying that the world will end if we don't censor them.

As far as I know, Facebook doesn't censor revolutionary Maoists or Khmir Rouge sympathizers, and their ideas are easily as dangerous, if any ideas are dangerous. No, we can work through this as a society.

Censoring everything is letting them win.

<comment><USER6> > Nazis don't exist. They used to, before most of us were born, but they don't > exist any more.

So what's the term for a bunch of white guys shouting "Jews will not replace us" while holding swastika, iron cross, and Nazi eagle symbols?

<comment><USER5> The term I would use is: 4chan users engaged in Live Action Role Playing. They were trying to spook you, and they did, apparently. The most they could muster from the entire world is < 200.

I remember when the media tried telling us about how "Anonymous" were "hackers on steroids" who treated things like a "real life video game".[1]

Yes, there are some creeps out there. But they are not a serious threat.

[1]: [https://www.youtube.com/watch?v=128IR21ZQa0](https://www.youtube.com/watch?v=128IR21ZQa0)

<comment><USER6> >The term I would use is: 4chan users engaged in Live Action Role Playing.

Role playing Nazis, by choice. And this wasn't a historical reenactment, or even a skit/play. This was an attempt at showing fear through symbology and crowd size.

You seem to sympathize with them in a very odd manor.

>They were trying to spook you, and they did, apparently.

LOL I see what you pathetically tried to do there.

Please explain, what's so friendly and cuddly about people who's ideology involves extermination of other humans.

>The most they could muster from the entire world is < 200

Not sure what you mean, feel free to clarify.

>I remember when the media tried telling us about how "Anonymous" were "hackers on steroids" who treated things like a "real life video game"

So because one news outlet was wrong one time, we should now disregard all reporting from any outlet till the end of time. Got it.

>Yes, there are some creeps out there. But they are not a serious threat.

You are an expert on what defines a serious threat now too! So in your clearly expert opinion (/s) what is the definition of a "serious threat?"

</comment></comment></comment></comment></comment></comment><comment><USER7> I set up a private Mastodon server for my family several weeks ago and we're loving it. It's a mighty impressive Rails/React app, very well designed and implemented. I have a few minor UI quibbles based on experience with twitter, but the design is really good, responsive, and snappy.

The kids run Fedilab on hand-me-down, wifi-only Android phones and just go to town with emojis, gifs, and photos. The parents share more of their days with the kids rather than only texting each other.

The setup guide is great--I followed it on a 1GB Vultr VPS and was up and running easily, except for one maddening hurdle, which is that on 1GB server you do need a swap file to successfully precompile JS assets. See [1] for the workaround. I guess this is just normal in 2019. Otherwise all the services run fine with 1GB once compiled.

I wish it supported locking down public posts at the admin level, but that's just my use case. The login system blocks most access, and public posts can be blocked with nginx authentication and tweaking the list of streams flagged public in the streaming service.

[1] [https://github.com/tootsuite/mastodon/issues/5836#issuecomme...](https://github.com/tootsuite/mastodon/issues/5836#issuecomment-352225419)

<comment><USER7> Ah, they have a card in progress for a one-click Digital Ocean install [1]. Fantastic! This is an incredible open source project to watch, not your typical library or system service, but a real, polished app.

Polls functionality is in master, just waiting for release. My kids are going to love this.

[1] - [https://github.com/tootsuite/mastodon/projects/8#card-185452...](https://github.com/tootsuite/mastodon/projects/8#card-18545229)

</comment></comment><comment><USER8> THe problem with Mastodon is a) discoverability and b) Twitter-like UI. Both of which I find hard enough to not bother with it at all. (disclaimer - <USER2> never used Twitter due to what I think is a horrible UI) I registered in Mastodon long time ago, when first articles appeared, followed some random maybe interesting people, tried writing a few posts and just dropped. I still have client installed on my phone but last time I opened it was half a year ago. And I use other social networks daily.

Instead of writing Twitter clone they should have done a LiveJournal of Facebook clone, with better interface, different approach to privacy and discoverability etc. But being Twitter-like it is simply not that interesting.

<comment><USER9> Mastodon is only a client to the ActivityPub protocol. There are other clients that look more like Facebook.

But yes discoverability is an issue because it's hard in a federated environment.

<comment><USER9> Actually StatusNet had a good solution for discoverability groups. An equivalent is an outstanding issue for ActtivityPub [https://github.com/w3c/activitypub/issues/328](https://github.com/w3c/activitypub/issues/328)

</comment></comment></comment><comment><USER10> Admittedly, I spent next to no real time investigating, but I did sign up and attempt to use it after some friends recommended it. There is no way any normal non-tech savvy person is going to go through that many hoops. I also couldn't figure out how to search other servers with any success. So unless I bring people I want to connect with into a server, I'm not grasping what good it's going to do as a social tool.

<comment><USER11> +1 to this, searching & browsing across instances is a major pain point. To me, joining a Mastodon instance has felt way more like joining a forum than getting on a federated social network.

I still like the instance I joined, though. Really nice people. Check out fosstodon.org if you're interested :)

</comment></comment><comment><USER12> one of the coolest things about mastodon and pleroma is that you can just start your own instance and immediately federate with the rest of the network.

this means that if youre unhappy with the moderation in a community, if you want to own your own data/privacy, or if you want to use self-hosted software, you can just do that with mastodon.

a few weeks back i went through a pleroma guide and set up my own pleroma instance. it only took a few hours, and it has been running for over a month on a $5 Linode (1 GB RAM). i migrated from mastodon.social, and it carried over everyone i was following seamlessly.

i get that the whole decentralized thing has been blown out of proportion in the past, but in the fediverse (mastodon, pleroma, GNU social, etc) it is working quite well.

</comment><comment><USER13> Never heard about Mastodon, it'd have been nice if the article explained what it is. Is this yet another startup promising they won't become a merciless corporation? Great value prop, lol

<comment><USER14> Technical answer:

Mastodon isn't a startup - it's not even a company. It's a _framework_ , like phpBB forums were (are, I guess). Basically, it's a way that _anyone_ with a web server can set up a Twitter-clone site, but - thanks to the ActivityPub protocol - all those twitter-clones can _talk to each other_ , making the whole universe of Mastodon instances functionally one giant social network.

Mastodon:Twitter::IRC:Facebook Messenger

Practical answer:

Mastodon is a Twitter clone that's currently pretty nice and enjoyable by virtue of being small. It's also decentralized, which is a cool buzzword, but also in practice means that it's nicely broken up into a bunch of small specialized communities in each Mastodon instance that can feel pretty cozy and friendly, way less overwhelming than Twitter.

The _really_ unique thing about it is that it's a weirdo open-source-y alternative social network that actually _doesn 't_ appear to be dying off like Diaspora or Voat or Steemit or whatever did - it seems to have reached a sort of social critical mass where people are sticking around because they genuinely enjoy the company and interactions with other users, and is growing at a sustainable rate.

<comment><USER15> Would enjoy an article that goes into more of these details. They have a pretty good video on it:

[https://www.youtube.com/watch?time_continue=1&v=IPSbNdBmWKE](https://www.youtube.com/watch?time_continue=1&v=IPSbNdBmWKE) [https://joinmastodon.org/](https://joinmastodon.org/)

</comment></comment></comment><comment><USER16> Will take a look, I'm in the alpha of Openbook.social ATM</comment>


Ask HN: How Does Instant Bank Verification (IBV) Work? - <AUTHOR> For example, on Venmo, you can link your bank account through IBV by selecting your institution and logging in. Once you do that, Venmo can pull money and put money back in there. But how does that actually work under the hood? <comment><USER2> A horrendous abomination of scraping, fake browsers, storing your credentials, probably. Probably also grab your transaction data while there. This seems to be par for the course for US accounts, unfortunately.

See: [https://news.ycombinator.com/item?id=20135693](https://news.ycombinator.com/item?id=20135693) [https://news.ycombinator.com/item?id=20134031](https://news.ycombinator.com/item?id=20134031) [https://news.ycombinator.com/item?id=20133882](https://news.ycombinator.com/item?id=20133882)</comment>