Digital dystopia: how algorithms punish the poor - <AUTHOR> https://www.theguardian.com/technology/2019/oct/14/automating-poverty-algorithms-punish-poor <comment><USER3> > Automating Poverty will run all week. If you have a story to tell about > being on the receiving end of the new digital dystopia, email > firstname.lastname@example.com

This is dangourous journalism. The reporter has clearly taken a stance and is not attempting to open up a nuanced and careful discussion of the factors at play here but is instead insighting fear and mistrust of technical progress in the public sphere.

I can understand the fear and concern around the changing digital landscapes and the potential impact on different parts of society. What's needed is a public discourse about these things that doesn't conflate the issues into one single problem of a "digital dystopia". Are we really talking about a "flawed algorithm", for example, or simply the encoding of a badly designed government process?

Technology _is_ changing how poor people interact with the state and this is an important topic to discuss openly and broadly. We'll never have the quality of discussion that's needed while there are fear mongering reporters like this subverting that conversation to the level of luddism.

<comment><USER4> I go back and forth on the value of a "devil's advocate". I hope someone else will chime in to help me out if I don't explain this eloquently.

This is a case where I believe the topic won't be properly discussed unless someone is willing to take a hard stance on the negative side. Someone has to point out all the potential failings of the system that so many people are pouring their lives into building. Even if that person ends up taking a much harder stance than they actually believe in, and becoming a little too disconnected from reality.

The situations being described in the article are horrifying, and sound an awful lot like what you might expect from software bugs in their early stages. That wouldn't be a problem if there was an appropriate human force behind the systems going online, but we've all seen stakeholders push systems into production before they're ready without adequate support.

You're probably on the right track to call it a "badly designed government process", but I'd wager the human element is what softened the blow from that bureaucracy. It shifts the burden of proof from the case worker to the support seeker. It changes the conversation from "You think you deserve benefits? Let's look over your evidence" to "You've already been rejected, why should we support you?". When you're talking about people surviving, that's a significant difference.

<comment><USER5> > _Even if that person ends up taking a much harder stance than they actually > believe in, and becoming a little too disconnected from reality._

Sure, that will get the issues discussed - discussed and then promptly rejected. A bad arguer arguing for the right side can do a lot of damage. For example, imagine how successful climate change skeptics would be if someone got in to the news by saying that New York was a month away from flooding.

</comment></comment></comment><comment><USER6> It's a bit like all the arbitrary account terminations we hear about with Google. A machine makes a decision with virtually no recourse for the user. When governments are applying this same concept to welfare, it does indeed create a dystopian scenario where people are left to starve because some algorithm got it wrong.

In essence, it's about cost cutting. Governments seem to think that machines can replace humans in making decisions where there is nuance involved and the stakes are extremely high. It is frightening and should not be accepted. The furthest the automation should go is to flag irregularities for review. Instead, the machines are given far too much autonomy, with the robodebt collection being a scary example.

<comment><USER7> A human isn't necessarily less arbitrary, in fact it can be much worse.

You (and Guardian, it seems) are falling for the fallacy of the danger of automation - "it's acceptable for bad decisions to be made as long as it's been decided by a human".

Fear mongering is not how you prove/disprove automation, you should pick a useful metric (e.g. are people starving?) and benchmark against that - <USER2> same way you would measure if a department comprised of humans were doing the same job.

<comment><USER8> > You (and Guardian, it seems) are falling for the fallacy of the danger of > automation - "it's acceptable for bad decisions to be made as long as it's > been decided by a human".

Not necessarily. Even bureaucratic systems tend to recognize that humans make imperfect decisions -- we're susceptible to everything from bribery to exhaustion. These systems then tend to not treat first decisions as final, and they allow an opportunity to appeal.

But what happens when the decision is made by an algorithm, especially one that wasn't built to give an explicable reason?

> Fear mongering is not how you prove/disprove automation, you should pick a > useful metric (e.g. are people starving?)

That's not a good metric.

Suppose the algorithm were in fact perfect and nobody starved -- except you. For some reason, it couldn't recognize your ID (and only your ID). By the "are people starving" metric, the algorithm would be doing a fantastic job compared to an imperfect human system, but it would also be profoundly unjust.

<comment><USER7> > Even bureaucratic systems tend to recognize that humans make imperfect > decisions -- we're susceptible to everything from bribery to exhaustion. > These systems then tend to not treat first decisions as final, and they > allow an opportunity to appeal.

So the problem is not automation, is lack of recourse.

Lack of recourse is a real problem that already exists today. People do go unattended by not knowing how to navigate bureaucratic structures, and not having money to pay a lawyer. It also affects the most the countries the Guardian article targets as "automating poverty".

> For some reason, it couldn't recognize your ID (and only your ID). By the > "are people starving" metric, the algorithm

This is not an algorithmic issue.

If you lost your ID, you wouldn't get your benefit even if you walked into a social security branch either.

<comment><USER9> > So the problem is not automation, is lack of recourse.

The problem _is_ automation, if it exaggerates the "lack of recourse". It means that, without being careful, adding automation would screw up most of such systems. Thus such articles to boost awareness and make sure automation is done well. I'm disappointed with what a hard time many on HN seem to have with accepting the need to be cautious. Especially with Gov, were you cannot just find or build a better alternative on the open market.

> If you lost your ID, you wouldn't get your benefit even if you walked into a > social security branch either.

Is that actually the case in the US? The programs (run by the church and heavily supported by german gov) I've come in contact with certainly did not. Never was at a soup kitchen, but would be surprised for them to ask for IDs.

<comment><USER10> I don't think you need to convince anyone here that automated systems fail. I think many of us have been affected by such systems. Accounts blocked for no reason, etc. Glitch.

The problem is not the automated systems. The problem is _zero_ recourse.

If nothing was automated then we'd have to wait hours on the phone to get to an actual human to do something trivial. Or make an appointment, travel somewhere, spend few hours to get something simple done.

In an automated world most of the time things work out.

The situation becomes despicable where there's no one to complain to and no way to seek redress.

In the tech world it became pretty common to seek support by Twitter shaming. Often there's no other route.

Still, it's not the automation that is at fault. It's the greedy humans. They could save 80% of the cost while keeping everyone happy but they'll chose to save 90% of the cost at the price of making many miserable.

Can't change the human nature?

</comment></comment></comment></comment></comment></comment><comment><USER11> This is just fearmongering; misrepresenting and oversimplifying the truth to the point where all meaning is lost.

\- There's no distinction between the different types of systems involved.

\- Human problems are blamed on the system (A lack of human support isn't the programs fault)

\- Individual stories are used not to underline broader evidence, but meant to _be_ the evidence ("a man died in india, therefore all computers are evil").

\- There's constant generalizations. It's not "some algorithms", it's "the algorithms" that are being blamed. (Ctrl+F shows the word "some" appears just once in the text FFS)

I don't know if the author is satisfied with this article; if they are, they should be fired. Misinformation like this is ammunition for those crying "fake news" to further destabilize the political landscape, which, in turn, harms the reputation of real journalism, the kind that our societies rely on to maintain a functioning democracy.

<comment><USER12> > \- Human problems are blamed on the system (A lack of human support isn't > the programs fault)

Oh, you can sue programs now? I wasn't aware of that!

I mean ... seriously? What are you even trying to argue there?!

</comment></comment><comment><USER13> Two books that dived well into this subject. Unlike this article they have a bit more meat to their research. I would recommend giving them a read before getting your pitchforks only due to this article. It's bad and needs to be fixed but a measured response instead of a purely reactionary response will just make things worse.

Weapons of Math Destruction [https://www.goodreads.com/book/show/28186015-weapons-of- math...](https://www.goodreads.com/book/show/28186015-weapons-of-math- destruction)

Automating Inequality [https://www.goodreads.com/en/book/show/34964830-automating-i...](https://www.goodreads.com/en/book/show/34964830-automating- inequality)

<comment><USER14> > a measured response instead of a purely reactionary response will just make > things worse.

I'm not sure I understand, would you clarify this for me? It's been my experience that pure reaction is usually bad for discourse, so I'd like to know more.

<comment><USER15> Going from the context of the comment, I believe parent meant something along the lines of:

"we need a measured response instead of a purely reactionary response which will just make things work"

At least, that was my first impression, definitely could be interpreting the comment incorrectly.

</comment></comment></comment><comment><USER16> There's an interesting assumption baked into this entire approach, which is that the human-evaluated system is better. I'll be interested to see if that's an artifact of this summary article or is reflected in the underlying reporting coming up, because it's wise to not leave that assumption unchallenged. FTA:

> Instead of talking to a caseworker who personally assesses your needs, you > now are channeled online where predictive analytics will assign you a future > risk score and an algorithm decide your fate.

If the caseworker is racist, this isn't a worse scenario... unless such racism has also been baked into the risk assessment algorithm, which is definitely a possibility. But at least from my personal experience, I trust our ability to identify and de-train racism from assessment networks more than I do our ability to consistently identify and hire non-racist caseworkers. Deprogramming racism out of humans is hard (and even if you succeed for one, you can't copy their state-vector into the brains of their peers).

<comment><USER17> I recall reading a story in the book _Weapons of Math Destruction_ [1] about a system used to assess people's recidivism risk which judges were relying on for sentencing hearings. The problem is that the system showed a clear racial disparity in the risk scores yet at the same time it was more accurate.

That means if we try to tune the system to make it less racist, we'll be making it less accurate. In essence, the system isn't really racist, it's a reflection of the racism in society which is leading to these outcomes. Ultimately, the problem is that putting someone in jail increases the likelihood that their relatives will commit crimes. It increases the likelihood that both they and their friends and family will reoffend. It's a vicious cycle and it doesn't appear to have any technical solution.

[1] [https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction](https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction)

<comment><USER18> It brings to mind a sarcastic way to get perfect prediction of patient outcomes. Step one is to decapitate the patient.

Predictability/accuracy isn't the important part except as means to the end - the outcome.

</comment></comment></comment><comment><USER19> This article is blaming technology. This is very dangerous because it diverts the blame from who it should be on and that's the governments for making bad decisions.

<comment><USER17> The whole point of switching to technology for systems like this is to block people from holding you accountable. It puts the burden of proof on the accuser to show that an inscrutable software black box is biased.

It very much resembles the shift from human tech support operators to software-based robo operators. In effect, it's a wall to keep you away.

The apotheosis of this concept is Google, a company that goes to ridiculous lengths to stop people from being able to contact a human being that works there, unless you've paid them money for the privilege.

</comment></comment><comment><USER20> Many here seem to cite this article as being low on fact, and high on emotions.

It is worth noting that this is merely the introduction piece to a series of articles that the Guardian seem to be running this week[1].

It's intention _is_ to be brief.

[1][https://www.theguardian.com/technology/series/automating- pov...](https://www.theguardian.com/technology/series/automating-poverty)

</comment><comment><USER21> This article miss the point, misrepresents and oversimplifies, it's not denouncing the culprits, it's dancing to their tune. This is what they want: "blame the algorithm, forget the real culprits".

</comment><comment><USER22> It tells you that a wrongly applied system can cause damage.

In respect of Aadhar in India, I know the Anti pattern where they deduplicated many undeserving people receiving 'welfare' payouts from the state.

Is this to state that either of these approaches are right?

No. Eliminate the skunk out of the politics, and such things won't happen.

<comment><USER23> I'm not sure what welfare payouts you're referring to; the Indian state mostly assists via subsidies rather than cash. Lately they have moved to direct benefits transfer (ie a payment into a bank account) for the cooking gas subsidy, which is an anti-corruption move meant to route around intermediaries who divert subsidized cylinders and sell them at market rate.

The problem is that Aadhar (introduced initially for subsidies) has become a biometric ID system to link more and more of a person's life to this government ID. A few weeks ago a child died of rabies and the first excuse the hospital gave to delay treatment was that they wanted her Aadhar: [https://timesofindia.indiatimes.com/city/agra/rabies- infecte...](https://timesofindia.indiatimes.com/city/agra/rabies-infected- minor-who-died-waiting-for-treatment-denied-vaccine-15-times-at- chc/articleshow/70832635.cms)

I am disappointed whenever I see Bill Gates mindlessly applauding Aadhar. It is easy to side with technocratic ideas from just reading headlines, but if a biometric surveillance state is such a great idea he should have it advocated for it in the US first. (Another example of a technocratic authoritarian move he initially made some positive murmurs about was demonetization, which now, three years later, has proven to have had a disastrous effect on India's economy.)

<comment><USER24> > I am disappointed whenever I see Bill Gates mindlessly applauding Aadhar. It > is easy to side with technocratic ideas from just reading headlines, but if > a biometric surveillance state is such a great idea he should have it > advocated for it in the US first.

I agree with this. It's not acceptable to use the developing world as a staging box for ideas and policies that are, at best, in beta and likely to have bugs.

<comment><USER5> India is a democracy that votes for its own laws, nobody is using it as a "staging box."

</comment></comment></comment></comment><comment><USER25> First six paragraphs offer no facts, but are very heavy with very emotionally loaded statements. Even if the whole article after that is completely true, it's impossible not to perceive this writing style as manipulative propaganda.

<comment><USER26> Reporting can be emotional and true. Journalism and activism can't be separated.

<comment><USER27> The article belongs in an Opinion section, not in a (Technology) News section.

 _> Journalism and activism can't be separated._

They should and they can. Either have a separate publication, or at the very least have a separate, clearly delineated "Opinion" section. Otherwise you run the risk of media trust becoming very low, and thus media losing its ability to guard the republic (or the democracy).

Consider a similar assertion - "journalism and advertising can't be separated" \- it's obviously false. Granted, there is some undue influence from advertisers to be expected. However this is considered a negative that should be avoided. And indeed the good practice is to have a "firewall" between the advertising department, and the news desk, in any publication.

<comment><USER24> > They should and they can. Either have a separate publication, or at the very > least have a separate, clearly delineated "Opinion" section. Otherwise you > run the risk of media trust becoming very low, and thus media losing its > ability to guard the republic (or the democracy).

Practically it is impossible to separate the two. Even news reporting doesn't usually come free of editorializing. There are places where you can go to get 'just the facts ma'am' but newspapers have never and will never be those places.

<comment><USER16> I agree that the mere act of choosing to observe or report something is, in some sense, activism, at least in that it indicates the author thinks the topic has some merit. But it's a sliding scale.

(1) And an author can identify words designed to elicit emotional response and work to minimize them in their writing if they want to focus on the facts vs. their interpretation of the facts.

(2) Listen friend, only an idiot doesn't understand words and phrases have emotional tone, and an author chooses whether they want to piss you off or stay out of your emotions when they choose their words.

... these two sentences have the same conceptual content.

</comment></comment></comment></comment></comment><comment><USER28> I, Daniel Blake

watch it</comment>


Nevada cops sued over forced occupation of private homes - <AUTHOR> http://rt.com/usa/nevada-third-amendment-lawsuit-710/

<comment><USER2> See also this excellent article at The Volokh Conspiracy:

A Real Live Third Amendment Case:

[http://www.volokh.com/2013/07/04/a-real-live-third- amendment...](http://www.volokh.com/2013/07/04/a-real-live-third-amendment- case)

</comment><comment><USER3> According to the local paper, the guy the cops were trying to get a "tactical advantage" over had all charges dismissed with prejudice.

[http://www.reviewjournal.com/news/crime-courts/henderson- fam...](http://www.reviewjournal.com/news/crime-courts/henderson-family-sues- police-breaking-their-homes)

</comment><comment><USER4> It seems to me the cops would have given away any tactical advantage they may have when they "loudly commanded" they be let inside".

Not that I deny the Mitchell's have a case, but I suspect there must be something else to this.

<comment><USER5> Yes - sending a clear message to other people who refuse to assist them in the future.</comment></comment>


Online video sceptic Mark Cuban invests in web video company Revision3 - <AUTHOR> http://venturebeat.com/2011/02/01/cuban-invests-in-revision3/

<comment><USER2> I don't think he was ever really an "online video skeptic". Mark wrote an article called "The future of tv is tv" where he challenged the _delivery_ of online video, not the existence of it. The masses don't want to hook up a Google/Apple TV box to their TV and fuss with it to make it work. That technology should be built in like with the Sony/Google TV. Tangent - I'd expect to see an Apple HDTV come out some time soon. I wrote an article on my startup's blog about this a while back.

</comment><comment><USER3> Where as Digg, Diggnation and Revision 3 founder Kevin Rose. Is not using Rev3 for his new podcast Foundation or his other new show "The Random Show".

The reason Kevin says for charging for Foundation is "I'm spending money on equipment, and editing"..

It all sounds silly to me, and shows just keep disappearing.

<comment><USER4> I see this happening with This Week In. When it first launched it had alot of fanfare. Initially it had only a few shows and was pretty straightfoward. But then they launched like 20 shows and it became noise with some of them being cancelled within a few airings with no reason given at all. I wonder if it will in fact be around in another year.</comment></comment>


PlotKit - Javascript Chart Plotting - <AUTHOR> http://www.liquidx.net/plotkit/ <comment><USER2> PlotKit is three years old and doesn't look to have been maintained since then. Why did it show up here?

<comment><USER3> I was going to say something along similar lines when I saw that it had been tested against Firefox 1.5...

</comment></comment><comment><USER4> Another one is flot for jquery.

</comment><comment><USER5> I've had such mixed results with the Javascript graphing libraries, Plotkit included. I always end up on a more traditional platform.

<comment><USER6> Almost the same here, I always found it difficult to do what I wanted with these frameworks. I ended writing my own custom javascript graphing code.

I realized it is not that hard to do, you can get nice graphs out quickly without too much code. I also found teh resulting code to be simpler than the one from libraries. I suspect that a lot of the complexity comes from making things generic, or maybe (in some cases) the authors being unfamiliar with javascript?

</comment></comment><comment><USER7> Also Flotr - <http://solutoire.com/flotr/> which is like Flot but using Prototype instead of jQuery. As such can be used in a Rails project more easily.

</comment><comment><USER8> See also Bluff, a JavaScript port of the Gruff graphing library for Ruby.

<http://bluff.jcoglan.com/></comment>


Why Tony Stark is better than you - <AUTHOR> http://julio-ody.tumblr.com/post/552745613/why-tony-stark-is-better-than-you <comment><USER2> Answer: Namely, because he is a fictitious character.

The article writers contention that programmers should also have solid HCI knowledge is not necessarily unreasonable, but the comparison to "2 meetings a week, and in a year..." is just ridiculous. Perhaps it was intended to be over the top, but division of labor exists for a good reason. What we really need are amazing back-end and interface engineers who _can work with eachother effectively_. That, and, as we are seeing with the iPhone becoming mainstream, a cultural willingness to put in extra work on the UI because _it matters_.

</comment><comment><USER3> I sorta fancy myself as a do-it-all. I do the design, write the backend, sell it to the client, do the testing, receive the feedback etc. I think it comes from the fact that I've been doing all of this for almost two decades now. I have spent three times more time fiddling in Paint Shop Pro than professional graphic artists in Photoshop. I may not be able to make designs as pretty as 280North but that doesn't stop me from giving it my best.

I've also spent more time and energy configuring Apache and various *nix apps than most sysadmins. But I don't feel comfortable blogging about "Top 5 Unix commands you must know" because I always feel there are 100 others who know more than me so I'll let them blog on these topics. I'll blog about what I'm good at or my own experiences. Which brings me back to the do-it-all topic - if it needs to be done, I want to learn how to do it and do it well. I delegate and outsource when I have resources available but if not, I just do whatever needs to be done. Heinlein's quote seems pretty appropriate here:

"A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects."

<comment><USER4> That is an excellent quote, although it's given me a few more things to strive for in my life. (butcher a hog, set a bone, write a sonnet, die gallantly)

</comment></comment><comment><USER5> _Stop believing Photoshop, Pixelmator, Illustrator, or anything else is too hard. These tools can have a steep learning curve (not as bad as you'd think), but so did programming when you were starting on this game._

Knowing how the tools and palettes work in Photoshop makes you a good designer like knowing Java's syntax makes you a good programmer.

<comment><USER6> Ask someone who doesn't know how to program to read just a few lines of code. Though it's very naturally readable to you -- and in general, if you didn't know a new language you could try reading it line-by-line like English -- it looks totally impenetrable to them. I've heard people tell me that they have no idea how I do anything [programming] when they look at just a simple script in PHP or Ruby.

Knowing the Photoshop interface is clearly not what makes you a good designer, but in general it is a necessary-but-not-sufficient prerequisite to being really good.

</comment></comment><comment><USER7> I wish I was better at design. On the other hand there's this wonderful principle called division of labor, and while I might spend 10 hours designing a mediocre website, a good designer will design a great one in half the time. This has actually happened to me. After spending a weekend building out some pages that I thought looked pretty good, my designer friend sent me a photoshop mockup that made me decide to totally scrap what I had made, and go with her design instead.

The most productive I've ever been has been while working in tandem with a good designer, because it frees me up to do what I do best, and that's write code.

Now, I will say that it's important for any web developer to understand principles of design, because understanding what your users actually want is crucial for both backend and frontend work, thinking about user interaction, your data model, etc.

<comment><USER8> Exactly....if one followed the OP's advice, I suppose maybe eventually I could get skilled in design and layout, but as far as I can tell, that is an entire profession in itself, I have enough trouble trying to stay current just on the pure programming stuff. The hours I've spent trying to get something to lay out properly across browsers using CSS could surely have been spent better elsewhere.

<comment><USER9> I don't personally know many people who understand all of the intricacies of CSS and browser support and writing HTML along with it that _aren't_ programmers.

To me, the design and the implementation are two very different things, and advanced CSS seems to be more in the programming realm (despite its goals to be accessible to the average designer).

</comment></comment></comment><comment><USER10> Ironically, your 3-column blog design is not very friendly, IMO. I had to scroll up and down to read that post.

<comment><USER11> When I got to the scroll up part of the post, I stopped reading...

Looks very nice at a glance, but fails its usability tests. ;)

<comment><USER12> Clearly one of those people who have every window maximised to take up the whole screen. Like people who wear white sneakers, do not trust them.

</comment></comment></comment><comment><USER13> I think this article is a perfect example of a what I've perceived as a growing trend of "developers" working at the "top of the stack" with no idea what is really going on under the hood.

That infact there are many, many, many developers who work on projects that are enormous that never see "end users" in the, guy who owns an iMac, sense of the word.

The implication that anyone who writes great software, but lacks "design" skills (in the graphical sense of the word) is so crazy... I wonder if this guy has ever thought about the guy who wrote the embedded realtime system in his car that runs his anti-lock breaks, no WOW factor there.

EDIT: never have I ever wished I had enough karma to vote something down before.

</comment><comment><USER14> Yeah but Superman's better than Tony Stark and he can't design anything, so this argument is entirely invalidated.

<comment><USER15> Dude. You're NOT seriously going to TRY start a DC vs. Marvel comic character thrown-down on HN?!

I pity 'da fool.

<comment><USER10> Well, there's no chance that Iron Man could defeat Superman in a head-to-head unless he uses kryptonite. OTOH, Superman vs Hyperion...

</comment></comment></comment><comment><USER16> I believe it would be good to look at stuff that is considered well-designed, analyze it and try to understand why it's considered to be well-designed before randomly coming up with stuff in Photoshop.

</comment><comment><USER17> WOW, so many commenters are complaining about the blog design, instead of the content of the post.

People should stop being so serious. People should take a look at things from a different perspective, and gain positive values from it.

</comment><comment><USER18> "it's ok for me to be ignorant about how to lay down a good interface, because I'm a right side of the brain kind of person."

I think he meant "left side of the brain". Does he ?

</comment><comment><USER19> Yes, because I so want to compare a fictional characters life to my own.

</comment><comment><USER20> This blog post is worthless shit.

Edit: let me clarify. This blog post is written by an infantile, narcissistic know-it-all and exudes the odor of vomit and feces. The fact that it appears here lowers my esteem of HN and further confirms my suspicion that the community is full of daft fight club-wannabe group-thinkers. Fuck you very much.</comment>


Vast Freshwater Reserves Found Beneath the Oceans - <AUTHOR> http://www.sciencedaily.com/releases/2013/12/131208085304.htm <comment><USER3> Regarding criticism that this isn't renewable and delays the move to renewable resources:

At the rate we are damming rivers, we will destroy many river ecosystems. This new potable water source seems less harmful than dams and could be a bridge until the energy costs of desalination are low enough.

If this discovery spurs investment in desalination (since this is simply cheaper desalination due to the lower salinity) that would be an added benefit as it would accelerate the move toward desalination.

Perhaps we could skip the dams entirely (which though seemingly renewable from a water perspective, are highly destructive to river ecosystems).

<comment><USER4> >until the energy costs of desalination are low enough.

Really? You think desalination is the ultimate goal? I much prefer free, ecosystem-provided desalination (aka rooftop rain-water harvesting).

As for agricultural water use, I highly suggest picking up a copy of P.A. Yeoman's classic _Water For Every Farm_. It details how proper runoff management, evaporation control, and correctly-scaled earthworks can provide reliable water availability with near-zero energy inputs [other than the sun, of course].

As for dams, it's not as simple as "all dams are bad". Oversized dams that are out of scale with the local hydrology are the problem.

But, I guess some people would prefer to re-invent the Earth's water cycle!

<comment><USER3> I agree that not all dams are bad. The real issue with dams is the scale. We are on track to dam the majority of the world's rivers over the next 50 years, and this will have significant effects on river and coastal ecosystems.

>> I much prefer free, ecosystem-provided desalination (aka rooftop rain-water harvesting).

I agree; however, this is not an infinite supply. Taking water from rivers is effectively a form of this and it is lower cost than many other "free, ecosystem-provided desalination" since the water has already been collected into a river due to gravity and land contours. As we remove water from these 'free' sources, we are depriving an ecosystem of that water.

The worst case scenario is something like the Aral Sea [http://en.wikipedia.org/wiki/Aral_Sea](http://en.wikipedia.org/wiki/Aral_Sea)

<comment><USER4> >The real issue with dams is the scale.

I couldn't agree more.

>I agree; however, this is not an infinite supply.

Well no system can provide an _infinite_ supply, but rainwater harvesting could support the current human population at a first-world standard of living (something our present system does not provide). Many areas famous for recent crippling droughts suffered from _flooding_ only months before. This is a spectacular failure in water management.

As to cost, I haven't seen any comparisons of the lifetime cost of rooftop rainwater harvesting vs centralized pumped underground pipe distribution. I know rainwater harvesting is much more common in Australia, so they might have more data. I suspect it's strongly dependent on the cost of energy.

One thing I forgot to mention about desalination: current technology (specifically the Affordable Desalination Collaboration's pilot plant) is at 30% of the thermodynamic limit. So we can expect a 3x improvement in performance, but no more. [http://www.usbr.gov/research/AWT/energy_use.html](http://www.usbr.gov/research/AWT/energy_use.html)

</comment></comment></comment></comment><comment><USER5> Good, now we can put off thinking about conservation yet again! Our great- grandchildren will have plenty of time to think of something.

<comment><USER6> Carbon credits today, water credits tomorrow.

<comment><USER7> Soon to be followed by shale fracking for water.

</comment></comment></comment><comment><USER8> We're contemplating going after harder-to-get fossil water because the easy stuff is depleted and can no longer keep up. Sounds like the situation we're in with oil.

In the case of water, which replenishes, the 'production cycle' is actually the global fresh-water cycle. If we dirty all of the fresh water faster than the earth can clean it, we're going to have to start drilling for the fossil stuff in hard-to-reach places. It's strange to think there would be drilling rigs in the ocean with people risking their lives and taking danger pay... for water. It raises a question in my mind: affordability. Who is going to be able to afford to buy all their water like this?

It's the next step now that there are several major rivers no longer reaching the ocean.

<comment><USER9> > Who is going to be able to afford to buy all their water like this?

The price of water can never really climb above the price of desalination.

</comment></comment><comment><USER10> hmmm, the ocean's pretty heavy. i wonder what extracting an incompressible fluid from the seafloor's support structure will do to weaken it. will they fill it with saltwater instead?

i guess it shouldnt be much different than oil drilling :\

<comment><USER11> This might be off-topic but water is in-fact compressible: [http://en.wikipedia.org/wiki/Properties_of_water#Compressibi...](http://en.wikipedia.org/wiki/Properties_of_water#Compressibility)

<comment><USER12> Getting further afield, water's density actually changes considerably with a change in temperature. [http://en.wikipedia.org/wiki/Properties_of_water#Density_of_...](http://en.wikipedia.org/wiki/Properties_of_water#Density_of_water_and_ice)

I'm not sure how common this is but I know Lake Michigan goes through an annual cycle where the water alternately mixes and segregates (by temperature). In the summer, the warm, less dense water rises to the top and stays there. In the winter, very cold water is actually lighter than slightly warmer water and a similar stratification happens. In spring and autumn the water temperatures near the surface equalize with water deeper in the lake and much if not all of the lake water circulates ("turns over").

[http://www.lakeaccess.org/ecology/lakeecologyprim4.html](http://www.lakeaccess.org/ecology/lakeecologyprim4.html)

<comment><USER13> Very common - this is one of the basic mechanisms that mix lakes. To clarify, the summer surface waters are heated by the sun, causing stratification. This is usually broken down in the fall by the combination of wind mixing and heat extraction (leading to denser surface waters which sink). This is an important component of ocean surface dynamics as well. The mechanism just described can mix the top few hundred meters of the North Atlantic in winter.

</comment></comment></comment></comment><comment><USER14> Newsflash - vast freshwater resources had been found sitting in form of ice on planetary caps. Those vast supplies of water are expected to last till 2030.

On a serious note, this could be big. Imagine oil like drilling platforms that pump water instead. Now we only need to figure out how economical this is.

</comment><comment><USER15> I guess there really is water at the bottom of the ocean.

</comment><comment><USER16> Once in a lifetime, water flowing underground.

</comment><comment><USER17> Govt is the cause of water shortages

[http://www.lewrockwell.com/2013/10/ryan-mcmaken/socialist- wa...](http://www.lewrockwell.com/2013/10/ryan-mcmaken/socialist-water- troubles/)

</comment><comment><USER18> Well, this is about the time we see crazy theories about some crazy civilizations living inside the planet with all this fresh water and heat freely available to them....

</comment><comment><USER19> Getting the water out of reserves beneath the ocean sounds expensive - <USER2> wonder how it stacks up energy wise vs desalination of ocean water.</comment>


Predictive Scaling for Amazon EC2, Powered by Machine Learning - <AUTHOR> https://aws.amazon.com/blogs/aws/new-predictive-scaling-for-ec2-powered-by-machine-learning/ <comment><USER2> Am I wrong in thinking this will cause some havok via incorrect predictions? It is not necessarily possible to predict machine loads.</comment>


Human speech may have a universal transmission rate: 39 bits per second - <AUTHOR> https://www.sciencemag.org/news/2019/09/human-speech-may-have-universal-transmission-rate-39-bits-second <comment><USER2> _But, he says, instead of being limited by how quickly we can process information by listening, we're likely limited by how quickly we can gather our thoughts. That's because, he says, the average person can listen to audio recordings sped up to about 120%—and still have no problems with comprehension._

Some years ago I worked on an accessibility project for an app and website designed for people with disabilities. One of the team members had low vision, and used a screen reader that must have been set to 3x or even higher. I usually listen to YouTube and podcasts at 1.5-2x and I could barely understand the audio. He seemed surprised, which indicated to me that 3x+ was the norm for people in his circle.

I wonder if his ability was trained through years of using fast screen readers, vs. a lower visual processing load leads to better audio processing, or some other explanation.

<comment><USER3> I've known a lot of people that push podcasts, videos, and audiobooks to extreme speed. I knew a guy who'd turn video speed up to 8x so he could binge watch a season of generic anime in an hour flat. I knew a girl who'd get through paperback romance novels by scanning each page diagonally, in 10 seconds each. And here in this thread we have a lot of people bragging along the same lines.

I just don't get the point. If you can process content much faster than it was meant to be played, it doesn't mean you're learning much faster than you could, it means the novel information density is low. Any content that can be sped up that much without loss is not worth listening to in the first place. You're just skipping the trite cliches, filler, and obvious facts.

I can read fast, and I typically go through fluffy NYT bestseller nonfiction at 600 WPM. But when I do this I constantly have a sneaking suspicion that I'm just wasting my time. When I read a good book full of new ideas, I barely go at 150 WPM, but the time always feels well-spent.

<comment><USER4> Exceedingly _slow_ narration, particularly what's normal for audiobooks, is annoying to me because it's slower than I process words. It's like walking with someone whose pace is far slower than your natural gait -- it takes more energy and concentration to slow down. It's why slow-talkers are so annoying.

This isn't "how fast can I go through this" but "what is a comfortable pace"?

So I bump the speed up, though usually fairly modestly: 1.25x - 1.5x is generally enough.

I've noticed that preferred speeds vary tremendously with the quality of the work and speaker -- high-density information and an exceedingly good speaker, and I'll slow down. Slapdash redundant content and poor speaker, I'll speed up.

The degree of polish in the production matters tremendously. I've listened to CPG Grey's YouTube videos (highly polished) and podcasts (a lot of chit-chat with his co-host). The videos work well at normal speed, or perhaps slighly sped up. The podcasts I find nearly unlistenable, though they improve at much higher speeds (1.75x - 2x).

<comment><USER5> Yes, and the painful slowness becomes even more evident if you speed it up to the 1.25/1.5x range, get comfortable with it, and then go back to 1x. IME, it sounds like the speaker is going over-the-top to enunciate, like you're a small child or have learning disabilities or something.

<comment><USER4> Audiences vary, and many listeners of audiobooks are visually disabled who may _also_ have hearing problems. Pitching the default to these seems a fair deal, particularly as speeding up is so straightforward.

</comment></comment></comment></comment></comment><comment><USER6> Apologies for being harsh, but this kind of thing is the phrenology of our time. I know it's utterly conventional to think this way about language in some circles that present themselves as doing legitimate science, but the view that you can calculate the amount of information in human speech, except in a super-technical sense that doesn't match any of the reporting on this study or the way people are interpreting it, has to be called out for the total nonsense that it is. It doesn't bear a moment's honest reflection.

And yes, I know information theory. It's language that these folks - many of them prominent and celebrated within their utterly normalized professions, just like in the days of phrenology - are fundamentally mistaken about. What quantity of information do you think there is in the word "trump," for instance? Is it the same over time, to bring up just one feature of how this funny thing called context informs human speech?

Wittgenstein's Philosophical Investigations is a good place to start if anyone's interested in understanding this issue.

<comment><USER7> They aren't talking about the semantic information of the word "trump". They explain the methodology for calculating information, and it's per syllable (based on the number of distinct syllables that are part of the language's phonetics). So, for English speakers, 'trump' has exactly 7 bits in it. That exact syllable may or may not exist in another language, but if so the same singly syllabic word "trump" would have a different number of bits to a speaker of that language. Maybe next time RTA?

<comment><USER8> >Maybe next time RTA?

I think it's you that has missed the point. Syllables have a very loose correlation to information. So great; we can stream out 39bits worth of syllables / second. In what way does that describe how information dense those syllables are? Context matters here.

<comment><USER9> I think the fact that context matters so much is why we don't try to quantify it. The word 'trump' can covey a lot of meaning or next to nothing, eg in a card game the word trump can covey a lot of information about the state of play and your reaction to it to your competitors. It doesn't take any longer to say and in the context of the game may take less time to think up as well.

</comment></comment></comment></comment><comment><USER10> Early on when Information Theory was emerging, there were attempts to measure the bandwidth of consciousness. They reckoned about 18 bits per second or less, which sounds very low.

Tor Norretranders book, The User Illusion, mentions some of the research:

W R Garner and Harold W Lake "The Amount of Information in Absolute Judgements" \- Psychological Review 58 (1951) - they attempted to measure people's ability to distinguish stimuli (such as light and sound) in bits. Result: 2.2 to 3.2 bits per second.

W E Hick "On the Rate of Gain of Information" \- Quarterly Journal of Experimental Psychology 4 (1952) - this experiment measured how much information a person could pass on if they acted as a link in a communication channel. That is, faced with a series of flashing lights, subjects had to press the right keys. Result: 5.5 bits per second.

Henry Quastler "Studies of Human Channel Capacity" \- Information Theory, Proceedings of the Third London Symposium (1956). Measured how many bits of information are expressed by a pianist while pressing keys on a piano. Result: 25 bits per second.

J R Pierce "Symbols, Signals and Noise" (Harper 1961) - used experiments involving letters and symbols. Result: 44 bits per second.

Discussion of the research, Tor Norretranders book, and what the research may have missed here:

[http://memebake.blogspot.com/2008/08/straw-dogs-and- bandwidt...](http://memebake.blogspot.com/2008/08/straw-dogs-and-bandwidth- of.html)

</comment><comment><USER11> > instead of being limited by how quickly we can process information by > listening, we're likely limited by how quickly we can gather our thoughts. > That's because, he says, the average person can listen to audio recordings > sped up to about 120%—and still have no problems with comprehension. "It > really seems that the bottleneck is in putting the ideas together."

Glad this paragraph was in the article, clears up their methodology. I wonder if it applies to writing too, or if skilled writers work faster.

<comment><USER12> Really depends on language, if you're writing java, you'll be putting out a lot more than that due to how stupidly verbose it is.

<comment><USER13> _> if you're writing java, you'll be putting out a lot more than that due to how stupidly verbose it is_

Being "verbose" means that each letter you type communicates fewer bits of information. If the bottleneck is putting ideas together then you would expect someone writing in a more verbose language to type more letters per minute but still take a similar amount of time to communicate the idea.

In practice most Java programmers are using IDEs with good auto-completion, though, so aren't actually needing to type as many letters as you'd think.

<comment><USER14> Like the sibling comment mentioned, it seems verbosity hinders _reading_ comprehension rather than writing. Many IDEs understand this and hide some of the boilerplate.

This raises the question: if the IDE autocompletes the boilerplate for you, and also hides it, why is it needed in the first place?

<comment><USER15> Consider the verbosity of XML compared to s-expressions. <html>...</html> vs (html ...)

The latter can trivially be used to output the former. The conclusion is obvious; some of these formats are objectively more verbose that others while having equivalent expressive power.

<comment><USER16> Interestingly, depending on the context I find one or the other to be more readable. XML is great when you're got a lot of content, because it provides additional context of where you are in the closing tag, but it's not as great when you don't have a lot of data since the closing tag is just visual clutter.

</comment></comment></comment></comment></comment></comment><comment><USER17> This is really cool. I am working in a related area and I think most of us have assumed that on average, the information rate is 'about the same' for the languages across the world. So it's exciting to see that their results confirm this assumption.

Two qualifying remarks.

1) The 'about the same' is important. Even in their data, there is still quite some variance. They found an average of 39bits, with a stdev of 5. That means that about 1/3 of the data falls outside of the range of 34-44bits.

2) Which brings me to the the uniform information density (UID) hypothesis. According to the UID, the language signal should be pretty smooth wrt how information is spread across it. For many years, the UID was thought to be pretty absolute: Even across a unit like a sentence, it was thought that information will spread pretty evenly. Now, there is an increasing amount of research that shows that esp. in spontaneous spoken language, there is a lot more variance within in the signal, with considerable peaks and troughs spread across longer sequences.

<comment><USER18> Why did everyone assume it would be the same on average? This seems weird to me.

Also, can you explain more about how the information density was calculated? Anything at the bit level seems crazy small to me. Words convey a lot of information. They cause your brain to create images, sounds, emotions, smells, etc. I guess we're calling language a compression of that? But even still, bits seems small.

<comment><USER17> > Why did everyone assume it would be the same on average? This seems weird to > me.

(see edit below; but i leave this up; it might be interesting, also) you mean that even for smaller sequences, the UID holds, right? the assumption was that even for a single sentence, there are a lot of ways to reduce or increase information density so that you get a smoother signal. e.g.: "It is clear that we have to help them to move on.", you could contract it to "it's clear we gotta help them move on" and contract it even further in the actual speech signal ('help'em'). or you could stretch it: "it is clear to us that we definitely have to help them in some way to move on", or alike. the assumption was that such increases / decreases would even be done to 'iron out' the very local peaks and troughs, particularly in speech.

bits: yeah, that took me a while to get used to, as well. the authors used (conditional) entropy as a way to measure information density (which is a good measure in this instance imv). and bits is just per definition the unit that comes out of information theoretical entropy: [https://en.wikipedia.org/wiki/Entropy_(information_theory)](https://en.wikipedia.org/wiki/Entropy_\(information_theory\)) . btw: while technically possible, i don't think that the comparison in the summary article between 39 bits in language and a xy bit modem is a helpful comparison. bits in the context of entropy are all about occurence and expectation in a given context. bits of a modem/in CS, they represent a low level information content for which we do not check context and expectation.

edit: ah, i realise you are asking why most in our community assumed that this universal rate applied across languages, right?

i guess the intuition was that all of us humans, no matter what language we speak, use the speech signal to transmit and receive information and that all of us have the same cognitive abilities. so the rate at which we convey information should be about the same. sure, there are probably differences according to some factors (spoken vs written language, differences in knowledge between speakers, etc.). but when the only factor that differs is English vs Hausa, esp. in spontaneous spoken language, then the information rate should be about the same.

<comment><USER18> > esp. in spontaneous spoken language, then the information rate should be > about the same.

This is entirely non-intuitive to me. I would think with language evolving that some would be faster than others. If language starts as conveying extremely simple thoughts then it should take longer to convey certain things. I would then assume that as the language develops it gets better at conveying ideas. I would think that thoughts could go much faster than how we process it with language. Like I have constant thoughts that are really fast and can be complex. There's no internal dialogue there. But when I think with an internal dialogue it is much slower.

</comment></comment></comment></comment><comment><USER19> So here is my own experience. I was avid audio book fan for last 3 years and while ago some guy on reddit told me about how he listen books on Audible using high-speed option like 2.x. I never tried that before last summer since at higher speed speech become incomprehensible for me.

What this guy told me is that it's just take time to adjust to it. So I basically started to listen for books at slightly higher speed. Then I gradually increased it and in a few days I could handle 2.0x speed no problem while listening for really complex fantasy (Malazan Book of the Fallen [1]). After two weeks I could handle 2.5x without a problem.

In the beginning it was harder to comprehend at high speed while walking or crossing the street since I lost attention, but in a few months I could do anything while listening without missing any information or emotions of narrator.

To give an example of how far this can go. This spring I was listening for The Expanse audiobook [2] at 4.0x speed. With some effort I could go even faster for like 5.x in case of these particular books, but obviously can not keep up for long.

I still usually listen books at 2.0-3.0x depend on narrator and quality of audio and this skill dont go away even if I have extended time between books like a month or so.

[1] [https://www.audible.com/pd/Reapers-Gale- Audiobook/B00M4LRBY6](https://www.audible.com/pd/Reapers-Gale- Audiobook/B00M4LRBY6)

[2] [https://www.audible.co.uk/pd/Abaddons-Gate- Audiobook/B00T6NZ...](https://www.audible.co.uk/pd/Abaddons-Gate- Audiobook/B00T6NZFWK)

UPD: Edit. s/can keep up/can not keep up/

<comment><USER20> One thing I'd also like to develop / wish was integrated into audible and the like is silence trimming. Some speakers leave outsized pauses in their narration which can be significantly shortened effectively increasing speed with less distortion.

I have the opposite problem where I have trouble paying attention to an audiobook at 1x. I get bored in between words and my mind wanders making it very difficult to keep track of what is being said (as in I hear individual words but have trouble keeping sentences in memory when everything comes too slow)

I wish I had realized this in university and had been able to somehow record and playback lectures at 2x. I always got so little out of lectures because the information wasn't coming in fast enough for me to process correctly.

<comment><USER21> Overcast (a podcasting app) has great features to optimize the high speed listening experience. They have variable speed, a great silence trimmer, and a voice boost that makes speech clearer.

<comment><USER22> I just wish I could listen to arbitrary audio with Overcast. As it is I have a blog set up to feed Overcast audio that I give it, but it feels super clunky.

</comment></comment></comment></comment><comment><USER23> I'm a bit confused, here. (I went and looked at the original paper.) They estimated information density for each of the subject languages as a whole, on average:

> In parallel, from independently available written corpora in these > languages, we estimated each language's information density (ID) as the > syllable conditional entropy to take word-internal syllable-bigram > dependencies into account.

But the experiment uses the same text translated into each language! Why introduce this extra variable (and source of error) of estimated language-wide information density, if you are controlling your experiment such that you have the exact same information encoded in each language? That is to say, why use an _estimated_ information density when you could measure it exactly for the texts that are being spoken? Or, conversely, why go to all the trouble of having the speakers read the same text translated into each language, if you aren't going to make use of that symmetry?

<comment><USER24> Information depends on probability. If something is very probable then it doesn't have much information (because you already saw it coming). If something is improbable then it has a lot of information.

In the paper they want to know how much information is in a syllable in context. To do that they need to know the probability of each syllable given the previous syllable. To estimate that probability distribution, you need to look at a lot of text, much more than just the passages that the authors used to measure speech rate.

</comment></comment><comment><USER25> Claude Shannon (of the information theory fame) did a similar research with his 1951 paper "Prediction and Entropy of Printed English". He used a particularly clever idea, leaving out words or letters from English text, he then measured how accurately people could predict the missing text. And then used those numbers for statistical information-theoretical analysis to estimate the information density at about 9-11 bits (IIRC) per letter.

<comment><USER26> Looked it up, 11.82 bits per word, 2.62 bits per letter.

<comment><USER25> Oh! Yes you're right. My mistake. That makes a lot more sense too, it would be weird if a letter was more bits than a byte, hm? :-)

</comment></comment></comment><comment><USER27> The researchers obviously have to keep the scope narrow in order to get numbers at all.

That said, we should be aware that a tech nerd audience will find simple answers to complex non-tech questions appealing, and we should not over- estimate our understanding here just because we have a number.

There is a large amount of data transmitted through sub-communication and context, particularly during an in-person interaction, which is what people are wired for. Overall tone, body language, eye contact, and various social cues make up the bulk of data being transferred in many interactions. There's a reason why talking to some people feels exhausting and others invigorating, and it's not just the transcript.

<comment><USER3> We can avoid reading too much into the study by just remembering the error bars. It's not like 39 is a universal constant. It's more like 39 with a standard deviation of 6. That's a wide spread, but it's _less_ wide than the spread you get from syllable rate alone, and that's all the study quantitatively tells us.

</comment></comment><comment><USER28> I'm not sure what the review process is for the source, but the paper [1] is pretty interesting. Lots of cool findings/references in there like:

\- There is a measurable difference in information density based on the sex of the speaker

\- Syllables were chosen as the base unit of measurement because morphemes (words) are too big/linguistically varied and phonemes (sound equivalent of letters) are too small and likely to be dropped in regular speech. I'd like to see the same analysis using phonemes to see how it changes, especially between dialects.

[1] [https://advances.sciencemag.org/content/5/9/eaaw2594](https://advances.sciencemag.org/content/5/9/eaaw2594)

</comment><comment><USER13> Here's the actual paper: [https://advances.sciencemag.org/content/5/9/eaaw2594](https://advances.sciencemag.org/content/5/9/eaaw2594)

</comment><comment><USER29> In my initial Army job I had to learn how to copy Morse code. I got up to a respectable 23 groups per minute (a group is 5 characters). If I remember right, beyond about 13-15 GPM, once the code stopped transmitting I would still write for another 20 seconds or so. It was not something I consciously did, it just backed up in my brain.

</comment><comment><USER30> I doubt that there's a universal rate - a universal mean (within the context of a shared language) makes more sense.

Every conversation acts as its own handshaking algorithm from which context is derived, and contexts will vary greatly in terms of amount of language required to convey concepts.

Jargon rich conversations between experts have the potential to transfer information at a rate far greater than average.

</comment><comment><USER31> So basically human speech processing is a noisy channel which has limited bandwidth. If the language we encode in has a higher rate of compression, we need to transmit with fewer errors, but if it has a low rate of compression (or even redundancy) we can support a high rate of transmission and correct for more errors.

Which is kind of neat. Thank you Claude Shannon!

</comment><comment><USER32> >Scientists started with written texts from 17 languages, including English, Italian, Japanese, and Vietnamese. They calculated the information density of each language in bits—the same unit that describes how quickly your cellphone, laptop, or computer modem transmits information.

I feel like the whole "bits" calculation is a neat way to get into the media, but not actually related to "information density".

Edit: Been informed I'm deeply ignorant on Information Theory.

<comment><USER33> Human speech uses the extremely dense lossy compression method called "A Lifetime of Experience and Biases"

<comment><USER34> It's only lossy when the decoder's dictionary doesn't have a similar volume of data points to properly reinflate the data stream. Things that didn't make sense to me as a teen make far too much sense now because my experience can properly contextualize the old data.

<comment><USER33> Or lossy when the decoder has a different volume of data points and extracts totally different information or even info that never existed in the first place. Kinda like those anime/retro game upscaling filters, except with way higher variance on what comes out between decoders, or the same decoder at different times. Gives me an idea for a new JPEG decoder with a floating ADC as input.

</comment></comment></comment></comment><comment><USER35> The constructed language Ithkuil is, among other things, designed to have a higher transmission rate. Here are documents and ideas about the upcoming new 2.0 language version:

[http://www.ithkuil.net/new_morpho- phonology_v_0_8.pdf](http://www.ithkuil.net/new_morpho-phonology_v_0_8.pdf)

[http://www.ithkuil.net/phonotaxis_v_0_4.pdf](http://www.ithkuil.net/phonotaxis_v_0_4.pdf)

[http://www.ithkuil.net/roots_v_0_1.pdf](http://www.ithkuil.net/roots_v_0_1.pdf)

[http://www.ithkuil.net/VxCs_Affixes_v_0_2.pdf](http://www.ithkuil.net/VxCs_Affixes_v_0_2.pdf)

[http://www.ithkuil.net/script_presentation_0_2.pdf](http://www.ithkuil.net/script_presentation_0_2.pdf)

[https://www.reddit.com/r/Ithkuil/comments/bvk1t5/tnil_number...](https://www.reddit.com/r/Ithkuil/comments/bvk1t5/tnil_numbering_system/)

[https://www.reddit.com/r/Ithkuil/comments/29wzfq/interesting...](https://www.reddit.com/r/Ithkuil/comments/29wzfq/interesting_video_touches_on_information_density/)

<comment><USER36> If your language is complex enough (and it's my understanding that Ithkuil is[0]) that the emitter takes more time to translate thoughts into it, the transmission rate is not going to raise regardless of information density (remember rate is over time).

I guess if we consider non-real time communication, but in that case (e.g. in English, which is limited by the medium's rate) the _reception_ rate is the main factor, which is probably not too far off the transmission rate.

I'd say Ithkuil is designed for information density and my guess is its actual max rate is pretty similar to the submission's 39bps.

[0] IIRC not even its creator is a fluent speaker.

</comment></comment><comment><USER37> From the paper [0]:

>Abstract

>Language is universal, but it has few indisputably universal characteristics, with cross-linguistic variation being the norm. For example, languages differ greatly in the number of syllables they allow, resulting in large variation in the Shannon information per syllable. Nevertheless, all natural languages allow their speakers to efficiently encode and transmit information. We show here, using quantitative methods on a large cross-linguistic corpus of 17 languages, that the coupling between language-level (information per syllable) and speaker-level (speech rate) properties results in languages encoding similar information rates (~39 bits/s) despite wide differences in each property individually: Languages are more similar in information rates than in Shannon information or speech rate. These findings highlight the intimate feedback loops between languages' structural properties and their speakers' neurocognition and biology under communicative pressures. Thus, language is the product of a multiscale communicative niche construction process at the intersection of biology, environment, and culture.

[0] [https://advances.sciencemag.org/content/5/9/eaaw2594](https://advances.sciencemag.org/content/5/9/eaaw2594)

</comment><comment><USER38> The Micro Machines commercial guy was transmitting at at least 900 baud.

[https://www.youtube.com/watch?v=yDBvEo1s6A4](https://www.youtube.com/watch?v=yDBvEo1s6A4)

<comment><USER39> John Moschitta held the title of World's Fastest Talker at one point: [http://nymag.com/speed/2016/12/is-the-micro-machines-guy- sti...](http://nymag.com/speed/2016/12/is-the-micro-machines-guy-still-the- fastest-talking-man-on-the-planet.html)

</comment></comment><comment><USER40> I wonder how much face-to-face communication adds to the effective transmission rate where the physical gestures, body language, winks, and other non-verbal communication are taken into account.

</comment><comment><USER41> As an interesting side note, ham radio operators heavily use a digital mode called PSK31. It stands for "Phase Shift Keying, 31 Baud".

As I understand it, the 31 bit/s transmission rate was chosen because it is close to the entropy that operators can generate by typing on their keyboards. PSK31 does not transmit 8 bit bytes, but instead uses what they call a Varicode, a kind of Fibonacci code. More frequent characters are encoded using fewer bits, thus the encoded bit rate is an approximation of the entropy in the text stream.

</comment><comment><USER42> Seems reasonable, from what is known about the capacity of speech, although extra cool if there is really some universal number we converge on. Kuyk and Kleijn found the upper bound of the speech channel to be 100 bits per second, in agreement with the lexical rate of about half that in "ON THE INFORMATION RATE OF SPEECH COMMUNICATION" ([https://ieeexplore.ieee.org/document/7953233](https://ieeexplore.ieee.org/document/7953233)) using an interesting method based on mutual information.

Some highlights from the introduction that are relevant:

 _Broadly speaking, two approaches to measuring the information rate of speech exist: the linguistic approach, and the acoustic approach. The linguistic approach describes speech as a sequence of discrete perceptual units such as phonemes, words, or sentences. Taking the average talking speed as 12 phonemes per second [3], and using the English phoneme probabilities tabulated in [4], the lexical information rate is approximately 50 b /s. When the dependencies of the phonemes are accounted for the rate will be decreased further. The lexical information rate does not include information about talker identification, emotional state, and prosody. However, these variables vary relatively slowly in time and contribute little to the overall information rate. As an example, [5] estimated that the total amount of talker-specific information (e.g., age, accent, sex) was of the order of 30 bits_

</comment><comment><USER43> I think it also really comes down to what videos or audio you listen to. In other words, the person delivering the audio and its quality.

I listen to most non-entertainment videos / podcasts at 1.5x and I would say about 80% of them are completely ok to comprehend from a "can I easily listen to this without struggling to figure out what they are saying at a language level". But as soon as I try 2x, that drops down to maybe 30-50% because the person speaking isn't speaking clear enough, they have an accent that overpowers any type of comprehension ability on my part or the audio quality is too poor and it introduces too many artifacts.

Sometimes I ask my friends to watch the same video at 2x to see if they can comprehend it and often times they can't (but sometimes they can). We're all in the same area.

I generally find a neutral accent and very clear annunciation helps the most. I've had a bunch of people say they've watched my videos / courses at 2x without issues because apparently I have no accent which is something I've heard from a number of people in different countries where English isn't their native language. I find it interesting because I've also heard a decent amount of people say I speak very fast at 1.0x speed, so I do believe accents and annunciation has at least some role in this.

Does anyone know of any software where you can feed it an English audio sample and it spits back the number of syllables per second? Seems like a pretty cool potential ML project.

<comment><USER44> Where are you from if people think you don't have an accent, if you don't mind me asking?

<comment><USER43> Long Island, New York.

I'm not trying to plug my channel but here's my latest public video from the other week: [https://www.youtube.com/watch?v=Kq_khHWovl4](https://www.youtube.com/watch?v=Kq_khHWovl4)

I do believe audio quality plays a -huge- role in this.

For comparison, here's the most recent Railscasts video from a few years ago by another screencast author: [https://www.youtube.com/watch?v=urPi4qZJeOE](https://www.youtube.com/watch?v=urPi4qZJeOE)

I can deal with him at 2x but it's mentally taxing because his audio has a metallic wispy sound at that speed and it makes his words sound blended together. I think he also talks slightly slower than me at 1.0x as well, so it's not just base talking speed. Does anyone else notice that metallic sound too?

Here's another sample of Joe Rogan and Bill Burr on a podcast: [https://www.youtube.com/watch?v=cS1KWv0das8](https://www.youtube.com/watch?v=cS1KWv0das8)

Listening to them talk at 2x feels like a joy. They are talking a little slower since it's a casual conversation but the audio is crystal clear and both of them have very good word annunciation (not surprising since they are on stage talking for a living).

<comment><USER45> I agree, you have a very neutral American accent, not much like a New Yorker at all. Other things I noticed include that you are indeed pronouncing each word very clearly, but also that you're speaking a bit more slowly than I would consider a typical speech rate (though not necessarily slower than the average Youtube video). In this video, you're also speaking at a bit higher pitch than a typical male voice, which I think further aids clarity because "sharper" syllables come out very clearly instead of being muddled like they are in very deep voices.

That said, I couldn't comprehend you well at all at 2x speed (using the Youtube controls). This might have just been to distortion caused by the Youtube player on my computer, I'm not sure. At 1.75x you were still very clear, though I suspect at that rate I would find myself pausing the video now and then to think about what you were saying.

<comment><USER43> Thanks for the listen. All of my tests are always using Youtube's playback controls btw.

Were you able to listen to Joe Rogan's podcast at 2x? Skip somewhere in the middle and listen for 15 seconds maybe.

You are right in that I speak slower in that video. Most of my more recent Youtube videos are unscripted so I'm just thinking about things with zero preparation, where as I script my courses word for word (which leads to faster speaking generally) but I don't have any course videos with the same audio equipment to compare side by side.

I didn't read the paper but I wonder what they classify comprehension as. Personally I wouldn't listen to hardcore technical things at 2x because understanding the words isn't usually the goal of listening to it. It would be to fully absorb and understand what you're listening to so you can apply it on your own later. There's a big difference between a mechanical understanding of the words and really "getting" what you're listening to.

I typically reserve 2x for listening to tech talks where my goal is to get a high level overview of something quickly.

<comment><USER45> > Were you able to listen to Joe Rogan's podcast at 2x? Skip somewhere in the > middle and listen for 15 seconds maybe.

Hmm, I skipped around to a few different spots. About half the time it was intelligible at 2x, but as soon as they started speaking faster it became a garble. Occasionally they would speak fast enough that I couldn't catch it even at 1.75x. So I'd say they have a lot more variability in their pacing than you do.

</comment></comment></comment></comment></comment></comment><comment><USER46> Typing at 60 words/minute, 5 characters/word and 8 bits/character gives a gross bit rate of 40 bits/second.

Of course, the information rate is a lot less, since there are fewer than 8 bits of information per character in English. The paper says "from 4.8 bits per syllable for Basque to 8.0 bits per syllable for Vietnamese" and there are multiple characters per syllable. So the typing information bit rate is probably somewhere around 10 to 15 bits/second.

<comment><USER47> IIRC 1.3 bits per character is the value used as a rule-of-thumb for english text entropy, which would yield 6.5bps at 60WPM and an average stenographer can go at 4x that rate for approximately 26bps.

</comment></comment><comment><USER45> I took a peak at the paper and discovered the following: among the languages they studied, English was second only to French in the number of bits of information per second conveyed on average. (English also had significantly fewer syllables per second than French.)

Thus, if you're choosing a language to communicate in on the basis of how fast it is to get an idea across, English and French are likely your best choices! (Among languages in the survey.)

</comment><comment><USER48> > Each participant read aloud 15 identical passages that had been translated > into their mother tongue

I doubt there is an objective way to ensure that no information is lost or smuggled in when you translate a text into another language. For example, English has more than 100 words for 'walk', whereas Toki Pona (a constructed language known for its extreme simplicity) has only one. But does 'stroll' encode the same amount of information as 'tawa'? Depends on what you want to use that information for, I guess. If you only want to know where I went this morning, they are equally good. If you want to know that my act of going there was a recreational activity, possibly part of my morning routine, they are not.

<comment><USER49> It's my interpretation that by deciding the bits of each word, they included subtext/classification that would differentiate 'walk' and 'stroll'.

</comment></comment><comment><USER50> _Scientists started with written texts from 17 languages, including English, Italian, Japanese, and Vietnamese. They calculated the information density of each language in bits—the same unit that describes how quickly your cellphone, laptop, or computer modem transmits information. They found that Japanese, which has only 643 syllables, had an information density of about 5 bits per syllable, whereas English, with its 6949 syllables, had a density of just over 7 bits per syllable. Vietnamese, with its complex system of six tones (each of which can further differentiate a syllable), topped the charts at 8 bits per syllable._

how can you encode 643 syllables using 5 bits? same for 6949 syllabes/7 bits?

<comment><USER51> If I understand this correctly, it isn't that they are uniquely encoding each syllable. It's that they are encoding the information in each syllable. Many syllables have very low information content and must be combined with other syllables to convey information. Many other syllables are redundant.

</comment></comment><comment><USER52> > After noting how long the speakers took to get through their readings, the > researchers calculated an average speech rate per language, measured in > syllables/second.

> No matter how fast or slow, how simple or complex, each language gravitated > toward an average rate of 39.15 bits per second

So does this mean that we are "understanding" only those 39 bits of syllables per second, or more like we are using those 39 bits to index something like an internal address space?

And if the latter is the case, how big would that address space be?

It would also be cool to see this complemented with the data rate (bits/second) of emotion communicated per second and see if that increases the total effective rate of communication between people.

<comment><USER53> The article explicitly concludes that the speaker is the bottleneck, not the listener. You can understand sped-up Youtube videos just fine.

<comment><USER52> Yes. And they say that we can listen at a rate of up to 120% more, which means 46.8 bits/second of syllables.

So then, does it mean we "understand" 46.8 bits of information, or that we are using those bit to address some other, maybe bigger/more complex or detailed, memory space?

</comment></comment></comment><comment><USER54> Being able to convey emotion with tone in speech likely adds a lot more "information".

<comment><USER55> The increased information density in tonal language appears to be offset by a reduced speaking rate [0].

[0] [http://ohll.ish- lyon.cnrs.fr/fulltext/pellegrino/Pellegrino_...](http://ohll.ish- lyon.cnrs.fr/fulltext/pellegrino/Pellegrino_to%20appear_Language.pdf)

<comment><USER56> I think jumelles was referring to something like "don't do that" uttered as fun, angry, etc.

</comment></comment></comment><comment><USER57> I would be interested if Rapid Serial Visual Presentation breaks this rule for any language. My guess is that it wouldn't, but I would be fascinated to find out. I would guess that languages with more general density of information per word would force the reader to slow their words per minute rate. Anecdotally, I can read with RSVP at around 620 words per minute in English and retain general comprehension. Sadly, I don't know another language well enough to compare that information with anything.

[https://en.m.wikipedia.org/wiki/Rapid_serial_visual_presenta...](https://en.m.wikipedia.org/wiki/Rapid_serial_visual_presentation)

</comment><comment><USER58> I recall a moment when, as an adult, I thought it might be fun to watch some cartoons that were my favorite as a child. I actually ended up giving up fairly quickly because the rate of the speech in many cartoons is apparently geared to be very slow. I remember firing up the first episode of the classic Thundercats show and being really struck at how much slower the characters spoke than my memory of the show.

On the flip side, I found noted radio show host Diane Rehm to be virtually unlistenable her rate of speech is sooooo incredibly slow. Her guests sound like they are all at 2x speed compared to her.

I'm pretty sure that this number changes as we age and our processing faculties gear up and then down.

<comment><USER59> Diane Rehm has spasmodic dysphonia, which affects the way she speaks. She's not just old.

<comment><USER58> True, but spasmodic dysphonia doesn't cause slow speech [1]. It's the source of other speech issues she's worked hard to over come.

Her pattern of speech is in general among the slowest I've ever heard -- sometimes approaching single digit words per minute. She's not always so slow, there's interstitials and other moments when her speaking is just kind of slow not unbearable.

Despite my personal feelings, I think her pace of speaking is part of what her appeal was. After being assaulted by other media all day, her show can also be a very relaxing listen and was nearly always a very intelligent conversation.

1 - [https://www.youtube.com/watch?v=SqzfsKMaLqk](https://www.youtube.com/watch?v=SqzfsKMaLqk)

</comment></comment></comment><comment><USER60> I didn't know the but rate, but I learned this studying in computational linguistics. Languages that seem "fast" tend to have more phonemes per semantic unit than "slow" languages. In general, languages with a lower lexicon of phonemes require more of them to make a "word", they get spoken faster. So the semantic units per period of time remains fairly constant across languages.

</comment><comment><USER61> Mm.Hmm... Not sure the researchers have yet encountered the Deep South (US). I estimate approximately 17 bits of information in "Mm.hmm..." based on the 128K possible meanings. It takes about 5-10 seconds for proper gestation, when uttered expertly during the course of a lively conversation.

So more like approximately 1.7 to 3.4 bits/s.

<comment><USER61> y'all

</comment></comment><comment><USER62> I don't get the methodology. If you have a short-story, and translate it into 10 languages (and translate very well), it should have the same amount of information in it in each translation, no?

Therefore the transmission rate will simply be proportional to the time to read the story. This idea contradicts what their study found, no?

</comment><comment><USER63> This has at-least tangential interest to me as one of my side projects is to create a custom "Number Station". Maybe out of a Raspberry Pi or similar, I'm still very much in the preliminaries. Since I'm in the US I am aware of the FCC.

</comment><comment><USER64> See also this story from 8 years ago, which was about a previous study by the same group

[https://news.ycombinator.com/item?id=2976044](https://news.ycombinator.com/item?id=2976044)

</comment><comment><USER65> I once saw research that said English speakers utter about 4 words per second. Shannon once empirically determined each English word carried about 6 bits of information. For what it's worth.

</comment><comment><USER66> I feel like there's an important follow up study in how much humans can compress data with respect to those bits (cat -> large -> mane ~= lion).

</comment><comment><USER67> I would love to see the impact of acronyms on information density. It would also be interesting to see how many bits per second the average human maximum is.

<comment><USER4> YMMV, TLAs increase s/n but leverage ROI on prefamiliarisation.

Language is symbolic, all words are pointers. Whether you collapse complexity through an Apollonian use of religious icons or through initialisations and acronyms likely matters little.

But: TANSTAAFL.

</comment></comment><comment><USER68> Are there weights applied for regional disparities? For instance sped up for American New Englanders and slowed down for American South Easterners?

</comment><comment><USER69> I wish I could provide references... I recall a tutor stating that human speech can be encoded at a very similar rate. This was in 1985 I think.

</comment><comment><USER50> i wonder if there's a constant average reading speed (in terms of bits per second) across written languages. chinese is obviously more dense per symbol than english, but is each character processed more slowly as a result, preserving constant data rate across languages? also interesting to consider how much harder real time, verbal translation would be if data rates differed.

</comment><comment><USER70> YouTube at 2x with captions can speed this to less than 78 seconds. "Less than " to account for advertising.

</comment><comment><USER71> But we convey a lot in a lot higher bandwidth, because we use parallel channels each with low bitrate limitations.

</comment><comment><USER72> Interesting, I did a similar analysis for music scores before and arrived at 30 bits per second.

</comment><comment><USER73> So what the title says is: It may or may not have that rate. Moving on, nothing to see here.

</comment><comment><USER74> I bet, elon musk is going use this data during his next talk about neuralink on human brain is limited to 39bit/sec during output of speech and BCI would surpass this limitation by output to smartphone speakers directly.

</comment><comment><USER75> So speech is a very ineffective way to communicate?

</comment><comment><USER76> the "text" portion? or already with intonation etc?

</comment><comment><USER77> 39.15/64 = phi

<comment><USER78> and 64 seconds is relevant here because?

<comment><USER77> It's 64 bits. The rate of information is measured in bits, and so 64 was the closest binary (2^6) denominator to 39.15 bits/sec.

</comment></comment></comment><comment><USER79> Curiously, this is also why memes are much better at conveying complicated political or sociological ideas faster and more efficiently than a few paragraphs on the topic.</comment>