Representing and Editing JSON with Spreadsheets (2018) - <AUTHOR> https://medium.com/@donhopkins/representing-and-editing-json-with-spreadsheets-8f8832db5239 <comment><USER2> When I see things like this, I try to think about:

      - this relative to my existing tools
      - think about what tools you I wish I had
      - think about what tools that other people from other disciplines wish they had

Tools and habits and thinking patterns are interwoven. Changing tools might seem, in some cases, to be a waste of time, relative to a rigid, short-term performance metrics. However, changing tools can change your thinking patterns, which can really help with long-term growth.

<comment><USER3> Totally this.

SpreadON (which I'm now calling this tool) comes from working with people who are quite comfortable and proficient with spreadsheets, use them for modeling, calculating, and storing data, and have a large investment in certain tools. I didn't want to make them change the way they worked or switch from using their favorite tools (like Excel), but I wanted to be forward-compatible with other tools (Google Sheets). And I definitely didn't want to ask them to write raw Python or JSON data structures.

I came up with seed of the idea more than a decade ago, working on a nutritional analysis and questionnaire system, which was configured with huge tables of structured data about nutritional ingredients, questions, and other structures, each including numbers, arrays, and dictionaries, all with the same structure.

We needed to be able to represent what is essentially a whole bunch of identical JSON structures in a spreadsheet, in a way that was compact, easy to produce, edit, and check.

So I came up with a dead simple way of formatting the spreadsheet headers to declare nested dictionary and array structures with numeric values, and implemented a parser in Python that read a CSV file and returned an array of nested Python dicts and arrays and numbers. (This was before JSON became so popular, but Python data structures are essentially the same as JSON, just calling them dicts instead of objects).

After the row of headers, each subsequent row represents a dict, and the corresponding headers were either names of the dict keys, or special markup like "begin array <name>", "end array <name>", "begin dict <name>", and "end dict <name>". (The name was optional if you were inside an array, but would typically be the numeric array index). The columns under the begin/end array/dict headers were left blank.

That worked great, and we're still using it. But using spreadsheets to represent repetitive structured data did (as you say) change my thinking patterns, which led me to implement a more advanced version that let you represent JSON data types, by adding a type after the dictionary key name (or array index), so the parser knew what type to convert the spreadsheet cell string to. (I object to guessing like YAML, or even parsing JSON syntax: just write the data type declaration once, then you don't have to use punctuation like quotation marks around strings, etc.)

SpreadON has a tag called "table" that implements this idea of DRY-ly and compactly representing repeated structures by declaring key names, dictionaries, arrays and structures in the headers, but it has a more convenient JSON-ish syntax using [ ] { } instead of begin/end array/dict, and it lets you interleave the structural syntax with the name and data type declarations in the headers, so it doesn't waste valuable columns on structural declarations (a disadvantage of the previous technique). But you can still leave blank columns, if that makes it easier to read and edit, and you can even use them for comments.

But there are also many cases when you want to represent structured but non- repetitive JSON data in spreadsheets, so I came up with a tree-like syntax for that, which is basically like pretty-printing typed JSON into the cells of the spreadsheet, but is more repetitive (declaring the data type for each cell, instead of guessing it like YAML), leaves more unused space (but which can be used for comments), and is not as compact and easily editable as rows and columns as the "table" representation. It's really great for writing JSON configurations of nested dictionaries, where keys aren't typically repeated, values have different types, and you might want to add comments or intermediate calculation formulas in the unused cells to the right.

I also implemented a "grid" tag for representing two-dimensional arrays of identically typed values, so you don't have to write out nested arrays, or repeat the data types for each cell. Besides not having to repeat the same data type for every cell, there are many obvious advantages to representing data as dense two-dimensional grids in spreadsheets (it's easy to edit, use formulas, and integrate with other data sources), so I wanted to support that well.

An elaboration of the "grid" tag is the "region" tag, which points to a named region by name, and declares it a data type. That lets you store the raw data in another sheet independently, easily adding rows and columns, without requiring you to insert and delete rows and columns in tree-structured sheets of JSON referring to it. You can also extract rows and columns and cells of a named region, transpose the region, flatten the region to row major or column major 1-dimensional arrays, get the number of rows and columns, etc.

Spreadsheet users are accustom to working with named regions, and also to being able to format the cells in various ways, which can be useful meta-data in itself. However, when you export a spreadsheet as a CSV/TSV file (or download a Google Sheet via its URL as CSV/TSV), you lose all of the named regions and formatting information.

To solve that problem, I wrote a Google Sheets script that exports all the named regions, as well as any layers of formatting information you're interested in, into parallel sheets (with corresponding named regions) that you can download. That way, the JSON consumer has access to all of the named regions as well as the formatting information from the spreadsheet! The named regions are necessary in order for the client to convert the CSV file into JSON, if it uses any "region" references. But they can also be used directly. The formatting information can be converted into parallel 2D JSON arrays with the "region" tag, and used along with the numeric or string values of the cells.

It eventually became obvious that there needed to be a better way to automatically define the named regions in spreadsheets, instead of creating and maintaining them by hand. So I made a way of defining user-defined named tags, which you could place next to regions you wanted to capture, which it could scan for empty cells as delimiters, and it will create named regions of rows and columns and grids relative to the position of the user-defined named tag. It also lets you capture named regions of formatting information. The user-defined tags and the regions they produce are defined in another sheet in "table" format, so you can create and edit them easily. (Kind of hard to explain here without an example, but it looks pretty obvious, works well in practice, and has been quite useful.)

This has all been the result of sitting down with the people who know how to use the tools, and finding out how they use them, which features are important, what they're willing to do, and discovering a fertile common ground between the worlds of spreadsheets and JavaScript/Python programming where we can both work together.

</comment></comment><comment><USER4> I made a Google spreadsheet add-on [1] that contains 3 built-in options for converting nested JSON to tabular format.

Method 1: drills into nested json objects and arrays and returns the key names as headers, key values as rows. In cases with multiple nested values, they get returned into multiple columns, differentiated by a number, e.g. orders > products > 1, orders > products > 2.

Method 2: same as above but returns all nested data into a single column, e.g. a single column named orders > products. This can break the association between JSON elements, but is more convenient for certain types of analysis.

Method 3: concatenates all the elements of each nested object into a single cell, separated with pipes.

In some cases (e.g. when the primary object of interest is nested inside another object) there are still problems recognizing what keys should represent rows vs columns, but in my tests the above 3 algorithms cover most of the use cases for spreadsheets.

[1] [https://mixedanalytics.com/knowledge-base/report- styles/](https://mixedanalytics.com/knowledge-base/report-styles/)

<comment><USER5> You could put the nested data into a separate sheet with a unique identifier for the row. Then include a foreign key back to the original sheet's primary id.

<comment><USER4> Interesting idea, thanks. There's often multiple levels of nested data, would you want to see each level in a new sheet?

<comment><USER5> Absolutely! It's relational data that could map to a defined relational schema, so why not?

<comment><USER3> As I described on another reply, SpreadOn not only supports a tree structured format, but also has tags for referring to entire sheets by name, for declaring grids of identically-typed data, for referring to named regions and extracting subsets of them, and compactly defining tables of repeating structures, which you can mix on a case-by as they make sense. (You can't embed a grid in another grid since it requires more than one cell, but you can make a grid of sheet references, since the sheet name only takes one cell and can be put into a grid.)

There is no "one best way" to represent JSON data in spreadsheets, because JSON data comes in all sizes and shapes, so SpreadON tries to support many different useful formats that you can link together, with a simple straightforward syntax that can be easily extended without breaking existing documents.

</comment></comment></comment></comment></comment><comment><USER6> Haha I did something similar at my company. Our customers are heavy spreadsheet users, so decided to create a generic Angular component that can take semi-arbitrary json and display it in a spreadsheet (handsontable).

The code flattens a deeply nested json object into a bunch of 'rows', similar to Spark's explode or Mongo's unwind.

The objects representing the flattend rows have their getters and setters hooked to the original json object. That way when you edit the spreadsheet the json is automatically updated, with parent values shown as merged cells for array child values.

The code was completely bonkers and I ripped it out later...but must say it was fun to write and see it doing its thing

</comment><comment><USER7> This looks awesome! Fun timing, too - I'm working on a related problem, trying to unlock multidimensional data in a similar way to how the spreadsheet unlocks data for non-technical stakeholders. So many businesses run on spreadsheets for good reason, but it's getting rare to be able to cleanly crush a data dump into a CSV and let an analyst or exec crunch away on it.

I'm sure it's confirmation bias, but I've been seeing tons of stuff on HN related to what I've been tinkering on, and it's really awesome to see other potential competitors thinking about the same problem and know I'm not way off in left field. :)

</comment><comment><USER8> When using VBA and Excel, I've found it easiest to just stick to XML for my REST calls and not try to deal with JSON. Talk about adding features to JSON to make it richer just makes me think everything goes around in circles.

What I'd like to see is an LLVM back-end that generates VBA, but I may never get around to it myself.

<comment><USER9> Everything goes around in circles because we hit walls and obstacles, work around them and move on. Then at a later date, when we hit new walls and obstacles, we re-visit old areas, many times to find that they are no longer problems (faster hardware, more network capacity, algorithmic breakthroughs), and so the industry half-pivots in a new direction.

Everything old is new again. Wash, rinse, repeat.

</comment></comment><comment><USER3> I've meant to contribute to this discussion and answer questions, but this article is more than a year old, and I've been it regularly and made lots of progress since then, I feel I should writing a fresh new article about it, including some actual documentation and examples.

I'll post it to HN as soon as I'm done, and put a link to it here. And I would appreciate hearing from anyone who's interested in or working on similar ideas, so we can collaborate to combine our forces! (Email in profile.)

It also needs a good name! (As a rule of thumb, I try not to get bogged down in naming things when I should be spending my time actually working on them and using them, to figure out what they actually are before naming them. But it's been more than a year and now I want to talk about it more mellifluously.)

I was calling it "JSONster", but that sounded too monstrous, don't describe it, and nobody gets the Napster reference these days.

So I tried to think of better name by repeatedly pounded my fist against my forehead, which gave me a headache, when suddenly a homeopathic solution to my naming problem sprang to mind: "SpreadON", short for "Spreadsheet Object Notation".

Elevator Pitch:

SpreadON applies JSON to spreadsheets like butter to waffles, so it melts into the nooks and crannies.

Marketing Ploy:

SpreadON: Apply directly to the spreadsheet. SpreadON: Apply directly to the spreadsheet. SpreadON: Apply directly to the spreadsheet.

[https://www.youtube.com/watch?v=f_SwD7RveNE](https://www.youtube.com/watch?v=f_SwD7RveNE)

[https://en.wikipedia.org/wiki/HeadOn](https://en.wikipedia.org/wiki/HeadOn)

>HeadOn is the brand name of a topical product claimed to relieve headaches. It achieved widespread notoriety in 2006 as a result of a repetitive commercial, consisting only of the tagline "HeadOn. Apply directly to the forehead", stated three times in succession. Originally sold as a homeopathic preparation, the brand was transferred in 2008 to Sirvision, Inc., who re- introduced the product with a new formulation.

>HeadOn's notoriety came in part because of its advertisements on cable and daytime programming on broadcast television which consisted of using only the tagline "HeadOn. Apply directly to the forehead", stated three times in succession, accompanied by a video of a model using the product without ever directly stating the product's purpose.

</comment><comment><USER3> I wrote some stuff in previous HN discussions about outliners and spreadsheets, and also some stuff about Dave Winer's Frontier, which I'll quote and link to here:

<comment><USER3> [https://news.ycombinator.com/item?id=20773851](https://news.ycombinator.com/item?id=20773851)

DonHopkins 44 days ago [-]

I love the collaborative features of Google Docs and Google Sheets.

The thing that's missing from "Google Docs" is a decent collaborative outliner called "Google Trees", that does to "NLS" and "Frontier" what "Google Sheets" did to "VisiCalc" and "Excel".

And I don't mean "Google Wave", I mean a truly collaborative extensible visually programmable spreadsheet-like outliner with expressions, constraints, absolute and relative xpath-like addressing, and scripting like Google Sheets, but with a tree instead of a grid. That eats drinks scripts and shits JSON and XML or any other structured data.

Of course you should be able to link and embed outlines in spreadsheets, and spreadsheets in outlines, but "Google Maps" should also be invited to the party (along with its plus-one, "Google Mind Maps").

More on Douglass Engelbart's NLS and Dave Winer's Frontier:

[https://news.ycombinator.com/item?id=20425970](https://news.ycombinator.com/item?id=20425970)

</comment></comment><comment> seph-reed Damn. I was going to do this soon. Guess I'll just have to do it even harder...

EDIT: I already do this a ton at work, but I haven't yet made a tool for going from json to srpeadsheet.

<comment><USER7> How are you doing it at work? Would love to pick your brain about the workflows you're seeing for some customer research purposes :)

<comment> seph-reed The route I was planning on going is more of an "someone smart sets it up for layman's" use case.

Basically, you outline the type defs for your json, then how you want that json to be displayed as csv.

</comment></comment></comment><comment> rr-geil-j Is there something like this but for YAML?

<comment><USER10> I was going to say, this looks like YAML! Using blank columns for indentation.

<comment><USER3> Part of SpreadON's design philosophy is to completely avoid the guess-work of YAML, by explicitly declaring the data type as a type name tag. This also enables extension tags for other formats and features like grids of identically typed data, sheet and named region references, etc. And also to avoid the inconvenient syntax of JSON as much as possible.

It addresses the most-complained-about problems with JavaScript: no comments, and no trailing commas. You can use the unused cells to the right as comments (or even intermediate calculation formulas), and not only is there no inconsistency about not being able to put a comma after the last element of an array or object, but it actually doesn't require any commas at all, no quotation marks around string, no backslashes in strings, no spaces or tabs for indentation, nor any other syntactic syrup of ipecac that JSON requires but YAML foolishly tries to avoid by guessing. Each cell is a separate syntactic token, so you don't need commas or spaces or tabs to separate them.

</comment></comment></comment><comment><USER11> Graph, not tree

<comment><USER3> JSON can only represent trees. JavaScript data can represent graphs, but JSON is a subset of that.</comment></comment>


Facebook Changed Everyone's Email to Facebook.com; Here's How to Fix It - <AUTHOR> http://lifehacker.com/5921095/facebook-just-changed-your-email-without-permission-heres-how-to-get-it-back?utm_campaign=socialflow_lifehacker_facebook&utm_source=lifehacker_facebook&utm_medium=socialflow <comment><USER2> People saying it's no big deal either never used Facebook to grab e-mails for parties/weddings/celebrations.

All my friends have emails, and most of them have Facebook, but not all of them check Facebook that often or only have Facebook to not become a social outcast. You either need to contact them by phone or by email if you need something from them in some immediate time frame. Add to that the fact that there are more emails listed on the site than phone numbers so your primary option is email them.

So now those people who haven't bothered checking Facebook cannot be contacted because that email address just routes back to Facebook. It negates the whole point of having your email listed, which is for your friends to contact you in some non-Facebook way.

And Facebook just has a fetish with mucking with your personal settings. It's the only company that routinely does this kind of thing without bothering to say anything about it.

<comment><USER3> They're trying to control the ecosystem and tighten the reigns. However you can't successfully control any ecosystem, you can only manage them. Continuing to abuse users is the absolute opposite thing to do. Users will start flooding away from Facebook once other competitive systems develop in its place.

<comment><USER4> What competitive systems ?

People keep talking like some magical Facebook competitor is just around the corner, ready to take on 400+ million fleeing users and assume its position at the top.

It simply isn't going to happen. The ghost town that is Google Plus is showing exactly what most competitors end up looking like.

<comment><USER5> Using "Google Plus" and "ghost town" in the same sentence is a red flag for not knowing what you're talking about. I get way more engagement on Google+ than I do on Facebook, and I've had a Facebook account for a long time.

<comment><USER6> Perhaps his Google+ experience is different then yours? I'm inclined to agree -- most people who I know have Google+ accounts, have added me to their circles (and vice-versa) but chose to keep all of their interaction on Facebook.

The groups of people you communicate with are different. Just like during the 90s, some groups used AIM, some MSN, some ICQ...

<comment><USER7> maybe its more like suburbia than a ghost town.

</comment></comment></comment></comment></comment></comment><comment><USER8> This is a really stupid move. If someone is viewing my profile and looking for an email address, they're not looking for this one. If that was the case they could just message me through the website.

This even affected profiles that listed 2 or more email addresses.

<comment><USER9> >This is a really stupid move. If someone is viewing my profile and looking for an email address, they're not looking for this one. If that was the case they could just message me through the website.

This seems painfully obvious to me. I am willing to assume that this change in visibility of email addresses was inadvertent or just a simple misthought choice, but it seems far more arrogant of Facebook to think that I'm going to use them as an email client. Ever. Besides the privacy implications, Facebooks messages feature and email, basically be definition and by 5+ years of usage, indicates that they're not the same thing.

This is the worst possible way I've seen someone use email, and I'm someone who will often use Facebook messages as it's a more reliable way of getting in touch with someone than email.

</comment></comment><comment><USER10> I can confirm that my profile was modified to hide my real email address without permission. This appears to be more than a hype piece.

<comment><USER11> They did this a couple of days ago; I think I noticed it on Saturday...

<comment><USER12> I'm willing to bet most news agencies/blogs sat on it until today to reach a more people.

</comment></comment></comment><comment><USER13> I just checked my profile and I saw that my facebook e-mail address was shared with 'only me', likely due to other privacy choices I made and that it stopped sharing my real e-mail address. I think this might have had the effect where people wouldn't see any e-mail address shared. (Although one can easily derive the @facebook.com e-mail address anyway.)

Also, this change just generally makes it harder to make use of other social sites. Need the person's e-mail address to add them on LinkedIn? Need to scrape e-mail addresses of your friends to invite them to another service (as was talked about a lot a few years ago)? Can't do either in many cases if your friends just show their @facebook addresses.

Edit: Well, for LinkedIn that would only work if the person associated their facebook.com address with LinkedIn, which I don't think people do. For the latter, it could still work but the person would strangely hear about the other service via Facebook (which I suppose Facebook could 'lose' those messages if it became a problem) and it could be hard to look people up by their more common e-mail addresses.

</comment><comment><USER14> Some context, from two days ago: <http://news.ycombinator.com/item?id=4151433>

</comment><comment><USER15> It's strange how much FB worships at the alter of engagement. Time and time again they intentionally poke their own users in the eye like this.

<comment><USER3> It's called abuse..

</comment></comment><comment><USER16> Here is how to resolve the issue: <http://www.wikihow.com/Permanently-Delete- a-Facebook-Account>

</comment><comment><USER17> Another one of Facebook's privacy "escapades". Sigh! I thought by now they would have got these things right but it looks like "privacy" is always going to be "bête noire" of Facebook.

<comment><USER18> Facebook and privacy in one sentence. Seriously ;)?

</comment></comment><comment><USER19> I sent a test email to my new @facebook.com email address from both my regular email account and another test email account I own. Neither ended up in either my Facebook messages list or being redirected to my real email. I have no idea where the messages went. Nice one!

At least I've found out about it now and have been able to remove the address from my profile.

</comment><comment><USER20> This is even lamer than general because FB Messages is an awesome product on its own (a reliable way to message or IM people with little risk of spamfiltering), and absolutely didn't need a heavy-handed push like this to be successful. Now, FB Messages is tainted with "product people didn't like, so we'll force them to use it".

</comment><comment><USER21> I've been trying to find out how much of this change has to do with iOS 6's Facebook integration. I remember having to re-do my contacts two weeks ago and this was a HUGE issue because, as Swang 5 said, i had to basically call or SMS everyone to get their e-mails into my contact lists.

Really big inconvenience.

</comment><comment><USER22> It sounds more like they're providing an email to every user with the options set to public by default. While that could be construed as bad, it shouldn't override another email if that other email is marked as public.

<comment><USER23> It is over-riding and replacing the normal email addresses.

<comment><USER24> It didn't replace, in my experience. I didn't know this was to be A Thing, and last week I was going through my settings and noticed the facebook.com address. "Huh, I guess they're adding email addresses to the existing URLs everybody has (or was able to sign up for those years ago)." My other (login) email address was still there, but as an alternate or whatever.

<comment><USER8> It replaced both of my public email addresses.

<comment><USER22> I bet it did if you click "Edit." I bet that your other email addresses are there and are marked as "Hide on Timeline."

</comment></comment></comment></comment></comment><comment><USER25> Is it time yet for a campaign of intentional misinformation across Facebook?

Is there anything that is more a threat to Facebook than misinformation? I am sure they track IP and where you are hitting the local switches and routers, on top of the location you provide, but is it time to provide false location and demographic info to spread a marketing virus. What would make advertisers shy off more than discovering that the demographic info is trash and even more useless than the trash Facebook now hocks.

</comment><comment><USER6> I can't imagine why they decided email addresses now needed two layers of privacy controls. First layer = who can see this, second layer = is it visible on timeline. If I didn't want it visible on the timeline, why wouldn't I just select "Only me" on the first option...? Am I trying to keep my email private from myself?

They also only made the change for email addresses -- not IM usernames, etc. If the two layers were to help with privacy somehow, why not apply it to everything?

</comment><comment><USER26> Wait a minute.. did they do this to just hide emails from spammers?? or did they actually replace your email address with your official facebook email address?

<comment><USER27> It's a real, working email address that sends emails to your facebook "messages".

</comment></comment><comment><USER28> Do Facebook EU have to conform to EU email handling law now they're an email service provider? Are they conforming to those laws?

<comment><USER29> I imagine they have to conform to those laws _for their email service_. And, given that they're a big company with plenty of lawyers, I have no doubt they are. However, this probably changes nothing about the core product.

<comment><USER28> > _And, given that they're a big company with plenty of lawyers, I have no > doubt they are_

Recent events in the UK make me have less confidence in the law abiding nature of large lawyered corporations. This is especially true with respect to privacy which is complex, often not in the corporation's interest, and poorly regulated.

I'll agree I sound a bit like a paranoid privacy nut here, and that there's nothing to suggest Facebook are doing anything illegal.

</comment></comment></comment><comment><USER30> Either they put everything back, or some accounts haven't gotten hit yet. My "primary" email is still listed as my personal email address. I can change it to be the @facebook.com one, but that's just listed as my "Facebook email".

</comment><comment><USER31> This is a big problem. But the only way it will be known as a big problem is if G+ puts some money behind a PR effort. I think this is a great opportunity for G+ to increase it's user base.

</comment><comment><USER18> How about not using facebook? I just don't see the value in it. The most people post any bullshit on facebook like "I'm in the train now. It's warm here." I think it's a waste of lifetime to read such things. There are a lot of other things to do and new things to explore, which are more fun and don't monitor your activities all the day.

</comment><comment><USER32> Am I the only one who's actually happy about not having my actual personal email exposed (even accidentally) on Facebook?

<comment><USER33> You always had that option by setting it to hidden. The issue is that if you wanted to expose your email to, say, your friends so that they could contact you in some non-Facebook way if they needed to, now you have to revert the changes that Facebook did.

</comment></comment><comment><USER34> It's time to dump Facebook and use Zurker (still in beta, you'd need an invite). Here's the invitation link:

<http://www.zurker.in/i-29371-ohhlpiiglq>

</comment><comment><USER35> Seems Zuck has lost it, after all.</comment>


(Designing) Paragraphs in the Web - <AUTHOR> https://lilii.co/aardvark/paragraphs

<comment><AUTHOR> Where I obsess about the design choices possible with a paragraph in the web, written about a month ago.</comment>


Text exceeding maxlength will no longer be truncated when pasted in Firefox 77 - <AUTHOR> https://www.fxsitecompat.dev/en-CA/docs/2020/text-exceeding-maxlength-will-no-longer-be-truncated-when-pasted-into-input-or-textarea/ <comment><USER2> This breaks my use case

        function shorten(text, length)
          const t = document.createElement('input')
          t.maxlength = length
          t.value = text
          return t.value
        }

<comment> jagged-chisel Indeed. I'd suggest a change in this code even if this change in FF hadn't arrived.

<comment><USER3> I think GP was joking. A far more straightforward way to shorten text to a given length is:

      const shorten = (text, length) => text.substring(0, length);

<comment><USER4> Probably way more efficient than relying on a side effect to a DOM update as well.

<comment><USER5> But it is the Web. Somebody must be using it.

</comment></comment></comment></comment></comment><comment><USER6> From the WHATWG/W3C definitions of the maxlength attribute:

> Constraint validation: If an element has a maximum allowed value length, its > dirty value flag is true, its value was last changed by a user edit (as > opposed to a change made by a script), and the code-unit length of the > element's value is greater than the element's maximum allowed value length, > then the element is suffering from being too long. > User agents may prevent > the user from causing the element's value to be set to a value whose code- > unit length is greater than the element's maximum allowed value length.

The key word, I think, is "may" in that user agents do not seem to be obligated to truncate text from what I can find in the standard.

While this does break the expectations from a developer point of view, I think it is perfectly in line with what users expect when they paste text. I think text falling off at the end after a paste with no explanation is more confusing than an the field glowing red with a message "you can only enter X characters here".

The old behaviour famously made people lose access to their PayPal account where the login form had a different maxlength as the registration form and where the password manager had put in a nice, long password. Preventing this sounds like a fine change for me, despite the compatibility break.

<comment><USER7> Arguably truncating the text is against the specification as it only specifies that the user agent may _prevent_ the user from going beyond the max-length, not that it may do arbitrary stuff to make the text fit.

I also hope it will make at least some developers realise that client-side validation is a bad idea.

<comment><USER6> I'd say the fact that it says that the user agent may, not must, truncate the text is a sign that this behaviour should not be expected. Failing the HTML5 validation is also perfectly reasonable. If people are building web pages right, this change will have little impact.

Big if, of course, but a man can dream...

</comment></comment></comment><comment><USER8> It's fairly common to copy a large amount of text and drop it into an input with length restrictions. For instance, I do it often when I submit HN titles. For regular text, it's a much better UX to be able to paste the whole thing then edit it down to meet the restrictions than to paste and have it auto- truncated.

When it comes to password inputs, where you can't necessarily see what you're pasting, it's extremely important than the user knows when truncation occurs. But could that be achieved while still respecting maxlength? Yes, I think it would be decent UX to alert the user when they attempt to paste text that exceeds the maxlength, without actually completing the paste. That way, the input remains empty so there's no confusion about whether the full password or a truncated password has been entered, and the user can take appropriate action.

<comment><USER9> You should just let the password field have kilobytes of data. Throw a warning, but it costs you almost nothing. You don't have to store the data, processing password hashes is designed to take a long time and use a lot of memory (way more than the password of a few KB).

Your web app should also not engage with the password field as much as possible. Don't make it a component, just leave it a password field. Don't read it except to immediately send the contents to the server. Bonus points for just having that be a form submission.

</comment></comment><comment><USER10> Highlights from the bug report[1][2]:

      - HTML spec allows it; says MAY, not MUST [3]
      - Affects only user pastes, not javascript edits
      - Affects all input boxes, not just password ones
      - New preference editor.truncate_user_pastes can restore old behavior

As a developer, I personally find the inconsistent behavior of maxLength unintuitive and am surprised a potentially-breaking change like this didn't have more discussion (although, the original bug report was open for 4 years). But as a user, I have some empathy for the team's desire to fix "broken" websites (e.g. where the login page has a shorter limit than the account creation page or backend).

[1] [https://phabricator.services.mozilla.com/D71689](https://phabricator.services.mozilla.com/D71689)

[2] [https://bugzilla.mozilla.org/show_bug.cgi?id=1320229](https://bugzilla.mozilla.org/show_bug.cgi?id=1320229)

[3] [https://html.spec.whatwg.org/multipage/form-control- infrastr...](https://html.spec.whatwg.org/multipage/form-control- infrastructure.html#limiting-user-input-length:-the-maxlength-attribute)

<comment><USER11> We discussed the problem on #security and we moved to bugzilla once we kinda had a solution (it is hard to discuss solutions on bugzilla. :) Here is a link to the chat: [https://matrix.to/#/!xSFwJMLGSLXLaSUrHr:mozilla.org/$o3a38gf...](https://matrix.to/#/!xSFwJMLGSLXLaSUrHr:mozilla.org/$o3a38gf7m-lLpimQ1HY2aYDcTVBwh9cGSBuj3tpVVUw?via=mozilla.org&via=matrix.org&via=synapse.travnewmatic.com)

</comment></comment><comment><USER12> Now how can we reclaim `onpaste` events? If I'm in an input box and paste text, are there any legit use cases of _blocking_ paste of text in text entry fields?

<comment><USER13> If you are browsing with Firefox, then yes, you can. Set this variable, "dom.event.clipboardevents.enabled", in about:config to false and websites can no longer block you from pasting into input fields.

<comment><USER14> There's also an extension called Don't Fuck With Paste.

<comment><USER15> Somehow it breaks Slack and Teams as collateral damage. I have to disable DFWP for certain sites.

<comment><USER9> Not collateral. They fuck with paste.

</comment></comment></comment></comment></comment><comment><USER16> This is a welcome change, but what would make it even more awesome is a little red bar at the last character that fits into the maxlength. A semi-common thing I do is paste a long thing of text into an exerpt text area, let it truncate to maxlength and manually tweak the ending. A little red bar to tell me where it would've gotten truncated would make that still possible, while fixing the dangerous behavior with truncating fields.

<comment><USER17> Why not a context menu entry to truncate the content to the correct length?

Or even better, some kind of warning when the paste happens, giving the user the option of truncating, canceling or continuing anyway?

</comment></comment><comment><USER18> I've definitely been bit by this and it definitely took hours to debug

<comment><USER19> I'm curious, could you describe the usecase?

<comment><USER20> I have used password-store (pass) to generate passwords and paste them to forms without realizing they were truncated and simultaneously those sites don't have the same maxlength on their login form.

<comment><USER19> Ah. On first read, I assumed you meant you had a webapp which was broken by these changes.

<comment><USER21> (This isn't the same person that answered you)

</comment></comment></comment></comment></comment><comment><USER22> At least this behaviour is better than iOS Safari, which will happily send an input field longer than maxlength to the server. You can find a whole bunch of StackOverflow questions about this [1][2]. So if you will need a server side check regardless of what Firefox does. (Of course, you need a server side check anyway, but you need one with proper feedback rather than throwing back an "Unacceptable" http status code).

[1] [https://stackoverflow.com/questions/33080103/ios-safari- igno...](https://stackoverflow.com/questions/33080103/ios-safari-ignores-html- maxlength-attribute) [2] [https://stackoverflow.com/questions/27319642/is- there-a-work...](https://stackoverflow.com/questions/27319642/is-there-a- workaround-for-text-input-maxlength-not-working-in-safari)

</comment><comment><USER23> Why would you have a maxlength on password in the first place?!

<comment><USER24> It's common in practice even if it shouldn't be.

Also many bcrypt implementations truncate input longer than 72 characters.

<comment><USER25> > It's common in practice even if it shouldn't be.

It should be though, the backend should reject overlong passwords, and the frontend should have such limits as well.

Though by "overlong" I mean kbyte range, not 32 character. The point of the limitation is to avoid randos feeding megabytes of data into your KDF and DOSing your server.

> Also many bcrypt implementations truncate input longer than 72 characters.

The alternative would be to error as bcrypt works on 18 words (of 32 bits). You need special handling (pre-hashing with a non-broken cryptographic hash function) to fix this issue.

Also it's 72 _bytes_ not characters. And your pre-hash needs to generate some sort of textual representation (hex, base64, base85, …), as bcrypt will also truncate at the first NUL byte.

The original paper actually specifies _56_ bytes.

<comment><USER26> A kilobyte limit is fine but

> The point of the limitation is to avoid randos feeding megabytes of data > into your KDF and DOSing your server.

You shouldn't be using a KDF that takes significantly longer when the password gets bigger. If you make that mistake, even a kilobyte is going to be annoyingly slow. If you don't make that mistake, then even MAX_POST_SIZE passwords won't DOS you.

<comment><USER25> > You shouldn't be using a KDF that takes significantly longer when the > password gets bigger.

Your KDF _necessarily_ takes longer when the password gets longer as it's a hash function and thus O(n).

For typical password sizes (typically under 64 bytes), you're below the hash's blocksize so the effect is nil and you can treat it as a constant but it _will_ start coming into play as the size of the key and thus the number of blocks to feed into the hash increases.

<comment><USER26> A KDF is not a hash function, and I said "significantly" for a reason.

If adding a megabyte of input causes the original megabyte to get hashed or otherwise processed once, then you pass the test.

If adding a megabyte of input causes the original megabyte to get fed into your algorithm 100000 times, then you fail the test.

</comment></comment></comment></comment></comment></comment><comment><USER27> I actually ran into a case where the input maxlength on the password field was longer than the maxlength on the "set password" field. When I used my password manager I could login, but when I pasted my password, I _couldn 't_ login.

</comment><comment><USER28> This will probably be helpful. I frequently come across fields which have a tight maxlength where there may or may not be spaces in the field (things like credit card numbers, sort codes, postcodes). When pasting in a version with spaces, this frequently deletes parts of the field which are relevant and required me to delete the spaces and then fill in the result. With this change I would be able to paste it in and then just delete the spaces.

</comment><comment><USER29> > _The form cannot be submitted until the user fixes the error, so the server > shall not receive an excessively long text or password_

Seems like win/win for everyone. In-spec, less confusing for users, and doesn't change behavior for normal form submissions (JS submitters are clearly opting out of browser safety nets).

</comment><comment><USER30> They should remove maxlength altogether; it breaks the expected behavior of textboses wherein pressing a key when focus is in a textbox inserts the character corresponding to that key.

<comment><USER31> Should they also remove type="number"? It breaks that expectation too.

</comment></comment><comment><USER32> Cue javascript-based workarounds

</comment><comment><USER33> For some reason I read that as Fortran 77.

</comment><comment><USER34> Not sure if this is a title length restriction on HN, but the omitted "...when pasted into..." here seems important.

<comment><USER3> And "a password field" also.

<comment><USER35> That's their justification, not a restriction on the change, this impacts non- password fields too.

<comment><USER36> Yeah that's the weird thing. They write "for password fields" and then apply it to non-password fields and even multi-line fields. Have you ever seen a multi-line password field?! I understand that people might abuse <textarea> for it but that's definitely not the common thing and just crazy talk. It's an excuse but I don't understand the reason behind this change.

I've been setting maxlength to generous values on my fields in applications since I started coding HTML, if now suddenly I have to revisit everything and add JavaScript magic to check form validity where previously the page was completely free of JS, well, I think I'd frankly refuse where possible and tell people to complain to their faulty implementation.

<comment><USER37> At the risk of stating the obvious, and sorry if so: doing input length validation client side is very useful for the user's experience, but the server should always validate this too. "... now suddenly I have to revisit everything and add JavaScript magic to check form validity ..." sounds suspiciously like you were relying on client side form validation without server validation, I hope that's not the case. :)

<comment><USER36> It prevents server-side errors, which I don't handle as gracefully. When the client sends content that it shouldn't have been able to send (i.e. someone tampered with the form) and the server bails out, I don't always bother making it pretty.

I'm well aware of the risks in client-side validation, but indeed, as I see in my job often enough (I'm a security consultant), it's a valid remark that not everyone has taken to heart quite yet so thanks for the comment :)

<comment><USER37> Ah yes, preventing uglier server-side errors is indeed a very valid reason. :)

</comment></comment></comment></comment></comment></comment></comment><comment><USER35> > The form cannot be submitted until the user fixes the error, so the server > shouldn't receive an excessively long text or password (a server-side > validation has to be put in place anyway.) However, this could potentially > affect a front-end implementation if it expects the entered text never to > exceed maxlength.

What century are Mozilla living in? Most, even simple forms, don't use <form> elements and submit buttons anymore, they're all aJax. Therefore this workaround will be commonly bypassed.

People can debate if this is a good or bad thing, but ultimately the problem remains: This change will cause unexpected behavior when maxlength-ed stuff no longer obeys on thousands of popular websites.

Even if sites check it server-side, that doesn't mean the user experience isn't substantially degraded relative to obeying HTML standards.

Their justification for this change is nonsensical too:

> This change mainly aims at preventing an unexpectedly truncated password > from being saved.

So why not limit it to input type=password? Heck why include textareas in this change, who is using a textarea for a password box?!

<comment><USER38> A lot of people still put them in forms. They just intercept the form submit to do with as the please. No putting them in a form also breaks accessibility

<comment><USER35> And a lot of sites don't. "This breaks a ton of things but not everything" isn't a good attitude to browser compatibility, particularly from a browser that already has a small market share.

> No putting them in a form also breaks accessibility

Nope. Screen readers have no concept of <form> fields, nor any concept of how the piping works below the surface when a <button> is pressed. I run a screen reader every single day.

<comment><USER39> Am I missing something? This seems less surprising than the alternative.

<comment><USER6> One breaks the expectations of users, the other breaks the expectations of developers. On a website full of developers you'll probably see a lot more from one side than from the other.

<comment><USER35> I'm pretty sure being able to input text longer than the maximum allowable text into a field breaks user's expectations too. Users don't "win" because you made how a site works more confusing.

Their justification for this change is the only user-win (password truncation), and they could have trivially restricted this to passwords then body is hurt.

<comment><USER40> Users don't expect the length limit! that info isnt presented to them until the text is over-long and the clue is: Surprise! Truncate w/o warning. On more than just password fields.</comment></comment></comment></comment></comment></comment></comment>


Instant messaging meets Yahoo! Answers - <AUTHOR> http://www.stumpedia.com/instantanswers.html

<comment><USER2> Really awesome idea. Though most of my queries didn't get answered, but can't see why this could not be helpful for certain kind of questions.

I was just talking with a friend of mine who is working on a project which, given data, lists the expertise of a user. We have plenty of information about our expertise on facebook, email clients, local documents, chats, and twitters. The only problem is that all the systems which have this data are disjoint. If a system processses all my communication - emails, IMs, forum posts, documents, maybe phone calls - it can learn easily which Stumpedia querries to direct to me.

My motivation to answer those querries is that I get visibility and I become a specialist.

I love Web 3.0!!

</comment><comment><USER3> I'm absolutely stunned at all the success HN people are witnessing (even if it is a small sample). I thought this would turn in to a prank/snark fest like chacha.com except in this case I envisioned the question askers and answerers as guilty parties.

</comment><comment><USER4> Very interesting idea, looks pretty slick too.

"Sorry, nobody is currently available to answer your question. If you leave your email address, we can alert you when it's possible to get an answer."

I'll stick to Google for the time being though.

</comment><comment><USER5> I used it and I am amazed by how well it works. I asked "what is a cheap place to stay in san francisco that has wireless internet access". Within two minutes I got a great response "monroe residence on sacremento st".

So far, stumpedia = WIN

</comment><comment><USER6> aardvark.im is another startup in this space -- but the idea is to only get answers from whom you have some kind of social connection (after giving facebook/whatever credentials..).

Interesting stuff.

</comment><comment><USER7> Wait, how is Stumpedia different from Muchobene itself?

<comment><USER8> From first glance it looks like Muchobene is the client that you use to answer questions, stumpedia is a site where people without Muchobene can ask questions. I could be wrong though.

<comment><USER8> So after this I thought: hey, why not pose the parent's question to Stumpedia itself? Here's what it gave:

~~ You asked: how is Stumpedia different from Muchobene itself? ~~ Connected in chat session (41 seconds). ~~ The other user says: Stumpedia is powered by Muchobene You say: so where does one begin and the other end? You say: is stumpedia just the web site you access muchobene from? He/she says: yep, like other partners, ubergizmo.com for isntance You say: ah, cool, thanks!

</comment></comment></comment><comment><USER9> Interesting concept, very smooth interface.

3 questions, 3 correct answers too. Who is answering, and what is their motivation?

<comment><USER10> interesting - i didnt know where to ask a question until i click on the welcome text. i was expecting a straight-forward textarea instead of a styled part of a note pad, though ;)

<comment><USER10> to follow up, though, its an awesome idea

</comment></comment></comment><comment><USER11> Why do you need a client to answer questions?

</comment><comment><USER12> This sounds like IRC to me.

<comment><USER13> That's exactly what IRC _was_ to many people. With the death of IRC, that problem is again unsolved.

<comment><USER12> Death? Come on freenode

</comment></comment></comment><comment><USER14> Looks very cool. Wonder if it works. Has anyone used it yet?

<comment><USER7> You asked: What's the difference between big-endian and little-endian? ~~ Connected in chat session (45 seconds). ~~ The other user says: In big-endian representation, the most significant byte is first (in lowest memory address), while in little-endian, the least significant byte is first.

</comment></comment><comment><USER15> Nice! I like it. I just hope it gains critical mass.</comment>


Piñatas delivered by drones - <AUTHOR> https://www.youtube.com/watch?v=7uCnxqXCghg <comment> J-dawg Startups have got so weird that I can't figure out whether this is a real thing or just a promotional piece for a digital agency.

I wonder if there is some potential in the "drone delivery as a novelty" concept though?

<comment><AUTHOR> They actually do sell piñatas. [https://medium.com/bakken- bæck/how-1000-piñatas-built-our-co...](https://medium.com/bakken- bæck/how-1000-piñatas-built-our-company-culture-412fc0137e8e#.2l4bzubal)

<comment> J-dawg Great story! But who buys 1000 piñatas for an office party?

<comment><AUTHOR> Actually I was supposed to attend that party but didn't for some reason that I can't recall. But these guys are usually quite serious about their parties. Last year they hacked their elevator for their space-themed party to have a tablet with controls for going up to their office. Ground floor was earth and their office on the top floor was displayed as the moon. They also ordered custom made Daft Punk helmets for the DJs.</comment></comment></comment></comment>


Malaysia Airlines didn't buy $10 upgrade that could have tracked missing MH370 - <AUTHOR> http://www.washingtonpost.com/world/malaysia-airlines-didnt-buy-computer-upgrade-that-could-have-given-data-on-missing-flight/2014/03/19/40e2484c-af7c-11e3-a49e-76adc9210f19_story.html

<comment><USER2> I asked this here 4 days ago:

[https://news.ycombinator.com/item?id=7411541](https://news.ycombinator.com/item?id=7411541)

I'm not sure why it's not repeatedly mentioned. The news talks about the futuristic hardware upgrades with GPS, etc. However, we can solve most of the problem today.

<comment><AUTHOR> It sounds unfortunate that if the upgrade was that cheap.

 _Malaysia Airlines said in a statement. "This installation is sufficient to meet all of MAS's operational requirements and at the same time meets all international requirements that enable us to fly international airways."_

This attitude is common, almost an industry standard IMO.

I watched a documentary on the Air France and at the end of the documentary Air France claimed their equipments met the regulatory compliance. What they said was "until the Board revises the requirements, [we] won't do anything about it."

So unless the regulation said so, or someone brave enough to suggest that has to be done, aviation companies will do less.

I was on a flight last summer from NY to SF and the kernel which they used to power the OS is about 5 years old. That's not to say they have to upgrade to the latest kernel but I wonder how old some of those equipments are. Since an aircraft can last for many years, how often do they upgrade software and hardware? What is the regulation like?

On the flip side, I wonder how simple a computer upgrade is on an aviation. Do they do testing on a few aircraft before they deploy on every single flight in the airline?

<comment><USER3> _This attitude is common, almost an industry standard IMO._

Common and standard in a hell of a lot of industries too. With something as tightly regulated as air travel, or pharma, or finance, you simply _do not add features that are not required_ unless it provides some measurable competitive benefit.

Everything feature or bit of equipment you add becomes something else you have to check, maintain, regulate, replace. You don't do that just because it's a good idea, but because you have to or it makes you more money. Neither of things apply in a situation that arises once in a lifetime, unfortunately.</comment></comment></comment>


Khan Academy React Coding Style Guides - <AUTHOR> https://github.com/Khan/style-guides/blob/master/style/react.md <comment><USER2> 80 character limit, yet uses 4 space indentation instead of 2. My advice: Use 2 space indentation.

I do like the `handleEvent` convention.</comment>


What do you miss about your country? Food? Cosmetics? - <AUTHOR> As Japanese expats abroad, we wanted to find a simpler + more cost effective way to access the products we love.
That's why we launch Torimo https://torimo.io/
Torimo matches shoppers searching for locally unavailable products with travelers looking to earn some cash while traveling.
Torimo will find you a traveler who is going to the destination where the product can be purchased and can bring it to you for a reasonable price. The traveler on the other hand, can earn commission to cover flights, hotels, and all other expenses while enjoying the trip with less concern about the budget.
Please check it out!
https://torimo.io/ <comment><USER2> In the "Upcoming Trips" page it shows the destination of the trip but it doesn't show the origin. It's weird because if someone buys something in Mexico, s/he must give it to me and I'm in Argentina.

If I assume that you are restricting the service to USA, it can be a problem if the traveler is from the East cost and the requester is in the West cost.

<comment><AUTHOR> Thank you for your comment! At this point, we focus on New York, so all users including travelers are New York residents and meetups to pick up the requested items happen in New York. We mention it several places in the platform, but it has to be more clear. Thank you for your feedback!

<comment><USER2> Focusing in a big city initially looks like a good idea.

In the list, there are travels in the past to NY. Does it mean that someone from abroad went there with a package? Perhaps that line should show the origin of the traveler.</comment></comment></comment>


GraphQL: Leaving technical preview - <AUTHOR> http://graphql.org/blog/production-ready/ <comment><USER2> Very excited about this! I believe that GraphQL's ease of composability can be a real game changer in terms of ease of development and discoverability.

</comment><comment><USER3> Nice. I can't wait till the end of version specific REST API endpoints! And I mean it from the viewpoint of my own apps too.</comment>


Think Stats, using python to learn stats  - <AUTHOR> http://www.greenteapress.com/thinkstats/ <comment><USER2> Thanks, I'll put this on my ipad and go through it because I am interested in stats.

Did a professor use a non-commercial book for their class? If so, that is amazing.

<comment><USER3> Yes. Olin, the college where the author teaches, encourages that kind of stuff. I hope they are the start to a trend.

</comment></comment><comment><USER4> I commend the effort to teach statistics to python programmers, but I'd recommend interfacing with R for creating the figures. R will create beautiful figures by default; it can also handle sophisticated stats, should the need arise. The current figures are filled with unneeded inwards-facing ticks that often overlap with data, and have tick and axis labels of inconsistent font sizes. The negative numbers in the y-axis in Fig 2.3 and x-axis in Fig 4.6 and 7.1 don't display properly, and the presentation of the data in Fig 3.1 is too noisy to deliver a useful message.

<comment><USER5> I think the point is to learn statistics using an easier language than R, ugly figures be damned.

<comment><USER6> I see this as a real issue. R is useful but sorta crappy as languages go. I have hopes for Incanter [<http://incanter.org/>] (a clojure-based stats environment), but it looks stalled out vs dead in the water. Even still, for basic stats it's pretty viable.

I would love to see a stats environment based on a sane, full-featured language that was ready to take advantage of multiple cores and GPU computing. Clojure has all the building blocks, but it's not built yet.

<comment><USER5> I have to confess that whenever I do any statistics (which isn't much, these days) I struggle with R for a bit and then say "screw it" and use MatLab.

<comment><USER4> Quick R, helped several of my friends try R: <http://www.statmethods.net/>

Once you're hooked, go to the R inferno: <http://www.burns- stat.com/pages/Tutor/R_inferno.pdf>

</comment></comment></comment></comment></comment><comment><USER7> To me, Allen Downey is the Salman Khan of programming education.

</comment><comment><USER8> How heavily is this tied to Python?

Can a ruby/JS hacker with no python experience jump into this book?

<comment><USER9> Come on young rubyist. embrace the dark side. You will like it. /jk

I went through the fist couple of chapters and all the python code I've seen so far is quite close to "pseudo-code" than deeply-idiomatic Python.

Considering the fact that the Prof. also wrote "How To Think Like a Computer Scientist: Learning with {Python/Java/C++}", you can be rest assured that he knows how not to trip up students with clever syntax.

<comment><USER10> _Come on young rubyist. embrace the dark side. You will like it. /jk_

I adopted Python because I was told Ruby _is_ the dark side. oO

</comment></comment></comment><comment><USER11> anyone knows of a similar resource for physics?

<comment><USER12> What area of physics do you want to know about? There are some free books (for example I know of one on small angle neutron scattering that you can find at NIST)--also there are many lecture notes online. For texts, though do you think it would be better to have a static PDF, or a wiki, where an author could add what they could and the community could improve it?

</comment></comment><comment><USER13> I looked through Downey's book.

(A) Definition of Probability.

As far as I could tell, the book defined probability in only one of two ways:

(1) Relative frequency assuming finitely many trials (page 13) and

(2) Bayesian prior intuitive belief (page 52).

No, here the book is doing readers a disservice.

(B) Exponential Distribution

The 'derivation' of the exponential distribution on page 37:

"I'll start with the exponential distribution because it is easy to work with. In the real world, exponential distributions come up when we look at a series of events and measure the times between events, which are called inter-arrival times. If the events are equally likely to occur at any time, the distribution of inter-arrival times tends to look like an exponential distribution."

is a bit too vague.

(C) Random.

The book also uses 'random' too frequently, loosely, and unnecessarily. Instead, essentially we can just drop the word 'random' and avoid questions about what 'random' means and if some data is 'truly random'.

For (A), essentially universally in advanced work, there is only one approach:

We have a non-empty set of 'trials'. Then an 'event' is a subset of the set of all trials. The set of all trials is an also an event. The set of all events is closed under countable unions and relative complements. In particular, the empty set is an event.

A 'probability' is a function P from the set of all events to the interval [0,1]. For an event A, P(A) is its probability. The probability of the set of all trials is 1.

P is 'countably additive': For countably infinitely many pair-wise disjoint events A_i, for i = 1, 2, ..., the probability of the the union of the A_i is the sum of the P(A_i).

A 'random variable' X is a function from the set of all trials to the real numbers such that for any real number x the set of all trials w such that X(w) <= x is an event. Usually we suppress the notation of the trials and just write the event X <= x. This is our only use of 'random', and we give no more definition of it.

In practice, essentially any number at all that we observe can be regarded as a random variable. In particular, the number might have been something observed about a person. The statement on page (52) that

"Anything involving people is pretty much off the table."

would be a silly foundation for probability.

For Bayesian and prior beliefs, can regard such a belief as an estimate of a conditional probability where we condition on what we know. E.g., we know a lot about people so that given a person, we can guess that with probability over 99% the person is less than 10 feet tall. This point does not mean that we have a 'Bayesian' foundation for probability. It's better just to drop mention of 'Bayesian'.

The 'distribution' of a random variable X is the function F_X(x) = P(X <= x).

Full details are in any of, say,

Jacques Neveu, 'Mathematical Foundations of the Calculus of Probability', Holden-Day.

Leo Breiman, 'Probability', ISBN 0-89871-296-3, SIAM.

M. Loeve, 'Probability Theory, I and II, 4th Edition', Springer-Verlag.

Kai Lai Chung, 'A Course in Probability Theory, Second Edition', ISBN 0-12-174650-X, Academic Press.

Yuan Shih Chow and Henry Teicher, 'Probability Theory: Independence, Interchangeability, Martingales', ISBN 0-387-90331-3, Springer-Verlag.

Essentially what is going on is that we are defining P as a non-negative real 'measure' with 'total mass 1' as in any of:

Paul R. Halmos, 'Measure Theory', D. Van Nostrand Company, Inc.

H. L. Royden, 'Real Analysis: Second Edition', Macmillan.

Walter Rudin, 'Real and Complex Analysis', ISBN 07-054232-5, McGraw-Hill.

This approach to probability is the one used in essentially all advanced work, e.g.,

Ioannis Karatzas and Steven E. Shreve, 'Brownian Motion and Stochastic Calculus, Second Edition', ISBN 0-387-97655-8.

Jean-Rene Barra, 'Mathematical Basis of Statistics', ISBN 0-12-079240-0, Academic Press.

Early in Neveu can see some ways we get essentially pushed into this foundation for probability.

With this approach, for a random variable X, we define its expectation E[X] as just its integral with respect to measure P in the sense of measure theory. Then essentially by a routine 'change of variable' we can get E[X] as an integral in terms of the distribution F_X and the measure it defines on the reals.

With this approach, if, say, for some positive integer n we measure the height of n people, then we say that we have the values of the n random variables

X_1, X_2, ..., X_n

and do not say that we have the values of n trials of one random variable X.

Indeed, with this approach, all our experience, all of the universe, is only one trial.

Given a set of events, we can define what it means for the set to be 'independent'. And we can proceed similarly for random variables.

Then we can give a more careful statement of the central limit theorem and also state the law of large numbers. We can also say why we use an average to estimate an expectation, e.g., as in

Paul R. Halmos, "The Theory of Unbiased Estimation", 'Annals of Mathematical Statistics', Volume 17, Number 1, pages 34-43, 1946.

On page 12 there is:

"The mean of this sample is 100 pounds, but if I told you 'The average pumpkin in my garden is 100 pounds,' that would be wrong, or at least misleading. In this example, there is no meaningful average because there is no typical pumpkin."

No: An 'average' does not have to be 'typical' to be 'meaningful'. The expectation of the weight X of a pumpkin in the garden remains meaningful. And the law of large numbers will still apply.

For the exponential distribution, there is a good 'qualitative' derivation: If we have arrivals where the inter-arrival times are stationary and independent, then the inter-arrival times are independent and have the same exponential distribution. Good details are early in the chapter on Poisson processes in

Erhan Cinlar, 'Introduction to Stochastic Processes', ISBN 0-13-498089-1, Prentice-Hall.

Downey's book places a lot of emphasis on distributions, e.g., Weibull, and descriptions, e.g., skewness. So an implication is, given some data, we should try to find its distribution and various summary properties of it, but usually this is unpromising. It is good to have the concept of a distribution but, given some data, usually not good to try to find its distribution. Instead, we get our results by manipulating the data in ways where we need to know little or nothing about the distribution.

The strong emphasis on hypothesis testing is curious. There is more in:

E. L. Lehmann, 'Testing Statistical Hypotheses', John Wiley and Sons.

E. L. Lehmann, 'Nonparametrics: Statistical Methods Based on Ranks', ISBN 0-8162-4994-6.

Downey's book mentioned the chi-square distribution: The main point is that if X_1, X_2, ..., X_k are random variables with Gaussian distribution with mean 0 and variance 1, then

Y = X_1^2 + X_2^2 + ... + X_k^2

has chi-squared distribution with k degrees of freedom.

Downey's book mentioned the variance of a finite population; while variance is of high interest, the variance of a finite population is not. What is usually wanted is an estimate of variance, and for that the formula in the book should be dividing by n - 1 instead of n; this change yields an 'unbiased' estimate.

It would be good for students to get at least an intuitive understanding of the most powerful hypothesis testing via the Neyman-Pearson result: Using an analogy from real estate investing, regard the false alarm rate as money to be spent and spend the money to get the best return on investment buy buying the property with the highest ROI first, then the one with the second highest, etc. Yes, in principal this process leads to a knapsack problem which is NP- complete. Likely the nicest proof of the Neyman-Pearson result is from the Hahn decomposition; this follows from the Radon-Nikodym result with a famous proof by von Neumann in Rudin.

The mention of 'resampling' is curious: There are cases where we won't have independent random variables but will have 'exchangeable' random variables, and exchangeability can be enough.

One example is extending nonparametric tests to multidimensional data and using it for anomaly detection in server farms and networks where multidimensional data is ubiquitous, e.g.,

N. B. Waite, "A Real-Time System-Adapted Anomaly Detector", 'Information Sciences', volume 115, April, 1999, pages 221-259.</comment>


Superconducting secrets solved after 30 years - <AUTHOR> http://m.phys.org/news/2014-06-superconducting-secrets-years.html

<comment><USER2> A model of clear writing:

 _In a superconductor, as in any electronic device, current is carried via the charge on an electron. What is different about superconductors is that the electrons travel in tightly bound pairs. When travelling on their own, electrons tend to bump into each other, resulting in a loss of energy. But when paired up, the electrons move smoothly through a superconductor 's structure, which is why superconductors can carry current with no resistance. As long as the temperature is kept sufficiently low, the electron pairs will keep moving through the superconductor indefinitely._</comment>


Patients do better when cardiologists are away at academic meetings - <AUTHOR> http://theincidentaleconomist.com/wordpress/patients-do-better-when-cardiologists-are-away-at-academic-meetings/ <comment><USER2> The FAA responded to a number of high profile aviation accidents by making a series of important changes in the cockpit. Their studies revealed that some mistakes happened because subordinates were too afraid to question the captain. After these changes, the captain was still left firmly in command but a new environment was created where subordinates felt (more) comfortable to voice any questions or concerns.

A similar thing needs to happen in medicine. Doctors walk around like gods in the hospital but they are still human and make mistakes. The problem is that everyone is too afraid to correct the doctors when they make obvious mistakes. Don't get me wrong---the doctors must remain in charge, but they will still goof up so people need to feel free to offer advice and criticism.

<comment><USER3> And it'd be nice if they treated their patients with some respect and as if they had a functioning brain instead of as objects. Obviously this does not apply to all doctors but this happens with alarming regularity.

<comment><USER4> How often has this been said to programmers/tech startups? The answer is typically shield the nerds from customer support by introducing extroverts. And doctors who are often nerds are forced by design to deal with people constantly. No easy solution here.

</comment></comment></comment><comment><USER5> The linked article glosses over the distinction between teaching hospitals, which did show a difference, and nonteaching hospitals, which did not see a significant difference. At teaching hospitals, there are research cardiologists and administrative types (division chair, officers of national organizations, journal editors) in addition to plain old clinicians. Everybody has to do time "on-service" directing the care of hospitalized patients. Naturally, the chiefs and the lab rats have to attend the big meetings, so their on-service time is in other months of the year. During the meetings the clinical cardiologists are mostly left in charge, and their clinical skills are better maintained, leading to better outcomes. Note that these outcomes had to do with acute high-risk cases, not elective interventions, etc.

</comment><comment><USER6> It seems that having most of the cardio guys away at the conferences means there are less invasive procedures.

I would guess that what is scewing the result though is that the invasive procedures carry short term risk for a long term health benefit. The study only sees the decrease in short term risk (less invasive procedures) but is blind to the long term consequences of not doing the procedure.

<comment><USER7> Your post brings to mind an article by (I think) Ben Goldacre in which he recounts that inserting a stent to "solve" narrowing of the arteries has never been shown to be a net benefit to the average patient (except for emergency cases).

Nevertheless, cardiologists have continued to carry out operations to insert "new! improved!" stents which successively claim to eliminate the problems of previous designs over what is now something like three decades of medical practice. None of them have worked.

I'll try and find the article, but a quick web search hasn't turned it up.

<comment><USER8> This was originally unknown and non-obvious (massively narrowed coronary arteries are ok?) but now known and modern clinical guidelines (which are written by physicians) have changed to reflect this reality. This was confirmed by well controlled multi-center trials. It's not cardiologists inserting "new! improved!" stents for the thrill of it.

This doesn't reflect the case of the example patient who has angina upon exercise with significant stenosis, but is not undergoing an active MI. That patient may still be stented and have their lifestyle improved drastically, despite there not being an urgent requirement.

<comment><USER7> It's not that it was unknown and non-obvious - it's that as each successive stent was shown not be an effective treatment, cardiologists continued to operate for decades, confident that the next "new! improved!" version would work, even though none of the previous ones had.

The reasons for this are a combination of the difficulty of publishing neutral or negative results & surgeons' entirely natural inclination to do something rather than nothing, even if nothing is actually the right thing to do, because operating is what they've trained their entire lives to do.

The same thing happened with extreme chemotherapy followed by bone marrow transplant as treatment for many cancers in the 80s: something like a $billion was wasted on treatments that didn't work. The reasons why it took so long to demonstrate that that particular treatment was ineffective were not entirely the same - read "The Emperor of All Maladies" for the details - but shared the optimism of doctors & patients combined with a plausible theory, which is what ultimately makes for a potentially toxic combination.

</comment></comment></comment></comment><comment><USER9> Fulltext: [https://pdf.yt/d/Ans8OYU3MwjkbN3J](https://pdf.yt/d/Ans8OYU3MwjkbN3J) / [https://www.dropbox.com/s/bexaqbczwjyi47f/2014-jena.pdf](https://www.dropbox.com/s/bexaqbczwjyi47f/2014-jena.pdf) / [http://sci- firstname.lastname@example.com...](http://sci- email@example.com@generic-88DD2FA3BEC9.pdf)

</comment><comment><USER10> When cardiologists are away, I'd imagine no non-emergency procedures/surgeries are performed (no heart transplants, quintuple bypasses, etc), which has to be a significant factor here.

<comment><USER11> Then that effect should appear in other disciplines, shouldn't it?

<comment><USER12> My understanding is that they only looked at cardiac patients, so that wouldn't contradict the study.

</comment></comment></comment><comment><USER8> This is complete clickbait.

It's because many fewer interventional procedures are done when there aren't any doctors to do them. This isn't even really very new. For example, during the Canadian physicians' strike, mortality also went down dramatically.

Any mortality differences would be temporary. The next study will show that mortality increases after cardiologists return from the conferences as they do all of the procedures that they put off, and then levels out again. Simple.

Really not very impressed with this headline, but any opportunity to rag on doctors is typically taken in the mainstream media.

</comment><comment><USER13> I didn't read the article, but only the summary:

>>> High risk patients admitted with heart failure during meetings had a 30-day mortality rate of 17.5%, compared to 24.8% when more cardiologists were there. Cardiac arrest 30-day mortality was 59% during meetings and 69.4% at other times. Not surprisingly, the rates of percutaneous coronary intervention were lower during meetings, too (20.8% versus 28.2%).

In my view, they could have done with one or two fewer significant digits based on the size of the error bars in the graph. My interpretation: Need to collect more data and see if this is reproducible.

<comment><USER14> Signifacant figure abuse is surprisingly widespread in science. At least 23.432361% (+-10%) of papers I have read have managed to get their significant figures wrong.

</comment></comment><comment><USER15> If the mods are around, the ars technica summary of this study covers it better than this post: [http://arstechnica.com/science/2014/12/when-the- doctors-away...](http://arstechnica.com/science/2014/12/when-the-doctors-away- the-patient-is-more-likely-to-survive/)

Among other things, more focus is put on the possible interpretations of this (and it explicitly cites the source of those possibilities as the research paper), and has a better explanation of the source of the data for the analysis.

<comment><USER16> There's nothing wrong with the original at all, it should stay.

In fact the original is far superior to the Ars "summary" because it provides a more neural headline, also provides link to the paper up front, provides the abstract, provides the key graph with error bars and information about comparison to other disciplines. It ends with a reasonable evaluation that this hasn't proven anything either way and calls for more investigation.

Furthermore it was also written December 23rd by a site specialising on these matters rather than by Ars, a pretty low brow generalist site, on December 27th. While the Ars author has a PhD in biology, the guy who got the scoop four days earlier is eminently qualified in this area.[1]

The original link is a better article, published earlier, by a more appropriate author on what appears to be (although that isn't hard) a better site. The HN mods shouldn't be changing the links except in extreme cases anyway.

[1]"Aaron E. Carroll, MD, MS is a Professor of Pediatrics and Assistant Dean for Research Mentoring at Indiana University School of Medicine. He is also the director of the Center for Health Policy and Professionalism Research. He earned a BA in chemistry from Amherst College, an MD from the University of Pennsylvania School of Medicine, and an MS in health services from the University of Washington, where he was also a Robert Wood Johnson Clinical Scholar.

Aaron's research focuses on the study of information technology to improve pediatric care and areas of health policy including physician malpractice, the pharmaceutical industry/physician relationship, and health care financing reform."

</comment></comment><comment><USER17> There is the anecdotal correlation to the contrary w/ May being the first or second most popular suicide month and also the month when the American Psychiatric Association has its annual meeting. I think that was mentioned in Infinite Jest or some other novel, but it is true as far as correlations go.

</comment><comment><USER18> See this is why I love science. Somebody does a study, and we learn something absolutely new that we wouldn't have known.

More follow up research needs to be done, but this potentially means that the doctors can change how they treat patients and significantly improve patient mortality.

</comment><comment><USER19> They need to add the individual doctors as factors in their regression.

Otherwise this could just be "inexperienced or checked-out doctors are more likely to go to the conferences.". It's no surprise that there are differences in survival that depend on which doctor you get.

</comment><comment><USER20> [http://www.bmj.com/content/316/7146/1734](http://www.bmj.com/content/316/7146/1734)

 _cough_ "Half of all doctors are below average" _cough_

</comment><comment><USER21> Linkbait headline is linkbait.

Certain patients (those classified as 'high risk') do better when cardiologists in certain hospitals (teaching hospitals) are away at academic meetings.</comment>