UTF-8 String Indexing Strategies - <AUTHOR> https://nullprogram.com/blog/2019/05/29/ <comment><USER2> Is there any real case where code point indexing is useful? It seems like all these attempts to restrict strings in such a way to accommodate code points is just introducing complexity with no gain.

UTF-8 was designed to be an encoding (of code points) on top of the "bytes" abstraction just as Unicode is designed to be an encoding (of human text) on top of the "code points" abstraction. I think it should be uncontroversial that there are very good reasons to at least handle arbitrary sequences of code points (eg, you want to be able to handle input from future versions of Unicode, and you don't know about the grapheme clustering of those code points), but I don't see a good reason _not_ to handle arbitrary sequences of bytes.

The only reason I can see is in ensuring that text is losslessly convertible to other UTFs, particularly UTF-16 (which exists for historical reasons), but this just seems like a matter of _when_ the information is lost (is it during conversion from string to UTF-16, or from bytes to string), not _if_ it is lost.

As far as I can tell with the Python story, for example, people decided to add special "Unicode" strings into Python 2, then presumably some code used the "Unicode" strings and some code used the "byte" strings, so this situation is obviously underisable .. then in Python 3 they tried fixing it by replacing which sort of string was the default. Why would it not have been better to just improve Unicode support for the existing strings instead of splitting the type into two and forcing everyone to decide whether their strings are for "bytes" or for "Unicode"?

<comment><USER3> Programmers are rarely interested in individual bytes, except sometimes if a string is abused as a byte array. In all other cases iterating or indexing over characters is the intention, and code points are the proper abstraction for characters, not bytes.

Also I might be wrong, but you can just look at the bytes to know how much bytes a UTF-8 character is, since the first byte with value 0 to 127 represents the final byte while 128 to 255 represents that the next byte is part of the character.

<comment><USER4> Except that code points aren't the proper abstraction for "characters". Most people would think of <Family: Woman, Woman, Girl, Boy>[1] as one character, but it's really five code points; woman, zero width joiner, woman, zero width joiner, girl, zero width joiner, boy. If you tried doing an operation like reversing a string or removing the last character, and you treated a unicode code point as a "character", you would end up with the wrong result. If you just removed the last code point to implement a backspace, you would end up with a string which ends in a zero width joiner, which makes little sense; and when the user wants to insert, say, a girl emoji, that emoji will end up as a part of the family due to that trailing joiner, when the user expected it to be a separate emoji.

This applies to more than just emojis by the way; there are languages whose unicode representation is much more complicated than english or other languages with latin characters.

[1]: [https://emojipedia.org/family-woman-woman-girl- boy/](https://emojipedia.org/family-woman-woman-girl-boy/)

EDIT: This comment originally used the actual emojis as examples, but hacker news just replaced every code point in the emoji with a space.

<comment><USER5> You don't even need emoji; eg U+63,U+300 (c̀) is one characters but two code points, and U+1C4 (DŽ) is two characters, but one code point. There's also U+1F3,U+308 (dz̈), which is two characters in two code points, but segments incorrectly if you split on code points instead of characters.

It's ambigous how to encode latin-small-a-with-ring-above (U+E4 vs U+61,U+30A). Decoding is also ambigous (most infamously Han grapheme clusters), but I'm not fluent enough in any of the affected languages to have a ready example.

Also, that's seven code points, not five.

<comment><USER6> You example is a little confusing from the use of the word "characters". I think glyphs would be more clear (U+1C4 is two glyphs). Though it might not actually be 2 glyphs, its dependent on how the font implements it.

At the end of the day, an OpenType font through substitutions can do far more crazy things than these "double glyph" examples. I once made a font that took a 3 letter acronym, substituted this for a hidden empty glyph in the font, then substituted this single hidden empty glyph into 20 glyphs. You were left with something like your U+1C4 in software where you could only highlight all 20 glyphs of none of them. And this was happening on text where all input code points were under 127. People often don't realize how much logic and complexity can be put into a font or how much the font is responsible for doing.

<comment><USER5> No, "characters" is the correct term; U+1C4 is two characters: latin-capital-d followed by latin-capital-z-with-caron (or whatever you want to call the little v thing). As you note, this means that non-buggy fonts will generally use two glyphs to render it, but that's a implementation detail; a font could process the entire word "the" as one glyph, or render "m" as dotless-i + right-half-of-n + right-half-of-n, but that wouldn't affect how many characters either string has.

<comment><USER6> Using characters is confusing because I don't know if you mean before or after shaping. U+1C4 is unquestionably a single Unicode code point. I've heard people call this 1 logical character. Other people might say how many characters it requires for encoding in UTF-8 or in UTF-16. After shaping, some people might say it is 1 or 2 "shaped characters". It's all horribly confusing. I find using the term code point more precise.

<comment><USER5> There's no shaping involved; I'm not talking about the implementation details of the rendering algorithm. There is a D, followed by a Ž. This only seems confusing because Unicode (and - to be fair - other, earlier character encodings) willfully misinterprets the term "character" for self-serving purposes.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> > One issue to consider is that strings typically feature random access > indexing of code points

True but I find it's much rarer to actually need random access to arbitrary code points. Most of the time I either use strings as opaque "things" or I'm iterating through characters to find something interesting (e.g. parsing) where I can build my own index, if and when necessary.

I do agree with the article that if an abstraction is very leaky it's better to be upfront about that.

<comment><USER8> I think that accessing characters by index is _probably_ a code smell in most places, especially if that string may contain arbitrary UTF-8.

<comment><USER9> Accessing string content by arbitrary indices is probably an error. Accessing string content by indices you got from previous lookups is useful for a number of situations.

<comment><USER10> If you have an index from a previous lookup, that can be a byte index.

</comment></comment></comment></comment><comment><USER11> Iterating code points is OK, as long as you know that know that iterating code points is not the same as iterating grapheme clusters aka user perceived characters. You get away with it most of the time, but you should know you are not dealing with full Unicode and have a plan to deal with exceptions. Unicode normalization does not solve it all.

Unfortunately almost all "Absolute minimum you must know about Unicode" articles don't cover the absolute minimum you have to know about Unicode.

Arbitrary well formed UTF-8 combined with advanced string algorithms and data structures where the unit is 'char' requires more than code points.

<comment><USER12> Iterating grapheme clusters is OK, as long as you know that iterating grapheme clusters is not the same as iterating unicode scalars (code points) aka the fundamental unit of textual parsing grammars.

This is something that really bugs me about how Swift changed its mind and made the String type a Collection of Characters (i.e. grapheme clusters). Originally they recognized this issue and required you to write `str.characters` to work with the grapheme clusters as a collection (and String itself wasn't a collection at all), but then in Swift 3 (I think) they changed course and said String is a collection after all. And the problem is now people work with Characters without even thinking about it when they really should be working with unicode scalars.

In my personal experience, I only ever actually want to work with grapheme clusters when I'm doing something relating to user text editing (for example, if the user hits delete with an empty selection, I want to delete the last grapheme cluster). Most of my string manipulation wants to operate on scalars instead.

<comment><USER13> The rules for what you want to do on backspace are complex - you want to delete the grapheme cluster if it's an emoji or ideograph with variation selector, but if it's a combining mark, most of the time you want to just delete that. One place this is written down is [1].

Of course, this might sound like a nitpick but only confirms the actual point you were making, that treating text as a sequence of grapheme clusters is often but not always the right way to view the problem.

If you're talking about cursor motion when hitting an arrow key, then yeah, grapheme cluster.

[1]: [https://github.com/xi-editor/xi- editor/blob/master/rust/core...](https://github.com/xi-editor/xi- editor/blob/master/rust/core-lib/src/backspace.rs)

<comment><USER12> macOS and iOS delete the entire grapheme cluster on backspace, not just the combining mark (which is to say, backspace with no selection is identical to shift-left to select the previous character and then hitting backspace).

<comment><USER14> Not sure what scripts you intended your comment about, but this is not true in general. If I type anything like किमपि ("kimapi") and hit backspace, it turns into किमप ("kimapa"). That is, the following sequence of codepoints:

        0915 DEVANAGARI LETTER KA
        093F DEVANAGARI VOWEL SIGN I
        092E DEVANAGARI LETTER MA
        092A DEVANAGARI LETTER PA
        093F DEVANAGARI VOWEL SIGN I

made of three grapheme clusters (containing 2, 1, and 2 codepoints respectively), turns after a single backspace into the following sequence:

        0915 DEVANAGARI LETTER KA
        093F DEVANAGARI VOWEL SIGN I
        092E DEVANAGARI LETTER MA
        092A DEVANAGARI LETTER PA

This is what I expect/find intuitive, too, as a user. Similarly अन्यच्च is made of 3 grapheme clusters but you hit backspace 7 times to delete it (though there I'd slightly have preferred अन्यच्च→अन्यच्→अन्य→अन्→अ instead of अन्यच्च→अन्यच्→अन्यच→अन्य→अन्→अन→अ that's seen, but one can live with this).

<comment><USER12> Looks like you're right. I don't have experience with languages like this one. I was thinking more of things like é (e followed by U+301), or 🇦🇧 (which is two regional indicator symbols that don't map to any current flag), or a snippet of Z̛̺͉̤̭͈̙A̧̦͉̗̩̞͙LG͈͎͍̺̖̹̘O̵̫ which has tons of combining marks but each cluster is still deleted with a single backspace.

<comment><USER13> Interesting. The rules seem to be different on different systems. Deleting two RIS symbols (whether they map to a flag or not) seems right in any case. Some other systems (Android included) will take the accents off separately when they are decomposed (but not for precomposed accented characters). Also note macOS takes just the accent off for Arabic (tested on U+062F U+064D).

</comment></comment></comment></comment></comment></comment></comment><comment><USER15> The article does not mention Python, other than to reference CPython's "Flexible String Representation". However, it's interesting that alternative Python implementations have decided against that model and indeed use UTF-8 strings internally.

MicroPython saves memory by simply making indexing into its strings O(n) [1], while PyPy's UTF-8 strings have "an optional extra index data structure to make indexing O(1)" [2].

For compatibility, of course, Python implementations have to provide indexing of code points - it would be interesting to examine the pros & cons of the different string representations. I wonder if new high-level languages would be better off using one of these representations, or taking the Go/Julia approach of only indexing bytes.

[1] [https://github.com/micropython/micropython/blob/a4f1d82757b8...](https://github.com/micropython/micropython/blob/a4f1d82757b8e95c21a095c99b7c3f04ded88104/py/objstrunicode.c#L156)

[2] [https://twitter.com/pypyproject/status/1095971192513708032](https://twitter.com/pypyproject/status/1095971192513708032)

<comment><USER16> "इंडेक्स" का क्या अर्थ है? [1]

Including the quote marks, spaces, and question mark, that's 18 characters. This isn't just about text editing, far from it. For a lot of string processing, indexing into the underlying codepoints is even less interesting than indexing into the underlying bytes.

[1] [https://translate.google.com/#view=home&op=translate&sl=hi&t...](https://translate.google.com/#view=home&op=translate&sl=hi&tl=en&text=%22%E0%A4%87%E0%A4%82%E0%A4%A1%E0%A5%87%E0%A4%95%E0%A5%8D%E0%A4%B8%22%20%E0%A4%95%E0%A4%BE%20%E0%A4%95%E0%A5%8D%E0%A4%AF%E0%A4%BE%20%E0%A4%85%E0%A4%B0%E0%A5%8D%E0%A4%A5%20%E0%A4%B9%E0%A5%88%3F)

<comment><USER17> I am not a linguist, but as a native speaker, shouldn't they be considered 15 characters? क्स, क्या and र्थ each form individual conjunct consonants. Counting them as two would then beget the question as to why डे is not considered two characters too, seeing as it is formed by combining ड and ए, much like क्स is formed by combining क् and स.

<comment><USER16> If you say they should be considered 15 characters then software and devs should support optionally indexing and counting them as 15 characters. This is the most important point.

And, as a corollary, software devs should aspire to have and know about string functions in software that recognize that the text string I used is 15 characters long in contexts where that's the right way to view it. Furthermore, those functions should asap be as easily available for use as they are today for recognizing that the text 'What does "index" mean?' is 23 characters long.

This notion of software and devs properly indexing and counting characters was the ultimate point of my comment, as I will elaborate below. I hope that you will reply to confirm you understand the gist of what follows; that would make my day and leave this exchange on HN to hopefully shine light where it's sorely needed. :)

\----

The OP title is "UTF-8 String Indexing Strategies". I could write that this begs the question What does "index" mean? _Unfortunately it seems it still doesn 't beg the question -- in 2019 -- for most western devs._

Last century devs generally assumed the index unit was bytes. So they created programming languages whose string type assumed indexing in bytes and functions and libraries that did the same. Nowadays they're starting to assume "codepoints", which is an equally broken assumption. (Codepoints are a Unicode notion and they're great for what they're great for. But being "characters" is, in the general case, something they're terrible for.)

Both these western devs and the OP are effectively ignoring the possibility that "इंडेक्स" का क्या अर्थ है? could be considered to be 15 characters (or 18). They're ignoring you, the half of the planet that are in a similar boat, and the whole of the planet that's coming together, sharing text like we are here.

\----

bakery2k demonstrated the problem. They wrote:

> MicroPython ... indexing ... O(n) ... PyPy's ... O(1)

Neither of these deals with indexing _characters_ , as one might expect based on an ordinary human's understanding of the word "characters". Instead they're myopically focused on indexing bytes and codepoints.

This goes hand-in-hand with Python's length function returning 26 for the text "इंडेक्स" का क्या अर्थ है?. It's counting codepoints, not characters, which is close to useless for that text.[1]

But you wouldn't have any clue about that from bakery2k's comment and it looks like bakery2k has no awareness of this:

> I wonder if new high-level languages would be better off using one of these > [byte and codepoint] representations, or taking the Go/Julia approach of > only indexing bytes.

Imo that's shockingly retrogressive given the lack of discussion of characters.

\----

Chances are good that if you try to select the text I wrote one character at a time you will find that you can cursor across 18 units.

Why/how does software do this? It relies on part of the Unicode standard for indexing that builds on the concept of "what a user thinks of as a character".[2]

This mechanism allows the string to be indexed/counted as N characters, where N varies according to the definition of "character". Software is supposed to choose the definition with appropriate adherence to the Unicode standard, which includes customizing it as necessary to produce practical results. And, as I noted, most good modern software dealing with cursoring/editing text gets it right per the Unicode standard.

My guess the Unicode standard by default has software consider क्स to be 2 characters because the consonant is comprised of क् and स placed visually side-by-side whereas it has डे be considered 1 character because it's comprised of ड and ए somehow overlapping visually. (That's a pure guess. Please let me know if it sounds crazy. :))

For some other use cases, like a native speaker just reading text abstractly, what's a character changes. You say the text I wrote is 15 characters; therefore software should be able to index and count it as 15 characters.

I hope that all makes sense. Thank you for your comment, reading my reply, and TIA for any reply. :)

[1] [https://tio.run/##K6gsycjPM/7/v6AoM69EIyc1T0Nd6cGS9gdLmh4sWf...](https://tio.run/##K6gsycjPM/7/v6AoM69EIyc1T0Nd6cGS9gdLmh4sWfhgKZAx9cHS3gdLdigpgJhL9inARNZDOK0PlmwA85cCOTsfLO2wV9fU/P8fAA)

[2] [https://unicode.org/glossary/#grapheme](https://unicode.org/glossary/#grapheme)

<comment><USER17> Sorry for the late reply, I don't use HN much. No idea if you'll actually notice this, does HN even have a "Reply Notification" feature?

Regarding what you wrote, I agree pretty much. As I said, I am not an expert in this field, so I am not aware of the most cutting edge stuff put there. But even the few languages I know and have seen are so different from each other (some more than others) that it seems unlikely that a single "theory of everything" would suffice for text, especially in the way we process text presently.

Perhaps there is some way to abstract out the differences, but I don't really see how. After all, characters are where the differences only begin. Start thinking about words or sentences and no single route seems viable for the way we do string processing today.

You probably expected a more substantial comment, but I don't really know enough of this field to make one.

Regarding क्स and डे, the difference between them is that the former is a combination of two consonants (pronounced "ks") while the latter is formed by a consonant and a vowel ("de"). However, looking at the visual representation is wrong, since डा (consonant+vowel) would also look like two characters. If you copy these into a text field and try to erase them through backspace or delete, you should see how it all works (assuming the text field functions correctly).

But again, these confusions only exist because Devnagari allows simple characters to form compound characters. That is obviously completely different than how Roman script works, which is probably completely different than various pictographic languages. So, how to reconcile the differences (except by hiring native speakers of every language out there)? I wish I knew, but currently I don't.

</comment></comment></comment></comment></comment><comment><USER9> It's sad and odd that Rust and (probably especially) Swift are missing from the article.

<comment><USER18> Why are there interesting technical differences in the way those languages do things compared to the other examples given?

The author obviously can't cover _all_ languages and strategies in a short article can they?

<comment><USER19> Yes: Swift groups by grapheme clusters, and Rust makes it difficult to do byte indexing.

<comment><USER9> > Rust makes it difficult to do byte indexing.

Not sure how. If you want to get a specific byte, just convert to a bytes slice (that's free) and index that. And you can slice strings (using byte- indexed indices), but your boundaries have to fall on codepoint boundaries. The only thing that's difficult is getting a codepoint at a specific index (byte or otherwise).

<comment><USER20> > Not sure how. If you want to get a specific byte, just convert to a slice > (that's free) and index that.

But then it is not automatic to cast that slice as a string.

<comment><USER21> If you want a single byte and `s` is a `&str`, then `s.as_bytes()[i]` returns a `u8` in `s` at index `i`. If the index `i` is out of bounds, then it panics, but no other UTF-8 checking is performed.

You do not need to do this if you're slicing. For example, if you know that `i..j` indexes a valid UTF-8 subslice of `s`, then `&s[i..j]` returns a subslice of `s` with type `&str`.

The only reason to subslice `s.as_bytes()` is if you want the raw bytes which may or may not be valid UTF-8. And in this case, it is a _good thing_ that it is not automatic to convert that back to a `&str` since it may not be valid UTF-8.

<comment><USER20> > it is a good thing that it is not automatic to convert that back to a `&str` > since it may not be valid UTF-8.

My comment was unclear in meaning, but the aim was to point out exactly this.

</comment></comment></comment></comment></comment></comment></comment><comment><USER22> _"So, Emacs pretends it has constant time access into its UTF-8 text data, but it's only faking it with some simple optimizations. This usually works out just fine."_

Usually, except when you're writing in, and searching for Chinese, Greek, Hindi, Korean, Russian, Turkish, etc, text, like 50+% of the world's population? It seems Emacs is made for programmers, who predominantly type and search for ascii text.

<comment><USER23> That sounds about right. You don't pick up vim for a shopping list. You pick it up because you're a programmer.

<comment><USER24> Some people use Emacs just for Org. That's a lot closer to a shopping list than to programming. And programmers sometimes write text in natural language.

<comment><USER23> I would guess that the parent comment's point is still true: Emacs (and Vim) are far more commonly used for programming and other work, probably ASCII heavy, than for natural language text editing.

I'd be willing to bet that for both Emacs and Vim, 90%+ characters by volume are ASCII. I wouldn't make a similar bet for Microsoft Word.

</comment></comment></comment></comment><comment><USER25> Thanks for the article, I'm curious of what the author think of string indexing in Rust. It's also explicit so I guess you would like it as well

<comment><USER26> From my personal experience I think Rust's string system is hard to beat at the moment. It's pretty darn good from a usability point of view and it also found a nice solution to work with UCS2 windows APIs by providing a OsStr type.

<comment><USER27> I'm glad that Rust strings aren't indexable by integer, but I think that making them indexable by range (of UTF-8 code unit offsets) was an error. `foo[0..10]` should have been `foo.slice(0..10)` or similar instead.

<comment><USER26> It's a bit of a footgun indeed but it's quite handy in combination with the char index iterator.

<comment><USER27> Sure, you do want to be able to index by code unit range, but it shouldn't have been with the Index trait.

</comment></comment></comment></comment></comment><comment><USER28> Could you store the string in multibyte form, and then keep a skip list (or other data structure) to get indexing in O(log n)?

</comment><comment><USER29> The xi rope science blog post series is, I think, the definitive answer to UTF-8 "indexing".

</comment><comment><USER30> Isn't it just best to do NFKC (or similar) normalization on input for indexing?

</comment><comment><USER31> It's a good article, but it would be nice if he'd also covered the Python 3 C API anti-pattern: forcing strings to be utf-8. This means that you have latent bugs in your code. Notably when trying to treat all filenames as strings, sooner or later your code will explode when it meets someone's filesystem which has ancient Latin1 filenames. Also when dealing with unfiltered user input.

(And to head off replies - yes I understand you can "just do X" where "X" is some complicated thing to avoid the bug if you remember to do "X" beforehand)

<comment><USER32> I don't really consider that an anti-pattern. Sure, you can get away with just blitting them to the terminal and hoping that they display properly, but sooner or later you're going to have to decode such byte strings anyway.

The real anti-pattern is conflating byte strings and character strings in the first place. We got away with it for decades, but in a UTF-8 world it just isn't possible.

<comment><USER33> I don't see why not? Go has a string type that contains arbitrary bytes, interpreted as UTF-8. This seems to work as well as anything else. If there are non-codepoints and it matters, you just have to deal with it (for example by printing an escape sequence or Unicode replacement character).

[https://blog.golang.org/strings](https://blog.golang.org/strings)

<comment><USER24> Because arbitrary bytes cannot be interpreted as UTF-8. I guess this kind of thing is tolerated by Go users because anyone who values a proper type system uses a language with generics.

<comment><USER33> How do you fix a file that has errors in it if the standard library of the language you're using won't even let you read it?

<comment><USER32> If you're fixing bytes then you load bytes and fix them.

You won't, though, fix bytes by loading characters and then trying … to fix the bytes … the characters encode to. Just doesn't make sense.

We were able to get away with stuff for a long time because bytes were characters and characters were bytes and we could think sloppily and not break anything. But with Unicode they really are different things, and we need to be tidier in our thinking.

<comment><USER33> Seems like you're just reasserting it doesn't make sense, without giving a reason. But it does make sense in Go.

<comment><USER32> > But it does make sense in Go.

No, Go doesn't work that way. You asked, 'How do you fix a file that has errors in it if the standard library of the language you're using won't even let you read it?' In Go, you don't read file as strings, but rather as bytes (proof: [https://golang.org/pkg/os/#Open](https://golang.org/pkg/os/#Open), which returns a File which implements Read: [https://golang.org/pkg/os/#File.Read](https://golang.org/pkg/os/#File.Read)).

You would do the same thing in Python: open the file in binary mode, and the iterate over the bytes it yields.

Now, the one thing that _would_ be annoying in Go is fixing a broken filename. I'd have to think a bit to figure that out.

<comment><USER33> You can cast between byte arrays and strings in Go. The difference is that strings are immutable (so it does a copy).

<comment><USER32> > You can cast between byte arrays and strings in Go.

Yes, you can. But, in the specific case you mentioned, no competent programmer would cast the bytes of an invalidly-encoded file to a string, then iterate through the runes of the string. That wouldn't even begin to make sense!

I really don't understand what you're trying to argue here.

<comment><USER33> Although it only works for smallish files, that seems fairly useful for getting as much info as you can out of a corrupt but mostly UTF-8 file?

Any runes that aren't valid will come back as the replacement character. And you can count newlines and print the location of the error(s). You also have the index of the error.</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment>


The Future of Sex: Androids, VR, and the Orgasm Button - <AUTHOR> http://singularityhub.com/2009/05/20/the-future-of-sex-androids-vr-and-the-orgasm-button/

<comment><USER2> Zzzz...the first article I read like this was written in the 1970s. I am so tired of this formula...

Thought you'd seen it all when (someone) did (that thing) (within the last year)? Think again, because (specific technology) is changing the way we (do something) faster than ever before.

Imagine a world where (well known person) does (the same thing with technology). Or where (those kinds of people) do (something else with technology). Or even (well known event) is (different with more technology).

But will things be better or worse? (something) would be awesome. But maybe (something we value) would disappear.

One thing's for sure: before long (repeat half of first paragraph) will seem like (some old example of technology chaning something). And maybe we'll (have some emotion) while we (do something familiar with new technology).

The best (or worst) example I've seen of this kind of lazy journalism was an article in the Economist speculating that musicians might one day take performance-enhancing drugs. You don't say.

<comment><USER3> The part of how "New treatments will cure STDs." That's been wished for since before the 70's. Still not there.

<comment><USER4> Some STDs have been treated with success, e.g. with antibiotics.

And even AIDS victims live vastly longer with the new drugs than before. We just do not have a silver bullet.

</comment></comment></comment><comment><USER5> With all sincerity, if we could have a button to induce sexual pleasure, would we want it? The root of this article is the promise/expectation/hope of quick, easy, sexual pleasure through the use of technology.

For a quick analogy of what might happen think about what cheap, easy calories are doing to us as a species. Now think about how much more emotionally powerful an orgasm is vs. say, a piece of chocolate...

<comment><USER6> _With all sincerity, if we could have a button to induce sexual pleasure, would we want it?_

If people did, masturbation might be common. Since it's not, I guess we have our answer.

<comment><USER3> Is that sarcasm? I thought that the Internet is for Porn, and the late 20th century was the start of the biggest explosion of...bothers! How the heck do you talk about this stuff without sounding like a /b/tard?

In any case, Internet Porn as it is now is close enough to a button to induce sexual pleasure. The dangerous thing is not producing raw pleasure. It's producing quality sexual experiences. If you could record and replay the experience of having sex with a very desirable partner, in the words of Dennis Miller, this will make Crack look like Sanka.

Given that the promise of sex is such a strong compulsion (there's a reason why advertisers and gangsters gravitate to it) is there any hope of resistance?

<comment><USER6> _How the heck do you talk about this stuff without sounding like a /b/tard?_

With sarcasm.

</comment></comment></comment></comment><comment><USER7> I'm having trouble finding the article, but AFAIK this was done at least once on a human volunteer (wiring the pleasure center). The experiment revolved around using positive reinforcement to "cure" homosexuality. Needless to say it was shut down when it became clear just how strong the induced feelings were (the subject became obsessed with the remote).

</comment><comment><USER8> How can you not click on that link... ;-)

<comment><USER9> Seriously, I'm impressed that the monkey (rta) only pushed the button for a mere 16 hours a day :-)

<comment><USER8> I seem to recall a ratty experiment in which lab rats had electrodes inserted into pleasure centres in the brain. They could stimulate the centre through pressing a bar. (Which is what lab rats do best.)

IIRC, the rats pleasured themselves to death, eschewing food and drink, in order the stimulate the pleasure centres.

<comment><USER10> Most animals don't have sex for very long at a time; I wouldn't be surprised if the pleasure wiring overrides any ability to feel any other urges, because of the rarity (and importance) of its activation. I imagine it would be quite different in the human case; things frequently "interrupt" people.

</comment></comment></comment></comment><comment><USER11> If VCRs and the web are any indication, robot sex will be a natural follow on to vacuuming floors and welding cars. I recommend Charlie Stross's Saturns Children for sex from the robot's point of view. And Love and Sex with Robots is on my wish list at Amazon. (Not the robot itself, thank you.)

The future is past. The bionic dick was a cure to impotence before the advent of Viagra.

</comment><comment><USER12> Scott Adams (of Dilbert fame) has been predicting this for years.

</comment><comment><USER13> There seems to be an online sexual revolution going on that I find shocking. ie ... ppl putting themselves on xtube and cam4 doing well.

I mean don't these people have jobs, friends and family?

<comment><USER3> There is a great Frontline documentary that addresses this on PBS:

<http://www.pbs.org/wgbh/pages/frontline/shows/porn/>

Basically, the widespread distribution of porn through emerging technologies has caused a huge cultural shift in the US and around the world.

</comment></comment><comment><USER14> All I can say is nope! :)</comment>


Ask YC: when is the next round of funding? - <AUTHOR>

I am curious to know when the next round of funding is, and when would the cycle start?  My guess was sometime in March is the application dates and the cycle would start end of June in Boston.
For those that are interested to apply I believe it would be nice to have some solid dates to make sure it wont be missed.
I think any sort of guide lines for the future applicants would be helpful, in order to plan things right and as early as possible.
Thanks in advance <comment><USER2> Some advice from applying last round:

1) Have a team in place you have evidence of working on projects on

2) Have great cofounders who have track record of doing stuff

3) Apply with an idea that is making a step change in a new or existing industry and has a way of making money directly or indirectly.

<comment><AUTHOR> thank you</comment></comment>


Show HN: I built a service to find cool people on Github - <AUTHOR> http://githubfriends.swizec.com

<comment><USER2> No way I'm signing in with my twitter account to any of your projects. I'm being cautious because I still remember your blogpost about storing passwords of 3rd party accounts in plain text.

Relevant: [http://swizec.com/blog/small-trick-for-seamless- base64-passw...](http://swizec.com/blog/small-trick-for-seamless- base64-password-storage-in-django/swizec/1378)

<comment><USER3> I don't think you understood his post, or how you sign into a third-party site with Twitter or Facebook.

His post was about sites that DO NOT do OAUTH, and the only way to store passwords to interface with those sites. He clearly states how insecure it is, and the base64 thing is only to prevent you from reading their password accidentally.

Twitter, however, DOES use OAUTH. There's no need to store your twitter password on his side at all.

His auth page is down right now so I can't check, but I'm betting that it ends up at a twitter.com OAUTH page that is perfectly safe.

<comment><USER2> I understood his post perfectly well. By using base64 to store passwords is just as bas as storing them in plain text, because you only rely on the good will of the person who has access to them.

When dealing with security, this sort of practice should be avoided at all costs e.g. a sacrifice to the functionality of your product.

<comment><USER4> Are you suggesting that Twitter is going to give him your password, so that he might store it in plain text?

I don't understand how that could happen. Twitter's OAuth implementation seems fairly sound.

</comment></comment></comment></comment><comment><USER5> I think the concept is cool but twitter is probably not the best vehicle of choice for me personally. Maybe it'll work well for others.

<comment><USER6> Just out of curiosity.

Where would you look instead?

Could be cool to get a list of alternative ways to twitter and facebook.

Are other sites having profiles that could be accessed such as SO etc?

<comment><USER5> I'm not sure as I haven't thought about it before so I'll think about this and hopefully try to post back but people I follow on twitter or friends I have on facebook probably aren't the best demographic. Chances are people I know, I'd be linked with on github if it mattered and for the vast majority of people linked together on facebook and twitter are probably not really all that targeted. Sure they'd be some correlation but it'd probably be a low percentage.

I'm not sure where you'd get a list like this (maybe you'd just create your own) but I think linking people to common languages (i.e. python guys to python guys) and similar projects/interests they like would have a higher correlation.

</comment></comment></comment><comment><USER7> The "sign in with Twitter" button doesn't appear to do anything (Safari 5.05 on OSX 10.5.8).

</comment><comment><USER8> Bug: it found some people on github but all link to github.com/${login}

</comment><comment><USER6> It would be great if you could bulk select those you wanted to follow.</comment>


College Cheating Scandal: An Admissions Officer Speaks Out - <AUTHOR> https://www.thecut.com/2019/03/college-cheating-scandal-an-admissions-officer-speaks-out.html <comment><USER2> I don't think it will really happen, but I can at least hope that this admissions scandal will shine a light on the scam that the "meritocracy" is, and I say this as a Harvard alum.

At any of the most elite universities there are many more qualified applicants than there are spots. For example, Harvard could fill its undergrad ranks with valedictorians alone, twice over. Thus, the fair thing to do would be to either (a) drastically increase the size of the incoming classes, or (b) set a minimum level of objective criteria (e.g. test scores, class standing, etc.) and then just run a lottery from those qualified students.

But instead, the admissions offices at schools run a system much _worse_ than random. It's a system that's highly subject to pay-to-play access (and usually not the outright fraud kind, but more the endow-a-chair kind) while at the same time allowing the universities to keep up the charade that it is all merit based. Even if there weren't undo influence, the current efforts to maintain "diversity" (and not just the racial kind, but the kind where we need 5 cello players and 7 chess club champions, etc) means the admissions criteria is already completely arbitrary if you meet the basic (albeit still tough) academic requirements.

The reason this won't change, though, is _the primary purpose of today 's elite universities is to maintain social stratification_. If universities did the fair thing and had a lottery for all qualified applicants, it would mean they would have to admit that they are not the objective arbiters of "potential" they purport to be.

<comment><USER3> I thought by "pay-to-play" you were going to point out the other opportunities to use money to perpetuate social stratification. It's not just "endow-a- chair" levels of wealth that improve the odds, it's also:

\- Live in a rich metro area? Now you have access to things like internships & research opportunities not available in low-SES areas.

\- Parents have high-status jobs? Now you have access to experiences offered by your parents' friends and colleagues, for example, to do research at a local university lab if your parents are professors.

\- Don't have to work summers? Now you can take a "finding yourself" trip or volunteer to build housing in Africa or whatever it is these days.

It's much easier to engage in extracurriculars that show how great/community- oriented/socially-conscious/talented at music/sports/research/debate/whatever you are when you grow up in an upper- or upper-middle-class family in a wealthy-enough neighborhood to parents with strong social and professional networks.

This is the social stratification that's perpetuated by elite school admissions processes, it's the top-20%, not the top-1%. It's not any less dangerous, and in fact may be more so -- and the income numbers alone don't tell the full story.

Lastly -- those wealthy enough to endow a chair don't need Harvard to perpetuate social stratification, they will simply inherit enough wealth to do so!

<comment><USER4> like 15-20% of Harvard admits come from top 1% of household income. I think like 35-40% come from households making more than 300k.

<comment><USER3> Indeed: [https://www.thecrimson.com/article/2017/1/25/harvard- income-...](https://www.thecrimson.com/article/2017/1/25/harvard-income- percentile/)

</comment></comment></comment></comment><comment><USER5> Nothing about this surprises me. Just another thing in the long list of ways that wealthy people are able to keep their wealth, and pass it on to their children.

I had college paid for in full by my parents, and they also were able to provide a good portion of the down payment on my house. My parents aren't exceptionally wealthy, but they have been able to give my siblings and I a very good head start on life. And if they were wealthier, they would have been able to provide even more advantages to us. The whole system is rigged, and has been for a very long time.

<comment><USER6> What's stopping you from going full Siddhartha, selling your house/earthly possessions, and giving the wad to the first homeless person you see on your way out of town to live your best ascetic life?

Caring for and providing as much of an advantage to your kids as one is able is a basic human (one could even argue 'mammalian') behavior. I'm disappointed that the narrative is trying to label such hard-coded behavior as everything from 'evil' to 'frowned upon'.

<comment><USER2> > "I'm disappointed that the narrative is trying to label such hard-coded > behavior as everything from 'evil' to 'frowned upon'."

The problem isn't so much that it is frowned up or declared evil, but it's that we have somehow built the worst of both worlds: we are most definitely _not_ a "privilege-free" meritocracy, but we like to pretend that we are. The British aristocracy gave their kids all the advantages they could, too, but at least everyone knew that system was unfair and the concept of "noblesse oblige" came out of that knowledge. We've replaced noblesse oblige with "I got all my success through my own brilliance so screw your lazy ass."

I'm not suggesting we return to an aristocratic system, but I am suggesting we fully acknowledge the role of unfair advantage, and luck, in anyone's success, in addition to their own skill and hard work.

One of the things I found most interesting about the current scandal is that one of the parents paid over a million dollars to commit this fraud. With numbers that big you have to wonder why they just didn't give a donation that large - probably would have had a similar effect. I think one main reason is that a donation would make everyone say "Oh, well your kid got in because of your huge donation." (See Jared Kushner). Committing the fraud allows the fiction that it was just skill and hard work that was the reason for the acceptance to continue.

<comment><USER7> I think you have an overly rosy view of the aristocracy of old. That social institution involved a hereditary monarchy, and you don't have to go very far back before the term _divine right_ was being thrown around. Talk about entitlement...

If you go back in history, I believe you'll find that most socially advantaged people simply thought they were inherently better than others. "I worked harder", whether true or not, is a better narrative.

<comment><USER8> _If you go back in history, I believe you 'll find that most socially advantaged people simply thought they were inherently better than others._

That's true now, is it not? It's certainly a common view. People who are rich deserve it; they are more deserving. They're better people. They must be, because how else would they be rich? In the US meritocracy (and indeed, many others), anyone who has done well must have done it through merit because that's how the system works, so they must be a better person.

<comment><USER7> My understanding is that a few hundred years ago most wealth was inherited land. Aristocracy was something you were born into. I'd love to see some statistics, but I believe the percentage of "self-made" people is _much_ higher today.

Modern society may have its flaws, but I do think this particular historical trend is positive.

<comment><USER8> Although the trend is now heading back towards that (in some places, including the US and the UK); the wealth of your parents is a stronger indicator of future success than it was fifty years ago.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> I find the nuance in the reporting to be quite fascinating depending on who the perpetrators are.

When it was rich Chinese international students doing the cheating, most articles would almost always allude to how it was cultural and how it reflects on the general population.

Now that's it's rich American students, the reporting focuses on wealth and class and how it's an isolated incident.

<comment><USER10> Well yeah. The poor certainly don't have the means to bribe Ivy League schools.

</comment></comment><comment><USER11> >I saw these decisions flipped frequently for students from affluent backgrounds, and rarely for students who'd applied for financial aid.

>Although our school advertised our "holistic" review process, our director typically used test scores to screen applicants. His rationale was that these were "riskier" students. The only time he didn't? If the student could pay full price to attend our institution, or a "full pay" student.

My main takeaway from this is a reminder on how reliant schools have become on gigantic, bloated tuitions to finance a large part of, if not all, their operations. Whee you need to keep paying for pricey administrators, new dorm buildings, and other likely-not-vital-for-education pieces but your state budgets have been slashed, endowment isn't paying out as much this year, and lower income students don't have access to as many loans - of course schools are going to put a preference for students from a wealthy background.

I think if you want to keep colleges fair, you need to keep their budgets more bare bones and focused on the essentials of schools (teaching and research), OR charge those who can afford the vast tuitions out of pocket with even higher tuitions to offset the payments for more qualified, but less able to pay students.

</comment><comment><USER12> What is the value of a college education? This is a contentious question that I believe more and more people should be asking, and debating about.

Is the value in the education that one receives, or in the the network that one gains access to? They're really both sides of the same coin: they both represent opportunity.

Ideally, we could have both – just admit all the high-achieving students from upper middle-class backgrounds. When there's no more space, just simply expand: build some dorms, hire a few more professors, etc.

But in reality, where resources are already scarce, schools face a tradeoff: do we admit the bright student from a poorer background, or the wealthier, albeit not as academically-inclined, student? Here, admissions basically become a zero-sum game.

I guess it boils down to an argument of whether intellectual capital is to be more valued, or of social capital.

</comment><comment><USER13> Students have slowly been reframed as customers - it's no surprise that admissions may turn into form of customer screening.

<comment><USER14> It is the Ivy League - there is no "turning" about it. Many practices were goal-post moving to keep model minorities out including ridiculously biased vocabulary in test questions like yaucht sailing vocabulary which is covered nowhere in even the most in depth historical or literaty sources.

</comment></comment><comment><USER15> I would rather hear from an admissions offer from HYP and Stanford, where average admit rate is 5 to 6%. This guy worked for schools whose admit rate ranges from 60 to 75%; so, the target for such schools is different from that for HYP.

<comment><USER16> I heard exactly the same from a friend who worked in admissions at an ivy league university. Applying for financial aid materially damages your application. Ability to pay is factored in just like test scores. The rich continue to get opportunities poor kids will never have.

<comment><USER17> [https://en.m.wikipedia.org/wiki/Need- blind_admission](https://en.m.wikipedia.org/wiki/Need-blind_admission)

It seems like there are only a few dozen U.S. colleges (43 by my count) that officially have need-blind admissions, so at any school not on this list ability to pay should be expected to be a factor.

<comment><USER16> I can tell you that list cannot be trusted. The university I spoke of is on that list, but seeing as someone who was in the room making admissions decisions said they consider financial need, it's clearly bullshit.

</comment></comment></comment></comment><comment><USER18> It's shocking to read justification of the bribery and fraud in this case as being just a manifestation of the drive to impart advantages to one's offspring.

I have that same drive. I can't imagine living with a morality that would enable me to pay bribes and falsify athletic achievement, or disability. I can't imagine involving my children in federal crimes.

There are lesser unfair practices, such as the ones the admissions officer in the article describes, and these should be fixed, too. But these lesser unfair practices are different from committing federal felonies. Especially when you are in technology entrepreneurial management, investing, or running a major law firm. It makes one wonder how deep the ethical rot goes.

</comment><comment><USER19> My family was part of that "full pay" segment of the population, at least when I was in school - it makes me wonder if my skew to private college (vs public) acceptances could be part of this, since I was an average student.

</comment><comment><USER20> Is it actually a bad thing if people who pay for it can get a good education?

Shouldn't we rather at the other end of the ladder, people who can't pay, and see to it that they can get a good education?

<comment><USER21> In that case you could make it more transparent and outright make it a bidding war over who gets in, rather than the whole charade of test scores and what- not.

</comment></comment><comment><USER22> TLDR; The admissions are heavily affected if students expressed need for financial aid. The OP worked at two schools, one that had little endowment liberal arts college not requiring SATs. This one doesn't surprise me. The other one is state college with 75% acceptance rate for in-state students. This one doesn't seem like problematic. OP isn't clear which school he/she is accusing of.

Overall, its well known fact that rich people's kid can chose great college as they prefer. Everything from Stanford to Harvard have seats reserved for donors.

<comment><USER23> I have friends that worked in admissions office of a state university. Normally, admissions rate will be high for in-state students because college- prep would have started long before application time. And through local alumni, teachers, and counselors students will have a good idea which state schools they'll be accepted into. But out of state and international admissions is completely different. When state budget cuts roll down that admission rate jumps from 10-15% to 25% and up. And if you can pay full freight that rate jumps to 100%.

When I found that out I was less shocked than I thought I would be. It just confirmed thoughts that I observed but failed to see.

</comment></comment><comment><USER24> Peter Thiel has been warning about education turning to exclusive clubs since years ago. i dont know if much can change though

</comment><comment><USER4> I read in an article somewhere that, American system heavily emphasis the brand of an Institution, which is markedly different from European institutions (Is this true?). This brand is valuable to people because they believe that the outcomes in their lives will become markedly different based on it. I think if we can somehow design a system which is indifferent to pedigree of an institution and rather afford opportunity based on individual merit (maybe university blind job admissions at the top companies), then the need to attend prestigious institutions will lessen.

<comment><USER25> European schools are taxpayer-funded, so students don't have to think as much about 5-figure differences between in-state tuition at a public university and full tuition at a private school.

The "pedigree" of US schools developed because the top schools have gigantic endowments from rich alumni, which allows them to build better facilities, hire top teaching talent and fund research. This is a cycle that is reinforced by alums (many at successful companies) who choose to recruit at those schools, thereby creating a new class of well-paid employees who're in a position to contribute to the endowment down the line.

For some industries like investment banking and finance, it's extremely important to do your undergrad at a "target school", where banks come to campus directly to run info sessions and do preliminary interviews. Otherwise you may not have a real shot unless you do an MBA, this time at a target school.

</comment></comment><comment><USER26> In this article, we report on the fact that businesses preferentially seek out customers who pay more.

</comment><comment><USER27> "75% in-state acceptance, 25% out-of-state" yet out-of-state usually pays double? Are 75% of applicants "affluent"? I'm sorry but a lot of these recent biased articles smell a bit of BS lately...

<comment><USER28> Already accepted students on first pass are likely not then questioned, so make up majority. The Deny/Check students that are affluent are subsequently 'reviewed' and allowed would be a percentage of the 75%

</comment></comment><comment><USER29> It seems like there is a balance of meritocracy and capitalism which is fine, but society should have a say in where that balance lies.

<comment><USER25> Especially since the top American schools are not-for-profits and aren't taxed on tuition fees received.

</comment></comment><comment><USER30> Make the SAT so difficult that on average 1 person gets a perfect score each year. Then do not allow any other criteria for admission.

To prevent bribery randomly sample 1% for a retest under much stricter security.

If you must, lower standards for protected classes to even out demographics.

Rich people can afford to burn their children's childhood on the enrichment activities and "volunteer" work Harvard requires.

Standardized testing is by far the most fair way to do this. SAT prep is not that effective and besides SAT prep is now available online for free.

<comment><USER31> Here's a solution. Make college free for all. Anyone with a HS diploma can get in. But, you MUST attend the classes and do the work or you're kicked out end of semester.

Totally based on merit. No biases on income or anything else.

<comment><USER32> Who pays the utility bills? Professor and administrative salaries? Is anyone allowed in just due to having a diploma? What keeps rampant cronyism and end of year cost splurges to keep high budget levels for the next year? Who sets the budget? How does enforcement work, government employees doing enforcing?

In short how do you pay for it given the already spiraling out of control costs of higher education?

<comment><USER33> All fairly simple issues that many countries have solved.

<comment><USER34> I suppose so, but it's worth noting that no other nation's tertiary education system holds a candle to the United States.

<comment><USER35> Switzerland and England both have schools in the top 10.

<comment><USER36> Those schools don't admit everyone, and certainly not everyone for free.

<comment><USER37> I can't speak for Switzerland but all universities in England charge the same for UK students with nothing upfront. Sure they don't have space for everybody who meets the minimum but that's inevitable. There are no legacy admissions, athletic admissions, positive discrimination etc. Everybody goes through the same process.</comment></comment></comment></comment></comment></comment></comment></comment>


Knuth and the Unix Way - <AUTHOR> http://www.leancrew.com/all-this/2011/12/more-shell-less-egg/ <comment><USER2> McIlroy's basic point is that, instead of building single-use programs, we should be directing our effort towards building reusable components, i.e. Unix utilities. Then, over time, most programs could be built as a simple piping together of high-level components.

This vision seems to have failed for about 5 decades now, but it is still a popular one. After no one was able to build serious applications by gluing together Unix utilities, it was thought that we would instead build serious applications by gluing together objects. This vision has also failed (or at least, not succeeded in a very apparent way). In the mean time there have been other promised types of "componentry", like COM, CORBA, WS-*, ..., all more or less complete failures (or at least, egregiously unapparent successes).

So, in my view, it is Knuth who has the last laugh.

<comment><USER3> I strongly disagree with the notion that the composition of small programs to build larger ones is a "failed vision". Shell scripting is an invaluable tool in the daily-use toolbox for a huge number of people, myself included. And what is a shell, besides a programming language for gluing together other programs?

Now, if you mean that no one is writing full-scale professional desktop or web applications by piping together sed and awk, then of course you are correct. But many projects come closer to this than you might think. Git, for instance, was originally just a large collection of shell scripts, which has over time been gradually rewritten in C. Many pieces of it are, even today, still written in the POSIX Bourne shell.

To me, the principal reason for the astounding longevity of Unix is its emphasis on composability. The tiny, ubiquitous Unix utilities are ancient compared to virtually every other piece of software in common usage, yet are no less useful today than when ken and dmr first conceived of them. By making the system modular and composable at a very fundamental level, they ensured (perhaps as much by accident as by intentional design) that users far into the future would be able to continue using their tools for purposes not yet dreamed of. And indeed, this is very much the case, and is likely to continue that way for some time to come. Whatever the eventual successor to Unix turns out to be, it's likely to have the same emphasis on small building blocks with enormous synergy when composed.

Likewise, when you say that the vision of "building serious applications by gluing together objects" has "not succeeded in a very apparent way", I frankly have no idea what you're talking about. OOP is the dominant paradigm in modern programming, bar none. You might argue that some other paradigm (FP, perhaps?) would have served us better in retrospect, but the last thing anyone can truthfully claim about the object-oriented approach is that it has not been successful. It's hard to think of even one popular programming language that doesn't borrow at least a few ideas from that school of thought.

I do agree with you about the "enterprisey" component approach being largely a failure. Thankfully, CORBA, SOAP, et al. seem to be mostly behind us, or at least rapidly receding.

<comment><USER4> "Git, for instance, was originally just a large collection of shell scripts, which has over time been gradually rewritten in C. "

Doesn't that prove the parent's point that the collection of supposedly reusable components could not be glued together to form GIT? There must be a reason why it was rewritten in C. And if it had to be rewritten then I don't think we can call the "large collection of shell scripts" a complete success.

"OOP is the dominant paradigm in modern programming, bar none."

There could be many reasons for that but it does not address the point that the actual gluing together of objects has somehow succeeded. How much of that OO work is actually shared between projects? How much of it is simply OOP for the sake of it because the language was chosen for convenience?

To me success would be using objects from one project without modification in a completely separate project. I just don't see that happening very often. If the objects are simple enough you might as well just rewrite them. If they are complicated enough to be of value to import, they usually require modification.

<comment><USER5> _Doesn't that prove the parent's point that the collection of supposedly reusable components could not be glued together to form GIT? There must be a reason why it was rewritten in C_.

Ironically the main reason for the rewriting was the poor performance of the fork system call in Windows (Cygwin) which make the system slow particularly for shell scripting.

The problem was Windows, not Git.

<comment><USER6>
        http://libgit2.github.com/

from the blurb: libgit2 is a portable, pure C implementation of the Git core methods provided as a re-entrant linkable library with a solid API, allowing you to write native speed custom Git applications in any language which supports C bindings.

and

        www.jgit.org

The point here seems to be that (ba)sh scripts are wonderful for one-time tasks. Not so much for anything else/permanent workflows.

</comment></comment></comment></comment></comment><comment><USER7> A cheap shot by McIlroy. Knuth was specifically asked to write in the literate style.

Word counting is one of those simple, domain-independent problems that lend themselves well to code reuse. It's a rare type of problem. Most tasks presented to a professional software developer could _not_ be solved by a small shell script. A large and unmaintainable one, maybe.

<comment><USER8> McIlroy's code lends itself very well to literate coding! Here's a try:

# First, _tr_ anslate multiple _s_ queezed occurrences of the _c_ omplement of A-Za-z (non-word-characters) into a line separator

      tr -cs A-Za-z '\n' |

# and then lowercase every word.

      tr A-Z a-z |

# Sort the words with a disk-based mergesort.

      sort |

# Count the unique characters :: [String] -> [(Int,String)]

      uniq -c |

# And do a reverse numerical disk-based merge sort.

      sort -rn |

# And write the first $1 lines and then quit.

      sed ${1}q

Personally, I'd change the last two to be

      sort -rn -k 1,1 | head -n $1

but that's just bikeshedding. And the parts it's made from are so modular, and so focused, that you can wrap your head around all of the problem, without worrying about how sort works, or how tr expands character ranges.

If I gave a "professional software developer" in 2011 a problem to count word frequencies, and I got back 10 pages of Pascal that didn't go much faster than code that fits on a Post-It note for my problem, I wouldn't trust that developer with anything else important.

<comment><USER9> Your code fails for French text.

<comment><USER8> As did Knuth's; see below the discussion about that.

If you wanted to change it, to split "words" by blanks and punctuation characters, you would have

      tr -s [:blank:][:punct:] "\n"

and then that piece would fit into the pipeline, the rest unscathed. This design is far easier to reason about than a dozen pages of Pascal.

<comment><USER10> Those globs should be protected from the shell otherwise ./bc is going to alter tr's arguments.

</comment></comment></comment></comment></comment><comment><USER11> To be fair to Knuth, it's not the sort of little utility program that he usually works on. At least from TeX, Metafont, and the TAOCP books, it's usually really big problems, or very low-level things where cycle counts matter. The two sorts in McIlroy's pipeline guarantee that his solution won't scale well at all, especially when only the top 10 are wanted.

</comment><comment><USER12> I know that this really isn't the point, but this is bugging me: McIlroy's pipeline doesn't properly handle words like _don't_ or _home-spun_.[1]

My first thought for a fix (still split on spaces, but now we need to remove left-over punctuation):

        tr -s '[:space:]' '\n' | tr -d '",.!?' | etc.

I doubt the _tr_ of that time had the character classes, but it gets at the general idea. I'm sure there are better alternatives, and I would be happy to hear them.

Oh, and also, we would need to remove single-quotes that might be left-over too, but _only_ if they are at word boundaries, rather than within words. I'm going back to work now...

[1] <https://gist.github.com/1447872>

<comment><USER13> Does Knuth's version handle those words? Since that would discredit the article's claim that McIlroy's version was bug-free.

<comment><USER14> No, Knuth's version doesn't handle those words either. Here's how he defines a _word_ in his literate program:

 _Let's agree that a word is a sequence of one or more contiguous letters; "Bentley" is a word, but "ain't" isn't. The sequence of letters should be maximal, in the sense that it cannot be lengthened without including a nonletter._

</comment></comment></comment><comment><USER15> Reminds me of this: <http://xkcd.com/664/>

<comment><USER16> The alt-text is sobering. How many ground-breaking ideas are put into everyday appliances?

<comment><USER17> _For every 0x5f375a86 we learn about, there are thousands we never see._ </quote>

I'd say 0x5f375a86/10000 ~ 160000 (known) / 1 (never see) is a highly optimistic ratio.

<comment><USER18> I'm not sure if you're joking, but that hex number isn't a count. It's a reference to a specific instance of an elegant solution to a problem (the fast reciprocal square root function from Quake).

<comment><USER19> For those who are curious, Wikipedia has a fascinating explanation, and the C code in question:

<http://en.wikipedia.org/wiki/Fast_inverse_square_root>

</comment></comment></comment></comment></comment><comment><USER20> > I can't help wondering, though, why he didn't use head -${1} in the last > line. It seems more natural than sed. Is it possible that head hadn't been > written yet?

I'm not sure when head(1) was added, but I doubt it was added by anyone in the Unix core team at Bell Labs, it certainly didn't make it to Plan 9 because it is redundant with sed as the example illustrates.

sed 11q is concise and clear, no need for shortcuts, but if you really have to, you can write your own head shell script that just calls sed.

<comment><USER21> sed is concise and clear if you know the command set, which was true of all developers at the time. But in the modern world, we all think in awk^H^H^Hperl^H^H^H^Hruby or whatever. On the command line, generic tools like sed, while they still work, have lost a lot of their relevance in favor of simpler stuff like head/tail which have much clearer semantics.

</comment></comment><comment><USER22> So, Knuth wrote it in 10 pages from the ground up?

But, the other guy just used unix commands.

I wonder how many lines of code all those utilities add up to.

Probably more than 10 printed pages.

<comment><USER23> I checked the Plan 9 tools, which are similar to the Unix ones. Though without access to Knuth's program, I can't draw a sensible conclusion from the following data.

      term% wc /sys/src/cmd/^(tr.c sort.c uniq.c sed.c)
          356     998    5993 /sys/src/cmd/tr.c
         1752    4526   28371 /sys/src/cmd/sort.c
          165     346    2185 /sys/src/cmd/uniq.c
         1455    4267   26848 /sys/src/cmd/sed.c
         3728   10137   63397 total

      term% wc <{man tr} <{man sort} <{man uniq} <{man sed}
           54     269    2028 /fd/8
          137     741    5542 /fd/7
           41     149    1225 /fd/6
          208    1078    8464 /fd/5
          440    2237   17259 total

</comment></comment><comment><USER24> Isn't there a middle ground where you can remain in the spirit of literate programming but still use libraries? It seems like that would be the best way . That seems like it would handle the biggest critique of the article unless I'm missing the point.

<comment><USER25> Yes, one can write libraries in literate style, and I have done so (<http://monday.sourceforge.net/>) in the past. However, from reading interviews with Knuth it seems that he does not find reusable software very interesting in itself.

</comment></comment><comment><USER26> I hate to nitpick, but if the explanation is being held up as wonderfully clear, must note that there's a typo:-

'Make one-word lines by transliterating the complement (-c) of the alphabet into newlines (note the quoted newline), and squeezing out (-a) multiple newlines.'

Surely should be 'squeezing out (-s)'? Unless I'm missing something?

<comment><USER12> The comments on the blog suggest that the typo is the blog author's, not McIlroy's.

<comment><USER26> Sorry probably wasn't clear - that's what I meant :). If he's holding it up as the paragon of clarity, he ought to reproduce it clearly.

It's a nit I know, but it confused me when I wanted to run through it and understand what was going on :)

</comment></comment></comment><comment><USER27> An interesting exercise would be to actually maintain both versions: e.g. make utf-8 friendly; add error handling. I think McIlroy's version would still come out on top but not by as much. Nice, informative error handling is one of the things most lacking in shell scripts.

<comment><USER28> It was fixed in Plan9, where programs return error strings, not numbers, the shell (rc) is better and errors are not lost in pipelines.

And UTF-8 was invented by Ken Thompson and Rob Pike while working on Plan9.

<comment><USER27> Yes, I'm aware of plan9. The 'what constitutes a word' and the error handling were and are the more interesting improvements.

</comment></comment></comment><comment><USER29> Being a PHP programmer, I struggle to espouse such programming practices.

For example, I copy my database class into each new project :(

How can I approach PHP development in a McIlroy manner?

Part of the problem is not understanding how to incorporate library repositories into my specific project repo.

</comment><comment><USER30> Imagine if the shell script had been written in a literate way? :)

</comment><comment><USER31> Try hyphenating a text with "simple pipeline".

<comment><USER23>
      pic challenge-accepted.ms | tbl | eqn | troff -ms

<comment><USER31> You're missing the point (intentionally, I suspect).

<comment><USER23> Your point isn't clear to me.

<comment><USER31> Once a problem becomes a non-toy problem, pipelines lose. troff(1) will do all the hyphenation (C or C++ code) in your code. Ironically troff is like TeX, it processes documents.

<comment><USER23> Of course troff hyphenates. Its job in the pipeline is to hyphenate, compute layouts, and output input for dpost. If your argument is that decoupling hyphenation from layout is difficult, I believe you are wrong. Splitting the hyphenator into a separate program is straightforward enough. Consider the text formatter

      hyphen < input | layout | topostscript

where hyphen inserts all possible hyphens, and layout discards those which are unnecessary.</comment></comment></comment></comment></comment></comment>


Trump executive order stresses skill over college degree in hiring - <AUTHOR> https://www.usatoday.com/story/news/politics/2020/06/26/trump-executive-order-stresses-skill-over-college-degree-hiring/3263074001/# <comment><USER2> > Ivanka Trump is co-chair of the American Workforce Policy Advisory Board, > which was created in 2018 and tasked with recommending ways to improve job > training.

Can someone explain why the presidents daughter is in such a position?</comment>


Show HN: Open Door, an appointment based marketplace for experts and influencers - <AUTHOR> https://opendoorconnect.com/ <comment><USER2> Congrats on Shipping!

Would be nice if you list existing experts in some sort of "Browse Influencers" page.

Also, do you vet or moderate the self-proclaimed "experts" / "influencers"? Otherwise, you risk quickly becoming the next Chatroulette.. (the chatroulette reference is cos you allow videos ) => [http://fskrealityguide.blogspot.com/2010/07/why- chatroulette...](http://fskrealityguide.blogspot.com/2010/07/why-chatroulette- failed.html)</comment>


Does Stimulus Spending Work? - <AUTHOR> http://www.learnliberty.org/videos/does-stimulus-spending-work

<comment><USER2> So according to his own arguments (which I don't accept btw @), he has just proven that increasing government spending does no damage to the economy? So we can now choose to spend more or less with no whining about the economy?

In that case, I choose more hospitals, libraries, healthcare and a social security net.

@ In any case, I think this is far too simplified due to:

a) Lack of a control group/proper statistical analysis/elimination of other factors.

b) No consideration of what that money is spent upon/ common factors / types of spending. His own graph shows massive variation. That is what he should be investigating. What actually makes the variation shown in the graph? I suspect it's down to types of spending. To illustrate my own bias, I'm happy to believe that giving money to the poor boosts the economy. Giving it to bankers, maybe less so.

c) No consideration of other benefits of spending. At the end, he says the only thing you have to show for spending is debt. If you spend it on infrastructure, you have roads, hospitals, etc, that you wouldn't have otherwise.</comment>


Unknown human relative discovered in Philippine cave - <AUTHOR> https://www.nature.com/articles/d41586-019-01152-3 <comment><USER2> Dupe: [https://news.ycombinator.com/item?id=19628128](https://news.ycombinator.com/item?id=19628128)</comment>


Facebook Gives Workers a Chatbot to Appease That Prying Uncle - <AUTHOR> https://www.nytimes.com/2019/12/02/technology/facebook-chatbot-workers.html <comment><USER2> This bot is an inside-company joke right? I can't believe that they'd spend time building this rather than just trusting their very intelligent employees to be able to answer those hard questions themselves.

If these employees can't answer these questions adequately, there are several implications, but I don't think the response of the company should ever seriously be "Just give them a bot that spits out PR-department talking points".

Weird.</comment>


Public Keys in Perl 6 - <AUTHOR> https://www.promptworks.com/blog/public-keys-in-perl-6 <comment><USER2> In case anyone was wondering, don't use this to do your encryption. It's an (interesting) intellectual exercise, nothing more.</comment>


Ask HN: Running startup with wife? - <AUTHOR> Do you know any successful examples of running startup with wife :) <comment><USER2> The book "Founder's Dilemma" has some case studies around this, if I remember correctly.

summary : Be cautious as hell.

</comment><comment><USER3> Y Combinator

<comment><AUTHOR> heh, you are right!</comment></comment>


Ask HN: Your favourite programming language dissertation - <AUTHOR>

Looking for a few high quality programming language dissertations. Thanks. :-) <comment><USER2> "Recursive Functions of Symbolic Expressions and Their Computation by Machine" <http://dspace.mit.edu/handle/1721.1/6096>

The lambda papers <http://library.readscheme.org/page1.html>

<comment><AUTHOR> Thanks. :-)</comment></comment>


China Sets Its Sights on Taiwan - <AUTHOR> http://bigthink.com/strange-maps/the-map-in-the-china-shop <comment><USER2> I don't think Taiwan is opposed to unification, I just don't think the mainland is ready to submit to Taiwanese rule. ;-)</comment>


C Corporation vs S Corporation vs LLC - <AUTHOR> http://www.themoneyalert.com/Corp-Entity-Table.html <comment><USER2> The advantage to going as far as an LLC is that LLCs can be invoiced and can cut PO's. It's also the bare minimum you need if you have cofounders.

Buried in this table is the major difference between LLC, S, and C:

* Any team member who has equity in an LLC is a member, a.k.a. partner, and members can't take W2 wages, only distributions.

* Conversely, employees of LLCs can't technically hold equity.

* Any external investor in an LLC (probably) must be an SEC "accredited investor", meaning that they have $1MM in the bank or several years of income over $200k.

* The major difference between S and C seems to be that C corps can have preferred (ie, VC) stock.

The long and the short of it is that equity is a mess until you're a C corp, and then everything else becomes a mess.

<comment><USER3> whats a PO?

<comment><USER4> purchase order

<http://en.wikipedia.org/wiki/Purchase_order>

</comment></comment></comment><comment><USER5> We just went through this decision process. Ended up doing S-Corp for the following reasons:

LLC vs S-Corp:

1\. Membership interest vs. Stock: Shares of stock are easier to deal with than membership interests, from an investor and giving to employee perspective.

2\. Taxes: Every dollar you take out of LLC is salary (and subject to payroll taxes). In an S-Corp, any money you take out of the business after paying yourself a "normal" salary is a distribution and free of payroll tax.

3\. The costs to set-up are nearly the same if you're able to create the articles of incorporation, corporate by-laws, and initial meeting minutes yourself (required for S-Corp). Not that difficult with Google and a $17 book from Barnes & Noble.

4\. Conversion to C-Corp if we raised VC money: Much easier to convert from S-Corp.

C-Corp vs. S-Corp:

1\. Losses: We expect to incur losses in year 1. In a C-Corp, those can be carried forward to subsequent years in which you have a gain, but in an S-Corp, you can take them on your personal return in that year.

2\. Double-taxation: If you only plan to pay a salary, then there is no double taxation, because that's deductible from the C-Corp, however, if you take a distribution, that would be taxed at both the corporate and personal level with the C-Corp (only at the personal level in S-Corp).

3\. Investors: Will have to convert to C if we raised VC money, but that's not the plan in the near term, and its easy to switch if we need to.

4\. A potential downside to the S-Corp for taxes is that the profits will be taxed at your marginal tax rate (because its all incremental income) as opposed to going through the levels of the progressive tax structure as it would in a C corp. If you only plan to pay yourself a normal salary, then C could be the way to go depending on how much you expect to make.

<comment><USER6> "2. Taxes: Every dollar you take out of LLC is salary (and subject to payroll taxes). In an S-Corp, any money you take out of the business after paying yourself a "normal" salary is a distribution and free of payroll tax."

\--> That one is HUGE in favor of S-Corp over LLC. That's 7% on all income over whatever you can justify as your base salary. If a company is going to be small-ish and profitable for a while, S-Corp is worth a few thousand dollars each year at the cost of added paperwork. LLC has more flexibility, but S-Corp can do all the same things if you don't mind filling out more documents and maybe thinking about it a bit more.

<comment><USER2> No, that's 7% on all income over what the IRS will justify as your base salary. =)

</comment></comment></comment><comment><USER7> S-Corps cannot have non-US residents.

That forced us into an LLC (and since, into a C-corp).

I'd suggest you talk to a lawyer. 90% of them will be able to cut you a deal on a simple C-corp.

</comment><comment><USER8> What do ycombinator startups incorporate as?

<comment><USER9> Delaware C corporations.

</comment></comment><comment><USER10> Has anyone switched from a C to S corp (yes, you read that right)? How painful was it?

Here is the situation: an existing small, moderately profitable company has decided to stay out of VC circuit, stay private, run it as a life-style business. C corp involves double taxation, which S corp avoids. They had grander dreams once (hence C corp) but now the situation has changed.

</comment><comment><AUTHOR> Newb question -- I don't really get the asset protection part of limited liability.

Say I sell my house to fund my business, then my LLC flops and I have to declare bankruptcy. I still lost my house, no?

Now say I take a loan out for $70K to fund my LLC and it flops and I have to declare bankruptcy. Then do I get to keep my house?

What's the difference?

<comment><USER11> If your LLC flops (or your C corp or Sub-S corp) you don't have to declare bankruptcy provided you haven't pledged your personal assets to the company or it's creditors. Your LLC/C/Sub-S creditors can't come after your house or other assets unless you have personally pledged them at some point.

<comment><AUTHOR> I'm still missing something. Clarified hypothetical situation:

Situation 1: I have a house worth $250K. I sell it and use $100K to fund my business. My business flops; I'm out the $100K.

Situation 2: I keep my $250K house and get a loan from my bank for $100K. My business flops; creditors can't come after my house because I'm behind the LLC firewall, right?

Why would anyone ever choose situation 1?

<comment><USER11> In Situation 2 you have pledged your house against the 100K loan, the fact that an LLC is involved is immaterial. There is nothing that can protect you when you want to take equity out of your house and you lose all of the money.

Situation 3, you borrow money to start your business, it takes off, you pay it back and your business continues to grow. Later you are sued and lose a 10M lawsuit wiping out all of the assets in your business and then some. If you have a corporation your personal assets are normally protected from this loss. Alternatively your large and growing business is struck by a meteorite (which can come in many forms) that wipes out all of your assets but leaves you with the liabilities. If you are incorporated winding down the business normally doesn't affect your personal assets.

<comment><AUTHOR> Good post, thanks.

(I know you're not a lawyer, but am trying to get an impression for this). Say instead of $10M I get sued for $50K. In the $10M case I clearly can't pay that but in the $50K case, I might be able to sell my car, move into a cheaper house, etc, in order to pay it. Am I protected in both cases? What about $10K? At what point is it "OK, you can't pay this, go declare bankruptcy so you get to keep your personal assets"?

I think I need to hit the books...

<comment><USER11> If you lose a lawsuit (or are hit by some flavor of meteor) the business has to pay for it to be able to continue. All that creating a corporation does is to clearly separate what are personal assets from corporate assets.

It may be easier to visualize if you imagine there are several shareholders. Each would want it clear what were the company's assets and what were their personal assets (in addition to their share of the company).

Look at it this way, you buy 100 shares of GM which subsequently goes bankrupt. Your shares may be worthless but GM's creditors cannot come after you as a shareholder for more than the equity that you contributed (the money you paid for your now worthless shares).

</comment></comment></comment></comment></comment></comment><comment><USER12> Interesting- in Missouri, an LLC can choose to be taxed as a Proprietorship or a Corporation. I don't know how other states do it.

</comment><comment><USER13> FYI: Massachusetts is the only jurisdiction in the US (that I am aware of) that requires at least 2 members for an LLC.

<comment><AUTHOR> "Contrary to what you may have read just a few years ago, you can now form an LLC with just one person in any state (or the District of Columbia)" - Pg 16, Nolo's Quick LLC

I happen to have read this last night :)

</comment></comment><comment><USER14> overall it's a useful chart... I just didn't like the liability part for LLCs. To me (a lawyer could prove me wrong - I'm not a lawyer), LLC (and I think S corp) members have the same liability as a proprietorship if they are company operators (ie not just a passive investor).

</comment><comment><USER15> Which one do most startups choose? LLC?

<comment><USER2> LLC, though you won't take outside funding without switching to a C corp.

<comment><USER16> How hard is it to switch?

<comment><USER2> From LLC to S/C? Annoying.

From S to C? Very easy.</comment></comment></comment></comment>


Show HN: Find it for me (Search engine for Facebook pages) - <AUTHOR> https://anikdas.github.io/finditforme/ <comment><AUTHOR> Before searching on Facebook pages were allowed, I wrote this tiny (dirty) JavaScript code to search posts through Facebook pages. Few things are still broken but keyword search works!</comment>


Redditor learns hard truths about machine learning - <AUTHOR> https://www.reddit.com/r/programmerhumor/comments/axi87h/_/ehtzp34 <comment><USER2> Wow. Reminds of this article from a year ago:

[https://petewarden.com/2018/03/19/the-machine-learning- repro...](https://petewarden.com/2018/03/19/the-machine-learning- reproducibility-crisis/)

 _In many real-world cases, the researcher won't have made notes or remember exactly what she did, so even she won't be able to reproduce the model. Even if she can, the frameworks the model code depend on can change over time, sometimes radically, so she'd need to also snapshot the whole system she was using to ensure that things work. I've found ML researchers to be incredibly generous with their time when I've contacted them for help reproducing model results, but it's often months-long task even with assistance from the original author._

 _Why does this all matter? I've had several friends contact me about their struggles reproducing published models as baselines for their own papers. If they can't get the same accuracy that the original authors did, how can they tell if their new approach is an improvement? It's also clearly concerning to rely on models in production systems if you don't have a way of rebuilding them to cope with changed requirements or platforms._

</comment><comment><USER3> Here's my machine learning "hard truth" I posted on reddit:

[https://www.reddit.com/r/ProgrammerHumor/comments/92s0du/me_...](https://www.reddit.com/r/ProgrammerHumor/comments/92s0du/me_trying_to_learn_machine_learning_after_being_a/)

Normally I wouldn't post a meme here, but I think it goes well with this person's experience.</comment>


The Visual ARM1 - <AUTHOR> http://blog.visual6502.org/2015/11/the-visual-arm1.html <comment><USER2> This is probably really amazing if you know what you're looking at. I have no idea what any of the parts do and what code is being run.

I would love to see someone give a "guided tour" though all the parts of the CPU, and step through some assembly to see how the things works.

I can see from the picture there are distinct areas, and large arrays of gates that might be memory or adders or I don;t know. It would be amazing to see an add interaction actually wire up some memory to some gate and see the result propagate through the CPU.

<comment><USER3> You could try to work through "Code, the hidden language" [http://www.charlespetzold.com/code/](http://www.charlespetzold.com/code/)

EDIT: The Art of Electronics has an accompanying book called the Student Manual, which will walk you through the process of (eventually) building a small computer.

</comment></comment><comment><USER4> slightly off topic, but...

> Credit goes to ... Lee Smith for spotting the variable record format used to > encode the file (an artifact of the VMS on Acorn's VAX that at first > appeared to be widespread corruption of the file),

These days, thanks to Unix and C I suppose, a file is just a collection of bytes. That wasn't always the case. Consider Pascal, where a file is a file of something (records, integers, etc).

So, on VMS, you had streaming files (just a collection of bytes) but also RMS -- Record Management System -- files, which were files of records. There were fixed length records (the record size was probably stored in the file metadata) and variable size records (record size was stored inline, as a byte or word prefix before the record). You might think you're just writing data to a file, but you're writing records (the OS adds the prefix for variable records or 0-pads for fixed length). Then when reading, you might request xx bytes but it will only read one record at a time. (Kind of like how fgets() will stop reading when it finds a newline or a tcp socket will stop reading when it finds a PUSH.)

There are advantages and disadvantages to this but if you're dealing with a VMS RMS file, you need to be aware or it will look corrupted.

</comment><comment><USER5> The arm1 was designed on paper by people at Acorn whod never made a chip before I believe? Turned out pretty good.

<comment><USER6> nah, I remember both Wilson and Furber mentioning buying Apollos for CAD.

</comment></comment><comment><USER7> What code is it running? When I click play, it seems to be in a tight loop from code at addresses 24-34.

Maybe it's simulating an entire OS run - if I leave it on for a few years it will finish zeroing the RAM and start booting :)

</comment><comment><USER8> I'm amazed at how small and (relatively) simple it all looks, that you can literally see the registers on that picture. Just 25K transistors for a whole 32-bit CPU with a 3-stage pipeline.

For those who would like a bit more detail about the feature of this CPU: [http://everything2.com/title/ARM1](http://everything2.com/title/ARM1)

One thing I was curious to know is whether the ARM1 had a hardware multiply instruction, whether they could fit that in this tiny CPU. It does not.

<comment><USER9> ARM2 had hardware multiply & more shadow registers and came in at 30k transistors according to Wikipedia. Everything2 thinks the ARM2 came in at just over 25k transistors though, with the ARM1 using less than 25k.

</comment></comment><comment><USER10> Would be nice if somebody with the knowledge could do some kind of guided tour where they would go over some of the structures what they are and what they do. For me as a software developer with no knowledge of electronics it looks nice but thats about it.

<comment><USER9> So the ARM1 had no cache & no microcode. 15 user visible 32 bit registers, with some (Nos 10-15 IIRC) duplicated for use during interrupts (FIQ mode). I think a couple more registers were also duplicated for use during other interrupts. So bottom left looks like it might be the register bank. The square block at top right might be the instruction decode PLA?

There's a block diagram for the ARM2 in this document: [http://www.riscos.com/support/developers/asm/cpu.html](http://www.riscos.com/support/developers/asm/cpu.html) I believe the ARM was much the same, with the exception of fewer banked registers & no integer multiplier, so the dedicated could probably trace the paths and make some guesses about where each block is located.

<comment><USER11> Yes, I think bottom left is the register bank -- you can see it is 25 elements horizontally by 32 vertically, and there are 25 total registers listed on the right (r0..r15 plus the svc/irq/fiq mode-banked copies of some of r10 through r14). Also I'm pretty sure the program counter (r15) is the far-left-hand- side, because that first row does not have cells in the bottom two bit positions or the top six (ARM1 used 26-bit addresses, hence the missing top six, and PC values are always 4-aligned, so no bottom two bits).

</comment></comment></comment><comment><USER12> I hope they tear apart the VIC-II chip next time.

I know it seems like the state-machines of VIC-II are fully documented, but perhaps there could be even more tricks undiscovered?</comment>